Tuesday, 2014-01-21

*** e0ne has quit IRC00:00
*** e0ne has joined #tripleo00:00
*** e0ne has quit IRC00:05
*** morazi has quit IRC00:18
*** edmund has quit IRC00:20
*** bauzas has quit IRC00:39
*** UtahDave has quit IRC01:04
StevenKlifeless: http://paste.openstack.org/show/61604/01:10
cody-somervillemordred: fyi, I have your laptop's charger.01:12
mikalcody-somerville: its ok, mordred doesn't have the laptop any more01:12
cody-somervilleIt's for his HP laptop01:12
cody-somervillethe one I think he's using now ;)01:12
cody-somervilleI'll give it back to you in Brussels (along with a short blue Ethernet cable that belongs in Sunnyvale).01:13
lifelessStevenK: roughly yes, but copy to the chosen output please, not /tmp/predictablepath01:14
lifelesse.g. if I pass -o foo give me foo.qcow2, foo.kernel, foo.ramdisk01:15
StevenKlifeless: Pass -o to a cleanup.d how?01:15
StevenKlifeless: And I'm guessing you don't want it under $TARGET_ROOT either01:15
lifelessits coming *from* target_root :)01:17
lifelessyou have the variable in the disk-image-create script already, and it's exported, I think inherited by01:17
StevenKRight, $IMAGE_NAME01:18
openstackgerritA change was merged to openstack-infra/tripleo-ci: Switch test environment users  https://review.openstack.org/6761401:20
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Add unstyled overcloud resource category page  https://review.openstack.org/6763201:29
*** greghaynes has quit IRC01:40
*** xuhanp_ has joined #tripleo01:41
*** weshay has joined #tripleo01:43
*** coolsvap has quit IRC01:51
*** e0ne has joined #tripleo01:57
*** nosnos has joined #tripleo01:59
*** jrist has quit IRC02:01
*** e0ne has quit IRC02:02
lifeless/dev/nbd15p1    1.4G  1.1G  273M  80% /tmp/openstack-vfs-localfsAvkjEz02:16
lifelessmethinks we haven't turned file injection off properly :)02:16
mordredlifeless: config-drive?02:19
lifelessshouldn't need nbd for config drive02:20
lifelessand it contains the full fs when I list it02:20
lifelessand look02:25
lifelesssuddently things boot about 5 billion times faster02:25
*** vkozhukalov has joined #tripleo02:34
*** sdake has quit IRC02:49
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Add unstyled overcloud resource category page  https://review.openstack.org/6763202:52
*** e0ne has joined #tripleo02:57
*** e0ne has quit IRC03:02
*** weshay has quit IRC03:05
*** ramishra has joined #tripleo03:06
*** ramishra_ has joined #tripleo03:08
*** ramishra has quit IRC03:11
*** greghaynes has joined #tripleo03:40
*** e0ne has joined #tripleo03:57
*** marun has joined #tripleo04:00
*** e0ne has quit IRC04:02
*** marun has quit IRC04:12
*** marun has joined #tripleo04:13
*** marun has quit IRC04:14
StevenKlifeless: http://paste.openstack.org/show/61605/04:14
SpamapS2014-01-21 04:12:51.987 2497 TRACE heat.openstack.common.rpc.common error: [Errno 104] Connection reset by peer04:16
SpamapSseeing errors for rabbitmq RPC04:16
*** marun has joined #tripleo04:16
SpamapSthat is just early boot crapola04:17
SpamapShm cfn reqs to seed are failing auth04:24
SpamapS2014-01-21 04:24:24.149 12222 WARNING keystone.openstack.common.db.sqlalchemy.session [-] This application has not enabled MySQL traditional mode, which means silent data corruption may occur. Please encourage the application developers to enable this mode.04:24
SpamapSthat is a LOVELY warning04:24
openstackgerritSteve Kowalik proposed a change to openstack/diskimage-builder: Create a new baremetal element  https://review.openstack.org/6801904:31
openstackgerritSteve Kowalik proposed a change to openstack/diskimage-builder: Create a new baremetal element  https://review.openstack.org/6801904:33
*** coolsvap has joined #tripleo04:38
*** ramishra_ has quit IRC04:40
*** ramishra has joined #tripleo04:55
*** e0ne has joined #tripleo04:57
*** e0ne has quit IRC05:01
*** marun has quit IRC05:08
StevenKlifeless: Thanks for the review. In terms of -get-kernel, you think a comment, or a echo with "DEPRECATED: Please consider using the `baremetal` element." ?05:33
lifelessStevenK: deprecate that shit :)05:34
*** rpodolyaka1 has joined #tripleo05:35
openstackgerritSteve Kowalik proposed a change to openstack/diskimage-builder: Create a new baremetal element  https://review.openstack.org/6801905:42
openstackgerritSteve Kowalik proposed a change to openstack/diskimage-builder: Create a new baremetal element  https://review.openstack.org/6801905:48
*** e0ne has joined #tripleo05:57
*** akuznetsov has quit IRC05:59
*** e0ne has quit IRC06:02
*** tzumainn has quit IRC06:07
*** kpavel has joined #tripleo06:22
*** akuznetsov has joined #tripleo06:26
*** akuznetsov has quit IRC06:37
*** akuznetsov has joined #tripleo06:37
*** ramishra has quit IRC06:38
*** mrunge has joined #tripleo06:43
*** mrunge has quit IRC06:44
*** mrunge has joined #tripleo06:44
*** vkozhukalov has quit IRC06:47
*** e0ne has joined #tripleo06:57
*** e0ne has quit IRC07:02
*** e0ne has joined #tripleo07:04
*** akuznets_ has joined #tripleo07:05
*** mrunge has quit IRC07:07
*** mrunge has joined #tripleo07:08
*** akuznetsov has quit IRC07:08
*** rdopieralski has joined #tripleo07:30
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Deployment design form  https://review.openstack.org/6715507:32
*** rlandy has joined #tripleo07:37
*** ifarkas has joined #tripleo07:41
*** rpodolyaka1 has quit IRC07:43
lifelessback for evening07:45
*** jprovazn has joined #tripleo07:52
*** jtomasek has joined #tripleo07:56
*** sdake has joined #tripleo07:58
*** pblaho has joined #tripleo08:03
*** bauzas has joined #tripleo08:08
*** e0ne has quit IRC08:09
GheRiveromorning TripleO!08:10
*** jcoufal has joined #tripleo08:13
*** jistr has joined #tripleo08:14
rlandyifarkas: hello - building the undercloud with tuskar went well but I deploying the undercloud failed with a WaitConditionTimeout. Should I have added anything additional to the "heat stack-create" command?08:14
rlandyside note: devtest was undergoing some changes last night (possible that the error is unrelated to tuskar addition)08:15
rlandyrebuilding from scratch this morning08:15
ifarkasrlandy, not that I know of.08:15
ifarkasrlandy, can you find anything useful in the logs?08:16
rlandyother than the timeout error - looked like authentication problem08:17
rlandywhich is why I was wondering if I needed to pass another password parameter for the extra element08:18
rpodolyakamorning all08:22
*** vkozhukalov has joined #tripleo08:28
ifarkasmorning rpodolyaka!08:36
jprovaznrlandy: can you ssh to undercloud?08:38
openstackgerritRalf Haferkamp proposed a change to openstack/diskimage-builder: Setup ccache symlinks on openSUSE  https://review.openstack.org/6796508:43
rlandyjprovazn: not anymore - I killed it to start again :)08:45
rlandyto answer your question - I could ping the undercloud08:45
rlandyissuing the keystone commands -  failed with could not connect08:46
jprovaznrlandy: I think wait condition is not matched until undercloud boots and openstack services are running on it, so it might be 1) your VM failed to boot or 2) after boot setup (the whole os-collect-config process) failed. Option 2) is more probable08:47
rlandyjprovazn: makes sense (nova showed the vm as running - heat stack-list showed it as CREATE_FAILED). I am doing the whole devtest process with tuskar in the undercloud again. If the failure is repeatable, we can inspect the machine/logs08:51
jprovaznrlandy: OK, you can use '-c' to use cached images to make it much more quicker08:52
*** boris-42 has quit IRC09:00
*** derekh has joined #tripleo09:00
*** boris-42 has joined #tripleo09:00
openstackgerritlifeless proposed a change to openstack-infra/tripleo-ci: Set TRIPLEO_ROOT  https://review.openstack.org/6804509:17
lifelessderekh: ^ is what you need I think09:18
derekhlifeless: yup, probably and it would explain why I did this https://review.openstack.org/#/c/67939/3/toci_devtest.sh09:20
*** markmc has joined #tripleo09:20
derekhlifeless: will abandon mine09:20
ifarkasI tried to deploy the undercloud but it failed for me. It seems that os-config-refresh couldn't fetch heat metadata from seed. Here's the log: http://pastebin.com/wthMqvns09:21
ifarkasAnyone has any idea what could be the problem?09:21
derekhifarkas: I got no idea, gonna run it locally to see if I can reproduce09:23
lifelessderekh: I've reviewed all your toci patches09:23
lifelessderekh: if you can fix the stack up (basically delete two patches and tweak the heat template) we can land it all09:24
ifarkasderekh, thanks, let me know if it works for you09:24
lifelessifarkas: whats that log from ?09:24
derekhlifeless: ok, on it now09:24
lifelessifarkas: it makes little sense to me :)09:24
ifarkaslifeless, sorry, the undercloud log is the output of os-collect-config -d and seed log is from journalctl09:25
rpodolyakaifarkas: I think, I'm seeing this too09:26
ifarkasrpodolyaka, cool, at least I am not the only one :-)09:26
rpodolyakathough I have no idea why this is happening :(09:27
lifelessoh, ok09:27
lifelessso that thats a heat service auth failure09:27
lifelesswith presigned request09:28
lifelessifarkas: check the heat-api log09:28
*** viktors has joined #tripleo09:28
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Fixup the handling of ssh keys  https://review.openstack.org/6793809:30
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Fix the server name  https://review.openstack.org/6759909:30
ifarkaslifeless, alright, I am on it09:30
derekhlifeless: done ^, heat template tweaked, the tweak is different to the one you suggested but I think its right09:32
ifarkaslifeless, there's nothing in the log on the seed vm from heat if I run os-collect-config on the undercloud machine09:33
rpodolyakaifarkas: http://paste.openstack.org/show/61611/ , sounds familiar?09:33
derekhlifeless: at least its what I used to create the testenv stack last night09:33
lifelessNg: https://review.openstack.org/#/c/67599/ please09:37
ifarkasrpodolyaka, yeah, I have the same09:37
*** athomas has joined #tripleo09:37
Nglifeless: sure09:37
openstackgerritDirk Mueller proposed a change to openstack/diskimage-builder: Add a service mapping for openSUSE  https://review.openstack.org/6805109:37
ifarkaslifeless, I've just set heat to debug and I have the same error in the log as rpodolyaka09:37
lifelessifarkas: rpodolyaka: I thinkDate:   Tue Jan 21 08:36:17 2014 +000009:40
lifeless    Merge "Use oslo crypto"09:40
lifelesslooks suspect to me09:40
lifelessI think try running without that patch - e.g. use 834eb42e2e1aa0048a10708b5a598204da98ec1b as your commitish for heat09:41
rpodolyakalifeless: thanks! will try09:42
lifelessfailing that, keystone becomes the culprit. looking in it's log for clues09:42
ifarkaslifeless, yeah, thanks, I will test it too09:42
lifeless    Merge "Replacing python-oauth2 by oauthlib"09:44
lifelessmight do it09:44
rpodolyakayeah, so actually keystone error is http://paste.openstack.org/show/61612/09:44
rpodolyakaand the only place where this exception is raised in keystone is here https://github.com/openstack/keystone/blob/master/keystone/contrib/ec2/controllers.py#L11909:47
lifelesswhich is the oauthlib patch I just mentioned09:47
lifelessso yeah, I'd back that patch out, restart keystone, heat stack-delete heat stack-create undercloud....09:48
rpodolyakalifeless: thanks for pointing this out!09:48
lifelessrpodolyaka: I'm just free associating with git logs :)09:49
rpodolyakasure, but I'm struggling to understrand how this is related to oauth :)09:49
* rpodolyaka was sure ec2/amazon used custom auth09:49
lifelessit looks like heat grabs a token then uses it to sign a request and hands that request (but not the token) into the metadata09:50
lifelessNg: https://review.openstack.org/#/c/67868/ and all its dependents09:51
Nglifeless: also sure. the previous stack is all Approved, whenever Jenkins decides to stop drinking beer and do some bending09:51
StevenKIt's Zuul, not Jenkins? :-)09:52
openstackgerritJiri Tomasek proposed a change to openstack/tuskar-ui: Number picker input widget and angular directive  https://review.openstack.org/6805309:53
*** e0ne has joined #tripleo09:53
NgStevenK: all these robots look the same to me ;)09:54
* StevenK confuses Ng by adding gearman into the mix.09:54
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: SSL support for openstack services.  https://review.openstack.org/6734309:55
StevenKI see Adobe are taking leaves from Amazon's book. "Thank you and enjoy Flash Player. Here is another product that might interest you."09:56
*** kwwii_ has joined #tripleo09:59
*** kwwii_ has left #tripleo10:00
*** kpavel has quit IRC10:00
*** martyntaylor has joined #tripleo10:07
openstackgerritA change was merged to openstack/diskimage-builder: Create a new baremetal element  https://review.openstack.org/6801910:07
openstackgerritJiri Tomasek proposed a change to openstack/tuskar-ui: Number picker input widget and angular directive  https://review.openstack.org/6805310:08
*** coolsvap has quit IRC10:09
*** coolsvap has joined #tripleo10:12
openstackgerritJiri Tomasek proposed a change to openstack/tuskar-ui: Number picker input widget and angular directive  https://review.openstack.org/6805310:13
Nglifeless: ok, both stacks approved up and down10:21
lifelessNg: might want to have a look for ancilliary fixes10:22
*** coolsvap has quit IRC10:23
*** coolsvap has joined #tripleo10:27
*** xuhanp_ has quit IRC10:29
*** martyntaylor has quit IRC10:35
*** akrivoka has joined #tripleo10:37
*** boris-42 has quit IRC10:38
*** ramishra has joined #tripleo10:49
*** martyntaylor has joined #tripleo10:51
openstackgerritA change was merged to openstack-infra/tripleo-ci: Add more detail to the json file  https://review.openstack.org/6768610:56
openstackgerritA change was merged to openstack-infra/tripleo-ci: Fixup the handling of ssh keys  https://review.openstack.org/6793810:56
openstackgerritA change was merged to openstack-infra/tripleo-ci: Fix the server name  https://review.openstack.org/6759910:56
*** max_lobur_afk is now known as max_lobur11:05
openstackgerritRonelle Landy proposed a change to openstack/tripleo-image-elements: Moving iptables rules to the relevant elements  https://review.openstack.org/6807511:08
rlandyderekh: ^^ https://review.openstack.org/68075 (marked workinprogress)11:09
derekhrlandy: cool, I'll try it out later today11:11
*** akrivoka has quit IRC11:14
*** e0ne has quit IRC11:15
*** e0ne has joined #tripleo11:15
rlandyrpodolyaka: ifarkas: are you able to deploy the undercloud now?11:17
lsmola_rpodolyaka: hello11:17
rpodolyakarlandy: unfortunately, no11:17
*** akrivoka has joined #tripleo11:17
lsmola_rdopieralski: i have the same problem11:17
lsmola_rlandy: nope11:17
rpodolyakarlandy: I tried to downgrade heat/keystone on seed vm but with no sucess11:17
*** jcoufal has quit IRC11:17
lsmola_rlandy: i have reverted keystone and heat, restart the services and it is not working11:18
lsmola_rlandy: any other ideas?11:18
rpodolyakalsmola_: no, looking through the keystone code trying to understand what is going on11:19
rpodolyakalsmola_: and how it is supposed to work11:19
rlandylsmola: rpodolyaka: can't say I have another bright idea right now ... I tripped over this last night doing a bug verification - so I wanted to check the standard deploy worked before I threw in the bug-related changes11:19
openstackgerritA change was merged to openstack/tuskar-ui: Add unstyled overcloud resource category page  https://review.openstack.org/6763211:19
lsmola_rdopieralski: ok11:20
* rdopieralski grins11:20
lsmola_rdopieralski: damn11:20
lsmola_rpodolyaka: I keep exchanging you with rdopieralski  :-)11:21
rlandydid anyone manage to deploy the undercloud yesterday?11:21
lsmola_rpodolyaka: ok, I trying to look what has changed on tripleo side11:21
rlandythis was my first attempt at devtest since last week - when it worked11:21
lsmola_rpodolyaka: I swear it was working for me on friday11:22
rlandyok - we got a check on Friday11:22
*** lucasagomes has joined #tripleo11:22
lsmola_rlandy: I think i was able to deploy it the whole last week, did it several times11:23
rlandyyeah - so do I11:23
rpodolyakayeah, worked for me last wee to11:23
rlandyok - so what happened on Friday night -> Monday?11:23
jprovaznsounds like a pretty good hangover :)11:29
lsmola_rlandy: can I test this with curl like this ? http://paste.openstack.org/show/61615/11:34
lsmola_rlandy: or I am missing something?11:34
rlandysec - on call ... brb11:34
rlandylsmola_:  just fyi ... I'm a newbie on the team and figuring this out as I go ... you may want to consult with others as well ;)11:35
lsmola_rlandy: :-) that sounds familiar to me11:36
*** martyntaylor has quit IRC11:42
openstackgerritA change was merged to openstack/tripleo-incubator: Upgrade setuptools to >=1 in a newly created virtualenv  https://review.openstack.org/6768811:42
openstackgerritA change was merged to openstack/tripleo-incubator: Read env-num from TE_DATAFILE  https://review.openstack.org/6768911:42
openstackgerritA change was merged to openstack/tripleo-incubator: Replace rsync with the copyseed command  https://review.openstack.org/6786711:42
openstackgerritA change was merged to openstack/tripleo-incubator: Replace usage of qemu+ssh with ssh <command>  https://review.openstack.org/6786811:42
*** jcoufal has joined #tripleo11:42
derekhlsmola_: rlandy so running tcpdump on the seed "tcpdump -i lo -A -nn port 5000" while running that curl command you get11:44
derekhlooks like the request is missing the auth token11:44
*** rlandy is now known as rlandy|brb11:45
derekhsurley there should be a X-Auth-Token heater in that request11:45
lsmola_derekh: hm, ok11:46
lsmola_derekh: so it is missing also in call that the os-colelct-config is doing11:46
lsmola_derekh: or it is being added there11:46
lsmola_derekh: ?11:47
lsmola_derekh: the log I posted is from os-collect-config -d11:47
derekhlsmola_: I *think* the call from os-collect-config to heat is ok (could be wrong), then heat tries to talk to keystone and isn't sending its auth token11:48
rpodolyakaseems so11:48
derekhlsmola_: not sure though, will have to dig a bit more, I'll let you knwo if I find anything out11:48
*** martyntaylor has joined #tripleo11:48
lsmola_derekh: hm, I am getting os_collect_config.cfn [-] 403 Client Error: AccessDenied , when running os-collect-config -d in my undercloud machine11:52
lsmola_derekh: i will try to tcpdump the call11:52
derekhlsmola_: yup, getting that here also11:53
lsmola_derekh: do you see AUTH token in the header?11:53
derekhlsmola_: nope, but don't think you should see it in that request11:57
lsmola_derekh: ok12:00
derekhunless of course I'm wrong :-)12:00
*** boris-42 has joined #tripleo12:05
lsmola_derekh: ok, so I have the same as you using os-collect-config -d http://paste.openstack.org/show/61617/12:05
lsmola_derekh: so the question is, what patch could broke the token passing12:06
*** morazi has joined #tripleo12:06
lsmola_derekh: cause it was working on friday12:06
derekhlsmola_: dunno, ok time to start sticking print statements in code12:10
derekhlsmola_: dunno, time to start stick print statments in code12:10
lsmola_derekh: ok12:12
derekhlsmola_: ok, looks like I'm going down the wrong path, here is the code that makes that request from heat to keystone12:20
derekhlookes like no attempt is made to add an auth token to the request and the file hasn't changed in 2 months12:21
lsmola_derekh: interesting12:21
lsmola_derekh: btw. it might be something on seed keystone side12:21
derekhlsmola_: yup, could be12:21
lsmola_derekh: cause it works for jprovazn with ovecloud <-> undercloud communication12:22
*** rlandy|brb is now known as rlandy12:22
lsmola_derekh: and he uses old undercloud12:22
rlandylsmola_: sorry  back ... catching up ^^12:22
*** coolsvap has quit IRC12:22
lsmola_derekh: so maybe Keystone ignored this till recently?12:23
rpodolyakahmm, is it ok that request to keystone is one on behalf of undercloud-User-ihb73467qruc?12:23
rpodolyaka*is done12:23
derekhrpodolyaka: so your saying that request to keystone looks ok?12:25
derekhso we gotta find out why keystone isn't allowing it12:25
rpodolyakaderekh: don't know, but it passes validation up to this point https://github.com/openstack/keystone/blob/master/keystone/contrib/ec2/controllers.py#L11912:25
rpodolyakaderekh: which makes me believe we have a problem with user/roles12:26
*** tserong has quit IRC12:27
lsmola_rpodolyaka: btw. I dont see this exception in my seed journal12:27
rpodolyakalsmola_:  hmm, this is what I see in keystone log on seed vm - http://paste.openstack.org/show/61618/12:28
rpodolyakalsmola_: and grep "User not valid for tenant" led me to https://github.com/openstack/keystone/blob/master/keystone/contrib/ec2/controllers.py#L11912:28
lsmola_rdopieralski: where do I find this log?12:29
*** tserong has joined #tripleo12:31
rpodolyakalsmola_: /var/log/upstart/keystone.log on ubuntu :)12:31
lsmola_rpodolyaka: hmm I don't see that one anywhere12:34
lsmola_rdopieralski: but I have reverted my keystone a way back to 05808a95e3058ea66dc1af1c09b0fdab3744215c12:36
lsmola_rpodolyaka: eh12:36
lsmola_rdopieralski: sorry :-D12:36
rpodolyakalsmola_: do you have debug=True in keystone?12:36
lsmola_rpodolyaka: nope12:36
lsmola_rpodolyaka: but that warning should be visible anyway right?12:36
rpodolyakalsmola_: I think so12:37
derekhI think on fedora we are leaking keystone logs into nothingness12:40
derekhI print statment confirms to me that this exception is being raised12:40
derekhonly if I start keystone on command line12:41
derekhsudo su -s /bin/bash -c '/opt/stack/venvs/keystone/bin/python /opt/stack/venvs/keystone/bin/keystone-all --config-dir /etc/keystone'12:41
lsmola_derekh: ok12:44
lsmola_derekh: should I ask why  "we are leaking keystone logs into nothingness" :-)12:45
rlandyjust recently?12:45
derekhlsmola_ rlandy don't know to both questions12:46
*** ramishra has quit IRC12:46
derekhok, got os-collect-config to run again with this12:47
derekhthe two ID's are my admin and Member roles12:48
derekhso we gotta find out why roles is empty12:48
rpodolyakaoh, nice :)12:49
rpodolyakahmm, is this user created by heat?12:50
derekhrpodolyaka: user is | c621ee19476046db85d8304885572d0e |       undercloud-User-ggcjcybrzoai       |   True  |                    |12:50
derekhso running this on the seed solved the problem12:55
derekhkeystone user-role-add --user undercloud-User-ggcjcybrzoai --tenant admin --role admin12:55
derekhkeystone user-role-add --user undercloud-User-ggcjcybrzoai --tenant admin --role Member12:55
derekhnot sure if both commands are needed12:55
rpodolyakajust did the same thing - works for me too12:56
rpodolyakabut the point is, isn't it heat who must do this?12:56
derekhrpodolyaka: yup, was just testing things out12:58
derekhso this did change recently12:58
derekhgoing to ask on the heat channel12:59
openstackgerritA change was merged to openstack/tripleo-heat-templates: Add license file  https://review.openstack.org/6780613:00
lsmola_derekh: great, that sounds promising13:01
rlandylsmola_: you rolled back keystone but not heat, right?13:01
lsmola_rlandy: I rolled heat, though not that far I assume13:02
lsmola_rlandy: checking13:02
rpodolyakaI rolled back heat, but didn't actually tried to do stack-delete/stack-create13:02
rlandyJan 19 doesn't seem that far away - but I guess there are a lot of commits per day13:03
derekhwhen you rolled back did you start from scratch again ?13:03
lsmola_rlandy: hmm, so I don't have it in my seed13:04
lsmola_rlandy: is it the right place?13:05
rpodolyakaderekh: nope, so my 'roll back' is not really demonstrative in case the commit you pointed is to blame13:05
lsmola_rdopieralski: eh, I didn't rolled back the client13:06
*** pblaho has quit IRC13:07
lsmola_rdopieralski: eh, again :-)13:07
lsmola_rpodolyaka: rlandy I didn't rolled back the client :-) trying13:07
rpodolyakalsmola_: me too13:08
lsmola_rpodolyaka: wait, the code is under heat13:09
*** Shrews_ has joined #tripleo13:10
rpodolyakalsmola_: yep :)13:10
lsmola_rdopieralski: so unless there is some magic, I did rolled it back, I dont see it in my git log13:11
lsmola_rdopieralski: come on, I have to cut my fingers of, and replace them with new ones13:11
*** Shrews has quit IRC13:12
*** Shrews_ is now known as Shrews13:12
lsmola_rpodolyaka: ^^ keep confusing you too :-D heeelp13:12
rpodolyakalsmola_: yeah, I just meant, I didn't do stack-delete/stack-create, so the use wasn't really recreated and this wasn't helpful in case of missing user roles13:12
*** Shrews_ has joined #tripleo13:13
lsmola_rpodolyaka: hm I did13:13
*** Shrews_ has quit IRC13:14
rlandyare you rolling back on the seed vm?13:14
*** Shrews_ has joined #tripleo13:15
*** Shrews has quit IRC13:16
*** Shrews_ has quit IRC13:16
lsmola_rlandy: yes13:16
lsmola_rlandy: then restarting 4 heat services and 1 keystone13:16
*** Shrews has joined #tripleo13:19
lsmola_derekh: so we should see the 2 roles with 'keystone user-role-list' on seed?13:25
derekhlsmola_: I got 6 roles13:26
lsmola_derekh: i have only one now13:26
derekhlsmola_: sorry, missunderstood13:26
derekhlsmola_: you should only see the heat_stack_user role for that user13:27
*** e0ne_ has joined #tripleo13:27
*** e0ne has quit IRC13:28
derekhlsmola_: just to confirm the code you roled back was in /opt/stack/venvs/heat13:29
*** nosnos has quit IRC13:30
*** jdob has joined #tripleo13:31
*** nosnos has joined #tripleo13:31
lsmola_derekh: ha13:31
lsmola_derekh: no, I rolled back the /opt/stack/heat13:32
lsmola_derekh: that could be the problem :-)13:32
derekhlsmola_: ok, that wouldn't do anything, that directory is just there to confuse people13:32
rpodolyakaI just installed it to venv using pip install -e :)13:32
lsmola_derekh: lol13:33
rpodolyakabut yes, we should probably remove those dirs13:33
derekhrpodolyaka: yup,13:33
lsmola_derekh: so how to I roll it back  in there?13:33
derekhrpodolyaka: although I do use them a lot13:34
rpodolyakaderekh: me too :)13:34
rpodolyakaderekh: for pip install -e and trying different commits easily13:35
derekhlsmola_: I usuaully do this13:35
derekh[root@localhost heat]# cd /opt/stack/venvs/heat/lib/python2.7/site-packages/13:35
derekh[root@localhost site-packages]# mv heat heat_13:35
derekh[root@localhost site-packages]# ln -s /opt/stack/heat/heat13:35
derekh[root@localhost site-packages]# cd !$13:35
derekhcd /opt/stack/heat/heat13:35
derekh[root@localhost heat]# git reset --hard <IDOFCOMMITIWANT>13:35
lsmola_derekh: ok13:36
*** nosnos has quit IRC13:38
*** martyntaylor has quit IRC13:40
derekhok, it may be only reproducable on tripleo only (or maybe only with latest keystone), going to try that minimal test that shardy linked to on #heat13:42
*** ramishra has joined #tripleo13:43
*** ramishra has quit IRC13:45
*** ramishra has joined #tripleo13:46
*** ramishra_ has joined #tripleo13:47
*** julim has joined #tripleo13:49
*** ramishra has quit IRC13:51
derekhshardy: so we start looking at keystone http://paste.openstack.org/show/61624/13:52
*** martyntaylor has joined #tripleo13:54
shardyderekh: Hmm, OK - just to confirm, the credentials you have sourced are for the admin tenant right?13:55
derekhshardy: yup13:55
shardyderekh: Ok, sounds like I need to pull latest keystone and try again13:55
shardyderekh: No clues in the keystone log?13:55
*** noslzzp has joined #tripleo13:56
derekhshardy: nope, just deprecated warnings13:58
*** ramishra_ has quit IRC13:59
shardyderekh: Hmm, I just pulled latest keystone and keystoneclient and it still works..13:59
* shardy is confused13:59
shardythere must be some other difference but I'm not sure what atm14:00
*** dprince has joined #tripleo14:00
*** lblanchard has joined #tripleo14:00
*** coolsvap has joined #tripleo14:00
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Move cinder iptables rules into scoped elements  https://review.openstack.org/6811814:01
shardyderekh: I'll build a fresh devstack VM and try there14:01
*** tzumainn has joined #tripleo14:01
derekhshardy: ok, I'm looking into our keystone, we may have the v3 api configured wrong or something14:03
*** max_lobur is now known as max_lobur_afk14:04
shardyderekh: What does your keystone endpoint URL look like?14:04
derekhthat would probably do it14:04
derekh| 91485fd74b6a4daba6667c1584ad0d99 | regionOne |       |       |       | 9ee566a842b64c28ae9ce14c824ed5c2 |14:04
shardyderekh: That should be OK, I substitute the v2.0 with v3 in the code14:05
derekhshardy: ok14:05
rlandyderekh: shardy: if it's helpful I have a lab machine with tripleo install (undercloud deploy just failed) - nothing rolled back yet.14:08
*** jdob_ has joined #tripleo14:08
rlandycan pm the creds to view/try changes14:09
rbradyhow are the docs in tripleo-incubator updated from the repo to http://docs.openstack.org/developer/tripleo-incubator/devtest_overcloud.html ?14:09
derekhrlandy: thanks but thats pretty much what I have here, shardy might be able to make use of it though14:09
openstackgerritRalf Haferkamp proposed a change to openstack/diskimage-builder: Setup ccache symlinks on openSUSE  https://review.openstack.org/6796514:09
*** jrist has joined #tripleo14:11
*** jrist has quit IRC14:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Install glance-manage into /usr/local/bin  https://review.openstack.org/6795614:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to horizon  https://review.openstack.org/6795214:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to heat  https://review.openstack.org/6795314:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Install heat-manage into /usr/local/bin  https://review.openstack.org/6795414:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to glance  https://review.openstack.org/6795514:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to neutron  https://review.openstack.org/6794914:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Install nova-baremetal-manage into /usr/local/bin  https://review.openstack.org/6794814:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Install keystone-manage into /usr/local/bin  https://review.openstack.org/6795114:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to keystone  https://review.openstack.org/6795014:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to novnc  https://review.openstack.org/6794514:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Install nova-manage into /usr/local/bin  https://review.openstack.org/6794714:12
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add package install support to nova  https://review.openstack.org/6794614:12
mariosabort... we've been slagled14:13
hewbrocc`patch bomb! take cover!14:13
akrivokawow impressive :)14:14
*** jrist has joined #tripleo14:15
lsmola_derekh: hmm, have you ever seen 'Got: ImportError('Class sslmiddleware_filter cannot be found', when I revert and restart Heat-API14:17
*** jrist has quit IRC14:17
derekhlsmola_: nope that one is new to me14:17
*** jrist has joined #tripleo14:18
shardyderekh: Aha, I've just reproduced, looks like a keystone regression but still investigating14:18
shardyThe roles.list(name='heat_stack_user') filter by name is not working14:19
shardyderekh: DOH! I had https://review.openstack.org/#/c/65985/ applied in my local test env which is not yet merged14:20
slaglemarios: hehe14:20
shardyderekh: I'll raise a heat bug and propose a workaround, sorry for the inconvenience14:20
slagleis there a way to resubmit just one change from a patchset to gerrit?14:21
rpodolyakaslagle: only, not rebase on master afaik (git review -R)14:21
rpodolyakaslagle: though, this won't help much if you need to change something in the middle of the series14:21
slagleright, that was the problem14:22
lsmola_shardy: cool, I am glad you have been able to find it so fas14:22
lsmola_shardy: fast14:22
*** jayg|g0n3 is now known as jayg14:23
*** e0ne has joined #tripleo14:24
*** CaptTofu has joined #tripleo14:25
*** martyntaylor1 has joined #tripleo14:26
derekhslagle: you could if you wanted just resubmit the updated patch and leave the later patches depended on the outdated version,14:26
derekhslagle: although probably only a good idea if you suspect it will have multiple itterations14:27
*** max_lobur_afk is now known as max_lobur14:27
*** e0ne_ has quit IRC14:27
derekhslagle: once it merges then you could rebase the rest14:27
lsmola_shardy: so it should be enough just to patch the keystoneclient ,right?14:29
*** martyntaylor has quit IRC14:29
slaglederekh: how do i resubmit just the updated patch though? make it a whole new change you mean?14:29
derekhshardy: cool, thanks14:29
shardylsmola_: No, I've been trying to get that patch accepted into keystoneclient, and they're being, uh, difficult, so I'll have to work around it in heat for now14:29
shardylsmola_: I'm working on a fix14:30
rbradyDoes anyone know the process for docs update in tripleo-incubator?  the docs dir in the repo is explicitly ignored in .gitignore14:30
lsmola_shardy: ok, great14:30
rbradyderekh, slagle: ^^14:30
openstackgerritDirk Mueller proposed a change to openstack/tuskar-ui: Remove dependencies on pep8, pyflakes and flake8  https://review.openstack.org/6812614:30
slaglerbrady: you update the devtest*.sh scripts themselves as the docs are generated from those scripts14:31
rbradyslagle: when does the doc actually get updated?  commit hook?14:31
rpodolyakarbrady: ... by a Zuul job that pushes them to http://docs.openstack.org/developer/tripleo-incubator/14:31
rbradyslagle: thanks!14:31
slaglerbrady: yes14:31
rpodolyakarbrady: after a change is merged14:31
derekhslagle: for that patchset if you wanted to just update https://review.openstack.org/#/c/67946/ you could do $ git review -d 6794614:32
derekhthen git review14:32
derekhslagle: non of the later commits would update but out be outdated14:32
derekhslagle: so they couldn't merge until you rebase them14:33
*** matty_dubs|gone is now known as matty_dubs14:33
derekhslagle: this would only really be usefull if you expected you would have to update https://review.openstack.org/#/c/67946/ multiple times14:33
derekhslagle: then once its merged you could rebase the rest14:33
openstackgerritRyan Brady proposed a change to openstack/tripleo-incubator: Update cinder usage  https://review.openstack.org/6812914:33
slaglederekh: ah, makese sense. thanks14:33
*** jdob_ has quit IRC14:34
rbradyrpodolyaka: thanks14:35
rpodolyakarbrady: btw, if you need to build docs locally (e.g. if you are fixing docs), you could do: tox -evenv -- python setup.py build_sphinx14:35
*** kwwii_ has joined #tripleo14:36
*** kwwii_ has left #tripleo14:36
*** coolsvap_away has joined #tripleo14:45
*** coolsvap has quit IRC14:47
*** coolsvap_away is now known as coolsvap14:48
*** bauzas has quit IRC14:51
*** bauzas has joined #tripleo14:53
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Add ability to use local cloud image  https://review.openstack.org/6813314:53
*** boris-42 has quit IRC14:54
*** jcoufal has quit IRC15:02
*** jcoufal_ has joined #tripleo15:02
*** spzala has joined #tripleo15:03
shardyderekh, lsmola_: https://review.openstack.org/6813515:04
*** vkozhukalov has quit IRC15:04
lsmola_shardy: excellent15:04
lsmola_shardy: thank you15:05
derekhlsmola_: thanks, will try it out15:05
derekhshardy: thanks ^15:05
shardylsmola_, derekh: np, sorry for breaking it ;)15:05
derekhshardy: no prob15:06
shardyIf you can +1 the review if it works for you that would be much appreciated :)15:06
lsmola_shardy: np, you are not first one who break it :-)15:06
lsmola_shardy: actually breaking it is our motto :-)15:07
derekhshardy: will do15:07
ccrouchWhy in overcloud-source.yaml do we pass CeilometerDSN, aka ceilometer.db, into nova-compute-instance.yaml when its not used by any of the elements used to build the overcloud compute image?15:10
ccrouchis this in anticipation of the ceilometer-agent-compute element going into the overcloud compute image?15:10
*** mrunge has quit IRC15:20
*** martyntaylor1 has quit IRC15:22
*** marun has joined #tripleo15:25
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Write public key to authorized_keys  https://review.openstack.org/6813915:27
ccrouchrwsu: shadower: any thoughts on my question above?15:27
openstackgerritA change was merged to openstack-infra/tripleo-ci: Set TRIPLEO_ROOT  https://review.openstack.org/6804515:28
*** lucasagomes is now known as lucas-hungry15:33
*** UtahDave has joined #tripleo15:37
*** martyntaylor has joined #tripleo15:38
*** martyntaylor has quit IRC15:38
*** martyntaylor has joined #tripleo15:38
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Destroy the remote seed domain  https://review.openstack.org/6814315:46
*** lblanchard has quit IRC15:48
*** lblanchard has joined #tripleo15:51
*** jprovazn has quit IRC15:52
*** jcoufal_ has quit IRC15:57
*** boris-42 has joined #tripleo16:02
*** rdopieralski has quit IRC16:12
*** lucas-hungry is now known as lucasagomes16:25
*** panda has joined #tripleo16:27
*** LinuxJedi is now known as LinuxJedi__16:29
*** panda_ has quit IRC16:30
*** panda_ has joined #tripleo16:31
*** taps has joined #tripleo16:33
*** panda has quit IRC16:33
*** LinuxJedi__ has quit IRC16:44
*** bauzas has quit IRC16:45
*** Aarongr_afk is now known as AaronGr16:46
*** markmc has quit IRC16:46
*** ftcjeff has joined #tripleo16:49
*** rlandy is now known as rlandy|bbl16:53
*** vkozhukalov has joined #tripleo17:06
*** CaptTofu_ has joined #tripleo17:07
*** julim has quit IRC17:09
*** hewbrocc` has quit IRC17:09
rwsuccrouch: the only parameter that is needed is the metering secret, i don't know how tomas was planning to use the other params17:09
*** ccrouch has quit IRC17:09
*** CaptTofu has quit IRC17:09
*** shadower has quit IRC17:09
*** lifeless has quit IRC17:09
*** ccrouch has joined #tripleo17:09
*** shadower_ has joined #tripleo17:09
*** lifeless1 has joined #tripleo17:09
*** jeblair has quit IRC17:09
*** ccrouch has quit IRC17:09
*** ccrouch has joined #tripleo17:09
*** julim has joined #tripleo17:10
*** jeblair has joined #tripleo17:10
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: set DIB_REPOLOCATION_<project> for projects  https://review.openstack.org/6817817:10
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Write public key to authorized_keys  https://review.openstack.org/6813917:10
*** matty_dubs is now known as matty_dubs|lunch17:14
*** jistr has quit IRC17:15
*** e0ne has quit IRC17:16
*** martyntaylor has quit IRC17:16
*** CaptTofu_ has quit IRC17:19
*** wendar has joined #tripleo17:27
*** morazi has quit IRC17:29
greghaynesMorning, TripleO17:30
cody-somervillegreghaynes: Hi Greg!17:30
cody-somervillegreghaynes: Welcome!17:30
greghaynesHello, thanks!17:30
*** viktors has left #tripleo17:30
*** marun has quit IRC17:38
*** marun has joined #tripleo17:44
*** jistr has joined #tripleo17:47
SpamapSg'morning tripleo17:48
SpamapSgreghaynes: ahoy there!17:49
greghaynesfinally made it :)17:50
*** matty_dubs|lunch is now known as matty_dubs17:52
*** mrunge has joined #tripleo17:53
*** CaptTofu has joined #tripleo17:53
*** coolsvap is now known as coolsvap_away17:54
*** mestery has quit IRC17:55
Nghey greghaynes :)17:56
greghaynesHi there17:56
*** derekh has quit IRC17:58
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Added initial tuskar api tests  https://review.openstack.org/6794018:01
*** athomas has quit IRC18:04
*** max_lobur is now known as max_lobur_afk18:07
*** martyntaylor has joined #tripleo18:09
SpamapShttps://review.openstack.org/#/c/68135/ <-- breaking TripleO18:16
SpamapScherry pick that into your heat to get devtest to work again18:16
*** martyntaylor has left #tripleo18:20
*** morazi has joined #tripleo18:22
*** akrivoka has quit IRC18:26
*** wendar has quit IRC18:28
*** wendar has joined #tripleo18:28
mordredlook it's a greghaynes !18:29
greghaynesyep yep18:31
mordredgreghaynes: great. now that you're here - first task is "fix everything"18:34
* mordred is now done for the day having imparted that wisdom18:34
*** hewbrocca has joined #tripleo18:34
greghaynessounds easy enough!18:35
SpamapSmordred: you have one other task.18:41
mordredSpamapS: wait - what is _my_ task?18:41
SpamapSmordred: _BREAK EVERYTHING_18:41
mordredSpamapS: DONE18:41
*** mrunge has quit IRC18:43
*** ccrouch has quit IRC18:43
greghaynesok, amateur devtest q incoming18:45
*** jprovazn has joined #tripleo18:47
greghaynesheat stack-create results in errord undercloud, got any tips on where to poke around to reverse engineer this set of tracebacks? https://gist.github.com/anonymous/8545751/raw/b91a8875d66604e82210ee9a0f9ac1f8a4c432f1/gistfile1.txt18:47
*** marun has quit IRC18:48
*** marun has joined #tripleo18:49
slagleit's look like ssh back to your devtest hold failed18:50
cody-somervillegreghaynes: Looks like the virtual power drive is failing to connect via ssh. Is ssh running on your host? and is the network setup so it can ssh in?18:50
slagleyea, that :)18:50
greghaynesyep, thats a winner18:51
* greghaynes considers adding this to devtest docs18:51
slaglemaking sure ssh is started and enabled would be a good patch for install-dependencies18:51
greghaynesoo, even better18:52
*** marun has quit IRC18:54
*** rpodolyaka1 has joined #tripleo18:55
*** jcoufal-mobile has joined #tripleo18:57
openstackgerritJiri Tomasek proposed a change to openstack/tuskar-ui: Number picker input widget and angular directive  https://review.openstack.org/6805318:57
greghaynesMorning, lifeless118:58
*** akuznets_ has quit IRC18:59
*** lifeless1 is now known as lifeless18:59
lifelessgreghaynes: hi18:59
lifelessmeeting time everyone :)18:59
*** shadower_ is now known as shadower19:02
*** rlandy|bbl is now known as rlandy19:02
*** ccrouch has joined #tripleo19:05
*** spzala has quit IRC19:07
*** ccrouch1 has joined #tripleo19:09
*** ccrouch has quit IRC19:10
*** e0ne has joined #tripleo19:11
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Fixes cinder element dependencies  https://review.openstack.org/6820419:18
openstackgerritRyan Brady proposed a change to openstack/tripleo-incubator: Update cinder usage  https://review.openstack.org/6812919:24
openstackgerritAaron Weitekamp proposed a change to openstack/diskimage-builder: Update default RHEL guest image  https://review.openstack.org/6820619:27
openstackgerritAaron Weitekamp proposed a change to openstack/diskimage-builder: Add support Red Hat Network (RHN)  https://review.openstack.org/6820719:27
openstackgerritAaron Weitekamp proposed a change to openstack/diskimage-builder: Support list of Red Hat channels and repos  https://review.openstack.org/6820819:27
openstackgerritAaron Weitekamp proposed a change to openstack/diskimage-builder: Add support for Red Hat Satellite  https://review.openstack.org/6820919:27
*** vkozhukalov has quit IRC19:27
SpamapSooo shiny RHEL stuff19:28
openstackgerritA change was merged to openstack/tuskar-ui: Deployment design form  https://review.openstack.org/6715519:29
jdobmarios: ping19:29
*** pblaho has joined #tripleo19:31
mariosjdob: hi19:41
*** mestery has joined #tripleo19:41
jdobmarios: hey, you mentioned your overcloud.yaml thing was blocked by the controllers and models, but i'm not sure why. the models are in place and the controllers shouldn't matter to what you're doing19:42
jdobeventually it'll need to be wired to a controller, but 99% of the actual work of building the template doesn't care about the controller19:42
mariosjdob: right. but i'm making assumptions about what the controllers will be (i.e where template generation will be invoked from). also, models are just in, and i expect may yet change19:43
mariosjdob: is all i meant19:43
jdobbut what you're doing is pretty much algorithmic, we can get that checked in and just not wired up to a controller19:43
jdobi agree the models may change, but all the more reason for you to start beating on them now to find the holes :)19:43
mariosjdob: yes i agree. and i am19:43
jdoboh, ok, i misread then, I thought you were saying you were blocked19:44
jdobsounds like we're on the same page19:44
mariosjdob: yeah, didn't say i was blocked mate, just that models/controllers are tbd19:45
mariosjdob: thanks19:45
jdobmy bad :)19:45
mariosjdob: heh np :)19:45
jdobi blame my distraction as I stare out the window and want to play in the snow19:45
mariosjdob: its almost 2200 here... ;)19:46
jdobwhere are you out of?19:46
mariosjdob: cyprus19:46
jdoboh wow, that's a new one for me19:46
mariosjdob: for most people ;) (small place)19:47
jdobthe "mate" thing threw me off, I didnt know of many places that use that term19:47
*** marun has joined #tripleo19:51
hewbroccajdob: marios was in the UK for quite a while19:53
jdobahh, ok19:54
*** e0ne has quit IRC19:54
*** GheRivero has quit IRC19:57
*** GheRivero has joined #tripleo19:58
ccrouch1can someone point me at where  to look to get a better understanding of how the stateful upgrades are going to work ?20:00
mariosjdob: yeah - was in the uk for many years and in fact i was born there cos my parents used to live there.20:00
mariosgoodnight tripleo20:00
lifelessccrouch1: hey, https://trello.com/b/0jIoMrdo/tripleo under current MVP20:01
*** markmc has joined #tripleo20:01
lifelessccrouch1: have a look within those three cards20:01
ccrouch1i think i've got a reasonable grasp of the current flow to push a new deployment to a new node with glance, nova-bm-helper the node being provisioned etc20:01
ccrouch1I'm trying to figure out the equivalent for upgrades20:01
SpamapSperhaps we should install horizon and tuskar in the seed and undercloud20:01
ccrouch1lifeless: thanks, looking...20:02
*** pblaho has quit IRC20:02
rlandySpamapS: did you try out the heat fix (https://review.openstack.org/#/c/68135/)?20:02
SpamapSrlandy: building now20:03
rlandySpamapS: k ... would be interested to know if it worked for you. I did the quick/sketchy thing - imported the fix to the seed, restarted heat and keystone services and deployed the undercloud. Maybe not surprisingly, still got the error.20:08
*** bauzas has joined #tripleo20:08
*** jistr has quit IRC20:11
*** epim has joined #tripleo20:16
*** jcoufal-mobile has quit IRC20:18
SpamapSrlandy: did you see the error in the seed's heat-engine log?20:23
*** ftcjeff_ has joined #tripleo20:24
SpamapSrlandy: for me, on seed, keystone user-role-list --user=undercloud-User-$randomthing shows heat_stack_user20:25
SpamapSmeaning problem is solved20:25
*** lucasagomes has quit IRC20:26
dprincerlandy: That fix works for me too. If you apply it to the /opt/stack/venv/heat in your seed VM and the 'service restart heat-engine' you should be good to go on the next heat stack-create for the undercloud20:27
*** jdob has quit IRC20:35
*** jdob has joined #tripleo20:36
rlandydprince: SpamapS: ok - will try it again .... good to know it works20:37
dprincerlandy: or just set these and rebuild your seed too:20:38
dprinceexport DIB_REPOLOCATION_heat=https://review.openstack.org/openstack/heat20:38
dprinceexport DIB_REPOREF_heat=refs/changes/35/68135/120:38
*** mrunge has joined #tripleo20:38
dprincerlandy: either should work...20:38
*** ftcjeff_ has quit IRC20:39
*** spzala has joined #tripleo20:42
*** e0ne has joined #tripleo20:43
*** e0ne has quit IRC20:44
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Make neutron-openvswitch-agent run cleanup.  https://review.openstack.org/6826120:45
*** rpodolyaka1 has quit IRC20:45
openstackgerritA change was merged to openstack/diskimage-builder: Move Babel and argparse from test-requirements.txt  https://review.openstack.org/6707020:46
*** marun has quit IRC20:46
*** jdob has quit IRC20:47
*** jdob has joined #tripleo20:47
lifelessgreghaynes: StevenK: oh btw - you'll want to update your affiliation mappings soonish :020:52
SpamapShah yeah lets not repeat the havana mess20:55
Nglifeless: SpamapS: I'm confused by the commentary on https://review.openstack.org/#/c/65492/3/elements/ssh-ephemeral/install.d/10-sshd-host-keys-ephemeral,unified20:57
NgI thought it was ok to register a state path for things that don't yet exist?20:57
Ngbut if I don't want to register the not-yet-existing host keys, why would I then want to register all of /etc/ssh? At that point wouldn't modifying the config to store in /mnt/state be sufficient?20:58
lifelessNg: give me a sec, will look at it20:58
openstackgerritRalf Haferkamp proposed a change to openstack/diskimage-builder: Setup ccache symlinks on openSUSE  https://review.openstack.org/6796521:01
openstackgerritDan Prince proposed a change to openstack/diskimage-builder: Only configure DHCP for real interfaces  https://review.openstack.org/6826521:04
lifelessNg: ok so21:04
lifelessNg: register-state-path is not a requirement for state paths21:04
lifelessNg: its specific plumbing to help with recalcitrant situations21:04
lifelessNg: a missing file is not such21:04
SpamapSNote that cloud-init will just always write to /etc/ssh21:05
lifelessNg: so thats the first bit on 'missing files'21:05
SpamapSwhich Ng patched out..21:05
*** lblanchard has quit IRC21:05
SpamapSbut which counts as recalcitrant and is why I suggest just taking all of /etc/ssh into /mnt/state21:05
*** e0ne has joined #tripleo21:06
lifelessnote that cloud-init also deletes the initial keys from the image21:06
SpamapSThe bad part of that is that we now own all the sshd configs.. but I think that may be O-K. :p21:06
lifelessits fundamentally broken if you ever reboot a cloud instance21:06
SpamapSlifeless: uh no21:06
lifelessoh, cfg option21:06
lifeless    if cfg.get("ssh_deletekeys", True):21:06
SpamapSlifeless: it only does that if instance-id changes21:06
lifelessSpamapS: no21:06
lifelessline 6121:07
lifelessdefaults to enabled21:07
SpamapSlifeless: right it only runs that code path if there is a new un-handled instance id21:07
lifelessSpamapS: that doesn't match my observed behaviour21:07
SpamapSwhich btw, suggests that we need to put /var/lib/cloud in state :)21:07
lifelessSpamapS: IMBW of course21:07
* SpamapS this close to saying /var/lib should be bind mounted into /mnt/state21:08
lifelessdoes ssh-keygen follow symlinks ?21:08
SpamapSexcept dpkg .. ugh21:08
lifelessSpamapS: theres a tonne of cruft we don't want21:08
lifelessbecause distros also conflate what the user wants and what the system wants21:08
SpamapSlifeless: I agree that keys are deleted before they are regenerated. That is important. But I am not agreeing that keys are generated on every reboot.21:09
lifelessSpamapS: sure, I moved on21:09
lifelessSpamapS: if the keys are deleted, a symlink will be deleted21:09
lifelessSpamapS: if we disable key deletion....21:09
greghayneslifeless: I just joined the openstack foundation the other day and added HP affiliation, is there anywhere else I should be worried about affiliation mappings?21:09
lifelessSpamapS: and ssh-keygen follows symlinks21:09
*** julim has quit IRC21:09
lifelessgreghaynes: about 3 dozen places [exaggerating]21:09
lifelessgreghaynes: git-dm for sure21:10
SpamapSlifeless: or.. /etc/ssh -> /mnt/state/etc/ssh and we just deal with owning sshd_config and the like.21:10
lifelessSpamapS: thats a possibility too21:10
lifelessSpamapS: we're exploring it ;)21:10
SpamapSI also think smoser posted a patch to let us configure cloud-init to set the ssh data dir21:11
SpamapSbut that is more to deal with us patching it directly in the image build21:11
lifelessSpamapS: to answer Ng's question, the proposal to move all of /etc is to do an end-run around cloud-init21:11
lifelessby making /etc resolve to the stateful location, so whatever it thinks its doing will be in the right place21:11
lifelessas long as key generation happens after ephemeral volume mounting21:11
lifelessotherwise, it will happen in / and be hidden after /mnt is mounted21:12
SpamapSYeah that may be a tricky thing but it definitely _should_21:12
lifelessso, I have no objection to having templates for all of /etc/ssh21:12
lifelesserm, I mean all of /etc/openssh or whatever, *not* all of /etc21:12
lifelessif we're going to own it, I think we should own it and make it consistent everywhere21:13
lifelessslagle: ^ in the same way I'm conerned about service names21:13
lifelessSpamapS: but I think we should try disabling key deletion as a first stab21:13
openstackgerritDan Prince proposed a change to openstack/diskimage-builder: Rename generate-interfaces-file.sh...  https://review.openstack.org/6826621:13
NgI'm not 100% sure why, but taking over all of /etc/ssh makes me feel a bit queasy21:13
openstackgerritDan Prince proposed a change to openstack/diskimage-builder: Rename generate-interfaces-file.sh...  https://review.openstack.org/6826621:14
lifelessSpamapS: because owning all of openssh configuration is a pretty big hammer21:14
SpamapSlifeless: looks like ephemeral is part of cloud-config and thus after ssh key generation21:15
Ngis the objection to just patching cc_ssh.py, that it's kinda rude and potentially fragile?21:15
SpamapSNg: well it is really fragile IMO21:15
SpamapSNg: also we have a patch now.. so worst case we upgrade cloud-init on all our images.21:16
slaglelifeless: what's the concern? having all our services not named the same?21:18
lifelessslagle: documentation for ops of a tripleo environment21:18
lifelessSpamapS: so the element just won't work right now21:19
lifelessNg: ^21:19
lifeless10:15 < SpamapS> lifeless: looks like ephemeral is part of cloud-config and thus after ssh key generation21:19
lifelessNg: e.g. it will get written to /mnt and then /mnt will get mounted21:19
lifelessSpamapS: you have a workaround for /mnt not being mounted by cloud config, right ?21:19
slaglelifeless: i added a os-svc-restart...so we could just document that. the service name will be the same regardless of distro or source/package install21:19
Ngthe test instances I booted ended up with the setup I was expecting21:19
lifelessslagle: the logs are affected too though, right?21:20
Ngbut it's possible I did something wronglike21:20
lifelessslagle: so any complexity is going to propogate all the way down into logstash and analysis tools21:20
lifelessslagle: I like os-svc-restart21:20
lifelessslagle: but I don't think it solves the issue21:20
lifelessNg: were they vms or physical? Did they have a /mnt partition mounted?21:21
Nghmm, actually they were probably bad test cases due to a lack of a separate partition being mounted21:21
*** rlandy has quit IRC21:21
SpamapSlifeless: yes.. https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/use-ephemeral/os-refresh-config/pre-configure.d/00-fix-ephemeral-mount21:21
slaglelifeless: yea, true. some services don't log to files at all right now, they just use journald21:22
slaglevs upstart service logs21:22
Ngwell, if I can replace the entire file with register-state-path /etc/ssh and make all this go away, I guess we're owning /etc/ssh ;)21:22
slaglebut, we control the logging configuration. even in the package installs21:22
SpamapSlifeless: I thought of one other thing.. we _could_ try to get ephemeral in the fstab. Doesn't it get a label? That would happen well before cloud-init tries to do any writing to disk.21:23
*** dprince has quit IRC21:23
* Ng hopes that our children will get to live in a world where there is only one init system on Linux distros21:23
lifelessNg: that wouldn't fix the mount race thats in discussion21:23
lifelessNg: 4 for ever21:23
lifeless(cloud-init, systemd, upstart, orc :P)21:23
*** spzala has quit IRC21:23
lifelessall 4 all at once21:23
SpamapS\o/ .. devtest actually worked ( with the heat patch cherry picked )21:23
Nglifeless: I hate your future21:24
lifelessNg: SpamapS: I am going to have to context switch; I suggest that in principle disabling deletion is a better fix than taking all of /etc - less scope creep21:24
lifelessand that we may need to work with cloud init to support /mnt , both the mount timing issue and data store for the keys21:25
lifelessboth seem like upstream patches to me21:25
*** e0ne has quit IRC21:27
*** markmc has quit IRC21:30
*** e0ne has joined #tripleo21:31
ccrouch1lifeless: https://etherpad.openstack.org/p/tripleo-image-updates21:33
ccrouch1was very helpful.21:33
ccrouch1I think i have a better idea how the upgrade-with-reboot will work, i.e. just leave the ephemeral partition alone when laying down new image.21:33
ccrouch1Are there more details about the upgrade-without-reboot flow?21:33
ccrouch1I see reference to  "takeovernode ", is that https://github.com/openstack/tripleo-incubator/blob/master/scripts/takeovernode21:33
ccrouch1but that still includes a reboot21:33
lifelessbut only needs one if the kernel changed21:34
lifelessits not finished, obviously21:34
jomara_tzumainn: kicked off irc internal?21:35
*** jprovazn has quit IRC21:37
bnemecjomara_: Yeah, looks like there was a problem with it.21:37
bnemecI just reconnected, but I was off for quite a while.21:38
ccrouch1lifeless: got it, I see the reboot OR...now in the todo. thx21:38
lifelessslagle: so look, I dunno21:38
lifelessslagle: I can see that we can tweak logs and we could tell folk to know about os-svc-restart21:38
lifelessslagle: but I worry that doing that - persistently masking users from the things their distro of choice installed21:39
lifelessslagle: basically makes us a virtual linux distro21:39
lifelessslagle: I'm speculating that a narrower focus - drill down on openstack - and say 'it will be X for our supported platforms' might avoid that21:40
lifelessslagle: so rather than documenting new-tool-Y, we document 'it will use your init system, but be called nova-api'21:40
jomara_bnemec: yeah it just popped me back on too21:40
slaglelifeless: and what would nova-api do?21:41
slaglelifeless: detect the init system restart the service correctly?21:41
*** CaptTofu has quit IRC21:41
*** e0ne has quit IRC21:43
lifelessslagle: I meant that we'd call the unit/job/$thing nova-api consistently21:43
slagleoh i see21:44
slaglei actually was going to go with a similar approach using symlinks for the systemd services21:44
slaglebut since systemd uses symlinks to disable/enable, you run into all sorts of problems21:45
slaglewe could rename the service files, but then we're modifying packaged files21:45
lifelessis that a concern? Since we're not using the package manager to upgrade...21:48
slaglei'm not super concerned21:50
slaglei would probably need to check if there is a concern from a support perspective21:51
slagleor maybe we could package something that fixes the names21:51
slagletbh, i wish fedora would just stick to upstream names in general. it makes searching for packages so much easier21:52
bnemecslagle: Yeah, symlinks for service files are an issue.  We discussed a workaround for mariadb here: https://bugzilla.redhat.com/show_bug.cgi?id=101431121:53
lifelessslagle: +1 :)21:54
lifelessI spoke to stewart @ LCA, he says that we can use the binary package for anything they don't have native repos21:54
lifelessso we should be good to go on that migration21:54
*** jcoufal has joined #tripleo21:56
slaglebnemec: oh, nice so we could package up a set of complimentary service files then21:57
bnemecslagle: Yeah, it seems like that should work fine.21:58
bnemecSadly they're dropping the mysql service compatibility for mariadb in rhel7. :-(21:58
*** jdob has quit IRC21:58
*** lblanchard has joined #tripleo22:02
*** CaptTofu has joined #tripleo22:05
ccrouch1 I see from overcloud-source.yaml that gre is the tunneling protocol neutron uses in the overcloud. I presume that means vxlan or vlan options are untested?22:09
lifelessmore fun...22:13
lifelesspleia2: hey so22:14
lifelesspleia2: you seemed a little unsure where the tripleo-ci stuff was at; perhaps if we had a brief chat that might clear thinsg up? I'm inferring (wrongly?) that as a result you aren't sure what to hack on...22:15
ccrouch1next random question :-)22:18
ccrouch1does anyone see any benefit from having neutron run on its own node in the overcloud separate from the other controller services?22:18
*** vipul is now known as vipul-away22:19
ccrouch1iiuc we've seen customer deployments that have/desire this separation22:20
ccrouch1i'm just wondering if anyone else sees a preference for that?22:20
openstackgerritJames Slagle proposed a change to openstack/tripleo-incubator: Document stable branches in devtest  https://review.openstack.org/6827422:22
pleia2lifeless: a chat would be great22:22
lifelesspleia2: gg+ ?22:23
slagleccrouch1: i believe you would need to do that if you had true isolated networks and desired public internet access for your tenant instances22:23
lifelessslagle: ccrouch1: hmm? no, its not needed for that22:23
slagleno? i thought i saw it on a diagram...somewhere in the docs22:23
lifelessits a scaling bottleneck though, and folk generally want network to be as fast as possible :)22:24
lifelessslagle: HP region has internal network, external network22:24
lifelesstwo NICs, done22:24
slagleok :)22:24
lifelessso breaking it out is a thing22:24
pleia2lifeless: yeah, just grabbing my headphones :)22:24
lifelessjust not required by that use case22:25
*** vipul-away is now known as vipul22:25
*** vipul is now known as vipul-away22:25
pleia2lifeless: ready when you are22:26
ccrouch1lifeless: so its a potential roadmap item to support the heat template that splits neutron out?22:26
lifelesssplit all the things22:26
slaglethis is what i was thinking of: http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#network-connectivity22:28
slaglebut, yea, that wouldn't be a requirement22:28
*** jwalls has joined #tripleo22:30
*** ftcjeff has quit IRC22:30
SpamapSccrouch1: just to add on.. splitting things out right now is painful because we have to keep things mergable or duplicate effort. With the hot software config changes we should be able to write-once-use-anywhere for template snippets that just include metadata.22:44
*** morazi has quit IRC22:46
*** spzala has joined #tripleo22:47
*** jayg is now known as jayg|g0n322:47
ccrouch1SpamapS: good to know22:52
lifelessSpamapS: well, also tuits; we haven't really pushed to make it be done23:00
*** morazi has joined #tripleo23:01
*** mrunge has quit IRC23:02
*** CaptTofu has quit IRC23:04
*** jcoufal has quit IRC23:04
*** lblanchard has quit IRC23:07
SpamapSlifeless: I've prioritized reviews that pave the way for it. :)23:12
*** matty_dubs is now known as matty_dubs|gone23:13
*** jtomasek has quit IRC23:19
*** jwalls has quit IRC23:28
*** lblanchard has joined #tripleo23:29
*** ccrouch1 has quit IRC23:41
*** blamar has quit IRC23:47
*** lblanchard has quit IRC23:52
openstackgerritClint "SpamapS" Byrum proposed a change to openstack/diskimage-builder: Add Copyright and License header to debian element  https://review.openstack.org/6697523:57
StevenKBleh, I can think of a few more places where I should do the same23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!