Thursday, 2014-04-24

lifelessmordred: btw - reinstall redphone :)00:01
*** nati_ueno has quit IRC00:05
openstackgerritMonty Taylor proposed a change to openstack-infra/tripleo-ci: Use /opt/git instead of /opt/stack/new  https://review.openstack.org/8997400:05
mordredlifeless: merry christmas ^^00:05
*** epim has quit IRC00:06
*** epim has joined #tripleo00:10
*** epim has quit IRC00:12
StevenKHm00:14
StevenKI think that won't work, since the devtest scripts won't know about the openstack/ in the directory, for setting things like PATH00:15
* mordred cries a little00:16
lifelessStevenK: hmm ?00:17
StevenKmordred: You could accidently ln -s $TRIPLEO_ROOT/openstack/tripleo-incubator to $TRIPLEO_ROOT/tripleo-incubator00:17
*** epim has joined #tripleo00:18
*** eguz has joined #tripleo00:19
*** eghobo has quit IRC00:23
*** epim has quit IRC00:29
*** fungi has quit IRC00:30
vinshgreghaynes, http://paste.openstack.org/show/76870/00:30
vinshthis thing.00:30
vinsh3 nodes. all running keepalived.  control0 prio300 control1 prio200 control2 prio30000:31
vinshsudo service keepalived stop on all of them00:31
vinshThis is a tcp dump with the service stopped... you can see the advertisements.. but its filled with a load of 100 vs 201 advertisements... that don't seem to have a labeled source00:32
vinshits the virtual ip.. advertising to vrrp "vrrp 192.0.2.254 > 224.0.0.18"00:36
vinshwth.00:36
*** nati_ueno has joined #tripleo00:36
openstackgerritMonty Taylor proposed a change to openstack/tripleo-incubator: Handle two-level-deep git repo structure  https://review.openstack.org/8997700:39
mordredStevenK: how about that ^^ ?00:39
*** nati_ueno has quit IRC00:40
StevenKmordred: Did you test that? :-)00:42
StevenKBecause I think it's going to die horribly.00:43
lifelessgreghaynes: you might enjoy the workflow vs convergence thing now00:47
lifelessclarkb: do you know where we are at on the f20 starvation thing?00:50
clarkbno idea00:50
*** kiall has quit IRC00:58
*** proffalken has quit IRC00:58
*** Shrews has quit IRC00:58
*** Shrews has joined #tripleo01:00
*** proffalken has joined #tripleo01:00
*** rlandy has quit IRC01:01
*** kiall has joined #tripleo01:02
*** nati_ueno has joined #tripleo01:10
lifelessugh01:15
lifeless2014-04-23 03:51:00.874 | No distributions at all found for oslo.config>=1.2.0 (from -r /opt/stack/nova/requirements.txt (line 32))01:15
lifeless2014-04-23 03:51:00.874 | Storing debug log for failure in /root/.pip/pip.log01:15
*** TravT has quit IRC01:15
lifelesscody-somerville: oh hai01:19
lifelesscody-somerville: around ?01:19
cody-somervillelifeless: Heya :)01:19
*** yamahata_ has joined #tripleo01:19
*** yamahata has joined #tripleo01:20
mordredlifeless: if I want to reference $(lsb_release -sc) in an element - (trusty) is that something already abstracted?01:24
StevenK$DIB_RELEASE perhaps?01:26
lifelesscody-somerville: heya01:27
mordredStevenK: awesome. thanks!01:27
*** fandi has quit IRC01:27
*** nati_ueno has quit IRC01:28
*** eguz has quit IRC01:33
vinshbtw.. anyone following the keepalived on 3 control node saga.. it all works.. if you choose br-ex as the keepalived interface.. but not if you chose eth0.01:33
vinshso. progress-ish.01:33
mordredStevenK: ok - you're my new goto person on questions. lucky you01:35
lifelessbbiab01:36
mordredStevenK: https://review.openstack.org/#/c/88479/3/modules/openstack_project/files/nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list01:36
*** nosnos has joined #tripleo01:36
mordredStevenK: if I have a python script in an element script01:36
mordredit doesn't seem to so much be getting env vars01:36
mordredam I just stupid?01:37
StevenKmordred: I'm struggling to see how it's nailed together01:47
mordredStevenK: I thinkI figured it out01:48
StevenKmordred: Then I was delighted I could help?01:48
mordredStevenK: the simple problem was not that the sript wasn't seeing the env var - it was that I was joining two paths that both start with '/'01:49
mordredwhich ends up with fail01:50
StevenKAh yeah01:50
StevenKI've hit that myself before01:50
StevenKos.path.join('/foo', '/bar') doesn't end with cute puppies and rainbows.01:50
mordredso this: Caching gerrit from git://git.openstack.org/openstack-infra/gerrit.git in /home/mordred/.cache/image-create/source-repositories/gerrit_68bdd7e0a4a59330c382812459e112857f1361da01:50
mordredif, in a day, there are new refs in that repo01:51
mordreddo we re-clone? or do we fetch and move the dir?01:51
clarkbmordred: fetching is tricky because by default it won't fetch all the things01:52
clarkbbut that is probably ok for caching01:52
mordredclarkb: btw - you should look in dib in elements/cache-url/bin/cache-url01:53
adam_ghas the bump of default release from saucy -> trusty caused  problems for anyone else?01:53
mordredclarkb: we should totally appropriate that code for the places in infra/config where we're downloading a url and wanting to NOT take action if the url contents have not changed :)01:53
mordredlifeless: btw - not that you'll be surprised or anything - but your suggestion to have an extra-data script write out a source-repositories file worked like a champ02:00
*** weshay has quit IRC02:02
*** jrist has quit IRC02:02
*** jrist has joined #tripleo02:09
lifelessmordred: great02:19
mordredlifeless: and I think I'm really close to having the image elements part of the project done02:23
*** nati_ueno has joined #tripleo02:26
lifelessStevenK: have you submitted your pyconau talk proposal?02:40
lifelessStevenK: I need brainstorm help for mine :/02:40
StevenKlifeless: I have not02:41
lifelessStevenK: duuuude02:41
lifelessjogo: have you ? and anteaya I think you were planning on one too ?02:41
clarkbits too bad it happens during seattle summer otherwise I totally would02:44
StevenKclarkb: Seattle has one week of summer? :-P02:44
clarkbStevenK: exactly02:44
clarkbI will have family visiting too02:45
clarkbbut yes first week of august is summer02:45
StevenKlifeless: I have an idea from SpamapS, which sounds pretty good. How can I help?02:45
*** newell has quit IRC02:45
*** NobodyCam has quit IRC02:46
StevenKclarkb: MAD did a send up of Northern Exposure, if you remember that show from the 90s. One bit of that send up was, "There's an awful lot of snow around, when is summer?" and the reply: "I don't rightly remember. Last year it was Tuesday."02:46
*** morganfainberg is now known as morganfainberg_Z02:47
*** yamahata has quit IRC02:57
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/131163103:00
uvirtbotLaunchpad bug 1311631 in tripleo "rabbitmq os-refresh config fails on the master due to mis-parsing of nodes list" [Undecided,In progress]03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/130288103:00
uvirtbotLaunchpad bug 1302881 in tripleo "incloud CIDR can overlap custom baremetal-network" [Undecided,Incomplete]03:00
*** untriaged-bot has quit IRC03:00
mordredlifeless: if you saw this:03:04
mordred/tmp/image.N7JkNvoL/hooks/extra-data.d/98-source-repositories: line 104: pushd: /tmp/image.N7JkNvoL/mnt/root/config: Permission denied03:04
mordredlifeless: would you have an immediate reaction as to "zomg, that's clear a fail in XXXX"03:04
lifelessextra-data runs outside the chroot03:05
lifelessas such its not running as root03:05
lifelessyou probably want to sudo there.03:05
lifelessI think.03:05
lifelessthats the path I'd look at first anyhow03:06
lifelessas well as obviously making sure that root/config is indeed a directory03:06
mordredbut - I mean, it's a source-repostories thing03:07
mordreddo I need to make an extra-data script that chmod's /root first?03:08
mordredand then a 99 that chmod's it back?03:08
mordredthe intent is that I want it to clone a repo and put it in /root/config03:08
mordredyou know - how about I just try something else03:08
lifelessso you should be able to03:16
lifelessI'd dig in a little and see whats going on03:16
lifelessit should be totally possible and straight forward03:16
*** geerdest has quit IRC03:19
cody-somervilleugh, realizing one of the earbuds in your headset is no longer working just before a long call is the worst. :(03:29
lifelessouch03:33
*** ramishra has joined #tripleo03:46
*** Rakesh6 has joined #tripleo03:49
*** vinsh is now known as vinsh_away03:53
*** nati_ueno has quit IRC03:55
mordredlifeless: are install.ds' interleaved of sequential?03:57
lifelesssequential today03:57
lifelesswe reserve the right to be more parallel in future but only for same-numbered things03:57
mordredso ALL of one element's stuff runs, then aother03:57
mordredor, rather, all of one elements' install.d runs, then the next, yeah?03:58
*** eghobo has joined #tripleo04:01
* StevenK tries to expand his 66 word abstract04:10
lifelessno04:13
lifelessall 66-foo runs04:13
lifelessthen all 67-foo runs04:13
lifelessacross all elements04:14
lifelessthis lets elements drop new stuff inbetween bits of another element04:14
lifelessmordred: ^04:15
lifelessStevenK: cool04:15
lifelessStevenK: got any other ideas?04:15
*** yamahata has joined #tripleo04:17
StevenKlifeless: I was pondering about os-cloud-config and friends04:18
lifelessulp, pickup C time04:21
*** lazy_prince has joined #tripleo04:25
*** morazi has quit IRC04:27
*** yamahata has quit IRC04:48
*** yamahata has joined #tripleo04:49
mordredlifeless: ok. that's what I thought happened04:49
mordredbut  then it was weird04:49
*** nati_ueno has joined #tripleo04:52
adam_ghmph https://bugs.launchpad.net/diskimage-builder/+bug/131198705:03
uvirtbotLaunchpad bug 1311987 in diskimage-builder "deploy-ironic ramdisks do not boot:  Failed to execute /init (error -28)" [Undecided,New]05:03
greghaynesadam_g: was waiting for the big bug to show05:04
adam_ggreghaynes, :)05:04
StevenKI thought it was /sbin/init, is /init an initramfs thing?05:04
greghaynesit says /sbin/init in trace05:05
adam_ggreghaynes, that makes 2 bugs blocking the ironic + devstack functional from fully passing. once it is, i'd like to get it added the pipeline for diskimage-builder to catch those sorts05:05
greghaynesI assume you mean ironic + devtest05:06
*** rwsu has quit IRC05:07
*** nati_ueno has quit IRC05:07
greghaynesadam_g: Thats just running with USE_IRONIC=1?05:08
adam_ggreghaynes, no, not devtest. devstack05:08
greghaynesoh05:08
adam_ghttp://logs.openstack.org/96/87396/8/check/check-tempest-dsvm-virtual-ironic/556a91a05:08
*** nosnos has quit IRC05:08
adam_gits the standard tempest + devstack check /w nova configured to use the ironic driver05:09
lifelessadam_g: is that a trusty ramdisk thing? cause saucy ironic ramdisk works for me05:13
adam_glifeless, yes. saucy works fine for me too05:13
adam_glifeless, the bump to trusty merged last night. noticed busted stuff today05:14
lifelessok05:17
lifelessso that might be a more-systemd thing05:17
StevenKWhy? Trusty is still upstart05:18
*** e0ne has joined #tripleo05:22
*** rwsu has joined #tripleo05:23
StevenKMy devstack-gate change finally passed Jenkins05:23
clarkbStevenK: which one?05:25
lifelessStevenK: hah, systemd-logind for instances05:25
lifelessStevenK: in saucy, replaces pm stuff...05:25
StevenKclarkb: https://review.openstack.org/8643505:25
StevenKlifeless: Sad. Face.05:25
StevenKclarkb: No, ignore me, that isn't the link.05:26
StevenKclarkb: https://review.openstack.org/#/c/89978/05:26
clarkb:P I was going to say, that isn't devstack-gate05:26
*** e0ne has quit IRC05:26
clarkbI am still trying to get sdague's removal of bc through05:26
lifelessStevenK: using occ as an arbitrary message bus for instance?05:26
lifelessdevananda: btw05:26
lifelessdevananda: is occ the agent backplane for ^ or something custom?05:26
StevenKclarkb: Oh, thanks05:27
StevenKlifeless: Hm, that sounds like an idea.05:27
StevenKlifeless: http://paste.openstack.org/show/176EKsX7HUwX4oMqghyd/ is my current abstract. Can you proof-read?05:30
lifelessI don't know05:30
lifelessI'll try05:30
lifelessStevenK: ok; instant thoughts05:31
lifelessthere's no title05:31
lifelessthe first sentence doesn't hook me05:31
lifelessand there's no cohesive theme except lessons-learnt05:31
StevenKlifeless: Title is a seperate field, currently I have "TripleO, git, pip, venvs, tox, oh my"05:31
*** nati_ueno has joined #tripleo05:33
*** nosnos has joined #tripleo05:35
*** saurabhs has left #tripleo05:35
*** eghobo has quit IRC05:39
*** eghobo has joined #tripleo05:39
*** eguz has joined #tripleo05:46
lifelessStevenK: sure, I get that. So what do you think the theme is?05:47
StevenKlifeless: "Lessons learnt, and where to go next" was what I was thinking05:48
*** eghobo has quit IRC05:50
*** nati_ueno has quit IRC05:58
*** jcoufal has joined #tripleo06:02
*** mrunge has joined #tripleo06:04
*** nati_ueno has joined #tripleo06:05
openstackgerritMatthew Macdonald-Wallace proposed a change to openstack/tripleo-image-elements: Install the "classic" icinga interface  https://review.openstack.org/8722306:10
openstackgerritMatthew Macdonald-Wallace proposed a change to openstack/tripleo-image-elements: Install check_mk server  https://review.openstack.org/8722606:11
*** jprovazn has joined #tripleo06:23
*** jeremydei has quit IRC06:31
lifelessStevenK: scars earnt and thumbs ruled?06:34
StevenKlifeless: Heh, yes06:36
* StevenK has finally convinced his laptop to upgrade to Trusty06:38
*** akrivoka has joined #tripleo06:44
*** ramishra has quit IRC06:45
*** ramishra has joined #tripleo06:46
*** rha has joined #tripleo06:47
rcarrillocruzmorning06:51
*** ifarkas has joined #tripleo06:59
*** lsmola has joined #tripleo07:08
*** e0ne has joined #tripleo07:11
*** eguz has quit IRC07:13
*** e0ne has quit IRC07:16
*** lucas-dinner has quit IRC07:17
*** e0ne has joined #tripleo07:17
openstackgerritDmitry Shulyak proposed a change to openstack/tripleo-heat-templates: Introduce configurable virtual ip in templates  https://review.openstack.org/8955607:17
*** bauzas has joined #tripleo07:21
*** ramishra has quit IRC07:24
*** ifarkas has quit IRC07:25
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Include default configuration for Sphinx  https://review.openstack.org/8978007:26
*** ifarkas has joined #tripleo07:26
*** d0ugal has quit IRC07:27
*** olaph has quit IRC07:28
*** d0ugal has joined #tripleo07:29
*** d0ugal has quit IRC07:29
*** d0ugal has joined #tripleo07:29
*** mrunge has quit IRC07:35
*** rpodolyaka has joined #tripleo07:37
rpodolyakamorning07:44
*** rpodolyaka has quit IRC07:47
*** hashar has joined #tripleo07:47
*** rpodolyaka has joined #tripleo07:47
openstackgerritAlexis Lee proposed a change to openstack/diskimage-builder: Map openjdk-7-jre-headless to RHEL+SUSE  https://review.openstack.org/8953107:49
*** andreaf has joined #tripleo07:50
*** akuznetsov has quit IRC07:51
*** akuznetsov has joined #tripleo07:52
*** mrunge has joined #tripleo07:52
*** rcarrill` has joined #tripleo07:53
*** rcarrillocruz has quit IRC07:56
*** andreaf has quit IRC08:06
*** pblaho has joined #tripleo08:11
*** e0ne has quit IRC08:15
*** e0ne has joined #tripleo08:17
*** nosnos has quit IRC08:18
*** derekh has joined #tripleo08:18
*** jistr has joined #tripleo08:21
*** lucasagomes has joined #tripleo08:23
*** darraghb has joined #tripleo08:24
*** athomas has joined #tripleo08:39
*** andreaf has joined #tripleo08:40
*** markmc has joined #tripleo08:42
*** jtomasek has joined #tripleo08:45
*** viktors|afk is now known as viktors08:45
lifelessStevenK: tchaypo: reminder: I'm in town wed through sunday; want to hack together work days?08:45
lifelessStevenK: dinner with sarah would rock; friday night or saturday would work - you choose08:46
*** nati_ueno has quit IRC08:46
lifelesstchaypo: we're likely doing yum cha too sunday am08:46
*** martyntaylor has joined #tripleo08:48
*** e0ne_ has joined #tripleo08:48
*** dkehn has quit IRC08:49
StevenKlifeless: Yes, I do -- where did you want to hack?08:51
*** e0ne has quit IRC08:51
StevenKlifeless: I'll get back to you re: dinner when I meet up with Sarah08:52
lifelessStevenK: lets see; have laptop, can travel. Staying at holiday inn darling harbour.08:52
StevenKlifeless: I can train out to you, you can train out to me, or we can meet at the HP office08:55
*** dkehn has joined #tripleo08:56
lifelessHP might work; but perhaps gatecrashing tchaypo's place might split the difference and have enough tablespace. Dunno - lets get his imput too.08:57
StevenKlifeless: I don't see how, given tchaypo is a lot closer to Darling Harbour :-)08:58
*** untriaged-bot has joined #tripleo09:01
untriaged-botUntriaged bugs so far:09:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/131198709:01
uvirtbotLaunchpad bug 1311987 in diskimage-builder "deploy-ironic ramdisks do not boot:  Failed to execute /init (error -28)" [Undecided,New]09:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/131198609:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/131163109:01
uvirtbotLaunchpad bug 1311986 in diskimage-builder "deploy-ironic ramdisks do not boot:  Failed to execute /init (error -28)" [Undecided,New]09:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/130288109:01
uvirtbotLaunchpad bug 1311631 in tripleo "rabbitmq os-refresh config fails on the master due to mis-parsing of nodes list" [Undecided,In progress]09:01
uvirtbotLaunchpad bug 1302881 in tripleo "incloud CIDR can overlap custom baremetal-network" [Undecided,Incomplete]09:01
*** untriaged-bot has quit IRC09:01
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Defaults TRIPLEO_ROOT to base of the current clone  https://review.openstack.org/8983809:05
StevenKHmmm. Not sure why I have two screensavers post Trusty upgrade.09:07
StevenKlifeless: Friday night for dinner09:08
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Separate Heat BM and VM configs for Nova-BM.  https://review.openstack.org/9003309:08
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Only use SSH keys for power mgmt if using VMs  https://review.openstack.org/8978709:08
openstackgerritVictor Sergeyev proposed a change to openstack/tuskar-ui: Replace Integer to Number in Type check  https://review.openstack.org/8978909:08
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Only use SSH keys for power mgmt if using VMs  https://review.openstack.org/8978709:11
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Only use SSH keys for power mgmt if using VMs  https://review.openstack.org/8978709:12
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Only use SSH keys for power mgmt if using VMs  https://review.openstack.org/8978709:14
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Separate Heat BM and VM configs for Nova-BM.  https://review.openstack.org/9003309:15
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Separate Heat BM and VM configs for Nova-BM.  https://review.openstack.org/9003309:16
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Separate Heat BM and VM configs for Nova-BM.  https://review.openstack.org/9003309:18
*** hashar has quit IRC09:19
lifelessStevenK: rendering at the same time ?09:31
*** giulivo has joined #tripleo09:35
*** jprovazn has quit IRC09:36
openstackgerritLadislav Smola proposed a change to openstack/tuskar-ui: Init code for Instack Undercloud  https://review.openstack.org/8795109:38
*** tzumainn has quit IRC09:45
*** tzumainn has joined #tripleo09:45
*** giulivo has quit IRC09:47
howleytHi, anyone know when the support for scaling of overcloud control nodes should land?09:50
howleytI realise that I can play with scaling myself by adding --scale controller to the make call, and figuring out the correct overcloud-source. yaml Just interested to know how far away it is from landing upstream.09:52
*** hashar has joined #tripleo09:52
*** rcarrillocruz has joined #tripleo09:54
*** rcarrill` has quit IRC09:56
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Only use SSH keys for power mgmt if using VMs  https://review.openstack.org/8978709:57
openstackgerritVictor Sergeyev proposed a change to openstack/tuskar-ui: Replace Integer to Number in Type check  https://review.openstack.org/8978910:10
*** hashar has quit IRC10:10
*** jprovazn has joined #tripleo10:12
*** nosnos has joined #tripleo10:15
*** e0ne_ has quit IRC10:20
openstackgerritstephen-mulcahy proposed a change to openstack/tripleo-incubator: Support custom PyPI mirrors for setup-clienttools  https://review.openstack.org/9004910:32
openstackgerritA change was merged to openstack/tuskar-ui: Replace Integer to Number in Type check  https://review.openstack.org/8978910:34
*** jcoufal has quit IRC10:41
*** darraghb has quit IRC10:48
*** darraghb has joined #tripleo10:48
viktorslifeless: hello10:55
smulcahyis there any obvious issue with https://review.openstack.org/90049 ?10:56
smulcahyIt's not clear if this needs to do anything magical for easy_install edge cases or whether pip options get properly propagated down to underlying tools10:57
openstackgerritLadislav Smola proposed a change to openstack/tripleo-incubator: Adding undercloud ceilometer  https://review.openstack.org/8691710:59
*** e0ne has joined #tripleo11:05
*** e0ne has quit IRC11:10
*** e0ne has joined #tripleo11:12
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Separate Heat BM and VM configs for Nova-BM.  https://review.openstack.org/9003311:12
*** ramishra has joined #tripleo11:34
openstackgerritLadislav Smola proposed a change to openstack/tripleo-incubator: Adding undercloud ceilometer  https://review.openstack.org/8691711:51
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Separate Heat BM and VM configs for Nova-BM.  https://review.openstack.org/9003312:02
*** hashar has joined #tripleo12:07
*** jdob has joined #tripleo12:16
anteayalifeless: I have submitted a pyconau talk proposal, yes12:24
anteayalifeless: thanks for asking12:24
anteayahas everyone in tripleo voted in the tc election?12:24
anteayavote vote vote12:24
anteayaand ask a friend12:24
derekhTwo clouds == twice as much fun, compute node 1 and 0 on rh ci-overcloud down....12:28
derekhHave rebooted compute1, os-collect-config problem, looking into it12:29
*** rlandy has joined #tripleo12:31
openstackgerritA change was merged to openstack/tuskar-ui: Include default configuration for Sphinx  https://review.openstack.org/8978012:38
*** nosnos has quit IRC12:42
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Separate Heat BM and VM configs for Nova-BM.  https://review.openstack.org/9003312:44
Ngmorning12:45
*** weshay has joined #tripleo12:52
*** spzala has joined #tripleo12:54
*** athomas has quit IRC12:58
*** NobodyCam has joined #tripleo12:58
openstackgerritDmitry Shulyak proposed a change to openstack/tripleo-image-elements: Add pacemaker and corosync as tripleo elements  https://review.openstack.org/8606913:01
*** absubram__ has joined #tripleo13:01
*** athomas has joined #tripleo13:07
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Use Tuskar endpoint from Keystone  https://review.openstack.org/9007413:09
*** Rakesh6 has quit IRC13:09
openstackgerritOm Kumar proposed a change to openstack/tripleo-image-elements: Adds local storage Boot support in case PXE boot fails.  https://review.openstack.org/7928913:09
*** dprince has joined #tripleo13:15
*** morazi has joined #tripleo13:19
*** julim has joined #tripleo13:20
*** ramishra_ has joined #tripleo13:21
*** ramishra has quit IRC13:22
*** matty_dubs|gone is now known as matty_dubs13:24
openstackgerritColeman Corrigan proposed a change to openstack/tripleo-image-elements: reset-db to get all db parameters from the config  https://review.openstack.org/8834013:30
*** gilliard_ has joined #tripleo13:31
*** lazy_prince has quit IRC13:35
dprincederekh: So the teworker signals, thing. Thanks for the feedback about my thread leak...13:39
derekhdprince: yup13:39
*** marun has joined #tripleo13:39
gilliard_Hi - what's the state of people's thoughts about CI on baremetal?13:40
gilliard_has there been a discussion on the ML?  I guess there's a lot to worry about w.r.t security.13:40
dprincederekh: Tinkering with it. So how do we feel about using globals in that script? I can either add a few globals, or do a more invasive refactoring to better support Threading for the signals feature13:40
derekhdprince: no prob, I like the idea so if can get it work asap great13:41
openstackgerritGonĂ©ri Le Bouder proposed a change to openstack/tripleo-image-elements: sort the element-deps to please dib-lint  https://review.openstack.org/9008313:41
derekhdprince: I'm good with either, if the globals avoids a big refactor then go for it13:41
dprincederekh: let me push that real quick and see what you think13:42
derekhdprince:13:42
derekhdprince: ok13:42
dprincederekh: Another question for you. Do we have a good way to query what each testenv-worker process is doing? i.e. what Job number, etc it is doing?13:43
*** CLOUDOUTAGE has joined #tripleo13:43
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- both ci-overclouds currently down https://etherpad.openstack.org/p/cloud-outage13:43
*** CLOUDOUTAGE has quit IRC13:43
dprincederekh: I was thinking we could have the testenv-worker modify its cmdline so that the process listing told us more meaningful info in that regard.13:43
dprincederekh: So something like a bin/testenv-ctl would allow us to stop/start/ and query the workers for what they are doing13:44
derekhdprince: it logs what was sent to it from the client, we could have the client send some better info13:45
derekhdprince: and also make it available through testenv-ctl13:45
dprincederekh: exactly. I more asking if something like this would be useful. For me... I think it would help visualize problem jobs where they occur.13:46
*** akuznetsov has quit IRC13:46
dprincederekh: we could for example put the job start time in there too...13:46
derekhdprince: yup, it normall parse the console logs to see where a job ran13:47
derekhdprince: yup, good idea13:47
derekhI'm working on the RH rack if anybody can take a look at HP rack it would be great, looks like we haven't had any passing tests in a while goodsquishy.com/downloads/tripleo-jobs.html13:53
*** vinsh_away is now known as vinsh13:53
*** geerdest has joined #tripleo14:01
*** mrunge has quit IRC14:02
*** akuznetsov has joined #tripleo14:05
*** CLOUDOUTAGE has joined #tripleo14:14
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- both ci-overclouds currently down https://etherpad.openstack.org/p/cloud-outage14:14
*** CLOUDOUTAGE has quit IRC14:14
*** rcarrill` has joined #tripleo14:25
*** rcarrillocruz has quit IRC14:27
openstackgerritA change was merged to openstack/tripleo-incubator: Set admin user tenant to 'admin'  https://review.openstack.org/8730214:27
*** funzo has quit IRC14:27
*** funzo has joined #tripleo14:28
*** funzo is now known as Guest1881814:28
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Add metadata header  https://review.openstack.org/8203814:29
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Add allow_empty to metadata header  https://review.openstack.org/8959714:29
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Allow multiple passes of template evaluation  https://review.openstack.org/8830814:29
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Add mode to metadata header  https://review.openstack.org/8959814:29
openstackgerritA change was merged to openstack/tripleo-image-elements: Add cinder-lio element  https://review.openstack.org/7846314:31
ccrouch1bnemec: \o/14:33
bnemecccrouch1: Indeed :-)14:34
*** jtomasek has quit IRC14:35
*** Guest18818 is now known as funzo14:37
*** TravT has joined #tripleo14:37
Nghmm, cloud outrage eh14:38
janggood lord. A week on holiday and EVERYTHING CHANGES14:42
jangtime to dust off some patches.14:43
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Make time out 60 mins like the Heat default.  https://review.openstack.org/9009914:45
*** CLOUDOUTAGE has joined #tripleo14:45
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- both ci-overclouds currently down https://etherpad.openstack.org/p/cloud-outage14:45
*** CLOUDOUTAGE has quit IRC14:45
*** hashar has quit IRC14:49
openstackgerritDan Prince proposed a change to openstack-infra/tripleo-ci: testenv-worker: use signals to enable/disable jobs  https://review.openstack.org/8797114:53
openstackgerritDan Prince proposed a change to openstack-infra/tripleo-ci: Add testenv stop/start processing scripts.  https://review.openstack.org/8797214:53
*** ramishra_ has quit IRC14:53
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/131198715:00
uvirtbotLaunchpad bug 1311987 in diskimage-builder "deploy-ironic ramdisks do not boot:  Failed to execute /init (error -28)" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/131163115:00
uvirtbotLaunchpad bug 1311631 in tripleo "rabbitmq os-refresh config fails on the master due to mis-parsing of nodes list" [Undecided,In progress]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/130288115:00
uvirtbotLaunchpad bug 1302881 in tripleo "incloud CIDR can overlap custom baremetal-network" [Undecided,Incomplete]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/130902715:00
uvirtbotLaunchpad bug 1309027 in tripleo "rabbitmq nodes repeatedly reporting the master node as being 'down' then 'up'" [Undecided,Incomplete]15:00
*** untriaged-bot has quit IRC15:00
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Add metadata header  https://review.openstack.org/8203815:06
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Add allow_empty to metadata header  https://review.openstack.org/8959715:06
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Allow multiple passes of template evaluation  https://review.openstack.org/8830815:06
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Add mode to metadata header  https://review.openstack.org/8959815:06
openstackgerritKiall Mac Innes proposed a change to openstack/tripleo-image-elements: Create haproxy chroot directory if necessary  https://review.openstack.org/9011015:06
*** ramishra has joined #tripleo15:08
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Add allow_empty to metadata header  https://review.openstack.org/8959715:11
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Allow multiple passes of template evaluation  https://review.openstack.org/8830815:11
openstackgerritAlexis Lee proposed a change to openstack/os-apply-config: Add mode to metadata header  https://review.openstack.org/8959815:11
*** gilliard_ has quit IRC15:12
*** CLOUDOUTAGE has joined #tripleo15:16
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- both ci-overclouds currently down https://etherpad.openstack.org/p/cloud-outage15:16
*** CLOUDOUTAGE has quit IRC15:16
*** ramishra has quit IRC15:20
*** rcarrill` is now known as rcarrillocruz15:25
*** markmc has quit IRC15:25
NobodyCamanyone seeing issues with DIB_REPOREF_*? I'm getting fatal: Couldn't find remote ref refs/changes/07/88307/6 /n fatal: The remote end hung up unexpectedly15:30
NobodyCamall of a sudden15:30
*** dshulyak has quit IRC15:31
*** dshulyak has joined #tripleo15:33
openstackgerritDmitry Shulyak proposed a change to openstack/tripleo-heat-templates: Introduce configurable virtual ip in templates  https://review.openstack.org/8955615:46
*** CLOUDOUTAGE has joined #tripleo15:47
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- both ci-overclouds currently down https://etherpad.openstack.org/p/cloud-outage15:47
*** CLOUDOUTAGE has quit IRC15:47
*** akrivoka has quit IRC15:54
SpamapSwhy does it say both overclouds are down?15:55
SpamapSI see running VMs and jobs?15:55
Ngcrap, I've tried to get into that like 10 times and been distracted by calls15:55
NgSpamapS: the bot just quotes the wikipage15:56
openstackgerritDmitry Shulyak proposed a change to openstack/tripleo-incubator: Add keepalived to overcloud, control vip to heat  https://review.openstack.org/8961315:56
Ngderekh: did the outrage engine just get started up again this afternoon?15:56
SpamapSpossibly15:56
NgI do keep losing my ssh to the undercloud machine, which is profoundly annoying15:59
NgI guess it's my turn to have the shitty networking issue16:00
*** lsmola has quit IRC16:00
openstackgerritPhil Neal proposed a change to openstack/tripleo-incubator: Updating ceilometer endpoint setup  https://review.openstack.org/9013216:04
*** cwolferh has joined #tripleo16:04
openstackgerritPhil Neal proposed a change to openstack/tripleo-incubator: Updating ceilometer endpoint setup on overcloud deploy  https://review.openstack.org/9013216:04
*** ramishra has joined #tripleo16:05
SpamapSNg: I've been getting sporadic ping failures so the insanity continues. :-P16:07
openstackgerritTom Hancock proposed a change to openstack/tripleo-incubator: Dont wait for a failed update  https://review.openstack.org/9013316:07
derekhSpamapS: Ng both were down, I've gotten the RH one back running, but have been on the phone ever sense16:07
derekhSpamapS: Ng the bot was always running but only spams irc when you uncomment the ircmessage:  lline on the therpad16:08
derekhSpamapS: Ng so basically all jobs were failing, but now its most http://goodsquishy.com/downloads/tripleo-jobs.html16:09
Ngah, doh16:09
derekhI'm off the phone now so will start looking into it again but will be leaving in an hour so if anybody else had time it would be great16:09
*** eghobo has joined #tripleo16:11
SpamapSlooks like more compute nodes down16:11
*** bauzas has quit IRC16:12
SpamapS8 again16:12
* derekh is thinking we should just clean slate the hp ci-overcloud , we have done so much messing on these its hard to know whats what16:12
*** hashar has joined #tripleo16:12
SpamapSderekh: agreed16:12
*** csd has joined #tripleo16:12
SpamapSmaybe try to save the db16:12
*** matty_dubs is now known as matty_dubs|lunch16:13
openstackgerritRoman Podoliaka proposed a change to openstack/tripleo-image-elements: Add rally element  https://review.openstack.org/9013516:14
*** viktors is now known as viktors|afk16:14
derekhI think lifeless wanted to finish scripting the whole process, has anybody started this? I got notes from the last time I did one but I wouldn't call it a script16:14
SpamapSheh16:15
SpamapSisn't that.. called TripleO ?16:15
SpamapS"scripting the deployment of OpenStack" ? ;)16:15
SpamapSone might laugh right now and say all we have are tools.. :)16:16
derekhSpamapS: :-)16:17
derekh the post deployment setup of test networks / infra quotas etc...16:17
*** CLOUDOUTAGE has joined #tripleo16:17
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage16:17
*** CLOUDOUTAGE has quit IRC16:17
derekhOh ya, I'm going to start say R1 = HP rack, and R2 = RH rack16:18
*** ramishra has quit IRC16:19
*** pblaho has quit IRC16:19
*** athomas has quit IRC16:19
*** ramishra has joined #tripleo16:22
*** eguz has joined #tripleo16:23
*** newell has joined #tripleo16:24
openstackgerritA change was merged to openstack/tripleo-incubator: Remove superfluous double pound in _variables  https://review.openstack.org/8742516:24
openstackgerritA change was merged to openstack/tripleo-image-elements: Update swift's selinux policies  https://review.openstack.org/8553916:24
*** eghobo has quit IRC16:27
*** athomas has joined #tripleo16:27
SpamapSOk, I'm investigating novacompute8 being down again16:28
SpamapSNg: I think we're going to need to report a ticket on that box. Has locked up hard twice now.16:28
derekhSpamapS: ok, I'll push up a WIP script to prepare a new ci-overcloud before I go (it will not work and will be messy but should be a start)16:29
*** ramishra has quit IRC16:33
openstackgerritDuncan Thomas proposed a change to openstack/tripleo-heat-templates: Make size of loop device used by cinder LVM configurable  https://review.openstack.org/8837716:35
SpamapSwait.. novacompute8 is _not_ locked up16:35
SpamapSit just looks really slow16:35
*** jistr has quit IRC16:40
SpamapSweird.. can't seem to log in to novacompute8 w/ stack user16:42
dprinceSpamapS: are we building w/ that element?16:44
SpamapSdprince: ci-overcloud yes16:45
SpamapSotherwise we'd have no way to login via ILO16:45
SpamapSAFAIK, ssh passwords are still disabled16:45
NgSpamapS: ooi are you losing ssh sessions to the rack, or just getting lag?16:45
Ng(talking to NOC folk atm)16:46
SpamapSOk so I'm seeing general networks uckage16:46
dprinceSpamaps: Right. I use it in my own setups as well. But if for example we override DIB_COMMON_ELEMENTS with a custom value it could be getting dropped.16:46
SpamapSNg: # mtr spamaps.org16:46
SpamapSFailed to resolve host: Name or service not known16:46
dprinceSpamaps: just saying it is worth checking the DIB logs perhaps...16:46
SpamapSoh wait16:46
SpamapSNg: ignore that16:46
*** marun_ has joined #tripleo16:46
*** marun has quit IRC16:47
*** marun_ is now known as marun16:47
SpamapSNg: I am seeing network starts and stops16:47
Ngok16:47
SpamapSNg: so I don't ever lose connection, but I definitely see things slow down or stop working. Seems like general packet loss16:48
Ngfrom here it's consistently stoppy enough that idle sessions are timing out, and interactive lag is getting as high as 5s16:48
SpamapSNg: just experienced it to the bastion16:48
NgSpamapS: ok, thanks. you're a handy test case in that you're by far the closest to the rack :)16:48
*** CLOUDOUTAGE has joined #tripleo16:48
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage16:48
*** CLOUDOUTAGE has quit IRC16:48
NgI am infinity lightparsecs away ;)16:48
SpamapSNg: aye16:48
SpamapSNg: my IRC host is 11 hops away16:49
SpamapSmy home network is 19 hops16:49
*** hashar has quit IRC16:50
SpamapSNg: seeing the effect right now on bastion16:51
SpamapSNg: from my home network which shows _no_ packet loss16:52
NgSpamapS: right, that's the weird thing, no packet loss, but super wonky connections16:53
SpamapSNg: http://paste.openstack.org/show/76970/16:54
NgI'd be guessing that we're going through an overloaded router/switch, but I have basically zero insight into how our network is structured, other than that it's pretty weird16:54
SpamapSNg: 10%16:54
SpamapSNg: and now it is snappy16:54
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: Add script to prepare a ci-overcloud  https://review.openstack.org/9014616:54
derekhlifeless: SpamapS: Ng just a dump of what I have ^^ for ci cloud preparation, will clean it up tomorrow but if you are redeploying you might find it helpful16:55
SpamapSNg: pinging the default route.. I see spikes up to 3-6ms often16:55
SpamapSderekh: I think we're going to diagnose the network first16:55
derekhSpamapS: ok, makes sense16:56
SpamapSNg: 93 packets transmitted, 83 received, 10% packet loss, time 92054ms16:56
SpamapSNg: thats just pinging the default route from bastion16:56
*** shardy is now known as shardy_afk16:57
SpamapSNg: 100 packets transmitted, 82 received, 18% packet loss, time 19958ms16:57
SpamapSrtt min/avg/max/mdev = 0.741/1.786/9.281/1.823 ms16:57
SpamapSNg: definitely something local to the rack16:57
SpamapSwhether it be the switch or the default route16:57
SpamapSor the nics themselves16:58
NgSpamapS: groovy, that's super useful to know16:58
NgI'll be opening a Net ticket just as soon as I can punch my way into Jira16:58
*** jpeeler has quit IRC16:58
*** jpeeler has joined #tripleo16:59
*** jpeeler has joined #tripleo16:59
jangSpamapS: is that on mellanox?16:59
*** matty_dubs|lunch is now known as matty_dubs16:59
mordredNg: while you're filing tickets, you should tell them to put in the cache controller17:00
SpamapSjang: yes, with Trusty kernel17:00
SpamapSjang: which has the 2.x drivers17:00
Ngmordred: the whatnow?17:00
mordredSpamapS: I believe your rack does not have any hw caching on the raid controllers, right?17:01
SpamapSmordred: correct17:01
mordredapparently it's a known thing and can be remedied by a known solution17:01
*** e0ne has quit IRC17:01
mordredit just has to be requested17:01
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Remove quotas on seed and undercloud's nova  https://review.openstack.org/8712617:01
Ngoh, we don't have the bbwc modules?17:01
mordredright17:02
Ngawesome17:02
SpamapSwatching ping -c 100 -i 0.2 $defaultroute run ... it pauses almost exactly 1 time per second17:03
*** darraghb has quit IRC17:04
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Allows user-specified PXE timeout for overcloud  https://review.openstack.org/8660517:06
*** martyntaylor has quit IRC17:07
SpamapSNg: it has gotten much worse now17:08
SpamapShopefully because they're doing something17:09
mordredhey guys - if I put a script in a bin/ dir in an element and I want to use it in an install.d script17:09
mordredis that a thing I do?17:09
mordredbecause it doesn't seem to work that way really17:09
SpamapSmordred: you'd need to install it first17:11
SpamapSmordred: bin is not magical17:11
mordredSpamapS: any examples of doing that I could look at?17:12
NobodyCamhas anyone been able to successfully use DIB_REPOLOCATION_ & DIB_REPOREF_ today?17:12
*** morganfainberg_Z is now known as morganfainberg17:12
SpamapSmordred: install -m 0755 -o root -g root $(dirname $0)/../bin/foo /usr/local/bin17:13
mordredSpamapS: in where? in pre-install.d ?17:13
mordredlike, I'm struggling with the layer context here for a sec?17:14
SpamapSmordred: in install.d17:14
mordredSpamapS: how does install.d see it?17:14
SpamapSmordred: before you use the script, install the script17:14
mordredI mean, by what path can install.d reference the script?17:14
mordredif it's not in the image context already17:14
SpamapS$(dirname $0)/..  <-- the root of your element17:15
*** derekh has quit IRC17:15
SpamapSactually, that is the root of all elements17:15
SpamapSas they're all copied into one space.. but for what you want.. that works17:15
*** lucasagomes has quit IRC17:15
SpamapSNg: bastion seems dead17:16
NgSpamapS: crap, I was about to ask if you still have a shell on it, to get me its default route IP, because I can't log in :/17:16
Ngbut yes, I agree, it seems to be unreachable17:16
SpamapSNg: 138.35.77.117:16
SpamapSNg: it is actually occasionally responding17:16
SpamapSProto Recv-Q Send-Q Local Address               Foreign Address             State17:17
SpamapStcp        0   1392 184.105.137.237:39345       138.35.77.10:22             ESTABLISHED17:17
SpamapSNg: taking about 30s to flush that Send-Q17:17
SpamapSahh and it's back17:17
SpamapS[108211.278851] ipmitool[9540]: segfault at 160 ip 0000000000415265 sp 00007fff2ccf1cb0 error 6 in ipmitool[400000+9a000]17:17
SpamapSNg: ok so I tweaked 'ethtool -K em3 gro off' and everything went to hell.. 'gro on' brought it back17:18
SpamapSNg: so it may be that newer drivers _must_ have GRO17:19
SpamapSI think that is a red herring though.17:19
*** CLOUDOUTAGE has joined #tripleo17:19
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage17:19
*** CLOUDOUTAGE has quit IRC17:19
SpamapSwith it back on.. pings still stalling every few seconds17:19
SpamapScould be the router throttling pings tho17:19
Ngyeah pings are a terrible diagnostic tool17:20
NgI sort of discounted the occasional ipmitool segfaults, they're happening pretty rarely and really shouldn't cause the kernel to fail to network17:20
SpamapSNg: well long Send-Q's suggest it is not ping's fault17:21
SpamapSas in, the router is actually sucking17:21
SpamapSNg: so is NOC looking into things?17:21
SpamapSjang: [   10.461980] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)17:23
SpamapSjang: I believe that is the driver that works o-k, yes?17:23
NgSpamapS: they are17:23
SpamapSNg: ok, I'll lay off for a bit17:24
*** panda has quit IRC17:30
*** panda has joined #tripleo17:31
*** akuznetsov has quit IRC17:32
*** akuznetsov has joined #tripleo17:32
mordredandybody know what an equiv of linux-headers-generic on fedora might be?17:35
*** eguz has quit IRC17:35
*** eghobo has joined #tripleo17:36
greghayneskernel-headers I think17:37
greghayneshttps://apps.fedoraproject.org/packages/s/kernel-headers17:37
greghaynesOnly because I checked before making that comment :p17:37
*** epim has joined #tripleo17:40
openstackgerritBen Nemec proposed a change to openstack/os-apply-config: Use FHS-compliant default template location  https://review.openstack.org/8966817:42
openstackgerritBen Nemec proposed a change to openstack/os-refresh-config: Use FHS-compliant default base directory  https://review.openstack.org/8966717:42
*** jtomasek has joined #tripleo17:43
mordredgreghaynes: thanks!17:43
mordredgreghaynes: (I'm slowly reading and working through your review comments btw - thanks - they were very helful)17:43
mordredgreghaynes: I assume I want to make a dib patch that does the kernel-headers mapping, right?17:43
greghaynesAwesome17:43
greghaynesYep17:43
openstackgerritMonty Taylor proposed a change to openstack/diskimage-builder: Add a mapping for kernel headers  https://review.openstack.org/9016417:45
SpamapSthey're going to bounce the switch R1 is on17:46
* Ng untypes that17:47
Ngexpected downtime of the switch is 3 mins17:47
*** jprovazn has quit IRC17:47
mordredgreghaynes: how do I avoid running 10-support-physical-hardware?17:48
*** CLOUDOUTAGE has joined #tripleo17:50
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage17:50
*** CLOUDOUTAGE has quit IRC17:50
greghaynesmordred: hrmm, looks like wed need to make that configurable somehow. probably by env var17:51
mordredgreghaynes: ok. I can ignore for now - I don't think it's hurting anything17:51
*** nati_ueno has joined #tripleo17:52
mordredgreghaynes: there's a thign like install-packages for ensuring services are/are not running right?17:53
slaglethere's a os-svc-{enable,restart}17:55
openstackgerritDan Prince proposed a change to openstack-infra/tripleo-ci: testenv-worker: use signals to enable/disable jobs  https://review.openstack.org/8797117:55
openstackgerritDan Prince proposed a change to openstack-infra/tripleo-ci: Add testenv-ctl to stop/start job processing  https://review.openstack.org/8797217:55
*** lucas-afk has joined #tripleo17:55
slagledon't think there is a disable or stop though17:55
*** andreaf has quit IRC18:02
*** jcoufal has joined #tripleo18:03
*** e0ne has joined #tripleo18:03
mordred(in this case, I've installed a package, but because $debian_insanity, it's started the service - but the service should not be running)18:05
*** ifarkas has quit IRC18:05
SpamapSmordred: the image build disables invoke-rc.d during image build18:05
mordredok. then  maybe my problem is something else18:06
SpamapSmordred: disabling services is not abstracted between OS's though, so you will need to have a conditional based on the output of dib-init-system18:06
*** absubram__ has quit IRC18:06
mordredwell, I want the service enabled when I boot the image18:06
SpamapSok it will be18:06
mordredI just don't want it running during image build18:06
SpamapSyeah, invoke-rc.d will refuse to start it18:06
*** nati_ueno has quit IRC18:06
mordredgreat.18:06
* mordred tries again18:06
mordredSpamapS: I figure you guys have most likely run in to and dealt with 99% of the things I'm doing, which is why I keep asking question - I pretty much assume there is an answer and I just don't know about it18:09
mordredwith the one exception of that what I really want to do is magically cache all of the things puppet wants to install in teh image and you haven't solved that for me yet :)18:09
*** athomas has quit IRC18:11
SpamapSmordred: why does puppet want to install things after image build?18:12
dprincemordred: you just need to build your images with a local package mirror in them, thats all18:13
*** nati_ueno has joined #tripleo18:13
mordredSpamapS: it doesn't18:14
*** nati_ueno has quit IRC18:14
mordredSpamapS: it wants to install things during image build18:14
mordreddprince: aroo?18:14
mordreddprince: that sounds like perhaps it's a thing I want18:14
SpamapSmordred: ah, so you need something more sophisticated that a squid?18:17
* SpamapS just had an image of a person walking into the book store from Portlandia and asking for a book about cephalapods.. "Ah.. so you need something more sophisticated than a squid?"18:18
*** jcoufal has quit IRC18:19
openstackgerritAdam Gandelman proposed a change to openstack/diskimage-builder: Revert default Ubuntu release back to Saucy  https://review.openstack.org/9017718:19
SpamapSadam_g: nooooooooooooooooooooooooooooooooooooooooooooooooooooooo18:20
adam_gSpamapS, :|18:20
adam_gi'm happy to work on figuring out whats wrong with trusty, but in the meantime...18:20
SpamapSActually that begs the question18:20
SpamapSwhy aren't we building ramdisks in CI?18:20
adam_gSpamapS, im not sure. aside from the trusty issue, im waiting on one merge to devstack before we have a functional ironic devstack gate check that exercises DIB and would have caught that. i think it should be part of the DIB check pipeline18:21
*** CLOUDOUTAGE has joined #tripleo18:21
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage18:21
*** CLOUDOUTAGE has quit IRC18:21
*** e0ne has quit IRC18:21
openstackgerritA change was merged to openstack/tripleo-image-elements: sort the element-deps to please dib-lint  https://review.openstack.org/9008318:22
*** darraghb has joined #tripleo18:22
*** e0ne has joined #tripleo18:22
*** nati_ueno has joined #tripleo18:22
mordredSpamapS: I don't understand what you're saying - are you saying that the answer is "run a squid on your laptop" - or that there is a feature or dib called "local package mirror" that I have not enabled?18:23
SpamapSmordred: run a squid on your laptop18:24
*** epim has quit IRC18:24
SpamapSmordred: but there is a debian-mirror element that will setup an image which, on boot, will build and expose a debian/ubuntu/etc. mirror18:25
*** darraghb has quit IRC18:25
mordrednah - neither of those thigns are what I'm looking for here - but thanks!18:25
*** rcarrill` has joined #tripleo18:25
SpamapSmordred: you want to prepopulate /var/cache/apt ?18:25
mordredyeah18:25
SpamapSmordred: that could very easily be done18:25
mordredyah- I'll work on that once this bad-boy is done18:26
bnemecSo I'm curious - what's our policy on the package mappings for distros not in the gate?18:26
SpamapScleanup.d -> copy the contents somewhere with a name that is locatable using the same arguments18:26
SpamapSextra-data.d -> look for that locatable name and pull it into the image18:26
bnemecFor example, opensuse and rhel both have map-packages too.18:26
mordredSpamapS: yup!18:26
SpamapSbnemec: get 'em in the gate. ;)18:26
openstackgerritKiall Mac Innes proposed a change to openstack/tripleo-image-elements: MySQL element - correct os-svc-restart arguments  https://review.openstack.org/9017918:27
*** epim has joined #tripleo18:27
bnemecSpamapS: Works for me.  Just wanted to make sure I wasn't off base there. :-)18:27
*** rcarrillocruz has quit IRC18:27
*** jml has quit IRC18:28
mordredSpamapS: right now I'm going for functional. once that's done, I'll start working on efficiency boost18:28
SpamapSbnemec: I also think we're not in a position to start kicking things out because they're not in the gate, until we get ourselves _fully_ into the gate and we have stable CI18:28
*** blamar_ has joined #tripleo18:29
SpamapSmordred: hah, we said that a year ago for TripleO ... "we'll optimize.. someday..." ;)18:29
*** blamar_ is now known as blamar18:29
mordredSpamapS: oh - I'll optimize - it'll just likely be saturday :)18:29
bnemecSpamapS: Yeah, as someone who just added support for rhel7 I'm not looking to kick anything out for not being in CI. ;-)18:30
*** darraghb has joined #tripleo18:30
*** derekh has joined #tripleo18:31
NobodyCamfyi reverting this: https://github.com/openstack/diskimage-builder/commit/f8416b1fe5905d8e9796aba51372ce33556bdfb5 seems to have fixed me issues with DIB_REPOLOCATION_ & DIB_REPOREF_18:31
*** jml has joined #tripleo18:31
*** dshulyak_ has joined #tripleo18:32
*** ramishra has joined #tripleo18:34
*** darraghb has quit IRC18:34
openstackgerritA change was merged to openstack/tripleo-image-elements: Move files common for mysql and mariadb under mysql-common  https://review.openstack.org/8687018:36
*** ramishra has quit IRC18:39
*** martyntaylor has joined #tripleo18:47
*** hashar has joined #tripleo18:49
*** nati_ueno has quit IRC18:49
NgNobodyCam: nuts, I really liked that commit18:49
NobodyCamcould just be my env...18:49
NobodyCamif no one else is having issues18:50
NobodyCam:-p18:50
*** sballe has quit IRC18:50
NobodyCambut it broke my ablity to use DIB_REPOLOCATION_ & DIB_REPOREF_18:50
NobodyCamreverting it seems to have things working again ... at least for me18:51
*** martyntaylor has left #tripleo18:51
*** CLOUDOUTAGE has joined #tripleo18:52
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage18:52
*** CLOUDOUTAGE has quit IRC18:52
*** ramishra has joined #tripleo18:55
*** ramishra has quit IRC19:00
*** sballe has joined #tripleo19:00
*** BadCub has joined #tripleo19:02
*** epim has quit IRC19:03
*** StevenK has quit IRC19:07
*** StevenK has joined #tripleo19:07
*** victor_lowther has quit IRC19:08
*** victor_lowther has joined #tripleo19:09
*** blamar has quit IRC19:12
*** CLOUDOUTAGE has joined #tripleo19:23
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage19:23
*** CLOUDOUTAGE has quit IRC19:23
*** nati_ueno has joined #tripleo19:30
*** dshulyak_ has quit IRC19:32
*** saurabhs has joined #tripleo19:36
SpamapSI reset ILO on novacompute8 ... since it has not been working.19:38
*** sdague has quit IRC19:41
*** sdague has joined #tripleo19:42
lifelessrun roh19:43
SpamapSlifeless: yeah, so novacompute8, which I thought was locked up yesterday, just had a very unresponsive ilo19:44
SpamapSlifeless: are ilos their own distinct port or do they share with the mellanox?19:44
Ngdistinct port19:44
SpamapSThe ilo for that box, even after reset, is almost totally unusable19:44
Nglifeless: just to catch you up a bit, we've been having fun with the NOC folk, resetting switches and suchlike, to try and get to the bottom of the weird network issue19:44
SpamapSyou can hit enter 10 times and only one enter will be registered19:44
Ngs/NOC/NetworkEngineering/19:45
lifelessmordred: padakges? I run an apt-mirror mirror19:45
lifelessSpamapS: distinct port and distinct CPU running distinct OS19:45
lifelessok19:46
lifelessso today is a public holiday here; while there are folk on deck I'm going to honour that :)19:46
SpamapSheh, I know what an ilo is. But some models make the ilo share w/ the first nic.19:46
lifelessif at EOD USA its still dead jim I'll jump on it19:46
lifelessSpamapS: sorry, wasn't quite sure of the question :P19:46
SpamapSok I'm on novacompute8 now..19:47
*** hashar has quit IRC19:47
SpamapScan't ping gateway19:48
SpamapSbut can ping some other compute nodes19:48
*** dprince has quit IRC19:51
SpamapShm19:52
SpamapSI wonder if we have a bad NIC19:52
SpamapSlink down/up fixed the problem19:52
SpamapSlifeless: so we were toying with the idea of deploying a new ci-overcloud19:53
*** geerdest has quit IRC19:53
lifelessSpamapS: mellanox driver version on compute8 ?19:53
openstackgerritJufang Wang proposed a change to openstack/tripleo-image-elements: configure keystone with apache  https://review.openstack.org/8974419:53
lifelessSpamapS: yes! new cloud needs finished deployment code because I'm damned if I want another round of manual fixups19:54
lifelessSpamapS: however if you want a close-eyes-its-horrible-but-have-to-do-it we could upgrade to trusty19:54
*** CLOUDOUTAGE has joined #tripleo19:54
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage19:54
*** CLOUDOUTAGE has quit IRC19:54
lifelessSpamapS: ... manually19:54
lifelesswhich is:19:55
lifeless- sudo do-release-upgrade19:55
lifeless- copy the initrd and vmlinuz to kernel & ramdisk of the instance uuid on the undercloud tftproot19:55
lifeless- reboot19:55
SpamapSlifeless: Right.. I would prefer if we could avoid that.19:56
SpamapSlifeless: and also, novacompute8 has old mellanox19:56
*** ramishra has joined #tripleo19:56
lifelessSpamapS: dmesg - and sa_allocs or cmd_alloc or whatever it is we see ?19:57
*** julim has quit IRC19:58
SpamapS[26463.763436] hpsa 0000:06:00.0: cmd_alloc returned NULL!19:59
SpamapSthat is about 20 hours ago19:59
lifelesstaaaa daaaaaa20:00
SpamapSlifeless: I'm more inclined to upgrade mellanox driver manually and focus on how we could actually do a roll out of trusty using our own tools than do-r-u'ing them all20:00
SpamapSlifeless: adam_g tells me that trusty is breaking ironic ramdisk btw20:00
adam_gSpamapS, actually im looking at that now20:01
*** ramishra has quit IRC20:01
derekhlifeless: I started a script to prepare the ci-overcloud earlier I should get it finished tomorrow (more of a brain dump at the moment) https://review.openstack.org/#/c/90146/20:01
adam_gSpamapS, seems to work fine on machines /w >= ~650MB of memory, but failing /w my default 512MB nodes20:01
SpamapSinteresting20:02
SpamapSour ramdisk isn't.. that big is it?20:02
SpamapS-rw-r--r--  1 clint clint  81M Apr 24 11:55 deploy-ramdisk.initramfs20:03
SpamapSnot sure that -28 == 28 == ENOSPC20:03
adam_gSpamapS, is that a trusty or saucy?20:03
SpamapSadam_g: trusty20:03
adam_gSpamapS, re ENOSPC, increasing memory on the VM fixes the issue. so..20:04
SpamapSadam_g: yeah that certainly points to it .. doh20:05
SpamapS[166854.436481] hpsa 0000:06:00.0: cmd_alloc returned NULL!20:06
SpamapSand just got one20:06
*** epim has joined #tripleo20:09
openstackgerritKiall Mac Innes proposed a change to openstack/tripleo-image-elements: MySQL Element - Support Debian libssl1.0.0 paths  https://review.openstack.org/9022320:11
mordredlifeless: if I ran an apt-mirror mirror on my laptop, how would I funnel activity in the image build through that but not have my laptop's apt-mirror thing be configured in the image so that when I boot it in a cloud and not my laptop apt still works?20:12
SpamapSmordred: elements/apt-mirror20:12
SpamapSerrr20:13
SpamapSapt-sources, and apt-conf20:13
SpamapSactually neither of those cleans up20:14
SpamapSmordred: I don't think anybody has made an apt source changer that changes it back to the default after20:15
SpamapSmordred: note that there is considerable effort put into utilizing http_proxy but then not storing it in the image20:16
*** epim has quit IRC20:16
SpamapSok lunch time20:16
mordredSpamapS: yeah - I think this is still too much and not enough at the same time20:18
mordredbtw:20:19
mordredFor APT based systems, if you require specific options for apt operations,20:19
mordredyou can define `DIB_APT_CONF` with your favorite apt.conf to override it,20:19
mordredbefore running devtest.sh.20:19
mordredthat's in elements/apt-conf/README.md20:19
mordredin diskimage-builder20:19
mordredbut devtest is a script in tripleo-incubator - and it's possible people might be using diskimage-builder for other things20:20
mordredjust saying :)20:20
greghayneshah, good point20:20
openstackgerritA change was merged to openstack/diskimage-builder: Disable splashimage for legacy grub.  https://review.openstack.org/8497320:20
*** akrivoka has joined #tripleo20:21
greghayneswhat, no splashimage!?20:22
mordredwow.20:22
Nggerrit needs to give me a +OMGYES for patches that remove serial/textmode issues20:22
SpamapSmordred: Agreed. I tend to -1 those changes if I notice.20:23
mordredSpamapS: http_proxy support really blows20:23
SpamapSmordred: eh?20:23
* mordred just tried to run dib while on the corp network - already giving up and I'll just do it later20:23
mordredoh. wait20:23
mordrednevermind20:23
mordredI actually suck20:23
SpamapSmordred: for corp network, I have two lines in my squid.conf that I have to change20:23
mordredI'm not actually on the corp network20:23
SpamapS# Both of these for times on HP corporate network20:24
SpamapS#cache_peer web-proxy.cup.hp.com parent 8080 0 default20:24
SpamapS#never_direct allow all20:24
SpamapSthe cup.hp.com is actually not needed btw20:24
SpamapSweb-proxy always chooses the closest one20:24
mordredI'm tethered to my phone, but I'm sitting in the office so I set the proxy settings20:24
SpamapSmordred: :)20:24
SpamapSmordred: so what I do is just always have everything on my laptop pointed at my local squid20:24
*** CLOUDOUTAGE has joined #tripleo20:24
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage20:24
*** CLOUDOUTAGE has quit IRC20:24
SpamapSmordred: and then when on corp network, I uncomment those things, and then my local squid goes through corporate evil20:25
SpamapSanyway, tummy rumbling20:25
SpamapSand only 35 minutes of indian buffet left20:25
Ngmordred: devtest runs through tethering? brave :)20:25
SpamapSNg: been there, done htat. ;)20:26
*** scott-millward has joined #tripleo20:27
mordredNg: not devtest20:28
mordredNg: I'm working on getting nodepool to use dib instead of the template/snap method we use right now20:28
mordredso I'm the new crazy customer in the corner with a whole new set of use cases20:28
Ngaha :)20:29
NgSpamapS: Vince is ready to reload the switch fabric. will take 10 mins. I say go for it, what say you?20:29
*** epim has joined #tripleo20:31
mordredok. I AM doing dib tethered through my phone20:34
mordredand, sadly, I think it's working better than my home wifi20:34
pleia2:\20:34
*** weshay has quit IRC20:34
lifelessSpamapS: okydoky20:35
Ngmordred: if only you lived in the centre of the technology universe or something :/20:35
mordredNg: right?20:37
*** lucas-afk has quit IRC20:37
Ngoh, duh, SpamapS went to fill his face with indian food20:38
Ngunilateral decision time!20:38
Ngwe're upgrading the ci cloud's switch fabric :)20:38
*** derekh has quit IRC20:39
*** marun has quit IRC20:40
lifelessNg: schwaeet20:40
clarkbmordred: have you tried not using wireless N? or another AP?20:43
clarkbsometimes there are weird interactions between particular drivers and devices and APs. I once had a router that would hard crash if a particular intel chipset spoke to it on wireless N20:44
Ngthe easy solution is to just buy apple computers and APs ;)20:45
NgI mean HP computers and APs! We probably make APs, right?20:45
lifelessclarkb: many people had that :)20:47
clarkblifeless: it is good to know I wasnt' the only one20:47
lifelessNg: we probably subcontract that to OEMs20:47
Nglifeless: sure, but it increases the chances that someone has actually tested them together20:49
mordredSpamapS: ugh20:50
mordrederr: /Stage[main]/Iptables/Service[iptables]: Failed to call refresh: Could not start Service[iptables]: Execution of '/etc/init.d/iptables-persistent start' returned 1:  at /opt/git/openstack-infra/config/modules/iptables/manifests/init.pp:3220:50
mordredSpamapS: the invoke-rc.d policy thing doesn't prevent my puppet from doing things :(20:50
mordredclarkb: so ... unbound is screwing me in dib - because it's installing, and it's installing an /etc/resolv.conf - but it's not actually running the service20:52
*** jdob has quit IRC20:52
mordredso subsequent operations can't do dns resolution anymore20:52
*** lucas-afk has joined #tripleo20:53
greghaynessounds like unbound element is needed20:53
mordredwell, put we manage unbound via puppet20:53
greghaynesI can see people wanting that20:53
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Add package uninstall support  https://review.openstack.org/9022820:53
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Support declarative package installs/uninstalls  https://review.openstack.org/9022920:53
clarkbmordred: oh right because it says resolve against localhost20:53
mordredthis is the fun place we're in here - is the mixing of tools20:53
mordredclarkb: :)20:53
lifelessmordred: ouch, puppet is being very naughty there20:53
clarkbmordred: maybe you need to have for extra.d run a resolver on localhost?20:53
clarkblifeless: why is that naughty?20:53
mordredclarkb: he's talking about the error above20:54
clarkboh20:54
mordredwhich is fine, I can fix that one20:54
lifelessclarkb: the iptables-persistent thing20:54
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Support declarative package installs/uninstalls  https://review.openstack.org/9022920:54
*** CLOUDOUTAGE has joined #tripleo20:55
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage20:55
*** CLOUDOUTAGE has quit IRC20:55
Ngif nothing else, it's a bit annoying that puppt is calling the init script directly, instead of using service(1) or whatever20:56
Ngat least that could be hobbled during builds20:56
mordredyah. no, that one is one that's easy to fix20:56
mordredthe unbound issue is much harder20:56
mordredbecause I need the node, when it boots up, to be prepped with the unbound - but dib is preventing unbound from running during the rest of image build20:57
*** ramishra has joined #tripleo20:57
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Add package uninstall support  https://review.openstack.org/9022820:57
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Support declarative package installs/uninstalls  https://review.openstack.org/9022920:57
mordredand I could, as greghaynes points out, do an unbound element - and could even parameterized the puppet install of unbound out20:57
mordredbut then I'm managing that in two places20:57
* mordred cries20:57
greghaynesYea.. seems like you really need to do some config post-boot with that one so not sure of a better way :/20:58
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/131198721:00
uvirtbotLaunchpad bug 1311987 in diskimage-builder "deploy-ironic ramdisks do not boot:  Failed to execute /init (error -28)" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/131163121:00
uvirtbotLaunchpad bug 1311631 in tripleo "rabbitmq os-refresh config fails on the master due to mis-parsing of nodes list" [Undecided,In progress]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/130288121:00
lifelesswhat does unbound specifically do that breaks things?21:00
uvirtbotLaunchpad bug 1302881 in tripleo "incloud CIDR can overlap custom baremetal-network" [Undecided,Incomplete]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/130902721:00
uvirtbotLaunchpad bug 1309027 in tripleo "rabbitmq nodes repeatedly reporting the master node as being 'down' then 'up'" [Undecided,Incomplete]21:00
*** untriaged-bot has quit IRC21:00
lifelesslike, shouldn't it be resolvconf integrated, so it dtrt at runtime ?21:00
* lifeless probably isn't helping21:01
lifelessand I need to run21:01
lifelessciao :)21:01
clarkbwe stopped doing resolveconf because it was broken and not on centos21:01
clarkbiirc21:01
clarkbwould need to reread git log21:01
*** ramishra has quit IRC21:02
*** jml has quit IRC21:16
*** jml has joined #tripleo21:17
*** hashar has joined #tripleo21:18
*** akrivoka has quit IRC21:18
*** e0ne has quit IRC21:21
*** yamahata has quit IRC21:24
*** epim has quit IRC21:26
*** CLOUDOUTAGE has joined #tripleo21:27
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage21:27
*** CLOUDOUTAGE has quit IRC21:27
*** epim has joined #tripleo21:30
*** nati_ueno has quit IRC21:33
* SpamapS returns21:37
*** hashar has quit IRC21:40
SpamapSmordred: I think wherever you cross paths with the fairly broken thing that is "puppet does build-y things at run time".. you're going to end up having to do things differently.21:41
hewbroccaand run-y things at build time21:41
hewbroccajust to make matters worse21:41
bnemecgithub recommends I name the new dib-run-parts repo bearded-octo-ninja :-)21:41
SpamapSmordred: the whole point of starting with "not puppet/chef/salt" in TripleO is that we are forced to do build things at build time, and config things at config time.21:41
* bnemec is tempted to take its suggestion21:41
SpamapShewbrocca: that's the worst really21:41
SpamapSbnemec: doooooooiitt21:41
hewbroccait's a horrible nightmare21:42
* hewbrocca no puppet fan21:42
SpamapShewbrocca: worse still, state-y things at build time.21:42
Ngensure => failure21:43
SpamapS#winning21:43
hewbroccaLOL21:44
*** nibalizer has joined #tripleo21:48
nibalizergreghaynes: hi21:48
greghayneshai21:49
mordredSpamapS: right. so, refactoring teh whole system is out of scope21:50
mordredright now, the ONLY thing I'm doing it changing image creation tools21:50
mordredrewriting the puppet is not on the table, for better or for worse21:50
*** e0ne has joined #tripleo21:52
*** derekh has joined #tripleo21:52
SpamapSmordred: you don't have to refactor the whole system. The bits of puppet that are doing build-time things could be separated and deactivated during engine builds though.21:53
mordredSpamapS: right - thign is - we use the unbound module on ALL of our servers21:54
mordredSpamapS: not just the slaves21:54
mordredso changes to it affect thigns that aren't related to dib thigns21:54
SpamapSI'm rusty on my puppet .. but you can conditionally do pieces, right?21:54
SpamapSSo like "if $BUILDING_IMAGE then noop else do as usual " ?21:55
mordredhttps://review.openstack.org/9023421:55
mordredyou mean  like this?21:55
SpamapSyes! :)21:55
mordredI grok what you're saying, and am trying to get it sanely21:55
SpamapSis that even something you could plug into facter?21:55
mordredbut what I DON'T want to do is have a bunch of craptastic puppet with a billion conditionals for if dib / if not dib21:56
mordredbecause that's very hard to reason about21:56
mordredI think this one is potentially not aweful21:56
SpamapSmordred: https://www.youtube.com/watch?v=sw2B9knw58U <-- and now you've arrived at the same conclusion she did..21:57
greghaynesDoes the puppet unbound code actually install the resolv.conf or is that just a result of the unbound package?21:57
*** e0ne has quit IRC21:57
mordredgreghaynes: our puppet module does21:57
mordredso the workaroudn here is either:21:57
greghaynes:( Was thinking maybe you could get clever and install unbound via element before puppet does but there goes that idea21:57
*** ramishra has joined #tripleo21:58
*** CLOUDOUTAGE has joined #tripleo21:58
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage21:58
*** CLOUDOUTAGE has quit IRC21:58
mordreda) block the resolv.conf install in puppet and then follow it up in a 99 install.d script21:58
mordredb) run a local resolver on my laptop so that the resolv.conf file doesn't break anything21:58
derekhmordred: incase it helps, at one stage we were using puppet to build RDO based images with packages, we had to do some weird stuff like swap out certain binaries during image builds to make it work with the openstack puppet modules  https://github.com/agroup/tripleo-puppet-elements/blob/master/elements/puppet/install.d/99-zuppet#L2221:58
mordredsince the nodepool server where this will run for real will have a resolver21:58
hewbroccafortunatly then we stopped doing that21:59
*** ramishra has quit IRC22:02
SpamapSmordred: a)... except, no cleanup.. wouldn't puppet run at first boot and make things the way they should be?22:04
*** geerdest has joined #tripleo22:05
*** saurabhs has quit IRC22:05
SpamapSmordred: that's why I was wondering about facter. If we could have a facter plugin which just set a variable "building_image" and then we remove that facter plugin.. your conditionals become very straight forward "this is something for doing / not doing at build time.22:05
SpamapSmordred: instead of 1000 different conditionals.. you just have one.. sprinkled through the puppet where it makes sense.22:06
mordredSpamapS: a) no. we dont' run puppet on boot at all22:06
mordredSpamapS: we only run puppet at image build time22:06
SpamapSoh lol of course22:06
mordredand never again22:06
SpamapSbtw, have I told you that w/ the Ubuntu font, lol looks like a ghost chasing somebody  ...     lol===---...22:07
SpamapS...---===/o/    <-- running away22:07
mordredthe main thing is taht one of the reasons we're doing that is that we've gota  bunch of base modules we need on the slaves AND on our long-lived servers whicha re puppet agent managemd22:07
SpamapSmordred: ok so you're just dealing with an entirely different build environment, not a difference between build/runtime.22:07
mordredthat's right22:08
* SpamapS gaining sense22:08
mordredso if we start putting conditionals in those, we're increasing complexity in other areas22:08
SpamapSthen yes, the cleanup at the end makes sense. Or maybe same thing, facter plugin, but you just run puppet again w/o the build variable set.22:08
*** yamahata has joined #tripleo22:09
*** saurabhs has joined #tripleo22:10
*** ccrouch1 has quit IRC22:10
mordredmeh - that would require writing ruby22:10
hewbrocca(now you have 2 problems)22:10
clarkbif you do that when do you puppet the extra stuff?22:11
clarkbdefeats the purpose to run puppet post boot22:11
SpamapSmordred: 3 lines or so22:11
SpamapSmordred: or you can use an executable with facter-customfacts-plugin22:12
SpamapSclarkb: if you do what?22:13
clarkbif you distinguish between build and not build22:13
clarkbwhen does not build puppet run?22:13
SpamapSclarkb: end of build :)22:13
clarkbso you do two passes? thats ugh22:14
SpamapSpost-install.d/99-zpuppet-postbuild22:14
clarkbnot worried about the dib side22:14
clarkbworried about the puppet side22:15
SpamapSclarkb: that's necessary because puppet isn't in control of everything.22:15
*** e0ne has joined #tripleo22:16
mordredclarkb: what are your thoughts on the resolver?22:17
mordredclarkb: I'm not sure what to do locally with that that won't screw network-manager22:17
mordredand I'd rather not have to say "you have to completely re-work your ubuntu to be able to build an infra slave image"22:17
*** e0ne_ has joined #tripleo22:18
clarkbmordred: nc port forward?22:19
clarkbthough that requires root, but dib does too right?22:19
SpamapSdib requires sudo22:20
clarkb`nc -lk -p 53 127.0.1.1 53` something like that may work?22:20
clarkbnot for udp though22:20
SpamapSwait what?22:20
clarkbSpamapS: you have a local resolver22:20
derekhfwiw we also has to set a factor fact to differentiate between build and run time puppet, the runtime puppet ran in a os-refresh-config script https://github.com/agroup/tripleo-puppet-elements/blob/master/elements/puppet/common.pp22:21
clarkbyou just need the chroot to ee it on 127.0.0.122:21
*** ccrouch has joined #tripleo22:21
*** e0ne has quit IRC22:21
SpamapSyeah just .. that seems evil.. ;)22:21
mordredderekh: yeah - we're trying very hard to not refactor our puppet at all right now22:21
SpamapSwell..22:21
clarkbmordred: ya no refactor is good *and* I think its a bug to runtime puppet22:22
SpamapSyour build host is the problem I think.. not dib per se22:22
mordredmy build host is my laptop22:22
mordredso this is what I'm saying though22:22
mordredmy build host is bog-standard trusty22:22
SpamapSdib wants unfettered sudo...22:22
mordredit has it22:22
*** e0ne_ has quit IRC22:22
SpamapSjust surprised that is o-k ;)22:23
mordredSpamapS: we don't have a choice do we?22:23
SpamapSRun in a throw-away build VM.22:23
mordredok. hang on22:23
mordredseriously22:23
derekhmordred: yup, fair enough, swapping binaries to make puppet thing it was starting things during image build was how we handled things we could refactor in the puppet modules, then swaped them back after image build22:23
SpamapSThat's the other choice.22:23
mordredthis has gotten truly ludicrou22:23
mordredludicrous22:23
mordredI want to build a glance image22:23
derekh*think22:23
mordredon my laptop22:24
mordredI do NOT want to have to spin up a VM to do that22:24
mordredbecause that's just the most ridiculous thing I've ever heard22:24
SpamapSmordred: just spelling out that you're giving the glance elements root on your box.22:24
mordredwhat?22:24
mordredI'm not doing tripleo22:24
SpamapSoh sorry22:24
* SpamapS OTP22:24
* mordred throws wet cat at SpamapS22:25
mordredderekh: nod22:25
*** lucas-afk has quit IRC22:25
*** CLOUDOUTAGE has joined #tripleo22:29
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage22:29
*** CLOUDOUTAGE has quit IRC22:29
*** epim has quit IRC22:30
*** epim has joined #tripleo22:38
greghaynesSpamapS: Im trying to grab a network uuid in heat, I imagine its some sub-property of {"Fn::Select": [ "ctlplane", {"Fn::GetAtt": [controller0, networks]} ]}... any idea how I can inspect the resource tree to see what that actually returns?22:42
*** derekh has quit IRC22:45
openstackgerritRichard Su proposed a change to openstack/tripleo-image-elements: Fix /var/lib/mysql selinux labeling  https://review.openstack.org/9024522:53
*** epim has quit IRC22:58
*** ramishra has joined #tripleo22:58
*** CLOUDOUTAGE has joined #tripleo23:00
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud currently down https://etherpad.openstack.org/p/cloud-outage23:00
*** CLOUDOUTAGE has quit IRC23:00
*** dkehn_ has joined #tripleo23:00
*** clarkb has quit IRC23:01
*** clarkb has joined #tripleo23:01
*** dkehn has quit IRC23:03
*** dkehn has joined #tripleo23:03
*** ramishra has quit IRC23:03
*** dkehnx has quit IRC23:04
*** lifeless has quit IRC23:04
*** CLOUDOUTAGE has joined #tripleo23:04
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud partially down https://etherpad.openstack.org/p/cloud-outage23:04
*** CLOUDOUTAGE has quit IRC23:04
*** epim has joined #tripleo23:05
*** lifeless has joined #tripleo23:05
SpamapSaaand novacompute9 paniced23:08
*** CaptTof__ has quit IRC23:08
SpamapSmellanox..23:08
SpamapSgrah23:08
*** CaptTofu has joined #tripleo23:09
*** epim has quit IRC23:11
*** CaptTofu has quit IRC23:14
*** e0ne has joined #tripleo23:18
openstackgerritA change was merged to openstack/diskimage-builder: indent: replace tab by 4 spaces  https://review.openstack.org/8853023:19
*** eguz has joined #tripleo23:19
*** eguz has quit IRC23:19
*** ccrouch has quit IRC23:22
*** e0ne has quit IRC23:22
*** eghobo has quit IRC23:23
*** saurabhs has left #tripleo23:27
*** CaptTofu has joined #tripleo23:29
SpamapSmmm fresh new devtest23:34
SpamapS[2014-04-24 16:03:01] Total runtime: 5012 s23:34
SpamapS[2014-04-24 16:03:01]   ramdisk    : 136 s23:34
SpamapS[2014-04-24 16:03:01]   seed       : 1041 s23:34
SpamapS[2014-04-24 16:03:01]   undercloud : 1189 s23:34
SpamapS[2014-04-24 16:03:01]   overcloud  : 2614 s23:34
*** CLOUDOUTAGE has joined #tripleo23:35
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- R1 ci-overcloud partially down https://etherpad.openstack.org/p/cloud-outage23:35
*** CLOUDOUTAGE has quit IRC23:35
*** e0ne has joined #tripleo23:43
*** e0ne has quit IRC23:47
*** eghobo has joined #tripleo23:55
*** yamahata has quit IRC23:57
*** ramishra has joined #tripleo23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!