Tuesday, 2013-12-03

*** rongze has quit IRC00:00
*** NobodyCam_afk is now known as NobodyCam00:04
*** jayg is now known as jayg|g0n300:04
*** cd-undercloud has joined #tripleo00:27
cd-undercloud************** overcloud complete status=1 ************00:27
*** cd-undercloud has quit IRC00:27
*** matsuhashi has joined #tripleo00:29
*** CaptTofu has quit IRC00:38
*** CaptTofu has joined #tripleo00:39
*** bcrochet is now known as bcrochet|whodat00:45
*** bcrochet|whodat is now known as bcrochet00:45
*** sdake_ has quit IRC00:47
*** sdake has quit IRC00:47
*** matsuhashi has quit IRC00:50
*** matsuhashi has joined #tripleo00:50
*** michchap_ has quit IRC00:53
*** matsuhas_ has joined #tripleo00:53
*** michchap has joined #tripleo00:53
*** matsuhashi has quit IRC00:54
*** krotscheck has quit IRC00:54
*** marun has joined #tripleo00:55
lifelessif anyone needs me, I'm at the doctors - ring01:01
*** marun has quit IRC01:02
*** sdake has joined #tripleo01:02
*** sdake has quit IRC01:15
openstackgerritJun Jie Nan proposed a change to openstack/os-refresh-config: `os-refresh-config` README enhancement  https://review.openstack.org/5931801:19
*** nosnos has joined #tripleo01:26
*** rongze has joined #tripleo01:27
*** cd-undercloud has joined #tripleo01:27
cd-undercloud************** overcloud complete status=1 ************01:27
*** cd-undercloud has quit IRC01:27
*** rongze has quit IRC01:31
*** jcooley_ has joined #tripleo01:37
*** CaptTofu has quit IRC01:42
*** CaptTofu has joined #tripleo01:43
*** jcooley_ has quit IRC01:45
*** kui has joined #tripleo01:46
*** arata has joined #tripleo01:54
*** jcooley_ has joined #tripleo02:01
*** jcooley_ has quit IRC02:02
*** rongze has joined #tripleo02:06
*** vkozhukalov has joined #tripleo02:19
*** CaptTofu has quit IRC02:21
*** CaptTofu has joined #tripleo02:21
lifelessback02:27
*** cd-undercloud has joined #tripleo02:28
cd-undercloud************** overcloud complete status=1 ************02:28
*** cd-undercloud has quit IRC02:28
*** matsuhas_ has quit IRC02:31
*** matsuhashi has joined #tripleo02:31
*** sdake has joined #tripleo02:32
*** ccrouch has joined #tripleo02:50
*** sdake_ has joined #tripleo02:56
*** sdake_ has quit IRC02:56
*** sdake_ has joined #tripleo02:56
*** CaptTofu has quit IRC02:56
*** CaptTofu has joined #tripleo02:57
*** nosnos has quit IRC02:58
*** nosnos has joined #tripleo02:59
*** cwolferh has quit IRC03:27
*** jcooley_ has joined #tripleo03:29
*** cd-undercloud has joined #tripleo03:31
cd-undercloud************** overcloud complete status=1 ************03:31
*** cd-undercloud has quit IRC03:31
*** matsuhashi has quit IRC03:42
*** sdake has quit IRC03:46
*** sdake has joined #tripleo03:46
*** sdake has quit IRC03:46
*** sdake has joined #tripleo03:46
*** arata has left #tripleo03:55
openstackgerritA change was merged to openstack/tripleo-incubator: Use cleanup_flavor in setup-baremetal.  https://review.openstack.org/5951204:09
*** michchap has quit IRC04:21
*** michchap has joined #tripleo04:21
*** matsuhashi has joined #tripleo04:25
*** rongze has quit IRC04:31
*** CaptTofu has quit IRC04:34
*** cd-undercloud has joined #tripleo04:35
cd-undercloud************** overcloud complete status=1 ************04:35
*** cd-undercloud has quit IRC04:35
*** CaptTofu has joined #tripleo04:35
*** matsuhashi has quit IRC05:00
*** jcoufal has joined #tripleo05:07
*** matsuhashi has joined #tripleo05:08
*** rushiagr has joined #tripleo05:14
openstackgerritA change was merged to openstack/diskimage-builder: Drop default distribution root element support  https://review.openstack.org/5860105:20
openstackgerritlifeless proposed a change to openstack/diskimage-builder: Revert "Store DIB_* only env variables"  https://review.openstack.org/5962105:22
openstackgerritlifeless proposed a change to openstack/diskimage-builder: Revert "Store DIB_* only env variables"  https://review.openstack.org/5962205:25
*** matsuhashi has quit IRC05:27
*** rpodolyaka1 has joined #tripleo05:36
lifelessrpodolyaka1: hi!05:37
*** cd-undercloud has joined #tripleo05:38
cd-undercloud************** overcloud complete status=1 ************05:38
*** cd-undercloud has quit IRC05:38
rpodolyaka1lifeless: morning!05:39
rpodolyaka1morning all05:39
rpodolyaka1lifeless: it's hard to intersect with you online these days :)05:40
rpodolyaka1lifeless: ok, so on nova-bm... I thought we were going to have some flag passed to rebuild API method and then down the stack to work_on_disk()05:41
rpodolyaka1but it seems to be odd, as I have to pass it as a kernel parameter, and then return it back to baremetal deploy helper05:41
rpodolyaka1and there seems to be no other way to do this except putting it to one of DB tables05:42
rpodolyaka1(I mean, baremetal-deploy-helper is a separate process, that just listens on 10000 for HTTP POSTs from nodes)05:43
rpodolyaka1on the other hand, having a flag is ok05:43
rpodolyaka1but when do we want to set it? in the handler of API rebuild method? or when a baremetal node is created (so all its rebuilds will preserve ephemeral)?05:44
lifelessrpodolyaka1: so nearly05:44
lifelessrpodolyaka1: API -> driver; driver -> DB; then deploy helper with the changes I've put up05:45
lifelessrpodolyaka1: so you're like 99% right05:45
rpodolyaka1ok, so are we postponing libvirt support a bit? and what the 1% left I am missing? :)05:46
rpodolyaka1btw, your comment on possible race condition - good point. I was thinking about that too05:47
lifelessrpodolyaka1: so the 1% a) needing a rebuild method on the driver to avoid the race condition and b) using the DB rather than kernel parameter tunneling05:48
rpodolyaka1ok. I think, we could for some kind of driver "extension". I.e. check if a driver supports 'native rebuild', call it, if not, do the existing procedure of destroy() + spawn()05:50
rpodolyaka1basically, change as little code as possible05:50
lifelessrpodolyaka1: huh? I'd do it in the base class05:50
lifelessrpodolyaka1: there is 20 LOC right ?05:50
lifelessrpodolyaka1: so we make a new base driver method rebuild(...) and we put those 20 LOC in there.05:50
rpodolyaka1which base do you mean? driver in virt?05:50
SpamapShm05:50
lifelessvirt.driver05:50
SpamapSno dnsmasq for neutron-dhcp-agent again05:51
lifelessrpodolyaka1: then, we override that baremetal; probably with one helper for the code between destroy and spawn05:51
rpodolyaka1lifeless: (precoffee)05:52
rpodolyaka1lifeless: yeah, seems reasonable05:52
lifelessthe rebuild method on driver would accept preserve_ephemeral as a parameter05:53
lifelessso that avoids the race05:53
lifelessand lets us set the preserve value in the DB + initiate a deploy05:53
SpamapSdnsmasq is segfaulting05:54
SpamapSwrite(11, "<30>Dec  3 05:54:31 dnsmasq-dhcp[29600]: DHCPOFFER(tap56efc4c6-55) 10.10.16.171 "..., 98) = 9805:54
SpamapS--- SIGSEGV (Segmentation fault) @ 0 (0) ---05:54
lifelessrotfl05:54
lifelessit's never easy05:55
SpamapSneutron-dhcp-agent never notices05:55
rpodolyaka1lifeless: ack05:55
rpodolyaka1I know, our deployers used some patches for dnsmasq05:55
rpodolyaka1not sure if they were related to segfaults though05:55
lifelessrpodolyaka1: if we get pushback on that from nova-core, we can discuss it in more detail with them05:56
lifelessrpodolyaka1: who would have those patches ?05:56
lifelessSpamapS: filing a bug on Neutron ?05:56
rpodolyaka1lifeless: sure, going to hack on this week, was side tracked mostly all the time last week :(05:57
rpodolyaka1lifeless: I meant deployers here at mirantis, I'm going to ask one of my colleagues about problems they had with dnsmasq05:58
lifelessrpodolyaka1: yeah, I figure thats what you meant :)05:59
*** matsuhashi has joined #tripleo05:59
rpodolyaka1oh, drop default distribution root element support is merged to diskimage-builder06:01
rpodolyaka1was waiting for it before doing a release of dib06:01
rpodolyaka1btw, in case I don06:02
rpodolyaka1don't make tonights meeting, I'm taking releases for the next week too :)06:03
lifelessrpodolyaka1: awesome, thanks!06:03
SpamapSlifeless: this is the problem with having a frozen undercloud :-P06:05
SpamapSlifeless: reporting bugs on pre-havana code base... dicey06:05
rpodolyaka1and on neutron bugs. the fix for policies bug was merged (though it is stated that it is partial in the commit message), and the one for multiple agents entries hasn't been reviewed yet. Review time in neutron is frustrating :(06:05
lifelessMedian wait time: 5 days, 20 hours, 31 minutes06:06
lifeless3rd quartile wait time: 20 days, 15 hours, 57 minutes06:06
lifelessdkehn: hey, if you're not already, I suggest you start doing two code-reviews a day in Neutron06:07
openstackgerritJun Jie Nan proposed a change to openstack/os-refresh-config: `os-refresh-config` README enhancement  https://review.openstack.org/5931806:08
lifelessdkehn: thats would put you at the same rate or more, than most of neutron-core, and should [once you're up to speed on the local culture etc] significantly help Neutron.06:08
*** arata has joined #tripleo06:15
*** noslzzp has quit IRC06:25
*** jpeeler has quit IRC06:34
*** jpeeler has joined #tripleo06:37
*** vkozhukalov has quit IRC06:47
*** tzumainn has quit IRC06:58
*** sdake_ has quit IRC07:09
*** toci-bot has joined #tripleo07:11
toci-botERROR during toci run, see http://54.228.118.193/toci/toci_logs_P7VbroC/07:11
*** toci-bot has quit IRC07:11
lifelessruh roh07:14
*** lsmola has joined #tripleo07:16
*** akuznetsov has quit IRC07:17
*** vipul has quit IRC07:32
*** vipul has joined #tripleo07:33
*** jcooley_ has quit IRC07:35
*** jcooley_ has joined #tripleo07:36
openstackgerritJun Jie Nan proposed a change to openstack/diskimage-builder: Modify /etc/selinux/config if it exists  https://review.openstack.org/5964207:41
*** jprovazn has joined #tripleo07:43
*** rpodolyaka1 has quit IRC07:51
GheRiveromorning TripleO07:53
*** jtomasek has joined #tripleo07:54
shadowerhey07:54
openstackgerritJun Jie Nan proposed a change to openstack/os-apply-config: os-apply-config readme enhancement  https://review.openstack.org/5932007:59
openstackgerritJun Jie Nan proposed a change to openstack/os-apply-config: os-apply-config readme enhancement  https://review.openstack.org/5932008:00
openstackgerritJun Jie Nan proposed a change to openstack/os-apply-config: os-apply-config readme enhancement  https://review.openstack.org/5932008:04
*** jtomasek has quit IRC08:07
*** jtomasek has joined #tripleo08:09
lifelessshadower: GheRivero: o/08:10
shadowerhey lifeless08:12
GheRiverohey lifeless and shadower08:13
*** pblaho has joined #tripleo08:20
*** rdopieralski has joined #tripleo08:24
*** vkozhukalov has joined #tripleo08:27
*** jcooley_ has quit IRC08:36
*** jcooley_ has joined #tripleo08:38
*** pblaho has quit IRC08:40
*** akrivoka has joined #tripleo08:45
*** jistr has joined #tripleo08:48
*** derekh has joined #tripleo08:57
*** rpodolyaka has quit IRC09:02
*** lynxman has quit IRC09:02
*** boris-42 has joined #tripleo09:03
*** rpodolyaka has joined #tripleo09:05
*** max_lobur_afk is now known as max_lobur09:14
Ngmorning09:18
*** martyntaylor has joined #tripleo09:19
*** rushiagr has quit IRC09:28
*** rushiagr has joined #tripleo09:28
*** athomas has joined #tripleo09:29
*** lucasagomes has joined #tripleo09:32
*** jcooley_ has quit IRC09:34
*** lynxman has joined #tripleo09:43
matelHi TripleO, what is the best starting point to understand how bare metal deployment happens? (AFAIK it's exporting the disk as iSCSI) Is there any documentation around that?09:47
*** markmc has joined #tripleo09:48
*** pblaho has joined #tripleo09:48
SpamapSlifeless: you working on cd-undercloud at all?09:49
*** kushi_ has joined #tripleo09:50
*** matsuhashi has quit IRC09:51
*** martyntaylor1 has joined #tripleo09:52
*** kui has quit IRC09:52
*** martyntaylor has quit IRC09:53
openstackgerritJun Jie Nan proposed a change to openstack/os-apply-config: Converted README to .rst format  https://review.openstack.org/5932009:54
*** matsuhashi has joined #tripleo09:54
*** kushi_ has quit IRC09:55
SpamapSlifeless: anyway, I think dnsmasq was segfaulting becuase there were no nameserver's in the host resolv.conf09:58
SpamapSnope.. still segfaulting10:02
SpamapSwtf10:02
lifelessSpamapS: I am not10:02
SpamapSlifeless: I'm a bit out of ideas for the segfaulting dnsmasq. Thoughts before I build one with debugging symbols and attach gdb?10:04
*** jcooley_ has joined #tripleo10:05
lifelessSpamapS: retracer?10:11
lifelessSpamapS: enable core files and retrace with the detached symbols Ubuntu builds for every package10:12
lifelessSpamapS: you can do that offline10:12
lifelessSpamapS: or do an apport run - I bet there are crashdumps just pending10:12
openstackgerritA change was merged to openstack/diskimage-builder: Modify /etc/selinux/config if it exists  https://review.openstack.org/5964210:12
*** jcooley_ has quit IRC10:12
*** boris-42 has quit IRC10:13
SpamapSlifeless: ubuntu does not actually build those for all packages. They try.10:13
SpamapSlifeless: I do have a core file already yes10:13
SpamapSlifeless: anyway, dnsmasq doesn't have a dbgsym package10:13
*** marun has joined #tripleo10:20
*** pblaho has quit IRC10:21
*** pblaho has joined #tripleo10:22
* SpamapS runs debsums just in case it is as simple as corrupted binaries10:24
lifelessSpamapS: uhm, you know about the dedicated archive with the symbols in it right10:27
lifeless?10:27
SpamapSlifeless: ddebs.ubuntu.com yes10:27
*** rongze has joined #tripleo10:28
lifelessSpamapS: ack, ok10:28
SpamapSlifeless: no dnsmasq unfortunately.10:28
SpamapSlifeless: probably means it is stripped by its own build process10:28
lifelessnuts10:29
lifelessbug filin' time10:29
lifelessalso what are you doing up at 4am?10:29
SpamapSlifeless: 2:3110:31
lifelessso much better10:31
SpamapSlifeless: drank a diet coke at 20:00 .. it is starting to wear off10:31
lifelessah pity the fool10:32
lifelessnight10:32
SpamapSthis is just eating away at my brain :-P10:32
*** akrivoka1 has joined #tripleo10:32
*** jcoufal has quit IRC10:34
*** akrivoka has quit IRC10:35
*** rongze has quit IRC10:46
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Add element to install a testenv worker  https://review.openstack.org/5830510:48
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Add element to install a testenv client  https://review.openstack.org/5830610:48
*** rdopieralski has quit IRC10:50
SpamapSDec  3 10:52:49 undercloud-notcompute-jws3awlsb2kh dnsmasq[7636]: bad IP address at line 8 of /var/run/neutron/dhcp/e85b44c7-1136-4217-954e-cdf0acdddfe1/opts10:53
SpamapS_hm_10:53
Ngthat sounds unfun10:54
SpamapSNg: fyi current problem is dnsmasq segfaults after DHCPOFFER10:54
SpamapSI have a backtrace, and I'm now trying trusty's dnsmasq to see if it fixes the problem10:55
Ng*wat*10:56
Ngthat's pretty crazy :)10:56
SpamapSyeah10:57
SpamapSwill wait through one more DHCP cycle then sleeeep10:57
SpamapSah, fixed10:59
SpamapSdnsmasq (2.67-1~ubuntu12.10.1) quantal-backports; urgency=low11:00
SpamapS  * No-change backport to quantal11:00
SpamapS -- Clint "Entropy" Byrum <clint@undercloud-notcompute-jws3awlsb2kh.novalocal>  Tue, 03 Dec 2013 10:51:17 +000011:00
*** rongze has joined #tripleo11:03
*** jcooley_ has joined #tripleo11:05
*** jcooley_ has quit IRC11:10
*** kui has joined #tripleo11:17
vkozhukalovHey all, can anybody know if tripleo-ci still alive? We want to create functional (aka integration) testing scheme for ironic based on emulated bare metal. Is there any activities connected to such a task?11:21
*** boris-42 has joined #tripleo11:27
vkozhukalovAnd another question: is there any documentation or something about emulated baremetal?11:28
*** sheep_ has joined #tripleo11:28
*** sheep_ is now known as rdopieralski11:29
rpodolyakavkozhukalov: hey, yep, tripleo-ci is alive11:30
rpodolyakavkozhukalov: but I would suggest to start from tripleo devtest story11:30
rpodolyakavkozhukalov: which is run on emulated baremetal11:30
rpodolyakavkozhukalov: though, you can tweak it to run on real baremetal11:31
*** jcoufal has joined #tripleo11:31
vkozhukalovrpodolyaka, thanks.11:32
vkozhukalovrpodolyaka, will have a look11:32
rpodolyakavkozhukalov: currently, tripleo-ci is duplicating devtest a lot, but I believe, the idea is to reuse devtest scripts in tripleo-ci11:32
rpodolyakavkozhukalov: http://docs.openstack.org/developer/tripleo-incubator/devtest.html11:32
rpodolyakavkozhukalov: https://github.com/openstack/tripleo-incubator     (the doc is generated from devtest*.sh scripts)11:34
vkozhukalovrpodolyaka, do you know why we need shell scripts to make VMs instead of using openstack itself (compute)?11:36
vkozhukalovrpodolyaka, at the moment compute can not boot VMs via PXE. I think this is the only problem.11:37
rpodolyakavkozhukalov: sort of. we are using SSH power driver of nova baremetal in order to do provisioning of nodes11:38
*** rongze has quit IRC11:39
*** jtomasek has quit IRC11:39
*** jtomasek has joined #tripleo11:39
vkozhukalovrpodolyaka, ok. FYI there is the blueprint https://blueprints.launchpad.net/nova/+spec/libvirt-empty-vm-boot-pxe connected with this task11:40
rpodolyakavkozhukalov: cool! I'm looking forward for this to be approved and implemented then ;)11:41
*** akrivoka1 has quit IRC11:43
*** arata has left #tripleo11:44
*** akrivoka has joined #tripleo11:44
*** agordeev has joined #tripleo11:45
*** jcooley_ has joined #tripleo11:51
GheRiverovkozhukalov: any reason to implement that blueprint in nova and not ironic? nova-bm is going to be deprecated11:51
GheRiverovkozhukalov: got it11:52
*** rongze_ has joined #tripleo11:56
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Add element to install a testenv client  https://review.openstack.org/5830611:58
vkozhukalovGheRivero, just to make it more clear. We already have advanced functionality in nova to make VMs. So instead of using shell scripts (devtest or tripleo-ci) and SshPower driver to deal with emulated baremetal VMs we could use nova for that. And the only obstacle here is that nova can not boot VMs via PXE.12:03
*** cd-undercloud has joined #tripleo12:04
cd-undercloud************** overcloud complete status=1 ************12:04
*** cd-undercloud has quit IRC12:04
*** boris-42_ has joined #tripleo12:05
*** max_lobur has quit IRC12:05
*** rpodolyaka has quit IRC12:05
*** max_lobur has joined #tripleo12:05
*** rpodolyaka has joined #tripleo12:06
*** boris-42 has quit IRC12:08
*** CaptTofu has quit IRC12:08
*** CaptTofu has joined #tripleo12:08
openstackgerritA change was merged to openstack/os-apply-config: Converted README to .rst format  https://review.openstack.org/5932012:09
*** jcooley_ has quit IRC12:25
*** boris-42_ has quit IRC12:27
*** lsmola has quit IRC12:30
*** morazi has joined #tripleo12:32
*** boris-42 has joined #tripleo12:33
*** akuznetsov has joined #tripleo12:38
*** lucasagomes is now known as lucas-hungry12:44
*** lsmola has joined #tripleo12:47
*** rbrady has joined #tripleo12:54
*** SpamapS has quit IRC12:59
*** noslzzp has joined #tripleo13:01
*** cd-undercloud has joined #tripleo13:06
cd-undercloud************** overcloud complete status=1 ************13:06
*** cd-undercloud has quit IRC13:06
*** jcooley_ has joined #tripleo13:08
*** jergerber has joined #tripleo13:11
*** jcooley_ has quit IRC13:13
*** CaptTofu has quit IRC13:15
*** CaptTofu has joined #tripleo13:16
*** jayg|g0n3 is now known as jayg13:19
*** jdob has joined #tripleo13:23
*** jcooley_ has joined #tripleo13:23
*** bcrochet has quit IRC13:25
*** bcrochet has joined #tripleo13:28
*** jcooley_ has quit IRC13:28
*** jdob has quit IRC13:35
*** jdob has joined #tripleo13:37
*** kui has quit IRC13:37
*** matsuhashi has quit IRC13:40
*** rushiagr has quit IRC13:42
openstackgerritMarios Andreou proposed a change to openstack/tripleo-heat-templates: WIP: make merge.py installable as a package using pbr  https://review.openstack.org/5882313:46
*** jdob has quit IRC13:51
*** jtomasek has quit IRC13:52
*** jdob has joined #tripleo13:52
*** jtomasek has joined #tripleo13:56
*** dprince has joined #tripleo13:59
*** pblaho has quit IRC14:01
*** julim has joined #tripleo14:01
*** tzumainn has joined #tripleo14:02
*** lucas-hungry is now known as lucasagomes14:04
openstackgerritRalf Haferkamp proposed a change to openstack/diskimage-builder: Add support for building openSUSE images  https://review.openstack.org/5615714:05
*** cd-undercloud has joined #tripleo14:08
cd-undercloud************** overcloud complete status=1 ************14:08
*** cd-undercloud has quit IRC14:08
*** hewbrocca has joined #tripleo14:09
*** john-n-seattle1 has joined #tripleo14:14
*** jcooley_ has joined #tripleo14:17
*** nosnos has quit IRC14:20
*** hewbrocca has quit IRC14:20
*** matty_dubs|gone is now known as matty_dubs14:21
*** hewbrocca has joined #tripleo14:21
*** jcooley_ has quit IRC14:21
*** rongze_ has quit IRC14:27
*** CaptTofu has quit IRC14:31
*** CaptTofu has joined #tripleo14:31
openstackgerritA change was merged to openstack/os-refresh-config: `os-refresh-config` README enhancement  https://review.openstack.org/5931814:31
*** rongze has joined #tripleo14:38
*** akuznetsov has quit IRC14:44
*** hewbrocca has quit IRC14:52
*** cd-undercloud has joined #tripleo15:10
cd-undercloud************** overcloud complete status=1 ************15:10
*** cd-undercloud has quit IRC15:10
*** agordeev has quit IRC15:16
*** jcooley_ has joined #tripleo15:17
*** jcoufal has quit IRC15:17
*** jcooley_ has quit IRC15:21
NobodyCamGood Morning TripleO15:24
jdobmorning :)15:28
NobodyCamgood morning jdob :)15:29
*** jprovazn has quit IRC15:33
*** akuznetsov has joined #tripleo15:33
*** vkozhukalov has quit IRC15:45
*** SpamapS has joined #tripleo15:51
openstackgerritMarios Andreou proposed a change to openstack/tripleo-heat-templates: WIP: make merge.py installable as a package using pbr  https://review.openstack.org/5882315:55
openstackgerritMarios Andreou proposed a change to openstack/tripleo-heat-templates: Make merge.py installable as a package using pbr  https://review.openstack.org/5882316:03
*** jcooley_ has joined #tripleo16:04
*** sdake_ has joined #tripleo16:06
*** sdake_ has quit IRC16:07
*** sdake_ has joined #tripleo16:07
*** sdake_ has quit IRC16:07
*** sdake_ has joined #tripleo16:07
*** cd-undercloud has joined #tripleo16:12
cd-undercloud************** overcloud complete status=1 ************16:12
*** cd-undercloud has quit IRC16:12
SpamapShm nope, still segfaulting dnsmasq16:13
*** john-n-seattle1 has quit IRC16:16
openstackgerritYuriy Zveryanskyy proposed a change to openstack/diskimage-builder: Add deploy ramdisk element for Ironic  https://review.openstack.org/5977016:17
*** jcooley_ has quit IRC16:19
*** jcooley_ has joined #tripleo16:19
*** jcooley_ has quit IRC16:20
*** john-n-seattle1 has joined #tripleo16:20
*** john-n-seattle1 has quit IRC16:25
*** dprince has quit IRC16:25
*** jcooley_ has joined #tripleo16:35
*** marun has quit IRC16:37
*** john-n-seattle1 has joined #tripleo16:39
*** jcooley_ has quit IRC16:47
*** rdopieralski has quit IRC16:49
*** mestery has quit IRC16:49
*** mestery has joined #tripleo16:50
*** SpamapS has quit IRC16:52
*** jtomasek has quit IRC16:55
*** marun has joined #tripleo16:57
*** jtomasek has joined #tripleo16:59
*** cwolferh has joined #tripleo17:05
*** SpamapS has joined #tripleo17:05
*** SpamapS has quit IRC17:05
*** SpamapS has joined #tripleo17:05
*** jistr has quit IRC17:10
*** jcooley_ has joined #tripleo17:12
*** UtahDave has joined #tripleo17:13
*** marun has quit IRC17:13
*** jcooley_ has quit IRC17:15
*** jcooley_ has joined #tripleo17:16
*** jcooley_ has quit IRC17:20
*** jcooley_ has joined #tripleo17:23
*** markmc has quit IRC17:24
*** bauzas has quit IRC17:31
*** matty_dubs is now known as matty_dubs|lunch17:35
*** max_lobur is now known as max_lobur_afk17:37
*** sharp_ has joined #tripleo17:37
*** boris-42 has quit IRC17:38
*** sharp_ is now known as kmsWork17:41
*** derekh has quit IRC17:41
*** martyntaylor1 has quit IRC17:52
*** vkozhukalov has joined #tripleo17:55
*** bauzas has joined #tripleo17:55
*** rongze has quit IRC17:58
openstackgerritChris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic to setup-undercloud-passwords  https://review.openstack.org/5980018:06
*** UtahDave has quit IRC18:08
* Ng dinners18:10
*** athomas has quit IRC18:15
*** vipul is now known as vipul-away18:15
*** sdake has quit IRC18:19
*** sdake has joined #tripleo18:19
*** sdake has quit IRC18:19
*** sdake has joined #tripleo18:19
*** sdake has quit IRC18:20
*** sdake has joined #tripleo18:20
*** sdake has quit IRC18:20
*** sdake has joined #tripleo18:20
*** SpamapS changes topic to "FIREDRILL: dnsmasq segfaults causing undercloud to fail | Using OpenStack to deploy OpenStack; meetings Tuesday 1900 UTC in #openstack-meeting-alt"18:22
*** matty_dubs|lunch is now known as matty_dubs18:30
*** krotscheck has joined #tripleo18:31
*** vipul-away is now known as vipul18:32
*** openstackgerrit has quit IRC18:33
*** openstackgerrit has joined #tripleo18:33
*** dprince has joined #tripleo18:39
*** A-G has quit IRC18:39
SpamapSggaaaahhh18:42
SpamapSwhy must servers take so dang long to boot18:42
SpamapSand now tftp is failing for some weird reason18:45
lifelessmorning18:47
*** rongze has joined #tripleo18:49
SpamapSlifeless: hello there18:50
lifelesshi18:52
*** rongze has quit IRC18:54
dprinceSpamapS: those extra boot time memory checks are quite painful18:55
SpamapSwhat sucks is..18:55
SpamapSI wait about 7 minutes for all the crap..18:56
dprinceSpamapS: if only you could have IPMI and turn off all those checks18:56
SpamapSthen it displays the error for about 0.75s18:56
*** pblaho has joined #tripleo18:58
SpamapSdprince: eh?18:58
SpamapSdprince: is there a way to streamline POST18:59
SpamapS?18:59
*** pblaho has left #tripleo18:59
lifelessmeeting time18:59
*** jistr has joined #tripleo19:00
dprinceSpamapS: on some machines you can tone down the wait time... but in general no it seems19:02
dprinceSpamapS: real servers with all the extra memory checks always seem to take the longest19:03
SpamapSdprince: and I have 96G of memory to check.. so... :-P19:03
lifelessSpamapS: there is a way, beat your vendor up :P19:03
pleia2dprince: apologies for not getting to your question yesterday re: ssh access, could use more details of exactly what you are asking19:04
*** slagle_ has joined #tripleo19:05
dprincepleia2: perhaps we can talk about it in our CI google hangout tomorrow too, but the issue is I'm getting pushback about obtaining public IPs for our CI/CD rack. I wanted to no exactly how many you had, and how many we might actually need.19:05
*** slagle has quit IRC19:05
*** slagle_ is now known as slagle19:06
dprincepleia2: which reminds me, I'll send out an invite for the meeting tomorrow.19:06
*** lucasagomes is now known as lucas-dinner19:07
pleia2dprince: ok, tomorrow would be good, I'm not sure19:08
SpamapShmmm19:10
SpamapSwe may have a rogue DHCP server19:10
SpamapSok no not rogue19:18
* SpamapS foiled by netns again19:18
lifelessdprince: can you get IPv6 for the test rack more easily?19:19
lifelessdprince: we can do IPv6 only19:19
lifelessdprince: I know IPv4 is super scarce19:19
*** shardy is now known as shardy_afk19:20
dprincelifeless: not sure about IPv6, I can push for IPv4 too but it is up in the air19:22
pleia2dprince: any CI updates for #openstack-meeting-alt meeting?19:22
lifelessdprince: so we need a fair naumber - one per jenkins slave + one for the undercloud control plane/endpoint + one for the overcloud control plane/endpoint - at minimum19:22
dprincelifeless: yeah, I figured :(19:23
lifelessdprince: 15 servers, say 8 end up being test environments and 2 end up being hypervisors, that would be 30+IPs19:23
dprincelifeless: My initial estimate was 20, I'll bump it a bit19:23
*** sdake has quit IRC19:24
*** sdake has joined #tripleo19:24
lifelessdprince: I really suggest going the IPv6 route.19:24
dprincepleia2: last week was mostly a holiday, so not much CI stuff yet, Derek has made progress on the gearman eleements though. Soon I plan on tackling making TOCI use the devtest scripts...19:24
SpamapSok so for some reason the freshly PXE dhcp'd machine doesn't see tftp on 10.10.16.17219:25
pleia2dprince: ok same here19:25
dprincelifeless: noted, will see what we can do19:25
lifelessdprince: because I'd rather not have to stress about hitting limits :)19:25
openstackgerritJames Slagle proposed a change to openstack/tripleo-incubator: Test MAC address should be 6 octets.  https://review.openstack.org/5981219:27
openstackgerritJames Slagle proposed a change to openstack/tripleo-incubator: Test MAC address should be 6 octets.  https://review.openstack.org/5981219:30
*** vipul is now known as vipul-away19:33
*** vipul-away is now known as vipul19:33
*** akrivoka has quit IRC19:34
lifelesssdake: got a pointer to the dib template for slagle19:34
*** jcooley_ has quit IRC19:37
*** jcooley_ has joined #tripleo19:37
*** jcooley_ has quit IRC19:41
*** jcooley_ has joined #tripleo19:42
SpamapSdkehn: ok, you have the baton19:44
dkehnSpamapS: thx19:44
SpamapSI need to do some reviews anyway19:44
lifelessdkehn: hold off fixing the live system, or filing the bug ?19:44
dkehnSpamapS: shouldn't take long19:44
lifelessSpamapS: 78:e7:d1:24:31:5d - I think thats one of ours, no?19:45
lifelessdprince: hah, I'm going to be difficult and suggest we hangout now; I have a dr appt tomorrow19:46
*** jcooley_ has quit IRC19:47
*** jcooley_ has joined #tripleo19:47
dprincelifeless: I can talk now (I think), But I'm not sure derekh can make it today19:47
lifelessah19:47
lifelessperhaps you me lyz today, and you derekh lyz tomorrow?19:48
pleia2I don't really have anything useful to say today19:48
SpamapSlifeless: yes that is bm-aw1az2-freecloud001019:48
dprincelifeless/pleia2: also, I'm working from home today so if we chat now I'll have my flaky headset only19:48
pleia2I think just talking with dprince and derekh tomorrow would be valuable19:49
lifelessSpamapS: which is just a regular trunk POC node, so we can fix it by giving it the same settings for those opts the other deployed nodes have19:49
lifelessdprince: pleia2: your call19:49
*** rongze has joined #tripleo19:51
*** jcooley_ has quit IRC19:51
*** jcooley_ has joined #tripleo19:52
sdakelifeless reading backscroll sec19:52
*** vipul is now known as vipul-away19:53
lifelesssdake: oh if you want context - https://review.openstack.org/#/c/59361/19:53
lifelessdkehn: did you see my suggestion about reviews - would have been your 1am or something19:54
sdakelifeless put in review19:55
dkehnlifeless: no I didn't but I saw that the terminal caught it, but didn;t seee where, will look at it19:55
*** rongze has quit IRC19:56
dkehnlifeless: lol, yepper trying to do 3 a day19:57
dkehnlifeless: reviews19:57
dkehnlifeless: unless my stuff gets in the way19:58
*** lynxman has quit IRC19:58
lifelessdkehn: ok, cool - is that new? - http://russellbryant.net/openstack-stats/neutron-reviewers-90.txt shows 10 reviews from you19:59
*** lynxman has joined #tripleo19:59
dkehnthat is new19:59
dkehn1219:59
dprincelifeless/pleia2: lets chat now (if I can make my flaky headset work)19:59
pleia2ok20:00
lifelessdkehn: Let me know if you need any input ;)20:00
dkehnlifeless: thx20:00
lifelessdkehn: sorry 12, yes :)20:00
dkehnnp20:00
*** epim has joined #tripleo20:03
*** vipul-away is now known as vipul20:04
*** jcooley_ has quit IRC20:06
lifelessdprince: https://wiki.openstack.org/wiki/TripleO/TripleOCloud/Regions#Contributing_a_region_to_the_TripleOCloud20:08
*** vipul is now known as vipul-away20:11
*** vipul-away is now known as vipul20:12
dprincehttp://upstart.ubuntu.com/cookbook/#disabling-a-job-from-automatically-starting20:19
*** vkozhukalov has quit IRC20:19
dkehnSpamapS: write the bug, if you pass a '' it doesn't get loaded, but a  public --fixed-ip subnet_id=dbf0a864-f836-4d39-ba81-787e8b6e9454,ip_address=172.24.4.230 --extra-dhcp-opt opt_name=bootfile-name,opt_value=0 --extra-dhcp-opt opt_name=tftp-server,opt_value=123.123.123.123 --extra-dhcp-opt opt_name=server-ip-address,opt_value='  ' gets it20:26
dkehnSpamapS: let me know the bug number and I'll put a patch on it20:27
*** vipul is now known as vipul-away20:27
SpamapSdkehn: seems like you have a grip, want to just write the bug yourself?20:28
SpamapSdkehn: I'm just aware of the symptoms.. :-P20:28
dkehnSpamapS: sure, will do20:28
dkehnSpamapS: thx for the catch20:28
SpamapSdkehn: thanks for looking into it. Thought I was going crazy. ;)20:32
dkehnSpamapS: hmm, can't be sure of the crazy part20:32
SpamapSmy parents said the same thing.. ;)20:33
SpamapShm20:36
SpamapSso tripleo-cd is now just stopping altogether if it hits the throttle20:36
SpamapSis that what we wanted?20:36
lifelessfor 10m20:38
lifelessit was20:38
lifelessif you stop and start it it will try again20:38
SpamapSit stops on its own20:38
SpamapSforever20:39
*** kui has joined #tripleo20:39
SpamapS    flock -x $THROTTLELOCK echo20:39
SpamapS    if [ "0" != "$RESULT" ]; then20:39
SpamapS        rm -f $THROTTLELOCK20:39
SpamapS        exit $RESULT20:39
slaglethe exit was there before wasn't it?20:39
SpamapSactually it just stops, period, if RESULT != 020:40
slagleit should get restarted by the service though20:40
slagleonce it exits20:40
SpamapSin theory yes20:40
SpamapSbut then the throttle is hit20:40
SpamapSand upstart says "you're restarting too fast"20:40
SpamapSI think20:40
SpamapSI actually don't understand the flock/throttle thing at all. Have not tried to think about it.20:41
SpamapSoh, block the exit if we failed in less than 10 minutes, ok.20:42
SpamapSnow I'm actually even more confused as to why we exit and don't just let the loop repeat. :-P20:43
lifelessthe lock file is random20:45
*** vipul-away is now known as vipul20:45
lifelessdprince: https://etherpad.openstack.org/p/tripleo-late-start-services20:48
*** jistr has quit IRC20:48
*** jcooley_ has joined #tripleo20:51
SpamapSok so I'm not really sure where to "fix" that network to work right20:52
*** jcooley_ has quit IRC20:53
SpamapSdkehn: ?20:53
*** rongze has joined #tripleo20:53
SpamapSI mean, I see tftp-server and server-ip-address are blank20:53
SpamapSwhich I presume is "the problem"20:53
dkehnSpamapS: check the nova.conf baremetal section , I'm thinking20:53
SpamapSnova/lib/python2.7/site-packages/nova/tests/virt/baremetal/test_driver.py:                    {'opt_name': 'server-ip-address', 'opt_value': CONF.my_ip},20:55
SpamapSdoh thats tests20:55
SpamapSah ok same diff20:55
SpamapS$ sudo grep my_ip /etc/nova/nova.conf20:56
SpamapSmy_ip=20:56
*** CaptTofu has quit IRC20:56
SpamapSandmy_ip={{local-ipv4}}20:57
SpamapSec2 metadata not working == the problem20:57
*** CaptTofu has joined #tripleo20:57
lifelessSpamapS: eyeball please - https://etherpad.openstack.org/p/tripleo-late-start-services20:58
pleia2eek, my chrome crashed20:58
*** rongze has quit IRC20:58
pleia2I think we were near the end anyway21:00
*** krotscheck has quit IRC21:00
SpamapSlifeless: hm.21:00
dprincepleia2: yep, we were done anyway.21:00
SpamapSlifeless: why are init files 'state' ?21:00
*** vipul is now known as vipul-away21:00
dprincepleia2: funny thing was your picture was still perfectly posed in the Google hangout. It really did look like you were still there, albiet mostly still/froze21:01
*** krotscheck has joined #tripleo21:01
pleia2heh :)21:04
SpamapSlifeless: did ec2 metadata ever work on cd-undercloud?21:04
lifelessSpamapS: yes21:06
lifelessSpamapS: if the undercloud has it wrong, its a <->seed issue21:07
*** boris-42 has joined #tripleo21:07
SpamapSlifeless: Destination     Gateway         Genmask         Flags Metric Ref    Use Iface21:07
SpamapS0.0.0.0         138.35.77.1     0.0.0.0         UG    0      0        0 vlan2521:07
SpamapSlifeless: is the gateway right there?21:07
*** shardy_afk is now known as shardy21:07
SpamapSlifeless: seems like we'd have to route through the seed to make it work21:07
lifelessSpamapS: no, the seed pushes a route21:08
SpamapSlifeless: _ahh_21:09
lifelessSpamapS: or should be; it may be too old21:09
SpamapSlifeless: we were still on file injection at the time21:09
lifelessSpamapS: yeah21:09
lifelessSpamapS: so I think we should add a 169.254.169.254 route to the dnsmasq config on the seed21:09
SpamapSlifeless:         post-up ip route add 169.154.169.154/32 via 10.10.16.16821:09
SpamapSits in /etc/network/interfaces21:09
SpamapSsomehow lost though?21:09
lifelessSpamapS: remember we had a ip binding issue on boot21:10
lifelessSpamapS: bridge vs eth21:10
lifelessadding the route to the seed dnsmasq21:10
SpamapSright21:10
lifelesswould let cloud-init get the data on boot21:10
lifelesswhich would let orc run properly21:10
SpamapSlifeless: ok, but we're not using the dhcp.. so?21:10
lifelesswhich would let it all come up normally I think21:11
lifelessSpamapS: we do, when we start the service when we reboot the undercloud node21:11
lifelessSpamapS: but right now, I'd be inclined to just a) file a note in trello, b) fix things right now, c) get onto automating our way down to the undercloud21:11
SpamapSright21:11
lifelessit's not even a bug21:12
lifelesssince it's in old code21:12
SpamapSright21:12
*** noslzzp has quit IRC21:13
SpamapShm, something happens where baremetal nodes get stuck as long dead instances21:15
lifelessSpamapS: how are init files state? They aren't *but* the assertion 'service X is properly configured' is.21:17
lifelessSpamapS: I'm not entirely happy with that etherpad, but I think it's strictly better than what we have (removes spurious errors in logs during deploy)21:18
*** vipul-away is now known as vipul21:19
SpamapSlifeless: so I may be wrong here, but I'm 90% sure upstart won't follow symlinks.21:21
*** CaptTofu has quit IRC21:22
SpamapSlifeless: we may have to have a set of 'start on starting' jobs that block until the right state has been achieved.21:24
SpamapSRESTRICTIONS The use of symbolic links in job configuration file directories is not supported since it can lead to unpredi‚Äź cable  behaviour  resulting  from  broken  or inaccessible links (such as would be caused by a link crossing a filesystem boundary to a filesystem that has not yet been mounted).21:24
*** jcooley_ has joined #tripleo21:24
SpamapSoh and I just found a spelling mistake in the man page21:24
* SpamapS submits patch21:24
lifelessSpamapS: ok; so goal is to support two things21:24
lifelessA) / readonly21:25
lifelessB) don't start services that don't have valid config21:25
lifelesswhich is both initial, and new image done via cold deploy21:25
lifelessSuggestions?21:25
SpamapSlifeless: a simple "orc has run successfully" coarse grained state that we can record in /mnt with touch would do the trick. We can have one job which just does 'start on starting (list all ..)' which runs a script polling for that file.21:26
SpamapSnewer upstarts can even start on inotify events21:28
*** CaptTofu has joined #tripleo21:29
SpamapSstart on file FILE=/mnt/occ.done EVENT=create21:29
lifelessSpamapS: orc needs to start these things during post-configure though; migrations may depend on it21:36
lifelessso occ.done would be too coarse I think21:36
SpamapSah right21:36
SpamapSlifeless: ok so configure.d 99-configuration-done touches /mnt/occ.configuration.done ?21:37
SpamapS/mnt/state rather21:37
SpamapSor something21:37
*** vipul is now known as vipul-away21:39
*** vipul-away is now known as vipul21:39
*** jdob has quit IRC21:41
SpamapSaaaaand now I've pissed off the ipmi21:41
*** bauzas1 has joined #tripleo21:42
*** bauzas has quit IRC21:43
lifelessSpamapS: or we could bind-mount (ugh) the file into /etc for upstart21:43
lifelessSpamapS: basically we need to unblock dprince though21:43
SpamapSlifeless: dirs in /etc/init will be used, so that could work.21:43
* SpamapS tries it21:43
SpamapSlifeless: so the jobs would have /'s in their names.21:46
*** A-G has joined #tripleo21:50
lifelessdirs?21:52
lifelessyou can bind mount files...21:52
SpamapSlifeless: another option is to have htem 'start on openstack-configured' and emit that in configure.d/75-openstack-configured on upstart systems.21:52
SpamapSlifeless: yeah I dislike bind mounting all those files. :-P21:52
*** rongze has joined #tripleo21:55
*** rpodolyaka1 has joined #tripleo21:56
*** rongze has quit IRC22:00
*** jcoufal has joined #tripleo22:02
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Add diskimage-builder element.  https://review.openstack.org/5936122:09
*** bauzas1 is now known as bauzas22:09
*** sdake_ has quit IRC22:10
*** sdake_ has joined #tripleo22:10
*** krotscheck has quit IRC22:14
*** noslzzp has joined #tripleo22:15
*** krotscheck has joined #tripleo22:17
*** dprince has quit IRC22:22
lifelessSpamapS: OTP22:24
SpamapS\o/ .. got a box to boot finally22:28
*** lsmola has quit IRC22:35
*** lsmola has joined #tripleo22:38
*** epim has quit IRC22:39
*** rpodolyaka1 has quit IRC22:41
lifelessSpamapS: \o/22:46
dkehnSpamapS:  dekehn proposed a change to openstack/neutron: Add checks for empty strings  https://review.openstack.org/59858, to deal with the issue earlier22:47
SpamapSdkehn: sweet thanks22:48
dkehnSpamapS: thanks for catching it22:48
SpamapSnow.. this is where we tell echohead he was right and we should fail os-apply-config on missing keys. ;)22:48
SpamapSI think nova should also fail if my_ip is blank22:49
SpamapSwhich is where this came from22:49
SpamapS"fail fast"22:49
SpamapSlifeless: is there any chance we could get the test rack split into two physical/vlan l2 segments?22:50
SpamapSlifeless: it would be good to be able to have two concurrent running underclouds.22:51
lifelessSpamapS: We had that, I think its a bad idea22:51
SpamapSmmk22:51
* SpamapS will keep trying to find things to do in between 40 minute fail cycles. :-/22:51
lifelessSpamapS: also we are already running two underclouds22:52
lifelessSpamapS: so its entirely not needed to do that22:52
*** epim has joined #tripleo22:52
SpamapSI suggested spinning up a new one and you said that it would cause problems. I may have misunderstood that to be a network segmentation issue.22:52
lifelessSpamapS: with native neutron dhcp there are no issues, as long as we don't have any machine owned by two at aonce22:53
lifelessSpamapS: but that sayid - why, what will we learn from a second undercloud ?22:53
SpamapSwhile the first is doing something, we can change something on the 2nd and run a second experiment.22:54
SpamapSbasically debugging anything on the undercloud takes _days_ of waiting.22:55
lifelessSpamapS: mmm, so when we're doing bare metal ironic gating we get that massively and automatically22:55
SpamapS"when we're doing" .. if that is sometime that could be qualified as "soon" then agreed, thats the way to go.22:56
*** michchap has quit IRC22:56
*** CaptTofu has quit IRC22:57
*** rongze has joined #tripleo22:57
*** michchap has joined #tripleo23:02
*** rongze has quit IRC23:02
*** jcoufal has quit IRC23:02
*** CaptTofu has joined #tripleo23:03
*** lsmola has quit IRC23:07
*** jtomasek has quit IRC23:07
*** noslzzp has quit IRC23:11
SpamapSHm ok so digging through my "bugs-to-file" notes.. neutron-dhcp-agent not noticing that its' dnsmasq has died...23:12
SpamapSTrying to decide what Neutron should do if dnsmasq does die.23:13
lifelessrun around in circles23:18
*** jcooley_ has quit IRC23:18
lifelessSpamapS: it should log it logger.exception(...)23:18
lifelessor logger.error(...)23:18
SpamapSI'm starting to see the code required to do this23:23
lifelessSpamapS: just file the bug IMO23:24
lifelessSpamapS: will be eventlet fiddling to respond to the child died event23:24
SpamapSlifeless: well I was going to provide a patch if that is easy23:24
SpamapSlifeless: the eventlet fiddling is relatively straight forward.. ignore SIGCHLD, use .wait() in its own greenthread.23:25
SpamapSassuming wait doesn't use SIGCHLD actually23:25
SpamapSI used to know this stuff.. :-P23:25
lifelessso thats why I say just file the bug :)23:25
lifelessthere's enough detail there to get right23:25
*** cd-undercloud has joined #tripleo23:29
cd-undercloud************** overcloud complete status=1 ************23:29
*** cd-undercloud has quit IRC23:29
*** matty_dubs is now known as matty_dubs|gone23:30
SpamapShttps://bugs.launchpad.net/neutron/+bug/125752423:30
lifelessmore on kexec - http://mjg59.dreamwidth.org/28746.html23:30
uvirtbotLaunchpad bug 1257524 in neutron "If neutron spawned dnsmasq dies, neutron-dhcp-agent will be totally unaware" [Undecided,New]23:30
lifelessI love mjg5923:30
SpamapSok well back to having ACTIVE servers, but they've timed out23:32
SpamapSec2 metadata now failed23:35
*** bauzas has quit IRC23:40
*** CaptTofu has quit IRC23:44
*** jtomasek has joined #tripleo23:45
*** jayg is now known as jayg|g0n323:48
*** jtomasek has quit IRC23:52
*** CaptTofu has joined #tripleo23:55
*** openstackgerrit has quit IRC23:56
*** openstackgerrit has joined #tripleo23:56
*** kui has quit IRC23:56
*** rongze has joined #tripleo23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!