Wednesday, 2014-12-17

*** andreaf has quit IRC00:05
*** marun has quit IRC00:06
*** lucas-dinner has joined #tripleo00:07
*** Marga_ has joined #tripleo00:15
*** Marga_ has quit IRC00:15
*** Marga_ has joined #tripleo00:15
*** lucas-dinner has quit IRC00:16
*** ccrouch1 has joined #tripleo00:19
*** ccrouch has quit IRC00:20
*** andreaf has joined #tripleo00:22
openstackgerritRichard Su proposed openstack/tripleo-incubator: SELinux developer guide
*** lblanchard has joined #tripleo00:38
openstackgerritRichard Su proposed openstack/tripleo-incubator: SELinux developer guide
openstackgerritRichard Su proposed openstack/tripleo-incubator: SELinux developer guide
*** chlong has quit IRC00:46
*** ccrouch1 has quit IRC00:47
*** jrist is now known as jrist-afk00:47
*** chlong has joined #tripleo01:03
*** lblanchard has quit IRC01:08
openstackgerritRichard Su proposed openstack/tripleo-specs: Switching SELinux to enforcing mode
*** penick has quit IRC01:11
*** rhallisey has quit IRC01:11
*** chlong has quit IRC01:12
*** chlong has joined #tripleo01:12
*** penick has joined #tripleo01:15
*** killer_prince is now known as lazy_prince01:20
*** vinsh has joined #tripleo01:23
*** vinsh has quit IRC01:27
*** penick has quit IRC01:39
*** chlong has quit IRC01:53
*** pradeep1 has joined #tripleo01:56
*** kes has joined #tripleo01:56
*** pradeep2 has joined #tripleo01:58
*** wqyuwss has quit IRC01:58
*** pradeep1 has quit IRC02:00
*** chlong has joined #tripleo02:00
*** chlong has quit IRC02:05
*** pradeep2 has quit IRC02:05
*** ccrouch has joined #tripleo02:08
*** ChuckC has quit IRC02:18
*** pradeep1 has joined #tripleo02:28
SpamapSwhy do hp1 and hp2 have PublicVirtualFixedIPs set in their _ci_env.json, but that value is never copied into devtest anywhere.02:30
*** pradeep2 has joined #tripleo02:34
*** pradeep1 has quit IRC02:35
*** pradeep2 has quit IRC02:40
*** pradeep1 has joined #tripleo02:40
*** rwsu has quit IRC02:43
*** pradeep2 has joined #tripleo02:50
*** pradeep1 has quit IRC02:51
*** Marga_ has quit IRC02:52
*** Marga_ has joined #tripleo02:53
*** chlong has joined #tripleo02:55
*** pradeep1 has joined #tripleo02:56
*** pradeep2 has quit IRC02:58
*** pradeep2 has joined #tripleo03:02
*** ChuckC has joined #tripleo03:03
*** lazy_prince is now known as killer_prince03:04
*** pradeep1 has quit IRC03:04
*** kes has quit IRC03:06
*** chlong has quit IRC03:07
*** kes has joined #tripleo03:09
*** chlong has joined #tripleo03:14
*** chlong has quit IRC03:16
*** pradeep1 has joined #tripleo03:19
*** pradeep2 has quit IRC03:21
*** kesmellon has joined #tripleo03:23
*** pradeep2 has joined #tripleo03:23
*** pradeep1 has quit IRC03:25
*** kes has quit IRC03:26
*** pradeep1 has joined #tripleo03:28
*** pradeep2 has quit IRC03:30
*** chlong has joined #tripleo03:35
*** pradeep2 has joined #tripleo03:36
*** pradeep1 has quit IRC03:38
*** chlong has quit IRC03:40
*** jerryz has joined #tripleo03:42
*** pradeep2 has quit IRC03:43
*** pradeep1 has joined #tripleo03:44
*** pradeep2 has joined #tripleo03:53
openstackgerritMartin André proposed stackforge/kolla: Add gitignore file
*** pradeep1 has quit IRC03:54
*** pradeep1 has joined #tripleo03:57
*** pradeep2 has quit IRC03:58
*** untriaged-bot has joined #tripleo04:00
untriaged-botUntriaged bugs so far:04:00
uvirtbotLaunchpad bug 1402928 in tripleo "RabbitClientUseSSL type error in overcloud-without-mergepy.yaml " [Undecided,New]04:00
uvirtbotLaunchpad bug 1402795 in tripleo "Keystone token_flush  cron job running at the same time in the controller node causes deadlock" [Undecided,In progress]04:00
uvirtbotLaunchpad bug 1387487 in tuskar "tripleo-image-elements/elements/tuskar-ui /os-refresh-config /post-configure.d/101-tuskar-ui  needs authentication from Keystone" [Undecided,New]04:00
*** untriaged-bot has quit IRC04:00
openstackgerritMartin André proposed stackforge/kolla: Remove unused libvirt-service.json file
*** pradeep1 has quit IRC04:09
*** vinsh has joined #tripleo04:13
openstackgerritMartin André proposed stackforge/kolla: Reduce code duplication in
*** ChuckC_ has joined #tripleo04:19
*** ChuckC has quit IRC04:20
*** chlong has joined #tripleo04:20
*** chlong has quit IRC04:22
openstackgerritMartin André proposed stackforge/kolla: Fix use of undefined KEYSTONE_SERVICE_HOST in Ceilometer
*** mandre has joined #tripleo04:34
*** yuanying_ has quit IRC04:43
*** yuanying has joined #tripleo04:43
*** andreaf has quit IRC04:45
*** andreaf has joined #tripleo04:46
openstackgerritMartin André proposed stackforge/kolla: Enable usage of containers outside of Kubernetes
*** gfidente has quit IRC04:54
*** al has quit IRC04:55
*** pradeep1 has joined #tripleo04:57
*** andrearosa has quit IRC04:59
*** Shrews has quit IRC04:59
*** gfidente has joined #tripleo04:59
*** andrearosa has joined #tripleo04:59
*** noslzzp has joined #tripleo05:00
*** Marga_ has quit IRC05:01
*** al has joined #tripleo05:03
*** weshay has quit IRC05:03
*** noslzzp has quit IRC05:08
*** pensu has joined #tripleo05:10
*** noslzzp has joined #tripleo05:11
SpamapShm this is odd05:37
SpamapSseems we may have something borken which disables os-collect-config from ever using the heat or cfn API.05:37
StevenKSpamapS: Which hotel did you book for the sprint, btw?05:47
* StevenK is currently looking at flights05:47
SpamapSStevenK: Westin05:49
SpamapSStevenK: short walk, nice beds. :)05:49
*** ishant has joined #tripleo05:49
* StevenK is flying in on Tuesday morning, leaving Saturday night05:49
SpamapSStevenK: The Sheraton is nicer, but super pricey and you have to book it w/o CWT because it's not HP Approved.05:50
StevenKDepart Sydney at Tuesday 11:45am, land into Seattle at Tuesday 11:29am05:50
SpamapSSheraton is literally across the street from 700 pike.05:50
StevenKI'm going backwards in time!05:50
SpamapSStevenK: time warp!!! nice05:50
SpamapSit had to be done05:51
StevenKSadly, I leave Saturday 6:55pm, and land into Sydney at 9:35am on Monday05:51
SpamapSStevenK: reverse timewarp. ;)05:52
StevenKSpamapS: Westin on 5th ave?06:01
greghaynesSpamapS: wah? I just did a devtest run that worked...d06:02
greghayneshrm, I should book summit travel06:08
greghaynesI wonder if cwt can do amtrak06:08
*** killer_prince is now known as lazy_prince06:09
StevenKgreghaynes: To Vancouver?06:11
greghayneser, sprint travel06:11
greghaynesalthough I was thinking of doing train to there also06:11
*** tzumainn has quit IRC06:17
*** noslzzp has quit IRC06:17
*** chlong has joined #tripleo06:19
*** ukalifon has joined #tripleo06:24
*** bvandenh has joined #tripleo06:28
*** avozza is now known as zz_avozza06:32
*** bvandenh has quit IRC06:36
*** dshulyak has joined #tripleo06:36
*** rlandy has joined #tripleo06:42
*** Shrews has joined #tripleo06:59
*** chlong is now known as chlong_hometime07:06
tchaypono bugsquashing for me yet, my workhours today are roughly late afternoon into the evening07:07
greghaynesits technically still tues here :007:07
tchaypoI think I’ll be working roughly american wednesday :)07:09
tchaypowhich probably means i won’t be seeing you...07:09
tchaypookay, email check complete, time for me to go deal with offline things07:10
*** pcaruana has joined #tripleo07:16
*** mrunge has joined #tripleo07:32
*** rdopiera has joined #tripleo07:32
openstackgerritMerged openstack/tripleo-incubator: Support --help as well as -h
*** bvandenh has joined #tripleo07:52
*** pblaho has joined #tripleo07:57
*** rdopiera has quit IRC07:59
GheRiveromorning Tripleo. Happy bug day!08:03
*** pcrews has quit IRC08:04
*** ifarkas has joined #tripleo08:06
*** pcrews has joined #tripleo08:07
*** jtomasek has joined #tripleo08:12
*** pcrews has quit IRC08:13
*** dshulyak has quit IRC08:18
*** pradeep1 has quit IRC08:25
*** jcoufal has joined #tripleo08:30
*** ukalifon has quit IRC08:36
*** wuhg has joined #tripleo08:37
*** pradeep1 has joined #tripleo08:46
*** jang2 has joined #tripleo08:49
*** ChuckC_ is now known as ChuckC08:52
*** pradeep2 has joined #tripleo08:57
*** akrivoka has joined #tripleo08:57
*** mandre has quit IRC08:58
*** pradeep1 has quit IRC08:58
*** kesmellon has quit IRC09:06
*** ukalifon1 has joined #tripleo09:07
*** pradeep2 has quit IRC09:08
*** noslzzp has joined #tripleo09:09
*** pradeep1 has joined #tripleo09:09
*** derekh has joined #tripleo09:16
*** mrunge has quit IRC09:18
*** noslzzp has quit IRC09:21
*** zz_avozza is now known as avozza09:23
*** mrunge has joined #tripleo09:25
*** mrunge has quit IRC09:25
*** mrunge has joined #tripleo09:25
*** gchamoul is now known as gchamoul|PTOs09:33
*** pradeep2 has joined #tripleo09:36
*** pradeep1 has quit IRC09:37
*** mrunge has quit IRC09:53
*** avozza is now known as zz_avozza09:59
*** untriaged-bot has joined #tripleo10:00
untriaged-botUntriaged bugs so far:10:00
uvirtbotLaunchpad bug 1402928 in tripleo "RabbitClientUseSSL type error in overcloud-without-mergepy.yaml " [Undecided,New]10:00
uvirtbotLaunchpad bug 1402795 in tripleo "Keystone token_flush  cron job running at the same time in the controller node causes deadlock" [Undecided,In progress]10:00
uvirtbotLaunchpad bug 1387487 in tuskar "tripleo-image-elements/elements/tuskar-ui /os-refresh-config /post-configure.d/101-tuskar-ui  needs authentication from Keystone" [Undecided,New]10:00
*** untriaged-bot has quit IRC10:00
*** mrunge has joined #tripleo10:00
*** lifeless_ is now known as lifeless10:04
*** zz_avozza is now known as avozza10:08
*** jp_at_hp has joined #tripleo10:22
*** jang2 has quit IRC10:24
*** athomas has quit IRC10:39
*** lucasagomes has joined #tripleo10:55
*** athomas has joined #tripleo11:05
openstackgerritClint 'SpamapS' Byrum proposed openstack/tripleo-image-elements: Use gearman to limit token flush to one host
*** pradeep1 has joined #tripleo11:11
*** pradeep2 has quit IRC11:14
*** thrash|g0ne is now known as thrash11:21
*** regebro has joined #tripleo11:53
*** pradeep2 has joined #tripleo11:53
openstackgerritDerek Higgins proposed openstack/tripleo-image-elements: Cleanup lockfile with trap
*** pradeep1 has quit IRC11:55
*** rhallisey has joined #tripleo12:29
*** jiangfei has quit IRC12:29
*** jiangfei has joined #tripleo12:29
openstackgerritDerek Higgins proposed openstack/tripleo-incubator: Increase size of volume used for user image
*** pradeep1 has joined #tripleo12:40
*** pradeep2 has quit IRC12:41
openstackgerritAndrea Frittoli  proposed openstack/tripleo-image-elements: Add a tempestrunrc for configuring run-tempest
openstackgerritAndrea Frittoli  proposed openstack/tripleo-image-elements: Configurable tests2skip file in tempest element
*** lazy_prince is now known as killer_prince12:49
*** ZhangYang has joined #tripleo12:49
*** ZhangYang has left #tripleo12:50
*** pradeep2 has joined #tripleo12:50
*** pradeep1 has quit IRC12:50
*** dprince has joined #tripleo12:52
*** ZhangYang has joined #tripleo12:56
*** ccrouch has quit IRC13:03
*** rhallisey has quit IRC13:03
*** rhallisey has joined #tripleo13:03
*** ccrouch has joined #tripleo13:04
*** vipul has quit IRC13:05
*** jesusaurus has quit IRC13:05
*** weshay has joined #tripleo13:08
*** vipul has joined #tripleo13:10
ZhangYanghello, tripleo.may i ask a question?  i met some problem building image with dib.13:13
ZhangYangi'm building an opensuse image with dib and tripleo-image-elements, do they support opensuse?13:14
*** tzumainn has joined #tripleo13:16
*** pradeep1 has joined #tripleo13:16
*** lblanchard has joined #tripleo13:17
*** pradeep2 has quit IRC13:17
*** jesusaurus has joined #tripleo13:18
*** Hazelesque_ has quit IRC13:22
*** rhallisey_ has joined #tripleo13:22
*** Hazelesque has joined #tripleo13:22
*** rhallisey has quit IRC13:26
*** jdob has joined #tripleo13:27
*** pensu has quit IRC13:31
*** wanghua has quit IRC13:36
*** jcoufal_ has joined #tripleo13:47
*** jcoufal has quit IRC13:50
*** ZhangYang has quit IRC13:51
*** ishant has quit IRC13:51
*** ZhangYang has joined #tripleo13:51
*** mrunge has quit IRC13:51
openstackgerritDan Prince proposed openstack/tripleo-image-elements: os-net-config: log at log level info again...
derekhZhangYang: some work was being done to support opensuse, I'm not sure if it was working but I cane definitely say it isn't being tested in CI so I wouldn't be surprised if there is problems14:02
*** akrivoka has quit IRC14:02
*** akrivoka has joined #tripleo14:03
*** julim has joined #tripleo14:07
*** radez_g0n3 is now known as radez14:11
gfidentegreghaynes, TheJulia, SpamapS is there a particular reason why we never implemented the sysctl settings to decrease keepalive interval?14:34
TheJuliagfidente: I don't think so, it likely just fell off the radar.  Why do you ask?14:41
gfidenteTheJulia, hi, because of this
uvirtbotLaunchpad bug 856764 in oslo.messaging "RabbitMQ connections lack heartbeat or TCP keepalives" [Critical,In progress]14:41
gfidenteI thought that combined with keepalive could make things to work so it seems about time to try that14:42
*** killer_prince is now known as lazy_prince14:44
gfidenteif there are no downsides, I'd try to push that14:44
gfidenteI am not sure which values you use though and I wouldn't mess up with that14:45
dprincederekh: any ideas why this is going nowhere?
*** pcaruana has quit IRC14:51
TheJuliagfidente: If I understand what your thinking, and if oslo.messaging has fixed the issues where they don't cleanly handle the connection loss, then in this case they should relatively quickly timeout due to keepalive if the controller node completely dies out from under it, and then messages could be reconsumed?14:51
*** jehb_ has joined #tripleo14:52
derekhdprince: no idea, it failed the python33 test, your reverify kicked if off again and its failed again (hasn't reported back to gerrit yet)14:52
gfidenteTheJulia, yeah they still don't handle the connection loss correctly14:52
tchaypoNameError: name 'StandardError' is not defined14:53
dprincederekh: oh, the job finished before I checked. I was looking on the zuul status page14:53
gfidentebut the changes in rabbit should cope with that14:53
TheJuliagfidente: then on the appropriate rabbit version, it should be less of an issue, or at least I would think.  sounds like worth giving a try, and walking it through the various failure scenarios to see what happens.14:53
gfidenteTheJulia, this is, as long as you enforce short keepalive interval as well14:53
gfidentecause client still misses heartbeat14:54
gfidenteso in the tripleo scenario, it seems to be just a few lines in template away14:55
gfidentewe won't need changes in either rabbit config or clients config14:55
tchaypowhen i run the tsts for 142120 locally, not even the py27 tests pass14:57
tchaypoConfigException: ("group 'root' not found in group database", 'os_apply_config/tests/chown_templates/')14:57
tchaypoand similar14:57
tchaypopy33 gets the "NameError: name 'StandardError' is not defined” error14:58
dprincetchaypo: yeah, all tests previously passed the check15:00
TheJuliagfidente: Sounds like its worth trying because the big issue that wait for a stale connection to die if the vip moves or the active holder of the vip is suddenly powered off15:01
gfidenteTheJulia, have some preferences for interval, probes?15:03
TheJuliagfidente: out of the box, i believe they are increadibly high15:05
gfidenteyeah 2hrs15:05
TheJulia~5 minutes wouldn't be too quickly15:06
TheJuliaand reduce the process confusion from the vip disappearing15:06
derekhNeed another +1 to push the mariadb fix through, anybody want to give it a whirl
*** jang1 has quit IRC15:17
*** jang has quit IRC15:17
bnemecslagle_: Would you be willing to take derekh and my word for it that ^ works? :-)15:17
*** jang has joined #tripleo15:17
*** jang1 has joined #tripleo15:17
bnemecAnd also the random anonymous who didn't log in before voting.15:18
*** lucasagomes is now known as lucas-hungry15:24
*** mcornea has joined #tripleo15:28
*** vinsh_ has joined #tripleo15:29
*** marun has joined #tripleo15:31
ukalifon1slagle_: Can I bug you with a few questions? First question is: the overcloud machines in a virt setup have no dns set up. So, for example, I can ping but I can't ping Is that a bug?15:32
*** vinsh has quit IRC15:33
*** vinsh_ has quit IRC15:33
*** vinsh has joined #tripleo15:48
*** mrunge has joined #tripleo15:56
*** untriaged-bot has joined #tripleo16:00
untriaged-botUntriaged bugs so far:16:00
uvirtbotLaunchpad bug 1402928 in tripleo "RabbitClientUseSSL type error in overcloud-without-mergepy.yaml " [Undecided,New]16:00
uvirtbotLaunchpad bug 1402795 in tripleo "Keystone token_flush  cron job running at the same time in the controller node causes deadlock" [Undecided,In progress]16:00
uvirtbotLaunchpad bug 1387487 in tuskar "tripleo-image-elements/elements/tuskar-ui /os-refresh-config /post-configure.d/101-tuskar-ui  needs authentication from Keystone" [Undecided,New]16:00
*** untriaged-bot has quit IRC16:00
*** pcrews has joined #tripleo16:00
*** spzala has joined #tripleo16:03
tchaypodprince: i have a sad.16:05
dprincetchaypo: oh no16:05
*** spzala has quit IRC16:05
tchaypolet me double-check to confirm my sad16:06
*** prad has joined #tripleo16:07
*** jerryz has quit IRC16:12
*** ukalifon1 has quit IRC16:13
tchaypohrm, i may be off track16:14
tchaypodocutils support for py3 is done by install-time translation of the files16:14
tchaypolooking at the two runs, the one that worked installed docutils then jinja2; the one that failed installed docutils as part of installing jinja216:14
*** rhallisey_ is now known as rhallisey16:15
tchayposorry - the two that worked both installed docutils first16:16
tchaypoDownloading/unpacking docutils>=0.10 (from sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3->-r /home/jenkins/workspace/gate-os-apply-config-python33/test-requirements.txt (line 11))16:16
tchaypothe one that failed has16:16
tchaypoDownloading/unpacking Jinja2>=2.3 (from sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3->-r /home/jenkins/workspace/gate-os-apply-config-python33/test-requirements.txt (line 11))16:16
tchaypolooks to me like line 11 of  /home/jenkins/workspace/gate-os-apply-config-python33/test-requirements.txt changed - mayhap someone decided docutils was redundant since jinja2 requires it?16:17
tchaypooh no - in both cases line 11 is installing sphinx. order of requirements changed for sphinx?16:20
*** lucas-hungry is now known as lucasagomes16:23
bnemectchaypo: dprince: It sounds like everyone's py33 gate is hitting this:
*** pblaho has quit IRC16:26
tchaypobnemec: wunderbar16:26
*** rlandy has quit IRC16:29
*** jrist-afk is now known as jrist16:30
*** jcoufal_ is now known as jcoufal16:31
dprincetchaypo/bnemec: thanks for the feedback. I can't chase this ATM but I'll stay tuned.16:32
bnemecYeah, and until zuul gets its act together  nothing's going to happen anyway.16:33
openstackgerritMerged openstack/os-net-config: Fix verbose/debug loglevels
bnemecAnd of course as I say that it looks like zuul might have gotten its act together.16:34
*** dsneddon has quit IRC16:36
*** rwsu has joined #tripleo16:46
*** marun has quit IRC16:49
openstackgerritAbel Lopez proposed openstack/diskimage-builder: Ignore stderr from pkg-map
openstackgerritMerged stackforge/tripleo-ansible: Workflow documentation is now in infra-manual
*** marun has joined #tripleo16:57
*** jtomasek has quit IRC17:01
*** Marga_ has joined #tripleo17:01
*** pensu has joined #tripleo17:04
*** spzala has joined #tripleo17:10
*** mcornea has quit IRC17:12
*** Marga_ has quit IRC17:15
*** lazy_prince is now known as killer_prince17:18
*** Marga_ has joined #tripleo17:19
*** athomas has quit IRC17:19
*** penick has joined #tripleo17:22
SpamapSah so my new challenge.. I assigned a static IP to my overcloud using NeutronPublicInterfaceIP.. so of course, everything is listening on "not that IP" and it isn't reachable. ARGH.17:25
*** jcoufal has quit IRC17:25
*** wuhg has quit IRC17:33
*** penick has quit IRC17:36
*** bvandenh has quit IRC17:41
*** avozza is now known as zz_avozza17:42
*** jp_at_hp has quit IRC17:51
*** penick has joined #tripleo17:56
greghaynesgfidente: I thought we did do that...17:58
*** akrivoka has quit IRC17:59
openstackgerritGiulio Fidente proposed openstack/tripleo-incubator: Add sysctl to all images
*** derekh has quit IRC18:00
*** ifarkas has quit IRC18:02
greghaynesSpamapS: So our TLS people had a good point about the keystone tls key not being securely transferred - our DB passwords also have this issue18:02
openstackgerritAbel Lopez proposed openstack/diskimage-builder: Ignore stderr from pkg-map
greghaynesI wonder if a good fix for now is to generate a secrets.tar with json for the properties we want and just scp it over, extract into local data collector18:08
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Set more aggressive keepalive timings
*** Marga_ has quit IRC18:13
*** Marga_ has joined #tripleo18:14
*** dshulyak_ has joined #tripleo18:14
tchaypoDo we still run the nova ec2 interface by default?18:14
tchaypofor some reason I thought we had turned it off but now I can’t find the emails I thought I saw saying that18:14
*** pensu has quit IRC18:16
openstackgerritBen Nemec proposed openstack/diskimage-builder: dracut-ramdisk: fix support for elements with ramdisk-install.d
*** marun has quit IRC18:20
*** pradeep2 has joined #tripleo18:22
*** penick has quit IRC18:23
*** pradeep1 has quit IRC18:23
SpamapSgreghaynes: yeah because even if we secure access via Heat/CFN, ec2 access isn't secure.18:24
SpamapStchaypo: I see no reason to turn it off.18:25
SpamapStchaypo: let the people use EC2 I say.18:25
*** pradeep1 has joined #tripleo18:27
*** pradeep1 has quit IRC18:28
*** pradeep2 has quit IRC18:29
tchaypoi may have been dreaming.18:29
tchaypoabout nova.18:29
tchaypoSpamapS: btw, in last night’s meeting we touched on having other people work on hp218:30
tchaypoif we have anyone who wants to work on it I’d love to get them set up and bring them up to speed18:30
*** penick has joined #tripleo18:31
tchaypogreghaynes: any chance we can get on ?18:32
tchaypoi have no idea what it is or how to get there18:32
SpamapSOh it's bug day!18:33
* SpamapS may file a few as well as fix some18:34
SpamapStchaypo: regarding hp2, I just want to make sure people aren't just stepping back because "tchaypo's got it". We should rotate so you can gain some perspective.18:34
openstackgerritgreghaynes proposed openstack/tripleo-image-elements: Enable debug logs
*** marun has joined #tripleo18:35
tchayporight now tchaypo don’t got it. the seed is ignoring arp and I can’t fix it. it’s one of those issues that I swear I understood back in 2.2 kernels but today it’s just in the “weird kernel network magic” black box18:35
*** greghaynes changes topic to "Happy Bugsquash day! | devtest on F20 broken | Mid-cycle meetup Feb 18 - 20 in Seattle | Using OpenStack to deploy OpenStack; meetings Tuesday 1900/0800 UTC in #openstack-meeting-alt"18:36
greghayneswow, we went from 299 open bugs to 264 already18:39
* SpamapS should probably turn off the random bug closer bot18:45
*** Hefeweizen has quit IRC18:45
greghaynesaw, damn18:46
*** Marga_ has quit IRC18:49
*** Marga_ has joined #tripleo18:49
greghaynesmorganfainberg: We have this bug:
uvirtbotLaunchpad bug 1272821 in tripleo "keystone isn't logging requests" [High,Triaged]18:49
morganfainberggreghaynes, which version of Keystone and... what deployment method... and...18:49
morganfainberguhm yeah those two questions?18:49
greghayneswow, that was fast18:50
morganfainberggreghaynes, :)18:50
morganfainberggreghaynes, my guess is apache deployed?18:50
greghaynes/bin/sh -e -c exec start-stop-daemon --start -c keystone --exec /opt/stack/venvs/openstack/bin/keystone-all -- --config-dir /etc/keystone 2>&1 | logger -t keystone /bin/sh18:51
greghaynesI think that means no apache?18:51
morganfainberghm eventlet deployed18:51
morganfainbergi'll bet it's a config option to squash logs from routes18:51
greghaynesroutes.middleware=WARN - maybe this is just high enough that it doesnt log them?18:52
morganfainbergthat might be it, 2x checking18:52
greghaynesso, I wonder, if all those are logger modules have individual defaults, does setting a different root default apply to them?18:53
morganfainbergit would be eventlet.wsgi.server18:53
morganfainbergand INFO would be the minimum to see the GET / POST /etc lines18:54
greghaynesah, gotcha18:54
morganfainbergthe logger might just be eventlet.wsgi as well or eventlet18:54
morganfainbergso look for those18:54
morganfainbergunder apache those logs go to your access.log, so if/when deploying under apache - i recommend making sure you grab / configure a sane access.log18:55
greghaynesyea, makes sense18:55
tchaypoSpamapS: I’m popping out to dinner now. Perhaps I should start including a “this week, tchaypo is stumped by …” in the ci/cd emails so that people are aware of what’s blocking hp2?18:55
*** jtomasek has joined #tripleo18:56
*** Marga_ has quit IRC19:01
*** Marga_ has joined #tripleo19:02
greghaynesmorganfainberg: so another thought - what do you think of making that the default setting?19:02
greghaynesmorganfainberg: the reason we want to lower it is that its pretty impossible to debug issues otherwise19:03
morganfainberggreghaynes, well, it was at one point a default setting. it's gone through a number of iterations and changes19:03
morganfainberggreghaynes, so, i'm not opposed to work to do that again. i don't actually know when this may have changed last :P19:03
greghaynescool, ill poke at it :)19:03
morganfainbergusability and sane defaults that support that usbility is important to us19:04
*** penick has quit IRC19:12
SpamapSmorganfainberg: production defaults is a core value for us, so yes, if the default doesn't make any sense lets change it. :)19:14
SpamapSlog destinations are a tough one though.19:14
SpamapSIMO writing to /dev/log kind of makes sense in most cases, but plenty of people will recoil in horror at that.19:14
greghaynesIts especially annoying for us because we have to maintain rsyslog and journald config for any kind of fancy outputting19:16
greghayneswhich you really want if you enable debug logs19:16
SpamapSjournald: because rsyslog was getting WAY too mature and awesome.19:18
greghaynesI referred to journald as 'a turducken of custom protocols' the other day19:19
SpamapSgreghaynes: fine artisan protocols ( ◔┏‸┓◔)19:21
SpamapShm so I'm trying to get my seed to be a full-fledge undercloud because I don't care about having an undercloud for this CI cloud..19:22
SpamapSI want it to make the public listening IP one from a certain CIDR that is on a separate interface...19:23
SpamapSthus far unable to unwind the network monster to figure this out19:23
SpamapSI think what I need to do is to create a 'public' network in my seed with the subnet of just the one IP.19:24
SpamapSSeems like what I want is a flat, not vlan, network.19:25
*** ChuckC has quit IRC19:39
*** penick has joined #tripleo19:42
openstackgerritgreghaynes proposed openstack/tripleo-image-elements: Disable set -x for passwords
*** jang2 has joined #tripleo19:46
*** jcoufal has joined #tripleo19:59
*** dshulyak_ has quit IRC20:00
*** rlandy has joined #tripleo20:10
*** dshulyak_ has joined #tripleo20:18
*** dshulyak_ has quit IRC20:20
*** lucasagomes is now known as lucas-dinner20:22
SpamapStchaypo: so re hp2 stuff.. I think we just need to make it a priority for all tripleo-cd-admins. Feels like we're silo'd a bit.20:22
*** penick has quit IRC20:24
tchaypoSo that probably requires communication about the current state and what needs to be worked on next20:26
tchaypothis feels a little bit like a trello board might be helpful?20:26
SpamapStchaypo: agree. We had one in the past and I felt like it was better than the current stalemate.20:27
SpamapSweird, why is init-keystone trying to connect with ssl to port 35357? :-P20:29
morganfainberghttps:// prefix?20:29
SpamapSinit-keystone --debug -o -t 455de70d5c41b036f7b5d44a04d019cdd2a6fb2e -e -p dd5a63b82c745e8141f7b7824bca7bd4ccfd9ec1 -u heat-admin -s --no-pki-setup20:30
* morganfainberg ducks before more cats are thrown... i'm sure mordred has thrown enough towards SpamapS that he has some to spare20:30
SpamapSmorganfainberg: if one is being added, it is entirely init-keystone's fault20:30
morganfainbergthat doesn't look wrong20:30
tchaypoon the topic of making stuff visible20:30
tchaypomaybe i should raise a bug for the current issue i have with the seed in hp2?20:31
morganfainbergoh wait a sec20:31
*** penick has joined #tripleo20:31
morganfainbergdoes keystone-init do something silly like use the service catalog to talk to keystone20:31
tchaypoi’m 45% certain it’s going to turn out to be a misconfiguration rather than a code bug, but at least it means the details of the problem are somewhere people can look at them and call me silly names when they see what I’m doing20:31
morganfainbergif it does and the SC has https in it, that could explain it20:31
SpamapShm maybe 35357 should be ssl? weird20:35
tchaypoI’ll do a bug and trello tomorrow. tonight i want to focus on bug-bashing.20:40
SpamapSStevenK: question for you about init-keystone's ssl bits20:40
SpamapSStevenK: seems like it expects 35357 to be SSL if an ssl host is passed in..20:41
SpamapSbut AFAICT, nothing sets up SSL on that port ever.20:41
SpamapSgreghaynes: you have any ideas?20:41
morganfainberggreghaynes, is it bad i always see \o/ or \O/ as
greghaynesaye, that bit of code has been through a few iterations20:42
* greghaynes rereads20:42
tchaypogreghaynes: you’re talking about 299 bugs just on the tripleo-incubator project? or are you looking at something else?20:42
SpamapSmorganfainberg: yes thats bad. (0)20:42
greghaynestchaypo: now 26420:42
*** panda has quit IRC20:43
*** panda has joined #tripleo20:43
greghaynesSpamapS: if ssl is defined it uses port 13000 AFAICT20:44
greghaynesthe endpoint created is never20:45
greghaynesthe endpoint created is never ssl though20:45
* greghaynes is backwards20:45
SpamapSgreghaynes: I'm talking about the fact that init-keystone, at this point, _cannot even talk to keystone_ on my cloud20:47
SpamapSit's using https://$host:35357/v2.0 as the keystone v2 url20:47
SpamapSbut that port is not SSL20:48
SpamapSand nothing in openstack-ssl ever configures it to be SSL20:48
morganfainbergand likely it shouldn't be.20:48
*** penick has quit IRC20:48
morganfainbergas it would imply under eventlet ... oh awful things.20:48
SpamapSmorganfainberg: oh come on, a modern CPU can push _at least _ 100Mbit of SSL all by itself. We don't need to do anything else with our one CPU allocated to keystone, right? ;)20:49
greghaynesso... how does this work at all20:50
SpamapSit doesn't20:50
SpamapSit's not tested20:50
*** jcoufal has quit IRC20:50
SpamapSI believe that we just need to drop the https condition there20:50
SpamapSyep that does it20:52
*** marun has quit IRC20:52
SpamapSgreghaynes: looks to me like our CI never tests HTTPS20:52
SpamapSlike I siad, I'm going to open bugs before I can squish a few today :)20:52
greghaynesawesome, was just trying to figure that out20:53
greghaynesour whole SSL story is in a pretty sorry state20:53
SpamapSSo we can go one of two ways. We can make a default SSL port for admin v2 keystone.20:53
SpamapSOr we can just not use SSL for init-keystone's v2 ops20:53
*** jcoufal has joined #tripleo20:53
SpamapSI vote for the latter20:54
morganfainbergSpamapS, or you could spin up nginx and make it SSL terminate for you :P20:54
SpamapSsince v2 is deprecated20:54
greghaynesHow are we doing it with v3?20:54
SpamapSmorganfainberg: thats what the former is, but stunnel, not nginx20:54
morganfainberg*cough* not deprecated - frozen *cough*20:54
SpamapSgreghaynes: v3 all goes through same stunnel-ized port20:54
SpamapSmorganfainberg: my heat engine logs in icehouse said different. ;)20:54
SpamapSit's been undeprecated. :-P20:54
greghaynesso if we wanted to ssl-ify it wed need to make a new stunnel port for v220:54
morganfainbergSpamapS, that should *not* be the case in icehouse stable20:55
SpamapSwhy actually, isn't v3 being pushed hard into all projects?20:55
morganfainbergwell, from the keystone perspective20:55
SpamapSmorganfainberg: yeah it was reverted late. :)20:55
SpamapSI'm just wanking20:55
morganfainbergSpamapS, it is - its a slow adoption, and there are things that are -- well bad ... just grep for is_admin in openstack20:55
*** spzala has quit IRC20:55
morganfainbergthen cry a little.20:55
morganfainberglook for hard-coded references :P20:55
greghaynesEither way, its broken right now and !ssl is better than broken when ssl is on20:55
morganfainbergit's *slow*20:55
greghaynesso +1 on that :)20:56
morganfainbergeither way works for me20:56
SpamapSgreghaynes: yeah, going to fix now20:56
morganfainbergbut least resistence is probably no-ssl on init20:56
SpamapSinit-keystone is run once20:58
SpamapSwhen the cloud is birthed20:58
SpamapSfrom the box that birthed it via pxe20:58
SpamapSso yeah, SSL is overkill for this20:58
tchaypoI want to close on the grounds that there are no specific actions for us to take at present; it’s more like a guiding principle for design than a bug20:58
uvirtbotLaunchpad bug 1188067 in tripleo "* listening services available on all addresses" [High,In progress]20:59
tchaypoand I think I’ve answered my own question - that means it’s invalid because it’s “not a bug"20:59
greghaynestchaypo: +1, reopen more specific ones20:59
*** penick has joined #tripleo21:00
greghayneswhy on earth are we doing horizon css compilation as part of apache startup21:00
SpamapStchaypo: agree, metabugs should be closed or converted to specs and/or documentation.21:01
SpamapSgreghaynes: cuz nodejs is bad fo yo healf21:01
greghaynesits node? I thought it was ruby21:01
greghayneswhich is only ever so slightly less of a pain21:01
SpamapSgreghaynes: I'm just playin, player21:02
greghaynesoh, well scss *is* bad for your health. No joke.21:02
SpamapScrap I need to figure out how to get /mnt to be /dev/sd$SOMETHINGELSE21:03
SpamapSthese blades have tiny DAS, but giant SAN LUN's21:03
*** dprince has quit IRC21:04
SpamapSgreghaynes: in this case I think I can just hack in an extra element that replaces use-ephemeral21:04
SpamapSsince those LUN's will never be touched21:04
openstackgerritMerged openstack/tripleo-image-elements: os-net-config: log at log level info again...
openstackgerritMerged stackforge/kolla: Fix use of undefined KEYSTONE_SERVICE_HOST in Ceilometer
openstackgerritMerged stackforge/kolla: Add gitignore file
openstackgerritMerged stackforge/kolla: Remove unused libvirt-service.json file
openstackgerritMerged stackforge/kolla: Reduce code duplication in
openstackgerritClint 'SpamapS' Byrum proposed openstack/os-cloud-config: Do not use https for keystone v2 admin
greghaynesoh wow
uvirtbotLaunchpad bug 1396695 in tripleo "Side by side Tripleo installs can break each other's keepalived clusters" [High,In progress]21:15
SpamapShm, setup-neutron doesn't seem to respect OS_CACERT21:16
SpamapSoh weird it's self signed21:18
*** Marga_ has quit IRC21:21
openstackgerritJonathan Brownell proposed openstack/tripleo-image-elements: Open firewall for VNC to Nova compute instances
*** rwsu has quit IRC21:36
*** Shrews has quit IRC21:37
*** Hefeweizen has joined #tripleo21:38
*** julim has quit IRC21:38
*** rwsu has joined #tripleo21:42
*** mrunge has quit IRC21:42
*** lblanchard has quit IRC21:43
*** marun has joined #tripleo21:44
*** chuckC_ has joined #tripleo21:45
SpamapSnice.. just updated the cert without having to redeploy21:48
*** lucas-dinner has quit IRC21:49
*** jdob has quit IRC21:59
*** untriaged-bot has joined #tripleo22:00
untriaged-botUntriaged bugs so far:22:00
uvirtbotLaunchpad bug 1402928 in tripleo "RabbitClientUseSSL type error in overcloud-without-mergepy.yaml " [Undecided,New]22:00
uvirtbotLaunchpad bug 1402795 in tripleo "Keystone token_flush  cron job running at the same time in the controller node causes deadlock" [Undecided,In progress]22:00
uvirtbotLaunchpad bug 1387487 in tuskar "tripleo-image-elements/elements/tuskar-ui /os-refresh-config /post-configure.d/101-tuskar-ui  needs authentication from Keystone" [Undecided,New]22:00
*** untriaged-bot has quit IRC22:00
*** rwsu has quit IRC22:03
tchaypoon the topic of
tchaypoJonathan (and I think one or two other people as well) have a whole bunch of similar patches that open up ports22:05
tchaypoknowing where the patches come from I’m certain that they’re useful patches22:06
*** pcrews has quit IRC22:06
tchaypomy question is how much scrutiny we should be applying to these; when i see 6 or so different requests for punching holes in the firewall I start seeing pictures of swiss cheese..22:07
*** Marga_ has joined #tripleo22:08
*** rlandy has quit IRC22:08
*** marun has quit IRC22:09
*** jcoufal has quit IRC22:10
*** marun has joined #tripleo22:10
*** jehb_ has quit IRC22:11
tchayponaturally it’s just as I hit the officially-brain-fried point that i notice the bugs assigned to me22:22
tchaypowhich are the ones I could easily have made progress on squashing, instead of reading random other bugs22:22
tchayposomething to work on tomorrow.22:23
*** prad has quit IRC22:26
*** spzala has joined #tripleo22:27
*** zz_avozza is now known as avozza22:28
SpamapStchaypo: thanks for squashing!22:33
*** rwsu has joined #tripleo22:33
*** jang2 has quit IRC22:37
openstackgerritJonathan Brownell proposed openstack/tripleo-image-elements: Add iptables rule for nova and mysql
*** jtomasek has quit IRC22:37
openstackgerritBen Nemec proposed openstack/diskimage-builder: Don't trace RHEL registration scripts
*** chlong_hometime is now known as chlong22:44
*** spzala has quit IRC22:54
*** rhallisey has quit IRC22:55
*** marun has quit IRC23:13
*** dsneddon has joined #tripleo23:19
*** avozza is now known as zz_avozza23:39
*** esp has joined #tripleo23:49
*** andreaf has quit IRC23:50
*** pradeep1 has joined #tripleo23:50
openstackgerritIan Wienand proposed openstack/diskimage-builder: Standarise tracing for scripts

Generated by 2.14.0 by Marius Gedminas - find it at!