Thursday, 2015-03-12

clarkbnotmyname: any other gotchas to consider?00:00
notmynamewho told you that?00:00
clarkbnotmyname: krtaylor00:00
jgriffithclarkb: hmm.. so that *should* work no?  I'm trying to figure out how to get rid of: http://paste.openstack.org/show/191718/00:00
notmynameclarkb: I don't think that's right, but let me check00:00
clarkbjgriffith: oh thats from the pretty_tox.sh script that mtreinish wrote00:00
*** banix has joined #openstack-infra00:01
clarkbjgriffith: I think there is a flag to turn that off in the script let me check00:01
*** _nadya_ has joined #openstack-infra00:01
*** sabeen has quit IRC00:01
anteayanotmyname: context 19:04:42 <jeblair> #topic Priority Efforts (Swift logs) http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-03-10-19.01.log.html00:01
*** michchap_ has quit IRC00:01
*** Longgeek has joined #openstack-infra00:03
jgriffithclarkb: I thought that was calling testr directly?00:03
clarkbjhesketh: subunit-trace appears to be the source of that but I do not see where that comes from00:03
*** otter768 has joined #openstack-infra00:03
clarkber jgriffith ^00:03
clarkbjgriffith: its calling testr but piping all output through subunit-trace which does the rendering00:04
jgriffithclarkb: got ya...00:04
jgriffithso run_tests script does that00:04
*** ujuc has joined #openstack-infra00:05
clarkbjgriffith: https://git.openstack.org/cgit/openstack/cinder/tree/run_tests.sh#n153 ya it uses subunit-trace too00:05
clarkbbut if I install python-subunit I don't get a subunit-trace00:05
jheskethnotmyname: https://review.openstack.org/#/c/163188/00:05
*** _nadya_ has quit IRC00:05
clarkblifeless: ^ any idea where subunit-trace comes from?00:05
*** annegentle has joined #openstack-infra00:06
jgriffithclarkb: haha... don't try just simply removing the wrapper :)00:06
*** bhuvan has quit IRC00:06
*** Longgeek has quit IRC00:07
clarkblooks like tempest lib installs subunit-trace, that is not super obvious00:07
jgriffithclarkb: hehe.. yeah, and then that in turn has the print output always00:08
*** otter768 has quit IRC00:08
clarkbjgriffith: https://github.com/openstack/tempest-lib/blob/master/tempest_lib/cmd/subunit_trace.py#L231-L244 those appear to be your options00:09
jgriffithclarkb: aha!  Excellent00:09
jgriffiththanks00:09
clarkbI don't think any of them provide the don't print stderr/stdout while running00:09
clarkbbut you can probably add taht in00:09
clarkb(or stop using pretty_tox.sh)00:09
clarkbbut pretty_tox.sh seems to be a thing now :/00:09
jgriffithclarkb: well, I'm still unclear on how we get to prettytox TBH00:10
notmynameclarkb: you can update the object expiry time00:10
clarkbjgriffith: tox.ini does it at https://git.openstack.org/cgit/openstack/cinder/tree/tox.ini#n16 then run_tests.sh emulates it but does not call it directly00:10
clarkbnotmyname: cool thats even better00:10
clarkbnotmyname: are there any gotchas you would point out?00:10
jgriffithclarkb: aha!00:11
notmynameclarkb: if you update the expiry time very close to the original delete time, you might not catch it in time00:11
clarkbnotmyname: I think we can live with that (we were also prepared to lie with not being able to update)00:12
notmynameclarkb: well, let me check one thing on that00:12
greghaynesclarkb: not sure if you saw my comment but my deadlock issues seem to happen only if I run testtools.run instead of tix00:12
greghayneser, instead of tox00:12
jgriffithwonder if maybe I'm chasing this needlessly... regardless, clarkb thanks for all the info00:12
openstackgerritSpencer Krum proposed openstack-infra/puppet-jenkins: Use @varname to supress warnings  https://review.openstack.org/16365000:12
clarkbgreghaynes: I did, not sure why that would be other than that the .testr.conf sets an env up that may be different than what you get via testtools directly00:13
greghayneshrmm00:13
notmynameclarkb: oh, nevermind. that corner case is covered in the swift code. not edge cases there00:13
clarkbgreghaynes: specifically log and stdout/stderr capturing00:13
notmynameclarkb: ie if you change the exipry time one second before it was set to expire, it won't expire00:13
greghaynesclarkb: hrmmm00:13
clarkbnotmyname: nice00:14
SpamapShrm, there seems to be something deep within pyOpenSSL that doesn't work with the gear py3 tests00:14
clarkbgreghaynes: try running it with those env vars set? see .testr.conf for the full list00:14
clarkbgreghaynes: curious if that makes a difference00:14
*** fdegir has quit IRC00:14
*** hdd_ has joined #openstack-infra00:14
anteayanotmyname: thanks, did you want to leave a review on 163188?00:15
*** boris-42 has joined #openstack-infra00:15
jlvillalclarkb: What is the correct procedure to ask for an infra patch to be reviewed that is blocking the openstack/python-ironicclient gate?  This is the patch: https://review.openstack.org/16360000:15
notmynameanteaya: hang on! I'm still looking at stuff! ;-)00:15
anteayanotmyname: sorry don't mean to rush00:15
*** xgerman has quit IRC00:16
notmynameclarkb: so, no, I don't think there are any corner cases to worry about. after the object expires, swift will return a 404 for it and a background process will eventually reap it00:16
clarkbnotmyname: that is awesome00:16
jgriffithclarkb: think you meant me... and yep00:16
jgriffithclarkb: setting those to False/0 eliminates the capture00:17
jgriffithclarkb: thanks a bunch for the help00:17
clarkbjgriffith: np00:17
jgriffithclarkb: think I'll add an option with the default to eat them00:17
clarkbgreghaynes: I just ran `.tox/py27/bin/python -m testtools.run nodepool.tests.test_nodepool` without trouble on your change00:18
greghaynes:/00:18
clarkbis that how you invoke testtools.run?00:18
clarkbmaybe i should try the specific test metho?00:18
greghaynesI was singleshotting the method00:18
clarkbjlvillal: requesting here is fine I will review it00:18
jlvillalclarkb: Thank you :)00:19
greghaynessounds like it might just be that I have some derp in my test environment00:19
greghayneswhich is very plausible00:19
*** Longgeek has joined #openstack-infra00:19
clarkbya `.tox/py27/bin/python -m testtools.run nodepool.tests.test_nodepool.TestNodepool.test_dib_upload_fail` works too00:20
clarkbjlvillal: maybe we should stop using httpretty?00:20
clarkbjlvillal: aiui its a constant source of problems00:21
*** bdpayne has quit IRC00:21
*** melwitt has quit IRC00:21
jlvillalclarkb: Uh....  That isn't really my call.  Probably would have to be something for the ironic-core team and any other projects which use it to decide.00:21
*** mwagner_lap has joined #openstack-infra00:21
clarkbjlvillal: sure00:21
* ianw happy i think i came to the same conclusion as lifeless , just half an hour later00:22
notmynameclarkb: anteaya: +1 in gerrit00:22
anteayanotmyname: thank you!00:22
anteayaand thanks for checking the update object options00:22
*** ddieterly has joined #openstack-infra00:23
anteayajhesketh: since I'm looking at the swift logging things, did you want to address clarkb's comment? https://review.openstack.org/#/c/163276/300:24
*** ddieterl_ has joined #openstack-infra00:24
*** ddieterly has quit IRC00:24
*** Longgeek has quit IRC00:26
jheskethclarkb, anteaya: Replied to the comment on https://review.openstack.org/#/c/163276/00:26
*** banix has quit IRC00:26
*** patrickeast has joined #openstack-infra00:27
anteayajhesketh: thanks00:27
clarkbjhesketh: oh in that case I can flip m vote00:28
anteayamtreinish: you still around to look at this patch which adds a new non-voting job to tempest that just tests the database tempest tests with trove: https://review.openstack.org/#/c/163521/100:28
jheskethif you're happy with it00:28
clarkbyup00:28
*** patrickeast has quit IRC00:29
mtreinishanteaya: sure I'll take a look now00:29
anteayaany reason 163276 can't merge?00:29
anteayamtreinish: thanks00:29
mtreinishjgriffith: did you figure out your issue with subunit-trace?00:29
anteayaclarkb: I'm mostly concerned with timing on the image builds00:30
*** mpopow__ has joined #openstack-infra00:30
*** gyee has quit IRC00:31
openstackgerritMerged openstack-infra/project-config: Add option to set swift delete after header  https://review.openstack.org/16318800:31
*** ashleighfarnham has quit IRC00:31
*** annegentle has quit IRC00:31
mtreinishjgriffith: looking at the backscroll if you use --no-failure-debug that will stop printing stdout during the run00:32
*** patrickeast has joined #openstack-infra00:32
mtreinishjgriffith: just make sure you have -f flag set too otherwise you won't see any tracebacks on failure00:32
SpamapSdhellmann: around?00:33
SpamapSdhellmann: https://github.com/python/cpython/blob/master/Lib/idlelib/PyShell.py#L1342 <-- that looks like a bug00:33
SpamapSdhellmann: shouldn't that be bytes, not str, for python3?00:34
*** mpopow_ has quit IRC00:34
mtreinishanteaya: +1 on both commits in that series00:35
*** ghostpl_ has joined #openstack-infra00:35
SpamapSahh n/m00:35
SpamapSdhellmann: ignore me, had it backwards00:35
* SpamapS is still confused why files want strings, and not bytes.. gah00:35
*** dimtruck is now known as zz_dimtruck00:36
*** esker has joined #openstack-infra00:36
*** annegentle has joined #openstack-infra00:38
clarkbanteaya: they will build in the morning so first thing to do is check on it tomorrow00:38
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/gear: Resolve python3 test failures  https://review.openstack.org/15814700:38
clarkbanteaya: shouldn't be a major problem00:38
anteayamtreinish: thank you00:39
anteayaclarkb: k thanks00:40
anteayaclarkb: I'm thinnking after these go in we should wait for the python-tests one00:40
anteayato see and make sure these work as expected00:40
anteayawhat do you think?00:40
*** woodster_ has joined #openstack-infra00:41
*** vishy has quit IRC00:41
clarkbanteaya: yup00:41
*** vishy has joined #openstack-infra00:42
*** ghostpl_ has quit IRC00:42
anteayak00:43
*** dannywilson has joined #openstack-infra00:44
*** kiseok7_ has quit IRC00:46
*** mtanino has quit IRC00:47
*** tsg_ has quit IRC00:47
*** annegentle has quit IRC00:48
*** annegentle has joined #openstack-infra00:49
*** zz_dimtruck is now known as dimtruck00:50
*** kiseok7_ has joined #openstack-infra00:50
openstackgerritMerged openstack-infra/project-config: zuul-swift-upload use logging for debug msgs  https://review.openstack.org/16327600:51
*** bhuvan has joined #openstack-infra00:51
*** tiswanso has joined #openstack-infra00:51
*** esker has quit IRC00:52
*** esker has joined #openstack-infra00:52
*** tiswanso has quit IRC00:54
openstackgerritMerged openstack-infra/project-config: Add non-voting trove tempest job  https://review.openstack.org/16352100:54
*** tiswanso has joined #openstack-infra00:55
*** chlong has quit IRC00:55
*** chlong has joined #openstack-infra00:55
*** rfolco_ has quit IRC00:59
*** marun has joined #openstack-infra01:00
*** dimtruck is now known as zz_dimtruck01:01
openstackgerritBrian Curtin proposed openstack/requirements: Add httpretty 0.8.8  https://review.openstack.org/16366201:03
*** rwsu is now known as rwsu-afk01:04
*** marun has quit IRC01:05
openstackgerritMerged openstack-infra/project-config: Add skip if for tempest running tempest on tempest commits  https://review.openstack.org/16299301:05
*** annegentle has quit IRC01:08
clarkbabout to sign off for the day but I notice that the status: ERROR messages from hpcloud seem to have stopped01:12
*** SumitNaiksatam has quit IRC01:13
clarkbso we likely need to figure out why/how ports leak01:13
lifelessclarkb: subunit-trace is in tempestlib01:13
clarkblifeless: ya I finally tracked that down01:14
lifelessI wrotes it for tempest01:14
lifelessits being reused now01:14
lifelessianw: my conclusion about?01:14
*** gokrokve has quit IRC01:15
anteayaclarkb: happy evening and yay cloud01:15
clarkblifeless: the testtools 1.7.0 thing01:15
*** AJaeger has quit IRC01:18
lifelessoh right, I see the mail.01:18
*** sigmavirus24 is now known as sigmavirus24_awa01:19
clarkbmordred: did ansible fix ever get pushed?01:20
clarkbmordred: we should try to get that in soon (tomorrow at this point) so that puppet works01:20
*** mtanino has joined #openstack-infra01:21
openstackgerritMerged openstack/requirements: Avoid httpretty 0.8.8 as it can break unittests  https://review.openstack.org/16360001:22
*** unicell has quit IRC01:25
*** markvoelker has quit IRC01:27
*** ivar-laz_ has joined #openstack-infra01:27
*** markvoelker has joined #openstack-infra01:27
*** ivar-lazzaro has quit IRC01:30
*** AJaeger has joined #openstack-infra01:30
*** ivar-laz_ has quit IRC01:32
*** markvoelker has quit IRC01:32
openstackgerritJoshua Harlow proposed openstack/requirements: Add 'shared_lock' package  https://review.openstack.org/16366601:32
*** hdd_ has quit IRC01:34
*** ayoung has joined #openstack-infra01:35
*** asettle-mtg is now known as asettle01:37
*** macjack has joined #openstack-infra01:37
*** Alexandra_ has joined #openstack-infra01:38
*** sdake_ has quit IRC01:42
*** Longgeek has joined #openstack-infra01:52
*** dims_ has quit IRC01:53
*** Longgeek has quit IRC01:56
*** salv-orlando has quit IRC01:58
*** yamahata has quit IRC01:59
*** armax has quit IRC02:01
*** macjack has quit IRC02:03
*** otter768 has joined #openstack-infra02:04
*** mpopow__ has quit IRC02:05
*** tqtran_ has quit IRC02:06
*** shardy has quit IRC02:07
*** shardy has joined #openstack-infra02:07
*** armax has joined #openstack-infra02:08
*** otter768 has quit IRC02:08
*** nelsnelson has joined #openstack-infra02:10
*** banix has joined #openstack-infra02:11
*** hdd_ has joined #openstack-infra02:14
*** patrickeast has quit IRC02:18
*** yfried|afk is now known as yfried|pto02:20
*** Qiming has joined #openstack-infra02:21
*** ivar-lazzaro has joined #openstack-infra02:23
*** jyuso1 has joined #openstack-infra02:25
*** ghostpl_ has joined #openstack-infra02:26
*** yfried|pto is now known as yfried|afk02:30
ianwlifeless: version pins for unittest202:33
*** ghostpl_ has quit IRC02:36
jeblairclarkb, mordred, fungi: it looks like over the post few hours we've been able to create many more hpcloud nodes02:42
*** achanda has quit IRC02:42
jeblairclarkb, mordred, fungi: so i think the port/ip leakages were probably the bulk of the problem02:42
*** nelsnelson has quit IRC02:42
*** Ryan_Lane has quit IRC02:43
*** zz_dimtruck is now known as dimtruck02:43
*** ivar-lazzaro has quit IRC02:43
*** david-lyle is now known as david-lyle_afk02:45
*** sdake_ has joined #openstack-infra02:48
*** yfried|afk is now known as yfried|pto02:50
*** oomichi has joined #openstack-infra02:52
*** unicell has joined #openstack-infra02:53
*** marun has joined #openstack-infra02:53
*** dims_ has joined #openstack-infra02:54
*** gokrokve has joined #openstack-infra02:54
*** spzala has quit IRC02:55
*** salv-orlando has joined #openstack-infra02:59
*** dims_ has quit IRC02:59
*** yfried|pto is now known as yfried|afk03:00
*** zz_jgrimm is now known as jgrimm03:03
openstackgerritHuang Rui proposed openstack-infra/project-config: Create nova-zvm-virt-driver project on StackForge  https://review.openstack.org/16368703:07
*** ddieterl_ has quit IRC03:09
*** dboik_ has quit IRC03:10
*** dkehn has quit IRC03:10
*** salv-orlando has quit IRC03:10
*** Qiming_ has joined #openstack-infra03:11
*** yfried|afk is now known as yfried|pto03:12
*** Qiming__ has joined #openstack-infra03:12
*** Qiming has quit IRC03:14
*** Ryan_Lane has joined #openstack-infra03:14
*** dkehn has joined #openstack-infra03:15
*** Qiming_ has quit IRC03:16
*** koolhead17 has joined #openstack-infra03:17
*** salv-orlando has joined #openstack-infra03:19
*** nelsnelson has joined #openstack-infra03:19
*** ivar-lazzaro has joined #openstack-infra03:19
*** SumitNaiksatam has joined #openstack-infra03:22
*** radez_g0n3 is now known as radez03:22
*** salv-orlando has quit IRC03:23
*** sputnik13 has quit IRC03:24
*** mtanino has quit IRC03:24
*** salv-orlando has joined #openstack-infra03:24
*** resker has joined #openstack-infra03:27
*** salv-orlando has quit IRC03:29
*** esker has quit IRC03:31
*** dannywilson has quit IRC03:31
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Modifying devstack plugin job template  https://review.openstack.org/16332303:33
*** Longgeek has joined #openstack-infra03:37
*** yfried|pto is now known as yfried|afk03:38
*** sdake_ has quit IRC03:38
*** Ryan_Lane has quit IRC03:40
*** yfried|afk is now known as yfried|pto03:41
*** banix has quit IRC03:41
*** hdd_ has quit IRC03:42
*** claudiub has quit IRC03:46
*** nelsnelson has quit IRC03:51
*** yfried|pto is now known as yfried|afk03:55
*** yfried|afk is now known as yfried|pto03:56
*** Ryan_Lane has joined #openstack-infra04:00
*** funzo has joined #openstack-infra04:01
*** Longgeek has quit IRC04:03
*** otter768 has joined #openstack-infra04:05
*** funzo has quit IRC04:06
*** sdake_ has joined #openstack-infra04:08
*** garyh has quit IRC04:09
*** ddieterly has joined #openstack-infra04:10
*** otter768 has quit IRC04:10
*** oomichi has quit IRC04:13
*** ddieterly has quit IRC04:14
*** Qiming_ has joined #openstack-infra04:16
*** gokrokve_ has joined #openstack-infra04:18
*** Qiming__ has quit IRC04:19
*** gokrokve has quit IRC04:21
*** radez is now known as radez_g0n304:21
*** ghostpl_ has joined #openstack-infra04:21
*** amotoki has joined #openstack-infra04:22
*** gokrokve_ has quit IRC04:22
*** devlaps has quit IRC04:25
*** ghostpl_ has quit IRC04:31
*** achanda has joined #openstack-infra04:31
*** ivar-lazzaro has quit IRC04:33
*** ivar-lazzaro has joined #openstack-infra04:34
*** macjack has joined #openstack-infra04:37
*** ivar-lazzaro has quit IRC04:39
*** ddieterly has joined #openstack-infra04:40
*** erlon has quit IRC04:41
*** ddieterly has quit IRC04:44
*** unicell has quit IRC04:44
*** unicell has joined #openstack-infra04:45
*** marun has quit IRC04:47
*** pblaho has quit IRC04:47
*** salv-orlando has joined #openstack-infra04:49
*** marun has joined #openstack-infra04:50
*** yfried|pto is now known as yfried|afk04:50
*** nelsnelson has joined #openstack-infra04:53
*** sdake__ has joined #openstack-infra04:55
*** Qiming_ is now known as Qiming04:56
*** Alexandra_ has joined #openstack-infra04:57
*** gokrokve has joined #openstack-infra04:58
*** sdake_ has quit IRC04:58
*** salv-orlando has quit IRC05:00
*** hdd_ has joined #openstack-infra05:02
*** teran_ has joined #openstack-infra05:02
*** Qiming has quit IRC05:03
*** dimtruck is now known as zz_dimtruck05:04
*** yamahata has joined #openstack-infra05:04
*** gokrokve_ has joined #openstack-infra05:04
*** achanda has quit IRC05:04
*** david-ly_ has joined #openstack-infra05:04
*** omnibus7 has joined #openstack-infra05:05
*** macjack1 has joined #openstack-infra05:05
*** vryzhenkin has joined #openstack-infra05:05
*** dvorak has quit IRC05:05
*** kevinbenton_ has joined #openstack-infra05:05
*** jd__` has joined #openstack-infra05:05
*** spiffxp- has joined #openstack-infra05:05
*** _qb has quit IRC05:06
*** devanand1 has joined #openstack-infra05:07
*** GheRiver1 has joined #openstack-infra05:07
*** mestery_ has joined #openstack-infra05:07
*** SumitNaiksatam has quit IRC05:07
*** ubuntu2 has joined #openstack-infra05:08
*** sdake__ has quit IRC05:08
*** mestery has quit IRC05:08
*** camunoz_ has joined #openstack-infra05:08
*** SlickN1k has joined #openstack-infra05:09
*** terryw has joined #openstack-infra05:09
*** jyuso has quit IRC05:09
*** freerunner has quit IRC05:09
*** david-lyle_afk has quit IRC05:09
*** adreznec has quit IRC05:09
*** SlickNik has quit IRC05:09
*** jyuso has joined #openstack-infra05:09
*** camunoz has quit IRC05:09
*** otherwiseguy has quit IRC05:09
*** sreshetnyak has quit IRC05:09
*** cebruns has quit IRC05:09
*** sreshetnyak has joined #openstack-infra05:09
*** mikal_ has joined #openstack-infra05:09
*** devananda has quit IRC05:09
*** gokrokve has quit IRC05:09
*** macjack has quit IRC05:09
*** jd__ has quit IRC05:09
*** davidlenwell has quit IRC05:09
*** kevinbenton has quit IRC05:09
*** teran has quit IRC05:09
*** spiffxp_ has quit IRC05:09
*** aarefiev has quit IRC05:09
*** gmann has quit IRC05:09
*** GheRivero has quit IRC05:09
*** frickler has quit IRC05:09
*** ubuntu1 has quit IRC05:09
*** mikal has quit IRC05:09
*** frickler has joined #openstack-infra05:09
*** raginbajin has quit IRC05:09
*** SlickN1k is now known as SlickNik05:09
*** jd__` is now known as jd__05:09
*** cebruns has joined #openstack-infra05:09
*** adreznec has joined #openstack-infra05:10
*** gmann has joined #openstack-infra05:10
*** aarefiev has joined #openstack-infra05:10
*** yfried|afk is now known as yfried|pto05:11
*** raginbajin has joined #openstack-infra05:13
*** dvorak has joined #openstack-infra05:14
openstackgerritHuang Rui proposed openstack-infra/project-config: Create nova-zvm-virt-driver project on StackForge  https://review.openstack.org/16368705:19
*** marun has quit IRC05:19
*** yfried|pto has quit IRC05:20
*** marun has joined #openstack-infra05:22
*** markvoelker has joined #openstack-infra05:25
*** achanda has joined #openstack-infra05:28
*** markvoelker has quit IRC05:30
*** reed has quit IRC05:30
*** WendellM_ has quit IRC05:32
*** BharatK has joined #openstack-infra05:37
*** enikanorov has joined #openstack-infra05:40
*** woodster_ has quit IRC05:40
*** harlowja_ is now known as harlowja_away05:41
*** ddieterly has joined #openstack-infra05:41
*** enikanorov__ has quit IRC05:41
*** marun has quit IRC05:42
*** sputnik13 has joined #openstack-infra05:42
*** esker has joined #openstack-infra05:44
*** deepakcs has joined #openstack-infra05:44
*** nelsnelson has quit IRC05:44
*** Longgeek has joined #openstack-infra05:45
*** ddieterly has quit IRC05:45
*** resker has quit IRC05:46
*** macjack1 has quit IRC05:46
*** yamamoto has joined #openstack-infra05:47
*** woodster_ has joined #openstack-infra05:49
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Add a python-ironicclient src job  https://review.openstack.org/16363205:50
*** wuhg has joined #openstack-infra05:56
*** Ryan_Lane has quit IRC05:56
*** hdd_ has quit IRC06:00
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/16370306:01
*** mrmartin has joined #openstack-infra06:01
*** sdake_ has joined #openstack-infra06:02
*** sdake__ has joined #openstack-infra06:04
*** otter768 has joined #openstack-infra06:06
*** sdake_ has quit IRC06:08
*** sdake_ has joined #openstack-infra06:10
*** bknudson has quit IRC06:10
*** otter768 has quit IRC06:11
*** sdake__ has quit IRC06:13
*** sdake__ has joined #openstack-infra06:14
*** Longgeek has quit IRC06:18
*** sdake_ has quit IRC06:18
*** gokrokve has joined #openstack-infra06:20
*** gokrokve_ has quit IRC06:23
*** afazekas is now known as __afazekas06:25
*** sputnik13 has quit IRC06:26
*** markvoelker has joined #openstack-infra06:26
*** macjack has joined #openstack-infra06:26
*** yamahata has quit IRC06:26
*** sputnik13 has joined #openstack-infra06:28
*** markvoelker has quit IRC06:32
*** dims_ has joined #openstack-infra06:32
*** _nadya_ has joined #openstack-infra06:35
*** _nadya_ has quit IRC06:35
*** sdake_ has joined #openstack-infra06:37
*** krisgan has quit IRC06:37
*** dims_ has quit IRC06:38
*** Alexandra_ has quit IRC06:38
*** sabari has quit IRC06:40
*** sdake__ has quit IRC06:41
*** ddieterly has joined #openstack-infra06:42
*** ddieterly has quit IRC06:46
*** funzo has joined #openstack-infra06:47
*** sabari has joined #openstack-infra06:49
*** yfried|pto has joined #openstack-infra06:51
*** funzo has quit IRC06:51
*** camunoz_ has quit IRC06:53
*** yfried|pto has quit IRC06:55
*** yfried|pto has joined #openstack-infra06:56
*** yfried|pto is now known as yfried|afk06:57
*** sdake__ has joined #openstack-infra06:58
*** yfried|afk is now known as yfried|pto07:00
*** yfried|pto is now known as yfried07:00
*** sdake_ has quit IRC07:02
*** mrunge has joined #openstack-infra07:02
*** sabari has quit IRC07:04
*** armax has quit IRC07:06
*** sdake__ has quit IRC07:08
*** sdake has quit IRC07:08
*** achanda has quit IRC07:11
*** sabari has joined #openstack-infra07:13
*** sdake has joined #openstack-infra07:19
*** garyk has joined #openstack-infra07:19
*** stevemar has quit IRC07:22
*** fdegir has joined #openstack-infra07:22
*** achanda has joined #openstack-infra07:24
*** HeOS__ is now known as HeOS_07:25
*** jcoufal has joined #openstack-infra07:27
*** salv-orlando has joined #openstack-infra07:27
*** markvoelker has joined #openstack-infra07:28
*** markvoelker has quit IRC07:33
*** sabari has quit IRC07:33
*** yfried is now known as yfried|afk07:34
*** yfried|afk is now known as yfried07:35
*** salv-orlando has quit IRC07:35
*** sputnik13 has quit IRC07:37
*** salv-orlando has joined #openstack-infra07:39
*** sputnik13 has joined #openstack-infra07:39
*** garyk has quit IRC07:41
*** garyk has joined #openstack-infra07:41
*** tnovacik has joined #openstack-infra07:41
*** salv-orl_ has joined #openstack-infra07:42
*** ujuc has quit IRC07:42
*** ddieterly has joined #openstack-infra07:42
*** sdake_ has joined #openstack-infra07:42
*** salv-orlando has quit IRC07:43
*** achanda has quit IRC07:44
*** salv-orl_ has quit IRC07:46
*** ddieterly has quit IRC07:47
*** terryw has quit IRC07:48
openstackgerritGhanshyam Mann proposed openstack-infra/devstack-gate: Provide option to tests Nova against v2 API  https://review.openstack.org/16372507:49
*** sdake_ has quit IRC07:49
*** sdake has quit IRC07:49
*** stevelle has left #openstack-infra07:50
*** sabari has joined #openstack-infra07:51
*** otherwiseguy has joined #openstack-infra07:53
*** HeOS has quit IRC07:53
*** jlanoux has joined #openstack-infra07:54
*** sdake has joined #openstack-infra07:56
openstackgerrityolanda.robla proposed openstack-infra/puppet-jenkins: Add configuration for HipChat in jjb  https://review.openstack.org/16215207:58
*** scheuran has joined #openstack-infra07:59
*** woodster_ has quit IRC08:00
*** salv-orlando has joined #openstack-infra08:01
*** Longgeek has joined #openstack-infra08:03
*** jistr has joined #openstack-infra08:03
*** sputnik13 has quit IRC08:03
*** ghostpl_ has joined #openstack-infra08:04
*** markus_z has joined #openstack-infra08:04
*** salv-orlando has quit IRC08:06
*** afazekas_ has joined #openstack-infra08:06
*** sputnik13 has joined #openstack-infra08:07
*** otter768 has joined #openstack-infra08:07
*** ghostpl_ has quit IRC08:10
*** bhuvan has quit IRC08:10
*** otter768 has quit IRC08:11
*** Longgeek has quit IRC08:12
*** gokrokve has quit IRC08:14
*** gokrokve has joined #openstack-infra08:14
*** GheRiver1 is now known as GheRivero08:15
*** _shaps_ has joined #openstack-infra08:16
*** gokrokve has quit IRC08:19
*** yfried is now known as yfried|afk08:20
*** hashar has joined #openstack-infra08:22
*** koolhead17 has quit IRC08:24
*** garyk has quit IRC08:25
*** garyk has joined #openstack-infra08:25
*** sdake_ has joined #openstack-infra08:25
*** mrda is now known as mrda-away08:27
*** markvoelker has joined #openstack-infra08:29
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: Remove Nova v21 neutron job & make one for v2.0  https://review.openstack.org/16373108:29
*** markvoelker has quit IRC08:34
*** salv-orlando has joined #openstack-infra08:34
*** doug-fish has quit IRC08:36
*** arxcruz has joined #openstack-infra08:39
*** mpavone has joined #openstack-infra08:39
*** yfried|afk is now known as yfried08:40
*** sputnik13 has quit IRC08:41
*** ddieterly has joined #openstack-infra08:43
*** mpaolino has joined #openstack-infra08:44
*** gokrokve has joined #openstack-infra08:45
openstackgerritMike Heald proposed openstack-infra/project-config: Project for puppet-bandersnatch  https://review.openstack.org/16340208:45
*** ddieterly has quit IRC08:48
*** gokrokve has quit IRC08:50
*** yfried is now known as yfried|afk08:50
*** Ala has joined #openstack-infra08:51
*** spredzy|afk is now known as spredzy08:52
*** sdake__ has joined #openstack-infra08:54
*** yfried|afk is now known as yfried08:54
*** skolekonov has joined #openstack-infra08:55
*** sdake has quit IRC08:57
GheRiveromorning all09:00
openstackgerritThomas Morin proposed openstack-infra/project-config: Add networking-bagpipe-l2 project  https://review.openstack.org/16129009:01
*** ghostpl_ has joined #openstack-infra09:06
*** HeOS has joined #openstack-infra09:07
yfriedanyone knows about personalize gerrit boards?09:08
*** sputnik13 has joined #openstack-infra09:13
*** _nadya_ has joined #openstack-infra09:14
*** gokrokve has joined #openstack-infra09:14
*** ghostpl_ has quit IRC09:14
*** gokrokve has quit IRC09:15
*** gokrokve has joined #openstack-infra09:16
*** che-arne has joined #openstack-infra09:17
*** ominakov_ has joined #openstack-infra09:19
*** gokrokve has quit IRC09:20
openstackgerritMax Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper  https://review.openstack.org/15668109:21
*** yamamoto has quit IRC09:21
*** derekh has joined #openstack-infra09:23
*** teran_ has quit IRC09:24
openstackgerritMatthias Runge proposed openstack/requirements: Raise cap for Django  https://review.openstack.org/15535309:25
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens  https://review.openstack.org/15359109:29
*** markvoelker has joined #openstack-infra09:30
*** dizquierdo has joined #openstack-infra09:31
*** ihrachyshka has joined #openstack-infra09:32
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Added story types  https://review.openstack.org/16341209:33
*** markvoelker has quit IRC09:35
*** asselin has quit IRC09:37
*** asselin has joined #openstack-infra09:38
*** _shaps_ has quit IRC09:38
openstackgerritMax Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper  https://review.openstack.org/15668109:38
openstackgerritMerged openstack-infra/devstack-gate: Set up ssh_known_host based on hostname  https://review.openstack.org/13659609:39
*** sarob has quit IRC09:40
*** sarob has joined #openstack-infra09:42
*** ddieterly has joined #openstack-infra09:44
*** Qiming has joined #openstack-infra09:46
*** sarob has quit IRC09:47
*** ddieterly has quit IRC09:48
*** yfried is now known as yfried|afk09:49
*** sputnik13 has quit IRC09:49
*** yfried|afk is now known as yfried09:50
*** yamamoto has joined #openstack-infra09:51
*** dims__ has joined #openstack-infra09:54
*** teran has joined #openstack-infra09:54
*** Longgeek has joined #openstack-infra09:54
*** e0ne has joined #openstack-infra09:55
*** ssam2 has joined #openstack-infra09:56
*** e0ne is now known as e0ne_09:56
*** macjack has quit IRC09:57
*** ujuc has joined #openstack-infra09:58
*** teran has quit IRC09:58
*** ghostpl_ has joined #openstack-infra10:01
*** e0ne_ has quit IRC10:05
openstackgerritThomas Morin proposed openstack-infra/project-config: Add networking-bagpipe-l2 project  https://review.openstack.org/16129010:05
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: Enable Neutron in neutronclient functional test job  https://review.openstack.org/16374610:06
*** zz_johnthetubagu is now known as johnthetubaguy10:07
*** otter768 has joined #openstack-infra10:08
*** e0ne has joined #openstack-infra10:09
*** e0ne has quit IRC10:09
*** bharath_ has joined #openstack-infra10:10
*** e0ne has joined #openstack-infra10:10
bharath_Hi, can we add launchpad group as a user in this file https://github.com/stackforge/stackalytics/blob/master/etc/default_data.json?10:10
*** sputnik13 has joined #openstack-infra10:12
*** jp_at_hp has joined #openstack-infra10:13
*** otter768 has quit IRC10:13
*** Longgeek has quit IRC10:14
*** sputnik13 has quit IRC10:14
*** gokrokve has joined #openstack-infra10:16
openstackgerrityolanda.robla proposed openstack-infra/system-config: Install puppet modules on the right order  https://review.openstack.org/16375210:17
*** jlanoux has quit IRC10:18
*** salv-orl_ has joined #openstack-infra10:18
*** sputnik13 has joined #openstack-infra10:18
*** imcsk8 is now known as imcsk8|zZz10:18
*** salv-orlando has quit IRC10:20
*** Qiming has quit IRC10:21
*** gokrokve has quit IRC10:21
*** fdegir has quit IRC10:22
*** fdegir has joined #openstack-infra10:22
*** salv-orl_ has quit IRC10:22
*** jedimike has joined #openstack-infra10:23
*** kiseok7_ has quit IRC10:24
*** andreykurilin_ has joined #openstack-infra10:26
*** markvoelker has joined #openstack-infra10:31
*** BobBall_AWOL is now known as BobBall10:33
*** yamamoto has quit IRC10:35
*** markvoelker has quit IRC10:36
*** jistr has quit IRC10:36
*** ominakov_ has quit IRC10:38
*** che-arne has quit IRC10:38
*** che-arne has joined #openstack-infra10:39
*** pelix has joined #openstack-infra10:41
*** teran has joined #openstack-infra10:42
*** sarob has joined #openstack-infra10:43
*** teran_ has joined #openstack-infra10:44
*** ddieterly has joined #openstack-infra10:45
*** teran has quit IRC10:46
*** mpaolino has quit IRC10:48
*** ddieterly has quit IRC10:49
*** sarob has quit IRC10:49
AJaegerbharath_, infra team does not run stackalytics - not sure what'S the appropriate channel is10:51
*** pc_m has joined #openstack-infra10:51
*** skraynev has quit IRC10:52
*** jlanoux has joined #openstack-infra10:52
*** skraynev has joined #openstack-infra10:52
*** skraynev has quit IRC10:53
*** jistr has joined #openstack-infra10:53
*** skraynev has joined #openstack-infra10:53
openstackgerrityolanda.robla proposed openstack-infra/puppet-zuul: Add fetch_url and poll parametesr to zuul.conf  https://review.openstack.org/16259110:54
jedimikei'm writing the initial version of puppet-bandersnatch, bit confused about names of dependencies. I see storyboard has openstackci-httpd as a dependency, does that provide apache? I can't find that module anywhere, the naming conventions seem a bit weird because i can find puppet-httpd which calls itself openstackinfra-httpd... can anyone explain?10:54
vponomaryovbharath_: you can contribute to stackalytics as well as to any other openstack projects proposing gerrit change.10:58
*** erlon has joined #openstack-infra10:58
bharath_vponomaryov, AJaeger okay thank you, I will do that10:59
*** kaisers has joined #openstack-infra11:00
kaisersHi!11:02
kaisersDoes anybody working on a thirdparty CI here have issues fetching the gerrit eventstream today?11:02
*** skraynev_ has joined #openstack-infra11:03
*** skraynev_ has quit IRC11:03
*** garyk has quit IRC11:04
*** garyk has joined #openstack-infra11:05
*** rlandy has joined #openstack-infra11:06
*** ominakov_ has joined #openstack-infra11:13
*** gokrokve has joined #openstack-infra11:14
*** andreykurilin_ has quit IRC11:18
*** gokrokve has quit IRC11:19
*** Qiming has joined #openstack-infra11:20
Qimingcan some infra-cores help review this, thanks! https://review.openstack.org/#/c/163292/11:21
*** ominakov_ has quit IRC11:22
*** Qiming_ has joined #openstack-infra11:27
*** Qiming has quit IRC11:31
*** yfried is now known as yfried|afk11:31
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Add py27-scenario-unit tox non-voting job for Sahara  https://review.openstack.org/16377611:32
*** markvoelker has joined #openstack-infra11:33
*** hichihara has quit IRC11:34
*** ldnunes has joined #openstack-infra11:36
*** ominakov_ has joined #openstack-infra11:37
*** markvoelker has quit IRC11:37
BobBalldoesn't look like it kaisers11:41
BobBallmine looks fine11:41
*** yfried|afk is now known as yfried11:42
*** ominakov_ has quit IRC11:42
*** esker has quit IRC11:44
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens  https://review.openstack.org/15359111:45
*** ddieterly has joined #openstack-infra11:45
*** scheuran_ has joined #openstack-infra11:46
*** doug-fish has joined #openstack-infra11:47
*** markus_z has quit IRC11:49
*** mpavone has quit IRC11:49
*** scheuran has quit IRC11:49
*** ddieterly has quit IRC11:50
*** markus_z has joined #openstack-infra11:50
*** isviridov_vction is now known as isviridov11:54
*** rfolco has joined #openstack-infra11:56
*** kgiusti has joined #openstack-infra11:58
*** spzala has joined #openstack-infra11:58
openstackgerritMerged openstack-infra/storyboard: Removed Process Pool execution.  https://review.openstack.org/16137411:59
*** Longgeek has joined #openstack-infra11:59
openstackgerritMerged openstack-infra/storyboard: Added python-dateutil  https://review.openstack.org/16101711:59
*** tsg_ has joined #openstack-infra11:59
*** salv-orlando has joined #openstack-infra12:00
*** mpavone has joined #openstack-infra12:01
*** adalbas has joined #openstack-infra12:02
*** yfried is now known as yfried|afk12:03
*** cdent has joined #openstack-infra12:03
*** yfried|afk is now known as yfried12:04
*** mwagner_lap has quit IRC12:04
*** salv-orlando has quit IRC12:08
*** baoli has joined #openstack-infra12:08
*** salv-orlando has joined #openstack-infra12:08
*** otter768 has joined #openstack-infra12:09
*** funzo has joined #openstack-infra12:11
*** otter768 has quit IRC12:14
*** gokrokve has joined #openstack-infra12:14
*** markvoelker has joined #openstack-infra12:15
*** esker has joined #openstack-infra12:15
*** gokrokve has quit IRC12:16
*** marcusvrn has joined #openstack-infra12:16
*** gokrokve has joined #openstack-infra12:16
*** marun has joined #openstack-infra12:17
*** Longgeek has quit IRC12:19
*** salv-orlando has quit IRC12:20
*** gokrokve has quit IRC12:21
*** skraynev has quit IRC12:23
*** skraynev has joined #openstack-infra12:23
*** sdake has joined #openstack-infra12:23
*** sdake__ has quit IRC12:23
*** tiswanso has quit IRC12:29
*** marcusvrn has quit IRC12:30
*** marcusvrn has joined #openstack-infra12:32
*** johnthetubaguy is now known as zz_johnthetubagu12:35
*** e0ne is now known as e0ne_12:35
*** dims__ has quit IRC12:35
*** dims_ has joined #openstack-infra12:36
*** spredzy is now known as spredzy|afk12:36
*** bswartz has quit IRC12:37
*** dprince has joined #openstack-infra12:37
kaisersBobBall: ok, thanks!12:39
*** Longgeek has joined #openstack-infra12:39
openstackgerritMatthias Runge proposed openstack/requirements: Raise cap for Django  https://review.openstack.org/15535312:41
*** marun has quit IRC12:41
*** aysyd has joined #openstack-infra12:43
*** michchap has joined #openstack-infra12:43
*** e0ne_ has quit IRC12:45
mkj_wkfungi: found that neither pexpect nor local reimplementation will help with my test case problem, because the output is stolen for logging by the test suite12:45
*** ddieterly has joined #openstack-infra12:45
mkj_wkfungi: wondering whether there's something in the testware that would let me wait for logged output12:46
*** deepakcs has quit IRC12:46
*** ujuc has quit IRC12:46
*** vipul has quit IRC12:48
*** radez_g0n3 is now known as radez12:48
*** sdake has quit IRC12:48
*** vipul has joined #openstack-infra12:48
*** sdake has joined #openstack-infra12:49
*** devanand1 has quit IRC12:49
*** devananda has joined #openstack-infra12:50
*** esker has quit IRC12:50
*** bharath_ has quit IRC12:51
*** hdd_ has joined #openstack-infra12:51
*** mwagner_lap has joined #openstack-infra12:52
kaisersBobBall: FYI - Seems to be project related. I was using ci-sandbox for testing which had the issue. Experimenting with cinder works...12:52
BobBallFair enough12:54
*** zhiwei has joined #openstack-infra12:56
*** amotoki has quit IRC12:57
*** esker has joined #openstack-infra12:59
*** zhiwei has quit IRC12:59
*** yfried_ has joined #openstack-infra13:00
*** saper has joined #openstack-infra13:01
*** pblaho has joined #openstack-infra13:01
*** pblaho has quit IRC13:02
*** _nadya_ has quit IRC13:02
*** pblaho_ has joined #openstack-infra13:02
fungimkj_wk: did you see my question yesterday about python 2.x vs 3.x (input vs raw_input)?13:02
*** _nadya_ has joined #openstack-infra13:02
*** dalgaaf has joined #openstack-infra13:03
*** esker has quit IRC13:03
*** yfried__ has joined #openstack-infra13:03
*** dboik has joined #openstack-infra13:03
*** hdd_ has quit IRC13:03
*** yfried has quit IRC13:03
fungiit seemed like you had said something earlier about it working in 3.x and not 2.x (or was it the other way around?)13:03
*** bswartz has joined #openstack-infra13:04
*** hdd_ has joined #openstack-infra13:05
*** ddieterly has quit IRC13:05
*** dboik_ has joined #openstack-infra13:05
*** yfried_ has quit IRC13:05
mkj_wkfungi: I didn't see the question; my only 2 vs. 3 thing was whether pexpect was already in the stack13:06
mkj_wkbut now I understand that pexpect wouldn't fix it either13:06
mkj_wkeven unbuffered Popen and a prod.stdout.read(1) hangs forever13:07
mkj_wkwhen the timeout hits, all the output from the process is in the test log13:07
*** asselin_ has joined #openstack-infra13:07
*** mestery_ has quit IRC13:08
mkj_wkI don't yet understand how that happens with subprocess.Popen since neither Popen nor subprocess.PIPE is mocked13:08
*** dboik has quit IRC13:08
mkj_wkfungi: all the advice I've seen on teh intarwebs about testing raw_input in subprocesses is "doctor doctor it hurts when I do this / don't do that then" advice that says you should not be invoking a subprocess, and should instead call the main method and mock out raw_input13:09
mkj_wkwhen I'm writing functional tests for my own work that's what I do13:09
fungimkj_wk: got it, so input() isn't problematic but raw_input() is?13:10
mkj_wkbut I hesitate to convert the entire git-review functional test suite to invoke main13:10
mkj_wkwell, I was only searching regarding raw_input :)13:10
mkj_wksince that's what do_input is an alias for13:10
fungipoint being we choose one or the other depending on which python major version is calling it13:11
fungiyeah13:11
mkj_wkif the functional tests wrapped cmd:_main() in a chdir we could just mock do_input13:11
mkj_wk(that's what I do)13:11
*** dboik_ has quit IRC13:11
mkj_wkbut I'm guessing there's a reason that git-review doesn't do that; I just don't know what it is13:12
fungiwell, i'm not at all opposed to a refactoring of the testsuite that's there. it's sort of hackish to start with but was really just a placeholder for better structured testing13:12
mkj_wkif you know the right way to get at the stolen output, I could do an "expect" o ntop of that13:12
*** yamamoto has joined #openstack-infra13:13
mkj_wkI also hook up coverage reporting -- helps me be sure I covered at least all the lines when adding new code13:13
fungi(before that functional test set it was me running a script to exercise git-review against a dev gerrit from the command line, and before that it was people running through an actual set of testing instructions)13:14
*** gokrokve has joined #openstack-infra13:14
fungiwe do have a coverage job we can add to it pretty trivially i think, though with the current structure i don't know if it would report very accurately13:15
mkj_wkI don't know how what I do would work under tox/testr13:16
jedimikei'm writing a new module, puppet-bandersnatch. what dependencies should i use in the metadata to make sure i have apache and logrotate available? (some of the naming conventions are confusing me)13:17
mkj_wkUsing nose, I always invoke with --with-coverage --cover-package <mypackage>13:17
mkj_wkat rPath we used a --patch-coverage that we used to make sure that all the lines we had added or modified were covered... :)13:18
fungimkj_wk: there may already be a cover env specified in tox.ini, but if there's not it's easy to add13:18
fungijedimike: if you use the apache module for puppet (or our httpd fork of it) then it will install apache automatically13:18
*** gokrokve has quit IRC13:18
*** stevemar has joined #openstack-infra13:18
funginot sure about logrotate, i'll have to hunt13:19
jedimikefungi, so I see puppet-httpd, but it lists its name as openstackinfra-httpd, and other modules seem to use openstackci-httpd, which is correct?13:19
fungijedimike: good question. i don't know if we've actually started using it as a dep anywhere yet. nibalizer could probably tell you13:20
*** mestery has joined #openstack-infra13:20
jedimikethanks :)13:20
fungii think at the moment we're still just using the apache module everywhere13:20
jedimikei'll grep the existing modules, see what i can find out and get working13:21
fungiand yeah, if you grep for logrotate you'll likely see how we're pulling it in elsewhere, i just don't remember off the top of my head13:21
mesteryInfra folks: Can someone help merge this one: https://review.openstack.org/#/c/163746/13:21
*** dkranz has joined #openstack-infra13:21
*** pblaho_ has quit IRC13:21
mesteryWe need that to get the python-neutronclient functional job running to move out our CLI tests from tempest /cc mtreinish13:21
*** _nadya_ has quit IRC13:22
fungimkj_wk: yeah, no cover env specified in tox.ini there yet, i'll try adding a typical one and see what i get13:23
mkj_wkcool -- this is my introduction to both testr and tox so I'm a bit out of depth :)13:24
fungimkj_wk: bear in mind that git-review is one of our least rigorously tested projects and could certainly use an overhaul of its test setup13:24
*** fdegir has quit IRC13:24
fungiso it's probably a poor intro there13:24
*** Longgeek has quit IRC13:24
openstackgerritMORITA Kazutaka proposed openstack-infra/project-config: Add non-voting Sheepdog test job to cinder check pipeline  https://review.openstack.org/15386813:24
*** bknudson has joined #openstack-infra13:26
*** peristeri has joined #openstack-infra13:27
mkj_wkfungi: I'll try to make fewer assumptions then :)13:27
fungiyeah, we're spread pretty thin and that project needs some extra love13:27
mkj_wkfungi: I've been grovelling through /proc looking to see if I can find what process has a file descriptor that is logging the output...13:27
mkj_wkbut no luck so far13:27
mkj_wkdo you happen to know how it happens that the output gets logged from the processes being started under the test suite??13:28
mkj_wk(it's not in git-review itself, it's just a detail I've never needed to know for testing other things)13:28
fungii don't but should be able to find it13:28
mkj_wkI'm feeling like an idiot after over ten years of writing tests for practically all my python code :)13:29
fungioh, that. i think subprocess.popen() gets wrapped in one of the testing tools13:29
fungiclarkb just had a patch to nodepool related to that. looking13:29
*** spredzy|afk is now known as spredzy13:30
mkj_wkthanks!13:31
*** dboik has joined #openstack-infra13:32
*** dboik has quit IRC13:32
* mkj_wk has been trying really hard not to ask you to "take a free puppy" with this new feature ☺13:32
*** dboik has joined #openstack-infra13:32
*** tiswanso has joined #openstack-infra13:34
*** ddieterly has joined #openstack-infra13:36
*** mriedem_away is now known as mriedem13:36
*** annegentle has joined #openstack-infra13:37
fungimkj_wk: ahh, it was jeblair's patch... so anyway we added popen logging to nodepool's tests like https://review.openstack.org/#/c/162673/2/nodepool/tests/__init__.py13:37
*** Longgeek has joined #openstack-infra13:37
fungii'm not immediately spotting a similar construct in teh git-review tests13:38
fungiwe do mock out run_command_exc()13:40
fungibut just in ConfigTestCase class13:41
mkj_wkyeah13:41
fungiand not adding any explicit logging even there13:41
mkj_wkthus why I think that what is happening is in unittest or testtools13:42
mkj_wknot like I haven't used subprocess.Popen a lot...13:42
fungiso if you look at run_command_status() we do have an explicit print call when VERBOSE is set13:42
fungimy money says we set VERBOSE when running under the testsuite13:43
*** gordc has joined #openstack-infra13:43
*** tnovacik has quit IRC13:43
mkj_wkif I don't put in the time.sleep(1) and instead do proc.stdout.read(1), the 2-minute timeout fires, and it shows the output I was trying to wait for13:43
fungiself._run_git_review('-v')13:44
mkj_wkso clearly, the output of the subprocess is going somewhere, and it's not to the stdout=subprocess.PIPE13:44
*** e0ne has joined #openstack-infra13:44
mkj_wkif I could just wait for the prompt to show up before writing the answer, the whole thing would work13:45
mkj_wksince it prints out the output after the test times out, the output clearly lives *somewhere*13:46
*** saper has quit IRC13:46
mkj_wkif I could just figure out where, I could expect/send instead of time.sleep(1)13:46
*** esker has joined #openstack-infra13:46
*** esker has quit IRC13:47
*** sarob has joined #openstack-infra13:48
openstackgerritMarton Kiss proposed openstack-infra/system-config: Upgrade openstackid to 1.0.5  https://review.openstack.org/16382013:49
*** dkliban_afk is now known as dkliban13:50
*** sigmavirus24_awa is now known as sigmavirus2413:51
*** sarob has quit IRC13:53
mkj_wkfungi: oh. testr is opening a pty13:53
mkj_wkI think testr is reporting output on that pty13:53
*** marcusvrn has quit IRC13:53
*** marcusvrn has joined #openstack-infra13:54
mkj_wkis git-review opening /dev/tty??13:54
mkj_wkdoesn't look like it13:54
funginot sure. i believe testr wants to grab stdio so it can attach it to the multiplexed subunit stream of test results, but i'm not much of a testing guru13:55
mkj_wkoh, huh13:55
fungias soon as clarkb's awake he may have some ideas13:55
mrmartiner13:56
openstackgerrityolanda.robla proposed openstack-infra/system-config: Conditionally manage iptables  https://review.openstack.org/16382913:57
mrmartinfungi: if you are here, we need a +2 on this patch, https://review.openstack.org/#/c/163820/ because openstack.org changed to openstackid as auth, and some users cannot login until a bug is fixed that is landed in 1.0.513:57
mkj_wkfungi: at least it clarifies to me that the output is not available in process; it's only being held in the supervisory testr process13:58
mkj_wkfungi: it explains why I couldn't find it when I walked up the stack printing out the locals() at each frame :)13:58
fungimrmartin: done--thanks for the quick heads up13:58
mrmartinfungi thnx13:58
AJaegeranteaya, please review https://review.openstack.org/#/c/159512/ again - both ' and " are valid quotation marks. Reviews by others on it are welcome as well ;)13:59
*** baoli has quit IRC14:00
*** bhunter71 has joined #openstack-infra14:01
*** baoli_ has joined #openstack-infra14:02
*** baoli_ has quit IRC14:02
*** r-daneel has joined #openstack-infra14:03
*** baoli_ has joined #openstack-infra14:04
*** esker has joined #openstack-infra14:04
*** esker has quit IRC14:04
*** nelsnelson has joined #openstack-infra14:06
*** zz_dimtruck is now known as dimtruck14:06
*** nelsnelson has quit IRC14:07
*** yamamoto has quit IRC14:07
*** mattfarina has joined #openstack-infra14:08
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Ensure filepath exists  https://review.openstack.org/16383414:09
mordredfungi: ^^ fix ansible please?14:09
*** pblaho_ has joined #openstack-infra14:09
*** otter768 has joined #openstack-infra14:09
openstackgerritMerged openstack-infra/system-config: Upgrade openstackid to 1.0.5  https://review.openstack.org/16382014:10
fungimrmartin: ^ i'll go ahead and pull that on openstackid.org right away14:10
mrmartinfungi: will you do a puppet apply ?14:10
fungiyes14:11
openstackgerritSomay Jain proposed openstack-infra/jenkins-job-builder: Adding more configurable options in Notifications plugin  https://review.openstack.org/16313714:11
openstackgerritMarc Koderer proposed openstack-dev/specs-cookiecutter: Fix example template  https://review.openstack.org/16383514:11
fungimrmartin: it's upgraded now. see if that's any better14:12
*** amotoki has joined #openstack-infra14:12
fungiNotice: /Stage[main]/Openstackid/Exec[update-site]/returns: executed successfully14:12
mrmartinnice14:14
mrmartinthnx.14:14
fungionce we have an openstackweb-dev hooked up to openstackid-dev there hopefully won't be scrambles like this14:14
*** gokrokve has joined #openstack-infra14:14
fungibut that's awesome that www.o.o is now authenticating against openstackid.org. i had no idea they were that close to switching over14:14
*** krotscheck has quit IRC14:14
*** yfried__ has quit IRC14:14
*** otter768 has quit IRC14:15
mordred++14:15
*** BrianB_ has joined #openstack-infra14:16
fungithe upshot being that was the blocker to being able to start removing any of the password management bits from the foundation website software, so that eventually it and openstackid.org no longer need to share a backend database14:17
*** krotscheck has joined #openstack-infra14:17
fungieventually the www site should just be asking openstackid for those bits14:18
openstackgerritMerged openstack-infra/ansible-puppet: Ensure filepath exists  https://review.openstack.org/16383414:19
*** gokrokve has quit IRC14:19
fungi(e.g. whatever bits openstackid is handling should be its and its alone)14:19
*** yamamoto has joined #openstack-infra14:19
mrmartinawesome14:20
*** yfried__ has joined #openstack-infra14:20
*** zhiwei has joined #openstack-infra14:20
*** ominakov_ has joined #openstack-infra14:21
*** yfried__ has quit IRC14:21
BrianB_I could use help with an error I am getting when doing git review, fatal: ICLA contributor agreement requires current contact information.14:22
*** mrmartin has quit IRC14:23
BrianB_Looking at bug/1317957, I verified my email, agreement signed,14:23
fungiBrianB_: https://ask.openstack.org/question/5672014:23
*** garyh has joined #openstack-infra14:24
fungiBrianB_: see if your profile is community or speaker or something rather than foundation member14:24
*** funzo has quit IRC14:24
*** vhoward has left #openstack-infra14:24
*** ominakov_ has quit IRC14:25
*** ominakov_ has joined #openstack-infra14:26
*** yfried has joined #openstack-infra14:27
openstackgerritJames E. Blair proposed stackforge/gertty: Don't display project column in project change list  https://review.openstack.org/16384314:27
*** timcline has joined #openstack-infra14:28
openstackgerritMerged openstack-dev/specs-cookiecutter: Fix example template  https://review.openstack.org/16383514:28
fungijeblair: ooh! room for useful data on my tiny screens ;)14:28
BrianB_fungi: my profile shows Current Member Level: Foundation Member14:28
*** wenlock has joined #openstack-infra14:29
fungiBrianB_: and you have tried to submit contact information at https://review.openstack.org/#/settings/contact ?14:29
jeblairfungi: heh, sorry about that.  apparently i broke that a while ago.14:30
openstackgerritMerged openstack-infra/project-config: Enable Neutron in neutronclient functional test job  https://review.openstack.org/16374614:30
fungijeblair: oh, i missed that it had ever worked any other way14:30
jeblairfungi: yeah, i'm not a fan of crowding that screen14:31
*** mattfarina has quit IRC14:31
BrianB_fungi: my contack info matches the info from git config --list (name and email)14:31
fungiBrianB_: that page says "Contact information last updated on [some date and time]"?14:32
*** gokrokve has joined #openstack-infra14:33
fungijeblair: oddly, the updated column is now 9 characters wide but needs 10 to fit a date without wrapping to a second line14:33
*** prad has joined #openstack-infra14:34
fungijeblair: not sure which patch is bringing that in... i can bisect if you aren't sure14:34
jeblairfungi: i don't think that change wolud have affected that (it wraps 2 chars onto the next line if i do a query that causes the project column to appear)14:35
jeblairfungi: my guess is that happened when we switched to iso dates14:35
*** yfried is now known as yfried|afk14:35
jeblairfungi: all the values in that column used to be shorter14:35
*** yfried|afk is now known as yfried14:35
*** yfried has quit IRC14:35
fungiahh, yep, latest release does it too14:35
fungii'll confirm14:35
jeblairfungi: but xkcd issued a comic so now we have no choice14:35
*** marun has joined #openstack-infra14:35
*** achanda has joined #openstack-infra14:36
fungiheh, well i already set my LC_TIME=en_DK.UTF-814:36
*** eharney has joined #openstack-infra14:39
*** _nadya_ has joined #openstack-infra14:40
*** mpaolino has joined #openstack-infra14:40
fungiyep, that change is the culprit. i'll see if i can figure out a quick fix14:41
*** r-daneel has quit IRC14:42
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Change Story to StorySummary in search  https://review.openstack.org/16385214:42
openstackgerritJames E. Blair proposed stackforge/gertty: Switch "Updated" column to fixed width  https://review.openstack.org/16385314:42
jeblairfungi: ^14:42
fungiyou're fast!14:43
*** woodster_ has joined #openstack-infra14:43
*** unicell1 has joined #openstack-infra14:43
openstackgerritAleksey Ripinen proposed openstack-infra/storyboard: Change Story to StorySummary in search  https://review.openstack.org/16385214:43
*** unicell has quit IRC14:43
*** achanda has quit IRC14:44
fungijeblair: that works great14:44
*** Somay has joined #openstack-infra14:44
*** r-daneel has joined #openstack-infra14:45
*** sarob has joined #openstack-infra14:48
SomayHello! Are there any commiters around who could review my patch for the openstack-infra/jenkins-job-builder project? https://review.openstack.org/#/c/163137/14:49
*** carl_baldwin has joined #openstack-infra14:49
*** sabeen has joined #openstack-infra14:49
*** tonytan4ever has joined #openstack-infra14:53
*** thedodd has joined #openstack-infra14:53
*** sarob has quit IRC14:54
*** Longgeek has quit IRC14:55
BrianB_fungi:I added my address to contact and saved it, now it is working. Thanks for your help14:56
*** xgerman has joined #openstack-infra14:57
*** mpaolino has quit IRC14:57
clarkbfungi: mkj_wk yes testr expects the individual test runners it starts to feed subunit back to it over stdout14:58
*** dboik_ has joined #openstack-infra14:58
fungiclarkb: does any spawned subprocess stdout pipe get renamed to something else?14:58
*** yfried has joined #openstack-infra14:59
*** ominakov_ has quit IRC14:59
*** tiswanso has quit IRC14:59
fungiBrianB_: you're welcome. it looks like the bit about entering contact information in the cla subsection of http://docs.openstack.org/infra/manual/developers.html#account-setup could stand to be a little more explicit. i'll try to remember to update that documentation14:59
*** Guest88753 has quit IRC15:00
clarkbmordred: fungi ansible seems to be working for puppet now. Just need to install puppet3 on the zm05-zm08 machines15:00
*** amitgandhinz has joined #openstack-infra15:00
*** baoli_ has quit IRC15:00
*** fdegir has joined #openstack-infra15:00
*** mattfarina has joined #openstack-infra15:00
clarkbfungi: no, you should still be able to use stdout and the subunit parser passes that data through if it isn't subunit15:00
*** dboik has quit IRC15:00
mordredclarkb: woot!15:00
*** e0ne is now known as e0ne_15:01
*** Hal has joined #openstack-infra15:01
*** claudiub has joined #openstack-infra15:01
*** Hal is now known as Guest3039415:01
clarkbI think that is just a simple matter of updating the package pin file and doing an apt-get update apt-get install puppet15:04
clarkbwill give that a go shortly15:04
*** BharatK has quit IRC15:06
*** mpopow has joined #openstack-infra15:06
*** e0ne_ has quit IRC15:06
*** gokrokve_ has joined #openstack-infra15:07
clarkbwe are using 588 ports in hpcloud region b15:07
*** tiswanso has joined #openstack-infra15:07
*** baoli has joined #openstack-infra15:08
*** MarkAtwood has joined #openstack-infra15:08
*** zz_johnthetubagu is now known as johnthetubaguy15:09
jeblairwow, uh, so do we need to run that script continuously?15:09
*** gokrokve has quit IRC15:11
clarkbmaybe? I/we should probably spend a bit more time trying to understand how ports can leak15:11
*** mpaolino has joined #openstack-infra15:11
*** esp has left #openstack-infra15:12
clarkbalso looks like we may be trying to use most of our capacity so its possible that this is normal use. Worth watching15:12
*** erikmwil_ has joined #openstack-infra15:13
*** erikmwilson has quit IRC15:13
*** erikmwil_ is now known as erikmwilson15:13
*** erikmwilson_ has joined #openstack-infra15:13
AJaegerjeblair, your proposal script works now: https://review.openstack.org/#/c/163703/15:13
jeblairyay!15:14
AJaegerjeblair, is that something we want to treat like many projects treat translation imports: A single +2 is enough - or do we want to do the usual 2 cores?15:14
clarkbAJaeger: jeblair I didn't get a chance to review the job, does it handle cases where the project may not be in gerrit yet?15:14
jeblairclarkb: no it does not.  so i think we should check that before approving.15:14
jeblairAJaeger: ^ so i think as long as we double check that the import actually happened, single +2 is probably ok.15:15
AJaegerjeblair, ok.15:16
jeblairAJaeger, clarkb: and maybe we can extend the job to check that automatically15:16
clarkbsounds good15:16
AJaegerand if it fails, we need an email telling us about it ;/15:16
openstackgerritChris Dent proposed openstack-infra/project-config: Add functional test job for python-ceilometerclient  https://review.openstack.org/16386515:16
*** yamahata has joined #openstack-infra15:17
*** prad has quit IRC15:17
openstackgerritKyle Mestery proposed openstack-infra/project-config: Make neutronclient-dsvm-functional gating for neutronclient  https://review.openstack.org/16386815:18
*** MarkAtwood_ has joined #openstack-infra15:19
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Add query for bug 1431408  https://review.openstack.org/16387115:19
openstackbug 1431408 in tempest "Race in tenant isolation unit tests" [High,Triaged] https://launchpad.net/bugs/143140815:19
*** BharatK has joined #openstack-infra15:20
*** dannywilson has joined #openstack-infra15:20
*** e0ne has joined #openstack-infra15:21
clarkbok zm05-zm08 should have puppet3 now and ansible will puppet them15:21
clarkbfungi: ask.o.o needs to be added to ansible's known hosts file15:22
clarkbci-backup-rs-ord is intentionally not key accepted15:22
*** dannywil_ has joined #openstack-infra15:22
clarkbwe should probably run puppet agent --disable on it too if we haven't already15:22
clarkbpleia2: ^ thoughts15:22
*** dannywilson has quit IRC15:23
*** david-ly_ is now known as david-lyle15:23
jeblairclarkb: we're using a hosts file? i thought we were using puppet cert inventory15:23
mordredclarkb: I've got a plan for ansible-side disabling that I think we can probably get landed very soon - let me go clean up a couple of patches15:23
mordredjeblair: we are - but ssh host keys need to be accepted15:23
*** tonytan4ever has quit IRC15:23
jeblairoh that sorry15:23
*** dkranz has quit IRC15:24
jeblairthe known_hosts file, not the file of known hosts ;)15:24
mordredjeblair: words are important15:24
fungiclarkb: i wouldn't add ask.o.o yet since it's not updated in dns?15:24
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/16370315:24
fungiclarkb: the puppeted ask.o.o is not the one in dns15:25
clarkbjeblair: the ssh known hosts file15:25
clarkbfungi: oh good point15:25
fungitesting the data import is on my to do list for today15:26
fungiand then if that checks out, we'll schedule a brief outage to shut down the current service, do one last export, import that into the replacement server and update dns15:26
*** ChuckC has joined #openstack-infra15:26
*** mpopow has quit IRC15:30
*** mpopow has joined #openstack-infra15:30
*** zhiwei has left #openstack-infra15:31
*** [HeOS] has joined #openstack-infra15:31
*** ddieterly has quit IRC15:31
*** HeOS has quit IRC15:31
*** eharney has quit IRC15:32
*** rwsu-afk is now known as rwsu15:33
*** dhritishikhar has joined #openstack-infra15:33
*** eharney has joined #openstack-infra15:33
*** reed has joined #openstack-infra15:34
fungineed to think hard about whether i'm putting in a proposal for all things open in october. i'd get back with possibly one day before heading out to tokyo if so15:34
fungiShrews should totally submit an abstract since he lives there15:34
*** kaisers has quit IRC15:35
clarkbnotmyname: `swift stat containername objectname` should show me the X-Delete-At info if it was set correct?15:35
* clarkb is trying to confirm we are properly setting that value on the objects now15:35
*** prad has joined #openstack-infra15:35
*** asselin_ has quit IRC15:36
openstackgerritAnthony Michon proposed openstack-infra/project-config: Add cerberus-dashboard, python-cerberusclient, stick, stick-dashboard, python-stickclient  https://review.openstack.org/16387615:36
*** zul has quit IRC15:40
openstackgerritMerged openstack-infra/elastic-recheck: Add query for bug 1431408  https://review.openstack.org/16387115:40
openstackbug 1431408 in tempest "Race in tenant isolation unit tests" [High,Triaged] https://launchpad.net/bugs/143140815:40
*** zul has joined #openstack-infra15:41
*** mrmartin has joined #openstack-infra15:41
*** Somay has quit IRC15:44
*** BharatK has quit IRC15:46
*** armax has joined #openstack-infra15:46
*** amitgandhinz has quit IRC15:47
pleia2good morning15:48
pleia2clarkb: wfm, I hadn't run disable on the agent15:48
pleia2(can do that now)15:48
*** nelsnelson has joined #openstack-infra15:48
*** apevec has joined #openstack-infra15:48
AJaegermorning, pleia215:49
apevechi, I need infra help, looks like I got this Icehouse backport into limbo: https://review.openstack.org/14141215:49
pleia2g'day AJaeger15:49
apevecit was in gate when my stable_freeze mistriggered and put -215:49
apevecit was approved and just about to merge15:50
apevecnow re-approval keeps saying it's in merge conflict even after I rebased it15:50
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: Add script for creating stable releases of libraries  https://review.openstack.org/16388015:50
fungiapevec: i'll double-check the rebase15:50
apevecso looks like it is somehow half-merged?15:50
apevecfungi, thanks!15:50
*** dhritishikhar has quit IRC15:50
*** amitgandhinz has joined #openstack-infra15:51
*** ihrachyshka has quit IRC15:52
*** asselin_ has joined #openstack-infra15:53
pleia2clarkb: elasticon was really great, they said they'd be getting videos online soon, sadly the security one was about one of their very few proprietary offerings (a security plugin), but when they went through what it covered it was *mostly* orthoginal to what we do since none of our data is secret15:54
clarkbI have confirmed that the X-Delete-After script change was used to upload http://logs.openstack.org/91/158891/1/check/gate-infra-puppet-apply-trusty/4e71f1b/ but swift stat on that object does not show the X-Delete-After info15:54
clarkbpleia2: nice (though less nice no the security thing)15:55
clarkbs/no/on/15:55
fungiapevec: interesting, it's definitely a fast-forward merge (its current parent is the tip of the target branch)15:55
fungiapevec: i'll go fishing in the zuul logs for clues15:56
*** Longgeek has joined #openstack-infra15:56
*** amitgandhinz has quit IRC15:56
apevecyeah, looks like I confused zuul15:56
clarkbalso apple swift makes googling for 'swift object *' really difficult15:56
clarkbyou get results on object reflection in the language15:56
*** dkranz has joined #openstack-infra15:56
*** sputnik13 has quit IRC15:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix return code of irc_tests.py  https://review.openstack.org/16388415:57
clarkbnotmyname: suggestions on double checking the X-Delete-At info much appreciated /me continues to dig15:57
AJaegerMy irc_tests for catching missing accessbot works fine - but always returns 0. Could I get some quick review, please? ^15:58
fungiapevec: in the meantime, i've rechecked it and zuul seems to have been able to calculate a merge commit for it just fine15:58
*** mrmartin has quit IRC15:58
*** AJaeger has quit IRC15:59
clarkbusing the most privileged account instead of the rw/ro swift specific accounts doesn't help either15:59
clarkbI can still list things just without seeing X-Delete-At info15:59
*** BharatK has joined #openstack-infra15:59
clarkbfungi: apevec that may be similar to the thing I was looking at yesterday16:00
*** asselin_ has quit IRC16:00
clarkbif the merger reconnects then it is treated as a merge failure for an in progress job. jeblair has a change up to get better logging around that, I should go review it now16:00
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: Add script for creating stable releases of libraries  https://review.openstack.org/16388016:00
*** thedodd has quit IRC16:00
jeblairmtreinish: does this look correct for the trove-only tempest job: http://logs.openstack.org/19/125819/6/check/gate-tempest-dsvm-trove/e4d2879/console.html  ?16:00
jeblair6 tests, apparently16:01
*** carl_baldwin has quit IRC16:01
mtreinishjeblair: yeah, there are only 6 tests for trove in tempest16:01
mtreinishand there mostly just sanity checks16:01
jeblairmtreinish: also, how can i find out the failure count of those tests with subunit2sql?16:01
openstackgerritQiming Teng proposed openstack-infra/project-config: new-project: stackforge/senlin  https://review.openstack.org/16329216:02
mtreinishjeblair: I have instructions up on: https://wiki.openstack.org/wiki/QA/Tempest-test-removal16:02
mtreinishin this case it would be something like: select * from tests where test_id like "tempest.api.database%";16:02
*** ashleighfarnham has joined #openstack-infra16:02
*** cody-somerville has joined #openstack-infra16:03
*** cody-somerville has quit IRC16:03
*** cody-somerville has joined #openstack-infra16:03
jeblairyep: http://paste.openstack.org/show/191865/16:03
*** tnovacik has joined #openstack-infra16:03
fungiman, a day of zuul debug logs uncompressed is up to ~1gb16:03
*** ddieterly has joined #openstack-infra16:03
*** andreykurilin has quit IRC16:03
*** prad has quit IRC16:04
mtreinishheh, I didn't expect any failures on those16:04
*** tqtran has joined #openstack-infra16:04
jeblairmtreinish: there is a .00011781401451254451 fail percentage16:04
*** tonytan4ever has joined #openstack-infra16:04
*** bhuvan has joined #openstack-infra16:06
mtreinishjeblair: heh, even so I was expecting 0, the tests that just do a bunch of api lists :)16:07
jeblairalso, they take 3.4 seconds to run16:08
*** cmluciano has left #openstack-infra16:08
*** krisgan has joined #openstack-infra16:09
*** claudiub has quit IRC16:09
*** scheuran_ has quit IRC16:09
*** eharney has quit IRC16:09
jeblairi approved the change that flips that to voting16:09
*** tiswanso has quit IRC16:09
*** unicell1 has quit IRC16:09
clarkbout of curiousity, if they only take 3.4 seconds to run why do we run them in a special job that needs ~25 minutes of setup?16:09
clarkbwhy not just run those 3.4 seconds of test along with the 35 minutes of test for the other things?16:10
zaromorning16:10
*** dboik_ has quit IRC16:10
*** tiswanso has joined #openstack-infra16:10
*** Somay has joined #openstack-infra16:10
*** otter768 has joined #openstack-infra16:10
*** dboik has joined #openstack-infra16:10
jeblairclarkb: that's a good question.16:11
*** mpaolino has quit IRC16:11
jeblairclarkb: i guess it's the principle?16:11
*** ParsectiX has joined #openstack-infra16:12
mtreinishjeblair: oh, it looks like the failures were from before we fixed the check queue filtering, I pulled to check: http://logs.openstack.org/52/135152/1/check/gate-tempest-dsvm-src-python-openstackclient/c37e7ac/16:12
jeblairah16:12
*** e0ne is now known as e0ne_16:13
jeblairclarkb: but yeah, people have been saying that gating on too many projects is a problem, but it's hard to imagine how that would be so for trove.16:13
clarkbjeblair: ya it just seems really odd to me. Even nodepool has more testing :)16:13
jeblairwell, now :)16:14
* clarkb notes that pip is failing to install keyring on static.o.o16:14
*** yfried has quit IRC16:14
clarkbwhich is used by os_loganalyze16:15
clarkbwill have to sort that out after breakfast16:15
*** otter768 has quit IRC16:15
*** dpyzhov_ has joined #openstack-infra16:15
*** dangers_away is now known as dangers16:16
mtreinishjeblair, clarkb: fwiw, when we get the tempest-lib bits in place for the trove client those tests will probably be removed from tempest16:16
ctlaughWhat's the best way to test a 3rd-party CI setup once it's running?  I used the os-ext scripts to deploy, and it's currently pointed at sandbox, but there's nothing out there.  Looking for recommendations of the best way to test that everything is running together correctly.16:16
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/project-config: Add python3-jobs for gear  https://review.openstack.org/15814816:16
*** tsg_ has quit IRC16:17
*** bswartz has quit IRC16:17
clarkbctlaugh push code to sandbox let it exercise your system16:17
*** e0ne_ is now known as e0ne16:17
clarkbalso you can run a silent pipeline against an actual project and check the results16:17
jeblairmtreinish: yeah, and when that's the case, and we move devstack bits into trove, it should be (automatically) removed from the integrated change queue16:18
jeblairmtreinish: (because we won't be running trove jobs on devstack and tempest)16:18
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add initial compute functional tests to Shade  https://review.openstack.org/15846316:18
clarkbI can install keyring using the version of pip on static.o.o locally16:19
*** koolhead17 has joined #openstack-infra16:19
*** koolhead17 has quit IRC16:19
*** koolhead17 has joined #openstack-infra16:19
*** mrmartin has joined #openstack-infra16:19
asselinjeblair, please +1 this change: https://review.openstack.org/#/c/163246/16:20
*** BrianB_ has quit IRC16:20
jeblairasselin: done16:20
jeblairthx16:20
*** claudiub has joined #openstack-infra16:20
asselinthank you16:20
*** Qiming_ has quit IRC16:22
*** eharney has joined #openstack-infra16:22
*** mrmartin has quit IRC16:22
*** prad has joined #openstack-infra16:23
clarkbdhellmann: did aeromancer acls sort themselves out?16:25
*** rkukura has quit IRC16:25
ctlaughclarkb: What makes a pipeline silent?  Just not having a "success:" or "failure:"  I like the idea of being able to run like that for a while, but I definitely don't want to generate noise.16:25
*** johnthetubaguy is now known as zz_johnthetubagu16:26
openstackgerritMerged openstack-infra/project-config: Make trove-only tempest job voting  https://review.openstack.org/16352216:26
*** rfolco has quit IRC16:26
*** Longgeek has quit IRC16:26
clarkbctlaugh: yes, not having reporters for success and fail that talk to gerrit. we have a silent pipeline in our zuul config at https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n11016:26
clarkbctlaugh: it is otherwise the same as the check pipeline just doesn't report anything to gerrit16:26
*** afazekas_ has quit IRC16:27
clarkblooks like the latest release of keyring may have broken us? ok really need food now, back in a bit to continue debugging that16:28
*** baoli has quit IRC16:28
*** jlanoux has quit IRC16:28
dhellmannclarkb: let me check...16:30
*** salv-orlando has joined #openstack-infra16:31
dhellmannclarkb: yes, thanks!16:31
clarkbgreat it was just the cache then16:31
asselinctlaugh, silent is the best way to go. You can also configure e-mail notifications.  http://ci.openstack.org/zuul/zuul.html#smtp http://ci.openstack.org/zuul/reporters.html#smtp-configuration16:32
*** garyk has quit IRC16:33
asselinctlaugh, you can pass in smtp host to zuul here: https://github.com/rasselin/os-ext-testing/blob/master/puppet/install_master.sh#L17016:33
ctlaughclarkb: asselin: excellent - thank you16:37
*** rkukura has joined #openstack-infra16:38
*** baoli has joined #openstack-infra16:38
*** arxcruz has quit IRC16:39
*** tnovacik has quit IRC16:40
*** BharatK has quit IRC16:40
*** tkelsey has joined #openstack-infra16:41
clarkbhrm looks like nodepool isn't getting job demand16:42
pleia2cinerama: while you're about, can we chat quickly about the zanata patch?16:42
*** rfolco has joined #openstack-infra16:43
cineramapleia2: you bet. btw, clarkb, saw your comments - thanks for the feedback@16:43
cineramafor values of '@' that equal '!'16:43
pleia2cinerama: I read the meeting minutes, thanks for providing an update16:43
*** _qb has joined #openstack-infra16:43
pleia2cinerama: I think re: apache proxy we might want to hold off on that until a later patch (like with haproxy for git, we got the base git working first)16:44
cineramapleia2: yep, i actually have a working patch in the wings ready to go16:44
pleia2I kind of want to get most basic thing working initially, and things like that should be optional in this module anyway16:44
pleia2great16:45
*** jistr has quit IRC16:45
clarkbnodepool is calculating demand just not showing very large deficits /me looks furhter up the chain16:45
pleia2cinerama: and we'll have to think about clamav, the fact that it's a stressed part of the install docs gives me reason to pause, but it's not really a think we've done (as noted in the meeting)16:46
*** tonytan4ever has quit IRC16:46
pleia2s/think/thing16:46
clarkbit appears that the zuul mergers may have not done a whole lot of work for a while?16:46
cineramapleia2: in the meeting iirc we decided that for our purposes clamav wasn't really a so much thing because the content to be uploaded is already reviewed code16:47
pleia2cinerama: plus, I think that's a 2nd patch kind of thing too :) like email (fwiw, we use exim, but postfix is more popular for ubuntu things as a default)16:47
pleia2cinerama: ah yes, you're right16:47
*** yamahata has quit IRC16:47
cineramathe main thing i am having trouble with is just separating out what goes in the base module vs what goes in a site file, etc16:47
clarkbhrm looks like lots of nova merges which are slower16:48
clarkbis that merge-check then?16:48
pleia2and regarding all the translations imports, AJaeger is the one who's familiar with all of our current scripts, so he's going to be the one to chat with about that (indeed, I think he volunteered to lead up the work there for Zanata)16:48
cineramaalso i briefly started work on openid integration...so it's supported, but it doesn't get along with openstackid.org yet16:48
pleia2cinerama: yeah, that's always tricky for us :\16:48
*** bdpayne has joined #openstack-infra16:48
*** markus_z has quit IRC16:48
*** yamamoto has quit IRC16:48
*** carl_baldwin has joined #openstack-infra16:48
clarkbcinerama: even after fungi fixed the cert it still complains?16:49
cineramahaving a more formal convention might be good here, both as a guideline for developers who want to help infra, and as a yardstick to review by16:49
pleia2cinerama: it's kind of an evolving discussion for us, historically we've been bad at separating (all of our puppet modules were all in system-config until not too long ago)16:50
cineramaclarkb: yeah i was testing against openstackid-dev.o.o, which gives a different error. the returned xrds is slightly different from, say, login.lp.n, but i don't know enough about it to know whether that would make a difference16:50
clarkbcinerama: gotcha16:50
cineramaclarkb: i tried standing up my own openstackid to play around, but then i got lost in a twisty maze of stuff16:50
pleia2fwiw, auth can be a "in the next patch" thing too16:50
cineramapleia2: yes that is in a separate patch16:50
SpamapShr16:51
SpamapShrm16:51
pleia2woo, on the same page \o/16:51
*** Somay has quit IRC16:51
clarkband I am happy to help with the translation import/export as I have some familiarity with the existing scripts too16:51
SpamapSzuul.openstack.org makes my firefox sad16:51
pleia2SpamapS: how about status.openstack.org/zuul ?16:51
pleia2actually, zuul.openstack.org kind of makes chrome sad too, seems to be loading up a lot of stuff16:52
pleia2(I hadn't navigated to it in forever)16:52
cineramaclarkb: that would be awesome if you could help with that. it's kind of early days for me so i am just starting to think past the immediate goal of getting it up and running16:52
*** Somay has joined #openstack-infra16:52
SpamapSpleia2: yeah that one works fine16:52
clarkbpleia2: SpamapS its working here with chromium16:52
SpamapSactually no16:52
SpamapSthat one also breaks in jquery16:52
pleia2doh16:53
SpamapSmaking me stop/debug/wait16:53
SpamapSah but at least that one eventually works16:53
SpamapSzuul.o.o just spins forever16:53
cineramapleia2: it would be good if you could think about anything else we're missing that this will need - i want to have scope on it as early as possible even if we don't deliver in full all at once16:53
*** ParsectiX has quit IRC16:54
pleia2cinerama: agreed, I'll work to make that happen by the end of the week (probably in a thread to the openstack-infra mailing list)16:54
clarkbcinerama: basically we have two sets of scripts. The first set run when code merges, they update the project's pot file (which contains a list of strings that need to be translated) and pushes that up into transifex. The second set run once a day and use the transifex api/client to pull any translations that have been made and propose them as a new patch in gerrit16:54
*** ParsectiX has joined #openstack-infra16:54
clarkbcinerama: so we basically need to take that second set and run them against transifex but instead of proposing them as a new patch in gerrit push them into zanata16:54
clarkbcinerama: let me get a link to those scripts16:55
pleia2there's also the migration of the less translated (under 75%) transifex translations to the new system16:55
clarkbpleia2: ya thats the above16:55
clarkbwe need to tweak the scripts to grab all translations too16:56
cineramaclarkb: ooh, sound like we need some machinery too16:56
pleia2but I haven't put a lot of thought into the scripts, since I've been trying to focus on the server bit for now16:56
pleia2(this is a big project, dealing with it in chunks helps me :))16:56
clarkbcinerama: pleia2 https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts the set of propose_translation_update scripts16:57
pleia2cinerama: also, please feel free to update the spec if things are unclear16:57
clarkbhttps://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/propose_translation_update.sh#n37 is the line to update (just remove the flag for minimum percentage) to get all the rtanslations16:57
cineramapleia2: it's not unclear, but it could be more detailed possibly16:58
pleia2cinerama: adding more details is good too :)16:58
*** ihrachyshka has joined #openstack-infra16:58
*** mwagner_lap has quit IRC16:59
pleia2bbiab, appointment with painter16:59
*** afazekas has joined #openstack-infra16:59
*** Somay has quit IRC17:03
clarkbpretty sure at this point that a large number of nova merge jobs essentially backed up zuuls ability to do other work since the merger workers were all busy doing nova things17:03
*** jlanoux has joined #openstack-infra17:04
*** Somay has joined #openstack-infra17:04
*** Ryan_Lane has joined #openstack-infra17:04
clarkbwhich is basically as designed but maybe we need more resources, so back to debugging keyring install on static.o.o17:04
*** thingee has joined #openstack-infra17:04
*** bhuvan has quit IRC17:04
*** dimtruck is now known as zz_dimtruck17:05
*** esker has joined #openstack-infra17:05
*** zz_dimtruck is now known as dimtruck17:06
*** harlowja_away is now known as harlowja_17:06
clarkbo_O manually running `sudo puppet agent --test` on static.o.o got it to install properly17:06
*** Ala has quit IRC17:06
clarkbI really have no idea why it failed under ansible puppet17:06
clarkbother than what the pip log said about it failing to run setup.py install17:07
*** thingee has quit IRC17:07
*** yamahata has joined #openstack-infra17:09
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/devstack-gate: Add oslo.versionedobjects to devstack-vm-gate-wrap.sh  https://review.openstack.org/16390817:09
*** esker has quit IRC17:10
*** woodster_ has quit IRC17:10
dhellmanndims_: that's a dupe of https://review.openstack.org/#/c/162218/ which is having trouble landing apparently17:11
fungicinerama: the cert chain on openstackid.org is at least no longer causing a problem though, right?17:11
dims_dhellmann: ah thanks17:12
cineramafungi: looks fine now. i've torn down my environment where i was working on openid stuff at the moment but i can check it later17:12
dhellmanndims_: it looks like my patch was hit by the unittest/testtools version conflict thing17:12
fungicinerama: okay, thanks for confirming17:12
dims_dhellmann: right, forgot about that one17:12
dims_that should pass now17:12
dhellmannyep17:13
*** Guest30394 has quit IRC17:13
cineramafungi: thanks for chasing that up...got to the point where us plebs couldn't quite work out what was happening :)17:15
*** bswartz has joined #openstack-infra17:16
*** patrickeast has joined #openstack-infra17:16
*** tonytan4ever has joined #openstack-infra17:17
notmynameclarkb: did you get any resolution to checking the delete-at header?17:19
*** tonytan4ever has quit IRC17:21
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: Log the job name info when running test mode  https://review.openstack.org/16391217:22
*** melwitt has joined #openstack-infra17:22
*** zz_johnthetubagu is now known as johnthetubaguy17:25
*** devlaps has joined #openstack-infra17:26
*** apevec has quit IRC17:27
clarkbnotmyname: no17:28
clarkbjeblair: fwiw I think we have only leaked 3 ports so far17:28
*** wuhg has quit IRC17:29
*** bhuvan has joined #openstack-infra17:29
*** jp_at_hp has quit IRC17:31
*** achanda has joined #openstack-infra17:32
*** ghostpl_ has quit IRC17:32
*** andreykurilin_ has joined #openstack-infra17:33
*** ivar-lazzaro has joined #openstack-infra17:33
*** mtanino has joined #openstack-infra17:34
SpamapSrcarrillocruz: Whats going on with the shade func tests? I thought you were adding some debug help?17:35
clarkbjeblair: I think zuul maybe merging in repo git tree deps directly rather than just merging the HEAD which brings along the parents17:36
clarkbjeblair: and that is what is causing the oscilations in zuul with the nova changes currently in check17:37
*** e0ne has quit IRC17:37
clarkbjeblair: seeings lots of merges across several mergers at around 1730 for 16335617:37
clarkbfunctionally it is correct but its slow17:37
*** amitgandhinz has joined #openstack-infra17:38
*** Somay has quit IRC17:39
*** cody-somerville has quit IRC17:40
*** woodster_ has joined #openstack-infra17:41
*** andreykurilin_ has quit IRC17:41
*** mpavone has quit IRC17:41
*** yfried has joined #openstack-infra17:42
*** tonytan4ever has joined #openstack-infra17:43
jesusaurusfungi: I just discovered that adding the notes fetch line to ~/.gitconfig causes a bad remote to get created on clone, which breaks 'git review -s'17:43
SpamapSYou know, why don't we just use zmq in our cloud?17:43
SpamapSI know we lurve gearman.17:43
SpamapSBut zmq is already written. Maybe we can just help improve it.17:44
clarkbSpamapS: I missed the discussion wher we said no rabbit17:44
clarkbalso zmq is :(17:44
clarkbI would not use it again for anything17:44
SpamapSclarkb: can you sumarize :( in 140 characters?17:45
SpamapSor less17:45
fungijesusaurus: oh, great point, git-review automatically assumes it doesn't need to create that remote any longer. could probably just name it something other than gerrit17:45
SpamapSclarkb: the discussion on no rabbit was guided by the same things sdague mentioned in the ops summary. Operating it is painful.17:45
clarkbSpamapS: it completely hides all error conditions from you then fails to act on errors properly17:45
SpamapSclarkb: is that by design, or a bad python implementation?17:46
clarkbSpamapS: by design (its the C lib that does it has nothing to do with python)17:46
SpamapSLike, we had the same thing with the python gearman lib right?17:46
sdagueright, so during the ops session on rabbit basically it was 'rabbit is bad, everything else is worse'17:46
SpamapSsdague: Right, and I'm worried that gearman could be a second system, while zmq might be something that could be improved.17:47
jesusaurusfungi: but isnt that telling git which remote to fetch from?17:47
clarkbSpamapS: so if it breaks in a way it can't recover it does not tell you and it is very hard to discover that it has broken. Basically oyu learn a day later when none of your services work despite being happily green17:47
clarkbSpamapS: so then you restart everything because zmq17:47
SpamapSbut it sounds like zmq might just be fundamentally bad, not poorly maintained or misunderstood.17:47
sdagueSpamapS: zmq has no HA story iirc17:47
*** skolekonov has quit IRC17:47
SpamapSsdague: zmq is inherently HA because there is no SPOF.17:47
clarkbsdague: correct you have to implement that on top of it17:47
jesusaurusfungi: it looks like a fresh clone creates a gerrit remote with no uri17:47
clarkbSpamapS: no :/17:47
clarkbSpamapS: thats not how HA works17:47
*** dustins has joined #openstack-infra17:47
fungijesusaurus: ahh, right. probably have to duplicate a little extra data to make that work17:47
clarkbSpamapS: you would have to implement your own retries and so on17:48
fungijesusaurus: like add connection details to the remote itself17:48
clarkbSpamapS: which is hard beacuse zmq doesn't tell you when it breaks17:48
greghaynesSpamapS: It might be better nowdays, but back when we were getting salt to operate using zmq there was a lot of issues where messages would get accepted and then go into a black hole and never be reported as such. Then there was a lot of finger pointing between zmq and the pyzmq folks and both ended up being at fault17:48
fungijesusaurus: at which point i guess you've devolved into needing to know repo-specific details17:48
SpamapSzomg wow17:48
greghaynesthis happened on like 3 separate occasions with 3 separate bugs17:48
SpamapSok now I see17:48
fungijesusaurus: this is actually something we can probably solve in git-review (like have it auto-add the notes fetch, perhaps make that a configurable option)17:49
sdagueit was actually a really fascinating session to be in, and see all the work arounds that folks have had17:49
SpamapSclarkb: I do think HA works if you eliminate all the spofs. It sounds like it's not HA that is lacking, but basic reliability of individual components.17:49
*** bhuvan has quit IRC17:49
SpamapSI'd expect a library purporting to be about simplifying async comms to be crazy pedantic about errors.17:50
jesusaurusfungi: oh good idea17:50
SpamapSSounds like it's the opposite.17:50
greghaynesyep. There was a continual "next major version will be better about this"17:51
fungijesusaurus: though i should also test just overriding origin. i'm pretty sure the notes get replicated so they don't _have_ to be fetched directly from gerrit17:51
clarkbSpamapS: eliminating spofs does not imply HA17:51
SpamapSsdague: I quite enjoyed the cron-to-restart-the-things comment. It's like Windows NT all over again.17:51
sdagueout of curiosity, what would the gear story be here? Because based on the event drops I've seen on zuul restarts that seems equally problematic on the HA side17:51
clarkbsdague: yes you would have to implement your own retries in gear17:51
SpamapSclarkb: If you have no spof, what challenges your availability?17:51
clarkbsdague: but gear will at least tell you when it fails17:51
jesusaurusfungi: oh? do the notes get replicated out to the mirrors? i assumed they didnt17:52
clarkbSpamapS: error handling17:52
clarkbSpamapS: if I don't have a single point of failure but they can't handle errors then what good is that17:52
clarkbSpamapS: my classic example is network load balancer arp storm explosion. There is no SPOF but also no HA17:53
SpamapSclarkb: I have not ever included that in my discussions of HA. If something is unreliable as a standalone component, that is just plain suckage.17:53
*** claudiub has quit IRC17:54
sdagueSpamapS: ummm... well from what I can gather that's a probably with a ton of this software space17:54
*** bhuvan has joined #openstack-infra17:54
sdagues/probably/problem/17:54
SpamapSclarkb: yeah, cascading fail is indeed the one area where 0 SPOF can still mean a failure to be available. Good point. :)17:55
*** _nadya_ has quit IRC17:55
*** dpyzhov_ has quit IRC17:55
*** sputnik13 has joined #openstack-infra17:55
sdagueso... http://kafka.apache.org/ was also brought up in discussion a little bit. No idea if that's a more fruitful target than gear.17:55
SpamapSBut I think if you have a standalone component that handles errors poorly and is unreliable, I put that in a bucket further away from desirability than "it isn't HA". That's in the "It isn't worth considering." bucket.17:55
sdagueI do know that the fanout queues get used enough that having to implement rebroadcase on top of gear might be challenging17:56
greghaynes++ on the kafka, just pretty out of scope for the infra cloud17:56
clarkbsdague: ya I think kafka is likely a good alternative17:56
SpamapSsdague: fanout isn't the same thing as broadcast. Or is it?17:56
clarkbsdague: especially with the reliability requirements that the openstack projects expect17:56
*** dizquierdo has quit IRC17:56
SpamapSTo me fanout means "you get a job. Now you get a different job. Now you get a different job." No?17:56
fungijesusaurus: here's my ~/.gitconfig and it seems to work as intended http://paste.openstack.org/show/191887/17:57
*** johnthetubaguy is now known as zz_johnthetubagu17:57
sdagueSpamapS: honestly, you're getting me a bit out of my depth in all the details of this part17:57
SpamapSsdague: mmk. :)17:57
mordredI like gear over kafka purely because we currently very successfully run gear at scale17:57
SpamapSBecause that is one thing gearman is crazy good at (job distribution)17:57
mordredwhich is not to say that kafka wouldn't work17:57
jesusaurusfungi: looks like you can fetch the notes from the git mirrors, so using origin Just Works™17:57
mordredbut we _know_ gear works well17:57
fungijesusaurus: when i clone and remote update/pull/fetch it still does the usual17:58
sdaguemordred: well, for a particular application17:58
mordredand until I know the same about kafka, it will remain the new shiny hipster toy to me17:58
mordredsdague: sure17:58
mordrednot saying it's the right or wrong choice for openstack17:58
sdaguewhich doesn't require reliable messaing17:58
*** cdent has quit IRC17:58
sdagueand I agree, kafka is completely unknown here as well17:59
*** SumitNaiksatam has joined #openstack-infra17:59
SpamapSsdague: ok I was wrong, and keep getting this wrong. Fanout == broadcast.18:00
*** annegentle has quit IRC18:00
greghaynesthats what it means in rmq land also18:00
*** yfried is now known as yfried|afk18:00
*** bhuvan has quit IRC18:00
*** purp is now known as purp_away18:00
fungijesusaurus: and after a remote update in any existing clones i also now have notes in my git log et cetera18:01
SpamapSfanout seems to be used in 3 places in nova18:01
SpamapS1: scheduler for resource updates18:01
*** katyafervent is now known as katyafervent_awa18:01
SpamapS2: cells for all the things18:01
jeblairclarkb: were you suggesting that the factorial merges from a long nova series was keeping the mergers busy enough that we weren't launching jobs in a timely manner?18:01
SpamapS3: nova-network for ... I refuse to spend any time caring about nova network.18:01
clarkbjeblair: yes18:02
clarkbjeblair: I haven't been able to confirm this by reading code but the zm0X logs seem to imply this is the case18:02
jesusaurusfungi: yep, I'm seeing the same thing after changing "gerrit" to "origin"18:02
clarkbjeblair: note the building spikes in the node graph, that should be relatively smooth18:02
*** derekh has quit IRC18:03
jeblairclarkb: i believe it will be the case, yes, though i would not necessarily expect the merging of the parents to take long, that should be very simple.18:03
*** dims__ has joined #openstack-infra18:03
jeblairclarkb: my guess is that a nova tree reset takes 30-60 seconds at this point, and actually merging the commit should be very fast.  let me investigate.18:03
SpamapSsdague: so anyway, for our cloud, we're not using cells, we're not using nova network, and we're not using host aggregates (the piece of the scheduler that uses fanout). So we might not even get a chance to find out if our fanout sucks.18:03
*** dims__ has quit IRC18:03
*** AJaeger has joined #openstack-infra18:04
clarkbjeblair: ok, anything else I should be looking at to help?18:04
*** katyafervent_awa is now known as katyafervent18:04
*** dims__ has joined #openstack-infra18:04
jeblairclarkb: i don't think so18:04
jeblairit's recovered now, so...18:04
*** koolhead17 has quit IRC18:05
AJaegerclarkb, could you review https://review.openstack.org/#/c/163884/ , please? That improves our gating on project-config18:05
sdagueSpamapS: yeh, especially as neutron beats up the message bus more than nova if I understand it correctly18:05
clarkbAJaeger: ya I will dive into reviews now that jeblair is looking at the zuul thing18:05
*** dpyzhov_ has joined #openstack-infra18:05
AJaegerclarkb: thanks!18:05
*** mwagner_lap has joined #openstack-infra18:06
*** jlanoux has quit IRC18:06
clarkbhuh sys.exit(True) has a return code of 118:07
*** dims_ has quit IRC18:07
clarkbI guess True is coerced to an int value?18:07
jeblairclarkb: the individual changes are not fast.18:07
*** esker has joined #openstack-infra18:07
clarkbjeblair: right but its overdoing them. Shouldn't it only do one per check queue item? instead it appears to do all of them18:08
AJaegerclarkb: yeah...18:08
clarkbnotmyname: still around? sorry I keep getting distracted. Is there a simple way to check the X-Delete-At value for a specific object?18:09
SpamapSsdague: analyzing neutron now, looks like it uses fanout for security groups updates.. which is.. interesting18:09
jeblairclarkb: if you upload a stack of 8 changes, there will be 8 merge jobs, one for each change.  the first will merge A. the second A,B, then A,B,C, etc.18:09
*** dims__ has quit IRC18:09
SpamapSah and for DHCP agent updates18:09
SpamapSwhich makes sense since the DHCP agent is HA18:09
clarkbjeblair: yes 8 merge jobs but each job is running a merge for each change instead of just merging the latest one18:10
SpamapS(or.. is it.... #letsNotDebateHATOdaykthxbai)18:10
sdagueSpamapS: heh18:10
clarkbjeblair: I think we can optimize out all of those unnecessary merges if the git tree already reflects them18:10
*** dpyzhov_ has quit IRC18:10
mordredSpamapS: well, we won't do many security group updates18:10
jeblairclarkb: correct.  give me some more time to analyze this :)18:10
mordredSpamapS: but dhcp agent might get updated more than occasionally18:10
SpamapSmordred: right, but we will do 2x DHCP updates per boot18:10
mordredyup18:10
SpamapSand 2x per delete18:10
SpamapSwell maybe 1x per boot and delete.. not sure18:11
mordredSpamapS: fanout is "send message A to every recipient" right?18:11
clarkbwait why does dhcp need any updates like that?18:11
SpamapSmordred: correct18:11
clarkbnode boots, it asks for IP, some dhcp server will win and respond18:11
*** Longgeek has joined #openstack-infra18:11
*** otter768 has joined #openstack-infra18:11
clarkbnode is deleted, short lease expires a minute later all is happy18:11
SpamapSclarkb: so you can have two capable of responding?18:11
mordredclarkb: the dhcp server has to be told what IP to give to the vm that just booted18:11
*** annegentle has joined #openstack-infra18:12
clarkbmordred: wait is that how its implemented but why18:12
clarkbthe whole point with dhcp in a dynamic environment is to not worry about that18:12
mordredclarkb: beacuse neutron owns the ips - and you can request via api call of neutron what to do with them18:12
SpamapSoh now this is even more interesting18:12
mordredclarkb: no, that's not the whole point of it18:12
clarkbmordred: no neutron owns the networks18:12
SpamapSNeutron dosn't use oslo.messaging's built in notifier thingy, but sends fanouts on the RPC bus18:12
mordredclarkb: the whoel point is a standard mechanism to communicate this information to clients18:12
mordredclarkb: the information may be statically or dynamically configured on the DHCP server side18:13
clarkbmordred: thats really easy to determine, check arp18:13
mordredlike, you may be saying "give me a random ip from the range"18:13
clarkbor ask libvirt18:13
mordredclarkb: or you may be saying "give me the IP that my MAC Address is mapped to"18:13
mordredbecause your machine's mac address may have an allocated IP18:13
clarkbbasically you make this a whole lot more reliable if you let it converge to where it goes rather than deciding upfront where it should go18:13
mordredthat doesn't change over time18:13
SpamapSclarkb: I'm not sure what you're arguing for. Neutron has information about which address each MAC should have. This is a useful thing. We don't want dynamic allocation at the DHCP server level.18:14
mordredSpamapS: ++18:14
clarkbSpamapS: I am saying neutron should not assign IPs like that18:14
clarkbSpamapS: DHCP should assign it18:14
mordredclarkb: it is18:14
mordredclarkb: DHCP is assigning what neutron tells it to18:14
SpamapSand we should poll DHCP for the current state of the world?18:14
clarkbmordred: but only assigning what neutron has told it18:14
mordredit HAS to work that way18:14
notmynameclarkb: ya, I'm getting distracted too. checking now18:14
clarkbno it does not need to work that way18:14
*** koolhead17 has joined #openstack-infra18:14
* clarkb has built a large cloud thing that absolutely did not work this way18:14
mordredclarkb: "neutron create-network A ; nova boot --network=A"18:14
mordredclarkb: did your network have API-driven network config?18:15
clarkbmordred: yes that should send a dhcp request on the l2 broadcast addr18:15
*** tkelsey has quit IRC18:15
*** annegentle has quit IRC18:15
mordredclarkb: yes. which is what it does18:15
clarkbmordred: then dhcp servers fight to assign an ip addr for the ips that live on that l2 netowrk18:15
mordredclarkb: right18:15
SpamapSclarkb: and how did your clients determine what address their box got?18:15
clarkbmordred: this IP can be arbitrary as long as it is on the network that belongs to the l2 net18:15
mordredclarkb: how did the client mac get mapped to the right network?18:15
*** annegentle has joined #openstack-infra18:15
*** baoli has quit IRC18:16
SpamapSsome kind of REST API in front of all the DHCP servers providing a union of all the address assignments?18:16
mordredclarkb: and also SpamapS wuestion18:16
clarkbmordred: because your broadcast comes in on interfaces that can be assigned to networks18:16
clarkbmordred: SpamapS one thing at a time18:16
clarkbso you get l2 broadcast on interface foo for l2 network foo. you then give response ack for IP on l3 network foo18:16
*** otter768 has quit IRC18:16
fungijesusaurus: are you overriding core.notesRef to refs/notes/review like i'm doing, or is there a saner solution to the fact that gerrit's not stashing its notes in the normal tree?18:16
clarkbVM says awesome my IP is now foo[ip]18:16
clarkbthen later when you want to say what is VM's ip address you ask arp18:17
clarkbdone18:17
SpamapSwho asks arp?18:17
clarkbSpamapS: your control plane in this case it would be neutron18:17
notmynameclarkb: ah, I think I see the issue18:17
*** achanda has quit IRC18:17
clarkbSpamapS: mordred but that is completely ditributed it converges to a stable state and does not have any broadcast rpc issues in the control plane18:17
clarkbSpamapS: mordred it turns out that netowrk engineers from the 80s have solved these problems18:17
SpamapSclarkb: yeah, I don't think that is any better than how neutron does it today, which is to just have neutron tell the dhcp server what mac<->ip mapping to use.18:17
*** dprince has quit IRC18:18
mordredSpamapS: no, I agree with clarkb that his is better18:18
clarkbSpamapS: thats a lot less reliable because it requires you to have multiple pieces of software all agree with each other in lockstep18:18
openstackgerritJames E. Blair proposed openstack-infra/zuul: Don't check out a commit if it is already checked out  https://review.openstack.org/16392218:18
jeblairclarkb: ^18:18
clarkbjeblair: thanks looking18:18
mordredSpamapS: but I do not think that clarkb's solution solves all of the use cases neutron is solving18:18
clarkbnotmyname: cool18:18
notmynameclarkb: `swift stat <cont> <obj>` reports the x-delete-at value. internally, an x-delete-after is translated to x-delete-at (a definitive timestamp). so you can get x-delete-at, but you won't see x-delete-after18:18
SpamapSmordred: hah, well I think we can agree we don't have most of the use cases neutron solves. :)18:18
notmynameclarkb: and I think it's a good idea that you're setting it with x-delete-after18:18
mordredSpamapS: totally18:18
clarkbnotmyname: I don't even get x-delete-at though, does that mean I am not setting either?18:19
clarkbnotmyname: its entirely possible we aren't setting this properlywhich I am trying to confirm18:19
notmynameclarkb: x-delete-after is seconds until it's deleted. x-delete-at is a time to delete it at18:19
SpamapSclarkb: ok, so I agree, you have a lower consistency requirement on the frontend, and thus gain scale. I do agree with that.18:19
notmynameclarkb: ah. maybe18:19
jeblairclarkb: i know this isn't exactly what you were thinking, but let me explain my thoughts.  here's the log from a recent big merge: http://paste.openstack.org/show/191893/18:19
notmynameclarkb: it's working on my box :-)18:19
mordredclarkb: I was going to argue that the reason you want to have neutron tell the dhcp server what the mapping is was so that you can handle flloating-ips, which are api-assigned ips to server mappings18:19
clarkbnotmyname: ok :) I think that means we did something wrong. thank you for confirming18:19
mordredclarkb: when I Remembered that floating ips don't dhcp18:19
mordredclarkb: because everyone is crazy18:19
clarkbmordred: :)18:19
SpamapSclarkb: except I very often want to assign an exact IP to a mac, which requires that we do things the way we have now AFAICT.18:19
mordredclarkb: so - I'm on board with you being right18:19
mordredSpamapS: no - but we dont' reallyd o that18:20
mordredSpamapS: see above re: floating-ips18:20
mordredSpamapS: that's teh only mechanism in openstack to assign a specific ip to a specific mac18:20
mordredSpamapS: and we do that wrong too18:20
SpamapSmordred: Yes, yes we do. :-/18:20
mordredSpamapS: because we do that with firewall rules, aiui18:20
*** [HeOS] has quit IRC18:20
*** yfried|afk is now known as yfried18:20
SpamapSThough you can actually cheat and get specific assignment if you can define your own subnets.18:20
clarkbmordred: yes its iptables NAT18:20
SpamapSbecause you can make 1 IP subnets.18:20
mordredwhen that SHOULD be done with dhcp - because of the reasons18:20
jeblairclarkb: if you look at it, it's the "checking out" part that takes a long time.  that's doing a working tree reset.  but it's resetting to the commit that it just created, so it shouldn't actually have to do anything.  yes, it's true that in the case of a git-parent, we could just check out the last thing in the tree, which is what we used to do, but i think it's much simpler if we can just be naive about it and treat git and commit-message18:21
SpamapSBut that's crazy sauce anyway. ;)18:21
mordredclarkb: you know - neutron I hear is crazy pluggable18:21
clarkbjeblair: oh is this related to the change I merged in zuul to do a reset?18:21
mordredclarkb: we could write a proper dhcp/arp based neutron plugin18:21
clarkbjeblair: it made sense for the zuul-cloner but I didn't consider if it would break the merger workers18:21
mordredclarkb: that also does floating-ips but does them via dhcp too18:21
mordredclarkb: and only uses neutron to configure broad-level dhcp network ranges and whatnot18:22
SpamapSThat part probably isn't as pluggable as you'd need.18:22
mordredSpamapS: I believe _everything_ in neutron is pluggable, no?18:22
SpamapSNo18:22
greghaynesclarkb: this is a bit beyond my networking fu, but if youre using something like DVR are you sure to have a consistent arp table?18:22
mordredSpamapS: I betcha I could write a patch to add a plugin point18:22
mordredgreghaynes: you dont need DVR with this18:22
greghaynesyes, we done18:22
greghayneser, we dont18:22
clarkbgreghaynes: if arp is inconsistent then nothing will work18:22
jeblairclarkb: so i think as long as we don't do those checkouts, the actual git merge (which takes 0.2 seconds) will be a trivial amount of extra work.  also, it is possible that later merges can re-use the refs for changes created halfway through the stack.  so it may not be completely wasted (but probably is)18:22
sdaguemaybe the neutron is wrong conversation might be better had in the neutron channel with neutron folks?18:22
clarkbgreghaynes: so I think by definition arp must be consistent18:22
openstackgerritMerged openstack-infra/project-config: New project: cue-dashboard to stackforge  https://review.openstack.org/16280418:22
greghaynesclarkb: I mean what if two nodes only have partial view of the world18:23
*** che-arne has quit IRC18:23
mordredsdague: nah - beacuse we'd jsut be yelling at them18:23
SpamapSmordred: L2 network management is highly pluggable. The rest is pretty rigid, and DHCP looks to be almost hard coded.18:23
clarkbgreghaynes: thats not how l2 networks work18:23
*** achanda has joined #openstack-infra18:23
jeblairclarkb: i don't think it's related to that change, and i don't think we've restarted the mergers since then?18:23
clarkbgreghaynes: even when virtualized every node on an l2 network should see the others via arp whohas18:23
*** tsg_ has joined #openstack-infra18:23
openstackgerritMax Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add support for 'raw' to inject direct xml to job configs.  https://review.openstack.org/16015818:23
jeblairclarkb: also, you are having _three_ conversations right now, right? :)18:23
clarkbjeblair: oh good point on restarting the workers would've happened on the 4th18:23
mordredjeblair: he's good like that18:23
greghaynesclarkb: arp replys are directed to a single mac IIRC?18:23
clarkbjeblair: yes sorry my brain is about to melt18:23
*** achanda has quit IRC18:23
mordredjeblair: we made him rage-converse :)18:23
clarkbgreghaynes: its a broadcast send then response unicast18:23
greghayneshrm, so yes, per l218:24
*** achanda has joined #openstack-infra18:24
sdaguemordred: maybe, but it would also mean folks that actually know neutron internals would be able to comment about which parts are factual and which aren't instead of speculation18:24
mordredjeblair: he's so correct though that I almost want to write the patch to neutron to enable this18:24
mordredsdague: bah. see, now you're talking about being productive18:24
SpamapSthe speculation right now is that fanout would suck w/ gearman18:24
jeblairmordred: you'll probably write it faster than clarkb since he's busy talking18:24
SpamapSit actually might be incredibly awesome. :)18:25
clarkbjeblair: ha18:25
* jeblair goes back to read two other conversations18:25
notmynameclarkb: I'm told that our new feature branch (feature/crypto) isn't having patches reported in the channel. is there a config somewhere that I need to tickle to get that to happen?18:25
mordredclarkb: right now, I want max's threaded IRC thing ...18:25
*** ssam2 has quit IRC18:25
greghaynesmordred: lol, its up18:25
clarkbnotmyname: yes, we need to add that branch to the list of branches to report in the bot config18:25
greghaynesmordred: its just not open sauce18:26
mordredgreghaynes: oh. well, I have no interest then18:26
clarkbnotmyname: willget a link for you18:26
notmynameclarkb: ah, thanks. what's the repo/file? ya, a like would be swell18:26
greghaynesmordred: yea, I gave a vigorous fist shaking at that18:26
SpamapSanyway, to summarize: dynamic allocation done all on dhcpd's side: #win ; static allocation done without NAT: #win18:26
SpamapSgo win18:26
clarkbnotmyname: https://git.openstack.org/cgit/openstack-infra/project-config/tree/gerritbot/channels.yaml#n974 update the branches list there18:26
*** afazekas has quit IRC18:26
notmynameclarkb: thanks18:26
mordredSpamapS: I agree with your tl;dr18:27
SpamapSso, reliable delivery in gearman is now something I hadn't really considered but needs considering.18:28
SpamapSit's actually pretty hard to know, just looking at call sites, if things require reliable delivery or not18:29
mordredSpamapS: I think the usual pattern is for the client to maintain context on the payload and to re-submit if the server goes away18:29
SpamapSwhich means we'd have ot assume they always do18:29
mordredSpamapS: no?18:29
SpamapSmordred: yes18:29
mordredwhich is also how multi-master gearman-server for HA works18:29
SpamapSmordred: and we can do that all deep within oslo.messaging, I think.18:29
jeblairSpamapS: so the interface to o.m is "here, deliver this message", and o.m makes a gearman job, submits it, resubmits it if needed, etc?18:30
SpamapSbut I think there are bits that expect .cast(...) to return immediately, but that the message will eventually be delivered come hell or high water.18:30
openstackgerritJohn Dickinson proposed openstack-infra/project-config: gerritbot report on swift's feature/crypto branch  https://review.openstack.org/16392718:30
SpamapSjeblair: right18:30
notmynameclarkb: ^18:30
SpamapSso we might have to write a store-and-forward worker18:30
*** yfried is now known as yfried|afk18:30
SpamapSwhich is actually really easy to write and I've written before. :)18:30
mordredjeblair, SpamapS: this is almost going to be like me saying AFS is the answer to everything ...18:30
mordredjeblair, SpamapS: but tell me why Cyrus IMAPd isn't the right answer?18:31
*** yfried|afk is now known as yfried18:31
SpamapShah, store the message on AFS. ;)18:31
*** Longgeek has quit IRC18:31
*** ujuc has joined #openstack-infra18:31
clarkbjeblair: your change lgtm and +1 to being naive and simple if possible18:31
mordredsince you want "hey, deliver this message kthxbai"18:31
SpamapSmordred: well that would be exim + Cyrus IMAPd right? ;)18:31
mordredSpamapS: nope18:31
mordredSpamapS: have oslo.messaging do direct message injection18:31
SpamapSoh just pop it into INBOX18:31
mordredno need for actual smtp mail routing18:31
mordredyah18:32
mordredOR18:32
mordredto a "shared queue"18:32
mordredfor things that need to be seen by multiple people18:32
mordredwhich would be a shared mailbox18:32
jeblairfunny thing -- cyrus does not actually work on afs due to memmap and range locking18:32
SpamapSmordred: on shared mailboxes, does each account get their own record of read/unread ?18:32
* mordred may have distracted with AFS mention18:32
mordredSpamapS: I dunno - jeblair ?18:32
SpamapSif not, then you really need nntp for fanout.. and yeah, I'm trolling myself at this point.18:33
*** vhoward has joined #openstack-infra18:33
jeblairSpamapS: i believe so, lemme double check18:33
mordredSpamapS: beacuse with a Cyrus Murder - you get HA - and you get scalability18:33
*** koolhead17 has quit IRC18:34
nibalizerjedimike: yep its not really used just yet i think18:34
SpamapSmordred: don't you get a lot of IO per message too?18:34
nibalizerand we will publish all the modules under openstackinfra so thats why the change18:34
nibalizergotta roll some updates through that18:34
mordredSpamapS: maybe - but it's scalable to VERY VERY large18:34
notmynameclarkb: jeblair: will that be available as soon as it lands, or does it take a bot/gerrit restart?18:34
mordredSpamapS: and not insane to operate18:34
clarkbnotmyname: it takes a bot restart which is automated by puppet as soon as it runs on that node18:34
*** esker has quit IRC18:34
SpamapSmordred: well a lot of IO per message may be unavoidable if reliable delivery is really truly necessary.18:35
notmynameclarkb: cool. thanks18:35
mordredSpamapS: yup18:35
*** dims has joined #openstack-infra18:35
mordredSpamapS: in fact, that's one of the rabbit problems I hear18:35
mordredis that you get message loss in an HA event18:35
mordredso - you know - it's lying about guaranteed delivery18:35
SpamapSbut I'm kind of wondering if it is. I'd bet 90% of the time it isn't, and we would do well to annotate that in call sites for all of the messaging backends to consume.18:35
mordredSpamapS: ++18:35
sdaguemordred: so it sounded more like the major issue wasn't that18:35
mordredsdague: sure - but that _Was_ an issue that got brought up18:35
sdaguebut that in an HA event something doesn't reconnect correctly18:35
mordredsdague: that too18:36
mordredyeah18:36
mordredbeing the main one18:36
sdagueso rabbit believes it's sending things18:36
sdaguebut no one is getting them18:36
SpamapSmordred: yes thats why the recommended method of failover for RabbitMQ is active/passive with shovel if you care about losing messages.18:36
*** claudiub has joined #openstack-infra18:36
SpamapSBut like sdague said, people aren't feeling that problem.18:36
* mordred is going to laugh if we wind up writing an IMAP plugin for oslo.messaging18:36
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Add a python-ironicclient src job  https://review.openstack.org/16363218:37
*** yolanda has quit IRC18:37
sdague``Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.''18:37
*** yolanda has joined #openstack-infra18:38
dimsa.k.a the "Dague Rule" :)18:38
jeblairmordred, SpamapS: i think shared "Seen" is actually configurable as either per-user or per-message....18:38
clarkbspeaking of I think I saw a thread on gerrit list that there is someone working on better email integration18:38
sdagueno, that's jwz18:38
clarkbso you can email your patches and reviews18:38
dims:)18:38
*** Sukhdev has joined #openstack-infra18:38
SpamapSjeblair: so that would suffice for tracking fanout recipients then.18:39
mordredjeblair: awesome18:39
* dhellmann wonders why mordred, SpamapS, and sdague are talking about changes to oslo.messaging here instead of in #openstack-oslo18:39
*** yolanda has quit IRC18:39
*** skraynev has quit IRC18:39
mordreddhellmann: we haven't actually gotten past trolling yet18:39
jeblairmordred, SpamapS: and that configuration is actually per-mailbox :)18:39
mordreddhellmann: when we have an actual suggestion that's worth oslo's time, we'll move there18:39
SpamapSdhellmann: yeah we don't want to start our reliationship with this insanity.18:39
mordredjeblair: AWESOME18:39
mordreddhellmann: although we might be getting close ... I may be coming at you with a spec for an imap backend ...18:40
dhellmannSpamapS: fair enough :-)18:40
SpamapSjeblair: so that's tracked in the mail store which is cyrus's special mdir thing?18:40
clarkbnotmyname: ok, I have mostly confirmed at this point we must not be setting X-Delete-After properly on our uploads18:40
dhellmannmordred: talk to sileht18:40
clarkbnotmyname: I did one by hand (I shoudl've just started with this) and it does have X-Delete-At in stat info18:40
notmynameclarkb: ok. what are you doing? is it that patch I added my +1 to?18:40
clarkbnotmyname: ya18:40
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Modifying devstack plugin job template  https://review.openstack.org/16332318:40
jeblairSpamapS: yeah, it'll be in one of the binary databases which hold message metadata18:41
clarkbnotmyname: so its likely either a logic bug or something we did wrong with requests18:41
notmynameclarkb: well that's what you get for trusting my reviews!! ;-)18:41
SpamapSmordred: the main reason this might not fly is that people will make fun of us. ;)18:41
mordredSpamapS: yes18:41
*** mtanino has quit IRC18:41
*** zul has quit IRC18:41
mordredSpamapS: that is, in fact, the only reason it won't fly18:41
jeblairi hope that won't actually stop us :/18:41
SpamapSNo18:41
SpamapSwe're nerds, we're used to being made fun of :)18:41
mordredjeblair, SpamapS: oh, it won't stop us :)18:42
mkj_wkclarkb: thanks for the confirmation (hours ago...) that testr is the process swallowing the output.  If the test case can't wait for the subprocess output any way, I have trouble knowing what to do other than time.sleep(1)...  I'd like your thoughts if you have a moment some time...18:42
clarkbmkj_wk: I don't think I understand the underlying issue, you can write to stdout just fine18:43
*** tkelsey has joined #openstack-infra18:43
SpamapSOk, it's worth an investigation and an experiment or two. Especially since it wouldn't require anything except an oslo.messaging driver18:43
mkj_wkI can't *read* it from the parent process18:43
*** zul has joined #openstack-infra18:43
fungimordred: clarkb: SpamapS: i gather the use case for floating ips is to be able to move (float) them between machines without needing to make interface changes. so while i agree that being able to dedicate static ip addresses to virtual machines would be a great feature independent of fip, i think that what fip does is the sane solution for what i assume was its primary use case18:43
SpamapSalso with this if we want to inspect the queue we can do it with roundcube or thunderbird. ;)18:43
mkj_wkI tried proc.stdout.read(1) with unbuffered pipes, and it hung until the timeout hit18:43
mordredfungi: but you can do that with arp18:44
mkj_wkat which point testr printed out the prompt it had helpfully intercepted18:44
mordredfungi: I used to do it all the time with heartbeat-based HA systems18:44
fungimordred: well, with hsrp/vrrp/carp sure18:44
jeblairSpamapS: also "less", since the actuall messages are still maildir18:44
SpamapSmordred: gratuitous arp on a shared network of VMs _might_ be something we don't want to encourage. :)18:44
dhellmannmordred, SpamapS, jeblair : the thing more likely to stop you is the paucity of reviewers contributing to oslo.messaging these days18:44
mordredSpamapS: fair18:44
clarkbmkj_wk: I see18:44
mordreddhellmann: good point18:44
fungimordred: that however requires coordination between the machines you're moving the address between though, while doing it at a router does not need the machines to even know18:44
mordredfungi: good point18:45
clarkbmkj_wk: I am not sure how to handle that better other than maybe give it a different fd to use?18:45
mordredfungi: I think I want both things, you know?18:45
SpamapSdhellmann: count me in as a reviewer for now. We can't let this infrastructure crumble. :)18:45
fungimordred: i do too18:45
mordredfungi: I want "I know best, put this IP on this machine"18:45
mordredfungi: and "please do a current floating-ip like thing and put the ip elsewhere"18:45
fungimordred: and i think the "add some addresses i can bind to my interface" nonexistent feature is the more useful one, unfortunately18:45
mordredfungi: of course, the floating-ip case is REALLY the simple case of load balacing18:45
mordredfungi: so you'd expect a good LBaaS thing to be able to handle the current floating-ip case18:46
mkj_wkclarkb: well, a hack is to wait until the prompt would have been issued18:46
mkj_wkclarkb: which was what I did18:46
mordredfungi: and then the rest to be able to use dhcp like sane people18:46
*** dprince has joined #openstack-infra18:46
fungimordred: it goes beyond load balancing, but yes it provides a good basis for simple failover solutions18:46
mkj_wkclarkb: the alternative is to completely change how the test suite works not to open a subprocess but instead call cmd._main(), and mock out lots more18:46
*** claudiub has quit IRC18:47
SpamapSevery good failover-floating-ip thing I've implemented has ended up on the load balancers. :)18:47
dhellmannSpamapS: it'll be good to have the help :-)18:47
mkj_wkclarkb: it's how I write my own python test suites for integration testing18:47
mordredSpamapS: yup18:47
mkj_wk(when I'm trying to test the main function, of course)18:47
*** tkelsey has quit IRC18:47
clarkbmkj_wk: ya that similar to have I have started to write nodepool cli command tests18:47
clarkb*how I have18:48
openstackgerritMerged openstack-infra/jenkins-job-builder: Add 'Build Flow Test Aggregator' publisher  https://review.openstack.org/15808318:49
* SpamapS goes off in search of food18:50
mordredSpamapS: get me some18:51
*** jedimike has quit IRC18:51
purp_awaySpamapS: me, too, please? =D18:52
fungiooh, if you're going out for burritos i want fried fish and cabbage with that habanero puree they use18:52
*** ghostpl_ has joined #openstack-infra18:52
openstackgerritMerged openstack-infra/jenkins-job-builder: Add tests for yaml anchor behaviour  https://review.openstack.org/15835018:52
mkj_wkclarkb: I hesitate to take on restructuring the entire command line testing in order to get this feature in, though :/18:53
*** jcoufal has quit IRC18:54
mkj_wkI'm hoping that an explained time.sleep() workaround can be good enough for now... :/18:55
*** esker has joined #openstack-infra18:55
clarkbmkj_wk: I will defer to fungi on that but generally if its documented (commented) I can be convinced that hacks are ok18:55
*** thedodd has joined #openstack-infra18:55
fungimkj_wk: yeah, while it would be awesome for git-review to get a better testsuite, i totally understand that being more work than someone would want to sign up for, so pragmatic workarounds are fine by me as long as they don't make the job flaky ;)18:57
fungimkj_wk: as it stands, the testsuite it has now wasn't exactly what i envisioned, but since i wasn't the one doing the work to write it and some testing is better than none, i'm just glad we have it at all18:58
mkj_wkthanks18:59
fungibefore git-review had testing, it tended to be a poster child for the "if it's not tested it's broken" mantra18:59
mkj_wk:)18:59
fungiwe had at least a few rather embarrassing broken releases18:59
mkj_wkI hate horribly to add new hacks, especially in the first feature I try to provide18:59
mkj_wkI'll make sure the comment is clear so it doesn't entrench...19:00
AJaegeranteaya: thanks for the review. could you review https://review.openstack.org/163884  as well, please?19:01
fungimkj_wk: also i keep meaning to test out the corner cases i'm worried about with 158877, so hopefully i can give you feedback on that later today19:02
wenlockhi, can anyone point me to any examples of where we create a tarball in a jenkins job builder template?  Is there a publisher or builder for that?19:02
fungiwenlock: the surprisingly-named "tarball" macro at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n35219:03
fungiwenlock: the scripts you see it running there are at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts19:04
*** yfried is now known as yfried|afk19:04
openstackgerritMerged openstack-infra/project-config: gerritbot report on swift's feature/crypto branch  https://review.openstack.org/16392719:04
wenlockfungi, i looked at that guy too... but it seemed like it was copying a tar bal... not creating one.19:04
mkj_wkfungi: cool, I'll submit a newer version now with a more detailed comment so by the time you get to it Jenkins should be happy with it. ☺19:04
wenlockfungi, is that what archive does?   how do you tell it the input or format in which to call the tar ?19:05
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose correct branch for rebase when submitting  https://review.openstack.org/15887719:05
*** yfried|afk is now known as yfried19:06
fungiwenlock: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-tarball.sh relies on a properly configured tox.ini and setup.py/setup.cfg to build a python package tarball19:06
wenlockfungi, ok, leme see if i can hunt a project down that uses that....otherwise, if the projects not python, i have to make something up?19:07
fungiwenlock: yeah, our "tarball" builder assumes you're making a python sdist tarball specifically19:08
wenlockfungi, ok cool, thanks!19:08
wenlockbtw, is sdist something made in openstack ?19:08
SlickNikjeblair / fungi: I have some folks from RedHat who want to run trove tests on fedora to make sure that they pass. Is there any OpenStack policy/guidance existing in that area? Do they need to set up a 3rd party CI to do it? Any docs you can point me to? Thanks!19:08
fungiwenlock: the standard git repository template we use for python packaging can be found at http://git.openstack.org/cgit/openstack-dev/cookiecutter/tree/%7b%7bcookiecutter.repo_name%7d%7d/19:09
wenlockfungi, awesome19:09
*** apevec has joined #openstack-infra19:09
*** tgohad has joined #openstack-infra19:09
fungiSlickNik: we have fedora job workers. do you have any more specific requirements from them?19:10
*** carl_baldwin has quit IRC19:10
*** garyk has joined #openstack-infra19:12
clarkbnotmyname: so in my debugging as best as I can tell we are handing the X-Delete-After header to requests.post just fine. Could it be because we are using tempurl to upload these files that that header doesn't stick?19:12
SlickNikfungi: Specifically they want to be able to run the tests that run as part of the current http://logs.openstack.org/27/159627/8/check/gate-trove-functional-dsvm-mysql/081de39/ gate on Fedora.19:13
*** tsg_ has quit IRC19:13
SlickNikgate-trove-functional-dsvm-mysql*19:13
SlickNik(too eager with the copy paste)19:13
clarkbnotmyname: looks like the server side may be configured with incoming_remove_headers19:13
clarkbnotmyname: any idea if we can whitelist things ourselves on a per tempurl or tenant/account/project basis19:14
*** pmackinn has joined #openstack-infra19:15
dhellmannhas anyone else reported seeing import errors with glob2 in zuul_swift_upload.py at the end of jobs?19:15
clarkbdhellmann: not yet, can you link to an example?19:16
dhellmannclarkb: https://jenkins01.openstack.org/job/check-dg-tempest-dsvm-full-reexec/364/console19:16
clarkbalso I am guessing that means the change merged that probably shouldn't have merged19:16
clarkboh wow that means we have a really old node19:17
fungiSlickNik: yeah, i think as long as the devstack-f21 workers we have are viable, that should be easy to add. might want to hit up ianw to find out if that platform's usable for us yet but we do seem to have some jobs running on it... e.g. https://jenkins04.openstack.org/job/check-tempest-dsvm-f21/296/console running right this moment19:17
* clarkb goes to check devstack-trusty in hpcloud-b319:17
clarkbdhellmann: huh our image there is ~4 hours old which is plenty new enough that it should have glob2 installed. THis requires more investigation19:18
dhellmannclarkb: when the current job fails should I reverify, or wait for you to do that investigation?19:18
SlickNikfungi: Awesome, thanks — I also see the jobs in project-config, and I can point them in that direction.19:19
clarkbdhellmann: you can reverify and roll dice that you get an image with glob2 in that venv19:19
dhellmannclarkb: ok, thanks19:19
clarkbdhellmann: it is working on some of the images I think because in general the infra jobs do work ok but msot don't run on devstack-* nodes19:19
openstackgerritMerged openstack-infra/project-config: Fix return code of irc_tests.py  https://review.openstack.org/16388419:21
apevecfungi, https://review.openstack.org/141412 is now in gate and all jobs are SKIPPED ??19:21
apevecwhat does that mean?19:21
jeblairapevec: if you mouse-over the black dot next to it you'll see the reason19:21
cineramahmm. so i'm a bit confused about where is the best place to put the mysql-related bits of my stuff for zanata given that the different puppet modules vary a bit on how they handle it.19:22
apevecso again merge conflict...19:22
jeblairapevec: the ceilo change has a merge conflict19:22
jeblairyeah19:22
apevecjeblair, but it doesn't19:22
*** ParsectiX has quit IRC19:22
*** crc32 has joined #openstack-infra19:22
fungiyeah, it's parent (according to gerrit) is the branch tip19:22
*** ParsectiX has joined #openstack-infra19:22
jeblairi'll look that up19:23
fungii went digging in the debug log to see if i could tell why zuul kicked it out for a merge conflict earlier and had no luck19:23
*** esker has quit IRC19:24
clarkbcinerama: I would have a zanata::mysql like you do, just don't put that in zanata's init.pp19:24
clarkbcinerama: that way you can choose to use it or choose not to use it when you compose your puppet manifests19:25
apevecjeblair, it was in the gate yesterday when I accidentaly set freeze -2, maybe that confused zuul?19:25
krtaylorclarkb, what is the problem you are seeing with X-Delete-After?19:25
clarkbkrtaylor: its not applying with the tempurl, I am pretty sure its because we use tempurl and rax must not allow us to19:25
jeblairapevec: it should have removed it from the gate at that point, but it shouldn't matter now19:25
cineramaclarkb: at least some of the modules seem to include their mysql stuff rather than bringing it in in the manifest19:25
*** pelix has quit IRC19:25
clarkbkrtaylor: I have been able to confirm that the header val is being set correctly before being given to requests.post19:26
krtaylorclarkb, you should be able to do a test and then go check the metadata on the object store console19:26
clarkbcinerama: I think we are trying to get away from that as a default19:26
*** claudiub has joined #openstack-infra19:26
clarkbkrtaylor: yes I have already checked `swift stat` for tempurl uploads and manaul not temprul uploads and its not set for the tempurl upload but is set otherwise19:26
cineramaclarkb:oh, ok. do we have a...thing...about this i should have read already?19:27
openstackgerritJames E. Blair proposed openstack-infra/zuul: Log merger job uuids  https://review.openstack.org/16393619:27
*** sigmavirus24 is now known as sigmavirus24_awa19:27
jeblairfungi: ^ we should merge that and its parent; it will greatly help with debugging this19:27
*** sigmavirus24_awa is now known as sigmavirus2419:28
krtaylorclarkb, in our swift upload we do it directly (not with the middleware), so hard to say, but it sounds like you may be right19:28
*** cody-somerville has joined #openstack-infra19:29
* krtaylor needs to go revisit the zuul swift upload helper19:29
fungijeblair: thanks, i was just looking for that since clarkb mentioned it earlier19:29
*** weshay has quit IRC19:30
krtaylorclarkb, I wish I could give you temp access to our swift store so you could see if it really is rax19:31
fungicinerama: i don't think we've documented that goal specifically, it's more just institutional knowledge that some deployments may want to have their database server somewhere other than on the application server (for example we might want to hook the application up to a "trove" saas db in the same service provider where the app server resides for ease of manageability)19:31
fungicinerama: so it's great flexibility to include19:32
*** bdpayne has quit IRC19:32
clarkbkrtaylor: its fine I can probably poke rax to get a concrete answer19:32
clarkbbut need to figure out these iamges first19:32
cineramafungi: right, i'm just saying that if folks are working off the existing examples then they are not necessarily going to have a clear picture of that19:33
cineramafungi: also it depends on whether we are developing for our own purposes at present, or for a more general use case where others would adopt it19:33
fungicinerama: i totally agree, it would probably be a low-hanging spec for someone to tackle (make database installation optional in existing modules)19:33
clarkbdhellmann: ok I think I understand what is going on and your job should work fine if it runs on a rax node19:34
fungicinerama: we always try to develop for others to use19:34
clarkbfungi: jeblair: et al new dib fun. If dib fails to build an image it appears that it will just upload the most recently succesfully build image during the daily image update19:34
dhellmannclarkb: let's hope that's where it's scheduled, then :-)19:34
clarkbfungi: jeblair so while it looks like we have a current dib image for hpcloud devstack-trusty we do not, its really old19:34
dhellmannoh, that's a fun failure mode19:35
clarkband we are hitting that ruby issue that asselin ran into19:35
clarkbdhellmann: ya :/19:35
clarkbasselin: what happeed with that ruby issue? /me needs to go figure that out now apparently19:35
fungicinerama: we've found that assuming up-front that others will want to run your software means you make early design choices to support that assumption which might be a lot harder to fix later once the code grows more complex19:35
*** ParsectiX has quit IRC19:36
clarkbdhellmann: so ya we are using an older image there that did not install glob2 into that venv yet19:36
*** pmackinn has left #openstack-infra19:36
clarkbdhellmann: once I figure out why the builds are failing I can trigger some rebuilds to get things moving, then work on making nodepool make these failures more obvious19:36
fungiclarkb: oh, ick!19:37
dhellmannclarkb: sounds reasonable19:37
clarkbbut its also lunchtime so I might eat first >_>19:37
*** timcline has quit IRC19:38
clarkbmordred: ^ I think this is related to how we cache packages in dib builds, I am going to try and reproduce in my local test bed to debug but maybe it stands out to you19:38
openstackgerritMerged openstack-infra/zuul: Improve merge client logging  https://review.openstack.org/16311019:38
mordredclarkb: I'm excited about this?19:38
*** pc_m has quit IRC19:38
clarkbmordred: well I think ou understand how we cache stuff with dib better than *19:38
*** e0ne has joined #openstack-infra19:39
*** openstackgerrit has quit IRC19:40
*** openstackgerrit has joined #openstack-infra19:40
greghaynesclarkb: are there logs of the bulid fails somewhere?19:40
greghaynesor is this the kind of thing you just have to hope you can repro19:41
fungigreghaynes: http://nodepool.openstack.org/ for image build logs19:41
greghaynesahoy!19:41
asselinclarkb, I worked around it.....19:42
asselinclarkb, greghaynes fyi: disabled set -e and added --ignore-missing http://paste.openstack.org/show/191911/19:43
greghaynesasselin: :/19:44
greghaynesasselin: what was the pakage it was failing on again? ruby1.8?19:45
clarkbon trusty19:45
clarkbon precise its docker.io19:45
rcarrillocruzSpamapS: i didn't want to pollute the change with the debug statemnts, latest patchset made the func test class to derive from TestCase, that has all fixtures OS_STDOUT_CAPTURE goodness19:46
asselinsorry...I needed to get dib working first.19:46
rcarrillocruzbut my devstack-gate VM is giving me pain, i'll put the debug statements on the change and push for review shortly, let's hope i get something useful from output...19:46
asselinI can take another look since it's reproducible on my dev box19:46
*** teran_ has quit IRC19:47
openstackgerritMerged openstack-infra/project-config: Add networking-bagpipe-l2 project  https://review.openstack.org/16129019:47
greghaynesclarkb: oh, youre hitting that. What I traced it down to is the trema deps file in devstack, but it seems thats been removed19:49
greghaynesclarkb: and looks like now ruby1.8 is no longer a dep in devstack19:49
greghaynesat least not explicitly19:49
cloudnullhey all, i want to convert the os-ansible-deployment project in stackforge to using specs instead of continuing to use Launchpad bp's. Is there a defacto way creating a specs repo for a stackforge project that I can read about without bothering you guys to much?19:49
greghaynesasselin: Yea, I wonder if its just 'fixed' now for you19:50
anteayacloudnull: http://docs.openstack.org/infra/manual/creators.html19:50
*** yfried is now known as yfried|afk19:51
* cloudnull reading19:51
cloudnullthanks anteaya19:51
asselingreghaynes, no..it's still broken...I saw the log message on newer dib builds19:51
clarkbya it failed today for us19:51
greghayneshuh19:51
anteayayou're welcome19:51
asselingreghaynes, just that the script keeps going. I justified it b/c if it's really needed, it will be installed during the job itself19:52
greghayneshttps://review.openstack.org/#/c/162455/ looks like its the commit that removed the dep19:52
asselinusing the not cached version....19:52
greghaynesasselin: and its still ruby1.8?19:52
asselingreghaynes, yes..let me see if I still have logs19:52
*** [HeOS] has joined #openstack-infra19:53
*** gokrokve_ has quit IRC19:53
*** gokrokve has joined #openstack-infra19:54
*** marun has quit IRC19:54
*** marcusvrn has quit IRC19:54
greghaynesclarkb: fungi so with those nodepool logs, is that multiple image bulids concatenated together?19:55
clarkbyes, just grep out the one you want19:55
fungigreghaynes: interleaved, but since dib builds serially they look concatenated19:55
*** gokrokve_ has joined #openstack-infra19:55
greghaynesis splitting that out a desired feature? looking for excuses to poke at things19:56
fungigreghaynes: yes, ianw is working on a possible solution there i believe19:56
*** dprince has quit IRC19:56
greghaynesah19:56
mtreinishjogo: on https://review.openstack.org/#/c/160933/4 do you know where in the logs to look and see if everything is working?19:57
fungigreghaynes: but in parallel to that, we also want to dramatically reduce the number of images we're building anyway19:57
fungiand have efforts underway to make that happen too19:57
greghaynesah, right, ive seen those patches19:57
*** gokrokve has quit IRC19:59
pabelangerDid gitweb ever run directly on gerrit for -infra?  Specifically behind the http proxy?  Trying to figure out the apache syntax to get it working with anonymous gerrit users19:59
fungipabelanger: you mean like https://review.openstack.org/gitweb?p=openstack-infra%2Fzuul.git19:59
pabelangerfungi, yup20:00
*** timcline has joined #openstack-infra20:00
pabelangerfor me, if unauthenticated user of gerrit access that URL, they get 40420:00
pabelangerI'm still doing 2.5 for gerrit20:00
fungipabelanger: i believe it was working for us anonymously even before 2.420:01
*** imcsk8|zZz is now known as imcsk820:01
pabelangerI suspect because you are using external cgit, it is working differently then my setup20:01
pabelangerfungi, okay, cool.  Checking git logs for info to see what apache config you did20:01
*** melwitt_ has joined #openstack-infra20:02
fungipabelanger: i don't see anything particularly special in our https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/templates/gerrit.vhost.erb20:02
*** mikal_ is now known as mikal20:02
*** melwitt has quit IRC20:02
*** melwitt_ is now known as melwitt20:02
*** garyk has quit IRC20:03
pabelangerfungi, same.20:03
pabelangermy is a little different, but same concept20:03
pabelangerlet me try using that syntax and see what happens20:03
*** baoli has joined #openstack-infra20:03
*** taron has quit IRC20:04
*** cody-somerville has quit IRC20:04
*** baoli has quit IRC20:04
*** taron has joined #openstack-infra20:05
dougwigclarkb: sorry i haven't gotten back to you on the unit test madness thing.  it's just a conversation that can wait until after kilo, and i've been buried.  i will close that loop when it gets a little saner.20:05
*** baoli has joined #openstack-infra20:05
anteayadougwig: it never gets saner20:05
anteayain 2 years I have never seen it get saner20:06
*** arxcruz has joined #openstack-infra20:06
*** jgrimm is now known as zz_jgrimm20:07
openstackgerritDoug Wiegley proposed openstack-infra/project-config: neutron-*aas pylint jobs moving into pep8 job  https://review.openstack.org/16091520:08
*** jebertol has joined #openstack-infra20:08
dougwiganteaya: i'd settle for brief lull.  :)20:09
*** AJaeger has quit IRC20:10
*** yfried|afk is now known as yfried20:11
*** gordc has quit IRC20:11
anteayasaw it once during grizzly20:11
*** otter768 has joined #openstack-infra20:12
jebertolHi - I'm getting an 'invalid committer' error when doing a git review.  Could use some help.  Looks like my git config and contacts on review.openstack.org match though.20:12
fungijebertol: can you post the output at http://paste.openstack.org/ and link it here?20:13
fungijebertol: chances are you used a different e-mail address when you ran 'git commit' or else you're trying to submit commits which aren't yours20:14
*** zz_jgrimm is now known as jgrimm20:14
jebertolfungi: here you go - http://paste.openstack.org/show/191930/20:15
*** Longgeek has joined #openstack-infra20:16
fungijebertol: is it possible you had quotes around the e-mail address in your git config and fixed those after you ran git commit?20:17
*** kgiusti has left #openstack-infra20:17
*** otter768 has quit IRC20:17
jebertolfungi: hmmm - yes I did20:18
fungijebertol: if so, a 'git commit --amend' should fix that20:18
*** sambetts has quit IRC20:18
jebertolfungi - ok then do the git review after the amend?20:18
fungijebertol: yep20:19
jebertolfungi: thanks some much - that worked20:20
*** woodster_ has quit IRC20:20
fungijebertol: great--let us know if you run into any other problems20:20
jebertolfungi: so much ...20:20
*** sambetts has joined #openstack-infra20:20
*** yfried is now known as yfried|afk20:21
*** cody-somerville has joined #openstack-infra20:22
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add initial compute functional tests to Shade  https://review.openstack.org/15846320:22
SpamapSrcarrillocruz: hah, you forgot the print statements in 19. ;)20:23
*** Longgeek has quit IRC20:23
*** dkranz has quit IRC20:24
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Log stdout from build_wheels in cache_devstack  https://review.openstack.org/16395820:24
mtreinishjogo: ^^^20:25
openstackgerritKevin Carter proposed openstack-infra/project-config: Updates for os-ansible-deployment adding specs and docs repos  https://review.openstack.org/16396020:26
mtreinishclarkb, fungi: If you get a sec ^^^ is needed to debug what is going on with the build_wheels script on nodepool20:27
mtreinishall it says in the logs is running, but jogo's d-g patch isn't seeing the dir being created20:27
*** dkranz has joined #openstack-infra20:27
fungijeblair: zaro: mordred: i have a custom trove config with character_set_server=utf8, collation_server=utf8_bin and wait_timeout=28800 which i can switch the review-dev instance to use (in place of the one it currently has to just override wait_timeout). does that cover what we were considering trying?20:28
jeblairfungi: i believe so20:28
*** e0ne has quit IRC20:30
jogomtreinish: ok running your patch locally in a docker env20:30
*** maxbit has joined #openstack-infra20:30
openstackgerritJan Hruban proposed openstack-infra/jenkins-job-builder: Add support for Sonar analysis builder  https://review.openstack.org/16396120:31
*** teran has joined #openstack-infra20:31
*** Somay has joined #openstack-infra20:31
openstackgerritMerged openstack-infra/nodepool: Accomodate ready nodes in min-ready checks  https://review.openstack.org/16271420:32
mtreinishjogo: I have a suspicion it's something with setting wheelhouse in the env for the subprocess, but that's just wild speculation20:32
openstackgerritMerged openstack-infra/nodepool: Don't allocate nodes to providers without images  https://review.openstack.org/16225520:33
openstackgerritSteve Baker proposed openstack-infra/project-config: Custom OVERRIDE_ENABLED_SERVICES for heat-dsvm-functional  https://review.openstack.org/16248720:33
*** e0ne has joined #openstack-infra20:34
jogomtreinish: hmm it worked for me20:34
*** maxbit has left #openstack-infra20:34
openstackgerritMerged openstack-infra/nodepool: Test that dib failures don't stop node builds  https://review.openstack.org/16223720:34
* mtreinish scatches his head20:34
openstackgerritMerged openstack-infra/nodepool: Gracefully handle upload failures  https://review.openstack.org/16323920:35
jogomtreinish: but my env isn't the same as the one this is being run in20:36
jogoso it could be subtle20:36
jogohmm this is failing in 15 seconds in nodepool20:37
jogowhich is pretty fast20:37
jogoerr finishing*20:37
fungijeblair: zaro: mordred: looks like it works, but the trove instance needs a restart after the config is chosen (i did the trove instance restart with gerrit stopped for safety): https://review-dev.openstack.org/525820:37
jogoI wonder if pip is installed20:38
*** Somay has quit IRC20:38
*** Sukhdev has quit IRC20:38
anteayaSpamapS: I recall not so long ago you waxing philosophical about a mythical village with a village square20:39
anteayait was a good email20:39
jeblairfungi: woot; so want to just do this next time gerrit is stopped for any reason (which is looking like next weekend for the os move anyway)?20:39
anteayamy vision got trampled by the crowd long ago20:39
fungijeblair: yep20:39
anteayathere was a part of me that hoped you still could maintain yours20:39
mtreinishjogo: I thought get_pip was done during nodepool but that's based on nothing but a blind assumption20:40
fungijeblair: i'll try to make sure we remember to switch the config to that one (it's called "sanity")20:40
jogomtreinish: nodepool/scripts/fix_pip.sh:if20:40
jeblairfungi: i have an idea for what to call the next one20:40
openstackgerritMerged openstack-infra/nodepool: Remove duplicate code  https://review.openstack.org/16323420:40
SpamapSanteaya: a village with a square sounds nice. :)20:40
fungijeblair: unsanity!20:40
anteayaSpamapS: tell if you find one20:41
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-kibana: Update kibana project url  https://review.openstack.org/16272720:41
jogomtreinish: but we also run http://logs.openstack.org/33/160933/4/check/check-tempest-dsvm-full/b1e7aa7/logs/devstacklog.txt.gz#_2015-03-12_18_27_23_73920:41
mtreinishjogo: we might just have to wait until tomorrow's image build after my stdout patch lands20:42
SpamapSanteaya: in this scenario, the village is hustling and bustling which is good, but do we really need the cobblers and taylors to run around in the street telling us about the suits and shoes they made this week? :)20:42
*** mrda-away is now known as mrda20:42
mtreinishjogo: but is it doing anything, the trace doesn't show anything being run, just some games with cd20:43
*** krisgan has quit IRC20:43
*** erikmwilson has quit IRC20:43
mtreinishjogo: I also know pip is there eventually because I pip install somethings for the subunit2sql side. But it could be an ordering thing20:44
jogo2015-03-12 14:38:16,261 INFO nodepool.image.build.rax-dfw.devstack-precise: Installing setuptools, pip...done.20:44
*** tonytan4ever has quit IRC20:44
anteayaSpamapS: ah your end of the village sounds pretty good, and yeah less noise would be good20:44
jogo2015-03-12 14:32:00,513 INFO nodepool.image.build.rax-dfw.devstack-precise: Running: ['bash', 'tools/build_wheels.sh']20:45
jogoso it maybe an ordering thing20:45
mtreinishjogo: yep, an ordering issue20:45
mtreinishwe probably will need to move the build_wheels call later in the process20:45
anteayaI've been trying to pick up the garbage and realize that just makes space for more people who can't find a garbage can20:46
openstackgerritMerged openstack-infra/project-config: Run nodepool-coverage as a check job  https://review.openstack.org/16323720:46
openstackgerritMerged openstack-infra/nodepool: Remove unneeded Label.*_providers  https://review.openstack.org/15275720:47
jogohmm nodepool/scripts/prepare_node.sh:wget https://git.openstack.org/cgit/openstack-infra/system-config/plain/install_puppet.sh20:48
*** ddieterly has quit IRC20:48
clarkbok I am back now to debug image builds20:48
pleia2anteaya: that should be the description of being a sysadmin :)20:48
*** ddieterl_ has joined #openstack-infra20:48
openstackgerritKevin Carter proposed openstack-infra/project-config: Updates for os-ansible-deployment adding specs and docs repos  https://review.openstack.org/16396020:48
*** carl_baldwin has joined #openstack-infra20:48
jogomtreinish: so what setsthe order?20:48
openstackgerritjose-idar proposed openstack-infra/project-config: Adding pep8 gate check to Opencafe  https://review.openstack.org/15964220:49
jeblairclarkb: you said gertty made something hard to detect in 161952; can you expand on that?20:50
anteayapleia2: well you get to clean up the machines20:50
*** thedodd has quit IRC20:50
openstackgerritMerged openstack-infra/nodepool: Do not try to upload building disk images  https://review.openstack.org/16148220:50
clarkbjeblair: ya, the diff view makes it raelly hard to see where the left edge of the "page" is to count whitespace towards the right20:50
anteayacleaning the garbage that _makes_ it into the can is way better20:50
clarkbjeblair: if we could draw a line on the right side of the line numbers that would help a bunch20:50
*** Somay has joined #openstack-infra20:50
anteayarather than trying to train them to hit the can20:50
pleia2anteaya: ah, indeed20:50
clarkbjeblair: because right now at the top you have 1lotsofspace20:50
clarkbjeblair: then at the bottom ou have 2123littlespace20:51
clarkbasselin: so I think your solution may actually be best20:51
clarkbasselin: the ignore-missing flag, want to propose that?20:51
clarkbasselin: the reason I think that is we look at all devstack branches regardless of base distro and release even though we know some images won't be used with some branches20:52
clarkbasselin: rather than encode that in the image builds I think we take the simple route of just ignoring missing packages20:52
mtreinishjogo: IIRC it's somewhere in system-config (or at least it used to be before the split)20:52
jeblairclarkb: ah, we should right-align the numbers first, which would help a lot.  and if you want more, we could give them a background color, then there will be a nice line20:52
mtreinishjogo: or the first script called was in there20:52
clarkbjeblair: that sounds good20:52
*** thedodd has joined #openstack-infra20:53
*** thedodd has quit IRC20:54
*** marun has joined #openstack-infra20:54
*** jebertol has quit IRC20:54
jogoahh entry point is prepare_node_devstack20:54
*** yfried|afk is now known as yfried20:55
*** Somay has quit IRC20:55
mtreinishjogo: yeah20:55
*** _nadya_ has joined #openstack-infra20:55
jogohmm maybe not a ordering thing20:57
jogo2015-03-12 14:16:28,444 INFO nodepool.image.build.rax-dfw.devstack-precise: Successfully installed pip-6.0.8 setuptools-14.020:57
mtreinishwell fungi just +A'd the patch so we can wait until the next image build to see what's going on20:58
fungimtreinish: i figured it was good output to have regardless20:58
*** dboik_ has joined #openstack-infra20:59
clarkbso are we still trying to build the wheels in the images?20:59
clarkbcan we instaed work on the mirroring of wheels?20:59
jogoclarkb: I don't see why not20:59
jogolets do both and see who finishes first21:00
*** _nadya_ has quit IRC21:00
clarkbbecause of all the problems with it...21:00
clarkband I would rather not duplicate work, instead just do the correct work21:00
clarkbjogo: how are you building wheels without installing deps first?21:00
clarkbmtreinish: ^21:01
clarkbif we can solve that problem I would be more on board but I am not sure that is solveable in a nice wa21:01
jogopip wheel21:01
clarkbjogo: no, you need dev headers of system packages to link against21:01
mtreinishclarkb: IIRC the the build_wheel script runs apt21:01
clarkblibxml2-dev for example21:01
mtreinishwhich is probably terrible21:01
*** ghostpl_ has quit IRC21:01
clarkbmtreinish: yes and thats a problem21:01
clarkbmtreinish: we need to not do that21:02
*** dboik has quit IRC21:02
jogoso this still comes back to the point that I don't think we should have one test to run devstack agaainst an empty box21:02
jogoand that would be gating for devstack21:02
mtreinishjogo: it already is21:02
openstackgerritStephanie Miller proposed openstack-infra/puppet-zanata: Initial commit of Zanata-specific sections  https://review.openstack.org/14794721:02
mtreinishit runs stack.sh and then unstack.sh and makes sure both exit 021:03
clarkbfungi: I think we need to consider reverting all of that wheels stuff instead of making more changes to it, we cannot sanely build wheels without installing a bunch of things we explicitly don't want to install21:03
jogoand another with as much as possible pre-installed/cached etc. and that would be for everything else21:03
mtreinishclarkb: fwiw, it doesn't look like it's doing anything now anyway it fails early for whatever reason21:03
jogomtreinish: oh right, what base image does that use?21:03
*** harlowja_ has quit IRC21:03
fungiclarkb: i built a temporary duplicate of git-fe02 at 104.239.141.161 and it looks like it dtrt with regard to login.defs but take a quick look if you don't mind21:03
clarkbfungi: will do21:03
mtreinishclarkb: I'm fine with a revert as long as we start working on mirroring the wheels (not sure where to start on that though)21:04
*** rlandy has quit IRC21:04
clarkbmtreinish: that would be my preference21:04
*** harlowja has joined #openstack-infra21:04
*** bswartz has quit IRC21:04
fungiclarkb: so it seems to me like the timing theory is validated, and our current configuration does what we want tof user/group creation now, we just have a handful of systems from that timeframe which need cleanup21:04
*** rfolco has quit IRC21:04
jogolong as we have something in place the entire time21:04
clarkbfungi: ya it seems to be correct there21:05
fungiawesome. i'll go ahead and delete that instance now21:05
jogoclarkb: so if we have a test that makes sure devstack installs on a generic base image21:05
*** patrickeast has quit IRC21:05
*** patrickeast_ has joined #openstack-infra21:05
jogowhat is wrong with pre-installing all the packages etc?21:05
clarkbjogo: so the whole point is that devstack can always deploy openstack exactly how it is now21:05
clarkbjogo: if we want to change that that is likely a qusetion for dtroyer and sdague21:06
jogoclarkb: I don't follow21:06
jogoclarkb: I am referring to the apt-get packages and things like that21:06
mtreinishclarkb: it's not something that's going to change21:06
clarkbjogo: the way we test today with devstack ensure to a high degree that devstack can always deploy openstack at all times21:06
*** sarob has joined #openstack-infra21:06
clarkbjogo: if you pull HEAD21:06
jogowe still get updates etc. just more is in the image we start from21:06
clarkbas soon as you start doing work for devstack you stop testing that stuff21:07
clarkband open yourself to races between devtsack changes and openstack changes breaking in those portions of devstack21:07
jogosorry I don't follow21:07
jogowe make the stack.sh/unstack.sh test use a blank image21:07
jogoto make sure we test the full standard devstack flow21:07
asselinclarkb, ok21:08
clarkbjogo: but thats only testing at a point in time21:08
clarkbjogo: so you oepn yourself to breaking21:08
clarkbasselin: I am about to test it locally for my own sanity but go ahead and push the change21:08
*** yfried is now known as yfried|afk21:08
clarkbjogo: the way we gate is that you test in lockstep as you go, this has the nice benefit of meaning cloning devstack and running it is almost always going to work on a clean machine21:08
clarkbjogo: that is not true if image builds or devstack-gate start doing devstack's job for it. We know this because we have broken devstack in the pst when devstack did too much21:09
*** aysyd has quit IRC21:09
jogoclarkb: how do we open our self up to breaking?21:10
jogowe still gate devstack on running devtack from an empty state21:10
*** Ryan_Lane has quit IRC21:10
jogocan you give a specific example21:10
clarkbjogo: because devstack runs about 20 something projects21:10
jogoand ... ?21:10
clarkbjogo: so its eg an interaction between nova and devstack21:10
clarkbwe never tested that because we only tested last nights devstack against latest nova21:11
clarkbnow on next image build the entire gate is wedged21:11
jogono that isn't what I am saying21:11
jogoI am saying we preinstall the following:21:11
clarkbalso on a related note we are explicitly trying to stop having specialized images21:11
*** ghostpl_ has joined #openstack-infra21:11
clarkbwe want the simplest image possible to run all the tests21:11
jogopackages, python stuff21:11
clarkbthis lends itself well to using the mirror21:11
jogoand devstack runs an update on them anyway (or can be made to)21:11
clarkbjogo: yes today that is devstacks job21:11
clarkbso lets contrive an example21:12
jogotoday we don't pre-install any packages or any python packages21:12
openstackgerritRamy Asselin proposed openstack-infra/project-config: Ignore missing packages  https://review.openstack.org/16397621:12
clarkbsay today devstack installs package A and nova depends on package A21:12
jogooptimizing for a single image isn't the right thing IMHO21:12
jogowe should optimize for run time21:12
clarkbdevstack and nova work together bceause A is installed by devstack21:12
asselinclarkb, ^^ I still need to double-check set -e is ok with --ignore-missing21:13
greghaynesasselin: if it ends up not being you can just || true, then we still have set -e for the rest of the script21:13
fungihrm... i do want to discuss how we pre-cache things on nodepool images (so that we can do it in a generic way for all projects' jobs rather than just for devstack-based jobs) but i need to pop out to dinner. i'll check scrollback in a little while and see where this goes ;)21:14
greghaynesasselin: didnt want to shed on that but youll end up making a change anyways if it doesnt work ;)21:14
*** woodster_ has joined #openstack-infra21:14
asselingreghaynes, thanks for the heads up21:14
clarkbjogo: now if package A is removed from devstack nova still works because using an old image21:14
*** thedodd has joined #openstack-infra21:14
openstackgerritJames E. Blair proposed stackforge/gertty: Right align line numbers  https://review.openstack.org/16397921:15
clarkbjogo: your test guards against that if devstack is gated by running from scratch21:15
clarkbso everything still good21:15
jeblairclarkb: ^ er, that took about 4 times as long as i thought it would, but i think it's good.  ;)21:15
fungii do agree at least that pre-building virtualenvs for the things we want tested is going to end in sorrow though21:15
fungianyway, back in a little while21:15
clarkbjogo: but, what if say that package is installed by the image build script similar to how we are doing21:15
*** ldnunes has quit IRC21:15
jogoif you remove package A from devstack, the devstack test that doesn't pre-install anything would catch it21:16
clarkbjogo: then your simple base image is inadverdently broken allowing devstack change to merge21:16
clarkbjogo: from this point on anyone using devstack is broken21:16
*** ghostpl_ has quit IRC21:16
jogoclarkb: see my comment above21:16
clarkbjogo: yes I provided an example where that is insufficient21:17
jogohow?21:17
jogoI still don't follow21:17
clarkbthe problem is in the managing of the images21:17
clarkbits hard to prevent leaking21:17
jogoyou made a change to devstack21:17
jogodevstack tests catch the issue21:17
clarkbyes keep reaidng21:17
*** yfried|afk is now known as yfried21:17
openstackgerritKevin Carter proposed openstack-infra/project-config: Updates for os-ansible-deployment adding specs and docs repos  https://review.openstack.org/16396021:18
jogoso your saying having multiple base images is why we can't do this?21:18
clarkbjogo: its one reason ye21:18
jogowhat is another reason?21:18
jogowe should really be optimizing this for shortest build time21:19
clarkbyes wheels on the mirror do that21:19
clarkbis what I am trying to say, we know how to solve this problem better21:19
*** mattfarina has quit IRC21:19
clarkbwe just need to do it21:19
jogoits not just wheels, deb packages21:20
clarkbthe other reason is that you can get clunky gate wedges by testing like this21:20
clarkbjogo: those can be mirrored too21:20
clarkbjogo: but that is less of a problem because we can cache them reliably21:20
jogonetworks break on us all the time21:20
jogoand installing things takes time21:20
clarkbunlike wheels you just shove the ar files in dirs21:20
jogoeven if its a simple install21:20
*** erikmwilson has joined #openstack-infra21:20
*** tiswanso has quit IRC21:21
jogoanyway ...21:21
jeblairwe have to have working network to at least be able to update from reliable mirrers (probably that we run)21:21
jogoI agree mirroring wheels is better then what mtreinish and I are trying to to today21:21
jeblairwe can't depend on having everything cached locally -- partly because we regularly go through phases where we can't build images for a week or more at a time21:22
jogoclarkb: just not even sure where to start for that one21:22
dstufftclarkb: you can totally just wheels in a dir21:22
dstufftjust shove*21:22
jogojeblair: whoa why can't we  build images for wheeks on end?21:22
clarkbdstufft: yes but the deps required to do that are a pita21:22
jeblairwe _should_ try to have everything locally though so that we don't saturate the network and to reduce the chances of impact21:22
jeblairjogo: because the world is a terrible place :(21:22
clarkbdstufft: we don't want to install libxml2-dev to build lxml for example21:22
jeblairjogo: all kinds of things21:22
*** bdpayne has joined #openstack-infra21:23
*** erikmwilson has quit IRC21:23
jeblairjogo: random github failures, external things change, base images change21:23
greghaynesor specifically, you dont want that installed on the resulting image, right?21:23
*** erikmwilson_ is now known as erikmwilson21:23
greghaynesI mention that because I think ive heard other hacky solutions, like using a second chroot for building the wheels21:23
clarkbgreghaynes: ya21:23
*** ayoung has quit IRC21:24
clarkbgreghaynes: and yes we could do hacky things like that but that becomes hard to debug, one chroot is hard enough21:24
greghaynesyep, agreed21:24
jogojeblair: I guess that is where we take all the hits for external network issues, that makes sense21:24
greghaynesand at the end of the day I think its more work than mirroring. Then if you want to cache a wheel in the built image you can just download the wheel from the mirror at build time21:24
clarkbgreghaynes: +100021:25
clarkbso ya how to mirror21:25
jeblairjogo: we try to keep the network traffic local though -- to our own mirrors.  that helps tremendously21:25
jeblairjogo: most of the network issues we see now, when talking to our own mirrors, are due to capacity limitations in neutron21:25
clarkbwe have a tool called pypi-mirror that will build wheels for every package given a list of deps. We can run this on centos/ubuntu/fedora nodes to make wheels when rsync them into our mirrors somehow21:25
*** dkranz has quit IRC21:25
jeblairjogo: apparently, there's only so much traffic you can shove through a virtual router21:25
greghaynes:D You know I love me some pypi-mirror21:26
jogojeblair: :( boo neutron21:26
clarkbit will also build them in a way that is distro release specific so that solves the problem that pypi itself isn't tackling21:26
clarkbit may need a change to handle arch details if we decide that is necessary. Right now I think its only distro-release specific and doesn't account for arch21:26
dstufftum21:27
greghaynesoh? I know it makes an arch specific dir21:27
dstufftit acoutns for arch I think21:27
clarkboh if it does even better21:27
dstufftor well wheels do I mean21:27
dstufftyou can stick 32bit and 64bit linux wheels in the same dir21:27
clarkbdstufft: wheels do, pypi-mirror doesnt21:27
clarkbbut maybe thats a non issue if wheels just work if in the same dir21:28
openstackgerritIan Wienand proposed openstack-infra/system-config: Add Fedora 21 dib image  https://review.openstack.org/16398221:28
*** koolhead17 has joined #openstack-infra21:28
*** koolhead17 has quit IRC21:29
jogointeresting, never seen this faiure before http://logs.openstack.org/58/163958/1/gate/gate-infra-puppet-apply-centos6/7ffba5f/console.html#_2015-03-12_21_16_09_45621:29
rcarrillocruzSpamapS: i think i found the problem: https://jenkins04.openstack.org/job/check-shade-dsvm-functional/1/console21:29
*** tgohad has quit IRC21:29
rcarrillocruzwhen we source openrc from devstack in the job, it contains OS_CACERT21:29
rcarrillocruzand that's loaded by os-client-config21:29
clarkbjogo: thats the git on centos6 is funny after libc update21:29
jogoclarkb: lol wow21:29
rcarrillocruzi'll put a change on the shade job to unset that variable after sourcing openrc21:30
rcarrillocruzwe should be good then21:30
greghaynesclarkb: There were some fixes to make it work on an arch specific setup recently, so it might 'just work'21:30
* jogo wonders if redhat can fix it21:30
*** koolhead_ has joined #openstack-infra21:30
greghayneshttp://git.openstack.org/cgit/openstack-infra/pypi-mirror/commit/pypi_mirror/cmd/run_mirror.py?id=4591e2abbc55c01a86180d1642a824f0188ffa8f21:30
clarkbjogo: we have been unable to root cause but git client deadlocks quite often since the great libc update21:30
*** kevinbenton_ is now known as kevinbenton21:30
*** koolhead_ has quit IRC21:30
clarkbjogo: it started right at the same time so we correlated the two but that may not be the actual cause21:31
rcarrillocruzor in the post-test-hook script rather...21:31
clarkbI really need a cron that keeps my local dib cache hot21:31
*** teran has quit IRC21:32
greghayneswell, if youre always building images with dib is stays pretty warm ;)21:32
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add initial compute functional tests to Shade  https://review.openstack.org/15846321:32
*** timcline has quit IRC21:32
greghaynesclarkb: so what would actually kick off the pypi-mirror runs?21:34
clarkbwoot I wasn't crazywith my idea why the shade test failed21:34
clarkbgreghaynes: zuul/jenkins21:34
greghaynesoh, just add a timer trigger?21:34
clarkbya21:34
greghaynes\O/ my new found zuul knowledge is becoming useful21:34
clarkband possibly trigger on global-requirements updates since we would likely just wheel the entirety of g-r21:34
*** hdd_ has quit IRC21:36
jeblairclarkb: is there a way to get wheels if available and fall back to what's in bandersnatch otherwise?21:37
clarkbyup21:37
clarkbjeblair: you set your main mirror index to be the wheels location (based on your distro release) then have an extra index url that points to the simple/ dir of the bandersnatch21:38
jeblairclarkb: so we'd still get new things instantly, and new versions instantly if someone had a req that required the new version.  we'd get new versions that match open ended reqs daily?21:38
*** ujuc has quit IRC21:38
clarkbexactly21:38
clarkband as greghaynes mentions the image builds could cache the wheels on the images from the mirror too21:39
jeblairoh nice21:39
jeblairthough if a project with an open ended req and a wheel breaks us, we end up not finding out for a day21:39
*** tqtran has quit IRC21:40
clarkbjeblair: no I think it would grab the latest from bandersnatch mirror and fail early21:40
greghayneshrm? youd do the same thing there, just extra-index-url into the on disk cache21:40
openstackgerritKevin Carter proposed openstack-infra/project-config: updates for os-ansible-deployment adding specs and docs repos  https://review.openstack.org/16396021:40
clarkbjeblair: because pip basically lists all the available options then picks the best match21:40
greghaynesyep21:40
jeblairclarkb: oh, i figured it would check wheels first and stop when it found a match there21:40
jeblairokay, then basically there's nothing lost?21:40
cloudnullsorry for all the reviews, that should be the "last one"-(tm).21:40
greghaynesAIUI that is the difference between using index-url and extra-index-url21:41
greghaynesnot so much that wheels win, but whether the first match in an index wins21:41
greghaynesbut I imagine dstufft will tell me im wrong ;)21:41
dstufftindex-url replaces the default, extra-index-urla dds one21:42
greghaynesalso, DIB has a pypi element to configure this kind of thing already (having multiple pip indexes)21:42
dstufftpip priortizes verison, then wheel, then order it finds it21:42
jeblairdstufft: so if we ask for foo >= 2.0 and index-url has egg 3.0 and extra-index-url has wheel 2.0, we get egg 3.0?21:43
*** dboik_ has quit IRC21:43
dstufftwell pip doesn't support egg, but if you pretend egg is sdist then yes21:43
jeblairdstufft: let's prentend i'm bad at picking the right words sometimes, yes :)21:44
*** dboik has joined #openstack-infra21:44
*** tqtran has joined #openstack-infra21:44
jeblairand speeling them too21:44
clarkbunlike gerrit caches I think this sort of caching will do what we want :)21:44
*** Sukhdev has joined #openstack-infra21:44
*** ihrachyshka has quit IRC21:44
clarkbthe important thing in dstufft's explanation of the best match is version is the highest weight21:44
jeblairsounds like a plan to me :)21:45
dstufftyea, we basically shove everytihng in a list in the order we find them, then do sorted(filter(ENSURE_MATCHES, the_list), key=lambda x: (x.version, x.is_wheel))[0]21:46
openstackgerritMerged openstack-infra/project-config: Log stdout from build_wheels in cache_devstack  https://review.openstack.org/16395821:46
dstufftsince pythons sort is stable that means we prefer version, format, then order we found it21:46
clarkbso ya if we can get a job template for running pypi-mirror to just make a wheel mirror I am sure we can resurrect the long lived slaves to run those jobs on21:47
pleia2nibalizer: about? I have a question re: https://review.openstack.org/#/c/163267/21:48
clarkbwoah did all of those nodepool test changes merge when I wasn't looking? thats awesome. time for me to rebase the cli tests change21:48
*** amotoki has quit IRC21:49
jogomtreinish: your patch is in https://review.openstack.org/16395821:49
jogomtreinish: now we wait and see / hope clarkb does his thing first21:50
nibalizerpleia2: lookin21:51
pleia2nibalizer: so, I don't actually understand how it works now :)21:51
nibalizerhaha21:51
*** prad has quit IRC21:52
pleia2nibalizer: I would assume that like $sysadmins we'd want to do something like = [], but apparently I don't know what's going on21:52
nibalizerwe do that on line 5?21:52
pleia2I mean, $root_rsa_key would be the same21:53
pleia2since it's a variable from hiera, I think21:53
pleia2setting it to xxx would break things, no?21:53
nibalizerthere are a couple ways to break21:53
openstackgerritRamy Asselin proposed openstack-infra/project-config: Ignore missing packages  https://review.openstack.org/16397621:53
nibalizerif you just want a the puppetmaster class to apply so you can prototype/test/kick21:53
nibalizerputting AAAA into /root/.ssh/id_rsa is fine21:53
asselingreghaynes, clarkb ^^21:53
nibalizerif you really want infra to come up, you'll need to pipe a real rsa key into that parameter21:54
pleia2nibalizer: yeah, so right now it doesn't magically pull in from hiera?21:54
*** bknudson has quit IRC21:55
pleia2(I swear, I am trying to say magic less WRT puppet)21:55
pleia2in short: right now it wouldn't actually work21:56
*** annegentle has quit IRC21:56
*** mriedem is now known as mriedem_away21:56
clarkbasselin: cool, I had just started a dib run to test that21:56
clarkbasselin: will rerun with latest patch as soon as current one fails21:56
asselinclarkb, ok21:57
nibalizerpleia2: it gets looked up here http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n17521:57
*** mjturek1 has left #openstack-infra21:58
*** che-arne has joined #openstack-infra21:58
pleia2nibalizer: right, so wouldn't hard-setting it in the config like that reset the value to xxx for our infra?21:58
nibalizerbut in the second box http://ci.openstack.org/puppet.html it says to use 'include o_p:puppetmaster'21:58
openstackgerritClark Boylan proposed openstack-infra/nodepool: Add nodepool command tests  https://review.openstack.org/15545821:58
nibalizerwhich doesn't work any more21:58
nibalizerpleia2: no if you explicitly pass in a value that takes precedence over the default21:59
pleia2ah, ok, so it will prefer the hiera value21:59
jeblairnibalizer: so it wasn't mordred's fault... i did a password reset by email, and still can't log in to puppet forge22:00
pleia2nibalizer: I think I understand now, thanks :)22:00
*** asettle has quit IRC22:00
clarkbjeblair: fungi greghaynes anyone else interested in nodepool testing ^ that above change should cover a good portion of our command line tools22:01
clarkbwil need more tests but that lays the groundwork for them22:01
*** melwitt_ has joined #openstack-infra22:01
*** sigmavirus24 is now known as sigmavirus24_awa22:02
*** melwitt_ has quit IRC22:02
*** harlowja has quit IRC22:02
openstackgerritMerged openstack-infra/zuul: Log merger job uuids  https://review.openstack.org/16393622:02
*** melwitt has quit IRC22:03
clarkbjroll: JayF trying to determine if rackspace swift implementation allows us to set X-Delete-At/X-Delete-After on tempurl posted files. Should I file a ticket to figure that out?22:03
*** peristeri has quit IRC22:03
JayFclarkb: I honestly don't know for 100% sure but I suspect yes; I know we expose functionality that seems similar to that in our UI (which is mostly how I interact with swift these days)22:04
jrollclarkb: that's the best way, but hang on lemme check22:04
jeblairnibalizer: hah, it was mordred's fault ;)  the account was not confirmed.22:04
*** sarob has quit IRC22:04
*** harlowja has joined #openstack-infra22:04
*** e0ne has quit IRC22:05
rcarrillocruzweeee22:05
rcarrillocruzhttps://jenkins06.openstack.org/job/check-shade-dsvm-functional/2/console22:05
rcarrillocruzSpamapS: clarkb ^22:05
rcarrillocruzunsetting OS_CACERT did it22:06
jeblairnibalizer, mordred: i have confirmed the puppetforge account and reset the password back to what's in hiera22:06
clarkbrcarrillocruz: nice22:06
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: fix test in include-raw-escape001.yaml  https://review.openstack.org/16399522:06
clarkbrcarrillocruz: I will have to rereview that change now22:06
jeblairnibalizer: what do i need to do next?  create an openstackinfra-httpd module?22:06
rcarrillocruzyeah, i will address your sync/async thing, was really focused on getting it working22:06
rcarrillocruzso feel free to re-review that again22:07
clarkbrcarrillocruz: ok22:07
rcarrillocruzwill also address GheRivero comments as well for the util module22:07
*** amitgandhinz has quit IRC22:07
jeblairnibalizer: or will that happen automatically when we tag / upload?  and my only task was to fix the password?  :)22:07
clarkbhttp://logs.openstack.org/58/155458/3/check/nodepool-coverage/4794ce2/cover/22:07
clarkbjeblair: ^ we can use that to focus what we test and track our progress22:08
jeblairclarkb: neato!22:08
jeblairah, that's the rebase cool22:08
*** Longgeek has joined #openstack-infra22:08
pleia2are we ok with adding bots our to #rdo- channels? https://review.openstack.org/#/c/163331/22:09
clarkbyup so it has coverage for the cli commands22:09
greghaynesoh hey, my test image uploading deal merged22:09
greghaynesso theres coverage on that too now22:09
anteayapleia2: ah not sure we want to bot all of freenode22:10
nibalizerjeblair: you must manually create the module on the web thing yes22:11
nibalizerthen tag /should/ upload22:11
jheskethMorning22:11
nibalizeror when you push a tag it should auto upload22:11
jeblairnibalizer: it wants me to upload a tarball22:11
pleia2anteaya: indeed22:11
anteayapleia2: not exactly sure where the line is, but pretty sure that is on the other side22:11
anteayamorning jhesketh22:11
nibalizerjeblair: orly22:11
pleia2anteaya: that's my feeling too22:11
crinklenibalizer: yeah you can't just create an empty module22:11
nibalizerjeblair: it wont let you 'create' without uploading a tarball22:11
jeblairnibalizer: http://paste.openstack.org/show/191960/22:11
jeblairnibalizer: text from page22:11
nibalizergahhhhhhhh puppetforge y u so terrible22:11
clarkbjhesketh: good morning. The change for X-Delete-After got in and didn't cause uploads to break but it isn't setting that header on objects. I think because rackspace swift may not allow us to set that header wen using tempurls22:11
*** tsg has joined #openstack-infra22:12
openstackgerritSumit Naiksatam proposed openstack-infra/project-config: Create experimental job for group-based-policy  https://review.openstack.org/16151122:12
crinklenibalizer: i don't think they thought having an empty module was a valid use case22:12
jeblaircrinkle, nibalizer: but maybe an initial upload via the automation will work?22:12
clarkbjhesketh: swift has several config options to limit headers for tempurl usage. My next step was to submit a ticket with rackspace to confirm22:12
nibalizerjeblair: worth a shot22:12
crinkleis there documentation on the blacksmith thing about it?22:12
clarkbjhesketh: unless you have other ideas? I also ran the upload script locally commenting out the request.post to check all the parameters to requests.post and they look good22:12
jeblairnibalizer: I39bafac1e7f85b0d14ec5d6626779985093c54f7 has merged; is the repo ready to go?22:12
*** ghostpl_ has joined #openstack-infra22:12
* anteaya goes for a walk22:13
jeblairnibalizer: version number?  semver style?  0.1.0 ?22:13
*** otter768 has joined #openstack-infra22:13
pleia2anteaya: enjoy22:13
anteayathanks22:13
* jeblair has finished typing his passphrase and is standing by :)22:14
nibalizerjeblair: version i metadata.json is 0.0.122:15
clarkbgreghaynes: looking at the coverage node deletion isn't well handled at all and the gearman/zmq/ssh client stuff is pretty untouched. Those  bug me less though because they do work and aren't touched much22:15
nibalizerand forge will listen to whats in metadata.json22:15
jeblairnibalizer: okay, will tag 0.0.1 then22:15
greghaynesclarkb: ah. I was going to try and poking at a few more command tests22:15
jheskethclarkb: hmm, a ticket is probably a good idea. mattoliverau had mentioned in the past that it was enabled on rax.22:15
greghaynesclarkb: since those are low hanging22:15
jeblairnibalizer: er, have we figured out how to put that into the json file during the tag build job?22:15
*** Ryan_Lane has joined #openstack-infra22:15
jeblairi'm very keen on tag-based post versioning22:16
jheskethmattoliverau: what do you know of X-Delete-After?22:16
nibalizeryou can also cd into puppet-httpd and 'puppet module build' and that will make a tarball and that tarball could be the inital upload if plan A doesn't work22:16
ianwjeblair: are your comments on https://review.openstack.org/#/c/153904/ (Add generate-log-config command) a -2?  it's not clear to me if there is a discussion to be had or it's not going in22:16
mattoliverauLots, but do you mean regarding formpost?22:16
*** eharney has quit IRC22:16
nibalizerjeblair: we're managin the json file in git... so unless we want to do the pbr thing i thing we just have to be responsible and make sure they line up22:16
mattoliveraujhesketh: ^22:16
jeblairnibalizer: we want to do the pbr thing22:16
clarkbgreghaynes: go for it. But I definitely think node deletion should be high on our list. Also the method that is run by the cron thing to periodically update images is not well covered22:17
*** nelsnels_ has joined #openstack-infra22:17
ianwclarkb: if you want to test all your new features, https://review.openstack.org/#/c/163982/ (https://review.openstack.org/#/c/163982/) ... works for me but always willing to be surprised by nodepool in production...22:17
clarkbgreghaynes: so low hanging fruit is good as is covering deletion and running the cron function22:17
jheskethmattoliverau: yes and it's deployment in public cloud22:17
greghaynesclarkb: sounds good22:17
jeblairnibalizer: it doesn't have to be the full pbr thing, but if we can omit the version info from metadata.json in git, and then have the build job insert the correct version info, that will be enough22:17
clarkbianw: I will need to restart nodepool daemon first22:17
clarkbianw: I can do that nowish22:17
jeblairnibalizer: we've done something similar with java, i believe.  possibly other systems22:18
crinkleblacksmith has a 'bump' task and a 'tag' task http://kartar.net/2014/10/using-puppet-blacksmith/22:18
crinklemaybe relevant22:18
*** carl_baldwin has quit IRC22:18
*** otter768 has quit IRC22:18
mattoliveraujhesketh: let me check to see if rax supports it.. X-delete-after in formpost has been in the code base for a year now!22:18
*** dimtruck is now known as zz_dimtruck22:19
jheskethmattoliverau: cool, thanks22:19
clarkbmattoliverau: so the behavior I see is that if I `swift upload --header 'X-Delete-After:300' container object` that works fine22:19
*** nelsnelson has quit IRC22:19
nibalizerjeblair: so the metadata.json not being it git is a problem for anyone trying to use the module from git22:19
nibalizerso two bad options i guess22:19
ianwclarkb: sure; new-ish redirects to images have made the f21 build just look like everything else22:19
jeblairnibalizer: does it need to have the version info in there?22:19
ianwwhich is nice22:19
clarkbmattoliverau: but the tempurl posts that we do doesn't seem to and I found taht swift has config options to whitelist/blacklist specific headers when using tempurl22:19
crinklejeblair: yes22:19
nibalizerjeblair: yea22:19
clarkbjhesketh: tempurl is what we are using right?22:19
*** ghostpl_ has quit IRC22:20
jeblairnibalizer, crinkle: does it have to be accurate?22:20
crinkleyes, the forge depends on it22:20
clarkbianw: did your yaml check thing get merged too? /me goes to check on it22:20
jeblaircrinkle: no not the forge -- the forge gets the correct version when we build the release and upload it22:20
*** hashar has quit IRC22:20
jeblairi'm just talking about people using something from git22:21
ianwclarkb: not yet, when it does i'll work on adding a job to use it22:21
crinkleoh, in that case there's not a technical reason for it to be accurate, but it might be confusing if it's not22:21
jheskethclarkb: not quite, we're using the formpost middleware22:21
mattoliverauclarkb, well we are using the form post middleware, but needs the tempurl.. For x-delete-after in form post you pass it in via the form22:21
clarkbmattoliverau: OH we pass it via the form not the header?22:21
jeblaircrinkle: it can't be accurate anyway.  the last 10 commits have all claimed to be version 0.0.1, but only the last one really is.22:21
jheskethAh, so it just needs to be part of the payload22:22
clarkbjhesketh: so thats probably it then we need to put the value in the payload22:22
*** yfried is now known as yfried|afk22:22
mattoliverauBut it'll only work if rax is using the latest version of form post22:22
jheskethclarkb: yep22:22
clarkbjhesketh: also other bug I noticed is that since the command like arg is a string if delete_after when delete_after is '0' will be true22:22
jheskethmattoliverau: do you have a way of determining that?22:22
*** luqas has joined #openstack-infra22:22
clarkbjhesketh: we should have a check there that says and delete_after != '0'22:22
crinklejeblair: i'm confused, do you want to have a new version on every commit?22:22
clarkbmattoliverau: jhesketh what is the purpose behind only accepting these headers in the form?22:23
mattoliverauYup, I'll take a look via info I hope.. Or rax docs or failing that just try22:23
clarkbI think I am missing the utility of that functionality22:23
*** dkliban is now known as dkliban_afk22:23
mattoliverauIts so you can simply use tempurl key + a simple http forms to push data22:24
jeblaircrinkle: so the way we have all of our python (and some other stuff like java, etc) set up is that when you tag the repo, our ci system automatically builds the package and the version embedded in the package is the name of the git tag.22:24
jeblaircrinkle: so all i have to do to release zuul is "git tag -s 3.0.0; git push gerrit 3.0.0".22:24
jheskethclarkb: can't we make argparse give us an int?22:24
jeblaircrinkle: no commits to change the version number that have to land in the right order, etc...22:24
nibalizerwe could generate metadata.json from a template pretty easily22:24
mattoliverauForm post pulls the data out of the form data and passes it along22:25
jeblaircrinkle: so my main concern is that i want that to be the release process for the puppet modules too22:25
clarkbjhesketh: ya we could do that too then coerce to string when we put it in the payload22:25
jeblaircrinkle: strictly speaking, that just means that the metadata.json embedded in the tarball that the tag job creates needs to get the version number from the tag22:25
nibalizercrinkle: do you think people who clone from git are expecting a metadata.json file? i think it is reasonable to expect that but I'm not sure what exactly would break22:25
clarkbjhesketh: pretty sure we need a string/bytes in the payload because http22:25
clarkbjhesketh: either way will work22:25
jeblaircrinkle, nibalizer: beyond that, i guess, whatever we need for git to work :)22:26
nibalizerjeblair: my plan was to have the person doing the tag make sure that the tag matched metadata.json22:26
crinklenibalizer: i don't think anything would break22:26
mordrednibalizer: the problem is that we have this code review system22:26
jeblairnibalizer: we're not smart enough to remember to do that22:26
mordredaslo, what jeblair said22:26
jheskethclarkb: oh the payload needs to be a string..22:26
mordredversion numbers in files in git repos is ew22:26
clarkbjhesketh: ya, but we can coerce after checking thats easy enough22:26
jheskethYep22:26
nibalizerso the way they do it on github is they make a review 'prep for 0.0.2 release'22:26
crinklei'm used to bumping versions being a code review process22:27
nibalizerwhich bumps metadata.json, changelog, whatevs22:27
clarkbjhesketh: should I write the change or would you like to?22:27
nibalizerthen when that lands they release it22:27
crinklebecause someone could be like "no that's a feature bump not a bugfix bump" or something22:27
jheskethclarkb: want me to make these changes since you seem busy22:27
jheskethHeh, I'm happy to :-)22:27
crinkleyeah what nibalizer said22:27
nibalizerfor us, it would be 'when that lands, pull it, tag, push'22:27
clarkbjhesketh: sure, I am only juggling a couple things at the moment :)22:27
clarkbjhesketh: but ya it would help22:27
mattoliveraujhesketh, clarkb: according to RAX cloudfiles docs, the X-Delete-At is in formpost.. good indication they are using the later formpost: http://docs.rackspace.com/files/api/v1/cf-devguide/content/Create_the_Form-d1a777.html22:27
jeblaircrinkle: so continuing the story about python -- we wrote a thing called pbr that we include in all our python projects which knows how to calculate a unique version number for something in a git repo, so you can still run a program and have it report a version, without having to have that version in code itself (it reads tags and appends a discriminator to the most recent version)22:28
jheskethclarkb: I'll take a look shortly (meeting first)22:28
jeblaircrinkle: we don't necessarily have to have something like that for puppet modules -- only if there's some reason that modules installed from git need to have a version number22:28
mordrednibalizer: right - thing is, we tend to think that process is ugly and convoluted22:28
mordrednibalizer: because the correct info is in the tag - so repeating it in a file is a waste of effort and prone to error22:29
clarkb+100022:29
clarkbgit is a version control system22:29
mordredyup22:29
mordredit's great at controlling versions22:29
jeblaironce you release something with 'git push' you'll never want to release something any other way22:29
nibalizerright and things like cvs and rcs had a magic flag that you could write into a file, and it would be expanded to version whenever it was checked out22:29
greghaynesYes, can confirm - the number of times I have tagged a release and forgotten to bump a version is > 022:29
mordred++22:29
nibalizergit doesn't have that22:29
*** patrickeast_ has quit IRC22:29
mordrednibalizer: nope. don't want22:29
nibalizermordred: its hacking around the same problem22:30
mordrednibalizer: that's all a hack for compatability with when tarballs were the main method of things22:30
mordrednibalizer: there is no problem if the tools aren't broken22:30
mordredand trying to exist in 199422:30
mordrednibalizer: I mean, dont' get me wrong ... the "let's put a version number ina  file our git repo" is a pretty common pattern22:31
*** camunoz has joined #openstack-infra22:31
mordrednibalizer: we've just fixed it for our other languages/repos22:31
clarkb161952 is another dib-nodepool test related change would be good to get that one reviewed22:31
*** andreykurilin_ has joined #openstack-infra22:32
mordrednibalizer: I'd be happy to write a patch for $something to have $something grok the pattern22:32
mordredbut I'm not sure what $something would be22:32
nibalizerya im trying to think of what the consumng points are22:32
mordredwoudl it be the puppet module command?22:32
*** Longgeek has quit IRC22:32
mordredbecasue, the only thing metadata.json in a git repo would be used for programmatically is figuring out depends, yeah?22:33
nibalizerprobably22:33
crinkleyes22:33
nibalizerso if you remove metadata.json and run 'puppet module list' it gives ├── concat (???) instead of ├── puppetlabs-concat (v1.1.2)22:33
nibalizerunicodepipebars are theirs not mine22:34
mordredif that's true - then perhaps a patch to puppet module that says "if metadata.json has no version number and it's in a git repo, look for git tag output instead"22:34
mordrednibalizer: aww... I thought you'd typed the pipebars :)22:34
jeblairdid we decide whether the version key is necessary in the metadata.json file in the git repo?  so could we try removing that key and then injecting it into the file during the tarball build?22:34
nibalizerjeblair: ill try that22:34
openstackgerritMerged openstack/requirements: Bump rtslib version to have config persistence  https://review.openstack.org/16129122:34
mordredif it works, I'll work on trying to make a patch that says the thing I said above22:35
jeblairnibalizer: i'll look into modifying the tarball job22:35
nibalizerjeblair: haha thats actually worse22:35
*** tiswanso has joined #openstack-infra22:35
nibalizerwhen metadata.json isn't present 'puppet module' kind rolls with it, when metadata.json doesn't have 'verision' it breaks pretty hard22:35
nibalizerpretend for a second that i can spell22:36
*** mpopow_ has joined #openstack-infra22:36
*** melwitt has joined #openstack-infra22:36
*** mpopow has quit IRC22:36
*** fdegir has quit IRC22:36
nibalizerthe thigns we produce are 1) a git repo and 2) a forge module22:36
crinkleso what if you just modified the job to sed the version in the metadata.json before building?22:36
crinklethen it would be accurate and puppet module would be happy22:36
jeblairso we could (a) remove it and generate the whole thing from a template (yaml file? ;)  or (b) stick "version": "0.0.0" in it and then overwrite it22:36
mordrednibalizer, crinkle: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/module_tool/metadata.rb is the place I'd do it if I wanted to try to make module smarterer22:36
jrollclarkb: "Just to be clear, you want to know if you can add them when using FormPOST"22:37
mordredjeblair: (b) sounds like it would break the least number of things right now22:37
mordred?22:37
nibalizerthe things that consume 1) and 2) are a) r10k b) puppet-librarian c) puppet-module tool d) random shell scripts22:37
jeblairand i think (b) is also crinkle's suggestion?22:37
mordredor - rather - less work, and break the same amount of things22:37
nibalizeri like (b)22:37
crinklethey are not super open to the idea of puppet module talking to git https://tickets.puppetlabs.com/browse/BKR-2122:38
clarkbjroll apparently. I got educated and mattoliverau says we can so I think we should be good22:38
jrollclarkb: oh, ok, cool22:38
*** krisgan has joined #openstack-infra22:38
mordredcrinkle: oh. that makes me sad22:38
clarkbjroll just need to move from header to form data22:38
jrollah22:38
nibalizeryea 0.0.0 as version is enough to say 'ya you did something funky, have fun with that'22:39
*** thedodd has quit IRC22:39
mordredcrinkle: it seems they aren't in favor of even using puppet mpdule22:39
mordredcinerama: and instead say "if you aren't using r10k go away"22:40
mordredgah22:40
mordredcrinkle: ^^22:40
mordred(at least, that's my reading of that issue)22:40
*** tiswanso has quit IRC22:40
nibalizeryea everybody loves r10k22:40
* nibalizer likes our shell script22:40
crinkleyeah i'm not sure they're considering `puppet module build` and `puppet module list`22:41
*** tim_o_ has joined #openstack-infra22:41
crinklewhich aren't in r10k yet afaik22:41
jeblairnibalizer: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/puppet-module-jobs.yaml#n6222:41
fungievery time i see r10k i think of sgi mips systems22:41
jeblairnibalizer: is it possible for a project to override "rake module:push" to do something locally?22:41
mordrednibalizer: I like our shell script too22:41
fungi(e.g. risc 10000)22:41
fungimy age is showing i guess22:42
jeblairfungi: isn't that usually rs10k?22:42
*** yfried|afk is now known as yfried22:42
fungiyep22:42
fungior r1000022:42
fungithe oxygen series22:42
nibalizerjeblair: probably.. but i think you just want to do the metadata.json munging before you call that22:42
* jeblair scores a fungus22:42
openstackgerritMerged openstack-infra/infra-manual: Add details on using git-review over https  https://review.openstack.org/16107822:43
jeblairnibalizer: i was actually asking for a different reason...22:43
cineramaaww i was all excited someone was talking to me :(22:43
pleia2cinerama: thanks for the latest patchset! :)22:43
jeblairnibalizer: we have a rube-goldberg thing in place for most of the build/upload jobs where we build on one node, put it in tarballs.o.o, then run another job on the pypi node which downloads that and uploads it22:43
jeblairnibalizer: because if someone can run arbitrary code on the pypi node, then they can steal the creds22:44
cineramapleia2: more on the way...just duking it out with Krusty's Non Narkotik Kough Syrup22:44
fungiheyhey kids22:44
pleia2cinerama: understandable, gives me time to scope out what exactly we want from this specific patch before we call it "finished" anyway :) (fwiw, I think we're close)22:44
*** tkelsey has joined #openstack-infra22:44
jeblairnibalizer: so i think we probably need to do that for this too, and then replace "rake module:push" with something that doesn't run arbitrary code22:44
nibalizerjeblair: nothing in rake module:push is executing arbitrary code though22:45
nibalizerif you consider blacksmith trusted i guess22:45
nibalizerwhich we may not22:45
fungii didn't get a photo with krusty, but i do have one from last week of me with duffman. i'll have to dig that up22:45
jeblairnibalizer: can you change Rakefile to make it do something else on "rake module:push" ?22:45
jeblairnibalizer: i think we can trust blacksmith22:45
jeblairnibalizer: it's the local repo we can't trust22:45
mordredcinerama: I mean, you can be involved in the r10k conversation if you'd like!22:45
*** dims_ has joined #openstack-infra22:46
jeblaircinerama: rs10k!22:46
*** dims has quit IRC22:46
mordredI want to install AIX on one of our nodes now ...22:46
pleia2haha22:47
fungiultrix ftw22:47
nibalizercrinkle: ^? can you sneak in to a rakefile?22:47
crinklenibalizer: i'm experimenting22:48
*** dboik has quit IRC22:48
dstufftmordred: oh man life decisions22:48
jesusaurusmordred: you could make a bare-aix... for.... puppet compatibility... i guess22:48
dstufftmordred: I have a TODO list to figure out how to run CI on AIX22:48
*** dboik has joined #openstack-infra22:48
*** r-daneel has quit IRC22:48
nibalizerjeblair: we could just overwrite the rakefile with only enough smarts to do blacksmith22:48
dstufftand QNX22:48
nibalizerobviously we are not gonna need to run lint tests or anything else at that point22:48
dstufftand a bunch of other random things22:48
* fungi now wonders if irix is still getting any love since it was so rudely abandoned22:48
cineramaboy i sure haven't done aix in a while22:48
*** tkelsey has quit IRC22:49
mordrednibalizer: does the rakefile need to be in the module upload?22:49
*** amotoki has joined #openstack-infra22:49
* fungi worries he has accidentally brought on friday an roughly hour early22:49
crinkleyeah you can totally overload it22:49
mordrednibalizer: by that, I mean, does rake module:push make a tarball? or can it operate on a tarball that's been made already22:49
crinklegross22:49
*** ayoung has joined #openstack-infra22:50
*** nelsnels_ has quit IRC22:50
fungicomputering is a gross and untidy business22:50
* mordred avoids computering as much as possible and washes his hands afterwards22:51
nibalizermordred: no the rakefile doesn't need to be in the artificat that is uploaded22:51
nibalizeralso i dont think there is really value in splitting module build and module upload22:52
nibalizerthey are both determinisitc tasks22:52
fungidebconf cfp is out now, if anybody wants to see heidelberg in august22:52
*** yfried is now known as yfried|afk22:52
nibalizerthe only question is can they be overrided22:52
*** achanda has quit IRC22:52
jeblairnibalizer: well, that's the only reason we split them :)22:52
jeblairand it's important22:52
mordrednibalizer: so, you're suggesting have build and publish go like "system writes out new Rakefile, does build, does upload"22:53
*** achanda has joined #openstack-infra22:53
crinklenibalizer: yes you can add a task to the rakefile with the name module:push and rake will run both22:53
mordredjeblair: I like the "write out Rakefile and don't use the one from the repo" approach22:53
jeblairbecause "cat /path/to/pypi/credentials; cat /path/to/puppetforge/credentials" would be a bad think to put in the rakefile22:53
clarkbwow google code is now deadish22:54
fungieh?!?22:54
jeblairclarkb: i think it may have been dead for a few years22:54
fungiyou mean even less usable than it used to be?22:54
mordredclarkb: but code hosting run by single companies is a great idea!22:54
jeblairokay, so we either need to start the job by overwriting the rakefile, or doing the split thing we do elsewhere22:54
mordred++22:54
*** amotoki has quit IRC22:55
fungieither of those options is fine by me22:55
fungiboth should be safe22:55
clarkbno new projects as of today22:55
mordredand if we do the split thing, we need to not use rake module:upload in the second job22:55
clarkband offline january of next year22:55
jeblairright, i forgot to add that includes something like a curl-based uploader22:55
fungiclarkb: woah! so add to the codehaus and gitlab die-offs then22:55
crinklejeblair: what is the risk when someone overrides the task? the task is only called when someone pushes a tag, correct? so only trusted people can run it?22:55
*** baoli has quit IRC22:55
jeblaircrinkle: yeah, but we don't trust everyone who can push a tag; we want this to be available to anyone on the system22:56
*** achanda_ has joined #openstack-infra22:56
jeblaircrinkle: so not just infra projects, but openstack and stackforge ones too22:56
crinkleoh I see22:56
*** annegentle has joined #openstack-infra22:56
*** achanda_ has quit IRC22:56
*** achanda_ has joined #openstack-infra22:56
fungier, not gitlab but gitorious (the former bought and is closing down the latter)22:57
*** ddieterl_ has quit IRC22:57
jesusaurusgitorious is shutting down?22:57
jesusaurus:(22:57
*** achanda has quit IRC22:57
pleia2jesusaurus: yes :(22:57
nibalizerso i think echo > Rakefile22:57
nibalizermaybe we need to require it22:58
nibalizerill do some testing22:58
*** melwitt has quit IRC22:59
*** sdake has quit IRC23:00
openstackgerritStephanie Miller proposed openstack-infra/puppet-zanata: Add apache configuration for zanata  https://review.openstack.org/16401123:00
*** dims_ has quit IRC23:00
*** otter768 has joined #openstack-infra23:00
*** dims has joined #openstack-infra23:00
*** erlon has quit IRC23:01
*** dpyzhov_ has joined #openstack-infra23:02
*** annegentle has quit IRC23:02
clarkbhttps://review.openstack.org/#/c/163976/ builds an image locally after it was failing locally23:02
clarkbmordred: jeblair fungi pleia2 ^ any chance we can get a second reviewer to look at that? I can kick off new dib build for precise and trusty after that merges23:03
fungilooking now23:03
zaroclarkb: i get these errors when running nodepool tests locally: "Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)".23:03
clarkbzaro: you need to have a mysql server running23:03
zarodo i need to install something to get it to work?23:03
zaroneed a local mysql?23:03
*** dangers is now known as dangers_away23:03
clarkbzaro: I have a docker container with mysql in it that I start and run tests in23:03
mordredclarkb, fungi: +2 - I left off +A because fungi was looking23:04
openstackgerritSpencer Krum proposed openstack-infra/puppet-httpd: Reduce version to 0.0.0  https://review.openstack.org/16401323:04
*** MarkAtwood has quit IRC23:04
*** melwitt has joined #openstack-infra23:05
*** dims has quit IRC23:05
*** dboik has quit IRC23:05
zaroclarkb: you run test in it?  why not just connect to it?23:06
zaroor is that not possible?23:06
fungiclarkb: asselin: i'm cool with 163976 and curious to know what packages specifically were causing grief there, but also wondering if we should look to do something similar with yum23:06
clarkbzaro: it is possible, its just easier to run in it because it looks for that unix socket23:07
clarkbfungi: ruby1.8 on trusty and docker.io on precise23:07
fungiclarkb: is that a recent change?23:07
jeblairnibalizer: i'm almost done with a job config change23:07
jeblairnibalizer: i'm futzing with some regexes23:07
*** ayoung has quit IRC23:07
clarkbfungi: the problem seems to be we take all packages for all devstack branches then try to cache them on a single release of ubuntu when reality is devstack isn't going to run all branches against both precise and trusty23:07
*** andreykurilin_ has quit IRC23:07
mordredclarkb: ++23:08
clarkbfungi: I think it will only complicate caching if we try to encode which branches go to which releases during our image builds23:08
clarkbinstead best effort caching is ideal23:08
fungiclarkb: ah! as soon as i whip bindep into shape i should look at integrating it into devstack23:08
mordredclarkb: devstack already has trusty/precise support internally23:08
clarkbfungi: its likely that yum will have similar problems23:08
*** achanda has joined #openstack-infra23:08
mordredclarkb: it should not be terribly hard to use it23:08
clarkbfungi: but I don't know enough about yum to make a good answer about it but I can look into it23:08
mordredclarkb: I mean, I thnk this patch is great23:08
fungiclarkb: that way devstack can declare distro release specific deps23:08
mordredbecause $things_happen23:09
mordredclarkb: it already does23:09
mordredgah23:09
mordredfungi: it already does23:09
*** hichihara has joined #openstack-infra23:09
fungioh, but not in a way our nodepool scripts are able to consume23:09
clarkbianw: any idea how to tell yum to download only and not complain too much if it can't download something in particular?23:09
* clarkb tries a centos7 build locally23:09
mordredfungi: disagree. I'll make a patch23:09
nibalizerjeblair: so what the way you detect the tag to use?23:10
fungione of the next items on my list is to get nodepool running bindep against projects to determine what to cache, so if devstack used bindep to describe those then we could pretty easily just reuse the same in our nodepool image builds23:10
mordredfungi: http://git.openstack.org/cgit/openstack-dev/devstack/tree/files/debs/heat#n123:11
jeblairnibalizer: $ZUUL_REF23:11
nibalizeroh23:11
ianwclarkb: yumdownloader23:11
mordredfungi: all we have to do is read the dist: entry when we loop23:11
nibalizerim glad i asked :)23:11
mordredfungi: now - sure to bindep23:11
nibalizerwould have taken a while23:11
mordredjust saying, it's not really a hard thing to process23:11
*** achanda_ has quit IRC23:11
ianwclarkb: in fact yum not complaining if it can't download has been an issue for devstack, although not sure about network errors v actually missing packages23:11
clarkbianw: thanks, I will look into that if the centos7 build breaks like precise/trusty were23:12
fungimordred: oh, so it does have distro release identifiers and we were just ignoring them? yeah that's a quick-ish fix23:12
mordredyes23:12
clarkbfungi: mordred I wouldn't trust those completely23:12
clarkbthe reason is master/kilo is currently only tested on trusty23:12
clarkbso we don't actually test if any of those deps are trusty only23:12
fungii guess ideally if projects used by devstack had bindep manifests then devstack might be able to drop its rpms and debs lists entirely at that point23:12
clarkbsame problem with icehouse being precise23:13
clarkbdevstack is going to best effort it but I wouldn't count on it23:13
openstackgerritJames E. Blair proposed openstack-infra/project-config: Update forge-upload job to use tags  https://review.openstack.org/16401623:13
mordredclarkb: sure- which is why I think the patch we just did was a good idea23:13
jeblairnibalizer: https://review.openstack.org/16401623:13
mordredclarkb: BUT - we can also only cache what devstack asks for23:13
*** nelsnelson has joined #openstack-infra23:13
jeblairnibalizer: i totally just guessed at what the rakefile should look like :)23:13
clarkbmordred: yup23:13
jeblairianw: did i ever answer your question?23:14
ianwjeblair: which question?23:14
*** patrickeast has joined #openstack-infra23:14
jeblairianw: about the logging patch23:14
jeblairi am apparently not able to carry on as many conversations as clarkb23:14
ianwjeblair: not yet, that's fine, i only meant it to go into a queue while i was looking at my reviews23:15
openstackgerritMerged openstack-infra/project-config: Ignore missing packages  https://review.openstack.org/16397623:15
clarkbjeblair: really my brain is melted. Also, I cleaned the shower earlier today and the bleach may have given me super powers23:16
*** ghostpl_ has joined #openstack-infra23:16
jeblairianw: i'm not going to -2 it if other people think that's the best approach, but i really would prefer one of the other approaches i listed.23:16
* fungi is reminded of a dead milkmen song about drinking bleach23:16
ianwjeblair: i think my issue is finding anyone else to give an opinion23:17
openstackgerritSpencer Krum proposed openstack-infra/project-config: Improvements to forge uploading process  https://review.openstack.org/16401823:17
*** yamamoto has joined #openstack-infra23:18
fungijeblair: ianw: i'm good with just hand-maintaining a logging config for the separate images with the understanding that we're working toward substantially reducing their count in the near future anyway23:18
nibalizerjeblair: crinkle https://review.openstack.org/#/c/164018/23:18
nibalizeroh you tried too23:18
nibalizeroops23:18
clarkbjeblair: ianw: I agree with jeblair that the logging config shouldn't be autogened by puppet and put in place. However, I don't having a utility like that that we can use to create and propose the logging config in gerrit changes23:19
ianwfungi: i'm not sure if i pasted the output of that command, it's not really maintainable if you split up by provider IMO23:19
*** crc32 has quit IRC23:20
jeblairclarkb: if we want a utility like that, it should probably be in system-config, not in nodepool itself23:20
jeblairsince it's _our_ logging config23:20
clarkbjeblair: that would be fine23:20
clarkb*I don't mind having a utility23:20
fungiyeah, that seems reasonable. again it would be hand-maintained configuration with a helper tool we can use to submit the updates23:21
ianwok that's where i started with https://review.openstack.org/#/c/147368/23:22
ianwbefore the config split, that could rather easily create the config file23:23
*** asettle has joined #openstack-infra23:23
ianwi will move it23:23
nibalizerjeblair: i've abandoned 164018 for 164016, and commented23:24
*** ghostpl_ has quit IRC23:25
morganfainbergdhellmann, jogo, sdague, so i have an idea for a hacking check i need to implement - but i don't want to wedge everyone. basically i need a way to say "don't consume the options in the [keystone_authtoken] section of the config if a project is using auth_token middleware. I am thinking we expose this hacking check via an entrypoint in middleware. but i want to do it "right" - however that is.23:26
openstackgerritJames E. Blair proposed openstack-infra/project-config: Update forge-upload job to use tags  https://review.openstack.org/16401623:26
jeblairnibalizer, crinkle: ^23:26
clarkbany opposition to me restarting nodepool now to pick up all the code fixes and things?23:26
*** otter768 has quit IRC23:26
SpamapSclarkb: Hey I just noticed that automatic nodepool coverage report link you pasted. I kind of wonder why we don't have that turned on in python-jobs.. ?23:26
jeblairclarkb: i want to restart zuul to get the logging changes23:26
clarkbSpamapS: because its super slow for many projects23:26
jeblairclarkb: let's coordinate!23:26
clarkbjeblair: +1 :)23:27
SpamapSclarkb: ahh23:27
SpamapStoo bad23:27
morganfainbergdhellmann, jogo, sdague, the key is projects can't rely on those options for anything besides middleware doing token validates (no you can't use those same options to talk to nova or neutron). any thoughts on the best way to approach this... softly to start23:27
*** tnovacik has joined #openstack-infra23:27
clarkbSpamapS: (it is in python-jobs on the jenkins side but not the zuul side, you can add it individually if you want)23:27
clarkbjeblair: what sort of coordinating should we do?23:27
*** wenlock has quit IRC23:28
jeblairclarkb: okay, i either need to open 8 terminal windows for all the zuul mergers or finally set up ansible locally23:28
clarkbjeblair: you can use ansible on the puppetmaster23:28
clarkbbut I don't know the exact incantation23:29
jogomorganfainberg: ping me tomorrow and I may be able to help brainstorm23:29
jeblairclarkb: oh good idea.. how do i do that?23:29
clarkbjeblair: let me look it up23:29
morganfainbergjogo, sounds good123:29
jogomorganfainberg: but not sure if hacking is the right thing for that etc.23:29
*** yamamoto has quit IRC23:29
morganfainbergjogo, this is why i'm asking, hacking was my first thought. but of course open to others23:29
jeblairclarkb: oh, bash history!23:29
clarkbjeblair: ansible 'zm0.*' -m shell -a '/etc/init.d/zuul-merger restart' -f823:30
clarkbjeblair: something like that23:30
jeblairwhat's the -f8?23:30
clarkbjeblair: its the parallelism23:30
clarkbit can be omitted23:30
*** zz_dimtruck is now known as dimtruck23:30
clarkbalso we still only have 3 leaked ports so that number hasn't changed over the course of the day \o/23:31
jeblairclarkb: okay.  so i'm going to save the queue, stop zuul, stop mergers; then you can stop nodepool, set all nodes to delete, start nodepool; then i'll start zuul, start mergers, reload queue.23:31
jeblairclarkb: sound good?23:31
*** achanda has quit IRC23:32
jeblairclarkb: actually, let's modify that23:32
clarkbjeblair: I think so. I have to figure a good way to set all nodes to delete23:32
*** achanda has joined #openstack-infra23:32
jeblairclarkb: mysql: "update node set state=4; update sub_node set state=4"23:32
clarkbjeblair: thanks23:32
jeblairclarkb: i'll save the queue, stop zuul, stop mergers; then you can stop nodepool, set all nodes to delete; i'll start geard; you start nodepool; wait until things have settled a bit, then i'll stop geard, start zuul, start mergers, reload queue.23:33
clarkbjeblair: sounds good, let me know when it is my turn23:33
jeblairclarkb: zuul is stopped; go for it23:34
*** psedlak has quit IRC23:34
jeblairclarkb: and geard is running so you can start nodepool whenever23:35
clarkbok starting it now23:35
SpamapShrm..23:35
SpamapStrying to track down how to pass --omit's to coverage report generation23:35
*** melwitt has quit IRC23:35
jeblairclarkb: so now let's wait until we have some new nodes so all the jobs get registered23:35
clarkbit is doing the things so we should just need to wait at this point23:36
jeblairit's fun to see geard as the countdown for that23:36
jeblairDEBUG:gear.Server:Polling 199 connections23:36
jeblairDEBUG:gear.Server:Polling 189 connections23:36
clarkbSpamapS: I think the easiest way is via a .coveragerc23:36
SpamapSclarkb: ah yeah thats the ticket23:37
clarkbSpamapS: pbr may also support it23:37
SpamapSclarkb: .coveragerc is perfect actually23:37
openstackgerritSpencer Krum proposed openstack-infra/puppet-logstash: Flesh out the readme  https://review.openstack.org/15382023:37
jeblairDEBUG:gear.Server:Polling 28 connections23:38
jeblairi think that was the nadir23:38
clarkbjeblair: http://paste.openstack.org/show/191971/ I think that happens when we switch a node to delete before it has a uuid?23:40
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/project-config: Turn on oslo.messaging coverage report  https://review.openstack.org/16402223:40
clarkbjeblair: I am guessing we might want to delete that row specifically then do an alien listing to kill aliens later?23:40
jeblairclarkb: i think that might be a database race, where the row was deleted out from under it23:40
clarkbjeblair: gotcha, I will see if it goes away then23:41
*** david-lyle is now known as david-lyle_afk23:41
jeblairclarkb: as in, we started a thread saying "delete node id foo" and by the time it tried to delete it, the row in the db was already gone23:41
jeblairclarkb: i think we have enough node types online now.23:41
jeblairi'll restart zuul23:41
clarkbok23:41
ianwjeblair / clarkb / fungi : so do you want https://git.openstack.org/cgit/openstack-infra/puppet-nodepool/tree/manifests/init.pp to a) have the generated log-config checked into it b) not deploy a log-config & leave it up to system-config c) something else?23:44
*** Sukhdev has quit IRC23:44
jeblairsdague: the logging to help figure out what's going on when a change is removed from the check queue because "a parent failed to merge" is in place; let me know if you see it again.23:44
jeblairianw: puppet-nodepool should also not have our logging configuration in it23:45
*** asettle has quit IRC23:45
fungiianw: system-config should be supplying it23:45
fungiianw: system-config is intended to be "systems configuration for the servers run by openstack infra" (ignoring for the moment that lots of people seem to want to also genericize that module to run copies of all our infrastructure without forking it)23:47
ianwfungi: any pointers on how to get puppet to use the template from system-config, rather than the local one?  is it like puppet://modules/system-config/path/to/template.erb maybe?23:47
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: zuul-swift-upload fix delete-after  https://review.openstack.org/16402323:47
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: zuul-swift-upload ensure delete after is an int  https://review.openstack.org/16402423:47
jheskethclarkb: ^23:47
clarkbok I have results for centos7 build. It caches packages ok but the wheel thing explodes on it so fails anyways23:47
fungiianw: we could make the logging config file path parameterized in the puppet-nodepool module, defaulting to the current one, but then supply an alternative path when we instantiate that module from the openstack_project module23:48
clarkbjhesketh: does requests.post with the data parameter coerce the types to the appropriate thing?23:48
clarkbjhesketh: looking at the other ints we have in there I am guessing so23:48
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Turn on post taskflow coverage report  https://review.openstack.org/16402523:48
jheskethclarkb: yeah, that's what I assumed23:49
ianwfungi: yep, that's sort of my idea, if you can think of anything else that does that i'd just like to see the syntax.  i'll dig though23:49
*** unicell has joined #openstack-infra23:49
jeblairfungi: i just traced one of the merge failures down to the merger itself quickly.23:50
jeblair2015-03-12 23:43:57,325 INFO zuul.MergeClient: Merge <gear.Job 0x7fefe83b9650 handle: H:127.0.0.1:79 name: merger:merge unique: 263d5fb242094ee2b9292b8d6207fbbb> complete, merged: False, updated: False, comm23:50
jeblairit: None23:50
jeblair2015-03-12 23:43:57,325 DEBUG zuul.Scheduler: Adding merge complete event for build set: <BuildSet item: <QueueItem 0x7fefe83b9610 for <Change 0x7fefea397bd0 154513,26> in gate> #builds: 0 merge state: PENDI23:50
jeblairNG>23:50
fungijeblair: oh! i was not having a lot of luck tracking down which merger processed which task previously23:50
jeblairfungi: yeah, i just restarted zuul with the new debug lines23:50
fungigot it23:50
jeblairfungi: so check " unique: 263d5fb242094ee2b9292b8d6207fbbb" from there23:50
jeblairansible zm* -m shell -a 'grep 263d5fb242094ee2b9292b8d6207fbbb /var/log/zuul/merger-debug.log'23:50
jeblairthen that ^23:51
clarkbjeblair: did you also restart the workers with your checkout change?23:51
jeblairzm05.openstack.org | success | rc=0 >>23:51
jeblair2015-03-12 23:42:59,338 DEBUG zuul.MergeServer: Got merge job: 263d5fb242094ee2b9292b8d6207fbbb23:51
jeblairthen that ^23:51
fungiaha23:51
jeblairand then read down a bit to find:23:51
jeblairfatal: merge program failed'23:51
jeblairstderr: 'error: Merge requires file-level merging23:51
jeblairso, real merge failure that one :)23:51
fungiyeah, distributed grep across the logs of all the workers is an awesome idea. i didn't think of that anyway23:51
*** dboik has joined #openstack-infra23:52
*** vladan has left #openstack-infra23:52
jeblairwe could probably report the worker in the server log line too; i didn't think of that.  but this is a lot better than before :)23:52
fungiabsolutely23:52
*** yamamoto has joined #openstack-infra23:52
jeblairso next time we get a merge failure that doesn't appear to be a merge failure, we should be better equipped23:52
*** dboik_ has joined #openstack-infra23:52
jeblairclarkb: i did not, because apparently that change fails all kinds of tests23:52
jeblairclarkb: so it may require actual thought23:53
jeblairi am not prepared to give it that right at this moment :)23:53
fungii wonder where the "requires file-level merging" is coming from. if it's the ceilometer icehouse patch apevec is trying to get merged, that was a pure ff23:53
clarkbjeblair: oh fun23:53
*** asettle has joined #openstack-infra23:53
jeblairfungi: oh, that was just the neutron change near the top failing23:54
apevecfungi, it merged23:54
fungiapevec: did it? strange23:54
apevecyeah...23:54
jeblairapevec: good for you, bad for our debugging :)23:54
apevechttps://review.openstack.org/14141223:54
fungithe only thing i hate more than bugs which won't go away is bugs which silently go away23:55
apevecps2 failed then merged in 2. attempt23:55
*** reed has quit IRC23:55
clarkbI have started a devstack-trusty dib build on nodepool.o.o and will upload it as soon as its done23:55
clarkbdhellmann: ^ this should fix your d-g test fail23:55
*** mtanino has joined #openstack-infra23:55
* SpamapS silently goes away23:56
*** gyee has joined #openstack-infra23:56
*** ChuckC has quit IRC23:56
*** dboik has quit IRC23:56
*** sabeen has quit IRC23:57
*** nelsnelson has quit IRC23:57
*** annegentle has joined #openstack-infra23:58
*** dannywil_ has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!