Wednesday, 2015-04-22

*** otter768 has quit IRC00:00
*** jpeeler has quit IRC00:00
*** annegentle has joined #openstack-infra00:01
*** agentle_ has joined #openstack-infra00:01
*** achanda has quit IRC00:01
*** markvoelker has joined #openstack-infra00:02
jogoianw: hopefully that will do the trick00:02
*** agentle_ has quit IRC00:05
clarkbmy naive attempt at that fix fails elsewhere00:05
clarkbbut I am not really debugging it00:05
greghayneslooks like theres already a pycparser fix up00:05
ianwthat's a fair trap that using __slots__ without __weakref__ breaks weak references ...00:05
greghaynesnot sure how long they will take to release00:06
lifelessfungi: ninjafixed?00:06
*** annegentle has quit IRC00:06
* dhellmann joins the fun late00:06
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a centos-6 node label
clarkboh my fix was incomplete00:06
fungihopefully ^ that will get lucky and test with fixed pycparser00:07
fungilifeless: the new pycparser release00:07
fungioh, not tagged yet00:07
*** markvoelker has quit IRC00:08
funginice. eliben says "I would advise cffi and other projects to lock their dependency on a "known-good" pycparser release and bump it occasionally after testing explicitly."00:08
clarkb it may not be complete00:08
clarkbbut I don't know anything about this project00:09
*** swat30 has joined #openstack-infra00:09
*** luqas has joined #openstack-infra00:09
greghayneswell, eliben says likely fixed00:09
greghaynesso *shrug*00:09
greghaynesbut yea, *everything* is broken00:09
greghaynesits kind of amazing00:09
lifelessat 2.x I'd have expected some feeling for stability00:09
lifelessOTOH we are using a 0.x library for crypto00:09
lifelessso foot, gun, trigger00:10
dhellmannfungi, lifeless : did we come up with a solution for our own issue, or do we have to have this pycparsing thing fixed to even tell? because if that's the case, I think clarkb's idea of turning off bandersnatch might be something to explore00:10
fungiespecially when going from 2.10 to 2.11 but whatever00:10
clarkbI guess if its an ABC then nothing else will have refs to instances of that class?00:10
lifelessdhellmann: I don't know if sdague's patch, which was the one we had group come up with, worked or not or just hit this fail00:10
clarkbthough if you just set __weakref__ in slots on the ABC would that not be sufficient?00:10
clarkboh no because slots is ioverridden in children00:10
*** isviridov_away is now known as isviridov00:11
KennanI also tried latest devstack, it also broke on tempest install with ' TypeError: cannot create weak reference to 'Struct' object'00:11
lifelessfungi: where does eliben say that?00:11
clarkbKennan: yes see scrollback00:11
fungiKennan: yeah, that's courtesy of latest pycparser00:11
dhellmannlifeless: yeah, it hit this fail, so we don't know. that's why I was saying maybe the bandersnatch thing would be a good idea, since we could explore the other fix we have to make00:11
lifelessclarkb: no, slots are concrete00:11
fungilifeless: latest comment in
*** jpeeler has joined #openstack-infra00:11
fungiactually just now no longer latest comment. he wants some testing and will then tag a new release asap00:12
clarkbI can test00:13
clarkbif I can figure out github00:13
ianwlifeless: yeah; playing with __slots__ is almost like fiddling with your ABI ... plenty of potential to mess up.  but i also think this is probably a corner-case, as there was no unit test of weak references.  who thinks of that, really00:13
*** luqas has quit IRC00:13
fungiafter the last couple days of reading debian-devel i've realized that i fail at github00:13
greghaynescryptography says it fixes for them00:13
*** sarob has joined #openstack-infra00:14
lifelessianw: my reaction was actually WTF are they taking weakrefs for00:14
clarkbya works here00:15
lifelessfungi: oh?00:15
fungii'm not even close to being a github apologist, but there were more than a few detractors unwilling to even interact with its ui00:16
fungijust mildly amused, but it's debian-devel so why am i surprised really00:17
*** tsg has quit IRC00:17
clarkbianw: a simple (and msotly reasonably) set of functional tests since this project exists to parse C for python is to run other projects C compiles00:17
clarkbianw: while that isn't a specific weakref test it would likely catch all kinds of fun issues and travis will run it for free00:18
clarkbwe could even run it here on a periodic basis ( we don't have github triggers so hard to trigger per commit)00:18
*** bhuvan has quit IRC00:18
openstackgerritJulia Kreger proposed openstack-infra/zuul: Adding doc reference of ZUUL_CHANGES
*** mtanino has quit IRC00:19
*** davideagnello has quit IRC00:20
fungiTheJulia: fyi hashar just today said he also has some documentation updates on the way for zuul, so those might be a good idea to keep an eye out for and review00:22
*** yamamoto has quit IRC00:22
fungiwe'll probably shoot for making sure its documentation is as polished as we can reasonably get it before tagging the impending releas3e00:23
*** achanda has joined #openstack-infra00:23
*** david-lyle has quit IRC00:24
*** tqtran has quit IRC00:24
lifelessfungi: I was rather amused at being (AFAICT) lumped in with the apologists00:26
*** davideagnello has joined #openstack-infra00:27
fungiyou proprietary software monger you00:27
*** david-lyle has joined #openstack-infra00:27
*** sputnik13 has quit IRC00:27
*** achanda has quit IRC00:28
jogoand pycparser is fixed
jogoback to one wedge00:29
*** tsg has joined #openstack-infra00:29
*** ddieterly has joined #openstack-infra00:31
swat30hey all00:32
swat30pushed my first review and am seeing quite a few tests fail, some seem unrelated to my changes00:32
fungijogo: one wedge... for now! ;)00:32
swat30can someone take a look and let me know if I'm doing anything wrong?00:32
clarkbswat30: its likely related to the recent pycparser explosion but I will look00:32
swat30clarkb, thanks! it does seem to be something odd going on00:33
fungiswat30: i find it interesting that your first change is to the stable/icehouse branch of neutron00:33
clarkbswat30: looks like the stevedore issue hit you instead which is the other wedge fungi and jogo are talking about00:33
*** davideagnello has quit IRC00:33
swat30fungi, yea, working on an upgrade on an old quantum install00:33
swat30ran into an issue w/ DB migrations00:34
swat30saw a bug already filed, but unfixed00:34
fungiswat30: so bug 1241577 is icehouse-only?00:34
openstackbug 1241577 in neutron "sqlalchemy.exc.OperationalError: (OperationalError) Cannot add a NOT NULL column with default value NULL" [Low,In progress] - Assigned to Jacob Godin (jacobgodin)00:34
*** gmurphy has joined #openstack-infra00:35
jogodhellmann: now that the crpyto issue is fixed, you have a fix for the other issue?00:35
swat30fungi, hm, good Q. the DB migration from grizz. is gone in Juno00:35
*** ddieterly has quit IRC00:35
fungiswat30: then sounds like icehouse is probably the only supported branch where it still exists, so that sounds right00:35
swat30that was my thought fungi00:36
swat30I'll look into it00:36
*** emagana has quit IRC00:36
dhellmannjogo: we have a couple of things in play. sdague submitted and I am working it from the other side with
fungiswat30: anyway, what clarkb says is true, we're running into some (hopefully very temporary) testing issues while trying to transition to a new requirements tracking workflow for the upcoming kilo release00:36
jogodhellmann: btw you may find the discussion at the end of funny00:37
jogore: python dependency hell00:37
*** jaypipes has quit IRC00:37
dhellmannjogo, fungi, sdague, ttx, lifeless: I started an etherpad for notes
jogofungi: yes00:38
swat30fungi, appreciate it. I saw some global req updates to a bunch of projects. I assume that's related?00:38
dhellmannjogo, fungi, sdague, ttx, lifeless: since this is only sort of related to the requirements plan, I didn't want to add all of the details there00:39
fungiswat30: lots of library releases too, yes00:39
*** zhiwei has joined #openstack-infra00:39
swat30fungi, cool. I'll let it be for a bit and give it a recheck in a day or two. thanks for your help00:39
*** zz_dimtruck is now known as dimtruck00:40
fungiheh... "maybe this is a pip problem"00:40
jogodhellmann: makes sense, looks like with a bit of luck we can be out of this mess in a few hours00:40
clarkbdhellmann: 176113 is the simple fix right?00:40
jogofungi: :)00:40
clarkbdhellmann: eg if that one works then we can just bump that req?00:41
* dhellmann goes to look up the change clarkb is referring to00:41
*** armax has quit IRC00:41
dhellmannclarkb: maybe. We don't know if that will actually work, yet00:41
fungidhellmann: "sean's patch" per the etherpad00:42
clarkbbut we should know in about an hour?00:42
dhellmannclarkb: and it may not be sufficient, we may need to also merge that change into the other projects00:42
dhellmannclarkb: assuming nothing else happens, yeah00:42
*** ddieterly has joined #openstack-infra00:42
fungiclarkb: 27 minutes so sayeth the zuul00:42
jogodhellmann: can we test this outside of the gate?00:42
clarkbjogo: if you have a tempest laying around ready to run00:43
dhellmannlarge-ops has already failed00:43
fungialso it's already failed gate-tempest-dsvm-large-ops though maybe that's unrelated?00:43
jogodhellmann: link?00:43
jogoahh found it00:43
*** dannywilson has quit IRC00:43
clarkb for those following along00:44
dhellmannthe job we care about is because that's the grenade test for the job that depends on the requirements change00:44
*** dannywilson has joined #openstack-infra00:44
clarkbI think this is just the next in line issue00:44
jogoclarkb: sigh00:45
fungiit's too bad they chain the mallet to this whack-a-mole machine00:45
jogofungi: invest in bolt cutters00:45
dhellmannI wish we could get a traceback from that command, so we could see whether the issue is in the cli program or the plugin00:45
fungiit is my honor to present to the qa team... this crate of whack-a-mole mallets. may they serve you well00:45
* jogo humbly accepts the crate on behalf of the qa team and hands them out00:47
* dhellmann bangs his head on his mallet00:47
jogodhellmann: the time stamp is interesting00:47
jogomorganfainberg: halp00:47
jogo127.0.0.1 - - [22/Apr/2015:00:42:36 +0000] "POST /v2.0/tokens HTTP/1.1" 401 365 "-" "python-keystoneclient"00:48
jogois the 401 error we are seeing at 42:3600:48
morganfainbergbut i need to jump out for food and an appointment very soon00:48
jogomorganfainberg: top level bug
morganfainbergAuthorization failed. Could not find user: admin (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from
morganfainbergis... that possible?00:49
morganfainbergi mean, the admin user should exist by this point00:49
jogoI don't think keystone was ready yet00:50
jogoat 42:3600:50
morganfainbergi mean, if the user legitimately isn't there this makes sense00:50
morganfainbergthat would be exactly what i would expect to have occured00:50
jogohmm is the error00:50
morganfainbergdep resolution issues00:51
jogoughhh dhellmann00:52
morganfainbergyay!  i mean... opposite of "yay"00:52
dhellmannjogo: ?00:52 Exception raised: (oslo.utils 1.2.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('oslo.utils>=1.4.0'), set(['python-novaclient']))00:52
*** yamamoto has joined #openstack-infra00:53
*** prad has quit IRC00:53
* morganfainberg goes to get food and head off for a couple hourse.00:53
jrollis it safe to assume nova jobs are failing right now?00:53
anteayajroll: lots of failure right now00:54
jrollthanks for confirmation anteaya :)00:54
anteayapyparser and a stevedore issue if my scorecard is correct00:55
jogoanteaya: pycparser is fixed00:55
Kennanthe pip00:55
anteayajroll: happy to do the least I can do00:55
anteayajogo: yay00:55
jogothe issue is now oslo.utils too00:55
anteayajogo: ah00:55
* anteaya updates her scorecard00:55
fungi(plus whatever problems are hiding behind that)00:55
morganfainberganteaya, i'm playing bingo00:55
jogofungi: :(00:56
fungimorganfainberg: it's all fun and games until someone loses an i00:56
anteayamorganfainberg: bingo whack-a-mole everyone has a game00:56
morganfainbergfung: hey, i think you lost one.00:56
* morganfainberg sees himself out for the bad joke.00:56
jogohow come devstack doesn't run with set -e00:57
*** marun has quit IRC00:57
anteayamorganfainberg: you might be greeted at the door with someone with a mallet insisting you stay00:57
morganfainberganteaya, hehe00:57
fungijogo: i think parts of it do, yeah?00:57
dtroyerjogo, it is supposed to, it appears to have been broken at some point00:57
fungithough it was not that long ago that we got most of devstack-gate running set -e00:58
morganfainbergit doesn't run set -e?00:58
* morganfainberg thought it did00:58
Kennanthe issue is not common hit nowdays, we found in requests package, it not install with requirements.txt, but used old ubuntu default package version00:59
dhellmannjogo: I think this whack-a-mole approach is wrong. I'm working on instead00:59
jogodhellmann: is that the right link? that points to sdague's patch01:00
*** tjones1 has joined #openstack-infra01:00
fungiKennan: yes, if you're pip installing into the system environment without -U then whatever version of requirements might have been installed from a distro package will get left in place, not replaced by any newer version from pypi01:00
dhellmannjogo: tab overload:
*** dannywilson has quit IRC01:00
fungiKennan: same goes for any python library01:01
*** johnsom has left #openstack-infra01:01
Kennanyes, :fungi it was for oslo-utils issue, same issue, I think01:02
*** ayoung has joined #openstack-infra01:02
openstackgerritMerged openstack-infra/system-config: Revert "Revert "Turn off HP in nodepool""
*** annegentle has joined #openstack-infra01:03
*** agentle_ has joined #openstack-infra01:03
dhellmanndtroyer: if you're still around, do you want to give a quick look to ? I'll self-approve if the tests pass, otherwise01:05
jogoharlowja: makes sense, better then whack-a-mole01:06
*** tjones1 has quit IRC01:06
jogodhellmann: ^01:06
jogodtroyer: test patch
dhellmannjogo: right, though we'll have to see if it does actually fix the issue01:06
jogodtroyer fungi: testing add set -e in devstack to see what happens01:06
dtroyerdhellmann: I'm in&out…  I think that looks fine but I'll admit that's on the edges of my setuptools understanding01:06
*** kaisers has quit IRC01:07
dhellmanndtroyer: there is similar code in stevedore, but I don't have to accommodate the openstackclient test code that creates a fake, so it's simpler01:07
dtroyerjogo: it's in there, w need to figure out where it is getting turned off and not back on01:07
*** agentle_ has quit IRC01:08
dhellmannI'll fix the fake separately, but won't be able to land that fix until the juno issues are resolved01:08
jogodtroyer: where is it a quick git grep didn't show anything01:08
*** prad has joined #openstack-infra01:08
dtroyerline 477 in my current branch - search for errexit01:08
*** annegentle has quit IRC01:08
lifelessis pycparser all good now ?01:09
jogolifeless: yes01:09
*** patrickeast has quit IRC01:11
*** gyee has quit IRC01:12
jogodtroyer: you may need If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment. The ERR trap is normally not inherited in such cases.01:13
dtroyerI suspect that there's a function where errexit is disabled that isn't re-enabling it01:15
*** sarob has quit IRC01:15
dtroyerwas trying to cut down on useless logging once upon a time01:15
*** shashankhegde has quit IRC01:16
*** sarob has joined #openstack-infra01:16
*** baoli has joined #openstack-infra01:17
jogodtroyer: I think this should do it
jogoas that fits the symptoms some things work for the trap and some don't01:18
* jogo heads out for some fresh air01:19
*** sarob has quit IRC01:21
*** baoli_ has joined #openstack-infra01:21
*** kaisers has joined #openstack-infra01:22
dtroyererr…I confused errexit and xtrace above, it's xtrace we toggle on and off…jeez, time for air for me too?01:23
*** baoli has quit IRC01:24
*** baoli_ has quit IRC01:26
*** asselin_ has joined #openstack-infra01:30
*** baoli has joined #openstack-infra01:34
HenryGWill rechecks work now, or not yet?01:38
*** ajmiller has joined #openstack-infra01:41
*** julim has joined #openstack-infra01:46
*** changbl has joined #openstack-infra01:48
*** otter768 has joined #openstack-infra01:56
*** stevemar has joined #openstack-infra01:56
*** tsg has quit IRC01:59
*** dboik has joined #openstack-infra02:00
*** agentle_ has joined #openstack-infra02:00
*** annegentle has joined #openstack-infra02:00
dtroyerjogo: here's a simple brute-force test of errexit (-e) and errtrace (-E) in functions and subshells:   the script exits early for every combination except when both are off and the error is in a function.  that's what we're seeing in the osc errors in these runs.  errtrace only affects when the err_trap is executed.02:00
*** otter768 has quit IRC02:01
*** unicell has quit IRC02:01
*** markvoelker has joined #openstack-infra02:01
*** harlowja is now known as harlowja_away02:03
*** dboik has quit IRC02:05
*** agentle_ has quit IRC02:09
*** annegentle has quit IRC02:09
*** xyang1 has quit IRC02:09
*** luqas has joined #openstack-infra02:10
*** shashankhegde has joined #openstack-infra02:13
*** luqas has quit IRC02:15
*** macjack has joined #openstack-infra02:20
*** kaisers has quit IRC02:24
*** agentle_ has joined #openstack-infra02:27
*** annegentle has joined #openstack-infra02:27
*** alexpilotti has quit IRC02:27
*** ChuckC has joined #openstack-infra02:29
*** a13x2121 has joined #openstack-infra02:34
*** Swami_ has joined #openstack-infra02:36
*** Swami has quit IRC02:37
*** esker has joined #openstack-infra02:38
*** jamielennox is now known as jamielennox|away02:39
*** sputnik13 has joined #openstack-infra02:39
*** kaisers has joined #openstack-infra02:39
*** sputnik13 has quit IRC02:44
*** sputnik13 has joined #openstack-infra02:44
Kennandid you limit the versionedobject becase 0.2.0 not work ?02:47
*** sputnik13 has quit IRC02:47
KennanI see it is lower than >0.1.1 <0.2.002:48
lifelessKennan: what?02:48
Kennan>=0.1.1 < 0.2002:48
lifelesssee what is02:48
*** ibiris is now known as ibiris_away02:48
Kennanbecause 0.2.0 the latest version breaks on openstack projects02:49
Kennanright ?02:49
lifelessKennan: I have *no* idea what you're talking about02:50
*** hichihara has joined #openstack-infra02:50
*** ivar-laz_ has joined #openstack-infra02:50
Kennanthe 0.2.0 has such change02:50
*** adreznec has quit IRC02:50
Kennanso we not use 0.2.0, because it breaks on some code02:51
lifelessHenryG: we're still wedged AIUI02:51
Kennanright ?02:51
*** mfink has quit IRC02:51
*** ivar-lazzaro has quit IRC02:51
jogodtroyer: so your saying errtrace doesn't do anything without errexit?02:51
lifelessKennan: no02:52
lifelessKennan: we have upper caps on many things because thats the current strategy being used to avoid situations like we're in now02:52
lifelessKennan: where whats installed doesn't work.02:52
*** ivar-laz_ has quit IRC02:53
Kennanok. I am ok with that. Just curious about that.02:53
Kennandid you have any comments about
*** davideagnello has joined #openstack-infra02:53
Kennan:lifeless I am not sure if we have clear of issue we talked before,  ubutun system install conflict with pip install02:54
Kennanis it pip someting ? or devstack issue ?02:55
lifelessnothing to do with system install02:55
lifelessits because of pip issue 98802:55
lifelesscombined with pip issue 268702:56
*** mfink has joined #openstack-infra02:58
Kennanok :lifeless so how to deal with that pip issue ? if pip not fix that03:00
KennanI think its better to raise some version03:00
lifelessKennan: there is ongoing debate about that03:00
lifelessKennan: please see the thread(s) on openstack-dev for context03:00
lifelessKennan: requirements is part of our mechanism for dealing with the issue03:00
Kennanyes. :lifeless. thanks. not subscribe openstack-dev threads recently03:01
*** adreznec has joined #openstack-infra03:01
ianwlifeless: so, as the testing guru, what's wrong with ? :)03:06
*** asselin__ has joined #openstack-infra03:09
lifelessianw: commented (twice)03:09
*** agentle_ has quit IRC03:10
*** annegentle has quit IRC03:10
*** asselin_ has quit IRC03:11
ianwlifeless: ok, thanks; i couldn't find many examples of projects sort of using themselves as functional tests for other projects.  i think it's a good smoke test, at least03:12
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed
openstackbug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,In progress] - Assigned to Bharat Kumar Kobagana (bharat-kobagana)03:12
dtroyerjogo: I'm saying its a no-op WRT failing, it only changes how the err trap is called03:13
*** agentle_ has joined #openstack-infra03:13
*** annegentle has joined #openstack-infra03:13
*** sputnik13 has joined #openstack-infra03:14
*** sputnik13 has quit IRC03:14
*** asalkeld has left #openstack-infra03:17
*** agentle_ has quit IRC03:20
*** annegentle has quit IRC03:20
*** annegentle has joined #openstack-infra03:21
*** agentle_ has joined #openstack-infra03:21
*** davideagnello has quit IRC03:25
*** agentle_ has quit IRC03:26
*** annegentle has quit IRC03:26
*** mescanef has quit IRC03:28
*** mescanef has joined #openstack-infra03:28
openstackgerritlifeless proposed openstack/requirements: raise stevedore requirements
*** jerryz has joined #openstack-infra03:32
jogodtroyer: ahh, my bashfoo is very weak03:33
*** baoli has quit IRC03:34
*** hichihara has quit IRC03:35
*** dboik has joined #openstack-infra03:36
*** dboik_ has joined #openstack-infra03:37
*** hichihara has joined #openstack-infra03:38
*** hichihara has quit IRC03:38
*** dboik has quit IRC03:40
*** Swami_ has quit IRC03:48
jogodtroyer: so what is causing devstack to not exit on failures?03:49
*** asselin__ has quit IRC03:50
dtroyerI don't know03:50
*** shashankhegde has quit IRC03:50
jogohow do $() and pipes change your test?03:51
dtroyernot sure I follow on $0,  in a pipeline only the last component causes an error exit with non-zero return unless PIPEFAIL is in play, which we do only in a couple of places.03:53
*** dboik_ has quit IRC03:54
jogodtroyer: I was thinking
jogothat fails03:54
jogobut doesn't cause devstack to exit03:55
*** ddieterly has quit IRC03:56
*** ajmiller has quit IRC03:57
*** otter768 has joined #openstack-infra03:57
jogoperhaps get_field is messing things up03:58
dtroyerwhen that is still used, possibly.  and that is new enough we can't use it in juno03:59
dtroyerer, we haven't backported the elimination of the pipelines in those function03:59
jogodtroyer: so this is one of the errors we see in successful runs
jogoahh, so we want to eliminate pipelines in those that makes sense04:00
jogoanyway its getting late here o/04:00
*** asselin_ has joined #openstack-infra04:00
dtroyerI'm actually ready to try to stop using our tools in jobs where they may also be under test04:00
dtroyerie, OSC isn't appropriate for utility use in DevStack, as much as I want it to be04:00
*** dannywilson has joined #openstack-infra04:01
dtroyerand I'm not kidding anymore about rewriting it in Go…spent >20 hours on that over the weekend...04:01
*** otter768 has quit IRC04:02
jerryzfungi: ping04:02
*** fawadkhaliq has joined #openstack-infra04:02
*** juggler_ has joined #openstack-infra04:05
*** juggler has quit IRC04:06
*** krtaylor has quit IRC04:06
*** dannywilson has quit IRC04:06
*** sslypushenko has quit IRC04:09
*** krtaylor has joined #openstack-infra04:10
*** luqas has joined #openstack-infra04:11
*** mahito has joined #openstack-infra04:15
*** patrickeast has joined #openstack-infra04:15
*** luqas has quit IRC04:16
*** BharatK has quit IRC04:20
*** enikanorov__ has quit IRC04:21
*** agentle_ has joined #openstack-infra04:22
*** annegentle has joined #openstack-infra04:22
*** agentle_ has quit IRC04:27
*** annegentle has quit IRC04:28
*** sdake has joined #openstack-infra04:29
*** sdake_ has joined #openstack-infra04:32
*** sdake has quit IRC04:33
*** unicell has joined #openstack-infra04:36
lifelessdhellmann: sdague:
lifelessdhellmann: that seems promising, only failure was grenade04:37
*** mrmartin has joined #openstack-infra04:42
*** fawadkhaliq has quit IRC04:43
*** ibiris_away is now known as ibiris04:43
*** prad has quit IRC04:45
dtroyersame ole thing, uncapped clients (neutronclient this time) getting stevedore 1.4.0 via cliff 1.7.0 and swift choking on it as expected04:50
lifelessswift chokes on stevedore 1.4?04:52
lifelessdtroyer: in 176113?04:52
dtroyerjuno swift does04:52
dtroyermaybe it's the celio middleware doing it?04:53
dtroyerbut that's not where it gets pulled in incorectly04:54
dtroyerjust the first thing to notice04:54", line 640, in find_egg_entry_point04:54
lifeless2015-04-22 03:44:32.983 |     pkg_resources.require(self.spec)04:54
lifelessthats the thing triggering the whinge04:54
lifelessso paste-deploy04:54
lifelessfrom File "/opt/stack/old/swift/swift/common/", line 316, in loadapp04:55
lifeless2015-04-22 03:44:32.983 |     ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)04:55
dtroyer is where it is being updated past the juno cap04:55
lifelessI wonder why it doesn't barf in regular juno tempest04:55
lifelesssince we have 1.4 there too04:56
*** asselin_ has quit IRC04:56
*** sarob has joined #openstack-infra04:56
dtroyerno idea.  I haven't duplicated this locally yet in plain devstack runs04:56
*** deepakcs has joined #openstack-infra04:57
*** ddieterly has joined #openstack-infra04:57
dtroyerbut further down it is downgraded to 1.3.004:58
dtroyerit's all a crapshoot04:58
lifelessanything that installs a git tree with a modified requirements04:59
lifelesswill force stuff into alignment04:59
lifelessas long as it has a direct reference to $stuff04:59
lifelessanything with no direct reference will only be aligned if there is something intermediate with a versioned cap that is on pypi05:00
lifelessthis is issue 268705:00
lifelessI might poke at that tomorrow05:00
dtroyerI don't know by number05:00
lifelessits conceptually straight forward to at least make it AND things together05:01
dtroyerI just need to get the venv-for-everyone bit finished and let the distros sort it out since they have a depsolver to work with ;)05:01
*** sarob has quit IRC05:01
lifelessdtroyer: I hope to have a solver in pip before vancouver05:01
*** patrickeast has quit IRC05:01
*** ddieterly has quit IRC05:02
dtroyerthen you'll be showered in beer I'm sure05:02
lifelesssheese, I can't drink beer.05:02
lifelessCider or bubbly wine plox :)05:02
dtroyerok, midnight just slid by my place now too on its journey west05:04
*** prad has joined #openstack-infra05:04
*** ibiris is now known as ibiris_away05:06
*** shashankhegde has joined #openstack-infra05:09
*** BharatK has joined #openstack-infra05:09
*** teran has quit IRC05:11
*** sks has joined #openstack-infra05:15
*** fawadkhaliq has joined #openstack-infra05:16
*** Hal has joined #openstack-infra05:19
*** Hal is now known as Guest2439605:20
*** emagana has joined #openstack-infra05:20
*** annegentle has joined #openstack-infra05:24
*** agentle_ has joined #openstack-infra05:24
*** agentle_ has quit IRC05:28
*** annegentle has quit IRC05:29
*** yamahata has joined #openstack-infra05:30
*** coolsvap|afk is now known as coolsvap05:31
*** sdake_ has quit IRC05:42
*** tsg has joined #openstack-infra05:45
*** achanda has joined #openstack-infra05:49
*** Krinkle is now known as Krinkle|detached05:49
*** mrmartin has quit IRC05:49
*** dannywilson has joined #openstack-infra05:50
*** hdd has joined #openstack-infra05:51
*** otter768 has joined #openstack-infra05:58
*** tsg has quit IRC05:58
*** ddieterly has joined #openstack-infra05:59
*** tsg has joined #openstack-infra05:59
*** sushilkm has joined #openstack-infra05:59
*** sushilkm has left #openstack-infra05:59
*** otter768 has quit IRC06:03
*** ddieterly has quit IRC06:03
*** mrunge_ has joined #openstack-infra06:03
*** mrunge_ has quit IRC06:04
*** yolanda has joined #openstack-infra06:08
*** markvoelker has quit IRC06:10
*** mrunge has joined #openstack-infra06:11
*** tnovacik has joined #openstack-infra06:11
*** tsg has quit IRC06:14
*** ildikov has quit IRC06:15
*** Ala has joined #openstack-infra06:17
*** achanda has quit IRC06:20
*** yolanda has quit IRC06:20
*** yolanda has joined #openstack-infra06:20
*** achanda has joined #openstack-infra06:21
*** agentle_ has joined #openstack-infra06:24
*** annegentle has joined #openstack-infra06:25
*** davideagnello has joined #openstack-infra06:26
*** fedexo has joined #openstack-infra06:26
*** agentle_ has quit IRC06:29
*** isviridov is now known as isviridov_away06:29
*** annegentle has quit IRC06:29
*** woodster_ has quit IRC06:30
*** HeOS has quit IRC06:31
*** davideagnello has quit IRC06:31
*** stevemar has quit IRC06:31
*** sdake has joined #openstack-infra06:34
*** mahito has quit IRC06:38
*** vlaza has joined #openstack-infra06:39
*** Guest24396 has quit IRC06:40
*** markvoelker has joined #openstack-infra06:40
*** _nadya_ has joined #openstack-infra06:43
*** markvoelker has quit IRC06:45
*** sarob has joined #openstack-infra06:45
*** _nadya_ has quit IRC06:48
*** sarob has quit IRC06:49
*** kaisers has quit IRC06:50
*** shashankhegde has quit IRC06:51
*** ptoohill has quit IRC06:53
*** emagana has quit IRC06:53
*** MaxV has joined #openstack-infra06:56
*** dtantsur|afk is now known as dtantsur06:56
*** ptoohill has joined #openstack-infra06:57
*** luqas has joined #openstack-infra06:57
*** luqas has quit IRC06:59
*** ddieterly has joined #openstack-infra06:59
yamamotozuul gurus around?07:00
*** mrmartin has joined #openstack-infra07:02
*** kaisers has joined #openstack-infra07:03
*** ddieterly has quit IRC07:04
*** shardy has joined #openstack-infra07:04
*** mrodden has quit IRC07:06
*** mrodden has joined #openstack-infra07:07
*** funzo has joined #openstack-infra07:09
*** weshay has joined #openstack-infra07:10
*** kaisers has quit IRC07:11
*** jyuso1 has joined #openstack-infra07:12
*** scheuran has joined #openstack-infra07:12
*** doude has joined #openstack-infra07:12
*** coolsvap is now known as coolsvap|afk07:13
*** luqas has joined #openstack-infra07:14
*** luqas has quit IRC07:15
*** luqas has joined #openstack-infra07:16
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Neutron QoS Meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Monasca Meeting details
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add OpenStack Community Infrastructure Meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add translation test meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Install guide meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Networking Gudie update hangout
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add the Win the Enterprise meetings
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Akanda Team Meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Neutron Service Chaining Meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add log WG meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add OpenStackClient Team Meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Cue team meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Chef Cookbook meeting.
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add magnetoDB Team Meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add DefCore Progress Update Meeting.
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add Community Office hours.
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Alternate Sahara meeting
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Update Neutron VPNaaS Meeting
*** mrodden has quit IRC07:20
*** mrodden has joined #openstack-infra07:21
openstackgerritMerged openstack-infra/shade: Invalidate user cache on user create
*** MaxV has quit IRC07:23
*** sergsh has joined #openstack-infra07:25
*** agentle_ has joined #openstack-infra07:25
*** annegentle has joined #openstack-infra07:25
*** kaisers has joined #openstack-infra07:26
*** achanda has quit IRC07:29
*** agentle_ has quit IRC07:29
*** liusheng has joined #openstack-infra07:30
*** annegentle has quit IRC07:31
*** achanda_ has joined #openstack-infra07:32
*** Longgeek has joined #openstack-infra07:35
*** sdake has quit IRC07:35
*** luqas has quit IRC07:36
*** hashar has joined #openstack-infra07:36
openstackgerritFabien Boucher proposed openstack-infra/puppet-openstackci: Add generic zuul manifests
*** kaisers has quit IRC07:39
*** markvoelker has joined #openstack-infra07:41
*** arxcruz has joined #openstack-infra07:41
*** luqas has joined #openstack-infra07:42
*** yolanda has quit IRC07:43
*** jcoufal has joined #openstack-infra07:43
nibalizerfungi: anteaya good ramp up projects for an opw intern might be: Run stackalytics in infra: and Run livegrep in infra:
*** btully has quit IRC07:44
*** andreykurilin__ has joined #openstack-infra07:44
*** fedexo has quit IRC07:44
*** achanda_ has quit IRC07:45
*** yolanda has joined #openstack-infra07:46
*** markvoelker has quit IRC07:46
jeblairpleia2, clarkb: ^07:46
jeblairnibalizer: great ideas!07:46
*** sdake has joined #openstack-infra07:47
*** jistr has joined #openstack-infra07:50
openstackgerritMichal Vanco proposed openstack-infra/jenkins-job-builder: Support for Xvfb plugin
*** ala_ has joined #openstack-infra07:53
*** Ala has quit IRC07:54
*** kaisers has joined #openstack-infra07:55
*** yolanda has quit IRC07:55
*** MaxV has joined #openstack-infra07:56
*** otter768 has joined #openstack-infra07:59
*** sdake has quit IRC08:00
*** ddieterly has joined #openstack-infra08:00
openstackgerritJiri Stransky proposed openstack-infra/project-config: Move tripleo-puppet-elements to tripleo-core
openstackgerritJiri Stransky proposed openstack-infra/project-config: Add tripleo-core to puppet-tripleo
*** yfried__ is now known as yfried|afk08:02
*** otter768 has quit IRC08:04
*** yamahata has quit IRC08:04
*** alexpilotti has joined #openstack-infra08:04
*** ddieterly has quit IRC08:05
*** isviridov_away is now known as isviridov08:09
*** ildikov has joined #openstack-infra08:10
*** ibiris_away is now known as ibiris08:10
*** Hal has joined #openstack-infra08:13
*** teran has joined #openstack-infra08:13
*** ildikov has quit IRC08:13
*** Hal is now known as Guest9611608:14
*** Guest96116 has quit IRC08:18
*** ajo has joined #openstack-infra08:18
openstackgerritAngus Salkeld proposed openstack/requirements: Exclude novaclient 2.24.0 as it breaks heat's gate
*** ildikov has joined #openstack-infra08:19
*** sandywalsh has quit IRC08:24
*** jamespage has quit IRC08:24
*** jamespage has joined #openstack-infra08:24
*** sandywalsh has joined #openstack-infra08:24
*** jamespage has quit IRC08:24
*** jamespage has joined #openstack-infra08:24
*** derekh has joined #openstack-infra08:25
*** yfried|afk has quit IRC08:26
*** yfried|afk has joined #openstack-infra08:26
*** __mimir has joined #openstack-infra08:26
*** annegentle has joined #openstack-infra08:26
*** _nadya_ has joined #openstack-infra08:26
jyuso1Hi infra.Intel PCI/NGFW/SRIOV/Hardware-Meters CI will be offline this week.08:28
jyuso1We have to tidy the server room.Our CI servers and log-server will be shutdown until next week.We¡¯ll resume those servers ASAP.Thanks:)08:29
*** krtaylor has quit IRC08:29
openstackgerritYanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag
*** annegentle has quit IRC08:31
nibalizerjeblair: so this is the thing that puppetlabs has 'vmpooler' kind of like nodepool, especially nodepool post zuulv3 I think08:32
*** smyers has quit IRC08:32
*** prad has quit IRC08:32
*** med_ has quit IRC08:32
*** hamzy has quit IRC08:32
*** hamzy has joined #openstack-infra08:34
*** prad has joined #openstack-infra08:34
*** krtaylor has joined #openstack-infra08:34
*** juggler has joined #openstack-infra08:35
*** med_ has joined #openstack-infra08:35
*** med_ has quit IRC08:36
*** med_ has joined #openstack-infra08:36
*** fawadkhaliq has quit IRC08:36
*** HeOS has joined #openstack-infra08:36
*** smyers has joined #openstack-infra08:37
*** juggler_ has quit IRC08:37
jeblairnibalizer: yay nodepool in ruby :)08:38
*** coolsvap|afk is now known as coolsvap08:40
*** juggler has quit IRC08:41
jeblairhrm, i get the following from etherpad:08:42
jeblairError: Attempt to load undefined module. in (line 1)08:42
*** markvoelker has joined #openstack-infra08:42
*** yolanda has joined #openstack-infra08:42
openstackgerritYanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag
jeblairthat might have been a local error.  weird.  i don't know if it was a network problem or if restarting my browser fixed it08:44
*** dannywilson has quit IRC08:45
*** dannywilson has joined #openstack-infra08:46
*** ssam2 has joined #openstack-infra08:46
*** markvoelker has quit IRC08:47
ttxjeblair: you up ?08:48
*** andreykurilin__ has quit IRC08:49
ttxjeblair: We could use some ninja-merging on
ttxsounds preferable to disabling grenade on stable/juno temporarily.08:50
*** dannywilson has quit IRC08:50
ttx(though the latter might prove necessary if the house of cards continues to fall)08:50
*** andreykurilin__ has joined #openstack-infra08:50
*** coolsvap is now known as coolsvap|afk08:51
*** devvesa has joined #openstack-infra08:52
*** prad has quit IRC08:53
*** jklare has quit IRC08:53
*** jklare has joined #openstack-infra08:55
*** _nadya_ has quit IRC08:55
*** zz_johnthetubagu is now known as johnthetubaguy08:55
*** devvesa has quit IRC08:57
*** GnomeKing has joined #openstack-infra08:57
*** devvesa has joined #openstack-infra08:57
*** sputnik13 has joined #openstack-infra08:58
*** GnomeKing is now known as Guest2888208:58
*** teran has quit IRC08:59
*** prad has joined #openstack-infra08:59
*** pelix has joined #openstack-infra09:01
*** ihrachyshka has joined #openstack-infra09:01
*** sputnik13 has quit IRC09:02
*** juggler has joined #openstack-infra09:05
*** fawadkhaliq has joined #openstack-infra09:07
*** jlanoux has joined #openstack-infra09:11
*** Hal has joined #openstack-infra09:11
*** __mimir has quit IRC09:11
*** Hal is now known as Guest536309:11
*** ala_ has quit IRC09:12
*** dimtruck is now known as zz_dimtruck09:14
*** Kennan2 has joined #openstack-infra09:19
*** Kennan has quit IRC09:19
*** afazekas_ has joined #openstack-infra09:21
*** adiantum has joined #openstack-infra09:22
*** __mimir has joined #openstack-infra09:25
*** agentle_ has joined #openstack-infra09:27
*** annegentle has joined #openstack-infra09:27
*** dtantsur is now known as dtantsur|brb09:29
*** e0ne has joined #openstack-infra09:30
*** tnovacik has quit IRC09:30
*** annegentle has quit IRC09:31
*** agentle_ has quit IRC09:32
*** ala_ has joined #openstack-infra09:34
jeblairttx: what does that error mean?09:34
*** yamamoto has quit IRC09:34
jeblairttx: (or on other words, why would force-merging that change fix the error reported in that change?)09:36
*** kaisers has quit IRC09:40
*** ajo has quit IRC09:41
*** markvoelker has joined #openstack-infra09:43
*** kaisers has joined #openstack-infra09:43
*** dguitarbite has joined #openstack-infra09:45
*** ajo has joined #openstack-infra09:46
*** shardy_ has joined #openstack-infra09:46
*** markvoelker has quit IRC09:47
*** shardy has quit IRC09:48
*** shardy_ has quit IRC09:52
*** shardy has joined #openstack-infra09:52
*** hashar is now known as hasharLunch09:54
*** ajo has quit IRC09:58
*** jogo has quit IRC10:01
*** palar has joined #openstack-infra10:01
*** __mimir has quit IRC10:02
*** ddieterly has joined #openstack-infra10:02
*** andreykurilin__ has quit IRC10:04
*** ssam2 has quit IRC10:04
*** jogo has joined #openstack-infra10:06
*** ddieterly has quit IRC10:07
openstackgerritSean Dague proposed openstack/requirements: uncap stevedore entirely
*** teran has joined #openstack-infra10:12
*** teran_ has joined #openstack-infra10:14
*** ajo has joined #openstack-infra10:15
*** teran__ has joined #openstack-infra10:15
*** teran__ has quit IRC10:16
*** teran__ has joined #openstack-infra10:16
*** teran has quit IRC10:16
*** teran_ has quit IRC10:18
*** ssam2 has joined #openstack-infra10:19
openstackgerritSean Dague proposed openstack/requirements: raise stevedore requirements
sdaguettx: I think if both of those get approved, things unwedge10:20
jeblairsdague: can they go through normal approval?10:22
sdagueI *think* so?10:22
sdaguehonestly, there is a lot of cascade fail here10:22
*** sarob has joined #openstack-infra10:23
sdagueand it takes an hour to get back results, so no idea10:23
sdaguealso, I've been awake for 20 minutes, so take all statements with "not fully awake yet"10:23
*** yfried|afk is now known as yfried__10:24
sdagueso, in the list of things we did not anticipate in the requirements cap path10:25
sdague1) it would require a giant flush like we had yesterday, which mean a bunch of bugs10:25
sdague2) we're actually still in a 2 release coupling for upgrade from juno -> liberty, because the grenade juno -> kilo, kilo -> liberty config is not merged10:26
*** ssam2_ has joined #openstack-infra10:26
sdaguewhich code wise isn't a problem10:26
sdaguebut with narrow requirements, the odds of getting deadlocked there is a high10:27
*** Guest28882 is now known as BobBall10:27
sdagueI also fully expect icehouse to just completely fail on for unrelated reasons10:27
sdaguegiven that I don't know if anyone has tried an icehouse patch since the flush10:27
*** nfedotov has joined #openstack-infra10:27
*** deepakcs has quit IRC10:28
*** annegentle has joined #openstack-infra10:28
*** ssam2 has quit IRC10:28
*** sarob has quit IRC10:28
sdagueso... the problem is we can't land liberty patches right now until we fix icehouse10:28
sdaguewhich, does not seem to be the kind of situation we want to be in10:28
sdagueand that a force merge of will fix10:28
*** annegentle has quit IRC10:33
*** yfried__ is now known as yfried|afk10:34
*** jlanoux has quit IRC10:35
*** yamamoto has joined #openstack-infra10:35
*** _nadya_ has joined #openstack-infra10:40
*** yamamoto has quit IRC10:40
ttxjeblair: the fix in itself fixes stable/juno, and therefore by extension stable/kilo and master. I'd rather unblock stuff now rather than lose a few more hours fixing stable/icehouse first10:41
ttxthe grenade fail is not an upgarde fail, it's a "stable/icehouse is wedged" fail10:41
ttxso +1 to force merging to break the cascade and at least get kilo & master unblocked10:43
ttxThen we can unblock juno by fixing icehouse.10:43
*** markvoelker has joined #openstack-infra10:43
ttxalternative would be to remove grenade tests from juno, which sounds (1) worse and (2) farther away10:45
*** hyakuhei has joined #openstack-infra10:46
ttxsdague: or we can get lucky and have 176243 miraculously fix icehouse.10:46
sdaguettx: yeh, no idea, are there other icehouse recent runs somewhere?10:46
ttxbut I fear it won't be enough, like you said10:46
*** yamamoto has joined #openstack-infra10:46
sdagueit fixes the tactical issue that I saw on a failed juno run10:46
ttxsdague: let's wait for the test results on that one, looks like we won't get anyone to ninja-merge it in the next hour anyway10:47
hyakuheiHey ttx, can you take a quick look at please - just a typo fix.10:48
*** markvoelker has quit IRC10:48
ttxhyakuhei: approved10:49
*** ssam2_ has quit IRC10:49
hyakuheiThanks :)10:49
*** e0ne is now known as e0ne_10:51
ttxfungi: you around ?10:52
ttxbah / me lunches.10:52
*** yfried|afk is now known as yfried__10:53
*** ssam2 has joined #openstack-infra10:53
*** isviridov is now known as isviridov_away10:53
*** juggler_ has joined #openstack-infra10:54
*** e0ne_ is now known as e0ne10:55
*** zhiwei has quit IRC10:56
*** dtantsur|brb is now known as dtantsur10:58
*** juggler has quit IRC10:58
*** hyakuhei has quit IRC10:58
*** erlon has joined #openstack-infra10:59
*** deepakcs has joined #openstack-infra11:01
*** ildikov has quit IRC11:01
openstackgerritSean Dague proposed openstack/requirements: add pycparser tracking to what-broke
*** yfried__ is now known as yfried|afk11:04
*** marcusvrn has joined #openstack-infra11:04
*** _nadya_ has quit IRC11:05
*** Kennan2 has quit IRC11:08
BharatKsdague: ping11:08
BharatKsdague: As you suggested, created a new for glusterfs, please review the patch
*** ildikov has joined #openstack-infra11:16
*** jlanoux has joined #openstack-infra11:18
*** afazekas_ has quit IRC11:18
*** Kennan has joined #openstack-infra11:23
*** doug-fish has joined #openstack-infra11:23
*** yfried|afk is now known as yfried__11:24
*** hasharLunch is now known as hashar11:26
fungittx: around now--what's up?11:26
*** e0ne is now known as e0ne_11:27
*** agentle_ has joined #openstack-infra11:28
*** shardy has quit IRC11:28
*** annegentle has joined #openstack-infra11:28
*** agentle_ has quit IRC11:32
*** afazekas_ has joined #openstack-infra11:34
*** annegentle has quit IRC11:34
ttxfungi: we could unblock gate using a ninjafix11:34
ttxwell, hmm, might not be necessary after all11:34
ttxcatching up11:34
fungilooks like 176113 passed tests11:35
* fungi too is catching up11:35
ttxyeah, so by miracle it may fix things11:35
funginot yet caffeinated11:35
ttxlet's see how it goes11:35
fungiestimated half an hour to merge11:36
*** woodster_ has joined #openstack-infra11:37
*** yfried__ is now known as yfried|afk11:37
*** e0ne_ has quit IRC11:38
*** smarcet has joined #openstack-infra11:38
*** esker has quit IRC11:38
*** bknudson has quit IRC11:40
*** ldnunes has joined #openstack-infra11:43
deepakcssdague, would appreciate if you can help review - will help cinder-glusterfs ci to turn back green :)11:43
*** wuhg has joined #openstack-infra11:43
*** mpaolino has joined #openstack-infra11:44
sdaguedeepakcs: please stop pinging me, I've been unwinding gate wedges, and this is very far down my priority list11:44
*** markvoelker has joined #openstack-infra11:44
*** sslypushenko has joined #openstack-infra11:45
deepakcssdague, sure, i had asked others, but they said you need to look at it!11:45
*** markvoelker has quit IRC11:45
*** markvoelker has joined #openstack-infra11:45
*** mugsie has quit IRC11:46
*** mugsie has joined #openstack-infra11:46
*** yfried|afk is now known as yfried__11:46
*** baoli has joined #openstack-infra11:53
*** yolanda has quit IRC11:55
fungideepakcs: it will probably need to wait until the kilo release. official project teams are all very busy trying to get the release completed for the next couple weeks11:55
*** ildikov has quit IRC11:56
*** yfried__ is now known as yfried|afk11:56
deepakcsfungi, thanks for the gracious answer, that helps :)11:57
*** e0ne has joined #openstack-infra11:58
*** otter768 has joined #openstack-infra12:00
*** ddieterly has joined #openstack-infra12:03
*** ayoung has quit IRC12:03
*** mmedvede has quit IRC12:04
*** otter768 has quit IRC12:05
sdaguefungi: and neutron rando fail12:05
*** smarcet has quit IRC12:05
fungisdague: which bug? i missed it in the scrollback i think12:06
sdagueso please ninja merge and
sdaguethe requirements fixes12:06
fungiuncap stevedore in icehouse, and raise the stevedore cap in juno12:07
fungilooks like 176243 passes tests but 176113 does not12:07
*** luqas has quit IRC12:07
sdague176113 passed tests12:07
*** ddieterly has quit IRC12:08
fungiahh, yep. so it did12:08
*** luqas has joined #openstack-infra12:08
fungii was looking at a partly loaded page. need to finish getting up and get to my normal computer12:08
*** ildikov has joined #openstack-infra12:09
fungitoo many channels full of fire. proof i should not look at irc before getting out of bed ;)12:09
*** deepakcs has quit IRC12:09
*** yolanda has joined #openstack-infra12:09
sdaguewell, if requirements-core had force push on requirements, we wouldn't have to wake you :)12:10
*** BharatK has quit IRC12:10
*** vladiskuz has quit IRC12:11
*** sarob has joined #openstack-infra12:12
*** dprince has joined #openstack-infra12:13
*** fawadkhaliq has quit IRC12:13
fungii don't think anyone should ever force-push a chance which can pass tests as long as the systems to run those jobs are up and working. they're enqueued at the head of the gate pipeline now12:13
sdaguefungi: so they can die on another rando fail and we can keep doing this for hours?12:14
sdaguethose are provably correct changes at this point12:14
*** mmedvede has joined #openstack-infra12:14
*** tnovacik has joined #openstack-infra12:14
*** jaypipes has joined #openstack-infra12:15
sdagueand this continues to block a most projects in OpenStack from getting stuff done12:16
fungisounds like we should probably stop running integration tests in the gate for now12:16
fungiif none of them can complete reliably in a reasonable amount of time12:16
fungii'll get the job change ready12:16
*** ociuhandu has joined #openstack-infra12:17
fungiwe can switch to run them in check only. should get things moving a whole lot faster and we can get this back on track sooner12:17
sdagueI'm confused.12:17
sdagueWe've seen these changes pass the relevant tests12:17
sdaguewhen they failed it was unrelated12:18
fungithey passed in check, right? we run them in check12:18
*** tsg has joined #openstack-infra12:18
fungiit sounds like most of the delay is getting integration jobs through the gate since they're unreliable12:18
*** sarob has quit IRC12:18
*** tiswanso has joined #openstack-infra12:18
*** cdent has joined #openstack-infra12:18
*** rfolco has joined #openstack-infra12:19
fungijust trying to figure out how to speed this up so we don't need to bypass gating completely instead12:19
fungisome testing is better than none, after all12:19
sdaguewe do it by relying on the fact that there are smart humans12:19
sdaguethat know better than machines some times12:19
ekarlsowhat's the recommended way for testing / mocking clients atm ?12:20
*** baoli has quit IRC12:21
fungiokay, so we do still want to run integration testing on other changes in the gate pipeline, just not these changes because we're pretty sure they shouldn't have adverse interactions with each other beyond maybe grenade, since they're patches to icehouse and juno?12:21
sdaguethat's been my frustration with this argument that we get into for stuff like this, there are times when smart humans actually have figured out the fix, and have data to support it, but are powerless to address that fix without fixing everything else first12:21
jeblairsdague, fungi: all i care about is that we stop wedging things.  is someone working on figuring out how we got into this state and preventing it in the future?  if so, i'm okay force-merging changes that unwedge us.12:21
ttxwell, we basically know that those two fixes get us closer to a world of good, and have to wait 6 hours to get them in12:22
sdaguefungi: not only are we pretty sure, we tested master patches Depends-On these changes, and saw them pass12:22
sdaguethat's about as solid data as I can give you12:23
ttx6 hours we can't really afford to lose at this point. I know they directly translate into 6 hours less of weekend time at this point12:23
fungii think those in favor of capping requirements did not realize it was going to result in a more time-consuming puzzle to solve than the weekly random breakages we could run down in an hour or two. but undoing all this now would probably be even more work12:23
*** dhritishikhar has joined #openstack-infra12:23
sdaguefungi: yes, there were many dragons12:23
fungiso i agree unwedging things at this point probably means pressing forward12:24
fungiand hoping we're not making it worse12:24
ttxpushing those at this point just means winning back 1+ hour12:24
ttxsince we know they will pass at some point12:24
ttxand we keep on pushing them12:25
ttxIf everything else was not blocked we could afford losing 1+ hour12:25
fungii'm fine manually merging them now if it gets everyone closer to having everything working again sooner12:25
sdaguejeblair: in honesty, no one has figured out a system that can prevent wedging given that python doesn't have a real package manager12:25
*** dhritishikhar has quit IRC12:26
ttxseriously, we can't lose another day at this point. This is the most fucked-up release period ever12:26
*** dhritishikhar has joined #openstack-infra12:27
ttxit will directly translate in pain and lack of quality, so every hour count12:27
fungiso anyway, it sounds like jeblair agrees that getting us un-wedged sooner even if it means officially bypassing gating is an acceptable solution12:27
fungii'll go ahead and do that for these two requirements changes12:28
*** rlandy has joined #openstack-infra12:29
*** annegentle has joined #openstack-infra12:29
*** arielb has joined #openstack-infra12:29
*** mwagner_lap has quit IRC12:31
openstackgerritMerged openstack/requirements: uncap stevedore entirely
*** agentle_ has joined #openstack-infra12:31
jeblairyeah.  my two criteria are that someone believes that it will really fix the problem and any current failing tests don't look like new regressions.  i'm happy with sdague's evaluation there12:31
openstackgerritMerged openstack/requirements: raise stevedore requirements
jeblairthe other part is that we figure out how to prevent something similar in the future.  i expect we may need to do that at the summit12:31
sdaguejeblair: honestly, given all the shortcomings of pip, I think that even in the medium term the solution includes human override by requirements core12:32
jeblairbecause clearly this is a really complex issue and i doubt we can do that right now12:32
sdaguebecause I think it's probably a year before we have a system robust enough to address this, even if we start now12:32
jeblairsdague: that's not really acceptable.  we need to understand what's causing the wedging.  this is a new behavior -- what's causing it?  is it because we're doing grenade testing now?12:33
sdagueit's because we've got 200 pip trees12:33
ttxjeblair: I think the new element in the equation is the capping.12:33
sdagueit's not new behavior, it's the same behavior we've always seen12:33
ttxAnd its ability to leak12:34
ttxwhich grenade has the ability to magnify12:34
* dhellmann catches up12:34
jeblairokay, so it sounds like our capping strategy isn't sufficient.  perhaps it needs complete tranitive dependencies?12:34
fungiwell, a big chunk of it is that pip install multiple times in the same environment produces deterministic but uncontrolled behavior, and that is how devstack operates right now12:35
sdagueright, so things that made this worse12:35
jeblairfungi: right, but complete fixed transitive deps works around that, right?12:35
*** dhritishikhar_ has joined #openstack-infra12:36
sdague1) installing more things from pypi instead of git trees12:36
dhellmannfungi: right, so we talked yesterday about making devstack build a list of requirements for the things it is going to install, and then install them all at once12:36
fungijeblair: yes12:36
*** dhritishikhar has quit IRC12:36
sdagueso the fact that we make projects use installed software, means we gave up control of the dependencies of that installed software12:36
sdaguewe no longer control python-barbicanclient dependencies12:37
sdagueat least not point in time12:37
sdaguewe have eventual consistency, which is not good enough12:37
*** Prophidys1 has left #openstack-infra12:37
jeblairsdague: ah, right because we aren't overwriting its requirements.txt file?12:37
dhellmannsdague: true, but if we feed pip 2 requirements that conflicting dependencies *at the same time* it will correctly report an error, right?12:37
dhellmann^that^that have12:38
sdaguedhellmann: sure, but the problem here isn't that, it's the transitive unwind12:38
EmilienMgood morning, can we get a chance to have a review on project-config/tripleo/puppet? (small patch)12:38
dhellmannsdague: right, my point was that a change to devstack might help with that12:38
sdaguejeblair: and the reality is, every time anyone thinks they've addressed 1 problem here, we uncover a couple of other dragons12:38
sdagueand then the energy to solve the long game gets lost in tactical unwind firefights12:39
sdaguewhich is why I believe human override is an important part of the medium term game12:39
jeblairsdague: right.  so have we learned enough to have a useful conversation at summit?12:39
sdaguedefine "useful conversation"12:40
sdaguewe've learned a lot12:40
*** claudiub has joined #openstack-infra12:40
*** dhritishikhar_ has quit IRC12:40
sdagueI don't think anyone has a solution sorted12:40
*** e0ne is now known as e0ne_12:41
jeblairsdague: i think we need to revsit the things that got us here -- what are we trying to do by installing from git, installing from pypi, capping requirements, etc.  i think we need to step back and consider all of those things and design a solution that takes all those into account rather than patching things one at a time.12:41
*** gordc has joined #openstack-infra12:41
sdaguejeblair: sure.12:41
sdaguethere are some attempted ML threads on that right now that lifeless started12:42
*** _nadya_ has joined #openstack-infra12:42
jeblairsdague: i'm supportive of human overrides to get us to the release and to a conversation about that.  i would really like to have that conversation before we start agreeing on things like "human overrides for a year"12:42
sdaguehonestly, it's probably a better ML conversation than face to face one, because there are *so* many details12:42
sdagueand if you lose one detail somewhere, the whole thing crumbles12:42
*** bknudson has joined #openstack-infra12:43
sdagueand we've convinced ourselves a few times that we had a new plan, and a detail was missed, and.... we get *yesterday*12:43
fungiand now i have a beatles song stuck in my head12:44
jeblairsdague: i think perhaps we didn't record that plan very well12:44
jeblairsdague: in particular, i was suprised that the requirements were capped _before_ the release12:44
sdaguejeblair: could be, I'll be honest, I mostly took a pass on this last go around because I was too burnt out on the problem space and the fire fighting12:44
ttxjeblair: agree that capping oslo libs on master was not such a great idea12:45
fungiagreed. the earlier discussions were that we'd cut release branches from master, then immediately post-release introduce requirements caps to protect stable12:45
ttxIt's because the general thinking was... stable/kilo requirements should only be cut when all RCs are in12:46
*** dannywilson has joined #openstack-infra12:46
ttxBut maybe it's the other way around12:46
*** kgiusti has joined #openstack-infra12:46
*** e0ne_ is now known as e0ne12:46
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Set EnablePacemaker for the HA job
fungiwhich would have allowed keeping momentum up through the release and then dealing with any fallout when there was less of a time crunch12:47
ttxfungi: a bit uncertain on the zuul status right now, why is it testing  176113 again12:47
*** yfried|afk is now known as yfried__12:48
fungittx: yes, i'm performing surgery to get those excised from the gate and requeue the tempest fix which depended on them12:48
* ttx holds on approving RC2 changes until surgery is complete12:49
*** radez_g0n3 is now known as radez12:50
*** dannywilson has quit IRC12:50
*** tiswanso has quit IRC12:50
dhellmannwe capped on master to avoid having the feature freeze in oslo last for an excessive period, since we freeze before the other projects and can only really thaw after their stable branches are cut12:50
fungiright, the pushback on waiting for the release to cap was that clients and libraries wanted to go ahead and start developing liberty features rather than being unable to introduce breaking changes prior to kilo release12:52
*** bswartz has quit IRC12:53
fungithough in retrospect, we could have added branches to all of them and capped them on stable/kilo nee proposed/kilo12:53
fungibut that's where the mixed proposed vs stable branch fallback logic issue came up12:54
*** amitgandhinz has joined #openstack-infra12:54
*** AJaeger has joined #openstack-infra12:55
*** dhritishikhar_ has joined #openstack-infra12:56
*** pc_m has joined #openstack-infra12:56
*** spzala has joined #openstack-infra12:56
*** baoli has joined #openstack-infra12:57
sdagueso, realistically, I think caps in libraries are part of the problems12:57
sdagueglobal-requirements works ok for a set of things that release all at once12:57
*** yfried__ is now known as yfried|afk12:58
sdaguebut it's having really weird behavior on things that don't, like these libraries12:58
*** eharney has joined #openstack-infra12:58
sdagueespecially once you think about what needs to be released for everything to have the same oslo.config definition12:59
*** yfried|afk is now known as yfried__12:59
fungittx: okay, the tempest fix is at the top of the gate now and zuul seems to be wrapping up processing its events/results queues now13:02
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add a ceph job
*** sergsh has quit IRC13:03
dhellmannyeah, I'm ok with not capping the dependencies within the libraries, but we still want the apps to have theirs capped at a point that lets us create stable branches of the libs and then move on with liberty work13:03
jeblairsdague, dhellmann: well regardless of whether we have a session at the summit about it, i look forward to talking to the two of you about it.  i think i need that in order to get on the same page.  you have learned a lot that i need to learn from you.  :)13:03
*** ociuhandu has quit IRC13:03
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Set EnablePacemaker for the HA job
jeblairsdague, ttx: if grenade is exacerbating the situation, can we make grenade non-voting on requirements stable branches?13:03
fungisdague: "libraries" is a fuzzy concept though... if you have library A which depends on B and it's incompatible with B>=2 then you need to record that somewhere such that you can run unit tests for A13:04
*** dhritishikhar_ has quit IRC13:04
sdaguejeblair: so, in this case, grenade wasn't really causing any issues directly13:04
sdagueit was just upleveling fails13:04
*** ildikov has quit IRC13:04
sdagueso stable/juno being broken, broke master patches13:04
sdaguethe real issue was ./ on stable/juno didn't work13:05
sdagueand the fix for that was blocked on ./ on stable/icehouse not working13:05
jeblairsdague: yeah, that's why i'm thinking that in the short/medium term that making it non-voting might be a reasonable compromise -- because we know it's a wedgeable situation so don't have the computer block human judgement there13:06
sdaguejeblair: yeh, or make grenade non-voting on non master in general13:06
fungianyway, i've come to realize that pip's current design assumes your requirements lists are not package-specific and that without modifying pip heavily to change that, the closest we can get to its intended behavior is to have job-specific (perhaps tox env specific) requirements lists rather than tying them to projects13:07
fungier, rather than tying them to packages i mean13:07
*** dkranz has joined #openstack-infra13:07
*** mattfarina has joined #openstack-infra13:07
sdaguefungi: yeh, that would be an option13:07
sdagueagain, if libraries were pretty wide in what they allowed, we could do all that upfront13:08
*** mrmartin has quit IRC13:10
sdaguejeblair: so if you think a summit session is the right format, I'm happy to participate13:12
*** afazekas_ has quit IRC13:13
*** yfried__ is now known as yfried|afk13:14
jeblairsdague: all i'm certain about at the moment is i have some things i need to learn from you and dhellmann  :)13:14
*** dhritishikhar has joined #openstack-infra13:15
sdaguehonestly, this might make for a reasonable sprint in the next cycle, because I feel like there is a lot to keep in ones head, but a few folks, and a few days, and a white board might go far13:15
jeblairsdague: yeah, that might be a good idea13:17
*** samueldmq has joined #openstack-infra13:17
*** ociuhandu has joined #openstack-infra13:18
*** yamahata has joined #openstack-infra13:19
*** ildikov has joined #openstack-infra13:19
*** agentle_ has quit IRC13:20
*** annegentle has quit IRC13:20
*** ddieterly has joined #openstack-infra13:21
EmilienMhow to you generally proceed to move some repos from X to Y (ie, from stackforge to openstack) ? My question is more about users.13:21
fungiusers move themselves?13:22
*** mriedem_away is now known as mriedem13:22
EmilienMfungi: yeah... but I'm sure we will have a lot of feedback from people having CI broken or I don't know13:22
fungioh, your users are consuming source code via git13:23
EmilienMI'm just wondering actually13:23
funginot packages13:23
EmilienMfungi: yes13:23
EmilienMfungi: RH does, but not sure about other users. Most of them are using git13:23
*** dhritishikhar has quit IRC13:23
EmilienMbefore moving everything to github/openstack I would like to figure out that one13:23
*** dustins has joined #openstack-infra13:23
fungiwell, on the github end it's pretty simple. we transfer the repos if possible and github creates redirects13:24
sdagueEmilienM: you probably should just send out a heads up with a 'get remote set-url'13:24
*** esker has joined #openstack-infra13:24
*** peristeri has joined #openstack-infra13:24
fungiEmilienM: but on the end we don't have a similar redirect service13:24
fungiso anyone consuming from there would need to change their remote urls (or we would need to design a server-side solution)13:25
*** bswartz has joined #openstack-infra13:25
EmilienMsdague, fungi: wfm13:25
marcusvrnhi guys, can some jenkins expert help me? I'm trying to trigger a job using zuul, but when jenkins tries to run the job on master, it stays as pending "There are no node with label '' ". If I run manually that job, it works...13:27
*** arielb has quit IRC13:27
EmilienMfungi: more a technical question - once I've communicated the information, how should I proceed to do the actual change ? An email to openstack-dev with [infra] tag as an official request?13:27
*** _nadya_ has quit IRC13:29
fungiEmilienM: a patch to openstack-infra/project-config and an entry at with the change url13:29
*** yfried|afk is now known as yfried__13:29
EmilienMfungi: awesome. Thx13:29
*** agentle_ has joined #openstack-infra13:29
*** annegentle has joined #openstack-infra13:29
*** stevemar has joined #openstack-infra13:29
fungiEmilienM: then we discuss project rename scheduling since it involves a nontrivial gerrit outage to make changes in its database and filesystem and rebuild search indexes13:29
fungiEmilienM: also we're postponing further rename maintenances until after the kilo release, just as a heads up13:30
EmilienMfungi: it makes sense13:30
jeblairfungi: this look good as a starting point?
fungijeblair: yep! that should do great13:31
*** agentle_ has quit IRC13:32
*** annegentle has quit IRC13:32
*** jpeeler has quit IRC13:32
*** annegentle has joined #openstack-infra13:32
*** ayoung has joined #openstack-infra13:33
openstackgerritSean Dague proposed openstack-infra/devstack-gate: add support for stable/kilo
*** andreykurilin__ has joined #openstack-infra13:33
openstackgerritSean Dague proposed openstack-infra/devstack-gate: add support for stable/kilo
sdaguejeblair: is it intentional that you can't have any other text on the Depends-On line?13:35
jeblairsdague: yes -- i think part of the idea was to try to be compatible with the syntax in the previous gerrit work13:37
*** stevemar has quit IRC13:37
jeblairsdague: but i mean if there's a compelling need to extend, i think we can talk about it13:38
sdaguejeblair: ok, that's fine. I mistakenly thought I could put a comment after it13:38
sdaguewith multiple Depends-On, it's nice to list what they do13:38
*** wenlock has joined #openstack-infra13:38
*** BharatK has joined #openstack-infra13:39
sdagueI think the current behavior is fine, I just wanted to confirm it was intentional13:39
*** palar has quit IRC13:39
openstackgerritIhar Hrachyshka proposed openstack-dev/hacking: Add support for flake8 off_by_default for optional checks
openstackgerritIhar Hrachyshka proposed openstack-dev/hacking: Add support for flake8 off_by_default for optional checks
openstackgerritIhar Hrachyshka proposed openstack-dev/hacking: Add support for flake8 off_by_default for optional checks
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Decrease PO files for manual imports
*** jpeeler has joined #openstack-infra13:44
*** jpeeler has quit IRC13:44
*** jpeeler has joined #openstack-infra13:44
*** ociuhandu has quit IRC13:44
*** sarob has joined #openstack-infra13:45
*** xyang1 has joined #openstack-infra13:45
*** ociuhandu has joined #openstack-infra13:46
*** e0ne is now known as e0ne_13:46
*** mwagner_lap has joined #openstack-infra13:48
*** mgedmin_ has joined #openstack-infra13:48
*** mgedmin_ has quit IRC13:48
*** radez is now known as radez_g0n313:49
*** sarob has quit IRC13:49
*** rlandy has quit IRC13:49
*** amotoki has joined #openstack-infra13:49
*** rlandy has joined #openstack-infra13:50
*** BobH has joined #openstack-infra13:55
*** tonytan4ever has joined #openstack-infra13:55
BobHGood morning all. I'm back to trying to get my 3rd party CI working after a bit of delay.13:56
*** e0ne_ has quit IRC13:56
BobHFirst issue following the instructions is that hiera-puppet is apparently already installed with puppet.13:57
*** annegentle has quit IRC13:57
*** johnthetubaguy is now known as zz_johnthetubagu13:57
BobHhowever, i do not see a hiera.yaml in /etc/puppet. Can I just simply copy the one in /etc?13:57
*** ddieterly has quit IRC13:58
*** jcoufal has quit IRC13:59
*** e0ne has joined #openstack-infra13:59
*** annegentle has joined #openstack-infra13:59
*** apmelton_away is now known as apmelton13:59
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Decrease PO files for Doc team imports
*** dhritishikhar has joined #openstack-infra14:01
*** dboik has joined #openstack-infra14:01
*** otter768 has joined #openstack-infra14:01
*** tqtran has joined #openstack-infra14:02
*** ala_ has quit IRC14:03
BobBallAny ideas why my zuul runs jobs against stable branches?  I thought the "branch: ^master.*$" restriction in the jobs stansa should ensure it only runs against master?14:04
*** ddieterly has joined #openstack-infra14:04
*** zhiwei has joined #openstack-infra14:05
fungiBobH: what are "the instructions" you're following?14:05
*** sigmavirus24_awa is now known as sigmavirus2414:05
*** dhritishikhar has quit IRC14:05
fungiBobBall: is that zuul running dsvm-tempest.* jobs against stable branches? do you have an example change it reported on?14:06
*** dhritishikhar has joined #openstack-infra14:06
*** otter768 has quit IRC14:06
BobBall,n,z so
*** btully has joined #openstack-infra14:06
BobBallIt's not running that job on it, no, but it's commenting14:06
BobBallwhich I don't understand.14:07
*** smarcet has joined #openstack-infra14:07
fungiBobBall: you told it not to run those jobs, so it didn't. sounds like you need to tell it to run in a pipeline which onlt triggers on changes to the branch(es) you want instead14:07
*** stevemar has joined #openstack-infra14:07
BobBallOK - so it's the pipeline definition I need to look at; thanks.14:07
smarcetfungi: hi, just want to continue discussion about openstackid resource server14:08
fungithe pipeline you're using is still triggering on all changes, but then if they're stable branch changes it has nothing to run for them, so successfully reports that it did so14:08
fungismarcet: you bet. just a sec and i'll finally have my coffee14:08
smarcetfungi: np :)14:08
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Set root disk == node disk for puppet jobs
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add a ceph job
*** erikmwilson is now known as Guest9565814:11
*** erikmwil_ has joined #openstack-infra14:11
*** Guest95658 has quit IRC14:11
*** erikmwil_ is now known as erikmwilson14:11
*** erikmwilson_ has joined #openstack-infra14:11
fungiahh, now i feel human14:12
fungismarcet: so you're writing a laravel-based resource api service from the ground up, with a similar design to openstackid but new codebase14:12
*** sks has quit IRC14:12
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Puppet: move from Stackforge to OpenStack
fungismarcet: and we're going to want to deploy that on a new server (or pair of servers, one for dev like we have for openstackid)14:13
smarcetfungi: rite, we will reusing existing code14:13
smarcetbut on a diferent code base14:13
smarcetfungi : so i think that on first place, we would need a new repo here
fungido you want to start with an empty project and then commit it all through code review from the start, or do you already have a separate repo you've started committing into that we'll need to import?14:14
fungialso, most importantly, does this repo already have a name ;)14:14
*** zz_dimtruck is now known as dimtruck14:15
smarcetfungi: i would rather prefer start with an empty project, and do all commit through code review workflow, btw, this workflow is still tied to ttps://
smarcetfungi: yeah we need a name14:15
*** dhritishikhar has quit IRC14:16
*** hdd has quit IRC14:16
smarcetfungi: what about openstackid-resource-server?14:16
*** ssam2_ has joined #openstack-infra14:16
*** ssam2 has quit IRC14:16
*** nelsnelson has joined #openstack-infra14:16
fungismarcet: creating the project just involves a very trivial change following the instructions at
*** dhritishikhar has joined #openstack-infra14:17
fungismarcet: as for the name, that's relatively long, not necessarily a problem but we should see if we can shorten it14:17
*** salv-orlando has quit IRC14:18
fungiis it completely tied to openstackid or is it something that could also be used independent of openstackid in some situations?14:18
smarcetfungi: re: name len, ok np, re: git repo i will take look at that link14:18
*** tonytan4ever has quit IRC14:18
* fungi just realized that step 1 in that guide is "pick a good name"14:18
smarcetfungi: is tied to openstackid, bc its need to query openstackid authorization server to check the token14:18
*** ddieterly has quit IRC14:19
fungibut for example, other software might also query the openstackid authorization server to check tokens as well. wondering if something else might make a more descriptive name14:19
*** ChuckC has quit IRC14:19
fungithis is the server where the marketplace api and other similar resources are going to be hosted, right?14:20
smarcetfungi: this guide states that is used for bugtracking14:20
fungier, service14:20
smarcetfungi: rite14:20
smarcetfungi: so what is the role of
fungismarcet: right, that guide is not 100% applicable to infra projects. it's written more from the standpoint of adding new repos to openstack overall14:21
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Drop postgres jobs and add multinode to tempest check
mtreinishsdague: ^^^14:21
mtreinishjogo: you'll like that too14:21
*** mtanino has joined #openstack-infra14:21
*** isviridov_away is now known as isviridov14:22
smarcetfungi: i am asking this, bc when i want to commit to gerrit , i need to refer the commit to a blue print id14:22
fungismarcet: there's an optional line we can add to openstack-infra/project-config:gerrit/projects.yaml to indicate that storyboard should set itself up to host issue tracking14:22
smarcetfungi: but i tried to do the same with storyboard and didnt work at all14:22
fungithe infrastructure team hasn't really been using blueprints. what specifically is the goal there?14:23
smarcetfungi: so not sure which app to use ( launchpad or storyboard)14:23
smarcetfungi: well, i think that was a requirement given by reed14:23
*** smccully has quit IRC14:24
fungiwe've generally used specifications like you see at for coordinating extremely complex activities, otherwise either linking to bug reports or just writing a good descriptive commit message with each change seems sufficient. but let's get up with reed and find out if that was a requirement for a reason14:24
*** smccully has joined #openstack-infra14:24
fungii expect he'll wander by any moment14:25
fungianyway, putting the blueprint discussion aside for the moment, how about a name like foundation-resources?14:25
fungior openstackid-resources if it's really tightly coupled to openstackid itself14:26
smarcetfungi: ok, meanwhile could i create thie project on launchpad?14:26
*** stevemar has quit IRC14:26
BobHfungi: here:
mtreinishsdague: :( way to rain on my parade14:27
smarcetfungi: re:name, i like openstackid-resources14:27
fungismarcet: there's no need to do the launchpad part of it for now. that can always be done later if it ends up being necessary14:27
smarcetfine with me14:27
smarcetfungi : roger14:27
fungismarcet: okay, i'll whip up a quick project-config change for this and you can see how that maps to the process in that guide14:28
mtreinishsdague: although with the schema input validation won't most of the negative stuff be caught earlier now?14:28
smarcetfungi: ok tnx :)14:28
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Decrease PO files for Doc team imports
openstackgerritTristan Cacqueray proposed openstack-infra/zuul: Add support for layout configuration split
*** wuhg has quit IRC14:29
openstackgerritJulia Kreger proposed openstack-infra/shade: Update unregister_machine to use tasks
smarcetfungi: btw, i was planning to use latest laravel version ( 5.x), but its needs php >= 5.414:30
NobodyCammorning INFRA folk, off the wall question: if someone was setting up third party testing, where would the best place to post the logs be?14:30
*** radez_g0n3 is now known as radez14:31
fungismarcet: we can deploy it on ubuntu trusty and that should have new enough php to meet that requirement i think14:31
smarcetfungi: does infra server supports that? i am asking bc we cant upgrade openstackid to use latest laravel so far14:31
fungismarcet: yes, at the time when the openstackid server was initially deployed ubuntu trusty hadn't been released yet. we should still look into an upgrade maintenance to be able to move it onto a newer server soon14:32
smarcetfungi: roger that tnx14:32
fungismarcet: looks like php 5.5.9 so should work?14:33
smarcetfungi: yup should do it14:33
*** isviridov is now known as isviridov_away14:34
*** dhritishikhar has quit IRC14:35
*** dizquierdo has joined #openstack-infra14:35
*** jerryz has quit IRC14:36
*** unicell1 has joined #openstack-infra14:36
*** unicell has quit IRC14:37
*** yfried__ is now known as yfried|afk14:38
*** dangers_away is now known as dangers14:38
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add the openstackid-resources repo
fungismarcet: that ^ is what will create the new project in our systems14:39
smarcetfungi: ok tnx14:40
fungii'm writing a governance change now (order of change dependencies is reversed based on a tc resolution yesterday, and the guide is about to get updated to reflect that with )14:40
jeblairmordred: why was abandoned?14:40
*** asselin_ has joined #openstack-infra14:42
fungismarcet: the corresponding governance bookkeeping addition is
*** sdake has joined #openstack-infra14:43
jeblairmordred: is it because master is shaping up for release?14:43
jeblairand the 0.10 branch is being abandoned?14:43
*** yamahata has quit IRC14:43
*** sdake_ has joined #openstack-infra14:44
*** yamahata has joined #openstack-infra14:44
fungijeblair: do you mind if i update your 173485 infra manual change based on AJaeger's feedback before i +2 it?14:44
*** yfried|afk is now known as yfried__14:44
smarcetfungi: who is the one in charge to approve those changes? reed?14:44
*** tsg has quit IRC14:45
*** tqtran has quit IRC14:45
openstackgerritJerry Zhao proposed openstack-infra/nodepool: add option to use ipv6 for image update and node launching
AJaegerjeblair, btw. could you review this infra-manual and approve it, please? It has two +2s but some wanted to give you a chance to review.14:47
*** MaxV has quit IRC14:47
*** sdake has quit IRC14:48
*** MaxV has joined #openstack-infra14:48
fungismarcet: these people...,members and,members14:48
smarcetfungi: ok tnx14:48
openstackgerritEmilien Macchi proposed openstack-infra/project-config: beaker: enable voting for puppet-(keystone/openstacklib)
*** jerryz has joined #openstack-infra14:48
fungismarcet: in irc that's AJaeger, anteaya, sdague, clarkb, pleia2, jeblair, me, jhesketh, mordred and SergeyLukjanov14:49
*** tonytan4ever has joined #openstack-infra14:49
smarcetfungi: cool tnx for the info :)14:49
fungias long as a couple of them look it over and are okay with it, that's all we need14:49
*** dkranz has quit IRC14:49
jerryzfungi: when i rebased on the change to switch to pymysql you added recently, unit test would fail due to mysql connection denied.14:50
jerryzfungi: pymysql somehow used my fqdn to connect to mysql server instead of localhost14:50
fungijerryz: which repo? nodepool?14:51
jerryzfungi: yes14:51
fungijerryz: i think the main behavior difference is that pymysql only supports connection via tcp socket, not unix socket (which was mysql-python's default behavior)14:51
fungijerryz: in our case it seems to connect to the loopback interface. let me double-check our config14:52
*** mrunge has quit IRC14:52
clarkbjerryz could it be a permissions issue? mysql treats the connection endpoints independently for access14:53
fungijerryz: right, so the dburi in our nodepool.yaml looks like mysql+pymysql://username:password@localhost/dbname14:53
fungi"localhost" being the key part there14:53
clarkbso you can have permission to connect via unix socket but not localhost tcp14:53
AJaegeris zuul/gerrit stuck? passed the check queue 35 mins ago and is not yet in the gate queue14:53
fungiAJaeger: the changes it depends on aren't in the gate14:54
fungiits parent commit has a -1 from jenkins still14:54
AJaegerfungi, ARgh - somebody rebased my patch...14:54
*** davideagnello has joined #openstack-infra14:54
AJaegerThanks, that explains it...14:54
*** tonytan4ever has quit IRC14:55
jerryzfungi, clarkb: this is the error i am seeing
fungijerryz: checking in our nodepool database, our users are just identified by "localhost" in the host column and their username in the user column (of the mysql.user table)14:56
clarkbNobodyCam: someplace accessible by the users of our gerrit server14:56
jerryzfungi: i did that as well14:56
clarkbNobodyCam: we are currently moving to storing logs in swift with a wsgi app in front to do the serving (so we can modify logs on the fly and so we don't have to deal with crazy cdn stuff at least not yet)14:57
jerryzfungi: but unit test still use fqdn to connect the database14:57
clarkbjerryz: yup that is a permissions issue14:57
*** yfried__ is now known as yfried|afk14:57
fungijerryz: oh! unit tests specifically, not your production server14:57
NobodyCamclarkb: ah so no real formal location14:57
fungijerryz: what patch are you seeing this from?14:58
clarkbits saing this is the user connecting to this location and trying to access this db and that isn't allowed14:58
jerryzfungi: the patch can pass on infra jenkins14:58
clarkbNobodyCam: no we don't provide hosting for you nor do we enforce a specific type of file hosting14:58
*** hdd has joined #openstack-infra14:58
fungijerryz: got it, so it's a database setup issue on the worker where that job runs14:58
jerryzfungi: but i can't make tox -epy27 to pass locally14:58
NobodyCamclarkb: :) awesome TY14:58
*** hdd has quit IRC14:59
fungijerryz: so you probably need your local mysql permissions set similar to how we do it on our job workers14:59
*** signed8bit has joined #openstack-infra14:59
jerryzfungi: i have granted the privileges to openstack_citest@localhost15:00
*** russellb has quit IRC15:00
fungijerryz: a summary of the commands needed can be found at
fungijerryz: so probably needs to be openstack_citest@% instead15:01
jerryzfungi: yes that is the key15:02
jerryzfungi: readme need to be updated15:02
fungijerryz: awesome--thanks for finding that15:03
*** zz_johnthetubagu is now known as johnthetubaguy15:03
*** johnthetubaguy is now known as zz_johnthetubagu15:03
fungijerryz: one of the things i want to get to next once bindep has been beaten into shape is some way to provide easy to consume database config scripts for projects that need some sort of setup like this15:04
*** zz_johnthetubagu is now known as johnthetubaguy15:04
fungisomething they can carry or easily access from their individual git repositories and be able to document consistently15:04
*** _nadya_ has joined #openstack-infra15:04
*** dkranz has joined #openstack-infra15:04
*** russellb has joined #openstack-infra15:05
fungibut those (mysql|pgsql)-(db|prep) builder macros will suffice initially15:05
*** annegentle has quit IRC15:06
*** agentle_ has joined #openstack-infra15:06
*** annegentle has joined #openstack-infra15:06
jerryzfungi: cool.15:07
jerryzfungi: yes but people need some knowledge of jjb and where to find the template.15:08
fungiultimately i think we need some sort of structured framework we can standardize on where a particular repo can say "to run these tests, first do this as root"15:09
fungiright, keeping that in jjb configuration is not the end goal15:09
fungiit's just a stepping stone15:09
fungiright now we're still actually doing that prep in our nodepool images when they're created15:10
fungiso step 1 is to do it at the start of each job instead15:10
gordcttx: do you have any concerns with making a ceilometerclient 1.0.14 release for kilo (with no requirements changes). we have a user who is using kilo in an upcoming product release15:10
*** yfried|afk is now known as yfried__15:10
fungijerryz: then step 2 is to make it so it can be overridden in each repo15:10
*** jamesmcarthur has joined #openstack-infra15:10
ttxgordc: yes we have a concern, would like to delay that a few more hours15:11
gordcttx: just for reference, 1.0.14 would have same changes as 1.1.0 (minus all the requirements changes)15:11
ttxgordc: we just unbroke the world and would like it to stay calm for a bit. But yeah worst case, we'll do them tomorrow15:11
gordcttx: sure thing. understandable given other gate issues... i'll hold off until you guys have it figured out15:11
gordcttx: sounds good. thanks!15:12
russellbhrm, seems to be down15:12
funginice. that's our server, so all sorts of things are down15:13
ttxthat cloud thing sure is flaky15:13
russellbttx: +115:13
*** agentle_ has quit IRC15:13
*** annegentle has quit IRC15:13
fungiapache seems to be running, not restarted recently15:13
funginothing recent in dmesg15:14
funginow it's finally returned content to me15:14
clarkbload is reasonable too so its not like it just stopped serving requests because there were too many15:14
fungiupdatedb.mlocate running there since 06:38 utc15:14
fungimight have been brief i/o starvation15:15
fungithat will eventually get better when the last of our logs expire from there15:15
russellbah, there it goes, yeah loaded eventually15:15
clarkbspeaking of anyone want to review the topic:enable_swift changes?15:15
fungiwant to? yes15:15
fungihere we go!
*** sabeen has joined #openstack-infra15:16
fungisomeone decided to download a ton of comething15:16
jogomtreinish: already removed postgres from neutron15:16
*** dboik_ has joined #openstack-infra15:16
clarkbfungi: its image builds15:16
*** fawadkhaliq has joined #openstack-infra15:16
clarkbfungi: grabbing images to cache for devstack15:16
clarkbwe should put those in swift15:17
*** dboik_ has quit IRC15:17
fungioh... the ironic agent images?15:17
*** dboik_ has joined #openstack-infra15:17
fungiyep, timing is right for that15:17
clarkband trove image15:17
fungidoesn't help that mlocate is reindexing the world at the same time15:18
fungiso i/o load and network load15:18
*** andreykurilin__ has quit IRC15:18
openstackgerritVictor Sergeyev proposed openstack-infra/project-config: Make oslo.db py3 job voting
fungii have to say, our log volume is looking emptier and emptier. we trimmed it back a lot around the end of last year
*** dboik has quit IRC15:20
russellbfungi: clarkb thanks for looking, and sorry to distract :)15:20
mtreinishjogo: well only partially right? there were 2 sets of those jobs iirc15:20
clarkbrussellb: no problem at all, I would much prefer to know that we should use swift more than irritate our users :)15:20
*** rlandy has quit IRC15:21
*** tqtran has joined #openstack-infra15:22
*** rlandy has joined #openstack-infra15:23
jogomtreinish: correct, I235f6fe1c0f37c8f3acaf92562d2d6320a86483c15:23
fungiclarkb: so i guess this ratchets up the priority a bit for getting tarballs.o.o into swift, and in such a way that downloads happen via http redirect or something rather than getting proxied through static.o.o15:25
*** arxcruz has quit IRC15:25
fungiunlike logs where we want to mangle them on the way through15:25
fungifor tarballs there is zero benefit to proxying15:25
*** yolanda has quit IRC15:25
*** tqtran has quit IRC15:26
fungi(tarballs, images, other binary blob artifacts)15:26
*** armax has joined #openstack-infra15:26
fungities in pretty nicely to too15:27
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Search methods for networks, subnets and routers
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Update volume API for new getters and dict retval
fungier, well even without that i guess, we could in theory start pushing dib nodepool images up to swift independent of having separate dib workers15:29
fungi(aside from where we have to for glance v2 api i mean)15:29
*** sabeen3 has joined #openstack-infra15:29
clarkbfungi: I think step one is getting them into swift, it will remove the disk IO and make it purely network which I think is cheaper for us15:30
clarkbfungi: then we can worry about the CDNing15:30
fungibut be able to serve them alongside tarballs and other non-log artifacts15:30
*** sabeen has quit IRC15:30
fungiyep, agreed15:30
*** dizquierdo has quit IRC15:30
*** vlaza has quit IRC15:31
clarkbthe other thing that helps this specific case is using dib to build those images since dib will cache them and only download updates when necessary15:32
*** claudiub has quit IRC15:32
clarkbso all the things are tied together in a web of fixit :)15:32
*** zhiwei has left #openstack-infra15:33
hogepodgeIs is just me, or is anyone else experiencing connection problems with etherpads?15:33
*** annegentle has joined #openstack-infra15:33
*** marun has joined #openstack-infra15:33
clarkbhogepodge: anteaya had trouble but no one else was able to reproduce15:33
hogepodgeI'm having to force reconnect and am losing edits every few minutes.15:34
clarkbya that was what anteaya saw, what browser and version are you using?15:34
*** dkranz has quit IRC15:34
*** dhritishikhar has joined #openstack-infra15:34
hogepodgeSafari 8.0.515:34
*** rwsu has joined #openstack-infra15:34
fungiat the moment i'm having trouble getting an etherpad to load15:34
*** ihrachyshka has quit IRC15:35
fungiso not similar to what anteaya described15:35
hogepodgeI've been seeing slow loads also, but I thought maybe it was a caching issue from the update.15:35
clarkb loads for me15:35
*** e0ne has quit IRC15:36
fungiit took a couple minutes for to finish loading for me, for some reason15:36
hogepodgeIt's not a matter of not loading, it's just taking a long time on older docs and when I'm editing the connection times out and I lose the edits15:36
hogepodgeI've taken to editing outside of the etherpad and pasting in to make sure I don't lose anything.15:36
clarkbfungi: that one loads straight up for me too15:37
*** e0ne has joined #openstack-infra15:37
fungihogepodge: yeah, that does sound similar to what anteaya ran into15:37
clarkbhogepodge: what etherpad are you editing?15:37
clarkbI can confirm the slowconnection disconnect in the logs15:37
* clarkb goes to look15:37
*** asselin_ has quit IRC15:38
fungithere's a chance it was a browser extension. she disabled all her extensions and restarted the browser and it went away, then she added back her extensions one by one and it kept working for a while, but eventually she got another disconnect15:38
clarkbDIAGNOSTIC-INFO: {"disconnectedMessage":"reconnect_timeout","padId":"DefCoreScale.13","socket":{}}15:38
clarkbthat is a different error15:38
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support to CFP in builders sections
hogepodgefor this morning, and last night this one
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Use new getters in update_subnet and update_router
*** dhritishikhar has quit IRC15:38
clarkb could you have it open in multiple tabs?15:39
hogepodgefungi: I'll take a look into that. I'm only running an ad blocker.15:39
*** yfried__ is now known as yfried|afk15:39
*** annegentle has quit IRC15:39
clarkbhogepodge: I do see two clients against that particular pad, hard to figure out if the have the same id becaus apahce is proxying15:40
hogepodgeclarkb: no, I have multiple eterpads open, but to different documents15:40
*** annegentle has joined #openstack-infra15:40
fungidepending on how the ad blocker is implemented, if it disconnects or blocks an http request for an ajax api call then it might cause this sort of symptom15:40
*** smoser has left #openstack-infra15:40
*** Sukhdev has joined #openstack-infra15:40
hogepodgefungi: I disabled it for all etherpad sites, so we'll see what happens.15:40
clarkbhogepodge: who is the dark green unknown user on that etherpad?15:40
*** yolanda has joined #openstack-infra15:40
anteayahogepodge: are you on ubuntu?15:41
fungiprobably not unless someone ported safari to linux15:41
anteayahogepodge: ubuntu has a default of 4 addons15:41
anteayaah sorry I haven't read all the backscroll15:41
hogepodgeclarkb heh, it could be me. It looks like I have a couple of users at the same time. I'll restart my browswer.15:41
*** dhritishikhar has joined #openstack-infra15:41
clarkbhogepodge: I bet that is your problem15:42
openstackgerritMerged openstack-infra/jenkins-job-builder: Initial work for supporting downstream-ext
openstackgerritMerged openstack-infra/jenkins-job-builder: Add FindBugs support to Publishers
clarkbits annoying that the error message isn't better from etherpad-lite though15:42
clarkbanteaya: you might also want to double check you aren't double connected to the etherpad(s) you had trouble with15:43
anteayahogepodge: I only had disconnects when I used firefox, I never disconnected when I used chromium15:43
*** yamahata has quit IRC15:43
anteayaI was double connected with chromium and had no disconnects15:43
anteayaI had problems with firefox when I was single connected15:43
hogepodgeclarkb: for some reason it's still bringing me in as two users. Cleared my cache and cookies.15:43
clarkbhogepodge: probably a second tab15:43
clarkbyou would need to close it15:44
hogepodgenope, no tab open15:44
*** gyee has joined #openstack-infra15:44
mtreinishhogepodge: looking at that etherpad, you know you need to add a include regex list file option to os-testr then you can just use the the idempotent ids to do the selection and not worry about moves15:44
mtreinishinstead of having to worry about passing the whole test_id everytime15:44
hogepodgemtreinish: if I use just the idempotent id won't I oversubscribe to the tests though? it's not a 1:1 mapping15:46
jerryzfungi: this is still causing issue for me.15:47
*** dhritishikhar has quit IRC15:47
mtreinishhogepodge: there is to a test method, which means given a config it'll completely test that feature15:47
*** dkranz has joined #openstack-infra15:47
*** dhritishikhar has joined #openstack-infra15:47
*** weshay has quit IRC15:47
mtreinishhogepodge: like if you tell tempest your cloud supports ipv4 and ipv6 would you want defcore validation to pass if one failed?15:47
mtreinishfor a particular test15:47
*** sdake_ has quit IRC15:48
hogepodgemtreinish: but we may not require all of the tests associated with the test method, so there could be false positive/negatives. The intent of that list is to only run the tests that are required and nothing more15:48
*** yfried|afk is now known as yfried__15:48
*** spredzy is now known as spredzy|afk15:49
fungijerryz: so you're getting an exception raised from the grant call, but not from the create call before it?15:49
mtreinishhogepodge: give me an example, because the only cases I can think of where subclassing is used are things things you shouldn't differentiate on in defcore (like ip version)15:49
mtreinishof if you did it should be a strong opinion, which you should force in the config file (like service api version)15:50
mtreinisherr, service api major version (ie cinder v1 or v2)15:50
jerryzfungi: no, grant privilege to localhost won't suffice. because pymysql attempts to connect to fqdn instead of localhost each time.15:50
mtreinishhogepodge: and in either case I don't think a failure would be a false result15:51
fungijerryz: i wonder if the difference is our /etc/hosts file15:51
jerryzfungi: probably15:51
hogepodgemtreinish: I'll speak with catherine about it. She's the expert on it, and initially raised the flag after analyzing the collisions.15:52
hogepodgemtreinish: it may not be an issue right now, though. Let me look and see if it is. I'll compare output from using the full names to the output from just using the ids15:53
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Sonatype CLM plugin
fungijerryz: here's an example of one of ours
hogepodgemtreinish: One of the things I'd like to do is add a non-voting gate job to defcore so that if tempest moves change anything we'll know.15:53
fungijerryz: having localhost mapped to is one thing to double-check you've got in yours15:53
*** smarcet has quit IRC15:54
*** ashleighfarnham has joined #openstack-infra15:55
mtreinishhogepodge: well tbh that's what the ids being added was meant to prevent the need for15:55
mtreinishhogepodge: I just feel like the concern on this stuff is coming from looking at the mechanics of listing and counting instead of looking at what the tests are actually doing15:56
openstackgerritDoug Hellmann proposed openstack/requirements: Remove OpenStack library caps for liberty
anteayaasselin: when you are about can you please read this email and propose your work to the etherpad:
*** funzo has quit IRC15:57
*** deepakcs has joined #openstack-infra15:57
anteayaasselin: that way we can get you a table as was discussed in the meeting yesterday15:57
hogepodgemtreinish: There's not a 1:1 mapping of test names to ids, so there's always the possibility of collision between required and non-required tests, regardless of the functionality.15:57
*** funzo has joined #openstack-infra15:58
*** dboik_ has quit IRC15:59
hogepodgemtreinish: you may disagree with that philosophically, but it's the reality of the process. My mistake was in not realizing the issue and pushing for the solution that sergei has proposed earlier.15:59
mtreinishhogepodge: well the overload only happens in a subclass, which will only happen if there is a config selector between tests. It's testing the same functionality over all the permuations you tell tempest it can run15:59
openstackgerritDoug Hellmann proposed openstack-infra/devstack-gate: add support for stable/kilo
*** ashleighfarnham has quit IRC15:59
*** ashleighfarnham has joined #openstack-infra16:00
*** dhritishikhar has quit IRC16:00
mtreinishhogepodge: I'm saying if defcore wants to say cloud X supports network feature Y. Then unless the feature Y test for all subclasses enabled via config passes you can't say that16:00
*** dhritishikhar has joined #openstack-infra16:00
*** dboik has joined #openstack-infra16:00
mtreinishhogepodge: because the runner of the test is saying my clouds supports ipv6 and ipv4 but the test for the feature doesn't work on both16:01
*** flip214 has joined #openstack-infra16:01
*** otter768 has joined #openstack-infra16:02
jerryzfungi: just checked it is similar to yours. i even went ahead to delete the line mapping to fqdn16:02
hogepodgemtreinish: what do you mean? It would be reasonable to say ipv4 is required, but ipv6 is advisory. We would need to distinguish between the two16:02
*** funzo has quit IRC16:02
jerryzfungi: still got the same error16:03
fungijerryz: is it the same permission error you pasted at or a new one?16:04
mtreinishhogepodge: no you don't. So you say ipv4 is required that means you force everyone to set that config option in tempest. But ipv6 is float, it can either be true or false it doesn't matter the burden is on the end user16:04
*** yfried__ is now known as yfried|afk16:04
mtreinishhogepodge: but if they set it true and the ipv6 tests fail then their cloud fails for that functionality because they're saying things should support ipv6 and they don't16:05
hogepodgebut by id only it's possible to run both tests when only one is required, and it's reasonable to say that one would pass and one would fail.16:05
asselinanteaya, reading16:05
*** hashar is now known as hasharAway16:05
mtreinishhogepodge: I'm saying its not reasonable, because it's all based on the config file. If you want that you just need to enforce the value of the option used to switch the tests16:06
mtreinishhogepodge: so if you want to say only ipv4 tell people they have to set ipv4 true and ipv6 false16:06
*** _nadya_ has quit IRC16:06
jerryzfungi: new one since i changed openstack_citest@‘%’ .
*** teran has joined #openstack-infra16:07
*** dprince has quit IRC16:07
*** scheuran has quit IRC16:07
hogepodgemtreinish: It does matter in the larger picture though, because we're going to be asking people to run many api tests (non required) to help guide future capabilities. From a management standpoint it means a lot more work to index and keep things straight16:07
*** otter768 has quit IRC16:07
*** shashankhegde has joined #openstack-infra16:07
mtreinishhogepodge: but if you leave it up to the user, then you can't say some functionality works when it doesn't for the configuration they say their cloud supports16:08
zaroclarkb, fungi : looks we will probably see this issue in our Gerrit openstack/2.10.2 branch,
zaroour branch contains sshd 0.14.016:08
mtreinishhogepodge: ok, maybe. I'm not exactly sure what you mean there.16:09
*** yfried|afk has quit IRC16:09
*** teran__ has quit IRC16:09
hogepodgemtreinish: They'll send passed results only, but we still have to sort the results. If we just use id we can't know if they pass ipv4 or ipv616:09
*** marun has quit IRC16:09
*** Guest5363 has quit IRC16:10
zaroclarkb, fungi : upgrade is scheduled in about two weeks so not sure what we want to do if upstream doesn't revert sshd back to a working version of SSHD16:10
hogepodgeI'm on a call right now, so can't respond for about an hour16:11
*** erikmwilson has quit IRC16:11
*** devvesa has quit IRC16:11
*** MaxV has quit IRC16:11
*** erikmwilson has joined #openstack-infra16:11
mtreinishhogepodge: well I'm not sure that matters if you're saying you don't care about ipv6. But you can do that because skips are still in the test results16:12
mtreinishso you have all the necessary information16:12
mtreinishhogepodge: ok sure16:12
*** jistr has quit IRC16:12
*** erikmwilson has quit IRC16:13
*** erikmwilson has joined #openstack-infra16:13
*** ddieterly has joined #openstack-infra16:14
*** dboik has quit IRC16:15
anteayaI will further note that I can attempt to open up an etherpad in firefox and while it is trying to load, open up the same etherpad in chromium and chromium is well finished while I close the tab on firefox because it is still trying to load16:16
*** unicell1 has quit IRC16:18
*** jlanoux has quit IRC16:20
openstackgerritRyan Petrello proposed openstack-infra/project-config: Set the proper homepage for akanda repositories.
*** radez is now known as radez_afk16:21
*** coolsvap|afk has quit IRC16:21
*** dboik has joined #openstack-infra16:24
*** coolsvap|afk has joined #openstack-infra16:24
*** dprince has joined #openstack-infra16:25
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove YamlParser from jenkins_jobs.builder
*** aduarte has joined #openstack-infra16:25
*** nfedotov has quit IRC16:26
*** ajmiller has joined #openstack-infra16:26
waynrpelix, zaro, clarkb (and anyone else familiar with JJB) I'm hoping to move quickly on since it is primarily a code organization change16:27
openstackgerritFabien Boucher proposed openstack-infra/infra-specs: Specification proposal about system-config testing using containers
*** Swami has joined #openstack-infra16:28
openstackgerritFabien Boucher proposed openstack-infra/infra-specs: Specification proposal about system-config testing using containers
fungizaro: got it. so we might want to only upgrade to 10.2 while they sort out whether there are sshd issues in 10.3?16:32
fungier, 2.10.2 as opposed to 2.10.316:32
sslypushenkomtreinish: I updated accouts-generator tool according to your comments16:32
*** ddieterly has quit IRC16:32
asselinwhat's the difference b/w fishbowl and workroom?16:32
jerryzfungi: granting the privilege to '%' nodepool/tests/ works for me. however, it is still unknown why the connection was resolved to my fqdn.16:33
jeblairfungi, pleia2:
mordredjeblair: zomg16:33
mordredfungi, jerryz: I have not read the whole scrollback - but it soudns like a case of having not deleted the anonymous user16:34
*** dannywilson has joined #openstack-infra16:34
jeblairnotnownikki just told me about that16:34
mordredjeblair: I would like to say that I'm a fan of that16:35
zarofungi: yep,  what would you suggest we do with current openstack/2.10.2 branch we have setup?16:35
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for create_image with glance v1
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Explicitly request cloud name in test_caching
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Add tests and invalidation for glance v2 upload
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for caching in list_images
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Test flavor cache and add invalidation
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Test that deleting user invalidates user cache
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Fix major update_user issues
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: create_user should return the user created
SpamapSmordred: ^^ rebases. :-P16:36
*** odyssey4me has left #openstack-infra16:36
zarowaynr: i like that change, will try to take a look today.16:36
mordredSpamapS: woot!16:36
openstackgerritStephanie Miller proposed openstack-infra/puppet-zanata: Make default from address mandatory
*** andreykurilin__ has joined #openstack-infra16:37
jerryzmordred: could you elaborate?16:37
*** luqas has quit IRC16:37
zarowaynr: may want to ping jhesketh on that jjb change as well.16:37
mtreinishanteaya, pleia2: so about that thread from a couple weeks ago about election tooling. I put together: as the basic idea behind on how we can use the ml16:37
*** ddieterly has joined #openstack-infra16:38
mordredjerryz: mysql ships with a user account with a name of ''16:38
mtreinishbut it's just idea code, I haven't actually tried to run it or anything16:38
mordredjerryz: it's a terrible idea16:38
mtreinishbut before I put any real effort into it I figured I should share16:38
*** dtantsur is now known as dtantsur|afk16:38
*** davideagnello has quit IRC16:38
mordredjerryz: and what's worse about it is that the username/hostname resolution rules in mysql do not produce the result they think you would if that user is present16:38
*** dboik has quit IRC16:39
mordredso often you'll wind up creating users and host matches that arent necessary, because the anonymous user was intercepting your login attempts16:39
mordredjerryz: but after lunch I'll read the whole scrollback16:39
pleia2mtreinish: cool, I'll have a look16:40
zarofungi: i guess we can leave it alone and wait for upstream to revert back to sshd 0.9 but i'm concerned that may not happen before we do upgrade so i'm thinking of a plan B16:40
*** mpaolino has quit IRC16:41
openstackgerritDoug Hellmann proposed openstack-infra/devstack-gate: add support for stable/kilo
*** ddieterly has quit IRC16:42
openstackgerritStephanie Miller proposed openstack-infra/system-config: Make default email mandatory
cineramapleia2: hi there16:42
pleia2cinerama: hey :)16:43
cineramapleia2: do you have some time today to talk through client stuff with me?16:44
pleia2cinerama: sure, maybe in an hour or so? I've just been working through some administrative/paperworky stuff this morning, should be done in a bit16:45
fungizaro: yeah, it seems like only upgrading as far as 2.10.2 until they work out the mina-sshd issues might be fine. then if it's solved in 2.10.4 or something, that's an easy upgrade to schedule whenever16:45
cineramapleia2: cool. i also have a couple of teensy patches put up16:45
pleia2jeblair, fungi, clarkb, anteaya: I added nibalizer's outreachy ideas here, so we can add more as needed
*** Krinkle|detached is now known as Krinkle16:48
*** unicell has joined #openstack-infra16:48
*** subscope_ has joined #openstack-infra16:48
*** dannywilson has quit IRC16:48
pleia2nibalizer: and thanks :) good ideas16:50
*** e0ne has quit IRC16:50
waynrjhesketh: hoping to get quick review and merge on this since it is primarily organizational:
*** tonytan4ever has joined #openstack-infra16:52
fungiooh... now i'm getting the pad reconnects too16:53
*** viglesias has quit IRC16:54
*** dhritishikhar has quit IRC16:54
*** mpaolino has joined #openstack-infra16:54
*** melwitt has joined #openstack-infra16:55
*** dhritishikhar has joined #openstack-infra16:55
*** yolanda has quit IRC16:55
anteayafungi: oh, mind if I'm happy about that?16:56
funginot at all16:56
anteayain that you are far better at debugging than myself16:57
fungiit seems to have been short-lived16:57
anteayawhat browser?16:57
funginope, there it went again16:57
fungifirefox 3116:57
*** bhuvan has joined #openstack-infra16:57
fungiwasn't doing it at all yesterday and i've not updated anything16:57
anteayaare you typing when it disconnects? or is it just open16:58
fungijust open16:58
fungiother people are typing16:58
anteayafunny, mine was usually while I was typing, but a few times when it was just open16:58
fungiit's possible there are leaks in wherever it handles its javascript stuff and i've bogged it down with a ton of gerrit tabs and whatnot16:58
anteayacan you open the same tab in chromium and see what your experience is?16:59
anteayaor another browser?16:59
fungibecause it's also been taking me 2+ minutes to open or force reload the pad16:59
*** harlowja_away is now known as harlowja16:59
*** deepakcs has quit IRC16:59
fungii don't have chromium installed but may test it later if i get around to it. for now i'm just going to see if restarting my browser helps at all17:00
*** dizquierdo has joined #openstack-infra17:00
anteayaI don't have a ton of tabs open but I do have a number of other pads open when it took forever to load a new pad this morning, the one I gave up on and just worked on in chromium17:00
anteayago you17:00
*** viglesias has joined #openstack-infra17:00
*** ddieterly has joined #openstack-infra17:00
*** derekh has quit IRC17:00
*** ddieterly has quit IRC17:02
*** bhuvan has quit IRC17:03
pelixwaynr: just a few minor nits on but looks real good, be nice to have the code base cleaned up a bit.17:03
*** shashankhegde has quit IRC17:03
*** andreykurilin__ has quit IRC17:03
*** ildikov has quit IRC17:04
*** jamesmcarthur has quit IRC17:05
*** tqtran has joined #openstack-infra17:05
anteayamtreinish: thank you17:06
*** subscope_ has quit IRC17:07
*** jamesmcarthur has joined #openstack-infra17:07
mtreinishanteaya: sure, np. I figured I should try and back up my assertions on the ML17:07
*** subscope_ has joined #openstack-infra17:07
*** salv-orlando has joined #openstack-infra17:07
anteayaand you did so17:08
anteayacan we discuss the potential values for elec_type:
anteayaI'm thinking we might be able to find a better value here :)17:09
jerryzfungi: adding skip-name-resolve fixed my issue. this time when i use pymysql with localhost as host to connect to the database, the connection won't get resolved to fqdn17:09
anteayamtreinish: acclaimed comes to mind17:09
mtreinishanteaya: hehe, well it's internal only. I doubt it would come up in openstack though17:10
jerryzfungi: i wonder if infra's slave has this mysqld option enabled17:10
mtreinishit's an election for a single spot without projects or a seat count17:10
fungijerryz: is that a parameter to the database connection, or a mysqld config option?17:10
*** ddieterly has joined #openstack-infra17:10
fungijerryz: ahh, i'll check17:10
jerryzfungi: in my.cnf17:10
anteayamtreinish: I like to plan for the possibility that it would17:11
jeblairanteaya: i'm seeing a patch in your future :)17:11
anteayaha ha ha17:11
*** AJaeger_ has joined #openstack-infra17:11
mtreinishanteaya: sure that's why I put it in there. We can change that value to be whatever17:12
zarofungi: did you get the info i provided yesterday on my Gerrit testing with regards to bouncycastle error?17:12
*** annegentle has quit IRC17:13
fungizaro: i may have missed it. last night is a bit of a blur17:13
anteayamtreinish: an election for a single spot without projects or a seat count: leader, lead?17:14
*** dizquierdo has quit IRC17:14
fungijerryz: i don't see skip-name-resolve in /etc/mysql/my.cnf on our workers17:14
*** yfried|afk has joined #openstack-infra17:15
*** sushilkm has joined #openstack-infra17:15
*** sushilkm has left #openstack-infra17:15
mtreinishanteaya: sure, either works. That value is just an internal keyword for when it does validation and eventually when it automates the election creation (I didn't look at the CIVS stuff yet)17:15
anteayathere is a word we haven't overloaded yet17:16
*** unicell has quit IRC17:16
*** unicell1 has joined #openstack-infra17:16
mtreinishanteaya: heh, yeah I like that17:16
anteayaoh personally I would be really hesitant to have it automate election creation17:16
*** russellb has quit IRC17:17
anteayaif we can clean up the nomination process I think we will have reduced the burden17:17
mordredanteaya: we could write a bot which generates an election for every patch proposal17:17
anteayamtreinish: have you created a poll with more than 1000 people on the CVIS?17:17
anteayamordred: we could17:17
*** ddieterly has quit IRC17:17
*** dizquierdo has joined #openstack-infra17:17
mtreinishanteaya: no, I haven't had a chance to look at it yet17:17
*** jamesmcarthur has quit IRC17:17
anteayaplease create youself a couple of test polls17:18
anteayafeel free to use my email to send me some ballots17:18
anteayausing the tools are really helpful, relying on the tools is somethign else entirely17:18
anteayais really helpful17:18
mtreinishanteaya: ah, it has a 1000 person limit17:19
anteayaI'm all for using helpful tools, I personally am very much against relying on them for elections17:19
mtreinishis there an api interface? or is it all web based?17:19
*** jamesmcarthur has joined #openstack-infra17:19
fungiwell, 1000 addresses per batch17:19
anteayaso you have to split the list and submit a second list17:19
anteayamtreinish: I have never looked myself, you can email the creater/maintainer17:19
*** tjones1 has joined #openstack-infra17:19
fungior download the source17:20
anteayait is only one person and he is very busy, be very polite please, as you always are17:20
mtreinishanteaya: sure, and it's all on github too:
zarodid a recheck in gerrit but can't find change in zuul queue.
*** davideagnello has joined #openstack-infra17:21
*** russellb has joined #openstack-infra17:21
anteayahe does reply to emails, I have offered to help him if he needed any help and he never took me up on it17:21
anteayaI figured it was the least I could do since we were using his service17:21
*** teran has quit IRC17:22
fungihrm, latest change in the check pipeline is from 34 minutes ago17:22
fungii bet stream-events tasks are hanging again17:22
*** teran has joined #openstack-infra17:22
fungii'll get a few gerrit thread dumps spaced 10 seconds or so apart, then restart it again17:22
*** lennyb has joined #openstack-infra17:23
anteayastream-events is hung for me17:23
*** jamesmcarthur has quit IRC17:24
*** dhritishikhar_ has joined #openstack-infra17:24
*** sdake has joined #openstack-infra17:24
clarkbget connection data too right?17:25
clarkbalso I am not here (Drs office)17:25
*** Shrews has quit IRC17:26
*** ChuckC has joined #openstack-infra17:26
*** Shrews has joined #openstack-infra17:26
zarofungi: so the info was that i tested Gerrit (without jeblair's change).  when i enabled contactstore in gerrit.config Gerrit doesn't startup completely.17:26
fungizaro: yep, i saw that much17:27
zarofungi: log with full debug:
*** dhritishikhar has quit IRC17:27
zarofungi: ohh.  so I'm not sure i can debug further on my VM.17:27
*** dizquierdo has quit IRC17:28
zarofungi: options now are to revert review-dev back to 2.8 and test it there or you try and run it manually on review.o.o17:28
*** dizquierdo has joined #openstack-infra17:28
*** dboik has joined #openstack-infra17:29
*** bhuvan has joined #openstack-infra17:29
*** sdake has quit IRC17:29
zarofungi: since you'll need to restart for stream-event hang maybe try to deploy
*** sdake has joined #openstack-infra17:30
*** dboik has quit IRC17:30
*** dboik has joined #openstack-infra17:31
*** dboik has quit IRC17:31
*** ssam2_ has quit IRC17:31
fungi#status notice gerrit is restarting to clear hung stream-events tasks. any review events between 16:48 and 17:32 utc will need to be rechecked or have their approval votes reapplied to trigger testing in zuul17:31
openstackstatusfungi: sending notice17:31
*** dboik has joined #openstack-infra17:31
zarofungi: just remember to install the core plugins from war as well:
-openstackstatus- NOTICE: gerrit is restarting to clear hung stream-events tasks. any review events between 16:48 and 17:32 utc will need to be rechecked or have their approval votes reapplied to trigger testing in zuul17:32
fungiokay, gerrit is back online now17:32
zarook. i guess no go on 1917:33
openstackstatusfungi: finished sending notice17:33
fungii wonder if the content of jstack dumps is security sensitive?17:33
*** Shrews has quit IRC17:34
fungii see some threads blocked in
*** Shrews has joined #openstack-infra17:35
KiallWeird... patch merged, then got a merged failed ;)
*** russellb has quit IRC17:35
*** ddieterly has joined #openstack-infra17:35
fungiKiall: i think that may be a race we see when merge-check pipeline processing gets backed up17:36
*** shashankhegde has joined #openstack-infra17:36
fungizaro: do you expect a jstack dump of the gerrit process to contain any passwords or keys?17:37
*** gyee is now known as chinese_gyee17:37
fungii'm looking through these now to see if it's safe to publish them or not17:37
*** chinese_gyee is now known as gyee17:38
*** luqas has joined #openstack-infra17:38
clarkbI wonder if a well behaved proxy in front of gerrit would help17:38
asselinanteaya, so do you think a fishbowl or workroom is better?17:38
*** pelix has quit IRC17:38
openstackgerritJerry Zhao proposed openstack-infra/nodepool: add option to use ipv6 for image update and node launching
clarkbso only one persistent ssh connection then everything else connects to a publiaher of wvents17:38
openstackgerritStephanie Miller proposed openstack-infra/system-config: Update wildfly puppet module version
fungioh, got it, a many-to-one event stream ssh proxy17:39
anteayaasselin: which would allow you to get the most work done? I think wasting time with more discussion doesn't help you create patches, the spec is up and approved, more time for chat is time wasted to my reconing, what do you think?17:39
*** ddieterly has quit IRC17:40
funginot sure how that would work on the ssh end of things17:40
*** hdd has joined #openstack-infra17:40
*** russellb has joined #openstack-infra17:40
zarofungi: sorry i have not used that tool17:40
anteayaasselin: fishbowl is talk, workroom is work17:40
fungiconnection session keys are negotiated between one client and one server17:40
*** Sukhdev has quit IRC17:40
asselinanteaya, ok thanks17:40
anteayaasselin: sure17:40
zarofungi: same thing as this?
clarkbwell it would need to be a smart app specific proxy then client side would be rabbit/zmq/protobufs/something17:40
clarkbfungi so only ssh between gerrit and proxy17:41
clarkbthen use a proper message bus for publishing17:41
Kiallfungi: actually, I just noticed that every designate patch (not sure about others) just have a merge conflict reported on it.. even ones I'm 150% sure are OK.. I can recheck, but, might be affecting everyone if it's really an issue :)17:42
fungizaro: sort of, except the threads don't have names, just thread numbers and states17:42
*** david-lyle has quit IRC17:42
fungizaro: though i had to use jstack -F so not sure if that makes a difference17:42
*** ivar-lazzaro has joined #openstack-infra17:43
anteayamtreinish: to submit a patch to your github repo, git add, git commit, git push?17:43
fungiKiall: it's possible a zuul merger worker tried to update from gerrit right at the moment i was restarting it. that looks like about the right timing17:43
mordredas a middle step - an ssh-based one that was a single gerrit ssh connection and published an ssh stream on the other side (just a multi-plexer-passthrough) might even be possible to hack together with regular ssh - and then we could just deploy 5 or 6 of them as needed17:43
*** luqas has quit IRC17:43
clarkbmordred ya17:44
*** lennyb has left #openstack-infra17:44
mordredclarkb: like, brainddead version - have a stream-events consumer writing the output to a file, and write a command on the server called "gerrit stream-events" that is just tail -f on the file17:44
Kiallfungi: makes sense :)17:44
* mordred waves hands17:44
mordredclarkb: but it could reduce load on gerrit itself, and should take about 15 minutes to spin up a test version17:44
pleia2cinerama: hey, so just pulled up the outstanding zanata topic reviews, I can review after we chat, what else do we have on our plate?17:44
fungimordred: clarkb: proxying the auth would get interesting. the usernames and keys live in gerrit's database17:45
* pleia2 opens up zanata-install etherpad too17:45
fungithough i suppose we could just have one "stream" user and auth via password or something17:45
mordredfungi: we could make the account no-auth - it's tied to running the stream events command17:45
*** radez_afk is now known as radez17:45
cineramapleia2: i've been updating the etherpad today and patching some bits and pieces. i think the main thing is getting the jenkins scripts doing the things correctly17:45
mordredfungi: why a password even? just passwordless - who cares if you consume the stream17:45
anteayamtreinish: nope17:45
*** mpaolino has quit IRC17:45
fungipasswordless how?17:46
mtreinishanteaya: it's github, I always forget the workflow. I think you have to fork it to your account push the commit to that and then submit a pull request17:46
anteayaah okay17:46
mordredfungi: dunno - haven't thought about it that much :)17:46
pleia2cinerama: cool, reading through now, give me a nudge if there's something I should pay particular attention to17:46
cineramapleia2: so - comparing the capabilities of the clients, deciding how we want to handle the account for it, etc17:46
anteayaI will attempt that17:46
fungimaybe there's some no-op auth method supported by openssh i'm unaware of17:46
mtreinishanteaya: do you want me to just submit the project-config change to add it to gerrit?17:46
*** dkranz has quit IRC17:46
cineramapleia2: i got a bit stuck on the client parts so i thought we could work through that together17:46
anteayaafter I go for a walk, I need to go for a walk17:46
mordredfungi: I've never had teh need to look for such a setting myself :)17:46
cineramapleia2: i think i just need to get unstuck17:46
funginor i!17:46
anteayamtreinish: I would like to get some opinions on what the elections officials would like17:47
anteayaI was just submitting a patch because I think language is important and jeblair encouraged me17:47
pleia2cinerama: yeah, I got as far as building the client the other day from the puppet manifest, but I haven't actually done anything with it17:47
anteayaI still think that having a repo for election nominations is easier than using the mailing list at all17:48
mtreinishanteaya: sure that's why I didn't do it to start. I just thought the workflow would be easier if you're ready for a patch :)17:48
fungizaro: aha, looks like to get thread names i need to run jstack as the gerrit2 user17:48
anteayamtreinish: thanks and I understand, know that regardless of the outcome where ever it lives I'm supportive of doyen over dictator17:49
cineramapleia2: do we have the zanata client on a dev proposal slave?17:49
mordredfungi:      PermitEmptyPasswords17:49
anteayabut I personally would like the current election officials to have the time after they are finished to offer their thoughts on what they would like17:49
mordred             When password authentication is allowed, it specifies whether the server allows login to accounts with empty password strings.  The default is “no”.17:49
mordredfungi: I think that would be it17:49
pleia2cinerama: yes, I believe clarkb tested it after the change merged a couple weeks back17:50
anteayamtreinish: and I'm appreciative of the fact you are willing to back up your ideas with code, thank you17:50
anteayamtreinish: and i need to get a walk in while it is not raining :)17:50
*** yamahata has joined #openstack-infra17:51
pleia2cinerama: confirmed, zanata-cli is there17:51
*** achanda has joined #openstack-infra17:51
cineramapleia2: so for translation, i also care about the propose_translation* scripts in project-config, and any more? clarkb?17:52
*** ddieterly has joined #openstack-infra17:52
fungimordred: poking around a little, looks like that may do it but you need to be mindful of which of the authentication methods can effectively negate that17:52
*** ddieterly has quit IRC17:52
fungimordred: it looks like it may not work with pam auth, or keyboard-interactive password auth, but testing will tell17:53
*** jerryz has quit IRC17:53
pleia2cinerama: refreshing my memory real quick on the spec to confirm17:54
mordredfungi: so we'd need to basically configure the daemon to only allow password auth17:54
mordredwith pam and keyboard-interactive turned off17:55
fungipossibly, yeah17:55
cineramapleia2: oh the other weird thing about the client is that at times it expects you to provide data on the command line which is in the zanata.ini file. not sure if it's a bug or i'm just not using it right17:55
cineramapleia2: the other thing is that the client is unhappy with self-signed certs - i think i imported the cert from translate-dev into the ssl keystore on my VM i was plaing with17:56
pleia2cinerama: we'll have a real cert on the live production cert, we just don't typically do them with dev servers17:57
*** sarob has joined #openstack-infra17:57
*** annegentle has joined #openstack-infra17:58
pleia2so if it's possible to work around it with the client, we can do that for now17:58
cineramapleia2: yep. does impact our testing a bit17:58
*** e0ne has joined #openstack-infra17:58
* pleia2 nods17:58
pleia2we should just document the "make it work" steps so everyone working on this knows how to do testing (particularly as we do reviews)17:58
pleia2cinerama: re: client + zanata.ini file, the issue is that you need access to the data in the zanata.ini on the server, but don't have it? (I assume for user auth or something?)17:59
*** sushilkm has joined #openstack-infra18:00
*** sushilkm has left #openstack-infra18:00
cineramapleia2: so the zanata.ini file is in place client side as it should be but the client whinges about not having the username, api key, and url, which are all in the ini18:00
* SpamapS looks at epic zuul queue and is reminded to do some work on documenting design of infra cloud.18:01
*** annegentle has quit IRC18:01
pleia2cinerama: hm, yeah, I'm not seeing a --config option either to point it at the .ini18:01
*** dkranz has joined #openstack-infra18:01
*** annegentle has joined #openstack-infra18:02
fungifor future reference: sudo apt-get install --no-install-recommends openjdk-7-jdk ; sudo -u gerrit2 jstack `cat ~gerrit2/review_site/logs/` ; sudo apt-get purge --auto-remove openjdk-7-jdk18:02
fungiredirect that middle command to files a few times 10 or so seconds apart (stdout and stderr)18:02
fungithat will get us some thread dumps with the thread names resolved correctly18:03
*** yamahata has quit IRC18:03
*** otter768 has joined #openstack-infra18:03
fungiclarkb: mordred: jeblair: pleia2: jhesketh: SergeyLukjanov: ^18:03
cineramapleia2: some of the subcommands seem to deal with finding that information & others don't. the ini file default location is where we put it in the client.pp18:04
SergeyLukjanovfungi, ack, interesting18:04
pleia2cinerama: yeah, on proposal it's in /home/jenkins/.config/zanata.ini18:04
cineramapleia2: so i think bug but...18:05
pleia2cinerama: does your zanata.ini have anything in it? it doesn't actually have the values populated on the proposal slave18:06
*** david-lyle has joined #openstack-infra18:07
*** johnthetubaguy is now known as zz_johnthetubagu18:07
*** dhritishikhar_ has quit IRC18:07
*** dhritishikhar_ has joined #openstack-infra18:07
*** otter768 has quit IRC18:08
*** isviridov_away is now known as isviridov18:11
*** eharney has quit IRC18:11
fungizaro: so looks like 0.14 is supposed to solve the stream-events hangs? i can confirm these are blocked in Object.wait() as described in
openstackgerritEmilien Macchi proposed openstack-infra/project-config: beaker: enable voting for puppet-(keystone/openstacklib)
fungi seems to have been when they imported 0.14.018:14
fungia couple weeks ago now18:14
EmilienMif some core-reviewers are around, we have two 'small' patches in infra that could help us to move forward with our CI: and
cineramapleia2: oops. i will look at the config for the proposal slave in a sec. we'll need to make an account that the client can should we do that?18:15
pleia2cinerama: in the long run for production, someone on infra should probably just create a jenkins user to interact with it in zanata, for now I think you can just add one on the dev server18:16
cineramapleia2: what email should we use for the dev jenkins user?18:17
annegentlefungi: hey, AJaeger and I want to pick your brain for ideas on sharing the FTP credentials for docs.openstack when you have a minute.18:17
pleia2cinerama: hm, does it need to do the auth process?18:18
annegentlefungi: sharing them so far has been me calling Jim. What can we do to expand to AJaeger and Lana incoming PTL?18:18
*** dangers is now known as dangers_away18:18
fungiannegentle: i can sort of multi-task. cloudsites lets you add multiple ftp accounts for one site, right?18:18
pleia2cinerama: actually, we can always change the email address later, just use one of yours for now18:18
cineramapleia2: it does. we can't currently use the local accounts at the same time as openid accounts18:18
annegentlefungi: oh! Hadn't thought of that but yes18:18
pleia2oh, it needs an openid account, hm18:18
annegentlefungi: and it really can be "when you have a minute" :)18:19
cineramapleia2: we would need to change zanata to do that. they did offer support if we wanted to contribute that iirc18:19
pleia2cinerama: to support both?18:19
fungiannegentle: though i suppose you need rackspace dashboard access to manage ftp accounts... so there's still likely a bottleneck even if we hand out individual ftp accounts18:20
cineramapleia2: yeah. they supply a login screen where you can choose from different openid providers but we'd need to whip up a login screen for the case where a specific openid provider is specified + using local accounts18:20
fungiannegentle: i'm _hopeful_ we'll be done with rackspace cloudsites hosting by the time the next ptl election rolls around, but that depends on how much time we get to put into implementation there18:20
pleia2cinerama: hm, does the client do openid?18:21
*** garyk has quit IRC18:21
*** SumitNaiksatam has joined #openstack-infra18:21
*** tonytan_brb has joined #openstack-infra18:22
*** ianw has quit IRC18:22
cineramapleia2: the client uses an API key which you generate on the web for that account. hmm. so we might not actually need to be able to login on the website. let me have a play for a sec18:23
annegentlefungi: jeblair said hopefully at the start of liberty so I'd love that18:23
*** tonytan4ever has quit IRC18:23
annegentlefungi: but yes, we should share these creds in the meantime18:23
annegentlefungi: and true on the dashboard / control panel access18:24
annegentlefungi: I can always do phone calls18:24
*** ayoung has quit IRC18:24
*** alexpilotti has quit IRC18:25
*** ianw has joined #openstack-infra18:25
cineramapleia2: heh. i don't think i can create a user on the web. i might be able to create a local user by messing with the database but i also can't manipulate other users' api keys from the web18:25
fungiannegentle: well, i mean phone calls are just one out-of-band communication mechanism. we could for example to openpgp-encrypted e-mails to hand off passwords if preferred18:26
zarofungi: i'm not sure about that.  IIRC that sshd fix happened, sshd released ver 0.14.0, gerrit updated to 0.14.0 but problem still occurs. i believe that's what happened, but would need to double check18:26
fungizaro: oh, the bug you pointed me at about mina-sshd 0.14.0 said that gerrit itself froze18:26
annegentlefungi: how much would it matter if Lana, Andreas and I haven't signed each other's keys? No biggie?18:26
*** ddieterly has joined #openstack-infra18:27
fungizaro: "the web-interface was dead" according to the initial message18:27
AJaeger_annegentle: my key was signed by fungi and others18:27
pleia2cinerama: hm, for some initial testing, can we use our own user accounts?18:27
annegentleAJaeger_: yeah18:27
annegentleto me, it's good enough18:27
fungiannegentle: no less secure than a phone call ;)18:27
cineramapleia2: yes18:27
annegentleright :)18:27
*** marun has joined #openstack-infra18:28
pleia2cinerama: I think we just document the issue we have now (what address+associated openstackid account to use, and who/how to generate the keys) and chat with the rest of the team when we need to go into prod18:28
fungiannegentle: "i think this is the right key and e-mail address" is probably at least as secure as "i think this is the right phone number and that sounds like her voice"18:28
pleia2cinerama: or at least get closer to prod :)18:28
annegentlefungi: heh18:29
cineramapleia2: we'll need to change system-config/modules/openstack_project/manifests/proposal_slave.pp to pass along the values18:29
*** jogo has quit IRC18:29
pleia2cinerama: yeah, probably something we'll need to add to hiera18:29
cineramapleia2: i will propose that in a sec18:30
zarofungi: maybe a slightly different problem than just ssh hang, not sure but looks like user definately confirmed some stability issues with updated sshd lib.18:30
pleia2cinerama: we just don't know what the values are yet until we discuss it with everyone :)18:30
cineramapleia2: yep but for dev we can at least chuck in our temporary values18:30
clarkbok I am back18:31
*** BharatK has quit IRC18:31
zarofungi: still not sure which path you wanted to take with testing on review-dev or review?18:31
fungizaro: yep, i agree that's a new risk and probably a worse issue, but not necessarily the same issue18:31
pleia2cinerama: well my hope is that once we get to actually using the proposal slave, we'll be closer to prod and have real values18:32
pleia2not sure we should be adding dev server values to the proposal slave18:32
cineramapleia2: so do you want to experiment with a local client for now to get things working, or just mod the zanata.ini on there by hand?18:32
zarofungi: maybe just 2 weeks out until upgrade so maybe just cherry-pick jeblair's change onto our 2.10.2 branch and test on review-dev?18:33
clarkbpleia2: cinerama ya we should be able to run the scripts by hand from an arbitrary location against the dev server18:33
*** ildikov has joined #openstack-infra18:33
pleia2cinerama: I think we leave the proposal slave alone until we're actually ready to go with most things18:33
fungizaro: that sounds reasonable18:33
openstackgerritMerged openstack-infra/shade: Search methods for networks, subnets and routers
*** yamahata has joined #openstack-infra18:33
clarkbpleia2: +118:33
openstackgerritMerged openstack-infra/shade: Use new getters in update_subnet and update_router
pleia2cinerama: there's nothing special about the proposal slave that we can't replicate in our local instances18:33
fungizaro: want to see if it applies cleanly and put up a change? i'll go ahead and approve it once the buck tests pass18:33
zarofungi: i'll do try that.18:33
*** ddieterly has quit IRC18:33
* zaro needs to head out for errands, bbl.18:34
fungizaro: at this point i'm hesitant to take offline long enough to replace the warfile between now and the kilo release18:34
cineramapleia2: just thoguht it might be easier for clarkb to give us a hand, etc18:34
fungiso waiting to have more debug info in 2.10 makes sense18:34
clarkbcinerama: I thnk as long as we are pushing the code someplace (a gerrit chagne) so we can all grab it and run it that should be fine18:34
*** sarob has quit IRC18:35
clarkbfungi: zaro catching up, latest 2.10 may not fix ssh problems?18:35
openstackgerritMerged openstack-infra/shade: Update volume API for new getters and dict retval
openstackgerritMerged openstack-infra/shade: Add test of OperatorCloud auth_type=None
clarkband we don't have jeblair's change deployed because gerrit won't start wiht that war?18:36
fungiclarkb: sounds like it's supposed to fix this problem but may introduce new and more exciting problems instead18:36
openstackgerritMerged openstack-infra/shade: Update register_machine to use tasks
openstackgerritMerged openstack-infra/shade: Update unregister_machine to use tasks
cineramawe need to be able to pass those values anyway so i'll at least put up the change that would be required18:36
clarkbthe best kind of update18:36
fungiclarkb: right, i couldn't get gerrit to start on review.o.o with the updated warfile for that debugging patch18:36
pleia2cinerama: ok, maybe just WIP it until we have the values to put into hiera?18:37
fungiclarkb: and if we upgrade via puppet we get a 10+ minute outage for reindexing to complete (which we could work around if we need to)18:37
*** dkranz has quit IRC18:37
cineramapleia2: sure18:38
*** dboik_ has joined #openstack-infra18:38
fungibut more just thinking that the number of restarts for stream-events hangs between now and release day are likely preferable to taking gerrit offline for a while to try and reapply that updated warfile18:38
fungigiven that it doesn't make these restarts unnecessary, just has a chance of allowing us to do some deeper debugging18:39
*** ddieterly has joined #openstack-infra18:39
clarkbfungi: ya18:39
fungiso in total, we'll most likely have more gerrit downtime between now and the release if we try that again, vs if we wait18:39
clarkbif it was a fix we should do it but its just debug info18:39
fungistab-in-the-dark debug info at that18:40
openstackgerritSamuel BARTEL proposed openstack-infra/project-config: add project fuel-plugin-nova-nfs
*** jogo has joined #openstack-infra18:40
pleia2cinerama: so I'm going to get myself up to speed with the client stuff, be prepared for stupid questions this afternoon as I make my way through :)18:40
*** yamahata has quit IRC18:40
*** samuelBartel has joined #openstack-infra18:41
*** dboik has quit IRC18:41
fungiclarkb: also there's a _chance_ that this new problem with mina-sshd 0.14.0 in gerrit 2.10.3 will be solved before time for us to upgrade18:41
cineramapleia2: np - in fact that will actually help a lot18:41
clarkbfungi: that would be nice18:41
clarkbfungi: zaro where is the link to the new gerrit ssh thing?18:41
fungithe discussion there is at least active and current18:41
*** isviridov is now known as isviridov_away18:42
*** yamamoto has quit IRC18:42
redrobotfungi I had a question about the changes you've been doing to the gerrit ACLs.  According to the Project Creator's Guide, we should have 2 gerrit groups barbican-core and barbican-release.  I see a third gropu barbican-milestone in gerrit, and I was wondering what the purpose of it was?18:42
AJaeger_redrobot: previously we used -milestone instead of -release18:43
AJaeger_so, it's the same18:43
fungiredrobot: you have the option of a separate group in charge of approving fixes proposed to milestones (generally just the proposed branch immediately prior to release now)18:43
*** Hal has joined #openstack-infra18:44
fungiredrobot: the release group was previously called "ptl" instead but confused people because it was mostly used for the ptl and their release delegates to be able to tag releases of libraries18:44
*** Hal is now known as Guest8923718:44
openstackgerritEmilien Macchi proposed openstack-infra/project-config: beaker: enable voting for puppet-(keystone/openstacklib)
EmilienMfungi: not sure I did correctly, let me know what you think18:45
funginow those milestone and release groups are probably mostly redundant in newer projects18:45
clarkbanything pressing I should look at? otherwise I am going to go back to trying to catch up on code review18:45
fungiredrobot: the "milestone" groups in older projects were at one time called "drivers" instead and were sync'd in from the groups of the same name in launchpad18:45
openstackgerritJoshua Harlow proposed openstack-infra/shade: Use a watch that isn't affected by time going backwards
*** ddieterly has quit IRC18:45
redrobotfungi I see...  so I can safely ignore xxx-milestone for now?  It doesn't seem to have any privileges in stable/kilo18:45
fungiredrobot: yeah, sounds like you probably won't have any need for it18:46
redrobotfungi right on, thanks!18:46
redrobotand thank you too AJaeger_18:46
*** eharney has joined #openstack-infra18:46
*** tonytan4ever has joined #openstack-infra18:48
fungiclarkb: i proposed the new repo creation change for the openstackid-resources api server split after discussing with smarcet:
*** ZZelle has quit IRC18:48
fungithat's a quickie review18:48
*** ZZelle has joined #openstack-infra18:49
*** dkranz has joined #openstack-infra18:49
*** thedodd has joined #openstack-infra18:49
fungitrying to encourage tipit to start new projects in gerrit from the beginning (he seems confident that he's going to write it from scratch rather than the other option we discussed of forking the openstackid codebase)18:49
*** tonytan_brb has quit IRC18:50
*** stevemar has joined #openstack-infra18:50
clarkbcool will review now18:51
clarkbiirc TC approved the change letting us all make projects without all the oversight in the world18:51
fungiyep. i'm about to do a couple cosmetic fixes to jeblair's corresponding infra-manual patch and then we can approve it as well i think18:52
fungibut i basically followed the new process from that18:52
fungiclarkb: also outreachy ideas brainstorming is happening in
clarkboh I had a couple ideas to add18:53
*** hdd has quit IRC18:53
fungiplease do. i haven't had a sec to try to come up with any, but the sooner we get up with vkmc on this stuff the more likely we are to have an intern18:53
*** ddieterly has joined #openstack-infra18:54
*** hdd has joined #openstack-infra18:54
clarkbok added, back to reviews18:56
stevemarfungi, clarkb qq for you guys, someone i know wants to do research on openstack reviews and such, is there any data available or would he have to skim everything?18:57
*** a13x2121 has quit IRC18:57
samueldmqstevemar, just the metrics for someone/some org ?18:57
clarkbstevemar: gerrit has a rest api18:57
fungistevemar: there are some pretty comprehensive apis to gerrit if he's not adverse to scripting something to query and distill what he needs18:57
fungistevemar: there's also and with some already baked analysis18:58
samueldmq^ I have used stackalytics api18:58
*** ddieterly has quit IRC18:58
stevemarwill performing a bunch of actions on the REST apis be painful?18:58
fungistevemar: nope. i have some scripts that query a ton of stuff via the gerrit rest api18:59
*** dizquierdo has quit IRC18:59
stevemarawesome, i let him know about the -infra channel18:59
stevemarcool cool18:59
fungistevemar: also if you've ever used gertty, it operates _entirely_ via gerrit's rest api, which is why i say it's comprehensive18:59
stevemarthanks guys :)18:59
stevemarvery preliminary discussion18:59
*** dizquierdo has joined #openstack-infra19:00
fungianyone with an interest in our development process and tools is welcome19:00
fungii love talking with people about this stuff19:00
*** jamesmcarthur has joined #openstack-infra19:00
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove YamlParser from jenkins_jobs.builder
clarkbstevemar: also you can grab all the git repos to do munging of git info19:01
clarkbso lots of options here, likely depends on what perspective you are looking for19:01
*** dizquierdo has quit IRC19:02
*** shashankhegde has quit IRC19:02
*** pblaho_ has joined #openstack-infra19:02
greghaynesmordred: looks like the big nodepool + shade patch stack now passes after rechecks19:02
*** dizquierdo has joined #openstack-infra19:02
greghaynes(and FYI for any other potential reviewers :))19:03
clarkbgreghaynes: you should recheck the children too?19:03
greghaynesI did19:03
clarkboh nevermind gertty sync lag19:03
greghaynesmuch rechecking was had19:04
mordredgreghaynes: woot19:04
stevemarclarkb, fungi just talking to a researcher/grad student about CI/CD, he did a paper on a smaller project, but something on openstack would be awesome19:04
mordredgreghaynes: I'm very excited about that19:04
* clarkb notes that none of the shade changes brought up yesterday are using the priority topic19:04
mordredstevemar: make sure he's aware of our graphite server as well19:04
*** dtantsur|afk has quit IRC19:04
clarkbso if people think they are required for nodepool + shade then that should be updated19:05
mordredclarkb: I think they're not actually essential to dib-nodepool19:05
fungistevemar: yes, i think we're probably the largest free/open ci/cd operation of all time, so an excellent research target19:05
stevemarmordred, whosa whatsa graphite server?19:05
mordredthey're more essential to fleshing out shade and making the ansible playbooks work19:05
clarkbmordred: ok, thats what was being argued, I agree19:05
fungistevemar: (it's what we use to generate the graphs at the bottom of the zuul status page among other things)19:05
openstackgerritMerged openstack-infra/shade: Test that deleting user invalidates user cache
mordredclarkb: they're important - but many of them are not directly related to nodepool consumption19:05
clarkbmordred: roger19:05
*** pblaho has quit IRC19:05
fungistevemar: statsd target and introspection interface19:06
clarkbstevemar: we shove timing data and counters in there for our tests and things then graphite has an api to do maths on them19:06
mordredSpamapS, Shrews: ^^ that said - with the interface update we're perpetrating right now ... we should get that applied to images and servers before we land nodepool patches that use those apis19:06
clarkbmordred: does that mean 168603 should be WIP?19:06
stevemarthat is pretty crazy19:06
fungistevemar: right now it's mostly storing performance, timing and counts from zuul activities, and some from nodepool as well19:06
mordredclarkb: no - that one is good to go19:07
mordredclarkb: that just uses the client passthrough objects - no interface changes happening there19:07
*** a13x212 has joined #openstack-infra19:07
*** BobH1 has joined #openstack-infra19:07
clarkbalso why are we changing interfaces after we made it stable for ansible?19:07
mordredclarkb: the ansible patches have not yet landed - and in working on those we found issues in the interface that we need to fix to provide sane ansible results19:08
fungistevemar: there's also an anonymously accessible mysql database we're sticking subunit trending information in from individual tests (via mtreinish's subunit2sql project) which can be mined for all sorts of job performance data19:08
*** Guest89237 has quit IRC19:08
mordredclarkb: basically, we had inconsistent semantics around get, list and search19:08
fungistevemar: and where we have a kibana front-end with more recent job logs for high-detail investigation19:09
clarkbmordred: oh I thought ansible stuff had landed19:09
Shrewsmordred: they're on the roadmap19:09
mordrednot yet19:09
openstackgerritMerged openstack-infra/shade: create_user should return the user created
mordredShrews, clarkb: and actually - I just checked - we have no nodepool patches up yet that would be affected19:09
*** zz_zz_ja is now known as zz_zz_zz_ja19:09
*** pblaho__ has joined #openstack-infra19:10
mordredShrews: this: is the only thing we have up that might want thought - I'm goign to WIP it real quick19:10
openstackgerritMerged openstack-infra/shade: Fix major update_user issues
stevemarmordred, fungi, this is all wonderful info. thanks guys19:10
*** pblaho_ has quit IRC19:10
*** dtantsur has joined #openstack-infra19:10
mtreinishfungi: there's a subunit2sql command for graphing test run_time over time :)19:11
* Shrews looks19:11
*** sarob has joined #openstack-infra19:11
mtreinishfungi: I actually just pushed some patches up to make it a bit more useful:,n,z19:11
mordredShrews: get_flavor_by_ram() seems to be a good candidate for the new search interface ... EXCEPT - it's a min ram number that gets passed in, so it still may want to be special in some ways19:12
fungioh, neat19:12
*** Hal has joined #openstack-infra19:12
*** sarob has quit IRC19:12
*** Hal is now known as Guest5353419:12
mtreinishwell really only the first 2 in that series are for improving the run time graphing19:13
mtreinishthe rest is for graphing/tracking test failure rates over time19:13
Shrewsmordred: i think we can keep get_flavor_by_ram() as it can self filter after calling list. a get_flavor() method should be made available, though, if we do not have it19:13
*** ArxCruz has joined #openstack-infra19:13
*** Guest53534 has quit IRC19:13
mordredfungi, clarkb, jhesketh: thoughts/opinions on i18n support in shade? since it's intended to be a friendly end user library - it might perhaps want to support translated strings for error messages and such?19:14
openstackgerritStephanie Miller proposed openstack-infra/system-config: WIP: Pass through zanata client parameters
*** annegentle has quit IRC19:14
mordredfungi, clarkb, jhesketh: the question arose as I was looking in to what gets dragged in by oslo.utils - because harlowja submitted a patch to change how elapsed time was calculated that uses timeutils in oslo.utils19:14
*** claudiub has joined #openstack-infra19:15
harlowjaafaik oslo.utils already getting dragged in by all the clients :-P19:15
openstackgerritJoe Gordon proposed openstack-infra/elastic-recheck: Generalize fingerprint for 1446847
clarkbmordred: these opinions are my own. I think it is a mistake to drag in half of oslo*. stdlib works great19:15
SpamapSIt's possible the whole infra toolchain would benefit from i18n to ease adoption for 3rd-party CI vendors who are not primarily english speaking.19:15
SpamapSharlowja: the clients are not going to be depended on in the mid-term future.19:15
openstackgerritStephanie Miller proposed openstack-infra/system-config: WIP: Pass through zanata client parameters
clarkbmordred: but i8ln is likely a good idea since shade is making its own error messages19:15
*** stevemar has quit IRC19:15
mordredclarkb: well, in this case, apparently there is a real issue in the stdlib that timeutils addresses19:15
Shrewsmordred: a bigger issue is that we still return image and server objects without making them dicts. that will break many things, i imagine19:15
greghaynesclarkb: there is a specific issue here that stdlib has19:15
mordredclarkb: which is also addressed by a pep targetted at python3.3 and later19:16
harlowjaSpamapS ok, so oslo.utils is trying to stay mostly minimalish19:16
clarkbgreghaynes: mordred can somone educate me then?19:16
harlowjasee ^19:16
SpamapSharlowja: I think it definitely has value in the waiting loops.19:16
clarkbharlowja: thats not very minimal19:16
greghaynes is the patch19:16
harlowjaclarkb ya, can't do much about that :-P19:16
harlowjaSpamapS is probably also useful for shade stuff19:17
clarkbmordred: that pep doesn't help me19:17
harlowjasince it looks like it has its own little task manager + future like thing19:17
openstackgerritJeremy Stanley proposed openstack-infra/infra-manual: Reverse dependency relationship in project creation
clarkbunder what circumstances does time.time go backwards19:17
SpamapSugh, netifaces19:17
clarkband under what circumstances is that actually a problem19:17
clarkbSpamapS: yup19:17
mordredclarkb: if you're calculating timeouts19:17
greghaynestime change of any kind, basically19:17
mordredclarkb: as we do19:17
mordredand the time changes for some reason19:17
clarkbmordred: ya if you do the math right it should still just work19:17
mordredclarkb: apparently there are cases when not19:18
*** david-lyle has quit IRC19:18
harlowjathe stop watch class from oslo.utils that i sucked over from taskflow should handle the cases for u19:18
clarkbmordred: no there are cases time.time can go backwards19:18
SpamapSntpd, btw, is not allowed to make the clock go backwards.19:18
harlowjabut up to u guys :-P19:18
SpamapSit _is_ allowed to make it stop.19:18
clarkbmordred: but that doesn't mean your timeout will fail19:18
*** annegentle has joined #openstack-infra19:18
harlowja (read at your own will, lol)19:18
clarkbmordred: your timeout may take more or less time19:18
clarkbmordred: but it should still timeout19:19
SpamapSSo you can see the same second for a few milliseconds, but ntpd won't adjust time backward fast _at all_.19:19
lifelessSpamapS: uhm, what ? ntp disagrees with you19:19
SpamapSlifeless: NTP, or NTPD?19:19
lifeless'This may on occasion cause the clock to be set backwards if the local clock time is more than 128 s in the future relative to the server. In some applications, this behavior may be unacceptable. If the -x option is included on the command line, the clock will never be stepped and only slew corrections will be used.'19:19
clarkbharlowja: right so my argument is that time going backwards isn't an issue19:19
harlowjaup to u guys and how much u care19:19
openstackgerritJames Slagle proposed openstack-infra/reviewstats: Sync TripleO projects.
SpamapSlifeless: it's been a while since I slogged through that man page. Ugh.19:20
clarkbharlowja: usually you say time_now = time.time() ; then later if time_now + timeout > time.time() then raise timeout19:20
openstackgerritMerged openstack-infra/project-config: Add the openstackid-resources repo
greghaynessure, so even more generally, the time is only used here in an output message19:20
lifelessSpamapS: I'm sorry :)19:20
clarkbharlowja: ^ will function if time goes back in time19:20
greghaynesso it doesnt matter in that sense19:20
SpamapSlifeless: assumption broken. Thanks. :)19:20
greghayneseven if its negative19:20
freerunnerHi there! Folks, review please .19:20
clarkbharlowja: that is not worth a super heavyweight dep imo19:20
harlowjaok, feel free to move that stopwatch stuff to another pypi library if u want :-P19:21
harlowjai just put it there,  but it could be anywhere19:21
clarkbharlowja: no I am saying just don't worry about it19:21
clarkbyou don't need a lib19:21
clarkbjust need smart  code19:21
greghayneswell, thats the point of libs in general19:21
harlowjai prefer smart lib, and dumb code, lol19:21
harlowjabut to each there own :-P19:22
greghaynesbut yes, its probably overkill here19:22
SpamapS is quite good btw. :)19:22
*** achanda has quit IRC19:22
*** hasharAway is now known as hashar19:23
openstackgerritJoe Gordon proposed openstack-infra/elastic-recheck: Generalize fingerprint for 1446847
*** jaypipes has quit IRC19:24
SpamapSclarkb: to harlowja's point, This is a generic problem that can be gotten wrong.19:24
SpamapSclarkb: also in your example, the timeout is not waited for if time goes backward.19:25
SpamapSclarkb: which means a clock correction can lead to spurious errors.19:25
clarkbSpamapS: yes you may timeout early19:25
clarkbSpamapS: but thats a non issue19:25
sdaguedid we lose a cloud again?19:26
clarkbSpamapS: because your code should already be time out tolerant19:26
clarkbSpamapS: if it isnt then *that* is the bug19:26
clarkbsdague: I turned off hpcloud again yesterday as its presence was making rax less performant19:26
SpamapSI think I could argue that spurious errors are an issue. But this is a seldom-hit issue. :)19:26
*** ddieterly has joined #openstack-infra19:26
sdagueclarkb: hmmm... ok?19:27
clarkbsdague: basically hpcloud wasn't booting nodes, but nodepool would still schedule half its workload to it19:27
sdagueso now we're just in backlog mode19:27
clarkbsdague: so we were running at half rax capacity + 0 hpcloud19:28
clarkbsdague: by turning off hpcloud we get 100% rax19:28
SpamapSclarkb: in shade's case, you're going to tolerate the timeout, but waste a lot of time and resources deleting and re-creating the server. BUT, I want to stress.. the error rate here, where nodepool's clock will go backwards, should be about once per ever.19:28
sdagueI'm surprised to see us waiting > 1 hr for pep8 nodes19:28
clarkbSpamapS: yes exactly19:28
mordredSpamapS: yup. if we timeout a second early once ever it's fine19:28
clarkbSpamapS: and when it does happen everything will work fine19:28
clarkbeven if it happend 100 times per day it would still be fine19:29
harlowjaSpamapS do we really know how often time goes backwards in nodepool (or vm(s)); that'd be something neat to know, cause I thought vm's are especially affected by clock-shifts (due to hypervisor load...)19:29
clarkbharlowja: hypervisor load should only shift time forward right?19:29
harlowjabusy neighbor, blah blah19:29
harlowjai'd hope so, but einstien and i haven't chatted recently19:29
clarkbif my cpu can't process the clock that doesn't make the clock go backwards19:30
clarkbbut it may jump ahead when I get cpu cycles19:30
harlowjais this involving gravity and/or spacetime?19:30
mordredclarkb: dunno man. don't forget systemd exists19:30
*** sandywalsh has quit IRC19:30
*** sushilkm has joined #openstack-infra19:30
mordredclarkb: I can no longer predict anything that happens on a linux system, because systemd may just decide to, youknow, lennart something19:30
*** sushilkm has left #openstack-infra19:30
SpamapSI like how systemd is basically the new thing that can cause anything to happen.19:30
SpamapSharlowja: As clarkb says, vms are known for skipping seconds, but not for repeating them.19:31
*** dizquierdo has quit IRC19:31
*** sandywalsh has joined #openstack-infra19:31
*** achanda has joined #openstack-infra19:31
SpamapSharlowja: It's a corner case. A real inconsistency, that is interesting, but I think not a priority.19:31
harlowjasure, i hope so, i'm not really sure since never ran much extensive testing of this19:31
clarkbsdague: I think some of that is the gerrit event stream died again too19:32
harlowjaSpamapS sure; take the review or not, either is fine with me :-P19:32
devanandaclock can go backwards if ntpd screws up, or is misconfigured19:32
harlowjai just work here, lol19:32
SpamapSIt goes up there with race conditions as something we'd love to have a real way to test, but since the matrix of inputs is every second that a test runs for...19:32
devanandathough ntpd _should_ merely slow a clock down (make seconds slightly longer) when it detects that it is skewed ahead19:32
SpamapSdevananda: you're a little late, we established that it's an unlikely problem to have. :)19:33
*** sarob has joined #openstack-infra19:33
devanandaSpamapS: cool. yah, i haven't read all the backscroll19:33
sdagueclarkb: ah19:33
SpamapSnot impossible, just unlikely.19:33
sdagueis that going to recover, or are we in lost job land?19:33
SpamapSAnd TIL that ntpd _is_ allowed to go backwards, just... very unlikely to do so.19:33
*** ayoung has joined #openstack-infra19:34
clarkbsdague: any jobs that did start should be fine19:34
clarkbsdague: s/start/queue/19:34
clarkbsdague: we may have missed starting jobs for changes during the time gerrit wasn't emitting events, I think likely many got rechecked resulting in the backlog19:34
openstackgerritEmilien Macchi proposed openstack-infra/project-config: beaker: enable voting for puppet-(keystone/openstacklib)
*** ddieterly has quit IRC19:36
*** jamesmcarthur has left #openstack-infra19:39
*** samuelBartel has quit IRC19:39
*** shakamunyi has joined #openstack-infra19:40
clarkbsdague: I have confirmed queued jobs are still starting, so we haven't stopped the system again for some reason19:40
sdagueclarkb: ok, no worries19:41
*** david-lyle has joined #openstack-infra19:41
clarkbmordred: greghaynes in 168603 why do we keep a global openstack cloud client?19:42
clarkbmordred: greghaynes that means we will use one client per test process which may leak state19:42
mordredclarkb: one sec19:43
clarkboh hrm I guess the old code did the same19:43
clarkbso meh19:43
mordredclarkb: yah.19:43
*** mmedvede has quit IRC19:43
greghaynesclarkb: Yes, its a fake - they need to share state19:43
mordredclarkb: minimal changes19:43
clarkbgreghaynes: you only need shared state per test not per test runner19:43
clarkbbut this is how the old code worked so meh19:43
*** sarob_ has joined #openstack-infra19:44
*** stevemar has joined #openstack-infra19:44
*** akshai has joined #openstack-infra19:45
*** ddieterly has joined #openstack-infra19:45
greghaynesclarkb: yea, not sure how to punch that through though? I guess in the test cleanup reset the client?19:46
clarkbgreghaynes: a19:46
clarkbbut not a problem I just didn't realize the old code did this too until I had scrolled further19:46
greghayneswell, I dont mind fixing it19:47
greghaynesill do it in followup19:47
*** sarob has quit IRC19:47
*** sarob_ is now known as sarob19:47
openstackgerritMerged openstack-infra/shade: Test flavor cache and add invalidation
openstackgerritMerged openstack-infra/shade: Add test for caching in list_images
*** mrunge has joined #openstack-infra19:48
*** mrunge has joined #openstack-infra19:48
*** mrmartin has joined #openstack-infra19:48
*** ZZelle_ has joined #openstack-infra19:49
*** ddieterly has quit IRC19:50
*** dkranz has quit IRC19:50
*** akshai has quit IRC19:50
EmilienMAJaeger: so you think doc-team mays be also interested by the backport-potential commit flag ?19:51
*** annegentle has quit IRC19:52
*** ociuhandu has quit IRC19:52
*** dangers_away is now known as dangers19:52
openstackgerritMerged openstack-infra/shade: Explicitly request cloud name in test_caching
openstackgerritJulia Kreger proposed openstack-infra/shade: Add update_machine method
*** openstackgerrit has quit IRC19:54
*** openstackgerrit has joined #openstack-infra19:55
*** mmedvede has joined #openstack-infra19:55
AJaeger_EmilienM: Yeah, see
EmilienMAJaeger_: cool. Though jogo is bring some thoughts about the 'automatic' thing, that will bring conflict issues...19:56
jogoEmilienM: yeah, cherry-picks need humans eyes19:57
jogowe already have a process for this I thaught19:57
*** ibiris is now known as ibiris_away19:57
EmilienMjogo: most of the puppet patches we backport are without conflict19:57
jogotagging bugs in launchpad19:57
EmilienM+1 for LP19:58
EmilienMAJaeger_: do you think we can go ahead with ?19:58
EmilienMjogo: what do you propose then?19:58
jogoEmilienM: just using LP19:58
AJaeger_jogo, EmilienM for openstack-manuals we have a few directories where we backport nearly everything - and there such an automatic tool would really be great19:58
*** radez is now known as radez_g0n319:58
jogoand having people go over the list of things marked as backport19:59
jogoit wouldn't be hard to make a tool to query LP for things to backport19:59
AJaeger_if you backport 3 patches in a different order than they were original, you might have conflicts19:59
*** subscope_ has quit IRC19:59
jogoand try to do so locally so a human can resolve conflicts manually19:59
*** Rockyg has joined #openstack-infra19:59
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for create_image with glance v1
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Add tests and invalidation for glance v2 upload
jogoAJaeger_ EmilienM: so I propose tools that help humans do this faster20:00
clarkbmordred: greghaynes one small thing on 16860320:00
SpamapSgreghaynes: ^ issues fixed20:00
SpamapSmordred: ^20:00
jogonot automated 100%20:00
AJaeger_EmilienM: 176070 has a very long line...20:00
EmilienMAJaeger_: is it an issue?20:00
openstackgerritMichael Krotscheck proposed openstack/requirements: Updated oslo.config to 1.11.0
AJaeger_jogo: we have a script that does this manually - just pass it the review and it backports and let's you review manually...20:00
mordredSpamapS: woot20:01
mordredclarkb: on it20:01
AJaeger_EmilienM: no, I'm not happy but I'll approve ;)20:01
jogoAJaeger_: 'git review -X' does that20:01
*** dimtruck is now known as zz_dimtruck20:01
jogoAJaeger_: but you can hook that up to launchpad to find the list of things to backport as well20:01
AJaeger_jogo: yes, the script uses git review -X20:02
openstackgerritMerged openstack-infra/project-config: Add third party CI working group tools repository
AJaeger_jogo, that's what we basically use - hooking into launchpad sounds nice20:03
fungiAJaeger_: i'm working on the new patch to attempt to address your consistency concern with conflation of "project" vs "repository" in the infra manual. it's daunting. you'll see in a bit if i manage to finish it20:03
mordredclarkb, greghaynes: I think we should do both - we should have one cloud with cloud and no auth info, one with cloud and auth info, and one with no cloud ... and we should test that the values you get back via shade/os-client-config match our expectations in each case20:04
jogoAJaeger_: thankfully LP has a decent REST API20:04
clarkbfungi: 171286 has my +2 and I rechecked it to get good test results post pycparser fix20:04
*** tqtran is now known as tqtran_afk20:04
clarkbmordred: +120:04
*** otter768 has joined #openstack-infra20:04
AJaeger_fungi: Wow! I agree with you that this should be separate patch - that's why I didn't -1 jeblair's patch.20:04
openstackgerritJoshua Harlow proposed openstack-infra/shade: Have the task class actually wait for results to be set
greghaynesmordred: sounds good20:04
jogogreghaynes: what ever happened to the wheel mirror?20:05
greghaynesjogo: sitting for reviews forever20:05
greghaynesstill there20:05
greghayneshad to rebase a bit ago20:05
clarkbI reviewed it :)20:05
*** ildikov has quit IRC20:05
AJaeger_fungi: looking forward to that, thanks!20:05
fungiclarkb: thanks!20:05
jogofungi: want to review it too?20:05
*** patrickeast has joined #openstack-infra20:06
greghaynesit being:,n,z20:06
fungijogo: review what?20:06
openstackgerritMerged openstack-infra/system-config: New nodes should install puppet from
jogofungi: greghaynes wheel mirror of goodness20:07
*** shashankhegde has joined #openstack-infra20:07
fungioh, yes i do want to review it. let's see if i manage to find time to review it20:07
*** zz_dimtruck is now known as dimtruck20:08
*** dangers is now known as dangers_away20:08
*** dangers_away is now known as dangers20:08
jogofungi: thanks20:09
*** otter768 has quit IRC20:09
clarkbmrmartin: good evening20:10
mrmartinfungi, I've question on this askbot-staging. So I had a quite nice progress today with the puppet scripts20:10
mrmartinbut how could we handle the situation when the puppet-askbot manifests breaks the compatibility with previous version, and we still need to operate the ask.o.o based on existing ones?20:11
mrmartincan we pinpoint ask.o.o's puppet module to the actual version somehow?20:11
*** sputnik13 has joined #openstack-infra20:11
mrmartinor just simply shutdown the puppet on that node until resolve all dependency issues?20:11
*** e0ne has quit IRC20:12
*** achanda has quit IRC20:12
*** teran has quit IRC20:13
clarkbmrmartin: we cannot, we would need to be backward and forward compatible20:14
*** sabeen3 has quit IRC20:14
clarkbmrmartin: it seems to work for the most part at least with jenkins-dev, etherpad-dev, review-dev and so on20:14
*** agentle_ has joined #openstack-infra20:14
*** annegentle has joined #openstack-infra20:14
clarkbenforces a bit of discipline that is good to have20:14
*** fawadkhaliq has quit IRC20:15
mrmartinI have a major issue with the deployment model, actually we are deploying from pip which brings in all of the requirements20:15
*** sabeen has joined #openstack-infra20:15
mrmartinbut if I checking out the project directly from github repo, what is the proper way to pull the missing deps? a, deploy the packages from ubuntu repo b, do a pip install -r requirements.txt ?20:16
clarkbmrmartin: if you want greater control of the dependencies you can install them all first then install askbot20:16
clarkbmrmartin: pip install /path/to/project should work20:16
mrmartinso typically, if a new askbot version brings in a different dependency in requirements.txt, the deploy can brake too20:17
openstackgerritMerged openstack-infra/project-config: Add puppet stackforge modules to tripleo jobs
*** HeOS has quit IRC20:18
clarkbmrmartin: yes20:19
clarkbmrmartin: but we want to know that so I am not sure we would change it?20:20
*** ildikov has joined #openstack-infra20:20
mrmartinok, deployment into a virtualenv can solve this new dependency problem?20:20
clarkbmrmartin: no, because the deps can still chnge within the virtualenv and break the deployment20:20
clarkbmrmartin: the virtualenv can isolate it, but not prevent it20:21
mrmartinand here comes the blue-green model :D20:21
mrmartindeploying into different venv slots?20:21
clarkbmrmartin: so I don't think this is a problem20:21
clarkbmrmartin: if the staging server breaks that is why it exists20:21
mrmartinI try to not over-engineer it20:21
clarkbmrmartin: then we only update the production server at known good points in staging20:21
mrmartinI guess some deps already missing from requirements.txt20:22
mrmartinok, great, thnx, I'll play with that tomorrow20:23
clarkbheads up system-config core, I think there are at least two bugs in the apply test that may make our apply integration test not quite an integration test20:23
clarkbI need to wrap my head around it but will share shortly20:24
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: networking-nec: add ability to create and manage branches
clarkb(but be careful approving module changes)20:24
*** rlandy has quit IRC20:25
*** ldnunes_ has joined #openstack-infra20:26
openstackgerritJeremy Stanley proposed openstack-infra/infra-manual: Reverse dependency relationship in project creation
openstackgerritJeremy Stanley proposed openstack-infra/infra-manual: Project does not mean repository? Inconceivable!
fungiAJaeger_: ^20:27
AJaeger_fungi: nice subject ;)20:28
fungiAJaeger_: i'm not personally convinced that change makes the guides more readable, but it's at least a suitable straw-man for discussion20:28
*** annegentle has quit IRC20:28
*** agentle_ has quit IRC20:28
clarkbok one bug is not a bug thats good20:29
fungiproblem is that even though the tc has decreed "project" means something different now, most of our tools and basically all our config files still use "project" to mean "repository"20:29
*** annegentle has joined #openstack-infra20:29
clarkb(I thought integration modules may have been overridden by after zuul-cloner runs but puppetdb is not in integration modules and the integration modules logic looks correct)20:29
fungithis is why changing the meaning of terms in use is so much more painful than picking new terms not yet in use20:29
waynrdang reviews are taking quite a while today20:30
*** yamamoto has joined #openstack-infra20:31
fungiwaynr: yep, zuul has about 800 jobs waiting on available workers at the moment
clarkbthe other issue is we can't have blank lines inside of a node def due to how our csplit works20:32
fungiwaynr: assuming by reviews you mean check results20:32
clarkbso the openstack_project::server decoupling is tripping over that in a few places20:32
*** sputnik13 has quit IRC20:32
waynryeah that's what i meant20:32
clarkband this is going to require better parsing than a regex to fix20:32
AJaeger_fungi, that's a significant change, I'll review tomorrow closely20:32
*** sandywalsh has quit IRC20:33
clarkbunless I make some other assumptions about the code which I acutally probably can maybe do20:33
*** sandywalsh has joined #openstack-infra20:34
fungiAJaeger_: i tried to clarify "project team" in some places and use "package" in other places where relevant, and use "repository" when talking about actual git repositories, but there are still many, many grey areas20:34
*** sputnik13 has joined #openstack-infra20:34
AJaeger_fungi: yeah, noticed - and that's when I decided I need to review this slowly ;)20:35
*** dannywilson has joined #openstack-infra20:35
AJaeger_It's not enough to review the change, this needs rewriting of the whole documents20:35
*** yamamoto has quit IRC20:35
fungiyes. yes it does20:36
AJaeger_enhancing your commit message with the above comment about these three words would help others to understand this better20:36
fungipreviously "project" potentially meant multiple things depending on the surrounding context20:36
AJaeger_I meant "reading again"20:36
fungican do20:37
AJaeger_fungi: yeah, project had multiple meanings indeed.20:37
AJaeger_thanks, fungi20:37
*** sambetts has quit IRC20:38
*** sambetts has joined #openstack-infra20:39
*** stevemar has quit IRC20:41
*** rbradfor has quit IRC20:41
sdagueclarkb / fungi - could use review - this adds the stable/kilo branch support, and Depends-On a series of fixes we needed that were discovered by trying to do that the first time20:42
*** hdd has quit IRC20:42
openstackgerritJeremy Stanley proposed openstack-infra/infra-manual: Project does not mean repository? Inconceivable!
fungiAJaeger_: updated20:42
*** dannywilson has quit IRC20:42
fungisdague: awesome. looking now20:43
*** dannywilson has joined #openstack-infra20:43
AJaeger_fungi, great!20:43
*** stevemar has joined #openstack-infra20:45
sdagueand we're holding requirements unfreeze until lands so that we're not doing juno -> liberty upgrades with new requirements20:45
pleia2cinerama: meant to ask you the other day, did you ever create an issue in the zanata tracker for project creation stuff? (bugzilla makes my brain hurt, cannot find!)20:46
*** ArxCruz has quit IRC20:46
*** pc_m has quit IRC20:46
*** mrmartin has quit IRC20:46
*** luqas has joined #openstack-infra20:47
cineramapleia2: oh to the effect that there's no configurability of roles so everyone can create a project?20:47
pleia2cinerama: that's the one20:47
pleia2saw mention of it when I was browsing the logs yesterday20:48
cineramapleia2: no - shall i do that now or would you like to do the honours?20:48
pleia2I'm honestly not sure what to do if we can't restrict that, just tell people to be good? :)20:48
pleia2cinerama: you should go for it20:48
cineramapleia2: i kind of got sidetracked dealing with some of the client issues20:48
pleia2cinerama: understandable :) just trying to close loops here20:48
*** ArxCruz has joined #openstack-infra20:49
*** AJaeger_ has quit IRC20:49
cineramapleia2: oh that's right, i have to create an account for that bugzilla and then file. that's probably how it went in my "ok, later" bin20:50
pleia2it's so nice that the interface auto-generates your zanata.ini for the client <320:51
*** luqas has quit IRC20:51
clarkbsdague: will look in just a moment, I have a system-config bug fix I need to push up20:52
*** achanda has joined #openstack-infra20:53
sdagueclarkb: no prob20:54
openstackgerritClark Boylan proposed openstack-infra/system-config: Allow blank lines in node{} with apply test
clarkbfungi: pleia2 ^ that is the bug, its still not perfect but should be a bit more flexible now20:55
clarkbnow I need to -1 some changes and have them rebase on htat20:55
*** hdd has joined #openstack-infra20:55
*** dboik_ has quit IRC20:56
fungioh that's a fun one to have to track down20:57
fungihave we let any other bugs slip in because of that?20:58
*** david-lyle_ has joined #openstack-infra20:58
fungior is the integration test working already with that patch?20:59
clarkbfungi: nothing is broken now but a bunch of proposed changes break without it20:59
sdaguealso, clarkb / fungi / pleia2 - is part of moving trove support out of devstack and into it's own plugin20:59
clarkbfungi: I was debugging why jenkins -1'd some of those new changes which haven't merged and discovered this20:59
*** david-lyle has quit IRC20:59
fungigot it20:59
*** adiantum has quit IRC20:59
clarkbfungi: so I think we are ok for now and am -1'ing things suggestiong they rebase on that fix20:59
pleia2clarkb: seems more perfect than splitting on blank lines20:59
clarkbpleia2: ya its still naieve but less so21:00
* pleia2 nods21:00
*** mattfarina has quit IRC21:01
clarkbI ran the string munging locally and it looks correct but I didn't run the full test fwiw21:01
pleia2k, I'll give it a go21:01
*** ldnunes_ has quit IRC21:01
openstackgerritSean Dague proposed openstack-infra/project-config: Create jobs for a wheel mirror
openstackgerritMerged openstack-infra/project-config: add project fuel-plugin-nova-nfs
*** baoli has quit IRC21:03
*** tqtran has joined #openstack-infra21:05
cineramapleia2: i'll be here for the next hour or so & then i have to go pick up a family member, so if you have any other questions for me, best to ask now21:06
openstackgerritSean Dague proposed openstack-infra/devstack-gate: only force off KEYSTONE_USE_WSGI up until kilo
cineramapleia2: in the meantime i have a few bugs to file with zanata21:06
openstackgerritMerged openstack-infra/project-config: Move tripleo-puppet-elements to tripleo-core
*** kgiusti has left #openstack-infra21:07
*** dizquierdo has joined #openstack-infra21:08
*** ivar-laz_ has joined #openstack-infra21:08
clarkbsdague: I have approved 175391 though it may not gate right away due to the depends on21:08
clarkbsdague: happy to push it again later21:08
sdagueclarkb: cool, thanks21:09
*** tqtran has quit IRC21:09
openstackgerritMerged openstack-infra/project-config: Add diskimage-builder functests
*** erikmwilson has quit IRC21:11
clarkbgreghaynes: ^21:11
*** ivar-lazzaro has quit IRC21:11
*** dboik has joined #openstack-infra21:12
*** teran has joined #openstack-infra21:13
*** dprince has quit IRC21:13
*** stevemar has quit IRC21:13
pleia2cinerama: noted, I'll let you know if I come across anything in the next hour :)21:15
*** david-lyle_ has quit IRC21:15
*** david-lyle has joined #openstack-infra21:15
openstackgerritMerged openstack-infra/reviewstats: Sync TripleO projects.
*** mrunge has quit IRC21:17
*** ldnunes has quit IRC21:17
openstackgerritMerged openstack-infra/elastic-recheck: Generalize fingerprint for 1446847
*** gyee has quit IRC21:20
*** HeOS has joined #openstack-infra21:20
*** david-lyle has quit IRC21:21
*** sarob has quit IRC21:25
*** gyee has joined #openstack-infra21:26
lifelessmordred: ping - tchaypo and i discussed, unless you see something more, going to close21:28
openstackLaunchpad bug 1446425 in PBR "pip only respects 1 version specifier for any particular name" [Undecided,New]21:28
*** peristeri has quit IRC21:29
*** openstackgerrit has quit IRC21:29
mordredlifeless: I defer to your judgemnt - my head is not deep in it21:29
*** openstackgerrit has joined #openstack-infra21:30
*** sarob has joined #openstack-infra21:31
*** dboik has quit IRC21:31
*** tnovacik has quit IRC21:31
*** hdd has quit IRC21:32
*** Rockyg has quit IRC21:32
tchaypolifeless: one thing that occurred to me21:34
*** yamahata has joined #openstack-infra21:34
tchaypoif pbr is re-invoking pip to deal does that mean you’ll end up with (potentially) different things being installed than if pip was doing all the processing?21:35
*** bswartz has quit IRC21:35
tchaypobut i think the answer is no, because when pbr re-invokes pip it passes in all the dependencies it already has, from memory21:35
lifelessyes, which is a different symptom and problem21:35
openstackgerritJoe Gordon proposed openstack-infra/system-config: Add subnode logs to logstash
clarkbpip bug 988 is going to be my most glorious bug opened21:35
openstackbug 988 in Launchpad itself "Malone result headers scrunched in Konqueror+Safari+Opera" [Medium,Fix released] - Assigned to Matthew Paul Thomas (mpt)21:35
tchaypoin any case I think the correct solution there is to fix pip to be consistent rather than break pbr to match pip21:36
tchaypoclarkb: much nicer than that other bug 98821:36
*** dboik has joined #openstack-infra21:37
*** david-lyle has joined #openstack-infra21:37
mordredclarkb: whats pip bug 988?21:38
openstackbug 988 in Launchpad itself "Malone result headers scrunched in Konqueror+Safari+Opera" [Medium,Fix released] - Assigned to Matthew Paul Thomas (mpt)21:38
lifelesstchaypo: was that a strawman? I don't think anyone suggested breaking pbr...21:38
*** dboik has quit IRC21:38
clarkbmordred: `pip needs a proper dep resolver`21:38
clarkbmordred: I opened a few years ago21:38
mordredclarkb: it is a glorious bug21:38
*** dboik has joined #openstack-infra21:38
tchaypolifeless: yes21:38
clarkbmordred: but it has become the nexus point of many python/pip related bugs21:38
mordredclarkb: you might wind up creating a black hole21:39
*** dustins has quit IRC21:40
*** tonytan4ever has quit IRC21:41
*** sdake has quit IRC21:44
*** annegentle has quit IRC21:45
*** ivar-laz_ has quit IRC21:45
openstackgerritMerged openstack-infra/project-config: Setup publishing to pypi for xstatic-*
*** ivar-lazzaro has joined #openstack-infra21:45
*** yamahata has quit IRC21:46
*** dprince has joined #openstack-infra21:46
*** bknudson has quit IRC21:46
openstackgerritMerged openstack-infra/project-config: Add stable acls for python-glanceclient
lifelessjogo: ECONTEXT?21:49
jogolifeless: pips recommendation for managing packages and ensuring repeatable installs21:49
jogolifeless: it says to use --no-deps, and even better it says to not use pip (and use peep?)21:50
lifelessso thats a little stale21:50
lifelessI also have a quarrel with it21:51
lifelessfreeze lists everything bar pip and setuptools21:51
lifelesswith ==21:51
lifelessso, having things randomly shift around with or without --no-deps is at best a bug21:51
lifelessI'd want an example to poke at21:51
jogolifeless: good point21:52
lifelessthe last point is false21:52
lifelessbecause pypi has been fixed21:52
lifelessno ninja-replacing-of-objects-on-pypi anymore21:52
jogolifeless: that is what I thought too but the docs are wrong for pip I guess21:52
jogoanyway since you are digging into this stuff now wanted to share the link with you21:52
lifelessyeah - good, just was like so many balls in air, not sure which one you refered to :)21:53
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove REST links from inventory metadata
openstackgerritMerged openstack-infra/project-config: Remove stable-compat-jobs for projects that have stable branchs
*** bhuvan has quit IRC21:54
openstackgerritMerged openstack-infra/project-config: Add glean to infra
openstackgerritMerged openstack-infra/puppet-zanata: Make default from address mandatory
*** dboik has quit IRC21:55
*** dboik has joined #openstack-infra21:55
*** cdent has quit IRC21:56
openstackgerritJoshua Harlow proposed openstack/requirements: Add cachetools requirement
jogolifeless: at least they aren't swords21:57
*** dprince has quit IRC21:58
*** ivar-laz_ has joined #openstack-infra21:58
*** aduarte has quit IRC22:00
*** ivar-lazzaro has quit IRC22:01
*** tqtran_afk is now known as tqtran22:01
*** Swami has quit IRC22:04
lifelessjogo: heh.22:04
*** andreykurilin__ has joined #openstack-infra22:05
*** otter768 has joined #openstack-infra22:05
*** bknudson has joined #openstack-infra22:05
*** esker has quit IRC22:07
*** otter768 has quit IRC22:09
*** signed8b_ has joined #openstack-infra22:09
*** openstackgerrit has quit IRC22:11
*** openstackgerrit has joined #openstack-infra22:11
clarkbjenkins04 appears dead22:12
*** signed8bit has quit IRC22:12
clarkbthe process is up but not responding to my browser, likely the thread leak again22:13
*** dboik has quit IRC22:14
clarkbthe other masters look fine. Working to get a thread count on 04 now22:14
tchaypodstufft is on the ball22:15
clarkb7,190 threads22:15
clarkbjenkins that is not healthy stop22:15
clarkbputting it in shutdown mode if I can22:16
clarkbI am also cleaning up ~10 stale nodes on rax that will bump our usable quota by ~10 nodes22:17
anteayaclarkb: is that the same leaking thread issue that we have had for a while?22:18
*** shashankhegde has quit IRC22:18
clarkbanteaya: ya22:19
*** ivar-laz_ has quit IRC22:19
clarkbgot 04 into shutdown mode, will likely leave it there then dig out the java gloves tomorrow and see if I can fix it22:19
anteayaand your upgrade didn't solve the issue :(22:19
*** ivar-lazzaro has joined #openstack-infra22:19
clarkbanteaya: it did not22:19
clarkbanteaya: we have observed the bug on both versions of the ssh-slaves plugin22:19
anteayaare java gloves made with dragon hide?22:19
*** jaypipes has joined #openstack-infra22:20
*** david-lyle_ has joined #openstack-infra22:20
*** david-lyle has quit IRC22:20
*** shashankhegde has joined #openstack-infra22:21
*** spzala has quit IRC22:21
*** erlon has quit IRC22:21
*** annegentle has joined #openstack-infra22:22
*** agentle_ has joined #openstack-infra22:22
zarofungi: i have verified that deploying jeblair's gerrit change on review-dev works without error:
*** ArxCruz has quit IRC22:23
*** dkranz has joined #openstack-infra22:24
anteayazaro: what do you have running on gerrit-dev at the moment?22:24
*** ddieterly has joined #openstack-infra22:25
openstackgerritMerged openstack-infra/system-config: Update wildfly puppet module version
*** hogepodge has quit IRC22:27
*** hashar has quit IRC22:27
zaroanteaya: i assume you mean review-dev, it's running  whatever is on tip of openstack/2.10.2 branch +
*** ddieterl_ has joined #openstack-infra22:28
*** ddieterly has quit IRC22:29
anteayaI do mean review-dev, sorry22:29
anteayazaro: that is a patch for an open wildfly module22:30
anteayaare you saying that is running on review-dev?22:31
*** sdake has joined #openstack-infra22:31
zaroopps, wrong #, it's actaully  and yes review-dev22:31
*** fbo has quit IRC22:31
anteayaah yes, thanks that makes sense22:32
*** hogepodge has joined #openstack-infra22:32
openstackgerritMerged openstack-infra/tripleo-ci: Set EnablePacemaker for the HA job
*** fbo has joined #openstack-infra22:37
*** pradk has joined #openstack-infra22:41
*** pradk has quit IRC22:41
*** dkranz has quit IRC22:43
*** bhuvan has joined #openstack-infra22:44
*** amitgandhinz has quit IRC22:45
*** agentle_ has quit IRC22:45
*** annegentle has quit IRC22:45
*** Swami has joined #openstack-infra22:46
*** shakamunyi has quit IRC22:46
*** gordc has quit IRC22:47
*** dimtruck is now known as zz_dimtruck22:48
openstackgerritJoshua Harlow proposed openstack-infra/release-tools: Include 'source_url' in notes and use readme parser function
*** apmelton is now known as apmelton_22:50
*** apmelton_ is now known as apmelton_away22:50
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Refactored event worker plugin structure into plugin directory.
*** nelsnelson has quit IRC22:51
*** Swami has quit IRC22:53
*** ZZelle_ has quit IRC22:54
*** david-lyle_ has quit IRC22:57
*** melwitt has quit IRC22:57
*** dizquierdo has quit IRC22:57
*** Rockyg has joined #openstack-infra22:58
*** andreykurilin__ has quit IRC22:59
clarkbgood morning23:00
*** ChuckC has quit IRC23:00
*** stevemar has joined #openstack-infra23:01
anteayamorning jhesketh23:01
anteayaI'm being diconnected again from my etherpad in firefox23:01
anteayaI have worked for several hours without being disconnected and I just got disconnected twice in about 5 minutes23:02
anteayait seems to come in waves but I don't know what the trigger is23:02
*** sabeen has quit IRC23:04
*** tjones2 has joined #openstack-infra23:04
*** tjones1 has quit IRC23:05
*** spzala has joined #openstack-infra23:06
*** jd__ has quit IRC23:08
*** jd__ has joined #openstack-infra23:08
*** signed8b_ is now known as signed8bit_ZZZzz23:08
*** doug-fish has left #openstack-infra23:08
*** eharney has quit IRC23:09
*** bswartz has joined #openstack-infra23:09
*** yamamoto has joined #openstack-infra23:10
*** ajo has quit IRC23:10
anteayaI will also note that etherpad in chromium is able to deal with a multiline copy/paste and format it the way it was pasted rather than in firefox which if I paste into a bullet point puts bullets at the beginning of every new line even if it is one passage23:13
*** thedodd has quit IRC23:13
*** yamamoto has quit IRC23:14
clarkbanteaya: did it work in firefox before the upgrade?23:15
clarkbor is chromium half fixing an existing bug?23:15
*** fawadkhaliq has joined #openstack-infra23:15
anteayafirefox worked fine as far as I can remember prior to about 2 days ago23:17
anteayaI never really worked in etherpad in chromium before yesterday as I am just used to firefox and use it without thinking about it23:17
anteayabut today I am thinking about it23:17
anteayaso I don't know what chromium did with regard to formating prior to today, nor for taht matter had I spent much time copy pasting and using bullets in etherpad in firefox23:18
anteayaI'm doing a nova-net to neutron ml round up23:18
anteayamuch to copy paste23:18
anteayaand for this work chromium, whatever it is doing, it proving itself to be less work for me to get what I want23:20
*** fawadkhaliq has quit IRC23:20
*** ChuckC has joined #openstack-infra23:21
clarkbmordred: SpamapS Shrews comment on 175699 about general shade behavior that you may find interesting23:22
clarkbanteaya: I think as a general rule chrom* has the largest user share so many applications work better with it23:23
clarkbanteaya: sort of like how IE6 was "best" for compatibility for so long23:23
*** david-lyle has joined #openstack-infra23:24
clarkbmordred: and for 175903 does links come from the cloud? if so wouldn't it be appropriate to let users decide if they don't want that information? I don't know what the links metadata is fwiw23:24
*** ChuckC has quit IRC23:25
*** Rockyg has quit IRC23:25
*** ChuckC has joined #openstack-infra23:25
anteayathat would make sense23:26
*** markvoelker has quit IRC23:26
*** hichihara has joined #openstack-infra23:26
*** mriedem is now known as mriedem_away23:26
anteayaand I have been disconnected from the etherpad in firefox 3 more times but yet to be disconnected in chrome23:26
*** amotoki has quit IRC23:27
clarkbmordred: found it in the api docs23:27
*** sigmavirus24 is now known as sigmavirus24_awa23:27
clarkbmordred: so its a link to that resource, but shade already knows how toget that resource23:27
clarkbmordred: so its not useful?23:28
*** sputnik13 has quit IRC23:29
greghaynesclarkb: thats why my reading of the API indicates23:30
*** ddieterl_ has quit IRC23:31
mordredclarkb: it's not useful in the inventory23:35
mordredclarkb: it's useful if it's part of the payload to do further rest calls23:35
mordredclarkb: also - yeah  list_images() should return a [] - thank you nice catch23:36
mordredclarkb, SpamapS, Shrews: also - I just rewrote that inventory patch (sorry, should have wip'd the one in gerrit) - new upload coming23:36
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Every worker now gets its own session.
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Pre-resolve author record for events.
mordredclarkb: I think I just found a MUCH nicer way to do it in shade that will let the ansible-specific things be in ansible and the re-usable things be in shade without making either one too weird23:37
mordred(Also, in so doing, found two other bugs - so it was worth hacking on regardless)23:37
*** sdake has quit IRC23:39
*** dannywilson has quit IRC23:39
*** claudiub has quit IRC23:42
*** Longgeek has quit IRC23:42
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove REST links from inventory metadata
openstackgerritMonty Taylor proposed openstack-infra/shade: Have get_image_name return an image_name
openstackgerritMonty Taylor proposed openstack-infra/shade: Return an empty list on volume_list
mordredgreghaynes: ^^ you'll enjoy 17654923:43
mordredShrews, SpamapS: we need to circle back on the behavior on volume_list I think23:44
mordredbecause right now it's just passing through cinder - which may not exist for the user in the cloud - this is one where we need to see if the user has cinder, if not, try nova23:45
greghaynesmordred: TIL, image is not name23:45
greghayneser, id is not name23:45
SpamapSmordred: Oh, yeah, thats entirely true. I just think we haven't run into a cloud like that yet.23:46
SpamapSthough, is nova-volume as prevalent as nova-network ?23:46
greghaynesmordred: what is with the WIP on ?23:46
SpamapSI thought cinder stomped it out of existence rather quickly.23:46
mordredSpamapS: well....23:47
mordredSpamapS: thing is - vexxhost and RunAbove do not have cinder at all23:47
cineramapleia2: back now23:47
mordredSpamapS: I mean, I can also put a try/except into the inventory meta code23:47
*** pc_m has joined #openstack-infra23:47
fungiclarkb: mind if i delete all ready nodes from jenkins04 so it doesn't keep squatting them?23:47
SpamapSmordred: but do they even have volumes then?23:48
clarkbfungi nope23:48
mordredSpamapS: prolly not23:48
mordredSpamapS: so yeah - we can prolly skip the nova volume api attempt23:48
SpamapSmordred: so isn't it appropriate to error out?23:48
mordredSpamapS: maybe so23:48
mordredSpamapS: it's possible I'm wrongthinking this23:48
fungiclarkb: done. thanks!23:48
*** yamamoto has joined #openstack-infra23:48
SpamapSAlso agreed that list_something should return a list btw. ;)23:48
fungihopefully that'll speed things up a little23:48
* SpamapS is just catching up.23:48
mordredgreghaynes: that's nodepool consuming a resource api in shade - so we should make sure that we're happy with its get/list/search interface23:49
greghaynesjust shade API instability23:49
mordredgreghaynes: yah - wanted to make sure we're ready to lock that api23:50
mordredgreghaynes: essentially, once nodepool starts using it, we need to consider it _stable_ and deal with transition paths and stuff23:50
*** ddieterly has joined #openstack-infra23:50
mordredSpamapS: I tink list_images returning a dict is a holdover from when we were caching in a dict23:50
*** ajmiller has quit IRC23:51
dansmithis the gate sick?23:51
SpamapSOr just a benign attempt at returning a more useful data structure.23:51
fungisomething seems to have broken all tempest-based jobs again (maybe all devstack-based jobs?)23:51
fungidansmith: just noticed the same. playing "spot the bug" now23:51
dansmithfungi: okay23:52
palendaefungi: Not necessarily, os-ansible-deployment isn't running either23:52
palendaeAnd we don't use devstack23:52
mordredSpamapS: we could ALSO do that - but I'd want it to be consistent across all of our list_foo calls23:52
dansmithI'm waiting for mine to die so I can look at the logs23:53
dansmithdevstack failed to start tho23:53
fungi"ImportError: No module named pathlib"23:53
mordredSpamapS, morganfainberg: OH WOW23:53
fungilooks like a django_pyscss issue23:53
morganfainbergmordred, hi23:53
dansmithfungi: now you play "who released a lib today" right?23:53
mtreinishdansmith, fungi: isn't pathlib in stdlib?23:53
morganfainbergdansmith: isn't that every day in openstack?23:53
mtreinishoh not on python 223:53
dansmithmorganfainberg: *hangs head* ... yes :/23:54
mordredmorganfainberg: the depth of that23:54
morganfainbergmordred: omg23:54
mordredmorganfainberg: yay for late binding23:54
morganfainbergmordred: i...23:54
mordredso, I construct a cinder_client with a keystone Session23:54
morganfainbergi half expected it to hit max depth there.23:54
fungidansmith: yeah, that's the current roll of the die anyway23:54
SpamapSmordred: impressive stacktrace ;)23:54
* mordred goes to look to see if there is a saner thing to do ... it _may_ still be reasonable to trap this rather than complain23:54
fungidansmith: hoping i don't draw a go to jail card23:55
dansmithfungi: heh23:55
greghaynesfungi: django_pyscss just released...23:55
fungisurvey says...
mtreinishSpamapS: heh, I still prefer this one:
SpamapSmordred: I think EndpointNotFound is a nice specific thing you can turn into a shade exception.23:56
mtreinishwhich gets dumped on ever cinder unit test run23:56
mtreinishmore than once per run I should say23:56
SpamapSmtreinish: best part about that one is that it is a perfectly normal thing to have happen. ;)23:56
SpamapSok, time to go fetch the children23:56
* SpamapS disappears23:56
clarkbmtreinish wat23:57
fungisdague's openstack/requirements:tools/ script puts django-pyscss 2.0.0 at the top of the suspect pile too, followed by python-glanceclient 0.17.1, kombu 3.0.26, libvirt-python 1.2.14, python-novaclient 2.24.1 and cliff 1.12.0 (all released 2015-04-22)23:57
fungibut yeah, given the traceback, django-pyscss is the clear winner this round23:58
fungi"you fail at python 2.x"23:58
greghaynesCan we start making awards23:58
fungi"Added support for pyScss 1.3 and Python 3" (and dropped support for any python older than that)23:59
fungiso! this raises an interesting challenge23:59

Generated by 2.14.0 by Marius Gedminas - find it at!