Wednesday, 2016-01-13

clarkbThelo: grep for LIBS_FROM_GIT00:00
clarkbyou will see its overridden00:00
*** zz_dimtruck is now known as dimtruck00:00
jeblair| local_gb                  | 915                                    |00:01
jeblairi get that from nova-hypervisor show00:01
jeblairer nova hypervisor-show00:01
jeblair| disk_available_least      | 835                                    |00:01
jeblair| free_disk_gb              | 865                                    |00:01
jeblair| local_gb_used             | 50                                     |00:01
asselin_Thelo, yes, what clarkb said, it's get soverride at 2016-01-12 01:28:48.742 | +++ LIBS_FROM_GIT=00:02
jeblair915-50=865; i don't know what disk_available_least is00:02
Theloasselin: mmm maybe my understanding of LIBS FROM GIT is wrong the, in my case I dont want the upstram but a specific pachtset, is it doable?00:03
jeblairah, it's available space including an overcommit allowance.00:03
jeblairat any rate, it's not looking like different classes of storage are bexing exposed through the parts of the nova api i'm seeing00:04
clarkbgreghaynes: otherwise I Think it is looking good00:04
asselin_Thelo, yes, but the issue is that you setting is getting ignored.00:04
*** bharathm has joined #openstack-infra00:04
*** rbrndt has quit IRC00:05
clarkbLIBS_FROM_GIT says install the lib from the git repo instead of pip. The problem here is that that variable is empty00:05
*** baoli has quit IRC00:07
Theloclarkb: true, so at which point should I specify LIBS FOR GIT so it is not ignored?00:07
greghaynesclarkb: fun00:08
clarkbThelo: well if you just set it once you'd be fine but it looks like its being set twice the second time its empty00:08
*** dims has joined #openstack-infra00:09
*** rkukura has quit IRC00:10
asselin_Thelo,  export DEVSTACK_PROJECT_FROM_GIT=00:10
*** tzn has quit IRC00:10
Theloasselin: Where is that line ? something to add?00:11
asselin_Thelo, you can include that in your jenkins job00:12
Theloasselin: ok ; I will try00:12
*** bcourt has joined #openstack-infra00:12
asselin_Thelo, this is the line ^^00:12
Theloasselin: what does it do?00:13
Theloasselin: O i get it00:13
Theloasselin, clarkb: thanks00:13
asselin_this is where it's actually set00:13
*** ddieterly has quit IRC00:13
*** salv-orl_ has quit IRC00:16
*** piet has quit IRC00:17
*** salv-orlando has joined #openstack-infra00:17
*** piet has joined #openstack-infra00:20
jeblairjhesketh: no, the zuul behavior you describe doesn't ring a bell00:20
*** bpokorny_ has quit IRC00:20
jeblairjhesketh: and sorry, i've gotten swamped this afternoon and haven't had a chance to discuss ansible yet00:20
*** bpokorny has joined #openstack-infra00:21
*** sigmavirus24 is now known as sigmavirus24_awa00:21
jeblairjhesketh: but really quickly, the gist is that i think it will be safer if we invoke ansible as a subprocess rather than using it directly in python00:21
*** eliqiao has quit IRC00:22
jeblairjhesketh: the python api is not very well documented, and seems mostly to exist for the purpose of driving ansible's command line clients, and they make some significant assumptions in doing that.00:23
*** eliqiao has joined #openstack-infra00:23
*** sdake has joined #openstack-infra00:24
jeblairjhesketh: so i'm going to revise my latest v3 change to do that and then turn it over to you00:24
*** jsavak has quit IRC00:24
jheskethjeblair: all good.. does that sound like incorrect behavour (re not rebasing onto the newer patchset)00:25
jheskethjeblair: good to know re the ansible lib. Once the zuul worker sets up the env and everything it should be a pretty simple call out to ansible00:26
jheskethie ansible will do all the heavy lifting of the job itself so I'm hoping that part is straight forward00:26
*** ZZelle_ has quit IRC00:28
*** rkukura has joined #openstack-infra00:29
*** tongli has joined #openstack-infra00:30
*** Swami has joined #openstack-infra00:32
*** bharathm has quit IRC00:33
*** bharathm has joined #openstack-infra00:34
*** otter768 has joined #openstack-infra00:35
*** tongli has quit IRC00:40
*** otter768 has quit IRC00:40
*** tongli has joined #openstack-infra00:40
openstackgerritMerged openstack-infra/jenkins-job-builder: Add more details about Emotional Jenkins plugin to docs
*** dimtruck is now known as zz_dimtruck00:44
mtreinishfungi: is it still running? This is pretty slow :)00:45
fungimtreinish: still running00:46
dimsoh man looking at zuul i don't even want to ship oslo releases this week00:48
clarkbits actually not that bad00:49
*** rkukura has quit IRC00:49
clarkbdims: what is the concern?00:49
dimsclarkb : gate depth, just in case i break something else00:50
tonybSo I'm trying to help a friend.  He has a change (in os-brick) that depnds-on changes to requirements and privsep.  Looking at the logs from the job zuul correctly set the ZULL_CHANGES envvar.  The requirements change was pulled in such that /opt/stack/new/requirements was the right git SHA but oslo.provsep was just installed from pip.  my gut feel is that I need to ass oslo.privsep to $PROJECTS for os-brick but I can't verify that.  Does it sound correct00:50
*** rkukura has joined #openstack-infra00:50
tonyb is the change in question.00:51
dimstonyb : use LIBS_FROM_GIT=oslo.privsep00:51
jeblairjhesketh: it sounds like an error -- uploading a new patchset should eject that item from the queue00:51
jheskethjeblair: it was a new patchset to a change above the one that wouldn't enter the gate00:51
tonybdims: but I only want it iff there is a Depends-On00:51
jeblairjhesketh: i may not understand the situation :)00:52
clarkbtonyb: adding it to PROJECTS is a necessary step, but it is not the only required step, you need libs from git too00:52
jheskethyeah difficult to explain sorry00:52
jheskethjeblair: 260628 depends on 260625/4 but it was 260625/6 that merged00:53
jhesketh260628/3 in the related changes section links to gitweb rather than to the change (which is a new thing to me)00:53
*** markmcclain has quit IRC00:53
jheskethin order to get it into the gate we had to rebase 260628 onto 260625/6 (well in reality since 260625/6  had merged I rebased onto master)00:54
dimstonyb : looking00:54
jheskethjeblair: you can kinda see what was going on in the comment section00:54
*** bcourt has quit IRC00:55
tonybdims: Thanks.  I'm struggling to find the linkage between what zuul decides is required, based on Depends-On and how the node is setup.00:55
*** zz_dimtruck is now known as dimtruck00:55
*** Sukhdev has quit IRC00:55
clarkbtonyb: all zuul is going to do is make sure the code is merged proprely and provide a zuul ref with that change00:55
*** amuller has quit IRC00:55
clarkbtonyb: your job(s) then need to properly checkout that zuul ref and install from source00:55
*** markmcclain has joined #openstack-infra00:55
clarkbPROJECTS list ensures that the listed projects are checked out properly and LIBS_FROM_GIT installs that lib from source00:56
jeblairjhesketh: looking00:56
jheskethjeblair: let me know if you want me to retry that bad explaination00:56
*** tlian has quit IRC00:57
* jeblair opens that patch in gertty to get correct (utc) times00:57
tonybclarkb: okay but I don't *always* want LIBS_FROM_GIT, I only want it if this change Depends-On one from oslo.privsep00:57
tonybclarkb: I'm not finding eveidnece that other projects have had to do that checking so I thought I was missing something00:58
clarkbtonyb: then write a one off change to do it00:58
clarkbthats probably the simplest way to do it00:58
clarkb(but this is the problem with not doing things the old way)00:59
clarkbthat said, you probably don't want both changes ot need atomic merging00:59
tonybclarkb: woudl the change be to devstack-gate? what is the old way?00:59
clarkbtonyb: the old way was we installed everything fomr source00:59
clarkbtonyb: I would probably make the change to devstack but either one works00:59
dimstonyb : the way i would do this with current as follows... 1) review for oslo.privsep if needed 2) review for os-brick if needed 3) review for devstack with depends-on for reviews 1 and 2 and LIBS_FROM_GIT set to both oslo.privsep and os-brick01:01
dimsthen run the tests in review #3 and keep adjusting #1 and #2 till you are happy01:01
*** eil397 has quit IRC01:02
*** ybathia has quit IRC01:03
clarkbtonyb: I think the idea behind having this setup is it forces you to not make changes that need multiple things to merge all toegher or release togethre01:03
dimstonyb : the other trick i use is modify my tox.ini to run "pip install -q -U -e git+"01:04
clarkbtonyb: so privsep should support the old and new thing, then you update brick, then you can deprecate the ol thing01:04
*** tlian has joined #openstack-infra01:04
clarkbbut either way what dims describes will work to test them prior to any releases01:04
tonybclarkb: I mostly get that but ...01:05
jeblair(really there should be some kind of job on the oslo lib that exercises this)01:05
tonybclarkb: we're trying to develop the 'new thing' in privsep and verify that brick works, and also trying to avoid lots of broken releases of privsep01:05
clarkbya if this is an ongoing thing a job that just does it may be worthwhile01:06
dimsclarkb : so it's master of oslo.privsep + master of os-brick tested with cinder/nova01:06
clarkbdims: no thats not quite right01:06
clarkbdims: its the proposed change of privsep + the proposed change of brick against cinder/nova01:07
*** piet has quit IRC01:07
dimsclarkb : right :)01:07
dimsmaster + review instead of library from pypi01:07
clarkbwell it doesn't have to be master01:07
dimsclarkb : explaining tonyb's situation. but yes you are right01:08
tonybdims: ate this point there is no cinder/nova in the picture it's just os-brick using privsep01:08
clarkbso ya I think your two options are make a temporary job that just does this for your changes as you go or push a change to devstack to configure it that way01:08
jeblair++temp job01:08
dimstonyb : then you can use the 2nd trick.01:08
dimstonyb : "pip install -q -U -e git+" in tox.ini just before you run tests01:09
jeblairand some day, someone will write a change to zuul to launch jobs on request01:09
tonybdims: sure. but how do I do that only when there is a Depnds-On01:09
jeblairtonyb: if a job changed behavior based on a depends on, then you would be able to break a gate01:10
*** tongli has quit IRC01:10
*** ekarlso has quit IRC01:10
jeblairtonyb: the next change after that might not have a depends on and would be testing different code01:10
jeblairso that's not a pattern we should use01:10
*** tzn has joined #openstack-infra01:10
*** tongli has joined #openstack-infra01:11
*** tongli has quit IRC01:11
tonybwait what?01:11
clarkbthe issue here is what is desired is an integration test between two repos which does not exist today01:11
*** annegent_ has joined #openstack-infra01:11
tonybwe use depends-on often01:11
*** tongli has joined #openstack-infra01:11
tonybhow is this different?01:11
clarkbthe best way to solve that problem is to make the integration test01:11
jeblairclarkb: ++01:11
clarkbtonyb: depends on requires the individual jobs to checkout and install the correct code01:13
*** annegentle has quit IRC01:13
dimsclarkb : when you do a Depends-On does the current setup pull down code from git if it is not present in PROJECTS?01:13
clarkbdims: no01:13
dimsi am guessing not01:13
tonybI don't mean to be dense.  but I'm celarly missing somethign important01:13
dimstonyb : see ^^^01:13
clarkbtonyb: all depends on does in zuul is make a zuul ref01:13
*** tlian has quit IRC01:13
clarkbtonyb: something needs to grab the zuul ref and install it. Devstack-gate and zuul-cloner will both do it for you01:13
*** tlian has joined #openstack-infra01:13
tonybclarkb: okay so is the zuul ref a cross project thing?01:14
clarkbtonyb: yes each project in that dependency gets the same ref in their repo01:14
tonybclarkb: okay.01:14
*** gongysh has joined #openstack-infra01:14
tonybclarkb: so I *think* just adding oslo.privsep to PROJECTS will work01:15
clarkbso in this case you need a job that checks out the zuul ref for privsep and brick, then you need to install them from that source, then you run whatever tests against them01:15
jeblairtonyb: a zuul ref is a potential future state of the world for all involved repositories -- including ones specified by depends on.01:15
dimstonyb : the other thing i do is work on my github repo and use the pip install -U trick to test01:15
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Initial add of gentoo support for diskimage-builder
tonybthen devstack-gate will use zuul-cloner to grab the zuul_ref01:15
tonybthat's the part that's missing.01:15
clarkbtonyb: it will do the equivalent thing yes, but what it probably won't do is install privsep from that location01:15
clarkbmost(all) libs are installed from pip01:16
* tonyb goes to check that bit01:16
gusok, so (being dense again) what does Depends-On actually help me with, if not this exact situation?01:16
clarkbgus: it does help this exact situation01:16
*** tzn has quit IRC01:16
clarkbgus: but you still need an integration test between the two repos01:16
gus.. except I need to predeclare all the combinations of Depends-On that I might use?01:16
clarkbsomething that says if I have this repo and that repo grab them both and test them together01:16
*** mriedem has joined #openstack-infra01:16
clarkbno, so thats my point earlier about how and why oslo no longer tests everything from source all together anymore01:17
clarkbthere was a conscious desecision that these libs should have interfaces that are more stable than that01:17
clarkbthere may be exceptions, but in those cases you will need a different job01:18
*** doug-fish has quit IRC01:18
gusclarkb: I'm all for that, and I'd be happy if the downstream patches couldn't merge until all the depends-on were worked out - but this makes development really hard :/01:18
*** rkukura has quit IRC01:18
jeblairthe oslo lib developers have _intentionally_ decided that new features should be developed in oslo first, then released, then they can begin to be used in other projects.01:18
jeblairand if those features don't work right, they will need to be adjusted in oslo01:18
jeblairand then oslo released again01:18
gusI basically can't iterate on an os-brick CI bot thinks of my change(s) without doing pypi release for every one.01:18
clarkbgus: you can...01:19
clarkbyou just need a new job01:19
gussure :P01:19
clarkbbecause the existing job setup has intentionally set it up such that brick changes will test against release privsep and privsep changes will test against release brick01:19
*** doug-fish has joined #openstack-infra01:19
dimsgus: you can review notes above from me on how i do this currently01:19
clarkb(its worth noting that infra iddn't enforce this, oslo did it)01:20
clarkb(we just run the jobs that oslo decided they wanted/needed to run)01:20
jeblairright, this is not a CI limitation, it's a choice the project developers have made01:20
gusoh yeah, no blame intended - I'm just a bit confused how people work in this environment.01:20
clarkbif oslo wants to install everything fomr source again that totally doable01:20
clarkband its how we tested everything for about 4 years01:20
dimsgus: i've shared all of my tips :)01:20
dimsclarkb : jeblair : sdague will have plenty to say on that :)01:21
jeblairdims: i wish sdague were here to explain this right now01:21
tonybokay so would a devstack-gate change that iterated over ZUUL_CHNAGES and added things to $PROJECTS and LIBS_FROM_GIT be laughed at?01:21
jeblairwe have a system that excels at testing coordinated releases of software, but we only use it for a subset of our projects.01:21
jeblairtonyb: it is important that a job have a consistent behavior -- it needs to either install each of its dependencies from source or pip every time it runs01:22
jeblairtonyb: we don't care which it does, only that it does the same thing for each dependency each time01:22
jeblairtonyb: it can be all source, all pip, or a mixture01:22
jeblairtonyb: but it can't change from one run to the next01:23
gusdims: so my challenge with os-brick is that I don't have any of the hardware to actually test most of these codepaths.01:23
jeblairtonyb: (of course, you can change the job configuration, but that's a static change)01:23
jeblairgus: are you working on the same thing as tonyb?01:23
tonybjeblair: run to run of the same review or ro run to run of the same repo?01:23
gusSo I'm looking closely at the 3rd party CI bots.  It seems there's no way to actually get them to run my unreleased privsep code.01:23
*** eil397 has joined #openstack-infra01:23
asselin_gus, there should be some 3rd party cis running to get more coverage in os-brick01:23
dimsgus : remember that tox will install stuff from requirements file too which is a pypi version01:24
jeblairtonyb: run to run of the same _job_.01:24
tonybjeblair: yeah gus is the friend I'm tryign to help01:24
jeblairhi gus :)01:24
tonybjeblair: okay01:24
gusjeblair: yes (or rather, he's bravely investigating my problem for me)01:24
jeblairgus, tonyb: clarkb's point earlier is the key -- you have found a cross-project integration test that is missing01:24
*** eliqiao has quit IRC01:24
asselin_I think Thelo  was asking about that too, and he discusseed the issue with hemna over in the cinder channel01:25
dimsjeblair : os-brick does not depend on oslo.privsep today (if you see master)01:25
asselin_gus, tonyb have you talked to hemna yet?01:25
tonybso I have to create a new job in in brick (and later nova/cinder) that pulls brick and privsep from source (as defined by the ZUUL_REF)01:25
jeblairgus, tonyb: for this situation, it seems like it would be worthwhile to add a job to perform that test, at least as long as this development effort runs, and then it can be removed when no longer relevant.01:25
gusjeblair: ok, but surely this is true for just about every oslo library?  and yet there was a decision not to just list all oslo libraries in this situation.01:25
dimstonyb : yes new experimental job for os-brick that adds oslo.privsep in PROJECTS.01:26
tonybasselin_: gus has, I just *thought* I understood this and could help him01:26
*** bharathm has quit IRC01:26
tonybdims: okay experimental for now later non-voting I guess01:26
dimstonyb : when you do that if you have Depends-On then the correct review is pulled down, you will still need to install it with pip commands in tox.ini01:27
dimsagainst the downloaded directory01:27
gusasselin_: I last did a few days ago..01:27
asselin_tonyb, gus please talk to hemna in #openstack-cinder. He's an os-brick core and familiar with this challenge and can help you find a path forward01:27
openstackgerritlifeless proposed openstack-infra/infra-manual: Update the manual to reference -constraints jobs.
*** Daisy has joined #openstack-infra01:28
*** mtanino has quit IRC01:28
*** ccrouch has quit IRC01:29
jeblairjhesketh: 2016-01-12 04:57:59,610 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added openstack/horizon master 260628,3>01:29
tonybokay I've never used the experimental queue before.  I know how to trigger it but where do I get the results/logs?01:29
jeblairjhesketh: zuul saw the jenkins+1 but did not report"Adding %s, %s to %s" % (project, change, pipeline))01:29
jeblairjhesketh: which suggests that it did not match the pipeline trigger for gate01:30
jeblairjhesketh: aha01:30
jheskethjeblair: right, it also didn't enter the gate when a +W was just added either01:30
*** tlian has quit IRC01:30
jeblairjhesketh: it's because jenkins already had a +101:30
fungitonyb: they get posted to the change where you leave a check experimental comment01:30
jeblairjhesketh: yeah, so i've been looking at the wrong event :)01:30
lifeless3rd-party should honour depends-on01:30
lifelessif it doesn't, its a bug, no?01:30
tonybfungi: ok.01:31
jeblairlifeless: we've not required that they run zuul01:31
* tonyb goes to look for that01:31
jeblairlifeless: many do, but some don't01:31
fungilifeless: also some of them are only set up to test changes to specific repos and need to be able to skip dependencies on changes to projects they don't have defined in zuul01:31
lifelessjeblair: Oh, I wasn't assuming running-zuul, just that they test the same combinations upstream does01:32
jeblairtonyb, gus: yeah if you are talking about third-party ci systems, all bets are off.01:32
lifelessjeblair: but I can see how that is coupled01:32
tonybjeblair: no just the infra run CI01:32
jeblairlifeless: afaik, depends-on is only implemented by zuul.  yes.  :)01:32
*** Keedya_ has joined #openstack-infra01:33
lifelessfungi: that doesn't make sense; if they are testing changes to cinder, but oslo.logging is a depends-on of a cinder change, they need the depends-on to be able to report usefully01:33
*** doug-fish has quit IRC01:33
lifelessfungi: perhaps I misinterpreted what you said01:33
jeblairlifeless: agreed in theory; i think in practice generally the 3rd-party ci test scope may be considered less thorough than what we would consider ideal01:34
gusjeblair: hrm, yes of course.01:35
gus(perhaps 3rd party bots that "don't support" Depends-On, should support it enough to give up on any change that includes a Depends-On?)01:35
*** Keedya_ has quit IRC01:36
fungilifeless: they often don't define the complete set of all the hundreds of repos that we do in our zuul config, and zuul can't create refs for repos it doesn't know about01:36
fungiso a depends-on to a change in an undefined repo gets handled as a no-op at this point (until a few months ago it caused zuul to throw an exception and not run any jobs for the depending change instead)01:37
jeblair(incidentally, something i want to change in v3)01:37
*** dims has quit IRC01:38
lifelessfungi: jeblair: thanks for the details.01:38
lifelessgus: I suspect contacting the third party ci in question perhaps ?01:39
*** hdd has quit IRC01:39
openstackgerritSteve Martinelli proposed openstack/requirements: bump versions for each keystone library
jeblair2016-01-12 04:03:17,803 DEBUG zuul.DependentPipelineManager: Checking for changes needed by <Change 0x7f06dd0bc6d0 260628,3>:01:40
jeblair2016-01-12 04:03:17,803 DEBUG zuul.DependentPipelineManager:   Change <Change 0x7f06dd0bc6d0 260628,3> needs change <Change 0x7f0827250890 260625,4>:01:40
jeblair2016-01-12 04:03:17,803 DEBUG zuul.DependentPipelineManager:   Needed change is not the current patchset01:40
jeblair2016-01-12 04:03:17,803 DEBUG zuul.DependentPipelineManager: Failed to enqueue changes ahead of <Change 0x7f06dd0bc6d0 260628,3>01:40
jeblairjhesketh: ^01:40
tonyblifeless: we're not targetting 3rd party ci just the infra's CI01:40
lifelesstonyb: to get test results on os-brick for oslo.privsep? I may have misunderstood the conversation01:41
jeblairjhesketh: i think that is in accord with what you described.01:41
guslifeless: I expect so.  I'm conscious of the fact that privsep is just an unimportant one amongst many project dependencies.  So it seems unreasonable to ask for things that are privsep+$project-specific just to save me some time on development iteration.01:41
*** eliqiao has joined #openstack-infra01:42
jeblairgus: the infrastructure is here to help everyone save time (and mistakes) in developer iteration :)01:42
gusjeblair: you'll excuse me for not feeling like that at the moment ;)01:42
lifelessgus: its a rather more invasive change than many01:42
jeblairgus: don't you have widespread support from those in channel for adding the job that will help you do your work?01:43
gusjeblair: I presume so.  I'll create the change and see what happens ;)01:44
gus(My alternative is to work on something else while I wait for the review+merge+release train)01:45
fungias to the reason we ended up with most jobs testing only with their released library dependencies, that was to mitigate a previous breakage risk where people were releasing unusable things which expected unreleased behavior in their dependencies01:45
fungiand didn't notice because the jobs were passing01:46
gusfungi: yeah, I can appreciate that risk.01:47
*** piet has joined #openstack-infra01:47
clarkbright it forced us away from the expectation of atomic releases01:47
clarkband makes you do more of a rolling system that allows users to upgrade01:47
gusIf we only had infinite development resources, then we could come up with a separate tests for "in development" vs "ready for release" or somesuch.01:48
jeblairgus: some repos have both01:48
fungiyep, which is a reasonable compromise even if still incomplete (as opposed to trying to test every possible combination of latest release and branch tip of all dependencies)01:49
fungiultimately people are going to either be installing from latest source or latest releases anyway01:49
*** amrith is now known as _amrith_01:50
fungiand not mixing and matching all over the place01:50
gusfungi: right, it's only the out-of-process dependencies that need to coexist during migration (assuming you can't stop-the-world).01:51
fungiso 1. make sure it works with things that have been released, and 2. make sure it works with things that have been merged but not yet released (and the interaction between proposed changes of two dependencies can be tested by the latter with change dependencies declared)01:51
clarkbthe mix and matching comes from mixing control planes01:51
clarkbit is very common to collocate many services01:52
clarkbbut not to upgrade them all at the same time01:52
gusclarkb: meh, "venvs" or some other jedi hand waving.01:52
fungigus: easier now than it used to be, but we still have plenty of packages which import from each other, so you can't silo them all01:53
gusdocker containers, chroots, etc.01:53
*** chlong is now known as chlong-afk01:53
*** gongysh has quit IRC01:54
clarkbI dunno I think its a good thing to be disciplined about01:54
clarkbsure you can work around it but the idea that you have to is silly01:54
fungipanaceas, magic bullets01:54
fungirainbow vomiting unicorns01:54
fungiall equally valid solutions to the problem01:55
Clintunicorn pancreas01:55
clarkbespecailly if we eve rwan to address the "openstack is hard to deploy problem"01:56
*** rkukura has joined #openstack-infra01:57
fungiClint: with some fava beans and a nice chianti01:57
Clinttoo bad i'm full01:58
jeblairjhesketh: i just re-checked that gerrit will (thankfully) not merge a change to a branch whose parent is not in the branch01:58
jeblairjhesketh: so zuul is doing the correct thing by preventing the change from being enqueued into the gate pipeline since it has no possibility of actually merging at the end01:59
*** eil397 has quit IRC01:59
asselin_hey any infra-root available to create a feature branch for me?,branches01:59
jeblairjhesketh: in short, i believe zuul's behavior is correct01:59
fungijeblair: is that new in 2.11? it sure did in 2.801:59
jeblairfungi: well, at least a change whose parent is outdated02:00
jeblairfungi: well, at least a change whose parent _change_ is outdated02:00
jeblairfungi: i haven't tested the 'change whose parent isn't even in gerrit' case :)02:00
fungijeblair: ahh, okay. the other issue was that it would fast-forward if the parent was on another branch and had no merge conflicts02:00
jeblairChange cannot be merged due to unsatisfiable dependencies.02:00
jeblairThe following dependency errors were found:02:00
jeblairDepends on patch set 1 of I7af6e09b, however the current patch set is 2.02:00
jeblairfungi: that ^02:00
fungiyep. that matches my expectations02:01
*** bardia has quit IRC02:01
jeblairfungi: yep.  i believe that was the behavior in 2.8 as well, and that's why zuul has enforced that for some time.02:01
jeblairfungi: i'm just being thorough with jhesketh's report :)02:01
*** piet has quit IRC02:01
jeblairi did not want to make any assumptions about behavior that may have changed in gerrit02:01
*** rkukura has quit IRC02:01
fungiwe likely still have the issue where if someone proposes a commit from say master to stable/liberty and stable/liberty hasn't diverged so all its commits are in master's history, then gerrit will drag in the intermediate commits02:01
fungiand won't provide much indication in the review ui that it's going to do so02:02
fungii don't think the bug report about that ever went anywhere02:03
*** thorst_ has quit IRC02:03
*** thorst has joined #openstack-infra02:04
openstackgerritDaniel Wallace proposed openstack-infra/shade: Save the adminPass if returned on server create
fungistated differently, gerrit is happy to consider changes mergeable even if their parents aren't in the current branch history, as long as they're in gerrit on another branch and all other mergability requirements are met02:05
*** bpokorny_ has joined #openstack-infra02:06
jheskethjeblair: shouldn't the gate pull down the newest patchset in that change?02:07
jheskethjeblair: or are what we're saying is that if a change ahead of itself is updated the child changes need rebasing02:07
*** rguillebert has quit IRC02:07
*** bpokorny has quit IRC02:09
*** tongli has quit IRC02:09
TheloHi all, on my CI zuul does not find out any new task (2016-01-13 10:08:33,086 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit.                    Can not get account information.)02:10
Thelois anyone familiar with that?02:10
jheskethThelo: those errors are safe to ignore02:10
*** tongli has joined #openstack-infra02:10
jheskethit's because of new events in a newer gerrit02:10
*** rkukura has joined #openstack-infra02:10
jheskeththere is a patch up to deal with them02:10
*** bpokorny_ has quit IRC02:10
jheskethnot finding a new task would be unrelated02:10
*** tzn has joined #openstack-infra02:13
*** Swami has quit IRC02:13
*** thorst has quit IRC02:14
*** tongli has quit IRC02:15
Thelojhesketh: is there a delay between the git review command and the moment where the log should appear in zuul/debug.log?02:15
clarkbasselin_: did you get your branch?02:15
clarkbfungi: ^ anything special you want ot instruct them on using a feature branch?02:16
*** Qiming has joined #openstack-infra02:16
fungiclarkb: nah, i referred them to our lovely infra manual bit about merge commits02:16
clarkbkk making branch now02:16
fungii thought i already did weeks ago02:16
clarkbfungi: not on ciwatch02:17
clarkbasselin_: feature/unit-tests?02:17
fungioh, ciwatch. i was thinking of the jjb refactoring feature branch02:17
clarkbI will go ahead and create it and point asselin_ at the manual doc too02:17
* jhesketh just read some missing scrollback02:18
fungiasselin_: what specifically are you wanting to do with said feature branch? i likely skimmed too lightly02:18
jheskethjeblair, fungi: are you saying gerrit 2.8 would fastfoward onto the newest patchset but 2.11 doesn't? (or am I reading the opposite)02:18
fungijhesketh: no, 2.8 wouldn't either02:18
clarkbfungi: aiui they are going to refactor ciwatch and add tests and felt that doing such heavy lifting on master would be problematic for keeping master working02:18
jheskethThelo: there will be a small delay, but it shouldn't be too long02:18
clarkbfungi: so basically let master keep running as is, then off to the side take a chainsaw to it to refactor and make it testable02:19
jheskethfungi: hmm okay.. I wonder why I didn't notice this in 2.8.. maybe I just missed it02:19
*** tzn has quit IRC02:19
fungijhesketh: 2.8 (and likely 2.11 as well) will fast-forward a branch with intermediate commits it knows about02:19
jheskethfungi: so when a parent moves forward we need to rebase the children?02:19
fungii wasn't referring to updating patchsets02:19
fungibut yes, just like in 2.8, if you change a parent change to add a new patchset, the child changes become "outdated" in gerrit and need to be rebased onto the new parent02:20
*** armax has joined #openstack-infra02:21
jheskethokay, for some reason I was under the impression that if a child was behind outdated parents a +W would do the correct thing and put it in the gate02:21
jheskethgood to know, thanks for your help :-)02:21
*** mudassirlatif has quit IRC02:22
fungiyeah, never did i'm pretty positive. i recall dealing with it regularly in 2.802:23
fungiand earlier gerrit for that matter02:23
openstackgerritMikhail S Medvedev proposed openstack-infra/puppet-ciwatch: Add initial implementation to deploy ciwatch
*** zhurong has joined #openstack-infra02:24
EmilienMhello, can I have a review on please ? it's openstack-infra/os-loganalyze02:25
jheskethEmilienM: will look02:26
EmilienMjhesketh: thanks!02:26
EmilienMthat's not urgent02:26
*** piet has joined #openstack-infra02:28
*** zhurong has quit IRC02:32
*** zhurong has joined #openstack-infra02:36
*** otter768 has joined #openstack-infra02:37
*** dimtruck is now known as zz_dimtruck02:38
*** markmcclain has quit IRC02:39
*** markmcclain has joined #openstack-infra02:40
*** otter768 has quit IRC02:41
Thelojhesketh: Ok I see the log now, right now I am tryting to trigger basic job each time there is a commit on the sandbox, but although I see the log, no job kick off, do you know what I should look at?02:41
jheskethThelo: you have to have a worker (eg jenkins) register with the job name or zuul will return a NOT_REGISTERED error02:42
jheskethThelo: and your pipeline triggers need to be set to trigger on a patchset created event02:43
TheloI do have jenkins how do I know if its register to zuul?02:44
jheskethThelo: connecting to gearman using telnet you can use 'status' to see what is registered02:45
jheskethThelo: although it sounds like you need to make sure the trigger is firing first02:45
Thelojhesketh: do you know on which port gearman listens by default?02:49
jheskethThelo: 473002:49
*** piet has quit IRC02:50
*** ccrouch has joined #openstack-infra02:50
Thelojhesketh: so here if I have the following configuration ( a job should be kick off on jenkins right?02:51
*** rossella_s has quit IRC02:52
*** rossella_s has joined #openstack-infra02:52
jheskethThelo: so status shows "total, running, workers"02:54
jheskethso you have 2 workers registered for the job which is good02:54
jheskethso the question is why it isn't queuing02:54
mmedvedeThelo: you might need to change last line of layout.yaml to '- dsvm-tempest-my-cinder-driver'02:55
Thelojhesketh: (thank for the explaination)02:55
*** hdd has joined #openstack-infra02:56
Thelommedvede: why?02:56
jheskethThelo: so in the logs you should see the event from gerrit and something comparing it to a trigger02:56
mmedvedeThelo: never mind, I did not see the dsvm-tempest-full in your gearman paste02:56
jheskethThelo: hmm, you have no source defined in your pipeline... I wonder if that has something to do with it02:56
jheskethThelo: did you look at the zuul/etc/zuul.conf.sample ?02:57
*** hdd has quit IRC02:57
*** dtardivel has quit IRC02:57
openstackgerritMerged openstack/requirements: Bump neutronclient in upper-constraints to 4.0.0
TheloI dont have this file? where should it be?02:58
Thelojhesketh: in the log I dont see any place where it is "comparing" it to a trigger, do you have a log example?03:00
*** zz_dimtruck is now known as dimtruck03:01
mmedvedeThelo: the conf sample is in zuul repo03:01
jheskethThelo: actually you don't need the source, it defaults to gerrit03:01
*** sigmavirus24_awa is now known as sigmavirus2403:02
*** kushal has quit IRC03:02
jheskethThelo: look for something like "Processing trigger event" in the logs03:03
Thelojhesketh : This is the patchset I made this morning : zuul.Scheduler: Processing trigger event <TriggerEvent patchset-created openstack-dev/sandbox master 266638,1>03:05
jheskethThelo: okay any clues around that log line03:05
openstackgerritMerged openstack/requirements: Adds networking-hyperv to projects.txt
*** yamamoto_ has joined #openstack-infra03:07
Thelothat line : ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit.                    Can not get account information.03:07
jheskethThelo: that event is for openstack-dev/sandbox, your layout.yaml is configured for the project openstack-dev/ci-sandbox03:08
jheskethThelo: so you'll probably want to update your layout03:08
jheskethThelo: the EventConnector issue doesn't matter03:08
*** mtanino has joined #openstack-infra03:08
Thelojhesketh: very true03:09
*** gildub has joined #openstack-infra03:10
*** thorst has joined #openstack-infra03:12
*** markmcclain has quit IRC03:12
Thelojhesketh, mmedvede: Many thanks I will try again03:13
jheskethno worries :-)03:13
*** kushal has joined #openstack-infra03:13
*** markmcclain has joined #openstack-infra03:15
*** apoorvad has quit IRC03:16
*** otter768 has joined #openstack-infra03:17
*** yamahata has quit IRC03:17
*** dims has joined #openstack-infra03:17
*** thorst has quit IRC03:19
*** ekarlso has joined #openstack-infra03:19
Thelojhesketh: I get the same result.03:19
jheskethThelo: got some new logs please?03:19
*** yuanying has quit IRC03:19
*** mriedem has quit IRC03:20
*** weshay_xchat has quit IRC03:21
*** unicell has quit IRC03:21
jheskethThelo: did you restart zuul after you updated your layout.yaml03:22
jheskethor reconfigure it03:22
TheloI didn't updated layout.yaml03:22
*** binbincong has joined #openstack-infra03:22
TheloI just did another commit on the right branch03:22
Thelobut when I try to stop zuul I get this error message : No process in pidfile '/var/run/zuul/' found running; none killed.03:23
*** links has joined #openstack-infra03:23
*** rkukura has quit IRC03:24
*** claudiub has quit IRC03:24
dougwigThelo: do you see the pipelines initialized in /var/log/zuul/zuul.log ?  Does 'service zuul stop' actually show it stopped in 'ps auxw | grep zuul' ?  If not, you might be running afoul of the scheduler lock bug I was having last week.03:25
jheskethThelo: you may have to kill the process and make sure the pid file is writeable by the daemon user03:25
jheskethfile/path that is03:25
jheskethThelo: once you've got it running again, do a reload so it prints out the pipeline configuration and paste-bin that if you can please03:26
*** edmondsw has quit IRC03:27
Thelodougwig : I dont see any initialization log03:28
Thelodougwig: the zuul process are still there, can I just kill them and restart?03:29
dougwigyou'll have to kill -9 them.  then one sec, i have something for you to try.03:29
dougwigi'd suggest trying this, if it's hung on you repeated, and you don't see your pipelines initializing03:31
*** sigmavirus24 is now known as sigmavirus24_awa03:32
Theloin zuul.log I only see this 2016-01-13 11:30:29,344 INFO zuul.Server: Starting scheduler03:32
openstackgerritTony Breeds proposed openstack-infra/project-config: Set BRANCH for periodic unittest bitrot jobs
*** dims has quit IRC03:33
*** yuanying has joined #openstack-infra03:33
*** gyee has quit IRC03:34
dougwigThelo: after you tried the above?  i'd expect that before, if you hit the same issue that I did.03:34
jheskethdougwig: why are you undoing the apscheduler bump?03:35
*** gongysh has joined #openstack-infra03:36
dougwigjhesketh: because that was the cause of my zuul hang.03:36
Thelodougwig I am trying03:36
*** vivekd has joined #openstack-infra03:36
*** woodster_ has quit IRC03:36
*** rlandy has quit IRC03:36
jheskethdougwig: do you have any more details about the zuul hang?03:36
jheskethie we should probably fix it if it's wrong03:37
dougwigjhesketh: yes, one sec.  phone just rang03:37
*** coolsvap|away is now known as coolsvap03:37
*** aeng_ has quit IRC03:37
*** markmcclain has quit IRC03:37
Thelook now I see things
*** Daisy has quit IRC03:38
*** markmcclain has joined #openstack-infra03:38
dougwigThelo: ok, yikes, you had the same bug.  jhesketh, standby.03:40
*** annegent_ has quit IRC03:40
dougwigThelo: you can continue with your CI with that revision of zuul, btw.03:41
Thelodougwig: I just ran into more issues03:43
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node
jheskeththat LOG PATH can be covered by
jheskethyou'll need to check your success/failure patterns in the layout.yaml03:46
*** unicell has joined #openstack-infra03:46
Thelothat one is more explicit03:47
jheskethThelo: line 46 looks promising there03:48
Thelothat does not look normal : zuul.Scheduler: Project None not found03:50
*** unicell1 has joined #openstack-infra03:50
*** unicell has quit IRC03:50
*** yuanying_ has joined #openstack-infra03:52
*** yuanying has quit IRC03:52
*** zhurong has quit IRC03:52
*** markmcclain has quit IRC03:52
mmedvedeThelo: that is normal in a sense, there is no project associated with ref-replicated event, ignore it03:53
Thelommedvede: ok thanks03:54
*** zhurong has joined #openstack-infra03:56
*** annegentle has joined #openstack-infra04:00
Thelojhesketh: INFO zuul.IndependentPipelineManager: Unable to merge change <Change 0x7fa720079850 266657,3>04:03
Thelojhesketh: I see some error in zull-merger now04:04
*** Daisy has joined #openstack-infra04:04
jheskethThelo: so possibly need to look at zuul-merger04:04
jheskethare your /var/lib/zuul's owned correctly04:04
*** otter768 has quit IRC04:04
Thelowho should be the owner? and what should be the right permission04:05
*** yuanying_ has quit IRC04:05
Thelothey should all belong to zuul right?04:05
*** yuanying has joined #openstack-infra04:06
TheloIn my case replacing zuul as mentioned by dougwig might have change some item to root owner04:06
dougwigjhesketh: ping, sorry for disappearing.04:07
dougwigThelo: shouldn't have, unless you manually started zuul by hand.04:07
*** yuanying_ has joined #openstack-infra04:07
jheskethThelo: looks like you don't have the merger ssh key set up correctly for cloning from gerrit04:07
jheskethThelo: see line 32 of paste04:07
dougwigThelo: you can ignore your ref-replicated errors.04:07
dougwigthe traceback is a problem with your file, usually.04:08
*** Daisy_ has joined #openstack-infra04:08
dougwigjhesketh: I mentioned the zuul issue last week, though it was just me.  my zuul started hanging right after the apscheduler bump, trying to launch a  thread for the timer trigger.  i'm not sure if it was an incompatible mix of package versions or what, but the apscheduler bump absolutely hosed my zuul.04:08
dougwig /though/thought04:09
*** bpokorny has joined #openstack-infra04:09
jheskethhmm okay04:09
jheskethdougwig: do you have logs?04:09
*** tongli has joined #openstack-infra04:10
dougwigmerger logs were fine.  the zuul logs had one line "Starting scheduler...", and *nothing* else.  I actually put LOG.debug's into the zuul py files to find that it was bombing while trying to import the timer trigger on startup.04:10
*** yuanying has quit IRC04:10
dougwigand nothing but a kill -9 would kill zuul-server. it was wedged up tight.04:10
*** salv-orl_ has joined #openstack-infra04:10
*** ccrouch has quit IRC04:11
Thelojhesketh: I dont see anything line 3204:11
*** ccrouch has joined #openstack-infra04:11
jheskethThelo: "Host key verification failed."04:11
jheskethThelo: you need to add gerrit to your zuul user's known_hosts file04:11
jheskethThe host keys are here:
*** Daisy has quit IRC04:12
*** Daisy_ is now known as Daisy04:12
dougwigjhesketh: there's only one in the current thirdparty ci puppet, so i'm guessing that's the missing key.04:13
*** salv-orlando has quit IRC04:13
*** kushal has quit IRC04:13
*** tzn has joined #openstack-infra04:16
*** thorst has joined #openstack-infra04:17
Thelojhesketh: Thanks I couldn't find the log you mentioned but after adding the key it seems to work04:19
*** kushal has joined #openstack-infra04:20
*** tzn has quit IRC04:20
Theloone more question, what is this noop-check-communication?04:21
*** tongli has quit IRC04:23
*** ccrouch has quit IRC04:24
*** markmcclain has joined #openstack-infra04:24
*** tongli has joined #openstack-infra04:24
*** thorst has quit IRC04:25
dougwigThelo: that just a starter job that does nothing but echo hello world, for testing that your zuul/gearman/jenkins and all are working properly.04:25
*** vivekd_ has joined #openstack-infra04:25
dougwig /that/that's/04:25
*** tongli has quit IRC04:26
Thelodougwig: THanks does it report something to gerrit?04:26
*** tongli has joined #openstack-infra04:26
dougwigif it's in silent in zuul layout.yaml, no.  if it's in check, yes.04:26
Thelommm in my case it didn't report anything04:27
Thelohowever dsvm-tempest-full was trigger in jenkins04:27
Theloso there is something wrong right?04:27
*** bharathm has joined #openstack-infra04:27
*** vivekd has quit IRC04:27
*** vivekd_ is now known as vivekd04:27
dougwigi had issues with that job sitting in jenkins queue and not running on master like it was supposed to.  i moved it to the dpc nodes and it started running fine. i think there's a setup bug in that part.04:28
*** markmcclain has quit IRC04:29
*** Daisy has quit IRC04:30
*** tongli has quit IRC04:30
jheskethThelo: now all the problems are solved, do you want to try upgrading to zuul head and seeing if it still works04:30
jheskethwould be helpful to see if you've got the same problem as dougwig if you don't mind04:30
openstackgerritsonu proposed openstack-infra/project-config: Adds the Bandit job for Designate in OpenStack Infra
dougwigjhesketh, Thelo: ooh yes, that would be a useful test.04:31
*** tongli has joined #openstack-infra04:31
Thelojust tell me how to do04:31
*** bharathm has quit IRC04:31
dougwigThelo: one sec04:33
Thelodougwig shouldn't I uninstall zuul before installing it again?04:35
*** tongli has quit IRC04:35
dougwigthe upgrade would be better, because you might keep some min rev packages.  you can always try fresh after if it works somehow.04:36
*** tongli has joined #openstack-infra04:36
*** treaki_ has joined #openstack-infra04:37
Thelodougwig: not sure that things were actually upgraded :
dougwigok, then "pip uninstall zuul ; pip install zuul".04:39
dougwigyou'll know you have it right when "pip list | grep apsched" returns a version of >3.x04:39
*** vivekd_ has joined #openstack-infra04:39
*** tongli has quit IRC04:40
*** dimtruck is now known as zz_dimtruck04:40
*** vivekd has quit IRC04:41
*** vivekd_ is now known as vivekd04:41
Thelopiplist | grep apsch returns nothing.04:41
*** treaki__ has quit IRC04:41
dougwigtry this:04:43
dougwigpython -c "import apscheduler; print apscheduler.__version__"04:43
TheloAttributeError: 'module' object has no attribute '__version__'04:43
dougwigthat's the old one.04:44
*** zhurong has quit IRC04:44
dougwigpip install -U git+
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: add py27 py34 pypy check for kolla project
*** zhurong has joined #openstack-infra04:46
*** sdake_ has joined #openstack-infra04:47
*** sdake has quit IRC04:48
Thelodougwig: I ok I have the right version now04:48
TheloI am triggering a new commit04:48
dougwigThelo: great.  fire up that zuul and let's see what happens.04:48
zarojhesketh: take another look?
Theloit seems ok04:49
Thelodougwig: For this commit I should trigger both noop-check-communication and dsvm tempest-fuul, in jenkins it looks like it proccess them one by one04:50
Thelowhy not in parrallel>04:50
dougwigif you see zuul pipelines, it's working. drat that it didn't repro.04:50
dougwigdo you have enough jobs available to run both at the same time?04:51
TheloI have 4 slaves, all of them are up but only one of them is working04:51
*** sdake_ has quit IRC04:53
TheloO I see it is node affected to any nodes type04:54
Thelois it normal?04:55
*** sdake has joined #openstack-infra04:55
*** hdd has joined #openstack-infra04:56
*** Daisy has joined #openstack-infra04:56
*** zhurong has quit IRC04:59
*** jamespage has quit IRC05:00
*** gnuoy has quit IRC05:00
*** gnuoy has joined #openstack-infra05:00
jheskethzaro: looking05:02
*** Sukhdev has joined #openstack-infra05:03
*** zhurong has joined #openstack-infra05:03
*** boris-42 has quit IRC05:03
*** piet has joined #openstack-infra05:04
dougwigThelo: if your node doesn't match your job, it's normal to not run. that's all about your config.05:06
*** Swami has joined #openstack-infra05:08
openstackgerritQiming Teng proposed openstack-infra/irc-meetings: Fix broken ptl list tool
*** markmcclain has joined #openstack-infra05:12
*** sdake has quit IRC05:16
*** tzn has joined #openstack-infra05:17
*** sridhar_ram has joined #openstack-infra05:18
*** jamespage has joined #openstack-infra05:19
*** markmcclain has quit IRC05:21
*** thorst has joined #openstack-infra05:21
Thelodougwig: Thanks05:22
*** tzn has quit IRC05:22
*** thorst has quit IRC05:29
*** annegentle has quit IRC05:29
*** mrmartin has joined #openstack-infra05:29
*** Qiming_ has joined #openstack-infra05:30
openstackgerritsonu proposed openstack-infra/project-config: Adds the Bandit job for Designate in OpenStack Infra
*** Qiming has quit IRC05:33
*** piet has quit IRC05:38
*** jaosorior has joined #openstack-infra05:41
*** mtanino has quit IRC05:43
*** _nadya_ has joined #openstack-infra05:48
*** markmcclain has joined #openstack-infra05:50
*** bpokorny has quit IRC05:53
*** markmcclain has quit IRC05:56
*** mrmartin has quit IRC06:01
*** mrmartin has joined #openstack-infra06:03
*** mrmartin has quit IRC06:08
*** vgridnev has joined #openstack-infra06:13
*** _nadya_ has quit IRC06:14
*** tzn has joined #openstack-infra06:18
*** ParsectiX has quit IRC06:22
*** tzn has quit IRC06:22
*** vivekd has quit IRC06:23
*** gongysh has quit IRC06:24
*** thorst has joined #openstack-infra06:26
*** markvoelker has quit IRC06:27
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** ParsectiX has joined #openstack-infra06:33
*** thorst has quit IRC06:34
*** kzaitsev_mb has joined #openstack-infra06:34
*** amit213 has quit IRC06:35
*** hdd has quit IRC06:35
*** amit213 has joined #openstack-infra06:36
openstackgerritsonu proposed openstack-infra/project-config: Add the Bandit gate to Designate section in OpenStack Infra
*** vivekd has joined #openstack-infra06:40
*** yamahata has joined #openstack-infra06:48
*** mrunge has joined #openstack-infra06:48
*** zhurong has quit IRC06:49
*** rossella_s has quit IRC06:52
*** Sukhdev has quit IRC06:52
*** rossella_s has joined #openstack-infra06:52
*** jaosorior has quit IRC06:57
*** aopincar_ has joined #openstack-infra06:58
*** sridhar_ram has quit IRC07:02
*** sridhar_ram has joined #openstack-infra07:06
*** Swami has quit IRC07:10
*** mudassirlatif has joined #openstack-infra07:12
*** med_ has quit IRC07:12
*** med_ has joined #openstack-infra07:14
*** med_ is now known as Guest7712107:14
*** zhurong has joined #openstack-infra07:15
*** vivekd has quit IRC07:15
*** isaacb has joined #openstack-infra07:17
*** tzn has joined #openstack-infra07:19
*** gildub has quit IRC07:19
*** zhurong has quit IRC07:20
*** zhurong has joined #openstack-infra07:23
*** tzn has quit IRC07:23
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: add py27 py34 pypy check for kolla project
*** markvoelker has joined #openstack-infra07:28
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: add py27 py34 check for kolla project
*** jtomasek has joined #openstack-infra07:28
*** thorst has joined #openstack-infra07:31
*** markvoelker has quit IRC07:33
*** cnesa has joined #openstack-infra07:33
*** thorst has quit IRC07:39
*** _nadya_ has joined #openstack-infra07:39
*** e0ne has joined #openstack-infra07:41
*** kzaitsev_mb has quit IRC07:41
*** vgridnev has quit IRC07:42
AJaegerThere's a job that has been approved which is not entering the gate - even after several rechecks: Could an infra-root investigate, please?07:45
*** tzn has joined #openstack-infra07:47
yolandahi AJaeger, morning07:50
yolandai can take a look07:50
AJaegermorning, yolanda !07:50
*** YorikSar has joined #openstack-infra07:52
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: add py27 py34 docs check for kolla project
yolandaAJaeger, so it will need another +1 from someone. recheck will make it go to check queue, but not to gate. It needs an extra +w from another reviewer, or -w +w from dims07:54
YorikSarOh, it looks like eavesdrop lags a bit here.07:56
YorikSaryolanda: I just added +W there and it got to Gate queue.07:56
YorikSar(I assume you're talking about
*** mpavone has joined #openstack-infra07:57
AJaegeryolanda: but why? Looking at history joshua harlow gave +2, then dims a +W, then dims a +207:57
AJaegerYorikSar: yes07:57
AJaegerSo, it should have gone to gate already after dims' +W07:57
yolandaAJaeger, maybe there was some restart, or some unexpected problem that caused that?07:58
YorikSarMy guess is that Zuul missed the original Verified+1 from Jenkins and it didn't trigger smth in Zuul.07:58
yolandayep, that is normally happening when for some reason gerrit fails to send the event, or zuul to pick it up07:58
*** ifarkas has joined #openstack-infra07:59
YorikSarLast reconfigured: Tue Jan 12 2016 14:36:48 GMT+030008:00
openstackgerritMerged openstack-infra/project-config: Fixed mistral experimental job
YorikSarMaybe Zuul forgot about some events after restart?08:00
*** otter768 has joined #openstack-infra08:00
AJaegerYorikSar: reconfigured is just reload, restart is different - and that one happened as well this week.08:00
*** ildikov has quit IRC08:00
AJaegeryolanda: might be. Ok, case solved for now - thanks!08:00
*** otter768 has quit IRC08:02
YorikSaryolanda: Do you know if Zuul periodically checks if it missed something from event stream?08:02
*** otter768 has joined #openstack-infra08:02
yolandano, it doesn't08:02
YorikSarI wonder if it would be a good idea to query Gerrit once in a while (an hour may be) for changes that should be in queues an verify that those are in queues.08:03
*** salv-orl_ has quit IRC08:04
*** otter768 has quit IRC08:04
*** salv-orlando has joined #openstack-infra08:04
yolandait's a recurring problem we have, but we are solving manually. It seems a useful idea to me, but i'm not sure about the complexity08:04
*** otter768_ has joined #openstack-infra08:04
AJaegerYorikSar: this is a corner case and worth doing after a restart I guess - better discuss with the Zuul experts at US time.08:04
yolandaYorikSar, you can throw that to the mailing list08:04
*** vgridnev has joined #openstack-infra08:04
*** Thelo has quit IRC08:04
AJaegerthanks for the suggestions, YorikSar08:06
*** otter768_ has quit IRC08:06
*** otter768 has joined #openstack-infra08:06
*** jistr has joined #openstack-infra08:07
*** tzn has quit IRC08:07
*** Thelo has joined #openstack-infra08:07
*** jistr is now known as jistr|doc08:07
*** otter768 has quit IRC08:08
*** otter768_ has joined #openstack-infra08:08
openstackgerrityolanda.robla proposed openstack-infra/ansible-puppet: Conditionally manage puppet config files
YorikSarOk, I'll write it up to ML later today.08:08
*** otter768 has joined #openstack-infra08:10
*** otter768_ has quit IRC08:10
*** Hal has joined #openstack-infra08:11
*** Hal is now known as Guest2264908:11
*** flepied1 has quit IRC08:11
*** otter768_ has joined #openstack-infra08:12
openstackgerritMerged openstack/requirements: ryu 3.29 incompatible with oslo.config
*** otter768_ has quit IRC08:13
*** markus_z has joined #openstack-infra08:13
*** otter768_ has joined #openstack-infra08:13
*** otter768 has quit IRC08:15
*** abregman has joined #openstack-infra08:15
*** otter768 has joined #openstack-infra08:15
*** otter768_ has quit IRC08:16
Thelojhesketh, dougwig : are you still here?08:17
jheskethThelo: in and out, but yes08:17
Thelozuul stops at Starting scheduler again08:17
*** otter768_ has joined #openstack-infra08:17
*** otter768 has quit IRC08:17
jheskethThelo: oh that's interesting... so you might be seeing what dougwig was08:18
jheskethThelo: can you post your logs somewhere per chance?08:18
*** e0ne has quit IRC08:18
*** otter768 has joined #openstack-infra08:19
*** otter768_ has quit IRC08:19
*** otter768_ has joined #openstack-infra08:21
*** otter768 has quit IRC08:21
jheskethThelo: could you do a pip freeze please?08:22
*** otter768_ has quit IRC08:23
*** otter768 has joined #openstack-infra08:23
Thelojhesketh: however I have already replace zuul08:23
jheskethThelo: you reverted to before the APScheduler update?08:24
*** shardy has joined #openstack-infra08:24
Thelojhesketh: not intentionnally at least08:25
*** otter768 has quit IRC08:25
jheskethThelo: what do you mean by replace zuul then?08:25
*** otter768 has joined #openstack-infra08:25
*** vgridnev has quit IRC08:25
*** otter768_ has joined #openstack-infra08:27
*** HeOS has quit IRC08:27
*** dtardivel has joined #openstack-infra08:28
*** otter76__ has joined #openstack-infra08:29
*** otter768 has quit IRC08:29
*** otter768_ has quit IRC08:29
Thelojhesketh: Even when I do doug's method it still stuck at the same point08:29
*** matrohon has joined #openstack-infra08:29
*** ildikov has joined #openstack-infra08:29
*** mudassirlatif has quit IRC08:29
jheskethhmm okay08:29
openstackgerrityolanda.robla proposed openstack-infra/ansible-puppet: Conditionally manage puppet config files
jheskethThelo: are you looking at the debug log, or just zuul.log?08:30
jheskethThelo: I assume you've restarted zuul and cleared out /var/lib/zuul etc08:30
Thelojhesketh: What should I clean in /var/lib/zuul?08:31
*** otter768 has joined #openstack-infra08:31
*** otter76__ has quit IRC08:31
jheskethwhen restarting it's just useful to clear out any state that zuul might have stored08:31
jheskethyou should be able to clear all of it out reasonably safely08:31
*** yaume has joined #openstack-infra08:32
*** yamahata has quit IRC08:32
*** otter768 has quit IRC08:32
*** samuelBartel has joined #openstack-infra08:32
*** otter768 has joined #openstack-infra08:33
*** GheRiver1 has joined #openstack-infra08:34
yolandaAJaeger, do you know if there is some problem with ansible-lint checks08:34
yolandai see my tests failing with 2016-01-13 08:32:50.448 | AttributeError: 'module' object has no attribute 'parse_yaml_from_file'08:34
*** otter768_ has joined #openstack-infra08:34
*** GheRiver1 has quit IRC08:35
*** esikachev has joined #openstack-infra08:36
*** otter768_ has quit IRC08:36
*** otter76__ has joined #openstack-infra08:36
AJaegeryolanda: No idea, do you have a change to look at?08:37
*** arxcruz has joined #openstack-infra08:37
*** thorst has joined #openstack-infra08:37
*** otter768 has quit IRC08:37
*** kushal has quit IRC08:37
AJaegeryolanda: ansible 2.0 was released yesterday, might be a fall out from that08:38
*** otter768 has joined #openstack-infra08:38
*** otter76__ has quit IRC08:38
yolandait may be a problem with the test itself, i'm running using ansible 2.0 and works fine08:39
*** nmagnezi has joined #openstack-infra08:39
AJaegercan't help with that - perhaps ask pabelanger08:39
yolandayep, i'll try find some time to take a look, or ping him later08:39
*** shardy has quit IRC08:40
*** otter768 has quit IRC08:40
*** otter768_ has joined #openstack-infra08:40
*** daemontool has joined #openstack-infra08:40
*** flepied has joined #openstack-infra08:40
*** fhubik has joined #openstack-infra08:41
*** otter768_ has quit IRC08:42
*** dingyichen has quit IRC08:42
*** shardy has joined #openstack-infra08:42
*** otter768 has joined #openstack-infra08:42
*** thorst has quit IRC08:43
*** ihrachys has joined #openstack-infra08:43
*** daemontool_ has joined #openstack-infra08:43
*** daemontool has quit IRC08:43
*** otter768_ has joined #openstack-infra08:44
*** daemontool_ is now known as daemontool08:44
*** otter768 has quit IRC08:44
*** jlanoux has joined #openstack-infra08:45
*** otter768 has joined #openstack-infra08:46
openstackgerritIWAMOTO Toshihiro proposed openstack/requirements: Bump ryu to 3.29.1
*** otter768_ has quit IRC08:46
*** otter768 has quit IRC08:48
*** otter768_ has joined #openstack-infra08:48
*** kushal has joined #openstack-infra08:49
*** otter768_ has quit IRC08:50
*** otter768 has joined #openstack-infra08:50
*** otter768_ has joined #openstack-infra08:52
*** otter768 has quit IRC08:52
Thelojhesketh: actually I just modified the layout.yaml then it stoped working, after I rolled back it seems fine...08:52
Thelojhesketh: do you see anything wrong in the commit before the last one (
*** otter768 has joined #openstack-infra08:54
*** markvoelker has joined #openstack-infra08:54
*** otter768 has quit IRC08:55
*** otter76__ has joined #openstack-infra08:55
*** otter768_ has quit IRC08:56
*** otter768 has joined #openstack-infra08:57
*** sshnaidm has quit IRC08:59
*** markvoelker has quit IRC08:59
*** otter768_ has joined #openstack-infra08:59
*** otter768 has quit IRC08:59
*** otter76__ has quit IRC09:00
*** otter768_ has quit IRC09:01
*** kzaitsev_mb has joined #openstack-infra09:01
*** salv-orlando has quit IRC09:02
*** MCoLo has joined #openstack-infra09:02
*** dizquierdo has joined #openstack-infra09:02
*** sfinucan has joined #openstack-infra09:02
*** salv-orlando has joined #openstack-infra09:03
*** gongysh has joined #openstack-infra09:03
*** otter768 has joined #openstack-infra09:03
*** otter768 has quit IRC09:05
*** otter768_ has joined #openstack-infra09:05
*** otter768 has joined #openstack-infra09:07
Thelojhesketh: problem fixed09:08
*** otter768 has quit IRC09:08
openstackgerritMerged openstack-infra/project-config: Update Watcher Tempest tests package name
*** otter768 has joined #openstack-infra09:09
oomichihi, can somebody take alook at a cliff patch?
*** amotoki has joined #openstack-infra09:10
oomichithe pep8 failure happens on the gate now, and the patch fixes it.09:10
*** otter768_ has quit IRC09:10
openstackgerritChris Jones proposed openstack-infra/project-config: Add project configuration for Vahana.
*** zhurong has quit IRC09:11
*** GheRiver1 has joined #openstack-infra09:11
*** otter768_ has joined #openstack-infra09:11
*** otter768 has quit IRC09:11
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: add py27 py34 docs check for kolla project
*** yaume_ has joined #openstack-infra09:11
*** MCoLo has quit IRC09:12
*** MCoLo has joined #openstack-infra09:12
*** GheRiver1 has quit IRC09:12
*** otter768_ has quit IRC09:13
*** otter768 has joined #openstack-infra09:13
*** yaume has quit IRC09:14
*** otter768_ has joined #openstack-infra09:15
*** vivekd has joined #openstack-infra09:15
*** otter768 has quit IRC09:15
openstackgerritMerged openstack-infra/project-config: Add osc-plugin-jobs check job for all OSC plugins
openstackgerritMerged openstack-infra/project-config: add ironic-inspector and tripleo to osc-check-plugins job
*** otter768 has joined #openstack-infra09:17
AJaegeroomichi: channel for cliff is #openstack-sdks - and probably also #openstack-oslo09:17
openstackgerritMerged openstack-infra/project-config: Use file permissions instead of ownership to grant access to files
*** jaosorior has joined #openstack-infra09:18
oomichiAJaeger: thanks, nice info :)09:18
*** otter768 has quit IRC09:19
*** otter76__ has joined #openstack-infra09:19
*** HeOS has joined #openstack-infra09:19
*** jlanoux has quit IRC09:19
*** otter768_ has quit IRC09:20
*** otter76__ has quit IRC09:20
*** jlanoux has joined #openstack-infra09:20
*** otter768 has joined #openstack-infra09:20
*** otter768 has quit IRC09:22
TheloHi all, my jenkins slave tried to git fetch the code from my CI server instead of the openstack review server ( does anyone know why?09:22
*** otter768 has joined #openstack-infra09:22
*** zeih has joined #openstack-infra09:24
*** otter768_ has joined #openstack-infra09:24
*** otter768 has quit IRC09:24
*** otter768_ has quit IRC09:26
*** otter768 has joined #openstack-infra09:26
*** AJaeger has quit IRC09:27
*** otter768 has quit IRC09:28
*** salv-orlando has quit IRC09:28
*** otter768 has joined #openstack-infra09:28
*** salv-orlando has joined #openstack-infra09:28
*** vgridnev has joined #openstack-infra09:29
*** otter768 has quit IRC09:30
*** otter768_ has joined #openstack-infra09:30
odyssey4meyolanda yes, ansible-lint is not yet compatible with ansible 2.009:31
*** otter768_ has quit IRC09:32
*** otter768 has joined #openstack-infra09:32
odyssey4meyolanda for those of us using ansible-lint/ansible, we've currently capped ansible at <2.0 until ansible-lint has released a fix09:32
*** e0ne has joined #openstack-infra09:33
*** otter768_ has joined #openstack-infra09:34
*** otter768 has quit IRC09:34
*** derekh has joined #openstack-infra09:35
*** dizquierdo has quit IRC09:36
*** otter768_ has quit IRC09:36
*** otter768 has joined #openstack-infra09:36
yolandaodyssey4me, thanks, i'll try that fix09:37
*** otter768_ has joined #openstack-infra09:38
*** tzn has joined #openstack-infra09:38
*** gongysh has quit IRC09:38
*** tzn has quit IRC09:39
*** otter76__ has joined #openstack-infra09:39
*** otter768_ has quit IRC09:40
*** otter768 has quit IRC09:40
*** Daisy has quit IRC09:40
*** thorst has joined #openstack-infra09:41
*** electrofelix has joined #openstack-infra09:41
*** otter76__ has quit IRC09:41
*** otter768 has joined #openstack-infra09:41
*** otter768 has quit IRC09:43
*** otter768_ has joined #openstack-infra09:44
*** otter768_ has quit IRC09:45
*** otter768 has joined #openstack-infra09:45
*** sshnaidm has joined #openstack-infra09:46
*** otter768 has quit IRC09:47
*** otter768 has joined #openstack-infra09:47
*** oomichi is now known as oomichi_away09:47
*** thorst has quit IRC09:48
Theloyolanda: Morning, in the openstack CI do you know where the variables "ZUUL_URL=""" is defined?09:49
*** otter768_ has joined #openstack-infra09:49
openstackgerritMerged openstack-infra/irc-meetings: Fix broken ptl list tool
*** otter76__ has joined #openstack-infra09:51
*** otter768_ has quit IRC09:51
yolandait is a parameter passed to zuul_scheduler or zuul_merger . For the merger it defaults to https://${::fqdn}/p, but you can override it09:51
*** otter768 has quit IRC09:52
Theloyolanda, I tried to override it in zuul.conf unsuccessfully. You meant directly in the slave jobs?09:52
*** otter768 has joined #openstack-infra09:53
*** otter76__ has quit IRC09:53
yolandai was talking about the puppet manifest itself09:54
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add support for --overcloud-update and use it for HA job
*** otter768_ has joined #openstack-infra09:55
Theloyolanda: but right value should be why is the localhost the default value?09:55
*** coolsvap is now known as coolsvap|away09:56
yolandaThelo, zuul_url is mostly used for the merger, to make it aware of the zuul server it needs to listen09:56
*** otter768_ has quit IRC09:57
*** otter76__ has joined #openstack-infra09:57 is gerrit, not zuul09:57
*** otter768 has quit IRC09:58
Theloyolanda: so why devstack try to get the code with this variable?09:58
*** otter768 has joined #openstack-infra09:59
yolandait may need a different var than the one from zuul config, sorry. That depends on job configuration09:59
*** otter768_ has joined #openstack-infra10:01
*** tzn has joined #openstack-infra10:01
Theloyolanda: you mean I should override this variable elsewhere?10:01
*** dtantsur|afk is now known as dtantsur10:02
*** otter76__ has quit IRC10:02
*** otter768_ has quit IRC10:02
*** otter768_ has joined #openstack-infra10:03
yolandasorry, i cannot help you with devstack tests so much10:03
yolandai thought you were talking about zuul configuration itself10:03
*** otter768 has quit IRC10:04
*** otter768_ has quit IRC10:04
*** otter768 has joined #openstack-infra10:05
*** otter768_ has joined #openstack-infra10:06
*** otter768 has quit IRC10:06
*** otter768 has joined #openstack-infra10:08
*** GheRiver1 has joined #openstack-infra10:08
*** salv-orl_ has joined #openstack-infra10:09
*** otter76__ has joined #openstack-infra10:10
*** otter768 has quit IRC10:10
*** otter768_ has quit IRC10:11
*** GheRiver1 has quit IRC10:12
*** otter768 has joined #openstack-infra10:12
*** otter76__ has quit IRC10:12
*** GheRiver1 has joined #openstack-infra10:12
*** salv-orlando has quit IRC10:13
*** electrofelix has quit IRC10:13
jheskethThelo: sorry was away.. so it was just a bad layout.yaml that stopped the scheduler from loading?10:13
*** otter768_ has joined #openstack-infra10:14
*** hashar has joined #openstack-infra10:15
*** cdent has joined #openstack-infra10:15
*** otter76__ has joined #openstack-infra10:16
*** otter768_ has quit IRC10:16
*** hashar has quit IRC10:16
*** salv-orl_ has quit IRC10:16
*** otter768 has quit IRC10:17
*** GheRiver1 has quit IRC10:17
*** salv-orlando has joined #openstack-infra10:17
*** GheRiver1 has joined #openstack-infra10:17
*** otter768 has joined #openstack-infra10:18
*** rguillebert has joined #openstack-infra10:19
*** Qiming_ has quit IRC10:19
*** otter768 has quit IRC10:19
*** jistr|doc is now known as jistr10:19
*** otter768 has joined #openstack-infra10:20
*** nilasae has joined #openstack-infra10:20
*** electrofelix has joined #openstack-infra10:20
*** otter76__ has quit IRC10:21
*** otter768_ has joined #openstack-infra10:22
*** otter768 has quit IRC10:22
*** vivekd_ has joined #openstack-infra10:22
*** vivekd has quit IRC10:23
*** vivekd_ is now known as vivekd10:23
*** otter768 has joined #openstack-infra10:23
*** otter768_ has quit IRC10:24
*** otter768 has quit IRC10:25
*** otter768 has joined #openstack-infra10:25
*** jistr has quit IRC10:26
*** otter768_ has joined #openstack-infra10:27
*** otter768 has quit IRC10:27
*** otter768 has joined #openstack-infra10:29
*** otter768_ has quit IRC10:29
*** gildub has joined #openstack-infra10:30
openstackgerritMerged openstack-infra/release-tools: stop sending folks to launchpad milestones for release details
*** vivekd has quit IRC10:33
*** vivekd_ has joined #openstack-infra10:33
*** vivekd_ is now known as vivekd10:33
*** dizquierdo has joined #openstack-infra10:34
*** otter768 has quit IRC10:34
openstackgerritThierry Carrez proposed openstack/requirements: Add missing license information
*** rfolco has joined #openstack-infra10:40
*** thorst has joined #openstack-infra10:46
*** _et_ has joined #openstack-infra10:47
*** savihou has joined #openstack-infra10:48
*** jistr has joined #openstack-infra10:51
*** rossella_s has quit IRC10:52
*** rossella_s has joined #openstack-infra10:52
*** thorst has quit IRC10:53
*** markvoelker has joined #openstack-infra10:55
*** markvoelker has quit IRC11:00
*** daemontool_ has joined #openstack-infra11:02
*** sputnik13_ has joined #openstack-infra11:02
*** aysyd has joined #openstack-infra11:03
*** daemontool has quit IRC11:05
*** jaosorior has quit IRC11:05
*** sputnik13 has quit IRC11:05
*** jaosorior has joined #openstack-infra11:05
*** jaosorior has quit IRC11:06
*** jaosorior has joined #openstack-infra11:07
*** lindsayk has quit IRC11:08
*** markvan has quit IRC11:09
*** markvan has joined #openstack-infra11:10
*** hughsaunders has quit IRC11:10
jklarehi, can anybody help me out with cleaning up the groups/acls and permissions for the openstack-chef project?11:12
*** lindsayk has joined #openstack-infra11:12
robcresswellHello! So I recently realised that the automatic py27 job that Horizon runs is duplicating a tox task (py27dj18) so is a waste of resources. My initial thought was to remove the dj18 task, but then when we move to django 1.9, we would have to reinstate it11:12
*** hughsaunders has joined #openstack-infra11:12
robcresswellSo I am wondering if a better long term solution is to stop using the python-jobs task, and just use tox-{envlist} with a few envs specified for Horizon11:13
robcresswellIs this viable, or have I misunderstood the usage?11:13
*** gildub has quit IRC11:15
*** Qiming has joined #openstack-infra11:16
*** lucas-dinner is now known as lucasagomes11:18
*** GheRiver1 has quit IRC11:18
*** aarefiev has quit IRC11:20
*** aarefiev has joined #openstack-infra11:21
*** fhubik is now known as fhubik_brb11:21
openstackgerritJan Klare proposed openstack-infra/project-config: acl cleanup for openstack-chef project
*** GheRiver1 has joined #openstack-infra11:21
*** GheRivero is now known as GheRivero_11:22
*** fhubik_brb is now known as fhubik11:25
*** sdake has joined #openstack-infra11:29
openstackgerritJan Klare proposed openstack-infra/project-config: acl cleanup for openstack-chef project
*** sdake has quit IRC11:34
*** jamespd has quit IRC11:34
*** jamespd has joined #openstack-infra11:35
*** aviau has quit IRC11:36
*** aviau has joined #openstack-infra11:38
openstackgerrityolanda.robla proposed openstack-infra/ansible-puppet: Pin to ansible < 2.0, it doesn't support ansible-lint
markus_zttx: Hi, do you have a minute for the meeting channel question I asked on the ML?11:42
savihouHi , any best practices regarding to troubleshooting zuul/gearman configuration?11:44
*** fhubik is now known as fhubik_brb11:44
*** kmartin has quit IRC11:45
*** flwang1 has quit IRC11:48
*** flwang1 has joined #openstack-infra11:49
*** thorst has joined #openstack-infra11:51
markus_zttx: nevermind, I checked again, and I can chose the same slots on another day of the week without losing any benefit.11:51
*** AJaeger has joined #openstack-infra11:53
*** daemontool_ has quit IRC11:54
*** markvoelker has joined #openstack-infra11:56
*** skraynev has quit IRC11:57
*** vgridnev has quit IRC11:57
*** daemontool has joined #openstack-infra11:57
*** esikachev has quit IRC11:58
*** thorst has quit IRC11:59
*** vgridnev has joined #openstack-infra11:59
betherlyhi all! i am in the process of setting up ironic-ui project. so far the governance and new project patches have merged12:00
betherlyreading through
*** markvoelker has quit IRC12:00
betherlyhow do i then go about accessing the repository and being added to groups etc?12:01
*** skraynev has joined #openstack-infra12:01
openstackgerrityolanda.robla proposed openstack-infra/project-config: Temporarily pin to ansible==1.9.4
*** doug-fish has joined #openstack-infra12:01
*** vgridnev has quit IRC12:03
yolandabetherly, i can help you12:04
yolandaif you send me the link with the project you proposed, i can add you in the groups you need12:04
openstackgerritJan Klare proposed openstack-infra/project-config: acl cleanup for openstack-chef project
betherlyyolanda: thanks so much! the review links that merged are here and
*** esikachev has joined #openstack-infra12:04
*** GheRiver1 has quit IRC12:05
*** salv-orlando has quit IRC12:05
*** baoli has joined #openstack-infra12:05
*** salv-orlando has joined #openstack-infra12:05
*** mase_x200 has joined #openstack-infra12:06
yolandabetherly, done12:06
*** baoli has quit IRC12:08
*** devananda has quit IRC12:08
*** devananda has joined #openstack-infra12:13
*** baoli has joined #openstack-infra12:16
*** baoli has quit IRC12:18
*** vgridnev has joined #openstack-infra12:18
*** rhallisey has joined #openstack-infra12:23
betherlyyolanda: thanks so much :)12:25
yolandayou are welcome12:26
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth
*** sdake has joined #openstack-infra12:27
*** sshnaidm has quit IRC12:28
*** vivekd has quit IRC12:28
*** dims has joined #openstack-infra12:28
yolandaAJaeger, infra-root, can you take a look at ? i want to fix the ansible-lint tests12:29
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth
AJaegeryolanda: It's a big hammer - can12:30
AJaegeryolanda: It's a big hammer - can't we do that only for the lint test?12:30
*** gordc has joined #openstack-infra12:30
yolandai can create a different builder for ansible-ilnt12:30
*** otter768 has joined #openstack-infra12:30
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth
robcresswellSorry for noise ^^12:31
AJaegeryolanda: that sounds better - but talk with ansible experts on the proper way here.12:31
AJaegeryolanda: I also prefer with such temporary changes a "# TODO(yolando): Pin to 1.9.4 since ansible-lint does not support 2.0" comment12:32
*** vivekd has joined #openstack-infra12:32
yolandaok i can amend12:32
AJaegeryes, we have history - but these are things that should be removed later ("temporarily")12:32
*** ociuhandu has quit IRC12:33
*** julim has joined #openstack-infra12:33
jklareAJaeger hi :) since you commented on the patch relating the changes for the openstack-chef groups: do you know what i have to do to make these adaptions to the gerrit groups / create the needed new ones ?12:33
ttxmarkus_z: good12:34
openstackgerrityolanda.robla proposed openstack-infra/project-config: Temporarily pin to ansible==1.9.4
*** otter768 has quit IRC12:35
AJaegerjklare: the new ones will be created as empty once the change merges ;)12:35
AJaegerjklare: I don't know whether we can rename groups, you have to ask others for that.12:36
jklareAJaeger and who will be able to add people to these groups?12:36
*** salv-orlando has quit IRC12:37
AJaegeryolanda: infra admins here will add *you* first and then you can add others12:37
*** salv-orlando has joined #openstack-infra12:37
*** fhubik_brb is now known as fhubik12:38
jklareAJaeger cool, ty12:40
AJaegerI meant jklare instead of yolanda above...12:41
AJaegeryolanda: your change looks fine to me12:41
yolandaAJaeger, i understood it :)12:42
*** _amrith_ is now known as amrith12:42
*** kaisers1 has joined #openstack-infra12:43
*** thorst has joined #openstack-infra12:43
AJaegerrobcresswell: you only "declare" the new jobs but you do not use them - don't "call" them.12:45
robcresswellAJaeger: Ah, I see12:46
*** kaisers1 has quit IRC12:47
*** hrubi has quit IRC12:47
*** weshay_xchat has joined #openstack-infra12:48
flip214mmedvede: ping?12:50
*** claudiub has joined #openstack-infra12:50
openstackgerritFabio Giannetti proposed openstack/requirements: Add python-monascaclient to the global requirements
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth
openstackgerritnikeshmahalka proposed openstack/requirements: Add google api python client
robcresswellAJaeger: The aim here is to establish a voting test for dj18, and non-voting for dj19. When we up requirements, I'll remove the dj19 test (as that will be covered by py27)12:54
AJaegerrobcresswell: you can also - as you mentioned earlier - remove py27.12:55
robcresswellYes, I was curious about that but nobody responded (or I missed it)12:56
robcresswellIs it reasonable to remove python-jobs and just use our tox tasks? It would slightly reduce our UT runs, which is always good.12:56
*** jcoufal has joined #openstack-infra12:57
*** markvoelker has joined #openstack-infra12:57
*** burgerk has joined #openstack-infra12:57
AJaegerrobcresswell: yes, that's reasonable.12:58
AJaegerAnd you should first add the job as non-voting to check queue, once it's voting add it to gate as well12:58
robcresswellAJaeger: Great. I'll propose a separate patch for that.12:58
*** xyang1 has joined #openstack-infra12:58
robcresswellAJaeger: Yep, I saw tsufiev comment on it. It's a good point, I'll change now.12:58
*** amuller has joined #openstack-infra12:59
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth
*** sshnaidm has joined #openstack-infra12:59
*** markvoelker has quit IRC13:02
*** vgridnev has quit IRC13:02
*** links has quit IRC13:04
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth
robcresswellAJaeger: Sorry, that regex was lazy on my part. Not sure what I was thinking.13:04
*** ociuhandu has joined #openstack-infra13:04
*** akshai has joined #openstack-infra13:04
AJaegerrobcresswell: please try again - the first regex misses one )13:05
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth
AJaegerfixed ;)13:05
*** vivekd has quit IRC13:05
AJaegerrobcresswell: LGTM, +213:06
robcresswellYeah, my vim autocompletes brackets, sometimes on double )) it gets muddled.13:06
robcresswellAJaeger: Thanks for working with me on that!13:06
AJaegeryou're welcome - let's hope Zuul is happy with it as well ;)13:06
*** aopincar_ has quit IRC13:06
openstackgerritFabio Giannetti proposed openstack/requirements: Add python-monascaclient to the global requirements
*** akshai has quit IRC13:08
*** rcernin has joined #openstack-infra13:09
openstackgerritFabio Giannetti proposed openstack/requirements: Add python-monascaclient to the global requirements
*** akshai has joined #openstack-infra13:11
*** hrubi has joined #openstack-infra13:13
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize entries for neutron subnet
*** sputnik13_ has quit IRC13:14
*** dslevin has quit IRC13:15
*** jsavak has joined #openstack-infra13:18
*** pradk has joined #openstack-infra13:19
AJaegerrobcresswell: gate-django-openstack-auth-tox-py27dj18 is wrong, your project uses _, so use gate-django_openstack_auth-tox-py27djXXX13:19
*** trown|outttypeww is now known as trown13:22
robcresswellAJaeger: ah, dang. Fixing now.13:22
*** kgiusti has joined #openstack-infra13:22
*** pradk has quit IRC13:23
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting dj1.9 test to django-openstack-auth
*** vivekd has joined #openstack-infra13:23
*** baoli has joined #openstack-infra13:25
*** dims has quit IRC13:25
*** dims has joined #openstack-infra13:26
*** dslev has joined #openstack-infra13:27
odyssey4medo the check and gate pipelines use different nodes from nodepool? we're seeing failures in the gate queue which aren't being exposed in the check queue13:28
*** vgridnev has joined #openstack-infra13:29
*** Daisy has joined #openstack-infra13:30
*** markvoelker has joined #openstack-infra13:31
*** ParsectiX has quit IRC13:31
*** amrith is now known as _amrith_13:35
*** MCoLo has left #openstack-infra13:37
*** baoli_ has joined #openstack-infra13:37
*** edmondsw has joined #openstack-infra13:39
*** berendt has joined #openstack-infra13:39
HeOSHello! I have an one question about gerrit and jeepyb, perhaps. Do you have a script to send time-to-time notifications to reviewers which were added into any request?13:39
*** burgerk has quit IRC13:40
*** baoli has quit IRC13:40
*** Daisy has quit IRC13:41
*** MCoLo has joined #openstack-infra13:41
*** hdd has joined #openstack-infra13:41
*** Daisy has joined #openstack-infra13:42
*** amitgandhinz has joined #openstack-infra13:44
fungiodyssey4me: should all be running on the same available image types and providers. have you attempted to correlate the failure to other factors with
*** Daisy has quit IRC13:47
fungiHeOS: we don't, though also that would get really annoying (i can't even use the default gerrit dashboard because of all the changes people subscribe me to as a reviewer, so the subscription feature is useless for many of us anyway)13:47
*** jistr has quit IRC13:48
*** jistr has joined #openstack-infra13:49
*** unicell has joined #openstack-infra13:49
*** unicell1 has quit IRC13:50
yolandahi fungi, can you check ? ansible-lint is broken and tests are failing13:50
fungimtreinish: the database pruning completed sometime while i was asleep, but rerunning this morning took ~15 minutes (for an idea of runtime when it's more or less keeping up)13:51
fungiyolanda: did we merge something which broke the test?13:51
yolandafungi, no, but ansible 2.0 landed13:51
yolandaand ansible-lint not compatible13:51
fungigot it. the 2.0 release was recent, i remember that now13:52
*** fhubik is now known as fhubik_brb13:52
*** vivekd has quit IRC13:53
fungii guess not upgrading until we can run our style checks on it is reasonable, so that we don't introduce stylistic issues we can't test. it's not possible to use 1.9.4 to run ansible-lint on roles which are used by ansible 2.0?13:53
*** akshai has quit IRC13:54
fungidoes it also affect our ability to syntax check roles?13:54
*** MCoLo has quit IRC13:55
yolandafungi, i limited the change to lint only13:55
*** dslev has quit IRC13:55
*** hdd has quit IRC13:55
*** mtanino has joined #openstack-infra13:55
yolandai tested ansible-lint with ansible 1.9.4 in a role that is used for 2.0, but i'm not sure on the new settings that are brought by ansible 2.0, lint may fail13:55
fungiyep, that works for me (the commit message was just vague and i hadn't pulled up the diff yet)13:55
*** ddieterly has joined #openstack-infra13:56
*** eharney has joined #openstack-infra13:56
*** jordanP has joined #openstack-infra13:57
*** Daisy has joined #openstack-infra13:57
*** fhubik_brb is now known as fhubik13:58
*** tlian has joined #openstack-infra14:00
odyssey4mefungi yolanda FYI
*** akshai has joined #openstack-infra14:00
odyssey4mefungi I'm busy doing the correlation, but the logstash processing is a little behind - the issue is something new I'm seeing this morning14:01
odyssey4mefungi so far hpcloud and rax-iad14:01
fungiodyssey4me: so also probably something related to 2.0?14:02
*** yamamoto_ has quit IRC14:02
odyssey4mefungi nope, we used to see this often with ansible 1.6 and below... but haven't seen it since... our tests have ansible capped at < 2.014:02
odyssey4mefungi ah, interesting - the last recheck passed on ovh14:03
*** bcourt has joined #openstack-infra14:03
odyssey4mefungi I have a patch going in to work around the issue, I was just wondering whether there was any known underlying hardware differences that could account for the issue14:04
*** links has joined #openstack-infra14:04
*** ddieterly has quit IRC14:05
*** dkranz has joined #openstack-infra14:05
openstackgerritMerged openstack-infra/project-config: Temporarily pin to ansible==1.9.4
*** zz_dimtruck is now known as dimtruck14:07
mordredfungi: yay! synchronize seems to be fixed for ipv6 in ansible 214:08
*** links has quit IRC14:09
*** rlandy has joined #openstack-infra14:09
mordredoh - for project-config we're pinning to 1.9.4 until ansible-lint is fixed14:09
mordredansible 2.0 on puppetmaster is working nicely14:09
yolandamordred, it just landed, i saw failures on ansible-lint this morning14:09
mordredansible-lint definitely no workie with ansible 2.014:09
*** links has joined #openstack-infra14:09
*** dslev has joined #openstack-infra14:10
pabelangeryolanda: mordred: still digging into it. Made a little progress yesterday on ansible-lint with 2.014:10
yolandapabelanger, do you know status of ansible-lint fixes? is there any estimate time to support ansible 2.0?14:11
*** annegentle has joined #openstack-infra14:14
*** ildikov has quit IRC14:15
*** ccrouch has joined #openstack-infra14:16
pabelangeryolanda: still a bit away, we've pinned to 1.9 for now14:17
yolandaok i pinned ansible-lint here to 1.9.4 , that shall work14:18
*** salv-orlando has quit IRC14:18
*** salv-orlando has joined #openstack-infra14:18
*** sshnaidm has quit IRC14:19
*** jpeeler has joined #openstack-infra14:19
*** jpeeler has quit IRC14:19
*** jpeeler has joined #openstack-infra14:19
*** GheRivero_ is now known as GheRivero14:20
*** baoli_ has quit IRC14:22
*** baoli has joined #openstack-infra14:23
*** dprince has joined #openstack-infra14:24
*** ayoung has joined #openstack-infra14:25
*** links has quit IRC14:25
*** Daisy has quit IRC14:25
*** links has joined #openstack-infra14:25
*** Daisy has joined #openstack-infra14:26
*** mriedem has joined #openstack-infra14:26
*** peristeri has joined #openstack-infra14:27
*** dimtruck is now known as zz_dimtruck14:27
pabelangeryolanda: should have looked there sooner.  That's bascially the code I've been working on. So, will attempt to work that PR and see what maintainer wants to do14:27
pabelangerodyssey4me: ^14:27
*** baoli_ has joined #openstack-infra14:27
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize vlan name on veth
*** dslev has quit IRC14:29
*** ari has joined #openstack-infra14:29
*** erlon has joined #openstack-infra14:29
*** links has quit IRC14:29
*** baoli has quit IRC14:30
*** links has joined #openstack-infra14:30
*** Daisy has quit IRC14:30
*** otter768 has joined #openstack-infra14:31
*** esker has quit IRC14:32
*** fhubik is now known as fhubik_brb14:33
*** lucasagomes is now known as lucas-hungry14:35
*** otter768 has quit IRC14:36
*** fhubik_brb is now known as fhubik14:36
odyssey4mepabelanger sorry, I must have missed you in telling everyone about the issue and associated PR :(14:37
*** piet has joined #openstack-infra14:37
openstackgerritMonty Taylor proposed openstack-dev/pbr: Split changelog on nulls instead of (
*** dslevin has joined #openstack-infra14:38
pabelangerodyssey4me: np14:38
arizaro# Thanks for the review is there someone else that I can get to give me a +2?14:39
*** ddieterly has joined #openstack-infra14:41
*** cdent has quit IRC14:42
*** bcourt has quit IRC14:44
*** peristeri has quit IRC14:44
pabelangerfungi: I know it is likely early, but has anybody here expressed interested in submitting a talk for the upstream developers track in Austin? I seen some -infra specific things already listed on and won't mind offering to aid in presenting14:44
*** Swami has joined #openstack-infra14:44
*** ddieterly has quit IRC14:46
fungipabelanger: funny you should ask! i was going to put out a specific call for volunteers at the next infra meeting, but you get first pick14:47
*** yamamoto has joined #openstack-infra14:48
fungiearly bird eating worms and all that14:48
fungitasty, tasty conference presentation worms14:48
pabelangerfungi: well, I was mostly thinking it would also be a good way to get into the conference talks for OpenStack14:49
fungiyou betcha14:49
pabelangerstepping stone if you will14:49
mriedemwtf, nova-net logs aren't even in this failed large ops job
*** pradk has joined #openstack-infra14:49
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: Add job for functional tests on py34 env to novaclient
pabelangerfungi: okay, add my name. I would be interested in helping out with that.14:50
*** regXboi has joined #openstack-infra14:50
*** levi_b-away has joined #openstack-infra14:50
*** levi_b-away is now known as levi_b14:50
*** levi_b has left #openstack-infra14:50
*** Qiming has quit IRC14:51
fungipabelanger: gladly, and thanks for volunteering!14:52
*** rossella_s has quit IRC14:52
*** hdd has joined #openstack-infra14:52
*** rossella_s has joined #openstack-infra14:52
*** peristeri has joined #openstack-infra14:52
*** davidlenwell has quit IRC14:57
*** yamamoto has quit IRC14:57
*** mase_x200 has quit IRC14:58
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Speed up story GET requests
mmedvedeflip214: ack14:58
*** bcourt has joined #openstack-infra14:58
*** davidlenwell has joined #openstack-infra14:59
*** ddieterly has joined #openstack-infra14:59
*** piet has quit IRC14:59
*** GheRiver1 has joined #openstack-infra14:59
*** piet has joined #openstack-infra15:00
flip214mmedvede: thanks for replying!15:00
flip214I submitted a patch for ci-watch about 4 weeks ago, and it got merged. But it's not deployed to yet. Is there anything else needed from my side?15:01
*** sridhar_ram has quit IRC15:01
openstackgerritMerged openstack-infra/project-config: Use scripts from horizon repo in integration job test hooks
mmedvedeflip214: I think apoorva did deploy your fix yesterday, he is the one maintaining ci-watch.tintri.com15:04
*** Qiming has joined #openstack-infra15:06
*** links has quit IRC15:07
anteayadhellmann: when did you want to discuss release script location?15:08
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for devstack setup bug 1457713
openstackbug 1457713 in devstack "ERROR: openstack Missing parameter(s)" [Undecided,Confirmed]
*** _amrith_ is now known as amrith15:10
openstackgerritBenjamin Koep proposed openstack-infra/jenkins-job-builder: * Added support for the Jenkins Slack plugin
*** burgerk has joined #openstack-infra15:10
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: [rally] Skip long-running jobs when it possible
*** sshnaidm has joined #openstack-infra15:11
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: [rally] Skip long-running jobs when it possible
*** HeOS has quit IRC15:11
openstackgerritBenjamin Koep proposed openstack-infra/jenkins-job-builder: * Added support for the Jenkins Slack plugin
*** fhubik is now known as fhubik_brb15:12
flip214mmedvede: thanks, right. the first day I didn't check the JS code, and it's done ;)15:12
flip214thank you very much!15:12
*** yamamoto has joined #openstack-infra15:13
flip214if I find any other simple optimizations, is there anything I can do to accelerate deployment?15:13
mmedvedeflip214: yw, sorry for the delay. fyi, some of third-party folks also hang out in #openstack-third-party-ci channel15:14
*** zz_dimtruck is now known as dimtruck15:15
*** fhubik_brb is now known as fhubik15:15
flip214mmedvede: thanks again!15:16
mmedvedeflip214: the delay was mainly due to most people having vacations. ci-watch.tintri is not managed by openstack infra team, and is not automatically deployed. Once we have a proper deployment, it should be faster15:16
pabelangerianw: greghaynes: diskimage-builder installed from RPM (epel7) :)  Version 1.4.0, but a start15:16
*** fhubik is now known as fhubik_brb15:17
*** levi_b has joined #openstack-infra15:17
*** levi_b has left #openstack-infra15:17
flip214mmedvede: ah, okay. I would have thought the people giving +2 (at least +1 on the workflow) would've done the deployment too...15:18
*** rbrndt_ has joined #openstack-infra15:18
mmedvedeflip214: if you're interested, I can give you the script to deploy your own ci-watch instance15:19
flip214mmedvede: thanks, but no thanks ;)15:19
mmedvedehaha, ok15:20
*** Qiming has quit IRC15:20
*** boris-42 has joined #openstack-infra15:20
*** fhubik_brb is now known as fhubik15:20
*** rbrndt__ has joined #openstack-infra15:21
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Speed up story GET requests
*** tdurakov has joined #openstack-infra15:22
tdurakovhi folks15:22
*** rbrndt_ has quit IRC15:23
tdurakovfast question about ansible, have you changed its version recently?15:23
odyssey4metdurakov ansible 2.0 released yesterday afternoon15:24
*** yamamoto has quit IRC15:25
tdurakovodyssey4me, got strange errors, line: $ANSIBLE subnodes --sudo -f 5 -i "$WORKSPACE/inventory" -m shell -a "mount -t nfs4 -o proto=tcp,port=2049 $primary_node:/ /opt/stack/data/nova/instances/" fails with 2016-01-13 15:04:00.719 | unsupported parameter for module: proto15:26
*** zeih has quit IRC15:27
*** rkukura has joined #openstack-infra15:27
*** armax has quit IRC15:28
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fix on refresh token worflow:
*** MarkAtwood has joined #openstack-infra15:28
pabelangerfungi: clarkb jeblair: nodepool question, do you see any downside to moving reconfigureGearmanClient() to the bottom of the updateConfig()? The reason being, if zuul is down (or not provisioned) gearman blocks, stopping DIBs from being created.15:28
pabelangergoing to test it locally now, but figured I ping you all first15:28
pabelangerHmm, now that I typed that out, I'm unsure if dibs will even build15:29
pabelangerokay, testing15:29
arizxiiro# Can you give me a review when you have time? -
cbader_is anyone else see issue with gerrit login. my zuul is getting  ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit.15:30
pabelangercbader_: Yup, a result of the gerrit upgrade a few weeks ago15:30
openstackgerritAndrea Frittoli proposed openstack-infra/puppet-log_processor: Allow for python-daemon already defined
*** yamahata has joined #openstack-infra15:30
cbader_pabelanger, I have been out the last few weeks, is there a fix I need to put in.15:31
igorbelikovhey folks, can someone review this request to add packetary project I know packetary guys are eager to start using gerrit and this review is hanging without infra comments for a long time15:31
clarkbpabelanger: I dont think thats true before the gearman image builder stuff merges since the devstack tests have no zuul/gearman. Will be rrue after though15:31
*** zeih has joined #openstack-infra15:31
pabelangerclarkb: okay, maybe I am just seeing a long timeout for the socket connection.  Either way, testing more locally15:32
clarkbwell you tell it not to connect to anything15:32
andreafyolanda, clarkb: tiny puppet review on log processor when you have a sec:
clarkbbut this all changes today hopefully when I get the image builder stuff in which will need a gearman15:33
pabelangercbader_: IIRC, it is more a warning then error. nodepool will just log it and work as expected15:33
tdurakovclarkb, hi, do you know what ansible version is now used by ci?15:33
cbader_clarkb: looks like i have run into a change in gerrit a few weeks ago, I am getting 'ref-replicated' from Gerrit is there a fix you can point me too.15:33
AJaegertdurakov: unless there are constraints: ansible 2.015:34
*** tonytan4ever has joined #openstack-infra15:34
pabelangerclarkb: Ya, in my setup. I want it to connect to gearman, its just that zuul hasn't been provisioned by ansible yet.  So, rather then block DIBs for another few minutes, I was looking to reorder the gearmanClient reconfigure15:34
AJaegertdurakov: ansible-lint uses 1.9.4 - yolanda added a constrainted there15:34
pabelangerclarkb: not a big deal, just have to wait a few minutes15:34
clarkbtdurakov: latest15:34
clarkbcbader_: thats benign, zuul will just ignore those events15:35
pabelangerclarkb: cbader_: maybe we should reduce the log level from ERROR to WARNING15:35
*** amitgandhinz has quit IRC15:35
*** FallenPegasus has joined #openstack-infra15:35
clarkbpabelanger: there is no real point to the reorder after the gearman image builder stuff merges because image building requires gearman at that point15:35
tdurakovAJaeger, clarkb, ok thank you!15:35
*** lucas-hungry is now known as lucasagomes15:36
pabelangerclarkb: Ya, that's what I wanted to confirm15:36
*** amitgandhinz has joined #openstack-infra15:36
openstackgerritMerged openstack-infra/elastic-recheck: Add query for devstack setup bug 1457713
openstackbug 1457713 in devstack "ERROR: openstack Missing parameter(s)" [Undecided,Confirmed]
openstackgerritChristian Berendt proposed openstack-infra/irc-meetings: Adjust meeting time for US/EU doc-install-team-meeting
clarkbpabelanger: cbader_ are those unknown event things logged as errors?15:38
*** fhubik is now known as fhubik_brb15:39
*** fhubik_brb is now known as fhubik15:39
pabelangerclarkb: ya15:40
*** Sukhdev has joined #openstack-infra15:40
asselinThelo, this is your issue: slave nodes can't access your CI node: Could not resolve host:
*** fawadkhaliq has joined #openstack-infra15:40
cbader_clarkb, pabelanger, yes as error. 2016-01-13 07:40:27,598 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit.                    Can not get account information.15:41
asselinThelo, so it will default to If you look at the script that checks out, there's a checkout order.15:41
pabelangercbader_: thanks, save me from digging up my error logs15:41
*** nmagnezi has quit IRC15:41
asselincbader_, I saw that issue yesterday in my test ci. When upgrading to the latest zuul they went away.15:41
pabelangerclarkb: cbader_ I can get a patch up to reduce it to a warning15:42
asselincbader_, or maybe that's coincidence....15:42
cbader_pabelanger, I made a change in my nodepool.nodepool.yaml and did an ./ which I guess updated my zuul and nodepool and now can't get nodepool vm's to start15:43
pabelangercbader_: if you run nodepoold -d manually, what do you see?15:44
*** yamamoto has joined #openstack-infra15:44
*** gongysh_ has joined #openstack-infra15:45
cbader_pabelanger: I get IOError: [Errno 13] Permission denied: '/etc/nodepool/secure.conf'15:46
cbader_pabelanger: and locks up15:46
waynrelectrofelix: so here's a shorter compare xml time:
*** berendt has quit IRC15:46
pabelangercbader_: so, need to fix your permission error15:46
cbader_pabelanger: don't know what would have changed to cause this error.15:47
Theloasselin: thanks I found the issue earlier this evening, however I dont quite get why we get the code from the CI server and the role of zuul-merger15:47
*** armax has joined #openstack-infra15:48
pabelangercbader_: not sure how you install nodepool. Are you using the same user as the service would?15:48
pabelangercbader_: either way, looks to be a simple fix15:48
tdurakovclarkb, it looks like there is regression with new ansible15:48
openstackgerritMerged openstack/requirements: Change OSprofiler constraints to >=0.4.0
asselinThelo, so the ci server, in particular the zuul merger, is responsible to prepare any git references needed by the project.15:50
asselinThelo, so if you make a change to the project 'cinder', it is the zuul-merger that will have that change prepared locally so that your ci jenkin slave can test the right patch.15:51
waynri've seen compare xml time on master branch take as long as 9m15:51
cbader_does anyone know where the CI Watch get the url for the logs it attaches too? I had to change my log file server and the CI Watch is not connecting to the correct URL?15:51
*** MarkAtwood has quit IRC15:51
asselinThelo, furthermore, if your cinder patch has a 'depends-on' to a nova patch, the zuul merger will prepare 2 refs, one for cinder and one for nova.15:51
asselinThelo, in this way, you ci- jenkins slave can get both patches and run the correct test.15:51
*** jpr has joined #openstack-infra15:52
asselincbader_, it gets if from the gerrit comment15:52
openstackgerritPaul Belanger proposed openstack-infra/zuul: Log 'Received unrecognized event type' as warning
*** jpr has left #openstack-infra15:53
Theloasselin: Got it, thanks15:53
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Add neutron -plus job template for advanced features
*** mgould has joined #openstack-infra15:53
*** jpr has joined #openstack-infra15:53
cbader_asselin: sorry don't know where the gerrit comment is configured.15:54
*** matrohon has quit IRC15:54
asselincbader_, I believe it is from zuul:
asselincbader_, /etc/zuul/zuul.conf 'url_pattern'15:56
*** bcourt has quit IRC15:56
*** sridhar_ram has joined #openstack-infra15:56
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Add neutron -plus job template for advanced features
asselincbader_, if that value is correct, you probably need to restart zuul15:58
*** bardia has joined #openstack-infra15:58
cbader_asselin: the ip in the /etc/zuul/zuul.conf is correct.15:58
*** yamamoto has quit IRC15:59
*** devananda has quit IRC15:59
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Improve "add card" UI
Zarawhoa I can see the speedup16:00
Zaraeven on my tiny instance16:00
*** kmartin has joined #openstack-infra16:00
*** sputnik13 has joined #openstack-infra16:01
anteayaZara: nice16:01
Zarahah, oops, I meant to type that in #storyboard but it still applies! xD16:02
*** devananda has joined #openstack-infra16:02
clarkbtdurakov: do you have more details than that?16:02
tdurakovi used adhoc shell module16:03
tdurakovwhen there is '=' in arguments it's treated as key-value16:03
*** jsavak has quit IRC16:03
*** Jeffrey4l has joined #openstack-infra16:04
tdurakovclarkb, $ANSIBLE subnodes --sudo -f 5 -i "$WORKSPACE/inventory" -m shell -a "mount -t nfs4 -o proto\=tcp,port\=2049 $primary_node:/ /opt/stack/data/nova/instances/"16:04
tdurakovneed to be escaped16:04
*** jsavak has joined #openstack-infra16:04
clarkbtdurakov: it is possible that was intentional, they did a 2.0 release because not everything is backard compatoble16:05
clarkbbut you best bet is to ask asnible16:05
cbader_asselin: I am getting the can not get account information still after restarting zuul and zuul-merger is there a fix for that?16:05
tdurakovclarkb, will try to ask16:05
tdurakovit affects my job that I'm working on in experimental pipeline, so not urgent at this moment, but it could be in other jobs/hooks^(16:06
asselincbader_, not sure where you see that....16:07
dhellmannanteaya, fungi, jeblair: do you have some time to talk about those release scripts quickly? I think ttx have come up with a plan that reduces my concern about moving some of them into project-config16:07
dhellmann* ttx and i16:07
*** austin81 has joined #openstack-infra16:07
anteayadhellmann: /me is here and standing by16:07
cbader_asselin: in the /var/log/zuul/zuul.log16:07
*** sigmavirus24_awa is now known as sigmavirus2416:07
jeblairdhellmann: o/16:08
cbader_asselin: getting this, 2016-01-13 08:08:03,648 ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replication-done' from Gerrit.                    Can not get account information.16:08
dhellmannanteaya , jeblair : let me run this past you, and I can catch up with fungi later if I need to16:08
asselincbader_,  not sure if that's impacting any real functionality16:08
anteayadhellmann: /me listens16:09
openstackgerritMerged openstack-infra/openstackid: Fix on refresh token worflow:
asselincbader_, seems more of a warning if anything16:09
dhellmannso my issues with moving all of the release tools into project-config are that (a) some are complex and change frequently and (b) converting them to work to use system packages for things that aren't packaged in ubuntu yet would be a hassle16:09
dhellmannwe think, now, that with some changes to just the script that submits the tags, we can move that one into project-config16:10
*** salv-orl_ has joined #openstack-infra16:10
dhellmannall of the other stuff related to generating release announcements contains the most complexity, and that we should be able to run on a non-privileged node so we shouldn't need to put it into project config or worry about using a virtualenv16:10
fungidhellmann: i'm around now, sorry16:11
fungicatching up16:11
dhellmannwhich leaves the script that adds comments related to released bugs, which falls somewhere in between: it's complex and needs launchpad tools and credentials16:11
dhellmannit's likely all of its dependencies are available as debs, though16:11
dhellmannfungi : np16:11
dhellmannso the idea is to do the refactoring of the tools that generate the tags and move those to project-config16:12
dhellmannand then see about the release announcement stuff as a separate unprivileged job16:12
dhellmannthough, do the unprivileged nodes have exim?16:12
*** bcourt has joined #openstack-infra16:12
dhellmannor some other way to send email, I suppose, it doesn't have to be exim16:13
*** ldnunes has joined #openstack-infra16:13
jeblairdhellmann: yes they do16:13
*** salv-orlando has quit IRC16:13
dhellmannok, cool16:13
fungiyeah, having them send e-mail to the listserv is probably fine16:13
dhellmannyeah, we might have some of the oslo releases announced to python-announce, too, but we don't do that right now16:13
fungiit's mostly things with access to teh signing key and lp account credentials we need to audit more closely16:14
jeblairdhellmann: the trick is that if you submit a message locally, the node needs to live long enough to deliver it16:14
dhellmannjeblair : I could have the script watch the exim log, I guess? feels dirty, though.16:14
fungiit might make more sense for the job to smtp directly to lists.o.o and block the job completion until it gets accepted?16:14
dhellmannyeah, that works, too16:15
dhellmannok, I'll make a note and make that change16:15
jeblairdhellmann: i would not expect delivery to anywhere but lists.o.o to work for now (and even then, we may have to tweak something); delivery to other sites will likely require a smarthost, and, heh, probably running on a trusted slave with smarthost credentials.16:15
*** katyafervent is now known as katyafervent_awa16:15
*** katyafervent_awa is now known as katyafervent16:15
fungisince it's not delivering to multiple destinations, a local mta would mostly be overkill for this16:15
dhellmannok, we can worry about other announcements later, then16:16
jeblairi think thats best16:16
*** rhallisey has quit IRC16:16
*** trown is now known as trown|brb16:17
dhellmannso that leaves figuring out what to do with the launchpad script. it runs out of the release tagging script right now, and we could keep doing that if I can fix it up to not need a virtualenv16:17
dhellmannit would be good to do that only after the tarball is built, though, and that either means a separate job or having the tagging job wait for the packaging job16:17
jeblairdhellmann: i think i did overstate the delay problem with slave_scripts yesterday... while it's true that if they are used on throwaway nodes it can take days for an image update to complete and for them to show up there -- when they are used on a static slave, that's usually not a problem; puppet will keep them updated so it should be <30 mins after merge16:18
*** trown|brb is now known as trown16:18
dhellmannok, that's good to know16:18
*** dims_ has joined #openstack-infra16:18
anteayaI also didn't know that, thanks jeblair16:18
*** yamamoto has joined #openstack-infra16:18
dhellmannwe pretty regularly discover some odd behavior because all of these repos are slightly different, and have to make adjustments for it16:19
fungiyeah, i mentioned that yesterday, but it eventually becomes a general problem in zuulv316:19
dhellmannthat's *mostly* in the release notes, though16:19
*** rhallisey has joined #openstack-infra16:19
jeblairfungi: yep16:19
anteayafungi: sorry I missed that16:19
mtreinishfungi: ok, cool yeah I noticed it was finished pruning at somepoint before I went to sleep last night16:19
dhellmannand if we can send release nodes from a job where the code is pulled from the release-tools repo as it's needed, then we have no delay16:19
*** rajinir has joined #openstack-infra16:19
mtreinishI'll drop my WIP from the patch adding the cron job to run it then16:19
mgouldhi everyone! are there any requirements cores about who could take a look at ?16:19
mgouldone-line patch to bump minor python-ironicclient version16:20
*** amitgandhinz has quit IRC16:20
dhellmannfungi, jeblair: the release notes script needs a repo and version as input, and then it scans the git tree for the history (among other things). how ugly is it to have a job triggered when a patch lands in the releases repository wait for the tarball to be built before generating the release notes?16:21
*** dims has quit IRC16:21
*** flepied has quit IRC16:21
dhellmannI think we already have a script for waiting for a tarball, so I'm really asking if there's an objection to doing that in CI vs. locally like we've been doing it16:21
*** flepied has joined #openstack-infra16:21
fungidhellmann: we could make it depend on the tarball job the same way the upload job does16:21
sfinucananteaya: fungi: Can we pick up the discussion from yesterday on this patch series?
jeblairdhellmann: a job should run after a patch landes in the release repo but it should wait until a tarball job from say the nova repo completes before running?16:22
dhellmannfungi : how would that job know what version was being built?16:22
fungiright now for things released to pypi we have one job generate the tarball/wheel, then a second job on another worker upload them to pypi if the first job succeeds16:22
anteayasfinucan: perhaps after we finish with dhellmann's thing16:22
*** ihrachys has quit IRC16:22
dhellmannjeblair : right16:22
*** sdake_ has joined #openstack-infra16:22
fungioh, patch to the release repo but tarball for another repo16:22
sfinucananteaya: Oh, of course :)16:22
clarkbjeblair: our first non gate dependent pipeline?16:23
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Add cron job to run weekly prune the data set of db
SergeyLukjanovfungi, jeblair, hi, could you please help. we've found out in sahara team that we created a new repo (sahara-scenario) incorrectly and we need to have repo named sahara-tests with git history (currently created empty repo), do you think it'll be okay to request new repo and move old one to attic eventually? other option is to rename repo and re-push with --force? what do you think guys about it?16:23
fungithat's a little weirder to work out16:23
*** angdraug has joined #openstack-infra16:23
jeblairfungi, dhellmann: neat.  i can't think of any way to do that right now, but cross-repo job dependencies may be a fun thing to explore.  :)16:23
anteayasfinucan: thank you :)16:23
jeblairclarkb: do you think a dependent pipeline could do it...? hrm....16:23
clarkboh except one job has to complete before the other whereas dependent pipeline would parallel16:23
dhellmannjeblair, fungi : ok, I can do it now with a thing that polls until the tarball exists on the server, so if that's ok as an interim solution I could help explore "better" solutions down the road, too16:23
jeblairdhellmann: what causes the tarball job to run?  tagging the repo?16:24
fungiSergeyLukjanov: can we fix the git history on sahara-scenario now and rename sahara-scenario to sahara-tests during a maintenance (we already have another rename requested we need to schedule anyway)16:24
dhellmannjeblair, fungi: my not quite secret goal is to have all releases going through this machinery by early in the next dev cycle, so I'd like to identify short-term and long-term solutions that support that16:24
dhellmannjeblair : yeah16:24
*** tonytan4ever has quit IRC16:24
jeblairdhellmann: and that's done by the previously mentioned tagging script ,right?16:24
dhellmannjeblair : right16:24
jeblairdhellmann: so -- commit to releases; tag pushed; tarball built; release notes job run  ?16:25
SergeyLukjanovfungi, I think yes - actually we have an empty repo now and need to init it from the external repo16:25
dhellmannjeblair : right16:25
SergeyLukjanovfungi, do you think it'll be okay?16:25
*** sdake has quit IRC16:25
*** sdake_ is now known as sdake16:25
jeblairdhellmann, fungi: can the release notes job be attached to the _project_ rather than release repo?16:26
fungidhellmann: ahh, so change merges to release repo, release repo job generates and pushes a tag for nova into gerrit, tag on nova triggers nova tarball job... and we want another release repo job to run after that completes16:26
clarkbjeblair: if we made a dependent pipeline with a window of 1 that never shifted that may work16:26
clarkbjeblair: major hacks though16:26
dhellmannjeblair : maybe? it needs the repo and a tag as input, so if that's available there may be a way?16:27
dhellmannI suppose we could use git describe to determine the tag16:27
fungiSergeyLukjanov: yeah, so push --force the correct content into the empty repo for now, then add it to the renames list on the meeting agenda to be scheduled. we can pick a date for the next renames batch when we meet on this coming tuesday16:27
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize vlan name on veth
dhellmannas long as the local copy of the working repo is checked out to the tagged commit16:27
jeblairdhellmann: it would have the tag (and commit that was tagged) in nova as well as the project, 'nova', as input... so it could go from there i think16:27
dhellmannok, I think we can make that part work, then16:28
jeblairdhellmann: no need for git describe, it will all be in ZUUL_ vars16:28
SergeyLukjanovfungi, cool, thx, I'll add it to the agenda16:28
fungiyeah, basically it would be a nova job which depends on the nova tarball job and fires the necessary machinery (the tag name is in the zuul parameters already)16:28
dhellmannjeblair : good, that seems more accurate16:28
jeblairdhellmann, fungi, clarkb: that means the release notes repo can just wait on the nova tarball release job then16:28
*** ildikov has joined #openstack-infra16:28
jeblairdhellmann, fungi, clarkb: that means the release notes job can just wait on the nova tarball release job then16:29
fungiyeah, parse error there16:29
jeblairi have not had breakfast yet16:29
funginp, that makes much more sense16:29
dhellmannyeah, that makes more sense, ok cool16:29
*** aopincar_ has joined #openstack-infra16:30
*** yamamoto has quit IRC16:30
*** yaume_ has quit IRC16:30
dhellmannfungi, jeblair : ok, the final question was how do we know when these jobs pass/fail? is there some way to see them easily in a dashboard or something?16:30
*** yaume has joined #openstack-infra16:30
*** danpb has joined #openstack-infra16:30
anteayatheir logs would be in post logs wouldn't they?16:31
fungidhellmann: not yet, other than looking at their logs or graphite or catching them in the zuul status, but there are efforts underway to add a better dashboard to things like post/release/periodic job logs16:31
fungiso that they're a little more discoverable16:31
openstackgerritSteve Martinelli proposed openstack-infra/project-config: set keystone's legacy_drivers job to voting
anteayaoh where is that dashboard going?16:31
fungiright now they're just going to be in the usual log path for teh repo+pipeline+jobname16:32
*** otter768 has joined #openstack-infra16:32
*** abregman has quit IRC16:32
dhellmannfungi : ok, when the time comes I'll get those path details from you. The only pattern I know is based on SHA values.16:33
fungianteaya: i think the hope was to get it integrated into the health dashboard, though rbradfor was also working on a proof-of-concept to do something similar for coverage jobs specifically16:33
anteayafungi: okey dokey, thanks16:33
dhellmannfungi, jeblair: I think you've answered my concerns. thanks for your time!16:34
fungidhellmann: thanks for making time to continue the discussion!16:34
anteayadhellmann: thank you16:34
anteayafungi clarkb jeblair danpb would like to be added to a new gerrit group16:35
anteayaif someone could assist16:35
anteayasfinucan: you are up16:35
*** mpavone has quit IRC16:35
sfinucananteaya: yay?16:35
anteaya,members link to os-vif-core16:36
anteayasfinucan: thanks for you patience16:36
anteayasfinucan: lets start with a link to the parent patch of that series from yesterday and see where we are16:36
sfinucananteaya: thanks for helping with this16:36
sfinucananteaya: this is it
anteayasfinucan: I do my best to listen to whatever I can16:37
sfinucanit's broken (see comment), but that's not the issue16:37
*** otter768 has quit IRC16:37
rbradforanteaya, I have a proof of concept for coverage jobs that builds a page to the logs from the zuul status (as the index page did not provide a link to the correct logs)16:37
*** Guest22649 has quit IRC16:38
anteayasfinucan: so the first two patches in the series fail unit tests and docs16:38
*** Guest91320 is now known as Vivek16:38
anteayasfinucan: then the third patch gives you this patch depends on a patch that fails to merge16:39
*** Vivek is now known as Guest1428916:39
fungirbradfor: hopefully that will be fixed once are merged16:39
sfinucananteaya: Yes, and the second patch is new to this "revision", i.e. patch 3 used to be patch 216:39
jeblairdanpb: who are you in gerrit?16:39
anteayasfinucan: now I can't say definitively but I personally would be curious if your fails to merge error goes away if the parent patch could pass unit and doc tests16:39
danpbjeblair: email addr berrange@redhat.com16:40
*** Guest14289 is now known as VivekVC16:40
*** Hal has joined #openstack-infra16:40
jeblairdanpb: done!16:40
anteayajeblair: danpb thank you16:40
*** Hal is now known as Guest7009416:40
rbradforfungi, woot!16:40
*** bhunter71 has joined #openstack-infra16:40
danpbjeblair: anteaya  great, thank you both for your help16:40
anteayadanpb: thanks for your patience16:40
sfinucananteaya: So patchset 6 of 257995 depended on patchset 6 of 25795616:41
anteayasfinucan: okay so I don't have access to the behind the scenes logs for gerrit and zuul16:41
sfinucanand the latter passed Jenkins tests16:41
anteayasfinucan: but I'm going to recommend getting unit and doc tests passing on the parent patch16:41
sfinucanbut the former still gave that failed to merge issue16:41
*** bpokorny has joined #openstack-infra16:42
*** Guest70094 has quit IRC16:42
sfinucananteaya: Sure, I can do that. I don't think it's the issues, however :/16:42
anteayasfinucan: well at the very least it will make the issue clearer if that isn't it16:43
sfinucananteaya: True. OK, I'll do that now and follow up tomorrow :)16:43
fungianybody remember what happened to this change? the api claims it's in a merged state but it looks incomplete (no successfully merged comment, no state displayed in the webui)16:43
anteayasfinucan: thank you, and thanks for asking when we had time to spend with you, makes me so much calmer :) I'm grateful16:44
fungii need to fix my middle-click to stop bouncing16:44
anteayafungi: I am not familiar with that change at all16:44
jeblairfungi: no idea16:44
*** cnesa has quit IRC16:44
*** yamamoto has joined #openstack-infra16:45
jeblairfungi: when was the rolled back gerrit upgrade?16:45
fungijeblair: may 11 i thought16:45
anteayamay I think16:45
fungiso this would have been a couple weeks prior16:45
fungii don't see any mention of it in logs of this channel (until just now)16:46
anteayayeah may 9th was the upgrade16:46
*** samuelBartel has quit IRC16:47
anteayathere is the rollback email:
openstackgerritGuido Günther proposed openstack/python-jenkins: Add support for handling promotions
jeblairfungi, anteaya: considering there is a verified+2 on apr26, it _probably_ merged on that date16:48
fungianyway, it's past my caring about threshold since it's just turning up in my tests to validate the new atc listing script against the september elections. it's not in scope for passes to the next summit and won't be in scope for the next election either16:48
anteayajeblair: I support that logic16:48
fungialso it's from someone who was a later contributor to the same repo, so doesn't skew my results16:48
jeblairfungi, anteaya: if we care to examine further, we may be able to pull data from backups.16:48
fungijust an anomaly i was running down from my warning output16:49
anteayaI'm satisfied16:49
*** arxcruz has quit IRC16:49
jeblairbtw, our backup partition is full so we're probably not currently backing anything up.16:49
electrofelixwaynr: so looks like it's just some variation due to build slave :)16:49
jeblairno one ever did step up to work on that :(16:49
anteayaif this was a one time patch I'd want to follow up for elections at least, but jd is a frequent contributor16:49
*** bradjones has quit IRC16:49
anteayajeblair: oh :(16:50
anteayawhere is the backup partition on cacti, I must have missed that16:50
*** bradjones has joined #openstack-infra16:50
*** bradjones has quit IRC16:50
*** bradjones has joined #openstack-infra16:50
*** vgridnev has quit IRC16:50
fungijeblair: what did we need there? just more space? pruning retention (is that possible with bup)? something else?16:50
jeblairi'm sorry if this rubs anyone the wrong way, but we really need people to care about boring important stuff.16:51
anteayait doesn't rub me the wrong way16:51
anteayaand i agree with you16:51
*** fhubik has quit IRC16:51
*** rcernin has quit IRC16:51
anteayaand I'm glad you stated it as such16:51
jeblairfungi: i'm not sure if we can prune retention :)16:52
*** tiswanso has joined #openstack-infra16:52
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Show Project Groups on search page
fungiright, bup storing it as git history16:52
fungiwe do have people who care about boring important stuff, but i agree we have more boring important stuff than we have people caring about it16:52
pleia2good morning16:53
jeblairfungi: yes, and, when faced point blank with the question of "will anyone take the time to validate our backup regime?" the answer has been silence.16:53
* jd__ is glad he will still have his ATC pass ;)16:53
*** amuller is now known as amuller_afk16:54
*** danpb has left #openstack-infra16:54
*** dtantsur is now known as dtantsur|afk16:54
anteayapleia2: good morning16:54
*** mtanino has quit IRC16:54
anteayajd__: yes thank you for being today's anomoly in fungi's script16:54
*** harlowja_at_home has joined #openstack-infra16:54
fungijeblair: we did have a volunteer, but the volunteer sort of disappeared after the validation plan was half written and i didn't follow up on it16:55
jeblairfungi: i will see about adding more space16:55
jeblairfungi: i will also attempt some restores to see if it's broken or not16:55
jeblairfungi: confirmed -- prune is not a feature of bup16:56
*** vgridnev has joined #openstack-infra16:56
*** vgridnev has quit IRC16:56
fungijeblair: thanks, just managed to get logged in myself (my ssh key on there is still old16:57
*** ifarkas has quit IRC16:57
fungiin good news, /opt/backups is already a logical volume16:59
*** MarkAtwood has joined #openstack-infra16:59
jeblairyeah, i'm cinder expanding it now17:00
rbradforI'm seeing some infra help in a merged patch #264977 that updated docs and includes code examples ( I'm getting a forbidden page e.g.
fungiso as long as we don't have corruption, we should hopefully just be able to add a second cinder volume behing it17:00
*** HeOS has joined #openstack-infra17:00
fungier, behind17:00
anteayayay good news17:00
cbader_I am trying to determine if my CI is functional and I tried to kickoff a previous check with the zuul command but I am not seeing the patch start or any errors, anywhere else I can look?17:00
*** jlanoux has quit IRC17:00
fungijeblair: thanks for shifting gears to work on that17:00
anteayarbradfor: I confirm I am also seeing forbidden at that link17:01
cbader_I ran the command: sudo zuul enqueue --trigger gerrit --pipeline check --project openstack/cinder --change 221441,14 it worked before has something changed so this doesn't work anymore.17:01
*** savihou has quit IRC17:02
rbradforanteaya, what's interesting, is the docs embed files, and that is visible (i.e the first code example), so the file I assume is actually there.17:02
*** salv-orl_ has quit IRC17:03
cbader_If I Cntl C the line I see this File "/usr/local/lib/python2.7/dist-packages/zuul/", line 57, in enqueue17:03
cbader_    return not self.submitJob('zuul:enqueue', data).failure17:03
cbader_  File "/usr/local/lib/python2.7/dist-packages/zuul/", line 45, in submitJob17:03
cbader_    time.sleep(0.1)17:03
rbradforanteaya, or perhaps it's generated and not using the actual example file.17:03
*** salv-orlando has joined #openstack-infra17:03
waynrelectrofelix zxiiro zaro clarkb last night I gave a presentation at a local meetup on the JJB 2.x API and a tool demonstrating its use, if you're interested the presentation and an example Python-defined jenkins pipeline can be found here:
electrofelixwaynr: thanks, taking a look17:04
anteayacbader_: please use a paste17:04
zxiirowaynr: cool thanks for sharing17:04
anteayarbradfor: both good questions17:04
* zxiiro adds it to his reading list17:05
cbader_anteaya: sorry for mistake17:05
waynrI still need up document the JJB 2.x demo tool's expected python module interface, actually there is much work to be done there17:05
anteayacbader_: the channel gets busy with many conversations at once, pastes make things easier, thank you17:06
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fix on user profile pic url:
anteayawaynr: thank you for sharing, I hope you had an enjoyable presentation experience17:06
cbader_anteaya: thank you for help17:06
anteayacbader_: you're welcome17:06
waynranteaya: it was great! lots of good questions, and I even plugged openstack-infra's zuul-based CI system when discussing the history of JJB17:07
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Improve "add card" UI
anteayawaynr: oh thank you17:07
waynr(in case it's not obvious I am a fan the openstack CI system)17:07
anteayawaynr: I'm glad you had an attentive audience17:07
anteayawaynr: always nice to state the obvious :)17:07
mordredwaynr: \o/17:08
*** jistr has quit IRC17:08
anteayajeblair fungi pleia2 clarkb mordred so thoughts on adding ci-backup to cacti?17:08
clarkbgreghaynes: looks like the nodepool stack is all +1 now by jenkins. Are you/we going to have time today to do this is the question17:08
clarkbgreghaynes: I really really want to get it merged but so many things to do17:09
anteayaI'll add it if we have no reason for it to not be in cacti17:09
* mordred also wants that merged17:09
mordredanteaya: I do not know of any reasons - but jeblair or clarkb may have one17:09
anteayamordred: thank you17:09
clarkbhrm, I don't think there is a reason not too, since that is just snmp17:09
* anteaya waits for more input while making soup for lunch17:10
anteayaclarkb: thank you17:10
clarkbbut I know we have been cautious with the backup hosts17:10
anteayamakes sense17:10
*** bardia has quit IRC17:10
greghaynesclarkb: I am free and clear (as much as you are due to housing)17:10
jeblairit's probably worth the very minor risk17:10
*** annegentle has quit IRC17:10
*** sfinucan has quit IRC17:10
*** zeih has quit IRC17:11
*** amotoki has quit IRC17:11
jeblairand actually, afaict, we're already running it and have the port open, so we're taking the 'risk' without getting the benefit :)17:11
zxiirocurious, what size VMs does openstack use for Jenkins instances?17:11
jeblairanteaya: want to add it to the list in system-config?17:11
anteayajeblair: I do, thank you17:11
fungiyeah, i see no real reason not to be monitoring the backup host(s) in cacti17:11
clarkbzxiiro: the jenkins masters are 8vcpu 30GB ram iirc17:11
* anteaya figures out how to offer that patch17:11
clarkbzxiiro: you can see them on http://cacti.openstack.org17:12
zxiiroclarkb: thanks17:12
clarkbgreghaynes: ok, why don't I ping you after I get back from the bank and we can try then17:12
jeblair/dev/mapper/main-backups  2.0T 1002G 1015G  50% /opt/backups17:12
*** apoorvad has joined #openstack-infra17:12
pleia2woo, new 2T backups overloard17:12
pleia2overlord too17:12
clarkbthere is no easy method for rolling old bcakups off with bup right?17:13
jeblairclarkb: s/easy//17:13
jeblairclarkb: well, i mean, they suggest starting a new backup tree17:13
clarkbah ok17:13
clarkbso you sort of manage that atop bup17:14
jeblairclarkb: so i mean, we could do a thing where we rotate trees by date or something17:14
mordredjeblair: AFS?17:14
openstackgerritGuido Günther proposed openstack/python-jenkins: Add support for handling promotions
jeblairmordred: nicely done17:14
* mordred wins today's "suggest AFS as the solution to a problem" game17:15
clarkbthis is going to turn into a drinking game17:15
jeblairmordred is already winning that too17:15
mordredclarkb: it's not already a drinking game?17:15
clarkbapparently I am just behind on the drinking :)17:16
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add non-voting Django 1.9 test to Horizon
*** esikachev has quit IRC17:18
anteayamorning zaro17:18
*** GheRiver1 has quit IRC17:19
clarkbgreghaynes: maybe if you get a chance you can skim through the stack and make sure it looks good to you17:19
*** dizquierdo has quit IRC17:21
*** flepied has quit IRC17:22
*** GheRiver1 has joined #openstack-infra17:23
*** krtaylor has quit IRC17:25
clarkbhogepodge: the state department has really good travel info for all the places as it related to USA passport carriers fwiw17:26
fungijeblair: thanks again17:26
*** krtaylor has joined #openstack-infra17:26
clarkber wrong window17:26
*** mudassirlatif has joined #openstack-infra17:26
clarkbbut hey useful to everyone :)17:26
fungianteaya: lmk when you post the cacti addition change and i'll fasttrack it17:26
openstackgerritMerged openstack-infra/storyboard: Speed up story GET requests
clarkb(as not all the data is MURICA specific)17:26
openstackgerritAnita Kuno proposed openstack-infra/system-config: Add to cacti
anteayafungi: thank you17:26
anteayathere it is, not sure if I have the name of the server correct17:27
anteayawill respin if I guessed wrong17:27
clarkbanteaya: the name is documented on the ci docs17:27
anteayaclarkb: I found that useful thank you17:27
anteayaclarkb: pleia2 found this for me:
*** _nadya_ has quit IRC17:28
*** markus_z has quit IRC17:28
anteayabut according to
*** ayoung has quit IRC17:28
anteayait should be ci-backup-rs-ord.openstack.org17:28
clarkbanteaya: ya that17:28
anteayaso I can change it17:28
* anteaya respins17:29
*** HeOS has quit IRC17:29
pabelangerjeblair: so, thinking of toggling the color of the singlestat panel for the pipelines (check).  Looking now we have 300+, to me that would be a good limit to indicate medium backlog17:29
pabelangermaybe 25017:29
*** xyang1 has quit IRC17:31
openstackgerritAnita Kuno proposed openstack-infra/system-config: Add to cacti
pabelangeranteaya: jeblair: AJaeger: maybe we can merge today too. Demoing template support in grafana17:31
*** aopincar_ has quit IRC17:31
anteayafungi clarkb ^^17:31
jeblairpabelanger: before we do that, maybe we can decide whether the value is correct17:31
*** Swami has quit IRC17:31
pabelangerjeblair: true17:31
pabelangerjeblair: I only see 219 on status.o.o/zuul17:32
jeblairpabelanger: could you look into whether that value is the number of live changes or the number of both live and non-live ones?17:32
*** mtanino has joined #openstack-infra17:32
*** fesp has joined #openstack-infra17:32
pabelangerjeblair: sure17:32
fungithanks anteaya17:33
*** annegentle has joined #openstack-infra17:33
anteayafungi: thank you17:33
* anteaya goes to have lunch17:33
*** xyang1 has joined #openstack-infra17:33
*** yamamoto has quit IRC17:34
*** fesp has quit IRC17:34
pabelangerjeblair: we are using the current value of stats.gauges.zuul.pipeline.check.current_changes17:35
*** kzaitsev_mb has quit IRC17:35
openstackgerritKaitlin Farr proposed openstack/requirements: Updates PyKMIP version
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Show Project Groups on search page
clarkbpabelanger: that is the same value the sparkline on the zuul status page uses17:35
jeblairpabelanger: what i'm asking you to do is to look into that value and see if it represents what we want it to17:36
clarkbbut the (226) on status.o.o/zuul is calcualted from the json iirc17:36
pabelangerclarkb: right, for some reason we have some driff between the grafana and it17:36
clarkbpabelanger: well I don't think there is a diff, since the number next to it is not from graphite17:36
*** bcourt has quit IRC17:36
jeblairpabelanger: i suspect that one or both of these may include non-live changes, and we should think about changing that17:36
pabelangerclarkb: Oh, that might be it17:36
pabelangerjeblair: okay, let me dig into it17:37
jeblairpabelanger: thanks17:37
jeblairanteaya: there was a point where gerrit was not rotating its logs, right?17:37
jeblairoh lunch17:38
*** rlandy has quit IRC17:38
jeblairfungi, clarkb: ^?17:38
pleia2and it was filling up the disk17:38
jeblairpleia2: 2015-03-21-053701 appears to be our last gerrit backup17:38
jeblairi'm extracting it, and so far so good, though it is pulling out a _lot_ of gerrit logs17:39
pleia2ah, yeah17:39
jeblairso i'm guessing/hoping that's before we implemented the rotation17:39
pleia2I think it was over the summer that we did the log thing17:39
* pleia2 nods17:39
jeblairpleia2: cool, that timeframe works then17:39
fungithat timeline sounds right17:39
*** rlandy has joined #openstack-infra17:39
jeblairdid we do something about having lots of those ^ directories?17:40
clarkbI don't recall doing anything for lots of tmp dirs17:41
jeblairi guess not, though it looks like it's not huge...17:41
pleia2nope (just checked on the gerrit server)17:41
clarkbjeblair: is that for melody? I think melody uses TMPDIR17:41
jeblairclarkb: i dunno?  it has gerrit.war unpacked and lots of jars17:41
pleia219M on the server, looks like a monthly rotation17:41
jeblairof gerrit dependencies17:41
clarkboh then no I don't think that is melody17:41
clarkbmelody uses tmp dirs to store the pngs17:41
pleia2actually, some other cadence for the .jar files17:42
jeblairclarkb: i think that's in /tmp/javamelody17:42
jeblairpleia2: maybe each time we upgrade or something?17:42
pleia2jeblair: too many for that, restarts maybe?17:42
jeblairpleia2: oh maybe17:43
jeblairpleia2: i restarted monday i think?17:43
pleia2-rw-r--r--  1 gerrit2 gerrit2 7.4K Dec 22 14:43 plugin_singleusergroup_151222_1443_889258577487934898.jar17:43
pleia2-rw-r--r--  1 gerrit2 gerrit2 7.4K Jan 11 22:05 plugin_singleusergroup_160111_2205_573377469593721012.jar17:43
pleia2those look like restart dates ^^17:43
jeblairpleia2: agreed17:43
jeblairzaro: do you know anything about gerrit unpacking things into /home/gerrit2/review_site/tmp/*/ every time it starts, and then not cleaning them up?17:44
fungii think we have something unpacking the default included plugins?17:46
fungii don't recall the exact details17:46
zarojeblair: not the top of my head, but i think i saw something about this in cthe configurations doc.17:46
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: Add job for functional tests on py34 env to novaclient
clarkbcacti fo rthe backup server is approved17:46
*** vgridnev has joined #openstack-infra17:46
*** Sukhdev has quit IRC17:47
zarofungi: not the same thing.  unpacking the plugins is our way to install plugins and it doesn't go into tmp folder17:47
pleia2thanks anteaya!17:47
*** unicell has quit IRC17:47
openstackgerritIan Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit
*** bcourt has joined #openstack-infra17:50
pabelangerjeblair: I suspect you are right. If I am reading this correct, formatStatusJSON() from the pipeline model is limiting the changes it is returning. While statsd seems to be logging all items in the queue17:50
*** harlowja_at_home has quit IRC17:52
*** tzn has quit IRC17:52
*** _et_ has quit IRC17:53
*** yamamoto has joined #openstack-infra17:53
*** electrofelix has quit IRC17:53
*** tzn has joined #openstack-infra17:54
openstackgerritMerged openstack-infra/openstackid: Fix on user profile pic url:
*** sridhar_ram has quit IRC17:54
anteayapleia2: thanks for your help17:55
openstackgerritIan Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit
*** shardy has quit IRC17:55
*** jordanP has quit IRC17:55
*** kushal has quit IRC17:57
*** amitgandhinz has joined #openstack-infra17:57
nibalizergood morning17:57
anteayamorning nibalizer17:58
nibalizeri am traveling today so wont be around much17:58
*** yamamoto has quit IRC17:58
*** tzn has quit IRC17:58
anteayajeblair: back now, reading backscroll looks like you found the answers you needed, I agree with what was stated re: gerrit logs17:58
anteayanibalizer: safe travels, thanks for telling us17:58
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: add py27 py34 docs check for kolla project
*** sabeen1 has joined #openstack-infra17:59
*** kushal has joined #openstack-infra17:59
*** rkukura has quit IRC18:00
*** mgould has quit IRC18:01
*** derekh has quit IRC18:01
greghaynesclarkb: ohai, sorry, had to be on the phone with bank for waaaay too long18:01
greghaynesclarkb: and sounds like you are doing the same but in person ;) - I checked over the nodepool stack last night and it looked good to me18:02
*** [1]Thelo has joined #openstack-infra18:02
yolandahi greghaynes, thanks for your review on dib lvm. I fixed the issues you noted18:02
greghaynesyolanda: ok, awesome18:02
openstackgerritMerged openstack-infra/storyboard-webclient: Improve "add card" UI
yolandacare taking another look? and also to, that is partitioning elemetn18:03
greghaynesyolanda: Ya, I'll check it out in a bit.18:03
*** yaume has quit IRC18:03
*** e0ne has quit IRC18:04
*** sabeen3 has joined #openstack-infra18:04
zarojeblair, fungi: ok. here's what the tmp folder is for!searchin/repo-discuss/tmp/repo-discuss/Z1STG5t3v6E/8T-i7Ucl_FwJ18:04
*** flepied has joined #openstack-infra18:04
*** Thelo has quit IRC18:04
*** [1]Thelo is now known as Thelo18:04
*** Swami has joined #openstack-infra18:05
zarolooks like cleanup has not been fixed18:05
anteayadims_: do you want to address mriedem's comment on this patch of yours I merged yesterday?
*** treaki_ has quit IRC18:06
anteayadims_: we seem to have overlooked an important leading comma18:06
mriedemanteaya: dims_: i'm fixing it18:06
*** sabeen1 has quit IRC18:06
fungizaro: jeblair: i guess it should be safe to have the initscript wipe them if we care?18:06
anteayamriedem: okey dokey, tell me when you have a patch up and I'll review, sorry about that18:06
dims_mriedem : thanks!18:07
fungizaro: though perhaps this is a regression. it looked like those fixes linked in the last comment merged years ago18:08
*** rajinir has quit IRC18:08
*** ganso has quit IRC18:08
*** boris-42 has quit IRC18:08
*** erlon has quit IRC18:08
*** rguillebert has quit IRC18:08
*** samyewell has quit IRC18:08
*** larainema has quit IRC18:08
*** jed56 has quit IRC18:08
*** yonglihe has quit IRC18:08
*** Ryan_Lane has quit IRC18:08
*** kgalanov has quit IRC18:08
*** igorbelikov has quit IRC18:08
*** dtardivel has quit IRC18:08
*** maximov has quit IRC18:08
*** briancurtin has quit IRC18:08
*** dougwig has quit IRC18:08
*** zhenguo has quit IRC18:08
*** xgerman has quit IRC18:08
*** mwhahaha has quit IRC18:08
*** auggy has quit IRC18:08
*** GheRivero has quit IRC18:08
*** skath_ has quit IRC18:08
*** patrickeast has quit IRC18:08
*** zxiiro has quit IRC18:08
*** sweston has quit IRC18:08
*** othermordred has quit IRC18:08
*** dham1 has quit IRC18:08
*** ams_ has quit IRC18:08
*** teran has quit IRC18:08
*** monester has quit IRC18:08
*** sballe has quit IRC18:08
*** ashtokolov has quit IRC18:08
*** ramishra_ has quit IRC18:08
*** tpeoples has quit IRC18:08
*** etoews has quit IRC18:08
*** ameade has quit IRC18:08
*** jraim has quit IRC18:08
*** fmccrthy has quit IRC18:08
*** DuncanT has quit IRC18:08
*** serverascode has quit IRC18:08
*** betherly has quit IRC18:08
*** mgkwill has quit IRC18:08
*** zhiyan has quit IRC18:08
*** evgenyl has quit IRC18:08
*** lindsayk_ has quit IRC18:08
*** rackertom has quit IRC18:08
*** kozhukalov has quit IRC18:08
*** simonmcc has quit IRC18:08
*** electrical has quit IRC18:08
*** edwarnicke has quit IRC18:08
*** warpc has quit IRC18:08
*** Ng has quit IRC18:08
*** fdegir has quit IRC18:08
zarofungi: it looks like a regression to me.  i'll enter an issue for it and see if i can get confirmation.18:10
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Fix ENABLED_SERVICES concat failure for stable branch jobs
mriedemanteaya: dims_: sdague: ^18:10
mtreinishwoah, someone must not like irccloud :)18:10
*** tonytan4ever has joined #openstack-infra18:10
*** rackertom has joined #openstack-infra18:10
*** ams_ has joined #openstack-infra18:11
*** GheRivero has joined #openstack-infra18:11
anteayamriedem: +218:12
*** jsavak has quit IRC18:12
mriedemdon't all +W at once now18:12
anteayamtreinish: I'm missing the context18:12
*** jsavak has joined #openstack-infra18:12
sdaguemriedem: +A18:12
fungianteaya: alt+=18:12
fungilook at all the kline quits18:12
fungione of the freenode operators just blocked a popular bouncer site i guess18:13
Clintit happens18:13
nibalizerany idea why?18:14
*** aopincar_ has joined #openstack-infra18:14
fungino mention of anything on their staff twitter, so you'd probably need to ask in #freenode18:14
*** serverascode has joined #openstack-infra18:14
jeblair18:13 < k6ka> Thero: k-line was accidental. collateral damage whilst trying to stop a troll.18:14
*** DuncanT has joined #openstack-infra18:14
jeblairi lurk in #freenode18:14
anteayadid they stop the troll?18:15
fungii should do that, thanks for the details18:15
*** ccrouch has quit IRC18:15
fungidoes anyone ever stop a troll? at best you slow them down while they look for a new way to come back and annoy people18:15
anteayaI mean if they took out so many others did they at least achieve their goal?18:15
*** edwarnicke has joined #openstack-infra18:15
anteayaha ha ha18:15
fungieventually you just make it cumbersome enough for them that they hopefully move on to something less tedious18:15
jeblairthey even updated their channel topic: "disruption to irccloud unintended - our apologies, it should be fixed now!"18:15
anteayathat was kind of them18:16
*** mwhahaha_ has joined #openstack-infra18:16
*** electrical has joined #openstack-infra18:16
*** warpc has joined #openstack-infra18:16
*** sweston has joined #openstack-infra18:16
*** mwhahaha_ is now known as mwhahaha18:16
*** sridhar_ram has joined #openstack-infra18:16
*** mgkwill has joined #openstack-infra18:17
anteayalike a cattle gate18:17
jeblairfungi: i don't highly recommend lurking in #freenode -- the highlight spam is near-continuous :)18:17
*** unicell has joined #openstack-infra18:17
*** jraim has joined #openstack-infra18:17
*** ccrouch has joined #openstack-infra18:18
fungioh, that's unfortunate18:18
*** larainema has joined #openstack-infra18:18
*** kozhukalov has joined #openstack-infra18:18
fungijeblair: on the other hand, i can just tell weechat not to highlight matches in that channel. quite simple to configure actually18:18
*** ccrouch has quit IRC18:18
*** ramishra_ has joined #openstack-infra18:18
*** ccrouch has joined #openstack-infra18:18
jeblairthere you go18:18
*** gyee has joined #openstack-infra18:18
anteayathey have cattle gates around university of toronto buildings in the middle of the city18:19
*** simonmcc has joined #openstack-infra18:19
fungii already have activity and highlight levels tuned differently for different channels, to help me prioritize and cope with my terribly slow reading speed18:19
anteayasince when the buildings were built there were cattle all around18:19
*** NikolaTesla has joined #openstack-infra18:19
*** Ng has joined #openstack-infra18:20
fungianteaya: that's pretty awesome. more cities need cattle grates18:20
*** lindsayk has quit IRC18:20
openstackgerritSean Dague proposed openstack-infra/project-config: remove nova-commit-message job
*** piet has quit IRC18:20
*** teran has joined #openstack-infra18:21
*** VivekVC is now known as Vivek18:21
anteayaI think they keep them for historical reasons now, they are really pretty wrought iron affairs18:21
*** etoews has joined #openstack-infra18:21
*** ameade has joined #openstack-infra18:21
*** Vivek has quit IRC18:21
*** Vivek has joined #openstack-infra18:21
*** sballe has joined #openstack-infra18:21
*** evgenyl has joined #openstack-infra18:21
*** lindsayk has joined #openstack-infra18:21
*** erlon has joined #openstack-infra18:22
*** amuller_afk is now known as amuller18:22
*** monester has joined #openstack-infra18:22
*** yonglihe has joined #openstack-infra18:22
*** rkukura has joined #openstack-infra18:22
anteayasdague: you let me know if you line up any support for 259569 and I too will support it18:22
*** zxiiro has joined #openstack-infra18:22
*** maximov has joined #openstack-infra18:23
*** betherly has joined #openstack-infra18:23
sdagueanteaya: well I feel like we got there on the thread18:23
*** samyewell has joined #openstack-infra18:23
* anteaya grimaces at the thought of wading through the thread18:23
fungimtreinish: not sure if you saw earlier, but daily catch-up pruning of the subunit2sql database is probably around 15 minutes18:23
*** patrickeast has joined #openstack-infra18:23
funginot too terrible18:23
anteayais there a post from lana that says she agrees?18:23
*** ganso has joined #openstack-infra18:23
*** skath_ has joined #openstack-infra18:23
*** othermordred has joined #openstack-infra18:23
*** zhiyan has joined #openstack-infra18:24
AJaegerlet's wait for loquacities (that's lana's nick) to wake up and discuss with her18:24
fungianteaya: that last thread it sounded like she was fine with the new direction, but yeah she can always confirm explicitly in here18:24
*** tpeoples has joined #openstack-infra18:24
fungier, in that last reply to the thread18:24
mtreinishfungi: yeah I did catch that. It's pretty good18:24
fungishe'll likely be around any time now18:24
*** lindsayk_ has joined #openstack-infra18:25
*** Jeffrey4l has quit IRC18:25
mtreinishI did set the cron job up to run weekly:
anteayawell in it sounds like she wants the defcore ptl to weigh in18:25
mtreinishbut I can change that to be daily if you think that's better18:25
*** ericksonsantos has quit IRC18:25
*** xgerman has joined #openstack-infra18:25
anteayaso yeah, I'll wait a bit, thanks18:25
fungimtreinish: should be fine. maybe an hour to complete weekly, but likely less18:25
*** esikachev has joined #openstack-infra18:25
*** briancurtin has joined #openstack-infra18:25
*** dougwig has joined #openstack-infra18:25
*** patrickeast_ has joined #openstack-infra18:26
*** rguillebert has joined #openstack-infra18:26
*** bardia has joined #openstack-infra18:26
mtreinishand that first run will likely be much slower it's got another 6months of data sitting there still18:26
anteayawould the defcore ptl be eglute?18:26
*** dham1 has joined #openstack-infra18:26
*** patrickeast_ has quit IRC18:26
anteayadefcore-release is empty:,members18:26
mtreinishanteaya: there's no such thing, the reference was for projects covered by defcore's ptl18:27
*** fdegir has joined #openstack-infra18:27
mtreinishanteaya: defcore is a board thing so ptl doesn't make sense for that18:27
anteayamtreinish: what does that entail?18:27
*** ashtokolov has joined #openstack-infra18:27
*** HeOS has joined #openstack-infra18:27
mtreinishanteaya: nova, glance, keystone, etc..18:27
*** Ryan_Lane has joined #openstack-infra18:27
anteayaI'm just trying to figure out who to get to review sdague's patch to avoid people coming back to me saying I didn't consult them18:28
anteayanow I'm fine with people thinking I made the wrong decision or a made a mistake18:28
*** ociuhandu has quit IRC18:28
anteayabut I do like folks to at least have a heads up, so trying to do that18:28
*** _nadya_ has joined #openstack-infra18:29
*** dtardivel has joined #openstack-infra18:29
*** auggy has joined #openstack-infra18:29
*** jed56 has joined #openstack-infra18:29
anteayaor we can just move ahead with out me18:29
anteayathat is an option18:29
*** _nadya_ has quit IRC18:30
openstackgerritMerged openstack-infra/project-config: Constraints jobs non-voting for neutron-vpnaas
*** _nadya_ has joined #openstack-infra18:30
*** dims_ has quit IRC18:30
*** mgagne has quit IRC18:30
*** rcernin has joined #openstack-infra18:30
*** trown is now known as trown|lunch18:31
*** rajinir has joined #openstack-infra18:31
*** mgagne has joined #openstack-infra18:31
*** kgalanov has joined #openstack-infra18:31
*** mgagne is now known as Guest8280318:31
*** dims has joined #openstack-infra18:32
*** igorbelikov has joined #openstack-infra18:32
*** fmccrthy has joined #openstack-infra18:32
*** Guest82803 is now known as mgagne18:32
*** kdas_ has joined #openstack-infra18:33
*** mgagne has quit IRC18:33
*** mgagne has joined #openstack-infra18:33
*** otter768 has joined #openstack-infra18:33
*** ayoung has joined #openstack-infra18:33
greghaynesclarkb: hrm, pep8 fail on (dib prune)18:33
greghaynesianw: ^18:33
clarkbis it running bashate?18:34
greghayneswe should rename that job to just be "lint" since it doen't do much pep818:34
greghaynesit runs dib-lint18:34
greghaynesand pep8 on some things18:34
*** sridhar_ram has quit IRC18:35
greghaynesoh actually, it is pep8 that failed. I wonder if that or something else updated...18:35
*** sridhar_ram has joined #openstack-infra18:35
*** piet has joined #openstack-infra18:35
*** kushal has quit IRC18:35
*** yamahata has quit IRC18:35
*** woodster_ has joined #openstack-infra18:36
*** kdas_ has quit IRC18:36
*** zhenguo has joined #openstack-infra18:36
*** esikachev has quit IRC18:36
*** kushal has joined #openstack-infra18:37
*** otter768 has quit IRC18:38
*** e0ne has joined #openstack-infra18:38
*** treaki_ has joined #openstack-infra18:40
*** jaypipes has quit IRC18:41
odyssey4megreghaynes yeah, pep8 released v1.7.0 today - so if you only have 'deps=flake8' as the deps in your tox config, then you'll have to do some python pep8 fixes18:42
*** darvon has quit IRC18:43
*** ig0r_ has joined #openstack-infra18:43
*** tonytan4ever has quit IRC18:43
greghaynesodyssey4me: ty18:43
jeblairanteaya: i'd ask loquacities to weigh in on it again too; her last comment was a -1, but i think further convergence happened on the thread after that.18:44
*** jaypipes has joined #openstack-infra18:44
zarojeblair, fungi : i just verified that the gerrit tmp folder cleanup works in 2.11.  the plugin files are probably left around from older versions of gerrit.  if you manually cleanup the tmp folder and start/stop gerrit you'll see that tmp is empty.18:45
*** darvon has joined #openstack-infra18:45
odyssey4megreghaynes sure thing :) another option is to change it to 'deps = -r{toxinidir}/test-requirements.txt' and then have constraints in test-requirements.txt that you're comfortable with and maintain18:45
jeblairzaro: we have one from decemeber 2218:45
*** rkukura has quit IRC18:45
anteayajeblair: yes that seems to be what I'm seeing too, yeah I'll wait for her to be online and she what she says18:45
greghaynesodyssey4me: Ya, I mostly just wanted to make sure our CI wasnt hosed, looks like we just have one line that needs changing so not a big deal18:45
*** bharathm has joined #openstack-infra18:45
odyssey4megreghaynes happy days then :)18:46
jeblairzaro: but we can try that and keep an eye on it18:46
*** _nadya_ has quit IRC18:46
*** rkukura has joined #openstack-infra18:48
*** kushal has quit IRC18:48
*** rcrit has joined #openstack-infra18:49
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fix pep8 import not at top of file
greghaynesSpamapS: ianw ^ needed to unbreak DIB CI18:49
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add wheel mirror to mirrors
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror
*** sridhar_ram1 has joined #openstack-infra18:50
SpamapSgreghaynes: thats fixed in the gentoo review18:50
SpamapSgreghaynes: so you could just +A that18:50
greghaynesSpamapS: hrm18:50
greghayneser, :/18:51
SpamapSand I'm ok with such a transgression ;)18:51
*** ayoung has quit IRC18:51
*** rossella_s has quit IRC18:52
*** rossella_s has joined #openstack-infra18:52
*** sridhar_ram has quit IRC18:54
*** daemontool has quit IRC18:54
greghaynesSpamapS: ok, done18:54
greghaynesw00t we support gentoo soon18:54
*** daemontool has joined #openstack-infra18:54
zarojeblair: i just tested it on review-dev.o.o and tmp folder was cleaned up.18:55
openstackgerritlifeless proposed openstack-dev/cookiecutter: Require tox 2.0
*** tonytan4ever has joined #openstack-infra18:57
openstackgerritIan Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit
*** peristeri has quit IRC19:00
*** yamahata has joined #openstack-infra19:01
*** ericksonsantos has joined #openstack-infra19:01
*** mriedem is now known as mriedem_meeting19:02
*** ybathia has joined #openstack-infra19:02
*** kushal has joined #openstack-infra19:02
*** ybathia_ has joined #openstack-infra19:03
lifelessNakato: when you're around
lifelessfungi: also you should look at I think19:03
openstackgerritMerged openstack/python-jenkins: Add support for handling promotions
*** rkukura has quit IRC19:03
fungidhellmann: ttx: you're probably the two other main stakeholders in the arrangement of the reference/extra-atcs file format... would you have any objection to extending the reference/projects.yaml schema to hold that information instead?19:04
*** jaosorior has quit IRC19:05
*** jaosorior has joined #openstack-infra19:05
egluteanteaya how can I help?19:06
anteayaeglute: hi19:06
anteayawell I'm not sure you can19:06
*** ybathia has quit IRC19:06
*** ybathia_ is now known as ybathia19:06
anteayawe were discussing a patch19:06
anteayaand I had thought you were referenced but I might be wrong19:07
anteayalet me get that19:07
eglutethank you19:07
*** dtardivel has quit IRC19:07
anteayathis is the mailing list thread:
openstackgerritIan Cordasco proposed openstack-dev/hacking: Do not call "rm -rf" on arbitrary user input
fungilifeless: sdague: having trouble context-switching to wrap my head around the implications of 267096 at the moment, but at a minimum the constraints-specific tox jobs are designed to be integration tests between the requirements repo and the repo for the change being tested, whereas the normal tox jobs definitely are not, so this would require pivoting normal tox-based jobs for all repos for the19:08
fungidesired behavior to take effect19:08
anteayaeglute: this is the patch under discussion:
sdaguefungi: ok, so here is where we stand19:09
krotscheckFound link http://localhost/wheel/Ubuntu-14.04/astroid/astroid-1.3.8-py2.py3-none-any.whl19:10
* krotscheck does a little dance19:10
sdagueI started doing the neutron style way, however, in the broader nova team we spent 3 years making a switch from to tox, and spend a non trivial amount of time with contributors debugging the fact they are running tests incorrectly19:10
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: Add job for functional tests on py34 env to novaclient
sdagueso lots of folks just wanted the current targets to do the right thing19:11
sdaguefungi: to the best of my understanding, the only real difference between the jobs is the  - zuul-git-prep-upper-constraints19:11
*** rkukura has joined #openstack-infra19:12
egluteanteaya you are right, I do not see that I can help, and it does not appear to affect defcore. Let me know if there is anything else!19:12
sdagueso if we could add that to all the existing jobs, it would make the tox -e py27 targets all be able to work this way (for instance)19:12
sdagueI guess that's the piece we need to get agreement19:12
anteayaeglute: thank you for taking a look, sorry for my mistake19:12
fungisdague: lifeless: we likely will still need a dichotomy of "test with constraints because this project uses requirements" job definitions and "this project doesn't participate in openstack global requirements so don't use constraints" job definitions unless we add some more scripted tooling to match against the projects.txt too19:12
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-pip: Added extra-index-url to pip.conf
egluteanteaya no worries! I am usually around on IRC, happy to help19:13
anteayaeglute: thank you :)19:13
sdaguefungi: whether or not to test with constraints is controlled project local in tox.ini19:13
lifelesssigh pyeclib19:13
*** vivekd has joined #openstack-infra19:13
mordredkrotscheck: WOOT!19:13
lifelessmordred: I am going to guess its a missing package dep19:13
sdagueas long as our infrastructure makes the constraints file always available, then I think we're done19:13
mordredlifeless: yah.19:13
sdagueand projects opt in by changing their targets19:13
mordrednotmyname: we have a pbr patch failing on a pyeclib thing - is there something special needed related to that?19:14
lifelessfungi: we use constraints for *everyone* in devstack jobs19:14
lifelessfungi: whether they participate in g-r or not19:14
fungisdague: i'll have to take a look at nova's configuration to see whether it's already using zuul-cloner to get the right refs when run in our ci, but that will have to wait for tomorrow, so removing my -1 for now19:14
lifelessfungi: and for unit tests and so on constraints tells us what versions work with any openstack deps they have19:15
notmynamemordred: what's the bug?19:15
fungiunder a tight deadline for getting the atc passes generation working19:15
krotscheckmordred: Indeed19:15
lifelessfungi: sdague: -> I don't think g-r participation is tightly coupled to constraints use19:16
notmynamemordred: ah yes19:16
mordredlifeless: I think we need to install liberasurecode-dev - but I'm not sure why it's not installed already for that job19:16
fungilifeless: okay, that seems maybe workable then19:16
fungilifeless: sdague: to make it work the way 267096 wants, any difference between the constraints and non-constraints job definitions probably needs to get copied into nova's tox.ini instead19:16
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to
notmynamemordred: as according to plan, recent pyeclib has stopped bundling up the liberasurecode library19:16
mordrednotmyname: yay!19:16
lifelessmordred: remember we install them ourselves in integration.sh19:17
notmynamemordred: so that's a good thing, right, but it does mean that you need to install liberasurecode itself now19:17
lifelessfungi: thats back to running zuul-cloner inside tox, which I thought we had significant objections to19:17
mordrednotmyname: cool. thanks!19:17
jeblairfungi: isn't it right in 267096?  it does not run zuul-cloner, it just pip install from a file over https19:17
krotscheckTwo questions. 1- Our puppet modules are installed from master, yes? 2- Should I be adding the wheel mirror location to /home/jenkins/.pydistutils.cfg as well?19:18
jeblairfungi: (i mean, isn't the answer to your question right there in 267096?)19:18
lifelessjeblair: if we do that, then depends-on and stable branches won't get the right constraints file19:18
jeblairlifeless: right19:18
fungijeblair: probably19:18
lifelessjeblair: to that end run-tox already exports upper-constraints to be a local path19:18
jeblairlifeless: i'm saying i think fungi is prematurely lifting his objection19:18
AJaegerrbradfor: if you build the docs locally, does that work?19:18
sdaguelifeless: we just have to set the branch in the url when we branch, right?19:19
fungijeblair: i just don't want to leave an objection on a change unless i have time to look closely at it19:19
lifelessjeblair: ack19:19
lifelesssdague: that takes care of branches, not of depends-on19:19
lifelesssdague: and we need depends-on for punch-through jobs19:20
sdaguelifeless: ok, explain what you mean there?19:20
jeblairfungi, lifeless, sdague: I have left a -1 on that change because i think it's fine to have an objection and ask that the proposer of a patch look into it even if i don't immediately have the time.19:20
fungineeds to respect $ZUUL_REF19:20
fungithanks jeblair19:20
openstackgerritMonty Taylor proposed openstack-dev/pbr: Split changelog on nulls instead of (
openstackgerritMonty Taylor proposed openstack-dev/pbr: Add libjpeg and liberasurecode for tests
sdagueexcept, in the jobs zuul cloner builds and exports that file19:20
mordredlifeless: ^^ (also notmyname in case you care- thanks)19:20
sdagueso overrides the url19:20
dhellmannfungi : I have no objection to moving atc data into the project files. What's the motivation for the change?19:20
lifelessmordred: hammer applied19:21
fungidhellmann: repeatedly writing parsers for our ever changing file format rather than using a standard19:21
mordredlifeless: yup19:21
mordredlifeless: although we should be able to remove the hammer in our new bindep overlord world19:21
openstackgerritRob Crittenden proposed openstack-infra/devstack-gate: Add tls-proxy as an available service
dhellmannfungi : so you're just tired of doing the same thing over and over? sheesh. ;-)19:21
lifelessmordred: indeedy19:22
fungidhellmann: also feeling icky whenever i for line in open('extra-atcs'):19:22
sdaguelifeless: does that not cover the issue?19:22
lifelesssdague: it does for -constraints jobs, not the normal ones today. Because AIUI historical reasons19:22
dhellmannfungi : haha, yeah. let's move things over. it'll probably make the governance repo rendering easier anyway. Want me to take stab at it, or are you going to?19:22
lifelesssdague: I was working on a patch set to unify all that19:22
sdaguelifeless: ok, where is that distinction?19:22
fungidhellmann: and then the fact that we inconsistently refer to project team names between it and projects.yaml, so it adds a need for mapping code to combine the two into a singular dataset19:23
lifelessbut it assploded in complexity19:23
dhellmannfungi : ew19:23
*** ayoung has joined #openstack-infra19:23
lifelesssdague: oh, my memory - I was working on neutron's stadium needs, which is different again.
fungidhellmann: for right now i'm just going to parse it and try to magically map it up to projects.yaml, but i'm happy to submit something in the future to simplify this for all of us19:23
sdagueright, we don't need all of that19:24
lifelesssdague: the distinction - I'm fuzzy on it - but my understanding is that zuul-cloner is new, and most of the jobs used some other mechanism to get code on disk19:24
sdagueok, but we're voting with at least some of them right?19:24
lifelesssdague: ECONFUSED19:24
mordredlifeless: all jobs _should_ be using zuul cloner to get git repos on disk19:24
*** peristeri has joined #openstack-infra19:25
lifelessmordred: thats my understanding yes19:25
sdagueok, so the change I want to make19:25
dougwigall neutron stadium jobs that i'm aware of are using zuul-cloner now.19:25
sdaguemake gerrit-git-prep == zuul-git-prep-upper-constraints19:25
fungii have a couple of builder macros that seem to be effective at replacing gerrit-git-prep and friends with zuul-cloner logic19:25
sdagueso we always have upper constraints available19:25
lifelessdougwig: via a tox install override though, which is not the same thing at all :)19:25
*** piet has quit IRC19:26
dougwiglifeless: oh, indeed, i like your change.19:26
sdaguethen I think this all works19:26
AJaegerrbradfor: are you still around?19:26
*** aopincar_ has quit IRC19:26
jeblairsdague: and alder the pip install command in 267096 to use the zuul-cloner installed constraints file?19:27
sdaguebecause otherwise neutron wouldn't be working19:27
sdaguejeblair: no19:27
*** Sukhdev has joined #openstack-infra19:27
sdaguebecause there is an env override19:27
sdaguewhich is set by run-tox.sh19:27
sdaguewhich is how the infrastructure tells us what to run19:27
sdaguewhen running in the infrastructure19:27
*** rockyg has joined #openstack-infra19:27
*** piet has joined #openstack-infra19:28
*** _nadya_ has joined #openstack-infra19:28
jeblairsdague: ah, so 267096 will use the env overriden constraints file which is the zuul-cloner installed constraints file; local non-zuul usage would use the internet fetched one?19:28
sdagueI believe that's what that line means19:28
AJaegerrbradfor: the files are on and have proper permissions. I do not know why you get a forbidden, might be some configuration on the host - and I can't help with that.19:29
lifelessjeblair: yes.what sdague's patch does thats unusual is make it the main targets that are constrainted, not the -constraints ones19:29
lifelessjeblair: this is different to what we wrote up for the project testing interface etc; but I see that social construct as fairly straight forward editable19:30
*** sridhar_ram1 is now known as sridhar_ram19:30
*** unicell1 has joined #openstack-infra19:30
*** fawadkhaliq has quit IRC19:30
lifeless and provide some context there19:30
jeblairlifeless, sdague: i think i agree that the mechanics of 267096 will work iff the standard tox env is run with the constraints zuul git prep19:31
jeblairlifeless, sdague: but that isn't the case now, is it?  so that would need to happen first?19:31
sdaguejeblair: yes19:31
sdaguewhich is actually where I was trying to start the conversation19:32
jeblairand in order to do that, we should probably update the PTI as lifeless says?19:32
sdaguewill I get objections to making that change?19:32
*** unicell has quit IRC19:33
sdaguewell, actually, it would mean we wouldn't have to do those updates19:33
sdaguebecause we'd be able to use the old names19:33
sdaguewhich would also save use retraining the world19:33
anteayaretraining the world takes about 12 - 18 months and is undone if other changes requireing retraining occur during that period19:34
anteayain my experience19:34
mordredanteaya: I'd like to retrain you on that19:34
jeblairsdague: side topic -- why don't we update the default envs in tox.ini?19:35
lifelesssdague: so the pti is already updated, the ones I linked to are updating the manual19:35
anteayamordred: it's just me, so depending on your approach you might be done in less than 7 days19:35
anteayamordred: if you can retrain the world in less than 12 months, I'd like to see how you do it19:35
jeblair(i love that the name for the consistent testing interface is inconsistent)19:36
lifelesssdague: when we were doing the constraints work, we got steered by infra to making everything explicit in the project testing interface19:36
sdaguejeblair: update them to what?19:37
lifelesssdague: so I'm assuming that keeping the docs and the expectations and so on all consistent is still important19:37
jeblairsdague: envlist = py34-constraints,py27-constraints,functional,pep8,pip-missing-reqs19:37
sdaguejeblair: that assumes that people use tox without -e for running tests19:37
sdaguewhich they rarely do, in my experience19:37
jeblairsdague: ok19:38
openstackgerritMerged openstack-infra/puppet-subunit2sql: Add cron job to run weekly prune the data set of db
jeblairsdague: i feel like ultimately, you are suggesting reversing the approach from a project opt-in to an opt-out?19:38
dougwigwell, we trained them not to use -e, by insisting on including py3 env's before they worked.19:38
jeblairsdague: as projects that don't use constraints would need to update their jobs to use new "py27-without-constraints" macros, templates, etc..19:38
jeblairdougwig: that was clever of us; considering the _only_ reason that's there is for developer convenience :)19:39
jeblairit certainly has nothing to do with ci :)19:39
*** Keedya_ has joined #openstack-infra19:39
fungidougwig: meh, we also included py26 envs for the longest time even though most devs didn't have a 2.6 interpreter installed19:39
sdaguejeblair: no, what I'm saying is that we have 1 developer interface for running tests19:39
* jeblair thinks it should just be py2719:39
sdaguetox -e py2719:39
openstackgerritMerged openstack-infra/project-config: Fix ENABLED_SERVICES concat failure for stable branch jobs
sdaguethe project decides what happens behind the scenes19:40
sdagueit's a project testing interface, not project testing implementation19:40
*** piet has quit IRC19:40
jeblairsdague: except your proposal is predicated on running zuul-cloner beforehand19:40
dansmith+1 for one tox target19:40
*** boris-42 has joined #openstack-infra19:40
jeblairsdague: which means updating the standard env for running tox on all projects (openstack, stackforge, infra, etc) to clone the requirements repo19:40
jeblairsdague: any project that doesn't need/want that to happen would have to opt out19:41
sdaguehow big of a cost is that?19:41
* jeblair checks if he's still allowed to say 'stackforge'19:41
jeblair(yes, it hasn't merged yet)19:41
sdagueso, I guess I don't understand19:41
sdaguebecause we're talking about zuul cloner, in upstream ci, making another repo available on disk19:41
sdaguefar away from developers laptop19:41
sdaguethat extra repo on disk doesn't hurt anyone that doesn't want to use it19:42
*** tsg has joined #openstack-infra19:42
jeblairsdague: it will generate some extra load and add time to every job, regardless of whether they use it19:43
jeblairsdague: i do not know how much, but i ask that you consider that19:43
*** trown|lunch is now known as trown19:43
sdaguecompared to the 40 minutes to spin the node?19:44
*** rkukura has quit IRC19:44
lifelessafk sorry, -> child prep19:44
sdaguethis seems really minor19:44
AJaegerdo we need an extra repo - or just a single file from it that we could just download or distribute with puppet ?19:45
jeblairsdague: it's an extra cost to *every* tox job.  i do not know the cost.  would you be willing to consider that in your proposal?19:45
*** darvon has quit IRC19:46
*** piet has joined #openstack-infra19:46
sdagueI'm not sure what I'm considering honestly, so I don't know19:46
jeblairsdague: okay, well we should keep it in mind19:47
*** rkukura has joined #openstack-infra19:47
jeblairsdague: so to your question about objections -- the only ones i can think of are: 1) we should agree on what we want to do globally and update the PTI as necessary to reflect the consensus and define the standard19:47
openstackgerritMerged openstack-infra/system-config: Add to cacti
sdaguejeblair: ok, I guess I don't see where in the PTI this level of implementation is specified19:48
jeblairsdague: "Projects desiring to support constrained unit tests must support: tox -epy27-constraints, tox -epy34-constraints"19:49
jeblairsdague: 2) as i understand it, we would need to zuul-clone the reqs repo before running 'tox -epy27' in zuul -- that has an unknown cost to job runtime and infrastructure (git server / zuul merger) load, that should be evaluated.  if we decide that we don't want to bear that cost for all tox jobs, we would need to invert the logic around running tox jobs to allow projects to opt-out of that.19:50
sdagueis that in the repo?19:50
sdaguegit grep is failing me19:50
jeblairsdague: it's at
sdagueah ok19:50
*** ayoung has quit IRC19:51
*** ldnunes_ has joined #openstack-infra19:51
jeblairsdague: those are the only objections i can come up with.  and to be clear, i think they are things we can work through and overcome.  i'm trying to be helpful by thinking of the negative things you asked for.  :)19:51
sdaguejeblair: ok, sorry, I guess it was more rhetorical question... :)19:52
*** mriedem_meeting is now known as mriedem19:52
sdagueif I push this patch, am I in for a -1 fest that means I should give up19:52
*** ldnunes has quit IRC19:53
jeblairsdague: i think it's not trivial -- i think this touches the project testing interface which we keep standard because our heads explode if things behave differently (in certain ways -- particularly this one where we're talking about dependencies) between projects.19:53
sdagueit is unit tests, when do we do dependenies between projects in unit tests?19:55
*** darvon has joined #openstack-infra19:55
*** mpavone has joined #openstack-infra19:55
jeblairsdague: there was a close-paren there.19:56
*** mudassirlatif has quit IRC19:56
jeblairsdague: the 'between projects' was 'behaving differently'.19:56
openstackgerritSean Dague proposed openstack-infra/project-config: always make upper-constraints available to python jobs
sdagueah, ok19:56
*** aopincar_ has joined #openstack-infra19:56
jeblairadding words made less clear19:56
*** davidlenwell has quit IRC19:57
sdaguewell, honestly I think there is too much belief that things aren't already different between projects :)19:57
sdagueI don't see this as substantially making that any worse19:57
sdagueand it hugely helps in reducing confusion in getting folks over19:57
jeblairsdague: i'm not saying it's a bad idea19:57
sdagueso I think is what is needed19:57
*** sridhar_ram has quit IRC19:58
*** davidlenwell has joined #openstack-infra19:58
mordredsdague: we'd like to in-general trend towards the Platonic idea rather than away from, even if the shadows and the objects are not the same today19:58
mordredsdague: but I'm just drive-by commenting at the moment, so ignore me19:58
sdaguemordred: sure, that's fine, but I feel like that sword gets wielded a lot in the abstract19:59
mordred(tried to figure out a way to get assymptotically into that sentence but failed)19:59
mordredsdague: indeed19:59
*** mpavone has quit IRC19:59
sdagueand it usually disregards all the actually important differences that come with separate team cultures19:59
mordredsdague: in a non-drive y comment way - I think that th abstract is very important to keep track of, and balancing the practicality of the moment with the goal of the abstract is the hard part of the job19:59
*** baoli_ has quit IRC20:00
mordredsdague: see, we were so close to being on the same page - I largely see the "important differences that come with separate team cultures" as a big bug in OpenStack. nova having its own culture is a problem, not a feature20:00
mordredsdague: and projects having that own culture is one of the thing that leads to cliques and boys-clubs and intra-project distrust20:01
jeblairsdague: i hope you are not dismissing my concerns as abstract -- i am VERY seriously concerned about the complexity of the constraints system which i believe no one really fully understands in its entirety, and i do not think we should lightly take the decision to ignore the cross-project standard we have established around it20:01
mordredhowever much it may be reality that it exists20:01
jeblairsdague: i repeat *we can change that standard*20:01
*** tonytan4ever has quit IRC20:03
*** bharathm has quit IRC20:04
*** eharney has quit IRC20:06
SamYaplemordred: For whats its worth I agree with you on the "own culture" thing. I have been trying to get Kolla to a more generic way of doing things where possible20:06
*** rguillebert has quit IRC20:07
*** ddieterly has quit IRC20:09
openstackgerritMerged openstack/diskimage-builder: Initial add of gentoo support for diskimage-builder
*** claudiub has quit IRC20:10
*** esikachev has joined #openstack-infra20:11
*** rkukura has quit IRC20:14
*** ldnunes_ has quit IRC20:14
*** rajinir has quit IRC20:15
jeblairproposed at least as a discussion starting point: Change the default set of tox jobs to run20:15
pc_mHi infra folks! I need some guidance on how to handle getting neutron to use the neutron-lib project that is being developed.20:16
*** annegentle has quit IRC20:17
pc_mdougwig created a commit,, that uses the neutron-lib. It fails several dsvm based tests.20:17
*** terriyu has joined #openstack-infra20:17
*** annegentle has joined #openstack-infra20:17
*** eliqiao has quit IRC20:17
pc_mIt *looks* like we could add neutron-lib to the list of PROJECTS in Would that be how to approach this?20:17
dhellmannfungi : move extra-atcs into projects.yaml20:18
*** dims has quit IRC20:18
fungidhellmann: thanks a bunch!20:18
*** esikachev has quit IRC20:18
*** dims has joined #openstack-infra20:18
dhellmannfungi : I'm pretty sure there are other tools that need to be updated, but that should get us started20:18
pc_mWould that work, or is there some other way to get neutron-lib installed so that neutron can reference it?20:19
pc_mThere are 6 voting tests that fail.20:20
*** eliqiao has joined #openstack-infra20:20
fungidhellmann: structure data ftw. also would be nice to have the expires-in be something easier to match on, but i guess that's mainly just helpful for the tc to know when to expunge them from the file20:20
*** rcernin has quit IRC20:20
pietSorry to randomize, but looking forward to meeting some of you at the mid-cycle in Fort Collins!20:20
*** lucasagomes is now known as lucas-dinner20:20
anteayapiet: what midcycle?20:21
dhellmannfungi : we can adjust that to be a date format we can parse or something. or just use all numbers? let's do that as a follow-on based on what the tool needs20:21
anteayapiet: the infra mid-cycle?20:21
fungipiet: you're coming to work on the infra-cloud deployment?20:21
*** Keedya_ has quit IRC20:21
anteayapc_m: um, have you chatted with dougwig about this?20:22
fungidhellmann: yeah, i don't actually have a use for that piece of metadata at the moment anyway20:22
dhellmannfungi : ok20:22
anteayapiet: do you mean dinner when we are in fort collins?20:22
pc_manteaya: sure. He asked if I could look into the faillures.20:22
anteayaoh okay yeah, well be pingable on irc and I'll do my best to relay dinner plans so you can join in where appropriate20:23
pietanteaya: This sprint -
anteayapiet: I'm looking forward to seeing you too20:23
anteayapiet: right20:23
dougwigpc_m: this is not the appropriate place for this.20:23
dougwigpc_m: let's go off channel.20:23
anteayapiet: also is ux having a sprint?20:23
anteayapiet: Zara mentioned a ux sprint, was that your group?20:23
pietanteaya; Yeah, one in London (assuming I get approval) that coincides with the Operators Summit in Manchester20:24
*** _nadya_ has quit IRC20:24
lifelesssdague: jeblair: I'm back. Where did the discussion get to?20:24
pietanteaya: One in Portland that coincide with the Horizon sprint20:24
anteayapiet: okay can you pencil in anything on the sprints wikipage?20:25
anteayapiet: Zara mentioned something and I had nothing to reference on the sprints page20:25
anteayapiet: thank you20:25
pietGive me a day to confirm travel approval20:25
*** e0ne has quit IRC20:25
anteayapiet: understood20:26
*** rkukura has joined #openstack-infra20:26
*** Sukhdev has quit IRC20:26
fungipleia2: do you have any pointers for how we can collect contributor details from zanata yet?20:26
pleia2fungi: I think Daisy was working directly with the Zanata folks on that, I don't have any updates20:27
gtmanfredcan I get someone to look at this?
*** ericksonsantos has quit IRC20:27
fungipleia2: okay, just hoping in can integrate something into this new tool i'm writing so that it doesn't end up being a manual process20:27
pleia2fungi: unfortunately at the summit all I learned is that they're overhauling parts of their API to better support it (and removing some old ooptions)20:28
sdaguelifeless: I don't know, I've proposed a pti change though -
pleia2so now is probably a bad time20:28
fungipleia2: oh, that's unfortunate. i don't suppose there's much we can parse out of its git backend either?20:28
pleia2fungi: nope20:29
clarkbthere is a mysql db ya?20:29
clarkbmaybe we can parse out of that?20:29
pleia2there is, but it's a bit... complicated20:29
pleia2this table, see that table for association20:29
fungii'm okay with just waiting for an api in that case (hopefully one that doesn't require admin privs)20:29
pleia2we specifically asked that whatever it is not require admin20:29
pleia2they were pushing for requiring it, since some projects aren't open source everyone-sees-everything like us20:30
pleia2but I think we won an option in the admin interface to allow for it20:30
*** jaosorior has quit IRC20:30
fungii'm just about at feature parity with our old atcs script, plus some nice-to-haves like duplicate address detection and extra-atc merging20:30
*** eharney has joined #openstack-infra20:30
*** jaosorior has joined #openstack-infra20:30
*** bharathm has joined #openstack-infra20:31
pleia2it's about time for me to drop an email to Daisy about it anyway20:31
anteayafungi: yay20:31
pleia2I think she actually has something to poll for ATC for i18n20:31
anteayaand yay for dup detection20:31
pleia2but it could have been something manual20:31
*** esikachev has joined #openstack-infra20:32
*** jsavak has quit IRC20:33
anteayapleia2: enjoy lunch20:33
*** mudassirlatif has joined #openstack-infra20:33
*** otter768 has joined #openstack-infra20:34
*** jaosorior has quit IRC20:34
*** tonytan4ever has joined #openstack-infra20:35
*** ddieterly has joined #openstack-infra20:35
*** jsavak has joined #openstack-infra20:35
openstackgerritSean Dague proposed openstack-infra/project-config: always make upper-constraints available to python jobs
sdaguejeblair: updated with Depends-On for governance change20:36
anteayamordred: so you are aware that currently nodepool nodes on bluebox are 100% error, yeah?20:36
mordredanteaya: I am not20:37
mordredjlk: ^^ your cloud is broken20:37
*** rguillebert has joined #openstack-infra20:37
*** daemontool has quit IRC20:37
mordredthat sure does look like error20:37
anteayabig box of red20:37
jeblairer wait20:37
* anteaya waits20:38
jeblairwe probably just haven't restarted after changing the flavor20:38
anteayaah ha20:38
mordredoh! yes20:38
mordredthat, in fact, would likely do it20:38
anteayajlk: might be our end20:38
*** daemontool has joined #openstack-infra20:38
*** otter768 has quit IRC20:38
* mordred hands jlk a nice cold towel20:38
*** markmcclain has joined #openstack-infra20:38
jeblairthough, we were in the middle of a conversation with jlk about storage and we were waiting on wrapping that up to see if we were going to change the flavor again :)20:38
jlkoh right.20:39
openstackgerritIan Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit
* mordred takes the towel back and uses it to chastise jlk20:39
jlkthat's. That's an interesting story20:39
* jeblair settles in for storytime20:39
* anteaya pours hot chocolate20:39
*** hashar has joined #openstack-infra20:39
jlkthis is something that shouldn't be spread around20:40
jlkyour cloud is slightly misconfigured.20:40
jeblairi won't tell anyone20:40
jlkThe machines we get from SoftLayer have two raid arrays. One array that is a mirror of "slow" disks for the OS. We asked for 500G per the BOM, we got 1TB.20:40
jlkThere is a second array, a 1.2TB array of 4 fast disks.20:40
* rockyg ,s ears perk up20:40
jlk10K RPM SAS20:40
*** ddieterly has quit IRC20:41
jlkAt the time your cluster was deployed, the code to deploy it hadn't yet learned how to deal with two disk sets20:41
jlkprior to using SoftLayer we just had one big fast array20:41
jlkso the code treated this system the same way, so all your instances are landing on that 2 disk slow mirror20:41
jlkand the 4 disk raid 10 mirror is sitting unused.20:41
jlkCapacity wise, they nearly matched up, looked right20:42
* anteaya waits for the part where bluebox gets faster20:42
jlkWe're working on the orchestration to migrate instance data from the slow mirror to the fast raid 1020:42
jlkand going to schedule the maint window to execute it20:42
jeblairanteaya: i think it's coming up!20:42
jlkWe're going to put y'all on the top of the list to execute20:42
mordredwe like this20:42
anteayajlk: thank you20:43
jlkWhat would help at that time, is if ya'll could purge your cloud20:43
jlkso we don't hav eto rsync over any instance data20:43
*** apoorvad has quit IRC20:43
jeblairjlk: so at the end of this, we will still have ~1TB per hypervisor to share among root and ephemeral volumes?20:43
jlkif there is ever a good time for you to do that, we could coordinate with you and get it done.20:43
anteayajlk: so purging the cloud is just setting bluebox to 0 max_servers in nodepool.yaml?20:43
jeblairjlk: just a different 1TB?20:43
anteayasorry that was for jeblair20:43
anteayajeblair: ^^20:43
jlkjeblair: correct. It's about 1.2TB usable.20:43
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Prepare neutron-lib for pypi releases
jeblairjlk, anteaya: it's easy for us to stop using the instances; but it's a moderate amount of work for us to rebuild the mirror (maybe ~1 or 2 days)... how hard would it be to have 1 instance with 300G of data to move?20:45
anteayaah, the mirror20:45
jlkjeblair: not hard, just makes the outage longer. We'll have to stop the instance and make sure all the file handlers are freed20:45
jlkI'm hoping we can do rsync's on the live data leading up to the outage window, so it'd be a quick sync up for any last minute changes20:46
rm_workthis issue is nothing to do with our tests and a recheck should resolve, right?
rm_work^^ just happened20:46
rm_workwell, semi-recently20:46
jeblairjlk: okay, that's probably best then if you can accomodate it; if we later find it's too much trouble, we can rebuild it20:46
*** dimtruck is now known as zz_dimtruck20:47
*** sridhar_ram has joined #openstack-infra20:47
jlkbig apologies on this one. Our 2.0 product is all set to handle the disk separation.20:47
jeblairno worries!  we like helping people find problems!20:47
jeblairand even solutions20:48
jeblairjlk, clarkb, mordred, anteaya: sounds like no changes for the moment so i'll go ahead and restart nodepool to pick up the new flavor now20:48
openstackgerritAnita Kuno proposed openstack-infra/project-config: REduce bluebox max_servers from 39 to 0
anteayathere's the patch for when we want to merge it20:49
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: refactor prior to move to project-config
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: refactor script for commenting on launchpad bugs prior to move to project-config
*** rcernin has joined #openstack-infra20:49
openstackgerritAnita Kuno proposed openstack-infra/project-config: Reduce bluebox max_servers from 39 to 0
jeblairgertty needs a history screen -- like "these are the last X changes you looked at"20:51
anteayajlk: do you want to -1 on 267160 for now and +1 when you know when the maintenance window is scheduled?20:51
dhellmannfungi, jeblair : I would appreciate your thoughts on those 2 patches in release-tools ^^ as precursors to moving the scripts over to project-config20:51
jlkanteaya: sure can!20:52
anteayajlk: thank you, helps us get the timing right20:52
jlkI really can't get used to the new gerrit UI20:53
anteayajlk: you are in good company20:53
jeblairjlk: how do you feel about text-mode console applications?20:54
jlkWell, I already use "hub" to manage github PRs, so... favorable. I know there are tools out there for gerrit, it's just not party of my day to day work life so I haven't bothered yet.20:54
jeblairjlk: well, if it every becomes a significant part of your work, i wrote gertty which is like 'mutt' for gerrit:
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: default to sending mail directly via
jlkmutt.... ouch.20:55
jeblairjlk: otherwise, they are working on an actual modern javascript interface called polygerrit, but it's a ways away20:56
jeblair(it's actually more like gnus for gerrit, but more people use mutt so i say that)20:56
*** armax has quit IRC20:56
mordredjlk: gertty is teh ossum20:56
jlkjust as long as it isn't full of ncurses20:56
jlkor emacs key bindings20:57
jeblairjlk: oh no.  no ncurses.  direct ansi codes.  emacs keybindings are default, vi are optional.20:58
*** baoli has joined #openstack-infra20:59
anteayamorning jhesketh20:59
jrollwhoa, vi bindings exist in gertty now?21:00
* jroll applies config21:00
jeblairwow, i'm watching the story of a node coming online by seeing spikes on the api graphs appear in grafana21:00
*** ari has quit IRC21:01
mordredjeblair: which daskboard?21:01
jeblairjroll: it has support for multi-key commands now, so you can ':q', etc... there's some examples of how to do that in the config -- if someone who actually knows vi writes a set of bindings we can merge it in to gertty so it's built-in21:01
*** ekarlso has quit IRC21:01
*** ekarlso has joined #openstack-infra21:01
jrolljeblair: ah, I see :)21:02
*** gildub has joined #openstack-infra21:03
openstackgerritSamuel de Medeiros Queiroz proposed openstack-infra/project-config: Identity API v3 only non-voting job in Tempest
*** rockyg has quit IRC21:03
*** rcernin has quit IRC21:04
samueldmqsdague: mtreinish ^ change for tempest21:04
samueldmqmtreinish: that was quick! thanks21:05
*** Keedya_ has joined #openstack-infra21:06
*** rockyg has joined #openstack-infra21:07
*** Keedya_ has quit IRC21:07
*** akshai has quit IRC21:08
*** zz_dimtruck is now known as dimtruck21:08
anteayasamueldmq: great commit, message21:08
anteayathank you21:08
anteayasorry for the errand comma in there21:08
*** ddieterly has joined #openstack-infra21:08
openstackgerritJames E. Blair proposed openstack/gertty: Add a vi keymap
samueldmqanteaya: thanks, I tried to make it very clear what the state is and what it is about21:09
*** baoli has quit IRC21:09
jeblairjroll: here i made it easy in  :)21:09
samueldmqanteaya: thanks for your feedback :)21:10
*** baoli has joined #openstack-infra21:10
anteayasamueldmq: you were very successful, what you gave me is what I hope for in all commit messages, please keep doing that21:10
jeblairjroll: i only know how to quit vi, so that's all i can do for now21:11
samueldmqanteaya: I will try with my best, thanks :)21:11
anteayasamueldmq: :)21:11
anteayajeblair: ha ha ha21:11
*** ayoung has joined #openstack-infra21:11
*** yonglihe has quit IRC21:11
jrolljeblair: ha! nice21:12
jrollI'll see what I can do21:12
anteayawhen I first opened vi, at home, far from user groups, fresh to linux from windows and no connection to the internet, I opened vi and couldn't close it21:12
*** akshai has joined #openstack-infra21:12
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-pip: Added extra-index-url to pip.conf
anteayaI've had car accidents that caused me less stress21:12
mtreinishjeblair: you're better than me I don't know how to exit emacs besides 'killall -9 emacs'21:12
openstackgerritJames E. Blair proposed openstack/gertty: Add a vi keymap
*** amrith is now known as _amrith_21:13
jeblairmtreinish: no that's right.21:13
*** esikachev has quit IRC21:13
dougwigthe modal editor that will not die.21:13
*** ig0r_ has quit IRC21:14
jeblairanteaya: if you've never read you may enjoy it :)21:15
jeblairvi and emacs have nothing on ed21:15
anteayaI've used ed21:15
anteayaI like it21:16
anteayaha ha ha21:16
anteayaoh but that was intentionally21:16
anteayathere was one time I said open the file and it opened in ed, boy that was another fun one21:16
anteayasome of my most terrifying life memories are text editor related21:17
jeblairi didn't think anyone had ever used ed intentionally21:17
anteayaI read a tutorial21:17
anteayaand though meh, I'll try it, so I did21:17
dougwigi've used it when /bin was the only thing not hosed. but not by choice.21:18
*** akshai has quit IRC21:19
*** baoli has quit IRC21:21
*** baoli has joined #openstack-infra21:22
anteayaI can feel your neck tendons creak21:22
anteayajeblair: thanks that was fun21:22
*** kgiusti has quit IRC21:22
*** kzaitsev_mb has joined #openstack-infra21:23
*** ig0r_ has joined #openstack-infra21:23
*** sdake has quit IRC21:26
*** angdraug has quit IRC21:28
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics
*** armax has joined #openstack-infra21:30
mordredjeblair: I want you to know I just fell into a hole of reading the texinfo manual for ed21:31
*** ig0r_ has quit IRC21:31
*** aysyd has quit IRC21:31
gtmanfredcan I get someone to look at this patch for shade?
*** jtomasek has quit IRC21:32
anteayajeblair: nicely done21:33
*** baoli has quit IRC21:33
*** baoli has joined #openstack-infra21:34
*** salv-orlando has quit IRC21:34
openstackgerritIan Cordasco proposed openstack-infra/project-config: Add integration jobs for Bandit
*** salv-orlando has joined #openstack-infra21:35
*** apoorvad has joined #openstack-infra21:35
anteayafungi: there is a gerrit bug up for the needed to parse change successfully merged for the timestamp of the merge if I remember correctly, could that be added to the commit message for your script as well?21:36
Shrewsgtmanfred: Could you educate me on exactly why this is needed? TBH, I'm quite concerned about the security implications of storing plain text passwords in possibly cached data.21:36
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add wheel mirror to mirrors
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror
* anteaya followed the link to the bug for viewing emails via REST and reviewed it in gerrit's gerrit21:37
gtmanfredShrews: for when servers are built, it is the initial password. I didn't think it cached the data from a build, but if you aren't passing a key for the initial build, I use the initial passwod, and then use my saltstack automation to put my ssh keys on the server and disable password authentication21:38
fungianteaya: it's actually mentioned in at least a couple code comments in the script itself with todo tasks called out21:39
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to
anteayafungi: oh sorry, was skimming, I will look closer21:40
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Bump APScheduler to >=3.0
fungianteaya: that way it survives a bit better than getting hidden away in a commit message21:40
*** ajmiller has quit IRC21:40
anteayaso it is21:40
anteayayup, your way is better21:41
anteayaclearly spelled out in line 111 for example21:41
fungianteaya: it will basically allow us to gut that really hacky function and just check a value in the change data we're already querying instead21:41
Shrewsgtmanfred: i'm still not clear as to why the server object must keep the initial password around AFTER the server has been built?21:41
greghaynesianw: just kicked off a recheck on the branch pruning patch - should be able to pass now21:42
*** rfolco has quit IRC21:42
gtmanfredbecause it is never returned, so if it is auto assigned by the provider, you never get it back21:42
anteayazaro: oh look this patch of yours has some helpful reviews!
fungianteaya: and it will also let us cut down on some additional query bloat since we no longer need to get gerrit to include comment details on each change21:42
gtmanfredrackspaces cloud auto assigns a password, and only the initial return of the server object has that admin_pass in it21:42
openstackgerritFei Long Wang proposed openstack-infra/project-config: Make zaqarclient-dsvm-functional job as voting
anteayafungi: yup, it will be much cleaner after 73651 merges and is backported21:43
anteayafungi: thanks sorry for being slow, yay script21:43
loquacitiesanteaya, jeblair: you rang?21:43
gtmanfredShrews: if it was only for passing admin_pass to the object, it wouldn't be needed, but for one that is assigned, there is no way to get the password after that first return21:43
anteayaloquacities: hello21:43
*** ajmiller has joined #openstack-infra21:44
anteayaloquacities: just wanted to get your on the record response to 259569 so I can review21:44
*** ig0r_ has joined #openstack-infra21:44
anteayaloquacities: sdague feels you have reached agreement21:44
*** akshai has joined #openstack-infra21:44
Shrewsgtmanfred: ah, i think i'm starting to understand21:44
*** ig0r_ has quit IRC21:45
loquacitiesagreement insofar as 259569 can go ahead, i guess21:45
*** baoli has quit IRC21:45
anteayaloquacities: I can accept that21:45
anteayathank you21:45
*** baoli has joined #openstack-infra21:45
Shrewsgtmanfred: to be honest, i'm not entirely sure this is something we'd want to support. mordred, do you have thoughts on this?21:46
openstackgerritgordon chung proposed openstack/requirements: bump ceilometerclient to 2.2.1
mordredI mean, it _is_ a part of the nova api thing - but I certainly personally have no interest in admin passwords21:47
mordredit doesn't bother  me supporting it though21:47
mordredShrews: what's the issue?21:47
Shrewsmordred: my concern would be security implications, especially if someone is using caching21:47
mordredit seems to be mainly just passthrough, no? oh - I see what you're saying21:48
anteayathanks loquacities21:48
Shrewsmordred: we could totally support it, but have the caveat that "hey, you could open yourself up for bad things"21:48
mordredit's because we're grabbing it from the initial return and then injecting it after the wait loop21:48
*** piet has quit IRC21:49
*** Sukhdev has joined #openstack-infra21:49
*** dimtruck is now known as zz_dimtruck21:49
Shrewsmordred: if someone chooses file-backed dogpile backend caching, for example, they suddenly have clear text passwords in the cache file21:49
Shrewsif i'm being overly paranoid, then the change is fine. just want us to be aware of the implications21:50
mordredwe could also put a filter in the cache-if for that ...21:51
*** aopincar_ has quit IRC21:51
mordredif the server has a password, don't cache it?... OH - wait, that's not going to get cached21:51
mordredthis is the return value of the create call21:51
*** dprince has quit IRC21:51
mordredwe cache list-servers21:51
gtmanfredthat was what I was thinking21:51
*** sdake has joined #openstack-infra21:53
Shrewsmordred: get_server() will return the server FROM the cache (once it appears). then it gets modified. doesn't that change go back into the cache?21:54
mordredShrews: not to my knowledge, no21:55
*** rcernin has joined #openstack-infra21:55
mordredShrews: MAYBE using the in-memory cache option, but that is a memory leak anyway and is not suggested21:55
mordredbut if it's a remote cache, like dbm or memcached, it'll cache-then-return and the objects after that point are not active-tied back to the cache21:56
Shrewsmordred: if that's the case (my knowledge of dogpile is limited), then my worry is for naught21:56
*** sdake_ has joined #openstack-infra21:56
*** hashar has quit IRC21:56
*** baoli has quit IRC21:57
mordredShrews: it's good to worry21:57
*** sdake has quit IRC21:57
*** baoli has joined #openstack-infra21:58
Shrewsgtmanfred: in that case, please add a unit test that makes sure the password returned from initial create survives (not everyone can run the functional tests, but everyone should be able to run unit tests)22:00
Shrewsbasically an equivalent of your functional test22:00
*** vgridnev has quit IRC22:00
*** ZZelle_ has joined #openstack-infra22:01
*** piet has joined #openstack-infra22:05
openstackgerritMonty Taylor proposed openstack/requirements: Update constraint for os-client-config
*** salv-orl_ has joined #openstack-infra22:09
*** baoli has quit IRC22:09
*** baoli has joined #openstack-infra22:10
*** burgerk has quit IRC22:10
*** zz_dimtruck is now known as dimtruck22:10
ianwgreghaynes: awesome, thanks, will keep an eye on it this time :)22:10
*** rcernin has quit IRC22:10
*** salv-orlando has quit IRC22:12
openstackgerritDaniel Wallace proposed openstack-infra/shade: Save the adminPass if returned on server create
gtmanfredShrews: done, just waiting for tests to run now :)22:13
*** jcoufal has quit IRC22:14
*** yamahata has quit IRC22:14
*** ddieterly has quit IRC22:16
*** jsavak has quit IRC22:18
openstackgerritRandall Burt proposed openstack-infra/project-config: Fix more syntax errors in gate-heat-templates-dsvm
*** rlandy has quit IRC22:18
*** dimtruck is now known as zz_dimtruck22:20
*** smckee has joined #openstack-infra22:21
anteayasmckee: hello22:21
*** baoli_ has joined #openstack-infra22:21
ianw ... there really is a xkcd for everything22:21
anteayaso smckee is working on a study tracking git habits22:21
*** trown is now known as trown|outttypeww22:21
anteayasmckee: and we were talking in -dev22:21
anteayasmckee: (s)he is interested in know what other projects use our workflow22:22
*** daemontool has quit IRC22:22
*** hdd has quit IRC22:22
anteayaoff the top of my head I had wikimedia and hpe22:22
anteayaanyone have any others?22:22
*** Sukhdev has quit IRC22:22
anteayain any case I have to go for a walk now22:23
anteayaeveryone this is smckee22:23
smckeeThanks again for all your help :)22:23
anteayasmckee: this is everyone22:23
*** daemontool has joined #openstack-infra22:23
anteayahappy to help, good luck with the study22:23
*** boris-42 has quit IRC22:23
*** jpr has quit IRC22:24
*** baoli__ has joined #openstack-infra22:24
*** baoli has quit IRC22:24
*** armax has quit IRC22:25
*** mase_x200 has joined #openstack-infra22:25
*** baoli_ has quit IRC22:26
*** isaacb has quit IRC22:26
*** baoli__ has quit IRC22:27
smckeeIf anybody is interested in the study that anteaya was talking about, we're aiming to study people who do major Git merges such as (but definitely not limited to) large backports that result in major conflicts.22:28
*** baoli has joined #openstack-infra22:28
smckeeBasically, the Git nightmare. I have come across people like this in other projects, but looking at openstack, I am also wondering if certain infra designs might prevent these sorts of issues altogether22:28
*** rlandy has joined #openstack-infra22:29
smckeeSo if anybody has any info in that general realm, I'd be interested in talking :)22:29
*** zz_dimtruck is now known as dimtruck22:29
*** baoli has quit IRC22:30
*** gildub has quit IRC22:30
*** baoli has joined #openstack-infra22:30
*** dkranz has quit IRC22:32
*** regXboi has quit IRC22:33
*** sridhar_ram has quit IRC22:33
*** Sukhdev has joined #openstack-infra22:33
*** sridhar_ram has joined #openstack-infra22:33
*** rockyg has quit IRC22:34
*** Sukhdev has quit IRC22:34
*** otter768 has joined #openstack-infra22:35
*** piet has quit IRC22:35
*** tiswanso has quit IRC22:37
*** xyang1 has quit IRC22:39
*** otter768 has quit IRC22:40
sputnik13we've had the following patches stuck and not merging for a few days now
sputnik13have no idea why they're not merging22:41
sputnik13can anyone help?22:41
*** baoli has quit IRC22:42
*** hdd has joined #openstack-infra22:43
*** baoli has joined #openstack-infra22:43
*** thorst has quit IRC22:43
*** thorst has joined #openstack-infra22:44
*** mase_x200 has quit IRC22:45
*** vivekd has quit IRC22:46
tonybjeblair: If you have a few minutes I'd appreciate a review on:
*** thorst has quit IRC22:48
tonybsputnik13: I looked at the first one, and it needs a rebase.22:49
tonybsputnik13: look for the little orange circle next to the parent SHA122:49
clarkbgreghaynes: you think we can attemp image builders late afternoon?22:51
tonybsputnik13: the others need more careful checking to work out why22:52
clarkbI think crazyness is dying down on my end22:52
sputnik13tonyb thanks, it looks like 3 of the patches have that, this one for example does not22:52
clarkbsmckee: so one thing that happens due to our use of gerrit is that changes tend to be small and logically contained22:53
*** rajinir has joined #openstack-infra22:53
*** tzn has joined #openstack-infra22:53
clarkbsmckee: this means that unlike github pull requests that can have massive numbers of commits that get merged into master we have many small changes that individually merge into master22:53
tonybsputnik13: Yeah I'm not sure about that one (and the others in a similar position)22:53
clarkbsmckee: it sort of distributes the problem of dealing with rebases to the authors most familiar and able to deal with rebases in any particular section of the code22:53
*** armax has joined #openstack-infra22:54
sputnik13tonyb that said, I don't think they're stuck because of the need for rebase, usually merge will take care of things for you unless there's a merge conflict22:54
*** baoli has quit IRC22:54
*** randallburt has joined #openstack-infra22:54
sputnik13and I was able to rebase using the rebase button, so it doesn't seem like a merge conflict22:55
*** baoli has joined #openstack-infra22:55
clarkbsmckee: for backports our stable branch policy is pretty restrictive and doesn't allow features, just high priority bugs and similar which also avoids needing to deal with massive merge conflicts22:55
*** sigmavirus24 is now known as sigmavirus24_awa22:55
randallburthi there, anyone from infra have time for a quick question?22:56
smckeeOK, that makes sense. Is that policy published anywhere, by any chance?22:56
*** tzn has quit IRC22:57
randallburtSo I submitted a fix for one of our Heat template tests and it merged. The blocked job then passed the gate once that fix merged. However, once my original patch was approved, the same error shows up again:
clarkbsmckee: yes let me find a link22:58
tonybsputnik13: no, they're not eligable to move forward without a rebase.22:58
openstackgerritMerged openstack-dev/pbr: Add libjpeg and liberasurecode for tests
tonybsputnik13: the mergeconflict is in the parent patch not the review that wont merge22:59
*** mriedem is now known as mriedem_away23:00
smckeeAwesome, thanks clarkb. I have to meet a student to help them for my class, but I really appreciate your help :)23:00
*** smckee has quit IRC23:01
*** armax has quit IRC23:01
clarkboh hey it ssomeone at oregon state23:01
clarkbwe could totally talk over beers23:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add ansible as a dependency
pabelangerjeblair: Woah, nice!23:03
jeblairjhesketh: ^ there's the rough 'hello world'23:03
*** thorst has joined #openstack-infra23:04
jheskethjeblair: awesome :-)23:04
jeblairjhesketh: my next task is getting a sensible playbook from the config over to the launcher for you to use23:05
jeblairjhesketh: in the interim, things i haven't tackled yet in the launcher include: having ansible route output to a file and performing the rsync of the git directory to the hosts23:05
jeblairjhesketh: and probably lots of other ansibly things23:06
*** ccrouch has quit IRC23:06
*** baoli has quit IRC23:06
*** ccrouch1 has joined #openstack-infra23:06
jeblairjhesketh: so there should be lots of stuff that can happen there that's not blocked on actually having the job playbook to run the job :)23:06
*** baoli has joined #openstack-infra23:07
*** thorst has quit IRC23:08
*** amitgandhinz has quit IRC23:11
*** dcentral has joined #openstack-infra23:15
*** dcentral has quit IRC23:16
*** thorst has joined #openstack-infra23:17
rm_workis the gate broken ATM?23:18
rm_workseeing this on every one of the lbaas gate tests:23:18
*** baoli_ has joined #openstack-infra23:18
*** annegentle has quit IRC23:18
rm_workor is that our issue23:18
rm_workpossibly due to the missing package name thing above it?23:19
clarkbrandallburt: you will need the fix to merge before it shows up in check tests23:19
clarkbrandallburt: check only checks against what is merged, the gate does speculative trees though so you can line them up there if things are working otherwise23:19
*** baoli__ has joined #openstack-infra23:20
randallburtclarkb:  but it did merge23:20
clarkbrandallburt: but did the fialing job run after it merged?23:20
randallburtclarkb:  yep23:20
clarkbrm_work: thats the actual issue23:20
clarkbrm_work: or right below it23:21
randallburtclarkb:  the patch that exposed the bug ran fine yesterday. it was approved today and the bug is back23:21
rm_workclarkb: so is that our issue?23:21
rm_workclarkb: or a bigger devstack/gate issue23:21
randallburtclarkb:  checked master and nothing stomped on my fix so I am confused.23:21
clarkbrm_work: I think it is your issue23:21
*** baoli has quit IRC23:21
clarkbdevstack doesn't run dib by default23:21
clarkbrandallburt: I am almost positive it is going to be a timing issue but I hvae to unravel it to be sure23:22
randallburtclarkb:  k. would just running recheck till it sorts itself help or hurt?23:22
rm_workclarkb: passed about 3 hours ago, failing now, nothing has merged inbetween I don't think :/23:23
clarkbwell if the fix isn't the fix it will just waste test nodes23:23
rm_workbut I will look into it23:23
*** ociuhandu has joined #openstack-infra23:23
*** baoli_ has quit IRC23:23
randallburtclarkb:  the fix should be the fix since the test for the original patch ran fine yesterday23:23
*** peristeri has quit IRC23:23
clarkboh ok the fix is to project-config23:23
randallburtclarkb:  yep23:23
clarkbso ya its going to be a timing issue I bet23:23
clarkbbecause puppet has to run then jjb hsa to run and we hva eto do it across 8 jenkins masters23:24
clarkblet me check the configs on the jenkins masters now23:24
randallburtclarkb:  ah, so just lucked out on the node selection yesterday?23:24
randallburtclarkb:  cool, thanks!23:24
clarkbwell its eventually consistent23:24
clarkbso you raced it23:24
clarkbI am going to confirm that it was just slow and not broken though23:25
randallburtclarkb:  k23:25
clarkbhrm jenkins01 still has the old config23:25
*** amotoki has joined #openstack-infra23:28
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Temporarily make lbaasv2-minimal job non-voting, due to breakage
*** baoli__ has quit IRC23:29
dougwigclarkb: yes, that lbaas breakage looks like a gate hook break.  see ^^ to get neutron sorted while they debug it.23:29
*** gordc has quit IRC23:29
jheskethjeblair: so without having looked at your new changes very closely yet, it looks like you've done parts of the 'create a zuul launcher' task which I thought I was working on?23:30
jhesketh(not that it matters because I hadn't started it yet)23:30
*** ZZelle_ has quit IRC23:30
*** Qiming has joined #openstack-infra23:31
clarkbI am doing a manual job update on 01. 04 will need it too and 06 is dead in the water23:32
clarkbI am going to guess puppet timed out on the exec23:33
clarkboh no there is an error23:33
clarkbthe cache is corrupt23:34
clarkbI am going t oremove the cache23:34
*** topol has quit IRC23:34
*** gildub has joined #openstack-infra23:34
*** armax has joined #openstack-infra23:37
*** topol_ has joined #openstack-infra23:37
*** topol_ is now known as Guest5713523:37
clarkbit is doing stuff now, the broken cache is in the .cache dir under a new name23:38
clarkbif we want to figure out how it broke23:38
jeblairjhesketh: very little -- just an outline; i'm turning it over to you now :)23:38
randallburtclarkb:  cool, thanks for sorting it. I'll try a recheck23:38
clarkbzaro: zxiiro what do you think about maybe having jjb rebuild the cache on its own if it is broken? log an error/warning then move on23:38
clarkbrandallburt: well not yet23:39
clarkbrandallburt: it hasn't applied the update yet23:39
randallburtclarkb:  oh, whoops.23:39
clarkbit is just in progress23:39
jeblairjhesketh: but i had some ideas about how it should vaguely work that i thought getting into code would be easiest23:39
jheskethjeblair: fair enough23:39
jheskethI had some different ideas :p23:39
*** chlong-afk has quit IRC23:40
clarkb8118 jobs generated23:40
jheskethjeblair: I was thinking nothing would change with the launcher.. but rather we'd just have a 'zuul-worker' or similar binary that would listen to gearman and do the work from there23:40
jheskethjeblair: that way the zuul-workers can scale similar to the zuul-mergers23:40
zxiiroclarkb: I think that'd be useful. Currently we have a separate weekly job that clears the cache as we've noticed on on rare occasions the cache needs to be cleared to update jobs.23:41
jeblairjhesketh: i think you'll find that's more or less what my change does23:41
jeblairjhesketh: except that the zuul-ansible-launcher contains a merger23:41
zaroclarkb: how would you tell that it's broken?23:42
jeblairjhesketh: mergers must be integral to the launcher now so that they can push git repo state to their workers23:42
clarkbzaro: if loading it throws an exception23:42
clarkbzaro: it is invalid yaml so it explodes23:42
jheskethjeblair: ah cool, I hadn't read it properly yet, but it looked like it was replacing the gearman launcher23:42
jeblairjhesketh: ah, no, i just renamed the gearman launcher because the filenames were getting out of hand23:42
gtmanfredShrews: is it possible to rerun jenkins? cause the failures were on code that didn't change between patchset 4 and 5, with 5 only being the addition of the unittest23:42
jheskethjeblair: okay and there's just no entry point yet to run the workers?23:43
*** Qiming has quit IRC23:43
*** Daisy has joined #openstack-infra23:43
jhesketh(except in the tests)23:43
zaroclarkb: there could be other reasons for it being broken though.  so in that use case it might make sense.23:43
jeblairjhesketh: correct, it only works in unit tests because i explicitly instantiated it.  but there should be a real entry point script.  the framework is just like a merge worker23:43
clarkbzaro: I mean, its a cache, if it breaks we can safely remove it23:43
jheskethjeblair: okay cool, looks like we're on the same page :-)23:44
clarkbit makes the update run longer but thats probably better than trying to repair it by hand23:44
jeblairjhesketh: it's all yours now, i'll stay out of there and work on the job config stuff for a while23:44
jheskethjeblair: so I'll review the outstanding v3 patches today.. Did you want me to just go ahead and merge these new ones similar to how we discussed previously and iterate by committing on top?23:44
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: 10 day count is too high
jeblairjhesketh: yeah, i think so23:45
jheskethjeblair: thanks, I have some catching up to do with the remaining reviews (the last lot took some time)23:45
*** dimtruck is now known as zz_dimtruck23:46
jeblairjhesketh: i really wanted to get an end-to-end demonstration of most of the big features as a skeleton -- and that's there now.23:46
*** Swami has quit IRC23:46
zaroclarkb: true but might be real annoying if it's to aggressive trying to repair.23:47
*** akshai has quit IRC23:47
clarkbsame issue on 04, so rerunning there with cache moved aside23:47
*** weshay_xchat has quit IRC23:48
jheskethjeblair: yeah, it's cool stuff :-)23:49
clarkbzaro: well you don't agressively repair you just remove it and start over23:49
anteayadougwig: reviewed23:50
anteayadougwig: also why does neutron have so few jobs in the gate pipeline compared to the check pipeline23:50
anteayadougwig: our policy is that a voting job is in both check and gate23:50
anteayanot just check23:50
pabelangerfungi: thanks for the invite to the summit!23:51
*** armax has quit IRC23:51
jeblairfungi: fwiw, i think i registered -- i was bounced to after doing so, but didn't really get any reassuring feedback (other than the url).23:51
*** amotoki has quit IRC23:52
jeblairi did get a reassuring email, so that's good.23:52
*** amotoki has joined #openstack-infra23:52
* anteaya is glad jeblair is reassured23:52
anteayaand registered23:52
jeblairwhat about openstack proposal bot?  i'm looking for its invite but i don't see it.23:52
fungijeblair: reassuring email is reassuring23:53
anteayaha ha ha23:53
anteayaI think script 2.0 removes the bots23:53
clarkbok and to fix jenkins06 I am going to stop it, remove the slaves fro mthe config.xml and start it23:53
anteayaclarkb: ack23:53
jeblairi will have to see if i can sneak a bot registration past script 2.023:53
*** pradk has quit IRC23:53
*** amotoki has quit IRC23:53
fungijeblair: the new hotness in 267161 supports an --ignore option (and takes multiples of them) or an ignore list in a local config file23:53
anteayaclarkb: thanks for fixing the jenkinsi23:53
fungijeblair: so proposal bot should not receive an invite this time23:54
dougwiganteaya: we did that because our failure rate is about 12% per job, and we wanted neutron devs to feel that pain, but not the integrated gate.23:54
anteayadougwig: well that is not in keeping with our policy23:54
*** Daisy has quit IRC23:54
*** akshai has joined #openstack-infra23:54
anteayaI'm all in favour of thumbscrewing neutron devs if neutron drivers think that will yield the best results23:55
jeblairanteaya: we did make some exceptions with neutron as a remedial step23:55
anteayahowever I am concerned with the cargo cult behaviour that zuul/layout.yaml does see23:55
*** Daisy has joined #openstack-infra23:55
jeblairit is worth discussing whether that's still necessary23:55
anteayajeblair: ah I must have missed that decision23:55
anteayadougwig: do you feel it is still necessary?23:55
jeblairanteaya: it was a long time ago, it may predate project-config even :)23:55
dougwiganteaya: granted, and they were discussed as exceptions, not the rule.23:55
anteayaI belive that23:55
randallburtclarkb:  job is running on 03 and is "over the hump", thanks much.23:55
jeblairanteaya: so anyway, carry on -- just wanted to provide history context23:56
anteayajeblair: I appreciate context, thank you23:56
*** rlandy has quit IRC23:56
* anteaya carries on23:56
anteayadougwig: okay so an exception has an end date or end criteria, yeah?23:56
*** rbrndt__ has quit IRC23:57
fungiSamYaple: are you around? did you get a summit code from me? the script decided to die on a connection timeout and yours was the last entry in my log so it may have been trying to send that one when it happened (needs to be more robust... guess i know which script i'm revamping next)23:57
anteayadougwig: so if we are going with the argument of exception, can we discuss some end criteria?23:57
mordredHow many previous OpenStack Summits have you attended?23:58
mordredI may have been doing this for a while23:58
anteayaha ha ha23:58
*** armax has joined #openstack-infra23:58
anteayasince the dawn of time the way I understand it23:58
fungimordred: they didn't have an option for "all of them"?23:58
dougwiganteaya: that's fair. I'd like to tie it to job failure rates. I don't think end criteria have been discussed yet.23:58
jeblairfungi: no, one has to count the alphabet each time23:58
clarkband 06 is starting up again23:58
dougwigOn my phone, so typing is hard. :)23:58
anteayadougwig: oh okay yes, let's do the philosphical conversation when you have a full keyboard23:59
mordredfungi: I have missed one23:59
mordredfungi: I was not at the Bexar summit23:59
fungimordred: nowai!23:59
anteayadougwig: but note I see only two non-template jobs in neutron gate and about a dozen in neutron check and that feels odd to me23:59

Generated by 2.14.0 by Marius Gedminas - find it at!