| *** Sukhdev has joined #openstack-infra | 00:00 | |
| *** Sukhdev_ has quit IRC | 00:00 | |
| fungi | ianw: yeah, getting clarkb's input would be good since i think he was behind the design of the db usage in those tests, so knowing whether that's really intentional or whether something like the mysql root account should be used to do those additional account grants instead would be helpful | 00:00 |
|---|---|---|
| clarkb | fungi: ianw yes grant is required | 00:01 |
| jeblair | anteaya: a quick survey of mainstream us online news sites suggests that if the purpose is to make the us aware of canada (or even the existence of other countries at all), it's not working. :| | 00:01 |
| anteaya | jeblair: colour me surprised | 00:01 |
| anteaya | jeblair: but thanks for looking | 00:01 |
| *** sridhar_ram1 has joined #openstack-infra | 00:01 | |
| anteaya | jeblair: I appreciate that at the very least, you are now aware | 00:01 |
| anteaya | mordred: sorry if I was obtuse earlier | 00:02 |
| fungi | watchers/readers of mainstream news in the usa likely don't realize canada has a government | 00:02 |
| clarkb | ok 15-20 will be puppeted this next pass | 00:02 |
| clarkb | 18-20 should alrady be done | 00:02 |
| anteaya | mordred: I had eroniously believed you might have known what I was talking about | 00:02 |
| pleia2 | fungi: it's a really big, unusual state, with moose in it | 00:02 |
| anteaya | fungi: most I think aren't aware canada has an atomsphere | 00:02 |
| *** rbrndt__ has quit IRC | 00:02 | |
| clarkb | anteaya: don't worry half the country is threatening to move to canada after novemeber | 00:03 |
| mtreinish | fungi: wait, canada has a government? That's not possible :) | 00:03 |
| clarkb | anteaya: which half depends on who gets eleected | 00:03 |
| anteaya | if you look at all the us weather reports, the weather stops at the 49th parallel | 00:03 |
| * fungi imagines the stalwart mounties logging firs and riding their faithful moose steeds | 00:03 | |
| anteaya | clarkb: I know it | 00:03 |
| *** sridhar_ram has quit IRC | 00:03 | |
| anteaya | they have been contacting me | 00:03 |
| pleia2 | fungi: I imagine that too | 00:03 |
| anteaya | I'm not joking | 00:03 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add ability to grant to openstack_citest mysql user https://review.openstack.org/291493 | 00:03 |
| ianw | fungi / clarkb: ^ | 00:03 |
| anteaya | mtreinish: it is a fable | 00:03 |
| anteaya | fungi: you can't ride moose | 00:04 |
| anteaya | the legs are too thin | 00:04 |
| anteaya | can't hold a load | 00:04 |
| anteaya | they tried | 00:04 |
| fungi | somehow i'm not surprised that they tried | 00:04 |
| jeblair | fungi: not for nothin' but i think i had something to do with https://review.openstack.org/#/c/82865/ | 00:04 |
| anteaya | fungi: ha ha ha | 00:04 |
| fungi | jeblair: ah, yep, that was your handiwork! | 00:05 |
| clarkb | anteaya: you can too ride a moose. Let me introduce you to the most MURICAN of presidents, teddy roosevelt | 00:05 |
| jeblair | i at least banged on my keyboard at it a bit | 00:05 |
| fungi | jeblair: so anyway, adding grant option for openstack_citest user in jobs sound like a good idea then? | 00:05 |
| *** Sukhdev has quit IRC | 00:05 | |
| jeblair | yep! :) | 00:05 |
| *** Swami has quit IRC | 00:06 | |
| clarkb | (its apparently a fake but the picture is famous) | 00:06 |
| fungi | jeblair: thanks!!! | 00:06 |
| *** Sukhdev has joined #openstack-infra | 00:06 | |
| anteaya | clarkb: tell me more | 00:06 |
| fungi | clarkb: the same could be said for ted himself | 00:06 |
| jeblair | also, i'm happy to continue to defer to clarkb on any future responsibility that comes from that patch | 00:06 |
| clarkb | anteaya: http://paleofuture.gizmodo.com/that-famous-photo-of-teddy-roosevelt-riding-a-moose-is-1355349896 | 00:06 |
| anteaya | ha ha ha | 00:06 |
| anteaya | in the water | 00:07 |
| jeblair | (but in all seriousness, it was kind of a team idea we all batted around for a few months, as i recall) | 00:07 |
| anteaya | I'm impressed so many americans can pronounce the word Canada | 00:07 |
| anteaya | warms my heart it does | 00:07 |
| fungi | jeblair: in fairness, i was only going to blame clarkb for the bad parts of it anyway | 00:07 |
| clarkb | I just know about the grant stuff because it is one of the things that got tricky with the puppet-ymsql upgrade | 00:08 |
| *** Qiming has quit IRC | 00:08 | |
| jroll | anteaya: it's pronounced kuh-nay-dee-ah, right? :P | 00:08 |
| anteaya | jroll: you've studied | 00:08 |
| anteaya | well done | 00:08 |
| *** Sukhdev has quit IRC | 00:08 | |
| jeblair | anteaya: sometimes in california, i get really confused about how to pronounce canada: https://en.wikipedia.org/wiki/La_Ca%C3%B1ada_Flintridge,_California | 00:08 |
| anteaya | ha ha ha | 00:08 |
| anteaya | jeblair: nice | 00:08 |
| fungi | clarkb: now i want to make a version of that photo with teddy riding a bearodactyl | 00:08 |
| jroll | jeblair: there's also http://www.canadacollege.edu/ | 00:09 |
| anteaya | the obama's pose for photos like art models | 00:09 |
| jeblair | jroll: they even have their own canadian cody | 00:10 |
| pleia2 | love the vintage photoshopping | 00:10 |
| anteaya | fungi: what is a bearodactyl? | 00:10 |
| jroll | jeblair: heh, weird | 00:10 |
| *** kzaitsev_mb has quit IRC | 00:12 | |
| jeblair | mordred: congrats on being a ptl! where DO you find the time! | 00:13 |
| ianw | fungi: ok, you can take back that host too; i think i have a legitimate bug in my image-size tracking code | 00:13 |
| anteaya | jeblair mordred wat? | 00:14 |
| clarkb | 10-14 will be this round | 00:14 |
| jeblair | anteaya: http://lists.openstack.org/pipermail/openstack-dev/2016-March/089029.html | 00:14 |
| *** akshai has quit IRC | 00:14 | |
| fungi | anteaya: a farcical hybrid animal invented for an online comic strip called "the oatmeal" | 00:15 |
| jeblair | anteaya: i mean, it's not official yet, but it sure looks like he has the race locked up. | 00:15 |
| *** tphummel has quit IRC | 00:15 | |
| mordred | jeblair: do I need to announce my candidacy any more explicity do you think? | 00:15 |
| *** doug-fish has joined #openstack-infra | 00:16 | |
| anteaya | ha ha ha | 00:16 |
| anteaya | mordred: I don't think anyone can challenge you | 00:16 |
| anteaya | fungi: ah, good comic, though it has been a while since I've read | 00:16 |
| fungi | nor can anyone else vote for him! | 00:16 |
| mordred | anteaya: I suppose I might have to start producing some git repos | 00:16 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 00:17 |
| anteaya | fungi: ha ha ha | 00:17 |
| *** doug-fish has quit IRC | 00:17 | |
| anteaya | mordred: I guess? | 00:17 |
| jeblair | mordred: i guess you'd have to push something to http://git.openstack.org/cgit/openstack/election/ soon to make it official | 00:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: CentOS AFS build : use homedir https://review.openstack.org/291498 | 00:17 |
| *** doug-fish has joined #openstack-infra | 00:17 | |
| ianw | fungi / jeblair : mind having a quick poke at ^ --> failing logs http://logs.openstack.org/03/291403/2/experimental/periodic-package-afs-centos7/b2a96ab/console.html | 00:18 |
| jeblair | ianw: already on it ;) | 00:18 |
| fungi | mordred: yes, after tristanC's question i was doing a quick peek through the electoral roll equivalent files created by my most recent summit invite batch run, and was surprised when the shortest list was just you. you are the lowest common denominator contributor across our projects, i think | 00:18 |
| anteaya | ha ha ha | 00:18 |
| jeblair | fungi: nice turn of phrase | 00:18 |
| anteaya | that needs a t-shirt | 00:18 |
| mordred | fungi: if I haven't contributed to your project, you're not really OpenStack | 00:18 |
| fungi | EXACTLY | 00:19 |
| anteaya | the new criteria | 00:19 |
| craige | heh | 00:19 |
| fungi | that goes on the big tent application checklist | 00:19 |
| anteaya | ha ha ha | 00:19 |
| fungi | wait, no commits from mordred? are you _sure_ this is a real project? | 00:19 |
| anteaya | easy enough benchmark to evaluate | 00:20 |
| jeblair | ianw: actually jenkins runs scripts with a CWD of $WORKSPACE which is something like ~jenkins/workspace/<JOBNAME> | 00:20 |
| jeblair | ianw: having said that, i think the fix is correct :) | 00:20 |
| fungi | gah, trying to load that change just crashed gertty | 00:21 |
| fungi | AttributeError: 'NoneType' object has no attribute 'topic' | 00:21 |
| jeblair | nuh uh | 00:21 |
| *** austin81 has quit IRC | 00:21 | |
| jeblair | fungi: have a full traceback? | 00:22 |
| *** doug-fish has quit IRC | 00:22 | |
| *** doug-fis_ has joined #openstack-infra | 00:22 | |
| fungi | jeblair: i do, collecting it now | 00:22 |
| fungi | jeblair: http://paste.openstack.org/show/490083 | 00:23 |
| *** openstackgerrit_ has quit IRC | 00:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Make all configuration in-repo configuration https://review.openstack.org/268386 | 00:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add job tags https://review.openstack.org/277987 | 00:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix race in tests https://review.openstack.org/290903 | 00:24 |
| *** yamamoto_ has joined #openstack-infra | 00:24 | |
| *** openstackgerrit_ has joined #openstack-infra | 00:24 | |
| jeblair | fungi: how did you open it? | 00:24 |
| *** bapalm has quit IRC | 00:24 | |
| *** doug-fish has joined #openstack-infra | 00:24 | |
| fungi | ctrl-o and then pasted the change number and hit enter | 00:24 |
| fungi | seeing if i can recreate, though likely it was state-dependent | 00:25 |
| jeblair | fungi: any chance you had full debug logging going on around then? | 00:25 |
| anteaya | mike meyers took too long in the entrance hall | 00:25 |
| fungi | yeah, it loaded fine after i restarted gertty. i also have the log but not on full debug no | 00:25 |
| anteaya | they had to hurry him along | 00:25 |
| *** doug-fis_ has quit IRC | 00:26 | |
| clarkb | 10-20 are all done. 01-09 are going to be done on the next pass | 00:26 |
| anteaya | clarkb: yay | 00:27 |
| anteaya | are 10-20 coming up again? | 00:27 |
| *** doug-fis_ has joined #openstack-infra | 00:27 | |
| * anteaya assumes we are talking logstash_workers | 00:28 | |
| jeblair | fungi: what change were you viewing when you opened it? | 00:28 |
| *** yamamoto_ has quit IRC | 00:28 | |
| *** doug-fish has quit IRC | 00:29 | |
| *** sc68cal has quit IRC | 00:29 | |
| *** bpokorny_ has quit IRC | 00:29 | |
| clarkb | anteaya: yes they are all fully operational | 00:29 |
| jeblair | fungi: do you have the activating the link in your terminal thing working? (any chance you could have remotely opened the change while copying the change number?) | 00:29 |
| clarkb | anteaya: on logstash 2.0 | 00:29 |
| *** doug-fish has joined #openstack-infra | 00:29 | |
| *** sdake has joined #openstack-infra | 00:29 | |
| *** bpokorny has joined #openstack-infra | 00:29 | |
| fungi | jeblair: i was loading ian's 291498 change, and no i don't have anything set up to trigger remote loading in gertty (yet at least) | 00:30 |
| jeblair | fungi: the error appears to be actually refreshing the change you were on, not the one you were going to, which is really weird. | 00:31 |
| anteaya | clarkb: yay! | 00:31 |
| *** gildub has joined #openstack-infra | 00:31 | |
| *** doug-fis_ has quit IRC | 00:32 | |
| fungi | clarkb: jeblair: any chance one of you want to officially weigh in on the grant option addition (291493) so i can get started reupdating jjb configs everywhere while things are quiet? | 00:32 |
| *** armax has quit IRC | 00:33 | |
| jeblair | done | 00:33 |
| fungi | it's going to impact lots of jobs, but hopefully this is the last tweak to these macros for a long while | 00:33 |
| fungi | thanks! | 00:34 |
| *** doug-fish has quit IRC | 00:34 | |
| fungi | once that lands i'll get jjb updates going again | 00:34 |
| anteaya | wooo | 00:35 |
| *** armax has joined #openstack-infra | 00:35 | |
| *** armax has quit IRC | 00:35 | |
| fungi | this is kinda a neat/convenient tool https://pypi.python.org/pypi/pythonpy | 00:36 |
| fungi | the auto-importing it does makes it a bit less of a hassle than straight python -c too | 00:36 |
| *** ashleighfarnham has quit IRC | 00:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Add ability to grant to openstack_citest mysql user https://review.openstack.org/291493 | 00:39 |
| fungi | and now i'll get ta' updatin' | 00:40 |
| *** bapalm has joined #openstack-infra | 00:41 | |
| fungi | clarkb: how were you suggesting to do multi-threaded updates? | 00:42 |
| fungi | i'll start looking at the jjb docs | 00:42 |
| fungi | just realized you had wanted me to benchmark that next | 00:43 |
| *** xarses has quit IRC | 00:43 | |
| clarkb | fungi: there is a flag to n way it | 00:44 |
| *** ddieterly has joined #openstack-infra | 00:44 | |
| fungi | --workers 0 | 00:45 |
| jeblair | increase that very very slowly :) | 00:45 |
| fungi | apparently that does as many workers as you have processors | 00:45 |
| fungi | --workers 2 may be a good place to start ;_ | 00:45 |
| jeblair | yes | 00:45 |
| *** claudiub has quit IRC | 00:46 | |
| *** sarob has quit IRC | 00:48 | |
| *** openstackgerrit_ has quit IRC | 00:48 | |
| *** EricGonczer_ has quit IRC | 00:49 | |
| *** openstackgerrit_ has joined #openstack-infra | 00:49 | |
| *** nwkarsten has joined #openstack-infra | 00:49 | |
| anteaya | ;_ looks sinister | 00:49 |
| clarkb | we be interseting to see if jenkins can handle N way updates to its configs without serializing | 00:50 |
| clarkb | s/we/will/ | 00:50 |
| *** pvaneck has quit IRC | 00:50 | |
| fungi | yeah, i think i remember when jjb briefly defaulted to --workers 0 by default. if memory serves that was not a great day | 00:52 |
| *** chlong has joined #openstack-infra | 00:52 | |
| fungi | anyway, we have 8 jenkins masters currently running 2-way parallel jjb updates | 00:52 |
| anteaya | yay | 00:52 |
| fungi | load average on zuul is hovering around 2 | 00:53 |
| clarkb | zuul is about ot merge a bunch of stuff too | 00:53 |
| fungi | actually its load is falling again | 00:53 |
| *** takashi has joined #openstack-infra | 00:53 | |
| fungi | yeah, it's more likely from that | 00:53 |
| jeblair | we should keep an eye on jenkins itself and nodepool | 00:54 |
| jeblair | with the gearman change, i think we're more likely to see trouble from api timeouts than gearman traffic | 00:55 |
| fungi | yeah, i'm expecting this to make jenkins01 and 02 unhappy since they already have a devil of a time avoiding api timeouts/proxy errors | 00:55 |
| anteaya | why are 01 and 02 so different from the others? | 00:57 |
| *** sridhar_ram1 has quit IRC | 00:57 | |
| *** tsg has quit IRC | 00:57 | |
| fungi | built earlier, i think on older/slower instance flavors | 00:57 |
| *** ajmiller has quit IRC | 00:57 | |
| anteaya | can they be upgraded? | 00:57 |
| fungi | yeah, they can be rebuilt, it's just nobody's gotten around to it because there are manual steps involved in copying configuration over | 00:58 |
| anteaya | okay | 00:58 |
| anteaya | so something for a slow day | 00:59 |
| anteaya | when we are all staring at each other and twiddling our thumbs | 00:59 |
| fungi | right, luckily we're thoroughly parallel wit these servers, so could take one offline for a while to rebuild it with no real negative impact | 00:59 |
| anteaya | or could we spin up two more and set them up and just drop these? | 01:00 |
| anteaya | nothing needs to be migrated, yes? | 01:00 |
| fungi | right, we could build 08 and 09 and phase these out. a fairly similar amount of work | 01:01 |
| *** ddieterly has quit IRC | 01:01 | |
| anteaya | yes, just nothing to be saved or transfered | 01:01 |
| fungi | we're no longer so tight on quota that it's a problem | 01:01 |
| *** sridhar_ram has joined #openstack-infra | 01:01 | |
| anteaya | think it needs a spec? | 01:02 |
| anteaya | I can put up a spec if it would be useful | 01:02 |
| fungi | no, just needs a root admin with available time to get it done | 01:02 |
| anteaya | a spec would be easier to come by | 01:02 |
| *** Qiming has joined #openstack-infra | 01:07 | |
| *** angdraug has quit IRC | 01:12 | |
| *** esker has quit IRC | 01:21 | |
| *** esker has joined #openstack-infra | 01:22 | |
| anteaya | jesusaur: merge conflict on your patch: https://review.openstack.org/#/c/190345/ | 01:23 |
| *** SumitNaiksatam has quit IRC | 01:23 | |
| fungi | so far the only ones i've found which weren't up to date by the time i started running on them were jenkins.o.o (27m2s to complete) and jenkins01 (which is still running) | 01:27 |
| jesusaur | anteaya: thanks for pointing that out | 01:27 |
| fungi | all the rest were "0 jobs updated" and completed in around 6-8 minutes | 01:27 |
| clarkb | fungi: I think somethibg is wrong with caching on 02 | 01:28 |
| clarkb | iirc the cache tells jjb whether or not its locsl jobs match the jenkins? | 01:28 |
| *** zhurong_ has joined #openstack-infra | 01:29 | |
| anteaya | jesusaur: welcome | 01:29 |
| fungi | clarkb: i can look at the cache and see if i spot anything odd, i guess | 01:29 |
| clarkb | fungi: maybe it isnt readable or being written? | 01:30 |
| fungi | it's empty at the moment | 01:30 |
| fungi | maybe it isn't being written | 01:30 |
| *** dmsimard|pto has quit IRC | 01:31 | |
| *** resker has joined #openstack-infra | 01:35 | |
| *** FallenPegasus has quit IRC | 01:35 | |
| *** sridhar_ram has quit IRC | 01:35 | |
| *** esker has quit IRC | 01:37 | |
| *** dimtruck is now known as zz_dimtruck | 01:39 | |
| *** tphummel has joined #openstack-infra | 01:40 | |
| *** sdake has quit IRC | 01:40 | |
| *** Sukhdev has joined #openstack-infra | 01:41 | |
| *** _ari_ has joined #openstack-infra | 01:41 | |
| jeblair | jhesketh: v3 is almost caught back up to master; see the parents of https://review.openstack.org/268386 | 01:42 |
| *** nikhil_k has quit IRC | 01:42 | |
| *** hamzy has joined #openstack-infra | 01:45 | |
| *** nwkarsten has quit IRC | 01:45 | |
| *** tzn_ has quit IRC | 01:45 | |
| jhesketh | jeblair: wouldn't we be better off doing another merge of master into zuulv3 rather than redoing tags? | 01:46 |
| *** nwkarsten has joined #openstack-infra | 01:46 | |
| jhesketh | jeblair: I actually had started that yesterday, but didn't finish resolving the merge conflicts | 01:46 |
| * jhesketh can do so now | 01:46 | |
| jeblair | jhesketh: if you want... it's mostly unimplemented because it affects parts that are radically different... it's basically just adding the test debt so we know to fix it later | 01:47 |
| *** Sukhdev has quit IRC | 01:47 | |
| jhesketh | jeblair: right, but if we redo it won't that make merging master into v3 (or even v3 into master if we choose to do it that way instead of just switching branches over) more difficult? | 01:48 |
| jeblair | jhesketh: yeah, it might make the git history nicer | 01:48 |
| jhesketh | jeblair: do you think we should get https://review.openstack.org/#/c/276395/4 into master first (and then merged into v3) | 01:49 |
| jeblair | jhesketh: probably so. i'll try to review it soon. | 01:50 |
| *** nwkarsten has quit IRC | 01:50 | |
| * craige cleared another Phabricator hurdle this morning. Feeling the momentum :-D | 01:50 | |
| jeblair | jhesketh: oh, you know what that change makes sense to me right now. +3. :) | 01:51 |
| *** xarses has joined #openstack-infra | 01:52 | |
| jhesketh | jeblair: cheers | 01:52 |
| jhesketh | jeblair: the follow on change (https://review.openstack.org/#/c/281166/2) needs some discussion if you have a moment too | 01:52 |
| jeblair | jhesketh: so feel free to go ahead and bash that in and do whatever you need to do to catch v3 up to master; then if you have a moment to look at 268386 that would be cool. | 01:52 |
| jhesketh | jeblair: I know you said you don't want to change the way connections do a reload, but if not, should we look at copying the cache across connections | 01:53 |
| jhesketh | jeblair: will do | 01:53 |
| jeblair | jhesketh: yeah, i want to think about that more in the v3 context rather than v2. i don't have everything i need in my head yet though. | 01:54 |
| jhesketh | jeblair: I guess it only matters for the v2 case if it's affecting the memory leak (that I've failed to track down) | 01:54 |
| jhesketh | it should (at least in theory, I haven't checked) decrease the load, but maybe we don't care about that until v3 | 01:55 |
| *** nwkarsten has joined #openstack-infra | 01:55 | |
| jeblair | jhesketh: yeah, i just desperately want v2 to be stable and stop moving at all so we can work on v3 | 01:55 |
| jeblair | we can deal with the load | 01:55 |
| *** tphummel has quit IRC | 01:55 | |
| jhesketh | agreed | 01:56 |
| openstackgerrit | Merged openstack-infra/zuul: Cache is held and managed by connections https://review.openstack.org/276395 | 01:57 |
| *** tphummel has joined #openstack-infra | 01:57 | |
| *** bpokorny_ has joined #openstack-infra | 02:00 | |
| openstackgerrit | Merged openstack-infra/project-config: CentOS AFS build : use homedir https://review.openstack.org/291498 | 02:02 |
| *** EricGonczer_ has joined #openstack-infra | 02:02 | |
| jeblair | jhesketh: each time we reload, we create a new driver object and register its use with a connection, right? | 02:02 |
| jhesketh | jeblair: correct.. we also create a new connection object | 02:02 |
| *** mtanino has quit IRC | 02:02 | |
| jeblair | jhesketh: do we? | 02:02 |
| jhesketh | jeblair: that's what that follow up change/discussion is about | 02:03 |
| *** bpokorny has quit IRC | 02:04 | |
| *** dingyichen has quit IRC | 02:04 | |
| jeblair | jhesketh: ok good. that's the safer thing to do for now. | 02:05 |
| *** bpokorny_ has quit IRC | 02:05 | |
| *** dims has quit IRC | 02:08 | |
| *** dalgaaf has quit IRC | 02:08 | |
| *** gyee has quit IRC | 02:10 | |
| *** bapalm has quit IRC | 02:11 | |
| *** nwkarsten has quit IRC | 02:13 | |
| *** nwkarsten has joined #openstack-infra | 02:14 | |
| *** asettle has quit IRC | 02:15 | |
| *** esikachev has joined #openstack-infra | 02:15 | |
| *** asettle has joined #openstack-infra | 02:15 | |
| *** tzn has joined #openstack-infra | 02:16 | |
| *** bapalm has joined #openstack-infra | 02:16 | |
| *** Jeffrey4l has joined #openstack-infra | 02:16 | |
| *** dims has joined #openstack-infra | 02:17 | |
| *** gildub has quit IRC | 02:17 | |
| *** nwkarsten has quit IRC | 02:18 | |
| *** esikachev has quit IRC | 02:19 | |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate grenade plugins list https://review.openstack.org/291514 | 02:20 |
| *** EricGonc_ has joined #openstack-infra | 02:24 | |
| *** yamamoto_ has joined #openstack-infra | 02:25 | |
| *** EricGonczer_ has quit IRC | 02:27 | |
| *** moravec has joined #openstack-infra | 02:27 | |
| *** rguillebert has quit IRC | 02:27 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/291515 | 02:28 |
| jhesketh | jeblair: ^ | 02:28 |
| jhesketh | jeblair: if you'd like I can rebase your changes (268386 and above) onto that merge change (probably dropping the re-implementation of tags) | 02:30 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Fail on bad options to fake-image-create https://review.openstack.org/291516 | 02:30 |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate tempest plugins list https://review.openstack.org/291517 | 02:31 |
| *** fawadkhaliq has quit IRC | 02:32 | |
| clarkb | #success Infra is running logstash 2.0 now | 02:32 |
| openstackstatus | clarkb: Added success to Success page | 02:32 |
| *** hichihara has joined #openstack-infra | 02:32 | |
| *** Guest38484 is now known as mfisch | 02:33 | |
| *** mfisch is now known as Guest13749 | 02:34 | |
| *** nikhil has joined #openstack-infra | 02:34 | |
| *** kevinshort has quit IRC | 02:34 | |
| fungi | w00t | 02:36 |
| fungi | jenkins01 is still jjb'ing | 02:36 |
| *** woodster_ has quit IRC | 02:37 | |
| *** kevinshort has joined #openstack-infra | 02:37 | |
| *** EricGonc_ has quit IRC | 02:38 | |
| *** kevinsho_ has joined #openstack-infra | 02:40 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: [wip] Revert "Add CentOS AFS build job to project-config experimental" https://review.openstack.org/291403 | 02:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: CentOS AFS build : move output to $WORKSPACE https://review.openstack.org/291521 | 02:42 |
| *** nikhil has quit IRC | 02:43 | |
| clarkb | looks like kibana4 wants to create an index and bails out early if it cna't | 02:44 |
| *** tqtran-afk has quit IRC | 02:45 | |
| *** nwkarsten has joined #openstack-infra | 02:45 | |
| clarkb | I amgoing to manually create that index and see if we can trick it into running without being able to write to it | 02:47 |
| clarkb | (this is a first step in deploying kibana4) | 02:47 |
| *** yamamoto_ has quit IRC | 02:48 | |
| clarkb | hrm that didn't do it... | 02:49 |
| *** nwkarsten has quit IRC | 02:50 | |
| *** gildub has joined #openstack-infra | 02:50 | |
| *** sdake has joined #openstack-infra | 02:51 | |
| clarkb | looks like kibana 4 lets anyone configure it via the web ui? will have to figure out how to avoid that | 02:52 |
| clarkb | "caution you can break things here" probably not great to expose that on the internet | 02:53 |
| *** Sukhdev has joined #openstack-infra | 02:54 | |
| *** sdake_ has joined #openstack-infra | 02:55 | |
| *** ajmiller has joined #openstack-infra | 02:56 | |
| *** sdake has quit IRC | 02:56 | |
| clarkb | this is very new and different | 02:58 |
| mtreinish | clarkb: where's your sense of adventure :) | 02:58 |
| *** dims has quit IRC | 02:58 | |
| *** ayoung has joined #openstack-infra | 02:58 | |
| *** sputnik13 has quit IRC | 03:00 | |
| *** ajmiller_ has joined #openstack-infra | 03:00 | |
| clarkb | I mean it queries things | 03:01 |
| clarkb | it just has so many other knobs and bells and whistles and scary config that appear exposed ot everyone that I need to figure out | 03:01 |
| *** kevinsho_ has quit IRC | 03:03 | |
| *** sdake_ has quit IRC | 03:03 | |
| *** ajmiller has quit IRC | 03:03 | |
| *** kevinsho_ has joined #openstack-infra | 03:06 | |
| *** kevinshort has quit IRC | 03:09 | |
| *** Sukhdev has quit IRC | 03:09 | |
| *** kevinshort has joined #openstack-infra | 03:09 | |
| *** kevinsho_ has quit IRC | 03:11 | |
| *** kevinsho_ has joined #openstack-infra | 03:11 | |
| fungi | grr, 01 finally hit a proxy error | 03:11 |
| *** kevinsho_ has quit IRC | 03:12 | |
| *** kevinsho_ has joined #openstack-infra | 03:12 | |
| *** nwkarsten has joined #openstack-infra | 03:13 | |
| *** dims has joined #openstack-infra | 03:13 | |
| *** kevinshort has quit IRC | 03:14 | |
| *** kevinshort has joined #openstack-infra | 03:14 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add some tracking for image sizes https://review.openstack.org/289678 | 03:15 |
| ianw | ^ that should make for a fun graph at some point | 03:15 |
| *** nwkarsten has quit IRC | 03:18 | |
| *** dims has quit IRC | 03:18 | |
| *** thorst__ has quit IRC | 03:20 | |
| *** thorst has joined #openstack-infra | 03:20 | |
| *** zul_ has joined #openstack-infra | 03:25 | |
| *** yamahata_ has joined #openstack-infra | 03:25 | |
| *** mknawabi_ has joined #openstack-infra | 03:25 | |
| *** rhallisey has quit IRC | 03:27 | |
| *** hieulq has joined #openstack-infra | 03:28 | |
| *** thorst has quit IRC | 03:29 | |
| *** mtreinish_ has joined #openstack-infra | 03:29 | |
| *** notmyname_ has joined #openstack-infra | 03:30 | |
| *** krotscheck has quit IRC | 03:30 | |
| *** yamahata has quit IRC | 03:30 | |
| *** ildikov has quit IRC | 03:30 | |
| *** zul has quit IRC | 03:30 | |
| *** notmyname has quit IRC | 03:30 | |
| *** mtreinish has quit IRC | 03:30 | |
| *** mknawabi has quit IRC | 03:30 | |
| *** ildikov has joined #openstack-infra | 03:30 | |
| *** mtreinish_ is now known as mtreinish | 03:30 | |
| *** notmyname_ is now known as notmyname | 03:30 | |
| *** krotscheck has joined #openstack-infra | 03:31 | |
| *** zz_dimtruck is now known as dimtruck | 03:31 | |
| *** hdd has joined #openstack-infra | 03:33 | |
| *** aeng has quit IRC | 03:33 | |
| *** aeng has joined #openstack-infra | 03:34 | |
| *** yamamoto has joined #openstack-infra | 03:36 | |
| *** yamahata_ has quit IRC | 03:36 | |
| *** aeng has quit IRC | 03:36 | |
| *** apoorvad has quit IRC | 03:37 | |
| *** tqtran-afk has joined #openstack-infra | 03:42 | |
| *** sdake has joined #openstack-infra | 03:42 | |
| fungi | initial pass on jenkins01 was 174m17s but i've resumed it now to deal with the api errors it hit that time around | 03:45 |
| *** links has joined #openstack-infra | 03:45 | |
| *** tqtran-afk has quit IRC | 03:46 | |
| *** dimtruck is now known as zz_dimtruck | 03:52 | |
| *** nikhil has joined #openstack-infra | 03:54 | |
| *** nikhil_k has joined #openstack-infra | 03:54 | |
| *** ildikov has quit IRC | 03:55 | |
| *** ildikov has joined #openstack-infra | 03:55 | |
| *** Nakato_ is now known as Nakato | 03:56 | |
| bswartz | shouldn't pushing a tag cause the post pipeline to run again? | 03:56 |
| bswartz | or am I missing something? | 03:56 |
| clarkb | release pipeline | 03:57 |
| bswartz | hrm | 03:57 |
| * bswartz might be an idiot | 03:57 | |
| bswartz | is there an example of a job that publishes tagged tarballs? | 04:01 |
| clarkb | to pypi? | 04:01 |
| bswartz | to tarballs.o.o | 04:01 |
| *** jpr has quit IRC | 04:03 | |
| *** baoli has joined #openstack-infra | 04:04 | |
| bswartz | actually I think I'm looking for the {name}-tarball job | 04:04 |
| bswartz | which nearly every project uses | 04:04 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml line 493 | 04:04 |
| clarkb | ya | 04:05 |
| *** jpr has joined #openstack-infra | 04:05 | |
| clarkb | but it is only run if you have the template on line 493 or explicitly run it | 04:05 |
| *** dingyichen has joined #openstack-infra | 04:07 | |
| bswartz | okay so what's the difference between "release" and "release-post"? | 04:07 |
| bswartz | wouldn't "release-post" be a more appropriate place to upload to tarballs.o.o? | 04:08 |
| bswartz | oh I see that pipe only has a single purpose | 04:08 |
| clarkb | I would have | 04:08 |
| clarkb | er | 04:09 |
| clarkb | would have tonread the release-post pipeline tonfigure out what it is for | 04:09 |
| *** tphummel has quit IRC | 04:09 | |
| bswartz | description: This pipeline runs release-process-critical jobs that operate after specific changes are merged. | 04:09 |
| bswartz | it's referred to in exactly 1 place | 04:09 |
| bswartz | the openstack/release project | 04:09 |
| *** amrith is now known as _amrith_ | 04:10 | |
| fungi | yeah, it's basically a separate "post" pipeline used for generating release notes so that we can up the priority on it, since the normal post pipeline gets lowest priority | 04:11 |
| *** sdake has quit IRC | 04:12 | |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Run publish job on release of manila-image-elements https://review.openstack.org/291544 | 04:15 |
| *** esikachev has joined #openstack-infra | 04:16 | |
| *** fedexo has joined #openstack-infra | 04:17 | |
| *** esikachev has quit IRC | 04:20 | |
| docaedo | bswartz: do you still think you'll want to link to an image in app catalog? (which I'm highly in favor of) - just want to make sure I'm staying on top of it in case you or anyone else on the manila team needs help | 04:20 |
| bswartz | docaedo: absolutely | 04:20 |
| bswartz | in fact that's what I'm working on atm | 04:20 |
| bswartz | on Wednesday I set out to push a change to app-catalog but I realized we didn't have a tagged build yet that we could use | 04:21 |
| bswartz | that let me to realize we had ACL issues and other project-config issues to solve first | 04:21 |
| docaedo | bswartz: sweet! I'm going to be stepping away for reading to the kids, but will be around either later tonight, or else early(ish) tomorrow morning, so I'm up to help however I can | 04:21 |
| bswartz | so I'm working through the issue so I can get a tagged build first | 04:22 |
| bswartz | once I have a tagged build you'll see my change go in | 04:22 |
| docaedo | awesome, sounds like that build will be a tomorrow thing then :) | 04:22 |
| bswartz | yeah don't worry about it tonight | 04:22 |
| bswartz | the wheels of zuul turn slowly | 04:23 |
| *** sdake has joined #openstack-infra | 04:23 | |
| docaedo | indeed .. also when you make your app catalog PR, be sure the first line of the commit message does not end in a period, and the body of the message should be in haiku format, though we'll accept a limerick if necessary | 04:23 |
| bswartz | lol | 04:24 |
| docaedo | as long as we can all agree that your poem conjures the majesty of OpenStack, we'll be able to merge quickly I'm sure. | 04:24 |
| fungi | grr, jenkins01 has hit another api timeout on the second jjb update pass as well | 04:26 |
| *** thorst has joined #openstack-infra | 04:26 | |
| *** sdake_ has joined #openstack-infra | 04:27 | |
| *** sdake has quit IRC | 04:29 | |
| *** baoli has quit IRC | 04:29 | |
| *** [1]Thelo has joined #openstack-infra | 04:30 | |
| *** hieulq has quit IRC | 04:32 | |
| *** jaosorior has joined #openstack-infra | 04:32 | |
| *** ajmiller_ has quit IRC | 04:33 | |
| *** Thelo has quit IRC | 04:33 | |
| *** [1]Thelo is now known as Thelo | 04:33 | |
| *** thorst has quit IRC | 04:34 | |
| *** treaki_ has joined #openstack-infra | 04:35 | |
| *** treaki__ has quit IRC | 04:39 | |
| openstackgerrit | Craige McWhirter proposed openstack-infra/system-config: Added RemoteUser Provider & OpenID configs https://review.openstack.org/254066 | 04:40 |
| *** sdake_ has quit IRC | 04:48 | |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Refactor manila check/gate queue https://review.openstack.org/291546 | 04:50 |
| *** zz_dimtruck is now known as dimtruck | 04:57 | |
| *** tqtran-afk has joined #openstack-infra | 04:58 | |
| *** jamespage has quit IRC | 05:00 | |
| *** gnuoy has quit IRC | 05:00 | |
| *** gongysh has quit IRC | 05:00 | |
| *** gnuoy has joined #openstack-infra | 05:00 | |
| *** jamespage has joined #openstack-infra | 05:00 | |
| *** tqtran-afk has quit IRC | 05:02 | |
| *** rossella_s has quit IRC | 05:03 | |
| *** rossella_s has joined #openstack-infra | 05:03 | |
| fungi | second pass on jenkins01: 84m27s | 05:09 |
| fungi | resuming again | 05:09 |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Refactor manila check/gate queue https://review.openstack.org/291546 | 05:17 |
| *** kdas__ is now known as kushal | 05:17 | |
| *** kushal has quit IRC | 05:18 | |
| *** kushal has joined #openstack-infra | 05:18 | |
| *** armax has joined #openstack-infra | 05:20 | |
| *** sridhar_ram has joined #openstack-infra | 05:23 | |
| *** _nadya_ has joined #openstack-infra | 05:23 | |
| *** nikhil has quit IRC | 05:23 | |
| *** fedexo has quit IRC | 05:27 | |
| *** _nadya_ has quit IRC | 05:30 | |
| *** thorst has joined #openstack-infra | 05:32 | |
| *** thorst has quit IRC | 05:39 | |
| *** heyongli has joined #openstack-infra | 05:39 | |
| *** heyongli has quit IRC | 05:41 | |
| *** heyongli has joined #openstack-infra | 05:41 | |
| *** sarob has joined #openstack-infra | 05:45 | |
| *** heyongli has quit IRC | 05:46 | |
| *** heyongli has joined #openstack-infra | 05:46 | |
| *** Jeffrey4l has quit IRC | 05:47 | |
| *** cburgess has quit IRC | 05:49 | |
| *** kingia_ has quit IRC | 05:50 | |
| *** rlandy has quit IRC | 05:50 | |
| *** kingia has joined #openstack-infra | 05:50 | |
| *** sarob has quit IRC | 05:50 | |
| *** rlandy has joined #openstack-infra | 05:50 | |
| *** amotoki has joined #openstack-infra | 05:50 | |
| *** amotoki has quit IRC | 05:50 | |
| *** melwitt has quit IRC | 05:51 | |
| *** Slower has quit IRC | 05:52 | |
| *** daemontool__ has quit IRC | 05:52 | |
| *** cburgess has joined #openstack-infra | 05:52 | |
| *** Slower has joined #openstack-infra | 05:52 | |
| *** armax has quit IRC | 05:53 | |
| *** zhurong_ has quit IRC | 05:53 | |
| *** tzn has quit IRC | 05:55 | |
| *** armax has joined #openstack-infra | 05:55 | |
| *** tzn has joined #openstack-infra | 05:55 | |
| *** dimtruck is now known as zz_dimtruck | 05:55 | |
| *** heyongli has quit IRC | 05:56 | |
| *** zhurong_ has joined #openstack-infra | 05:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: puppet: fix cookiecutter job template https://review.openstack.org/290906 | 05:56 |
| *** jaosorior has quit IRC | 05:56 | |
| *** ayoung has quit IRC | 05:56 | |
| *** Douhet has quit IRC | 05:56 | |
| *** bryan_att has quit IRC | 05:56 | |
| *** fkautz has quit IRC | 05:56 | |
| *** Ng has quit IRC | 05:56 | |
| *** zxiiro has quit IRC | 05:56 | |
| *** fdegir has quit IRC | 05:56 | |
| *** simonmcc has quit IRC | 05:56 | |
| *** ashtokolov has quit IRC | 05:56 | |
| *** dansmith has quit IRC | 05:56 | |
| *** yonglihe has quit IRC | 05:56 | |
| *** evgenyl has quit IRC | 05:56 | |
| *** mwhahaha has quit IRC | 05:56 | |
| *** Volundr has quit IRC | 05:56 | |
| *** mescanef has quit IRC | 05:56 | |
| *** cfarquhar has quit IRC | 05:56 | |
| *** heyongli has joined #openstack-infra | 05:57 | |
| *** melwitt has joined #openstack-infra | 05:57 | |
| *** yamahata_ has joined #openstack-infra | 05:59 | |
| *** mariojv has quit IRC | 05:59 | |
| *** krotscheck has quit IRC | 06:01 | |
| *** mariojv has joined #openstack-infra | 06:02 | |
| *** ayoung has joined #openstack-infra | 06:02 | |
| *** Douhet has joined #openstack-infra | 06:02 | |
| *** fkautz has joined #openstack-infra | 06:02 | |
| *** bryan_att has joined #openstack-infra | 06:02 | |
| *** Ng has joined #openstack-infra | 06:02 | |
| *** zxiiro has joined #openstack-infra | 06:02 | |
| *** fdegir has joined #openstack-infra | 06:02 | |
| *** simonmcc has joined #openstack-infra | 06:02 | |
| *** ashtokolov has joined #openstack-infra | 06:02 | |
| *** dansmith has joined #openstack-infra | 06:02 | |
| *** yonglihe has joined #openstack-infra | 06:02 | |
| *** evgenyl has joined #openstack-infra | 06:02 | |
| *** mwhahaha has joined #openstack-infra | 06:02 | |
| *** Volundr has joined #openstack-infra | 06:02 | |
| *** mescanef has joined #openstack-infra | 06:02 | |
| *** cfarquhar has joined #openstack-infra | 06:02 | |
| *** vponomaryov has quit IRC | 06:03 | |
| *** vponomaryov has joined #openstack-infra | 06:04 | |
| *** jaosorior has joined #openstack-infra | 06:04 | |
| *** heyongli has quit IRC | 06:06 | |
| *** heyongli has joined #openstack-infra | 06:07 | |
| *** krotscheck has joined #openstack-infra | 06:09 | |
| *** yamahata_ has quit IRC | 06:10 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/291557 | 06:11 |
| *** esikachev has joined #openstack-infra | 06:16 | |
| *** heyongli has quit IRC | 06:17 | |
| *** heyongli has joined #openstack-infra | 06:17 | |
| *** openstackgerrit_ has quit IRC | 06:17 | |
| *** cloudtrainme has joined #openstack-infra | 06:17 | |
| *** openstackgerrit_ has joined #openstack-infra | 06:18 | |
| *** cloudtrainme has quit IRC | 06:18 | |
| *** Guest68942 has joined #openstack-infra | 06:18 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet: fix cookiecutter job template https://review.openstack.org/290906 | 06:20 |
| *** esikachev has quit IRC | 06:21 | |
| *** hdd has quit IRC | 06:22 | |
| *** eezhova has quit IRC | 06:23 | |
| *** eezhova has joined #openstack-infra | 06:23 | |
| *** heyongli has quit IRC | 06:27 | |
| *** Sukhdev has joined #openstack-infra | 06:28 | |
| *** hieulq has joined #openstack-infra | 06:35 | |
| *** thorst has joined #openstack-infra | 06:37 | |
| *** baoli has joined #openstack-infra | 06:41 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 06:42 |
| *** thorst has quit IRC | 06:43 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 06:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 06:44 |
| *** baoli has quit IRC | 06:46 | |
| *** noorul has joined #openstack-infra | 06:46 | |
| *** mrmartin has joined #openstack-infra | 06:47 | |
| noorul | I am trying to setup openstack-ci locally | 06:48 |
| noorul | I see mysql reference everywhere but can't find puppet manifest to setup mysql | 06:48 |
| *** dalgaaf has joined #openstack-infra | 06:49 | |
| *** rcernin has joined #openstack-infra | 06:49 | |
| *** watanabe_isao has joined #openstack-infra | 06:50 | |
| *** Jeffrey4l has joined #openstack-infra | 06:55 | |
| *** sridhar_ram has quit IRC | 06:56 | |
| *** _nadya_ has joined #openstack-infra | 06:59 | |
| nibalizer | noorul: most of the time we use trove instances | 06:59 |
| *** gildub has quit IRC | 07:04 | |
| *** sbalukoff has joined #openstack-infra | 07:05 | |
| *** asettle has quit IRC | 07:07 | |
| *** camunoz has quit IRC | 07:07 | |
| *** asettle has joined #openstack-infra | 07:08 | |
| *** jaosorior has quit IRC | 07:10 | |
| *** mariojv has quit IRC | 07:10 | |
| *** ayoung has quit IRC | 07:10 | |
| *** Douhet has quit IRC | 07:10 | |
| *** bryan_att has quit IRC | 07:10 | |
| *** fkautz has quit IRC | 07:10 | |
| *** Ng has quit IRC | 07:10 | |
| *** zxiiro has quit IRC | 07:10 | |
| *** fdegir has quit IRC | 07:10 | |
| *** simonmcc has quit IRC | 07:10 | |
| *** ashtokolov has quit IRC | 07:10 | |
| *** dansmith has quit IRC | 07:10 | |
| *** yonglihe has quit IRC | 07:10 | |
| *** evgenyl has quit IRC | 07:10 | |
| *** mwhahaha has quit IRC | 07:10 | |
| *** Volundr has quit IRC | 07:10 | |
| *** mescanef has quit IRC | 07:10 | |
| *** cfarquhar has quit IRC | 07:10 | |
| *** zhurong_ has quit IRC | 07:11 | |
| *** zhurong_ has joined #openstack-infra | 07:12 | |
| *** rlandy has quit IRC | 07:16 | |
| noorul | nibalizer: thank you! | 07:17 |
| nibalizer | mysql::server from the mysql module works pretty well | 07:17 |
| *** asettle has quit IRC | 07:18 | |
| *** jaosorior has joined #openstack-infra | 07:21 | |
| *** mariojv has joined #openstack-infra | 07:21 | |
| *** ayoung has joined #openstack-infra | 07:21 | |
| *** Douhet has joined #openstack-infra | 07:21 | |
| *** fkautz has joined #openstack-infra | 07:21 | |
| *** bryan_att has joined #openstack-infra | 07:21 | |
| *** Ng has joined #openstack-infra | 07:21 | |
| *** zxiiro has joined #openstack-infra | 07:21 | |
| *** fdegir has joined #openstack-infra | 07:21 | |
| *** simonmcc has joined #openstack-infra | 07:21 | |
| *** ashtokolov has joined #openstack-infra | 07:21 | |
| *** dansmith has joined #openstack-infra | 07:21 | |
| *** yonglihe has joined #openstack-infra | 07:21 | |
| *** evgenyl has joined #openstack-infra | 07:21 | |
| *** mwhahaha has joined #openstack-infra | 07:21 | |
| *** Volundr has joined #openstack-infra | 07:21 | |
| *** mescanef has joined #openstack-infra | 07:21 | |
| *** cfarquhar has joined #openstack-infra | 07:21 | |
| *** scheuran has joined #openstack-infra | 07:24 | |
| *** Sukhdev has quit IRC | 07:24 | |
| *** hieulq has quit IRC | 07:24 | |
| *** hieulq has joined #openstack-infra | 07:26 | |
| *** jtomasek has joined #openstack-infra | 07:27 | |
| *** amotoki has joined #openstack-infra | 07:29 | |
| noorul | nibalizer: I installed mysql manually for the time being | 07:30 |
| noorul | nibalizer: I am now stuck at http://paste.openstack.org/show/490100/ | 07:30 |
| *** mikelk has joined #openstack-infra | 07:40 | |
| *** thorst has joined #openstack-infra | 07:41 | |
| *** gildub has joined #openstack-infra | 07:42 | |
| openstackgerrit | Andreas Scheuring proposed openstack-infra/system-config: Correct Logstash Query Example https://review.openstack.org/257501 | 07:44 |
| *** Guest68942 has quit IRC | 07:45 | |
| openstackgerrit | Andreas Scheuring proposed openstack-infra/system-config: Correct Logstash Query Example https://review.openstack.org/257501 | 07:45 |
| *** asettle has joined #openstack-infra | 07:45 | |
| *** jaosorior has quit IRC | 07:47 | |
| *** mariojv has quit IRC | 07:47 | |
| *** ayoung has quit IRC | 07:47 | |
| *** Douhet has quit IRC | 07:47 | |
| *** bryan_att has quit IRC | 07:47 | |
| *** fkautz has quit IRC | 07:47 | |
| *** Ng has quit IRC | 07:47 | |
| *** zxiiro has quit IRC | 07:47 | |
| *** fdegir has quit IRC | 07:47 | |
| *** simonmcc has quit IRC | 07:47 | |
| *** ashtokolov has quit IRC | 07:47 | |
| *** dansmith has quit IRC | 07:47 | |
| *** yonglihe has quit IRC | 07:47 | |
| *** evgenyl has quit IRC | 07:47 | |
| *** mwhahaha has quit IRC | 07:47 | |
| *** Volundr has quit IRC | 07:47 | |
| *** mescanef has quit IRC | 07:47 | |
| *** cfarquhar has quit IRC | 07:47 | |
| *** nikhil has joined #openstack-infra | 07:47 | |
| *** thorst has quit IRC | 07:49 | |
| *** asettle has quit IRC | 07:50 | |
| *** mrmartin has quit IRC | 07:50 | |
| *** mrmartin has joined #openstack-infra | 07:52 | |
| *** fhubik has joined #openstack-infra | 07:52 | |
| *** jaosorior has joined #openstack-infra | 07:54 | |
| *** mariojv has joined #openstack-infra | 07:54 | |
| *** ayoung has joined #openstack-infra | 07:54 | |
| *** Douhet has joined #openstack-infra | 07:54 | |
| *** fkautz has joined #openstack-infra | 07:54 | |
| *** bryan_att has joined #openstack-infra | 07:54 | |
| *** Ng has joined #openstack-infra | 07:54 | |
| *** zxiiro has joined #openstack-infra | 07:54 | |
| *** fdegir has joined #openstack-infra | 07:54 | |
| *** simonmcc has joined #openstack-infra | 07:54 | |
| *** ashtokolov has joined #openstack-infra | 07:54 | |
| *** dansmith has joined #openstack-infra | 07:54 | |
| *** yonglihe has joined #openstack-infra | 07:54 | |
| *** evgenyl has joined #openstack-infra | 07:54 | |
| *** mwhahaha has joined #openstack-infra | 07:54 | |
| *** Volundr has joined #openstack-infra | 07:54 | |
| *** mescanef has joined #openstack-infra | 07:54 | |
| *** cfarquhar has joined #openstack-infra | 07:54 | |
| *** mrmartin has quit IRC | 07:56 | |
| AJaeger | Good morning! I have one cleanup for the ubuntu-trusty jobs: Rename of tox-nodb to to: https://review.openstack.org/#/c/291349/ Could a second project-config core review this, please? | 07:58 |
| *** kzaitsev_mb has joined #openstack-infra | 07:59 | |
| *** harlowja_at_home has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Delay release for charm projects https://review.openstack.org/291421 | 08:01 |
| *** dizquierdo has joined #openstack-infra | 08:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/291557 | 08:02 |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-tuning-box' https://review.openstack.org/286137 | 08:04 |
| *** jlanoux has joined #openstack-infra | 08:08 | |
| *** fhubik has quit IRC | 08:13 | |
| *** gongysh_ has joined #openstack-infra | 08:16 | |
| *** claudiub has joined #openstack-infra | 08:18 | |
| *** e0ne has joined #openstack-infra | 08:21 | |
| *** claudiub|2 has joined #openstack-infra | 08:21 | |
| *** HeOS has quit IRC | 08:21 | |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-tuning-box' https://review.openstack.org/286137 | 08:23 |
| *** vincentfrancoise has joined #openstack-infra | 08:23 | |
| *** jamielennox is now known as jamielennox|away | 08:23 | |
| *** claudiub has quit IRC | 08:24 | |
| *** dtardivel has joined #openstack-infra | 08:26 | |
| *** matrohon has joined #openstack-infra | 08:28 | |
| *** hashar has joined #openstack-infra | 08:32 | |
| *** arxcruz has joined #openstack-infra | 08:33 | |
| *** _nadya_ has quit IRC | 08:34 | |
| *** bauzas is now known as bauwser | 08:34 | |
| *** shardy has joined #openstack-infra | 08:35 | |
| *** yamahata_ has joined #openstack-infra | 08:35 | |
| *** sarob has joined #openstack-infra | 08:37 | |
| *** flepied has quit IRC | 08:37 | |
| *** vincentll has joined #openstack-infra | 08:38 | |
| *** pcaruana has joined #openstack-infra | 08:41 | |
| *** Jeffrey4l has quit IRC | 08:41 | |
| *** sarob has quit IRC | 08:42 | |
| *** ihrachys has joined #openstack-infra | 08:45 | |
| *** abregman has joined #openstack-infra | 08:45 | |
| *** thorst has joined #openstack-infra | 08:46 | |
| *** HeOS has joined #openstack-infra | 08:48 | |
| *** jeeloo_ has joined #openstack-infra | 08:49 | |
| *** chlong has quit IRC | 08:49 | |
| *** zeih_ has joined #openstack-infra | 08:51 | |
| *** jeeloo has joined #openstack-infra | 08:52 | |
| *** jeeloo_ has quit IRC | 08:52 | |
| *** thorst has quit IRC | 08:54 | |
| *** kzaitsev_mb has quit IRC | 08:54 | |
| *** ifarkas has joined #openstack-infra | 08:55 | |
| *** noorul has quit IRC | 08:57 | |
| *** watanabe_isao has quit IRC | 09:01 | |
| *** rossella_s has quit IRC | 09:03 | |
| *** dizquierdo has quit IRC | 09:03 | |
| *** rossella_s has joined #openstack-infra | 09:03 | |
| *** sfinucan has joined #openstack-infra | 09:04 | |
| jeeloo | Hi, I'm trying to set up zuul in my own environment and I have managed to get most of the flow working. The issue that I have run into at the moment is that every time there is more than 2 changes in ZUUL_CHANGES the zuul-cloner fails with "IndexError: No item found with id u'origin/HEAD'" I'm on Zuul 2.1.0, any thougts or suggestions wold be appreciated. | 09:06 |
| *** lucas-afk is now known as lucasagomes | 09:06 | |
| openstackgerrit | Alexey Ovchinnikov proposed openstack-infra/project-config: [WIP] Add publishing capability to buildimage-lxd job https://review.openstack.org/290418 | 09:06 |
| *** salv-orlando has joined #openstack-infra | 09:09 | |
| *** zhurong_ has quit IRC | 09:10 | |
| *** hichihara has quit IRC | 09:10 | |
| *** sbelous_ has joined #openstack-infra | 09:11 | |
| *** flepied has joined #openstack-infra | 09:13 | |
| *** electrofelix has joined #openstack-infra | 09:15 | |
| *** daemontool has joined #openstack-infra | 09:16 | |
| *** esikachev has joined #openstack-infra | 09:18 | |
| *** ramishra has quit IRC | 09:19 | |
| *** jordanP has joined #openstack-infra | 09:22 | |
| *** abregman is now known as abregman|afk | 09:23 | |
| *** jistr has joined #openstack-infra | 09:24 | |
| *** ramishra has joined #openstack-infra | 09:25 | |
| *** salv-orlando has quit IRC | 09:25 | |
| *** yamahata_ has quit IRC | 09:26 | |
| *** pblaho has joined #openstack-infra | 09:27 | |
| *** tzn has quit IRC | 09:28 | |
| *** tzn has joined #openstack-infra | 09:28 | |
| *** takashi has quit IRC | 09:29 | |
| *** jordanP has quit IRC | 09:29 | |
| *** jordanP has joined #openstack-infra | 09:31 | |
| *** Daisy has joined #openstack-infra | 09:32 | |
| *** Daisy has quit IRC | 09:37 | |
| *** derekh has joined #openstack-infra | 09:40 | |
| *** yamamoto has quit IRC | 09:40 | |
| *** vgridnev has joined #openstack-infra | 09:42 | |
| *** _nadya_ has joined #openstack-infra | 09:45 | |
| *** _nadya_ has quit IRC | 09:50 | |
| *** _nadya__ has joined #openstack-infra | 09:50 | |
| *** thorst has joined #openstack-infra | 09:51 | |
| *** esikachev has quit IRC | 09:53 | |
| *** placko_ has joined #openstack-infra | 09:57 | |
| *** abregman|afk has quit IRC | 09:57 | |
| *** thorst has quit IRC | 09:59 | |
| *** esikachev has joined #openstack-infra | 10:02 | |
| *** rvasilets__ has joined #openstack-infra | 10:02 | |
| *** daemontool_ has joined #openstack-infra | 10:02 | |
| *** andreykurilin__ has joined #openstack-infra | 10:06 | |
| *** nijaba has quit IRC | 10:06 | |
| *** daemontool has quit IRC | 10:06 | |
| *** hieulq has quit IRC | 10:07 | |
| *** nijaba has joined #openstack-infra | 10:07 | |
| *** _degorenko|afk is now known as degorenko | 10:08 | |
| *** tpsilva has joined #openstack-infra | 10:09 | |
| *** Qiming has quit IRC | 10:11 | |
| *** Guest20150 has quit IRC | 10:14 | |
| *** kushal has quit IRC | 10:14 | |
| *** Hal has joined #openstack-infra | 10:16 | |
| *** rguillebert has joined #openstack-infra | 10:16 | |
| *** Hal is now known as Guest69862 | 10:16 | |
| yolanda | morning | 10:18 |
| openstackgerrit | mathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 10:23 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/storyboard: Replace deprecated LOG.warn with LOG.warning https://review.openstack.org/291649 | 10:25 |
| *** kzaitsev_mb has joined #openstack-infra | 10:28 | |
| *** jcoufal has joined #openstack-infra | 10:30 | |
| *** sdague has joined #openstack-infra | 10:30 | |
| *** sambetts|afk is now known as sambetts | 10:31 | |
| AJaeger | morning, yolanda ! | 10:35 |
| yolanda | how are things today? | 10:35 |
| yolanda | i'm seeing gerrit a bit slow, but i checked logs and there are no relevant errors | 10:36 |
| *** samuelBartel has joined #openstack-infra | 10:36 | |
| AJaeger | yeah, it's slowing down, had a few 502s ;( | 10:37 |
| *** gongysh_ has quit IRC | 10:38 | |
| yolanda | AJaeger, about https://review.openstack.org/#/c/284281 ... do we have any policy for enabling new vhosts in infra? | 10:40 |
| openstackgerrit | Merged openstack-infra/project-config: Publish Anchor to pypi https://review.openstack.org/291433 | 10:40 |
| openstackgerrit | Merged openstack-infra/project-config: Modify freezer job for tempest tests https://review.openstack.org/289442 | 10:41 |
| AJaeger | yolanda: I don't know ;( If you look at the referenced email, fungi even suggested it but you might want to double check with him. | 10:42 |
| AJaeger | yolanda: I fear that adding the chnage needs some manual work for DNS. | 10:42 |
| * AJaeger would love to see this move forward | 10:42 | |
| yolanda | yes, that's sure | 10:42 |
| *** daemontool__ has joined #openstack-infra | 10:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Added npm-install macro to swagger build job https://review.openstack.org/291422 | 10:43 |
| yolanda | i can +2 but wait for approval until we check with more infra-root | 10:43 |
| AJaeger | yolanda: please include https://review.openstack.org/291349 and https://review.openstack.org/287414 in your review session | 10:44 |
| yolanda | seems that fungi agrees so this should move forward | 10:44 |
| AJaeger | yolanda: it needs a second +2 anyhow ;) | 10:44 |
| yolanda | yep, system-config... | 10:44 |
| *** daemontool__ is now known as daemontool | 10:45 | |
| * AJaeger rebases 291349 now... | 10:45 | |
| yolanda | i was just going to tell you :) | 10:46 |
| *** daemontool_ has quit IRC | 10:46 | |
| AJaeger | the anchor change broke it ;( | 10:47 |
| *** sarob has joined #openstack-infra | 10:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename tox-nodb-ENV to tox-ENV https://review.openstack.org/291349 | 10:48 |
| AJaeger | rebased ^ | 10:48 |
| yolanda | cool i'll review | 10:48 |
| yolanda | gerrit is slowering for moment, i think is time for restart | 10:49 |
| yolanda | i started to get 502 on every review | 10:49 |
| AJaeger | yeah, it's getting worse. Let's restart while the load is light if you have time | 10:50 |
| yolanda | #status alert Gerrit is going to be restarted due to bad performance | 10:50 |
| openstackstatus | yolanda: sending alert | 10:50 |
| AJaeger | yolanda: Send the status ok only after the #status alert has finished... you might confuse the bot otherwise. I think last time you were too quick ;) | 10:51 |
| AJaeger | but it worked out just fine :) | 10:51 |
| yolanda | ah yes, or alert too slow :) | 10:51 |
| AJaeger | too many channels ;) | 10:52 |
| sbelous_ | sdague: ping | 10:52 |
| *** sarob has quit IRC | 10:53 | |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to bad performance | 10:53 | |
| *** ChanServ changes topic to "Gerrit is going to be restarted due to bad performance" | 10:53 | |
| *** dims has joined #openstack-infra | 10:54 | |
| Zara | that message always makes me feel so sorry for gerrit | 10:54 |
| yolanda | poor gerrit... it has been restarted already, but bot is too slow announcing... | 10:55 |
| openstackstatus | yolanda: finished sending alert | 10:56 |
| yolanda | here it comes... | 10:56 |
| AJaeger | yolanda: next time perhaps just a #status notice? That does not change the topic and just leaves a message... | 10:56 |
| yolanda | #status ok Gerrit has been restarted successfully | 10:56 |
| openstackstatus | yolanda: sending ok | 10:56 |
| yolanda | yes, it may work better | 10:56 |
| *** thorst has joined #openstack-infra | 10:56 | |
| AJaeger | Not sure what's the best practice is - or if one exists. But it looks like the restart is soo quick that nobody notices with the 502s;) | 10:57 |
| *** asettle has joined #openstack-infra | 10:57 | |
| *** asettle has quit IRC | 10:57 | |
| *** mrmartin has joined #openstack-infra | 10:57 | |
| yolanda | yes, it takes much more time to send the alerts than to restart gerrit | 10:57 |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 10:58 |
| AJaeger | yolanda: can you kill https://jenkins03.openstack.org/job/magnum-coverage/359/consoleFull ? Looking at zuul status, the job runs for four hours but jenkins03 hasn't seen any action for 3+ hours ;( | 10:58 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 10:58 | |
| -openstackstatus- NOTICE: Gerrit has been restarted successfully | 10:58 | |
| yolanda | sure | 10:58 |
| openstackgerrit | Ilya Shakhat proposed openstack-infra/project-config: Add new project "performa" https://review.openstack.org/291657 | 10:59 |
| AJaeger | that's on vexxhost for the record | 10:59 |
| *** matrohon has quit IRC | 10:59 | |
| yolanda | pabelanger told that vexxhost was having problems due to slow i/o yesterday | 10:59 |
| yolanda | killed | 11:00 |
| AJaeger | thanks | 11:00 |
| *** Qiming has joined #openstack-infra | 11:01 | |
| openstackstatus | yolanda: finished sending ok | 11:01 |
| *** kzaitsev_mb has quit IRC | 11:02 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| openstackgerrit | Simon McCartney proposed openstack-infra/infra-ansible: Reworked setup_env.sh to support Debian & Darwin/OSX https://review.openstack.org/291658 | 11:02 |
| openstackgerrit | Merged openstack-infra/system-config: Add a simple LP OpenId reverse lookup utility https://review.openstack.org/286284 | 11:03 |
| *** thorst has quit IRC | 11:04 | |
| *** yamamoto_ has joined #openstack-infra | 11:06 | |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Use default ruby packages for Xenial https://review.openstack.org/276811 | 11:06 |
| openstackgerrit | Merged openstack-infra/infra-manual: creators.rst: Fix broken internal hyperlinks https://review.openstack.org/287083 | 11:06 |
| *** dizquierdo has joined #openstack-infra | 11:08 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Update translation setup instructions https://review.openstack.org/284047 | 11:08 |
| *** yamamoto has quit IRC | 11:09 | |
| *** ociuhandu has quit IRC | 11:11 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:15 | |
| *** daemontool_ has joined #openstack-infra | 11:16 | |
| *** vgridnev has quit IRC | 11:16 | |
| *** d0ugal has quit IRC | 11:16 | |
| *** d0ugal has joined #openstack-infra | 11:17 | |
| *** mrmartin has quit IRC | 11:18 | |
| *** daemontool has quit IRC | 11:19 | |
| *** Daisy has joined #openstack-infra | 11:20 | |
| *** mrmartin has joined #openstack-infra | 11:21 | |
| *** gnuoy_ has joined #openstack-infra | 11:22 | |
| *** gnuoy_ has quit IRC | 11:22 | |
| *** e0ne has quit IRC | 11:23 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix file link to groups.txt in sysadmin docs https://review.openstack.org/284378 | 11:23 |
| *** vincentfrancoise has quit IRC | 11:23 | |
| openstackgerrit | Milan Kováčik proposed openstack-infra/project-config: enable stack sudo for ironic discovery gate job https://review.openstack.org/291664 | 11:24 |
| *** Daisy has quit IRC | 11:24 | |
| *** yamamoto_ has quit IRC | 11:25 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 11:26 |
| *** yamamoto has joined #openstack-infra | 11:26 | |
| *** e0ne has joined #openstack-infra | 11:26 | |
| *** salv-orlando has joined #openstack-infra | 11:28 | |
| *** kzaitsev_mb has quit IRC | 11:30 | |
| *** yamamoto has quit IRC | 11:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental gate to kolla-mesos https://review.openstack.org/289042 | 11:30 |
| *** mrmartin has quit IRC | 11:31 | |
| openstackgerrit | Merged openstack-infra/system-config: Add roles for CI users https://review.openstack.org/288160 | 11:31 |
| *** asilenkov has quit IRC | 11:31 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 11:32 |
| *** tesseract has joined #openstack-infra | 11:33 | |
| *** jtomasek has quit IRC | 11:34 | |
| *** tesseract has quit IRC | 11:34 | |
| *** Guest16009 has joined #openstack-infra | 11:34 | |
| *** HeOS has quit IRC | 11:36 | |
| *** vgridnev has joined #openstack-infra | 11:36 | |
| *** HeOS has joined #openstack-infra | 11:36 | |
| *** asilenkov has joined #openstack-infra | 11:37 | |
| *** Guest16009 has quit IRC | 11:37 | |
| *** ildikov has quit IRC | 11:37 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add a tooltip when hovering on the "Task notes" button https://review.openstack.org/291391 | 11:38 |
| *** salv-orlando has quit IRC | 11:39 | |
| *** jtomasek has joined #openstack-infra | 11:39 | |
| *** tesseract- has joined #openstack-infra | 11:40 | |
| yolanda | mm AJaeger, look at that failure in puppet linters.. | 11:40 |
| yolanda | http://logs.openstack.org/47/275247/1/check/gate-puppet-etherpad_lite-puppet-lint/f8c02f4/console.html | 11:40 |
| *** kushal has joined #openstack-infra | 11:40 | |
| yolanda | 2016-03-11 11:35:36.012 | + sudo rm -f /etc/sudoers.d/jenkins-sudo | 11:40 |
| yolanda | 2016-03-11 11:35:36.013 | + sudo -n true | 11:40 |
| yolanda | 2016-03-11 11:35:36.013 | sudo: a password is required | 11:40 |
| yolanda | looks related to the move? | 11:40 |
| AJaeger | it does not ring a bell ;( | 11:41 |
| yolanda | going to hold a node and test | 11:41 |
| AJaeger | let me check the job definition and change to it... | 11:41 |
| *** mrmartin has joined #openstack-infra | 11:42 | |
| *** yamamoto has joined #openstack-infra | 11:42 | |
| AJaeger | yolanda: EmilienM updated this job already on the 26th of February... | 11:43 |
| AJaeger | https://review.openstack.org/#/c/285542/ looks fine | 11:43 |
| *** yamamoto has quit IRC | 11:43 | |
| *** erlon has joined #openstack-infra | 11:44 | |
| *** _amrith_ is now known as amrith | 11:44 | |
| *** HeOS has quit IRC | 11:44 | |
| yolanda | apart from this error, i see | 11:45 |
| yolanda | 2016-03-11 11:35:36.455 | + rake lint | 11:45 |
| yolanda | 2016-03-11 11:35:36.455 | /tmp/hudson2623652897521899231.sh: line 23: rake: command not found | 11:45 |
| *** HeOS has joined #openstack-infra | 11:45 | |
| AJaeger | yolanda: exactly ;) The sudo is fine, the missing rake is the problem | 11:45 |
| *** thorst has joined #openstack-infra | 11:46 | |
| yolanda | we need to add exec rake macro? | 11:46 |
| AJaeger | I wonder why it suddenly fails ;( | 11:47 |
| yolanda | yes, that's strange | 11:47 |
| *** mrmartin has quit IRC | 11:48 | |
| *** ociuhandu has joined #openstack-infra | 11:48 | |
| openstackgerrit | Merged openstack-infra/project-config: CentOS AFS build : move output to $WORKSPACE https://review.openstack.org/291521 | 11:49 |
| *** vgridnev has quit IRC | 11:49 | |
| yolanda | AJaeger, it may be a particular use case of that project | 11:49 |
| yolanda | no metadata.json or Gemfile... so it just executes rake lint | 11:50 |
| yolanda | check in jenkins/jobs/macros.yaml, for the puppet-lint job | 11:50 |
| AJaeger | I agree. The repository has not seen merges since November and I guess the missing Gemfile or metadata.json is the culprit. | 11:50 |
| yolanda | ok going to amend it | 11:51 |
| yolanda | but seems our fallback for rake lint is broken anyway | 11:51 |
| AJaeger | yes, it is - it should install rake | 11:51 |
| AJaeger | might be that bare-trusty had rake on it so this didn't hit us ;( | 11:52 |
| yolanda | looks like. We are hitting some interesting bugs with the move to ubuntu-trusty :) | 11:52 |
| AJaeger | won't be the last one ;( | 11:52 |
| yolanda | i will send a fix | 11:53 |
| AJaeger | thanks | 11:53 |
| AJaeger | then get that one in - and afterwards recheck the puppet-etherpad changes... | 11:53 |
| *** vgridnev has joined #openstack-infra | 11:54 | |
| yolanda | mm, git review is taking ages | 11:55 |
| yolanda | is that the issue you had yesterday? | 11:55 |
| *** thorst has quit IRC | 11:55 | |
| *** thorst has joined #openstack-infra | 11:56 | |
| AJaeger | I had the issue with a review that contained a stacked change | 11:56 |
| yolanda | it may be that my connection sucks... | 11:56 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-etherpad_lite: Add missing Gemfile to the project https://review.openstack.org/291678 | 11:57 |
| yolanda | took a life! | 11:57 |
| *** kzaitsev_mb has joined #openstack-infra | 11:59 | |
| AJaeger | are you going to fix the job as well so that the fallback works? | 11:59 |
| *** aysyd has joined #openstack-infra | 11:59 | |
| yolanda | yep | 11:59 |
| yolanda | working on it, but i want to test it | 12:00 |
| AJaeger | thanks! | 12:00 |
| yolanda | anytime | 12:01 |
| *** jaosorior has quit IRC | 12:02 | |
| *** jaosorior has joined #openstack-infra | 12:03 | |
| *** thorst has quit IRC | 12:05 | |
| *** aysyd has quit IRC | 12:05 | |
| *** aysyd has joined #openstack-infra | 12:05 | |
| *** ldnunes has joined #openstack-infra | 12:05 | |
| *** rvasilets__ has quit IRC | 12:06 | |
| *** rhallisey has joined #openstack-infra | 12:06 | |
| *** lucasagomes is now known as lucas-hungry | 12:07 | |
| *** zhenguo_ has quit IRC | 12:10 | |
| *** rfolco has joined #openstack-infra | 12:10 | |
| *** |-paul-| has joined #openstack-infra | 12:12 | |
| *** sdake has joined #openstack-infra | 12:16 | |
| *** kushal has quit IRC | 12:22 | |
| *** sdake has quit IRC | 12:23 | |
| *** jtomasek has quit IRC | 12:25 | |
| *** jtomasek has joined #openstack-infra | 12:27 | |
| *** gongysh has joined #openstack-infra | 12:28 | |
| yolanda | AJaeger, stlll looking because it seems for this case, lint will break even with rake installed | 12:29 |
| AJaeger | ;( | 12:31 |
| *** yamamoto has joined #openstack-infra | 12:31 | |
| *** ildikov has joined #openstack-infra | 12:32 | |
| *** dims has quit IRC | 12:33 | |
| *** amrith is now known as _amrith_ | 12:33 | |
| yolanda | it may be a problem of the project itself, because Rakefile needs another deps, such as spec_helper | 12:35 |
| *** kushal has joined #openstack-infra | 12:36 | |
| *** sdake has joined #openstack-infra | 12:36 | |
| *** dims has joined #openstack-infra | 12:36 | |
| *** dguitarbite has quit IRC | 12:37 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** thorst has joined #openstack-infra | 12:40 | |
| *** baoli has joined #openstack-infra | 12:41 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Install rake lint when no Gemfile is found https://review.openstack.org/291700 | 12:44 |
| yolanda | AJaeger, that made it work for me, although i'm not 100% sure of all cases | 12:44 |
| pabelanger | yolanda: AJaeger: Ya, I've seem some general IO slowness with vexxhost, we should have them check into why | 12:45 |
| yolanda | pabelanger, i have not been much involved on that setup. Do you know the person to contact? | 12:47 |
| *** _amrith_ is now known as amrith | 12:47 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add domain_id param to project operations https://review.openstack.org/291701 | 12:48 |
| openstackgerrit | Merged openstack-infra/system-config: login.defs/redhat: set SYS_UID_MIN and SYS_GID_MIN https://review.openstack.org/288244 | 12:49 |
| *** abregman|nb has joined #openstack-infra | 12:49 | |
| *** |-paul-| has quit IRC | 12:50 | |
| pabelanger | yolanda: I cannot remember of the top of my head, need to check logs | 12:50 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-etherpad_lite: Add missing Gemfile to the project https://review.openstack.org/291678 | 12:51 |
| pabelanger | AJaeger: yolanda: are we ready to land 291349? | 12:51 |
| *** Guest69862 has quit IRC | 12:51 | |
| yolanda | i think is fine for me, we'll need to monitor jjb runs | 12:52 |
| *** Hal has joined #openstack-infra | 12:52 | |
| pabelanger | okay, I can try and help too, if you don't mind mentoring on it :) | 12:53 |
| *** Hal is now known as Guest75056 | 12:53 | |
| yolanda | pabelanger, i see dmsimard involved in some conversations about vexxhost. Do you remember that name? | 12:53 |
| *** kaisers1 has joined #openstack-infra | 12:54 | |
| *** e0ne has quit IRC | 12:54 | |
| yolanda | pabelanger, what i normally do for monitoring jjb, is that i tail puppet logs on puppetmaster, to check if jenkins is successful | 12:54 |
| *** yamamoto has joined #openstack-infra | 12:54 | |
| yolanda | also is good to see config history on all jenkins, do you have access to do it? | 12:54 |
| *** e0ne has joined #openstack-infra | 12:54 | |
| pabelanger | yolanda: Ya, dmsimard will know the contact, they are both from MTL | 12:54 |
| pabelanger | yolanda: okay, let me ssh into puppetmaster and tail | 12:55 |
| pabelanger | yolanda: where do you check config history? | 12:55 |
| yolanda | pabelanger for example on https://jenkins04.openstack.org/job/gate-system-config-puppet-syntax/jobConfigHistory/ | 12:56 |
| yolanda | if you have perms you will see something, if not, that will be empty | 12:56 |
| yolanda | you need to be sure you have openstack-ci-admins group enabled | 12:56 |
| yolanda | when login from sso | 12:56 |
| pabelanger | yolanda: Ah, no empty here | 12:56 |
| pabelanger | after sso | 12:56 |
| pabelanger | so will ask fungi | 12:57 |
| * yolanda needs to run to have lunch | 12:57 | |
| yolanda | will be back in short | 12:57 |
| *** amrith is now known as _amrith_ | 13:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Rename tox-nodb-ENV to tox-ENV https://review.openstack.org/291349 | 13:00 |
| *** yamamoto has quit IRC | 13:01 | |
| *** rossella_s has quit IRC | 13:03 | |
| *** alaski is now known as lascii | 13:03 | |
| *** e0ne has quit IRC | 13:03 | |
| *** dims_ has joined #openstack-infra | 13:03 | |
| *** rossella_s has joined #openstack-infra | 13:03 | |
| *** e0ne has joined #openstack-infra | 13:04 | |
| *** dims has quit IRC | 13:04 | |
| *** EricGonczer_ has joined #openstack-infra | 13:05 | |
| *** sdake has quit IRC | 13:10 | |
| *** sdake has joined #openstack-infra | 13:11 | |
| *** jaosorior has quit IRC | 13:12 | |
| *** jaosorior has joined #openstack-infra | 13:12 | |
| *** mugsie has quit IRC | 13:13 | |
| *** mugsie has joined #openstack-infra | 13:13 | |
| yolanda | pabelanger, when you login into jenkins, you see a checkbox asking for openstack-ci-admins membership? | 13:17 |
| *** akshai has joined #openstack-infra | 13:19 | |
| *** dprince has joined #openstack-infra | 13:19 | |
| *** sdake_ has joined #openstack-infra | 13:20 | |
| *** notnownikki has joined #openstack-infra | 13:23 | |
| *** vincentfrancoise has joined #openstack-infra | 13:23 | |
| pabelanger | yolanda: no, just the SSO page from launchpad, then redirected back to jenkins | 13:23 |
| *** sdake has quit IRC | 13:23 | |
| *** yamamoto has joined #openstack-infra | 13:26 | |
| *** pradk_ has joined #openstack-infra | 13:26 | |
| *** pradk_ has quit IRC | 13:26 | |
| *** dingyichen has quit IRC | 13:28 | |
| *** jpr has quit IRC | 13:28 | |
| pabelanger | yolanda: okay, so I think puppet just failed on jenkins | 13:28 |
| *** zeih_ has quit IRC | 13:28 | |
| *** lucas-hungry is now known as lucasagomes | 13:29 | |
| *** yaume has joined #openstack-infra | 13:29 | |
| *** akshai has quit IRC | 13:29 | |
| yolanda | pabelanger, then there is a playbook to run jjb manually on the failed jenkins | 13:30 |
| *** akshai has joined #openstack-infra | 13:30 | |
| AJaeger | morning, pabelanger ! | 13:30 |
| yolanda | the run_jjb.yaml | 13:30 |
| yolanda | you can execute that from puppetmaster, but need to be sure that jjb is not already running on the jenkins host, to avoid cache corruption | 13:30 |
| pabelanger | yolanda: okay, can we do a screen share, so I can watch you do it? | 13:31 |
| pabelanger | AJaeger: o/ | 13:31 |
| *** baoli_ has joined #openstack-infra | 13:31 | |
| AJaeger | yolanda: I'm not sure about your rake change, the if -f GEMFILE condition looks wrong, you reorder commands. | 13:32 |
| AJaeger | pabelanger, yolanda, please double check | 13:32 |
| AJaeger | pabelanger: thanks for reviewing | 13:33 |
| yolanda | checking | 13:33 |
| *** Daisy has joined #openstack-infra | 13:34 | |
| *** baoli has quit IRC | 13:34 | |
| yolanda | AJaeger, i will amend, it was consequence of my tests i guess | 13:34 |
| *** jeeloo1 has joined #openstack-infra | 13:36 | |
| *** jeeloo has quit IRC | 13:36 | |
| *** keedya has joined #openstack-infra | 13:36 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove interpret_puppet_exitcodes https://review.openstack.org/291249 | 13:36 |
| *** abregman|nb is now known as abregman|afk | 13:37 | |
| *** links has quit IRC | 13:39 | |
| *** dizquierdo has quit IRC | 13:40 | |
| *** Daisy has quit IRC | 13:40 | |
| *** Daisy has joined #openstack-infra | 13:41 | |
| *** matrohon has joined #openstack-infra | 13:41 | |
| *** links has joined #openstack-infra | 13:42 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Install rake lint when no Gemfile is found https://review.openstack.org/291700 | 13:42 |
| AJaeger | seems that the tox job rename was small enough and it was applied to all Jenkinses - if my interpretation of puppetboard is correct | 13:42 |
| *** eharney has quit IRC | 13:42 | |
| AJaeger | EmilienM: could you check https://review.openstack.org/#/c/291700 , please? | 13:44 |
| EmilienM | AJaeger: sure | 13:44 |
| EmilienM | AJaeger: nice catch, and +1 | 13:45 |
| AJaeger | EmilienM: background : We have at least one repo that does not include a Gemfile and ubuntu-trusty has no rake isntalled by default. | 13:45 |
| EmilienM | yeah | 13:45 |
| *** Daisy has quit IRC | 13:45 | |
| AJaeger | EmilienM: thanks for reviewing! | 13:45 |
| *** kushal has quit IRC | 13:45 | |
| EmilienM | you will need a Gemfile though :-) | 13:46 |
| *** jeeloo1 has quit IRC | 13:47 | |
| *** jeeloo has joined #openstack-infra | 13:47 | |
| pabelanger | AJaeger: Ya, that is what I am trying to confirm ATM | 13:48 |
| *** kushal has joined #openstack-infra | 13:49 | |
| AJaeger | thanks, pabelanger ! | 13:49 |
| *** akshai has quit IRC | 13:51 | |
| *** hieulq has joined #openstack-infra | 13:53 | |
| yolanda | ok back.. going to check job config history to be sure | 13:53 |
| *** yaume has quit IRC | 13:53 | |
| *** doug-fish has joined #openstack-infra | 13:54 | |
| yolanda | thx for the review EmilienM | 13:54 |
| *** Qiming has quit IRC | 13:54 | |
| pabelanger | AJaeger: yolanda: Ya, things seem good looking at puppet now | 13:54 |
| *** Qiming has joined #openstack-infra | 13:54 | |
| AJaeger | glad to hear! | 13:55 |
| *** kgiusti has joined #openstack-infra | 13:55 | |
| *** akshai has joined #openstack-infra | 13:56 | |
| * AJaeger has no plans for further mass renames this week ;) | 13:56 | |
| pabelanger | no doubt | 13:57 |
| fungi | pabelanger: yeah, need to add you to a group on lp we're using for openid-based access control on the jenkinses. i can take care of it in a few moments | 13:57 |
| AJaeger | yolanda's change touches a few dozen jobs... | 13:57 |
| AJaeger | morning, fungi! | 13:57 |
| pabelanger | fungi: thanks | 13:57 |
| AJaeger | fungi, yolanda asked earlier whether there are policies for adding vhosts in the context of this change: https://review.openstack.org/#/c/284281/ - the neutron review board. | 13:58 |
| * AJaeger would love to see this land to help neutron folks with their workflow | 13:58 | |
| yolanda | AJaeger, so as a sample, that job was created on jenkins01 https://jenkins01.openstack.org/view/All/job/gate-horizon-tox-py27dj18/jobConfigHistory/ | 13:58 |
| yolanda | going to check all jenkins | 13:59 |
| AJaeger | yolanda: great! | 13:59 |
| *** sdake_ has quit IRC | 13:59 | |
| *** HeOS has quit IRC | 14:00 | |
| fungi | yolanda: AJaeger: yeah, when i mentioned "add a vhost" i meant something generic for aggregating review information (as a replacement for or alternative to reviewday). i didn't realize they wanted to run something neutron-only, so am not really in favor of that choice | 14:00 |
| yolanda | fungi, i misunderstood your comments on the thread then. So you prefer an independent project to be created? | 14:01 |
| openstackgerrit | Merged openstack-infra/project-config: Install rake lint when no Gemfile is found https://review.openstack.org/291700 | 14:01 |
| *** jeeloo has left #openstack-infra | 14:01 | |
| Shrews | fungi: do you know if there are any developer docs that describe running nodepool tests locally? none seem to be in the repo, so i'm guessing "no" | 14:01 |
| *** HeOS has joined #openstack-infra | 14:02 | |
| yolanda | AJaeger, pabelanger i see that renames applied in all jenkins properly | 14:02 |
| *** gildub has quit IRC | 14:03 | |
| *** nwkarsten has joined #openstack-infra | 14:05 | |
| *** sdake has joined #openstack-infra | 14:05 | |
| *** nwkarsten has quit IRC | 14:05 | |
| *** nwkarsten has joined #openstack-infra | 14:06 | |
| fungi | yolanda: doesn't have to be an independent project, just the job which publishes review details doesn't need to be an exclusive neutron-only thing and the site it's published under shouldn't be either | 14:06 |
| *** dims_ has quit IRC | 14:07 | |
| fungi | Shrews: by running nodepool tests locally you mean other than the unit tests which are in its repo? | 14:07 |
| AJaeger | rossella_s: see above | 14:07 |
| AJaeger | fungi, so rossella_s misunderstood you... | 14:08 |
| pabelanger | what is the status of the pypi.*.openstack.org servers? They've been replaced with AFS right? | 14:08 |
| *** kushal has quit IRC | 14:08 | |
| pabelanger | notice they are still referenced in puppet and wonder if they need to be cleaned up | 14:08 |
| fungi | Shrews: mmmpork wrote up some documentation on making a test deployment of nodepool to perform functional testing of it. let me see if that landed in the repo too | 14:08 |
| * AJaeger should have known on Tuesday when he was with rossella_s at the bug smash - we could have worked something out then... | 14:09 | |
| rossella_s | AJaeger, thanks for pinging me...so it's not clear to me what I should do | 14:09 |
| Shrews | fungi: just trying to run the unit tests. apparently there is some db setup that is assumed. trying to reverse engineer that part | 14:09 |
| fungi | Shrews: oh! databases, yes. i can show you the script we use to set up databases on our job workers | 14:10 |
| AJaeger | fungi, did you had something in mind like specs.openstack.org for rossella_s? Where each project could run a periodic job that publishes to let's say reviewlist.openstack.org and then there's a central index page with one entry per project? | 14:13 |
| fungi | Shrews: the mysql-prep script at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n642 followed by the mysql-db script at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n642 | 14:13 |
| *** ajmiller has joined #openstack-infra | 14:13 | |
| fungi | AJaeger: rossella_s: can the periodic job which generates the review details be extended to run for any team, not just neutron? then it doesn't need to run multiple times and can just create the index when it updates review info for whatever teams it's configured to support | 14:14 |
| Shrews | fungi: thx | 14:14 |
| *** mriedem has joined #openstack-infra | 14:14 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: Use IPv6 on the ceph gate job https://review.openstack.org/289445 | 14:14 |
| fungi | AJaeger: rossella_s: i haven't seen the publication job yet, so not sure what the options are there | 14:14 |
| *** gordc has joined #openstack-infra | 14:14 | |
| * AJaeger checks the job at https://review.openstack.org/#/c/284284/ | 14:15 | |
| fungi | aha, i see you just added a link to it in a review comment. that helps me gain some context on how this is expected to work | 14:15 |
| *** ajmiller_ has joined #openstack-infra | 14:15 | |
| rossella_s | fungi depends on the team, we don't have the same way of assigning priorities to reviews I think. My script was customized according to the requests of the neutron cores. | 14:15 |
| fungi | so tools/milestone-review-dash.py is something that lives in the neutron repo? | 14:16 |
| AJaeger | fungi: yes | 14:17 |
| *** dims has joined #openstack-infra | 14:17 | |
| AJaeger | rossella_s: you can't hardcode mitaka-3 either in the jenkins job... | 14:17 |
| *** links has quit IRC | 14:17 | |
| rossella_s | AJaeger, I can fix that | 14:17 |
| *** hdd has joined #openstack-infra | 14:18 | |
| fungi | my goal is just that if this becomes popular and a lot of other teams decide it's cool and want to do the same thing, that we make sure we don't end up with dozens of disconnected copies of this tooling and dozens of slightly different websites under separate domain names | 14:18 |
| AJaeger | rossella_s: that's a second step ;) Let's figure out with fungi the first one ;) | 14:18 |
| *** akshai has quit IRC | 14:18 | |
| *** ajmiller has quit IRC | 14:18 | |
| fungi | so planning to make it usable for everyone (a solution for openstack) instead of only one team (a solution for neutron) is preferable | 14:18 |
| AJaeger | fungi: I should have challenged you, I misunderstood you as neutron specific suggestion as well ;( | 14:18 |
| fungi | yeah, it's more that i'd like to see someone fix/take over/replace reviewday since it seems to have fallen into disuse | 14:19 |
| *** sdake_ has joined #openstack-infra | 14:19 | |
| *** akshai has joined #openstack-infra | 14:20 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on missing Summit Entity Events https://review.openstack.org/291735 | 14:20 |
| rossella_s | fungi, I agree with you...the script can be extended for other teams actually, it's not so hard. I imagine the most of the requirements are in common | 14:20 |
| *** _ari_ has joined #openstack-infra | 14:20 | |
| fungi | rossella_s: so to start, let's pick a generic name for the site which isn't team-specific, and we could publish this into a team-specific subdirectory while you try it out and see what needs adjusting | 14:21 |
| *** xyang1 has joined #openstack-infra | 14:21 | |
| AJaeger | and once more projects come up, we can generalize, add index page etc? | 14:21 |
| rossella_s | fungi, great! let's do that | 14:21 |
| *** sdake has quit IRC | 14:21 | |
| *** vincentfrancoise has left #openstack-infra | 14:22 | |
| *** kaisers1 has quit IRC | 14:22 | |
| fungi | then later, if it's something others want to do, the tooling can be extracted from the neutron repo and put somewhere more central, generalized and made more configurable, and we could start running it for more projects and publishing an index to their team dashboards or whatever | 14:22 |
| AJaeger | fungi, good plan. | 14:22 |
| fungi | but that should be the step that happens when a second team decides to use it, so we don't get a messy proliferation | 14:23 |
| *** yaume has joined #openstack-infra | 14:23 | |
| AJaeger | So, for now: Rename the vhost in system-config | 14:23 |
| rossella_s | fungi, AJaeger thanks a lot, it's a good plan! | 14:23 |
| *** dims has quit IRC | 14:23 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on missing Summit Entity Events https://review.openstack.org/291735 | 14:23 |
| AJaeger | For project-config change: Resolve merge conflict, remove mitaka-3 as argument and publish to neutron subdirecotry | 14:24 |
| AJaeger | rossella_s: Add to the system-config commit message a "Needed-By: GerritID-of-project-config" line | 14:24 |
| * AJaeger will be back later... | 14:25 | |
| AJaeger | thanks, fungi! | 14:25 |
| *** ikutukov has joined #openstack-infra | 14:26 | |
| rossella_s | AJaeger, thanks a lot! | 14:26 |
| *** ericksonsantos has quit IRC | 14:26 | |
| *** nikhil has quit IRC | 14:26 | |
| *** ericksonsantos has joined #openstack-infra | 14:27 | |
| *** nikhil has joined #openstack-infra | 14:27 | |
| fungi | pabelanger: you are now a member of https://launchpad.net/~openstack-ci-admins | 14:27 |
| *** placko_ has quit IRC | 14:28 | |
| pabelanger | fungi: thanks, so just so I know, what does that actually mean :) | 14:28 |
| fungi | pabelanger: log out of the jenkins webui, then when you're logging back in pay attention to the openid details confirmation page where it prompts you to check the checkbox to include group membership details for that | 14:28 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add version numbers to tracking dashboard https://review.openstack.org/291741 | 14:28 |
| fungi | pabelanger: that's pretty much the only thing we use that group for any more | 14:28 |
| pabelanger | fungi: ack | 14:28 |
| pabelanger | fungi: okay, I see the difference now | 14:29 |
| fungi | pabelanger: it should remember you've checked that when logging into _that_ jenkins master in the future, but you'll need to do it again for any one of them you haven't logged into yet | 14:29 |
| *** burgerk has joined #openstack-infra | 14:30 | |
| *** vgridnev has quit IRC | 14:31 | |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/system-config: Add vhost for Neutron reviews dashboard https://review.openstack.org/284281 | 14:31 |
| *** [1]Thelo has joined #openstack-infra | 14:32 | |
| *** _amrith_ is now known as amrith | 14:32 | |
| rossella_s | fungi, I fixed it, hope I got what you meant, thanks! | 14:32 |
| *** dims has joined #openstack-infra | 14:32 | |
| *** mriedem is now known as dontrump | 14:32 | |
| fungi | rossella_s: thank you, i'll have a look | 14:32 |
| rossella_s | fungi thanks a lot :) | 14:33 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add playbook to manage power state of ironic servers https://review.openstack.org/284981 | 14:33 |
| *** Thelo has quit IRC | 14:34 | |
| *** [1]Thelo is now known as Thelo | 14:34 | |
| *** dansmith is now known as superdan | 14:36 | |
| *** zeih has joined #openstack-infra | 14:36 | |
| *** rlandy has joined #openstack-infra | 14:37 | |
| *** vgridnev has joined #openstack-infra | 14:37 | |
| *** hdd has quit IRC | 14:38 | |
| EmilienM | a patch merged in project config like 8 hours ago but JJB has not been updated everywhere I think https://review.openstack.org/#/c/290906/ http://logs.openstack.org/28/291728/1/check/gate-puppet-openstack-cookiecutter-puppet-3-cookiecutter-check-nv/f679b7d/console.html#_2016-03-11_14_11_50_104 | 14:38 |
| fungi | that's disheartening if so. i was up until after midnight my time getting jjb configs updated and in sync on all our jenkins masters | 14:40 |
| EmilienM | if you look at the job logs, the path is still not correct, something we fixed in project-config | 14:41 |
| fungi | i should be able to take a closer look in a few minutes | 14:43 |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/project-config: Add job to update the link for Neutron Review Dashboard https://review.openstack.org/284284 | 14:44 |
| *** amitgandhinz has joined #openstack-infra | 14:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 14:45 |
| jeblair | fungi, rossella_s: is that creating an actual gerrit dashboard? | 14:46 |
| *** BigWillie has joined #openstack-infra | 14:46 | |
| *** vgridnev has quit IRC | 14:46 | |
| rossella_s | jeblair, yes using gerrit-dash-creator | 14:46 |
| *** vgridnev has joined #openstack-infra | 14:46 | |
| jeblair | fungi, rossella_s: those can be hosted in gerrit itself, per-project | 14:47 |
| *** ajmiller_ has quit IRC | 14:48 | |
| rossella_s | jeblair, oh really? do you have any link or any pointer that explain how to do that? | 14:48 |
| *** jaosorior has quit IRC | 14:48 | |
| pabelanger | fungi: EmilienM: jenkins.o.o is correct | 14:49 |
| pabelanger | checking others now | 14:49 |
| jeblair | fungi, rossella_s: someone just needs to update jeepyb to support installing them | 14:49 |
| jeblair | fungi, rossella_s: https://review.openstack.org/Documentation/user-dashboards.html#project-dashboards | 14:50 |
| *** zz_dimtruck is now known as dimtruck | 14:50 | |
| jeblair | fungi, rossella_s: there's just stored under refs/meta/dashboards/... | 14:50 |
| rossella_s | jeblair, wow, didn't know that. Thanks for the hint, I will look into that! | 14:51 |
| rossella_s | fungi, jeblair, AJaeger I might bother you again on Monday...now I need to go, thanks a lot again for your help!! | 14:51 |
| mordred | jeblair, AJaeger, pabelanger, fungi: did we get anywhere with databases and nodepool and gate tests yesterday? | 14:51 |
| *** hieulq has quit IRC | 14:51 | |
| pabelanger | fungi: jenkins02, 03, 04, 05, 06 and 07 looks out of sync | 14:52 |
| zigo | pabelanger: Hi there! I was wondering if I could get voluptuous upgraded to its latest upstream version. | 14:52 |
| pabelanger | mordred: I think ianw submitted a fix last night | 14:52 |
| zigo | pabelanger: I need it for the latest version of tooz, which itself is needed by the latest Gnocchi. | 14:52 |
| zigo | pabelanger: Did you see the Debian bug I filed? | 14:52 |
| *** rbrndt__ has joined #openstack-infra | 14:52 | |
| *** sc68cal has joined #openstack-infra | 14:53 | |
| mordred | pabelanger: cool. I'll go look for it | 14:53 |
| pabelanger | mordred: 291493 I believe | 14:54 |
| pabelanger | zigo: upgraded where? | 14:54 |
| *** krtaylor has quit IRC | 14:54 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Display task notes from board https://review.openstack.org/291755 | 14:54 |
| jeblair | AJaeger: ^ also see discussion above about dashboards | 14:55 |
| mordred | pabelanger: https://jenkins04.openstack.org/job/nodepool-coverage/175/console coverage job is not happy | 14:55 |
| zigo | pabelanger: Upgrade the voluptuous package in Debian Sid. | 14:55 |
| zigo | pabelanger: You are one of the package maintainers of this package. | 14:55 |
| zigo | pabelanger: Which is why I'm asking you the permission to do it. | 14:56 |
| zigo | pabelanger: Though I don't have access to the DPMT Git repo, so I wont be able to upload my changes there. | 14:56 |
| pabelanger | zigo: Ya, it's likely going to my old email address. Let me check | 14:56 |
| mordred | pabelanger, AJaeger: what's the right way to get the db stuff installed for a coverage job? | 14:56 |
| pabelanger | zigo: I can also add you to update too | 14:56 |
| mordred | oh - looks like the db is installed on converage jobs ... | 14:57 |
| jeblair | mordred: i think there's a macro | 14:57 |
| pabelanger | mordred: that's what I am checking | 14:57 |
| mordred | so perhaps that's just related to "fungi: jenkins02, 03, 04, 05, 06 and 07 looks out of sync" | 14:57 |
| mordred | yah | 14:57 |
| zigo | pabelanger: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=817269 | 14:57 |
| openstack | Debian bug 817269 in src:voluptuous "Please package voluptuous >= 0.8.6" [Important,Open] | 14:57 |
| * mordred thinks pabelanger is already working on this :) | 14:57 | |
| *** rfolco has quit IRC | 14:57 | |
| *** Qiming has quit IRC | 14:58 | |
| pabelanger | mordred: not the outof sync part yet, I don't know how we deal with that ATM | 14:58 |
| mordred | kk | 14:58 |
| *** Qiming has joined #openstack-infra | 14:58 | |
| zigo | pabelanger: Even if you add me as Uploaders, I wont have Git write access to /git/python-module, so the only thing I can do is ugrade the package as an NMU. Is this fine by you? You could later on commit the changes. | 14:58 |
| *** Guest13749 is now known as mfisch | 14:58 | |
| *** mfisch is now known as Guest88074 | 14:59 | |
| pabelanger | mordred: Once we solve the sync issue with JJB, I belive it will be fixed | 14:59 |
| *** rossella_s has quit IRC | 14:59 | |
| mordred | k | 14:59 |
| mordred | SO | 14:59 |
| jeblair | pabelanger, mordred: yes, the job defn on 04 lacks the grant option which is in the current jjb config, so this is jenkins not updated | 14:59 |
| mordred | kk | 14:59 |
| mordred | woot | 14:59 |
| Shrews | neat | 15:00 |
| *** salv-orlando has joined #openstack-infra | 15:00 | |
| pabelanger | So, I don't know how we kick off resyncs manually, yolanda send me a playbook, but figure I'll hold off a bit and just watch it happen | 15:00 |
| pabelanger | zigo: sure, that works for me | 15:00 |
| yolanda | yes, the run_jjb playbook , do you want me to throw it? | 15:01 |
| *** Guest88074 is now known as mfisch | 15:01 | |
| *** mfisch has quit IRC | 15:01 | |
| *** mfisch has joined #openstack-infra | 15:01 | |
| jeblair | before we run it, we should see what fungi learned about the --workers option to jjb.... | 15:01 |
| *** jcoufal has quit IRC | 15:02 | |
| jeblair | he was trying it yesterday with 2 workers, but i haven't checked scrollback to learn how well that worked | 15:02 |
| pabelanger | zigo: FWIW: I don't think I have every upload voluptuous before. hashar might be a better person | 15:02 |
| fungi | jeblair: no conclusion. most of them were already in sync via puppet/ansible in the few minutes it took me to get around to start running them | 15:03 |
| zigo | hashar: Hi there! | 15:03 |
| fungi | jeblair: of the ones that weren't, jenkins.o.o took just shy of 30 minutes and jenkins01 needed several tries and about 4 hours of wall clock time | 15:03 |
| pabelanger | zigo: I'm actually surprised I have uploads :) | 15:04 |
| hashar | zigo: pabelanger: the Debian package i build for voluptuous was my very first deb package. It should have been migrated in the alioth git repos | 15:04 |
| zigo | hashar: Can you allow me to NMU a new upstream version of Voluptuous in Sid? | 15:04 |
| hashar | I might just have pilled a bunch of emails in the source package :-} | 15:04 |
| pabelanger | hashar: hehe | 15:04 |
| *** rfolco has joined #openstack-infra | 15:04 | |
| hashar | zigo: can you be added as a maintainer as well ? Your .deb skills are way above my league :} | 15:05 |
| jeblair | fungi: it doesn't sound like there was a 2x difference. | 15:05 |
| jeblair | fungi: it sounds like it was a 1x difference. | 15:05 |
| *** tonytan4ever has joined #openstack-infra | 15:05 | |
| zigo | hashar: Yeah, I can add myself, though I have lost access to the DPMT Git repository, so I wont be able to push the changes there. | 15:05 |
| pabelanger | Stepping out for an hour, have to run an errand. | 15:05 |
| fungi | jeblair: or even necessarily a discernible difference given our variability of performance | 15:05 |
| hashar | zigo: such a pity | 15:05 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:06 | |
| SamYaple | hello. who wants to review some shade code?! https://review.openstack.org/#/c/287504/ | 15:06 |
| hashar | zigo: should I craft a commit to add you to maintainer list and then add your commit and push it for you ? | 15:06 |
| fungi | jeblair: the only consistent number i managed to get out of the exercise is that when updating 0 jobs a full jjb run on most of our jenkins masters takes somewhere in the 7 minute range | 15:07 |
| *** kmartin has quit IRC | 15:07 | |
| zigo | hashar: I can send to your email some "git format-patch" series, yes. | 15:07 |
| fungi | jeblair: most of that time is spent compiling xml and checking it against the cache, i think | 15:07 |
| hashar | zigo: please spam me at hashar@free.fr and I will be more than happy to push ! | 15:07 |
| *** dims_ has joined #openstack-infra | 15:07 | |
| jeblair | fungi: yeah, i think so | 15:07 |
| zigo | hashar: Thanks, doing that right away. | 15:08 |
| *** dims has quit IRC | 15:08 | |
| jeblair | so it sounds like we shouldn't bother with --workers | 15:08 |
| hashar | zigo: the X-Vcs still reference subversion but afaik the team has moved to git http://anonscm.debian.org/cgit/python-modules/packages/voluptuous.git/ | 15:08 |
| fungi | jeblair: i'm inclined to agree | 15:08 |
| hashar | zigo: will push later tonight or tomorrow | 15:08 |
| *** xarses has quit IRC | 15:08 | |
| jeblair | i guess pabelanger isn't going to run it; yolanda did you want to? | 15:09 |
| yolanda | yes i can do it | 15:09 |
| fungi | jeblair: rossella_s: oh! i didn't realize all that was doing was creating gerrit dashboards. i figured there was a lot more going on | 15:09 |
| yolanda | infra-root, btw, can i get feedback about https://review.openstack.org/#/c/284462/ ? this is to prevent the cache corruption on jjb plays | 15:10 |
| fungi | mordred: on databases and nodepool tests, yes ianw discovered we needed with grant option so we updated all the jobs using the mysql-prep macro (thousands) to add that and then i made sure all the jenkins masters were in sync, so nodepool tests should be passing now | 15:11 |
| *** woodster_ has joined #openstack-infra | 15:11 | |
| nibalizer | yolanda: i don't see how you'd get into a place where jjb had a lockfile but didn't exist in the process table | 15:11 |
| *** dims_ has quit IRC | 15:11 | |
| nibalizer | maybe if the puppet cron hit a timeout | 15:11 |
| fungi | mordred: if it's still not passing, i suppose it's possible that just updating /etc/project-config and then rerunning jjb update isn't updating our jobs for some reason | 15:12 |
| yolanda | nibalizer, by your comments, you meant introducing a lockfile feature on jjb? | 15:12 |
| nibalizer | i think it was paul that suggested it but it makes a ton of sense to me | 15:13 |
| jeblair | (jjb doesn't need to know about the lockfile) | 15:13 |
| fungi | for everything else, there's flock(1) | 15:13 |
| yolanda | yep, pabelanger had some valid suggestions. Also limiting the number of jobs that jjb updates per run | 15:13 |
| *** hdd has joined #openstack-infra | 15:13 | |
| yolanda | so add a lockfile extra layer on our puppet jjb runs? | 15:13 |
| *** dims has joined #openstack-infra | 15:13 | |
| *** salv-orlando has quit IRC | 15:14 | |
| jeblair | we could do that. though, since jjb writes a cache, it may make sense for it to understand a lockfile internally. | 15:14 |
| nibalizer | why would you limit the number of jobs that jjb updates per run? | 15:14 |
| jeblair | write the lockfile next to the cache | 15:15 |
| nibalizer | that just sounds like you'd have all the masters in super inconsistent states | 15:15 |
| SamYaple | dont everyone jump on reviewing that shade patch all at once now | 15:15 |
| jeblair | nibalizer: to be fair, that's the actual result of what we get now with puppet | 15:15 |
| nibalizer | jeblair: nice | 15:15 |
| fungi | mordred: also i have a somewhat bleary-eyed recollection of grepping "with grant option" in our jenkins xml on one of the masters which claimed to have 0 jobs updated, and saw it in numerous jobs, so pretty sure at least my manual updates last night were complete | 15:15 |
| nibalizer | do we know why it takes 87 minutes to apply our jjb to a master? | 15:15 |
| nibalizer | is it becuase the master is busy or something? | 15:15 |
| fungi | nibalizer: yeah, calls to the jenkins api can be slooooow | 15:16 |
| yolanda | nibailzer it depends on the amount of jobs affected | 15:16 |
| fungi | and also sometimes break/timeout completely | 15:16 |
| *** salv-orlando has joined #openstack-infra | 15:16 | |
| fungi | when our masters get into their thread leak condition, it gets a lot worse too | 15:16 |
| jeblair | fungi: the job mordred was looking at does not have the grant option: https://jenkins04.openstack.org/job/nodepool-coverage/configure | 15:16 |
| *** ajmiller has joined #openstack-infra | 15:17 | |
| yolanda | i think it will be a good idea to add this lockfile internally to jjb, as concurrent runs have proved to break cache | 15:17 |
| *** harlowja_at_home has joined #openstack-infra | 15:17 | |
| jeblair | yolanda: wfm | 15:18 |
| *** thiagop is now known as thiagop-afk | 15:18 | |
| fungi | indeed it is missing on that one | 15:18 |
| jeblair | fungi: it does have it on jenkins.o.o | 15:18 |
| nibalizer | yolanda: i think thats a great idea | 15:18 |
| jeblair | fungi: and 01 and 02 | 15:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Highlight the notes button if a task has notes https://review.openstack.org/291404 | 15:19 |
| fungi | and on 07 | 15:20 |
| fungi | what gives?!? | 15:20 |
| fungi | i ran `sudo grep -ir 'with grant option' /var/lib/jenkins/jobs` on jenkins03 last night to double-check | 15:21 |
| jeblair | fungi: it's in gate-nodepool-python27 on jenkins03 | 15:22 |
| *** vincentfrancoise has joined #openstack-infra | 15:22 | |
| jeblair | fungi: but not in nodepool-coverage on jenkins03 | 15:22 |
| fungi | yeah | 15:22 |
| fungi | how does that happen? is caching wacky? | 15:22 |
| fungi | they should be using the same macro | 15:22 |
| jeblair | caching + workers=2 ? | 15:23 |
| fungi | and jjb said it updated 0 jobs when i ran it, implying the cache matched | 15:23 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715 | 15:23 |
| fungi | ohh, maybe --workers 2 bit me on the rump with this | 15:23 |
| AJaeger | jeblair: the challenge with rossella_s' dashboard is that neutron looks up first bugs in launchpad and prioritizes according to them. | 15:23 |
| AJaeger | jeblair: they want to generate a dashboard based on current bug priorities - so would need to update the gerrit dashboard daily | 15:24 |
| jeblair | AJaeger: yep, so we just need to figure out (a) where to run that and (b) how to push that to the special gerirt ref | 15:24 |
| fungi | so maybe some job which pushes updates into gerrit dashboards? | 15:24 |
| fungi | anyway, is anyone rerunning jjb on the jenkins masters yet? if not, i'll go ahead and fire it up again | 15:25 |
| *** kzaitsev_mb has quit IRC | 15:25 | |
| yolanda | fungi, i'm doing that | 15:26 |
| yolanda | jenkins.openstack.org finished | 15:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Highlight the notes button if a task has notes https://review.openstack.org/291404 | 15:26 |
| fungi | yolanda: okay, thanks | 15:26 |
| fungi | yolanda: without --workers 2 hopefully ;) | 15:26 |
| *** gongysh has quit IRC | 15:26 | |
| yolanda | just the run_jjb play | 15:26 |
| *** tsg has joined #openstack-infra | 15:26 | |
| fungi | i still haven't tried that one. how does it deal with jjb hitting jenkins api timeouts? does it loop rerunning it there until it completes, or do you end up rerunning it manually after that? | 15:27 |
| yolanda | i think it doesn't work good with timeouts | 15:28 |
| yolanda | have to launch again if that fails | 15:28 |
| fungi | running jjb on the jenkins masters is not hard. it's keeping on top of which ones broke and restarting them that's annoying | 15:28 |
| *** jcoufal has joined #openstack-infra | 15:28 | |
| AJaeger | jeblair: so, that's a system wide option to push a dashboard? Coool... That makes it much easier! | 15:28 |
| yolanda | fungi but is becoming a daily task lately... | 15:28 |
| *** sdake_ has quit IRC | 15:29 | |
| yolanda | jenkins01 finished | 15:29 |
| fungi | yolanda: well, for the past week we've been iterating on modifications that hit thousands of jobs | 15:29 |
| fungi | as we discover corner cases and need to adjust job macros | 15:29 |
| *** bdemers has quit IRC | 15:29 | |
| yolanda | yes, that massive reconfig is the cause, i know | 15:29 |
| fungi | i've been babysitting jjb runs almost every day to keep things in sync | 15:30 |
| *** kzaitsev_mb has joined #openstack-infra | 15:30 | |
| yolanda | yep, same here in the mornings, along with AJaeger | 15:30 |
| AJaeger | And I thought that yolanda's fix for the puppet-lint build was not touching that many jobs that we run into this ;( | 15:30 |
| fungi | oh, so we've been doing it twice a way? :/ | 15:31 |
| yolanda | looks as some days it was the case | 15:31 |
| fungi | er, twice a day | 15:31 |
| AJaeger | fungi, not every day as far as I remember. | 15:31 |
| *** sdake has joined #openstack-infra | 15:31 | |
| yolanda | not daily , just on massive jobs | 15:31 |
| fungi | i just meant we've had massive job updates almost daily for the past week, so i've resorted to running jjb manually for each of those | 15:32 |
| *** esikachev has quit IRC | 15:33 | |
| *** regXboi has joined #openstack-infra | 15:34 | |
| fungi | unfortunately today i've got a bunch of errands i need to disappear to take care of, which is why i was making sure things were synced up before i went to sleep last night (or so i thought anyway) | 15:34 |
| openstackgerrit | Russell Bryant proposed openstack/requirements: Update ovs to 2.5.0. https://review.openstack.org/291769 | 15:34 |
| fungi | sorry about that! | 15:34 |
| *** HeOS has quit IRC | 15:34 | |
| *** esikachev has joined #openstack-infra | 15:34 | |
| *** tonytan4ever has quit IRC | 15:35 | |
| *** salv-orlando has quit IRC | 15:35 | |
| *** xarses has joined #openstack-infra | 15:35 | |
| *** tonytan4ever has joined #openstack-infra | 15:35 | |
| fungi | my worry now is that we've got a cache out of sync with reality. so if the current run doesn't actually make any difference, we may need to blow away the cache and try again | 15:36 |
| AJaeger | fungi, I think on Tuesday morning it was just a jjb run on jenkins02 - the others were fine... | 15:36 |
| yolanda | it can take hours | 15:36 |
| *** vgridnev has quit IRC | 15:36 | |
| openstackgerrit | Russell Bryant proposed openstack/requirements: Add ovs with Python 3 support. https://review.openstack.org/291775 | 15:38 |
| *** esikachev has quit IRC | 15:39 | |
| *** annegentle has joined #openstack-infra | 15:39 | |
| *** fedexo has joined #openstack-infra | 15:40 | |
| *** HeOS has joined #openstack-infra | 15:40 | |
| AJaeger | should we schedule job approvals somehow? Or will we run into this JJB problems more often if we have them all together since more jobs get updated - and thus should not mass-approve? | 15:42 |
| ikutukov | Hello! I have a review https://review.openstack.org/#/c/288635/ that stuck in 'Patch in Merge Conflict’, i’ve tried rebasing it all possible way and the only result was ’no changes’, in local merge there is no conflicts. How i could troubleshoot what gone wrong in Jenkins merge process? Thank you! | 15:42 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715 | 15:43 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 15:44 |
| AJaeger | ikutukov: try the following: Create new branch from master HEAD, run git review -x 288635, send it in... | 15:44 |
| *** vgridnev has joined #openstack-infra | 15:44 | |
| ikutukov | Ok, thank you | 15:45 |
| *** Jeffrey4l has joined #openstack-infra | 15:45 | |
| *** jaypipes is now known as leakypipes | 15:45 | |
| AJaeger | ikutukov: that gives me a conflict on nailgun/nailgun/db/sqlalchemy/models/deployment_graph.py | 15:45 |
| AJaeger | ikutukov: you always need to pull from master first and rebase on top of master to resolve merge conflicts | 15:45 |
| *** claudiub|2 has quit IRC | 15:45 | |
| *** yamahata_ has joined #openstack-infra | 15:46 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Highlight the notes button if a task has notes https://review.openstack.org/291404 | 15:46 |
| ikutukov | There are a chain of dependant changes under this one, maybe this have played role | 15:47 |
| fungi | AJaeger: i have a feeling today's issue is more due to a bug in jjb, but we'll need some more focused attempts to reproduce before we can say for sure. i don't think it's necessary to schedule approvals for normal job additions/changes, but we do still need to be mindful of any which change widely-used templates/macros | 15:47 |
| ildikov | hi infra team | 15:48 |
| ildikov | I have a basic meeting orgainzation question | 15:48 |
| *** scheuran has quit IRC | 15:49 | |
| *** yamamoto has quit IRC | 15:49 | |
| ildikov | do I have to reserve a meeting slot if the meeting is only for one occasion or it is ok to just pick a channel which is available for that slot and organize/announce the meeting on the ML? | 15:49 |
| *** mtanino has joined #openstack-infra | 15:49 | |
| *** yamamoto has joined #openstack-infra | 15:49 | |
| openstackgerrit | Ana Krivokapic proposed openstack-infra/project-config: New project for TripleO UI https://review.openstack.org/291227 | 15:50 |
| fungi | ildikov: for a one-time meeting, i wouldn't reserve it as long as there's not a significant risk of someone else picking the same channel and time | 15:50 |
| *** denisra has joined #openstack-infra | 15:50 | |
| *** yamamoto has quit IRC | 15:50 | |
| fungi | ildikov: though if it's a cross-project meeting you may want to use #openstack-meeting-cp for it | 15:50 |
| ildikov | fungi: I guess if I pick a slot where multiple channels are available, then it should be ok then | 15:50 |
| fungi | ildikov: yeah, that seems safest | 15:50 |
| *** ifarkas has quit IRC | 15:51 | |
| fungi | that way it's easy to move the meeting to another available channel quickly in the case a conflict does arise | 15:51 |
| ildikov | fungi: hmm, I wasn't aware of #openstack-meeting-cp :) | 15:51 |
| *** yamamoto has joined #openstack-infra | 15:51 | |
| fungi | ildikov: it was created specifically to host cross-project meetings (for things like tc formed working groups, cross-project specs, et cetera) | 15:52 |
| ildikov | fungi: now I found it, it's just not listed explicitely on the wiki I found | 15:52 |
| fungi | oh, we might have forgotten to update the channel list in the wiki when that was created | 15:52 |
| anteaya | I seem to not be around at the moment and I will continue to not be around for a bit longer as I have errands to run in town | 15:53 |
| anteaya | yay logstash 2.0 | 15:53 |
| pabelanger | back | 15:53 |
| fungi | anteaya: i'm about to be the same way. enjoy errands (as much as you can, i guess) | 15:53 |
| ildikov | fungi: I will try to find a slot when that's available as I'm trying to organize a cross-project one and pick another which is also available as a backup option | 15:53 |
| ildikov | fungi: thanks much for the help! | 15:53 |
| pabelanger | jeblair: Ya, just holding off for a few days and just watch how it is done :) | 15:53 |
| anteaya | fungi: and to you | 15:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 15:53 |
| ildikov | fungi: is it ok, if I update the wiki too with this channel? | 15:54 |
| openstackgerrit | Sergey Belous proposed openstack-infra/project-config: Initial job for Nova with os-vif https://review.openstack.org/286085 | 15:54 |
| fungi | ildikov: i'm already doing it | 15:54 |
| ildikov | fungi: ok, cool, that was fast :) | 15:54 |
| ildikov | fungi: tnx again! :) | 15:54 |
| yolanda | mordred, around? did you see my proposal to fix glean tests? | 15:56 |
| *** yamamoto has quit IRC | 15:56 | |
| fungi | ildikov: i've added it to https://wiki.openstack.org/wiki/IRC and https://wiki.openstack.org/wiki/Meetings now | 15:56 |
| *** ldnunes has quit IRC | 15:57 | |
| *** tonytan_brb has joined #openstack-infra | 15:57 | |
| *** tonytan4ever has quit IRC | 15:58 | |
| ildikov | fungi: I cannot see it at the latter one also there is now an item that looks odd at the top of the page: #openstack-meeting-alt</code> | 15:59 |
| ildikov | fungi: I found this page: https://wiki.openstack.org/wiki/Meetings/CreateaMeeting | 15:59 |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Refactor manila check/gate queue https://review.openstack.org/291546 | 15:59 |
| fungi | ildikov: yeah, i mistyped, just a sec | 15:59 |
| ildikov | fungi: meeting channels are listed there as well | 15:59 |
| fungi | i should look closer at the preview | 15:59 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/infra-specs: Add extension to openstackci for next phase of work https://review.openstack.org/239810 | 16:00 |
| fungi | i'll update that too, thanks | 16:00 |
| ildikov | fungi: I hear ya, I mistype everything all the time | 16:00 |
| *** bdemers has joined #openstack-infra | 16:00 | |
| ildikov | fungi: np, at least my question turned out to be useful :) | 16:00 |
| *** eharney has joined #openstack-infra | 16:01 | |
| *** hieulq has joined #openstack-infra | 16:02 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Stop updating the board in-place when moving lanes https://review.openstack.org/291786 | 16:02 |
| fungi | ildikov: yep, corrected the typo and updated the other page you mentioned. thanks! | 16:03 |
| *** kencjohnston has joined #openstack-infra | 16:03 | |
| *** nwkarsten has quit IRC | 16:05 | |
| *** nwkarsten has joined #openstack-infra | 16:05 | |
| ildikov | fungi: cool, looks good now everywhere, tnx | 16:06 |
| *** thorst is now known as thorst_afk | 16:06 | |
| *** vgridnev has quit IRC | 16:06 | |
| *** ramishra has quit IRC | 16:08 | |
| *** kmartin has joined #openstack-infra | 16:08 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715 | 16:08 |
| *** esikachev has joined #openstack-infra | 16:08 | |
| *** ramishra has joined #openstack-infra | 16:09 | |
| *** nwkarsten has quit IRC | 16:10 | |
| *** vgridnev has joined #openstack-infra | 16:10 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Report number of changed jobs https://review.openstack.org/291789 | 16:10 |
| AJaeger | fungi, what do you think of the above debug output? ^ | 16:11 |
| *** Qiming has quit IRC | 16:11 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: DO NOT MERGE - Testing https://review.openstack.org/291790 | 16:12 |
| fungi | AJaeger: seems like a fine addition to me | 16:12 |
| *** yaume has quit IRC | 16:12 | |
| fungi | could also use grep -c instead of piping to wc -l | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 16:12 |
| AJaeger | will update... | 16:13 |
| jeblair | AJaeger: thank you :) | 16:13 |
| fungi | though also, looking for ^--- gets you the number of "removed or changed" jobs | 16:13 |
| AJaeger | fungi, that change showed me that the puppet-lint change touched 120 jobs - more than I thought | 16:13 |
| *** esikachev has quit IRC | 16:13 | |
| *** rhedlind has quit IRC | 16:14 | |
| *** xwizard_ has quit IRC | 16:14 | |
| fungi | and i wonder whether putting it after the potentially verbose diff output might make it easier to spot | 16:14 |
| *** hieulq has quit IRC | 16:14 | |
| *** akrivoka has joined #openstack-infra | 16:15 | |
| yolanda | jenkins02 failed when removing old jobs | 16:15 |
| yolanda | going to launch again | 16:15 |
| AJaeger | fungi, added as well. But if you have a better grep line, I can change it | 16:15 |
| AJaeger | I could also check for ^diff -r - see http://logs.openstack.org/85/286085/5/check/gate-project-config-compare-xml/ced0e9c/console.html#_2016-03-11_16_05_07_375 | 16:16 |
| *** skolekonov is now known as _skolekonov | 16:16 | |
| AJaeger | that adds a new job | 16:16 |
| * AJaeger changes the test.. | 16:16 | |
| *** nwkarsten has joined #openstack-infra | 16:16 | |
| openstackgerrit | Martin Oemke proposed openstack-infra/project-config: Enable Translation for tacker https://review.openstack.org/274855 | 16:17 |
| *** _nadya__ has quit IRC | 16:18 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715 | 16:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Report number of changed jobs https://review.openstack.org/291789 | 16:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: DO NOT MERGE - Testing https://review.openstack.org/291790 | 16:19 |
| dolphm | what does it take to get a channel added to eavesdrop? | 16:19 |
| *** bdemers_ has joined #openstack-infra | 16:20 | |
| *** markvoelker has quit IRC | 16:21 | |
| dolphm | i assumed all it would take would be adding it to accessbot https://review.openstack.org/#/c/289557/1/accessbot/channels.yaml | 16:21 |
| clarkb | dolphm: our meetbot is actually the bot that provides channel logging | 16:21 |
| clarkb | there is a hiera file to edit iirc let me get a link | 16:21 |
| dolphm | clarkb: ah thanks | 16:22 |
| clarkb | dolphm: https://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/common.yaml#n112 add to that list | 16:22 |
| dolphm | clarkb: thank you, will do! | 16:22 |
| *** bdemers has quit IRC | 16:23 | |
| clarkb | fungi: sounds like using multple workers may not actually update things properly? fun | 16:23 |
| openstackgerrit | Martin Oemke proposed openstack-infra/project-config: Enable Translation for tacker https://review.openstack.org/274855 | 16:24 |
| *** jlanoux has quit IRC | 16:24 | |
| *** asettle has joined #openstack-infra | 16:25 | |
| *** thiagop-afk_ has joined #openstack-infra | 16:26 | |
| openstackgerrit | Dolph Mathews proposed openstack-infra/system-config: Add #osic to meetbot https://review.openstack.org/291798 | 16:26 |
| clarkb | logstash 2.0 looks happy | 16:27 |
| clarkb | happier than 1.3 so far | 16:27 |
| openstackgerrit | Ana Krivokapic proposed openstack-infra/project-config: New project for TripleO UI https://review.openstack.org/291227 | 16:27 |
| bookwar | hi, there is a small patch to add delay-release option to couple of projects https://review.openstack.org/#/c/291390/ reviews welcome :) | 16:28 |
| *** mikelk has quit IRC | 16:28 | |
| yolanda | jenkins04 finished | 16:28 |
| *** sbelous_ has quit IRC | 16:28 | |
| *** HeOS has quit IRC | 16:29 | |
| *** nwkarsten has quit IRC | 16:29 | |
| *** nwkarsten has joined #openstack-infra | 16:30 | |
| *** asettle has quit IRC | 16:30 | |
| rcarrillocruz | btw yolanda, fungi, i have this change https://review.openstack.org/#/c/280189/5/playbooks/run_jjb.yaml | 16:32 |
| rcarrillocruz | not sure if taht would've help on the jjb runs you are doing | 16:32 |
| yolanda | ah i remember that | 16:32 |
| *** zeih_ has joined #openstack-infra | 16:32 | |
| *** penguinolog has quit IRC | 16:32 | |
| yolanda | adding flush cache , or workers, looks useful to me | 16:32 |
| *** bpokorny has joined #openstack-infra | 16:33 | |
| clarkb | bookwar: I don't see delay-release as a jeepyb flag, any idea what that affects? | 16:33 |
| bookwar | clarkb: bot which changes bug status when patch is merged | 16:33 |
| clarkb | yolanda: we used workers yesterday and it is why we are in this situation | 16:33 |
| bookwar | clarkb: i'll try to find a reference.. | 16:33 |
| clarkb | yolanda: so I don't think we should use that until it is working properly | 16:34 |
| clarkb | yolanda: and flush cache means we will update all jobs every time which we definitely do not want | 16:34 |
| *** sdake_ has joined #openstack-infra | 16:34 | |
| yolanda | clarkb, workers was a sample... rcarrillocruz change is a feature for enabling additional parameters | 16:34 |
| yolanda | so you can pass that on demand | 16:34 |
| bookwar | clarkb: http://lists.openstack.org/pipermail/openstack-dev/2015-December/081724.html | 16:34 |
| clarkb | yolanda: I am just pointing out I don't think we want either of those flags | 16:34 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test https://review.openstack.org/289445 | 16:35 |
| *** zeih has quit IRC | 16:35 | |
| yolanda | clarkb, --flush-cache can be useful to be able to run jjb with a clean cache, in case we detect corruption | 16:35 |
| yolanda | right now, if that happens, we need to run it directly on jenkins | 16:35 |
| *** nwkarsten has quit IRC | 16:35 | |
| clarkb | yolanda: jjb can and should address corruption direclt | 16:35 |
| clarkb | y | 16:35 |
| yolanda | clarkb how it can? because we've hit that cache corruption quite often | 16:36 |
| *** sdake has quit IRC | 16:36 | |
| clarkb | yolanda: the cache corrupts when puppet times it outand kills it | 16:36 |
| clarkb | yolanda: one way to avoid that is to write it to a tmpfile until completion then mv it over | 16:36 |
| clarkb | and if on start a corrupt cache is found jjb should just delete it | 16:37 |
| yolanda | well yes, but that happens... i agreed to work on some lockfile for jjb, but i think that having the ability to send some extra parameters on jjb runs cannot hurt | 16:37 |
| *** nikhil has quit IRC | 16:37 | |
| clarkb | I am not saying that | 16:37 |
| *** zeih_ has quit IRC | 16:37 | |
| *** bdemers_ has quit IRC | 16:37 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add reno jobs to some Puppet OpenStack modules https://review.openstack.org/291805 | 16:37 |
| clarkb | I am saying those two flags shoudl be considered harmful for these reasons | 16:37 |
| *** bdemers has joined #openstack-infra | 16:37 | |
| clarkb | and both should be fixed in jjb | 16:38 |
| yolanda | i agree with you, jjb should not arrive to fail due to corrupted cache | 16:38 |
| *** dims has quit IRC | 16:38 | |
| yolanda | jenkins02 finished now | 16:38 |
| yolanda | i want to work on that next week | 16:38 |
| *** matrohon has quit IRC | 16:38 | |
| *** apoorvad has joined #openstack-infra | 16:39 | |
| *** apoorvad has quit IRC | 16:39 | |
| *** vgridnev has quit IRC | 16:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Report number of changed jobs https://review.openstack.org/291789 | 16:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: DO NOT MERGE - Testing https://review.openstack.org/291790 | 16:40 |
| clarkb | the worker one is odd, it claimed ot update but apparently didn't | 16:40 |
| clarkb | maybe it isn't splitting the job uploads properly between workers? | 16:40 |
| clarkb | so some subset of jobs gets updated but the others dont | 16:40 |
| *** apoorvad has joined #openstack-infra | 16:40 | |
| *** pblaho has quit IRC | 16:42 | |
| *** vgridnev has joined #openstack-infra | 16:43 | |
| yolanda | how is worker feature supposed to work? it divides the load of jobs to update? | 16:43 |
| *** zul_ has quit IRC | 16:43 | |
| clarkb | ya, I think it uses a single thread to compile and determine which jobs need updates then it splits the update of those jobs between different threads | 16:43 |
| yolanda | maybe there are races updating the cache? | 16:44 |
| *** ikutukov has quit IRC | 16:44 | |
| yolanda | ok all jenkins claimed to be updated now | 16:44 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Add NFS devstack plugin https://review.openstack.org/289589 | 16:45 |
| *** zul has joined #openstack-infra | 16:45 | |
| *** esikachev has joined #openstack-infra | 16:47 | |
| pabelanger | just a heads up, will be offline a few hours this afternoon. Should be back my 20:00 UTC | 16:47 |
| openstackgerrit | Ana Krivokapic proposed openstack-infra/project-config: New project for TripleO UI https://review.openstack.org/291227 | 16:48 |
| *** krtaylor has joined #openstack-infra | 16:48 | |
| *** automagically_ has joined #openstack-infra | 16:48 | |
| *** tonytan_brb is now known as tonytan4ever | 16:48 | |
| automagically_ | FYI, seeing problems download check log: http://logs.openstack.org/93/291793/1/check/gate-openstack-ansible-os_glance-ansible-func-ubuntu-trusty/65abd76/console.html, I get http://logs.openstack.org/93/291793/1/check/gate-openstack-ansible-os_glance-ansible-func-ubuntu-trusty/65abd76/console.html#_2016-03-11_16_22_11_587 and nothing more. Been seeing similar issues throughout the week | 16:48 |
| *** pcaruana has quit IRC | 16:50 | |
| *** dims has joined #openstack-infra | 16:50 | |
| *** ashleighfarnham has joined #openstack-infra | 16:50 | |
| *** salv-orlando has joined #openstack-infra | 16:51 | |
| clarkb | automagically_: wgetting that file similarly fails which means it isn't the wsgi app that annotates the file failing | 16:51 |
| *** abregman|afk is now known as abregman | 16:53 | |
| clarkb | thta file doesn't exist on disk on the log server which implies it is a swift upload | 16:54 |
| automagically | appreciate you looking into it clarkb | 16:54 |
| clarkb | either we aren't downloading a complete console log, we aren't uplodaing a complete console log, or swift is funny | 16:54 |
| *** automagically_ has left #openstack-infra | 16:54 | |
| SamYaple | doesnt it have to be one? | 16:54 |
| SamYaple | or does it rather | 16:54 |
| clarkb | I think the fact that the connection attempts to continue implies swift is being funny | 16:54 |
| automagically | And not funny haha | 16:55 |
| *** tesseract- has quit IRC | 16:55 | |
| clarkb | baically swift is saying there is more data here and it is never showing up | 16:55 |
| * clarkb looks at more logs | 16:55 | |
| *** thiagop-afk_ has quit IRC | 16:55 | |
| *** annegentle has quit IRC | 16:56 | |
| fungi | more like "funny uhoh" | 16:56 |
| *** dtantsur is now known as dtantsur|afk | 16:56 | |
| yolanda | AJaeger, my puppet change passed. The fix for puppet-lint worked fine | 16:56 |
| fungi | anyway, have fun with that. i need to go do errands for a while now. should be back in a few hours worst case | 16:56 |
| clarkb | http://paste.openstack.org/show/490201/ | 16:57 |
| *** dims_ has joined #openstack-infra | 16:58 | |
| automagically | Same issue with: http://logs.openstack.org/93/291793/1/check/gate-openstack-ansible-os_glance-ansible-func-ubuntu-trusty/04768ad/console.html | 16:58 |
| clarkb | should also find the console log on the jenkins master /me looks for it | 16:58 |
| *** vincentll has quit IRC | 16:59 | |
| *** dims has quit IRC | 16:59 | |
| *** ashleighfarnham has quit IRC | 17:00 | |
| clarkb | automagically: https://jenkins04.openstack.org/job/gate-openstack-ansible-os_glance-ansible-func-ubuntu-trusty/7/console is the full log on the jenkins side | 17:01 |
| automagically | Thank you sir | 17:01 |
| clarkb | automagically: there are no errors reported by the swift upload | 17:01 |
| *** Jeffrey4l has quit IRC | 17:01 | |
| *** rcernin has quit IRC | 17:01 | |
| *** samuelBartel has quit IRC | 17:01 | |
| clarkb | and the file downloaded to be uploaded to swift reports 2423kb | 17:01 |
| *** vincentfrancoise has left #openstack-infra | 17:02 | |
| clarkb | which is much larger than what we are getting back out of swift again | 17:02 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add reno jobs to some Puppet OpenStack modules https://review.openstack.org/291805 | 17:02 |
| clarkb | I will check swift directly next | 17:03 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add domain_id param to project operations https://review.openstack.org/291701 | 17:04 |
| *** e0ne has quit IRC | 17:05 | |
| *** BigWillie has quit IRC | 17:05 | |
| *** bpokorny has quit IRC | 17:06 | |
| automagically | clarkb, thanks again. That Jenkins log link gave me everything I needed | 17:08 |
| *** HeOS has joined #openstack-infra | 17:08 | |
| automagically | Big props to the tireless efforts of the infra team | 17:08 |
| *** dims_ has quit IRC | 17:08 | |
| *** markvoelker has joined #openstack-infra | 17:09 | |
| *** salv-orlando has quit IRC | 17:09 | |
| yolanda | pabelanger, more failures with vexxhost, git failures | 17:09 |
| clarkb | judging by the speed at which the file brwoser thing is going we may be giving it a sad | 17:09 |
| clarkb | might have to switch to the cli client | 17:09 |
| pleia2 | good morning | 17:10 |
| clarkb | pleia2: good morning | 17:10 |
| nibalizer | ohai pleia2 | 17:10 |
| *** roxanagh_ has joined #openstack-infra | 17:10 | |
| *** annegentle has joined #openstack-infra | 17:11 | |
| *** gyee has joined #openstack-infra | 17:11 | |
| clarkb | 14048946 files using about 1.5TB of disk | 17:12 |
| jlvillal | Should the gate start using pip > 8 again? I notice it keeps using pip 7.1.2. I had posted this patch: https://review.openstack.org/#/c/271526/ to allow it to follow the pip releases again. | 17:12 |
| *** abregman is now known as abregman|afk | 17:13 | |
| *** dims has joined #openstack-infra | 17:13 | |
| clarkb | jlvillal: probably | 17:13 |
| clarkb | I will approve it now | 17:13 |
| jlvillal | clarkb: Thanks! | 17:13 |
| clarkb | automagically: I can get the full file from swift directly | 17:16 |
| clarkb | maybe it is something in our wsgi app | 17:16 |
| *** sdague has quit IRC | 17:17 | |
| automagically | clarkb: Hmm…probably worth opening a bug about, since I know I’ve seen it at least 5-8 times this week | 17:17 |
| automagically | And it prompts a recheck typically, which just consumes more resources | 17:17 |
| *** MarkAtwood has joined #openstack-infra | 17:17 | |
| *** hashar has quit IRC | 17:19 | |
| pabelanger | and back | 17:20 |
| clarkb | os-loganalyze and swiftclient are both older, so I don't think a code change has caused this | 17:20 |
| pabelanger | yolanda: okay, lets ask fungi or clarkb who the vexxhost contact it again | 17:20 |
| clarkb | pabelanger: yolanda: mnaser | 17:20 |
| pabelanger | clarkb: thanks | 17:20 |
| *** ldnunes has joined #openstack-infra | 17:21 | |
| pabelanger | mnaser: do you have a moment to help troubleshoot some IO issue we are seeing on vexxhost? | 17:21 |
| *** arxcruz has quit IRC | 17:21 | |
| *** BobBall_ has joined #openstack-infra | 17:21 | |
| mnaser | finishing up lunch | 17:21 |
| mnaser | afterwards sure! | 17:21 |
| *** BobBall_ is now known as Guest45383 | 17:21 | |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Pin pip to less than version 8 in install_puppet.sh" https://review.openstack.org/271526 | 17:22 |
| *** jordanP has quit IRC | 17:23 | |
| *** BobBall has quit IRC | 17:24 | |
| *** vgridnev has quit IRC | 17:25 | |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite: fix module to work with latest mysql version https://review.openstack.org/275247 | 17:25 |
| *** Guest45383 has quit IRC | 17:26 | |
| *** BobBall has joined #openstack-infra | 17:26 | |
| *** BobBall is now known as Guest65124 | 17:26 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Edit task notes from board https://review.openstack.org/291755 | 17:27 |
| *** vgridnev has joined #openstack-infra | 17:27 | |
| *** derekh has quit IRC | 17:27 | |
| *** leakypipes has quit IRC | 17:30 | |
| *** nwkarsten has joined #openstack-infra | 17:31 | |
| mnaser | pabelanger: im a bit busy with something for most of the day today, however, if you have a list of instance id's that have been getting these problems, i could likely track them down and see if (for example) it's a bad host or something. | 17:34 |
| *** Guest65124 is now known as GnomeKing | 17:34 | |
| *** aysyd has quit IRC | 17:34 | |
| *** jaypipes has joined #openstack-infra | 17:34 | |
| automagically | clarkb: Yet another example: http://logs.openstack.org/93/291793/2/check/gate-openstack-ansible-os_glance-ansible-func-ubuntu-trusty/4cd5c53/console.html | 17:34 |
| *** krotscheck has quit IRC | 17:34 | |
| *** aysyd has joined #openstack-infra | 17:34 | |
| clarkb | pabelanger: you can get that from the nodepool log by grepping for the nodepool ids of those servers, that should get you vexxhost uuids | 17:35 |
| clarkb | mtreinish: looking at logstash the push-subunit job claims to have 253 jobs running but only 2 workers | 17:35 |
| *** pcaruana has joined #openstack-infra | 17:35 | |
| clarkb | mtreinish: not sure if that is expected but could be source of the OOMs if it is actually running more than one job per worker | 17:35 |
| *** nwkarsten has quit IRC | 17:36 | |
| *** krotscheck has joined #openstack-infra | 17:37 | |
| clarkb | automagically: my first suggestion would be to switch away from swift logging for that job | 17:37 |
| *** ericksonsantos has quit IRC | 17:37 | |
| clarkb | automagically: just to work around the problem for now while we sort out what is causing it | 17:37 |
| clarkb | I cna push that chnage up if you are ok with it | 17:38 |
| automagically | clarkb: I’m good with that | 17:38 |
| *** vgridnev has quit IRC | 17:38 | |
| *** arxcruz has joined #openstack-infra | 17:39 | |
| automagically | Again, appreciate your help | 17:40 |
| automagically | clarkb: Will they just be accessible via Jenkins? | 17:40 |
| *** admin0 has joined #openstack-infra | 17:41 | |
| *** stevelle has joined #openstack-infra | 17:41 | |
| *** bpokorny has joined #openstack-infra | 17:41 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Ansible role jobs copy console to disk not swift https://review.openstack.org/291834 | 17:41 |
| *** vgridnev has joined #openstack-infra | 17:42 | |
| clarkb | automagically: no, they will still be accessible via logs.openstack.org, just changing the backend that we store the data in | 17:42 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Edit task notes from board https://review.openstack.org/291755 | 17:42 |
| automagically | Ah okay, clarkb | 17:42 |
| clarkb | automagically: right now we have the ability to save to a 12TB disk array or to a swift container | 17:42 |
| *** sfinucan has quit IRC | 17:42 | |
| automagically | I’m +1 on that | 17:43 |
| automagically | And let the community in #openstack-ansible know | 17:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Make all configuration in-repo configuration https://review.openstack.org/268386 | 17:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix race in tests https://review.openstack.org/290903 | 17:45 |
| *** dizquierdo has joined #openstack-infra | 17:46 | |
| openstackgerrit | Merged openstack-infra/zuul: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/291515 | 17:47 |
| *** kzaitsev_mb has quit IRC | 17:48 | |
| clarkb | pleia2: fungi: for the sprint we are sharing with ironic and qa again or is that copy pasta from last time? | 17:48 |
| *** ildikov has quit IRC | 17:48 | |
| *** bpokorny has quit IRC | 17:49 | |
| *** bdemers has quit IRC | 17:49 | |
| jeblair | there's something strange going on with nodepool allocations | 17:50 |
| *** bpokorny has joined #openstack-infra | 17:50 | |
| jeblair | we've been consistently under capacity for a while, but we still have post jobs waiting for hours | 17:50 |
| pleia2 | clarkb: copy pasta, fixing :) | 17:50 |
| clarkb | jeblair: its because we have a large extra ready node | 17:50 |
| *** baoli_ has quit IRC | 17:50 | |
| clarkb | which may be due to bare-trusty? not sure if we have cleaned that out yet | 17:51 |
| *** lucasagomes is now known as lucas-afk | 17:51 | |
| *** stevelle has left #openstack-infra | 17:51 | |
| jeblair | clarkb: yeah, there are a bunch of bare-trusty nodes than are 4h ready | 17:51 |
| *** EricGonc_ has joined #openstack-infra | 17:52 | |
| jeblair | clarkb: ah, we need a zuul geard restart now don't we? | 17:52 |
| clarkb | jeblair: yes I think that is still pending | 17:52 |
| *** _nadya_ has joined #openstack-infra | 17:52 | |
| jeblair | clarkb: which dovetails nicely into why i was checking that page anyway | 17:52 |
| jeblair | clarkb: i want to restart it to pick up jhesketh's cache change (and there's an outside chance that may have an effect on the leak too, though i don't think so) | 17:53 |
| jeblair | so i will restart zuul now | 17:53 |
| *** nwkarsten has joined #openstack-infra | 17:53 | |
| clarkb | then we can tell nodepool to delete any leftover bare-trusty | 17:53 |
| jeblair | yep | 17:53 |
| clarkb | crinkle: re summit planning, what do you think the hcances are of infra-cloud hosts being in their new home by summit time? we could possibly use the sprint to bring it back up again and put it into production | 17:54 |
| clarkb | crinkle: I know west is sort of in a weird layer 8 state but maybe we can do east? | 17:55 |
| *** EricGonczer_ has quit IRC | 17:55 | |
| jeblair | zuul restarted; nodepool deleting | 17:55 |
| * clarkb adds it to the list, worst case we pick other things off the list | 17:55 | |
| jeblair | clarkb: i like the idea | 17:55 |
| *** e0ne has joined #openstack-infra | 17:56 | |
| *** vgridnev has quit IRC | 17:56 | |
| jeblair | all bare-trusty nodes are in delete state | 17:56 |
| *** samuelBartel has joined #openstack-infra | 17:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Report number of changed jobs https://review.openstack.org/291789 | 17:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: DO NOT MERGE - Testing https://review.openstack.org/291790 | 17:57 |
| pabelanger | clarkb: sure, let me check now | 17:57 |
| *** auggy is now known as frauggy | 17:57 | |
| *** nwkarsten has quit IRC | 17:57 | |
| clarkb | pabelanger: note the nodepool log is on an 8 hour rotation so you may have to look at the previous several log files | 17:57 |
| AJaeger | yolanda: great to hear that the puppet-lint works! | 17:58 |
| pabelanger | clarkb: okay | 17:58 |
| crinkle | clarkb: if everything goes as expected then at least east will be rehomed by the summit (but i worry about counting on things going as expected) | 17:58 |
| *** esikache1 has joined #openstack-infra | 17:58 | |
| *** esikachev has quit IRC | 17:58 | |
| clarkb | crinkle: thats fair, I put it on the list and we can tackle it if we are able | 17:59 |
| crinkle | clarkb: timeline was about three weeks from last monday which gives a couple extra weeks of buffer | 17:59 |
| *** sarob has joined #openstack-infra | 18:00 | |
| *** asettle has joined #openstack-infra | 18:00 | |
| *** tongli has joined #openstack-infra | 18:01 | |
| *** yamahata_ has quit IRC | 18:01 | |
| *** vgridnev has joined #openstack-infra | 18:01 | |
| *** samuelBartel has quit IRC | 18:02 | |
| openstackgerrit | Merged openstack-infra/zuul: Fix race in tests https://review.openstack.org/290903 | 18:02 |
| *** dims_ has joined #openstack-infra | 18:04 | |
| *** dims has quit IRC | 18:04 | |
| *** asettle has quit IRC | 18:05 | |
| *** tphummel has joined #openstack-infra | 18:05 | |
| *** dizquierdo has quit IRC | 18:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 18:06 |
| *** ihrachys has quit IRC | 18:06 | |
| *** admin0 has left #openstack-infra | 18:07 | |
| *** vgridnev has quit IRC | 18:08 | |
| *** annegentle has quit IRC | 18:09 | |
| *** sbelous_ has joined #openstack-infra | 18:09 | |
| *** annegentle has joined #openstack-infra | 18:09 | |
| *** annegentle has quit IRC | 18:10 | |
| *** rossella_s has joined #openstack-infra | 18:10 | |
| *** icey has quit IRC | 18:11 | |
| *** flepied has quit IRC | 18:11 | |
| *** dizquierdo has joined #openstack-infra | 18:11 | |
| *** mriedem has joined #openstack-infra | 18:12 | |
| * AJaeger sees a NOT_REGISTERED for gate-bindep-fallback-devstack-centos7 in zuul | 18:12 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 18:13 |
| AJaeger | We didn't change that job AFAIK | 18:13 |
| clarkb | AJaeger: zuul was just restarted, we may not have had a devstack-centos7 ready to register | 18:13 |
| AJaeger | clarkb: ah... | 18:13 |
| AJaeger | ok | 18:13 |
| *** icey has joined #openstack-infra | 18:13 | |
| *** binbincong has quit IRC | 18:13 | |
| *** dontrump has quit IRC | 18:13 | |
| *** shardy has quit IRC | 18:13 | |
| *** baoli has joined #openstack-infra | 18:14 | |
| *** nwkarsten has joined #openstack-infra | 18:14 | |
| *** fernnest_ has joined #openstack-infra | 18:14 | |
| *** fernnest has quit IRC | 18:15 | |
| *** vgridnev has joined #openstack-infra | 18:15 | |
| *** fedexo has quit IRC | 18:16 | |
| *** annegent_ has joined #openstack-infra | 18:16 | |
| *** hdd has quit IRC | 18:17 | |
| *** noorul has joined #openstack-infra | 18:17 | |
| *** annegent_ has quit IRC | 18:17 | |
| *** nwkarsten has quit IRC | 18:18 | |
| *** annegentle has joined #openstack-infra | 18:18 | |
| noorul | hi | 18:20 |
| clarkb | hello | 18:20 |
| noorul | I am trying to setup openstack ci locally | 18:20 |
| noorul | I think I am almost there | 18:20 |
| *** binbincong has joined #openstack-infra | 18:21 | |
| *** hdd has joined #openstack-infra | 18:21 | |
| *** esikache1 has quit IRC | 18:21 | |
| noorul | Once I add new project information to project-config repo, I suppose I should run puppet apply on gerrit machine | 18:21 |
| clarkb | AJaeger: ya no ready or used centos7 right now | 18:21 |
| clarkb | one building in internap | 18:21 |
| noorul | So that it adds this new project to gerrit | 18:22 |
| *** sbelous_ has quit IRC | 18:22 | |
| *** jistr has quit IRC | 18:22 | |
| noorul | puppet apply fails at /usr/local/bin/manage-projects -d -l /var/log/manage_projects.log | 18:22 |
| *** austin81 has joined #openstack-infra | 18:22 | |
| *** degorenko is now known as _degorenko|afk | 18:22 | |
| clarkb | that is what adds the project, check the log file there | 18:23 |
| clarkb | could be authentication or permissions issue | 18:23 |
| noorul | /usr/local/lib/python2.7/dist-packages/paramiko/client.py:645: UserWarning: Unknown ssh-rsa host key for | 18:23 |
| clarkb | that means that you need to accept the ssh-rsa key for gerrit | 18:23 |
| AJaeger | clarkb: thanks... | 18:23 |
| *** annegent_ has joined #openstack-infra | 18:24 | |
| *** resker has quit IRC | 18:24 | |
| *** abregman|afk has quit IRC | 18:24 | |
| *** esikache1 has joined #openstack-infra | 18:24 | |
| *** annegent_ has quit IRC | 18:24 | |
| noorul | clarkb: Not sure what needs to be done | 18:24 |
| *** annegent_ has joined #openstack-infra | 18:25 | |
| clarkb | noorul: you have to accept the ssh host key for the gerrit service | 18:25 |
| clarkb | as the user running manage-projects (probably root) ssh to gerrit's ssh server (so port 29418 not 22) and accept the key | 18:25 |
| *** austin81 has left #openstack-infra | 18:26 | |
| *** rbrndt__ has quit IRC | 18:26 | |
| AJaeger | so, my change for reporting the number of changed jenkins jobs is ready for review if anybody wants some more information: https://review.openstack.org/#/c/291789 | 18:26 |
| *** annegentle has quit IRC | 18:28 | |
| *** sdake_ has quit IRC | 18:28 | |
| noorul | clarkb: I am not sure which key and user it is using to ssh | 18:28 |
| clarkb | noorul: if using our puppet modules it will be root | 18:29 |
| noorul | and the key /home/gerrit2/review_site/etc/ssh_host_rsa_key ? | 18:29 |
| clarkb | yes thats the key, but you need the ssh client side to store it in known hosts | 18:29 |
| clarkb | so just run the ssh command against it | 18:29 |
| *** asselin has left #openstack-infra | 18:30 | |
| noorul | But I get access denied | 18:30 |
| clarkb | thats fine, did you accept the host key? if so that problem is solved | 18:30 |
| clarkb | as for access denied you probably have to set the username for the remote side | 18:30 |
| clarkb | since root isn't going to be a user in gerrit | 18:30 |
| *** tsg has quit IRC | 18:30 | |
| *** vgridnev has quit IRC | 18:30 | |
| pabelanger | clarkb: mnaser: I've added 3 nodes for veexhost that are failing. https://etherpad.openstack.org/p/vexxhost-io-timeouts | 18:30 |
| pabelanger | trying to find a few with slow apt-get install issues | 18:31 |
| *** binbincong has quit IRC | 18:31 | |
| noorul | but when using puppet it will anyhow use root, right? | 18:32 |
| clarkb | noorul: it will run the command as root but you can always spcify the remote username | 18:32 |
| noorul | clarkb: Where should I specify that? | 18:32 |
| *** binbincong has joined #openstack-infra | 18:33 | |
| clarkb | noorul: it is part of the jeepyb config iirc | 18:33 |
| noorul | clarkb: I am not sure whether I missed any step here | 18:34 |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Update upper-constraints for fairy-slipper 0.2.0 https://review.openstack.org/291853 | 18:34 |
| *** claudiub|2 has joined #openstack-infra | 18:34 | |
| *** nwkarsten has joined #openstack-infra | 18:34 | |
| noorul | I am following http://docs.openstack.org/infra/system-config/running-your-own.html | 18:35 |
| *** arxcruz has quit IRC | 18:35 | |
| noorul | clarkb: I am trying to find out where is jeepyb config located | 18:36 |
| *** nwkarste_ has joined #openstack-infra | 18:36 | |
| noorul | I am running this from gerrit node | 18:37 |
| clarkb | noorul: looks like you set the user in the projects.yaml | 18:37 |
| clarkb | gerrit-user: foo | 18:37 |
| *** dtardivel has quit IRC | 18:38 | |
| clarkb | noorul: so hceck your projects.yaml and project.ini | 18:38 |
| clarkb | thats the config jeepyb/manage-projects uses | 18:38 |
| *** sdake has joined #openstack-infra | 18:39 | |
| AJaeger | wow, still no dsvm-centos7 nodes ;( | 18:39 |
| clarkb | noorul: that is in /home/gerrit2 | 18:39 |
| *** nwkarsten has quit IRC | 18:39 | |
| pabelanger | AJaeger: did we loose it? | 18:39 |
| *** thiagop-afk is now known as thiagop | 18:39 | |
| clarkb | AJaeger: the one in internap has been building for an hour :/ | 18:39 |
| clarkb | and I tried deleting the one in ovh that is in a deleting state manualy and that failed | 18:40 |
| openstackgerrit | Simon McCartney proposed openstack-infra/infra-ansible: Reworked setup_env.sh to support Debian & Darwin/OSX https://review.openstack.org/291658 | 18:40 |
| *** claudiub has joined #openstack-infra | 18:40 | |
| AJaeger | pabelanger: NOT_REGISTERED for any job that uses dsvm-centos7 ;( | 18:40 |
| clarkb | I can try deleting the one that is building | 18:40 |
| *** sdake_ has joined #openstack-infra | 18:41 | |
| AJaeger | min-ready is 1 for that one | 18:41 |
| *** nwkarste_ has quit IRC | 18:41 | |
| clarkb | AJaeger: we may want to rebalance that to a higher number | 18:41 |
| clarkb | ok we have one ready now | 18:41 |
| *** yamahata_ has joined #openstack-infra | 18:41 | |
| AJaeger | great, thanks | 18:42 |
| *** claudiub|2 has quit IRC | 18:42 | |
| AJaeger | we also do not run dsvm-centos7 in all clouds. Is that intentional? | 18:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Move gerrit logic from source to connection https://review.openstack.org/291854 | 18:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Sever the forward reference from connections to their users https://review.openstack.org/291855 | 18:42 |
| AJaeger | rax is missing | 18:42 |
| clarkb | AJaeger: may be due to dib and glean | 18:42 |
| clarkb | we probably could add it to rax now if we test that networking is configured properly on boot | 18:42 |
| *** sputnik13 has joined #openstack-infra | 18:43 | |
| pabelanger | clarkb: Once ianw gets a new release of diskimage-builder we can do 290974 | 18:44 |
| pabelanger | last I checked centos-minimal was working on rax, according to ianw | 18:44 |
| *** sdake has quit IRC | 18:44 | |
| clarkb | pabelanger: that has the opposite problem of only adding it to rax :) | 18:44 |
| *** thorst_afk is now known as thorst | 18:44 | |
| clarkb | but ya with that in place we can rax and everywhere else because glean | 18:45 |
| pabelanger | clarkb: Ya, I think ianw wanted to get it working on rax first, then other coulds. Since he has always battled rax in the past | 18:45 |
| AJaeger | 817 jobs run on devstack-centos7 - complete distribution (but one day old tree): http://paste.openstack.org/show/490223/ | 18:45 |
| mnaser | pabelanger does git.openstack.org run on ipv6 too? | 18:45 |
| AJaeger | So, more devstack-centos7 (817) jobs than devstack-trusty (539)? | 18:46 |
| jeblair | mnaser: yes | 18:46 |
| pabelanger | mnaser: ya, it resolves to 2001:4800:7819:103:be76:4eff:fe06:63c | 18:47 |
| slagle | hi, we're seeing NOT_REGISTERED for all of the gate-tripleo-* jobs | 18:47 |
| mnaser | pabelanger jeblair ok, we've been working on bringing ipv6 up and we noticed a few networking issues that we're trying to iron out. i suspect that is related. | 18:47 |
| AJaeger | clarkb: what value do you suggest for devstack-centos7 min-ready instead of 1? | 18:47 |
| SamYaple | shameless begging for an ack review on my shade service w/ keystone v3 support patch https://review.openstack.org/#/c/287504/ | 18:47 |
| slagle | is there any maintenance going on, or something we can check in the tripleo cloud? | 18:47 |
| pabelanger | dprince: ^ | 18:48 |
| clarkb | slagle: we restarted the gearman server which means that all the jenkins slaves had to reregister themselves, if there are no tripleo slaves available then they won't register | 18:48 |
| jeblair | slagle: i restarted nodepool and zuul. all tripleo nodes are in use except one which has been in delete state for 14 days | 18:48 |
| *** flepied has joined #openstack-infra | 18:48 | |
| jeblair | slagle: i can delete all those nodes, and some should be rebuild and will register jobs | 18:48 |
| jeblair | (this would happen on its own once the jobs finish, but they take a while) | 18:49 |
| slagle | ok, thanks | 18:49 |
| pabelanger | mnaser: clarkb: Ya, I noticed some network issue with the AFS mirror in vexxhost too. So we are seeing some network issues too | 18:49 |
| pabelanger | 404 timeout for example | 18:49 |
| pabelanger | err | 18:50 |
| pabelanger | 404 not found | 18:50 |
| mnaser | we're working on it, i don't think it'll be much longer to clear things out now | 18:50 |
| jeblair | slagle: all deleted, should correct itself soon | 18:50 |
| dprince | jeblair: thanks | 18:50 |
| slagle | jeblair: thanks | 18:50 |
| *** ajmiller_ has joined #openstack-infra | 18:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Increase min-ready for devstack-centos7 https://review.openstack.org/291861 | 18:51 |
| AJaeger | clarkb: very conservative change to 2 only ^ - I'm fine to increase further if there's some guidance | 18:52 |
| *** jcoufal has quit IRC | 18:52 | |
| prometheanfire | while we were down, I proposed this patch https://review.openstack.org/#/c/291859/ increaded the libvirt-python upper constraint to 1.3.2 | 18:52 |
| *** ajmiller__ has joined #openstack-infra | 18:53 | |
| *** ajmiller_ has quit IRC | 18:53 | |
| *** ajmiller has quit IRC | 18:53 | |
| noorul | clarkb: I remember creating Project Bootstrappers group | 18:54 |
| noorul | clarkb: Adding account project-creator | 18:54 |
| noorul | clarkb: That is in projects.ini | 18:54 |
| clarkb | noorul: then you need to make sure the ssh key for project-creater is the one that manage_projects uses and that the ssh host key is accepted by ssh | 18:55 |
| noorul | clarkb: http://docs.openstack.org/infra/system-config/gerrit.html#users | 18:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Increase min-ready for devstack-centos7 https://review.openstack.org/291861 | 18:56 |
| fungi | only a couple hundred lines of scrollback while i was erranding. surprisingly non-chatty, but i guess that's a friday for you | 18:57 |
| *** nwkarsten has joined #openstack-infra | 18:57 | |
| AJaeger | fungi, have a look at the numbers of nodes and jobs in 291861 or http://paste.openstack.org/show/490223/ - that was interesting to me | 18:58 |
| *** sridhar_ram has joined #openstack-infra | 18:58 | |
| fungi | clarkb: that's what we _had_ last time. what i told ttx and thingee was "shared sprint space" was fine for us again as it worked out well before | 18:59 |
| noorul | clarkb: I am not sure what is wrong, somehow auth is denied | 18:59 |
| clarkb | noorul: are you still getting the ssh host key error? | 18:59 |
| clarkb | noorul: if so that is the problem | 18:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add missing node to puppet-cookiecutter check https://review.openstack.org/291863 | 19:00 |
| noorul | I have project-creator account on gerrit | 19:00 |
| noorul | I have the public ssh key added to the account | 19:00 |
| AJaeger | somehow we lost the node declaration on two jobs, would be nice to get the change above in today ^ | 19:00 |
| AJaeger | EmilienM: FYI ^ | 19:00 |
| fungi | jeblair: i wanted to restart zuul last night but was trying to get the last of the jjb updates in sync everywhere first to make sure we didn't reregister bare-trusty on any. then lenkins01 took so long/many retries to finish i was too sleepy to be confident i wouldn't make a mess and figured we could restart zuul today or tomorrow when it gets quiet again | 19:00 |
| openstackgerrit | Merged openstack-infra/project-config: Setup neutron-lbaas-dashboards for release https://review.openstack.org/290878 | 19:01 |
| clarkb | noorul: yup, the ssh host key is what the ssh server identifies itself with as an assurance you are not getting man in the middled | 19:01 |
| *** sridhar_ram1 has joined #openstack-infra | 19:01 | |
| *** nwkarsten has quit IRC | 19:01 | |
| fungi | as for the cloud formerly known as infra-cloud west, i suspect nobody knows an actual timeline for that hardware rematerializing in a connected location. at least that's how the last update sounded | 19:02 |
| noorul | clarkb: Is there a way I can go and see the user information in gerrit for those accounts created using gerrit create-account command? | 19:02 |
| fungi | buried under a data center full of red tape | 19:02 |
| clarkb | noorul: if you are getting the host key errors still that isn't an issue with the gerrit user, it is a problem with the ssh client side configuration | 19:02 |
| clarkb | noorul: if you run the ssh command and accept the key that should go away | 19:02 |
| pabelanger | fungi: where did you see the update by chance? | 19:03 |
| EmilienM | Ajaeger: ack | 19:03 |
| mtreinish | clarkb: so there are only 2 workers, but each should only be handling 1 thing at a time on the gearman | 19:03 |
| *** sridhar_ram has quit IRC | 19:03 | |
| clarkb | mtreinish: ok, that isn't what gearman is claiming. May possibly be related to OOMing? | 19:03 |
| clarkb | noorul: once that is done, rerun manage-projects and see if the error changes and go from there | 19:04 |
| noorul | clarkb: This is what I am getting http://paste.openstack.org/show/490231/ | 19:04 |
| mtreinish | clarkb: they pull the subunit files from the log server and as soon as it downloads it puts it on a queue and returns job complete | 19:04 |
| *** sdake_ has quit IRC | 19:05 | |
| mtreinish | and if something goes wrong with the pulling it returns a job exception (or whatever it is) | 19:05 |
| clarkb | noorul: that says your ssh key is not a valid ec key | 19:05 |
| clarkb | noorul: I would use rsa not sure how well gerrit does ecdsa | 19:05 |
| mtreinish | clarkb: the ooming I assumed was that it was putting things on the queue faster than it could pop them off | 19:05 |
| clarkb | mtreinish: I see | 19:05 |
| mtreinish | which is why we moved to 2 workers | 19:05 |
| mtreinish | I don't think it's OOmed since nibalizer and I got subunit-worker02 up | 19:05 |
| *** doug-fis_ has joined #openstack-infra | 19:05 | |
| *** tqtran-afk has joined #openstack-infra | 19:06 | |
| noorul | clarkb: http://docs.openstack.org/infra/system-config/running-your-own.html#stage-3-gerrit | 19:07 |
| noorul | clarkb: I used ssh-key-gen command from that section | 19:07 |
| *** doug-fish has quit IRC | 19:08 | |
| nibalizer | mtreinish: would it say it oomed in syslog? | 19:08 |
| clarkb | noorul: ok you need to figure out why it is thinking that file is an ecdsa key | 19:08 |
| clarkb | noorul: examine it and see if the first bytes are not ssh-rsa | 19:08 |
| fungi | pabelanger: http://lists.openstack.org/pipermail/openstack-infra/2016-March/003969.html | 19:09 |
| mtreinish | nibalizer: syslog, dmesg, or kern.log. I forget where things end up in ubuntu land | 19:09 |
| clarkb | fungi: ya but it sounds like maybe we will have east | 19:09 |
| mtreinish | but you'd see it in 1 of those places (if not all of them, assuming they all exist) | 19:09 |
| fungi | yolanda: did jjb updates complete successfully? | 19:09 |
| clarkb | and we can go ahead an work on making east a thing | 19:09 |
| *** eharney has quit IRC | 19:09 | |
| nibalizer | mtreinish: i see no evidence | 19:10 |
| *** sdake has joined #openstack-infra | 19:10 | |
| nibalizer | also our subunit worker has been running since Mar07 and puppet isn't configured to restart it anyways | 19:10 |
| fungi | yolanda: i think it must have worked because https://jenkins04.openstack.org/job/nodepool-coverage/configure has "WITH GRANT OPTION" now | 19:11 |
| fungi | so that's a good sign at least | 19:11 |
| yolanda | fungi, everything worked | 19:11 |
| fungi | mordred: should be safe to recheck i think, if you haven't already ^ | 19:11 |
| pabelanger | fungi: Ah, ya, that was a little cryptic | 19:11 |
| *** tqtran-afk is now known as tqtran | 19:11 | |
| *** alex_xu has quit IRC | 19:12 | |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite: Fix races in httpd install and config https://review.openstack.org/215169 | 19:12 |
| mtreinish | nibalizer: yeah that's probably because I'm pretty sure we'd drop whatever's been queued when we restart it :) | 19:12 |
| fungi | pabelanger: if you followed the discussion at the sprint, that's basically that they didn't get around to unracking and removing our hardware before the deadline for that data center to go on "lockdown" for a full reinventory | 19:12 |
| *** e0ne_ has joined #openstack-infra | 19:13 | |
| mordred | fungi: woot! | 19:14 |
| *** e0ne has quit IRC | 19:14 | |
| pabelanger | fungi: wow | 19:14 |
| *** binbincong has quit IRC | 19:14 | |
| *** ferhat has quit IRC | 19:16 | |
| *** alex_xu has joined #openstack-infra | 19:16 | |
| *** nwkarsten has joined #openstack-infra | 19:17 | |
| jeblair | pabelanger: (even though we shut it down early to make sure they could do so) | 19:17 |
| automagically | clarkb: Any idea why https://review.openstack.org/#/c/291834/ didn’t verify. I don’t see any Jenkins checks with a failure | 19:17 |
| *** ildikov has joined #openstack-infra | 19:18 | |
| fungi | yeah... them: we're worried this hardware may get stuck in the datacenter for a reinventory if we don't remove it soon... us: okay, we'll shut it down today so you can get started right away! | 19:18 |
| clarkb | automagically: its the NOT_REGISTERED job if you recheck that should be fixed now | 19:18 |
| automagically | aha, its been submitted for recheck, so I’ll wait and see | 19:19 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing node to puppet-cookiecutter check https://review.openstack.org/291863 | 19:19 |
| crinkle | fungi: latest comment on the jira ticket is "shipment postponed while Switch and HP finalize the move out" | 19:19 |
| clarkb | crinkle: for west or east or both? | 19:20 |
| crinkle | clarkb: for west | 19:20 |
| crinkle | east is on its way | 19:20 |
| jeblair | clarkb: okay... so after the restart..... there's something weird going on with nodepool allocations; post jobs are waiting 1h for jobs even though we're well under capacity. :) | 19:20 |
| fungi | i guess "Switch" is a department? or a vendor? | 19:20 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Use dd to create swapfile, ensure swap amount == 8GB https://review.openstack.org/290776 | 19:20 |
| clarkb | jeblair: hrm and the ready nodes are all gone | 19:20 |
| crinkle | fungi: i am guessing a vendor | 19:21 |
| noorul | clarkb: Looks like an issue with the key | 19:21 |
| fungi | jeblair: last night i saw a bunch of ubuntu-trusty nodes in a building state in rax-iad for up to an hour, so nodepool was counting those toward demand fulfillment | 19:21 |
| noorul | clarkb: I used admin account to create the project | 19:21 |
| jeblair | fungi: ah, that could do it | 19:21 |
| pabelanger | jeblair: Exactly | 19:21 |
| fungi | if iad is still having trouble... | 19:21 |
| jeblair | fungi: most building nodes are <5m; maybe ~10 at 30+m | 19:22 |
| openstackgerrit | Aleksandra Fedorova proposed openstack-infra/project-config: Add missing delay-release option in Fuel projects https://review.openstack.org/291390 | 19:22 |
| jeblair | fungi: all 30m+ are iad | 19:22 |
| jeblair | i wonder if we should lower our build timeout | 19:22 |
| jroll | crinkle: fungi: last I saw "Switch" was building a big colo - not sure if they only do colo or what | 19:23 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 19:23 |
| pabelanger | 1hr boot is pretty gracious | 19:24 |
| jeblair | there is a switch dc in vegas | 19:24 |
| pabelanger | I'd be raging after 5mins :) | 19:24 |
| jeblair | so maybe that's where west is moving out of | 19:24 |
| mordred | yes | 19:25 |
| mordred | that is where west was | 19:25 |
| mordred | it's an AMAZING datacenter | 19:25 |
| Clint | with slot machines in every rack | 19:25 |
| jeblair | mordred: they have another one called "tahoe-reno" even though it would be *slightly* more accurate to call it "black rock city" | 19:26 |
| mordred | jeblair: something tells me that calling it black rock city would get way to many people showing up at its doors | 19:27 |
| *** annegent_ has quit IRC | 19:28 | |
| SamYaple | mordred: trying to figure out the ending of LOST still huh? black rock was just a ship move on | 19:29 |
| *** ajmiller__ is now known as ajmiller | 19:29 | |
| mordred | SamYaple: heh. sorry man - wrong cultural reference | 19:29 |
| openstackgerrit | Merged openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 19:30 |
| *** annegentle has joined #openstack-infra | 19:30 | |
| SamYaple | ;) | 19:30 |
| jeblair | okay i'm wrong, it is slightly closer to lake tahoe than brc | 19:30 |
| noorul | Now the project is created. I submitted a patch for review | 19:31 |
| noorul | Zuul log shows http://paste.openstack.org/show/490238/ | 19:31 |
| jeblair | pabelanger: yes, though we also have to account for the fact that nodepool is slow when under load, so it probably wouldn't be good to make it shorter than 30m at the moment | 19:31 |
| *** Guest75056 has quit IRC | 19:32 | |
| *** sambetts is now known as sambetts|afk | 19:33 | |
| pabelanger | jeblair: Was just to see if we logged the boot times in statsd, I don't think so | 19:34 |
| noorul | Can someone help me to figure what is wrong? | 19:34 |
| *** ociuhandu has quit IRC | 19:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Ansible role jobs copy console to disk not swift https://review.openstack.org/291834 | 19:37 |
| *** akrivoka has quit IRC | 19:37 | |
| *** amrith is now known as _amrith_ | 19:38 | |
| *** daemontool_ has quit IRC | 19:39 | |
| *** e0ne_ has quit IRC | 19:40 | |
| jeblair | mordred: i don't know about actual hosting, but looking at the gallery, i will give them *high* marks for theater. | 19:40 |
| jeblair | mordred: https://www.supernap.com/gallery/#4 <-- actual internet tubes :) | 19:40 |
| mordred | jeblair: yup :) | 19:40 |
| jeblair | pabelanger: best we have is 'time to ready' | 19:41 |
| mordred | jeblair: so - one of the neat things they have there is that their air handling system mixes outside air depending on air quality and humidity | 19:41 |
| mordred | jeblair: cause it turns out the air in the desert at night is actually very much the type of air you want | 19:41 |
| jeblair | pabelanger: which is what nodepool is paying attention to. the trick is that because of the nodepool thread starvation under heavy load, it can actually be several minutes before nodepool gets around to processing a node that has become ready | 19:41 |
| jeblair | pabelanger: we will fix that in v3 | 19:41 |
| jeblair | mordred: i love the air in the desert at night | 19:42 |
| pabelanger | jeblair: Ah, interesting | 19:42 |
| jeblair | almost as much as i love it at dawn | 19:42 |
| mordred | the other thing is that they have ex-marines outside who patrol a few-miles-wide perimeter and who are empowered to commandeer fuel in case of crisis for the generators | 19:42 |
| *** HeOS has quit IRC | 19:43 | |
| jeblair | so, like, don't drive past in a diesel truck in times of crisis | 19:43 |
| *** bdemers has joined #openstack-infra | 19:43 | |
| mordred | nope | 19:43 |
| jeblair | mordred: or they will take your truck and fuel and put you in the holding cell: https://www.supernap.com/gallery/#45 | 19:43 |
| *** HeOS has joined #openstack-infra | 19:44 | |
| mordred | jeblair: it's not a bad holding cell to be in in times of crisis that are crisis enough that the hoover dam is not producing electricity | 19:44 |
| * jeblair wonders if the empowerment comes from the constitution or the guns their holding, and whether that's an academic question | 19:44 | |
| mordred | I mean, let's be clear about what all has to fail before those fellows are out taking your truck | 19:44 |
| *** _nadya_ has quit IRC | 19:44 | |
| jeblair | they're even | 19:45 |
| mordred | jeblair: apparenlty, currently, it's an AND | 19:45 |
| jeblair | the constitution is weird in nevada | 19:45 |
| mordred | jeblair: the empowerment comes from the DOD hosting strategic servers in the facility | 19:45 |
| mordred | the power grid shuts off power to hospitals before that data center for the same reason | 19:46 |
| mordred | which is, in itself, creepy | 19:46 |
| jeblair | noorul: is that your zuul or ours? | 19:46 |
| jeblair | er | 19:46 |
| jeblair | noorul: sorry, meant is that your gerrit or ours? | 19:46 |
| noorul | mine | 19:47 |
| noorul | jeblair: mine | 19:47 |
| jeblair | noorul: i'd try running "ssh -p 29418 $hostname gerrit stream-events" with the ssh key zuul is using and see what it's outputting | 19:47 |
| noorul | I did that | 19:48 |
| noorul | and I get | 19:48 |
| noorul | Capability streamEvents is required to access this resource | 19:48 |
| noorul | zuul user I created is under Continuous Integration Tools group | 19:48 |
| *** esikache1 has quit IRC | 19:48 | |
| noorul | I am figuring out how to give access to stream-events for users under that group | 19:48 |
| noorul | any idea? | 19:49 |
| *** sdake_ has joined #openstack-infra | 19:49 | |
| jeblair | noorul: you'll want to modify acls for the "All-Projects" project | 19:49 |
| *** sridhar_ram1 is now known as sridhar_ram | 19:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable Translation for tacker https://review.openstack.org/274855 | 19:49 |
| *** sdake has quit IRC | 19:49 | |
| *** tonytan4ever has quit IRC | 19:50 | |
| jeblair | noorul: under 'global capabilities' you should be able to add the 'stream events' capability and add the group to that capability | 19:50 |
| *** nwkarsten has quit IRC | 19:50 | |
| jeblair | noorul: here's ours: https://review.openstack.org/#/admin/projects/All-Projects,access | 19:50 |
| *** electrofelix has quit IRC | 19:50 | |
| *** tonytan4ever has joined #openstack-infra | 19:51 | |
| jeblair | noorul: (we give it to all users, but giving it to any group your zuul user is in, such as ci tools, will be fine) | 19:51 |
| *** thiagop has quit IRC | 19:51 | |
| fungi | mordred: my old employer's boston datacenter did similar outside air mixing to save on cooling in the winter | 19:52 |
| mordred | fungi: ++ | 19:52 |
| *** Slower has quit IRC | 19:53 | |
| noorul | jeblair: Instead I added zuul user to Non Interactive Users group | 19:54 |
| fungi | it was actually a very smart system, did inline (de)humidification and filtration on the intake | 19:54 |
| noorul | jeblair: Now I get http://paste.openstack.org/show/490243/ | 19:54 |
| openstackgerrit | Merged openstack-infra/nodepool: Be more specific in logging timeout exceptions https://review.openstack.org/287835 | 19:54 |
| *** esikache1 has joined #openstack-infra | 19:54 | |
| *** esikache1 has quit IRC | 19:54 | |
| *** _nadya_ has joined #openstack-infra | 19:56 | |
| openstackgerrit | Simon McCartney proposed openstack-infra/infra-ansible: Reworked setup_env.sh to support Debian & Darwin/OSX https://review.openstack.org/291658 | 19:56 |
| *** sdake_ has quit IRC | 19:59 | |
| *** tonytan4ever has quit IRC | 20:02 | |
| *** nwkarsten has joined #openstack-infra | 20:03 | |
| *** nwkarsten has quit IRC | 20:04 | |
| *** nwkarsten has joined #openstack-infra | 20:04 | |
| *** pots has quit IRC | 20:09 | |
| *** gnuoy_ has joined #openstack-infra | 20:11 | |
| *** madorn_ is now known as madorn | 20:11 | |
| *** pots has joined #openstack-infra | 20:12 | |
| *** nikhil_k has quit IRC | 20:12 | |
| *** flepied has quit IRC | 20:13 | |
| AJaeger | EmilienM: the cookiecutter job still fails ;( I have no idea why ;( Hope somebody else can help and tell what's wrong with paths in https://jenkins07.openstack.org/job/gate-puppet-openstack-cookiecutter-puppet-3-cookiecutter-check-nv/1/console | 20:15 |
| jeblair | noorul: that'll be a config file syntax error; you need a "projects:" section in the layout.yaml file | 20:16 |
| *** sdake has joined #openstack-infra | 20:17 | |
| EmilienM | Ajaeger, OK I'm currently on my phone afk but I'll look tonight. It's not urgent at all and thanks a lot for your help. I'll debug | 20:17 |
| *** ociuhandu has joined #openstack-infra | 20:19 | |
| *** eil397 has joined #openstack-infra | 20:19 | |
| *** x00350071 has quit IRC | 20:19 | |
| *** x00350071 has joined #openstack-infra | 20:20 | |
| *** ihrachys has joined #openstack-infra | 20:21 | |
| noorul | jeblair: You are right. | 20:22 |
| noorul | jeblair: I fixed that! | 20:22 |
| noorul | Now Zuul is up | 20:22 |
| jeblair | noorul: yay! | 20:22 |
| *** pots has quit IRC | 20:23 | |
| noorul | But it is not responding to events | 20:23 |
| noorul | I posted a review | 20:23 |
| jeblair | noorul: can you paste your layout.yaml? | 20:23 |
| *** pots has joined #openstack-infra | 20:24 | |
| noorul | https://github.com/noorul/os-project-config/blob/master/zuul/layout.yaml | 20:24 |
| *** zul has quit IRC | 20:25 | |
| *** pots has quit IRC | 20:25 | |
| *** eharney has joined #openstack-infra | 20:26 | |
| *** hashar has joined #openstack-infra | 20:27 | |
| jeblair | noorul: two things: 1) you may want to start with a much simpler layout.yaml that only has the jobs you're working on, and doesn't use our paratemer functions, etc. | 20:27 |
| jeblair | noorul: 2) the debug log will usually help you figure out why zuul ignored an event or didn't run a job, etc. | 20:27 |
| noorul | Another issue I have is | 20:28 |
| noorul | when I run zuul using service command it is not starting properly | 20:29 |
| noorul | but when I run "sudo -u zuul zuul-server -d", it is running | 20:29 |
| *** pots has joined #openstack-infra | 20:29 | |
| *** ajmiller_ has joined #openstack-infra | 20:30 | |
| *** ajmiller has quit IRC | 20:30 | |
| *** pots has quit IRC | 20:30 | |
| noorul | Looks like it is getting the event | 20:31 |
| noorul | http://paste.openstack.org/show/490248/ | 20:31 |
| *** apoorvad has quit IRC | 20:32 | |
| fungi | AJaeger: pwd on the job is likely the git worktree at /home/jenkins/workspace/gate-puppet-openstack-cookiecutter-puppet-3-cookiecutter-check-nv/ and so the relative path to the script would be ./contrib/bootstrap.sh not ./contrib/openstack/puppet-openstack-cookiecutter/contrib/bootstrap.sh | 20:33 |
| *** |-paul-| has joined #openstack-infra | 20:33 | |
| *** kencjohnston has quit IRC | 20:34 | |
| fungi | AJaeger: it's probably adding an extra $ZUUL_PROJECT/contrib/ in the middle of that or something | 20:34 |
| *** pots has joined #openstack-infra | 20:34 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-elasticsearch: Make total_shards_per_node configurable https://review.openstack.org/291901 | 20:35 |
| *** twm2016_ has joined #openstack-infra | 20:35 | |
| twm2016_ | Has anyone here encountered problems related to installing deps while running a tox test? | 20:35 |
| *** asselin has joined #openstack-infra | 20:35 | |
| pleia2 | fungi: doh, thanks | 20:35 |
| twm2016_ | I've tried the solution here: https://github.com/GoogleCloudPlatform/gcloud-python/issues/1103 but haven't had any luck. | 20:36 |
| *** kencjohnston has joined #openstack-infra | 20:36 | |
| *** thorst has quit IRC | 20:38 | |
| fungi | twm2016_: "problems related to installing deps while running a tox test" is... well the word vague would be an understatement here. can you link to an example of the particular "problems" you have? | 20:39 |
| *** thorst has joined #openstack-infra | 20:39 | |
| AJaeger | thanks for checking fungi, let me look at it again and udpate... | 20:40 |
| fungi | twm2016_: are you seeing the same "Not uninstalling py at /usr/lib/python2.7/dist-packages, owned by OS" message mentioned in that github issue? | 20:40 |
| twm2016_ | fungi: sorry for the vague description. Here is the output I get after the test runs http://paste.openstack.org/show/490249/ | 20:40 |
| AJaeger | fungi, I should have never approved that last change ;( It made it worse... | 20:40 |
| AJaeger | thanks, fungi | 20:40 |
| pleia2 | AJaeger: heh, I missed it too | 20:41 |
| fungi | twm2016_: for some reason it's calling /opt/stack/swift/.tox/py27/bin/easy_install instead of pip. do you happen to know why? | 20:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix path for puppet-openstack-cookiecutter test https://review.openstack.org/291905 | 20:42 |
| AJaeger | pleia2, fungi, EmilienM : Let's try again ^ | 20:42 |
| fungi | twm2016_: is it possible you're using a very old release of tox? or maybe a heavily modified configuration? | 20:42 |
| *** asettle has joined #openstack-infra | 20:42 | |
| AJaeger | thanks for reviewing! | 20:44 |
| twm2016_ | fungi: after running tox --version it says it is "2.3.1 imported from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc" | 20:45 |
| *** krtaylor has quit IRC | 20:45 | |
| AJaeger | twm2016_: that version works fine for me ;) | 20:45 |
| fungi | twm2016_: okay, that's the latest version. have you modified the tox.ini for swift? | 20:45 |
| twm2016_ | fungi: I have not made any modifications to tox.ini. | 20:46 |
| openstackgerrit | Merged openstack-infra/project-config: enable stack sudo for ironic discovery gate job https://review.openstack.org/291664 | 20:46 |
| *** asettle has quit IRC | 20:47 | |
| fungi | twm2016_: and workinf with a recent checkout of swift source code? | 20:48 |
| twm2016_ | Ajaeger: I believe has something to do with my proxy. I'm looking at this patch to help: https://review.openstack.org/#/c/207790/9/doc/source/development_saio.rst and when I run the pip commands I get "ConnectTimeoutError". I found this link to confirm my problem: https://github.com/pypa/pip/issues/1805 | 20:48 |
| fungi | twm2016_: yeah, the tox.ini for swift definitely says "install_command = pip install --allow-external netifaces --allow-insecure netifaces -U {opts} {packages}" but instead of pip it's calling easy_install | 20:49 |
| twm2016_ | fungi: yes cloned, thirty minutes ago. | 20:49 |
| fungi | in your example output | 20:50 |
| fungi | and then easy_install thinks you're asking it to install a package called "install" because it doesn't take subcommands like that | 20:50 |
| *** apoorvad has joined #openstack-infra | 20:50 | |
| *** annegentle has quit IRC | 20:51 | |
| twm2016_ | fungi: I've tried both, sorry I must have given you different output. | 20:51 |
| fungi | is it possible there's a very, very old version of virtualenv being used? what does `/opt/stack/swift/.tox/py27/bin/pip --version` say? | 20:51 |
| fungi | twm2016_: tried both what? | 20:51 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Add python3 non-voting job to kolla-mesos https://review.openstack.org/289046 | 20:52 |
| *** rbrndt__ has joined #openstack-infra | 20:52 | |
| twm2016_ | fungi: both pip install and easy_install | 20:52 |
| twm2016_ | fungi: The version output is : pip 8.1.0 from /opt/stack/swift/.tox/py27/local/lib/python2.7/site-packages (python 2.7) | 20:52 |
| fungi | oh, can you provide another paste where you haven't tried to modify it to use easy_install? the error in the one you provided is definitely along the lines of "easy_install has no idea what you're asking it to do" | 20:53 |
| fungi | so it's not going to have anything at all to do with whatever error you're getting from proper pip usage | 20:54 |
| twm2016_ | fungi: Here it is, http://paste.openstack.org/show/490254/ | 20:54 |
| fungi | twm2016_: the big bold errors there telling you to install the header packages on your system for liberasurecode are your problem | 20:55 |
| AJaeger | Regarding JJB: I think it would be nice to see the number of changed jobs inthe compare output merged sooner - reviews for https://review.openstack.org/291789 are welcome | 20:55 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing delay-release option in Fuel projects https://review.openstack.org/291390 | 20:55 |
| fungi | twm2016_: looks like you're on debian, so `sudo apt-get install liberasurecode-dev` | 20:55 |
| * AJaeger waves good-bye and wishes everybody a great weekend! | 20:56 | |
| fungi | thanks AJaeger and have a great weekend! | 20:56 |
| pleia2 | AJaeger: take care | 20:56 |
| twm2016_ | fungi: You know I tried doing that before, I must have got the package name wrong because it couldn't find it. But that command definetly worked. | 20:56 |
| twm2016_ | Thank you for your help! | 20:56 |
| fungi | twm2016_: no problem, have a great weekend | 20:57 |
| AJaeger | fungi, pleia2 enjoy the weekend as well! | 20:57 |
| *** sdake has quit IRC | 20:58 | |
| jeblair | if that doesn't work, just try this: http://xkcd.com/1654/ | 21:03 |
| pleia2 | haha | 21:04 |
| bswartz | XKCD always has something appropriate | 21:04 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix path for puppet-openstack-cookiecutter test https://review.openstack.org/291905 | 21:05 |
| *** gordc has quit IRC | 21:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Add python3 non-voting job to kolla-mesos https://review.openstack.org/289046 | 21:07 |
| *** gildub has joined #openstack-infra | 21:08 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 21:12 |
| *** dprince has quit IRC | 21:13 | |
| *** alex_xu has quit IRC | 21:14 | |
| ianw | pabelanger: afs rpms are at least building now : https://jenkins07.openstack.org/job/periodic-package-afs-centos7/1/console | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:15 | |
| *** alex_xu has joined #openstack-infra | 21:15 | |
| mordred | ianw: woot! | 21:16 |
| *** yolanda has quit IRC | 21:18 | |
| *** yolanda has joined #openstack-infra | 21:19 | |
| ianw | pabelanger: ahh, so close! http://tarballs.openstack.org/package-afs-centos7/ it's one directory deeper nested than i thought it would be. | 21:19 |
| clarkb | is plan to have puppet grab from tarballs and install that rpm? | 21:20 |
| *** kgiusti has quit IRC | 21:20 | |
| clarkb | when building the wheel builder slave? | 21:20 |
| *** frauggy is now known as auggy | 21:20 | |
| ianw | clarkb: https://review.openstack.org/#/c/290266/ is the plan | 21:21 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 21:22 | |
| *** roxanagh_ has quit IRC | 21:23 | |
| *** roxanagh_ has joined #openstack-infra | 21:24 | |
| *** baoli has quit IRC | 21:24 | |
| *** baoli has joined #openstack-infra | 21:25 | |
| *** nikhil_k has joined #openstack-infra | 21:28 | |
| *** nwkarste_ has joined #openstack-infra | 21:28 | |
| *** baoli has quit IRC | 21:31 | |
| *** nwkarsten has quit IRC | 21:32 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: [wip] Revert "Add CentOS AFS build job to project-config experimental" https://review.openstack.org/291403 | 21:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Centos AFS rpm : copy to workspace https://review.openstack.org/291919 | 21:32 |
| *** devananda has quit IRC | 21:35 | |
| *** nwkarste_ has quit IRC | 21:35 | |
| *** thorst has quit IRC | 21:35 | |
| *** nwkarsten has joined #openstack-infra | 21:35 | |
| *** mriedem has quit IRC | 21:36 | |
| *** rlandy has quit IRC | 21:37 | |
| *** nwkarste_ has joined #openstack-infra | 21:37 | |
| *** sarob has quit IRC | 21:37 | |
| *** annegentle has joined #openstack-infra | 21:38 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Don't install ES on logstash workers https://review.openstack.org/291920 | 21:39 |
| *** annegentle has quit IRC | 21:39 | |
| *** nwkarsten has quit IRC | 21:40 | |
| clarkb | anteaya: ^ last step in logstash 2.0 upgrade | 21:41 |
| *** annegentle has joined #openstack-infra | 21:41 | |
| anteaya | clarkb: any other post logstash 2.0 install patches to review? | 21:41 |
| clarkb | anteaya: that should be it | 21:41 |
| anteaya | yup, caught it, thanks | 21:41 |
| anteaya | wonderful, you have my +1 | 21:41 |
| *** noorul has quit IRC | 21:41 | |
| anteaya | so far so good? | 21:41 |
| *** nwkarste_ has quit IRC | 21:42 | |
| jesusaur | clarkb: do we want to also ensure the service is stopped and the package in purged? | 21:42 |
| jesusaur | or will that be done manually? | 21:43 |
| *** _nadya_ has quit IRC | 21:43 | |
| clarkb | jesusaur: the current puppet-elasticsearch module can't express that, I was just going to do it manually | 21:44 |
| jesusaur | clarkb: right, it can't be done through the module, but we could replace the class resource with a service and package resource, then remove those in a follow-up patch | 21:45 |
| *** sarob has joined #openstack-infra | 21:45 | |
| *** annegentle has quit IRC | 21:45 | |
| clarkb | nah I will just clear it out manually, but can make the puppet richer if people prefer that | 21:45 |
| *** annegentle has joined #openstack-infra | 21:46 | |
| jesusaur | nah, just wanna make sure that step doesn't get missed | 21:46 |
| anteaya | I have no preference | 21:46 |
| anteaya | as long as we soon get to be able to discuss steps to impliment kibana 4 | 21:46 |
| anteaya | and as long as anyone else should they have to touch a logstashworker, knows what to do | 21:47 |
| *** devananda has joined #openstack-infra | 21:48 | |
| mtreinish | anteaya: why the big push for kibana 4? what's so much better about it? | 21:48 |
| * mtreinish hasn't even looked at it or the differences | 21:48 | |
| *** kencjohnston has quit IRC | 21:48 | |
| fungi | i've recently heard it referred to as "munroe's law" (akin to godwin's law) that once the discussion has been summarized by an xkcd link there's no point in continuing it | 21:48 |
| *** nwkarsten has joined #openstack-infra | 21:49 | |
| *** aysyd has quit IRC | 21:50 | |
| anteaya | fungi: ha ha ha | 21:50 |
| anteaya | mtreinish: it is our end state for front ends | 21:50 |
| *** gnuoy_ has quit IRC | 21:50 | |
| SamYaple | its crazy how many laws there are that are similiar fungi. we should come up with one law that is a standard | 21:50 |
| anteaya | mtreinish: I took an an action item at the last summit, to bring on more folks able to debug the gate | 21:51 |
| *** tongli has quit IRC | 21:51 | |
| anteaya | mtreinish: then we had to address logstash/kibana upgrades | 21:51 |
| anteaya | mtreinish: the kibana interface for 3 was much different than 2 | 21:51 |
| anteaya | and I'm told, the kibana 4 interface is much different again | 21:51 |
| SamYaple | anteaya: I would be happy help but the learning curve is super high... | 21:51 |
| mtreinish | anteaya: well it's the end state until they release kibana 5 :) | 21:51 |
| mtreinish | anteaya: yeah, they seem to like to change things a lot between releases | 21:52 |
| mtreinish | (and IMO not always for the better) | 21:52 |
| anteaya | so I didn't see the point to try to teach folks kibana 3 | 21:52 |
| anteaya | SamYaple: yes the learning curve is high, even when the interfaces are not changing out from under you | 21:52 |
| anteaya | mtreinish: points on both | 21:52 |
| anteaya | so I made a decision | 21:52 |
| anteaya | and didn't communicate it widely | 21:52 |
| anteaya | prefering to focus my energy on doing what I could to support achieving the end state | 21:53 |
| anteaya | hence my actions | 21:53 |
| SamYaple | who would have thought deploying openstack would be simplier than debugging the gate? | 21:53 |
| *** nwkarsten has quit IRC | 21:53 | |
| anteaya | so anytime there was an opportunity to support the move to logstash 2.0 to get to kibana 4, I tried to help | 21:54 |
| anteaya | and i hope my actions are construed as helping | 21:54 |
| *** gnuoy_ has joined #openstack-infra | 21:54 | |
| anteaya | SamYaple: anyone who hasn't tried to debug the gate | 21:54 |
| fungi | SamYaple: to debug gate jobs you basically become an openstack operator at a very small it shop where you're the only one around to support the entire cloud and it's suddenly suffered a catastrophic cascade failure | 21:55 |
| *** sc68cal has quit IRC | 21:56 | |
| *** thorst has joined #openstack-infra | 21:56 | |
| *** tsg has joined #openstack-infra | 21:56 | |
| anteaya | mtreinish: so that's my reason | 21:56 |
| *** tpsilva has quit IRC | 21:57 | |
| anteaya | now I have to go back to the cross project session this summit and hang my head in shame as I did not do what I said I would do this release | 21:57 |
| *** rguillebert has quit IRC | 21:57 | |
| anteaya | but I do hold the hope we can get to kibana 4 by austin so at least I can offer a silver lining | 21:57 |
| SamYaple | fungi: i was an openstack operator and i run a cloud with public ips at my house.... but no one dpends on me.... | 21:58 |
| anteaya | but I didn't tell the whole story before because clark has had enough on his plate this release | 21:58 |
| anteaya | so decided to just offer to help | 21:58 |
| fungi | anteaya: well, the up side is that most of the answers to "kibana 3 is terrible" were "wait for kibana 4, we've totally solved that" | 21:58 |
| mtreinish | anteaya: ok, so there isn't a killer feature or any known improvement. It's just to make sure we have the latest | 21:58 |
| anteaya | fungi: exactly | 21:58 |
| mtreinish | fungi: well I'm sure they said the same things about 2 vs 3 | 21:58 |
| anteaya | I'm hoping for that | 21:58 |
| anteaya | clarkb says it is better than 3 | 21:58 |
| anteaya | I am trusting clarkb | 21:59 |
| fungi | and indeed, it was probably true. what they never tell you is what other features you depended on that they decided to completely break | 21:59 |
| mtreinish | fungi: like permalink :) | 21:59 |
| *** e0ne has joined #openstack-infra | 21:59 | |
| *** scottda is now known as scottda_on_vacat | 21:59 | |
| fungi | that's the main one i had in mind when i said that | 21:59 |
| fungi | so painful clarkb basically reimplemented it in javascript | 22:00 |
| *** scottda_on_vacat is now known as scottda_vacation | 22:00 | |
| *** _ari_ has quit IRC | 22:00 | |
| fungi | or somebody did (that was clarkb right?) | 22:00 |
| clarkb | well its "better" | 22:00 |
| *** rfolco has quit IRC | 22:01 | |
| clarkb | I ran it locally last night and it needs some work | 22:01 |
| clarkb | they expose all the config knobs to everyone | 22:01 |
| clarkb | and have big scary things like "lol this can break things" | 22:01 |
| clarkb | thanks kibana for letting everyone break things | 22:01 |
| *** ajmiller has joined #openstack-infra | 22:02 | |
| *** ajmiller_ has quit IRC | 22:02 | |
| *** jaypipes has quit IRC | 22:02 | |
| fungi | i bet it's a really cool feature, as long as you don't ever let anyone use it | 22:02 |
| clarkb | its also a lot more visually noisy | 22:03 |
| *** dmsimard has joined #openstack-infra | 22:03 | |
| *** e0ne has quit IRC | 22:04 | |
| clarkb | on the list of things then for someone (maybe me) to figure out is how to configure or otherwise hobble kibana 4 to prevent that access | 22:06 |
| clarkb | it does still use elasticsearch itself to save the data for links and stuff in a .kibana index but we would give the appliation access to that not the user or their browser so that is fine | 22:07 |
| anteaya | do we have a not noisy option? | 22:08 |
| anteaya | 2 was friendly in that it was very quiet | 22:08 |
| clarkb | anteaya: no thats part of the new and improved kibana | 22:08 |
| clarkb | you can save visualizations and dashboards | 22:08 |
| anteaya | :( | 22:08 |
| clarkb | and share those | 22:08 |
| *** lascii is now known as alaski | 22:08 | |
| anteaya | queries | 22:09 |
| anteaya | all I want to be able to do is compose a query | 22:09 |
| anteaya | share a query | 22:09 |
| clarkb | yup thats still there too | 22:09 |
| anteaya | and teach others to do those two actions | 22:09 |
| *** GnomeKin1 has joined #openstack-infra | 22:09 | |
| anteaya | okay thanks | 22:09 |
| clarkb | https://www.elastic.co/guide/en/kibana/current/advanced-options.html is what needs to be hobbled | 22:10 |
| anteaya | oh my | 22:10 |
| anteaya | yes, can you just remove the html that makes that available? | 22:10 |
| anteaya | html/javascript/language of choice | 22:10 |
| *** GnomeKing has quit IRC | 22:12 | |
| clarkb | or set up a proxy that null routes that | 22:14 |
| clarkb | or maybe there is an on disk config to update | 22:14 |
| *** GnomeKin1 has quit IRC | 22:14 | |
| *** GnomeKing has joined #openstack-infra | 22:14 | |
| anteaya | we can hope? | 22:14 |
| *** GnomeKing is now known as Guest82807 | 22:14 | |
| *** amitgandhinz has quit IRC | 22:15 | |
| *** dmsimard is now known as dmsimard|pto | 22:15 | |
| *** melwitt has quit IRC | 22:18 | |
| *** melwitt has joined #openstack-infra | 22:18 | |
| anteaya | I am on day three of working on this expense report | 22:21 |
| clarkb | ya, i am sure there is a way even if it is making the proxy smarter than it should be | 22:21 |
| clarkb | but the logstsah upgrade itself looks good | 22:21 |
| anteaya | I bet I spend 30% of my time logging back in after it kicks me out | 22:21 |
| *** regXboi has quit IRC | 22:21 | |
| anteaya | :( | 22:21 |
| clarkb | it is keeping up with demand better than old version likely due to the extra available memory | 22:21 |
| anteaya | clarkb: yay logstash 2.0 | 22:21 |
| anteaya | clarkb: ohhh, good to hear | 22:21 |
| *** krtaylor has joined #openstack-infra | 22:22 | |
| sc` | i'm attempting to get a centos7 gate working, but the build fails because ifconfig isn't there, despite net-tools being installed | 22:25 |
| clarkb | sc`: probably a path issue | 22:26 |
| clarkb | ifconfig is in /sbin/ I htink? and that isn't always in default paths | 22:26 |
| sc` | looks like for c7 it lives in /usr/sbin | 22:26 |
| clarkb | yup just checked on a devstack-centos7 machine | 22:27 |
| clarkb | I would make sure that is in your path | 22:28 |
| sc` | yeah. trying with that in my path | 22:28 |
| *** twm2016_ has quit IRC | 22:29 | |
| *** gnuoy_ has quit IRC | 22:30 | |
| *** Sukhdev has joined #openstack-infra | 22:30 | |
| *** Slower has joined #openstack-infra | 22:31 | |
| *** esker has joined #openstack-infra | 22:32 | |
| *** asettle has joined #openstack-infra | 22:33 | |
| *** krtaylor has quit IRC | 22:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Report number of changed jobs https://review.openstack.org/291789 | 22:35 |
| *** melwitt has quit IRC | 22:36 | |
| *** melwitt has joined #openstack-infra | 22:36 | |
| *** krtaylor has joined #openstack-infra | 22:37 | |
| clarkb | infra-root care to ack https://review.openstack.org/#/c/291920/ then I can clear out the ES installs on the logstash workers? | 22:37 |
| *** andymaier has joined #openstack-infra | 22:37 | |
| *** david-lyle has quit IRC | 22:37 | |
| clarkb | last step in logstash 2.0 upgrade | 22:37 |
| pleia2 | having a look | 22:37 |
| nibalizer | clarkb: lgtm | 22:37 |
| *** andymaier has quit IRC | 22:37 | |
| *** jtomasek has quit IRC | 22:38 | |
| *** asettle has quit IRC | 22:38 | |
| pleia2 | clarkb: there you go | 22:39 |
| clarkb | thank you! | 22:39 |
| anteaya | yay! | 22:39 |
| *** krtaylor has quit IRC | 22:42 | |
| *** krtaylor has joined #openstack-infra | 22:43 | |
| *** jlk has quit IRC | 22:43 | |
| clarkb | mordred: so I have discovered that https://review.openstack.org/#/c/285477/ seems to need a complete rewrite now? | 22:43 |
| clarkb | this diff is not very lcean | 22:43 |
| mordred | clarkb: oh. piddle | 22:44 |
| mordred | clarkb: I thought that had landed | 22:44 |
| mordred | clarkb: I'll port your wor | 22:44 |
| mordred | work | 22:44 |
| clarkb | mordred: ok thanks, I think the actually tricky stuff is in the shell scripts and those don't conflict | 22:44 |
| mordred | yah | 22:44 |
| clarkb | its just in the logic/workflow that needs merging | 22:44 |
| mordred | the patch in launch-node itself all makes immediate sense to me | 22:44 |
| openstackgerrit | Merged openstack-infra/system-config: Don't install ES on logstash workers https://review.openstack.org/291920 | 22:45 |
| *** dimtruck is now known as zz_dimtruck | 22:46 | |
| *** rcernin has joined #openstack-infra | 22:47 | |
| sc` | looks like i'm unable to override PATH? | 22:47 |
| sc` | never mind. fat finger | 22:47 |
| jeblair | ianw: see comments in 291919 | 22:48 |
| *** jlk has joined #openstack-infra | 22:48 | |
| *** jlk has quit IRC | 22:48 | |
| *** jlk has joined #openstack-infra | 22:48 | |
| *** Slower has quit IRC | 22:48 | |
| clarkb | mordred: oh is build server not ansibling? I could probably do this if you want me to | 22:48 |
| clarkb | mordred: its just the file copying that is ansibling now? | 22:49 |
| *** Sukhdev has quit IRC | 22:49 | |
| *** alex_xu has quit IRC | 22:49 | |
| clarkb | hrm does git appy have a -e flag? this will be awesome if so | 22:49 |
| mordred | clarkb: build server is not yet ansibling - ricky actually has some work I want to build on - but I don't think we need to ansible it yet | 22:49 |
| *** tsg has quit IRC | 22:49 | |
| mordred | clarkb: right now it's just ansibling the initial puppet run | 22:49 |
| clarkb | no git apply -e :( | 22:50 |
| *** alex_xu has joined #openstack-infra | 22:51 | |
| sc` | PATH appears to be getting set, but not making it to the point when 'env' is run | 22:51 |
| clarkb | sc`: how are you setting it? is it on the other side of a sudo? | 22:52 |
| *** dmsimard|pto has quit IRC | 22:52 | |
| *** claudiub has quit IRC | 22:52 | |
| sc` | before sudo. might have to put it in one of the sudo commands i'm thinking | 22:53 |
| clarkb | yes sudo wipesout env | 22:53 |
| *** ihrachys has quit IRC | 22:53 | |
| *** IlyaG has joined #openstack-infra | 22:53 | |
| SamYaple | unless using -E | 22:54 |
| clarkb | the first easy check would be to use a full path to ifconfig | 22:54 |
| clarkb | if that works then figure out PATH | 22:54 |
| sc` | i'm not directly calling ifconfig | 22:55 |
| sc` | "something" is | 22:56 |
| *** EricGonczer_ has joined #openstack-infra | 22:56 | |
| sc` | ah. never mind. reverse engineering fun times. i think i see it | 22:57 |
| *** sarob has quit IRC | 22:57 | |
| *** Slower has joined #openstack-infra | 22:57 | |
| *** sarob has joined #openstack-infra | 22:58 | |
| *** EricGonc_ has quit IRC | 22:59 | |
| *** bauwser is now known as bauzas | 23:00 | |
| *** xyang1 has quit IRC | 23:01 | |
| *** jlk has quit IRC | 23:01 | |
| *** asettle has joined #openstack-infra | 23:01 | |
| *** jlk has joined #openstack-infra | 23:01 | |
| *** jlk has joined #openstack-infra | 23:01 | |
| *** bpokorny_ has joined #openstack-infra | 23:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Pass tags through to jobs https://review.openstack.org/291937 | 23:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add neutron tags to some jobs https://review.openstack.org/291938 | 23:03 |
| jeblair | mtreinish: our running zuul supports that now ^ | 23:03 |
| jeblair | mtreinish: (and tags can be a list; i just added that neutron tag as an example more or less) | 23:04 |
| *** bpokorny_ has quit IRC | 23:04 | |
| *** amotoki has quit IRC | 23:04 | |
| *** bpokorny_ has joined #openstack-infra | 23:04 | |
| anteaya | I'd like to see them in action | 23:05 |
| anteaya | thanks jeblair | 23:05 |
| clarkb | looking at puppet runs and puppetboard we could stand to do some cleanup | 23:05 |
| jeblair | anteaya: yeah, nothing uses them yet, but we can merge that and see the output on the jenkins params page for those jobs | 23:05 |
| *** bpokorny has quit IRC | 23:05 | |
| anteaya | clarkb: anything a non-root can help with? | 23:05 |
| clarkb | the old pypi mirrors all fail and appear to still be in puppetboard | 23:05 |
| clarkb | anteaya: mostl yI just need to understand how to clear it up | 23:06 |
| anteaya | jeblair: yeah, I figured it wouldn't hurt anything to merge them and see where they take us | 23:06 |
| clarkb | I think we need a puppet cert delete and maybe a cache removal for ansible | 23:06 |
| jeblair | clarkb: if i had a veto on deleting those, i hereby lift it if i hadn't already. :) | 23:06 |
| anteaya | clarkb: ah, do say if I can help at all | 23:06 |
| mnaser | clarkb pabelanger ipv6 issues should be resolved by now. | 23:06 |
| mnaser | so the git checkout timeouts should be improved and favor ipv6 too | 23:06 |
| anteaya | mnaser: yay! | 23:06 |
| mnaser | also | 23:06 |
| mnaser | ipv6! | 23:06 |
| *** rcernin has quit IRC | 23:06 | |
| jeblair | mnaser: ipv6! | 23:06 |
| anteaya | ipv6! | 23:06 |
| mnaser | dual stack by default for all machines | 23:07 |
| clarkb | when I am not cleaning up after ES I will try to look into that more | 23:07 |
| anteaya | mnaser: nice work | 23:07 |
| clarkb | (I am watching puppet run before cleaning up es) | 23:07 |
| anteaya | clarkb: yup | 23:07 |
| jeblair | i'll look into the pypi stuff | 23:07 |
| mtreinish | jeblair: very cool, I'll push a patch up shortly to expand on that | 23:07 |
| anteaya | happy to help if I can figure out what to do (and have permissions) | 23:07 |
| mnaser | clarkb: looking at the addressing, mirror.ca-ymq-1.vexxhost.openstack.org can now resolve to 2604:e100:1:0:f816:3eff:feb1:40bd and it'll #justwork | 23:08 |
| anteaya | woooooo | 23:08 |
| *** rcernin has joined #openstack-infra | 23:08 | |
| clarkb | mnaser: neat we will have to add AAAA records | 23:08 |
| mnaser | i dont have ipv6 at home but according to online pings it work | 23:08 |
| clarkb | mnaser: are you slaacing? | 23:09 |
| mnaser | we're doing a few more tweaks because one of our upstreams is involved in a little peering drama | 23:09 |
| mnaser | yes clarkb | 23:09 |
| clarkb | Wondering if we need to do anyhting on our side to pick that up | 23:09 |
| mnaser | nothing at all | 23:09 |
| clarkb | awesome | 23:09 |
| mnaser | all your instances are all pingable by ipv6 now automatically | 23:09 |
| mnaser | and also they have ipv6 listed in nova show / api | 23:09 |
| *** asettle has quit IRC | 23:09 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Make delete_project to call get_project https://review.openstack.org/291940 | 23:12 |
| mnaser | if any network issues happen again, please let me know, however, there shouldn't be | 23:12 |
| jeblair | clarkb: we deleted the pypi servers.... | 23:12 |
| jeblair | where is the ansible cache? | 23:13 |
| clarkb | launch node has a path to it iirc | 23:13 |
| jeblair | /var/cache/ansible-inventory/ansible-inventory.cache ? | 23:13 |
| anteaya | mnaser: thanks for being so available | 23:13 |
| clarkb | /var/cache/ansible-inventory/ansible-inventory.cache | 23:13 |
| mnaser | anteaya: no problem, it's my pleasure! | 23:13 |
| anteaya | mnaser: makes it so lovely and easy | 23:13 |
| anteaya | :) | 23:13 |
| jeblair | clarkb: no pypis in there | 23:14 |
| mnaser | i make an effort! | 23:14 |
| clarkb | huh why do they show up in puppet_run_all.log | 23:14 |
| mnaser | anyways, time for a little break. take care folks | 23:14 |
| anteaya | mnaser: you do so | 23:15 |
| anteaya | mnaser: have a good weekend | 23:15 |
| anteaya | thanks again | 23:15 |
| jeblair | they are in /etc/ansible/hostfile and hosts/generated-groups | 23:15 |
| *** daemontool has joined #openstack-infra | 23:16 | |
| *** akshai has quit IRC | 23:16 | |
| jeblair | and they show up in ansible 'pypi*' --list-hosts | 23:16 |
| jeblair | which is how those get generated | 23:16 |
| clarkb | and list-hosts is generating those via the inventory script? | 23:17 |
| jeblair | clarkb: the generated groups are via a script that runs ansible --list-hosts; my understanding is that should be consulting the cache, but clearly my understanding is incomplete | 23:18 |
| jeblair | mordred: pypi.dfw.openstack.org and similar hosts appear in "ansible 'pypi*' --list-hosts" but do not appear in /var/cache/ansible-inventory/ansible-inventory.cache. what am i missing? | 23:18 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add periodic-qa tag to periodic qa jobs https://review.openstack.org/291943 | 23:20 |
| mtreinish | jeblair: ^^^ | 23:20 |
| *** gyee has quit IRC | 23:21 | |
| jeblair | clarkb, mordred: i think i figured it out | 23:21 |
| jeblair | clarkb, mordred: it's slightly circular... | 23:21 |
| clarkb | elasticsearch is removed on all logstash workers | 23:21 |
| pleia2 | clarkb: hooray | 23:21 |
| jeblair | clarkb, mordred: /etc/ansible/hosts/generated-groups appears to _also_ be input to ansible --list-hosts | 23:21 |
| clarkb | jeblair: nice! | 23:21 |
| jeblair | clarkb, mordred: so once a host exists and gets put in that file, it will be in the inventory forever | 23:22 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:22 | |
| clarkb | we should probably make the only input the inventory cache? | 23:23 |
| clarkb | or rathe rthe only "these are the hosts that exist" input | 23:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Remove pypi ansible group https://review.openstack.org/291944 | 23:23 |
| jeblair | clarkb, mordred: in this specific case, of removing all matching hosts, this will correct the problem, but we should still solve for the case of deleting a single host that's part of a group ^ | 23:24 |
| anteaya | clarkb: yay | 23:24 |
| anteaya | mtreinish: any way we could regex that up a bit? | 23:25 |
| mtreinish | jeblair: once we have some examples in the subunit2sql db with tags I think it's probably worth a ML post about tags and periodic jobs | 23:25 |
| anteaya | mtreinish: 291943 | 23:25 |
| mtreinish | anteaya: I didn't see an easy way | 23:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Remove pypi hosts from cacti https://review.openstack.org/291945 | 23:25 |
| *** rguillebert has joined #openstack-infra | 23:25 | |
| mtreinish | because there are other jobs that fit the basic pattern | 23:25 |
| anteaya | harumph | 23:26 |
| mtreinish | anteaya: like: periodic-tempest-dsvm-oslo-latest-full-master | 23:26 |
| mtreinish | which isn't in the set I wanted to tag | 23:26 |
| jeblair | mtreinish: ++ on the change and the ml post | 23:26 |
| jeblair | anteaya: to be fair, my neutron tags may be too loose as well. | 23:27 |
| anteaya | okay | 23:27 |
| anteaya | so we'll make them work and hopefully tighten them up a bit when we can? | 23:27 |
| jeblair | they'll look a lot nicer in zuulv3 too. | 23:28 |
| anteaya | oh good | 23:28 |
| anteaya | that makes me feel better, thanks | 23:28 |
| SamYaple | eta on zuulv3? | 23:28 |
| SamYaple | waiting for that sweet sweet reconnect feature | 23:28 |
| anteaya | I'm +2 on all of them, but won't +A mtreinish's since the others haven't been approved yet | 23:29 |
| anteaya | SamYaple: that is entirely based on patches and reviews done on patches | 23:29 |
| anteaya | so reviewing and writing patches will speed things along | 23:29 |
| anteaya | if you leave it all up to jeblair it is as fast as he is able to create space to write code | 23:30 |
| SamYaple | last time this rolled around i did ask if tehre was a spec/blueprint/wrok items where i could pick up some slack on | 23:30 |
| *** mgkwill has quit IRC | 23:30 | |
| *** claudiub|2 has joined #openstack-infra | 23:30 | |
| SamYaple | so if someone links me to that I am sure i can help there | 23:30 |
| *** eil397 has quit IRC | 23:30 | |
| *** rcernin has quit IRC | 23:31 | |
| *** eil397 has joined #openstack-infra | 23:31 | |
| *** annegentle has quit IRC | 23:32 | |
| anteaya | SamYaple: this would be a good place to begin: http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html | 23:32 |
| *** annegentle has joined #openstack-infra | 23:32 | |
| SamYaple | thanks anteaya will review | 23:32 |
| anteaya | and if you would like to review: https://review.openstack.org/#/q/branch:feature/zuulv3+status:open | 23:33 |
| anteaya | thank you | 23:33 |
| *** flepied has joined #openstack-infra | 23:34 | |
| *** annegentle has quit IRC | 23:37 | |
| *** bryan_att has quit IRC | 23:37 | |
| * rcarrillocruz waves as his latest change has voting tests in green | 23:37 | |
| rcarrillocruz | have a nice weekend all | 23:37 |
| rcarrillocruz | laterz | 23:37 |
| anteaya | rcarrillocruz: thanks, you too | 23:39 |
| anteaya | congratulations for all green on your change | 23:39 |
| anteaya | have a nice weekend | 23:39 |
| sc` | i keep running into permission denied bits. sudo wrapping the command gets me past it, but i shouldn't be using sudo for everything | 23:41 |
| *** notnownikki has quit IRC | 23:41 | |
| *** daemontool_ has joined #openstack-infra | 23:41 | |
| jeblair | SamYaple: nodepool-shade is in the critical path for zuulv3, so thanks for reviewing that | 23:42 |
| *** thorst is now known as thorst_afk | 23:42 | |
| jeblair | SamYaple: we're going to branch nodepool after it lands | 23:42 |
| *** asettle has joined #openstack-infra | 23:42 | |
| *** daemontool has quit IRC | 23:43 | |
| *** mgkwill has joined #openstack-infra | 23:45 | |
| *** annegentle has joined #openstack-infra | 23:46 | |
| SamYaple | jeblair: okiedokie. ill try to keep up with the reviews since it sounds like that may be where im of most help | 23:47 |
| *** erlon has quit IRC | 23:48 | |
| *** hdd has quit IRC | 23:49 | |
| *** asettle has quit IRC | 23:49 | |
| *** annegentle has quit IRC | 23:54 | |
| *** roxanagh_ has quit IRC | 23:54 | |
| *** annegentle has joined #openstack-infra | 23:54 | |
| docaedo | anyone feel like taking a few minutes to help debug an app-catalog issue? Would require either ssh'ing to the box, or critiquing a fun pipe-chain shell script :) | 23:54 |
| *** nwkarsten has joined #openstack-infra | 23:55 | |
| docaedo | (but I figure most everyone is weekending by now, so can pick this up Monday) | 23:55 |
| *** roxanagh_ has joined #openstack-infra | 23:56 | |
| openstackgerrit | Simon McCartney proposed openstack-infra/infra-ansible: Reworked setup_env.sh to support Debian & Darwin/OSX https://review.openstack.org/291658 | 23:56 |
| sc` | it seems as though /home/jenkins/workspace/stuff isn't writeable by the jenkins user... | 23:57 |
| *** rbrndt__ has quit IRC | 23:57 | |
| mordred | SamYaple, jeblair: why nodepool-shade fail??? | 23:57 |
| sc` | seems that the git clone is happening as root. hm... have to see where/why/how | 23:58 |
| *** annegentle has quit IRC | 23:59 | |
| anteaya | as a follow in to last night's conversation, some may enjoy this: http://cbiftrumpwins.com/#intro | 23:59 |
| sc` | may as well just use sudo in this case. no sense in debugging permissions and other sundry things | 23:59 |
| anteaya | *follow on | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!