| fungi | zaro: see what is in that file for the dc_local_interfaces value | 00:00 |
|---|---|---|
| fungi | chances are it's set to 'lo' or something | 00:00 |
| *** salv-orlando has joined #openstack-infra | 00:00 | |
| jeblair | zaro: why not run it on the same server? | 00:01 |
| *** irenab has joined #openstack-infra | 00:02 | |
| zaro | jeblair: yeah, got it working from my mac (dev machine) but thought it would be nicer to have a dedicated server for that, where i can connect gerrit from other machines. | 00:02 |
| fungi | and yeah, if you just need to configure an smtp relay (smarthost) for a gerrit service, you can do that locally on your gerrit server (that's what we do on ours) | 00:02 |
| *** piet has quit IRC | 00:02 | |
| *** markvoelker has joined #openstack-infra | 00:02 | |
| jeblair | zaro: also, be aware that if you're planning on having the server accept mail from any host, unauthenticated, and then send that mail off to other hosts rather than delivering locally, that particular configuration is called an 'open relay', and is a really bad idea. so make sure it either only delivers locally or only accepts authenticated connections. | 00:02 |
| fungi | oh, you're running gerrit on your mac and don't want to configure the mac as a smart host? | 00:02 |
| *** salv-orl_ has quit IRC | 00:03 | |
| *** irenab has quit IRC | 00:03 | |
| zaro | fungi: sorry, i don't know the term 'smart host. | 00:03 |
| fungi | zaro: common industry term for an smtp relay | 00:03 |
| zaro | alright i guess jeblair says bad idea so i'll just decomission and use the local one i've setup. | 00:04 |
| fungi | yeah, if you do set it up to receive and forward e-mail, then you should either set it up to only accept authenticated submissions (over tls encryption), or restrict the source ip addresses which it will allow to relay through | 00:04 |
| *** apuimedo has quit IRC | 00:04 | |
| fungi | and of course test it to make sure you can't send through it from random places without authentication, because open relays are a bad scene | 00:04 |
| *** apuimedo_ has quit IRC | 00:05 | |
| zaro | got it, bad idea! | 00:05 |
| *** apuimedo has joined #openstack-infra | 00:06 | |
| fungi | you can restrict the source ip addresses allowed to relay with the dc_relay_nets in that file, and you can also restrict what destination domains to which you'll allow relaying with the dc_relay_domains option | 00:06 |
| *** apuimedo_ has joined #openstack-infra | 00:06 | |
| *** admin0 has quit IRC | 00:07 | |
| *** markvoelker has quit IRC | 00:07 | |
| *** sdake has quit IRC | 00:07 | |
| zaro | fungi: yeah, i see that but might as well just setup local smtp if i have to maintain that. so i'll just stick with the localhost setup. | 00:07 |
| fungi | but to jeblair's earlier point, the reason it sets itself up by default to only listen on the loopback interface is because of years of the internet struggling to stamp out people cluelessly running open relays without realizing they were doing it | 00:07 |
| *** piet has joined #openstack-infra | 00:08 | |
| fungi | one of those "i hope you know what you're getting yourself into" configuration steps | 00:08 |
| * zaro is really clueless | 00:08 | |
| *** irenab has joined #openstack-infra | 00:09 | |
| zaro | well learned my lesson after only a few hours of mucking :) | 00:09 |
| fungi | running mailservers is pretty esoteric, especially these days, so it's pretty much expected that most people, even many sysadmins, may be clueless about it | 00:09 |
| Clint | :( | 00:09 |
| lifeless | sudo apt-get install clueless-smtpd ? | 00:10 |
| lifeless | should be a thing | 00:11 |
| *** irenab has quit IRC | 00:11 | |
| *** rbrndt has quit IRC | 00:11 | |
| *** coolsvap|away has quit IRC | 00:12 | |
| *** xarses has quit IRC | 00:12 | |
| *** otter768 has joined #openstack-infra | 00:12 | |
| *** jswarren_ has joined #openstack-infra | 00:12 | |
| *** sdake has joined #openstack-infra | 00:12 | |
| *** asselin has quit IRC | 00:13 | |
| *** apuimedo_ has quit IRC | 00:13 | |
| *** BobBall_AWOL has quit IRC | 00:14 | |
| *** jswarren has quit IRC | 00:14 | |
| *** BobBall_AWOL has joined #openstack-infra | 00:14 | |
| *** notmorgan has quit IRC | 00:14 | |
| *** coolsvap|away has joined #openstack-infra | 00:14 | |
| *** apuimedo_ has joined #openstack-infra | 00:15 | |
| *** zeih has quit IRC | 00:15 | |
| *** zz_dimtruck is now known as dimtruck | 00:19 | |
| *** apuimedo_ has quit IRC | 00:19 | |
| docaedo | Does a change to jenkins-slave manifest (like https://review.openstack.org/#/c/260230 for instance) take effect relatively soon after merge? | 00:19 |
| *** apuimedo has quit IRC | 00:20 | |
| *** apuimedo has joined #openstack-infra | 00:20 | |
| *** apuimedo_ has joined #openstack-infra | 00:21 | |
| *** Qiming has joined #openstack-infra | 00:22 | |
| *** weshay_xchat has quit IRC | 00:23 | |
| fungi | Clint: yep, officially cynical in my old age i guess | 00:26 |
| *** tzn has joined #openstack-infra | 00:27 | |
| Clint | fungi: this is why we can't have nice anti-spam software | 00:27 |
| *** EricGonczer_ has quit IRC | 00:27 | |
| *** irenab has joined #openstack-infra | 00:28 | |
| *** Jay__ has joined #openstack-infra | 00:28 | |
| *** irenab has quit IRC | 00:29 | |
| * fungi has nice anti-spam software | 00:29 | |
| fungi | at least spamassassin is working pretty well for me anyway | 00:29 |
| *** mudassirlatif has quit IRC | 00:31 | |
| *** apuimedo has quit IRC | 00:32 | |
| *** eliqiao has joined #openstack-infra | 00:33 | |
| *** apuimedo has joined #openstack-infra | 00:33 | |
| *** eliqiao_ has quit IRC | 00:35 | |
| *** gildub has quit IRC | 00:36 | |
| *** apuimedo_ has quit IRC | 00:37 | |
| *** apuimedo has quit IRC | 00:38 | |
| *** apuimedo has joined #openstack-infra | 00:38 | |
| *** apuimedo_ has joined #openstack-infra | 00:39 | |
| *** dims has quit IRC | 00:42 | |
| jeblair | docaedo: should within 30m or so... | 00:42 |
| *** apuimedo has quit IRC | 00:43 | |
| docaedo | jeblair: cool, thanks - now mordred's patch just needs a little love :) | 00:43 |
| *** apuimedo_ has quit IRC | 00:43 | |
| *** notmorgan has joined #openstack-infra | 00:44 | |
| *** notmorgan has quit IRC | 00:44 | |
| *** notmorgan has joined #openstack-infra | 00:44 | |
| *** apuimedo has joined #openstack-infra | 00:44 | |
| *** irenab has joined #openstack-infra | 00:45 | |
| *** apuimedo_ has joined #openstack-infra | 00:45 | |
| *** irenab has quit IRC | 00:45 | |
| *** xarses has joined #openstack-infra | 00:45 | |
| *** asselin has joined #openstack-infra | 00:47 | |
| SpamapS | is it just me, or is login.launchpad.net down? | 00:49 |
| *** EricGonczer_ has joined #openstack-infra | 00:49 | |
| lifeless | SpamapS: what? | 00:49 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure the Gerrit's sendmail.includeDiff option https://review.openstack.org/260283 | 00:49 |
| lifeless | SpamapS: that isn't a host, is it? | 00:49 |
| SpamapS | lifeless: it's the one that gerrit sends me to | 00:49 |
| *** apuimedo_ has quit IRC | 00:50 | |
| SpamapS | for my https://launchpad.net/~clint-fewbar openid url | 00:50 |
| lifeless | hmm, LP should really update its openid redirects | 00:50 |
| lifeless | anyhow | 00:50 |
| lifeless | its up | 00:50 |
| SpamapS | oh yeah it worked now | 00:50 |
| *** irenab has joined #openstack-infra | 00:50 | |
| SpamapS | lifeless: I think it actually goes to login.ubuntu.com now | 00:50 |
| *** salv-orlando has quit IRC | 00:51 | |
| SpamapS | hm still different IPs | 00:51 |
| lifeless | yes, because old ssl stacks | 00:51 |
| *** apuimedo_ has joined #openstack-infra | 00:51 | |
| *** hdd has quit IRC | 00:51 | |
| *** apuimedo has quit IRC | 00:51 | |
| *** irenab has quit IRC | 00:51 | |
| *** apuimedo has joined #openstack-infra | 00:52 | |
| *** apuimedo_ has quit IRC | 00:56 | |
| *** apuimedo has quit IRC | 00:57 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add experimental job to check Rally installation on Centos 7 https://review.openstack.org/260258 | 00:57 |
| *** apuimedo has joined #openstack-infra | 00:58 | |
| *** eil397 has quit IRC | 00:58 | |
| *** apuimedo_ has joined #openstack-infra | 00:58 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Include unified diffs in Gerrit notification emails https://review.openstack.org/260284 | 00:59 |
| *** irenab has joined #openstack-infra | 00:59 | |
| *** tzn has quit IRC | 01:01 | |
| *** irenab has quit IRC | 01:02 | |
| *** apuimedo_ has quit IRC | 01:03 | |
| *** apuimedo_ has joined #openstack-infra | 01:04 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure the Gerrit's sendmail.includeDiff option https://review.openstack.org/260283 | 01:06 |
| *** apuimedo has quit IRC | 01:07 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure the Gerrit's sendmail.includeDiff option https://review.openstack.org/260283 | 01:07 |
| *** angdraug has quit IRC | 01:07 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Include unified diffs in Gerrit notification emails https://review.openstack.org/260284 | 01:08 |
| *** Swami has quit IRC | 01:08 | |
| *** irenab has joined #openstack-infra | 01:08 | |
| *** apuimedo has joined #openstack-infra | 01:08 | |
| *** irenab has quit IRC | 01:08 | |
| *** apuimedo_ has quit IRC | 01:09 | |
| *** sdake has quit IRC | 01:10 | |
| *** apuimedo_ has joined #openstack-infra | 01:10 | |
| *** baoli has joined #openstack-infra | 01:11 | |
| *** irenab has joined #openstack-infra | 01:12 | |
| *** Jay__ has quit IRC | 01:13 | |
| *** Jay__ has joined #openstack-infra | 01:13 | |
| *** apuimedo has quit IRC | 01:15 | |
| *** baoli has quit IRC | 01:16 | |
| *** dims has joined #openstack-infra | 01:16 | |
| *** apuimedo has joined #openstack-infra | 01:16 | |
| *** zeih has joined #openstack-infra | 01:16 | |
| fungi | okay, well i'm just going to take a leap of faith in 260213 and then go through the manual jjb updates everywhere | 01:16 |
| *** apuimedo_ has quit IRC | 01:17 | |
| fungi | jhesketh: ^ just a heads up this is happening in your daytime, pass it along | 01:17 |
| *** zhenguo has joined #openstack-infra | 01:17 | |
| *** apuimedo_ has joined #openstack-infra | 01:17 | |
| fungi | added a note in my approval comment about what i'll be running where, in case anyone needs to revert and redo | 01:19 |
| *** lykinsbd has quit IRC | 01:20 | |
| *** zeih has quit IRC | 01:21 | |
| *** apuimedo has quit IRC | 01:21 | |
| *** lykinsbd has joined #openstack-infra | 01:22 | |
| anteaya | fungi: sounds good | 01:22 |
| *** apuimedo_ has quit IRC | 01:22 | |
| anteaya | I have to be up in about 6 hours so I'm going to go to bed now | 01:22 |
| *** apuimedo has joined #openstack-infra | 01:22 | |
| anteaya | I'll read backscroll when I'm awake again | 01:23 |
| anteaya | night | 01:23 |
| *** apuimedo_ has joined #openstack-infra | 01:23 | |
| *** lykinsbd_ has joined #openstack-infra | 01:23 | |
| tonyb | I saw this come up somewhere but what does | 01:23 |
| tonyb | "NOT_REGISTERED" mean from a Jenkins POV? | 01:24 |
| openstackgerrit | Merged openstack-infra/project-config: Get rid of precise||trusty job node declarations https://review.openstack.org/260213 | 01:25 |
| *** lykinsbd has quit IRC | 01:27 | |
| *** PsionTheory has joined #openstack-infra | 01:27 | |
| fungi | tonyb: means that zuul couldn't find a registration for the job it wanted run, possibly because the job hadn't been created on the jenkins masters yet (they tend to update more slowly than zuul does) or because it's looking for it in combination with the wrong node type (260213 in combination with its child change seek to address that second complexity) | 01:27 |
| *** Daisy has joined #openstack-infra | 01:28 | |
| *** gyee has quit IRC | 01:29 | |
| tonyb | fungi: Thanks. | 01:29 |
| *** sputnik1_ has quit IRC | 01:30 | |
| fungi | normally zuul passes along jenkins job states (success, fail, unstable, abort) but in this case it's made up one because jenkins doesn't have a "state" for nonexistent jobs (for obvious reasons) | 01:32 |
| fungi | well, s/state/result/ | 01:32 |
| *** irenab has joined #openstack-infra | 01:33 | |
| *** irenab has quit IRC | 01:33 | |
| *** apuimedo_ has quit IRC | 01:34 | |
| tonyb | fungi: okay. I'll look into when the job. It's a f23 job so I don't think the work you're doing for precise is related. | 01:35 |
| *** apuimedo_ has joined #openstack-infra | 01:35 | |
| *** sputnik13 has joined #openstack-infra | 01:35 | |
| tonyb | Ahh I see someone recently switched from f21 -> f23 | 01:35 |
| *** larainema has quit IRC | 01:36 | |
| *** jyuso1 has joined #openstack-infra | 01:36 | |
| *** apuimedo has quit IRC | 01:36 | |
| tonyb | fungi: Thanks again. | 01:36 |
| *** irenab has joined #openstack-infra | 01:36 | |
| *** irenab has quit IRC | 01:36 | |
| fungi | yeah, and we've been having trouble building fedora 23 workers | 01:36 |
| fungi | ianw has some updates to diskimage-builder which should get that going | 01:37 |
| ianw | yep, should be working soon! | 01:37 |
| * ianw is a bit sick of saying that, but i believe it, sort of, now | 01:37 | |
| *** irenab has joined #openstack-infra | 01:38 | |
| *** irenab has quit IRC | 01:38 | |
| *** apuimedo has joined #openstack-infra | 01:38 | |
| *** larainema has joined #openstack-infra | 01:39 | |
| *** Daisy has quit IRC | 01:39 | |
| *** Daisy has joined #openstack-infra | 01:40 | |
| *** lykinsbd has joined #openstack-infra | 01:40 | |
| *** dims has quit IRC | 01:40 | |
| *** apuimedo_ has quit IRC | 01:42 | |
| *** PsionTheory_ has joined #openstack-infra | 01:42 | |
| *** dimtruck is now known as zz_dimtruck | 01:43 | |
| *** apuimedo_ has joined #openstack-infra | 01:43 | |
| *** asselin__ has joined #openstack-infra | 01:43 | |
| *** lykinsbd_ has quit IRC | 01:43 | |
| *** lykinsbd_ has joined #openstack-infra | 01:44 | |
| *** gongysh_ has joined #openstack-infra | 01:44 | |
| *** Daisy has quit IRC | 01:44 | |
| *** triggerwarning has joined #openstack-infra | 01:45 | |
| *** asselin_ has quit IRC | 01:45 | |
| *** lykinsbd_ has quit IRC | 01:45 | |
| *** triggerwarning has quit IRC | 01:45 | |
| *** triggerwarning has joined #openstack-infra | 01:45 | |
| *** PsionTheory has quit IRC | 01:45 | |
| *** TongMaster is now known as ruagair | 01:45 | |
| *** ruagair is now known as Guest56931 | 01:46 | |
| *** Daisy has joined #openstack-infra | 01:46 | |
| *** lykinsbd has quit IRC | 01:46 | |
| *** Qiming has quit IRC | 01:47 | |
| *** EricGonczer_ has quit IRC | 01:47 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Include query results in graph https://review.openstack.org/260188 | 01:48 |
| *** Guest56931 is now known as TongMaster | 01:48 | |
| *** jerryz has quit IRC | 01:49 | |
| *** oomichi_away is now known as oomichi | 01:49 | |
| *** TongMaster is now known as craige | 01:49 | |
| *** zz_dimtruck is now known as dimtruck | 01:50 | |
| *** dims has joined #openstack-infra | 01:50 | |
| *** Daisy has quit IRC | 01:51 | |
| *** gongysh_ has quit IRC | 01:51 | |
| *** irenab has joined #openstack-infra | 01:52 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config: Build a Debian simple-init images https://review.openstack.org/254892 | 01:52 |
| ianw | greghaynes / SpamapS : appreciate any comments on -> https://www.technovelty.org/openstack/platform-bootstrap-for-openstack-infrastructure.html | 01:53 |
| *** mriedem_away has quit IRC | 01:53 | |
| *** irenab has joined #openstack-infra | 01:54 | |
| greghaynes | ianw: oooo | 01:54 |
| *** apuimedo_ has quit IRC | 01:54 | |
| *** apuimedo has quit IRC | 01:54 | |
| *** irenab has quit IRC | 01:54 | |
| *** apuimedo has joined #openstack-infra | 01:55 | |
| *** apuimedo_ has joined #openstack-infra | 01:55 | |
| *** mriedem has joined #openstack-infra | 01:58 | |
| *** apuimedo_ has quit IRC | 02:00 | |
| *** apuimedo_ has joined #openstack-infra | 02:00 | |
| *** apuimedo has quit IRC | 02:02 | |
| *** apuimedo has joined #openstack-infra | 02:03 | |
| *** markvoelker has joined #openstack-infra | 02:03 | |
| *** dims has quit IRC | 02:06 | |
| *** apoorvad has quit IRC | 02:07 | |
| *** markvoelker has quit IRC | 02:07 | |
| *** apuimedo has quit IRC | 02:07 | |
| *** apuimedo_ has quit IRC | 02:07 | |
| *** Daisy has joined #openstack-infra | 02:08 | |
| fungi | okay, looks like it's been long enough for 260213 to be on the jenkins masters according to my grep of the local config copies on them, so i'm starting manual job update runs on all 8 now | 02:08 |
| *** apuimedo has joined #openstack-infra | 02:08 | |
| fungi | will keep tabs on them over the next few hours and restart any as needed until they all complete | 02:08 |
| *** apuimedo_ has joined #openstack-infra | 02:08 | |
| *** [1]Thelo has joined #openstack-infra | 02:11 | |
| *** zhurong has joined #openstack-infra | 02:11 | |
| *** apuimedo has quit IRC | 02:13 | |
| *** Thelo has quit IRC | 02:13 | |
| *** [1]Thelo is now known as Thelo | 02:13 | |
| *** apuimedo has joined #openstack-infra | 02:13 | |
| * craige notes it's after midday on his last working day of the year and opens a beer. Happy holidays! | 02:14 | |
| *** sdake has joined #openstack-infra | 02:16 | |
| *** Qiming has joined #openstack-infra | 02:17 | |
| fungi | cheers, craige! | 02:17 |
| fungi | try not to break anything on your way out | 02:17 |
| fungi | ;) | 02:17 |
| *** zeih has joined #openstack-infra | 02:17 | |
| *** apuimedo has quit IRC | 02:18 | |
| *** apuimedo_ has quit IRC | 02:18 | |
| *** apuimedo has joined #openstack-infra | 02:19 | |
| *** apuimedo_ has joined #openstack-infra | 02:20 | |
| *** zeih has quit IRC | 02:22 | |
| *** apuimedo has quit IRC | 02:24 | |
| *** apuimedo has joined #openstack-infra | 02:26 | |
| *** apuimedo_ has quit IRC | 02:27 | |
| *** apuimedo_ has joined #openstack-infra | 02:28 | |
| *** dims has joined #openstack-infra | 02:29 | |
| *** piet has quit IRC | 02:30 | |
| *** Daisy has quit IRC | 02:30 | |
| *** Daisy has joined #openstack-infra | 02:31 | |
| *** hichihara has quit IRC | 02:32 | |
| *** bardia has quit IRC | 02:32 | |
| *** apuimedo_ has quit IRC | 02:33 | |
| *** apuimedo has quit IRC | 02:33 | |
| *** apuimedo has joined #openstack-infra | 02:34 | |
| *** sdake_ has joined #openstack-infra | 02:34 | |
| *** apuimedo_ has joined #openstack-infra | 02:34 | |
| *** sdake has quit IRC | 02:34 | |
| *** Daisy has quit IRC | 02:35 | |
| *** unicell1 has quit IRC | 02:37 | |
| craige | I'm going to find who wrote launchpadlib and give them the longest, most uncomfortable hug ever. | 02:38 |
| *** apuimedo has quit IRC | 02:38 | |
| *** harlowja_at_home has joined #openstack-infra | 02:39 | |
| *** apuimedo has joined #openstack-infra | 02:39 | |
| *** zzzeek has quit IRC | 02:44 | |
| *** apuimedo has quit IRC | 02:44 | |
| *** apuimedo_ has quit IRC | 02:44 | |
| *** apuimedo has joined #openstack-infra | 02:44 | |
| *** apuimedo_ has joined #openstack-infra | 02:45 | |
| *** dims has quit IRC | 02:45 | |
| *** zzzeek has joined #openstack-infra | 02:45 | |
| *** dims has joined #openstack-infra | 02:47 | |
| *** apuimedo has quit IRC | 02:51 | |
| *** apuimedo_ has quit IRC | 02:52 | |
| *** apuimedo has joined #openstack-infra | 02:53 | |
| *** apuimedo_ has joined #openstack-infra | 02:53 | |
| *** salv-orlando has joined #openstack-infra | 02:55 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Update query for nova db deadlock bug 1480305 https://review.openstack.org/260300 | 02:55 |
| openstack | bug 1480305 in OpenStack Compute (nova) "DBDeadlock inserting into instance_extra" [Medium,Confirmed] https://launchpad.net/bugs/1480305 | 02:55 |
| *** apuimedo_ has quit IRC | 02:57 | |
| *** tzn has joined #openstack-infra | 02:58 | |
| *** apuimedo_ has joined #openstack-infra | 02:59 | |
| *** apuimedo has quit IRC | 02:59 | |
| *** apuimedo has joined #openstack-infra | 03:01 | |
| *** appprod0 has joined #openstack-infra | 03:01 | |
| *** appprod0 has left #openstack-infra | 03:01 | |
| *** appprod0 has joined #openstack-infra | 03:02 | |
| *** tzn has quit IRC | 03:03 | |
| *** apuimedo_ has quit IRC | 03:03 | |
| *** markvoelker has joined #openstack-infra | 03:04 | |
| *** apuimedo_ has joined #openstack-infra | 03:04 | |
| *** baoli has joined #openstack-infra | 03:05 | |
| *** apuimedo has quit IRC | 03:05 | |
| *** rvasilets___ has quit IRC | 03:05 | |
| *** links has joined #openstack-infra | 03:06 | |
| *** apuimedo has joined #openstack-infra | 03:06 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Update query for nova db deadlock bug 1480305 https://review.openstack.org/260300 | 03:07 |
| openstack | bug 1480305 in OpenStack Compute (nova) "DBDeadlock inserting into instance_extra" [Medium,Confirmed] https://launchpad.net/bugs/1480305 | 03:07 |
| *** baoli has quit IRC | 03:08 | |
| *** dims has quit IRC | 03:08 | |
| *** markvoelker has quit IRC | 03:08 | |
| *** harlowja_at_home has quit IRC | 03:09 | |
| *** harlowja_at_home has joined #openstack-infra | 03:09 | |
| *** apuimedo has quit IRC | 03:10 | |
| *** apuimedo has joined #openstack-infra | 03:12 | |
| *** piet has joined #openstack-infra | 03:13 | |
| *** ramineni has joined #openstack-infra | 03:13 | |
| *** yamahata has quit IRC | 03:14 | |
| *** harlowja_at_home has quit IRC | 03:14 | |
| *** thorst has joined #openstack-infra | 03:14 | |
| *** thorst has quit IRC | 03:16 | |
| *** apuimedo_ has quit IRC | 03:16 | |
| *** greghaynes has quit IRC | 03:16 | |
| *** thorst has joined #openstack-infra | 03:16 | |
| *** apuimedo_ has joined #openstack-infra | 03:17 | |
| *** greghaynes has joined #openstack-infra | 03:17 | |
| *** mriedem has quit IRC | 03:19 | |
| *** thorst has quit IRC | 03:20 | |
| *** gongysh_ has joined #openstack-infra | 03:21 | |
| *** coolsvap|away is now known as coolsvap | 03:21 | |
| *** vilobhmm111 has quit IRC | 03:21 | |
| *** apuimedo_ has quit IRC | 03:21 | |
| *** apuimedo_ has joined #openstack-infra | 03:22 | |
| fungi | jenkins03-7 took around 1.25 hours to finish updating, 01-2 are still underway | 03:25 |
| *** rlandy has quit IRC | 03:26 | |
| *** apuimedo_ has quit IRC | 03:26 | |
| *** apuimedo has quit IRC | 03:27 | |
| *** apuimedo has joined #openstack-infra | 03:27 | |
| *** baoli has joined #openstack-infra | 03:27 | |
| *** apuimedo_ has joined #openstack-infra | 03:27 | |
| *** baoli has quit IRC | 03:28 | |
| *** piet has quit IRC | 03:28 | |
| *** Daisy has joined #openstack-infra | 03:31 | |
| *** apuimedo has quit IRC | 03:31 | |
| *** apuimedo has joined #openstack-infra | 03:33 | |
| *** baoli has joined #openstack-infra | 03:34 | |
| *** salv-orlando has quit IRC | 03:35 | |
| *** baoli has quit IRC | 03:35 | |
| *** harlowja_at_home has joined #openstack-infra | 03:39 | |
| *** apuimedo_ has quit IRC | 03:39 | |
| *** apuimedo_ has joined #openstack-infra | 03:40 | |
| *** Daisy has quit IRC | 03:40 | |
| *** apuimedo has quit IRC | 03:40 | |
| janonymous | Can someone review a Hacking patch : https://review.openstack.org/#/c/259729/ | 03:40 |
| *** Daisy has joined #openstack-infra | 03:41 | |
| *** apuimedo has joined #openstack-infra | 03:41 | |
| *** apuimedo_ has quit IRC | 03:45 | |
| *** Daisy has quit IRC | 03:45 | |
| *** apuimedo has quit IRC | 03:45 | |
| *** apuimedo has joined #openstack-infra | 03:46 | |
| *** apuimedo_ has joined #openstack-infra | 03:47 | |
| *** gildub has joined #openstack-infra | 03:48 | |
| *** zhurong has quit IRC | 03:48 | |
| fungi | jenkins01 is done, 02 still going | 03:50 |
| *** apuimedo_ has quit IRC | 03:51 | |
| *** apuimedo has quit IRC | 03:53 | |
| *** apuimedo has joined #openstack-infra | 03:53 | |
| *** apuimedo_ has joined #openstack-infra | 03:54 | |
| *** gongysh_ has quit IRC | 03:55 | |
| *** apuimedo has quit IRC | 03:58 | |
| *** apuimedo has joined #openstack-infra | 03:59 | |
| *** irenab has joined #openstack-infra | 03:59 | |
| *** tzn has joined #openstack-infra | 03:59 | |
| *** apuimedo_ has quit IRC | 04:00 | |
| *** irenab has quit IRC | 04:00 | |
| *** apuimedo_ has joined #openstack-infra | 04:01 | |
| *** apuimedo has quit IRC | 04:03 | |
| *** tzn has quit IRC | 04:04 | |
| fungi | janonymous: you're more likely to find reviewers for the openstack-dev/hacking repo active in #openstack-qa rather than here, since it's under the control of the qa team http://governance.openstack.org/reference/projects/quality-assurance.html | 04:04 |
| *** apuimedo has joined #openstack-infra | 04:04 | |
| *** irenab has joined #openstack-infra | 04:06 | |
| *** PsionTheory_ has quit IRC | 04:06 | |
| *** irenab has quit IRC | 04:06 | |
| *** apuimedo_ has quit IRC | 04:06 | |
| *** apuimedo_ has joined #openstack-infra | 04:06 | |
| *** irenab has joined #openstack-infra | 04:10 | |
| *** irenab has quit IRC | 04:10 | |
| *** apuimedo_ has quit IRC | 04:11 | |
| fungi | jenkins02 done (2 hours 5 minutes wall clock time) | 04:13 |
| sputnik13 | is there a way to get the demo or admin user credentials in devstack short of sourcing openrc? | 04:13 |
| sputnik13 | I mean as part of a plugin | 04:13 |
| sputnik13 | not as a user | 04:13 |
| *** apuimedo_ has joined #openstack-infra | 04:13 | |
| *** irenab has joined #openstack-infra | 04:14 | |
| janonymous | fungi: hey thanks, i would ask for review in there | 04:14 |
| fungi | removed my -2 from 260214 now that the jjb manual updates are all completed | 04:16 |
| *** irenab has joined #openstack-infra | 04:17 | |
| *** apuimedo_ has quit IRC | 04:18 | |
| *** irenab has quit IRC | 04:18 | |
| *** apuimedo has quit IRC | 04:19 | |
| *** zeih has joined #openstack-infra | 04:19 | |
| *** apuimedo has joined #openstack-infra | 04:20 | |
| *** apuimedo_ has joined #openstack-infra | 04:20 | |
| *** Thelo has quit IRC | 04:22 | |
| *** Ramanjaneya has joined #openstack-infra | 04:24 | |
| clarkb | sputnik13 read the vars directly? | 04:24 |
| *** fawadkhaliq has joined #openstack-infra | 04:25 | |
| *** zeih has quit IRC | 04:25 | |
| sputnik13 | right, the obvious answer :) | 04:26 |
| sputnik13 | except I see ADMIN_PASSWORD, but don't see a DEMO_PASSWORD | 04:26 |
| clarkb | they are the same iirc | 04:26 |
| sputnik13 | must have tunnel vision, don't see it >_< | 04:26 |
| *** apuimedo has quit IRC | 04:27 | |
| *** apuimedo_ has quit IRC | 04:27 | |
| *** irenab has joined #openstack-infra | 04:27 | |
| sputnik13 | duh, ok openrc actually has a comment "# Set ADMIN_PASSWORD to set the password for 'admin' and 'demo'" | 04:27 |
| *** irenab has quit IRC | 04:27 | |
| sputnik13 | thx | 04:28 |
| *** apuimedo has joined #openstack-infra | 04:28 | |
| *** apuimedo_ has joined #openstack-infra | 04:28 | |
| clarkb | fungi still around? | 04:30 |
| clarkb | I can approve 260214 but not really babysit | 04:30 |
| fungi | clarkb: yeah, were 260213/260214 sane? | 04:30 |
| fungi | i can babysit for another couple hours | 04:30 |
| clarkb | (tending to real life babies) | 04:30 |
| fungi | hah | 04:30 |
| clarkb | 260214 looks sane | 04:30 |
| fungi | also trying to make sure jhesketh is aware, and yolanda once she wakes up | 04:31 |
| * jhesketh is here | 04:31 | |
| fungi | just in case any of this needs undoing because zomgsobroken | 04:31 |
| jhesketh | I saw 260213 go in and look okay | 04:32 |
| fungi | as opposed to a job here or there which was somehow using the wrong label or something | 04:32 |
| clarkb | +2'd approve at will | 04:32 |
| jhesketh | looking at 214 now | 04:32 |
| *** irenab has joined #openstack-infra | 04:32 | |
| fungi | i'll let jhesketh take a peek before i do | 04:32 |
| fungi | that one's short at least | 04:32 |
| fungi | the parent change was freaking huge, but also likely to merge conflict for all eternity if we didn't approve quickly | 04:33 |
| *** irenab has quit IRC | 04:33 | |
| *** nikhil has joined #openstack-infra | 04:33 | |
| *** fawadkhaliq has quit IRC | 04:34 | |
| *** irenab has joined #openstack-infra | 04:36 | |
| *** vilobhmm11 has joined #openstack-infra | 04:36 | |
| *** nikhil_k has quit IRC | 04:36 | |
| *** apuimedo has quit IRC | 04:37 | |
| *** apuimedo_ has quit IRC | 04:37 | |
| *** irenab has quit IRC | 04:37 | |
| *** apuimedo has joined #openstack-infra | 04:38 | |
| jhesketh | fungi: with that change I'm confused how devstack gets the right node now | 04:38 |
| *** apuimedo_ has joined #openstack-infra | 04:39 | |
| fungi | jhesketh: er, because the nodes that can run dsvm jobs are the ones which register them? | 04:39 |
| fungi | or do we have a hole there? | 04:40 |
| jhesketh | hmm I haven't looked at the node allocation in ages... I remember the jenkins/gearman/zuul stuff being a little bit magic | 04:40 |
| ramineni | hi , can someone do quick review on https://review.openstack.org/#/c/255654/ , it has 2 +2Å› already | 04:40 |
| fungi | right now the jjb configs for them should only be listing node: bare-trusty | 04:40 |
| jhesketh | in the way that the host type is added to gearman for the jenkins-gearman plugin | 04:40 |
| fungi | er, sorry, node: devstack-trusty i mean | 04:41 |
| jhesketh | right, it is possible we have some jobs missing that, but we can catch those | 04:41 |
| *** unicell has joined #openstack-infra | 04:41 | |
| jhesketh | how does jenkins know which slaves are which for the jobs though | 04:41 |
| *** irenab has joined #openstack-infra | 04:41 | |
| fungi | and i thought as long as zuul isn't told a specific node label to look for it just went with build:job-name instead of build:job-name:node-label | 04:41 |
| fungi | the parent patch of that one replaced all the "node: devstack-precise || devstack-trusty" with just "node: devstack-trusty" so there should no longer be multiple node types registering the same jobs | 04:43 |
| jhesketh | right, zuul should do that | 04:44 |
| jhesketh | and yes, the jobs should have their node defined in jenkins | 04:44 |
| jhesketh | what I'm not sure is how does jenkins know what slave is what... | 04:44 |
| jhesketh | is that nodepool registration that does that | 04:44 |
| * jhesketh checks | 04:44 | |
| *** apuimedo has quit IRC | 04:45 | |
| fungi | yeah, nodepool sets a common node label on each one as it connects it to the jenkins master as a slave | 04:45 |
| *** apuimedo_ has quit IRC | 04:46 | |
| *** apuimedo has joined #openstack-infra | 04:47 | |
| *** apuimedo_ has joined #openstack-infra | 04:47 | |
| *** rguillebert has quit IRC | 04:47 | |
| fungi | but definitely thanks for walking back through all this and confirming my recollections. it's a change with a large potential for impact to pretty much all jobs | 04:47 |
| jhesketh | okay I can see that the nodes are created with the name given.. does the jenkins-gearman plugin use that failing a ZUUL_NODE from gearman? | 04:48 |
| * jhesketh now checks jenkins-gearman | 04:48 | |
| fungi | if you look at the job registrations for a job, you'll see it gets workers registered with and without the :<node-label> suffix | 04:49 |
| fungi | you can see them in the gearman status output | 04:49 |
| *** markvoelker has joined #openstack-infra | 04:49 | |
| jhesketh | yeah I see the suffix, I'm wondering once zuul stops giving that suffix to jenkins what it'll fall back on | 04:50 |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: congress: disable tempest plugin for stable branches https://review.openstack.org/258799 | 04:50 |
| *** piet has joined #openstack-infra | 04:51 | |
| *** apuimedo_ has quit IRC | 04:52 | |
| *** apuimedo_ has joined #openstack-infra | 04:53 | |
| *** markvoelker has quit IRC | 04:54 | |
| *** apuimedo has quit IRC | 04:58 | |
| *** eil397 has joined #openstack-infra | 04:58 | |
| *** apuimedo has joined #openstack-infra | 05:00 | |
| *** jamespage has quit IRC | 05:00 | |
| *** gnuoy has quit IRC | 05:00 | |
| *** tzn has joined #openstack-infra | 05:00 | |
| *** gnuoy has joined #openstack-infra | 05:00 | |
| *** armax has quit IRC | 05:01 | |
| *** Daisy has joined #openstack-infra | 05:02 | |
| jhesketh | fungi: so I think the presence of nodelabelparameter jenkins plugin is what does the magic for us | 05:03 |
| *** fawadkhaliq has joined #openstack-infra | 05:04 | |
| *** apuimedo has quit IRC | 05:04 | |
| *** tzn has quit IRC | 05:05 | |
| *** apuimedo_ has quit IRC | 05:05 | |
| *** piet has quit IRC | 05:05 | |
| fungi | This plugin adds two new parameter types to job configuration - node and label, this allows to dynamically select the node where a job/project should be executed. | 05:06 |
| fungi | yep, we're definitely using it | 05:06 |
| fungi | so that's what allows us to specify a job should run on a node with a particular label | 05:06 |
| *** apuimedo has joined #openstack-infra | 05:06 | |
| fungi | i hadn't realized that was a plugin, just always thought it was a builtin feature | 05:07 |
| *** apuimedo_ has joined #openstack-infra | 05:07 | |
| jhesketh | fungi: yep, so I've confirmed nodepool sets a label for each node it attaches to jenkins | 05:08 |
| jhesketh | the documentation on https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin makes it sound like a label is set on a job | 05:09 |
| jhesketh | but it seems like the label is set on the node and a "node" type is set on the job | 05:09 |
| fungi | yeah, that's basically it | 05:09 |
| jhesketh | assuming the jjb defintion of "node" sets it correctly for the job and jenkins matches that to a node label we shoudl be all good | 05:09 |
| * jhesketh is possibly just failing to grep the plugin doc | 05:10 | |
| fungi | nodes are labeled, and you can optionally specify a label or the name of a specific node to which the job should be restricted | 05:10 |
| jhesketh | fungi: either way, I'm +2 on the patch, but it certainly needs baby sitting | 05:10 |
| fungi | yeah, i'll stick around and fire some test jobs once it merges | 05:10 |
| jhesketh | I was about to say I'm happy to watch it | 05:11 |
| fungi | the earlier in the week we have this done the more likely it is we'll catch any omissions that need fixing before i'm gone for holidays | 05:11 |
| jhesketh | given it must be silly o'clock for you | 05:11 |
| *** apuimedo has quit IRC | 05:11 | |
| fungi | well, your eye will be less bleary than mine ;) | 05:11 |
| fungi | approve at will, and i'm around for an hour or two just in case | 05:11 |
| *** apuimedo_ has quit IRC | 05:11 | |
| jhesketh | fungi: will do | 05:11 |
| * jhesketh +w | 05:12 | |
| fungi | i also have a job which is currently not_registered which should in theory be fixed by this | 05:12 |
| fungi | thanks! | 05:12 |
| fungi | which was what lit a fire under me to just get it cleaned up finally | 05:12 |
| *** apuimedo has joined #openstack-infra | 05:12 | |
| *** apuimedo_ has joined #openstack-infra | 05:13 | |
| *** piet has joined #openstack-infra | 05:13 | |
| *** apuimedo has quit IRC | 05:19 | |
| *** apuimedo has joined #openstack-infra | 05:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove precise/trusty label mapping in Zuul config https://review.openstack.org/260214 | 05:21 |
| fungi | and now the fun begins | 05:21 |
| jhesketh | heh | 05:22 |
| *** sdake_ has quit IRC | 05:23 | |
| tangchen | Hi everyone, is anybody familiar with XenProject CI in Nova ? Please take a look this patch. https://review.openstack.org/#/c/202605/ | 05:23 |
| tangchen | The XenProject CI always fails to merge the patch. But I don't think it is the patch's problem. Would anyone please tell me how to recheck it ? | 05:24 |
| tangchen | Anyway, the gating job started. Not sure if the XenProject CI failure matters. | 05:26 |
| zaro | tangchen: i believe 3rd party CI votes do not block approvals. | 05:29 |
| tangchen | zaro: Oh, I see. Thanks a lot. :) | 05:31 |
| *** apuimedo_ has quit IRC | 05:32 | |
| *** apuimedo_ has joined #openstack-infra | 05:33 | |
| *** apuimedo has quit IRC | 05:37 | |
| *** apuimedo_ has quit IRC | 05:37 | |
| *** apuimedo has joined #openstack-infra | 05:38 | |
| *** abregman has joined #openstack-infra | 05:38 | |
| *** apuimedo_ has joined #openstack-infra | 05:38 | |
| *** salv-orlando has joined #openstack-infra | 05:41 | |
| *** eil397 has quit IRC | 05:41 | |
| *** harlowja_at_home has quit IRC | 05:42 | |
| *** apuimedo_ has quit IRC | 05:43 | |
| *** apuimedo_ has joined #openstack-infra | 05:44 | |
| openstackgerrit | Gal Sagie proposed openstack-infra/project-config: Add rally-plot publisher to OVN, Dragonflow and Kuryr rally jobs https://review.openstack.org/260330 | 05:46 |
| *** baoli has joined #openstack-infra | 05:47 | |
| *** apuimedo has quit IRC | 05:47 | |
| *** apuimedo_ has quit IRC | 05:48 | |
| *** apuimedo has joined #openstack-infra | 05:49 | |
| fungi | jhesketh: i see zuul was last reconfigured 27 minutes ago and any changes enqueued for less than that aren't getting workers assigned to their jobs | 05:49 |
| fungi | jhesketh: looking at debug logs now | 05:49 |
| *** apuimedo_ has joined #openstack-infra | 05:49 | |
| *** markvoelker has joined #openstack-infra | 05:50 | |
| *** maishsk has quit IRC | 05:50 | |
| *** baoli has quit IRC | 05:51 | |
| fungi | okay, this is strange. gearman status shows build:gate-tempest-dsvm-full:devstack-trusty registered but no build:gate-tempest-dsvm-full | 05:52 |
| fungi | nevermind. grep fail | 05:52 |
| jhesketh | :-) | 05:53 |
| jhesketh | fungi: chances are the full name will stick around for a while | 05:53 |
| jhesketh | I can't recall when gearman forgets about them, but I think it might only be on restart | 05:53 |
| fungi | build:gate-tempest-dsvm-full:devstack-trusty 8 8 111 | 05:53 |
| fungi | build:gate-tempest-dsvm-full 0 0 111 | 05:53 |
| jhesketh | which means we won't get any not-registered | 05:53 |
| clarkb | ya I think it remembers forever until you kill the process | 05:54 |
| fungi | so it is claiming to have the same workers | 05:54 |
| jhesketh | actually those workers will keep registering as that name | 05:54 |
| fungi | right, no i was expecting to see them | 05:54 |
| fungi | i simply had a malformed grep | 05:54 |
| fungi | so that part looks right | 05:54 |
| *** tlian has quit IRC | 05:54 | |
| *** zhurong has joined #openstack-infra | 05:55 | |
| *** markvoelker has quit IRC | 05:55 | |
| *** apuimedo has quit IRC | 05:56 | |
| *** apuimedo_ has quit IRC | 05:56 | |
| *** apuimedo has joined #openstack-infra | 05:57 | |
| *** apuimedo_ has joined #openstack-infra | 05:58 | |
| *** otter768 has quit IRC | 05:59 | |
| *** yamahata has joined #openstack-infra | 05:59 | |
| *** tzn has joined #openstack-infra | 06:01 | |
| fungi | okay, i see the problem, i think | 06:02 |
| fungi | http://paste.openstack.org/show/482470/ | 06:02 |
| *** _nadya_ has joined #openstack-infra | 06:04 | |
| jhesketh | fungi: woops, do you have a patch incoming? | 06:04 |
| *** apuimedo has quit IRC | 06:04 | |
| jhesketh | otherwise I can put it togheter | 06:04 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Correct set_node_options argument list https://review.openstack.org/260334 | 06:05 |
| fungi | jhesketh: ^ plain brown wrapper | 06:05 |
| *** apuimedo has joined #openstack-infra | 06:05 | |
| *** tzn has quit IRC | 06:05 | |
| jhesketh | fungi: +w | 06:05 |
| fungi | much obliged | 06:06 |
| jhesketh | fungi: however I think we'll need to apply manually | 06:06 |
| fungi | oh? you think the current state will prevent it from reloading its config? | 06:06 |
| fungi | oh! | 06:06 |
| fungi | right, it won't get tests run :/ | 06:06 |
| jhesketh | yeah because all jobs have that function applied | 06:06 |
| fungi | i'll do that now | 06:06 |
| jhesketh | actually we could just merge manually, not necessarily apply | 06:07 |
| *** apuimedo_ has quit IRC | 06:07 | |
| fungi | i just manually changed the parameter function file. does zuul need a signal to reread it? | 06:07 |
| *** apuimedo_ has joined #openstack-infra | 06:08 | |
| jhesketh | I think a reload should do it | 06:08 |
| fungi | but yeah, i'll go ahead and manually submit the change to merge as well | 06:08 |
| jhesketh | won't need a restart I don't think | 06:08 |
| *** Daisy has quit IRC | 06:08 | |
| fungi | reloaded | 06:08 |
| *** Daisy has joined #openstack-infra | 06:09 | |
| *** HeOS has joined #openstack-infra | 06:11 | |
| *** sputnik13 has quit IRC | 06:12 | |
| *** apuimedo_ has quit IRC | 06:12 | |
| *** Daisy has quit IRC | 06:12 | |
| *** Daisy has joined #openstack-infra | 06:13 | |
| *** sputnik13 has joined #openstack-infra | 06:14 | |
| *** apuimedo_ has joined #openstack-infra | 06:14 | |
| *** apuimedo has quit IRC | 06:14 | |
| *** apuimedo has joined #openstack-infra | 06:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Correct set_node_options argument list https://review.openstack.org/260334 | 06:16 |
| fungi | there we go | 06:16 |
| fungi | they made that harder in the new change screen | 06:16 |
| *** salv-orlando has quit IRC | 06:16 | |
| *** sputnik13 has quit IRC | 06:16 | |
| fungi | anyway, it also looks like manually applying the change on zuul.o.o and reloading its config got things lubricated | 06:17 |
| fungi | so changes are getting workers assigned to run their jobs again | 06:17 |
| jhesketh | yay :-) | 06:17 |
| *** sabeen has quit IRC | 06:17 | |
| *** sputnik13 has joined #openstack-infra | 06:18 | |
| fungi | now to try my previously broken job | 06:18 |
| *** apuimedo_ has quit IRC | 06:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/260340 | 06:19 |
| *** sputnik13 has quit IRC | 06:20 | |
| *** apuimedo_ has joined #openstack-infra | 06:20 | |
| fungi | there's a guinea pig, just in time! | 06:20 |
| jhesketh | hah | 06:20 |
| *** rossella_s has quit IRC | 06:21 | |
| *** HeOS has quit IRC | 06:21 | |
| *** rossella_s has joined #openstack-infra | 06:21 | |
| *** zeih has joined #openstack-infra | 06:21 | |
| *** sputnik13 has joined #openstack-infra | 06:22 | |
| fungi | jhesketh: anyway, assuming no major broken across the board, any jobs people report failing because of using a new node type probably just need their job definitions corrected | 06:23 |
| jhesketh | yep, I hope so | 06:23 |
| jhesketh | I wonder if they'll fail silently though | 06:24 |
| jhesketh | in that they'll hang in zuul waiting to be picked up | 06:24 |
| jhesketh | since the job definition may still be there | 06:24 |
| *** apuimedo has quit IRC | 06:24 | |
| fungi | i expect at least a few because this may have been hiding some sort of broken job configuration previously | 06:24 |
| jhesketh | oh yeah, totally | 06:25 |
| *** apuimedo has joined #openstack-infra | 06:25 | |
| *** zeih has quit IRC | 06:26 | |
| *** vdrok has quit IRC | 06:27 | |
| *** vilobhmm11 has quit IRC | 06:28 | |
| *** apuimedo has quit IRC | 06:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Turn bluebox servers off in nodepool. https://review.openstack.org/260247 | 06:30 |
| *** apuimedo_ has quit IRC | 06:31 | |
| *** apuimedo has joined #openstack-infra | 06:31 | |
| *** _nadya_ has quit IRC | 06:32 | |
| *** apuimedo_ has joined #openstack-infra | 06:33 | |
| *** pasquier-s has quit IRC | 06:34 | |
| *** pasquier-s has joined #openstack-infra | 06:34 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/260346 | 06:37 |
| *** vgridnev has joined #openstack-infra | 06:37 | |
| *** apuimedo_ has quit IRC | 06:37 | |
| *** apuimedo_ has joined #openstack-infra | 06:38 | |
| *** Jay_ has joined #openstack-infra | 06:38 | |
| *** apuimedo has quit IRC | 06:38 | |
| *** apuimedo has joined #openstack-infra | 06:39 | |
| *** dimtruck is now known as zz_dimtruck | 06:40 | |
| *** Jay__ has quit IRC | 06:41 | |
| *** apuimedo_ has quit IRC | 06:42 | |
| *** irenab has quit IRC | 06:43 | |
| *** hurgleburgler has quit IRC | 06:43 | |
| *** apuimedo_ has joined #openstack-infra | 06:44 | |
| *** apuimedo has quit IRC | 06:44 | |
| *** tangchen has quit IRC | 06:44 | |
| *** apuimedo has joined #openstack-infra | 06:44 | |
| openstackgerrit | Anton Tcitlionok proposed openstack-infra/jenkins-job-builder: Clarify 'trigger-parametrized-builds' description https://review.openstack.org/259870 | 06:46 |
| *** apuimedo_ has quit IRC | 06:48 | |
| *** apuimedo_ has joined #openstack-infra | 06:50 | |
| *** apuimedo has quit IRC | 06:51 | |
| *** apuimedo has joined #openstack-infra | 06:52 | |
| *** maishsk has joined #openstack-infra | 06:52 | |
| *** apuimedo_ has quit IRC | 06:54 | |
| *** pahuang has quit IRC | 06:55 | |
| *** apuimedo_ has joined #openstack-infra | 06:55 | |
| *** aeng has quit IRC | 06:55 | |
| *** apuimedo has quit IRC | 06:56 | |
| *** piet has quit IRC | 06:57 | |
| *** apuimedo has joined #openstack-infra | 06:57 | |
| *** sdake has joined #openstack-infra | 06:57 | |
| *** coolsvap is now known as coolsvap|away | 06:58 | |
| openstackgerrit | Anton Tcitlionok proposed openstack-infra/jenkins-job-builder: Clarify 'trigger-parameterized-builds' description https://review.openstack.org/259870 | 07:01 |
| *** tzn has joined #openstack-infra | 07:02 | |
| *** apuimedo_ has quit IRC | 07:02 | |
| *** AJaeger has joined #openstack-infra | 07:03 | |
| *** apuimedo_ has joined #openstack-infra | 07:03 | |
| AJaeger | good morning, is everything working fine, fungi? | 07:04 |
| *** coolsvap|away is now known as coolsvap | 07:04 | |
| *** hdd has joined #openstack-infra | 07:04 | |
| *** aopincar has joined #openstack-infra | 07:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/jeepyb: Add docimpact tag for bugs that target project queues https://review.openstack.org/253229 | 07:05 |
| *** markvoelker has joined #openstack-infra | 07:06 | |
| *** tzn has quit IRC | 07:06 | |
| *** apuimedo has quit IRC | 07:06 | |
| AJaeger | reading backscroll, you seem to have manged fine - great! | 07:06 |
| *** apuimedo has joined #openstack-infra | 07:07 | |
| *** markvoelker has quit IRC | 07:10 | |
| *** apuimedo_ has quit IRC | 07:10 | |
| * AJaeger does not see any UNREGISTERED jobs right now - great! | 07:11 | |
| *** irenab has joined #openstack-infra | 07:12 | |
| AJaeger | I'll review now all changes for project-config and -1 those with multiple node declarations... | 07:12 |
| *** apuimedo has quit IRC | 07:12 | |
| *** apuimedo has joined #openstack-infra | 07:12 | |
| *** apuimedo_ has joined #openstack-infra | 07:13 | |
| AJaeger | jhesketh, yolanda: could you review the stack at 260202, please? it would unblock us to move to a general linters target - and unblock bandit. | 07:13 |
| AJaeger | yolanda: please ping me once you're around | 07:13 |
| jhesketh | AJaeger: taking a look | 07:15 |
| *** sdake has quit IRC | 07:15 | |
| odyssey4me | anyone around to sponsor another vote on https://review.openstack.org/259922 ? | 07:16 |
| *** chmouel_ is now known as chmouel | 07:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/260340 | 07:17 |
| jhesketh | AJaeger: looks good to me.. I only just noticed we're validating the jjb here.. Not sure I'm a fan of that. I'd rather see JJB handle it, perhaps with a flag to fail on unknown vars | 07:18 |
| AJaeger | jhesketh: Thanks! https://review.openstack.org/#/c/259480/ would be nice as well | 07:19 |
| AJaeger | jhesketh: we merged that validation recently. I'm happy to remove it once jjb does it. | 07:19 |
| *** apuimedo has quit IRC | 07:19 | |
| AJaeger | the change for the validation was month up for review - as WIP | 07:19 |
| AJaeger | We failed to notice typos like builder/builders etc. | 07:19 |
| jhesketh | I'm not complaining that I missed the review, just surprised that we are doing it | 07:20 |
| AJaeger | And that broke a few jobs... | 07:20 |
| *** apuimedo has joined #openstack-infra | 07:20 | |
| AJaeger | jhesketh: that was the quickest way - if you can teach jjb about it, that would be even better... | 07:20 |
| jhesketh | okay fair enough | 07:21 |
| jhesketh | If I remember tomorrow I might poke at it | 07:21 |
| AJaeger | 260192 and 260215 are the ones I need it in from the stack - and if I hadn't used description in one, I wouldn't have needed this change ;) | 07:22 |
| *** Daisy has quit IRC | 07:22 | |
| AJaeger | jhesketh: that would be a nice christmas present ;) | 07:22 |
| *** Daisy has joined #openstack-infra | 07:23 | |
| *** ParsectiX has joined #openstack-infra | 07:24 | |
| *** ParsectiX has quit IRC | 07:24 | |
| *** ParsectiX has joined #openstack-infra | 07:25 | |
| *** ParsectiX has quit IRC | 07:25 | |
| *** shardy has joined #openstack-infra | 07:25 | |
| *** ParsectiX has joined #openstack-infra | 07:25 | |
| *** ParsectiX has quit IRC | 07:26 | |
| *** ParsectiX has joined #openstack-infra | 07:26 | |
| *** ParsectiX has quit IRC | 07:27 | |
| jhesketh | AJaeger: lgtm | 07:27 |
| jhesketh | thanks for always keeping all our jobs nice and tidy :-) | 07:27 |
| *** Daisy has quit IRC | 07:27 | |
| *** binbincong has joined #openstack-infra | 07:28 | |
| *** ParsectiX has joined #openstack-infra | 07:28 | |
| AJaeger | thanks, jhesketh ! | 07:29 |
| *** maishsk has quit IRC | 07:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix constraint jobs branch https://review.openstack.org/259480 | 07:29 |
| *** maishsk has joined #openstack-infra | 07:32 | |
| *** hdd has quit IRC | 07:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Do the sdist check in a virtualenv https://review.openstack.org/255466 | 07:33 |
| *** ParsectiX has quit IRC | 07:35 | |
| AJaeger | #success The node handling in our CI setup has been greatly simplified, just specify the node you want and no need for the complex logic to figure out precise or trusty. Thanks, fungi and reviewers! | 07:44 |
| openstackstatus | AJaeger: Added success to Success page | 07:44 |
| *** apuimedo_ has quit IRC | 07:45 | |
| *** apuimedo_ has joined #openstack-infra | 07:46 | |
| AJaeger | jhesketh: another cleanup review request if you have time, please: https://review.openstack.org/#/c/259814/ | 07:50 |
| *** apuimedo has quit IRC | 07:52 | |
| *** apuimedo has joined #openstack-infra | 07:53 | |
| *** apuimedo_ has quit IRC | 07:57 | |
| *** apuimedo_ has joined #openstack-infra | 07:59 | |
| *** fawadkhaliq has quit IRC | 07:59 | |
| *** mpavone has joined #openstack-infra | 07:59 | |
| *** otter768 has joined #openstack-infra | 08:00 | |
| *** gildub has quit IRC | 08:02 | |
| *** apuimedo has quit IRC | 08:02 | |
| *** tzn has joined #openstack-infra | 08:03 | |
| *** apuimedo has joined #openstack-infra | 08:03 | |
| *** apuimedo_ has quit IRC | 08:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove proposed section from gerrit ACLs https://review.openstack.org/260376 | 08:04 |
| *** vilobhmm11 has joined #openstack-infra | 08:05 | |
| *** otter768 has quit IRC | 08:05 | |
| *** apuimedo_ has joined #openstack-infra | 08:05 | |
| *** ifarkas has joined #openstack-infra | 08:05 | |
| *** Hal1 has joined #openstack-infra | 08:06 | |
| *** gildub has joined #openstack-infra | 08:07 | |
| *** Daisy has joined #openstack-infra | 08:07 | |
| *** apuimedo has quit IRC | 08:07 | |
| *** apuimedo has joined #openstack-infra | 08:08 | |
| *** tzn has quit IRC | 08:08 | |
| *** e0ne has joined #openstack-infra | 08:11 | |
| *** apuimedo has quit IRC | 08:12 | |
| *** apuimedo_ has quit IRC | 08:12 | |
| *** apuimedo has joined #openstack-infra | 08:13 | |
| *** apuimedo_ has joined #openstack-infra | 08:13 | |
| *** yaume has joined #openstack-infra | 08:13 | |
| *** zhurong has quit IRC | 08:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove ACL for refs/heads/release- for mistral https://review.openstack.org/260379 | 08:14 |
| *** irenab has quit IRC | 08:15 | |
| anteaya | AJaeger: glad to hear this are moving smoothly | 08:15 |
| anteaya | AJaeger: do you need anything from me or may I return to sleep? | 08:15 |
| yolanda | hi AJaeger, sorry i'm late, but i started holiday today | 08:16 |
| yolanda | do you need something? | 08:16 |
| anteaya | I skimmed backscroll | 08:16 |
| AJaeger | anteaya: go to sleep! | 08:16 |
| anteaya | AJaeger: very good | 08:16 |
| AJaeger | anteaya: looks fine here... | 08:16 |
| * anteaya does as told | 08:16 | |
| anteaya | AJaeger: so happy to hear that | 08:16 |
| odyssey4me | anyone around to sponsor another vote on https://review.openstack.org/259922 in addition to that provided by AJaeger ? | 08:16 |
| AJaeger | yolanda: If you have holiday, no need to be here. Just wanted to bring you up to speed on an important change that fungi did yesterday: The node handling has been greatly simplified, no need anymore to say "node: precise || trusty" - or to say it differently: Just use the *one* node you need. | 08:17 |
| AJaeger | Please keep that in mind when reviewing. I -1 all changes that are affected. | 08:17 |
| lifeless | AJaeger: how does that work ? | 08:18 |
| AJaeger | fungi, anteaya, and myself are expecting at least some fallout from this - so if people show up, we know where to direct them. | 08:18 |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Removing perModuleEmail paramater management https://review.openstack.org/259987 | 08:18 |
| yolanda | so mostly trusty? | 08:18 |
| AJaeger | yolanda: yes, trusty - or centos7. | 08:19 |
| *** coolsvap is now known as coolsvap|away | 08:19 | |
| yolanda | cool | 08:19 |
| AJaeger | lifeless: we had some logic in zuul/*functions* that set the node to precise if a job was run on stable/icehouse but to precise otherwise. | 08:19 |
| AJaeger | Since icehouse is end-of-life, we do not need this logic anymore and can simplify the job templates in a significant way. | 08:20 |
| lifeless | cool | 08:20 |
| AJaeger | Main changes are https://review.openstack.org/#/c/260213/ and https://review.openstack.org/#/c/260214/ | 08:20 |
| *** coolsvap|away is now known as coolsvap | 08:21 | |
| AJaeger | lifeless: could you +2A an addition to requirements, please? https://review.openstack.org/#/c/257157/ | 08:21 |
| *** zeih has joined #openstack-infra | 08:23 | |
| lifeless | done | 08:23 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 08:24 |
| AJaeger | thanks, lifeless ! | 08:24 |
| *** apuimedo has quit IRC | 08:25 | |
| *** apuimedo has joined #openstack-infra | 08:25 | |
| *** zeih has quit IRC | 08:27 | |
| *** maishsk has quit IRC | 08:28 | |
| *** e0ne has quit IRC | 08:28 | |
| *** unicell1 has joined #openstack-infra | 08:31 | |
| *** unicell has quit IRC | 08:31 | |
| *** apuimedo_ has quit IRC | 08:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Gate requirements repo on docs https://review.openstack.org/259814 | 08:32 |
| *** apuimedo_ has joined #openstack-infra | 08:32 | |
| *** apuimedo has quit IRC | 08:32 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 08:33 |
| *** e0ne has joined #openstack-infra | 08:33 | |
| *** yamamoto has joined #openstack-infra | 08:33 | |
| *** apuimedo has joined #openstack-infra | 08:33 | |
| *** yamamoto has quit IRC | 08:36 | |
| *** yamamoto has joined #openstack-infra | 08:36 | |
| AJaeger | infra-root, looking at status.openstack.org/zuul, the post queue seems stuck - and we have lots of available workers. This looks like a fallout from fungi's node change, could anybody investigate, please? I'm not seeing anything obvious directly from here | 08:36 |
| yolanda | AJaeger, i need to leave for a school festival just now, sorry :( but i can take a look when i come back if there is no one available | 08:37 |
| *** apuimedo_ has quit IRC | 08:37 | |
| *** maishsk has joined #openstack-infra | 08:37 | |
| AJaeger | yolanda: enjoy! | 08:37 |
| AJaeger | yolanda: no CritSit here, just the post jobs ;) | 08:38 |
| yolanda | ok i feel less guilty then :) | 08:38 |
| *** apuimedo_ has joined #openstack-infra | 08:38 | |
| AJaeger | yolanda: this can even wait for the US to wake up if my assessment is correct - just wanted to inform everybody in case somebody has time... | 08:39 |
| AJaeger | yolanda: so, stop feeling guilty and enjoy yourself! | 08:39 |
| yolanda | leaving now then, bye! | 08:39 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add instructions on how to troubleshoot bots https://review.openstack.org/260068 | 08:39 |
| *** salv-orlando has joined #openstack-infra | 08:40 | |
| AJaeger | bye, yolanda ! | 08:40 |
| *** yamamoto has quit IRC | 08:41 | |
| *** gildub has quit IRC | 08:43 | |
| *** vgridnev has quit IRC | 08:43 | |
| *** egonzalez has joined #openstack-infra | 08:44 | |
| *** apuimedo has quit IRC | 08:44 | |
| *** christiana_ has quit IRC | 08:45 | |
| *** apuimedo has joined #openstack-infra | 08:45 | |
| *** Daisy has quit IRC | 08:46 | |
| *** Daisy has joined #openstack-infra | 08:47 | |
| *** markus_z has joined #openstack-infra | 08:47 | |
| *** abregman has quit IRC | 08:48 | |
| *** andreaf has quit IRC | 08:48 | |
| *** andreaf has joined #openstack-infra | 08:49 | |
| *** abregman has joined #openstack-infra | 08:49 | |
| *** apuimedo has quit IRC | 08:49 | |
| *** apuimedo has joined #openstack-infra | 08:50 | |
| *** Daisy has quit IRC | 08:51 | |
| *** salv-orl_ has joined #openstack-infra | 08:51 | |
| *** salv-orl_ has quit IRC | 08:51 | |
| *** salv-orlando has quit IRC | 08:51 | |
| *** salv-orlando has joined #openstack-infra | 08:52 | |
| *** apuimedo has quit IRC | 08:54 | |
| *** apuimedo has joined #openstack-infra | 08:55 | |
| *** salv-orl_ has joined #openstack-infra | 08:55 | |
| *** apuimedo_ has quit IRC | 08:56 | |
| *** apuimedo_ has joined #openstack-infra | 08:57 | |
| *** fawadkhaliq has joined #openstack-infra | 08:57 | |
| *** fawadkhaliq has quit IRC | 08:57 | |
| *** christiana_ has joined #openstack-infra | 08:58 | |
| *** fawadkhaliq has joined #openstack-infra | 08:58 | |
| *** salv-orlando has quit IRC | 08:58 | |
| *** apuimedo has quit IRC | 09:00 | |
| *** apuimedo has joined #openstack-infra | 09:00 | |
| *** hashar has joined #openstack-infra | 09:02 | |
| *** tzn has joined #openstack-infra | 09:04 | |
| *** admin0 has joined #openstack-infra | 09:04 | |
| *** apuimedo_ has quit IRC | 09:04 | |
| *** apuimedo_ has joined #openstack-infra | 09:05 | |
| *** apuimedo has quit IRC | 09:05 | |
| *** apuimedo has joined #openstack-infra | 09:06 | |
| *** markvoelker has joined #openstack-infra | 09:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove redundant node declarations https://review.openstack.org/260400 | 09:08 |
| *** jamespage has joined #openstack-infra | 09:08 | |
| *** tzn has quit IRC | 09:09 | |
| *** _nadya_ has joined #openstack-infra | 09:11 | |
| *** markvoelker has quit IRC | 09:11 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add non-voting linters-nv job-template https://review.openstack.org/260402 | 09:12 |
| jhesketh | AJaeger: just noticed your message | 09:12 |
| AJaeger | jhesketh: as I said to yolanda: No critsit AFAIU ;) | 09:13 |
| jhesketh | critsit sorry? | 09:13 |
| *** apuimedo has quit IRC | 09:13 | |
| AJaeger | Critical situation | 09:13 |
| AJaeger | Something that needs our full attention - urgently... - this is not the case here | 09:14 |
| *** apuimedo has joined #openstack-infra | 09:14 | |
| *** e0ne has quit IRC | 09:16 | |
| jhesketh | right | 09:16 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: Revert "Add limited tempest job to swift3 check pipeline" https://review.openstack.org/259761 | 09:17 |
| openstackgerrit | Evgeniy L proposed openstack-infra/project-config: Create bareon-api repository https://review.openstack.org/259352 | 09:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Create linters test target https://review.openstack.org/260405 | 09:18 |
| *** HeOS has joined #openstack-infra | 09:18 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 09:19 |
| *** apuimedo_ has quit IRC | 09:19 | |
| *** apuimedo_ has joined #openstack-infra | 09:19 | |
| *** vilobhmm11 has quit IRC | 09:20 | |
| *** e0ne has joined #openstack-infra | 09:20 | |
| *** dizquierdo has joined #openstack-infra | 09:22 | |
| AJaeger | jhesketh: the graph for ready nodes looks interesting - over 40 for each jenkins: http://grafana.openstack.org/dashboard/db/nodepool This is since the last change by fungi merged | 09:22 |
| *** tzn has joined #openstack-infra | 09:22 | |
| AJaeger | jhesketh: and post queue contains jobs that do not move forward at all since the last merged node change | 09:22 |
| jhesketh | AJaeger: yeah I think nodepool is responding to the demand but the jobs aren't being assigned | 09:22 |
| jhesketh | I think they aren't moving as they run on persistent nodes | 09:23 |
| jhesketh | so there's something funky with the job assignment there | 09:23 |
| AJaeger | I don't see why fuel-specs-publish-specs should run on persistent node before the change | 09:23 |
| *** aopincar has quit IRC | 09:24 | |
| jhesketh | I could be wrong, that was a theory I'm investigating | 09:24 |
| *** e0ne has quit IRC | 09:24 | |
| *** apuimedo_ has quit IRC | 09:24 | |
| *** apuimedo has quit IRC | 09:24 | |
| AJaeger | jhesketh: you might be right as well ;) | 09:25 |
| AJaeger | https://jenkins01.openstack.org/view/All/job/fuel-specs-publish-specs/lastBuild/consoleFull is a run - it run on bare-trusty | 09:25 |
| *** apuimedo has joined #openstack-infra | 09:25 | |
| AJaeger | and the node was taken offline afterwards if I interpret https://jenkins01.openstack.org/view/All/job/fuel-specs-publish-specs/lastBuild/parameters/ correct | 09:26 |
| *** apuimedo_ has joined #openstack-infra | 09:26 | |
| *** apuimedo has quit IRC | 09:30 | |
| *** apuimedo_ has quit IRC | 09:30 | |
| *** apuimedo has joined #openstack-infra | 09:31 | |
| *** apuimedo_ has joined #openstack-infra | 09:32 | |
| *** derekh has joined #openstack-infra | 09:35 | |
| *** apuimedo has quit IRC | 09:36 | |
| *** apuimedo_ has quit IRC | 09:36 | |
| *** apuimedo has joined #openstack-infra | 09:37 | |
| *** apuimedo_ has joined #openstack-infra | 09:38 | |
| *** aopincar has joined #openstack-infra | 09:39 | |
| *** _nadya_ has quit IRC | 09:39 | |
| *** Hal1 has quit IRC | 09:39 | |
| *** yamamoto has joined #openstack-infra | 09:42 | |
| *** apuimedo_ has quit IRC | 09:42 | |
| *** Hal1 has joined #openstack-infra | 09:43 | |
| *** apuimedo_ has joined #openstack-infra | 09:43 | |
| *** apuimedo has quit IRC | 09:44 | |
| *** sshnaidm has quit IRC | 09:45 | |
| *** apuimedo has joined #openstack-infra | 09:46 | |
| *** triggerwarning has quit IRC | 09:47 | |
| *** dcentral has quit IRC | 09:47 | |
| tdurakov | hi folks | 09:47 |
| *** dcentral has joined #openstack-infra | 09:47 | |
| AJaeger | hi tdurakov. Is everything fine? | 09:47 |
| tdurakov | got question about devstack, is there any reason why when we set USE_SCREEN to false, we don't use services? | 09:48 |
| *** apuimedo_ has quit IRC | 09:48 | |
| tdurakov | AJaeger, ^ | 09:48 |
| AJaeger | tdurakov: I cannot help with that - for devstack you better ask on #openstack-qa | 09:48 |
| *** yamamoto has quit IRC | 09:48 | |
| jhesketh | there are 47 workers claiming to be able to do fuel-specs-publish-specs, but for some reason jenkins aren't picking it out of the gearman queue | 09:49 |
| tdurakov | AJaeger, ok, thank you | 09:49 |
| *** tzn has quit IRC | 09:49 | |
| *** vgridnev has joined #openstack-infra | 09:49 | |
| AJaeger | 47 should be enough ;) This is strange but beyond my debug foo ;( | 09:50 |
| *** apuimedo_ has joined #openstack-infra | 09:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters for system-config https://review.openstack.org/260420 | 09:51 |
| openstackgerrit | Ramana Raja proposed openstack-infra/project-config: Add Manila with CephFSNative driver CI job https://review.openstack.org/259323 | 09:52 |
| *** BobBall_AWOL is now known as BobBall | 09:53 | |
| *** Daisy has joined #openstack-infra | 09:53 | |
| *** apuimedo_ has quit IRC | 09:55 | |
| *** apuimedo_ has joined #openstack-infra | 09:55 | |
| *** sshnaidm has joined #openstack-infra | 09:57 | |
| *** apuimedo has quit IRC | 09:57 | |
| *** yamamoto has joined #openstack-infra | 09:58 | |
| *** apuimedo has joined #openstack-infra | 09:58 | |
| *** fawadkhaliq has quit IRC | 10:00 | |
| *** zhenguo has quit IRC | 10:01 | |
| *** otter768 has joined #openstack-infra | 10:01 | |
| *** zhenguo has joined #openstack-infra | 10:01 | |
| *** maishsk has quit IRC | 10:01 | |
| *** apuimedo has quit IRC | 10:03 | |
| *** maishsk has joined #openstack-infra | 10:03 | |
| *** apuimedo has joined #openstack-infra | 10:03 | |
| *** otter768 has quit IRC | 10:06 | |
| *** apuimedo has quit IRC | 10:08 | |
| *** aopincar has quit IRC | 10:09 | |
| *** apuimedo has joined #openstack-infra | 10:09 | |
| *** apuimedo_ has quit IRC | 10:09 | |
| *** apuimedo_ has joined #openstack-infra | 10:10 | |
| *** bexelbie has joined #openstack-infra | 10:10 | |
| *** yamamoto has quit IRC | 10:10 | |
| *** apuimedo has quit IRC | 10:14 | |
| *** apuimedo_ has quit IRC | 10:14 | |
| AJaeger | jhesketh: did the last puppet run complete on all jenkins? | 10:15 |
| *** sshnaidm has quit IRC | 10:15 | |
| AJaeger | or did it run into any timeouts and we need to update them first with jjb config? | 10:15 |
| jhesketh | AJaeger: haven't checked that in particular, but the job definition has the update | 10:15 |
| AJaeger | jhesketh: http://puppetboard.openstack.org/ gives me an internal server error | 10:15 |
| AJaeger | ;( | 10:16 |
| *** apuimedo has joined #openstack-infra | 10:16 | |
| jhesketh | yeah I saw that but figured I'd solve one problem at a time :p | 10:16 |
| *** Qiming has quit IRC | 10:16 | |
| *** apuimedo_ has joined #openstack-infra | 10:16 | |
| * AJaeger was wondering why gate-requirements-docs is still NOT_REGISTERED (approved a few hours ago) - and a not-running puppet could explain that and might play a role in the post jobs. But since you checked that those are fine... | 10:17 | |
| AJaeger | jhesketh: sure | 10:17 |
| AJaeger | vponomaryov: glad to hear that your manila-image problem is solved! | 10:18 |
| *** irenab has joined #openstack-infra | 10:18 | |
| vponomaryov | AJaeger: CI side - yes )) | 10:19 |
| AJaeger | vponomaryov: one step at a time ;) | 10:19 |
| vponomaryov | AJaeger: do you have link to change that fixed Ci part? | 10:19 |
| AJaeger | Main changes are https://review.openstack.org/#/c/260213/ and https://review.openstack.org/#/c/260214/ | 10:20 |
| AJaeger | vponomaryov: fungi reworked the node handling completely with the two changes above | 10:20 |
| *** Daisy has quit IRC | 10:21 | |
| *** apuimedo has quit IRC | 10:23 | |
| vponomaryov | AJaeger: thanks for info and fungi for work! | 10:23 |
| *** yamamoto has joined #openstack-infra | 10:23 | |
| *** apuimedo_ has quit IRC | 10:23 | |
| *** yamamoto has quit IRC | 10:24 | |
| *** apuimedo has joined #openstack-infra | 10:24 | |
| openstackgerrit | Jaume Devesa proposed openstack-infra/project-config: Add puppet-midonet to OpenStack infrastructure https://review.openstack.org/260093 | 10:24 |
| *** apuimedo_ has joined #openstack-infra | 10:24 | |
| *** zeih has joined #openstack-infra | 10:24 | |
| AJaeger | jhesketh: Don't work through the whole night - if you find anything, tell it here and let others take over later... | 10:25 |
| *** yamahata has quit IRC | 10:25 | |
| jhesketh | AJaeger: night is young yet ;-) | 10:25 |
| *** e0ne has joined #openstack-infra | 10:26 | |
| *** Hal1 has quit IRC | 10:27 | |
| *** sshnaidm has joined #openstack-infra | 10:27 | |
| AJaeger | ;) | 10:27 |
| *** zeih has quit IRC | 10:29 | |
| *** apuimedo has quit IRC | 10:29 | |
| *** apuimedo_ has quit IRC | 10:29 | |
| *** apuimedo has joined #openstack-infra | 10:30 | |
| *** apuimedo_ has joined #openstack-infra | 10:30 | |
| *** daemontool has joined #openstack-infra | 10:31 | |
| *** fawadkhaliq has joined #openstack-infra | 10:34 | |
| *** apuimedo has quit IRC | 10:35 | |
| *** fawadkhaliq has quit IRC | 10:35 | |
| *** apuimedo has joined #openstack-infra | 10:36 | |
| *** anshprat has quit IRC | 10:36 | |
| *** vmbrasseur has quit IRC | 10:36 | |
| *** vmb has joined #openstack-infra | 10:36 | |
| *** vmb is now known as vmbrasseur | 10:37 | |
| *** apuimedo_ has quit IRC | 10:38 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 10:38 |
| *** apuimedo_ has joined #openstack-infra | 10:38 | |
| *** Hal1 has joined #openstack-infra | 10:39 | |
| *** zhurong has joined #openstack-infra | 10:39 | |
| *** apuimedo has quit IRC | 10:40 | |
| *** apuimedo has joined #openstack-infra | 10:41 | |
| *** apuimedo_ has quit IRC | 10:43 | |
| *** apuimedo_ has joined #openstack-infra | 10:45 | |
| *** apuimedo has quit IRC | 10:45 | |
| *** apuimedo has joined #openstack-infra | 10:46 | |
| *** maishsk has quit IRC | 10:46 | |
| ttx | AJaeger: any infra issue I should be aware of ? | 10:46 |
| * ttx tries to parse the IRC logs | 10:47 | |
| AJaeger | ttx: We changed the node setup in project-config - greatly simplified it. | 10:48 |
| AJaeger | Seems like some post-jobs are not run currently. | 10:48 |
| *** _nadya_ has joined #openstack-infra | 10:48 | |
| AJaeger | But everything else looks fine. | 10:48 |
| ttx | AJaeger: I just W+1ed https://review.openstack.org/#/c/230699/ 10 min ago and it does not get picked up | 10:48 |
| *** rvasilets___ has joined #openstack-infra | 10:49 | |
| *** maishsk has joined #openstack-infra | 10:49 | |
| AJaeger | ttx, that's strange and not known. We expected some fallout from the node handling change but not this kind ;( | 10:50 |
| ttx | hmm | 10:50 |
| ttx | Let me try something else. I tried to use the W+1 1-click gerrit button, maybe that fails | 10:50 |
| *** apuimedo has quit IRC | 10:51 | |
| AJaeger | Ah, the one-click button does not +2! | 10:51 |
| ttx | hmm, though I don't really see why/how | 10:51 |
| ttx | that repo does not require +2 | 10:51 |
| ttx | it requires CodeRevbiew+1 RollCall+1 | 10:51 |
| AJaeger | And we have both. | 10:51 |
| *** apuimedo_ has quit IRC | 10:52 | |
| *** apuimedo has joined #openstack-infra | 10:52 | |
| ttx | hmm, /me tries again | 10:52 |
| *** apuimedo_ has joined #openstack-infra | 10:52 | |
| *** markvoelker has joined #openstack-infra | 10:52 | |
| AJaeger | still not there ;( | 10:53 |
| ttx | AJaeger: yeah. And can't see why it would block it | 10:53 |
| *** hashar has quit IRC | 10:53 | |
| ttx | it just parsed a series of events | 10:54 |
| AJaeger | ttx, one more issue for infra-root to debug - let me just log it here: 230699 has been approved but Zuul is not putting it into the gate | 10:54 |
| * ttx will hold on processing releases | 10:54 | |
| AJaeger | ttx, thanks for telling here. jhesketh is currently looking at the post queue and fungi et all will help later... | 10:55 |
| ttx | AJaeger: it's probably an unrelated issue | 10:55 |
| jhesketh | AJaeger, ttx: http://paste.openstack.org/show/482496/ | 10:56 |
| AJaeger | ttx, agreed | 10:56 |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/project-config: Add python-brick-cinderclient-ext to OpenStack https://review.openstack.org/243080 | 10:57 |
| AJaeger | jhesketh: thanks for that line. Something is indeed wrong here. | 10:57 |
| ttx | jhesketh: hmm, so for some reason the original check is missing | 10:57 |
| jhesketh | yeah | 10:57 |
| *** markvoelker has quit IRC | 10:57 | |
| * jhesketh is looking | 10:57 | |
| * ttx tries recheck | 10:57 | |
| *** fawadkhaliq has joined #openstack-infra | 10:57 | |
| ttx | could be it fell in the cracks of earlier issues | 10:57 |
| ttx | ok, it's in the check queue now | 10:58 |
| *** hashar has joined #openstack-infra | 10:58 | |
| ttx | OK, I suspect the check failed to trigger earlier, and it does not automatically get retriggered on W+1 | 10:58 |
| ttx | should be ok now | 10:59 |
| *** zeih has joined #openstack-infra | 10:59 | |
| jhesketh | yeah but it's not clear why the original check never happened | 10:59 |
| ttx | unrelated issue, probably a consequence of an earlier issue | 10:59 |
| jhesketh | mind you, that was a week ago | 10:59 |
| ttx | right | 10:59 |
| AJaeger | 14th of December - not sure what hit us at that time ;( | 11:00 |
| jhesketh | ttx: so with the releases maybe just start with one and see how it goes.. I'm still trying to debug the post queue | 11:00 |
| AJaeger | ok, issue solved ;) | 11:00 |
| ttx | jhesketh: yeah, I'll wait a bit anyway | 11:00 |
| jhesketh | admittedly I'm running out of ideas | 11:01 |
| jhesketh | it appears unique to the post pipeline and only jobs on non-persistent nodes | 11:01 |
| *** fawadkhaliq has quit IRC | 11:02 | |
| jhesketh | I'm tempted to try restarting a jenkins master, but I believe that'll lose/screw up the work that it has in progress | 11:02 |
| *** fawadkhaliq has joined #openstack-infra | 11:02 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| *** yamamoto has quit IRC | 11:02 | |
| *** apuimedo has quit IRC | 11:06 | |
| *** apuimedo_ has quit IRC | 11:06 | |
| *** apuimedo has joined #openstack-infra | 11:08 | |
| *** dtantsur|afk is now known as dtantsur | 11:08 | |
| *** apuimedo_ has joined #openstack-infra | 11:08 | |
| AJaeger | jhesketh: you can schedule it for downtime somehow - so that no new jobs get run on that jenkins and then restart it once it's idle... | 11:09 |
| jhesketh | oh, that'd be handy | 11:10 |
| * jhesketh looks for that thing | 11:10 | |
| AJaeger | http://docs.openstack.org/infra/system-config/jenkins.html#safe-master-restarts | 11:11 |
| AJaeger | jhesketh: might be this information ? ^ | 11:11 |
| jhesketh | AJaeger: very helpful, thankyou :-) | 11:12 |
| *** smarcet has joined #openstack-infra | 11:12 | |
| *** apuimedo has quit IRC | 11:12 | |
| *** apuimedo_ has quit IRC | 11:13 | |
| *** apuimedo has joined #openstack-infra | 11:13 | |
| *** apuimedo_ has joined #openstack-infra | 11:14 | |
| *** shardy has quit IRC | 11:14 | |
| *** lucas-dinner is now known as lucasagomes | 11:15 | |
| *** irenab has quit IRC | 11:16 | |
| *** Qiming has joined #openstack-infra | 11:17 | |
| *** smarcet has quit IRC | 11:17 | |
| *** Jay_ has quit IRC | 11:18 | |
| *** binbincong has quit IRC | 11:19 | |
| *** apuimedo_ has quit IRC | 11:20 | |
| *** apuimedo_ has joined #openstack-infra | 11:22 | |
| *** egonzalez has quit IRC | 11:23 | |
| *** Jay_ has joined #openstack-infra | 11:24 | |
| *** triggerwarning has joined #openstack-infra | 11:25 | |
| *** Jay_ has quit IRC | 11:25 | |
| *** apuimedo has quit IRC | 11:25 | |
| *** triggerwarning has quit IRC | 11:25 | |
| *** triggerwarning has joined #openstack-infra | 11:25 | |
| *** Jay_ has joined #openstack-infra | 11:25 | |
| *** apuimedo has joined #openstack-infra | 11:26 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/tripleo-ci: Remove the tripleo.sh Flavor creation call https://review.openstack.org/260448 | 11:26 |
| AJaeger | jhesketh: which jenkins are you taking down? | 11:26 |
| jhesketh | AJaeger: jenkins01 | 11:27 |
| *** apuimedo_ has quit IRC | 11:27 | |
| jhesketh | AJaeger: however it looks like things are unstuck | 11:27 |
| jhesketh | but not due to restarting 01 | 11:27 |
| jhesketh | so I'm not sure what's going on just yet | 11:27 |
| AJaeger | indeed, there's progress! | 11:27 |
| jhesketh | 01 is still being restarted though so I have to let that finish | 11:27 |
| *** apuimedo_ has joined #openstack-infra | 11:27 | |
| AJaeger | load is light enough that we can restart it... | 11:27 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: remove one of ec2-api gating job https://review.openstack.org/260449 | 11:27 |
| *** tzn has joined #openstack-infra | 11:30 | |
| *** Daisy has joined #openstack-infra | 11:30 | |
| *** fawadkhaliq has quit IRC | 11:31 | |
| *** fawadkhaliq has joined #openstack-infra | 11:31 | |
| *** ramineni has left #openstack-infra | 11:31 | |
| *** apuimedo_ has quit IRC | 11:32 | |
| *** apuimedo_ has joined #openstack-infra | 11:33 | |
| *** apuimedo has quit IRC | 11:35 | |
| *** egonzalez has joined #openstack-infra | 11:36 | |
| *** Daisy has quit IRC | 11:36 | |
| *** apuimedo has joined #openstack-infra | 11:37 | |
| *** Daisy has joined #openstack-infra | 11:37 | |
| *** apuimedo_ has quit IRC | 11:37 | |
| *** apuimedo_ has joined #openstack-infra | 11:38 | |
| *** zeih has quit IRC | 11:38 | |
| *** rvba` is now known as rvba | 11:39 | |
| *** Daisy has quit IRC | 11:40 | |
| *** Daisy has joined #openstack-infra | 11:41 | |
| jhesketh | AJaeger: so it got through some, and now it is waiting again | 11:41 |
| jhesketh | AJaeger: I wonder if it's just due to the low precedence on the post pipeline | 11:42 |
| jhesketh | however nodepool seems to be over allocating demand at the moment | 11:42 |
| *** apuimedo_ has quit IRC | 11:42 | |
| jhesketh | which it's doing so we can pick up on all of the queued jobs | 11:43 |
| *** apuimedo has quit IRC | 11:43 | |
| *** apuimedo has joined #openstack-infra | 11:44 | |
| *** salv-orl_ has quit IRC | 11:44 | |
| AJaeger | the post pipeline has indeed a low precedence - but stuck for several hours is rare, especially if enough nodes are available. | 11:44 |
| *** apuimedo_ has joined #openstack-infra | 11:44 | |
| jhesketh | oh yeah, something is still fishy, but that might be why a few snuck through | 11:46 |
| AJaeger | jhesketh: we can wait for the US to wake up as well and see how this evolves... | 11:46 |
| jhesketh | yeah I don't think it's bad enough to warrant reverting or anything too drastic | 11:48 |
| jhesketh | it can likely wait for people much smarter than me to wake up | 11:48 |
| AJaeger | much smarter -> more familiar ;) | 11:48 |
| *** apuimedo has quit IRC | 11:48 | |
| *** jyuso1 has quit IRC | 11:48 | |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Create fuel-ui project https://review.openstack.org/260455 | 11:48 |
| *** lxsli_festive has quit IRC | 11:48 | |
| *** apuimedo has joined #openstack-infra | 11:49 | |
| *** lxsli_festive has joined #openstack-infra | 11:52 | |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Create fuel-ui project https://review.openstack.org/260455 | 11:52 |
| *** apuimedo has quit IRC | 11:54 | |
| *** apuimedo_ has quit IRC | 11:54 | |
| *** apuimedo has joined #openstack-infra | 11:54 | |
| *** isaacb has joined #openstack-infra | 11:55 | |
| *** apuimedo_ has joined #openstack-infra | 11:56 | |
| *** Daisy has quit IRC | 11:56 | |
| *** Daisy has joined #openstack-infra | 11:56 | |
| *** xwizard_ has quit IRC | 11:57 | |
| *** apuimedo has quit IRC | 11:59 | |
| *** apuimedo has joined #openstack-infra | 12:00 | |
| *** apuimedo_ has quit IRC | 12:00 | |
| *** Daisy has quit IRC | 12:00 | |
| *** Jay_ has quit IRC | 12:01 | |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Create fuel-ui project https://review.openstack.org/260455 | 12:01 |
| *** apuimedo_ has joined #openstack-infra | 12:01 | |
| *** otter768 has joined #openstack-infra | 12:02 | |
| *** hashar has quit IRC | 12:03 | |
| *** Hal1 has quit IRC | 12:05 | |
| *** apuimedo_ has quit IRC | 12:06 | |
| *** egonzalez has quit IRC | 12:06 | |
| *** apuimedo_ has joined #openstack-infra | 12:07 | |
| *** apuimedo has quit IRC | 12:07 | |
| *** hashar has joined #openstack-infra | 12:07 | |
| *** otter768 has quit IRC | 12:07 | |
| *** apuimedo has joined #openstack-infra | 12:08 | |
| *** jaypipes has joined #openstack-infra | 12:09 | |
| AJaeger | jhesketh: jenkins01 should be idle now... | 12:10 |
| *** claudiub has joined #openstack-infra | 12:11 | |
| *** apuimedo_ has quit IRC | 12:11 | |
| *** apuimedo has quit IRC | 12:12 | |
| *** apuimedo has joined #openstack-infra | 12:13 | |
| *** _nadya_ has quit IRC | 12:13 | |
| *** apuimedo_ has joined #openstack-infra | 12:13 | |
| jhesketh | AJaeger: one to go | 12:13 |
| *** ociuhandu has joined #openstack-infra | 12:13 | |
| *** apuimedo has quit IRC | 12:17 | |
| *** apuimedo_ has quit IRC | 12:18 | |
| *** yamamoto has joined #openstack-infra | 12:19 | |
| *** apuimedo has joined #openstack-infra | 12:19 | |
| *** daemontool has quit IRC | 12:19 | |
| *** apuimedo_ has joined #openstack-infra | 12:19 | |
| AJaeger | I see... | 12:21 |
| *** egonzalez has joined #openstack-infra | 12:21 | |
| jhesketh | AJaeger: okay restart done | 12:21 |
| *** daemontool has joined #openstack-infra | 12:21 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: TEST: this is supposed to trigger a major upgrade https://review.openstack.org/260466 | 12:23 |
| *** yamamoto has quit IRC | 12:23 | |
| *** Jay_ has joined #openstack-infra | 12:23 | |
| *** apuimedo has quit IRC | 12:23 | |
| *** hashar has quit IRC | 12:24 | |
| *** apuimedo has joined #openstack-infra | 12:24 | |
| openstackgerrit | Ramana Raja proposed openstack-infra/project-config: Add Manila with CephFSNative driver CI job https://review.openstack.org/259323 | 12:24 |
| *** yamamoto has joined #openstack-infra | 12:25 | |
| *** Jay_ has quit IRC | 12:25 | |
| *** Jay_ has joined #openstack-infra | 12:26 | |
| *** Jay_ has joined #openstack-infra | 12:27 | |
| *** egonzalez has quit IRC | 12:29 | |
| *** gordc has joined #openstack-infra | 12:30 | |
| *** apuimedo_ has quit IRC | 12:31 | |
| *** apuimedo_ has joined #openstack-infra | 12:32 | |
| AJaeger | jhesketh: doesn't seem to have changed anything from what I can see | 12:34 |
| *** dprince has joined #openstack-infra | 12:34 | |
| jhesketh | no unfortunately :-( | 12:34 |
| *** apuimedo has quit IRC | 12:36 | |
| *** apuimedo_ has quit IRC | 12:37 | |
| *** apuimedo has joined #openstack-infra | 12:37 | |
| *** apuimedo_ has joined #openstack-infra | 12:38 | |
| *** markvoelker has joined #openstack-infra | 12:38 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/elastic-recheck: Correct typos https://review.openstack.org/260474 | 12:38 |
| *** ZZelle has joined #openstack-infra | 12:40 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** rhallisey has joined #openstack-infra | 12:41 | |
| *** rhallisey has quit IRC | 12:42 | |
| *** apuimedo_ has quit IRC | 12:42 | |
| *** pblaho has joined #openstack-infra | 12:43 | |
| *** markvoelker has quit IRC | 12:43 | |
| *** apuimedo has quit IRC | 12:44 | |
| *** apuimedo has joined #openstack-infra | 12:44 | |
| *** apuimedo_ has joined #openstack-infra | 12:44 | |
| *** salv-orlando has joined #openstack-infra | 12:45 | |
| *** Hal1 has joined #openstack-infra | 12:48 | |
| *** apuimedo has quit IRC | 12:48 | |
| *** apuimedo_ has quit IRC | 12:49 | |
| *** yamamoto has joined #openstack-infra | 12:50 | |
| *** apuimedo has joined #openstack-infra | 12:50 | |
| *** apuimedo_ has joined #openstack-infra | 12:50 | |
| *** sshnaidm has quit IRC | 12:53 | |
| *** amuller has joined #openstack-infra | 12:54 | |
| *** sdake has joined #openstack-infra | 12:54 | |
| *** apuimedo_ has quit IRC | 12:55 | |
| *** apuimedo_ has joined #openstack-infra | 12:55 | |
| *** baoli has joined #openstack-infra | 12:56 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: add python-aodhclient to openstack repo https://review.openstack.org/259637 | 12:56 |
| *** rguillebert has joined #openstack-infra | 12:56 | |
| *** apuimedo has quit IRC | 12:57 | |
| *** apuimedo has joined #openstack-infra | 12:57 | |
| *** mudassirlatif has joined #openstack-infra | 12:58 | |
| *** _nadya_ has joined #openstack-infra | 12:59 | |
| *** abregman_ has joined #openstack-infra | 13:01 | |
| *** dims has joined #openstack-infra | 13:01 | |
| *** apuimedo has quit IRC | 13:02 | |
| *** apuimedo has joined #openstack-infra | 13:03 | |
| *** Jay_ has quit IRC | 13:04 | |
| *** abregman has quit IRC | 13:05 | |
| *** sshnaidm has joined #openstack-infra | 13:06 | |
| *** apuimedo_ has quit IRC | 13:06 | |
| *** abregman_ has quit IRC | 13:07 | |
| *** apuimedo_ has joined #openstack-infra | 13:07 | |
| *** Jay_ has joined #openstack-infra | 13:07 | |
| *** apuimedo has quit IRC | 13:08 | |
| *** Jay__ has joined #openstack-infra | 13:08 | |
| *** apuimedo has joined #openstack-infra | 13:08 | |
| *** Jay_ has quit IRC | 13:12 | |
| *** apuimedo has quit IRC | 13:13 | |
| *** apuimedo has joined #openstack-infra | 13:14 | |
| *** dizquierdo is now known as dizquierdo_afk | 13:14 | |
| *** apuimedo_ has quit IRC | 13:14 | |
| *** apuimedo_ has joined #openstack-infra | 13:15 | |
| *** sshnaidm has quit IRC | 13:17 | |
| *** ihrachys has joined #openstack-infra | 13:17 | |
| *** salv-orlando has quit IRC | 13:18 | |
| *** salv-orlando has joined #openstack-infra | 13:18 | |
| *** sdake has quit IRC | 13:19 | |
| *** ZZelle has quit IRC | 13:19 | |
| *** apuimedo_ has quit IRC | 13:25 | |
| *** apuimedo_ has joined #openstack-infra | 13:26 | |
| *** weshay_xchat has joined #openstack-infra | 13:28 | |
| *** sshnaidm has joined #openstack-infra | 13:29 | |
| *** tzn has quit IRC | 13:30 | |
| *** shardy has joined #openstack-infra | 13:34 | |
| *** dcentral has quit IRC | 13:34 | |
| *** triggerwarning has quit IRC | 13:35 | |
| *** apuimedo_ has quit IRC | 13:36 | |
| *** apuimedo_ has joined #openstack-infra | 13:36 | |
| *** ZZelle has joined #openstack-infra | 13:39 | |
| *** piet has joined #openstack-infra | 13:44 | |
| *** mudassirlatif has quit IRC | 13:46 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: rename ceilometer-specs to telemetry-specs https://review.openstack.org/248897 | 13:46 |
| *** triggerwarning has joined #openstack-infra | 13:46 | |
| *** yamamoto has quit IRC | 13:47 | |
| *** dcentral has joined #openstack-infra | 13:47 | |
| *** EricGonczer_ has joined #openstack-infra | 13:47 | |
| *** abregman has joined #openstack-infra | 13:47 | |
| *** yamamoto has joined #openstack-infra | 13:48 | |
| *** EricGonczer_ has quit IRC | 13:49 | |
| *** admin0 has left #openstack-infra | 13:52 | |
| *** abregman has quit IRC | 13:53 | |
| anteaya | AJaeger: I'm here for about 30 minutes before I have to depart for errands, what can I do to help? | 13:55 |
| *** tlian has joined #openstack-infra | 13:55 | |
| jd__ | anteaya: some outage I guess? | 13:56 |
| anteaya | jd__: we changed some configuration for allocated nodes yesterday, AJaeger and jhesketh seem to be tracking an issue | 13:56 |
| jd__ | ok, I indeed get 502 error on Gerrit | 13:56 |
| anteaya | jd__: I just woke up, anyone following backscroll knows more than me at this point | 13:56 |
| anteaya | jd__: okay thank you | 13:57 |
| * anteaya tries do gerrit | 13:57 | |
| jd__ | there's no much in the backlog actually :) | 13:57 |
| anteaya | okay thanks | 13:57 |
| *** thorst_ has joined #openstack-infra | 13:58 | |
| AJaeger | anteaya: Zuul/Jenkins is slow, not sure what'S going on. | 13:59 |
| anteaya | I think we have an overabundance of available test nodes | 13:59 |
| anteaya | AJaeger: very good, thanks | 13:59 |
| AJaeger | anteaya: Run your errands and let's wait for fungi and clarkb to figure it out. | 13:59 |
| *** links has quit IRC | 13:59 | |
| openstackgerrit | Dong Ma proposed openstack-infra/jenkins-job-builder: Add support for OWASP Dependency-Check Plugin https://review.openstack.org/179604 | 13:59 |
| anteaya | gerrit seems fine for me so far | 13:59 |
| *** piet has quit IRC | 13:59 | |
| anteaya | AJaeger: very good, thanks | 13:59 |
| AJaeger | anteaya: Yes, far too many test nodes - and Jenkins not using them ;( | 13:59 |
| anteaya | AJaeger: ah, yeah that is an issue | 13:59 |
| AJaeger | jd__: gerrit works fine here as well | 13:59 |
| * jd__ retries | 14:00 | |
| jd__ | it works again indeed, though it's quite slow | 14:01 |
| *** EricGonczer_ has joined #openstack-infra | 14:01 | |
| jd__ | or not, just got a 502 | 14:01 |
| AJaeger | anteaya: but reviewing my linters changes today would be great - see https://review.openstack.org/260192 - but that's part of the stack | 14:01 |
| AJaeger | jd__: yeah, it's slow now ;( | 14:01 |
| AJaeger | but no 502 yet for me | 14:02 |
| jd__ | âž” curl https://review.openstack.org/changes/260168/revisions/72f5156fa29c9d5d4894f99a2b2ff32bbc513b12/comments | 14:02 |
| jd__ | <title>502 Proxy Error</title> | 14:02 |
| *** apuimedo_ has quit IRC | 14:02 | |
| anteaya | I confirm 502 here | 14:02 |
| jd__ | <address>Apache/2.4.7 (Ubuntu) Server at review.openstack.org Port 443</address> | 14:02 |
| *** thorst_ has quit IRC | 14:02 | |
| jd__ | (I truncated the html a bit but you got it) | 14:02 |
| AJaeger | right now zuul is taking again 3 changes from the post queue - after those have been sitting there for ages... | 14:02 |
| *** otter768 has joined #openstack-infra | 14:02 | |
| anteaya | jd__: the gerrit 502 may be an artifact of the zuul/jenkins issue | 14:02 |
| * jd__ nods | 14:03 | |
| * AJaeger got all his requests server but it was slow | 14:03 | |
| *** apuimedo_ has joined #openstack-infra | 14:03 | |
| anteaya | I think I will have breakfast | 14:03 |
| anteaya | that isn't very helpful but at least I'm not adding to the problem | 14:03 |
| AJaeger | ;) | 14:03 |
| *** markvoelker has joined #openstack-infra | 14:04 | |
| AJaeger | Found the bottom of my stack for review - once you have time after your errands, anteaya : https://review.openstack.org/#/c/260202 | 14:04 |
| AJaeger | anteaya: enjoy! | 14:04 |
| jd__ | AJaeger: you're privileged ;) | 14:05 |
| AJaeger | LOL | 14:05 |
| *** hamzy is now known as hamzy_vacay | 14:05 | |
| *** weshay_xchat has quit IRC | 14:06 | |
| *** peristeri has joined #openstack-infra | 14:06 | |
| *** thorst_ has joined #openstack-infra | 14:07 | |
| *** weshay_xchat has joined #openstack-infra | 14:07 | |
| *** otter768 has quit IRC | 14:07 | |
| *** sdake has joined #openstack-infra | 14:08 | |
| *** sdake has quit IRC | 14:08 | |
| *** abregman has joined #openstack-infra | 14:08 | |
| *** markvoelker has quit IRC | 14:08 | |
| *** sdake has joined #openstack-infra | 14:09 | |
| *** abregman has quit IRC | 14:09 | |
| *** apuimedo_ has quit IRC | 14:10 | |
| *** apuimedo_ has joined #openstack-infra | 14:11 | |
| *** markvoelker has joined #openstack-infra | 14:12 | |
| *** markvoelker has quit IRC | 14:13 | |
| *** markvoelker has joined #openstack-infra | 14:14 | |
| *** thorst_ has quit IRC | 14:14 | |
| *** dkranz has joined #openstack-infra | 14:15 | |
| dhellmann | is something going on with gerrit? I'm getting errors loading https://review.openstack.org/#/c/194868/ | 14:17 |
| markus_z | dhellmann: I'm experiencing issues too. I don't know the cause. | 14:17 |
| Shrews | dhellmann: i believe so, looking at scrollback | 14:17 |
| *** apuimedo_ has quit IRC | 14:18 | |
| dhellmann | Shrews : thanks | 14:19 |
| dhellmann | markus_z : thanks for confirming | 14:19 |
| *** apuimedo_ has joined #openstack-infra | 14:19 | |
| dulek | I can confirm proxy errors too. And yes - I'm behind a corporate proxy. | 14:20 |
| AJaeger | dhellmann: somehow gerrit is slow... | 14:20 |
| AJaeger | infra-root, is anybody already awake to check what's going on with gerrit? | 14:21 |
| dhellmann | AJaeger : ok. I'm seeing actual page load failures, like jd__ was | 14:21 |
| * dhellmann wonders if this is a sign we should all take the day off ;-) | 14:21 | |
| *** rossella_s has quit IRC | 14:21 | |
| d0ugal | +1 | 14:21 |
| AJaeger | dhellmann: it is a sign ;) | 14:21 |
| d0ugal | s/day/week/? | 14:21 |
| *** rossella_s has joined #openstack-infra | 14:21 | |
| AJaeger | dhellmann: a sign to take the rest of the year off ;) | 14:22 |
| * dhellmann thinks that escalated quickly | 14:22 | |
| *** yamamoto has quit IRC | 14:22 | |
| *** rbrndt has joined #openstack-infra | 14:22 | |
| jd__ | haha | 14:23 |
| *** rlandy has joined #openstack-infra | 14:24 | |
| notmorgan | hm.. getting 503s from review.o.o | 14:24 |
| notmorgan | and 502s | 14:24 |
| *** apuimedo_ has quit IRC | 14:24 | |
| AJaeger | jd__: Now the 502s reached me as well ;( | 14:24 |
| bdemers | yup, 502s here too | 14:24 |
| *** apuimedo_ has joined #openstack-infra | 14:24 | |
| *** admin0 has joined #openstack-infra | 14:24 | |
| notmorgan | dhellmann: take the day off :P | 14:24 |
| monester | also got 502 errors on gerrit | 14:24 |
| notmorgan | it is intermittant | 14:25 |
| notmorgan | i have had a a few pages loading ok for me | 14:25 |
| mescanef | "The proxy server received an invalid response from an upstream server." | 14:25 |
| jd__ | AJaeger: looks like you got unprivileged! :) | 14:25 |
| notmorgan | yeah, this is likely the java process having some issues | 14:26 |
| AJaeger | Unfortunately the US admins are not yet here and the Australian ones have left, so let's all take a break... | 14:26 |
| jd__ | who got a nice story to tell then? :p | 14:26 |
| * AJaeger configures his out-of-office replies... | 14:26 | |
| notmorgan | AJaeger: you're much nicer than I am | 14:27 |
| *** bgmccollum has joined #openstack-infra | 14:27 | |
| notmorgan | i don't bother with out-of-office replies | 14:27 |
| *** vdrok has joined #openstack-infra | 14:28 | |
| AJaeger | ;) | 14:28 |
| * fungi is here, though likely not terribly functional yet since was up late | 14:28 | |
| AJaeger | morning fungi | 14:28 |
| AJaeger | Let me give a quick summary: | 14:29 |
| fungi | seeing what i can piece together from scrollback | 14:29 |
| notmorgan | fungi: drink moar caffiene | 14:29 |
| mordred | fungi: high load average | 14:29 |
| notmorgan | fungi: we can also wait for you to be more functional | 14:29 |
| mordred | gerrit not responding ish | 14:29 |
| mordred | fungi: I was considering bouncing it | 14:29 |
| AJaeger | Zuul was kind of slow the whole day, it only looked every two hours at the post queue and we already thought it was stuck | 14:29 |
| *** EricGonczer_ has quit IRC | 14:29 | |
| AJaeger | And now gerrit gives 502s ;( | 14:29 |
| notmorgan | mordred: is the server status from apache showing ick or just the java ? | 14:30 |
| AJaeger | fungi: No actual reports of problems in jobs due to the node change | 14:30 |
| mordred | notmorgan: well, there's at least one apache pegged at 95% cpu | 14:30 |
| notmorgan | wondering if this is mod_proxy being cranky because something happened with gerrit. | 14:31 |
| *** piet has joined #openstack-infra | 14:31 | |
| fungi | oh the post pipeline backup is proably gearman incorrectly offlining one of the multi-use workers. checking | 14:31 |
| *** markus_z has quit IRC | 14:31 | |
| elmiko | sounds like others are seeing the 502s from review.os.o? | 14:31 |
| *** apuimedo_ has quit IRC | 14:31 | |
| AJaeger | elmiko: welcome to the 502 party ;) | 14:32 |
| notmorgan | elmiko: yep, it's being looked at | 14:32 |
| notmorgan | :) | 14:32 |
| elmiko | ack, thanks =) | 14:32 |
| *** apuimedo_ has joined #openstack-infra | 14:32 | |
| notmorgan | mordred: can you take a look at server-status? | 14:32 |
| notmorgan | mordred: [since it's 403'd from the outside world] | 14:32 |
| fungi | hrm, no offline workers after all | 14:32 |
| AJaeger | fungi: There'S also a very large amound of available nodes | 14:32 |
| AJaeger | right now around 50 per jenkins, see http://grafana.openstack.org/dashboard/db/nodepool | 14:33 |
| * notmorgan is leaning towards an issue with gerrit itself, but wans to be sure it isn't apache misbehaving | 14:33 | |
| AJaeger | And http://puppetboard.openstack.org/ is dead | 14:33 |
| mordred | notmorgan: do I just curl localhost/server-status? | 14:33 |
| fungi | oh i see, their jobs ran but haven't reported | 14:33 |
| AJaeger | "Internal Server Error" | 14:33 |
| notmorgan | mordred: i think so. | 14:33 |
| fungi | i'll check zuul's debug log to see why it's not reporting | 14:33 |
| *** mattt has joined #openstack-infra | 14:33 | |
| notmorgan | because i'm getting massively slow response for even the 403 trying to hit that externally | 14:34 |
| dmsimard | o/ reporting errors/timeouts on review.o.o as well | 14:34 |
| notmorgan | the whole server seems verrrry cranky atm | 14:34 |
| *** mattymo has joined #openstack-infra | 14:34 | |
| fungi | mordred: you need to reach server-status over an ssh tunnel with a browser | 14:35 |
| mattymo | I'm sure everyone is aware of gerrit and 502 errors | 14:35 |
| anteaya | mattymo: you are correct | 14:35 |
| AJaeger | mattymo: yes | 14:35 |
| notmorgan | fungi: curl should be sufficient shouldn't it? i mean it's just text last i saw. | 14:35 |
| notmorgan | fungi: or did they wind up other magic? | 14:35 |
| *** mriedem has joined #openstack-infra | 14:35 | |
| bauzas | gerrit is down and I see a convo here, could we please have a topic for all the IRC chans? | 14:35 |
| fungi | notmorgan: oh, yeah a text-based browser or curl or something would, probably. it's just bigger than my terminal i think | 14:36 |
| *** ihrachys has quit IRC | 14:36 | |
| notmorgan | fungi: ah yeah | 14:36 |
| *** dizquierdo_afk is now known as dizquierdo | 14:37 | |
| notmorgan | esp. with a bunch of workers | 14:37 |
| fungi | no clue what's going on with puppetboard | 14:38 |
| notmorgan | yeah i'm seeing 4-5 seconds to get a 403 trying to hit server-status externally. so the workers are all bound up at the very least. i don't have insight into the java process health so can't say which side it is, likely if a bounce of gerrit is needed, so will apache need to be bounced. | 14:38 |
| *** sshnaidm has quit IRC | 14:38 | |
| notmorgan | just to clear things up more quickly. | 14:39 |
| anteaya | my company has arrived and I have to go do errands now, back in a few hours | 14:39 |
| *** sshnaidm has joined #openstack-infra | 14:40 | |
| *** apuimedo_ has quit IRC | 14:41 | |
| mordred | fungi: any problems with me bouncing our friend mr. gerrit? | 14:41 |
| *** mohankumar has joined #openstack-infra | 14:42 | |
| *** apuimedo_ has joined #openstack-infra | 14:42 | |
| *** notnownikki has joined #openstack-infra | 14:42 | |
| *** dslev has joined #openstack-infra | 14:42 | |
| fungi | mordred: go for it. i don't see anything interesting in zuul's logs yet to explain the jobs in the post pipeline waiting to report | 14:42 |
| fungi | though i wouldn't expect them to be blocking on gerrit anyway | 14:43 |
| *** breton has joined #openstack-infra | 14:43 | |
| dmsimard | review.o.o is back and snappy for me now. | 14:44 |
| mattt | same | 14:44 |
| mordred | k. restarted | 14:44 |
| mordred | I also restarted apache, fwiw | 14:44 |
| AJaeger | fungi, interesting: zuul seems to have nearly finished the post queue | 14:44 |
| AJaeger | thanks, mordred | 14:45 |
| notmorgan | mordred: looks much happier | 14:45 |
| notmorgan | just from a responsive standpoint | 14:46 |
| notmorgan | yeah 403 is now taking partial seconds so the apache processes are happier | 14:46 |
| notmorgan | too | 14:46 |
| *** apuimedo_ has quit IRC | 14:46 | |
| AJaeger | post queue is empty, fungi - this is strange... | 14:46 |
| fungi | doesn't look like we had a runaway ci commenting this time | 14:47 |
| fungi | AJaeger: well, they looked like their jobs were all finished but zuul hadn't gotten around to processing the results | 14:48 |
| *** apuimedo_ has joined #openstack-infra | 14:48 | |
| *** eharney has quit IRC | 14:49 | |
| *** xwizard_ has joined #openstack-infra | 14:50 | |
| notmorgan | fungi: i'm going to look into tuning down our proxy timeouts a little so apache is quicker to fail/less likely to bind up if gerrit gets unhappy | 14:51 |
| fungi | mordred: so before the restart, the server status seemed to indicate that all 6 apache worker threads had 25 connections each | 14:51 |
| AJaeger | fungi: that was only now, for several hours nothing happend. We had this morning them in the queue for 4 hours with only the translation jobs running, then jhesketh checked and the first 3 or 5 changes merged. | 14:51 |
| fungi | notmorgan: ^ | 14:51 |
| notmorgan | fungi: yeah | 14:51 |
| AJaeger | fungi: and then it was stuck again - until you poked and suddenly all of them finished... | 14:51 |
| *** zhurong has quit IRC | 14:51 | |
| notmorgan | fungi: but to be fair almost all of them were blocking in "sending reply" could be blocked on getting data from gerrit | 14:51 |
| notmorgan | fungi: based on server-status | 14:52 |
| fungi | AJaeger: i have a feeling zuul was stuck getting through to gerrit, even though it didn't need to report post pipeline results to gerrit, not sure since i didn't get a thread dump before things cleared up | 14:52 |
| AJaeger | gerrit worked this morning - but the recent slowdown might have started earlier and build up somehow... | 14:53 |
| pabelanger_ | nibalizer: asselin: Patches for puppet-nodepool are up, so I think it is just a matter of rotating through puppet modules now. | 14:53 |
| *** pabelanger_ is now known as pabelanger | 14:53 | |
| AJaeger | let's wait and see whether everything continues fine on that front... | 14:53 |
| *** links has joined #openstack-infra | 14:54 | |
| asselin__ | pabelanger, how does that help for https://review.openstack.org/#/c/253619/ ? | 14:55 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for manila-neutron job timeout bug 1528597 https://review.openstack.org/260575 | 14:55 |
| openstack | bug 1528597 in Manila "manila (neutron) gate jobs are timing out" [Undecided,New] https://launchpad.net/bugs/1528597 | 14:55 |
| pabelanger | asselin__: we'll be using puppetlabs-apache directly, IIRC they don't have the issue | 14:58 |
| pabelanger | asselin__: so, I believe your check is simpler | 14:58 |
| pabelanger | but you'd need to test | 14:58 |
| asselin__ | pabelanger, so it automatically includes the necessary mods? | 14:58 |
| *** lucasagomes is now known as lucas-hungry | 14:59 | |
| pabelanger | You still still need to include them | 14:59 |
| pabelanger | but I believe the original reload issue is gone | 14:59 |
| *** maishsk has quit IRC | 14:59 | |
| pabelanger | honestly, I'd have to test the patch under puppetlabs-apache and see what would be needed | 14:59 |
| *** apuimedo_ has quit IRC | 15:00 | |
| SpamapS | ianw: I read your post, and it looks great. Thanks for making it! | 15:00 |
| *** apuimedo_ has joined #openstack-infra | 15:00 | |
| mordred | SpamapS: link? | 15:00 |
| SpamapS | mordred: https://www.technovelty.org/openstack/platform-bootstrap-for-openstack-infrastructure.html | 15:01 |
| SpamapS | mordred: you're more likely to find anything infra-wrong with it than I am. ;) | 15:01 |
| *** hashar has joined #openstack-infra | 15:03 | |
| mordred | ianw: awesome. typo: "Now you'll need to setup a cloud.yaml file to give yourself the permissions to upload the image. " s/cloud.yaml/clouds.yaml/ | 15:03 |
| skolekonov | join #openstack-meeting-4 | 15:04 |
| skolekonov | sorry | 15:04 |
| mordred | :) | 15:04 |
| mordred | ianw: great post! | 15:04 |
| *** dtantsur is now known as dtantsur|brb | 15:04 | |
| *** maishsk has joined #openstack-infra | 15:05 | |
| *** cdent has joined #openstack-infra | 15:05 | |
| cdent | Does anybody know if this change that gordc has made is the right way to deal with independent release projects used in other jobs: https://review.openstack.org/#/c/258642/ | 15:06 |
| *** weshay_xchat has quit IRC | 15:08 | |
| *** eharney has joined #openstack-infra | 15:08 | |
| *** xyang1 has joined #openstack-infra | 15:09 | |
| *** apuimedo_ has quit IRC | 15:10 | |
| *** sdake has quit IRC | 15:10 | |
| *** apuimedo_ has joined #openstack-infra | 15:10 | |
| mordred | infra-root: heads up - ansible 2.0 is estimated/slated to be released on Jan 6. it SHOULD continue to work for us with no differences, but we should be aware in case something goes weird | 15:11 |
| krotscheck | What could possibly go wrong | 15:11 |
| krotscheck | 0:) | 15:11 |
| mordred | krotscheck: :) | 15:12 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Generate puppet-openstack ci docs https://review.openstack.org/259246 | 15:14 |
| *** thorst_ has joined #openstack-infra | 15:15 | |
| asselin__ | AJaeger, just a commit message update ^^ so that it can merge | 15:15 |
| *** maishsk has quit IRC | 15:16 | |
| notmorgan | mordred: at least it's happening AFTER new years. | 15:17 |
| notmorgan | mordred: they could have been really mean and released it on the 28th or 30th | 15:17 |
| notmorgan | and then jumped out for the year ... | 15:17 |
| notmorgan | :P | 15:17 |
| *** esker has quit IRC | 15:18 | |
| *** mattt has left #openstack-infra | 15:18 | |
| *** Qiming has quit IRC | 15:19 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for manila-neutron job timeout bug 1528597 https://review.openstack.org/260575 | 15:19 |
| *** zigo_ is now known as zigo | 15:19 | |
| openstack | bug 1528597 in Manila "manila (neutron) gate jobs are timing out" [Undecided,Confirmed] https://launchpad.net/bugs/1528597 | 15:19 |
| *** apuimedo_ has quit IRC | 15:19 | |
| * AJaeger does not understand why the fuel team proposes one repo after the other - but does not want to have them officially as part of the fuel team, see https://review.openstack.org/#/c/260208/ or https://review.openstack.org/#/c/259403/ | 15:20 | |
| *** apuimedo_ has joined #openstack-infra | 15:20 | |
| mwhahaha | because it's an official fuel team thing as defined by those instructions | 15:20 |
| mwhahaha | the problem is the plugins aren't really part of fuel | 15:20 |
| mwhahaha | The question "is this part of the fuel team" is different then "is this part of the fuel project under the big tent" | 15:21 |
| AJaeger | mwhahaha: I don't understand why you need to make this difference... | 15:21 |
| mwhahaha | a plugin can be developed by anyone and may not be tied to fuel | 15:22 |
| mwhahaha | the core project | 15:22 |
| mwhahaha | but we want to put it out there for community development rather than assign it to a specific person | 15:22 |
| mwhahaha | it does not necessarily track with the fuel project itself | 15:23 |
| AJaeger | so, fuel-plugin-designate: Who will work on that? | 15:23 |
| *** thorst_ has quit IRC | 15:23 | |
| mwhahaha | at the moment it probably has two devs who are responsible for it | 15:23 |
| mwhahaha | i have no idea. i can only speak to the etckeeper one which will be myself initially and will probably be handed off over time to different folks | 15:24 |
| fungi | it was a little easier to see in the days of the stackforge git namespace, but official and unofficial projects generally don't share groups in their gerrit acls because they're necessarily developed/maintained by different teams of people | 15:24 |
| *** esker has joined #openstack-infra | 15:25 | |
| AJaeger | fungi: they all have different - separate - acls | 15:25 |
| pabelanger | AJaeger: thanks again for the help landing the tox patches yesterday | 15:26 |
| AJaeger | pabelanger: you're welcome | 15:26 |
| *** zz_dimtruck is now known as dimtruck | 15:26 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Correct typos https://review.openstack.org/260474 | 15:27 |
| AJaeger | mwhahaha, fungi: I'm a bit surprised that we now have 82 fuel repos - and every week we get at least another one - and for each of them I hear: This is not part of the big fuel tent. And most of these repos come from the same person that requests them | 15:27 |
| *** rbrndt has quit IRC | 15:27 | |
| *** mohankumar has quit IRC | 15:28 | |
| mwhahaha | yea probably the guy who knows what is required | 15:28 |
| AJaeger | mwhahaha: I would have expected something else when every repo starts with fuel... | 15:28 |
| fungi | repos with "fuel" in their names account for 10% of the repos in our gerrit | 15:29 |
| AJaeger | mwhahaha: I'm not going to block your change - I'm just trying to understand how you work together on this. | 15:29 |
| mwhahaha | Sure and i appreciate that, I'm working through this as best I can because I don't know what is expected | 15:29 |
| asselin__ | AJaeger, can you +A it? | 15:30 |
| AJaeger | asselin__: I see yolanda gave already a +2A before... | 15:30 |
| *** bcourt has joined #openstack-infra | 15:30 | |
| asselin__ | yeah | 15:30 |
| AJaeger | in that case, I do ;) | 15:31 |
| asselin__ | :) thanks...helps with progress on the other patch... | 15:31 |
| *** otter768 has joined #openstack-infra | 15:32 | |
| AJaeger | fungi, coming back to post queue: Top of queue is now 40 min in the queue, we have lots of available workers (didn't check which type) and post queue only does translation update. Is that ok with our lowered precedence? | 15:33 |
| fungi | AJaeger: mwhahaha: it sounds like these are independently maintained repos then, which act as extensions or plugins to the fuel suite while not being maintained or overseen by the fuel team? we have some parallels in, for example, the neutron plugin ecosystem | 15:33 |
| openstackgerrit | Kaitlin Farr proposed openstack/requirements: Cap PyKMIP version https://review.openstack.org/260599 | 15:33 |
| AJaeger | fungi: yeah, that's how it looks like to me. Still, strikes me as odd - especially since most of the plugin repos get created by mescanef, but seems he does repo-creation-as-a-service ;) | 15:34 |
| *** weshay_xchat has joined #openstack-infra | 15:34 | |
| AJaeger | have to run some errands, bye for now! | 15:35 |
| jeblair | fungi, mordred: did you get any info from gerrit? like from javamelody or show-queues or show-caches? | 15:35 |
| mwhahaha | yes plugins themselves follow the neutron plugin model more than they are tied to core fuel | 15:35 |
| *** HeOS has quit IRC | 15:36 | |
| *** mudassirlatif has joined #openstack-infra | 15:36 | |
| jeblair | fungi, mordred: cacti basically just shows a cpu pegged, which could be apache, but we still don't know if the underlying cause was apache or (more likely) gerrit, yeah? | 15:36 |
| AJaeger | mwhahaha: the neutron plugins are nearly all part of neutron ... | 15:36 |
| *** AJaeger has quit IRC | 15:36 | |
| fungi | jeblair: i did not in the moments before it was restarted. i was looking at zuul logs at the time | 15:36 |
| *** HeOS has joined #openstack-infra | 15:36 | |
| fungi | also i think nodepool is mistaken about how many ready nodes there actually are | 15:37 |
| * mwhahaha shrugs | 15:37 | |
| mwhahaha | I'll bring it up to the ptl and get his take on if they should get moved under the fuel umbrella | 15:38 |
| mwhahaha | I don't care one way or the other to be honest | 15:38 |
| mwhahaha | i just need a repo in gerrit to put code | 15:38 |
| *** hurgleburgler has joined #openstack-infra | 15:38 | |
| fungi | oh, weird. for some reason nodepool decided to create a ton of bare-precise and devstack-precise nodes | 15:39 |
| fungi | hundreds | 15:40 |
| *** sabeen1 has joined #openstack-infra | 15:40 | |
| fungi | is this perhaps an upshot of zuul no longer knowing what node labels are associated with the jobs it needs to run? did nodepool's demand calculation grow dependent on our node label double-accounting for jobs? | 15:41 |
| jeblair | [2015-12-22 13:23:52,774] WARN org.eclipse.jetty.io.SelectorManager : Could not process key for channel java.nio.channels.SocketChannel[connected local=/127.0.0.1:8081 remote=/127.0.0.1:44097] | 15:42 |
| jeblair | we started getting *a lot* of those errors in the gerrit log starting then | 15:42 |
| jeblair | [2015-12-22 13:23:52,774] WARN org.eclipse.jetty.util.thread.QueuedThreadPool : HTTP{STARTED,5<=25<=25,i=17,q=50} rejected org.eclipse.jetty.io.AbstractConnection$2@476f11b | 15:43 |
| openstackgerrit | Diana Whitten proposed openstack-infra/project-config: Add openstack-ux repo https://review.openstack.org/259239 | 15:43 |
| *** weshay_xchat has quit IRC | 15:43 | |
| fungi | the bulk of the precise nodes seem to have been built around when the parameter functions were updated last night, though there have been more built since even though almost none of them are needed for running jobs | 15:44 |
| *** kgiusti has joined #openstack-infra | 15:44 | |
| jeblair | fungi, mordred: so i suspect something was stuck in gerrit; those error messages are related to the http connection queue | 15:45 |
| *** sabeen3 has joined #openstack-infra | 15:46 | |
| *** apuimedo_ has quit IRC | 15:46 | |
| fungi | jeblair: that sounds likely. and if zuul was waiting for some timeout while reporting i suppose that could account for why it was taking a while to process results even for pipelines that don't report back into gerrit (e.g. post)? | 15:47 |
| fungi | as for the nodepool precise demand, it definitely seems to think there is som | 15:47 |
| fungi | 2015-12-22 15:37:30,210 DEBUG nodepool.NodePool: Demand from gearman: bare-precise: 72 | 15:47 |
| fungi | 2015-12-22 15:37:30,211 DEBUG nodepool.NodePool: Demand from gearman: devstack-precise: 163 | 15:47 |
| *** apuimedo_ has joined #openstack-infra | 15:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Generate puppet-openstack ci docs https://review.openstack.org/259246 | 15:47 |
| pabelanger | Wow, looking at zuul-status, I've never seen that many nodes available! :D | 15:48 |
| jeblair | fungi, mordred: confirmation the error is related to the incoming request queue: https://groups.google.com/forum/m/#!msg/repo-discuss/-2hnEOqQA7M/BEq1UZhZCgAJ | 15:49 |
| *** sabeen1 has quit IRC | 15:49 | |
| *** yamamoto has joined #openstack-infra | 15:49 | |
| jeblair | fungi, mordred: basically "gerrit was slow doing _something_ and the queue filled"; possible causes include database issues... so a mysql show processlist might be a good idea | 15:49 |
| fungi | pabelanger: yeah, nodepool's building node types that zuul doesn't actually need. i think this is an upshot of my change to drop the job->node map from the zuul parameter functions | 15:49 |
| *** odlD2 has quit IRC | 15:50 | |
| pabelanger | fungi: darn! I thought christmas came early in the form of more clouds! hehe | 15:51 |
| jeblair | fungi: what was your change number? | 15:51 |
| fungi | jeblair: there were several, it was part of a series to get rid of the old precise defaults for icehouse-era branches. digging it up now | 15:52 |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Adding setup of credentialsId in stashNotifier https://review.openstack.org/260606 | 15:53 |
| clarkb | 260213 and 260214 | 15:53 |
| fungi | jeblair: 260213, then 260214, then 260334 | 15:53 |
| *** apuimedo_ has quit IRC | 15:54 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add astara to projects https://review.openstack.org/257314 | 15:54 |
| *** apuimedo_ has joined #openstack-infra | 15:54 | |
| *** baoli has quit IRC | 15:55 | |
| fungi | oh! i wonder whether zuul needs a restart so that we clear out all the old precise node registrations | 15:55 |
| *** baoli has joined #openstack-infra | 15:56 | |
| fungi | maybe nodepool is assuming it may need precise nodes when zuul queues a label-unqualified work request which is registered by both types? | 15:56 |
| *** bexelbie_ has joined #openstack-infra | 15:57 | |
| *** davideagnello has quit IRC | 15:58 | |
| *** bexelbie has quit IRC | 15:59 | |
| jeblair | fungi: for unqualified jobs, nodepool picks the first node type it sees for that job when assigning demand to nodes | 15:59 |
| *** davideagnello has joined #openstack-infra | 15:59 | |
| *** apuimedo_ has quit IRC | 15:59 | |
| fungi | jeblair: makes sense, so i can delete this excess temporarily and then we can do a controlled restart of zuul to clean up the stale registrations | 16:00 |
| jeblair | fungi: that likely means that now that the bulk of our demand is for unqualified jobs, it's picking the wrong node type | 16:00 |
| fungi | because once the gearman registrations are repopulated from scratch, we should only see jobs registered for one label each | 16:00 |
| *** apuimedo_ has joined #openstack-infra | 16:00 | |
| jeblair | this is probably something that could be corrected by paying attention to function registration count, but considering this happens once every few years, probably easier to just defer to zuulv3 | 16:01 |
| jeblair | fungi: you can delete the exccess, but it will immediately create more | 16:01 |
| fungi | sounds good. mystery solved | 16:01 |
| fungi | oh. hrm. okay i'll plan to do a zuul restart shortly in that case | 16:02 |
| jeblair | i think that's probably best | 16:02 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/infra-manual: Update release management chapter https://review.openstack.org/260610 | 16:03 |
| *** HeOS has quit IRC | 16:04 | |
| *** odlD2 has joined #openstack-infra | 16:04 | |
| *** lucas-hungry is now known as lucasagomes | 16:04 | |
| *** lezbar has joined #openstack-infra | 16:05 | |
| *** apuimedo_ has quit IRC | 16:05 | |
| *** apuimedo_ has joined #openstack-infra | 16:07 | |
| ttx | long overdue ^ | 16:08 |
| *** ifarkas has quit IRC | 16:09 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Update barbican and related projects https://review.openstack.org/257321 | 16:09 |
| openstackgerrit | Merged openstack-infra/reviewstats: Update infrastructure projects https://review.openstack.org/257337 | 16:09 |
| *** HeOS has joined #openstack-infra | 16:09 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Add cloudkitty to projects https://review.openstack.org/257327 | 16:10 |
| *** rbrndt has joined #openstack-infra | 16:11 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Update cinder and related projects https://review.openstack.org/257319 | 16:11 |
| openstackgerrit | Merged openstack-infra/reviewstats: Add chef related projects https://review.openstack.org/257316 | 16:11 |
| openstackgerrit | Merged openstack-infra/reviewstats: Update ironic and related projects https://review.openstack.org/257336 | 16:11 |
| *** FallenPegasus has joined #openstack-infra | 16:14 | |
| *** armax has joined #openstack-infra | 16:14 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Update nova and related projects https://review.openstack.org/257333 | 16:14 |
| *** dtantsur|brb is now known as dtantsur | 16:15 | |
| *** vgridnev has quit IRC | 16:15 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Add app-catalog to projects https://review.openstack.org/257330 | 16:15 |
| openstackgerrit | Merged openstack-infra/reviewstats: Add additional TripleO repositories https://review.openstack.org/222618 | 16:15 |
| openstackgerrit | Merged openstack-infra/reviewstats: Remove projects not in OpenStack repo https://review.openstack.org/253490 | 16:16 |
| openstackgerrit | Merged openstack-infra/reviewstats: Update climate to blazar https://review.openstack.org/257128 | 16:16 |
| openstackgerrit | Merged openstack-infra/reviewstats: Add cue to projects https://review.openstack.org/255079 | 16:16 |
| *** apuimedo_ has quit IRC | 16:16 | |
| *** nelsnelson has joined #openstack-infra | 16:16 | |
| *** daemontool has quit IRC | 16:17 | |
| *** apuimedo_ has joined #openstack-infra | 16:17 | |
| *** ZZelle_ has joined #openstack-infra | 16:17 | |
| *** esker has quit IRC | 16:18 | |
| *** ZZelle_ has quit IRC | 16:19 | |
| *** yamamoto has quit IRC | 16:19 | |
| *** esker has joined #openstack-infra | 16:19 | |
| *** pblaho has quit IRC | 16:20 | |
| *** thorst_ has joined #openstack-infra | 16:21 | |
| *** harlowja_at_home has joined #openstack-infra | 16:23 | |
| *** baoli has quit IRC | 16:24 | |
| *** shardy has quit IRC | 16:24 | |
| *** baoli has joined #openstack-infra | 16:26 | |
| *** coolsvap is now known as coolsvap|away | 16:26 | |
| *** _nadya_ has quit IRC | 16:26 | |
| fungi | okay, dumping check/gate pipelines and restarting zuul now | 16:27 |
| fungi | unless there are any objections | 16:27 |
| clarkb | go for it | 16:28 |
| fungi | here goes | 16:28 |
| fungi | it's on its way back up now | 16:28 |
| *** thorst_ has quit IRC | 16:28 | |
| *** apuimedo_ has quit IRC | 16:28 | |
| *** apuimedo_ has joined #openstack-infra | 16:29 | |
| clarkb | jeblair: greghaynes: I need to catch up on where reviews are, but if the image builders changes for nodepool are ready to go I would like to approve them today, restart nodepool, trigger a manual image build, then watch image builds tomorrow morning | 16:29 |
| jeblair | clarkb: sounds good to me | 16:30 |
| fungi | looks like we have all the requisite node types registered already, so i'll reenqueue check and gate now | 16:31 |
| krotscheck | jeblair: Could I trouble you to refresh your +2 on https://review.openstack.org/#/c/253236/ ? | 16:32 |
| *** fawadkhaliq has quit IRC | 16:32 | |
| *** mpavone has quit IRC | 16:33 | |
| krotscheck | Oooh, neat, a "diff against patchset" dropdown. | 16:34 |
| * krotscheck keeps discovering things in new gerrit that he likes. | 16:34 | |
| *** tonytan4ever has joined #openstack-infra | 16:34 | |
| *** gyee has joined #openstack-infra | 16:34 | |
| *** dprince has quit IRC | 16:35 | |
| *** apuimedo_ has quit IRC | 16:36 | |
| jeblair | krotscheck: that's always been there :) | 16:37 |
| *** smarcet has joined #openstack-infra | 16:37 | |
| smarcet | fungi: morning | 16:37 |
| smarcet | :) | 16:37 |
| smarcet | when u have some time, could u approve https://review.openstack.org/#/c/259598/? | 16:37 |
| smarcet | thx u | 16:37 |
| fungi | okay, all the exported gate and check pipeline changes have been reenqueued. working on cleaning up the unnecessary precise node surplus in nodepool now | 16:38 |
| jeblair | krotscheck: ps7 seems to be a rebase; was there any other change from ps6? | 16:38 |
| krotscheck | jeblair: Nope. | 16:38 |
| *** apuimedo_ has joined #openstack-infra | 16:38 | |
| krotscheck | jeblair: It just lost your +2 for some reason, maybe because of the merge conflict? | 16:38 |
| *** maishsk has joined #openstack-infra | 16:38 | |
| jeblair | krotscheck: ah, yeah, merge conflicts don't count as simple rebases so votes don't carry over. i trust that you resolved it correctly :) | 16:39 |
| krotscheck | jeblair: Urr.... well, I dunno. Past-me and Current-me don't really trust each other. | 16:39 |
| krotscheck | (Let's not talk about future-me. He's crazy) | 16:39 |
| jeblair | krotscheck: if your different selves could have separate gerrit accounts, that will make things easier. :) | 16:40 |
| *** tsg has joined #openstack-infra | 16:40 | |
| krotscheck | jeblair: I keep meaning to do that, but past-me keeps changing the password so I can't lock him out :) | 16:41 |
| *** piet has quit IRC | 16:41 | |
| *** dims has quit IRC | 16:42 | |
| *** apuimedo_ has quit IRC | 16:42 | |
| asselin__ | clarkb, it would be nice to have this too for new image builds https://review.openstack.org/#/c/217914/ | 16:43 |
| *** resker has joined #openstack-infra | 16:43 | |
| *** piet has joined #openstack-infra | 16:44 | |
| *** resker has quit IRC | 16:44 | |
| *** apuimedo_ has joined #openstack-infra | 16:44 | |
| *** fawadkhaliq has joined #openstack-infra | 16:44 | |
| *** yamamoto has joined #openstack-infra | 16:44 | |
| *** resker has joined #openstack-infra | 16:44 | |
| *** esker has quit IRC | 16:46 | |
| *** apuimedo_ has quit IRC | 16:49 | |
| *** AJaeger has joined #openstack-infra | 16:49 | |
| *** apuimedo_ has joined #openstack-infra | 16:50 | |
| *** admin0 has quit IRC | 16:50 | |
| *** maishsk has quit IRC | 16:52 | |
| flip214 | The XenProject CI says FAILED, but doesn't provide logs for https://review.openstack.org/#/c/256292/3 ... is anyone interested in that data point? | 16:53 |
| AJaeger | flip214: there's a contact email for XenProject CI, see the comment. They usually react pretty fast if you tell them | 16:54 |
| *** apuimedo_ has quit IRC | 16:55 | |
| AJaeger | fungi, jeblair: Glad to thear that the node mystery is solved! | 16:55 |
| *** maishsk has joined #openstack-infra | 16:55 | |
| flip214 | AJaeger: right now it started to work again... strange. | 16:55 |
| *** kashyap has left #openstack-infra | 16:55 | |
| *** ayoung has quit IRC | 16:55 | |
| *** apuimedo_ has joined #openstack-infra | 16:56 | |
| *** yamamoto has quit IRC | 16:56 | |
| flip214 | AJaeger: but the failure is due to the setup... http://117.218.56.3:9090/logs/73/259973/2/check/dsvm-tempest-full/20df5c7/logs/devstacklog.txt | 16:56 |
| flip214 | 2015-12-22 13:48:36.761 | + sudo ip -6 addr add 2001:db8::2/64 dev br-ex | 16:56 |
| *** yamamoto has joined #openstack-infra | 16:56 | |
| flip214 | 2015-12-22 13:48:36.776 | RTNETLINK answers: File exists | 16:56 |
| flip214 | 2015-12-22 13:48:36.778 | + exit_trap | 16:56 |
| AJaeger | flip214: this is the wrong channel to discuss them, I can't help with that... | 16:57 |
| flip214 | asselin__: ^^ interested in that? | 16:57 |
| isaacb | flip214: see https://review.openstack.org/#/c/239363/ | 16:57 |
| *** Hal1 has quit IRC | 16:58 | |
| *** yamamoto has quit IRC | 16:58 | |
| flip214 | isaacb: thanks for the info. | 16:58 |
| *** Hal1 has joined #openstack-infra | 16:59 | |
| *** abregman has joined #openstack-infra | 17:00 | |
| *** bgmccollum has left #openstack-infra | 17:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:00 | |
| *** apuimedo_ has quit IRC | 17:01 | |
| *** yaume has quit IRC | 17:01 | |
| BobBall | flip214: That was a known bug... but if you recheck things seem to be working now | 17:01 |
| AJaeger | fungi, clarkb, jeblair: A review of the linters changes would be great - bottom of stack is https://review.openstack.org/#/c/260202/ . | 17:01 |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Fix on bower test file exist condition: https://review.openstack.org/259598 | 17:01 |
| BobBall | flip214: I see on the change you've not used the correct syntax to recheck the XenProject CI; it tells you how to recheck in the message from the CI (click Toggle CI if you don't see it) - type "Xen: recheck" to ask for the CI to run the test again | 17:02 |
| *** apuimedo_ has joined #openstack-infra | 17:02 | |
| *** amuller is now known as amuller_afk | 17:03 | |
| fungi | is anybody interested in looking into why puppetboard is broken? if not, i'll dig into that next | 17:03 |
| *** zeih has joined #openstack-infra | 17:03 | |
| flip214 | BobBall: thank you very much, I already wasted a few tries... | 17:04 |
| flip214 | Yeah right, "Toggle CI" does help ;\ | 17:04 |
| *** hashar has quit IRC | 17:05 | |
| *** maishsk has quit IRC | 17:05 | |
| *** apuimedo_ has quit IRC | 17:09 | |
| *** apuimedo_ has joined #openstack-infra | 17:10 | |
| *** sshnaidm has quit IRC | 17:11 | |
| *** dims has joined #openstack-infra | 17:12 | |
| *** yamahata has joined #openstack-infra | 17:12 | |
| *** daemontool has joined #openstack-infra | 17:13 | |
| *** boris-42 has quit IRC | 17:13 | |
| openstackgerrit | Yuriy Zveryanskyy proposed openstack/diskimage-builder: [EXPERIMENTAL] Ansible deploy ramdisk for Ironic https://review.openstack.org/239447 | 17:13 |
| *** HeOS has quit IRC | 17:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Bump APScheduler to >=3.0 https://review.openstack.org/260637 | 17:14 |
| *** vgridnev has joined #openstack-infra | 17:14 | |
| pabelanger | jeblair: would be interested in your thoughts ^ | 17:14 |
| pabelanger | bumping to APScheduler 3.0, but need to dig more to see what configuration changes there are | 17:15 |
| openstackgerrit | Yuriy Zveryanskyy proposed openstack/diskimage-builder: [EXPERIMENTAL] Ansible deploy ramdisk for Ironic https://review.openstack.org/239447 | 17:15 |
| pabelanger | jeblair: main reason for this, fedora has python-APScheduler 3.0.5 | 17:15 |
| *** piet has quit IRC | 17:15 | |
| *** davideagnello has quit IRC | 17:15 | |
| *** apuimedo_ has quit IRC | 17:17 | |
| *** harlowja_at_home has quit IRC | 17:18 | |
| *** apuimedo_ has joined #openstack-infra | 17:19 | |
| *** unicell1 has quit IRC | 17:19 | |
| *** isaacb has quit IRC | 17:20 | |
| AJaeger | therve: regarding 259643: Will your change work with stable/kilo and stable/liberty as well? Please comment in the review... | 17:22 |
| *** apuimedo_ has quit IRC | 17:23 | |
| fungi | 2015-12-22 17:07:15,356 ERROR [o.a.a.b.BrokerService] Failed to start ActiveMQ JMS Message Broker (localhost, null). Reason: org.apache.activemq.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero). | 17:23 |
| *** davideagnello has joined #openstack-infra | 17:23 | |
| *** kzaitsev_mb has quit IRC | 17:23 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Add astara to projects https://review.openstack.org/257314 | 17:23 |
| fungi | that seems to be causing puppetdb to shut down immediately when started | 17:23 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Bump APScheduler to >=3.0 https://review.openstack.org/260637 | 17:24 |
| *** apuimedo_ has joined #openstack-infra | 17:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental job to check Rally installation on Centos 7 https://review.openstack.org/260258 | 17:24 |
| *** eil397 has joined #openstack-infra | 17:25 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Bump os-client-config constraint and minimum https://review.openstack.org/260092 | 17:25 |
| fungi | nibalizer: you aware of anything that might have caused puppetdb to start going toes-up recently? | 17:25 |
| fungi | i see we also have warnings about postgresql <=9.1 support being deprecated, and another warning about being unable to install optimal indexing | 17:26 |
| *** thorst_ has joined #openstack-infra | 17:27 | |
| fungi | "For maximum index performance, we recommend using PostgreSQL 9.3 or greater." | 17:27 |
| fungi | so much fun stuff in the puppetdb log | 17:27 |
| SpamapS | wow.. every time I click a review link now, the new horrible UI just slaps me in the face | 17:27 |
| *** dizquierdo has quit IRC | 17:27 | |
| fungi | SpamapS: gerritslap! | 17:27 |
| SpamapS | I wonder if the new UI is as hated in other gerrit user communities as it is here? | 17:28 |
| BobBall | Is that the new trout slap? | 17:28 |
| *** apuimedo_ has quit IRC | 17:28 | |
| crinkle | fungi: cacti says puppetdb is almost out of space? | 17:28 |
| fungi | crinkle: ooh! | 17:28 |
| *** apuimedo_ has joined #openstack-infra | 17:29 | |
| *** derekh has quit IRC | 17:29 | |
| fungi | crinkle: right you are, that probably explains the corrupt transaction log. i bet it's truncated | 17:29 |
| *** peristeri has quit IRC | 17:29 | |
| fungi | i should have looked at disk utilization first | 17:29 |
| *** woodster_ has joined #openstack-infra | 17:29 | |
| *** zeih has quit IRC | 17:30 | |
| *** devananda has quit IRC | 17:30 | |
| AJaeger | mestery, dougwig, armax : Could one of you neutron liasons review https://review.openstack.org/#/c/256719/ , please? | 17:30 |
| nibalizer | fungi: crinkle neat | 17:31 |
| fungi | we have about 8.5gb in /var (more than a third of the rootfs) | 17:32 |
| fungi | interestingly, we have >3gb in /usr/src | 17:32 |
| fungi | headers packages for ~70 kernel versions. that seems pretty unnecessary | 17:33 |
| *** thorst_ has quit IRC | 17:33 | |
| *** apuimedo_ has quit IRC | 17:33 | |
| fungi | apt-get autoremove is cleaning up 4gb on disk now | 17:34 |
| *** dtantsur is now known as dtantsur|afk | 17:35 | |
| *** apuimedo_ has joined #openstack-infra | 17:36 | |
| *** tmcpeak has joined #openstack-infra | 17:36 | |
| fungi | and apt-get clean freed up almost another 4gb | 17:36 |
| tmcpeak | AJaeger, fungi: o/ | 17:36 |
| *** jerryz has joined #openstack-infra | 17:37 | |
| *** maishsk has joined #openstack-infra | 17:37 | |
| *** piet has joined #openstack-infra | 17:38 | |
| *** dprince has joined #openstack-infra | 17:39 | |
| *** devananda has joined #openstack-infra | 17:40 | |
| nibalizer | fungi: cool | 17:42 |
| tmcpeak | fungi: can you take a look at this bad boy when you get a chance? https://review.openstack.org/#/c/260192/ | 17:42 |
| nibalizer | what caused apt to keep so many files around? | 17:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Support project templates https://review.openstack.org/260649 | 17:43 |
| *** apuimedo_ has quit IRC | 17:43 | |
| *** zeih has joined #openstack-infra | 17:43 | |
| *** apuimedo_ has joined #openstack-infra | 17:43 | |
| *** resker has quit IRC | 17:44 | |
| *** zns has joined #openstack-infra | 17:44 | |
| *** zns has quit IRC | 17:44 | |
| *** jyuso1 has joined #openstack-infra | 17:45 | |
| fungi | nibalizer: unattended-upgrades doesn't autoremove or clean by default | 17:45 |
| *** notnownikki has quit IRC | 17:46 | |
| fungi | there was a change in review to switch that on recently, not sure if it got approved and isn't taking effect for some reason, or is still under discussion | 17:46 |
| dougwig | AJaeger: looking | 17:46 |
| *** zeih has quit IRC | 17:47 | |
| fungi | crinkle: nibalizer: since the puppetdb service is complaining about corruption in /var/lib/puppetdb/mq/localhost/KahaDB/db-12340.log is there any reason i shouldn't move that transaction log out of the way and retry starting? we'd just lose some history? and we only keep the last 10 reports for a node anyway, right? | 17:47 |
| *** e0ne has quit IRC | 17:47 | |
| nibalizer | fungi: im fine with that | 17:48 |
| fungi | giving it a try | 17:48 |
| *** abregman is now known as abregman|nb | 17:49 | |
| *** piet has quit IRC | 17:50 | |
| *** cburgess has quit IRC | 17:50 | |
| fungi | no such luck. still complaining that it can't start the broker, "probably due to KahaDB corruption or version incompatibility after a PuppetDB downgrade" | 17:50 |
| nibalizer | 'Unable to start broker in "/var/lib/puppetdb/mq". This is probably due to KahaDB corruption or version incompatibility after a PuppetDB downgrade (see "KahaDB Corruption" in the PuppetDB manual)' | 17:50 |
| nibalizer | haha jinx | 17:51 |
| nibalizer | have you issued a service puppetdb restart? | 17:51 |
| fungi | yeah, that's after a restrat | 17:51 |
| fungi | restart | 17:51 |
| fungi | tips on an easy way to reinitialize an empty state? | 17:51 |
| *** jyuso1 has quit IRC | 17:52 | |
| *** cburgess has joined #openstack-infra | 17:52 | |
| *** apuimedo_ has quit IRC | 17:52 | |
| fungi | looks like it's complaining about /var/lib/puppetdb/mq/localhost/KahaDB/db-12340.log still even though i moved it out of there entirely | 17:53 |
| *** apuimedo_ has joined #openstack-infra | 17:53 | |
| fungi | looks like you started it again? | 17:53 |
| *** apoorvad has joined #openstack-infra | 17:53 | |
| nibalizer | i haven't started it no | 17:53 |
| * anteaya is back | 17:54 | |
| anteaya | anything I can do to help? | 17:54 |
| fungi | oh, or maybe i was reading the uncaught exceptions at shutdown as related to startup | 17:54 |
| * anteaya goes to read backscroll | 17:54 | |
| nibalizer | fungi: maybe its fixed? | 17:54 |
| fungi | seems to be happier now according to the logs | 17:54 |
| fungi | yeah | 17:54 |
| nibalizer | the logs is hapier and puppetboard is functional | 17:54 |
| fungi | yep, web interface is good again | 17:54 |
| *** admin0 has joined #openstack-infra | 17:54 | |
| fungi | need to break for lunch, then spend a couple minutes prepping for our meeting an hour from now | 17:55 |
| anteaya | fungi: enjoy lunch | 17:56 |
| *** Swami has joined #openstack-infra | 17:56 | |
| *** apuimedo_ has quit IRC | 18:00 | |
| *** apuimedo_ has joined #openstack-infra | 18:02 | |
| *** openstackgerrit has quit IRC | 18:02 | |
| *** openstackgerrit has joined #openstack-infra | 18:03 | |
| *** jab is now known as zz_jab | 18:03 | |
| openstackgerrit | Merged openstack-infra/system-config: Initial commit of codesearch docs https://review.openstack.org/257626 | 18:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Support project templates https://review.openstack.org/260649 | 18:05 |
| *** amuller_afk is now known as amuller | 18:05 | |
| *** rvasilets___ has quit IRC | 18:05 | |
| *** xyang1 has quit IRC | 18:06 | |
| AJaeger | thanks, dougwig ! | 18:06 |
| *** apuimedo_ has quit IRC | 18:06 | |
| *** apuimedo_ has joined #openstack-infra | 18:08 | |
| *** cdent has quit IRC | 18:08 | |
| *** thorst_ has joined #openstack-infra | 18:12 | |
| *** apuimedo_ has quit IRC | 18:13 | |
| *** apuimedo_ has joined #openstack-infra | 18:14 | |
| *** piet has joined #openstack-infra | 18:14 | |
| *** rbrndt has quit IRC | 18:14 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add 'testing' node to site.pp https://review.openstack.org/241497 | 18:14 |
| *** bexelbie_ has quit IRC | 18:15 | |
| anteaya | AJaeger: I agree with your concern, 82 repos in gerrit without any kind of belief that they are all useful | 18:15 |
| *** links has quit IRC | 18:15 | |
| anteaya | I'm sure some are, just not sure all 82+ are | 18:15 |
| *** kmartin has quit IRC | 18:16 | |
| AJaeger | anteaya: even if they some would be usefull, I wonder how they are organized... | 18:17 |
| AJaeger | this does not scale | 18:18 |
| *** apuimedo_ has quit IRC | 18:18 | |
| *** thorst_ has quit IRC | 18:18 | |
| anteaya | AJaeger: agreed, I think it is time for some overview and accounting here | 18:19 |
| *** thorst_ has joined #openstack-infra | 18:19 | |
| anteaya | to share some understanding | 18:19 |
| openstackgerrit | Merged openstack-infra/project-config: add more jobs for networking-fortinet https://review.openstack.org/256719 | 18:19 |
| pabelanger | fungi: so, openstackid on ubuntu trusty seems to work, but some issues. Maybe we can stand up a new dev server and can work on it there. | 18:19 |
| *** apuimedo_ has joined #openstack-infra | 18:20 | |
| pabelanger | things like enabling php modules (mcrypt) needs to be done | 18:20 |
| *** zeih has joined #openstack-infra | 18:20 | |
| AJaeger | anteaya: if you have time, I'd like the stack starting at 260202 to move forward, please | 18:21 |
| *** rossella_s has quit IRC | 18:21 | |
| nibalizer | SpamapS: re: changescreen you can look at the future future here: http://polygerrit.nibalizer.com/q/status:open | 18:21 |
| *** rossella_s has joined #openstack-infra | 18:22 | |
| anteaya | AJaeger: okay I think I have a sense of things again | 18:22 |
| anteaya | AJaeger: I'll take a look | 18:22 |
| *** kmartin has joined #openstack-infra | 18:23 | |
| AJaeger | thanks, anteaya | 18:23 |
| SpamapS | nibalizer: nice and clean | 18:24 |
| nibalizer | and unable to diff lol | 18:24 |
| anteaya | AJaeger: so when I look here: https://review.openstack.org/#/c/260202 and it says in the top right hand corner that there are 5 related changes, these are the children of this parent patch? | 18:24 |
| nibalizer | but it used to be able to diff so I am thinking that is an issue with my deployment | 18:24 |
| AJaeger | anteaya: correct | 18:24 |
| *** unicell has joined #openstack-infra | 18:25 | |
| clarkb | jeblair: can you make sure that https://review.openstack.org/#/c/232800/15 looks good to you now? the commit message needs an update I think but the code seems to address your comments | 18:25 |
| anteaya | AJaeger: and when you say the stack is starting at 260202 is that the parent of all the other changes, or the last child? | 18:25 |
| clarkb | jesusaurus: for your comments on that test I can write a follow up change to address those comments to avoid rebases | 18:25 |
| *** breton has left #openstack-infra | 18:25 | |
| AJaeger | anteaya: parent of all, so 260202 will merge first | 18:25 |
| anteaya | AJaeger: thank you | 18:25 |
| *** bharathm has joined #openstack-infra | 18:25 | |
| AJaeger | 260202 is a prerequisite for 260192 - 260192 adds a description and the test failed, so I updated the testsuite first | 18:26 |
| jeblair | clarkb: yeah, i agree the commit msg is out of date but didn't think that was worth holding up on so i previously applied +2; i don't think there's anything substatially diffirent since then, just a rebase yeah? | 18:26 |
| jeblair | clarkb: (i have +2d) | 18:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-openstackid: Enable php5-mcript https://review.openstack.org/260660 | 18:27 |
| pabelanger | fungi: so far, only issue ^ | 18:27 |
| clarkb | jeblair: oh I was just diffing against where I least voted where you had comments must've missed a +2 that went away due to rebase | 18:27 |
| anteaya | AJaeger: so what happens when 260202 merges and there is no description? | 18:27 |
| jeblair | clarkb: ah cool | 18:27 |
| *** thorst_ has quit IRC | 18:28 | |
| fungi | pabelanger: awesome--thanks! | 18:28 |
| anteaya | AJaeger: is this allowing a space for their to be descriptions or forcing descriptions? | 18:28 |
| clarkb | ok so I think whole stack is ready to go in this case, I will approve after the meeting and get the service restarted | 18:28 |
| clarkb | jesusaurus: and I will push a followup chane ot address your -1 to avoid another rebase on that stack | 18:28 |
| AJaeger | anteaya: description is an optional parameter | 18:28 |
| anteaya | AJaeger: thank you | 18:28 |
| jesusaurus | clarkb: a follow up is fine by me | 18:28 |
| AJaeger | For builders, see ".Required('builders')" - so that's a required one | 18:29 |
| *** piet has quit IRC | 18:29 | |
| *** baoli has quit IRC | 18:29 | |
| openstackgerrit | Philipp Marek proposed openstack-infra/ciwatch: Remove jQuery filter to improve page load time https://review.openstack.org/260193 | 18:29 |
| fungi | pabelanger: is that safe on precise as well? the challenge in this exercise is to be able to upgrade one server (development) without immediately upgrading the other (production) | 18:30 |
| *** zeih has quit IRC | 18:30 | |
| *** zeih has joined #openstack-infra | 18:30 | |
| *** notnownikki has joined #openstack-infra | 18:30 | |
| greghaynes | clarkb: w00t | 18:30 |
| *** zeih has quit IRC | 18:30 | |
| *** zeih has joined #openstack-infra | 18:31 | |
| *** zeih has quit IRC | 18:31 | |
| *** zeih has joined #openstack-infra | 18:31 | |
| anteaya | AJaeger: ah there it is, thank you | 18:31 |
| *** dimtruck is now known as zz_dimtruck | 18:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Refactor pipeline loading https://review.openstack.org/260662 | 18:31 |
| pabelanger | fungi: Ya, I need to test that for sure. | 18:31 |
| *** zeih_ has joined #openstack-infra | 18:32 | |
| *** xarses has quit IRC | 18:35 | |
| anteaya | AJaeger: so this file you take away the filters on the newly created gate-project-config-example-pep8 job, since it is no longer filtered using gate-project-config-pep8: https://review.openstack.org/#/c/260192/6/zuul/layout.yaml | 18:35 |
| anteaya | AJaeger: so that is a difference for the -example job, yes? | 18:35 |
| *** harlowja has quit IRC | 18:36 | |
| *** zeih has quit IRC | 18:36 | |
| *** apuimedo_ has quit IRC | 18:36 | |
| *** changbl has quit IRC | 18:36 | |
| *** harlowja has joined #openstack-infra | 18:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Add description label to jenkins checker https://review.openstack.org/260202 | 18:37 |
| *** piet has joined #openstack-infra | 18:37 | |
| *** e0ne has joined #openstack-infra | 18:37 | |
| *** vilobhmm11 has joined #openstack-infra | 18:37 | |
| *** apuimedo_ has joined #openstack-infra | 18:37 | |
| crinkle | clarkb: want to look at 235005, 258193, and 257640 for me today? (mysql) | 18:38 |
| clarkb | sure | 18:38 |
| *** _nadya_ has joined #openstack-infra | 18:38 | |
| *** _nadya_ has quit IRC | 18:38 | |
| AJaeger | anteaya: running the job on both projects was bad style | 18:38 |
| AJaeger | I could add the filter back for project-config-example but with the few changes going into that one, we don#t need it. | 18:39 |
| openstackgerrit | Ian proposed openstack-infra/jenkins-job-builder: Adding Build Keeper plugin. https://review.openstack.org/260664 | 18:39 |
| anteaya | AJaeger: no argument from me | 18:39 |
| AJaeger | ;) | 18:40 |
| jesusaurus | clarkb: is your logstash curator cron working? I just noticed that mine still has unescaped %s in the crontab | 18:40 |
| anteaya | okay fine, just acknowledging that the code change does affect that job | 18:40 |
| anteaya | asselin__: ^^ | 18:40 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade: No Mutable Defaults https://review.openstack.org/260665 | 18:40 |
| clarkb | jesusaurus: let me check | 18:41 |
| anteaya | AJaeger: mind including that point in the commit message on 260192? | 18:41 |
| *** piet has quit IRC | 18:41 | |
| AJaeger | will do directly | 18:42 |
| clarkb | jesusaurus: ya looks like it is working now | 18:42 |
| anteaya | AJaeger: thank you | 18:42 |
| *** angdraug has joined #openstack-infra | 18:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Replace bashate/pep8 in project-config by linters https://review.openstack.org/260192 | 18:43 |
| *** _nadya_ has joined #openstack-infra | 18:43 | |
| jesusaurus | clarkb: ah, looks like maybe a problem with install_modules.sh on my end | 18:43 |
| AJaeger | anteaya: done ^ | 18:43 |
| *** peristeri has joined #openstack-infra | 18:44 | |
| clarkb | crinkle: so looking at these changes we can interrogate the version that is installed to determine how to use the mysql module? | 18:44 |
| anteaya | thanks | 18:44 |
| clarkb | crinkle: I didn't realize this was possible, thats a neat trick | 18:46 |
| crinkle | clarkb: yes | 18:46 |
| crinkle | clarkb: yep it's a new feature in stdlib from nibalizer | 18:46 |
| anteaya | I am trying to +2 https://review.openstack.org/#/c/260192/6 and when I click the reply button I have a text box and two buttons post and cancel but zero radio buttons to vote | 18:47 |
| anteaya | sigh | 18:47 |
| *** hdd has joined #openstack-infra | 18:47 | |
| clarkb | crinkle: do the paste and wiki use of mysql also need the version checking like nodepool? | 18:48 |
| *** salv-orlando has quit IRC | 18:48 | |
| AJaeger | anteaya: switch to revision 7 | 18:48 |
| crinkle | clarkb: I didn't use it there since it's all in system-config and can be done all at once | 18:48 |
| AJaeger | top-left corner | 18:48 |
| anteaya | somehow I was on patchset 6/7 yet refreshing the patch did not take me to revision 7 | 18:48 |
| *** apuimedo_ has quit IRC | 18:48 | |
| clarkb | crinkle: gotcha | 18:49 |
| anteaya | AJaeger: yeah, very time consuming | 18:49 |
| AJaeger | your URL contains a 6 | 18:49 |
| anteaya | ah | 18:49 |
| anteaya | refreshing used to just take me to the most recent version | 18:49 |
| *** EricGonczer_ has joined #openstack-infra | 18:49 | |
| *** apuimedo_ has joined #openstack-infra | 18:49 | |
| clarkb | cool so I guess I should just confirm that innodb is the default in puppet-mysql for my own sanity, but otherwise this set of changes lgtm | 18:50 |
| crinkle | clarkb: cool | 18:50 |
| crinkle | clarkb: this will cause mysql to restart but I don't forsee any other issues | 18:51 |
| *** piet has joined #openstack-infra | 18:51 | |
| anteaya | AJaeger: this patch is in a can't merge state: https://review.openstack.org/#/c/260420/1 | 18:52 |
| *** _nadya_ has quit IRC | 18:52 | |
| clarkb | crinkle: ok | 18:52 |
| clarkb | maybe we should stop the services manually then and schedule this | 18:52 |
| nibalizer | crinkle: i think i have reviewed all the relevant patches | 18:52 |
| nibalizer | let me know if i missed something | 18:52 |
| clarkb | "wiki and paste and nodepool will be offline for 30 minutes while we update infrastructure management of these services" | 18:52 |
| clarkb | crinkle: nibalizer ^ what do you think | 18:53 |
| crinkle | sgtm | 18:53 |
| crinkle | nibalizer: ty | 18:53 |
| dhellmann | AJaeger : can you join us in #openstack-oslo to discuss a translation question? | 18:53 |
| nibalizer | clarkb: meeting in 10 minutes though right? | 18:53 |
| nibalizer | maybe want to wait until after | 18:53 |
| AJaeger | dhellmann: will do | 18:53 |
| *** zeih has joined #openstack-infra | 18:53 | |
| clarkb | nibalizer: yes after, I am saying we should properly shcedule this since the services will flap | 18:54 |
| *** hdd has quit IRC | 18:54 | |
| clarkb | rather than just do it | 18:54 |
| *** zeih_ has quit IRC | 18:55 | |
| clarkb | I am having a hard time seeing how override_optons does anything but affect hte user used | 18:55 |
| *** notnownikki has quit IRC | 18:56 | |
| clarkb | oh because puppet can't update a var in place so the name changes | 18:56 |
| *** rbrndt has joined #openstack-infra | 18:56 | |
| anteaya | clarkb: this is puppet-mysql changes to those services? | 18:57 |
| clarkb | anteaya: we are upgrading the puppet-mysql version which will apparently restart mysql | 18:58 |
| anteaya | okey dokey, thanks | 18:58 |
| *** apuimedo_ has quit IRC | 18:58 | |
| clarkb | anteaya: the services and mysql themselves shouldn't change, but the restart of the db will cause an outage | 18:58 |
| *** ihrachys has joined #openstack-infra | 18:58 | |
| fungi | clarkb: crinkle: if we want to do this after the meeting, can we coordinate a paste outage to also move its db to trove? i'm happy to drive that part, and that way we don't need additional outages for it | 18:59 |
| anteaya | clarkb: cool | 18:59 |
| clarkb | fungi: sure? I thought it was still broken? | 18:59 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for manila volume create NoValidHost bug 1528665 https://review.openstack.org/260670 | 18:59 |
| openstack | bug 1528665 in Manila "Share instance fails to create due to volume creation fails" [Undecided,New] https://launchpad.net/bugs/1528665 | 18:59 |
| fungi | clarkb: we never retried after we discovered how to fix the timeouts, afaik | 18:59 |
| fungi | it's that (weekly meeting) time once again! find us in #openstack-meeting for the next hour | 19:00 |
| clarkb | though I am already doing image builders after the meeting | 19:00 |
| clarkb | so don't wantto commit to too much all at once | 19:00 |
| *** apuimedo_ has joined #openstack-infra | 19:00 | |
| *** lyz_ is now known as lyz | 19:00 | |
| fungi | ahh, okay | 19:00 |
| *** zz_dimtruck is now known as dimtruck | 19:00 | |
| fungi | well, if i do the paste migration first, then it shouldn't require an outage when the wiki/nodepool outage happens | 19:01 |
| clarkb | yup | 19:01 |
| *** qb_ has joined #openstack-infra | 19:04 | |
| *** ianw has quit IRC | 19:06 | |
| *** ianw has joined #openstack-infra | 19:07 | |
| *** dimtruck is now known as zz_dimtruck | 19:07 | |
| *** apuimedo_ has quit IRC | 19:07 | |
| *** fawadkhaliq has quit IRC | 19:07 | |
| *** apuimedo_ has joined #openstack-infra | 19:08 | |
| *** piet has quit IRC | 19:08 | |
| *** vilobhmm11 has quit IRC | 19:09 | |
| *** vilobhmm11 has joined #openstack-infra | 19:09 | |
| *** smarcet has quit IRC | 19:09 | |
| *** aopincar has joined #openstack-infra | 19:10 | |
| openstackgerrit | Merged openstack-infra/system-config: Upgrade the stdlib module to 4.10.0 https://review.openstack.org/235005 | 19:11 |
| *** apuimedo_ has quit IRC | 19:12 | |
| *** FallenPegasus has quit IRC | 19:13 | |
| *** xarses has joined #openstack-infra | 19:14 | |
| *** dmsimard is now known as caffeine | 19:14 | |
| *** apuimedo_ has joined #openstack-infra | 19:14 | |
| *** caffeine is now known as dmsimard | 19:14 | |
| *** xarses has quit IRC | 19:15 | |
| *** xarses has joined #openstack-infra | 19:15 | |
| *** mdenny has joined #openstack-infra | 19:16 | |
| *** piet has joined #openstack-infra | 19:17 | |
| *** ihrachys has quit IRC | 19:18 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for manila volume create NoValidHost bug 1528665 https://review.openstack.org/260670 | 19:20 |
| openstack | bug 1528665 in Manila "Share instance fails to create due to volume creation fails due to NoValidHost" [Undecided,Confirmed] https://launchpad.net/bugs/1528665 | 19:20 |
| *** baoli has joined #openstack-infra | 19:21 | |
| *** baoli_ has joined #openstack-infra | 19:22 | |
| *** admin0 has left #openstack-infra | 19:22 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-openstackid: Enable php5-mcript https://review.openstack.org/260660 | 19:22 |
| *** thorst_ has joined #openstack-infra | 19:25 | |
| *** baoli has quit IRC | 19:26 | |
| *** apuimedo_ has quit IRC | 19:26 | |
| *** apuimedo_ has joined #openstack-infra | 19:28 | |
| *** claudiub has quit IRC | 19:28 | |
| *** rossella_s has quit IRC | 19:30 | |
| *** thorst_ has quit IRC | 19:32 | |
| *** apuimedo_ has quit IRC | 19:32 | |
| *** piet has quit IRC | 19:33 | |
| *** vgridnev has quit IRC | 19:33 | |
| *** apuimedo_ has joined #openstack-infra | 19:33 | |
| *** apuimedo_ has quit IRC | 19:38 | |
| *** hashar has joined #openstack-infra | 19:39 | |
| *** apuimedo_ has joined #openstack-infra | 19:39 | |
| *** zeih_ has joined #openstack-infra | 19:40 | |
| *** vgridnev has joined #openstack-infra | 19:44 | |
| *** zeih has quit IRC | 19:44 | |
| *** dims has quit IRC | 19:45 | |
| *** apuimedo_ has quit IRC | 19:46 | |
| *** salv-orlando has joined #openstack-infra | 19:47 | |
| *** apuimedo_ has joined #openstack-infra | 19:47 | |
| *** _nadya_ has joined #openstack-infra | 19:48 | |
| *** _nadya_ has quit IRC | 19:48 | |
| *** eharney has quit IRC | 19:50 | |
| *** Ramanjaneya has quit IRC | 19:51 | |
| *** hdd has joined #openstack-infra | 19:54 | |
| *** lucasagomes is now known as lucas-pto | 19:55 | |
| *** apuimedo_ has quit IRC | 19:56 | |
| clarkb | I have approved the nodepool image builders change stack | 19:56 |
| greghaynes | \O/ | 19:57 |
| *** apuimedo_ has joined #openstack-infra | 19:57 | |
| *** markvoelker has quit IRC | 19:57 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Bump os-client-config constraint and minimum https://review.openstack.org/260092 | 19:58 |
| *** aopincar has quit IRC | 19:59 | |
| *** terriyu has joined #openstack-infra | 19:59 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-nodepool: Add logic to work with old and new mysql module https://review.openstack.org/258193 | 20:00 |
| *** xyang1 has joined #openstack-infra | 20:00 | |
| pleia2 | my turn to be sick, away resting for a bit | 20:01 |
| *** iraj has joined #openstack-infra | 20:01 | |
| *** ihrachys has joined #openstack-infra | 20:01 | |
| *** iraj has left #openstack-infra | 20:01 | |
| nibalizer | clarkb: did you approve the puppet change too? | 20:01 |
| *** apuimedo_ has quit IRC | 20:01 | |
| fungi | get well, pleia2! | 20:01 |
| clarkb | nibalizer: no, it isn't necessary | 20:01 |
| greghaynes | nibalizer: I think the plan is to run default first | 20:01 |
| anteaya | pleia2: :( | 20:01 |
| jeblair | fungi: i think the way i have been looking at the scripts situation is that basic things that don't need to change often should be slave scripts. i figured the 'install stuff using bindep' script would be among those | 20:01 |
| anteaya | pleia2: hope you feel better soon | 20:02 |
| AJaeger | pleia2: get well soon! | 20:02 |
| jeblair | fungi: but if it's changing enough to be a concern, maybe we should pull it back into jjb. and maybe we should rethink that and pull more into jjb too. | 20:02 |
| clarkb | nibalizer: step 2 is to update puppet to start splitting the services apart, but it should run the service as is against zuul geard just fine | 20:02 |
| jeblair | fungi: none of that addresses the testing aspect; only ease of changes | 20:02 |
| clarkb | nibalizer: on the same host and all that so not what we want long term but good for "does this work" | 20:02 |
| fungi | jeblair: yeah, agreed but those scripts start somewhere and having them self-tested while they're on their way to stabilizing would help reduce risks | 20:02 |
| *** ihrachys has quit IRC | 20:02 | |
| *** apuimedo_ has joined #openstack-infra | 20:03 | |
| nibalizer | clarkb: we don't need a service definition and an init script? | 20:03 |
| fungi | like, for example, now i want the script to confirm that it did in fact install everything we wanted (because you can't trust yum's exit codes on nonexistent packages) | 20:03 |
| clarkb | nibalizer: nope nodepoold starts one for us by default | 20:03 |
| *** ybathia has joined #openstack-infra | 20:03 | |
| nibalizer | oh ok | 20:03 |
| *** zeih_ has quit IRC | 20:03 | |
| *** dims has joined #openstack-infra | 20:04 | |
| greghaynes | Yep, by default it operates the same way as before, you just have the option to run them separately now. | 20:04 |
| fungi | and might be nice to add some loops around things like package list updates and package retrieval to mitigate some network-related breakage | 20:04 |
| fungi | being able to demonstrate in our ci that these additions work would go a long way to making them easier to review | 20:04 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-specs: Gerrit-2.11 spec is implemented https://review.openstack.org/260694 | 20:04 |
| AJaeger | here's my action item ^ | 20:05 |
| nibalizer | greghaynes: oh okay that makes more sense then | 20:05 |
| *** bharathm has quit IRC | 20:05 | |
| fungi | as it stands, i'll probably just continue trickling improvements into it while leaving the jobs using it only experimental for a while still | 20:05 |
| *** eharney has joined #openstack-infra | 20:05 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Upgrade mysql module to 3.6.1 https://review.openstack.org/257640 | 20:05 |
| anteaya | AJaeger: thank you | 20:06 |
| *** terriyu has quit IRC | 20:06 | |
| AJaeger | anteaya: thanks for the linters reviews. Anybody here to give another +2 for https://review.openstack.org/#/c/260192 ? jhesketh gave his +2 already and then I updated the commit message... | 20:06 |
| clarkb | greghaynes: and unit tests failed :( | 20:06 |
| crinkle | clarkb: I updated the puppet-nodepool and system-config patches to be explicit about the default-storage-engine setting | 20:06 |
| fungi | jeblair: and keeping the logic inside the job configuration isn't a ton better. it's still not self-tested, just has a slightly faster turn-around on the proposal->implementation cycle | 20:07 |
| AJaeger | anteaya: I'll rebase 260420 once its dependencies are merged... | 20:07 |
| clarkb | crinkle: thanks | 20:07 |
| greghaynes | clarkb: pymysql fixture? | 20:07 |
| clarkb | greghaynes: osclientconfig | 20:07 |
| greghaynes | clarkb: wah | 20:07 |
| clarkb | greghaynes: that may be a sign | 20:07 |
| fungi | jeblair: though in zuul v3 that changes of course | 20:07 |
| anteaya | AJaeger: awesome | 20:07 |
| greghaynes | did they release? | 20:07 |
| clarkb | Region fake-region is not a valid region name for cloud envvars. Valid choices are . Please note that region names are case sensitive. | 20:07 |
| greghaynes | clarkb: there was a release ~1 day ago :( | 20:08 |
| *** ybathia_ has joined #openstack-infra | 20:08 | |
| *** hdd has quit IRC | 20:08 | |
| openstackgerrit | Merged openstack/diskimage-builder: yum-minimal: do not configure eth0 & eth1 for DHCP automatically https://review.openstack.org/257172 | 20:09 |
| *** thorst_ has joined #openstack-infra | 20:09 | |
| *** apuimedo_ has quit IRC | 20:10 | |
| *** apuimedo_ has joined #openstack-infra | 20:10 | |
| clarkb | Ie2f3d2ca3ccbe7e3dd674983136b42c323544997 is the change in oscc that added the error | 20:10 |
| *** ybathia has quit IRC | 20:10 | |
| *** ybathia_ is now known as ybathia | 20:10 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Make server variable expansion optional https://review.openstack.org/257437 | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Have inventory use os-client-config extra_config https://review.openstack.org/257436 | 20:11 |
| clarkb | greghaynes: I think I may have a fix | 20:11 |
| clarkb | give me a few | 20:11 |
| *** dims_ has joined #openstack-infra | 20:11 | |
| greghaynes | clarkb: sweet | 20:12 |
| mriedem | hmm, tox not liking some gate runs? ValueError: No closing quotation | 20:13 |
| mriedem | http://logs.openstack.org/88/256388/1/gate/gate-designate-dsvm-powerdns/9a48601/console.html.gz#_2015-12-16_13_56_43_534 | 20:13 |
| AJaeger | mriedem: tox 2.x incompatiblity | 20:13 |
| AJaeger | which project is this on? | 20:13 |
| mriedem | several | 20:13 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22ValueError:%20No%20closing%20quotation%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:%5C%221%5C%22%20AND%20NOT%20build_queue:%5C%22experimental%5C%22 | 20:13 |
| fungi | mriedem: mostly on stable branches? | 20:14 |
| mriedem | no | 20:14 |
| mriedem | mostly master | 20:14 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-nodepool: Nodepool image build workers https://review.openstack.org/240093 | 20:14 |
| *** dims has quit IRC | 20:14 | |
| nibalizer | greghaynes: clarkb i think thats ^^ the way we want the puppet to look | 20:14 |
| greghaynes | nibalizer: cool | 20:14 |
| *** hdd has joined #openstack-infra | 20:14 | |
| clarkb | greghaynes: basically we need to write a cloud config for all tests now :/ | 20:15 |
| clarkb | greghaynes: and populate it iwth a valid region | 20:15 |
| *** apuimedo_ has quit IRC | 20:15 | |
| AJaeger | mriedem: give me one repo name and I'll propose a patch... | 20:15 |
| *** apuimedo_ has joined #openstack-infra | 20:15 | |
| greghaynes | clarkb: wah? What was it doing before for regions by default? | 20:15 |
| openstackgerrit | Merged openstack/diskimage-builder: yum-minimal: leave behind dummy /etc/resolv.conf https://review.openstack.org/257173 | 20:16 |
| clarkb | greghaynes: it would return None I think | 20:16 |
| openstackgerrit | Merged openstack/diskimage-builder: yum-minimal : install selinux policy packages https://review.openstack.org/259867 | 20:16 |
| mriedem | AJaeger: https://github.com/openstack/designate/blob/master/devstack/gate/run_tempest_tests.sh#L32 | 20:16 |
| *** tzn has joined #openstack-infra | 20:16 | |
| mriedem | AJaeger: ahhhh https://github.com/openstack/designate/commit/0f5fd246c7ecb851aff1ec489bca1904110fe9fa | 20:16 |
| greghaynes | clarkb: oh, it is because we are setting a region_name | 20:17 |
| AJaeger | mriedem: exactly, remove the comments | 20:17 |
| clarkb | greghaynes: ya, maybe we just unset region_name? | 20:17 |
| greghaynes | clarkb: hrm, this is wierd. If we pass in a region_name then why would that force us to have one in the config | 20:17 |
| clarkb | greghaynes: but also, the best part is we pass validate = False | 20:17 |
| greghaynes | seems backwards | 20:17 |
| clarkb | greghaynes: because if it doesn't find one in the config oscc raises | 20:18 |
| mriedem | hence the question about stable, we have to backport those tox.ini changes | 20:18 |
| clarkb | greghaynes: I agree that seems broken in oscc, particularly since we say validate = False | 20:18 |
| AJaeger | so, designate is fixed, mriedem . Let me fix devstack-plugin-ceph... | 20:18 |
| mriedem | ironic is fixed too | 20:18 |
| *** zeih has joined #openstack-infra | 20:19 | |
| *** rfolco has quit IRC | 20:19 | |
| greghaynes | clarkb: Ya, so I think occ should not fail here if you pass in a region as an arg | 20:20 |
| greghaynes | clarkb: regardless of validate | 20:20 |
| mriedem | AJaeger: tracking with 1527562 | 20:20 |
| mriedem | bug 1527562 | 20:20 |
| openstack | bug 1527562 in Ironic "tox fails when run - ValueError: No closing quotation" [High,Fix released] https://launchpad.net/bugs/1527562 - Assigned to Lucas Alvares Gomes (lucasagomes) | 20:20 |
| greghaynes | mordred: ^ | 20:20 |
| clarkb | greghaynes: agreed | 20:20 |
| clarkb | greghaynes: reading the code it seems pretty redundant | 20:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Make server variable expansion optional https://review.openstack.org/257437 | 20:20 |
| clarkb | greghaynes: you pass in a region and only if that region is in the config does it accepti | 20:20 |
| AJaeger | mriedem: thanks | 20:20 |
| greghaynes | clarkb: yes, it is effectively sanity checking that the arg you passed in is valid | 20:21 |
| *** boris-42 has joined #openstack-infra | 20:21 | |
| clarkb | greghaynes: except that has the side effect of requiring that config everywhere | 20:21 |
| clarkb | so we can fix it in nodepool's tests by writing that config everywhere | 20:21 |
| jesusaurus | so that check isn't respecting validate = False? | 20:21 |
| clarkb | or make oscc less picky | 20:21 |
| clarkb | jesusaurus: nope | 20:21 |
| greghaynes | clarkb: I want to fix occ | 20:22 |
| clarkb | greghaynes: ++ | 20:22 |
| greghaynes | clarkb: not sure if we want to do both since nodepool is going to be broken until that releases | 20:22 |
| *** EricGonczer_ has quit IRC | 20:22 | |
| AJaeger | mriedem: training-labs is fixed already | 20:23 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for tox ValueError fail bug 1527562 https://review.openstack.org/260701 | 20:23 |
| openstack | bug 1527562 in devstack-plugin-ceph "tox fails when run - ValueError: No closing quotation" [Undecided,In progress] https://launchpad.net/bugs/1527562 - Assigned to Andreas Jaeger (jaegerandi) | 20:23 |
| clarkb | jesusaurus: apparently validate = False means don't make a ksc auth plugin instance | 20:23 |
| clarkb | mordred: are you around? | 20:23 |
| mordred | clarkb: yes | 20:23 |
| AJaeger | mriedem: releases is fixes AFAIK as well, is devstack fixed? | 20:24 |
| mordred | clarkb, greghaynes point me at the issue? | 20:24 |
| clarkb | mordred: oscc now requires that the region name be in your clouds.yaml before you can use the region name | 20:24 |
| greghaynes | mordred: http://logs.openstack.org/75/195175/36/gate/gate-nodepool-python27/f01d50c/console.html | 20:24 |
| mordred | thanks | 20:24 |
| mriedem | https://github.com/openstack-dev/devstack/commit/f59e6df23a430f0622215fec5bbbe18cfe7fee9c | 20:24 |
| clarkb | so we pass in region-name = fake-region to get_one_cloud and it raises and exception because we don't have a clouds.yaml with a region name in it | 20:24 |
| mriedem | AJaeger: looks like it | 20:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Bump python-daemon to 2.1.0 https://review.openstack.org/260702 | 20:24 |
| *** EricGonczer_ has joined #openstack-infra | 20:25 | |
| mordred | so, the _intent_ there is that if your cloud defines a list of regions in the vendors yaml and you request a bad region name, that we give you a "that's an invalid region name" error | 20:25 |
| mordred | this is, obviously, a bug in that :) | 20:25 |
| *** hdd has quit IRC | 20:25 | |
| mordred | I will see if I can come up with a somewhat sane patch to fix on the occ side | 20:26 |
| clarkb | mordred: I can fix that if that is what the intent is | 20:26 |
| *** baoli_ has quit IRC | 20:26 | |
| *** e0ne has quit IRC | 20:28 | |
| *** tmcpeak has quit IRC | 20:28 | |
| clarkb | mordred: greghaynes https://review.openstack.org/260704 something like that | 20:29 |
| clarkb | I think we also need to handle the case where regions and region_name are nil equivalents | 20:29 |
| pabelanger | if anybody wants to look at python-daemon patch, checkout zuul ^. Seems they added new logic to setup supplementary groups, which looks to need super user creds | 20:29 |
| clarkb | not sure if we just return '' as region_name in that case or if that is an error | 20:30 |
| mordred | clarkb: that looks about right | 20:30 |
| pabelanger | not sure why they didn't default False out of the box | 20:30 |
| clarkb | mordred: if regions and region_name are falsey should it also raise and require one or the other? | 20:30 |
| jesusaurus | pabelanger: ah, that issue strikes again | 20:30 |
| greghaynes | clarkb: if region_name is '' and regions is [] then we might still want to error there | 20:31 |
| pabelanger | jesusaurus: Oh, where did you see it? | 20:31 |
| clarkb | greghaynes: ya thats what I am wondering about, is that avalid config or an error? | 20:31 |
| hashar | pabelanger: Joshua had a patch for zuul / python-daeon | 20:31 |
| * greghaynes looks at mordred | 20:31 | |
| pabelanger | hashar: Ya, mine bumps the min requirements to support it :) | 20:31 |
| pabelanger | hashar: fedora packaging | 20:31 |
| jesusaurus | pabelanger: in log_processor | 20:31 |
| hashar | pabelanger: got it by hit a few weeks ago ( https://review.openstack.org/#/c/250315/ ) | 20:31 |
| pabelanger | jesusaurus: Ah, going to check code | 20:32 |
| *** tzn has quit IRC | 20:32 | |
| *** e0ne has joined #openstack-infra | 20:32 | |
| pabelanger | hashar: Ha ha! | 20:32 |
| pabelanger | nice | 20:32 |
| pabelanger | will abandon mine | 20:32 |
| mordred | clarkb: no, region is not required in all places | 20:32 |
| greghaynes | mordred: clarkb ok, LGTM then | 20:33 |
| *** irenab has joined #openstack-infra | 20:33 | |
| hashar | pabelanger: sorry for my crappy english. It is late here :D | 20:33 |
| pabelanger | any zuul-core interested in 250315? :) | 20:33 |
| *** xarses has quit IRC | 20:33 | |
| pabelanger | hashar: np | 20:33 |
| *** odlD2 has quit IRC | 20:34 | |
| clarkb | cool, now just to figure out how to test this | 20:34 |
| pabelanger | hashar: Oh, hit me up tomorrow or next week. Want to chat about your nodepool SIGINT patch you landed a while back | 20:34 |
| pabelanger | hashar: can't get it working as expected | 20:34 |
| *** xarses has joined #openstack-infra | 20:34 | |
| greghaynes | clarkb: so, it is probably a good time for me to foods since were gonig to just be waiting on this to merge/release | 20:34 |
| clarkb | mordred: in the case of no regions and no region_name what should we return? is the expanded region of '' correct? | 20:34 |
| greghaynes | ? | 20:34 |
| clarkb | greghaynes: ya | 20:35 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for tox ValueError fail bug 1527562 https://review.openstack.org/260701 | 20:35 |
| openstack | bug 1527562 in devstack-plugin-ceph "tox fails when run - ValueError: No closing quotation" [Undecided,In progress] https://launchpad.net/bugs/1527562 - Assigned to Andreas Jaeger (jaegerandi) | 20:35 |
| hashar | have fun folks, I am off *wave* | 20:36 |
| *** apuimedo_ has quit IRC | 20:36 | |
| mordred | clarkb: dunno. I need to read keystoneauth code ... notmorgan ^^ ? you know ? | 20:37 |
| notmorgan | hmm? | 20:37 |
| * notmorgan reads up | 20:37 | |
| *** hashar has quit IRC | 20:37 | |
| *** apuimedo_ has joined #openstack-infra | 20:38 | |
| notmorgan | oh weird | 20:38 |
| notmorgan | uhmm... | 20:38 |
| notmorgan | i don't know off the top of my head, but i think '' | 20:39 |
| *** hdd has joined #openstack-infra | 20:39 | |
| clarkb | notmorgan: { 'region': '', 'values': {} } is what we will return | 20:40 |
| clarkb | with my change as is | 20:40 |
| mordred | yah - '' looks like it will work | 20:40 |
| clarkb | mordred: notmorgan so not the dict above? just ''? | 20:40 |
| clarkb | I will have ot update my change if so | 20:41 |
| AJaeger | asselin: I'll update project-config-example... | 20:41 |
| asselin__ | AJaeger, thanks | 20:42 |
| *** zeih has quit IRC | 20:44 | |
| mordred | clarkb: the dict | 20:44 |
| mordred | the region value should be '' | 20:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config-example: Use linters tox environment https://review.openstack.org/260710 | 20:45 |
| *** apuimedo_ has quit IRC | 20:45 | |
| AJaeger | asselin__: ^ | 20:45 |
| clarkb | mordred: cool my change should be correct then, working on a set of tests though | 20:45 |
| mordred | \o/ | 20:45 |
| *** apuimedo_ has joined #openstack-infra | 20:45 | |
| *** zeih has joined #openstack-infra | 20:46 | |
| *** odlD2 has joined #openstack-infra | 20:47 | |
| *** maishsk has quit IRC | 20:48 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: Add the log of package list for pip3 https://review.openstack.org/246017 | 20:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters job for project-config-example https://review.openstack.org/260711 | 20:49 |
| AJaeger | asselin__: and this one ^ | 20:49 |
| *** salv-orlando has quit IRC | 20:49 | |
| *** baoli has joined #openstack-infra | 20:49 | |
| AJaeger | all changes for the linters job now use the linters topic: https://review.openstack.org/#/q/topic:linters+status:open | 20:50 |
| *** tsg has quit IRC | 20:50 | |
| *** apuimedo_ has quit IRC | 20:50 | |
| *** apuimedo_ has joined #openstack-infra | 20:50 | |
| *** zeih has quit IRC | 20:51 | |
| asselin__ | anyone know how to you see parent/child relationships in new gerrit ui? | 20:52 |
| AJaeger | asselin__: the "Related changes" on the top left | 20:52 |
| anteaya | asselin__: top right | 20:52 |
| *** kgiusti has left #openstack-infra | 20:52 | |
| *** bharathm has joined #openstack-infra | 20:53 | |
| asselin__ | oh, so those are the depends-on...ok | 20:53 |
| asselin__ | what about needed-by? | 20:53 |
| *** baoli has quit IRC | 20:54 | |
| asselin__ | nevermind...goes up in that case..ok not as intuitive, but good enough | 20:54 |
| anteaya | the patches taht used to be considered dependencies are also listed there | 20:54 |
| anteaya | yeah, I'm trying to figure it out as well myself | 20:55 |
| *** baoli has joined #openstack-infra | 20:55 | |
| * asselin__ goes to lunch | 20:55 | |
| Shrews | asselin__: yeah, i find that entire view totally unintuitive myself. one of the reasons i switched to gertty | 20:55 |
| anteaya | ha ha ha | 20:55 |
| *** apuimedo_ has quit IRC | 20:55 | |
| anteaya | lunch is never a bad idea | 20:55 |
| *** apuimedo_ has joined #openstack-infra | 20:55 | |
| *** thorst_ has quit IRC | 20:57 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list https://review.openstack.org/254337 | 20:57 |
| *** thorst_ has joined #openstack-infra | 20:57 | |
| *** markvoelker has joined #openstack-infra | 20:58 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/puppet-openstackci: Do initial git commit with stock puppet files https://review.openstack.org/260714 | 20:58 |
| *** nelsnelson has quit IRC | 20:59 | |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add notes to see reference-gertty.yaml for more info https://review.openstack.org/260715 | 21:00 |
| *** bharathm has quit IRC | 21:00 | |
| *** nelsnelson has joined #openstack-infra | 21:01 | |
| openstackgerrit | Victor Ryzhenkin proposed openstack/requirements: Add openstack/murano-agent to projects.txt https://review.openstack.org/256555 | 21:01 |
| *** markvoelker has quit IRC | 21:03 | |
| *** maishsk has joined #openstack-infra | 21:03 | |
| *** otter768 has quit IRC | 21:05 | |
| *** apuimedo_ has quit IRC | 21:05 | |
| *** edmondsw has joined #openstack-infra | 21:06 | |
| *** edmondsw has quit IRC | 21:06 | |
| *** thorst_ has quit IRC | 21:06 | |
| *** apuimedo_ has joined #openstack-infra | 21:06 | |
| *** zeih has joined #openstack-infra | 21:06 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Make install-distro-packages.sh more robust https://review.openstack.org/260717 | 21:07 |
| clarkb | mordred: ok updated with tests now | 21:07 |
| clarkb | mordred: should cover the entirety of that one method | 21:07 |
| *** eharney has quit IRC | 21:07 | |
| clarkb | mordred: if that looks good to you a release would be awesome | 21:07 |
| clarkb | oh wait I need to add one more test case | 21:08 |
| *** eharney has joined #openstack-infra | 21:08 | |
| *** odlD2 has quit IRC | 21:09 | |
| *** bharathm has joined #openstack-infra | 21:09 | |
| *** vgridnev has quit IRC | 21:11 | |
| *** qadaemon has joined #openstack-infra | 21:11 | |
| *** maishsk has quit IRC | 21:11 | |
| *** tmcpeak has joined #openstack-infra | 21:12 | |
| *** maishsk has joined #openstack-infra | 21:12 | |
| *** otter768 has joined #openstack-infra | 21:13 | |
| *** maishsk has quit IRC | 21:13 | |
| clarkb | ya my change is broken | 21:14 |
| *** electrofelix has joined #openstack-infra | 21:15 | |
| *** woodster_ has quit IRC | 21:16 | |
| electrofelix | Wondering if anyone else has encountered issues trying to use the new gerrit with android mobile browsers? | 21:16 |
| clarkb | electrofelix: it works fine for me so far except for the diff pages | 21:17 |
| clarkb | scrolling is almost impossible on the diffs | 21:17 |
| openstackgerrit | Merged openstack-infra/ciwatch: Remove jQuery filter to improve page load time https://review.openstack.org/260193 | 21:17 |
| electrofelix | clarkb: Don't use it too much,but found out useful to be able to reply to comments, but now it keeps preventing me adding inline comments. | 21:18 |
| clarkb | I havne't tried to do inline comments because I can't really see what I am commenting on | 21:18 |
| electrofelix | Ok, I'll look to log an issue about it. Presume storyboard? | 21:21 |
| *** abregman|nb has quit IRC | 21:21 | |
| jeblair | electrofelix: https://code.google.com/p/gerrit/issues/list | 21:21 |
| *** ajmiller_ is now known as ajmiller | 21:21 | |
| *** tonytan4ever has quit IRC | 21:21 | |
| *** apuimedo_ has quit IRC | 21:22 | |
| *** odlD2 has joined #openstack-infra | 21:22 | |
| *** apuimedo_ has joined #openstack-infra | 21:23 | |
| *** davidlenwell has quit IRC | 21:24 | |
| *** Mmike has quit IRC | 21:25 | |
| clarkb | mordred: ok current patchset should be good | 21:26 |
| clarkb | test case covers nodepool testing situation and additonal tests cover the old behavior | 21:26 |
| *** Mmike has joined #openstack-infra | 21:27 | |
| *** Mmike has joined #openstack-infra | 21:27 | |
| electrofelix | jeblair: so no need to track it locally? | 21:27 |
| *** thingee has quit IRC | 21:27 | |
| *** adam_g has quit IRC | 21:27 | |
| *** baoli has quit IRC | 21:27 | |
| *** eil397 has quit IRC | 21:27 | |
| *** trown is now known as trown|outttypeww | 21:28 | |
| *** baoli has joined #openstack-infra | 21:28 | |
| clarkb | greghaynes: maybe you want to review https://review.openstack.org/260704 when you have a chance | 21:28 |
| *** apuimedo_ has quit IRC | 21:28 | |
| greghaynes | clarkb: now with 100% more tests? | 21:28 |
| clarkb | greghaynes: ya | 21:28 |
| clarkb | oh bah I broke other tests though | 21:29 |
| clarkb | grumble grumble /me fixes | 21:29 |
| *** apuimedo_ has joined #openstack-infra | 21:29 | |
| anteaya | electrofelix: tracking locally doesn't do anything, we can't change the upstream code any faster than gerrit devs | 21:29 |
| anteaya | actually they are faster at it | 21:29 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Make install-distro-packages.sh more robust https://review.openstack.org/260717 | 21:29 |
| fungi | must appease the bash8 overlords | 21:29 |
| greghaynes | clarkb: I have a nit | 21:30 |
| greghaynes | clarkb: if youre updating it | 21:30 |
| clarkb | oh too late but I can reupdate | 21:30 |
| *** thingee has joined #openstack-infra | 21:31 | |
| clarkb | greghaynes: we should hit that | 21:31 |
| clarkb | greghaynes: but we just want t oreturn None in that case | 21:31 |
| greghaynes | oh | 21:31 |
| clarkb | no known regions | 21:32 |
| clarkb | return value == None | 21:32 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add containers job to tripleo.org CI status https://review.openstack.org/259177 | 21:32 |
| clarkb | the problem is _get_regions was overloaded between list known regions and return valid default fallback | 21:32 |
| *** tzn has joined #openstack-infra | 21:32 | |
| clarkb | so I split it up a bit to make that division of duties | 21:32 |
| greghaynes | Ah, ok, I like | 21:33 |
| clarkb | greghaynes: otherwise we get the valid default when trying to determine if the region name we passed in should just be used and it gets all confused | 21:33 |
| *** bhunter71 has joined #openstack-infra | 21:33 | |
| *** adam_g has joined #openstack-infra | 21:33 | |
| *** adam_g has quit IRC | 21:34 | |
| *** adam_g has joined #openstack-infra | 21:34 | |
| anteaya | AJaeger: what is the purpose of non-voting lint jobs? https://review.openstack.org/#/c/260402/1 | 21:34 |
| anteaya | AJaeger: to me that just translates as noise | 21:34 |
| AJaeger | anteaya: I was looking at changes like https://review.openstack.org/#/c/251223/2/zuul/layout.yaml | 21:35 |
| *** davidlenwell has joined #openstack-infra | 21:36 | |
| anteaya | AJaeger: I don't understand | 21:36 |
| anteaya | apparently I don't have as much context as I need for that to make sense to me as an argument | 21:37 |
| AJaeger | that change adds a non-voting lint job (using bandit) | 21:37 |
| anteaya | what is bandit? | 21:37 |
| AJaeger | anteaya: bandit is a security linter | 21:37 |
| anteaya | I still don't see the argument for wanting a non-voting lint job | 21:38 |
| *** tzn has quit IRC | 21:38 | |
| anteaya | security linter or no | 21:38 |
| electrofelix | anteaya: not expecting it to be fixed any faster, but how do you keep an eye on issues affecting the instance/version run for openstack usage? | 21:38 |
| AJaeger | so, is this a single case where people add a linter initially as non-voting and then migrate it to voting once it passes | 21:38 |
| tmcpeak | anteaya: security things can be harder to teach for developers | 21:38 |
| tmcpeak | so a project might want to run non-voting to educate their developers what things will not be done in the future | 21:38 |
| AJaeger | or is that a pattern - the pattern we use for introduction of new changes: First experimental, thne non-voting, then voting. | 21:38 |
| *** bharathm has quit IRC | 21:38 | |
| anteaya | electrofelix: I ask zaro personally, and he reads the gerrit dev output of bugs and patches | 21:39 |
| tmcpeak | generally though I'd expect this to be mostly voting | 21:39 |
| anteaya | so a security linter is in a lint category of its own? | 21:39 |
| fungi | yeah, having more developers from our community get deeply embedded in the developer communities of the tools on which we rely is the best model we have | 21:39 |
| anteaya | AJaeger: if it is a single case why are we adding a template? | 21:39 |
| AJaeger | anteaya: I made this change to have *one* target for all linters: pep8, bashate, bandit... | 21:40 |
| tmcpeak | no, it's pretty much the same. Let's say a project had a bunch of pep8 violating things. You might say: "ok guys, we're going to start enforcing this in a month. Meanwhile please look at the output to see what not to do" | 21:40 |
| fungi | regarding gerrit issue tracking impacting our user base | 21:40 |
| *** e0ne has quit IRC | 21:40 | |
| anteaya | AJaeger: yeah I see that, I just don't get the non-voting part | 21:40 |
| AJaeger | anteaya: I ask the question whether it's a single case, or the usual pattern... | 21:40 |
| tmcpeak | in this particular use of Bandit though, I'd expect people to go straight to voting | 21:40 |
| anteaya | lint or don't lint | 21:40 |
| anteaya | I don't see why non-voting is useful | 21:40 |
| tmcpeak | honestly it probably isn't | 21:40 |
| AJaeger | anteaya: that -nv change is not crucial for my patch series ;) It's not used anywhere, so I'm not pushing for it. | 21:41 |
| anteaya | tmcpeak: I don't say okay guys for anything, as I see zero benefit to excluding the women | 21:41 |
| tmcpeak | I'd be ok with voting only for the 'linters' target | 21:41 |
| AJaeger | I see your point, we can come back to this in case somebody has real arguments... | 21:42 |
| tmcpeak | anteaya: hmm, sorry, I didn't mean it offensively, I'm from California and we just use it as a a figure of speech. I say "ok guys" to groups that include women equally | 21:42 |
| AJaeger | and so far go with tmcpeak'S voting only for linters... | 21:42 |
| anteaya | okay well I have reviewed as much as makes sense to me | 21:42 |
| AJaeger | interesting gate-sqlalchemy-migrate-pep8 has voting false - as only job | 21:43 |
| AJaeger | which supports anteaya's view IMO. | 21:43 |
| AJaeger | anteaya: I'll abandon it, thanks | 21:43 |
| anteaya | clarkb: I know you are busy with other things but this logstash 2.0 change is in merge conflict and jenkins is -1 on it: https://review.openstack.org/#/c/247714/ | 21:43 |
| *** davidlenwell has quit IRC | 21:43 | |
| anteaya | clarkb: once you have finished your current thing | 21:44 |
| anteaya | AJaeger: thanks for understanding | 21:44 |
| AJaeger | thanks for reviewing and asking, anteaya ! | 21:44 |
| anteaya | AJaeger: glad I could elbow myself enough space where I could hold a thought in my head to do so | 21:45 |
| *** e0ne has joined #openstack-infra | 21:45 | |
| openstackgerrit | Kaitlin Farr proposed openstack/requirements: Cap PyKMIP version https://review.openstack.org/260599 | 21:46 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Initial sphynx doc setup for publishing https://review.openstack.org/259245 | 21:48 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add ci logs helper script https://review.openstack.org/237992 | 21:50 |
| *** apuimedo_ has quit IRC | 21:50 | |
| *** christiana_ has quit IRC | 21:51 | |
| *** electrofelix has quit IRC | 21:51 | |
| *** apuimedo_ has joined #openstack-infra | 21:51 | |
| *** adam_g has quit IRC | 21:52 | |
| *** bharathm has joined #openstack-infra | 21:54 | |
| *** davidlenwell has joined #openstack-infra | 21:55 | |
| *** adam_g has joined #openstack-infra | 21:56 | |
| *** adam_g has quit IRC | 21:56 | |
| *** adam_g has joined #openstack-infra | 21:56 | |
| *** apuimedo_ has quit IRC | 21:56 | |
| *** apuimedo_ has joined #openstack-infra | 21:57 | |
| *** changbl has joined #openstack-infra | 21:58 | |
| *** qadaemon has quit IRC | 21:59 | |
| *** e0ne has quit IRC | 21:59 | |
| *** markvoelker has joined #openstack-infra | 21:59 | |
| *** e0ne has joined #openstack-infra | 21:59 | |
| *** ociuhandu has quit IRC | 21:59 | |
| *** peristeri has quit IRC | 22:00 | |
| *** nelsnels_ has joined #openstack-infra | 22:00 | |
| *** piet has joined #openstack-infra | 22:00 | |
| *** nelsnelson has quit IRC | 22:00 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix growroot device detection https://review.openstack.org/259748 | 22:01 |
| openstackgerrit | Merged openstack/diskimage-builder: Package installs defaults to tracing off https://review.openstack.org/259717 | 22:02 |
| openstackgerrit | Merged openstack/diskimage-builder: Don't print trace unless trace is on in pkg-map https://review.openstack.org/259716 | 22:02 |
| *** davidlenwell has quit IRC | 22:02 | |
| *** markvoelker has quit IRC | 22:03 | |
| *** thorst_ has joined #openstack-infra | 22:04 | |
| *** adam_g has quit IRC | 22:06 | |
| *** apuimedo_ has quit IRC | 22:06 | |
| *** apuimedo_ has joined #openstack-infra | 22:07 | |
| tmcpeak | AJaeger: how are we looking on the original patch? | 22:07 |
| tmcpeak | that one seems to have pretty good support | 22:08 |
| *** xarses has quit IRC | 22:08 | |
| *** xarses has joined #openstack-infra | 22:08 | |
| *** thorst_ has quit IRC | 22:08 | |
| openstackgerrit | Kaitlin Farr proposed openstack/requirements: Cap PyKMIP version https://review.openstack.org/260599 | 22:09 |
| clarkb | anteaya: huh how did that merge conflict | 22:09 |
| clarkb | anteaya: I will check on it | 22:09 |
| anteaya | dunno | 22:09 |
| anteaya | clarkb: thanks | 22:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Alter config format to lists of dictionaries https://review.openstack.org/260727 | 22:09 |
| AJaeger | tmcpeak: yeah, 260192 needs another review... | 22:10 |
| tmcpeak | ahh ok cool | 22:10 |
| *** adam_g has joined #openstack-infra | 22:10 | |
| *** adam_g has quit IRC | 22:11 | |
| *** adam_g has joined #openstack-infra | 22:11 | |
| clarkb | anteaya: local git says no conflict so maybe just gerrit being gerrit? | 22:11 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Logstash 2.0 compat ES output rule https://review.openstack.org/247714 | 22:11 |
| clarkb | rebase pushed to fix it | 22:11 |
| jeblair | if anyone is curious https://review.openstack.org/260727 is most of the way to the zuul v3 configuration syntax, setting aside most of the job configuration for the moment | 22:11 |
| *** dprince has quit IRC | 22:12 | |
| jeblair | it's a good one to look at just to see the config files in the tests | 22:12 |
| *** davidlenwell has joined #openstack-infra | 22:12 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fix shade tests with OCC 1.13.0 https://review.openstack.org/260730 | 22:14 |
| *** piet has quit IRC | 22:14 | |
| Shrews | mordred: jeblair: ^^^ shade tests has a sad | 22:14 |
| *** AJaeger has quit IRC | 22:15 | |
| clarkb | I am looking at https://review.openstack.org/#/c/260704/3 and it doesn't show jenkins' +1 | 22:15 |
| clarkb | is that a known issue? | 22:15 |
| *** e0ne has quit IRC | 22:16 | |
| jeblair | clarkb: jenkins+1 appears in gertty | 22:16 |
| clarkb | jeblair: yes :) I think this is actually a bug in hideci.js though | 22:16 |
| clarkb | it seems to be replacing the vote table with the test results table | 22:17 |
| *** david-lyle has joined #openstack-infra | 22:17 | |
| anteaya | jeblair: thanks | 22:18 |
| *** Jay_ has joined #openstack-infra | 22:18 | |
| asselin__ | anyone curious to see 3rd party ci docs gerenated? http://docs-draft.openstack.org/45/259245/2/check/gate-puppet-openstackci-docs/b5086c2/doc/build/html/ | 22:18 |
| jeblair | clarkb: there's a vote table? | 22:19 |
| jeblair | i thought it just threw stuff into the reviewers area | 22:19 |
| jeblair | i hesitate to call that area a table | 22:19 |
| clarkb | sure it does something like that | 22:19 |
| clarkb | but its not there even though jenkins has +1'd | 22:19 |
| jeblair | oh yeah, there's supposed to be a table there | 22:20 |
| clarkb | ok its not just me and my browser | 22:20 |
| anteaya | a table would be an improvement | 22:21 |
| *** apuimedo_ has quit IRC | 22:21 | |
| anteaya | clarkb: I also do not see jenkins give a +1 in the spot I would expect it | 22:21 |
| anteaya | regardless of how we are defining that spot | 22:21 |
| *** Jay__ has quit IRC | 22:21 | |
| openstackgerrit | Merged openstack/diskimage-builder: Extend root device hints for different types of WWN https://review.openstack.org/245839 | 22:22 |
| anteaya | given the speed which which tourists are able to trigger their car alarms to go off after they arrive, I expect most suburban areas to be non-stop car alarm activity these days | 22:22 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config-example: Fix typo referencing '17', and provide an example randomizer https://review.openstack.org/260734 | 22:23 |
| *** apuimedo_ has joined #openstack-infra | 22:23 | |
| *** yamamoto has joined #openstack-infra | 22:24 | |
| *** tzn has joined #openstack-infra | 22:24 | |
| jeblair | i asked if the gerritforge folks had plans to make their .deb packages use online reindexing for their automatic upgrades... | 22:24 |
| jeblair | they said "you can propose a patch". | 22:25 |
| jeblair | <sigh> | 22:25 |
| *** tzn has quit IRC | 22:25 | |
| *** yamamoto has quit IRC | 22:25 | |
| clarkb | yolo packaging | 22:25 |
| clarkb | oh! | 22:25 |
| clarkb | its because I had patchset 3 selected not 4 | 22:26 |
| clarkb | this is the most confusing /grump | 22:26 |
| clarkb | I really like the old/gertty way of showing you a change not a patchset | 22:26 |
| clarkb | well that mystery is solved | 22:26 |
| jeblair | if the new way were really consistent, it wouldn't show you comments for ps4. | 22:26 |
| greghaynes | clarkb: this is how I was not able to find the rebase button | 22:26 |
| clarkb | jeblair: ++ | 22:26 |
| openstackgerrit | Kaitlin Farr proposed openstack/requirements: Cap PyKMIP version https://review.openstack.org/260599 | 22:26 |
| jeblair | so it's like, half the screen is from the past, and half is from the future. | 22:27 |
| greghaynes | yep | 22:27 |
| *** thingee has quit IRC | 22:27 | |
| anteaya | the twilight zone approach to code review | 22:27 |
| anteaya | like we needed something more to distract us | 22:28 |
| fungi | also worth noting, when leaving votes now if you have submit permission, there is no save and submit. the submit button now seems to only be available as a separate action from reviewing | 22:28 |
| fungi | discovered that last night when we had to bypass running job on a change to fix the parameter functions so that we could start running jobs again | 22:29 |
| fungi | confused me for a bit since i thought maybe we were missing a permission for the project-bootstrappers group in 2.11 acls | 22:30 |
| anteaya | :( | 22:31 |
| *** mriedem has quit IRC | 22:32 | |
| *** esker has joined #openstack-infra | 22:33 | |
| *** dkranz has quit IRC | 22:34 | |
| *** apuimedo_ has quit IRC | 22:34 | |
| *** apuimedo_ has joined #openstack-infra | 22:35 | |
| *** thingee has joined #openstack-infra | 22:35 | |
| *** xyang1 has quit IRC | 22:36 | |
| openstackgerrit | Joey D. proposed openstack/diskimage-builder: Fix discoverd bug when dmidecode reports GB https://review.openstack.org/258757 | 22:37 |
| *** apuimedo_ has quit IRC | 22:39 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config-example: Fix typo referencing '17', and provide an example randomizer https://review.openstack.org/260734 | 22:40 |
| *** apuimedo_ has joined #openstack-infra | 22:41 | |
| *** Qiming has joined #openstack-infra | 22:45 | |
| *** pahuang has joined #openstack-infra | 22:46 | |
| *** bharathm has quit IRC | 22:46 | |
| *** bharathm has joined #openstack-infra | 22:47 | |
| *** xarses has quit IRC | 22:47 | |
| *** Jay_ has quit IRC | 22:47 | |
| *** larainema has quit IRC | 22:49 | |
| *** Qiming has quit IRC | 22:50 | |
| *** apuimedo_ has quit IRC | 22:51 | |
| *** bharathm_ has joined #openstack-infra | 22:52 | |
| *** bharathm has quit IRC | 22:52 | |
| *** apuimedo_ has joined #openstack-infra | 22:52 | |
| *** mudassirlatif has quit IRC | 22:52 | |
| *** amuller has quit IRC | 22:53 | |
| jhesketh | Hey all | 22:55 |
| *** bharathm_ has quit IRC | 22:55 | |
| *** bharathm has joined #openstack-infra | 22:55 | |
| jhesketh | looks like the post pipeline issue was solved | 22:55 |
| jhesketh | what was the fix? | 22:55 |
| *** mfisch` is now known as mfisch | 22:55 | |
| *** mfisch is now known as Guest13696 | 22:56 | |
| *** apuimedo_ has quit IRC | 22:57 | |
| *** apuimedo_ has joined #openstack-infra | 22:59 | |
| fungi | jhesketh: zuul needed a restart to clear the defunct precise label registrations | 22:59 |
| *** apoorvad has quit IRC | 22:59 | |
| jhesketh | yeah just caught up on that piece of scrollback | 22:59 |
| jhesketh | and it looks like the post pipeline grew because gerrit was being slow? | 23:00 |
| fungi | since zuul was requesting unqualified jobs and nodepool was just picking the first label it saw registered for those jobs to determine demand, which ~half of the time was precise | 23:00 |
| *** markvoelker has joined #openstack-infra | 23:00 | |
| fungi | oh, yes also that, gerrit was apparently (as best as we can tel after the fact) blocked fulfilling http requests | 23:00 |
| fungi | and that slowed zuul's reporting to a crawl, so there were lots of completed jobs it wasn't clearing from the pipeline | 23:01 |
| fungi | s | 23:01 |
| fungi | (theory) | 23:01 |
| jhesketh | ah right | 23:01 |
| jhesketh | nice find | 23:01 |
| jhesketh | fungi: has the zuul restart been done yet? | 23:01 |
| fungi | as soon as gerrit was restarted, zuul quickly burned down the results backlog | 23:02 |
| fungi | jhesketh: yeah, i restarted zuul shortly after we determined where the precise node surplus was originating | 23:02 |
| fungi | and then i deleted the remnants to make room for more productive node types | 23:02 |
| jhesketh | cool | 23:02 |
| jhesketh | the restart was smooth? | 23:02 |
| fungi | completely | 23:02 |
| jhesketh | is this the first restart since connections? | 23:02 |
| fungi | yeah, also even rare node types reregistered in gearman very quickly after the zuul restart | 23:03 |
| *** bharathm_ has joined #openstack-infra | 23:03 | |
| fungi | basically as soon as i checked they were already there | 23:03 |
| fungi | so much faster than i was used to previously... not sure what we've changed that might alter that | 23:03 |
| *** bharathm has quit IRC | 23:04 | |
| jhesketh | possibly just a smaller gearman? | 23:04 |
| jhesketh | I don't think gearman cleans things up much, so a restart might have been appreciated by it | 23:05 |
| *** markvoelker has quit IRC | 23:05 | |
| fungi | anyway, thanks for keeping tabs on that stuff last night. much less impact that i feared | 23:05 |
| jhesketh | any time :-) | 23:05 |
| jhesketh | sorry I failed to find the gerrit performance | 23:06 |
| fungi | really the biggest scramble of the morning was whatever mysterious malady gerrit came down with, which should be entirely unrelated to that change | 23:06 |
| jhesketh | indeed | 23:06 |
| fungi | hopefully next time it happens we can catch it for a little longer, check database queues, process list within the jvm, et cetera | 23:07 |
| fungi | also puppetboard ran out of disk, so that wasn't helping things (made us slightly more blind to what might be going on network-wide) | 23:07 |
| clarkb | it wasn't the normal garbage collection spins cpu leak fun? | 23:07 |
| *** apuimedo_ has quit IRC | 23:08 | |
| fungi | clarkb: i don't know. i was focused on zuul logs at the time thinking we were looking at two unrelated issues, so mordred was checking on gerrit | 23:08 |
| fungi | i briefly saw apache stats from it, which indicated it had a lot of sockets active writing | 23:09 |
| fungi | not sure what else he looked at | 23:09 |
| *** apuimedo_ has joined #openstack-infra | 23:10 | |
| *** bradjones has quit IRC | 23:10 | |
| *** larainema has joined #openstack-infra | 23:11 | |
| *** bradjones has joined #openstack-infra | 23:13 | |
| *** bradjones has quit IRC | 23:13 | |
| *** bradjones has joined #openstack-infra | 23:13 | |
| *** apuimedo_ has quit IRC | 23:16 | |
| *** apuimedo_ has joined #openstack-infra | 23:18 | |
| *** dims has joined #openstack-infra | 23:18 | |
| *** dims_ has quit IRC | 23:19 | |
| *** mudassirlatif has joined #openstack-infra | 23:21 | |
| *** rbrndt has quit IRC | 23:21 | |
| *** apoorvad has joined #openstack-infra | 23:21 | |
| *** apuimedo_ has quit IRC | 23:22 | |
| *** apuimedo_ has joined #openstack-infra | 23:23 | |
| *** harlowja has quit IRC | 23:25 | |
| *** harlowja has joined #openstack-infra | 23:25 | |
| *** zeih has quit IRC | 23:25 | |
| *** apuimedo_ has quit IRC | 23:28 | |
| *** apuimedo_ has joined #openstack-infra | 23:29 | |
| *** triggerwarning has quit IRC | 23:30 | |
| *** triggerwarning has joined #openstack-infra | 23:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Remove boilerplate zuul.conf from test fixtures https://review.openstack.org/260749 | 23:32 |
| *** FallenPegasus has joined #openstack-infra | 23:33 | |
| *** apuimedo_ has quit IRC | 23:33 | |
| *** apuimedo_ has joined #openstack-infra | 23:33 | |
| *** rbrndt_ has joined #openstack-infra | 23:34 | |
| *** sputnik1_ has joined #openstack-infra | 23:35 | |
| *** ayoung has joined #openstack-infra | 23:35 | |
| *** apuimedo_ has quit IRC | 23:38 | |
| *** apuimedo_ has joined #openstack-infra | 23:40 | |
| *** apuimedo_ has quit IRC | 23:44 | |
| *** apuimedo_ has joined #openstack-infra | 23:46 | |
| *** salv-orlando has joined #openstack-infra | 23:47 | |
| *** jyuso1 has joined #openstack-infra | 23:48 | |
| *** ChuckC has joined #openstack-infra | 23:49 | |
| *** apuimedo_ has quit IRC | 23:50 | |
| *** apuimedo_ has joined #openstack-infra | 23:51 | |
| *** jyuso1 has quit IRC | 23:54 | |
| *** Verilium has joined #openstack-infra | 23:54 | |
| *** apuimedo_ has quit IRC | 23:56 | |
| *** apuimedo_ has joined #openstack-infra | 23:56 | |
| *** bharathm_ has quit IRC | 23:58 | |
| *** hdd has quit IRC | 23:58 | |
| *** dahoo has joined #openstack-infra | 23:58 | |
| *** rlandy has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!