| *** openstack has joined #openstack-infra | 05:58 | |
| *** amitgandhinz has joined #openstack-infra | 05:59 | |
| *** tkelsey has joined #openstack-infra | 06:00 | |
| *** jyuso1 has quit IRC | 06:00 | |
| *** jyuso1 has joined #openstack-infra | 06:01 | |
| *** alex_xu has quit IRC | 06:03 | |
| *** kzaitsev_mb has quit IRC | 06:03 | |
| *** amitgandhinz has quit IRC | 06:03 | |
| *** tkelsey has quit IRC | 06:05 | |
| *** alex_xu has joined #openstack-infra | 06:06 | |
| *** andreas_s has joined #openstack-infra | 06:16 | |
| *** javeriak has joined #openstack-infra | 06:17 | |
| *** sarob has joined #openstack-infra | 06:18 | |
| *** yolanda has joined #openstack-infra | 06:18 | |
| *** yolanda has quit IRC | 06:21 | |
| *** ramishra has quit IRC | 06:21 | |
| *** sarob has quit IRC | 06:23 | |
| *** yolanda has joined #openstack-infra | 06:24 | |
| *** rcernin has joined #openstack-infra | 06:24 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/system-config: OpenStack-CI Apt mirror for Ubuntu Cloud Archive https://review.openstack.org/330152 | 06:25 |
|---|---|---|
| yolanda | good morning | 06:26 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/330335 | 06:26 |
| AJaeger | good morning, yolanda ! | 06:26 |
| *** ramishra has joined #openstack-infra | 06:26 | |
| *** javeriak has quit IRC | 06:31 | |
| *** _nadya_ has quit IRC | 06:31 | |
| *** javeriak has joined #openstack-infra | 06:32 | |
| AJaeger | yolanda: rax-ord still has old images, do you have a chance to manually upload the current ubuntu-trusty image to it, please? | 06:33 |
| yolanda | AJaeger, sure | 06:34 |
| *** jcoufal has quit IRC | 06:36 | |
| *** Daisy_ has joined #openstack-infra | 06:36 | |
| AJaeger | thanks a lot! I have a failing job due to old jenkins/scripts on that image, and rax-ord seems to be the only provider without that change now. | 06:37 |
| *** Daisy has quit IRC | 06:39 | |
| *** sree_ has joined #openstack-infra | 06:40 | |
| *** sree_ is now known as Guest21396 | 06:41 | |
| yolanda | image uploading | 06:41 |
| *** aeng has quit IRC | 06:42 | |
| *** sree has quit IRC | 06:43 | |
| *** thorst_ has joined #openstack-infra | 06:44 | |
| *** hichihara has joined #openstack-infra | 06:45 | |
| *** Daisy has joined #openstack-infra | 06:49 | |
| *** salv-orlando has quit IRC | 06:49 | |
| *** salv-orlando has joined #openstack-infra | 06:50 | |
| *** Sukhdev has quit IRC | 06:51 | |
| *** Daisy_ has quit IRC | 06:51 | |
| *** afazekas is now known as afazekas|dentist | 06:52 | |
| *** javeriak has quit IRC | 06:52 | |
| *** thorst_ has quit IRC | 06:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/330335 | 06:53 |
| *** salv-orl_ has joined #openstack-infra | 06:55 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add specifications repository for charms https://review.openstack.org/330054 | 06:56 |
| *** salv-orlando has quit IRC | 06:57 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:59 | |
| *** _nadya_ has joined #openstack-infra | 06:59 | |
| *** camunoz has quit IRC | 06:59 | |
| *** Daisy_ has joined #openstack-infra | 06:59 | |
| *** ramishra has quit IRC | 07:00 | |
| *** amitgandhinz has joined #openstack-infra | 07:00 | |
| *** Daisy has quit IRC | 07:00 | |
| *** markvoelker_ has quit IRC | 07:01 | |
| *** javeriak has joined #openstack-infra | 07:01 | |
| *** javeriak has joined #openstack-infra | 07:01 | |
| *** markvoelker has joined #openstack-infra | 07:01 | |
| *** kzaitsev_mb has quit IRC | 07:03 | |
| *** Douhet has quit IRC | 07:04 | |
| *** Daisy_ has quit IRC | 07:04 | |
| *** amitgandhinz has quit IRC | 07:04 | |
| *** mikelk has joined #openstack-infra | 07:05 | |
| *** amoralej|off is now known as amoralej | 07:05 | |
| *** ramishra has joined #openstack-infra | 07:05 | |
| *** sorantis has joined #openstack-infra | 07:05 | |
| *** hdaniel has joined #openstack-infra | 07:06 | |
| *** tesseract has joined #openstack-infra | 07:09 | |
| *** esikachev has joined #openstack-infra | 07:10 | |
| *** tristanC has joined #openstack-infra | 07:12 | |
| *** jamielennox is now known as jamielennox|away | 07:12 | |
| *** jed56 has joined #openstack-infra | 07:16 | |
| *** zeih has joined #openstack-infra | 07:16 | |
| *** pcaruana has joined #openstack-infra | 07:17 | |
| *** vincentll has joined #openstack-infra | 07:18 | |
| *** permalac has quit IRC | 07:19 | |
| *** ifarkas has joined #openstack-infra | 07:19 | |
| *** ramishra has quit IRC | 07:19 | |
| *** ramishra has joined #openstack-infra | 07:19 | |
| *** sarob has joined #openstack-infra | 07:20 | |
| *** matrohon has joined #openstack-infra | 07:20 | |
| *** yaume has joined #openstack-infra | 07:22 | |
| openstackgerrit | watanabe isao proposed openstack-infra/zuul: WIP: Improve zuul proxy feature https://review.openstack.org/240131 | 07:25 |
| *** sarob has quit IRC | 07:25 | |
| *** ramishra has quit IRC | 07:26 | |
| *** roxanaghe has joined #openstack-infra | 07:27 | |
| *** ramishra has joined #openstack-infra | 07:28 | |
| *** arxcruz has joined #openstack-infra | 07:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Add specifications repository for charms https://review.openstack.org/330054 | 07:29 |
| *** shardy has joined #openstack-infra | 07:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Add stable branch jobs for tripleo-ci https://review.openstack.org/297390 | 07:31 |
| *** roxanaghe has quit IRC | 07:32 | |
| *** ramishra has quit IRC | 07:32 | |
| *** abregman has quit IRC | 07:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Use https to transfer builds https://review.openstack.org/330325 | 07:32 |
| *** abregman has joined #openstack-infra | 07:33 | |
| *** reduxio has quit IRC | 07:34 | |
| *** ccamacho has joined #openstack-infra | 07:35 | |
| *** jlanoux has joined #openstack-infra | 07:36 | |
| *** ramishra has joined #openstack-infra | 07:37 | |
| *** Guest21396 has quit IRC | 07:37 | |
| *** dizquierdo has joined #openstack-infra | 07:38 | |
| *** praneshp has quit IRC | 07:39 | |
| *** jlanoux_ has joined #openstack-infra | 07:41 | |
| *** jlanoux has quit IRC | 07:41 | |
| openstackgerrit | Merged openstack-infra/project-config: Added new repo for fuel-plugin-datera-cinder https://review.openstack.org/315651 | 07:41 |
| *** acabot has joined #openstack-infra | 07:42 | |
| openstackgerrit | Merged openstack-infra/project-config: Shorten the name of the kolla-kubernetes gates https://review.openstack.org/330123 | 07:44 |
| openstackgerrit | Merged openstack-infra/project-config: Enable python3 jobs for git-upstream https://review.openstack.org/329475 | 07:44 |
| *** ramishra has quit IRC | 07:46 | |
| *** ramishra has joined #openstack-infra | 07:46 | |
| *** salv-orl_ has quit IRC | 07:49 | |
| *** thorst_ has joined #openstack-infra | 07:49 | |
| *** dtardivel has joined #openstack-infra | 07:49 | |
| *** salv-orlando has joined #openstack-infra | 07:49 | |
| *** samP has joined #openstack-infra | 07:49 | |
| AJaeger | yolanda: how long does the upload take? | 07:49 |
| yolanda | it's still on progress | 07:49 |
| AJaeger | ok | 07:50 |
| *** dtantsur|afk is now known as dtantsur | 07:50 | |
| *** Douhet has joined #openstack-infra | 07:51 | |
| *** sree has joined #openstack-infra | 07:52 | |
| *** mhickey has joined #openstack-infra | 07:53 | |
| *** samuelBartel has joined #openstack-infra | 07:53 | |
| *** thorst_ has quit IRC | 07:56 | |
| *** _nadya_ has quit IRC | 07:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Updated two projects in gerrit/projects.yaml https://review.openstack.org/329943 | 07:57 |
| *** hashar has joined #openstack-infra | 07:58 | |
| *** sarob has joined #openstack-infra | 07:59 | |
| *** ggnel_t has joined #openstack-infra | 07:59 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** amitgandhinz has joined #openstack-infra | 08:00 | |
| *** acoles_ is now known as acoles | 08:01 | |
| esikachev | AJaeger: hi! question about https://review.openstack.org/#/c/329497/7/jenkins/jobs/sahara.yaml | 08:01 |
| AJaeger | esikachev: yes? | 08:02 |
| openstackgerrit | Merged openstack-infra/project-config: Add jobs for Monasca-Analytics https://review.openstack.org/329376 | 08:02 |
| *** tkelsey has joined #openstack-infra | 08:02 | |
| esikachev | Why need remove the project name from job name? | 08:03 |
| AJaeger | esikachev: why do you need it there at all? | 08:03 |
| AJaeger | esikachev: for those kind of integration jobs, we normally use one job and then add it to both. It's the same job, isn't it? Where do you differentiate them? | 08:04 |
| esikachev | you mean unused var? | 08:04 |
| *** sarob has quit IRC | 08:04 | |
| AJaeger | YEs. | 08:04 |
| *** kzaitsev_mb has quit IRC | 08:04 | |
| esikachev | got it | 08:04 |
| AJaeger | esikachev: you might need to enhance the PROJECTS var to checkout the proper projects. | 08:05 |
| *** hichihara has quit IRC | 08:05 | |
| AJaeger | esikachev: REview the change some more - perhaps you should use the variable ;) | 08:05 |
| *** sflanigan has quit IRC | 08:05 | |
| *** amitgandhinz has quit IRC | 08:06 | |
| *** jaosorior has quit IRC | 08:07 | |
| *** binbincong has quit IRC | 08:07 | |
| *** jaosorior has joined #openstack-infra | 08:08 | |
| openstackgerrit | Illia Khudoshyn proposed openstack-infra/project-config: Add Rally job against deployment with existing users https://review.openstack.org/328249 | 08:09 |
| *** permalac has joined #openstack-infra | 08:09 | |
| *** mhickey has quit IRC | 08:11 | |
| *** dizquierdo has quit IRC | 08:11 | |
| *** chem```` has joined #openstack-infra | 08:12 | |
| *** e0ne has joined #openstack-infra | 08:13 | |
| *** chem``` has quit IRC | 08:14 | |
| *** zeih has quit IRC | 08:14 | |
| *** zeih has joined #openstack-infra | 08:15 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add Rally job against deployment with existing users https://review.openstack.org/328249 | 08:15 |
| *** Julien-zte has joined #openstack-infra | 08:15 | |
| openstackgerrit | Illia Khudoshyn proposed openstack-infra/project-config: Add Rally job against deployment with existing users https://review.openstack.org/328249 | 08:17 |
| *** Daisy has joined #openstack-infra | 08:17 | |
| *** Daisy_ has joined #openstack-infra | 08:18 | |
| *** Daisy has quit IRC | 08:18 | |
| openstackgerrit | Illia Khudoshyn proposed openstack-infra/project-config: Add Rally job against deployment with existing users https://review.openstack.org/328249 | 08:19 |
| *** binbincong has joined #openstack-infra | 08:19 | |
| *** zeih has quit IRC | 08:19 | |
| *** abregman has quit IRC | 08:19 | |
| *** chem```` is now known as chem | 08:20 | |
| ikhudoshyn | AJaeger: https://review.openstack.org/#/c/328249 <-- fixed that. sry for inconvenience. sometimes i might get dumber in mornings | 08:21 |
| AJaeger | ikhudoshyn: I put it to WIP to fix it myself, why did you push one nevertheless? Patchset 5 is fine, I commented on 7 - and will not touch it again | 08:22 |
| * AJaeger sends some fresh coffee to ikhudoshyn ;) | 08:22 | |
| ikhudoshyn | AJaeger: thanks | 08:24 |
| * ikhudoshyn reaches for 'patience' potion | 08:25 | |
| yolanda | AJaeger, image finally has been uploaded | 08:25 |
| *** mhickey has joined #openstack-infra | 08:26 | |
| AJaeger | yolanda: yeah, thanks! | 08:26 |
| ikhudoshyn | AJaeger: I actually first thought to fix it in the right place yet gerrit's colored diff confused me | 08:27 |
| AJaeger | ;) | 08:28 |
| *** lucas-afk is now known as lucasagomes | 08:28 | |
| AJaeger | ikhudoshyn: do you understand my most recend comment? | 08:28 |
| ikhudoshyn | yep, I won't do myself anything, just keep waiting | 08:29 |
| ikhudoshyn | right? | 08:30 |
| *** BobBall_AWOL is now known as BobBall | 08:30 | |
| *** zeih has joined #openstack-infra | 08:31 | |
| openstackgerrit | Chaoyi Huang proposed openstack-infra/project-config: Add the Tricircle functional test job https://review.openstack.org/329740 | 08:31 |
| *** abregman has joined #openstack-infra | 08:32 | |
| AJaeger | ikhudoshyn: ok, I'll update in a few minutes... | 08:33 |
| notmorgan | lifeless: so, looks like io.FileIO raises the wrong error, instead of an IOError, it raises ValueError when you perform .read(0) on a stream only open for writing. I think we need to add ValueError to the exception here: https://github.com/testing-cabal/subunit/blob/master/python/subunit/__init__.py#L1298 - there was a bug with python that some cases raised ValueError, looks like io.FileIO has | 08:33 |
| notmorgan | mimiced this error case. | 08:33 |
| notmorgan | lifeless: i'll look at spinning up a patch and a PR to confirm this. | 08:33 |
| notmorgan | (with tests) | 08:33 |
| ikhudoshyn | AJaeger: oh, see, coffee helped, i'm getting smarter back | 08:33 |
| *** Daisy_ has quit IRC | 08:34 | |
| *** savihou has joined #openstack-infra | 08:34 | |
| notmorgan | lifeless: fwiw, py3 does the right thing | 08:34 |
| *** Daisy has joined #openstack-infra | 08:34 | |
| *** jordanP has joined #openstack-infra | 08:34 | |
| *** mhickey has quit IRC | 08:34 | |
| *** dingyichen has quit IRC | 08:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add Rally job against deployment with existing users https://review.openstack.org/328249 | 08:36 |
| *** Daisy has quit IRC | 08:36 | |
| *** Daisy has joined #openstack-infra | 08:36 | |
| AJaeger | ikhudoshyn: looks now fine to me ;) | 08:36 |
| openstackgerrit | Sampath Priyankara proposed openstack-infra/project-config: Request to add new project masakari https://review.openstack.org/330370 | 08:37 |
| ikhudoshyn | AJaeger: great, thanks | 08:37 |
| *** hferenc has quit IRC | 08:38 | |
| samP | Hi! | 08:38 |
| AJaeger | infra-root, panko has a job in the post queue and one in periodic - and I don't see those moving forward. Panko has not merged any change yet - is that confusing our CI system? | 08:40 |
| *** sree has quit IRC | 08:42 | |
| samP | Hi!, I (Sampath Priyankara) have just push a patch for request to add new project "masakari". According to the doc, I must wait here.. should I send a mail to openstack-infra ML as well? | 08:42 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add infrstucture scripts to prepare rh2 https://review.openstack.org/295243 | 08:46 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI https://review.openstack.org/329521 | 08:46 |
| *** derekh has joined #openstack-infra | 08:47 | |
| *** hferenc has joined #openstack-infra | 08:47 | |
| AJaeger | samP: please have patience - wait can be a couple of days. We will review it in due time... | 08:48 |
| *** sree has joined #openstack-infra | 08:48 | |
| AJaeger | samP: Looking at your change: Please add a paragraph explaining what masakari his and why it should be hosted in the OpenStack infrastructure to your change. | 08:48 |
| lennyb | Hi, Infra. is there a way not to upgrade pip during ./stack.sh or request a specific pip version? I am getting error in pip 1.8.2 ( known issue ) when checking LIBS_FROM_GIT param | 08:49 |
| *** sree has quit IRC | 08:49 | |
| *** sree has joined #openstack-infra | 08:49 | |
| samP | Ajaeger: sure, thank you. That explanation in commit msg? | 08:50 |
| AJaeger | samP: yes | 08:50 |
| samP | AJaeger: Sure, I will add. Thank you again. | 08:51 |
| *** kdas__ has joined #openstack-infra | 08:51 | |
| *** mescanef has joined #openstack-infra | 08:52 | |
| *** Daisy has quit IRC | 08:53 | |
| *** Daisy has joined #openstack-infra | 08:53 | |
| *** mdbooth_ is now known as mdbooth | 08:53 | |
| *** thorst_ has joined #openstack-infra | 08:54 | |
| *** salv-orlando has quit IRC | 08:55 | |
| *** salv-orlando has joined #openstack-infra | 08:56 | |
| *** dizquierdo has joined #openstack-infra | 08:56 | |
| *** Daisy_ has joined #openstack-infra | 08:56 | |
| AJaeger | yolanda: thanks for uploading, my test finally passed! | 08:57 |
| yolanda | nice! | 08:57 |
| *** Daisy has quit IRC | 08:58 | |
| *** Daisy has joined #openstack-infra | 08:58 | |
| *** Daisy has quit IRC | 08:58 | |
| *** Daisy has joined #openstack-infra | 08:58 | |
| *** roxanaghe has joined #openstack-infra | 09:00 | |
| *** dtantsur is now known as dtantsur|brb | 09:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:00 | |
| *** Daisy_ has quit IRC | 09:01 | |
| *** amotoki has quit IRC | 09:01 | |
| *** thorst_ has quit IRC | 09:01 | |
| *** amitgandhinz has joined #openstack-infra | 09:01 | |
| *** andreykurilin has joined #openstack-infra | 09:02 | |
| *** dmk0202 has joined #openstack-infra | 09:02 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute031.uswest metadata for better tracking https://review.openstack.org/330390 | 09:03 |
| *** kzaitsev_mb has quit IRC | 09:05 | |
| *** amitgandhinz has quit IRC | 09:06 | |
| *** nijaba has quit IRC | 09:06 | |
| *** roxanaghe has quit IRC | 09:06 | |
| *** ifarkas has quit IRC | 09:06 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add python logger configuration https://review.openstack.org/325409 | 09:07 |
| openstackgerrit | Merged openstack/diskimage-builder: Introspect logging testing more https://review.openstack.org/328071 | 09:08 |
| *** eezhova has joined #openstack-infra | 09:09 | |
| openstackgerrit | Alina Nesterova proposed openstack-infra/project-config: Migrated sahara-client-functional-job https://review.openstack.org/329497 | 09:09 |
| *** _nadya_ has joined #openstack-infra | 09:10 | |
| openstackgerrit | Elena Ezhova proposed openstack-infra/project-config: Add a functional job for neutron-lbaas https://review.openstack.org/322812 | 09:10 |
| esikachev | AJaeger: take a look, is correct? https://review.openstack.org/#/c/329497/ | 09:11 |
| openstackgerrit | Tomoki Sekiyama proposed openstack-infra/project-config: Create new functional job for glance_store-cinder https://review.openstack.org/329827 | 09:12 |
| *** nijaba has joined #openstack-infra | 09:13 | |
| *** kdas__ is now known as kushal | 09:14 | |
| *** kushal has joined #openstack-infra | 09:15 | |
| *** sree has quit IRC | 09:16 | |
| openstackgerrit | Sampath Priyankara proposed openstack-infra/project-config: Request to add new project Masakari https://review.openstack.org/330370 | 09:19 |
| *** eezhova has quit IRC | 09:19 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:20 | |
| *** Na3iL has joined #openstack-infra | 09:20 | |
| *** eezhova has joined #openstack-infra | 09:20 | |
| *** zeih has quit IRC | 09:22 | |
| *** ihrachys has joined #openstack-infra | 09:26 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute088.useast metadata for better tracking https://review.openstack.org/330401 | 09:26 |
| *** electrofelix has joined #openstack-infra | 09:27 | |
| *** mkoderer__ has quit IRC | 09:28 | |
| *** javeriak has quit IRC | 09:28 | |
| *** esikachev has quit IRC | 09:29 | |
| openstackgerrit | Tom Fifield proposed openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 09:30 |
| *** markusry has joined #openstack-infra | 09:31 | |
| openstackgerrit | Sampath Priyankara proposed openstack-infra/project-config: Request to add new project Masakari https://review.openstack.org/330370 | 09:32 |
| *** Daisy has quit IRC | 09:32 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute016.useast metadata for better tracking https://review.openstack.org/330407 | 09:33 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute089.useast metadata for better tracking https://review.openstack.org/330409 | 09:36 |
| *** zeih has joined #openstack-infra | 09:36 | |
| *** tosky has joined #openstack-infra | 09:36 | |
| *** ociuhandu has quit IRC | 09:39 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute086.useast metadata for better tracking https://review.openstack.org/330412 | 09:39 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Remove pagination for timeline events https://review.openstack.org/327682 | 09:40 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Rework timeline events into a timeline https://review.openstack.org/327231 | 09:40 |
| *** HeOS_ has joined #openstack-infra | 09:43 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute085.useast metadata for better tracking https://review.openstack.org/330416 | 09:43 |
| *** samP has quit IRC | 09:45 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute087.useast metadata for better tracking https://review.openstack.org/330418 | 09:45 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Start using pre-prov creds in dsvm-layer4 https://review.openstack.org/329723 | 09:47 |
| *** esikachev has joined #openstack-infra | 09:47 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute084.useast metadata for better tracking https://review.openstack.org/330420 | 09:48 |
| *** javeriak has joined #openstack-infra | 09:50 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute014.useast metadata for better tracking https://review.openstack.org/330421 | 09:51 |
| *** kzaitsev_mb has quit IRC | 09:54 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute013.useast metadata for better tracking https://review.openstack.org/330422 | 09:54 |
| *** bunting_ is now known as bunting | 09:55 | |
| *** samuelBartel has quit IRC | 09:56 | |
| *** ifarkas has joined #openstack-infra | 09:57 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute012.useast metadata for better tracking https://review.openstack.org/330423 | 09:57 |
| *** wfoster_ is now known as wfoster | 09:57 | |
| *** tosky has quit IRC | 09:59 | |
| *** thorst_ has joined #openstack-infra | 09:59 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute015.useast metadata for better tracking https://review.openstack.org/330424 | 09:59 |
| wznoinsk | anteaya are you around? | 10:00 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute010.useast metadata for better tracking https://review.openstack.org/330426 | 10:02 |
| *** amitgandhinz has joined #openstack-infra | 10:02 | |
| *** ajmiller_ has joined #openstack-infra | 10:03 | |
| *** ajmiller has quit IRC | 10:03 | |
| *** kushal has quit IRC | 10:04 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute009.useast metadata for better tracking https://review.openstack.org/330430 | 10:04 |
| *** lvdongbing has quit IRC | 10:04 | |
| *** abregman has quit IRC | 10:05 | |
| *** thorst_ has quit IRC | 10:06 | |
| *** tosky has joined #openstack-infra | 10:06 | |
| *** ajmiller_ has quit IRC | 10:06 | |
| *** ajmiller has joined #openstack-infra | 10:06 | |
| *** amitgandhinz has quit IRC | 10:06 | |
| *** salv-orlando has quit IRC | 10:09 | |
| *** sambetts|afk is now known as sambetts | 10:09 | |
| *** zqfan has joined #openstack-infra | 10:10 | |
| *** _degorenko|afk is now known as degorenko | 10:11 | |
| *** yanyanhu has quit IRC | 10:11 | |
| *** dtantsur|brb is now known as dtantsur | 10:13 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute011.useast metadata for better tracking https://review.openstack.org/330435 | 10:13 |
| *** zhurong has quit IRC | 10:14 | |
| *** sarob has joined #openstack-infra | 10:14 | |
| *** jamielennox|away is now known as jamielennox | 10:15 | |
| *** javeriak has quit IRC | 10:16 | |
| openstackgerrit | Alina Nesterova proposed openstack-infra/project-config: Migrated sahara-client-functional-job https://review.openstack.org/329497 | 10:18 |
| *** abregman has joined #openstack-infra | 10:18 | |
| *** sarob has quit IRC | 10:19 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute017.useast metadata for better tracking https://review.openstack.org/330437 | 10:19 |
| *** javeriak has joined #openstack-infra | 10:19 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute019.useast metadata for better tracking https://review.openstack.org/330440 | 10:23 |
| *** sorantis has quit IRC | 10:23 | |
| *** ociuhandu has joined #openstack-infra | 10:25 | |
| *** sirushti has quit IRC | 10:26 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute020.useast metadata for better tracking https://review.openstack.org/330441 | 10:26 |
| *** sorantis has joined #openstack-infra | 10:26 | |
| *** sree has joined #openstack-infra | 10:29 | |
| *** jtomasek_ has joined #openstack-infra | 10:30 | |
| *** mikelk has quit IRC | 10:32 | |
| *** eezhova has quit IRC | 10:32 | |
| *** sirushti has joined #openstack-infra | 10:33 | |
| *** kushal has joined #openstack-infra | 10:34 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute026.useast metadata for better tracking https://review.openstack.org/330445 | 10:35 |
| *** abregman has quit IRC | 10:38 | |
| openstackgerrit | Jiri Tyr proposed openstack-infra/jenkins-job-builder: Adding property-file into the pipeline publisher https://review.openstack.org/311150 | 10:40 |
| *** sdake has joined #openstack-infra | 10:40 | |
| *** e0ne has quit IRC | 10:41 | |
| *** e0ne has joined #openstack-infra | 10:41 | |
| *** yamamoto has quit IRC | 10:42 | |
| *** sdake_ has joined #openstack-infra | 10:42 | |
| *** esikachev has quit IRC | 10:42 | |
| openstackgerrit | Tom Fifield proposed openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 10:43 |
| *** sdake has quit IRC | 10:45 | |
| *** jtyr has joined #openstack-infra | 10:49 | |
| jtyr | Please could somebody review this very useful patch for Jenkins Job Builder? ... https://review.openstack.org/#/c/311150/ | 10:49 |
| *** esikachev has joined #openstack-infra | 10:54 | |
| *** ccamacho is now known as ccamacho|lunch | 10:56 | |
| *** HeOS has quit IRC | 10:57 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:57 | |
| *** HeOS has joined #openstack-infra | 10:57 | |
| *** jtomasek_ has quit IRC | 10:59 | |
| *** eezhova has joined #openstack-infra | 10:59 | |
| *** kzaitsev_mb has quit IRC | 11:02 | |
| *** YorikSar has quit IRC | 11:02 | |
| *** roxanaghe has joined #openstack-infra | 11:02 | |
| *** amitgandhinz has joined #openstack-infra | 11:03 | |
| *** thorst_ has joined #openstack-infra | 11:03 | |
| *** YorikSar has joined #openstack-infra | 11:04 | |
| *** javeriak has quit IRC | 11:06 | |
| *** roxanaghe has quit IRC | 11:07 | |
| *** salv-orlando has joined #openstack-infra | 11:07 | |
| *** amitgandhinz has quit IRC | 11:07 | |
| *** salv-orlando has quit IRC | 11:10 | |
| *** sdake_ has quit IRC | 11:10 | |
| *** ramishra has quit IRC | 11:10 | |
| *** salv-orlando has joined #openstack-infra | 11:11 | |
| *** thorst_ has quit IRC | 11:11 | |
| *** ramishra has joined #openstack-infra | 11:16 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Enable Tempest test job also for sahara-tests https://review.openstack.org/329590 | 11:17 |
| *** yamamoto has joined #openstack-infra | 11:22 | |
| *** esikachev has quit IRC | 11:26 | |
| *** YorikSar has quit IRC | 11:31 | |
| *** lucasagomes is now known as lucas-hungry | 11:31 | |
| *** abregman has joined #openstack-infra | 11:31 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add the ability to permalink to a comment https://review.openstack.org/289078 | 11:32 |
| *** jed56 has quit IRC | 11:34 | |
| *** kaisers1 has joined #openstack-infra | 11:34 | |
| *** jed56 has joined #openstack-infra | 11:34 | |
| *** e0ne has quit IRC | 11:35 | |
| *** weshay has joined #openstack-infra | 11:37 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/tripleo-ci: Update Fedora Atomic URL https://review.openstack.org/330477 | 11:38 |
| openstackgerrit | Merged openstack-infra/project-config: setup release notes job for sahara-dashboard https://review.openstack.org/327729 | 11:42 |
| lifeless | notmorgan: cut out at 'looks like io.FileIO has' | 11:43 |
| lifeless | notmorgan: saw your patch, would like a test, which needs to say which Python its relevant to so we know when we can drop it | 11:43 |
| *** kaisers1 has quit IRC | 11:44 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Make task notes button clearer https://review.openstack.org/325292 | 11:44 |
| *** salv-orlando has quit IRC | 11:45 | |
| *** kaisers1 has joined #openstack-infra | 11:45 | |
| *** salv-orlando has joined #openstack-infra | 11:46 | |
| *** ddieterly has joined #openstack-infra | 11:47 | |
| *** andreykurilin has quit IRC | 11:47 | |
| *** esikachev has joined #openstack-infra | 11:49 | |
| *** Julien-zte has quit IRC | 11:49 | |
| *** ramishra has quit IRC | 11:52 | |
| *** YorikSar has joined #openstack-infra | 11:52 | |
| *** thorst_ has joined #openstack-infra | 11:53 | |
| *** dprince has joined #openstack-infra | 11:54 | |
| *** claudiub has joined #openstack-infra | 11:55 | |
| *** thorst__ has joined #openstack-infra | 11:55 | |
| *** TheJulia_ is now known as TheJulia | 11:56 | |
| eliqiao | AJaeger: hi around? can you help to take a look at https://review.openstack.org/#/c/326306/ ? (zun irc renaming) | 11:58 |
| *** thorst_ has quit IRC | 11:58 | |
| *** amoralej is now known as amoralej|lunch | 11:58 | |
| lifeless | notmorgan: that said, I thought we had direction specific tests - e.g. write('') vs write(b'') for write streams, so perhaps thats a better and longer term fix | 11:58 |
| *** maishsk has joined #openstack-infra | 12:00 | |
| AJaeger | eliqiao: not right now, need to figure out a few bits first... | 12:02 |
| *** ramishra has joined #openstack-infra | 12:03 | |
| *** roxanaghe has joined #openstack-infra | 12:03 | |
| *** ldnunes has joined #openstack-infra | 12:03 | |
| *** amitgandhinz has joined #openstack-infra | 12:03 | |
| *** shardy has quit IRC | 12:04 | |
| *** sdake has joined #openstack-infra | 12:05 | |
| *** ddieterly is now known as ddieterly[away] | 12:05 | |
| eliqiao | AJaeger: yes? can you tell a bit more? | 12:06 |
| AJaeger | eliqiao: I don't have time to look at this right now and need to double check a few bits first | 12:07 |
| AJaeger | eliqiao: sorry - hope I manage later today. But there are others to review as well... | 12:08 |
| *** roxanaghe has quit IRC | 12:08 | |
| *** amitgandhinz has quit IRC | 12:08 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Enable Tempest test job also for sahara-tests https://review.openstack.org/329590 | 12:09 |
| *** ramishra has quit IRC | 12:10 | |
| eliqiao | AJaeger: oh sorry, not in a hurry, thanks. Just take your time. I can check it tomorrow(my time) | 12:11 |
| *** maishsk has quit IRC | 12:12 | |
| *** winggundamth_ has joined #openstack-infra | 12:12 | |
| *** ramishra has joined #openstack-infra | 12:12 | |
| *** claudiub has quit IRC | 12:12 | |
| *** rhallisey has joined #openstack-infra | 12:15 | |
| *** sarob has joined #openstack-infra | 12:15 | |
| *** slagle_ is now known as slagle | 12:16 | |
| *** ramishra has quit IRC | 12:16 | |
| *** samuelBartel has joined #openstack-infra | 12:17 | |
| *** claudiub has joined #openstack-infra | 12:19 | |
| *** sarob has quit IRC | 12:19 | |
| *** ramishra has joined #openstack-infra | 12:21 | |
| *** lucas-hungry is now known as lucasagomes | 12:22 | |
| *** jtyr has quit IRC | 12:23 | |
| *** gema has quit IRC | 12:23 | |
| *** rcernin has quit IRC | 12:24 | |
| *** gema has joined #openstack-infra | 12:25 | |
| *** zhurong has joined #openstack-infra | 12:25 | |
| *** dizquierdo has quit IRC | 12:26 | |
| *** ccamacho|lunch is now known as ccamacho | 12:26 | |
| *** kgiusti has joined #openstack-infra | 12:29 | |
| *** maishsk has joined #openstack-infra | 12:29 | |
| *** lvdongbing has joined #openstack-infra | 12:30 | |
| *** trown|outtypewww is now known as trown | 12:30 | |
| *** thiagop has joined #openstack-infra | 12:30 | |
| *** pradk has joined #openstack-infra | 12:31 | |
| *** ddieterly has joined #openstack-infra | 12:32 | |
| *** jcoufal has joined #openstack-infra | 12:34 | |
| openstackgerrit | Tom Fifield proposed openstack-infra/project-config: Add governance-uc repo https://review.openstack.org/330403 | 12:34 |
| *** javeriak has joined #openstack-infra | 12:34 | |
| *** pas-ha has joined #openstack-infra | 12:39 | |
| *** rcernin has joined #openstack-infra | 12:39 | |
| *** pas-ha has left #openstack-infra | 12:39 | |
| *** julim has joined #openstack-infra | 12:39 | |
| *** e0ne has joined #openstack-infra | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:40 | |
| *** zul_ has joined #openstack-infra | 12:41 | |
| *** ddieterly is now known as ddieterly[away] | 12:41 | |
| *** ddieterly[away] has quit IRC | 12:41 | |
| *** yujunz has joined #openstack-infra | 12:41 | |
| *** zul_ has quit IRC | 12:42 | |
| *** javeriak has quit IRC | 12:42 | |
| *** zul_ has joined #openstack-infra | 12:42 | |
| *** zul_ has quit IRC | 12:43 | |
| *** zul_ has joined #openstack-infra | 12:44 | |
| *** zeih has quit IRC | 12:44 | |
| pabelanger | infra-root: looks like zl01 is no longer updating nodepool properly. I am going to graceful stop it for now until we schedule a nodepool reboot | 12:44 |
| *** eharney has quit IRC | 12:44 | |
| *** edmondsw has joined #openstack-infra | 12:45 | |
| mordred | pabelanger: k. is that a behavior we should debug? | 12:45 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a temporary tripleo cloud "rh2" https://review.openstack.org/330514 | 12:45 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a experimental tripleo job on rh2 https://review.openstack.org/330515 | 12:45 |
| pabelanger | mordred: Yes, going to look at the logs and see what happen. See if anything specific stands out | 12:46 |
| mordred | cool | 12:46 |
| *** rodrigods has quit IRC | 12:46 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds API reference publishing jobs for Swift https://review.openstack.org/313015 | 12:46 |
| *** rodrigods has joined #openstack-infra | 12:46 | |
| *** isaacb has joined #openstack-infra | 12:46 | |
| *** baoli has joined #openstack-infra | 12:47 | |
| *** amitgandhinz has joined #openstack-infra | 12:48 | |
| *** amitgandhinz has quit IRC | 12:48 | |
| derekh | In https://review.openstack.org/330514 I'm hoping to at a new (temporary) deployment for tripleo , whats the best way to handover credentials? | 12:48 |
| *** amitgandhinz has joined #openstack-infra | 12:49 | |
| *** zul_ has quit IRC | 12:49 | |
| *** annegentle has joined #openstack-infra | 12:49 | |
| *** zul has quit IRC | 12:49 | |
| isaacb | yolanda: hi | 12:49 |
| *** zul has joined #openstack-infra | 12:50 | |
| lennyb | jordanP: regarding https://review.openstack.org/#/c/330331/ , we are facing some difficulties debugging test_router_rescheduling that failed. Do you know who can assist us with it? Thanks | 12:51 |
| *** winggundamth_ has quit IRC | 12:51 | |
| jordanP | lennyb, do you have log of a failure ? | 12:51 |
| jordanP | I mean, a direct http link | 12:52 |
| *** Julien-zte has joined #openstack-infra | 12:52 | |
| lennyb | jordanP: http://13.69.151.247/31/330331/1/check-sriov-tempest/Tempest-Sriov/a62d113/testr_results.html.gz | 12:52 |
| *** baoli_ has joined #openstack-infra | 12:53 | |
| isaacb | AJaeger: Hi | 12:53 |
| *** zeih has joined #openstack-infra | 12:53 | |
| *** rcernin has quit IRC | 12:54 | |
| *** ramishra has quit IRC | 12:54 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-diskimage_builder: Upodate pip provider to use openstack_pip provider https://review.openstack.org/330524 | 12:54 |
| jordanP | lennyb, you sure if you revert that OO patch, the test pass ? | 12:54 |
| jordanP | Or maybe another regression got in | 12:55 |
| *** jsavak has joined #openstack-infra | 12:55 | |
| *** mikelk has joined #openstack-infra | 12:55 | |
| *** baoli has quit IRC | 12:55 | |
| jordanP | not, that unlikely, nothing else was merged recently | 12:56 |
| lennyb | jordanP: did not check it. | 12:56 |
| *** Goneri has joined #openstack-infra | 12:57 | |
| jordanP | in the log I see "2016-06-16 13:04:11,434 1628 DEBUG [tempest.common.fixed_network] No valid network provided or created, defaulting to fixed_network_name", was this message present a couple of days ago ? | 12:58 |
| *** gouthamr has joined #openstack-infra | 12:58 | |
| *** aysyd has joined #openstack-infra | 12:58 | |
| *** abregman is now known as abregman|intervi | 12:58 | |
| lennyb | jordanP: sorry, I did check it. http://13.69.151.247/46/303346/114/check-mlnx/Neutron-ML2-Sriov/ea40c91/testr_results.html.gz with tempest cf2150a534659964255a9682ecb32c7b78943ea6 | 12:58 |
| *** thorst__ is now known as thorst_ | 12:58 | |
| *** ramishra has joined #openstack-infra | 12:58 | |
| pabelanger | mordred: jeblair: at first glance, it looks like nodepool was no longer receiving gearman updates from zl01. Looks to be the same issue we have then jenkins loose a connection to nodepool. | 12:58 |
| mordred | pabelanger: interesting | 12:59 |
| lennyb | jordanP: I am running CI with temporal tempest git checkout cf2150a534659964255a9682ecb32c7b78943ea6 | 12:59 |
| pabelanger | mordred: jeblair: zl01 was running jobs, nodepool was missing the used event and seeing them only as ready | 12:59 |
| *** abregman|intervi has quit IRC | 12:59 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-jenkins: Upodate pip provider to use openstack_pip provider https://review.openstack.org/330528 | 12:59 |
| pabelanger | and zl01 was getting more ready nodes, nodepool was sending them just not receiving | 13:00 |
| pabelanger | I won't restart nodepool until jeblair is around to review | 13:00 |
| isaacb | pabelanger: can you assist with https://review.openstack.org/#/c/330054 ? | 13:02 |
| isaacb | this cause install_modules.sh to fail | 13:02 |
| isaacb | the script tries to clone non existent "charm-specs" repo | 13:03 |
| *** eezhova has quit IRC | 13:03 | |
| pabelanger | isaacb: where are you seeing the failure? Trying to see how they are related as install_modules.sh is for our puppet modules | 13:03 |
| *** roxanaghe has joined #openstack-infra | 13:04 | |
| *** gouthamr_ has joined #openstack-infra | 13:04 | |
| *** kushal has quit IRC | 13:04 | |
| isaacb | pabelanger: I'm trying to create an image. | 13:04 |
| *** kzaitsev_mb has joined #openstack-infra | 13:04 | |
| *** ramishra has quit IRC | 13:05 | |
| *** amoralej|lunch is now known as amoralej | 13:05 | |
| pabelanger | isaacb: with diskimage-builder? | 13:05 |
| *** winggundamth_ has joined #openstack-infra | 13:05 | |
| *** mescanef has quit IRC | 13:05 | |
| *** ramishra has joined #openstack-infra | 13:05 | |
| isaacb | pabelanger: no. snapshot | 13:05 |
| pabelanger | okay, that aside, I can see https://git.openstack.org/cgit/openstack/charm-specs doesn't exist yet | 13:05 |
| pabelanger | let me look and see why it failed | 13:05 |
| isaacb | pabelanger: charm-specs is in github | 13:06 |
| isaacb | install_modules works with git.openstack.org | 13:06 |
| *** banix has joined #openstack-infra | 13:06 | |
| pabelanger | isaacb: URL? | 13:06 |
| pabelanger | I just looked at our nodepool DIB builds, they failed too | 13:07 |
| pabelanger | so something is wrong | 13:07 |
| *** rcernin has joined #openstack-infra | 13:07 | |
| *** gouthamr has quit IRC | 13:07 | |
| isaacb | https://github.com/openstack-charmers/charm-specs.git | 13:07 |
| *** psilvad has joined #openstack-infra | 13:08 | |
| isaacb | you can see that gerrit/projects.yaml point to github as upstream | 13:08 |
| *** roxanaghe has quit IRC | 13:08 | |
| pabelanger | isaacb: right, but it will get imported into openstack/charm-specs, using manage_projects.py, that import process failed for some reason | 13:09 |
| phschwartz | nibalizer: fungi: https://review.openstack.org/330528 https://review.openstack.org/330524 | 13:09 |
| *** e0ne has quit IRC | 13:09 | |
| phschwartz | nibalizer: fungi: those so far fix my issue. Looking to see if there are others. | 13:09 |
| rcarrillocruz | pabelanger, yolanda :heya, added a bunch of changes for infracloud inventory | 13:10 |
| *** mdrabe has joined #openstack-infra | 13:10 | |
| yolanda | hi rcarrillocruz , seems you are having fun today! :) | 13:10 |
| pabelanger | rcarrillocruz: Nice, will review them shortly. Was also going to ping you this morning and see where things were with a controller | 13:11 |
| rcarrillocruz | :-) | 13:11 |
| *** Hal has joined #openstack-infra | 13:11 | |
| rcarrillocruz | not now, but once they move the machines again | 13:11 |
| yolanda | going to review | 13:11 |
| rcarrillocruz | we will have to check with networking | 13:11 |
| *** Hal is now known as Guest58170 | 13:11 | |
| *** zeih has quit IRC | 13:11 | |
| rcarrillocruz | we want to have two different clouds, one vanilla another chocolate | 13:11 |
| *** gouthamr has joined #openstack-infra | 13:11 | |
| rcarrillocruz | for potentially different versions, etc | 13:11 |
| *** pgadiya has quit IRC | 13:11 | |
| rcarrillocruz | not sure if we also want to split them in terms of provisioning | 13:12 |
| *** e0ne has joined #openstack-infra | 13:12 | |
| rcarrillocruz | before they were physically split, currently they are not | 13:12 |
| rcarrillocruz | so one baremetal00 dhcp to rule them all | 13:12 |
| *** zeih has joined #openstack-infra | 13:12 | |
| rcarrillocruz | yolanda: which baremetal did we use for east? | 13:13 |
| rcarrillocruz | as there were no config on system-config, i wonder where it is | 13:13 |
| rcarrillocruz | baremetal01 | 13:13 |
| rcarrillocruz | ? | 13:13 |
| rcarrillocruz | baremetal00 shows just west machines on the ironic inventory | 13:13 |
| wznoinsk | would anyone know whether there will be any CI talks proposed by infra team for Barcelona? | 13:13 |
| yolanda | i think baremetal01 | 13:13 |
| phschwartz | yolanda: These are a couple of puppet changes if you don't mind a quick review. https://review.openstack.org/330528 https://review.openstack.org/330524 | 13:14 |
| *** psilvad has quit IRC | 13:14 | |
| *** hemanthm_ is now known as hemanthm | 13:14 | |
| rcarrillocruz | wznoinsk: pabelanger was about to propose a thread for potential talks for Barcelona | 13:14 |
| *** psilvad has joined #openstack-infra | 13:14 | |
| rcarrillocruz | yolanda: ++, let me check | 13:14 |
| *** gouthamr_ has quit IRC | 13:15 | |
| Shrews | pabelanger: any chance zl01 was running an older version (before jun 8) of zuul? | 13:15 |
| pabelanger | Shrews: I don't believe so, we did a gracefully restart last night. So we should have used the latest release at that time. However, I will confirm | 13:15 |
| *** eharney has joined #openstack-infra | 13:17 | |
| wznoinsk | rcarrillocruz: do you discuss these on infra meetings? I'd like to talk about what I'm thinking to submit - 3rd party CI, it may be a good one for collaboration | 13:17 |
| *** pcarver_ has joined #openstack-infra | 13:17 | |
| *** mescanef has joined #openstack-infra | 13:17 | |
| Shrews | pabelanger: don't have the access you do, so can't help you much. was just looking at recent zuul & nodepool changes for possible causes | 13:17 |
| rcarrillocruz | i think we agreed to discuss on the mailing list, but i don't see why you couldn't raise that topic on next meeting | 13:17 |
| pcarver_ | Is anybody here familiar with the code that generates the IRC meeting entries on eavesdrop.o.o? | 13:18 |
| rcarrillocruz | if the thread has not gone live by the time, you could talk about it on open discussion too | 13:18 |
| *** pcarver_ is now known as pcarver | 13:18 | |
| *** zeih has quit IRC | 13:18 | |
| wznoinsk | ok, will see what's on ML and join meeting too, thanks | 13:18 |
| *** zeih has joined #openstack-infra | 13:19 | |
| *** matt-borland has joined #openstack-infra | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:19 | |
| *** psilvad has quit IRC | 13:19 | |
| pabelanger | puppet-user[8927]: (/Stage[main]/Openstack_project::Puppetmaster/Package[shade]/ensure) change from 1.9.0 to latest failed: Could not get latest version: HTTP-Error: 403 Must access using HTTPS instead of HTTP | 13:19 |
| pabelanger | Did we figure out the fix to ^ last night? | 13:20 |
| pabelanger | puppet is failing to apply on puppetmaster.o.o which is stopping ansible from running properly | 13:20 |
| phschwartz | pabelanger: I just fixed 2 and we need to fix that one | 13:20 |
| pabelanger | phschwartz: okay, checking | 13:20 |
| phschwartz | pabelanger: there was a change yesterday to pypi where instead of redirecting http to https where it now 403's | 13:20 |
| *** psilvad has joined #openstack-infra | 13:21 | |
| phschwartz | pabelanger: https://review.openstack.org/330528 https://review.openstack.org/330524 | 13:21 |
| *** salv-orlando has quit IRC | 13:21 | |
| phschwartz | pabelanger: I will put up one for that one, but we need to find the others | 13:21 |
| *** salv-orlando has joined #openstack-infra | 13:21 | |
| pabelanger | phschwartz: what is openstack_pip? first time seeing that | 13:21 |
| phschwartz | pabelanger: it is a provider that fixes the issue that was created by crinkle | 13:22 |
| pabelanger | ya, quick google leads me to 302396 | 13:22 |
| anteaya | wznoinsk: what can I help you with? | 13:22 |
| *** jcoufal has quit IRC | 13:23 | |
| *** pgadiya has joined #openstack-infra | 13:23 | |
| anteaya | wznoinsk: I'm here briefly then off running errands this morning | 13:23 |
| *** bhavik1 has joined #openstack-infra | 13:23 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: fix the pattern matching in the constraint update logic https://review.openstack.org/330542 | 13:24 |
| mordred | phschwartz: thanks | 13:24 |
| *** sree has quit IRC | 13:24 | |
| pabelanger | isaacb: okay, found the issue. Looks to be an issue with our puppet wheel not running properly. Until fixed, openstack/charm-specs will continue to fail | 13:25 |
| *** sree has joined #openstack-infra | 13:25 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/system-config: Upodate pip provider to use openstack_pip provider https://review.openstack.org/330547 | 13:25 |
| *** psilvad has quit IRC | 13:25 | |
| *** akshai has joined #openstack-infra | 13:25 | |
| *** pgadiya has quit IRC | 13:25 | |
| phschwartz | pabelanger: ^^ | 13:26 |
| phschwartz | pabelanger: that is fixing it in openstack_project | 13:26 |
| *** rfolco has joined #openstack-infra | 13:26 | |
| phschwartz | pabelanger: I am sure there will be others in the infra scope, I haven't gone throgh all puppet-* yet | 13:26 |
| phschwartz | mordred: hey, fixing it upstream is good. It is breaking me so why not help others :) | 13:26 |
| pabelanger | phschwartz: great, thanks | 13:27 |
| *** akshai_ has joined #openstack-infra | 13:27 | |
| phschwartz | pabelanger: just need to get those 3 reviews reviewed and merged in. | 13:27 |
| phschwartz | I will keep looking for others | 13:27 |
| wznoinsk | anteaya: I was wondering whether you know anyone doing any talks about CI/3rdparty CIs in Barcelona (or plan to do it) | 13:27 |
| *** raildo-a` is now known as raildo | 13:27 | |
| *** psilvad has joined #openstack-infra | 13:27 | |
| pabelanger | yolanda: mordred: do you mind reviewing: https://review.openstack.org/#/q/topic:change_pip_provider | 13:27 |
| *** vincentll has quit IRC | 13:28 | |
| *** vincentll has joined #openstack-infra | 13:28 | |
| mordred | pabelanger: done | 13:29 |
| pabelanger | mordred: phschwartz: great, thank you | 13:29 |
| phschwartz | mordred: pabelanger: there might be more. I will alert on the topic if there are. | 13:30 |
| *** sree has quit IRC | 13:30 | |
| pabelanger | phschwartz: ack | 13:30 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Remove special config for west bifrost host https://review.openstack.org/330552 | 13:30 |
| anteaya | wznoinsk: I am not aware of anyone | 13:31 |
| *** akshai has quit IRC | 13:31 | |
| *** ramishra has quit IRC | 13:31 | |
| anteaya | wznoinsk: I'm guessing someone or multiple someones will put proposals forward, the talk submission process usually has many enteries, but noone has come to me or said anything that I have seen that allows me to say joe shmoe is putting in a talk | 13:32 |
| *** roxanaghe has joined #openstack-infra | 13:32 | |
| isaacb | pabelanger: what about: modules/openstack_project/manifests/template.pp | 13:32 |
| phschwartz | pabelanger: mordred: cloning all puppet-* modules we have in the projects set takes a long time. lol | 13:33 |
| *** roxanaghe has joined #openstack-infra | 13:33 | |
| mordred | phschwartz: right? | 13:33 |
| wznoinsk | I'm planning to submit a talk about our journey with 3rdparty CI, I'm looking if anyone would want to join forces on that... I'll ask on the mailing list then | 13:34 |
| phschwartz | mordred: it has been going for over an hour. lol | 13:34 |
| anteaya | wznoinsk: good idea | 13:35 |
| phschwartz | wznoinsk: I might be interested. Not 100% sure I will make it yet though. My wife is due any day so I am not sure I want to leave her alone with the 6 year old and a 4 month old as that will still be during the partial sleepless period. | 13:35 |
| *** mtanino has joined #openstack-infra | 13:35 | |
| *** Na3iL has quit IRC | 13:36 | |
| *** eezhova has joined #openstack-infra | 13:36 | |
| isaacb | wznoinsk: for which project of openstack you run the CI? | 13:36 |
| *** ramishra has joined #openstack-infra | 13:37 | |
| pabelanger | isaacb: I am not sure your question about template.pp | 13:37 |
| wznoinsk | phschwartz how likely the delivery could be clashing with the prep/summit? | 13:38 |
| wznoinsk | isaacb: neutron, nova, networking-ovs-dpdk, networking-odl, devstack - Intel NFV CI | 13:38 |
| phschwartz | wznoinsk: she is due a couple days from now, aka 4 months old by the summit. | 13:39 |
| *** zeih has quit IRC | 13:39 | |
| *** eezhova has quit IRC | 13:39 | |
| *** Na3iL has joined #openstack-infra | 13:39 | |
| pabelanger | wznoinsk: anteaya: You might consider adding it to https://etherpad.openstack.org/p/barcelona-upstream-openstack-infa . See if any other people are interested in helping out | 13:40 |
| wznoinsk | phschwartz: which CI you're running? | 13:40 |
| *** dizquierdo has joined #openstack-infra | 13:40 | |
| *** ddieterly has joined #openstack-infra | 13:40 | |
| phschwartz | wznoinsk: a mixed 3rd party + a full clone of infra for IBM Cloud, along with Rax prior. | 13:40 |
| pabelanger | wznoinsk: anteaya: nothing official by any means but a place for speakers to collaborate | 13:41 |
| *** furlongm_ is now known as furlongm | 13:41 | |
| yolanda | sorry, meeting. Going through the review queue now | 13:42 |
| *** eezhova has joined #openstack-infra | 13:42 | |
| *** inc0 has joined #openstack-infra | 13:42 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Fix baremetal00 name attribute still referencing west https://review.openstack.org/330560 | 13:42 |
| anteaya | just to be clear, I'm not endorsing a talk in third party ci | 13:43 |
| anteaya | those talks tend not to make the roster that I have seen | 13:43 |
| anteaya | but I answered the question posed to me to the best of my ability | 13:43 |
| wznoinsk | anteaya you're worried we won't have enough interest in the topic? | 13:44 |
| anteaya | that isn't what I said | 13:44 |
| *** ddieterly is now known as ddieterly[away] | 13:44 | |
| anteaya | you asked me a question and I answered it | 13:45 |
| anteaya | I'm not endorsing anyone's talk proposal | 13:45 |
| *** _sigmavirus24 is now known as sigmavirus24 | 13:46 | |
| *** sigmavirus24 has joined #openstack-infra | 13:46 | |
| *** abregman has joined #openstack-infra | 13:47 | |
| *** ddieterly[away] is now known as ddieterly | 13:49 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Replace hpuswest name for vanilla https://review.openstack.org/330567 | 13:49 |
| *** rbrndt has joined #openstack-infra | 13:50 | |
| wznoinsk | anteaya: understood, thanks | 13:51 |
| anteaya | wznoinsk: thank you | 13:52 |
| *** jcoufal has joined #openstack-infra | 13:52 | |
| anteaya | my wish for you and for everyone submitting any talk proposal is that you clearly state your intent and may you give a good presentation should you be selected | 13:52 |
| *** aviau has quit IRC | 13:54 | |
| *** aviau has joined #openstack-infra | 13:54 | |
| *** tpsilva has joined #openstack-infra | 13:58 | |
| *** hongbin has joined #openstack-infra | 13:58 | |
| *** xarses_ has quit IRC | 13:59 | |
| anteaya | I'm off doing errands today, back later | 14:00 |
| *** garyk1 has quit IRC | 14:01 | |
| *** woodster_ has joined #openstack-infra | 14:01 | |
| jeblair | pabelanger, mordred: one thing that might be helpful if this happens again is to run a zmq listener process on the zuul server connecting to localhost to see if the problem is that zl is not transmitting or whether nodepool is not listening. | 14:02 |
| pabelanger | jeblair: can we try that now? We can bring zl01 back online and see what happens | 14:03 |
| wznoinsk | pabelanger: I've replied in the above 'infa' etherpad | 14:03 |
| isaacb | Core reviewers: can someone please review: https://review.openstack.org/#/c/321480 | 14:03 |
| pabelanger | wznoinsk: ack | 14:03 |
| *** lvdongbing has quit IRC | 14:03 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Replace hpuseast name for chocolate https://review.openstack.org/330580 | 14:05 |
| jeblair | pabelanger: sure | 14:05 |
| *** abregman has quit IRC | 14:05 | |
| pabelanger | okay, bringing zl01 back online | 14:05 |
| jeblair | i'm running a stream listener | 14:05 |
| pabelanger | started | 14:06 |
| *** annegentle has quit IRC | 14:07 | |
| *** nwkarsten has joined #openstack-infra | 14:07 | |
| *** PollyZ has joined #openstack-infra | 14:07 | |
| *** bhavik1 has quit IRC | 14:07 | |
| pabelanger | okay, zl01 got a job | 14:07 |
| *** infra-red has joined #openstack-infra | 14:08 | |
| jeblair | it's receiving events | 14:08 |
| jeblair | er sending them rather | 14:08 |
| fungi | jeblair: pabelanger: rather than running an extra zmq listener, couldn't you just fire up tcpdump? | 14:09 |
| jeblair | fungi: that would probably work as well | 14:10 |
| jeblair | pabelanger, fungi: nodepool did not get that event | 14:10 |
| jeblair | so it seems to be on the nodepool side | 14:10 |
| jd__ | hello guys; could you add myself in panko-core so I can add more team members and merge patches? :) | 14:10 |
| pabelanger | okay | 14:10 |
| fungi | is each listener in nodepool a separate thread? | 14:10 |
| *** infra-red has quit IRC | 14:11 | |
| jeblair | pabelanger, fungi: i'm running tcpdump on nodepool | 14:11 |
| fungi | jd__: no problem--just a sec | 14:11 |
| jeblair | fungi: i believe so (zmq is a bit of a black box about that sort of thing) | 14:11 |
| fungi | jd__: i've added you to panko-core and panko-stable-maint | 14:12 |
| fungi | jeblair: just wondering whether a thread dump of nodepool would yield anything useful (assuming we could even tell which thread went with which publisher) | 14:13 |
| jd__ | thanks fungi! | 14:13 |
| fungi | jd__: happy hacking! | 14:14 |
| jd__ | hehe | 14:14 |
| fungi | jd__: oh, also AJaeger pointed out a couple of stuck jobs for that repo | 14:14 |
| fungi | just a sec and i'll see if that's still the case | 14:14 |
| jd__ | fungi: yeah I just send a patch to fix those jobs :) | 14:14 |
| fungi | oh, perfect! | 14:14 |
| jd__ | https://review.openstack.org/#/c/330431/ makes it all green :) | 14:14 |
| jd__ | thanks AJaeger for noticing btw | 14:14 |
| fungi | jd__: oh, not failing but actually stuck waiting for a worker | 14:16 |
| fungi | panko-upstream-translation-update (in post) and panko-propose-translation-update (in periodic) | 14:16 |
| jeblair | fungi, pabelanger: tcpdump on nodepool is not seeing traffic from zl01 | 14:16 |
| fungi | jeblair: have tcpdump running on zl01? | 14:17 |
| jeblair | fungi: not yet | 14:17 |
| *** zeih has joined #openstack-infra | 14:17 | |
| fungi | assuming no firewall or network connectivity issues, it's probably not emitting anything | 14:17 |
| *** ri0 has joined #openstack-infra | 14:18 | |
| jeblair | fungi: agreed; it's likely the zmq bits in nodepool stuck, and not having an active connection | 14:18 |
| jeblair | yep, tcpdump on zl01 shows nothing as well | 14:18 |
| fungi | jeblair: ooh, wonder if we have a half-dead socket and there's no keepalive/timeout for it | 14:18 |
| jeblair | fungi: maybe so | 14:18 |
| *** zz_dimtruck is now known as dimtruck | 14:18 | |
| pabelanger | fungi: Ah, right. | 14:18 |
| fungi | netstat showing anything for that source/destination? | 14:18 |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-ansible: Update pip provider to use openstack_pip provider https://review.openstack.org/330589 | 14:19 |
| *** abregman has joined #openstack-infra | 14:19 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-bandersnatch: Update pip provider to use openstack_pip provider https://review.openstack.org/330590 | 14:19 |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-gerritbot: Update pip provider to use openstack_pip provider https://review.openstack.org/330591 | 14:19 |
| *** yaume has quit IRC | 14:19 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-jeepyb: Update pip provider to use openstack_pip provider https://review.openstack.org/330592 | 14:19 |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-lodgeit: Update pip provider to use openstack_pip provider https://review.openstack.org/330594 | 14:19 |
| *** daemontool has joined #openstack-infra | 14:19 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-log_processor: Update pip provider to use openstack_pip provider https://review.openstack.org/330595 | 14:19 |
| jeblair | fungi: no, actually | 14:19 |
| *** yaume has joined #openstack-infra | 14:19 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-logstash: Update pip provider to use openstack_pip provider https://review.openstack.org/330596 | 14:20 |
| *** winggundamth_ has quit IRC | 14:20 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-meetbot: Update pip provider to use openstack_pip provider https://review.openstack.org/330597 | 14:20 |
| jeblair | so this is probably all on zmq | 14:20 |
| fungi | huh, so really just not connected at all | 14:20 |
| fungi | wowzers | 14:20 |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-odsreg: Update pip provider to use openstack_pip provider https://review.openstack.org/330598 | 14:20 |
| jeblair | it's why i want to get rid of zmq and never touch it again in infra | 14:20 |
| fungi | yes. i totally get that | 14:20 |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-subunit2sql: Update pip provider to use openstack_pip provider https://review.openstack.org/330599 | 14:20 |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/puppet-zuul: Update pip provider to use openstack_pip provider https://review.openstack.org/330600 | 14:21 |
| pabelanger | jeblair: this is what zookeeper is going to replace right? | 14:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: return nodes to nodepool via gearman https://review.openstack.org/330601 | 14:21 |
| *** kaisers_ has joined #openstack-infra | 14:21 | |
| phschwartz | pabelanger: mordred: yolanda: fungi: https://review.openstack.org/#/q/topic:change_pip_provider | 14:21 |
| *** burgerk has joined #openstack-infra | 14:21 | |
| phschwartz | about 13 more reviews for infra related things moving to openstack_pip to fix the pypi issue | 14:22 |
| jeblair | pabelanger: yes, i hope. we haven't actually agreed on that / landed that spec yet, because we're going to try it out in nodepool first | 14:22 |
| jeblair | pabelanger, fungi, mordred: i'm thinking about doing this as a temporary measure to increase our reliability: Ansible launcher: return nodes to nodepool via gearman https://review.openstack.org/330601 | 14:22 |
| jeblair | (untested) | 14:23 |
| mordred | jeblair: looking | 14:24 |
| mordred | phschwartz: thanks! those all look great | 14:24 |
| fungi | okay, this is weird. build:panko-upstream-translation-update and build:panko-propose-translation-update are registered in gearman with no available workers | 14:24 |
| fungi | zlstatic01 should be reporting proposal.slave.o.o as available to run them, right? | 14:25 |
| jeblair | fungi: theoretically | 14:26 |
| fungi | presumably it was available at some point because it registered it into gearman | 14:27 |
| *** zeih has quit IRC | 14:27 | |
| *** jistr is now known as jistr|mtg | 14:28 | |
| *** ifarkas has quit IRC | 14:28 | |
| *** bhavik has joined #openstack-infra | 14:28 | |
| jeblair | i'm going to get the worker list from gearman | 14:28 |
| jeblair | nobody else do that because it's huge | 14:29 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Move gate-manila-buildimage-docker job to check pipeline https://review.openstack.org/330604 | 14:29 |
| fungi | k | 14:29 |
| *** ri0 has quit IRC | 14:29 | |
| *** amotoki has joined #openstack-infra | 14:29 | |
| jeblair | yeah, it is not on proposal's list of jobs | 14:30 |
| jeblair | but proposal does have many translation-update jobs | 14:31 |
| fungi | however zlstatic01 does seem to have an up to date jjb config with panko instantiating the translation-jobs group | 14:32 |
| jeblair | jjb seems to want to create that job | 14:33 |
| jeblair | i will point my local zuul-launcher at the config and see if it wants to register it | 14:33 |
| *** EricGonczer_ has joined #openstack-infra | 14:34 | |
| *** ramishra has quit IRC | 14:34 | |
| jeblair | build:panko-upstream-translation-update:proposal001 | 14:36 |
| jeblair | that's local ^ | 14:36 |
| *** ramishra has joined #openstack-infra | 14:37 | |
| mordred | oh - out of date files on zuul launcher | 14:37 |
| jeblair | mordred: which ones? | 14:37 |
| mordred | we're still recovering from the pip outage causing puppet to not run | 14:37 |
| fungi | is it possible puppet put the jjb configs on teh launcher but didn't exec jjb? | 14:38 |
| mordred | jeblair: the copies of the jjb files on the launcher host I image are not up to date | 14:38 |
| jeblair | mordred: fungi checked the jjb config, and zuul reconfigured at 2016-06-16 11:52:44,955 | 14:38 |
| mordred | oh. bother. don't listen to me | 14:38 |
| *** sarob has joined #openstack-infra | 14:38 | |
| jeblair | fungi: we don't need to run jjb, just update the files and reconfigure zuul ^ | 14:38 |
| fungi | but yeah, broken puppet because of the pypi behavior change has been lingering in the back of my mind | 14:38 |
| *** hongbin has quit IRC | 14:39 | |
| fungi | jeblair: oh, right, zl imports jjb as a lib | 14:39 |
| jeblair | fungi: was this recently added? | 14:39 |
| fungi | oldthink dies hard | 14:39 |
| fungi | jwbdded yesterday | 14:39 |
| fungi | gah | 14:39 |
| fungi | added yesterday | 14:39 |
| jeblair | oh... i may have an idea... | 14:40 |
| jeblair | i think we may only re-register jobs if a node is not running a job | 14:40 |
| jeblair | which makes sense for a single-use node, but not so much for a long-running one | 14:41 |
| mordred | ++ | 14:41 |
| jeblair | the most recent reconfiguration *was* during a proposal job | 14:41 |
| *** pcaruana has quit IRC | 14:41 | |
| fungi | oh... | 14:42 |
| jeblair | 2016-06-15 17:39:12,186 DEBUG zuul.NodeWorker.proposal.slave.openstack.org: Received reconfigure request | 14:42 |
| fungi | would definitely explain it | 14:42 |
| jeblair | that was the most recent reconfiguration received outside of running a job | 14:42 |
| *** jamesmcarthur has joined #openstack-infra | 14:42 | |
| *** sarob has quit IRC | 14:43 | |
| notmorgan | lifeless: ack and i have a test just was super tired at 2am | 14:43 |
| notmorgan | lifeless: so, i'll revisit | 14:43 |
| fungi | yeah, i believe that was before the panko patch merged and got puppeted | 14:43 |
| jeblair | fungi: it's possible the registration in gearman came from jenkins.o.o | 14:44 |
| *** trown is now known as trown|brb | 14:44 | |
| jeblair | okay, i think i can fix this fairy quickly | 14:44 |
| *** kaisers_ has quit IRC | 14:44 | |
| fungi | makes sense | 14:45 |
| *** mixos has joined #openstack-infra | 14:45 | |
| *** gordc has joined #openstack-infra | 14:45 | |
| *** kmartin has joined #openstack-infra | 14:45 | |
| notmorgan | jeblair: doh. but that makes sense based upon what you're saying | 14:45 |
| *** xarses_ has joined #openstack-infra | 14:47 | |
| *** roxanaghe has joined #openstack-infra | 14:47 | |
| *** kaisers_ has joined #openstack-infra | 14:48 | |
| *** sarob has joined #openstack-infra | 14:49 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch back to current-tripleo pre promotion https://review.openstack.org/111011 | 14:49 |
| *** jaosorior has quit IRC | 14:49 | |
| *** mriedem1 has joined #openstack-infra | 14:50 | |
| *** EricGonczer_ has quit IRC | 14:50 | |
| *** bhavik has quit IRC | 14:50 | |
| openstackgerrit | Mário Santos proposed openstack-infra/shade: Add support for changing metadata of compute instances https://review.openstack.org/319395 | 14:50 |
| *** bhavik has joined #openstack-infra | 14:50 | |
| *** mriedem has quit IRC | 14:51 | |
| jeblair | fungi: oh, you can probably run 'zuul-launcher reconfigure' if proposal is not running a job right now and the immediate problem will be fixed | 14:51 |
| jeblair | fungi: er, it is running a job | 14:51 |
| jeblair | fungi: but when it goes idle, you could do that | 14:51 |
| fungi | will do--thanks! | 14:51 |
| jeblair | fungi: "nc proposal.slave.openstack.org 8088" :) | 14:51 |
| *** roxanaghe has quit IRC | 14:52 | |
| *** trown|brb is now known as trown | 14:52 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: update registration between jobs https://review.openstack.org/330624 | 14:52 |
| jeblair | fungi, notmorgan, mordred: ^ | 14:52 |
| *** jlanoux_ has quit IRC | 14:53 | |
| *** hongbin has joined #openstack-infra | 14:53 | |
| jeblair | oh, i want some debug lines in there, one sec | 14:53 |
| mordred | jeblair: looks good in general, pending debug lines | 14:53 |
| *** sarob has quit IRC | 14:54 | |
| notmorgan | jeblair: looking | 14:54 |
| *** sdake has quit IRC | 14:54 | |
| *** bhavik has quit IRC | 14:55 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: update registration between jobs https://review.openstack.org/330624 | 14:55 |
| jeblair | there we go | 14:55 |
| jeblair | they also serve as comments :) | 14:56 |
| *** tphummel has joined #openstack-infra | 14:56 | |
| notmorgan | jeblair: much better w/ debug | 14:56 |
| *** salv-orlando has quit IRC | 14:56 | |
| *** dizquierdo has quit IRC | 14:56 | |
| *** ramishra has quit IRC | 14:57 | |
| *** psachin has quit IRC | 14:57 | |
| *** amotoki_ has joined #openstack-infra | 14:57 | |
| notmorgan | jeblair: lgtm. | 14:57 |
| *** salv-orlando has joined #openstack-infra | 14:57 | |
| *** mriedem1 is now known as mriedem | 14:57 | |
| *** dtantsur is now known as dtantsur|bbl | 14:58 | |
| *** ramishra has joined #openstack-infra | 14:59 | |
| *** chem has quit IRC | 14:59 | |
| *** zhurong has quit IRC | 14:59 | |
| *** sdake has joined #openstack-infra | 14:59 | |
| *** maishsk has quit IRC | 15:00 | |
| *** salv-orlando has quit IRC | 15:00 | |
| *** chem has joined #openstack-infra | 15:00 | |
| *** amotoki has quit IRC | 15:00 | |
| *** salv-orlando has joined #openstack-infra | 15:00 | |
| *** mixos has quit IRC | 15:00 | |
| *** david-lyle_ is now known as david-lyle | 15:00 | |
| fungi | we seem to have a glut of ready nodes. i'm in a meeting right now but can try to look into it in a bit if nobody beats me to the punch | 15:02 |
| *** Na3iL has quit IRC | 15:02 | |
| pabelanger | oh, we need to take zl01 out of server again | 15:03 |
| pabelanger | I can do that now | 15:03 |
| pabelanger | jeblair: ^ are you finished with tcpdump? | 15:03 |
| jeblair | pabelanger: yep | 15:03 |
| zigo | pabelanger: Hi there! | 15:03 |
| zigo | pabelanger: Did you have a chance to work on the debian repo thing? | 15:03 |
| pabelanger | zl01 in graceful mode | 15:03 |
| zigo | pabelanger: Because it hasn't went far enough, now the TC wants to remove my project from the big tent: https://review.openstack.org/#/c/329430/ | 15:04 |
| pabelanger | zigo: sorry now, been head down working on our zuul-launcher migration | 15:04 |
| zigo | pabelanger: So we need to show progress. | 15:04 |
| zigo | pabelanger: We *just* need that repo stuff, then I can do the rest alone with a few review from infra. | 15:04 |
| zigo | pabelanger: If it's not you, who can I ask? | 15:04 |
| zigo | Monty didn't help at all. :( | 15:05 |
| *** yujunz has quit IRC | 15:05 | |
| *** cbader_ has quit IRC | 15:05 | |
| odyssey4me | zigo what do you need done? | 15:06 |
| *** amotoki_ has quit IRC | 15:06 | |
| zigo | odyssey4me: Packages are currently being published on each merge jobs, like this: http://tarballs.openstack.org/%7Bproject%7D/uploads/220dead31374edabced0d890bfaf2f900da4b66c/ | 15:06 |
| zigo | odyssey4me: We need a pickup job that would store these packages into a Debian repo. | 15:07 |
| pabelanger | zigo: Ya, I don't have the bandwidth at the moment. Apologies for that. My only suggestion at the moment would be to see how to create the repo external to infra at the moment. Then when bandwidth opens up, get our per project reprepro going again | 15:07 |
| zigo | Then make that debian repo available for other packages to use. | 15:07 |
| zigo | pabelanger: We already have that... | 15:07 |
| pabelanger | Or publisher the reprepro bits into static.o.o like we do for our periodic centos-7 afs client | 15:07 |
| zigo | Though there's then no point. | 15:07 |
| *** markmcd has quit IRC | 15:08 | |
| zigo | I'm not sure what that last line means. | 15:08 |
| odyssey4me | zigo ah, so you need to build a repo based on a set of published packages... ? | 15:08 |
| banix | Had a question about a gate job. Cam I add a file to /etc/sudoers.d for a gate job? If yes, how do I go about doing it? | 15:08 |
| zigo | odyssey4me: I need a repo to pickup packages produced after a merge of the packaging-deb git repo, yes. | 15:09 |
| lennyb | jordanP: revert passed https://review.openstack.org/#/c/330554 http://13.69.151.247/54/330554/1/check-sriov-tempest/Tempest-Sriov/6b189d3/testr_results.html.gz | 15:09 |
| *** kaisers_ has quit IRC | 15:09 | |
| *** kzaitsev_mb has quit IRC | 15:09 | |
| zigo | odyssey4me: I've been told to use the AFS of infra, but that's all I know, I have no idea how to integrate with the rest of infra. | 15:09 |
| *** kaisers_ has joined #openstack-infra | 15:10 | |
| *** markmcd has joined #openstack-infra | 15:10 | |
| *** markmcd1 has joined #openstack-infra | 15:10 | |
| *** mattymo_ is now known as mattymo | 15:10 | |
| odyssey4me | zigo hrm, that's a little out of my league at this point... I thought it might just be putting together another repo - if so, I've put together two patches which pabelanger needs to review *hint* ;) and it hasn't been too bad... there's some prior art | 15:10 |
| jordanP | lennyb, ack. Unfortunately I think we can't revert this patch. It's really useful... I feel you have to keep looking for the root cause of the CI failure | 15:10 |
| zigo | odyssey4me: Could you point to me at such a patch? | 15:11 |
| odyssey4me | zigo pabelanger perhaps what may be useful is if we can put a time slot together to understand what needs to be done better and to point at prior art... then perhaps you can put a review together yourself and maybe I can help | 15:11 |
| zigo | That would be nice. | 15:12 |
| odyssey4me | zigo these are my two new repo mirror requests: https://review.openstack.org/#/q/status:open+project:openstack-infra/system-config+branch:master+topic:ci-mirror-change | 15:12 |
| zigo | Anything going further would be nice at this point. | 15:12 |
| odyssey4me | zigo just seeing the files that get changed and inspecting them might give you enough to get started | 15:12 |
| zigo | Right. | 15:12 |
| zigo | odyssey4me: That public key block is abnormally big. | 15:13 |
| zigo | gpg --armor --export-options export-clean,export-minimal --export 0x<KEY-ID> is the way to do it. | 15:14 |
| zigo | Without the --export-options, then you get such large key. | 15:14 |
| odyssey4me | zigo are they? I quite possibly did something wrong there... I'm happy to receive feedback in the review for improvements :) | 15:14 |
| odyssey4me | ah ok, I did an apt-key export | 15:14 |
| *** ramishra has quit IRC | 15:14 | |
| odyssey4me | can you add that to the review please so that I can pick that change up later | 15:14 |
| * odyssey4me is technically in a meeting :p | 15:15 | |
| zigo | Here's my public key http://paste.debian.net/739700/ it's a 4096 key as well, see how smaller it is. | 15:15 |
| zigo | And I got *A LOT* of signatures on this key, plus a photo, etc. | 15:15 |
| *** ramishra has joined #openstack-infra | 15:16 | |
| *** piet has joined #openstack-infra | 15:16 | |
| zigo | With the --export-options export-clean,export-minimal, my public key is 51 lines, without, its 388111 lines... | 15:16 |
| lennyb | jordanP: in that case, please approve https://review.openstack.org/#/c/330331/2 since that fix is ok. We will see what causes 1 test to fail | 15:17 |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder: Upodate pip provider to use openstack_pip provider https://review.openstack.org/330524 | 15:17 |
| *** bhavik has joined #openstack-infra | 15:17 | |
| pabelanger | odyssey4me: zigo: Right now, we need to bike shed of how this looks in AFS. I think we agreed to deb-openstack in AFS, with that in mind, we'd basically do the same process as we do with our wheel-build / wheel-release jobs | 15:17 |
| pabelanger | wheel-build is already done by zigo | 15:18 |
| pabelanger | so the -release part is downloading the bits from tarball.o.o, copy them into /afs/.openstack/... then run afs release | 15:18 |
| *** jistr|mtg is now known as jistr | 15:19 | |
| odyssey4me | zigo thanks for the feedback :) | 15:19 |
| jeblair | pabelanger: any reason not to restart nodepool? | 15:20 |
| *** wojdec has joined #openstack-infra | 15:21 | |
| pabelanger | zigo: odyssey4me: The work I was going to do, create new builder based on wheel-build builder (jenkins/jobs/wheel-mirror.yaml), update it to download package bits into right directory, then add afs-release builder to add to AFS | 15:21 |
| *** EricGonczer_ has joined #openstack-infra | 15:21 | |
| *** maishsk has joined #openstack-infra | 15:21 | |
| pabelanger | jeblair: no, nodepool-builder failed so we are fine | 15:21 |
| pabelanger | zigo: odyssey4me: that's basically what is left to do | 15:22 |
| *** dizquierdo has joined #openstack-infra | 15:22 | |
| wojdec | Hi Folks. I've hit an issue with the Ostack gerrit system: When submitting a patch I'm told I need to provide contact information. On entering contact information via the dashboard, I get a "Server Error - Cannot Store Information" | 15:23 |
| wojdec | Is this the right forum to raise this, if not where should I report it? | 15:24 |
| *** ifarkas has joined #openstack-infra | 15:24 | |
| openstackgerrit | Merged openstack-infra/system-config: Upodate pip provider to use openstack_pip provider https://review.openstack.org/330547 | 15:25 |
| *** ihrachys has quit IRC | 15:25 | |
| *** tphummel has quit IRC | 15:25 | |
| *** ihrachys_ has joined #openstack-infra | 15:25 | |
| zigo | I'll have a look to that job. | 15:25 |
| *** jlanoux has joined #openstack-infra | 15:25 | |
| *** maishsk_ has joined #openstack-infra | 15:27 | |
| *** Swami has joined #openstack-infra | 15:27 | |
| *** maishsk has quit IRC | 15:28 | |
| *** maishsk_ is now known as maishsk | 15:28 | |
| *** Swami has quit IRC | 15:29 | |
| *** andreas_s has quit IRC | 15:29 | |
| *** wojdec has quit IRC | 15:29 | |
| *** hdaniel has quit IRC | 15:31 | |
| *** andreww has joined #openstack-infra | 15:33 | |
| *** andreww has quit IRC | 15:33 | |
| *** xarses_ has quit IRC | 15:33 | |
| fungi | wow, people /join, ask questions and then /part again 5 minutes later | 15:33 |
| *** andreww has joined #openstack-infra | 15:34 | |
| *** openstackgerrit has quit IRC | 15:34 | |
| *** GheRivero has left #openstack-infra | 15:34 | |
| fungi | before anyone has a chance to answer them | 15:34 |
| *** openstackgerrit has joined #openstack-infra | 15:34 | |
| bkero | I feel that's standard IRC newbie behavior | 15:34 |
| bkero | rule 1) don't ask to ask, rule 2) stick around for an answer | 15:34 |
| fungi | it's standard makes-me-hard-to-care behavior at any rate | 15:34 |
| fungi | er, makes -it-hard-for-me-to-care | 15:35 |
| *** mikelk has quit IRC | 15:35 | |
| *** EricGonczer_ has quit IRC | 15:35 | |
| *** vincentll has quit IRC | 15:35 | |
| odyssey4me | pabelanger ah, so perhaps the name decision needs to be something discussed and decided in the next infra meeting? | 15:35 |
| *** eezhova has quit IRC | 15:36 | |
| clarkb | pabelanger: all image builds failed this morning? | 15:36 |
| odyssey4me | then perhaps zigo can try to fathom how to do it on his own, with consultation to yourself and perhaps some others who know how the spaghetti falls :) | 15:36 |
| fungi | clarkb: probably the pip provider issue? | 15:36 |
| pabelanger | clarkb: yes, puppet apply is currently broken | 15:36 |
| clarkb | pabelanger: also I recall questions about ovh images, is it ossible that it is related to the switch to qcow2 over raw? | 15:36 |
| pabelanger | working on fixing it | 15:36 |
| pabelanger | clarkb: Ohh..... maybe. I think I might know the problem | 15:37 |
| pabelanger | clarkb: I need to check something | 15:37 |
| *** _nadya_ has quit IRC | 15:37 | |
| pabelanger | crinkle: phschwartz: possible bug in openstack_pip? http://logs.openstack.org/28/330528/1/check/gate-infra-puppet-apply-ubuntu-trusty/7fb30a9/console.html#_2016-06-16_14_48_17_844 | 15:37 |
| fungi | clarkb: short story is that pypi switched its redirect behavior for http->https, which broke puppet<=3.x pip package provider's ability to scrape the xml api | 15:37 |
| *** eharney_ has joined #openstack-infra | 15:38 | |
| clarkb | fun | 15:38 |
| mordred | clarkb: yes. fun is a word in the english language | 15:38 |
| mordred | pabelanger: I agree- looks like a bug | 15:38 |
| *** Guest58170 has quit IRC | 15:38 | |
| fungi | clarkb: there are i hear fixes being proposed upstream in puppet for it, but also crinkle's rewritten pip provider is sane and does not suffer this blight so we're switching everything to that | 15:38 |
| *** Hal has joined #openstack-infra | 15:38 | |
| clarkb | new house gets internets today and we have started migrating boxes of stuff so likely not to be here long | 15:39 |
| *** eharney has quit IRC | 15:39 | |
| mordred | clarkb: just use containers | 15:39 |
| *** Hal is now known as Guest41851 | 15:39 | |
| clarkb | mordred: yes if I had each room of my house deployed as a cloud native containeried application moving would be so easy | 15:39 |
| greghaynes | Ya, containers are good for shipping I hear, maybe they work for moving too | 15:39 |
| fungi | boxes are containers, and a house is really just a bigger container to put your smaller containers in | 15:39 |
| phschwartz | pabelanger: hmm, I am looking and to note that didn't error on my review to puppet-diskimage_builder and system-config | 15:39 |
| clarkb | I could just clone all of my furniture, ship them to new location, and have an orchestration engine migrate for me | 15:39 |
| pabelanger | clarkb: no, I think raw images are fine. I looked just now and images uploaded properly and displayed in nodepool correctly today | 15:40 |
| clarkb | pabelanger: you mean qcow2s are ok? | 15:40 |
| pabelanger | clarkb: sorry, yes qcow2 | 15:40 |
| clarkb | kk | 15:40 |
| mordred | phschwartz: perhaps the others didn't use the ensure => parameter? | 15:40 |
| pabelanger | I manually upload ubuntu-trusty yesterday for AJaeger. Then nodepool did its uploads too | 15:41 |
| pabelanger | mordred: I think that is correct | 15:41 |
| pabelanger | phschwartz: mordred: ansible-role-puppet should fail too because we set ensure => '2.0.2.0' | 15:41 |
| pabelanger | or 2.1.0.0 | 15:41 |
| phschwartz | mordred: yeah, the others were ensure => latest | 15:42 |
| *** EricGonczer_ has joined #openstack-infra | 15:42 | |
| *** maishsk has quit IRC | 15:42 | |
| nibalizer | phschwartz: did you see that https://review.openstack.org/#/c/330528/ failed | 15:42 |
| phschwartz | mordred: so I will need to look at crinkle's code to see if it handles it correctly | 15:42 |
| *** kzaitsev_mb has joined #openstack-infra | 15:42 | |
| jeblair | pabelanger: restarted nodepool | 15:42 |
| phschwartz | nibalizer: yeah, that is what we are discussing right now | 15:43 |
| fungi | aha, looks like we have a bunch of stale ready nodes on jenkins05, 06, 07 and zl01 | 15:43 |
| *** eezhova has joined #openstack-infra | 15:43 | |
| jeblair | fungi: maybe the nodepool restart will take care of the jenkinses too | 15:43 |
| phschwartz | nibalizer: looks like crinkle's openstack_pip provider doesn't support versioned installs only latest. | 15:43 |
| *** markusry has quit IRC | 15:43 | |
| fungi | jeblair: entirely possible | 15:43 |
| jeblair | pabelanger, fungi: can you clean up those stale nodes while i go test my zuul patches? | 15:43 |
| fungi | jeblair: we also have a bunch of stale delete nodes on the jenkins masters | 15:44 |
| *** eharney_ is now known as eharney | 15:44 | |
| pabelanger | jeblair: sure | 15:44 |
| fungi | jeblair: so i'm worried that "cleanup" involving nodepool delete will just mean more stuck deletes | 15:44 |
| pabelanger | fungi: ya, I notices that today too. Think our clouds are having some hiccups this morning | 15:44 |
| *** shahid_ has quit IRC | 15:45 | |
| *** tphummel has joined #openstack-infra | 15:45 | |
| fungi | i guess if we delete the jenkins-assigned ones from nodepool's db, then the alien cleanup pass will get rid of them | 15:45 |
| fungi | ? | 15:45 |
| *** kzaitsev_mb has quit IRC | 15:46 | |
| pabelanger | I usually grep for used nodes that are older then 3 hours, then delete them. But would be curious if alien cleanup does the same thing | 15:46 |
| *** johnny_ has joined #openstack-infra | 15:47 | |
| fungi | the alien cleanup will only remove instances which are not in nodepool's db | 15:47 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/system-config: OpenStack-CI Apt mirror for MariaDB https://review.openstack.org/307831 | 15:47 |
| fungi | it's basically node leak mitigation | 15:47 |
| *** roxanaghe has joined #openstack-infra | 15:48 | |
| nibalizer | so I don't really understand this subsystem but this is an odd bug for sure | 15:48 |
| nibalizer | because all the openstack_pip provider is supposed to be doing is just inheriting the pip provider and overwriting latest | 15:48 |
| nibalizer | overriding? | 15:48 |
| fungi | actually, the glut of stuck deletes on jenkinses seems to be clearing up now | 15:49 |
| *** mdrabe has quit IRC | 15:49 | |
| fungi | pabelanger: were you working on cleaning up stale ready nodes, or shall i? | 15:50 |
| crinkle | phschwartz: it inherits from the upstream one and only overrides the latest method | 15:50 |
| crinkle | it otherwise works the same | 15:50 |
| *** reduxio has joined #openstack-infra | 15:51 | |
| nibalizer | crinkle: yah? | 15:51 |
| nibalizer | maybe the capabilities statement doesn't get inherited? | 15:51 |
| reduxio | Hello, Can anyone help with third party CI setup? | 15:51 |
| nibalizer | im running a test with just declaring the capabilities in the openstack_pip provider :shrug: | 15:51 |
| *** ddieterly is now known as ddieterly[away] | 15:51 | |
| crinkle | nibalizer: oh you said that | 15:51 |
| crinkle | nibalizer: uh i'm not sure | 15:51 |
| *** ddieterly[away] is now known as ddieterly | 15:51 | |
| phschwartz | crinkle: ok, so that is where the issue is. The upstream one checks for ensure to be a value or latest and performs an action based on that. The openstack_pip doesn't ever look at the value from what it looks like | 15:52 |
| phschwartz | crinkle: nibalizer: https://github.com/puppetlabs/puppet/blob/3.8.7/lib/puppet/provider/package/pip.rb#L92-L96 | 15:52 |
| crinkle | phschwartz: like i said this one is just a subclass of the upstream one | 15:52 |
| jeblair | fungi: i note zmq events from jenkins07 now | 15:52 |
| *** roxanaghe has quit IRC | 15:53 | |
| *** sorantis has quit IRC | 15:53 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-pip: Declare capabilities in openstack_pip provider https://review.openstack.org/330657 | 15:53 |
| nibalizer | crinkle: so in one quick test ^^ that does work | 15:53 |
| crinkle | phschwartz: the latest method only checks what the latest available is, it doesn't mess with the install method | 15:53 |
| crinkle | nibalizer: okay | 15:54 |
| *** coreyob has quit IRC | 15:54 | |
| nibalizer | crinkle: idunno, feels wrong to do it that way? | 15:54 |
| nibalizer | or put another way, it feels like we shouldn't have to do that | 15:54 |
| fungi | pabelanger: i'm going to go ahead and delete the stale ready nodes (any in a ready state for >10 minutes) | 15:54 |
| crinkle | nibalizer: no i think it makes sense since we have the commands thing there too | 15:54 |
| nibalizer | :shrug: ok | 15:54 |
| nibalizer | should i update and remove the question mark from the commit message? | 15:55 |
| crinkle | nibalizer: yes please | 15:55 |
| mordred | nibalizer: I dunno. I like commits that ask questions | 15:55 |
| nibalizer | kk | 15:55 |
| *** isaacb has quit IRC | 15:56 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-pip: Declare features in openstack_pip provider https://review.openstack.org/330657 | 15:56 |
| *** e0ne has quit IRC | 15:56 | |
| pabelanger | fungi: becareful of zl01, they are likely running jobs | 15:56 |
| pabelanger | fungi: okay, looks like zl01 is good | 15:57 |
| nibalizer | crinkle: mordred ^ | 15:57 |
| *** fernnest has joined #openstack-infra | 15:57 | |
| *** andreww is now known as xarses | 15:57 | |
| mordred | nibalizer: yup. it has been +Ad | 15:57 |
| fungi | pabelanger: oh, were we still not getting zmq events from zl01? i thought that had been resolved | 15:57 |
| mordred | thanks! | 15:57 |
| reduxio | anteaya: Hi anteaya, i was told you may be able to help me with setting up third party CI, or at least point me in the right direction? | 15:57 |
| pabelanger | fungi: Ya, looks like it stopped correctly before your deletes | 15:58 |
| pabelanger | jeblair: I'll hold off on restart zl01 until you are ready | 15:58 |
| jeblair | pabelanger: i'm ready | 15:58 |
| pabelanger | ack | 15:58 |
| *** akshai has joined #openstack-infra | 15:58 | |
| *** Apoorva has joined #openstack-infra | 15:58 | |
| pabelanger | zl01 restarted | 15:59 |
| nibalizer | huh, my gerrit now longer shows the table of votes? | 15:59 |
| clarkb | nibalizer: you are probably on an older patchset then | 15:59 |
| *** bhavik1 has joined #openstack-infra | 16:00 | |
| nibalizer | oh | 16:00 |
| nibalizer | </3 gerrit | 16:00 |
| *** bpokorny has joined #openstack-infra | 16:00 | |
| *** tesseract has quit IRC | 16:00 | |
| *** reduxio has quit IRC | 16:00 | |
| *** mriedem1 has joined #openstack-infra | 16:00 | |
| *** kaisers_ has quit IRC | 16:01 | |
| *** dprince has quit IRC | 16:01 | |
| *** akshai_ has quit IRC | 16:01 | |
| *** dprince has joined #openstack-infra | 16:02 | |
| *** mriedem has quit IRC | 16:02 | |
| *** bhavik has quit IRC | 16:02 | |
| *** samuelBartel has quit IRC | 16:02 | |
| *** bhavik1 is now known as bhavik | 16:02 | |
| *** mriedem1 is now known as mriedem | 16:03 | |
| *** bpokorny has quit IRC | 16:05 | |
| *** bpokorny has joined #openstack-infra | 16:06 | |
| *** lucasagomes is now known as lucas-afk | 16:06 | |
| *** gyee has joined #openstack-infra | 16:06 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a temporary tripleo cloud "rh2" https://review.openstack.org/330514 | 16:07 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a experimental tripleo job on rh2 https://review.openstack.org/330515 | 16:07 |
| *** hashar is now known as hasharAway | 16:07 | |
| rcarrillocruz | nibalizer , clarkb , mordred : when you get a sec, https://review.openstack.org/#/c/327789/4 | 16:08 |
| *** mescanef has quit IRC | 16:08 | |
| rcarrillocruz | been testing it with logdest, without, with /var/log/ansible-puppet and without | 16:09 |
| *** javeriak has joined #openstack-infra | 16:09 | |
| *** ansiwen_ is now known as ansiwen | 16:10 | |
| *** akshai_ has joined #openstack-infra | 16:10 | |
| *** roxanaghe has joined #openstack-infra | 16:10 | |
| *** mdrabe has joined #openstack-infra | 16:11 | |
| *** akshai has quit IRC | 16:13 | |
| *** javeriak_ has joined #openstack-infra | 16:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add Zuul as a first-party CI name https://review.openstack.org/330671 | 16:14 |
| jeblair | mordred, fungi, pabelanger: ^ i created a Zuul account in gerrit. this should let us start using that to report on changes | 16:15 |
| *** kzaitsev_mb has joined #openstack-infra | 16:15 | |
| *** javeriak has quit IRC | 16:15 | |
| *** permalac has quit IRC | 16:15 | |
| clarkb | you don't want to keep "jenkins" around in name if not executables? | 16:15 |
| fungi | clarkb: we could switch back to hudson | 16:16 |
| jeblair | heh | 16:16 |
| pabelanger | jeblair: nice | 16:16 |
| jeblair | i think that the time when we stop running jenkins is a good time to make the gerrit account name switch | 16:16 |
| jeblair | (while it would be amusing to keep calling it jenkins, it could be confusing) | 16:17 |
| *** coreyob has joined #openstack-infra | 16:17 | |
| mordred | jeblair: ++ | 16:19 |
| pabelanger | I still need to update out dib elements for devstack cache adding support for /home/zuul | 16:19 |
| pabelanger | should dig into that later today | 16:19 |
| *** johnny_ has quit IRC | 16:19 | |
| *** dprince has quit IRC | 16:20 | |
| *** sdake has quit IRC | 16:20 | |
| *** maishsk has joined #openstack-infra | 16:21 | |
| openstackgerrit | Merged openstack-infra/puppet-log_processor: Update pip provider to use openstack_pip provider https://review.openstack.org/330595 | 16:21 |
| *** dprince has joined #openstack-infra | 16:21 | |
| nibalizer | jeblair: so lets rename it to hudson... | 16:22 |
| *** ramishra has quit IRC | 16:22 | |
| jeblair | nibalizer: fungi already made that joke. ;) | 16:22 |
| nibalizer | dangit | 16:22 |
| openstackgerrit | Scott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend https://review.openstack.org/330678 | 16:22 |
| * nibalizer hangs head in shame | 16:23 | |
| * fungi prides himself at being quick with terrible jokes | 16:23 | |
| openstackgerrit | Merged openstack-infra/puppet-lodgeit: Update pip provider to use openstack_pip provider https://review.openstack.org/330594 | 16:24 |
| openstackgerrit | Merged openstack-infra/puppet-meetbot: Update pip provider to use openstack_pip provider https://review.openstack.org/330597 | 16:24 |
| jeblair | mordred: hrm, i don't think there's a good way to handle the name change for the 'clean-check' gate pipeline requirement. | 16:24 |
| openstackgerrit | Merged openstack-infra/puppet-gerritbot: Update pip provider to use openstack_pip provider https://review.openstack.org/330591 | 16:24 |
| *** jsavak has quit IRC | 16:24 | |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Update pip provider to use openstack_pip provider https://review.openstack.org/330599 | 16:24 |
| mordred | jeblair: hrm | 16:24 |
| mordred | jeblair: that sucks a little bit | 16:24 |
| jeblair | mordred: it will require us to either synthetically create a bunch of zuul votes that == jenkins votes in gerrit, or alter zuul to support regexes in usernames | 16:25 |
| mtreinish | jeblair: if you rename the jenkins user that will probably break e-r: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/elastic_recheck/elasticRecheck.py#n213 | 16:25 |
| mtreinish | there might be other things out there, which are looking for something similar too | 16:25 |
| jeblair | mtreinish: i'm sure there are | 16:26 |
| *** ramishra has joined #openstack-infra | 16:26 | |
| *** jsavak has joined #openstack-infra | 16:27 | |
| openstackgerrit | Scott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend https://review.openstack.org/330678 | 16:28 |
| *** cbader has joined #openstack-infra | 16:28 | |
| *** sdake has joined #openstack-infra | 16:28 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul: Use safe_dump() for Ansible playbooks https://review.openstack.org/330683 | 16:29 |
| *** yamamoto has quit IRC | 16:29 | |
| *** ddieterly is now known as ddieterly[away] | 16:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/elastic-recheck: Support Zuul as a Gerrit user equivalent to Jenkins https://review.openstack.org/330684 | 16:29 |
| jeblair | mtreinish: ^ | 16:29 |
| mtreinish | jeblair: actually asselin has a patch which makes that a config var: https://review.openstack.org/#/c/251515/ | 16:31 |
| mtreinish | we probably should just land that and update it in the puppet | 16:31 |
| *** dtantsur|bbl is now known as dtantsur | 16:32 | |
| mtreinish | although your patch will have a more graceful transition | 16:32 |
| jeblair | mtreinish: okay. since that's a binary, we'll probably have a short 'outage' during the transition. but we can try to land it close to the event | 16:32 |
| *** esikachev has quit IRC | 16:32 | |
| *** matrohon has quit IRC | 16:33 | |
| *** ramishra has quit IRC | 16:34 | |
| mtreinish | jeblair: we can also land your patch first, and switch to asselin's after the rename is done | 16:34 |
| mtreinish | doesn't bug me either way | 16:34 |
| *** david-lyle has quit IRC | 16:35 | |
| *** EricGonc_ has joined #openstack-infra | 16:35 | |
| jeblair | mtreinish: i +2d asselin's. that plan sounds like the best approach (no critical timing). your call. | 16:36 |
| *** pvaneck has joined #openstack-infra | 16:36 | |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Update pip provider to use openstack_pip provider https://review.openstack.org/330590 | 16:36 |
| openstackgerrit | Merged openstack-infra/puppet-jeepyb: Update pip provider to use openstack_pip provider https://review.openstack.org/330592 | 16:37 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch back to current-tripleo pre promotion https://review.openstack.org/111011 | 16:37 |
| *** EricGonczer_ has quit IRC | 16:37 | |
| *** jordanP has quit IRC | 16:37 | |
| mtreinish | jeblair: I just fast approved your change, I'll handle rebasing and approving asselin's change after the rename is done | 16:38 |
| *** yamahata has joined #openstack-infra | 16:39 | |
| *** jlanoux has quit IRC | 16:39 | |
| pabelanger | fungi: are you okay adding python3 wheels (329094) back into the mix? python2 job is working correctly again: http://logs.openstack.org/12/12c28df295d7e0039339706ac5e3c2ce5b99a4f3/release/wheel-build-ubuntu-trusty-amd64-python2/e68ae0e/ | 16:39 |
| *** ramishra has joined #openstack-infra | 16:40 | |
| *** amitgandhinz has quit IRC | 16:40 | |
| jeblair | mtreinish: cool, thanks | 16:40 |
| pabelanger | fungi: just confirming scripts are correct again on 329094 | 16:41 |
| *** amitgandhinz has joined #openstack-infra | 16:41 | |
| fungi | pabelanger: yeah, my only remaining concern is that they may race and step on each other since for universal wheels both jobs would create a version of the same file | 16:41 |
| jeblair | anyone seen sdague lately? | 16:41 |
| pabelanger | fungi: Right, but since we have only 1 slave atm, is that going to be the case? | 16:41 |
| fungi | though i'm not sure how to tell the python 3 job to only create wheels for non-universal | 16:42 |
| pabelanger | fungi: Oh, I see what you are saying | 16:42 |
| fungi | pabelanger: oh, fair point they run on the same server. so worst is that we unnecessarily create a lot of the same wheels multiple times | 16:42 |
| pabelanger | fungi: ya | 16:42 |
| *** dmk0202 has quit IRC | 16:42 | |
| pabelanger | would be a good optimization to disable the dupe packages between python2 / python3 | 16:42 |
| fungi | there's also the corner case that supposedly universal wheels created under python 3 don't behave like the ones created under python 2 and that we'd want to prefer the latter over the former | 16:43 |
| fungi | i have no idea if that's actually a risk though | 16:43 |
| pabelanger | right | 16:43 |
| pabelanger | me neither | 16:43 |
| pabelanger | wheel-build-ubuntu-trusty-amd64-python3 looks correct | 16:44 |
| pabelanger | just confirmed locally | 16:44 |
| *** erlon has joined #openstack-infra | 16:44 | |
| fungi | cool, jobs passed for 330624. +2 from me | 16:44 |
| pabelanger | thanks | 16:44 |
| pabelanger | fungi: 329094 is python3 wheel | 16:46 |
| fungi | i've still been waiting for a lull on proposal.slave.o.o but the backlog in post probably means that won't happen for quite a while | 16:46 |
| *** rossella_s has joined #openstack-infra | 16:46 | |
| fungi | pabelanger: yeah, i was talking about the static worker job registration fix for zuul | 16:46 |
| fungi | that's in the gate now | 16:46 |
| pabelanger | fungi: Ah | 16:46 |
| *** EricGonc_ has quit IRC | 16:46 | |
| pabelanger | sorry for the noise | 16:46 |
| fungi | i'll take a look at 329094 shortly | 16:46 |
| fungi | need to heat up some lunch first | 16:47 |
| *** mixos has joined #openstack-infra | 16:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Make pipeline approval username requirement a regular expression https://review.openstack.org/330696 | 16:47 |
| *** eezhova has quit IRC | 16:47 | |
| jeblair | mordred, notmorgan: ^ | 16:47 |
| fungi | pabelanger: since i think the wheelhouse on that node persists between wheel build jobs, i _hope_ that by running the python 2 job first we ensure that the python 3 job doesn't bother building any universal wheels because it finds them already present in the wheelhouse | 16:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Change Zuul's username to zuul https://review.openstack.org/330698 | 16:49 |
| jeblair | mordred, notmorgan: ^ and that | 16:49 |
| jeblair | fungi: where's the wheelhouse? | 16:50 |
| *** david-lyle has joined #openstack-infra | 16:50 | |
| *** EricGonczer_ has joined #openstack-infra | 16:50 | |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: update registration between jobs https://review.openstack.org/330624 | 16:50 |
| jeblair | fungi: i updated some long running jobs to rm -rf /home/jenkins/workspace at the beginning of each job | 16:50 |
| jeblair | fungi: i don't know if it was on the wheel builders or not | 16:51 |
| fungi | jeblair: oh, no it's not persisted between jobs. we explicitly blow it away each run http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/wheel-mirror.yaml | 16:51 |
| fungi | pabelanger: ^ | 16:51 |
| fungi | so much for that idea | 16:52 |
| *** nwkarsten has quit IRC | 16:52 | |
| fungi | i wonder if we should reconsider that | 16:52 |
| fungi | it's passed in as WHEEL_DIR=/opt/wheel/workspace | 16:52 |
| jeblair | fungi: it was only on the proposal slave that i do the rm workspace | 16:52 |
| jeblair | fungi: so that should be fine if we want to reconsider that | 16:53 |
| openstackgerrit | Merged openstack-infra/puppet-kerberos: Don't have the server install the client https://review.openstack.org/305968 | 16:53 |
| openstackgerrit | Merged openstack-infra/puppet-logstash: Update pip provider to use openstack_pip provider https://review.openstack.org/330596 | 16:53 |
| fungi | well, even if it were on the proposal slave, we specify a directory outside the job workspace | 16:53 |
| jeblair | ok | 16:53 |
| fungi | so clearing the job workspace wouldn't impact the wheel cache | 16:54 |
| openstackgerrit | Scott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend https://review.openstack.org/330678 | 16:54 |
| openstackgerrit | Merged openstack-infra/puppet-meetbot: Fix deprecated warnings https://review.openstack.org/322320 | 16:54 |
| *** esikachev has joined #openstack-infra | 16:54 | |
| *** ddieterly[away] is now known as ddieterly | 16:54 | |
| *** ramishra has quit IRC | 16:55 | |
| openstackgerrit | Merged openstack-infra/puppet-openafs: Increase memory available to network stack on fileservers https://review.openstack.org/314707 | 16:55 |
| notmorgan | jeblair: oh nice | 16:55 |
| notmorgan | jeblair: so we know which is commenting | 16:55 |
| fungi | notmorgan: not really. it was always zuul commenting | 16:55 |
| notmorgan | fungi: right, but in this case we could make it anything | 16:56 |
| fungi | or at least it was after we stopped using the gerrit plugin for jenkins | 16:56 |
| notmorgan | fungi: rather than must be jenkins named | 16:56 |
| notmorgan | it could be *any* tool as long as the account matches, e.g. having multiple/different tools for different things (in theory) | 16:56 |
| jeblair | notmorgan: we can currently make it anything, it's just a string match, so we only get to make it one thing. this lets us make it two things at once. :) | 16:56 |
| notmorgan | jeblair: yeah this fits our needs for sure here :) | 16:57 |
| fungi | easier transition this way | 16:57 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute031.uswest metadata for better tracking https://review.openstack.org/330390 | 16:57 |
| mtreinish | jeblair: iirc, sdague is going camping over the weekend and said today was prep day | 16:57 |
| jeblair | also, i think that's just about the last thing in that part of the config that *isn't* a regex | 16:57 |
| * notmorgan gets caffiene... | 16:57 | |
| mordred | jeblair: \o/ | 16:57 |
| jeblair | mtreinish: oh fun! well, there might be some dashboards that need a'fixin when he gets back | 16:58 |
| mordred | jeblair: what email address is it using? | 16:59 |
| *** derekh has quit IRC | 16:59 | |
| fungi | yay! i caught the proposal.slave.o.o idle and reconfigured zl on zlstatic01 | 16:59 |
| jeblair | mordred: it does not have a preferred email address (like jenkins). it does have one though. why? | 16:59 |
| fungi | and now those jobs have workers | 16:59 |
| jeblair | mordred: (this way, the email address does not show up in the display) | 16:59 |
| *** PollyZ_ has joined #openstack-infra | 16:59 | |
| fungi | in gearman | 16:59 |
| mordred | jeblair: mostly wondering what email will show up on the merge commits | 17:00 |
| jeblair | fungi: cool, sounds like confirmation we're on the right track | 17:00 |
| jeblair | mordred: i put zuul@zuul.openstack.org | 17:00 |
| mordred | cool. I will add that to the robot exclusion list in stackalytics | 17:00 |
| *** wojdec has joined #openstack-infra | 17:00 | |
| *** mhickey has joined #openstack-infra | 17:00 | |
| jeblair | we might update that to zuul@openstack.org | 17:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-github: Update pip provider to use openstack_pip provider https://review.openstack.org/330702 | 17:00 |
| mordred | I'll add both :) | 17:00 |
| pabelanger | phschwartz: ^ | 17:00 |
| jeblair | mordred: i'll go add an alias for zuul, and switch it to zuul@openstack.org | 17:01 |
| *** EricGonczer_ has quit IRC | 17:01 | |
| *** kushal has joined #openstack-infra | 17:02 | |
| *** PollyZ has quit IRC | 17:02 | |
| *** yamamoto has joined #openstack-infra | 17:05 | |
| odyssey4me | pabelanger thanks for the review of https://review.openstack.org/330152 - I've replied with a question for clarity when you have a moment | 17:05 |
| *** xarses has quit IRC | 17:05 | |
| jeblair | mordred, pleia2: created zuul@openstack.org alias for infra-root along with a server-side filter | 17:06 |
| *** xarses has joined #openstack-infra | 17:06 | |
| *** akshai_ has quit IRC | 17:06 | |
| fungi | jeblair: further confirmation, shortly after workers showed up in gearman for those jobs they ran and cleared out of zuul's queues | 17:06 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute016.useast metadata for better tracking https://review.openstack.org/330407 | 17:07 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute086.useast metadata for better tracking https://review.openstack.org/330412 | 17:07 |
| *** daemontool has quit IRC | 17:08 | |
| mordred | jeblair: I've submitted a stackalytics change to add zuul@ and zuul@zuul to the list: https://review.openstack.org/330711 | 17:08 |
| *** trown is now known as trown|lunch | 17:08 | |
| *** PollyZ_ has quit IRC | 17:08 | |
| *** PollyZ has joined #openstack-infra | 17:08 | |
| *** ifarkas has quit IRC | 17:08 | |
| *** salv-orl_ has joined #openstack-infra | 17:08 | |
| asselin | mtreinish, jeblair default name 'jenkins' no longer valid for upstream? | 17:09 |
| pabelanger | odyssey4me: replied | 17:09 |
| jeblair | asselin: for now it is, we will be changing it soon | 17:09 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute089.useast metadata for better tracking https://review.openstack.org/330409 | 17:09 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute085.useast metadata for better tracking https://review.openstack.org/330416 | 17:10 |
| jeblair | mordred: and updated in gerrit | 17:10 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute087.useast metadata for better tracking https://review.openstack.org/330418 | 17:10 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute084.useast metadata for better tracking https://review.openstack.org/330420 | 17:10 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute013.useast metadata for better tracking https://review.openstack.org/330422 | 17:10 |
| mtreinish | asselin: yeah, they're switching it to 'zuul' | 17:10 |
| mordred | asselin: we've very close to not having any software running named jenkins | 17:11 |
| pabelanger | mordred: \o/ | 17:11 |
| fungi | this could probably stand a heads-up notice to the -dev ml because i bet a lot of people have e-mail filters and stuff that assume the old name/address | 17:11 |
| *** roxanaghe has joined #openstack-infra | 17:11 | |
| jeblair | fungi: i'm planning on sending one soon | 17:11 |
| fungi | i figured! | 17:11 |
| * fungi notes to adjust his own .forward filter file | 17:12 | |
| *** salv-orlando has quit IRC | 17:12 | |
| jeblair | i'm thinking: finish the transition; once no jenkins servers are running, send an announcement (i have one prepped); then we can effect the name change after that | 17:12 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute012.useast metadata for better tracking https://review.openstack.org/330423 | 17:13 |
| jeblair | we could do the name change tomorrow during the zuul maintenance, or we could wait longer if we think that's necessary | 17:13 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute015.useast metadata for better tracking https://review.openstack.org/330424 | 17:13 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute009.useast metadata for better tracking https://review.openstack.org/330430 | 17:13 |
| *** Daisy has joined #openstack-infra | 17:13 | |
| openstackgerrit | Merged openstack-infra/system-config: Add compute010.useast metadata for better tracking https://review.openstack.org/330426 | 17:13 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute011.useast metadata for better tracking https://review.openstack.org/330435 | 17:13 |
| mordred | jeblair: think it's worth it, while we're doing name changes, to make a zuul account in launchpad and replace hudson-openstack? | 17:13 |
| fungi | if pabelanger hasn't written up the announcement for tomorrow yet, we may want to reschedule out a week | 17:13 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute017.useast metadata for better tracking https://review.openstack.org/330437 | 17:13 |
| jeblair | mordred: zuul doesn't touch launchpad | 17:14 |
| pabelanger | speaking of that, I have the etherpad here: https://etherpad.openstack.org/p/upgrade-zuul-trusty but we still haven't sent it | 17:14 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute019.useast metadata for better tracking https://review.openstack.org/330440 | 17:14 |
| fungi | it's getting to be pretty short notice at this point | 17:14 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute026.useast metadata for better tracking https://review.openstack.org/330445 | 17:14 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute020.useast metadata for better tracking https://review.openstack.org/330441 | 17:14 |
| pabelanger | ya, I left data empty so we can revisit it | 17:14 |
| pabelanger | date* | 17:14 |
| openstackgerrit | Merged openstack-infra/system-config: Fix baremetal00 name attribute still referencing west https://review.openstack.org/330560 | 17:14 |
| mordred | jeblair: right. but gerrit does, and it uses the hudson-openstack account to do it - so I guess we could change the account to openstack-gerrit instead of openstack-zuul | 17:14 |
| fungi | mordred: i'd rather hold on to hope that we simply won't need an account in launchpad for our automation at some point in the future and leave it alone for now. adjusting membership in everybody's bug groups would be a pain | 17:14 |
| mordred | fungi: good call | 17:14 |
| fungi | that would be a lot of cat herding since i doubt they're all owned by openstack-admins | 17:15 |
| *** roxanaghe has quit IRC | 17:16 | |
| *** BobBall is now known as BobBall_AWOL | 17:16 | |
| pabelanger | fungi: see eitherpad above, but pusing back to next week (R-15) seems to be clear for newton | 17:16 |
| *** Daisy has quit IRC | 17:18 | |
| jeblair | pabelanger: what hosts are we upgrading? | 17:18 |
| fungi | zuul and static | 17:18 |
| *** _ari_ has joined #openstack-infra | 17:18 | |
| pabelanger | jeblair: zuul.o.o and static.o.o need to upgraded | 17:18 |
| jeblair | static. that's the one i was forgetting that makes this complex :) | 17:18 |
| fungi | static is the bigger concern since we need to take everything offline and move 14 cinder volumes | 17:18 |
| fungi | we could probably do the zuul upgrade fairly quickly/seamlessly by comparison | 17:19 |
| pabelanger | agreed | 17:19 |
| *** amitgandhinz has quit IRC | 17:19 | |
| jeblair | which is terrifying since the last time we tried to detach a cinder volume it took 8 hours to fail | 17:19 |
| fungi | but if we can't run any jobs while static.o.o is being moved, then might as well have someone else upgrade zuul at teh same time | 17:19 |
| _ari_ | zxiiro: I keep hearing rumors that JJB is going to go away is this true? | 17:20 |
| pabelanger | Ya, I would recommend we stop the server over detaching while it is running | 17:20 |
| jeblair | _ari_: it is not true | 17:20 |
| *** amitgandhinz has joined #openstack-infra | 17:20 | |
| _ari_ | jeblair: phew | 17:20 |
| fungi | jeblair: well, my suggestion to halt the server worked when pabelanger tried it, after we gave up waiting for the fanatics to respond | 17:20 |
| mordred | _ari_: we are eventually going to stop using it in infra - JJB itself has a lovely team of dedicated humans | 17:20 |
| pabelanger | fungi: ++ | 17:20 |
| *** dtantsur is now known as dtantsur|afk | 17:20 | |
| jeblair | _ari_: so it will live on with our support as long as they want it | 17:20 |
| *** sambetts is now known as sambetts|afk | 17:21 | |
| *** rcernin has quit IRC | 17:21 | |
| fungi | _ari_: at the moment, jjb is actually being used directly by zuul 2.5 and translated into ansible plays | 17:21 |
| _ari_ | jeblair: mordred So I heard there is a plan to replace Jenkins and use Zuul + Ansible | 17:21 |
| zxiiro | _ari_: I hope not, we (the linux foundation) depend on JJB so we plan on keeping it supported | 17:22 |
| _ari_ | So what is the need to get rid of Jenkins out of curiosity | 17:22 |
| fungi | _ari_: so it's technically now the "jenkins and zuul job builder" ;) | 17:22 |
| _ari_ | lol | 17:22 |
| jeblair | _ari_: yes, we are in the process of replacing jenkins with zuul now, with playbooks created by jjb. in zuulv3, we will stop using jjb and write playbooks directly as job definitions | 17:22 |
| *** eharney has quit IRC | 17:23 | |
| *** bauzas has quit IRC | 17:23 | |
| pabelanger | _ari_: see http://logs.openstack.org/92/330192/2/check/gate-kolla-dsvm-build-oraclelinux-binary/3bfdbc7/ for an example of zuul-launcher (ansible). Note the _zuul_ansible folder to see out ansible bits | 17:23 |
| fungi | but yeah, we hope jjb continues to have a long life even once we're no longer in need of it for the openstack infrastructure. if anything, the current jjb contributors will have greater freedom and autonomy to take the project in the directions they want without worry for our possibly esoteric use cases | 17:23 |
| jeblair | zxiiro: ^ | 17:23 |
| mordred | _ari_: jenkins is complex in support of a ton of features we don't and can't use. it's not designed for our kind of scale and it keeps breaking. we've been working towards moving off of it for about 4 years now, but it's along and involved process, given the central role of the gate | 17:23 |
| _ari_ | jeblair: So is the plan to reproduce the functionality that you already get with Jenkins plugins with playsbooks? | 17:23 |
| mordred | _ari_: we do not really use jenkins plugins | 17:24 |
| *** bauzas has joined #openstack-infra | 17:24 | |
| mordred | for us, jenkins is a remote execution engine that is driven by REST calls | 17:24 |
| _ari_ | mordred: ah I see | 17:24 |
| *** ddieterly is now known as ddieterly[away] | 17:24 | |
| _ari_ | pabelanger: thanks! | 17:24 |
| fungi | we basically use jenkins to run scripts on job workers, and collect results/logs | 17:24 |
| jeblair | _ari_: the only plugins we use are the scp and ftp publishers | 17:24 |
| *** ihrachys_ has quit IRC | 17:24 | |
| zxiiro | _ari_: my organizations use many Jenkins plugihns so we can't move away from JJB so we will be providing support it for even if openstack stops | 17:25 |
| mordred | yay for open source! | 17:25 |
| pabelanger | I really like how JJB has taken on life of its own. | 17:25 |
| mordred | me too | 17:25 |
| zxiiro | one thing that concerns me a bit that came to my attention yesterday was this https://pypi.python.org/pypi/jenkins-job-builder-addons | 17:25 |
| _ari_ | zxiiro: jeblair I used many Jenkins plugins as well and it gives a lot of hooks into different infrastructures like github, Amazon S3,Docker, etc. but I can see if you don't use those why it is not worth while | 17:26 |
| zxiiro | I'd rather see the author try to push those features into JJB proper rather than providing a separate package | 17:26 |
| *** wojdec1 has joined #openstack-infra | 17:26 | |
| _ari_ | zxiiro: +1 | 17:26 |
| pabelanger | zxiiro: ouch | 17:26 |
| jeblair | zxiiro: well, we did design it to use entry points to facilitate that sort of thing | 17:26 |
| mat128 | zxiiro: https://github.com/jimbydamonk/jenkins-job-builder-addons/issues/2 | 17:26 |
| *** PollyZ has quit IRC | 17:27 | |
| *** wojdec has quit IRC | 17:27 | |
| jeblair | mostly so that people could have site-local things | 17:27 |
| *** boltR_ has quit IRC | 17:27 | |
| fungi | but yeah, addons make sense if you're only using them internally. at the point where you're going to publish them, might as well sibmit a patch to jjb? | 17:27 |
| jeblair | but also, for more esoteric plugins | 17:27 |
| fungi | s/sibmit/submit/ | 17:27 |
| jeblair | we used to not accept jjb support for non-open-sounce jenkins plugins | 17:28 |
| jeblair | i think that slipped when i started reviewing less :) | 17:28 |
| *** akshai has joined #openstack-infra | 17:28 | |
| _ari_ | fungi: definitely it is very easy to extend and contribute the plugin feature upstream | 17:28 |
| *** inc0 has quit IRC | 17:28 | |
| *** eharney has joined #openstack-infra | 17:28 | |
| _ari_ | zxiiro: one question that has been asked by me and others is where does the pipeline plugin come in to JJB since they no longer store XML on disk for jobs any longer | 17:29 |
| pabelanger | eager to see if we are going to have a plugin system for zuulv3+ | 17:29 |
| *** inc0 has joined #openstack-infra | 17:29 | |
| mordred | pabelanger: yes | 17:29 |
| _ari_ | pabelanger: if you do that aren't you just reinventing what Jenkins already does | 17:29 |
| mordred | pabelanger: it's called "ansible roles" | 17:29 |
| jeblair | pabelanger: ansible | 17:29 |
| *** johnny_ has joined #openstack-infra | 17:29 | |
| jeblair | best part, we don't have to write it | 17:29 |
| mordred | yup | 17:29 |
| pabelanger | IKR | 17:29 |
| *** pvaneck has quit IRC | 17:30 | |
| *** ccamacho has left #openstack-infra | 17:30 | |
| zxiiro | _ari_: I'm not sure. I haven't had any cycles to look at the pipelining stuff yet | 17:30 |
| zxiiro | _ari_: we currently don't use it though so it's not a priority for us | 17:30 |
| pabelanger | I should have rephrased, interested what our ansible roles for zuulv3 will looking. And if we'll have some sort of community spring up around it | 17:31 |
| _ari_ | zxiiro: sure I am able to get the same type of functionality using multijob and matrix plugins | 17:31 |
| pabelanger | s/looking/ look like | 17:31 |
| *** tqtran has joined #openstack-infra | 17:31 | |
| fungi | pabelanger: i expect a lot of it will be shaped by the ability for projects to have their own job definitions in-repo | 17:31 |
| pabelanger | fungi: Ya, looking forward to what those projects produce | 17:32 |
| zxiiro | will zuulv3 support actual pipeline? we have issues with switching to zuul because we can't create triggers that trigger off of other jobs. | 17:32 |
| _ari_ | jeblair: what do you mean you don't have to write it you still have to create a role to do the things Jenkins do right? | 17:32 |
| zxiiro | for example a downstream job is watching an upstream job and triggers if the upstream job passes | 17:32 |
| phschwartz | mordred: when was openstack-infra/ansible-puppet renamed to openstack-infra/ansible-role-puppet? | 17:32 |
| fungi | phschwartz: a couple weeks ago | 17:32 |
| zxiiro | not sure what the words to use to describe it but it's like reverse pipelines? i guess? | 17:32 |
| phschwartz | fungi: hmm, seems to have not complained till today | 17:33 |
| phschwartz | fungi: easy enough fix on my side | 17:33 |
| *** xyang1 has joined #openstack-infra | 17:33 | |
| fungi | zxiiro: we have dependent jobs, where job y only runs if job x passes | 17:33 |
| *** tosky has quit IRC | 17:34 | |
| fungi | zxiiro: that sounds like what you're describing, and it's also been suggested that we could consider inverting the definition so that a job gets marked as depending on some other job rather than a job having a list of dependent jobs | 17:35 |
| jeblair | zxiiro: here's what that looks like today: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n609 | 17:35 |
| zxiiro | fungi: yeah we have jobs where we don't care that it votes or not. just simply that x job finished (vote happens) and then run a build and not report anything. It's so that downstream projects can see if upstream pushed a new feature that broke them. | 17:36 |
| jeblair | zxiiro: on release, build the tarball, if that succeeds, upload the tarball, if that succeeds, rebuild the trusty wheel mirror, if that succeeds, release the trusty wheel mirror to production | 17:36 |
| mordred | 335407 | 17:36 |
| jeblair | mordred: that looks like a change number but doesn't appear to be one | 17:38 |
| *** nwkarsten has joined #openstack-infra | 17:38 | |
| *** nwkarsten has quit IRC | 17:38 | |
| fungi | it will be in a couple weeks at our current pace | 17:38 |
| fungi | (or sooner) | 17:38 |
| *** nwkarsten has joined #openstack-infra | 17:38 | |
| jeblair | i can't wait to see what it ends up being! :) | 17:39 |
| mordred | jeblair: it's not a change number | 17:39 |
| jeblair | mordred promises it will be a good one | 17:39 |
| *** acoles is now known as acoles_ | 17:39 | |
| mordred | jeblair: its 6 numbers that have no current meaning :) | 17:39 |
| * fungi guesses it's mordred's yubikey getting stepped on by a cat | 17:39 | |
| jeblair | fungi: mordred's cat has opposable thumbs, so probably just actually pressed it | 17:39 |
| *** dizquierdo has quit IRC | 17:40 | |
| mordred | jeblair: ++ | 17:40 |
| *** abregman has quit IRC | 17:40 | |
| *** kzaitsev1mb has joined #openstack-infra | 17:40 | |
| *** kzaitsev_mb has quit IRC | 17:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add install-guide jobs for trove https://review.openstack.org/330725 | 17:46 |
| *** eharney has quit IRC | 17:46 | |
| *** Guest41851 has quit IRC | 17:47 | |
| *** Hal has joined #openstack-infra | 17:48 | |
| *** wojdec1 has left #openstack-infra | 17:48 | |
| *** Hal is now known as Guest83979 | 17:49 | |
| * AJaeger has just watched the proposal slave output - cool. thanks, jeblair ! | 17:49 | |
| *** rbrndt has quit IRC | 17:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add zl05 - zl07 to nodepool https://review.openstack.org/330729 | 17:51 |
| *** kushal has quit IRC | 17:52 | |
| jeblair | we're going to finish this today :) | 17:52 |
| pabelanger | mortal combat style | 17:52 |
| fungi | i'll feel a lot more comfortable about the accuracy of our nodepool counts once we're not running a mix of zl and jenkins | 17:53 |
| jeblair | zmq will still be a weak link :( | 17:53 |
| fungi | yep, but it will be a consistent weak link | 17:54 |
| mordred | jeblair: I would like to request that the final patch removing jenkins things has the word "fatality" attached to it | 17:54 |
| AJaeger | jeblair: great! | 17:54 |
| fungi | and easier to orchestrate around in the short term than our current ansible playbook for jenkins restarts | 17:54 |
| *** jamesmcarthur has quit IRC | 17:54 | |
| AJaeger | jeblair: do we really need the same number of zuul-launchers? Or is that something to figure out over time? | 17:55 |
| andreaf | d-g change to switch tempest layer4 job to pre-provisioned test accounts: https://review.openstack.org/#/c/329723/ - could I have reviews / comments? | 17:55 |
| fungi | AJaeger: it looked like the load balance/cpu/memory was about right at 1:1 | 17:55 |
| jeblair | AJaeger: we *think* we need about the same. however, the zl hosts have 1/4 the ram :) | 17:55 |
| fungi | AJaeger: noting that the zuul-launchers are much smaller servers than the giant flavors we ran our jenkins masters on | 17:56 |
| fungi | er, yeah, what jeblair said ;) | 17:56 |
| AJaeger | thanks for explanation | 17:56 |
| *** vhosakot has joined #openstack-infra | 17:57 | |
| *** PollyZ has joined #openstack-infra | 17:57 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:58 | |
| *** pvaneck has joined #openstack-infra | 17:58 | |
| *** electrofelix has quit IRC | 17:58 | |
| *** mtanino has quit IRC | 17:59 | |
| *** yaume has quit IRC | 17:59 | |
| pabelanger | on demand zuul-launchers would be very cool | 18:00 |
| pabelanger | automated of course | 18:00 |
| *** baoli_ has quit IRC | 18:00 | |
| *** PollyZ has quit IRC | 18:00 | |
| fungi | in/after zuul v3 | 18:01 |
| *** dimtruck is now known as zz_dimtruck | 18:01 | |
| mordred | yup | 18:01 |
| fungi | i wouldn't want to heap any more effort on nice-to-haves in 2.5, this is just our "quick" solution for dropping jenkins | 18:01 |
| *** vhosakot has quit IRC | 18:01 | |
| fungi | any unnecessary effort put into 2.5 saps development resources from 3 | 18:02 |
| openstackgerrit | Merged openstack-infra/puppet-pip: Declare features in openstack_pip provider https://review.openstack.org/330657 | 18:02 |
| *** kzaitsev1mb has quit IRC | 18:02 | |
| pabelanger | I think feature requests for v25 are closed :) | 18:02 |
| *** vhosakot has joined #openstack-infra | 18:02 | |
| pabelanger | I'm imagining the count down clock for zuulv25 starts as soon as the last jenkins master goes offline | 18:03 |
| *** jsavak has quit IRC | 18:03 | |
| *** Sukhdev has joined #openstack-infra | 18:03 | |
| russellb | will there be cake? | 18:03 |
| *** amoralej is now known as amoralej|off | 18:04 | |
| mordred | russellb: there is always time for cake | 18:04 |
| *** kmartin has quit IRC | 18:04 | |
| russellb | ++ | 18:04 |
| openstackgerrit | Merged openstack-infra/project-config: Create new functional job for glance_store-cinder https://review.openstack.org/329827 | 18:05 |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Add partition image src jobs for agent driver https://review.openstack.org/330738 | 18:06 |
| *** jsavak has joined #openstack-infra | 18:06 | |
| *** rcernin has joined #openstack-infra | 18:07 | |
| *** akshai has quit IRC | 18:08 | |
| *** trown|lunch is now known as trown | 18:09 | |
| *** akshai has joined #openstack-infra | 18:11 | |
| *** mtanino has joined #openstack-infra | 18:11 | |
| *** PollyZ has joined #openstack-infra | 18:11 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Support Zuul as a Gerrit user equivalent to Jenkins https://review.openstack.org/330684 | 18:13 |
| *** eharney has joined #openstack-infra | 18:14 | |
| *** hdaniel has joined #openstack-infra | 18:14 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Get user list by domain https://review.openstack.org/330739 | 18:15 |
| oomichi | fungi: hi, mkoderer has written the detail of our meetup on https://wiki.openstack.org/wiki/Sprints/QAInfraNewtonSprint | 18:15 |
| oomichi | fungi: so we can get attendees now :) | 18:15 |
| *** baoli has joined #openstack-infra | 18:16 | |
| mtreinish | oomichi: you might want to put it on the ML | 18:16 |
| oomichi | mtreinish: yep, I am searching old mail as a sample now ;) | 18:16 |
| *** HeOS has quit IRC | 18:18 | |
| fungi | oomichi: awesome news--thanks! | 18:18 |
| fungi | mkoderer: thanks! | 18:18 |
| *** _nadya_ has joined #openstack-infra | 18:18 | |
| openstackgerrit | Merged openstack-infra/zuul: Make pipeline approval username requirement a regular expression https://review.openstack.org/330696 | 18:19 |
| fungi | pabelanger: i approved 329094 but we'll want to keep an eye out for potentially unusual behavior since a lot (most?) of our universal wheels in the mirror will be built under python 3 now instead of 2 | 18:22 |
| pabelanger | fungi: that is fair | 18:23 |
| fungi | i wonder if we could adjust the build script to only copy non-universal wheels when called for python 3 | 18:23 |
| *** mriedem1 has joined #openstack-infra | 18:23 | |
| jeblair | russellb: oh yeah, we should have a retirement party for jenkins tomorrow | 18:23 |
| *** jyuso1 has quit IRC | 18:24 | |
| openstackgerrit | Dong Ma proposed openstack-infra/jenkins-job-builder: Remove the colon after default in docstring https://review.openstack.org/330741 | 18:24 |
| *** mkoderer__ has joined #openstack-infra | 18:24 | |
| fungi | pabelanger: though that introduces another corner case because of environment markers. module version dependencies may be different under different interpreter versions | 18:24 |
| *** jyuso1 has joined #openstack-infra | 18:24 | |
| *** ddieterly[away] has quit IRC | 18:24 | |
| fungi | so we might end up not building some universal wheels which were needed by versions obtained under python 3 that we didn't need under python 2 | 18:24 |
| fungi | s/building/mirroring/ | 18:25 |
| *** gyee has quit IRC | 18:25 | |
| pabelanger | fungi: ya | 18:25 |
| *** mriedem has quit IRC | 18:26 | |
| *** PollyZ has quit IRC | 18:27 | |
| AJaeger | could somebody help me with reviewing the higgins rename change logic, please? I'm only concerned right now on the IRC setup: https://review.openstack.org/#/c/326306/ adds a new channel, then we have https://review.openstack.org/#/c/330017/ to rename channels in system-config - and then we should be able to remove the old channel. Correct? | 18:27 |
| *** kmartin has joined #openstack-infra | 18:27 | |
| mordred | 469166 | 18:28 |
| mordred | gah | 18:28 |
| mordred | ok. that's not a good place for that | 18:29 |
| AJaeger | mordred: that change number will take even longer ;) | 18:29 |
| jeblair | mordred is a numbers station | 18:29 |
| *** mriedem1 is now known as mriedem | 18:29 | |
| fungi | indeed. a few more and i'll have the code cracked | 18:29 |
| * mordred knows how to make integers | 18:29 | |
| *** ddieterly has joined #openstack-infra | 18:30 | |
| *** mixos has quit IRC | 18:30 | |
| Shrews | mordred: i remember my first yubikey | 18:30 |
| *** nwkarsten has quit IRC | 18:31 | |
| AJaeger | fungi, can we add another channel to gerritbot - or do we have to wait for the limit to be changed? | 18:31 |
| *** maishsk_ has joined #openstack-infra | 18:31 | |
| *** maishsk has quit IRC | 18:32 | |
| *** maishsk_ is now known as maishsk | 18:32 | |
| *** hdaniel has quit IRC | 18:33 | |
| notmorgan | mordred: lol | 18:33 |
| notmorgan | mordred: welcome to my world | 18:34 |
| *** nwkarsten has joined #openstack-infra | 18:34 | |
| *** nwkarsten has quit IRC | 18:34 | |
| *** baoli has quit IRC | 18:34 | |
| mordred | notmorgan: dude. I'm just following your instructions | 18:34 |
| *** nwkarsten has joined #openstack-infra | 18:34 | |
| clarkb | bte the qa/infra sprint coincides with a gerrit meetup at the same place | 18:34 |
| *** sarob has joined #openstack-infra | 18:35 | |
| jeblair | clarkb: neat | 18:35 |
| *** sarob has quit IRC | 18:36 | |
| fungi | AJaeger: we can add them, just it may not join them all if there are more than 120 | 18:36 |
| jeblair | what a coincidence | 18:36 |
| fungi | tomaw isn't in channel at the moment :/ | 18:36 |
| *** baoli has joined #openstack-infra | 18:37 | |
| AJaeger | fungi, if we add them, it will be more than 120. So, I wonder whether we just approve - or wait. | 18:37 |
| nibalizer | I think I'm still not allowed to hack on gerrit :/ | 18:37 |
| *** PollyZ has joined #openstack-infra | 18:37 | |
| nibalizer | which might actually be a feature | 18:37 |
| fungi | AJaeger: let's wait to add more channels to gerritbot until we can get CHANLIMIT upped for its account | 18:38 |
| *** rbrndt has joined #openstack-infra | 18:38 | |
| notmorgan | mordred: and clearly you're doing it right if you found out that issue (finding the right USB port was a challenge for me) | 18:38 |
| nibalizer | fungi: ack | 18:38 |
| *** sarob has joined #openstack-infra | 18:38 | |
| notmorgan | mordred: i'm using the one next to the HDMI port on the right (x1c), so i don't put my hands there that often | 18:38 |
| AJaeger | fungi, fine with me. | 18:39 |
| *** hasharAway is now known as hashar | 18:39 | |
| AJaeger | the higgins rename would be very brief - adding one, rename in system-config, removing the old one. But to do this, we would need to wait for a time without meetings... Or should the rename done differently than 326306 and 330017 ? | 18:40 |
| AJaeger | (I know it's missing the last bit of the rename for project-config) | 18:40 |
| mordred | notmorgan: I'm also doing that - and I also don't put my hands there- except I do? | 18:41 |
| *** zz_dimtruck is now known as dimtruck | 18:41 | |
| *** tonytan4ever has joined #openstack-infra | 18:43 | |
| notmorgan | mordred: it also was really bad when i started | 18:43 |
| notmorgan | it gets better as soon as you are used to the thing being there | 18:44 |
| *** dmk0202 has joined #openstack-infra | 18:44 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul: Update pip provider to use openstack_pip provider https://review.openstack.org/330600 | 18:45 |
| *** dmk0202 has quit IRC | 18:45 | |
| *** akshai_ has joined #openstack-infra | 18:45 | |
| *** chem is now known as chem|off | 18:46 | |
| fungi | AJaeger: the higgins channel move is a net gain of 0, so i wouldn't hold that up, would just suggest a cut-over change and merge that immediately after the channel forward is set | 18:46 |
| fungi | shouldn't need a multi-stage series of changes for a channel move | 18:46 |
| openstackgerrit | Merged openstack-infra/puppet-github: Update pip provider to use openstack_pip provider https://review.openstack.org/330702 | 18:47 |
| AJaeger | fungi, then my guidenance was wrong ;( So, just two moves? Ok, let me update the changes for them since I misguided them... | 18:47 |
| *** akshai has quit IRC | 18:47 | |
| AJaeger | jeblair: your change to add the new zuuls failed - but exit code is 0: http://logs.openstack.org/29/330729/1/gate/gate-project-config-layout/c6e6d0f/console.html | 18:47 |
| openstackgerrit | Merged openstack-infra/zuul: Use safe_dump() for Ansible playbooks https://review.openstack.org/330683 | 18:48 |
| jeblair | AJaeger: | 18:48 |
| jeblair | 2016-06-16 18:39:27,495 p=7644 u=root | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/zuul/ansible/library/zuul_log.py' | 18:48 |
| jeblair | also | 18:49 |
| jeblair | um | 18:49 |
| *** maishsk has quit IRC | 18:49 | |
| jeblair | why does the ansible log say 'u=root' ? | 18:49 |
| jeblair | zuul-launcher is running as 'zuul' according to ps | 18:49 |
| *** jamesmcarthur has joined #openstack-infra | 18:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Zun: rename higgins to zun (1) https://review.openstack.org/326306 | 18:50 |
| *** akshai_ has quit IRC | 18:50 | |
| jeblair | Shrews, mordred: any ideas about the ansible log u=root thing? | 18:50 |
| mordred | jeblair: launcher may be running as zuul. but what user is it running that task as remotely? | 18:50 |
| *** kmartin has quit IRC | 18:50 | |
| *** sarob_ has joined #openstack-infra | 18:50 | |
| *** xarses has quit IRC | 18:50 | |
| jeblair | mordred: jenkins? | 18:50 |
| *** amit213 has joined #openstack-infra | 18:51 | |
| mordred | hrm | 18:51 |
| EmilienM | fungi: I have an outstanding request to bring our TripleO CI back to green: can you promote 330661 jobs please? | 18:51 |
| mordred | jeblair: where is that log? | 18:51 |
| *** cody-somerville has joined #openstack-infra | 18:51 | |
| jeblair | mordred: | 18:51 |
| jeblair | http://logs.openstack.org/29/330729/1/gate/gate-project-config-layout/c6e6d0f/_zuul_ansible/logs/ansible.log | 18:51 |
| *** xarses has joined #openstack-infra | 18:51 | |
| *** shamail has joined #openstack-infra | 18:53 | |
| Shrews | http://logs.openstack.org/29/330729/1/gate/gate-project-config-layout/c6e6d0f/_zuul_ansible/inventory says jenkins | 18:53 |
| mordred | jeblair: I do not see anything in those playbooks which would lead me to believe it should be root | 18:53 |
| fungi | EmilienM: i'm assuming you mean you want it enqueued into the gate, skipping the check pipeline? | 18:53 |
| EmilienM | fungi: yes, we would like it merged asap | 18:53 |
| *** sarob has quit IRC | 18:53 | |
| Shrews | jeblair: this is where -vvv would be helpful | 18:54 |
| *** salv-orl_ has quit IRC | 18:54 | |
| EmilienM | fungi: thx a ton | 18:55 |
| jeblair | Shrews, mordred: when i run locally, i get u=corvus, which is the user my zuul-launcher is running as | 18:55 |
| *** salv-orlando has joined #openstack-infra | 18:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Add python3 wheel-builds https://review.openstack.org/329094 | 18:55 |
| jeblair | so i think that's supposed to mean u=<user ansible is running as locally> | 18:55 |
| *** maishsk has joined #openstack-infra | 18:56 | |
| *** baoli has quit IRC | 18:56 | |
| *** _nadya_ has quit IRC | 18:58 | |
| *** e0ne has joined #openstack-infra | 18:58 | |
| *** mhickey has quit IRC | 18:59 | |
| jeblair | user = getpass.getuser() | 18:59 |
| jeblair | logger = logging.getLogger("p=%s u=%s | " % (mypid, user)) | 18:59 |
| *** yolanda has quit IRC | 19:00 | |
| *** _nadya_ has joined #openstack-infra | 19:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:01 | |
| jeblair | okay, so we need to pause the migration and solve these two mysteries first: | 19:02 |
| jeblair | 1) why does the log say u=root | 19:02 |
| *** ociuhandu has quit IRC | 19:02 | |
| jeblair | 2) what caused the missing zuul_log module error | 19:02 |
| jeblair | and i need to grab lunch now. biab. | 19:02 |
| Shrews | jeblair: oh, i don't think that's coming from ansible | 19:04 |
| jeblair | oh i have a theory on #2 | 19:04 |
| jeblair | i think we re-installed zuul *exactly* as that task was running | 19:04 |
| jeblair | apparently re-installing zuul means that file disappears briefly | 19:05 |
| *** arxcruz has quit IRC | 19:05 | |
| jeblair | Jun 16 18:39:29 zl04 puppet-user[20639]: (/Stage[main]/Zuul/Exec[install_zuul]) Triggered 'refresh' from 1 events | 19:05 |
| jeblair | 2016-06-16 18:39:27,495 p=7644 u=root | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/zuul/ansible/library/zuul_log.py' | 19:05 |
| *** jsavak has quit IRC | 19:05 | |
| jeblair | Shrews: what's not coming from ansible? | 19:06 |
| *** roaet has joined #openstack-infra | 19:06 | |
| *** hdaniel has joined #openstack-infra | 19:06 | |
| *** xarses_ has joined #openstack-infra | 19:06 | |
| roaet | hello infra! | 19:06 |
| Shrews | jeblair: eh, nm | 19:08 |
| *** EricGonczer_ has joined #openstack-infra | 19:08 | |
| *** xarses has quit IRC | 19:09 | |
| jeblair | okay, lunch for real now; if anyone has ideas on #1 they would be greatly appreciated | 19:09 |
| openstackgerrit | Brian Stajkowski proposed openstack-infra/project-config: New Project Quark Plugin https://review.openstack.org/319368 | 19:12 |
| *** bhavik has quit IRC | 19:12 | |
| *** xarses_ has quit IRC | 19:12 | |
| *** roxanagh_ has joined #openstack-infra | 19:13 | |
| *** EricGonczer_ has quit IRC | 19:13 | |
| *** javeriak has joined #openstack-infra | 19:14 | |
| *** javeriak_ has quit IRC | 19:15 | |
| Shrews | jeblair: mordred: ok, so, the ansible-playbook log (captured via the log_path variable) runs from the local machine (not remote), so we could expect to see the username in that log output to be the zuul-launcher user | 19:17 |
| *** roxanagh_ has quit IRC | 19:17 | |
| Shrews | which i'm assuming is root? | 19:18 |
| Shrews | if that is actually 'zuul', then i have no good answer. | 19:21 |
| *** e0ne has quit IRC | 19:22 | |
| Shrews | infra-root: anyone care to see what user zuul-launcher is running as on zl04 ? | 19:22 |
| mordred | Shrews: yah - zuul-launcher shows up in ps as running as zuul | 19:22 |
| Shrews | mordred: on zl04? | 19:22 |
| mordred | zuul 9980 1 0 Jun15 ? 00:12:29 /usr/bin/python /usr/local/bin/zuul-launcher | 19:22 |
| mordred | Shrews: yes. that's on zl04 | 19:23 |
| Shrews | mordred: i hate computers | 19:23 |
| mordred | yup | 19:23 |
| *** _nadya_ has quit IRC | 19:26 | |
| *** mixos has joined #openstack-infra | 19:27 | |
| *** dprince has quit IRC | 19:27 | |
| Shrews | mordred: https://t37.net/on-getting-the-running-process-user-under-python-and-supervisord.html | 19:28 |
| Shrews | not sure that the supervisord is relevant, but the env var stuff is | 19:29 |
| fungi | oh, fun | 19:29 |
| *** _ari_ has quit IRC | 19:30 | |
| *** Jeffrey4l has quit IRC | 19:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Add zl05 - zl07 to nodepool https://review.openstack.org/330729 | 19:30 |
| fungi | yeah, so LOGNAME getting carried along into the child after changing effective uid confuses it? | 19:30 |
| Shrews | fungi: maybe? i can only speculate | 19:31 |
| *** chem|off` has joined #openstack-infra | 19:31 | |
| Shrews | but that sounds plausible to me | 19:31 |
| fungi | well, the initscript is run by root | 19:31 |
| *** Jeffrey4l has joined #openstack-infra | 19:31 | |
| *** csomerville has joined #openstack-infra | 19:31 | |
| fungi | which then tells start-stop-daemon to change user when launching the daemon | 19:32 |
| *** chem|off has quit IRC | 19:33 | |
| *** sarob_ has quit IRC | 19:34 | |
| fungi | i'm guessing one of the envvars getpass.getuser() queries isn't being reset to match | 19:34 |
| Shrews | so, i'll change my statement to "i hate python" | 19:34 |
| *** Guest83979 has quit IRC | 19:34 | |
| Shrews | so, i'll change my statement to "i hate python, too" | 19:35 |
| fungi | yeah, i would have expected (on *nix systems) that it would look at the process uid and then map that via the usual mechanisms to a username | 19:35 |
| fungi | maybe this is an attempt to make fewer calls to the filesystem? | 19:35 |
| *** cody-somerville has quit IRC | 19:35 | |
| fungi | regardless, violates principle of least surprise for me and apparently lots of other people | 19:36 |
| *** dtardivel has quit IRC | 19:37 | |
| *** ociuhandu has joined #openstack-infra | 19:37 | |
| *** Hal has joined #openstack-infra | 19:37 | |
| *** Hal is now known as Guest17731 | 19:37 | |
| *** asselin_ has joined #openstack-infra | 19:38 | |
| *** flepied1 has joined #openstack-infra | 19:38 | |
| *** dimtruck is now known as zz_dimtruck | 19:39 | |
| *** hdaniel has quit IRC | 19:39 | |
| *** asselin__ has quit IRC | 19:39 | |
| *** dmk0202 has joined #openstack-infra | 19:40 | |
| *** EricGonczer_ has joined #openstack-infra | 19:41 | |
| *** flepied has quit IRC | 19:42 | |
| *** PollyZ has quit IRC | 19:42 | |
| *** dmk0202 has quit IRC | 19:43 | |
| *** esikachev has quit IRC | 19:44 | |
| *** nwkarsten has quit IRC | 19:45 | |
| *** EricGonczer_ has quit IRC | 19:45 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Remove the colon after default in docstring https://review.openstack.org/330741 | 19:46 |
| *** _nadya_ has joined #openstack-infra | 19:47 | |
| jeblair | Shrews: aha! | 19:49 |
| jeblair | Shrews, fungi, mordred: are we satisfied with that as an explanation for #1, and my theory for #2? | 19:49 |
| lifeless | hmm, gerrit seems to be signing me out after 5m? I thought the bug fix to my config patch was deployed? | 19:50 |
| hongbin | Hi, I tried to rename IRC channel from #openstack-higgins to #openstack-zun. I followed this instruction http://docs.openstack.org/infra/system-config/irc.html#renaming-an-irc-channel but it turns out I didn't have permission to execute all those commands. Could anyone know how to fix it? | 19:51 |
| Shrews | jeblair: fairly confident about #1. #2 theory seems highly likely to me | 19:52 |
| fungi | lifeless: it's not signing me out quickly. are you sure you don't have different session ids fighting between browser tabs/windows? | 19:54 |
| fungi | lifeless: (possibly painful) workaround is to log out of gerrit in all your tabs, then log back in with just one and refresh all the others after that | 19:55 |
| *** jsavak has joined #openstack-infra | 19:55 | |
| *** hdaniel has joined #openstack-infra | 19:55 | |
| fungi | at that point they _should_ (in my experience) all be using the same session | 19:55 |
| AJaeger | lifeless: see I95c20373d572b061e870103a813fef1d25de4b23 - 2m means 2mins, so it was reverted... | 19:57 |
| AJaeger | lifeless: so, it should be at 7 days - but it's less than a day it seems to me... | 19:57 |
| fungi | definitely doesn't seem to be 5 minutes though | 19:58 |
| *** tkelsey has quit IRC | 19:58 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: treat missing metadata as a warning not an error https://review.openstack.org/330764 | 19:58 |
| JayF | I can easily reproduce the 5m issue: open multiple review tabs; login to >1 of them | 19:59 |
| timothyb89 | does anyone have experience running diskimage-builder behind a proxy? with vars set and the local-config element added I'm still hitting proxy errors from pip at 40-glean | 19:59 |
| JayF | and it sorta spirals down because if it logs you out frequently you're encourages to just keep logging back in so you can vote/comment on apatch | 19:59 |
| JayF | timothyb89: that might be a good question for the tripleo folks, I know they use that frequently | 20:00 |
| fungi | JayF: yep, it's been a very long-standing flaw with the session design in gerrit | 20:00 |
| *** salv-orlando has quit IRC | 20:00 | |
| *** salv-orlando has joined #openstack-infra | 20:01 | |
| jeblair | fungi, Shrews, mordred: so i think we can continue. i'm not too worried about #2 -- it will be nice to find a fix for that in the future, but it shouldn't happen too often. for #1, it would be good to fix that if possible just so we don't look bad, but i don't think we have to block on it. | 20:01 |
| JayF | I will second that it anecdotally seems 'worse' in the recent past | 20:01 |
| timothyb89 | JayF: thanks for the suggestion, I'll ask over there! | 20:01 |
| JayF | but I know how useless of a bugreport that makes ;D | 20:01 |
| fungi | jeblair: yeah, #1 seems like we could prove it pretty quickly by echoing some envvars if we're really concerned that's not the cause. for #2 we may want some sort of atomicity solution for replacing the full fileset? | 20:01 |
| jeblair | fungi, Shrews, mordred: maybe we can set LOGNAME or something in our ansible invocation | 20:01 |
| fungi | export LOGNAME=$USER | 20:02 |
| *** Apoorva has quit IRC | 20:02 | |
| fungi | sure | 20:02 |
| *** PollyZ has joined #openstack-infra | 20:03 | |
| jeblair | oh, the change landed | 20:03 |
| jeblair | that means nodepool needs a restart | 20:04 |
| *** ffrank has joined #openstack-infra | 20:04 | |
| fungi | i'm happy to do that unless you're already working on it | 20:04 |
| lifeless | fungi: oh yeah, this time was tabs | 20:04 |
| jeblair | fungi: on it | 20:04 |
| lifeless | fungi: that said, yeah, the multi day fix ain't in... | 20:04 |
| fungi | awesome | 20:04 |
| fungi | lifeless: right, i thought someone was going to repropose it with an actually longer (not 2-minute) value, but no idea if that happened yet | 20:05 |
| Shrews | jeblair: http://paste.openstack.org/show/516724/ | 20:05 |
| jeblair | Shrews: okay, so we just add that to the env when we invoke ansible and make it say whatever we want :) | 20:05 |
| jeblair | Shrews: also, you have a 'bubba' account too? | 20:06 |
| openstackgerrit | Christian Fetzer proposed openstack-infra/jenkins-job-builder: Add support for Clang Scan-Build plugin version 1.7 https://review.openstack.org/330150 | 20:06 |
| *** shamail has quit IRC | 20:06 | |
| Shrews | jeblair: i want to call it 'cthulu' | 20:06 |
| Shrews | cthulhu... however you spell it | 20:06 |
| fungi | i recommend the more traditional spelling, "cthulhu" | 20:06 |
| Shrews | that too | 20:06 |
| fungi | yeah, that one | 20:06 |
| jeblair | do not upset it | 20:07 |
| fungi | ktulu also works though | 20:07 |
| xenogear | hi guys, i am trying to update task status on wiki but unable to save? i click on save button and nothing happens. thoughts? | 20:07 |
| *** matrohon has joined #openstack-infra | 20:07 | |
| AJaeger | xenogear: scroll to the top and answer the question | 20:07 |
| fungi | xenogear: did you see the captcha up at the top? you need to answer that (spam mitigation) | 20:07 |
| *** zz_dimtruck is now known as dimtruck | 20:07 | |
| xenogear | oh...must've missed it...let me try again, thx | 20:07 |
| *** baoli has joined #openstack-infra | 20:07 | |
| xenogear | :D thanks! worked | 20:09 |
| fungi | jeblair: Shrews: if we wanted to get really pedantic, maybe LOGNAME=`id -un` | 20:09 |
| *** sarob has joined #openstack-infra | 20:09 | |
| pabelanger | review.o.o looks to be back until control of puppet now | 20:10 |
| fungi | (or whoami if we assume coreutils is installed) | 20:10 |
| jeblair | fungi, Shrews: i figure for this purpose, we can just hardcode zuul :) | 20:10 |
| jeblair | though, who knows what else might use it | 20:10 |
| jeblair | (presumably nothing important if nothing else is broken) | 20:11 |
| fungi | yep, seems fine to me | 20:11 |
| *** nwkarsten has joined #openstack-infra | 20:12 | |
| *** nwkarsten has quit IRC | 20:12 | |
| *** sarob has quit IRC | 20:12 | |
| *** jcoufal has quit IRC | 20:12 | |
| *** nwkarsten has joined #openstack-infra | 20:12 | |
| *** sarob has joined #openstack-infra | 20:13 | |
| *** akshai has joined #openstack-infra | 20:13 | |
| *** ddieterly is now known as ddieterly[away] | 20:14 | |
| jeblair | #status log zl05 - zl07 are in production; jenkins05 - jenkins07 are in prepare for shutdown mode pending decomissioning | 20:15 |
| * jeblair chases after statusbot | 20:16 | |
| mordred | lifeless: https://review.openstack.org/#/c/311888/ | 20:17 |
| *** ayoung has quit IRC | 20:18 | |
| jeblair | it was running but idle; maybe a dead connection (maybe whatever killed zmq) | 20:18 |
| jeblair | restarted it | 20:18 |
| *** openstackstatus has joined #openstack-infra | 20:19 | |
| *** ChanServ sets mode: +v openstackstatus | 20:19 | |
| Shrews | jeblair: we can set the env variables with the 'env' param to Popen(), but that means no other env variables will be inherited. i wonder if that would be ok? | 20:19 |
| jeblair | Shrews: we can make a copy of the current env first, then update it with that | 20:19 |
| jeblair | #status log zl05 - zl07 are in production; jenkins05 - jenkins07 are in prepare for shutdown mode pending decomissioning | 20:19 |
| jeblair | (it's still joining) | 20:19 |
| *** tonytan4ever has quit IRC | 20:19 | |
| Shrews | ah, yeah | 20:19 |
| openstackstatus | jeblair: finished logging | 20:20 |
| *** lucas-afk has quit IRC | 20:21 | |
| *** zul has quit IRC | 20:21 | |
| jeblair | what's with the ovh quota situation? | 20:21 |
| jeblair | it's basically constant exceptions from that in the nodepool log | 20:22 |
| *** zul has joined #openstack-infra | 20:22 | |
| *** xarses_ has joined #openstack-infra | 20:22 | |
| pabelanger | we also get it from rackspace too | 20:22 |
| *** Apoorva has joined #openstack-infra | 20:22 | |
| lifeless | mordred: has it been deployed? | 20:22 |
| lifeless | mordred: cause, I'm definitely not getting 7 days of login | 20:23 |
| *** inc0 has quit IRC | 20:23 | |
| *** dmk0202 has joined #openstack-infra | 20:23 | |
| *** gyee has joined #openstack-infra | 20:24 | |
| *** kgiusti has left #openstack-infra | 20:24 | |
| jeblair | we have 58 servers in ovh-gra1. the error we are receiving is Forbidden: Quota exceeded for instances: Requested 1, but already used 200 of 200 instances (HTTP 403) (Request-ID: req-a73a12b8-c13d-4fb5-bda0-835a080e932a) | 20:24 |
| jeblair | that makes no sense. | 20:25 |
| *** flepied has joined #openstack-infra | 20:25 | |
| lifeless | jeblair: quota skew would explain it | 20:26 |
| mordred | lifeless: as in - something on the backend is not updating quota properly? | 20:27 |
| *** ihrachys has joined #openstack-infra | 20:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Remove jenkins05 - jenkins07 from nodepool https://review.openstack.org/330774 | 20:28 |
| *** piet has quit IRC | 20:28 | |
| *** lucasagomes has joined #openstack-infra | 20:28 | |
| *** flepied1 has quit IRC | 20:29 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling https://review.openstack.org/319591 | 20:30 |
| phschwartz | nibalizer: looks like we are still getting the failure on ensure => ### | 20:30 |
| phschwartz | nibalizer: I am back and awake so I will dig back in | 20:30 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul: Set LOGNAME env var to fix Ansible logging https://review.openstack.org/330775 | 20:31 |
| *** jsavak has quit IRC | 20:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Remove jenkins05 - jenkins07 https://review.openstack.org/330776 | 20:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Decomission jenkins05 - jenkins07 https://review.openstack.org/330776 | 20:31 |
| pabelanger | phschwartz: It think those failures just need a recheck | 20:31 |
| jeblair | that's the final jenkins decomissioning ^ | 20:31 |
| *** jsavak has joined #openstack-infra | 20:32 | |
| * rbergeron casually hangs out in case anything fun is happening | 20:32 | |
| rbergeron | ;) | 20:32 |
| Shrews | everyone flee! it's rbergeron | 20:33 |
| *** javeriak has quit IRC | 20:33 | |
| phschwartz | pabelanger: there were 2 rechecks already, I will do another just to see | 20:33 |
| rbergeron | hey nowwwwwwww | 20:33 |
| pabelanger | phschwartz: ya, we rechecked before nibalizer patch landed | 20:33 |
| *** _nadya_ has quit IRC | 20:34 | |
| Shrews | jeblair: 330775 should fix the logging stuff | 20:34 |
| pabelanger | jeblair: congrats, awesome work | 20:34 |
| phschwartz | pabelanger: he had a recheck at 2:10 est and it ran at 2:30 est | 20:35 |
| phschwartz | pabelanger: was that still before his patch? | 20:35 |
| pabelanger | phschwartz: I believe so. | 20:36 |
| mordred | jeblair: I +A'd the system-config one and +2'd the other | 20:36 |
| jeblair | rbergeron: the exciting thing right now is watching jenkins run down the clock :) | 20:36 |
| pabelanger | phschwartz: We should have used the Depends-On header on your commits to ensure the patch was in the pipeline before recheck | 20:36 |
| pabelanger | phschwartz: but we didn't | 20:36 |
| pabelanger | it has merged now however | 20:37 |
| phschwartz | pabelanger: I will keep an eye on it | 20:37 |
| *** _nadya_ has joined #openstack-infra | 20:38 | |
| *** sarob_ has joined #openstack-infra | 20:38 | |
| jeblair | i'm a little puzzled by zl05 and zl07 having received 0 nodes, and zl06 only having received 1 | 20:39 |
| jeblair | okay, 05 just got one | 20:39 |
| *** esikachev has joined #openstack-infra | 20:40 | |
| Shrews | jeblair: is distribution normally evenly weighted? | 20:41 |
| *** sarob has quit IRC | 20:42 | |
| jeblair | Shrews: not even remotely. i think we probably need to fix that nowish. | 20:42 |
| Shrews | jeblair: something with geard wakeup algorithm i suspect? | 20:43 |
| rbergeron | jeblair: I am not quite sure I know what you mean (I think i do, but :D) -- but I am content to just watch y'all collaborating on the things. | 20:43 |
| pabelanger | we restarted nodepool recently right? Trying to account for the reason for the stale nodes | 20:43 |
| jeblair | Shrews: yeah, i think i'll add an exponential delay tied to the number of currently held nodes in between noop and grab_job | 20:43 |
| pabelanger | maybe 45mins ago? | 20:43 |
| jeblair | pabelanger: yeah | 20:43 |
| pabelanger | okay, doing some clean up | 20:43 |
| jeblair | pabelanger: at 20:04 | 20:43 |
| openstackgerrit | Merged openstack-infra/puppet-odsreg: Update pip provider to use openstack_pip provider https://review.openstack.org/330598 | 20:44 |
| pabelanger | jeblair: Yup, that lines up with that I am seeing | 20:44 |
| Shrews | jeblair: maybe just a simple LRU algorithm would work | 20:44 |
| *** ddieterly[away] is now known as ddieterly | 20:44 | |
| mordred | rbergeron: tl;dr - we're currently waiting for the jobs that are running on the last 3 jenkins masters to complete before we turn them off | 20:45 |
| *** PollyZ has quit IRC | 20:45 | |
| *** hemanthm is now known as hemanthm|afk | 20:46 | |
| *** sarob has joined #openstack-infra | 20:46 | |
| rbergeron | mordred: I figured -- I was mostly like, "hm, i'm not sure where to see that" (and thus, "am i insane thinking that's the meaning?") | 20:46 |
| *** bhavik has joined #openstack-infra | 20:46 | |
| Shrews | i wonder what the real gearman does for wakeup | 20:46 |
| mordred | rbergeron: ah. yah. that's not actually a thing you can effectively watch right now. so 'watch' is metaphorical | 20:47 |
| rbergeron | ;) | 20:47 |
| *** sarob_ has quit IRC | 20:49 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Enable storyboard for js-openstack-lib https://review.openstack.org/330779 | 20:49 |
| jeblair | Shrews: it wakes everyone up at ~the same time. we could have gear lru or randomize that and it may help, but still, network and host latency may end up dwarfing that. | 20:50 |
| dmsimard | mordred: o/ figured you'd be interested in the UI work we've been doing for ARA http://logs.openstack.org/76/330576/6/check/gate-ara-integration-ubuntu-trusty/6edcb0a/logs/build/ | 20:51 |
| dmsimard | still a WIP of course but better than the "before": http://logs.openstack.org/56/329256/2/gate/gate-ara-integration-centos-7/d099746/logs/build/ | 20:51 |
| *** ihrachys has quit IRC | 20:52 | |
| *** zul has quit IRC | 20:52 | |
| *** baoli has quit IRC | 20:53 | |
| *** ihrachys has joined #openstack-infra | 20:53 | |
| jeblair | i think the older zl servers are gaininng enough workload that they're slowing down enough for the new ones to pick stuff up now. | 20:54 |
| *** tkelsey has joined #openstack-infra | 20:54 | |
| *** aeng has joined #openstack-infra | 20:54 | |
| jeblair | Shrews: you can see the disparity here: http://grafana.openstack.org/dashboard/db/nodepool (thanks pabelanger!) | 20:55 |
| *** PollyZ has joined #openstack-infra | 20:55 | |
| mordred | dmsimard: ooh. that's nice | 20:56 |
| dmsimard | jeblair: that building node graph is so sexy | 20:56 |
| dmsimard | you can see there is no jenkins, just zuul now :P | 20:56 |
| mordred | yah | 20:56 |
| mordred | that's the best picture ever | 20:57 |
| pabelanger | jeblair: I would agree with your statement. | 20:57 |
| *** rhallisey has quit IRC | 20:58 | |
| *** tkelsey has quit IRC | 20:59 | |
| openstackgerrit | Stephanie Miller proposed openstack/diskimage-builder: Ironic agent kernel should be owned by user building image https://review.openstack.org/330783 | 20:59 |
| *** trown is now known as trown|outtypewww | 21:00 | |
| nibalizer | pabelanger: recheck all the things! | 21:01 |
| nibalizer | also the patch upstream in puppet landed, but I don't know where they are on a release | 21:01 |
| *** _nadya_ has quit IRC | 21:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Remove Jenkins masters from grafana dashboard https://review.openstack.org/330784 | 21:01 |
| Shrews | mordred: let's make t-shirts with that picture | 21:01 |
| *** _nadya_ has joined #openstack-infra | 21:02 | |
| *** _nadya_ has quit IRC | 21:02 | |
| *** baoli has joined #openstack-infra | 21:02 | |
| *** sarob has quit IRC | 21:03 | |
| *** rfolco has quit IRC | 21:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: delay node assignment under load https://review.openstack.org/330786 | 21:04 |
| jeblair | mordred, Shrews: ^ | 21:04 |
| *** sarob has joined #openstack-infra | 21:04 | |
| *** aysyd has quit IRC | 21:06 | |
| mordred | nibalizer: are they also patching 3.x? | 21:07 |
| *** chem|off` has quit IRC | 21:07 | |
| nibalizer | yah, already landed | 21:07 |
| nibalizer | i actually only patched 3.x, I think we deterimed 4 is unaffected | 21:08 |
| EmilienM | jeblair: we have some jobs that fail for unknown reason, like http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/652634b/console.html#_2016-06-16_20_40_28_161119 | 21:09 |
| mordred | nibalizer: cool | 21:09 |
| jeblair | 2016-06-16 20:40:27,502 p=20099 u=zuul | fatal: [node]: FAILED! => {"changed": false, "failed": true, "msg": "async task did not complete within the requested time"} | 21:10 |
| jeblair | EmilienM: it's a timeout ^ | 21:10 |
| jeblair | EmilienM: that's in http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/652634b/_zuul_ansible/logs/ansible.log | 21:10 |
| pabelanger | Wow, that's the first time I see a job get killed back of async | 21:10 |
| jeblair | EmilienM: we should find a way to put that in the console log | 21:10 |
| jeblair | pabelanger: yeah... :| | 21:10 |
| EmilienM | jeblair: a timeout ? but Gerrit says 19m 46s | 21:11 |
| *** thorst_ has quit IRC | 21:11 | |
| pabelanger | We are still getting some watchdog events, but haven't debugged it more | 21:11 |
| jeblair | (but the clue is that it did job complete without task complete) | 21:11 |
| *** thorst_ has joined #openstack-infra | 21:11 | |
| jeblair | EmilienM: so do the timestamps. | 21:12 |
| EmilienM | "async task did not complete within the requested time" | 21:12 |
| *** psilvad has quit IRC | 21:12 | |
| Shrews | jeblair: so, that looks fine. but if we WERE to manage fairly even distribution, then they're all sleeping unnecessarily for the approximate same amount of time. but it should be fairly short, so maybe that's not too big of an issue | 21:12 |
| jeblair | Shrews: very short most of the time, 10 seconds at their design capacity; 40 seconds at 2x their design capacity | 21:13 |
| EmilienM | jeblair: what should I do? recheck? | 21:14 |
| *** roxanagh_ has joined #openstack-infra | 21:14 | |
| jeblair | EmilienM: i'm looking to see if i can find any more info | 21:14 |
| EmilienM | ack | 21:14 |
| jeblair | EmilienM: in the mean time, yes, recheck please | 21:14 |
| EmilienM | ok | 21:14 |
| EmilienM | our CI is broken we need this patch landed asap | 21:15 |
| *** gouthamr has quit IRC | 21:15 | |
| EmilienM | (puppet CI this time) | 21:15 |
| *** krtaylor has quit IRC | 21:15 | |
| Shrews | jeblair: that might be good enough for now | 21:15 |
| *** ldnunes has quit IRC | 21:16 | |
| jeblair | Shrews: yeah, it's a quick fix for sure | 21:16 |
| *** thorst_ has quit IRC | 21:16 | |
| jeblair | it doesn't look like we log the timeout value we supply to ansible | 21:16 |
| pabelanger | ya, I cannot find it either | 21:16 |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Upodate pip provider to use openstack_pip provider https://review.openstack.org/330528 | 21:18 |
| openstackgerrit | Merged openstack-infra/zuul: Fix referenced before assignment for BuildCompletedEvent https://review.openstack.org/327274 | 21:18 |
| *** roxanagh_ has quit IRC | 21:18 | |
| jeblair | i will add a log line for that | 21:19 |
| *** eharney has quit IRC | 21:19 | |
| *** ihrachys_ has joined #openstack-infra | 21:19 | |
| lifeless | mordred: as in, the backend data is plain wrong. There have been lots and lots of consistency holes in quota /usage tracking in nova | 21:20 |
| *** baoli has quit IRC | 21:21 | |
| *** esikachev has quit IRC | 21:21 | |
| *** baoli has joined #openstack-infra | 21:21 | |
| *** amitgandhinz has quit IRC | 21:22 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: Log ansible command line https://review.openstack.org/330795 | 21:22 |
| jeblair | Shrews, pabelanger: ^ | 21:22 |
| *** ihrachys has quit IRC | 21:22 | |
| *** bhavik1 has joined #openstack-infra | 21:23 | |
| *** thorst_ has joined #openstack-infra | 21:23 | |
| *** bhavik has quit IRC | 21:24 | |
| *** bhavik1 is now known as bhavik | 21:24 | |
| pabelanger | jeblair: Ya, almost wish ansible would also output the timeout value in the FAILED message you linked above too | 21:24 |
| pabelanger | going to check ansible and see how to add it | 21:24 |
| openstackgerrit | Merged openstack-infra/puppet-ansible: Update pip provider to use openstack_pip provider https://review.openstack.org/330589 | 21:25 |
| *** mdrabe has quit IRC | 21:25 | |
| jeblair | pabelanger: yeah, we could also put a zuul_log line at the start of the playbook indicating what the job timeout is | 21:25 |
| Shrews | jeblair: maybe log it with ' '.join(cmd) instead? | 21:25 |
| pabelanger | jeblair: Ya, that would be nice | 21:26 |
| jeblair | Shrews: could -- but seeing the individual components gives me a warm fuzzy | 21:26 |
| jeblair | i don't have to second guess shell parsing | 21:26 |
| jeblair | 2016-06-16 14:21:09,567 DEBUG zuul.NodeWorker.test: Ansible post command: ['ansible-playbook', '/tmp/tmpbfY2ke/ansible/post_playbook', '-e', 'success=True', '-v'] | 21:26 |
| jeblair | it looks like that ^ | 21:26 |
| jeblair | 2016-06-16 14:21:01,708 DEBUG zuul.NodeWorker.test: Ansible command: ['ansible-playbook', '/tmp/tmpbfY2ke/ansible/playbook', '-e', 'timeout=600', '-v'] | 21:27 |
| jeblair | and that ^ | 21:27 |
| *** rossella_s has quit IRC | 21:27 | |
| Shrews | jeblair: far be it from me to remove your warm fuzzies | 21:27 |
| * mordred hands Shrews a box of used warm fuzzies | 21:27 | |
| * jeblair curls up with his blanket | 21:28 | |
| *** thorst_ has quit IRC | 21:28 | |
| openstackgerrit | Stephanie Miller proposed openstack/diskimage-builder: Ironic agent kernel should be owned by user building image https://review.openstack.org/330783 | 21:28 |
| jeblair | the last jenkins job just completed. | 21:30 |
| jeblair | for the record, it was http://logs.openstack.org/50/326250/1/check/gate-grenade-dsvm-ironic-nv/fc21c0b// | 21:30 |
| jeblair | and it failed | 21:30 |
| Shrews | how _ironic_ is that | 21:31 |
| jeblair | there are some leaked nodes in nodepool, i'm cleaning them up now | 21:31 |
| *** hdaniel has quit IRC | 21:31 | |
| * Shrews cracks himself up | 21:31 | |
| *** bpokorny_ has joined #openstack-infra | 21:32 | |
| mordred | devananda, jroll: congrats. you snagged the last jenkins job in openstack ever | 21:32 |
| mordred | jeblair: the decomission patch is all yours for the +Aing | 21:32 |
| jroll | O_O | 21:33 |
| jroll | it's happening?! | 21:33 |
| * jroll so excite | 21:33 | |
| Shrews | the excite is much | 21:33 |
| *** bpokorny has quit IRC | 21:34 | |
| openstackgerrit | Merged openstack-infra/zuul: Set LOGNAME env var to fix Ansible logging https://review.openstack.org/330775 | 21:35 |
| asselin | are upstream dib images working? | 21:35 |
| mordred | asselin: we're fixing our puppet ... there were issues with pypi changes that phschwartz made a bunch of patches for | 21:36 |
| mordred | I think most of them have landed | 21:36 |
| pabelanger | should be fixed now | 21:36 |
| *** rcernin has quit IRC | 21:36 | |
| pabelanger | Shrews: jeblair: https://github.com/ansible/ansible/pull/16333 | 21:37 |
| *** dimtruck is now known as zz_dimtruck | 21:37 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Add a build for the gerrit its-storyboard plugin https://review.openstack.org/330796 | 21:37 |
| *** gouthamr has joined #openstack-infra | 21:37 | |
| *** zz_dimtruck is now known as dimtruck | 21:37 | |
| pabelanger | future proof | 21:37 |
| *** dimtruck is now known as zz_dimtruck | 21:37 | |
| *** sarob has quit IRC | 21:37 | |
| fungi | it's the unjenkinsing! | 21:37 |
| *** matrohon has quit IRC | 21:38 | |
| pabelanger | :) | 21:38 |
| *** dmk0202 has quit IRC | 21:38 | |
| krotscheck | Urgh. | 21:39 |
| * krotscheck is trying to figure out how to build documentation for javascript projects without littering the project with confusing python build harnesses | 21:39 | |
| *** matt-borland has quit IRC | 21:40 | |
| mordred | krotscheck: I recognize the problem and salute you for diving in | 21:40 |
| fungi | krotscheck: what do you want to build the docs from and to? | 21:40 |
| *** banix has quit IRC | 21:40 | |
| *** Guest17731 has quit IRC | 21:40 | |
| mtreinish | krotscheck: for devstack and other bash things we gave up trying to do that and just used sphinx | 21:41 |
| mordred | I actually used sphinx to document C++ before I used sphinx to document python | 21:41 |
| asselin | mordred, have you seen this error before? http://paste.openstack.org/show/516731/ | 21:42 |
| EmilienM | jeblair: http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/cbe1fbd/console.html#_2016-06-14_16_56_00_056599 | 21:42 |
| EmilienM | again | 21:42 |
| krotscheck | Let's try this again, this time with less bad coffeeshop internet | 21:42 |
| krotscheck | fungi: I would to use as much of the existing infrastructure as possible, with an additional "oh yeah go do API docs on this javascript" piece. | 21:42 |
| EmilienM | this time, job did nothing | 21:42 |
| fungi | krotscheck: worth noting, there are command-line wrappers around some of what we think of as python doc build tooling (e.g. the rst2html command provided by the python-docutils package) | 21:42 |
| krotscheck | All the pieces are there, however to use the existing jobs I need a setup.cfg, setup.py, tox.ini, etc, and the last thing I want to have is someone looking at our projects and going: Urr... is this python or Javascript? | 21:43 |
| pabelanger | EmilienM: looks like the remote node died | 21:43 |
| pabelanger | http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/cbe1fbd/_zuul_ansible/logs/ansible.log | 21:43 |
| fungi | krotscheck: but yeah if you want to reuse a lot of our existing jobs and theming then that does get fiddly | 21:43 |
| mordred | krotscheck: think stopping at sphinx and leaving out setup.* and tox.ini would be better | 21:43 |
| mordred | krotscheck: the glue to run sphinx right shouldn't be too much to duplicate in js land | 21:43 |
| *** thorst_ has joined #openstack-infra | 21:44 | |
| krotscheck | fungi, mordred: Ya know, if I do this right, it might even work for python things. | 21:44 |
| asselin | mordred, w/o lines cut off: http://paste.openstack.org/show/516734/ | 21:44 |
| mordred | krotscheck: in fact, a generic gulp plugin (or whatnot) to deal with sphinx in javascript repos wouldn't be a terrible thing, right? | 21:45 |
| *** Goneri has quit IRC | 21:45 | |
| asselin | that failing curl works fine outside of DIB | 21:45 |
| EmilienM | pabelanger: recheck again? | 21:45 |
| pabelanger | EmilienM: jeblair: I would have expected zuul_launcher to error out: http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/cbe1fbd/console.html#_2016-06-14_13_54_55_030725 | 21:45 |
| mordred | weird | 21:45 |
| jeblair | EmilienM, pabelanger: oh, i wonder if losing connectivity to the remote node could also have manifested as the async timeout? | 21:45 |
| mordred | greghaynes: ^^ see line from asselin ? | 21:45 |
| pabelanger | checking logs to see why not | 21:45 |
| jeblair | EmilienM, pabelanger: so maybe both errors are the same, and they are both because we stopped being able to log into the node | 21:46 |
| asselin | greghaynes, http://paste.openstack.org/show/516734/ | 21:46 |
| *** thorst__ has joined #openstack-infra | 21:46 | |
| jeblair | EmilienM: any chance this job could affect zuul's ability to log into the node? | 21:46 |
| *** burgerk has quit IRC | 21:46 | |
| pabelanger | jeblair: right, I think this last case, we lost the SSH connection. But, I would have expected us to requeue in that case. Maybe we didn't trap it properly | 21:46 |
| pabelanger | checking the exit code on ansible-playbooks now | 21:46 |
| EmilienM | jeblair: I don't think so | 21:47 |
| jeblair | pabelanger: we handle ansible not being able to initially ssh in. this is it failing to do so after the job has run for 20 minutes | 21:47 |
| pabelanger | 2016-06-16 21:29:46,148 p=30339 u=root | fatal: [node]: FAILED! => {"failed": true, "msg": "failed to transfer file to /home/jenkins/.ansible/tmp/ansible-tmp-1466112561.94-113511581390802/source:\nsftp> put /tmp/tmpvMj3p4/ansible/scripts/5ed90b291603471e8a5c3cc477bacd59.sh /home/jenkins/.ansible/tmp/ansible-tmp-1466112561.94-113511581390802/source\n\nmux_client_request_session: read from master | 21:48 |
| pabelanger | failed: Broken pipe\r\nWarning: Permanently added '192.237.175.227' (RSA) to the list of known hosts.\r\nremote open(\"/home/jenkins/.ansible/tmp/ansible-tmp-1466112561.94-113511581390802/source\"): No such file or directory\r\n"} | 21:48 |
| pabelanger | sorry | 21:48 |
| pabelanger | let me pastebin | 21:48 |
| greghaynes | asselin: huh, that looks like it is running outside the chroot so that kills most of my theories | 21:48 |
| pabelanger | http://paste.openstack.org/show/516736/ | 21:48 |
| pabelanger | that is concerning to me | 21:48 |
| pabelanger | http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/cbe1fbd/_zuul_ansible/logs/ansible.log | 21:48 |
| *** woodster_ has quit IRC | 21:48 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Add a build for the gerrit its-storyboard plugin https://review.openstack.org/330796 | 21:49 |
| *** thorst_ has quit IRC | 21:49 | |
| greghaynes | asselin: are you potentially exporting some proxy settings that only effect dib? | 21:49 |
| mordred | I was just about to ask the same question | 21:50 |
| asselin | greghaynes, yes, we're behind a corp proxy | 21:50 |
| bknudson_ | I'm wondering why https://review.openstack.org/#/c/324020/ isn't being picked up so that it can verify? | 21:50 |
| *** maishsk_ has joined #openstack-infra | 21:50 | |
| *** thorst__ has quit IRC | 21:50 | |
| *** chem has joined #openstack-infra | 21:50 | |
| *** mriedem has quit IRC | 21:51 | |
| *** inc0 has joined #openstack-infra | 21:51 | |
| greghaynes | asselin: what you can do that will help debug is prepend a break=after-error to your dib command, that will give you a shell right after it fails | 21:51 |
| greghaynes | then you can test the command and try and debug what is actually going on | 21:51 |
| jeblair | EmilienM: wait, that job was from 2 days ago? | 21:51 |
| *** yamahata has quit IRC | 21:51 | |
| greghaynes | asselin: but I wonder if maybe youre only setting the proxy for dib or vice versa? | 21:52 |
| asselin | greghaynes, this is via nodepool. do you know how to add that? | 21:52 |
| *** mriedem has joined #openstack-infra | 21:52 | |
| asselin | greghaynes, should be both...can't get out either way w/o it | 21:52 |
| *** mriedem has quit IRC | 21:52 | |
| mordred | asselin: adding that string to the list of elements should work | 21:52 |
| *** chem is now known as chem|off | 21:52 | |
| mordred | oh. wait. poo. no, I'm stupid | 21:53 |
| pabelanger | EmilienM: okay, that is an old job too. back on the 06-14, we've made some changes to zuul-launcher to then. I would recheck | 21:53 |
| mordred | asselin: please do not listen to me :) | 21:53 |
| *** maishsk has quit IRC | 21:53 | |
| *** maishsk_ is now known as maishsk | 21:53 | |
| pabelanger | EmilienM: so, unsure if you linked the right log. because that job didn't run today | 21:53 |
| *** jsavak has quit IRC | 21:53 | |
| *** jsavak has joined #openstack-infra | 21:53 | |
| *** zz_dimtruck is now known as dimtruck | 21:53 | |
| greghaynes | asselin: youll have to just run it by hand - the command should be somewhere in the nodepool log | 21:54 |
| asselin | i need to refresh my memory on running it manually then to reproduce | 21:54 |
| asselin | ok will look there | 21:54 |
| EmilienM | what? | 21:54 |
| EmilienM | this job is from today | 21:54 |
| EmilienM | jeblair, pabelanger | 21:54 |
| EmilienM | it's still in zuul pipeline, see 330694 | 21:54 |
| EmilienM | and I pushed the patch today | 21:54 |
| EmilienM | there is timestamp issue somewhere | 21:55 |
| fungi | gonna go get something to eat, then come back and try to work on artifact signing automation some more | 21:55 |
| pabelanger | EmilienM: You linked the following log: http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/cbe1fbd/console.html#_2016-06-14_16_56_00_056599 from 06-14 | 21:56 |
| mordred | wow | 21:56 |
| jeblair | the log at http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/cbe1fbd/console.html does not seem to match that job | 21:56 |
| EmilienM | pabelanger, jeblair: and I confirm, no ssh issue, other jobs seem to work fine | 21:56 |
| mordred | pabelanger: yah - I just walked through to the link from the zuul status page | 21:56 |
| mordred | 2016-06-14 16:56:00.056599 [Zuul] Job complete, result: FAILURE | 21:56 |
| mordred | 2016-06-16 21:26:28.983831 [Zuul] Job complete, result: FAILURE | 21:56 |
| mordred | that's at the bottom | 21:56 |
| EmilienM | pabelanger: I found the link on zuul.openstack.org | 21:56 |
| mordred | in the console.log | 21:56 |
| jeblair | so we have a console log with apparently 3 jobs | 21:56 |
| *** tkelsey has joined #openstack-infra | 21:56 | |
| EmilienM | and see timestamp in http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-xenial/cbe1fbd/ | 21:56 |
| EmilienM | and then in console.log | 21:56 |
| *** dimtruck is now known as zz_dimtruck | 21:57 | |
| EmilienM | they are 2 days different o_O | 21:57 |
| pabelanger | it this an NTP issue? | 21:57 |
| mordred | EmilienM: congratulations - something is weird | 21:57 |
| mordred | pabelanger: even if it was - why would we have to FAILURE lines | 21:57 |
| jeblair | and the log from a different job | 21:57 |
| mordred | yah | 21:57 |
| jeblair | 2016-06-14 13:48:11.702524 JJB template: gate-{name}-puppet-unit-{puppet_version}-{node} | 21:58 |
| JayF | That explains what I was about to ask about. I have a job that is showing connectivity issues to git.openstack.org in console log, but continues and appears to work | 21:58 |
| JayF | (until it fails for another reason) | 21:58 |
| jeblair | it starts as a puppet-unit | 21:58 |
| jeblair | JayF: can you supply a link? | 21:58 |
| JayF | http://logs.openstack.org/89/330789/1/check/gate-tempest-dsvm-ironic-agent_ssh-coreos-src/c240d37/console.html#_2016-06-16_21_27_08_336204 | 21:58 |
| mordred | jeblair: did we maybe get the same node attached to two launchers? | 21:58 |
| JayF | the failure of the job is unrelated to that (it's an apache restart issue we're already aware of) | 21:58 |
| *** hashar has quit IRC | 21:59 | |
| jeblair | JayF: i don't think that's an error | 21:59 |
| jeblair | JayF: i think that just says "ovh does not have ipv6 connectivity" | 21:59 |
| JayF | jeblair: okay, thanks. I'll keep that in mind if I see that again. | 21:59 |
| *** somerville32 has joined #openstack-infra | 21:59 | |
| EmilienM | did I break something again? | 22:00 |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: Log ansible command line https://review.openstack.org/330795 | 22:01 |
| *** tkelsey has quit IRC | 22:01 | |
| *** thiagop has quit IRC | 22:01 | |
| pabelanger | I am trying to find with zuul-launcher we ran the latest job on | 22:01 |
| jeblair | EmilienM, pabelanger, mordred: https://etherpad.openstack.org/p/536mveQy8Y | 22:01 |
| *** amotoki has joined #openstack-infra | 22:03 | |
| *** csomerville has quit IRC | 22:03 | |
| *** gouthamr_ has joined #openstack-infra | 22:04 | |
| *** dmk0202 has joined #openstack-infra | 22:04 | |
| pabelanger | I think we lost SSH access to the node from ansible-playbook for some reason | 22:04 |
| *** xyang1 has quit IRC | 22:04 | |
| *** sigmavirus24 is now known as sigmavirus24_ | 22:05 | |
| *** mixos has quit IRC | 22:06 | |
| *** xarses_ has quit IRC | 22:06 | |
| *** xarses_ has joined #openstack-infra | 22:07 | |
| *** bpokorny_ has quit IRC | 22:07 | |
| jeblair | okay, the two jobs ran on a host with the same ip | 22:07 |
| *** gouthamr has quit IRC | 22:07 | |
| *** dingyichen has joined #openstack-infra | 22:07 | |
| *** ayoung has joined #openstack-infra | 22:07 | |
| *** bpokorny has joined #openstack-infra | 22:08 | |
| pabelanger | 2 days apart it seems | 22:08 |
| jeblair | nodepool claims to have deleted the server | 22:09 |
| pabelanger | that make sense, one is centos-7 the other ubuntu-xenial | 22:10 |
| jeblair | they have different nova ids | 22:10 |
| EmilienM | "/pup | 22:10 |
| EmilienM | oops | 22:11 |
| *** akshai has quit IRC | 22:11 | |
| *** baoli has quit IRC | 22:11 | |
| *** bhavik has quit IRC | 22:12 | |
| jeblair | http://logs.openstack.org/31/323431/7/check/gate-puppet-tripleo-puppet-unit-3.8-centos-7/f4dec3c/ was the older job | 22:12 |
| *** PollyZ has quit IRC | 22:12 | |
| *** baoli has joined #openstack-infra | 22:12 | |
| *** PollyZ has joined #openstack-infra | 22:12 | |
| *** bhavik has joined #openstack-infra | 22:13 | |
| jeblair | 2016-06-14 13:51:55,142 DEBUG zuul.NodeWorker.centos-7-rax-ord-1656082: Ansible output: fatal: [node]: FAILED! => {"ansible_job_id": "649974018264.27026", "changed": false, "failed": true, "finished": 1, "msg": "could not find job", "started": 1} | 22:13 |
| jeblair | that's what caused the first job to fail | 22:13 |
| jeblair | which is also weird | 22:13 |
| mordred | so .... | 22:14 |
| *** sarob has joined #openstack-infra | 22:14 | |
| mordred | if the original node that was requested to be was managed by a compute node that went down, if cleanup didn't happen properly, it's possible that the compute node came back up and fought the new vm that had claimed that IP for the IP - and because the keys are the same for us, ansible would happily log in to the old vm if it was there for a sec | 22:15 |
| jeblair | mordred: in both instances, we have ansible unable to find things on the host that it expected to find | 22:16 |
| mordred | vms coming back from downed compute nodes due to resource tracker stuff is one of the failure cases in nova that I hear people talk about | 22:16 |
| jeblair | mordred: ansible's non-persistent ssh connections could me more suceptible to that than jenkins | 22:17 |
| *** weshay has quit IRC | 22:18 | |
| mordred | yah | 22:18 |
| pabelanger | so between SSH attempts, a new server came online and claim the IP address between SSH attempts? | 22:18 |
| mordred | or the old server erroenously came back on line for a sec | 22:18 |
| pabelanger | also, between SSH attempts | 22:18 |
| *** sarob has quit IRC | 22:19 | |
| mordred | it's exceptionally odd enough that unless we see a second instance somewhere, I kinda have a hunch it's going to be a cloud-sun-spot | 22:20 |
| jeblair | i found the 3rd job (the one in the middle) | 22:20 |
| jeblair | 2016-06-14 16:59:14,314 DEBUG zuul.NodeWorker.ubuntu-trusty-rax-ord-1665155: Ansible output: fatal: [node]: FAILED! => {"ansible_job_id": "709933771768.1907", "changed": false, "failed": true, "finished": 1, "msg": "could not find job", "started": 1} | 22:20 |
| jeblair | it also failed with a similar error ^ | 22:20 |
| pabelanger | In that case, I think we could make the argument to somehow enable host_key_checking per playbook run some how | 22:20 |
| pabelanger | so we know the IP is for the right server | 22:20 |
| pabelanger | then clean up when job is finished | 22:20 |
| mordred | we could chose a known-hosts file just for the job, and set ssh to "auto accept but then die if it changes" | 22:21 |
| jeblair | EmilienM: this seems like a reasonable hypothesis that i'd say go ahead and recheck now if you haven't | 22:21 |
| *** inc0 has quit IRC | 22:21 | |
| EmilienM | ok | 22:21 |
| pabelanger | mordred: ya, that might be safe to do | 22:21 |
| krotscheck | mordred: So, I can write a makefile for sphinx things, but that won't get updates from global-requirements. | 22:21 |
| jeblair | by 'reasonable' i mean 'pull my hair out insane' but... | 22:21 |
| krotscheck | In fact, anything that doesn't use the python tooling doesn't get useful updates from global requirements. | 22:22 |
| krotscheck | mordred: Any suggestions? | 22:22 |
| *** PollyZ has quit IRC | 22:23 | |
| mordred | krotscheck: I think you're either going to wind up with a weird python-files-in-javascript-repos thing, or you're going to need to write a few pieces of the glue over again in js as a proper js build plugin thing | 22:23 |
| mordred | that thing can maybe install sphinx and oslo-sphinx into a build venv and then invoke them on the doc/source dir perhaps? | 22:24 |
| *** sarob has joined #openstack-infra | 22:24 | |
| jeblair | mordred, pabelanger: all 3 instances were in rax-ord, so certainly could have been on the same compute host | 22:24 |
| mordred | krotscheck: or does that sound too gross? | 22:24 |
| mordred | jeblair: yah | 22:24 |
| *** PollyZ has joined #openstack-infra | 22:24 | |
| pabelanger | jeblair: ya, weird for sure | 22:24 |
| krotscheck | mordred: That's what I'm trying to do via make, but keeping the dependencies up to date is the sticking point | 22:25 |
| jeblair | pabelanger, mordred: i think we should call this a reasonable working hypothesis, see if jroll, JayF, or other rackspace folks find it interesting, and then proceed to protect against (and definitively log errors) by checking the host key | 22:25 |
| *** rlandy has quit IRC | 22:25 | |
| *** jsavak has quit IRC | 22:26 | |
| JayF | jeblair: I haven't been following the whole thing; what's the hypothesis/ | 22:26 |
| mordred | krotscheck: yah - I don't think make is usable for exactly that reason | 22:26 |
| mordred | krotscheck: we looked at make for the python stuff a few years ago and avoided it in favor of python-based for that exact reason | 22:26 |
| *** baoli has quit IRC | 22:27 | |
| pabelanger | jeblair: mordred: in the mean time, I think we can add [ssh_connection] into our ansible.cfg file, modifying ssh_args to use a per job UserKnownHostsFile, pointing to a file that contains the current host key of the node | 22:27 |
| pabelanger | if that isn't too complicated | 22:27 |
| jeblair | JayF: that a deleted instance (briefly?) came back online while another instance had its same ip address. | 22:27 |
| *** baoli has joined #openstack-infra | 22:27 | |
| jeblair | JayF: mordred hypothesizes this might happen in the case of a host failure... | 22:28 |
| JayF | jeblair: I don't know enough about the virt side of the house to rule it out; but I find it exceptionally unlikely. If you all have access to file tickets, please do so with the instance uuids of the instances in question | 22:28 |
| jeblair | i'm collecting nova ids | 22:28 |
| JayF | jeblair: and I can poke someone to look at the ticket interally | 22:28 |
| JayF | jeblair: because I agree that's ... really strange behavior | 22:28 |
| *** sarob has quit IRC | 22:28 | |
| mordred | it's almost never what you'd want :) | 22:28 |
| *** vhosakot has quit IRC | 22:29 | |
| *** salv-orlando has quit IRC | 22:29 | |
| krotscheck | mordred: Well, in that case I could just use jsdoc and ignore sphinx altogether | 22:29 |
| JayF | Yeah; I mean, I want to conjecture but I don't have enough real information to do anything more than theorycraft. | 22:29 |
| *** jamesmcarthur has quit IRC | 22:30 | |
| *** salv-orlando has joined #openstack-infra | 22:30 | |
| *** camunoz has joined #openstack-infra | 22:30 | |
| jeblair | the info for the three nodes is at the top of https://etherpad.openstack.org/p/536mveQy8Y | 22:30 |
| mordred | krotscheck: I think for programmer docs that's not insane - javascript devs are going to expect jsdoc docs, no? | 22:31 |
| krotscheck | mordred: Honestly, they're going to expect stackoverflow | 22:31 |
| jeblair | i'm going to open a ticket now | 22:31 |
| mordred | krotscheck: they are, aren't they? | 22:31 |
| krotscheck | mordred: Yep. | 22:31 |
| * krotscheck makes his doc build create stackoverflow | 22:32 | |
| pabelanger | I wonder if we could use pretask to run ssh-keyscan delegate_to: 127.0.0.1 so when tasks run, ssh will then find the new know_hosts file | 22:32 |
| *** ddieterly is now known as ddieterly[away] | 22:32 | |
| *** mriedem has joined #openstack-infra | 22:32 | |
| pabelanger | will test that later tonight | 22:32 |
| *** ddieterly[away] has quit IRC | 22:32 | |
| *** darrenc is now known as darren_afk | 22:34 | |
| *** salv-orlando has quit IRC | 22:34 | |
| *** salv-orlando has joined #openstack-infra | 22:35 | |
| jeblair | JayF: 160616-dfw-0001883 | 22:36 |
| JayF | jeblair: thanks, I've passed it along. | 22:38 |
| *** dmk0202 has quit IRC | 22:39 | |
| jeblair | pabelanger, mordred: i'm going to send my announcement email now; okay? | 22:40 |
| mordred | jeblair: \o/ | 22:40 |
| *** baoli has quit IRC | 22:40 | |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr: generate the reno cache during sdist if reno is installed https://review.openstack.org/330814 | 22:40 |
| dhellmann | mordred, lifeless, ttx: ^^ | 22:41 |
| dhellmann | fungi : ^^ | 22:41 |
| mordred | dhellmann: oooh | 22:41 |
| dhellmann | it feels a bit hacky, but it seems to work | 22:41 |
| jeblair | #success There is no Jenkins, only Zuul. | 22:42 |
| openstackstatus | jeblair: Added success to Success page | 22:42 |
| jeblair | #status log Jenkins is retired: http://lists.openstack.org/pipermail/openstack-dev/2016-June/097584.html | 22:43 |
| openstackstatus | jeblair: finished logging | 22:43 |
| jeblair | infra-root, Shrews, notmorgan, AJaeger, anteaya: ^ | 22:43 |
| jeblair | approving 774 now | 22:44 |
| mordred | woohoo! https://twitter.com/e_monty/status/743574595437879296 | 22:44 |
| pleia2 | :D | 22:44 |
| *** PollyZ_ has joined #openstack-infra | 22:45 | |
| *** asselin__ has joined #openstack-infra | 22:45 | |
| asselin | wow...very cool. | 22:46 |
| nibalizer | jeblair: congratz! | 22:46 |
| *** asselin_ has quit IRC | 22:47 | |
| EmilienM | jeblair: again | 22:47 |
| EmilienM | http://logs.openstack.org/94/330694/4/check/gate-puppet-openstack-integration-3-scenario002-tempest-ubuntu-xenial/6f306a2/console.html#_2016-06-16_22_41_30_123784 | 22:47 |
| * mordred hands everyone most of a slice of cake | 22:47 | |
| jeblair | i deleted 2 stuck tripleo nodes | 22:47 |
| mordred | EmilienM: it happened again? | 22:47 |
| EmilienM | zuul says job is complete but it's not, we're still running tempest :( | 22:48 |
| jeblair | EmilienM: this is back to the 2016-06-16 22:41:29,490 p=19835 u=root | fatal: [node]: FAILED! => {"changed": false, "failed": true, "msg": "async task did not complete within the requested time"} | 22:48 |
| *** PollyZ has quit IRC | 22:48 | |
| jhesketh | Morning | 22:48 |
| jeblair | EmilienM: so the kind of failure that looks more like a timeout -- i don't think we hit the phantom instance issue again | 22:49 |
| *** nwkarsten has quit IRC | 22:49 | |
| jeblair | EmilienM: okay, i think we should recheck that and hold the node so we can poke at it and see if there's a connection issue | 22:49 |
| bknudson_ | zuul works with jenkins job builder? | 22:49 |
| jeblair | bknudson_: yes, it reads jjb yaml files and translates them to playbooks | 22:50 |
| bknudson_ | ah. interesting | 22:50 |
| mordred | bknudson_: you'll find the generated playbooks along side the console logs for jobs | 22:50 |
| *** nwkarsten has joined #openstack-infra | 22:51 | |
| bknudson_ | dsvm jobs | 22:51 |
| *** rbrndt has quit IRC | 22:51 | |
| jeblair | EmilienM: let me know when the job starts after you recheck it and i'll hold the node | 22:52 |
| mordred | bknudson_: all of them. like: http://logs.openstack.org/40/327840/5/gate/gate-cinder-pep8/fcff1aa/_zuul_ansible/ | 22:53 |
| bknudson_ | I must have been looking in the wrong place | 22:53 |
| *** darren_afk is now known as darrenc | 22:55 | |
| *** gordc has quit IRC | 22:55 | |
| *** nwkarsten has quit IRC | 22:56 | |
| *** bpokorny_ has joined #openstack-infra | 22:56 | |
| *** ffrank has quit IRC | 22:56 | |
| bknudson_ | I'm glad I'm not writing this playbook by hand -- http://logs.openstack.org/40/327840/5/gate/gate-cinder-pep8/fcff1aa/_zuul_ansible/playbook | 22:57 |
| bknudson_ | mode: 365 :) | 22:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Re-enable in-progress job url links https://review.openstack.org/330815 | 22:58 |
| jeblair | pabelanger, mordred: ^ | 22:58 |
| EmilienM | jeblair: done, 330694 | 22:58 |
| *** tkelsey has joined #openstack-infra | 22:58 | |
| bknudson_ | found it -- http://logs.openstack.org/40/327840/5/gate/gate-cinder-pep8/fcff1aa/_zuul_ansible/scripts/622aa5d867b547a988d666538c25c782.sh -- finally runs pep8 | 22:59 |
| mordred | jeblair: and until we add it, zl runs just won't have a url while it's running, right? | 22:59 |
| *** xarses_ has quit IRC | 22:59 | |
| jeblair | mordred: they have a url -- telnet:// | 22:59 |
| mordred | oh. duh. | 22:59 |
| *** xarses_ has joined #openstack-infra | 23:00 | |
| jeblair | so, granted, this will not immediately work as a link for most people | 23:00 |
| *** bpokorny has quit IRC | 23:00 | |
| anteaya | jeblair: thank you | 23:00 |
| jeblair | but i'll mention that in a followup announcement | 23:00 |
| anteaya | jeblair: congratulations | 23:00 |
| openstackgerrit | Merged openstack-infra/project-config: Remove jenkins05 - jenkins07 from nodepool https://review.openstack.org/330774 | 23:00 |
| *** mixos has joined #openstack-infra | 23:01 | |
| *** piet has joined #openstack-infra | 23:01 | |
| jeblair | anteaya: thanks -- be sure to have some cake at jenkins' retirement party tomorrow | 23:01 |
| anteaya | zaro: what is its-base? | 23:01 |
| anteaya | jeblair: wooo, sounds delicious | 23:01 |
| anteaya | zaro: I didn't configure anything to do with its-base that I recall | 23:01 |
| *** alaski_ is now known as alaski | 23:02 | |
| EmilienM | jeblair: I need to leave a bit, I might come back later -- feel free to play with https://review.openstack.org/#/c/330694/ - and thanks for your help | 23:02 |
| anteaya | zaro: where are the instructions to configure its-base? | 23:03 |
| *** tkelsey has quit IRC | 23:03 | |
| *** PollyZ_ has quit IRC | 23:03 | |
| mordred | pabelanger: I should be able to land your console-log last patches now, yeah? | 23:04 |
| mordred | pabelanger: since we no longer need to watch jenkins carefully | 23:04 |
| anteaya | zaro: I found https://gerrit.googlesource.com/plugins/its-base/+doc/master/src/main/resources/Documentation/config.md but the first 3 steps are just links and all 3 links 404 for me | 23:04 |
| *** ayoung has quit IRC | 23:07 | |
| jeblair | EmilienM: i held the node it just started on | 23:07 |
| jeblair | EmilienM: | 1745771 | rax-ord | None | ubuntu-xenial | zuul | zl04 | ubuntu-xenial-rax-ord-1745771 | ubuntu-xenial-rax-ord-1745771 | a5aad02d-8bb0-46de-b777-2f2cfd4e194b | 104.239.192.196 | hold | 00:00:00:00 | | 23:07 |
| jeblair | EmilienM: nc 104.239.192.196 8088 | 23:07 |
| *** pradk has quit IRC | 23:07 | |
| *** sarob has joined #openstack-infra | 23:07 | |
| *** edmondsw has quit IRC | 23:07 | |
| jeblair | i also have an open ssh connection to that host | 23:07 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Remove special config for west bifrost host https://review.openstack.org/330552 | 23:07 |
| EmilienM | jeblair: ack | 23:08 |
| mordred | nc the log is so cool | 23:08 |
| *** salv-orl_ has joined #openstack-infra | 23:08 | |
| *** sarob has quit IRC | 23:09 | |
| EmilienM | yeah | 23:09 |
| EmilienM | oh see | 23:09 |
| EmilienM | it exited | 23:09 |
| EmilienM | ah no, it was staled | 23:09 |
| jeblair | yeah, i think we have ~15 minutes to go based on previous runs | 23:10 |
| *** salv-orlando has quit IRC | 23:11 | |
| jeblair | pabelanger: jenkins05-07 still exist | 23:12 |
| jeblair | pabelanger: which means ansible will run jjb on them for the console-log jjb change | 23:12 |
| jeblair | pabelanger: i'm waiting for 330776 to land | 23:12 |
| jeblair | mordred: oh, you approved those ^ | 23:13 |
| mordred | jeblair: I can unapprove real quick | 23:13 |
| jeblair | mordred: might be a good idea | 23:13 |
| openstackgerrit | Merged openstack-infra/system-config: Add Zuul as a first-party CI name https://review.openstack.org/330671 | 23:14 |
| mordred | jeblair: I dropped a -2 block | 23:14 |
| mordred | jeblair: batch mode in gertty would not let me just remove my +A fwiw | 23:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Pre-install pip/virtualenv packages https://review.openstack.org/327472 | 23:15 |
| *** roxanagh_ has joined #openstack-infra | 23:15 | |
| jeblair | mordred: hrm | 23:16 |
| mordred | jeblair: I figured blocking the merge was more important than troubleshooting | 23:16 |
| *** claudiub|2 has joined #openstack-infra | 23:16 | |
| mordred | jeblair: when I opened the "vote" box, it did not load that my previous one was W+1 - so when I voted, it was with +0s which it seemingly took to be no-op | 23:17 |
| *** hongbin has quit IRC | 23:17 | |
| *** asselin has quit IRC | 23:17 | |
| jeblair | mordred: makes sense; that case may not be handled well. | 23:18 |
| *** sarob has joined #openstack-infra | 23:18 | |
| openstackgerrit | Merged openstack-infra/system-config: Decomission jenkins05 - jenkins07 https://review.openstack.org/330776 | 23:19 |
| *** claudiub has quit IRC | 23:20 | |
| *** roxanagh_ has quit IRC | 23:20 | |
| *** ihrachys_ has quit IRC | 23:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Pre-install pip/virtualenv packages https://review.openstack.org/327472 | 23:21 |
| *** ihrachys has joined #openstack-infra | 23:21 | |
| jeblair | restarted nodepool after the final config change | 23:21 |
| *** bhavik has quit IRC | 23:22 | |
| *** amitgandhinz has joined #openstack-infra | 23:23 | |
| *** claudiub has joined #openstack-infra | 23:23 | |
| bknudson_ | oh, now I know why I didn't see the ansible script is because this just started | 23:24 |
| *** isaacb has joined #openstack-infra | 23:24 | |
| jeblair | bknudson_: we've been phasing it in over the past 2 weeks | 23:24 |
| bknudson_ | oh, so just unlucky and picked one that didn't have it | 23:25 |
| jeblair | but yeah, hit the 100% mark a couple hours ago | 23:25 |
| mordred | jeblair: k. so, it should be safe to re-approve now that 330776 has landed, yeah? | 23:26 |
| jeblair | mordred: after i delete the hosts, after ansible runs | 23:26 |
| mordred | jeblair: there's no jenkins.o.o in site.pp - ansible might run on them, but it shouldn't run jjb? | 23:27 |
| mordred | it should relaly just ensure our ssh keys are still there | 23:27 |
| *** claudiub|2 has quit IRC | 23:27 | |
| *** amitgandhinz has quit IRC | 23:27 | |
| jeblair | mordred: good point | 23:27 |
| mordred | but I'll wait anyway | 23:27 |
| mordred | just cause | 23:27 |
| jeblair | mordred, pabelanger: 119 nodes are currently involved in testing that patchset :( | 23:29 |
| EmilienM | jeblair: tempest still running /me cross finger | 23:29 |
| *** sarob has quit IRC | 23:29 | |
| jeblair | er, patch series | 23:29 |
| mordred | pabelanger: how about we squash it into one patch | 23:29 |
| jeblair | i'd like us to reinstate the files stanzas for project-config too | 23:30 |
| mordred | we don't need to phase it in anymore | 23:30 |
| jeblair | multiple patches can still be handy for dealing with rebases in this sort of thing | 23:30 |
| mordred | yah. but in this case, if I push it up squashed now, it should cancel the other builds, and then do less when we his +A in just a smidge | 23:31 |
| pabelanger | mordred: jeblair: Ya, squashing is fine. not that jenkins is gone | 23:31 |
| pabelanger | now* | 23:31 |
| * mordred on that | 23:32 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Ensure console-log is our last publisher https://review.openstack.org/329984 | 23:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Revert "Remove skip test logic for project-config" https://review.openstack.org/330821 | 23:33 |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: use .txt as logfile extension https://review.openstack.org/330213 | 23:33 |
| jeblair | pabelanger: https://review.openstack.org/330821 | 23:34 |
| pabelanger | jeblair: Ya, we'll need to also add other-requirements.txt now, but we can do that in a follow up patch | 23:34 |
| openstackgerrit | Merged openstack-infra/system-config: Remove Jenkins documentation; replace with zuul-launcher https://review.openstack.org/330200 | 23:35 |
| openstackgerrit | Merged openstack-infra/system-config: Remove jenkins playbooks and tools https://review.openstack.org/330202 | 23:35 |
| EmilienM | jeblair: it worked /me afk now | 23:35 |
| fungi | jeblair: awesome work! | 23:36 |
| jeblair | pabelanger: er, i meant for us to hold off on approving that until we were sure we had consensus | 23:36 |
| jeblair | pabelanger: cause i don't want to have a revert war with other infra-roots | 23:36 |
| mordred | jeblair: should we drop compare-xml ? like, is it valuable anymore? | 23:36 |
| pabelanger | jeblair: Oh | 23:36 |
| pabelanger | sure | 23:36 |
| *** sarob has joined #openstack-infra | 23:36 | |
| mordred | jeblair, pabelanger: I'm in favor, so count me in | 23:36 |
| pabelanger | jeblair: I read it as, revert, because original patch didn't have consensus | 23:37 |
| jeblair | oh, i guess i could have worded that more clearly | 23:37 |
| * fungi is catchign up now on how/why we had several jobs in one log | 23:37 | |
| pabelanger | bp | 23:37 |
| pabelanger | np* | 23:37 |
| jeblair | either way, it's not so important we can't let this one simmer for a bit | 23:38 |
| jeblair | at least long enough for clarkb to -2 it if he wants ;) | 23:38 |
| jeblair | fungi: yeah. um. wow. there's a ticket. | 23:38 |
| clarkb | aroo? | 23:39 |
| clarkb | I have internets at new house now and truck ewnted saturday so next few days are sort of busy | 23:40 |
| anteaya | yay! | 23:40 |
| fungi | mordred: krotscheck: i'd vote for using distro packages of the utilities/libs you need to generate docs and add an other-requirements.txt file listing those as [test] profile entries, so that we'll autoinstall the tools needed to build javascript docs during the job run. relying on distro packages for non-javascript things seems slightly less weird than expecting someone to pip install stuff to build | 23:41 |
| fungi | javascript docs | 23:41 |
| mordred | clarkb: \o/ | 23:42 |
| *** chlong has quit IRC | 23:42 | |
| mordred | clarkb: also, while you were out, jeblair finished getting rid of jenkins | 23:42 |
| *** ayoung has joined #openstack-infra | 23:44 | |
| jeblair | it seems like geard thinks it has more workers than it really does | 23:45 |
| *** piet has quit IRC | 23:45 | |
| jeblair | i'm unsure of the cause, but we may want to restart zuul | 23:45 |
| *** bpokorny_ has quit IRC | 23:45 | |
| jeblair | including workers for nodes which have been deleted | 23:46 |
| *** piet has joined #openstack-infra | 23:46 | |
| fungi | if the zuul scheduler/geard hasn't been restarted since the last jenkins master went down, we probably ought to do that just so that we're only getting job registrations from zuul-launchers | 23:46 |
| *** bpokorny has joined #openstack-infra | 23:46 | |
| jeblair | oh huh | 23:47 |
| fungi | though in theory they _should_ be identical entries | 23:47 |
| *** bhavik has joined #openstack-infra | 23:47 | |
| jeblair | i think this may be a case where nodepool deleted the node before zuul cleanly shut it down | 23:47 |
| fungi | oh that's fun. race? | 23:47 |
| *** salv-orl_ has quit IRC | 23:47 | |
| jeblair | well, i think that without a keepalive on that connection, geard won't notice it's gone | 23:47 |
| jeblair | oh, but the connection comes from the launcher | 23:48 |
| jeblair | so, erm, i dunno. | 23:48 |
| *** salv-orlando has joined #openstack-infra | 23:48 | |
| *** xarses_ has quit IRC | 23:48 | |
| *** kzaitsev_mb has quit IRC | 23:49 | |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: delay node assignment under load https://review.openstack.org/330786 | 23:51 |
| zaro | anteaya: its-base is a common code that all its (issue tracking system) plugins depend on. | 23:51 |
| clarkb | mordred: I am happy to report that my fios experience so far has been good. they terminate rj45 not coax which is nice and I can get up to gigabits if I pay lots of money | 23:51 |
| *** ihrachys has quit IRC | 23:51 | |
| mordred | clarkb: I'll be interested to hear if you get intermittent pauses from time to time | 23:51 |
| mordred | clarkb: mine actually seems to be better since I've been back | 23:51 |
| mordred | so it might have just be a point in time thing too | 23:51 |
| *** Julien-zte has quit IRC | 23:52 | |
| jeblair | restarting zuul | 23:52 |
| anteaya | zaro: I don't recall installing and configuring its-base in my local gerrit | 23:54 |
| anteaya | zaro: have you a link to any instructions on how I should install and configure? | 23:54 |
| mordred | jeblair: aw crap. we have a bunch of conference talk slides to update now | 23:54 |
| *** johnny_ has quit IRC | 23:54 | |
| jeblair | hehe | 23:54 |
| jeblair | mordred: i don't plan on speaking at conferences until i have something to say | 23:54 |
| zaro | anteaya: you don’t need to install it. just installing its-storyboard.jar was all that was needed | 23:54 |
| anteaya | zaro: oh, I was hoping that was the thing I did wrong on my installation | 23:55 |
| anteaya | zaro: since you call out its-base specificially in the commit message for your patch | 23:55 |
| jeblair | #status log all jenkins servers have been deleted | 23:56 |
| jeblair | mordred: ^ | 23:56 |
| openstackstatus | jeblair: finished logging | 23:56 |
| *** sdake has quit IRC | 23:57 | |
| jeblair | and i cleared the ansible inventory cache | 23:57 |
| zaro | anteaya: we need to build its-storyboard and put it on tarballs.o.o so we can install it with puppet | 23:59 |
| zaro | anteaya: building requires building its-base. | 23:59 |
| zaro | so its-base is only required for building not required to run | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!