| ianw | #status log graphite restarted on graphite.o.o to pick up logging changes from https://review.openstack.org/#/c/541488/ | 00:06 |
|---|---|---|
| openstackstatus | ianw: finished logging | 00:06 |
| *** HeOS has quit IRC | 00:19 | |
| *** anticw has joined #openstack-infra | 00:20 | |
| *** gyee has quit IRC | 00:21 | |
| *** masber has quit IRC | 00:22 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add a documentation index page https://review.openstack.org/555505 | 00:22 |
| openstackgerrit | melissaml proposed openstack-infra/zuul master: fix typos in documentation https://review.openstack.org/555537 | 00:23 |
| openstackgerrit | Merged openstack-infra/system-config master: Add limestone credentials to nodepool's cloud.yaml https://review.openstack.org/555470 | 00:24 |
| openstackgerrit | Merged openstack-infra/system-config master: Add NO_TIMEOUT flag for reprepro https://review.openstack.org/550683 | 00:24 |
| *** weshay is now known as weshay_PTO | 00:25 | |
| *** boden has joined #openstack-infra | 00:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove gerrit legacy jobs https://review.openstack.org/552723 | 00:30 |
| *** hamzy has joined #openstack-infra | 00:30 | |
| *** boden has quit IRC | 00:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Test growroot in boot tests https://review.openstack.org/555103 | 00:30 |
| *** danpawlik has joined #openstack-infra | 00:31 | |
| *** danpawlik has quit IRC | 00:36 | |
| *** iyamahat has quit IRC | 00:38 | |
| *** yamahata has quit IRC | 00:38 | |
| *** odyssey4me has quit IRC | 00:38 | |
| *** odyssey4me has joined #openstack-infra | 00:39 | |
| *** agopi|out has joined #openstack-infra | 00:43 | |
| *** edmondsw has quit IRC | 00:46 | |
| dmsimard | #thanks smcginnis for doing those release countdowns on openstack-dev :D | 00:46 |
| openstackstatus | dmsimard: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 00:46 |
| openstackgerrit | Merged openstack-infra/system-config master: Add review-dev01.o.o to cacti.o.o https://review.openstack.org/555498 | 00:53 |
| *** eernst has quit IRC | 00:53 | |
| *** iyamahat has joined #openstack-infra | 00:58 | |
| *** e0ne_ has quit IRC | 01:00 | |
| *** esberglu has joined #openstack-infra | 01:03 | |
| *** ihrachys has quit IRC | 01:04 | |
| *** wolverineav has quit IRC | 01:04 | |
| *** esberglu has quit IRC | 01:08 | |
| *** danpawlik has joined #openstack-infra | 01:09 | |
| *** mriedem has quit IRC | 01:10 | |
| *** masber has joined #openstack-infra | 01:12 | |
| *** harlowja_ has quit IRC | 01:13 | |
| *** danpawlik has quit IRC | 01:14 | |
| *** yamahata has joined #openstack-infra | 01:14 | |
| *** hongbin has joined #openstack-infra | 01:15 | |
| *** rlandy|bbl is now known as rlandy | 01:22 | |
| *** bhagyashris has quit IRC | 01:26 | |
| *** germs has left #openstack-infra | 01:26 | |
| *** bhagyashris has joined #openstack-infra | 01:27 | |
| *** chrisyang_0660 has joined #openstack-infra | 01:32 | |
| *** samuelbeau has quit IRC | 01:33 | |
| *** wolverineav has joined #openstack-infra | 01:36 | |
| *** jamesmcarthur has joined #openstack-infra | 01:41 | |
| *** danpawlik has joined #openstack-infra | 01:47 | |
| *** danpawlik has quit IRC | 01:51 | |
| *** iyamahat has quit IRC | 01:52 | |
| *** zhurong has joined #openstack-infra | 01:52 | |
| *** ykarel has joined #openstack-infra | 02:05 | |
| *** rlandy has quit IRC | 02:15 | |
| *** jamesmcarthur has quit IRC | 02:20 | |
| *** danpawlik has joined #openstack-infra | 02:26 | |
| *** germs has joined #openstack-infra | 02:28 | |
| *** danpawlik has quit IRC | 02:31 | |
| *** edmondsw has joined #openstack-infra | 02:34 | |
| *** ramishra has joined #openstack-infra | 02:39 | |
| *** edmondsw has quit IRC | 02:39 | |
| *** masber has quit IRC | 02:41 | |
| *** ykarel has quit IRC | 02:44 | |
| *** salv-orl_ has joined #openstack-infra | 02:47 | |
| *** andreas_s has joined #openstack-infra | 02:48 | |
| *** salv-orlando has quit IRC | 02:50 | |
| *** rosmaita has quit IRC | 02:50 | |
| *** psachin has joined #openstack-infra | 02:51 | |
| *** andreas_s has quit IRC | 02:53 | |
| *** wolverineav has quit IRC | 02:54 | |
| *** wolverineav has joined #openstack-infra | 02:55 | |
| *** jamesmcarthur has joined #openstack-infra | 02:55 | |
| *** ykarel has joined #openstack-infra | 02:57 | |
| *** wolverineav has quit IRC | 02:59 | |
| *** danpawlik has joined #openstack-infra | 03:00 | |
| *** danpawlik has quit IRC | 03:05 | |
| *** dhajare has joined #openstack-infra | 03:11 | |
| *** ykarel has quit IRC | 03:15 | |
| *** masber has joined #openstack-infra | 03:19 | |
| *** eernst has joined #openstack-infra | 03:20 | |
| *** danpawlik has joined #openstack-infra | 03:36 | |
| *** danpawlik has quit IRC | 03:41 | |
| *** liusheng has quit IRC | 03:47 | |
| *** liusheng has joined #openstack-infra | 03:47 | |
| *** hongbin has quit IRC | 03:47 | |
| *** eernst has quit IRC | 03:54 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: [WIP] Move environment create/cleanup to python https://review.openstack.org/555463 | 04:01 |
| *** eernst has joined #openstack-infra | 04:01 | |
| *** germs has quit IRC | 04:01 | |
| *** germs has joined #openstack-infra | 04:02 | |
| *** zhurong has quit IRC | 04:04 | |
| *** VW has quit IRC | 04:04 | |
| *** VW has joined #openstack-infra | 04:06 | |
| *** danpawlik has joined #openstack-infra | 04:11 | |
| *** VW has quit IRC | 04:12 | |
| *** niska has quit IRC | 04:16 | |
| *** danpawlik has quit IRC | 04:17 | |
| *** harlowja has joined #openstack-infra | 04:19 | |
| *** VW has joined #openstack-infra | 04:20 | |
| *** persia has quit IRC | 04:20 | |
| *** eernst has quit IRC | 04:20 | |
| *** weshay_PTO is now known as weshay | 04:21 | |
| *** persia has joined #openstack-infra | 04:21 | |
| *** niska has joined #openstack-infra | 04:21 | |
| *** edmondsw has joined #openstack-infra | 04:23 | |
| *** VW has quit IRC | 04:25 | |
| *** links has joined #openstack-infra | 04:26 | |
| *** edmondsw has quit IRC | 04:27 | |
| *** ykarel has joined #openstack-infra | 04:28 | |
| *** pgadiya has joined #openstack-infra | 04:29 | |
| *** kambiz has quit IRC | 04:36 | |
| *** kuromagi has quit IRC | 04:36 | |
| *** gus has quit IRC | 04:36 | |
| *** kuromagi has joined #openstack-infra | 04:36 | |
| *** gus has joined #openstack-infra | 04:37 | |
| *** kambiz has joined #openstack-infra | 04:38 | |
| *** udesale has joined #openstack-infra | 04:41 | |
| *** VW has joined #openstack-infra | 04:41 | |
| *** pgadiya has quit IRC | 04:43 | |
| *** danpawlik has joined #openstack-infra | 04:45 | |
| *** pickle has quit IRC | 04:45 | |
| *** VW has quit IRC | 04:47 | |
| *** ykarel_ has joined #openstack-infra | 04:47 | |
| *** danpawlik has quit IRC | 04:50 | |
| *** ykarel has quit IRC | 04:50 | |
| *** pgadiya has joined #openstack-infra | 04:58 | |
| *** harlowja has quit IRC | 05:12 | |
| *** lpetrut has joined #openstack-infra | 05:12 | |
| *** VW has joined #openstack-infra | 05:14 | |
| *** imacdonn has quit IRC | 05:15 | |
| *** imacdonn has joined #openstack-infra | 05:15 | |
| *** danpawlik has joined #openstack-infra | 05:17 | |
| *** pgadiya has quit IRC | 05:19 | |
| *** danpawlik has quit IRC | 05:22 | |
| *** gyankum has joined #openstack-infra | 05:23 | |
| *** VW has quit IRC | 05:24 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean master: [DNM] Fix handling of openrc based init systems https://review.openstack.org/548604 | 05:26 |
| prometheanfire | clarkb: cloudnull ^ | 05:26 |
| prometheanfire | first pass at the networkd stuff | 05:27 |
| prometheanfire | no tests, missing vlan | 05:27 |
| prometheanfire | and bonding is probably broken | 05:27 |
| prometheanfire | but works for dhcp :D | 05:28 |
| prometheanfire | also... https://review.openstack.org/493443 has been out for a LONG time | 05:30 |
| *** pgadiya has joined #openstack-infra | 05:33 | |
| *** agopi|out has quit IRC | 05:33 | |
| openstackgerrit | Merged openstack/diskimage-builder master: proliant-tools: add net-tools package to support hpsum utility https://review.openstack.org/548154 | 05:36 |
| *** yamahata has quit IRC | 05:38 | |
| jlvillal | gerritbot review request: https://review.openstack.org/#/c/545469/ Some cleanup/refactoring and adding unit tests. Has one +2 Thanks! | 05:42 |
| ianw | AJaeger: do you know if there's any non-legacy replacement for /usr/local/jenkins/slave_scripts/version-properties.sh ? | 05:48 |
| *** danpawlik has joined #openstack-infra | 05:50 | |
| dirk | AJaeger: as a openstack-zuul-job it might be good to go.. | 05:54 |
| *** danpawlik has quit IRC | 05:56 | |
| *** armaan has joined #openstack-infra | 06:09 | |
| *** armaan has quit IRC | 06:10 | |
| *** VW has joined #openstack-infra | 06:15 | |
| *** VW has quit IRC | 06:20 | |
| *** zhurong has joined #openstack-infra | 06:21 | |
| *** lpetrut has quit IRC | 06:22 | |
| *** VW has joined #openstack-infra | 06:23 | |
| *** VW has quit IRC | 06:29 | |
| *** danpawlik has joined #openstack-infra | 06:30 | |
| *** dbecker has quit IRC | 06:30 | |
| *** danpawlik has quit IRC | 06:34 | |
| *** lpetrut has joined #openstack-infra | 06:36 | |
| *** vivsoni_ has quit IRC | 06:40 | |
| *** vivsoni_ has joined #openstack-infra | 06:41 | |
| *** vivsoni_ has quit IRC | 06:43 | |
| *** vivsoni_ has joined #openstack-infra | 06:43 | |
| *** armaan has joined #openstack-infra | 06:44 | |
| *** dbecker has joined #openstack-infra | 06:45 | |
| *** alexchadin has joined #openstack-infra | 06:50 | |
| frickler | ianw: just noticed a new error in the cron mail from zuul01: "error: '/root/.bup/' is not a bup repository; run "bup init"" do you want to continue with fixing this? | 06:55 |
| ianw | frickler: did we merge the bup removal thing? | 06:55 |
| ianw | frickler: no, https://review.openstack.org/#/c/553697/ want to review that first? | 06:56 |
| *** VW has joined #openstack-infra | 06:57 | |
| *** ykarel_ is now known as ykarel|lunch | 06:58 | |
| *** lihi has quit IRC | 07:00 | |
| *** lihi has joined #openstack-infra | 07:01 | |
| *** VW has quit IRC | 07:02 | |
| *** kiennt26 has quit IRC | 07:05 | |
| frickler | ianw: oh, I didn't check that, but assumed that it had been merged and thus caused the new error to show up. +3 | 07:06 |
| ianw | frickler: cool, thanks; i've fixed all the setup for it, so it should work, once bup is installed properly & doesn't keep disappearing :) | 07:08 |
| ianw | it might be questionable if we really need to backup all the zuul logs ... better safe than sorry i guess | 07:09 |
| *** ethfci has quit IRC | 07:13 | |
| ianw | pabelanger / clarkb : so I think the stack from https://review.openstack.org/555633 is GTG. the gerrit-buck job publishes the war to the logs (http://logs.openstack.org/33/555633/7/check/gerrit-buck/89b2998/gerrit/gerrit/buck-out/gen/) and the post job should use the tarball/artifacts stuff to publish to tarballs.o.o | 07:15 |
| *** pgadiya has quit IRC | 07:16 | |
| *** jtomasek has joined #openstack-infra | 07:18 | |
| *** pcaruana has joined #openstack-infra | 07:21 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Don't remove libjs-jquery https://review.openstack.org/553697 | 07:21 |
| *** VW has joined #openstack-infra | 07:27 | |
| AJaeger | ianw: no, there's no replacement yet that I'm aware of | 07:28 |
| *** pgadiya has joined #openstack-infra | 07:30 | |
| *** zhurong has quit IRC | 07:30 | |
| *** VW has quit IRC | 07:34 | |
| *** florianf has joined #openstack-infra | 07:43 | |
| *** zhaochao has joined #openstack-infra | 07:48 | |
| *** lpetrut has quit IRC | 07:56 | |
| *** danpawlik has joined #openstack-infra | 07:58 | |
| *** edmondsw has joined #openstack-infra | 07:59 | |
| *** kjackal has quit IRC | 07:59 | |
| *** ianychoi has quit IRC | 08:00 | |
| *** ianychoi has joined #openstack-infra | 08:00 | |
| *** danpawlik has quit IRC | 08:01 | |
| *** electrofelix has joined #openstack-infra | 08:01 | |
| *** danpawlik has joined #openstack-infra | 08:01 | |
| *** edmondsw has quit IRC | 08:03 | |
| *** VW has joined #openstack-infra | 08:08 | |
| *** VW has quit IRC | 08:12 | |
| *** HeOS has joined #openstack-infra | 08:14 | |
| *** tesseract has joined #openstack-infra | 08:20 | |
| *** VW has joined #openstack-infra | 08:28 | |
| *** _alastor_ has quit IRC | 08:31 | |
| *** ralonsoh has joined #openstack-infra | 08:34 | |
| *** gfidente has joined #openstack-infra | 08:35 | |
| *** amoralej|off is now known as amoralej | 08:36 | |
| *** andreas_s has joined #openstack-infra | 08:37 | |
| *** cshastri has joined #openstack-infra | 08:38 | |
| *** VW has quit IRC | 08:38 | |
| *** armaan has quit IRC | 08:40 | |
| *** pgadiya has quit IRC | 08:41 | |
| *** pgadiya has joined #openstack-infra | 08:41 | |
| *** armaan has joined #openstack-infra | 08:44 | |
| *** tosky has joined #openstack-infra | 08:45 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** armaan has quit IRC | 08:51 | |
| *** greghaynes has quit IRC | 08:54 | |
| *** greghaynes has joined #openstack-infra | 08:55 | |
| *** andreas_s has quit IRC | 08:56 | |
| *** andreas_s_ has joined #openstack-infra | 08:58 | |
| *** cshastri has quit IRC | 08:58 | |
| *** Adri2000 has joined #openstack-infra | 08:59 | |
| *** Adri2000 has joined #openstack-infra | 08:59 | |
| *** andreas__ has joined #openstack-infra | 08:59 | |
| *** bauzas is now known as bauwser | 08:59 | |
| *** andrea___ has joined #openstack-infra | 09:01 | |
| *** bauwser is now known as bauzas | 09:01 | |
| *** hasharAway is now known as hashar | 09:02 | |
| *** bauzas is now known as bauwser | 09:02 | |
| *** andreas_s_ has quit IRC | 09:02 | |
| *** andrea___ has quit IRC | 09:03 | |
| *** andreas_s has joined #openstack-infra | 09:04 | |
| *** andreas__ has quit IRC | 09:04 | |
| openstackgerrit | ShaoHe Feng proposed openstack-infra/project-config master: add python-cyborgclient to openstack repo https://review.openstack.org/555707 | 09:05 |
| *** jpich has joined #openstack-infra | 09:06 | |
| *** andreas_s has quit IRC | 09:11 | |
| *** lucas-afk is now known as lucasagomes | 09:13 | |
| *** alexchadin has quit IRC | 09:14 | |
| *** alexchadin has joined #openstack-infra | 09:18 | |
| *** alexchadin has quit IRC | 09:18 | |
| *** alexchadin has joined #openstack-infra | 09:18 | |
| *** alexchadin has quit IRC | 09:19 | |
| *** andreas_s has joined #openstack-infra | 09:19 | |
| *** alexchadin has joined #openstack-infra | 09:19 | |
| *** andreas_s has quit IRC | 09:19 | |
| *** alexchadin has quit IRC | 09:20 | |
| *** alexchadin has joined #openstack-infra | 09:20 | |
| *** alexchadin has quit IRC | 09:20 | |
| *** rcernin has quit IRC | 09:20 | |
| *** alexchadin has joined #openstack-infra | 09:21 | |
| *** andreas_s has joined #openstack-infra | 09:22 | |
| *** gibi is now known as giblet | 09:23 | |
| *** alexchadin has quit IRC | 09:25 | |
| *** lpetrut has joined #openstack-infra | 09:27 | |
| *** lpetrut_ has joined #openstack-infra | 09:28 | |
| *** lpetrut has quit IRC | 09:28 | |
| openstackgerrit | ShaoHe Feng proposed openstack-infra/project-config master: add python-cyborgclient to openstack repo https://review.openstack.org/555707 | 09:30 |
| *** dhajare has quit IRC | 09:30 | |
| *** panda|off is now known as panda | 09:33 | |
| *** e0ne has joined #openstack-infra | 09:37 | |
| *** derekh has joined #openstack-infra | 09:37 | |
| openstackgerrit | ShaoHe Feng proposed openstack-infra/project-config master: add python-cyborgclient to openstack repo https://review.openstack.org/555707 | 09:38 |
| *** VW has joined #openstack-infra | 09:39 | |
| *** dbecker has quit IRC | 09:39 | |
| *** ykarel|lunch is now known as ykarel | 09:40 | |
| *** VW has quit IRC | 09:43 | |
| *** dsariel has quit IRC | 09:44 | |
| *** edmondsw has joined #openstack-infra | 09:47 | |
| *** edmondsw has quit IRC | 09:51 | |
| *** alexchadin has joined #openstack-infra | 09:52 | |
| *** bmace has quit IRC | 09:54 | |
| *** bmace has joined #openstack-infra | 09:54 | |
| *** andreas_s has quit IRC | 09:57 | |
| *** dsariel has joined #openstack-infra | 09:57 | |
| *** alexchadin has quit IRC | 09:57 | |
| openstackgerrit | ShaoHe Feng proposed openstack-infra/project-config master: Set up python-cyborgclient repository https://review.openstack.org/555722 | 09:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Add ansible-role-statsd zuul jobs https://review.openstack.org/554648 | 10:00 |
| *** andreas_s_ has joined #openstack-infra | 10:00 | |
| *** dsariel has quit IRC | 10:03 | |
| *** yamamoto has quit IRC | 10:05 | |
| *** yamamoto has joined #openstack-infra | 10:06 | |
| *** rcernin has joined #openstack-infra | 10:08 | |
| *** andreas_s_ has quit IRC | 10:09 | |
| *** andreas_s has joined #openstack-infra | 10:09 | |
| *** yamamoto has quit IRC | 10:11 | |
| *** andreas_s has quit IRC | 10:14 | |
| *** larainema has quit IRC | 10:15 | |
| *** VW has joined #openstack-infra | 10:18 | |
| *** andreas_s has joined #openstack-infra | 10:18 | |
| *** stephenfin is now known as finucannot | 10:19 | |
| *** alexchadin has joined #openstack-infra | 10:20 | |
| *** VW has quit IRC | 10:23 | |
| *** andreas_s has quit IRC | 10:23 | |
| *** alexchadin has quit IRC | 10:25 | |
| *** dbecker has joined #openstack-infra | 10:26 | |
| *** alexchadin has joined #openstack-infra | 10:28 | |
| *** andreas_s has joined #openstack-infra | 10:28 | |
| *** andreas_s has quit IRC | 10:32 | |
| *** sambetts|afk is now known as sambetts | 10:32 | |
| *** boden has joined #openstack-infra | 10:33 | |
| *** andreas_s has joined #openstack-infra | 10:34 | |
| *** andreas_s has quit IRC | 10:34 | |
| *** andreas_s has joined #openstack-infra | 10:37 | |
| *** alexchadin has quit IRC | 10:37 | |
| *** andreas_s_ has joined #openstack-infra | 10:39 | |
| *** dtantsur|afk is now known as dtantsur | 10:41 | |
| *** andreas_s has quit IRC | 10:42 | |
| *** zoli is now known as zoli|lunch | 10:42 | |
| *** andreas_s has joined #openstack-infra | 10:42 | |
| *** pblaho has joined #openstack-infra | 10:43 | |
| *** andreas_s_ has quit IRC | 10:43 | |
| *** pbourke has quit IRC | 10:43 | |
| *** pbourke has joined #openstack-infra | 10:44 | |
| *** alexchadin has joined #openstack-infra | 10:44 | |
| *** larainema has joined #openstack-infra | 10:47 | |
| *** alexchadin has quit IRC | 10:49 | |
| *** dsariel has joined #openstack-infra | 10:50 | |
| *** alexchadin has joined #openstack-infra | 10:55 | |
| *** VW has joined #openstack-infra | 10:56 | |
| *** alexchadin has quit IRC | 10:56 | |
| *** alexchadin has joined #openstack-infra | 11:01 | |
| *** VW has quit IRC | 11:01 | |
| *** annp has quit IRC | 11:06 | |
| *** udesale has quit IRC | 11:06 | |
| *** yamamoto has joined #openstack-infra | 11:07 | |
| *** alexchadin has quit IRC | 11:10 | |
| *** alexchadin has joined #openstack-infra | 11:12 | |
| *** yamamoto has quit IRC | 11:13 | |
| *** dprince has joined #openstack-infra | 11:14 | |
| *** adarazs is now known as adarazs_lunch | 11:16 | |
| *** katkapilatova has joined #openstack-infra | 11:22 | |
| *** katkapilatova has left #openstack-infra | 11:22 | |
| *** VW has joined #openstack-infra | 11:27 | |
| *** pcaruana has quit IRC | 11:27 | |
| *** HeOS has quit IRC | 11:28 | |
| *** HeOS has joined #openstack-infra | 11:28 | |
| *** VW has quit IRC | 11:32 | |
| *** HeOS has quit IRC | 11:43 | |
| *** HeOS has joined #openstack-infra | 11:47 | |
| *** arxcruz|rover is now known as arxcruz|rover|lu | 11:50 | |
| *** pgadiya has quit IRC | 11:53 | |
| *** electrofelix has quit IRC | 11:55 | |
| *** jaosorior has quit IRC | 11:56 | |
| *** yamamoto has joined #openstack-infra | 11:56 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** rfolco|off is now known as rfolco|ruck | 11:57 | |
| *** dbecker has quit IRC | 11:57 | |
| *** sree_ has joined #openstack-infra | 11:59 | |
| *** sree_ is now known as Guest19457 | 11:59 | |
| *** edmondsw has joined #openstack-infra | 12:03 | |
| *** jlabarre has joined #openstack-infra | 12:03 | |
| *** VW has joined #openstack-infra | 12:04 | |
| *** VW has quit IRC | 12:08 | |
| *** yamamoto has joined #openstack-infra | 12:10 | |
| *** odyssey4me has quit IRC | 12:12 | |
| *** odyssey4me has joined #openstack-infra | 12:12 | |
| *** tosky has quit IRC | 12:15 | |
| *** tosky has joined #openstack-infra | 12:15 | |
| *** adarazs_lunch is now known as adarazs | 12:17 | |
| bhagyashris | Hi Team can anyone please guide me how to pass specific microversion while writing test in tempest | 12:18 |
| fungi | bhagyashris: probably a better question for the tempest maintainers in #openstack-qa | 12:19 |
| *** greghaynes has quit IRC | 12:19 | |
| bhagyashris | fungi: Thank you | 12:19 |
| *** greghaynes has joined #openstack-infra | 12:19 | |
| * fungi doesn't know at least, sorry | 12:19 | |
| bhagyashris | no issues thanks for inputs :) | 12:20 |
| fungi | you're welcome | 12:21 |
| dhellmann | based on http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all it looks like zuul's memory use is ratcheted up a notch this morning | 12:21 |
| dhellmann | is it still safe for me to be submitting lower-constraints job patches or should I hold off? | 12:22 |
| *** zoli|lunch is now known as zoli | 12:22 | |
| dhellmann | fwiw, I have 230 more patches to go | 12:23 |
| fungi | dhellmann: yeah, while a handful at a time likely won't shift that, we're definitely at the point where we probably need to restart the scheduler again to clear all the (presumably unused most of the time) memory the python process has claimed | 12:25 |
| *** yamamoto has quit IRC | 12:25 | |
| fungi | looks like we made it a month without a scheduler restart this time though? wow | 12:26 |
| fungi | infra-root: ^ | 12:26 |
| fungi | not sure if there were any pending things we wanted to adjust/check during a restart, given how long it's been now | 12:27 |
| *** efried is now known as fried_rice | 12:27 | |
| dhellmann | ok, I'll push another batch and continue to keep an eye on that chart | 12:28 |
| *** kjackal has joined #openstack-infra | 12:29 | |
| *** alexchadin has quit IRC | 12:29 | |
| *** pcaruana has joined #openstack-infra | 12:30 | |
| *** VW has joined #openstack-infra | 12:30 | |
| *** rcernin has quit IRC | 12:30 | |
| *** alexchadin has joined #openstack-infra | 12:33 | |
| *** rlandy has joined #openstack-infra | 12:34 | |
| *** HeOS has quit IRC | 12:35 | |
| *** HeOS has joined #openstack-infra | 12:37 | |
| *** arxcruz|rover|lu is now known as arxcruz|rover | 12:37 | |
| dmsimard | infra-root: just a reminder that I stopped a ~month long process on nl03 puppet apply process that was capping CPU on nl03 which handles inap, vexxhost and linaro in case you had noticed any odd behavior with the node, everything seems to be back to normal now: | 12:40 |
| dmsimard | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64813&rra_id=0&view_type=tree&graph_start=1519303020&graph_end=1521808620 | 12:40 |
| *** VW has quit IRC | 12:41 | |
| *** dhajare has joined #openstack-infra | 12:42 | |
| *** lucasagomes is now known as lucas-hungry | 12:43 | |
| frickler | seems the swapping activity on zuul01 is slowly growing. not critical yet, but if someone could do a scheduled restart later today that would probably be safer than waiting for the next memory bump http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64793&rra_id=all | 12:45 |
| *** trown has joined #openstack-infra | 12:47 | |
| *** jamesmcarthur has quit IRC | 12:49 | |
| *** jamesmca_ has joined #openstack-infra | 12:49 | |
| dmsimard | kashyap: so just to recap, your solution to git review issues was to drop the automatic rebase ? | 12:52 |
| kashyap | dmsimard: Yes | 12:52 |
| dmsimard | kashyap: I wonder if it would be worth considering documenting that | 12:52 |
| *** chrisyang_0660 has quit IRC | 12:52 | |
| dmsimard | kashyap: like in a "troubleshooting" section, or faq, or whatever | 12:52 |
| fungi | had you previously been explicitly asking it to rebase and push your changes? | 12:53 |
| fungi | or are you talking about the temporary rebase it performs to check that your change doesn't have merge conflicts wit the target branch? | 12:54 |
| *** dsariel has quit IRC | 12:57 | |
| *** HeOS has quit IRC | 12:58 | |
| dmsimard | fungi: I'm not 100% sure, I've just been following up with kashyap during the different mornings of this week :p I believe there was a significant overhead to submitting his nova patches when rebases were enabled | 12:59 |
| * kashyap goes into a call; bbiab | 13:00 | |
| dmsimard | I was questioning whether that had to do with the sheer size of the nova repo | 13:00 |
| kashyap | dmsimard: Yes, should be documented for sure | 13:00 |
| kashyap | fungi: Yeah, it's the temporary rebase thing | 13:01 |
| *** mriedem has joined #openstack-infra | 13:03 | |
| *** hemna_ has quit IRC | 13:03 | |
| fungi | aha, interesting | 13:05 |
| fungi | was it cpu-bound? | 13:05 |
| *** Goneri has joined #openstack-infra | 13:06 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** yamamoto has quit IRC | 13:08 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** yamamoto has quit IRC | 13:08 | |
| smcginnis | dmsimard: Aw, thanks. And you're welcome. :) | 13:13 |
| *** yamamoto has joined #openstack-infra | 13:15 | |
| *** alexchadin has quit IRC | 13:15 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Import python_venv_build ansible role https://review.openstack.org/553392 | 13:17 |
| *** alexchadin has joined #openstack-infra | 13:17 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add base jobs for python_venv_build ansible role https://review.openstack.org/553396 | 13:17 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul master: Add instructions for reporting vulnerabilities https://review.openstack.org/554352 | 13:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Set up cyborg-specs repository https://review.openstack.org/553976 | 13:18 |
| jamesmca_ | mrmartin: https://twitter.com/campuscodi/status/976580119685935104?s=12 - This is a pretty massive security issue for Drupal. | 13:19 |
| *** yamamoto has quit IRC | 13:19 | |
| *** dtantsur is now known as dtantsur|brb | 13:20 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Rename python-openstacksdk to openstacksdk https://review.openstack.org/554662 | 13:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Drop python-openstacksdk for rename https://review.openstack.org/555767 | 13:21 |
| fungi | jamesmca_: thanks for the heads up. a quick search across all our (infra) properties suggests this only impacts the groups.o.o and groups-dev.o.o servers | 13:22 |
| fungi | or at least, those seem to be the only servers where we're exposing a drupal-based service | 13:23 |
| jamesmca_ | fungi: yeah, i think that's correct | 13:24 |
| *** eharney has joined #openstack-infra | 13:24 | |
| *** jamesmca_ has quit IRC | 13:24 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add limestone to nodepool builder and launcher configuration https://review.openstack.org/555476 | 13:25 |
| *** jamesmcarthur has joined #openstack-infra | 13:25 | |
| *** Guest19457 has quit IRC | 13:25 | |
| *** dklyle has quit IRC | 13:27 | |
| *** ethfci has joined #openstack-infra | 13:29 | |
| *** jamesmcarthur has quit IRC | 13:30 | |
| *** yamamoto has joined #openstack-infra | 13:30 | |
| *** lucas-hungry is now known as lucasagomes | 13:32 | |
| *** yamamoto has quit IRC | 13:35 | |
| mordred | fungi: where's your patch with the updated rename instructions? | 13:35 |
| fungi | mordred: https://review.openstack.org/554261 | 13:35 |
| mordred | awesome thakns | 13:35 |
| *** panda is now known as panda|lunch | 13:36 | |
| fungi | mordred: change(s) for step 1 and a paste with the yaml for step 2 would be good to have well in advance if we can | 13:37 |
| mordred | agree - I'm in prep mode atm | 13:37 |
| fungi | thanks! | 13:37 |
| mordred | fungi: so - in this case, I think we're going to need to force-merge more than just the project-config change | 13:38 |
| mordred | fungi: (I think) | 13:38 |
| fungi | mordred: makes sense. i expect we'll need to tweak that entry slightly | 13:39 |
| mordred | fungi: because openstacksdk shares jobs with openstackclient and each define a job thatuse sdk that both projects use | 13:39 |
| *** dansmith is now known as superdan | 13:39 | |
| mordred | fungi: so neither project will be able to self-test landing the fix | 13:39 |
| fungi | i may have written it in anticipation of the scheduler running the fix to ignore projects with incorrect configuration | 13:40 |
| *** dsariel has joined #openstack-infra | 13:40 | |
| pabelanger | morning | 13:40 |
| mordred | fungi: I'm actually concerned about job content - not just zuul config | 13:40 |
| fungi | right, also makes sense | 13:40 |
| *** VW has joined #openstack-infra | 13:40 | |
| mordred | fungi: in order for changes to land, I think we're going to need to force-merge changes to openstacksdk, python-openstackclient and devstack | 13:40 |
| *** rosmaita has joined #openstack-infra | 13:41 | |
| AJaeger | mordred: or disable jobs for now, rename, enable jobs? | 13:41 |
| *** esberglu has joined #openstack-infra | 13:41 | |
| mordred | fungi: this one is a kind of pathological example case - but it should be a good one for us to post-mortem and understand every way in which this can break | 13:41 |
| pabelanger | yah, looks like 1.95GB free memory on zuul01.o.o. Happy to help restart after I power up with coffee | 13:41 |
| fungi | mordred: though that brings up a great point related to my earlier point about being overdue for a zuul scheduler restart... i think we've landed some zuul changes to help support the rename process and aren't running them yet? | 13:41 |
| mordred | AJaeger: not good enough - devtsack itself is going to just flat stop working | 13:41 |
| AJaeger | mordred: what fun ;) | 13:41 |
| mordred | yah | 13:41 |
| *** yamahata has joined #openstack-infra | 13:41 | |
| fungi | mordred: i agree, pathological case is great for shaking out invalid assumptions | 13:42 |
| *** yamamoto has joined #openstack-infra | 13:45 | |
| *** yamamoto has quit IRC | 13:50 | |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| mrmartin | jamesmca_this drupal thing looks interesting. | 13:54 |
| dmsimard | Do we need to restart the scheduler after a rename ? | 13:54 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-storyboard master: Use utf8mb4 for MySQL database charset https://review.openstack.org/555787 | 13:55 |
| dmsimard | Or is it going to realize that somehow new project B is project A and not duplicate the configuration/layout or something | 13:55 |
| *** yamamoto has quit IRC | 13:56 | |
| fungi | dmsimard: the scheduler will (we hope!) figure it out when it does a full configuration reload | 13:56 |
| *** d0ugal_ has joined #openstack-infra | 13:56 | |
| fungi | if it doesn't, we've found a bug | 13:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Create review group for hieradata https://review.openstack.org/555790 | 13:57 |
| *** d0ugal has quit IRC | 13:57 | |
| fungi | mrmartin: i'm having a hard time figuring out what version we're currently running... dsd-status says 7.x-0.49 but i can't tell whether that corresponds to a 7.50 pre-release or is just a red herring | 13:58 |
| *** links has quit IRC | 13:58 | |
| fungi | (their release versions seem to jump from 7.44 to 7.50) | 13:58 |
| *** andreww has quit IRC | 13:59 | |
| mrmartin | we are on 7.56 | 14:00 |
| mrmartin | the dsd status is our internal release number. | 14:01 |
| fungi | cool, thanks! so what command are you using to check the installed version of drupal? | 14:01 |
| mrmartin | would be great to upgrade this thingy to a composer made environment, as it is a standard on D8 now. | 14:01 |
| mrmartin | it is in the git repo. | 14:01 |
| mrmartin | https://github.com/openstack-infra/groups/blob/master/drupal-org-core.make | 14:02 |
| fungi | aha, so just hard-coded version and we bump it when we want to pull a new one (e.g., the 7.57 which seems to be another security fix from last month)? | 14:02 |
| *** andreas_s has quit IRC | 14:02 | |
| pabelanger | infra-root: I'll be doing the following step to bring review01.o.o online today, if you'd like to validate steps: https://etherpad.openstack.org/p/review01-xenial-upgrade | 14:02 |
| *** kgiusti has joined #openstack-infra | 14:02 | |
| *** agopi has joined #openstack-infra | 14:05 | |
| openstackgerrit | Rishabh Dave proposed openstack-infra/openstack-zuul-jobs master: Add manila-tempest-plugin as a requirement for manila-ui-dsvm job https://review.openstack.org/555800 | 14:06 |
| *** eernst has joined #openstack-infra | 14:06 | |
| dmsimard | pabelanger: we should probably let corvus look if he can find anything peculiar before restarting the scheduler ? Maybe there's something to troubleshoot regarding the memory usage. | 14:07 |
| *** dave-mccowan has joined #openstack-infra | 14:07 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add instructions for reporting vulnerabilities https://review.openstack.org/554352 | 14:09 |
| *** jpena is now known as jpena|lunch | 14:12 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add instructions for reporting vulnerabilities https://review.openstack.org/554352 | 14:13 |
| *** yamahata has quit IRC | 14:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Rename python-openstacksdk to openstacksdk https://review.openstack.org/554662 | 14:13 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: Use utf8mb4 for MySQL database charset https://review.openstack.org/555806 | 14:14 |
| *** amoralej is now known as amoralej|lunch | 14:14 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add instructions for reporting vulnerabilities https://review.openstack.org/554352 | 14:14 |
| pabelanger | dmsimard: sure, I think we have improvements that will be picked up once we reboot too | 14:15 |
| mordred | fungi: ok. I think https://review.openstack.org/#/q/topic:project-rename+status:open has all the patches needed | 14:17 |
| *** gouthamr has joined #openstack-infra | 14:17 | |
| *** yamahata has joined #openstack-infra | 14:17 | |
| fungi | mordred: thanks! | 14:18 |
| *** panda|lunch is now known as panda | 14:18 | |
| *** gus has quit IRC | 14:18 | |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: add lower-constraints job https://review.openstack.org/555297 | 14:19 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: update expected value in test to match canonical form https://review.openstack.org/555808 | 14:19 |
| dmsimard | corvus: if you happen to be around, would you like to look in the ram usage before we restart the scheduler ? http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64792&rra_id=0&view_type=tree&graph_start=1521728443&graph_end=1521814843 | 14:20 |
| corvus | dmsimard: there's nothing for me to debug at the moment; go for it | 14:21 |
| dmsimard | pabelanger: ^ | 14:21 |
| pabelanger | okay, lets give release team a heads up | 14:22 |
| pabelanger | I see some patches in release-post | 14:22 |
| dmsimard | infra-root/config-core: this last patch will allow us to test nodepool with limestone before truly opening the floodgates: https://review.openstack.org/#/c/555476/ | 14:22 |
| pabelanger | +2 | 14:24 |
| *** jamesmcarthur has joined #openstack-infra | 14:24 | |
| *** garyk has joined #openstack-infra | 14:27 | |
| corvus | dmsimard: how are you going to test? | 14:28 |
| *** zhaochao has quit IRC | 14:29 | |
| *** gus has joined #openstack-infra | 14:30 | |
| *** dave-mcc_ has joined #openstack-infra | 14:30 | |
| pabelanger | okay, release team has given the go a head for a zuul restart | 14:30 |
| dmsimard | corvus: the plan was to let nodepool upload the image without sending any jobs there yet (hence max-servers 0) and then manually boot a node on that image to see if the network looks good on the node and to the mirror. The networking is the more concerning part of limestone. | 14:30 |
| dmsimard | Getting the mirror up required a bit of fiddling both on our end and on limestone's | 14:31 |
| corvus | dmsimard: gotcha. there are some more automated ways of testing, but under the circumstances, that's probably the way to go here. | 14:31 |
| dmsimard | corvus: happy to hear about automated ways of testing, do you have an example ? | 14:31 |
| odyssey4me | if any project-config cores are available to look through this repo import I'd appreciate it: https://review.openstack.org/553392 | 14:32 |
| corvus | dmsimard: (for instance, you could stand it up with max-servers=1 and unique labels, and push up DNM changes which run, say, pep8 on limestone-xenial) | 14:32 |
| *** dave-mccowan has quit IRC | 14:32 | |
| dmsimard | corvus: oh, that's not a bad idea, I'll write that down (I'm also taking this opportunity to try and document the process of adding a nodepool provider) | 14:33 |
| pabelanger | okay, I've dump queues in zuul | 14:33 |
| pabelanger | everybody ready for a restart? | 14:33 |
| corvus | pabelanger: yep | 14:33 |
| pabelanger | proceeding | 14:34 |
| *** dkehn_ has joined #openstack-infra | 14:35 | |
| pabelanger | zuul starting now | 14:35 |
| mordred | infra-root: everybody ok with me stopping puppet on puppetmaster in prep for the renames? | 14:35 |
| corvus | mordred: ++ | 14:35 |
| corvus | mordred: do we need to force-merge all those changes? | 14:36 |
| corvus | mordred: (openstack-manuals looks like maybe one we shouldn't, for example) | 14:36 |
| mordred | corvus: not all of the - I'm about to prepare a list of which ones need to be force-merged | 14:36 |
| *** sai-mtg is now known as sai-intrw | 14:36 | |
| corvus | mordred: cool | 14:36 |
| pabelanger | corvus: does zuul-web still need a restart or has that been fixed? | 14:36 |
| corvus | pabelanger: i don't believe it has | 14:36 |
| pabelanger | k | 14:36 |
| fungi | mordred: find by me | 14:36 |
| *** sree has joined #openstack-infra | 14:37 | |
| openstackgerrit | Rafael Folco proposed openstack/diskimage-builder master: Revert "Remove tripleo jobs" https://review.openstack.org/555814 | 14:37 |
| pabelanger | zuul-web restarted | 14:37 |
| *** dtantsur|brb is now known as dtantsur | 14:37 | |
| mordred | infra-root: it seems like puppet is already disabled on puppetmaster | 14:37 |
| mordred | oh -nevermind | 14:37 |
| mordred | there it is | 14:37 |
| openstackgerrit | Rafael Folco proposed openstack/diskimage-builder master: Revert "Remove tripleo jobs" https://review.openstack.org/555814 | 14:38 |
| *** florianf has quit IRC | 14:39 | |
| pabelanger | 2018-03-23 14:39:48,462 WARNING zuul.Scheduler: Tenant openstack isn't loaded | 14:40 |
| pabelanger | I think something failed | 14:40 |
| pabelanger | maybe not | 14:40 |
| dmsimard | pabelanger: that's "normal" until it eventually loads I think | 14:40 |
| dmsimard | zuul-web tends to complain about that sort of "error" as well | 14:40 |
| pabelanger | yah, I was a little to fast | 14:40 |
| pabelanger | 2018-03-23 14:40:14,996 INFO zuul.Scheduler: Full reconfiguration complete | 14:40 |
| pabelanger | loading queues | 14:40 |
| *** psachin has quit IRC | 14:42 | |
| mordred | http://paste.openstack.org/show/710105/ <-- there's the repo list yaml if anyone wants to double check it | 14:43 |
| corvus | frickler, pabelanger, dmsimard can you review https://review.openstack.org/555104 please? i'm hoping if we can get all the folks who originally voted rollcall on that we can speed up the process a bit. | 14:44 |
| *** r-daneel has joined #openstack-infra | 14:44 | |
| pabelanger | corvus: +1 | 14:45 |
| corvus | mordred: lgtm | 14:46 |
| *** pickle has joined #openstack-infra | 14:47 | |
| pabelanger | okay, queues have been restored | 14:48 |
| pabelanger | #status log zuul.o.o has been restarted to pick up latest code base and clear memory usage. Both check / gate queues were saved, be sure to check your patches and recheck when needed. | 14:49 |
| openstackstatus | pabelanger: finished logging | 14:49 |
| *** rosmaita has quit IRC | 14:49 | |
| pabelanger | maybe that should be notice | 14:49 |
| pabelanger | #status notice zuul.o.o has been restarted to pick up latest code base and clear memory usage. Both check / gate queues were saved, be sure to check your patches and recheck when needed. | 14:50 |
| openstackstatus | pabelanger: sending notice | 14:50 |
| mordred | infra-root: since we need to force-merge anyway, I think we should squash https://review.openstack.org/#/c/500768/ and https://review.openstack.org/#/c/513554 - yeah? | 14:50 |
| -openstackstatus- NOTICE: zuul.o.o has been restarted to pick up latest code base and clear memory usage. Both check / gate queues were saved, be sure to check your patches and recheck when needed. | 14:51 | |
| corvus | mordred: probably a good idea | 14:51 |
| *** yamamoto has joined #openstack-infra | 14:52 | |
| corvus | mordred: the fewer force-merges we have to do, the fewer reconfiguration cycles zuul will have to go through. | 14:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Add limestone to nodepool builder and launcher configuration https://review.openstack.org/555476 | 14:52 |
| openstackstatus | pabelanger: finished sending notice | 14:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Rename collectd-ceilometer-plugin to collectd-openstack-plugins https://review.openstack.org/500768 | 14:53 |
| *** danpawlik has quit IRC | 14:54 | |
| *** florianf has joined #openstack-infra | 14:54 | |
| mordred | infra-root: https://etherpad.openstack.org/p/project-renames-2018-03-23 <-- etherpad for the renames | 14:54 |
| openstackgerrit | Merged openstack-infra/zuul master: fix typos in documentation https://review.openstack.org/555537 | 14:57 |
| *** david-lyle has joined #openstack-infra | 14:57 | |
| *** yamamoto has quit IRC | 14:57 | |
| pabelanger | dmsimard: make sure to add grafyaml for limestone too, helpful getting stats from the cloud | 14:57 |
| *** ykarel has quit IRC | 14:57 | |
| dmsimard | pabelanger: yup, will do. | 14:57 |
| *** jpena|lunch is now known as jpena | 14:58 | |
| *** ramishra has quit IRC | 15:00 | |
| *** garyk has quit IRC | 15:01 | |
| *** alexchadin has quit IRC | 15:01 | |
| *** d0ugal__ has joined #openstack-infra | 15:01 | |
| *** d0ugal_ has quit IRC | 15:01 | |
| *** d0ugal__ has quit IRC | 15:02 | |
| *** HeOS has joined #openstack-infra | 15:02 | |
| *** d0ugal has joined #openstack-infra | 15:02 | |
| *** ihrachys has joined #openstack-infra | 15:03 | |
| mordred | infra-root: ok. we're at the appointed time for renames - everybody ready? also - before we start, could people take a look at the force-merge patches and make sure I didn't mistype any of the repo renames? | 15:04 |
| *** r-daneel has quit IRC | 15:04 | |
| mordred | corvus: actually - should we squash the two project-config changes into a single force merge? | 15:04 |
| pabelanger | looking | 15:04 |
| corvus | mordred: yeah, even better | 15:05 |
| dhellmann | I'm running into an issue with stackviz on py3 in a cinder job https://review.openstack.org/#/c/555359/ and smcginnis thinks that should have been fixed yesterday/early today but my recheck also failed. Is there something else going on, or did I just recheck too early to get the fix? | 15:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Rename collectd-ceilometer-plugin and python-openstacksdk https://review.openstack.org/500768 | 15:07 |
| fungi | dhellmann: yeah, i think that ran before the update to make the stackviz task completion non-required | 15:08 |
| dhellmann | fungi : ok, thanks, I'll recheck again | 15:08 |
| fungi | though there's also a separate fix to stackviz itself making it more robust under python 3, which should hopefully be in place soon if it's not already | 15:08 |
| *** amoralej|lunch is now known as amoralej | 15:09 | |
| *** eernst has quit IRC | 15:09 | |
| *** eernst has joined #openstack-infra | 15:09 | |
| pabelanger | mordred: lgtm (patches) | 15:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Enable storyboard for openstacksdk https://review.openstack.org/554663 | 15:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Enable storyboard for python-openstackclient https://review.openstack.org/555829 | 15:10 |
| *** _alastor_ has joined #openstack-infra | 15:11 | |
| mordred | pabelanger: thanks | 15:11 |
| mordred | corvus: you feel like giving https://review.openstack.org/#/c/555770/ a look as a devstack core? | 15:12 |
| *** hongbin has joined #openstack-infra | 15:14 | |
| corvus | lgtm; i'm still going over the list of required force-merges | 15:19 |
| mordred | infra-root: #status notice gerrit is going down for a short while for project renames. we don't anticipate a long outage | 15:19 |
| pabelanger | wfm | 15:19 |
| mordred | how does that look? the outage should _basically_ be about the length of a gerrit restart this time, yeah? | 15:19 |
| mordred | maybe I should just say ... | 15:20 |
| mordred | infra-root: #status notice gerrit is restarting for project renames | 15:20 |
| mordred | ? | 15:20 |
| fungi | lgtm | 15:20 |
| fungi | in theory it should be quick since the ansible playbook does it all rapid-fire | 15:20 |
| mordred | ++ | 15:21 |
| mordred | # TODO: online reindex accounts | 15:21 |
| mordred | # TODO: online reindex projects | 15:21 |
| mordred | those are in the playbook | 15:21 |
| mordred | is there a manual step we need to do to cause online reindex? | 15:22 |
| AJaeger | mordred: scare everybody and then have us surprised how quick it goes ;) | 15:22 |
| dmsimard | mordred: Gerrit will be temporarily unreachable as we restart it to complete the rename of some projects. <-- ? | 15:22 |
| dmsimard | (Just a little bit less dry) | 15:22 |
| mordred | dmsimard: wfm | 15:22 |
| dmsimard | fungi, pabelanger ^ | 15:22 |
| mordred | AJaeger: I'd LOVE to surpise everyone with a less-scary-than-expected maint :) | 15:22 |
| mordred | corvus: I think I'm ready to rock and roll whenever you're happy with the force-merge patches | 15:23 |
| *** trown has quit IRC | 15:23 | |
| * dmsimard wary of mordred rock and rolling | 15:24 | |
| *** bhagyashris has quit IRC | 15:24 | |
| *** trown has joined #openstack-infra | 15:24 | |
| pabelanger | mordred: have to defer to fungi, I think we need to issue a command to gerrit | 15:24 |
| pabelanger | gerrit index start accounts; gerrit index start changes | 15:24 |
| corvus | mordred: just finished going through them. lgtm. | 15:25 |
| mordred | corvus: sweet | 15:25 |
| mordred | pabelanger: thanks | 15:25 |
| pabelanger | but, maybe that could be done via ansible now, unsure | 15:25 |
| mordred | I bet it could - but Im just going to put it in the etherpad for onw | 15:25 |
| *** jtomasek has quit IRC | 15:25 | |
| pabelanger | ack | 15:25 |
| *** melwitt is now known as jgwentworth | 15:26 | |
| fungi | lemme find the command. it's a gerrit ssh api command | 15:26 |
| AJaeger | mordred: may I touch https://review.openstack.org/#/c/500768/12/gerrit/projects.yaml? | 15:26 |
| AJaeger | That does not pass tests due to ordering, I'd like to fix quickly... | 15:26 |
| AJaeger | or will you do? | 15:26 |
| mordred | AJaeger: go for it! | 15:26 |
| AJaeger | ok, on it.... | 15:27 |
| *** kaisers has joined #openstack-infra | 15:27 | |
| mordred | AJaeger: thanks - and sorry for missing the ordering | 15:27 |
| *** VW_ has joined #openstack-infra | 15:27 | |
| AJaeger | mordred: you fixed all other places except that one ;) | 15:29 |
| *** VW__ has joined #openstack-infra | 15:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Rename collectd-ceilometer-plugin and python-openstacksdk https://review.openstack.org/500768 | 15:29 |
| AJaeger | updated ^ | 15:29 |
| *** VW_ has quit IRC | 15:29 | |
| fungi | looks like the reindex commands are already on lines 45 and 56? | 15:30 |
| *** VW has quit IRC | 15:30 | |
| fungi | sorry, also in the middle of the release team meeting | 15:31 |
| mordred | fungi: I do not see those in /opt/system-config/production/playbooks/rename_repos.yaml ? | 15:31 |
| openstackgerrit | Raoul Scarazzini proposed openstack-infra/project-config master: Create tripleo-ha-utils project https://review.openstack.org/555832 | 15:31 |
| mordred | fungi: oh - in the etherpad - yes, I just added them there | 15:32 |
| fungi | mordred: lines 45 and 46 of the etherpad | 15:32 |
| mordred | fungi: yup! | 15:32 |
| fungi | though does the openstaci-project-creator account use /home/gerrit2/review_site/etc/ssh_project_rsa_key ? i thought that was just for the 'Gerrit Code Review' account | 15:32 |
| fungi | easy enough to test i suppose | 15:33 |
| fungi | i think part of the challenge with ansiblifying that step is that you need to wait until the gerrit api is up and responding | 15:34 |
| fungi | also, in the past i've noticed that if you do `gerrit index start accounts` it actually starts doing a full reindex | 15:34 |
| mordred | fungi: there's a pause in the playbook already - so I think it shold be ok | 15:34 |
| mordred | fungi: but I just grabbed the -i /home/gerrit2/review_site/etc/ssh_project_rsa_key openstack-project-creator@review.openstack.org | 15:35 |
| fungi | oh, and i believe we need --force in there because gerrit thinks reindexes are only for upgrades | 15:35 |
| mordred | from the rename-group command | 15:35 |
| mordred | gerrit index --force start accounts? | 15:35 |
| fungi | you'll see it tell you there's nothing to do when you try it without --force if memory serves | 15:35 |
| *** shardy has quit IRC | 15:35 | |
| * fungi checks gerrit docs, thinks order of parameters matters there | 15:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add reindex commands to rename playbook https://review.openstack.org/555833 | 15:36 |
| mordred | fungi: looks like changes|accounts and --force are parameters to the start subcommand from the docs | 15:37 |
| fungi | mordred: https://gerrit-documentation.storage.googleapis.com/Documentation/2.13/cmd-index-start.html says --force comes at the end | 15:37 |
| mordred | oh - at the end end. cool | 15:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add reindex commands to rename playbook https://review.openstack.org/555833 | 15:37 |
| mordred | fungi: so like that? | 15:37 |
| fungi | i think in the past when i've seen `gerrit index start --force accounts` reindex everything, it's been because the api thought i left off the index name (since it was after --force and so ignored by its parser?) | 15:38 |
| fungi | mordred: yeah, that looks fine | 15:39 |
| mordred | fungi: cool | 15:39 |
| kashyap | Hi folks, is there a Gate env with: Ubuntu 18.04 (Bionic LTS)? | 15:39 |
| kashyap | I realize the GA date is: 2018-04-26 | 15:39 |
| kashyap | Will Infra be immediately using that on GA? | 15:40 |
| fungi | kashyap: pretty sure we have ubuntu-bionic images already, we've just been recommending you only use them for non-voting jobs until it's been released | 15:40 |
| kashyap | Ah-ha, noted. | 15:40 |
| kashyap | fungi: Trying to bump the next MIN libvirt / QEMU version | 15:40 |
| *** links has joined #openstack-infra | 15:40 | |
| fungi | always a fun time | 15:40 |
| kaisers | Hey Infra! Quick q: Our CIs results aren't reported back to gerrit, while it fetches and runs jobs ok. Where there changes to gerrit in the last 3/4 days that might cause this and i missed out on? | 15:41 |
| kashyap | So Nova has to align with roughly what we're testing in the Gate | 15:41 |
| *** gyee has joined #openstack-infra | 15:41 | |
| kashyap | fungi: Yeah, have to do a bunch of looking up and clean up unneeded version constants, etc :-) | 15:41 |
| corvus | kaisers: not that i'm aware of, are you running zuul? | 15:41 |
| kashyap | fungi: Got a link to the Bionic image, when you get a sec? | 15:41 |
| kaisers | corvus: nope, running sos-ci fork | 15:41 |
| kaisers | corvus: weird thing is i don't see any errors in the ssh connection for the results. I just restarted the CIs today and wondered while no results are turning up... | 15:42 |
| corvus | kaisers: gerrit hasn't been restarted since mar 6. what's the account name? | 15:42 |
| kaisers | corvus: sec | 15:42 |
| kaisers | corvus: quobyteci | 15:43 |
| fungi | kashyap: i think if you write your job to use "ubuntu-bionic" as its nodeset, like in this shade job: http://git.openstack.org/cgit/openstack/ansible-role-shade/tree/.zuul.yaml#n19 | 15:44 |
| corvus | kaisers: i see 4 connections on that account | 15:44 |
| *** VW__ has quit IRC | 15:44 | |
| corvus | what's our current max? | 15:44 |
| pabelanger | kashyap: fungi: that is right | 15:44 |
| kaisers | corvus: max? | 15:45 |
| kashyap | ₥e clicks | 15:45 |
| fungi | corvus: 64 concurrent authenticated ssh sessions for a single account, or 100 concurrent tcp socket connections to the api port from a single ip address | 15:45 |
| * kashyap wonders if there are any Nova jobs using that | 15:45 | |
| corvus | so that's not it | 15:45 |
| AJaeger | dmsimard: could you add a grafana dashbaord for the new cloud as well, please? | 15:46 |
| mordred | infra-root: I think we're ready to go - any last minute concerns? | 15:46 |
| corvus | kaisers: i don't think you should be running into any issues on our side | 15:46 |
| fungi | kashyap: not that i saw (yet) doing http://codesearch.openstack.org/?q=ubuntu-bionic | 15:46 |
| corvus | kaisers: we're about to restart gerrit though, so you can try again in a few minutes and see if anything has changed | 15:46 |
| corvus | mordred: +1 | 15:46 |
| *** links has quit IRC | 15:46 | |
| kaisers | corvus: ok, thanks for checking & the hints!! | 15:46 |
| fungi | mordred: i don't know of anything else lingering | 15:46 |
| corvus | i'm monitoring zuul's log | 15:46 |
| pabelanger | kashyap: fungi: there is a patch up to add support to devstack, needs to land first | 15:47 |
| pabelanger | https://review.openstack.org/550218/ | 15:47 |
| kashyap | pabelanger: Thanks! | 15:47 |
| mordred | #status notice Gerrit will be temporarily unreachable as we restart it to complete the rename of some projects. | 15:47 |
| openstackstatus | mordred: sending notice | 15:47 |
| dmsimard | AJaeger: yes, just haven't had time to do it yet. I'll do it before enabling workloads on it. | 15:47 |
| * kashyap clicks | 15:47 | |
| AJaeger | dmsimard: thanks | 15:48 |
| pabelanger | kashyap: frickler is doing a bit of testing with bionic too | 15:48 |
| pabelanger | https://review.openstack.org/547986/ for tempest | 15:48 |
| pabelanger | mordred: ready here | 15:48 |
| corvus | we're starting the gerrit renames now, so if folks could keep other conversations to a minimum while we perform the maintenance, that would be appreciated. | 15:49 |
| -openstackstatus- NOTICE: Gerrit will be temporarily unreachable as we restart it to complete the rename of some projects. | 15:49 | |
| * mordred will kick the ansible playbook as soon as the notice as finished sending | 15:49 | |
| openstackstatus | mordred: finished sending notice | 15:51 |
| corvus | anyone want to start work on the github moves? | 15:51 |
| mordred | infra-root: I'm going to run the playbook | 15:52 |
| mordred | oh good. my lag dropped back to 0 | 15:52 |
| pabelanger | corvus: I can, never done it before | 15:52 |
| mordred | it had shot up to 66 for a second | 15:52 |
| fungi | gertty just threw a connection error. must be a good sign | 15:53 |
| mordred | pabelanger: I put liks in the etherpad to the settings pages | 15:53 |
| *** VW has joined #openstack-infra | 15:53 | |
| mordred | pabelanger: just click the link, at the top is a project name bot | 15:53 |
| mordred | box | 15:53 |
| *** yamamoto has joined #openstack-infra | 15:53 | |
| pabelanger | mordred: k, so just update and click rename? | 15:53 |
| mordred | yup | 15:53 |
| corvus | yep. that'll make an internal redirect in github, as opposed to delete/add which won't. | 15:54 |
| pabelanger | cool, are we ready for that now? | 15:54 |
| kashyap | Is Gerrit down just for me? | 15:54 |
| corvus | pabelanger: i believe so. | 15:54 |
| AJaeger | kashyap: see status message | 15:54 |
| kashyap | Ah, just saw the NOTICE above | 15:54 |
| kashyap | Thx | 15:54 |
| pabelanger | okay, doing openstacksdk first | 15:54 |
| pabelanger | https://github.com/openstack/openstacksdk | 15:55 |
| pabelanger | now the other | 15:55 |
| *** VW_ has joined #openstack-infra | 15:55 | |
| corvus | zuul has noticed gerrit is offline | 15:55 |
| clarkb | sorry later start than normal because my water is supposed to be shut off shortly so needed to do all pre trip water related activities early today | 15:55 |
| mordred | ok. the playbook finished but gerrit failed to start | 15:55 |
| *** VW_ has quit IRC | 15:55 | |
| clarkb | mordred: gerrit init script will say that because it has a timeout | 15:56 |
| corvus | mordred: it mayb be starting | 15:56 |
| pabelanger | https://github.com/openstack/collectd-openstack-plugins | 15:56 |
| pabelanger | renames done | 15:56 |
| clarkb | mordred: check ps if it actually started | 15:56 |
| mordred | yah | 15:56 |
| corvus | i'm guessing the playbook doesn't handle that though? | 15:56 |
| mordred | it does not appear to be running | 15:56 |
| * mordred is going ot try starting again | 15:56 | |
| corvus | mordred: wait | 15:56 |
| mordred | oh. wait | 15:56 |
| mordred | I'm dumb | 15:56 |
| corvus | gerrit2 40203 19.4 1.1 54748588 688456 ? Sl 15:53 0:31 GerritCodeReview -Xmx48g -jar /home/gerrit2/review_site/bin/gerrit.war daemon -d /home/gerrit2/review_site --run-id=1521820433.40050 | 15:56 |
| *** VW_ has joined #openstack-infra | 15:56 | |
| mordred | corvus: yah - I was running ps on puppetmaster | 15:56 |
| clarkb | ya basically gerrit startup exceeds the init script timeout now :( | 15:57 |
| mordred | amazingly, gerrit is not running there | 15:57 |
| corvus | mordred: whew | 15:57 |
| clarkb | I'm half hoping java8 will make that better but not really expecting it to | 15:57 |
| corvus | it's still not "up" yet though | 15:57 |
| mordred | so - yah - the next step in the playbook is ""Make sure that Gerrit ssh api is accepting requests. Use Enter or ^C c to continue. ^C a to abort." | 15:57 |
| pabelanger | yah, I see that at PTG, it took a little bit to start, mostly because of large load | 15:57 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint Add Ticket Type https://review.openstack.org/555839 | 15:58 |
| AJaeger | gerrit up again? ^ | 15:58 |
| corvus | yeah, zuul is actually working through a backlog of trigger events | 15:58 |
| corvus | which is kinda cool | 15:58 |
| *** VW has quit IRC | 15:59 | |
| pabelanger | there we go, I see review.o.o now | 15:59 |
| corvus | mordred: i think we're okay to proceed now | 15:59 |
| *** ldnunes has joined #openstack-infra | 15:59 | |
| mordred | corvus: awesome | 15:59 |
| *** yamamoto has quit IRC | 15:59 | |
| *** gyankum has quit IRC | 16:00 | |
| mordred | ok. all of teh playbook tasks are done | 16:00 |
| mordred | the groups have been renamed | 16:00 |
| AJaeger | nice | 16:01 |
| mordred | shall we force-merge the changes? | 16:01 |
| corvus | zuul's event queues have emptied | 16:01 |
| corvus | mordred: ++ | 16:01 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint Add Ticket Type https://review.openstack.org/555839 | 16:01 |
| * mordred is forcemerging | 16:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rename collectd-ceilometer-plugin and python-openstacksdk https://review.openstack.org/500768 | 16:02 |
| pabelanger | I see a merger_failure on zuul.o.o for 555004,1 I guess because gerrit was down? | 16:03 |
| corvus | 2018-03-23 16:03:02,805 DEBUG zuul.Scheduler: Submitting tenant reconfiguration event for openstack due to project openstack-infra/project-config | 16:03 |
| corvus | 2018-03-23 16:03:02,964 INFO zuul.Scheduler: Tenant reconfiguration beginning | 16:03 |
| corvus | pabelanger: probably | 16:03 |
| mordred | corvus: I think we need the reindex to be able to force-merge the openstacksdk change | 16:03 |
| mordred | corvus: https://review.openstack.org/#/c/555765/ is giving me a 500 | 16:03 |
| corvus | mordred: that's unfortunate... | 16:04 |
| mordred | yah | 16:04 |
| corvus | the reindex will take many many hours | 16:04 |
| mordred | corvus: I could try to push it from locally | 16:04 |
| clarkb | oh thats an annoying behavior | 16:04 |
| corvus | mordred: under the circumstances, that's probably worth doing. be very careful. :) | 16:04 |
| mordred | ok. I pushed the openstacksdk change | 16:05 |
| pabelanger | has re-index started? | 16:06 |
| corvus | 2018-03-23 16:06:07,728 DEBUG zuul.Scheduler: Submitting tenant reconfiguration event for openstack due to project openstack-infra/shade | 16:06 |
| mordred | okI believe I have landed all the needed force-merge changes | 16:06 |
| *** dkehn_ has quit IRC | 16:06 | |
| mordred | pabelanger: I will now start the reindex | 16:06 |
| *** jroll has quit IRC | 16:07 | |
| corvus | mordred: wait a sec | 16:07 |
| mordred | corvus: I will wait a sec | 16:07 |
| corvus | mordred: devstack? | 16:07 |
| corvus | oh, that's not a zuul change, nm | 16:07 |
| *** gtmanfred has quit IRC | 16:07 | |
| corvus | zuul did not complete the reconfiguration after python-openstackclient | 16:08 |
| AJaeger | mordred: are you also renaming files.openstack.org/docs/python-openstacksdk ? | 16:08 |
| corvus | let me try to find out why | 16:08 |
| AJaeger | mordred: or do you want to publish something from fresh? | 16:08 |
| mordred | AJaeger: there's an sdk patch queued up to land already - so I think we can just publish fresh | 16:09 |
| *** mriedem is now known as mriedem_afk | 16:09 | |
| AJaeger | mordred: ok, works for me... | 16:09 |
| pabelanger | mordred: I have to step away from renames, RL issue. That okay? | 16:09 |
| AJaeger | mordred: most of your open changes have a depends-on an abandoned change, you need t oremove that line... | 16:09 |
| *** gtmanfred has joined #openstack-infra | 16:10 | |
| *** jroll has joined #openstack-infra | 16:10 | |
| mordred | pabelanger: yah. thanks! | 16:10 |
| mordred | AJaeger: oh piddle | 16:10 |
| corvus | ah, i see the problem -- we haven't done a full reconfig because puppet is off; i'll do one now | 16:10 |
| pabelanger | mordred: thanks | 16:10 |
| mordred | corvus: I suppose I can go ahead and re-enable puppet as well, yeah? | 16:10 |
| fungi | mordred: _if_ the github updates have been completed, _and_ the project-config changes have merged, then yes it should be safe to start puppet again | 16:11 |
| corvus | mordred: probably, but let's wait just a sec in case we find something else | 16:11 |
| mordred | fungi, corvus: kk | 16:11 |
| fungi | we don't want puppet to run before project-config is updated or else it'll recreate empty versions of the old repo names | 16:12 |
| *** dkehn has joined #openstack-infra | 16:12 | |
| fungi | manage-projects is just helpful like that | 16:12 |
| mordred | fungi: so helpful | 16:12 |
| fungi | and new replacements for the old renamed groups too | 16:12 |
| *** jamesmcarthur has quit IRC | 16:12 | |
| fungi | and the old project names on github | 16:12 |
| corvus | Exception: Gerrit error executing git-upload-pack openstack/collectd-ceilometer-plugin | 16:13 |
| fungi | basically everything | 16:13 |
| corvus | mordred: zuul can't load config because of that | 16:13 |
| *** jamesmcarthur has joined #openstack-infra | 16:13 | |
| *** jamesmcarthur has quit IRC | 16:13 | |
| mordred | corvus: why is it trying to do that? | 16:13 |
| *** jamesmcarthur has joined #openstack-infra | 16:14 | |
| corvus | mordred: because we haven't run puppet to update main.yaml | 16:14 |
| AJaeger | mordred: your force-change is not visible in http://git.openstack.org/cgit/openstack-infra/project-config/log/ | 16:14 |
| corvus | so i think we need to do a manual puppet run | 16:14 |
| corvus | AJaeger: that's probably just replication delay | 16:14 |
| fungi | AJaeger: git replication events will be blocked on the reindex tasks | 16:14 |
| corvus | mordred: how about i do kick.sh on zuul01 ? | 16:14 |
| AJaeger | oh | 16:14 |
| mordred | corvus: ++ | 16:15 |
| fungi | git servers are, if previous experience is any indicator, lag by some hours (maybe as long as 12 hours) | 16:15 |
| *** jroll has quit IRC | 16:15 | |
| corvus | /opt/system-config/production/tools/kick.sh zuul01.openstack.org | 16:16 |
| *** jroll has joined #openstack-infra | 16:16 | |
| AJaeger | wow, show-queue says "12785 tasks" | 16:16 |
| *** harlowja has joined #openstack-infra | 16:16 | |
| *** timrc has quit IRC | 16:17 | |
| *** agopi is now known as agopi|lunch | 16:17 | |
| corvus | er, that didn't update main.yaml -- is it because project-config is updated outside of the puppet run? | 16:18 |
| mordred | corvus: I think it's that run_all updates project-config | 16:18 |
| corvus | so i should do what run_all does for project-config, then run kick again? | 16:18 |
| mordred | corvus: oh - or, also, things pull from git.o.o - so if the change hasn't replicated to git.o.o yet ... | 16:18 |
| *** danpawlik has joined #openstack-infra | 16:18 | |
| *** timrc has joined #openstack-infra | 16:18 | |
| corvus | /opt/system-config/production/playbooks/remote_puppet_git.yaml pulls from git.o.o | 16:20 |
| mordred | corvus: yah. but git.o.o doesn't have the relevant change | 16:20 |
| corvus | yep. er, actually, it just gets the sha and stores it in a register... | 16:20 |
| corvus | then we pass that to the puppet role on zuul | 16:20 |
| mordred | oh - you konw - it's probably just the 'it takes a long time for gerrit to start replication after a restart' | 16:22 |
| mordred | (since we haven't started reindexes yet) | 16:22 |
| corvus | so maybe if we just update the p-c repo on zuul and run kick, we'll be good? | 16:22 |
| clarkb | ya that 12k number AJaeger posted earlier should probably be down to 10k ish now | 16:22 |
| corvus | mordred: it starts replication immediately, we just have a lot to replicate | 16:22 |
| mordred | corvus: right | 16:22 |
| mordred | corvus: you'll hve to fetch the ref from gerrit, but yes, I think that should be great | 16:23 |
| fungi | oh, if reindexing hasn't started yet, then yeah it's just a half hour or so in past experience | 16:23 |
| *** jamesmcarthur has quit IRC | 16:23 | |
| *** danpawlik has quit IRC | 16:23 | |
| fungi | once reindexing starts, further replication will block behind it | 16:23 |
| mordred | we might want to, in the future, consider force-merging the project-config change and letting it replicate before we do the gerrit restart | 16:23 |
| *** jamesmcarthur has joined #openstack-infra | 16:24 | |
| corvus | updated repo on zuul, running kick again | 16:24 |
| *** kaisers has quit IRC | 16:25 | |
| corvus | hrm, i think that ended up reverting the project-config change | 16:26 |
| mordred | corvus: actually - update pc on zuul01 and then you can just run puppet locally on zuu01 | 16:26 |
| *** germs has quit IRC | 16:26 | |
| mordred | corvus: puppet apply /opt/system-config/production/manifests/site.pp - should do the trick | 16:26 |
| corvus | running that now | 16:27 |
| *** ykarel has joined #openstack-infra | 16:27 | |
| *** germs has joined #openstack-infra | 16:27 | |
| *** germs has quit IRC | 16:27 | |
| *** germs has joined #openstack-infra | 16:27 | |
| corvus | mordred: oh, i think if we don't pass in the ref, it just lets vcsrepo update | 16:27 |
| corvus | mordred: so that's what caused it to revert | 16:28 |
| corvus | so we need to run puppet with the project_config_ref fact | 16:28 |
| *** jamesmcarthur has quit IRC | 16:28 | |
| corvus | maybe i can do "FACTER_project_config_ref=... puppet apply ..." ? | 16:29 |
| mordred | corvus: I think so? | 16:29 |
| corvus | FACTER_project_config_ref=57781963bd17a5a639ab29da6977fe55d8bd94d0 puppet apply /opt/system-config/production/manifests/site.pp | 16:29 |
| corvus | 2018-03-23 16:29:57,260 INFO zuul.Scheduler: Full reconfiguration beginning | 16:30 |
| mordred | \o/ | 16:30 |
| *** jamesmcarthur has joined #openstack-infra | 16:30 | |
| *** ralonsoh has quit IRC | 16:30 | |
| corvus | mordred: i like your idea of pre-merging p-c changes; i think that would have made this smoother | 16:32 |
| mordred | corvus: ++ | 16:33 |
| mordred | corvus: I think the replication lag post-gerrit-restart was the thign we weren't accounting for this time | 16:33 |
| corvus | mordred, fungi: we should probably rework the instructions for that, but also, note the process we just worked out so that if we need to force-merge a followup, we know how to do that | 16:33 |
| clarkb | ya also might be worth talking to upstream on the replication plugin to see if we can avoid replicating all repos on startup | 16:33 |
| clarkb | in theory that is only needed if you make a new server with existing contnet | 16:33 |
| clarkb | so maybe it can just be a command instead | 16:34 |
| mordred | clarkb: yah. OR - it might be time for us to dig in and help finish the online-rename plugin | 16:34 |
| corvus | Zuul encountered a syntax error while parsing its configuration in the | 16:34 |
| corvus | repo openstack-infra/shade on branch stable/queens | 16:34 |
| corvus | you know, like that ^ kind of a followup :) | 16:34 |
| *** smarcet has joined #openstack-infra | 16:35 | |
| mordred | corvus: that's still fatal and we need to fix is asap yeah? | 16:35 |
| corvus | mordred: you want to write that (those?) changes? | 16:35 |
| corvus | mordred: yep | 16:35 |
| mordred | yup. on it | 16:35 |
| mordred | blast | 16:35 |
| corvus | mordred: it's a openstack/python-openstacksdk ref in required-projects on shade-tox-py27-tips | 16:35 |
| smarcet | fungi: afternoon how are u doing ? i am having issues with gerrit/zull | 16:35 |
| mordred | yup | 16:35 |
| *** fried_rice is now known as fried_rolls | 16:35 | |
| mordred | corvus: we'll need patches for openstackclient shade openstacksdk and os-client-config | 16:36 |
| *** caphrim007 has joined #openstack-infra | 16:36 | |
| smarcet | fungi: its saying that this change is merged https://review.openstack.org/#/c/555839/ but not reflected here https://git.openstack.org/cgit/openstack-infra/openstackid-resources | 16:36 |
| clarkb | smarcet: we are in the middle of rename gerrit projects which creates lag in the replication process | 16:36 |
| fungi | smarcet: we just restarted gerrit, and it usually delays by around 30 minutes replicating changes to git.openstack.org and github.com after a restart | 16:36 |
| smarcet | fungi: oh ok thx u for the info :) | 16:37 |
| dmellado | corvus: mordred are you checking the issue on shade? | 16:37 |
| dmellado | it breaks dhellmann patch for lower-reqs | 16:37 |
| corvus | dmellado: can you point to the error? we may or may not be working on the issue you've identified | 16:38 |
| dmellado | corvus: https://review.openstack.org/#/c/555755/ | 16:39 |
| corvus | dmellado: yep that's what we're fixing | 16:39 |
| *** smarcet has quit IRC | 16:39 | |
| dmellado | corvus: awesome, glad to hear it | 16:39 |
| dmellado | thanks! | 16:39 |
| mordred | corvus: ok. I've pushed and merged changes for sdk, osc and shade | 16:39 |
| *** smarcet has joined #openstack-infra | 16:39 | |
| mordred | corvus: hopefully zuul is happy this time | 16:39 |
| dmellado | mordred: let's see, I'm rechecking :P | 16:40 |
| *** smarcet has quit IRC | 16:40 | |
| corvus | dmellado: not yet | 16:40 |
| mordred | dmellado: well- hang on a sec ... | 16:40 |
| corvus | mordred: something reverted the project-config repo again | 16:40 |
| dmellado | heh, not yet xD | 16:41 |
| corvus | is puppetmaster running? | 16:41 |
| mordred | corvus: yes. disabling again. I'm not sure how it got re-enabled | 16:41 |
| *** kaisers_ has joined #openstack-infra | 16:41 | |
| mordred | corvus: oh. kick | 16:42 |
| corvus | mordred: ok. i think we need to keep it disabled until replication has caught up. | 16:42 |
| mordred | corvus: kick runs puppet on the puppetmaster, which re-added the crontab entry | 16:42 |
| corvus | oh. oops | 16:42 |
| fungi | yikes, i hope it didn't get far enough to run manage-projects | 16:42 |
| fungi | otherwise we have some significant cleanup to do | 16:43 |
| fungi | and another gerrit outage for that | 16:43 |
| mordred | fungi: I am pretty certain it did -it's running remote_puppet_else right now | 16:43 |
| fungi | if it ran create-cgit-repos on the git servers, we may also need to clean that up | 16:43 |
| fungi | i have an appointment at 17:00z i need to get going to, so can't really stick around any longer (and had promised to work on storyboard today so still need to spend the afternoon on that when i get back) | 16:45 |
| *** VW_ has quit IRC | 16:45 | |
| corvus | i don't see a python-openstacksdk.git repo in /home/gerrit2/review_site/git | 16:45 |
| fungi | also wanted to note that once reindexing starts (and again when it completes) we'll have entries in ~gerrit2/review_site/logs/error_log for com.google.gerrit.lucene.OnlineReindexer | 16:45 |
| mordred | should I kill run_all.sh ? | 16:46 |
| fungi | please please kill it | 16:46 |
| mordred | fungi: killed | 16:46 |
| fungi | thank you | 16:46 |
| corvus | i guess gerrit is the last stage of that process? | 16:46 |
| clarkb | there is a manage projects log on review.o.o under /var/log we could check | 16:47 |
| corvus | it is empty | 16:47 |
| corvus | last ran on 3-20 | 16:47 |
| *** dkehn has quit IRC | 16:47 | |
| clarkb | oh because it subscribes to the file right? | 16:47 |
| clarkb | so if the file didn't change it won't run | 16:48 |
| clarkb | whew | 16:48 |
| fungi | i've seen very little problem from killing off the run_all when i've accidentally fired it in the past, so odds are whatever we might have to clean up from that is going to be minimal compared to if manage-projects and create-cgit-repos fired prematurely | 16:48 |
| corvus | what file? | 16:48 |
| mordred | wow. we got luck on that one | 16:48 |
| *** vabada has quit IRC | 16:48 | |
| clarkb | corvus: the projects.yaml file | 16:48 |
| mordred | corvus: the projects.yaml file | 16:48 |
| corvus | it changed | 16:48 |
| mordred | it didn't - the repo is stale | 16:48 |
| clarkb | corvus: not on review.o.o | 16:48 |
| mordred | so the repo doesn't have the updated version | 16:48 |
| mordred | ya- the run_all script updated puppet - to the old copy - then ran puppet everywhere, which was a no-op for manage-projects due to replication lag \o/ | 16:49 |
| corvus | oh you're right -- p-c did not change. gerrit.config changed. | 16:49 |
| fungi | okay, disappearing now before i'm late for my appointment. fingers crossed that the remainder of the maintenance goes better, and i'll check back in a couple hours when i get home | 16:49 |
| mordred | fungi: I am confident that IRC will still be here when you're back | 16:49 |
| corvus | i also don't see python-openstacksdk on git01 | 16:50 |
| smcginnis | Should I be concerned about the last zuul comment on https://review.openstack.org/#/c/555311/ ? | 16:50 |
| fungi | mordred: probably so | 16:50 |
| *** ykarel has quit IRC | 16:50 | |
| smcginnis | Or did we just catch things at a bad time? | 16:50 |
| corvus | so i think we're good. i'll refresh p-c on zuul again, and reload | 16:50 |
| *** pgaxatte has left #openstack-infra | 16:50 | |
| *** pgaxatte has quit IRC | 16:50 | |
| mordred | corvus: ++ | 16:50 |
| corvus | smcginnis: bad time, gives us a few more mins | 16:51 |
| smcginnis | corvus: OK, sounds good. Thanks! | 16:51 |
| corvus | 2018-03-23 16:51:49,246 INFO zuul.Scheduler: Full reconfiguration beginning | 16:51 |
| clarkb | mordred: in addition to stopping the running puppet run did you kill the cron entry too? | 16:52 |
| clarkb | (don't want it restarting) | 16:52 |
| *** david-lyle has quit IRC | 16:52 | |
| *** armaan has joined #openstack-infra | 16:52 | |
| *** dhajare has quit IRC | 16:53 | |
| mordred | clarkb: yes | 16:53 |
| jlvillal | gerritbot review request: https://review.openstack.org/#/c/545469/ Some cleanup/refactoring and adding unit tests. Has one +2 Thanks! | 16:54 |
| *** Swami has joined #openstack-infra | 16:54 | |
| *** bnemec is now known as beekneemech | 16:54 | |
| *** yamamoto has joined #openstack-infra | 16:55 | |
| corvus | mordred: http://paste.openstack.org/show/710276/ | 16:56 |
| *** lucasagomes is now known as lucas-afk | 16:57 | |
| mordred | corvus: on it | 16:57 |
| corvus | mordred: that project has stable branches | 16:57 |
| corvus | i'm guessing we'll need changes to all of em | 16:58 |
| mordred | yah | 16:58 |
| *** zoli is now known as zoli|gone | 16:58 | |
| *** zoli|gone is now known as zoli | 16:58 | |
| mordred | so - I'm not going to do it right now - but this is using legacy jobs in ozj and the playbooks there are going to need updating | 16:58 |
| corvus | yeah, should be able to do that once we're in sync again | 16:59 |
| mordred | remote: https://review.openstack.org/555865 Rename to collectd-openstack-plugins | 17:00 |
| *** yamamoto has quit IRC | 17:00 | |
| mordred | corvus: no .zuul.yaml on stable branches | 17:00 |
| mordred | corvus: collectd-openstack-plugins merged | 17:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Log error messages when reconfiguration fails https://review.openstack.org/555867 | 17:01 |
| corvus | mordred: so ready to go again? | 17:01 |
| mordred | corvus: yes | 17:01 |
| *** hemna_ has joined #openstack-infra | 17:01 | |
| corvus | 2018-03-23 17:03:50,072 INFO zuul.Scheduler: Full reconfiguration beginning | 17:03 |
| *** VW has joined #openstack-infra | 17:05 | |
| *** andreas_s has joined #openstack-infra | 17:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add names to rename playbook tasks https://review.openstack.org/555869 | 17:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add online reindex tasks to project rename playbook https://review.openstack.org/555870 | 17:06 |
| *** masber has quit IRC | 17:06 | |
| corvus | mordred: http://paste.openstack.org/show/710296/ | 17:08 |
| mordred | I thought I'd gotten that one already | 17:08 |
| corvus | mordred: last i see on stable/pike is nov 30 2017 | 17:10 |
| mordred | yah - doing it now | 17:10 |
| *** jamesmcarthur has quit IRC | 17:10 | |
| *** jamesmcarthur has joined #openstack-infra | 17:11 | |
| corvus | mordred: i think that's the only branch needing the update; looks like ocata has no zuul, and queens is done | 17:11 |
| mordred | corvus: yes. that is my take as well | 17:12 |
| mordred | corvus: I have landed that patch | 17:12 |
| corvus | 2018-03-23 17:12:22,736 INFO zuul.Scheduler: Full reconfiguration beginning | 17:12 |
| mordred | corvus: and I checked python-openstacksdk for pre-queens stable .zuul.yaml and there are none | 17:12 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: builddoc: Treat '[pbr] autodoc_tree_excludes' as a multi-line opt https://review.openstack.org/552100 | 17:14 |
| *** mriedem_afk has quit IRC | 17:14 | |
| *** dtantsur is now known as dtantsur|afk | 17:16 | |
| corvus | mordred: i *think* we're over the hump, and zuul is rebuilding all the in-queue dynamic layouts now | 17:17 |
| *** krenczewski has quit IRC | 17:17 | |
| mordred | \o/ | 17:17 |
| *** smarcet has joined #openstack-infra | 17:17 | |
| mordred | corvus: phew. that was more fun than I would have preferred it to be | 17:18 |
| corvus | yeah, should get a little better soon | 17:18 |
| *** masber has joined #openstack-infra | 17:18 | |
| *** agopi|lunch is now known as agopi | 17:19 | |
| mordred | corvus: looking at https://review.openstack.org/#/q/status:merged+topic:project-rename - it took 9 force-merge patches to rename openstacksdk - and 2 to rename collect-openstack-plugins - with one of each of those sets being the same for a total of 10 patches | 17:19 |
| corvus | 2018-03-23 17:19:18,450 INFO zuul.Scheduler: Full reconfiguration complete | 17:19 |
| mordred | corvus: \o/ | 17:19 |
| *** e0ne has quit IRC | 17:22 | |
| *** pcaruana has quit IRC | 17:23 | |
| corvus | smcginnis, dmellado: we should be good now | 17:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove redundant acl_config entry https://review.openstack.org/554662 | 17:23 |
| corvus | mordred: time to kick off the reindex? | 17:24 |
| dulek | corvus: I can confirm that jobs are rechecking fine. | 17:24 |
| mordred | corvus: when I made the folowup patches for the rename, I made them all depends-on https://review.openstack.org/554662 - but then I squashed that patch with the collectd one | 17:24 |
| corvus | so 554662 is now a minor cleanup we want to merge asap to unblock the followups? | 17:25 |
| mordred | corvus: that patch there ^^ is now a trivial patch so that we don't have to go update 10 patches | 17:25 |
| mordred | yah | 17:25 |
| corvus | +3 | 17:25 |
| mordred | thanks | 17:25 |
| mordred | I will now start the reindexes | 17:25 |
| mordred | the reindexers have been started | 17:27 |
| *** agopi is now known as agopi|out | 17:28 | |
| *** jpich has quit IRC | 17:28 | |
| *** beekneemech has quit IRC | 17:29 | |
| *** diablo_rojo has quit IRC | 17:30 | |
| *** bnemec has joined #openstack-infra | 17:32 | |
| *** bnemec is now known as beekneemech | 17:33 | |
| *** agopi|out has quit IRC | 17:33 | |
| *** beekneemech has quit IRC | 17:34 | |
| *** bnemec has joined #openstack-infra | 17:34 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint get ticket type by summit https://review.openstack.org/555877 | 17:34 |
| *** jamesmcarthur has quit IRC | 17:35 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint get ticket type by summit https://review.openstack.org/555877 | 17:36 |
| *** dprince has quit IRC | 17:36 | |
| *** masber has quit IRC | 17:37 | |
| *** trown has quit IRC | 17:40 | |
| dhellmann | is there a way to ask gerrit for patches under review that don't have a Verified setting at all? I'm trying "is:open path:^.*zuul.* NOT label:Verified+1 and NOT label:Verified-1" and it's telling me I have too many query parameters | 17:41 |
| *** mriedem has joined #openstack-infra | 17:41 | |
| dhellmann | oh, duh, it's the "and" | 17:41 |
| mordred | dhellmann: if you have a sec, http://logs.openstack.org/89/555789/2/check/build-tox-manuals-checkbuild/f2cfedc/job-output.txt.gz#_2018-03-23_16_03_42_129604 | 17:42 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Add a documentation index page https://review.openstack.org/555505 | 17:42 |
| mordred | dhellmann: I'm not 100% sure what the right thing to do there is | 17:43 |
| dhellmann | it's a little complicated | 17:43 |
| clarkb | dhellmann: I think you can do not >0 and not <-1 | 17:43 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Publish docs to zuul-ci.org https://review.openstack.org/555501 | 17:43 |
| *** jpena is now known as jpena|off | 17:43 | |
| clarkb | er <0? I want ot say searches for 0 don't work unless someone explicitly votes 0 | 17:43 |
| clarkb | but check just a search for 0 too I guess | 17:43 |
| *** psachin has joined #openstack-infra | 17:43 | |
| *** armaan has quit IRC | 17:43 | |
| dhellmann | mordred: there's a thing earlier in the htaccess template that produces redirects for all official repos to a "latest" directory. Because the governance repo lists python-openstacksdk, there is a rule generated | 17:44 |
| dhellmann | and because of that, an unexpected redirect is happening based on the later rule being added for the rename | 17:44 |
| dhellmann | after the governance patch merges that test should pass | 17:44 |
| dhellmann | clarkb : removing the "not" gave me what I want | 17:44 |
| mordred | dhellmann: ah - ok | 17:44 |
| mordred | dhellmann: cool | 17:44 |
| dhellmann | clarkb : https://review.openstack.org/#/q/is:open+path:%255E.*zuul.*+NOT+label:Verified%252B1++NOT+label:Verified-1+NOT+label:Verified-2 | 17:45 |
| mordred | dhellmann: that's a pretty cool test | 17:45 |
| dhellmann | yeah, I wanted to make the redirect tests very strict so we knew exactly what we were getting | 17:45 |
| dhellmann | all of that is in the whereto tool | 17:45 |
| *** derekh has quit IRC | 17:46 | |
| dhellmann | I always spend a ton of time ensuring I haven't introduced some unexpected loop or whatever into the set of redirects, and now I don't have to think about it any more | 17:46 |
| clarkb | dmsimard: re https://review.openstack.org/#/c/555433/ my second comment's point was to point out you can make a change that will work with all versions of shade | 17:47 |
| clarkb | dmellado: so I think we can make that change then worry less about the specific version and work on modernizing the process and docs separately | 17:47 |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Explicitly read setup.cfg as utf-8 on Python 3 https://review.openstack.org/555366 | 17:47 |
| clarkb | er dmsimard ^ sorry dmellado | 17:47 |
| dhellmann | clarkb : I'm hoping that query gives me a reasonable approximation of the changes currently up for review and being tested that change a zuul config, so I don't submit too many at one time | 17:47 |
| mordred | clarkb, dmsimard: crap. I owe a patch update to that patch | 17:48 |
| clarkb | mordred: is it to do my suggestion? | 17:48 |
| mordred | clarkb: that will actually not work with cloud.list_servers() at all | 17:48 |
| mordred | clarkb: nope - we strip links dicts in shade :) | 17:48 |
| clarkb | gotcha so the fallback is flawed anyways | 17:48 |
| mordred | clarkb: yup | 17:49 |
| clarkb | (I was relying on the idea that hte fallbcak would work) | 17:49 |
| mordred | clarkb: but I can make a simple patch that fixes | 17:49 |
| clarkb | (and realized we didn't needto fallback in that case just use it) | 17:49 |
| openstackgerrit | Merged openstack-infra/nodepool master: Publish docs to zuul-ci.org https://review.openstack.org/555497 | 17:49 |
| clarkb | arg and of course its the weird dns api that needs the href rather than the uuid | 17:49 |
| mordred | yup | 17:50 |
| *** harlowja has quit IRC | 17:50 | |
| dmsimard | mordred: so do I abandon that one then ? | 17:52 |
| dmsimard | clarkb: also, dmellado and I are used to be mistaken :P | 17:52 |
| clarkb | looks like limestone image uploads are failing (I can't currently look into that as I need to do the lsat of my packing and eat lunch then find the airport) | 17:52 |
| clarkb | also nl02.openstack.org/ uses an ubuntu apache default index.html :/ | 17:52 |
| corvus | i need to head out for a bit; i should be back after lunch | 17:53 |
| AJaeger | mordred: I'll rebase your storyboard changes, they have a conflict after my change... | 17:53 |
| mordred | AJaeger: kk - and thanks! | 17:54 |
| *** felipemonteiro has joined #openstack-infra | 17:55 | |
| *** yamamoto has joined #openstack-infra | 17:56 | |
| *** yamahata has quit IRC | 17:57 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint update ticket type https://review.openstack.org/555892 | 17:58 |
| *** david-lyle has joined #openstack-infra | 17:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update launch-node's dns.py to work with openstacksdk https://review.openstack.org/555433 | 18:01 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint update ticket type https://review.openstack.org/555892 | 18:01 |
| mordred | clarkb, dmsimard: ^^ that should do it | 18:02 |
| *** yamamoto has quit IRC | 18:02 | |
| *** ykarel has joined #openstack-infra | 18:02 | |
| prometheanfire | can I get a quick review on this dib change? https://review.openstack.org/552952 | 18:02 |
| clarkb | prometheanfire: we don't appear to have a boot test for gentoo? maybe we can add one as part of that to check it works? | 18:03 |
| AJaeger | mordred: which group will you use for python-openstackclient? That is missing in your change | 18:03 |
| clarkb | though maybe we would test not systemd by default in that case? | 18:03 |
| clarkb | so many gentoo options | 18:04 |
| mordred | AJaeger: python-openstackclient? | 18:04 |
| AJaeger | mordred: I assume python-openstackclient, is that correct? | 18:04 |
| mordred | yah | 18:04 |
| AJaeger | ;) | 18:04 |
| bkero | Gentoo has all the best options | 18:04 |
| mordred | bkero: you have all the best options | 18:04 |
| bkero | mordred: everybody says so | 18:04 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Enable storyboard for openstacksdk https://review.openstack.org/554663 | 18:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Enable storyboard for python-openstackclient https://review.openstack.org/555829 | 18:08 |
| AJaeger | mordred: please double check ^ | 18:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Publish zuul-jobs docs to zuul-ci.org https://review.openstack.org/555503 | 18:09 |
| mordred | AJaeger: look great | 18:10 |
| prometheanfire | clarkb: I think once we have systemd working with dib and glean working with networkd we can make a boot test | 18:10 |
| prometheanfire | atm I still need to finish up the networkd/glean stuff (doesn't boot without it really) | 18:10 |
| clarkb | prometheanfire: gotcha, I've already approved the change since it seems to follow the existing all the options setup for gentoo | 18:10 |
| prometheanfire | ya, dib builds it just fine (have a test instance booted locally) | 18:11 |
| *** kaisers_ has quit IRC | 18:11 | |
| *** smarcet has quit IRC | 18:12 | |
| *** agopi|out has joined #openstack-infra | 18:15 | |
| *** hashar is now known as hasharAway | 18:15 | |
| *** slagle has quit IRC | 18:15 | |
| *** sambetts is now known as sambetts|afk | 18:16 | |
| * fungi returns to a computer and surveys the damage | 18:16 | |
| clarkb | fungi: things seem happy and reindex is running now | 18:17 |
| fungi | i see that | 18:20 |
| fungi | though `grep OnlineReindexer ../gerrit2/review_site/logs/error_log` confuses me a bit | 18:20 |
| *** armaan has joined #openstack-infra | 18:21 | |
| fungi | http://paste.openstack.org/show/710388/ | 18:21 |
| fungi | i guess it's the [Reindex v32-v32] which is relevant here | 18:21 |
| clarkb | fungi: I think there are two different indexes now | 18:22 |
| fungi | not sure what [Reindex v3-v3] is all about | 18:22 |
| *** bobh has joined #openstack-infra | 18:22 | |
| clarkb | I'm guessing they have different sets of versions? | 18:22 |
| fungi | ahh, perhaps so | 18:22 |
| *** cshastri has joined #openstack-infra | 18:22 | |
| fungi | i'll buy that, and get back to being efficient at worrying about more important matters | 18:22 |
| openstackgerrit | Borne Mace proposed openstack-infra/project-config master: kolla-cli https://review.openstack.org/555897 | 18:23 |
| *** smarcet has joined #openstack-infra | 18:24 | |
| *** smarcet has left #openstack-infra | 18:24 | |
| *** psachin has quit IRC | 18:26 | |
| *** sree has quit IRC | 18:26 | |
| *** sree has joined #openstack-infra | 18:27 | |
| openstackgerrit | Borne Mace proposed openstack-infra/project-config master: Add Zuul project for kolla-cli https://review.openstack.org/555898 | 18:27 |
| *** yamahata has joined #openstack-infra | 18:30 | |
| *** andreas_s has quit IRC | 18:31 | |
| *** sree has quit IRC | 18:31 | |
| openstackgerrit | Borne Mace proposed openstack-infra/project-config master: kolla-cli https://review.openstack.org/555897 | 18:32 |
| openstackgerrit | Borne Mace proposed openstack-infra/project-config master: Add Zuul project for kolla-cli https://review.openstack.org/555898 | 18:32 |
| *** gfidente has quit IRC | 18:32 | |
| *** harlowja has joined #openstack-infra | 18:33 | |
| *** bhavik1 has joined #openstack-infra | 18:33 | |
| *** eharney has quit IRC | 18:35 | |
| *** r-daneel has joined #openstack-infra | 18:37 | |
| rm_work | anyone know kevin fox's IRC nick? | 18:37 |
| *** harlowja has quit IRC | 18:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove redundant acl_config entry https://review.openstack.org/554662 | 18:40 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: new diskimage-build command with config https://review.openstack.org/554002 | 18:40 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: Move fail early checks in to python https://review.openstack.org/555462 | 18:40 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: Move environment create/cleanup to python https://review.openstack.org/555463 | 18:40 |
| fungi | rm_work: kfox1111 i think (seen talking in #openstack-kolla recently) | 18:41 |
| fungi | yeah, he seems to be in channel now | 18:42 |
| rm_work | kk thanks | 18:42 |
| clarkb | the reindex seems to be down to nova and openstack manuals | 18:42 |
| mordred | yah - openstacksdk repo is properly re-indexed and changes are correctly showing up in gerrit again | 18:43 |
| fungi | i think once we get below 4 the replication tasks get some bandwidth again | 18:43 |
| clarkb | fungi: ya replication is done according to show-queue | 18:43 |
| clarkb | and with that I really need to afk. I think I'll see some of you soonish | 18:44 |
| mordred | fungi, clarkb: project-config repo has replicated properly ... | 18:44 |
| mordred | so I think we can re-enable puppet now, yes? | 18:44 |
| fungi | all good signs | 18:45 |
| fungi | safe travels, clarkb! (and anyone else headed to ons) | 18:45 |
| fungi | i'm happy to let the rest of you who live closer to that coast take care of discussions there | 18:45 |
| *** andreas_s has joined #openstack-infra | 18:47 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: Use utf8mb4 for MySQL database charset https://review.openstack.org/555806 | 18:49 |
| *** r-daneel_ has joined #openstack-infra | 18:50 | |
| *** andreas_s has quit IRC | 18:51 | |
| mordred | fungi: that patch looks great - but I believe will only work on mysql | 18:51 |
| mordred | fungi: so instead of 'supports alter' if might be better to check to make sure it's a mysql dialect? | 18:51 |
| fungi | mordred: nice! i was just about to ask someone in #openstack-oslo how to limit it to openstack and its derivatives | 18:52 |
| *** r-daneel has quit IRC | 18:52 | |
| fungi | i grepped around in the existing migrations in sb and worry that we're already doing mysql-only things without appropriate conditionals | 18:52 |
| *** r-daneel_ is now known as r-daneel | 18:52 | |
| *** dbecker has joined #openstack-infra | 18:53 | |
| *** dkehn has joined #openstack-infra | 18:53 | |
| mordred | fungi: yah. the postgres support seems potentially young still | 18:53 |
| fungi | so have a feeling we're only actually testing with mysql anyway | 18:53 |
| fungi | but happy to not be part of the problem, at least | 18:53 |
| mordred | fungi: ++ | 18:53 |
| pabelanger | and back, catching up on backlog | 18:53 |
| fungi | grr, "s/openstack and its derivatives/mysql and its derivatives/" | 18:54 |
| mordred | fungi: I actually read the sentence as you intended | 18:54 |
| *** dave-mcc_ has quit IRC | 18:55 | |
| *** armaan has quit IRC | 18:55 | |
| *** rosmaita has joined #openstack-infra | 18:57 | |
| *** yamamoto has joined #openstack-infra | 18:58 | |
| openstackgerrit | Merged openstack/diskimage-builder master: enable systemd profile for Gentoo https://review.openstack.org/552952 | 19:01 |
| *** yamamoto has quit IRC | 19:02 | |
| *** jesusaur has quit IRC | 19:05 | |
| *** ykarel has quit IRC | 19:07 | |
| *** lpetrut_ has quit IRC | 19:08 | |
| *** harlowja has joined #openstack-infra | 19:09 | |
| *** cshastri has quit IRC | 19:11 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: Use utf8mb4 for MySQL database charset https://review.openstack.org/555806 | 19:12 |
| *** jesusaur has joined #openstack-infra | 19:13 | |
| mordred | infra-root: everybody ok with me re-enabling puppet? | 19:14 |
| clarkb | I think it is probably fine assuming github was done (I'm not sure it was) but also not going to be able to help much at this point | 19:15 |
| *** hemna_ has quit IRC | 19:15 | |
| fungi | mordred: yeah, if everything's merged and github updates were done (sounds like they were) then puppet can be safely reenabled | 19:19 |
| mordred | ok. done | 19:20 |
| *** diablo_rojo has joined #openstack-infra | 19:22 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:23 | |
| *** vivsoni__ has joined #openstack-infra | 19:23 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean master: [DNM] Fix handling of openrc based init systems https://review.openstack.org/548604 | 19:25 |
| mnaser | Around what time are any OpenStack infra people getting to LA? | 19:25 |
| clarkb | mnaser: 6:25pm tonight | 19:26 |
| *** felipemonteiro has quit IRC | 19:26 | |
| *** vivsoni_ has quit IRC | 19:26 | |
| clarkb | mnaser: happy to share a cab to downtown if that lines up with other people's travel. I think modrred joins us tomorrow | 19:26 |
| *** dbecker has quit IRC | 19:27 | |
| mnaser | clarkb: not sure who else is coming I’m on mobile but I finally checked in now | 19:27 |
| clarkb | also happy to meet up and grab dinner if airport schedules are not working | 19:27 |
| mnaser | Exhausted of travels | 19:27 |
| clarkb | I expect I will be in hunt of a burrito | 19:27 |
| mnaser | I’d be up for dinner but not sure how tired I’ll be, been up since 3 am EST | 19:27 |
| clarkb | mnaser: ouch | 19:27 |
| fungi | mnaser: i'd already be asleep. you have a superior constitution | 19:28 |
| mnaser | there’s a restaurant in Santa Monica which overnights pasta from Italy and makes it here, reasonably priced too | 19:28 |
| fungi | wow! | 19:28 |
| mnaser | I try to eat there at least once when I’m in LA | 19:28 |
| clarkb | ha mnaser has his la restaurant | 19:28 |
| mnaser | Sorry that’s as hip as I’ll get | 19:29 |
| mnaser | fungi: struggling to stay up not to mess up my sleep! | 19:29 |
| fungi | granted, i wonder whether overnighting your pasta from italy is better than just making fresh pasta in your restaurant. it's not like it's _hard_ compared to other things restaurants already do | 19:29 |
| *** bhavik1 has quit IRC | 19:29 | |
| fungi | but maybe that's the gimmick | 19:30 |
| mnaser | Apparently the Italian eggs taste different and the skill set of people who know how to make it “the right way” isn’t something you replicate easily | 19:30 |
| mnaser | It is really really good though, by far the best lasagna I’ve ever had there | 19:30 |
| fungi | i'd buy that. especially the part about the eggs | 19:30 |
| fungi | easy enough to import labor, harder to get the right feed for the right chickens to make the right eggs | 19:31 |
| clarkb | also we have silly egg food standards in this country | 19:31 |
| fungi | though i'd be willing to bet the italian eggs are coming from chickens who are just eating bugs. healthier chickens, tastier eggs, harder to come by in 'murika | 19:32 |
| fungi | we seem to prefer to feed our chickens chemical-laden grain substitute and ground-up bits of their siblings | 19:32 |
| fungi | mmm, tastes like artificial hormones, antibiotics and cannibalism | 19:33 |
| *** openstackgerrit has quit IRC | 19:34 | |
| clarkb | mnaser: I guess I'll ping you once I've made it to hotel and am ready to find food. Then either you'll be awake and willing or not :) | 19:36 |
| clarkb | looks like there will be no shortage of tacos which I am happy with over my initial desire for a burrito | 19:36 |
| mnaser | clarkb: tacos are a risky choice though. But what’s the worst that can happen | 19:37 |
| mnaser | Hahah | 19:37 |
| *** fried_rolls is now known as fried_rice | 19:40 | |
| *** openstackgerrit has joined #openstack-infra | 19:41 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to delete ticket type https://review.openstack.org/555908 | 19:41 |
| *** agopi|out has quit IRC | 19:43 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to delete ticket type https://review.openstack.org/555908 | 19:45 |
| *** camunoz has joined #openstack-infra | 19:46 | |
| *** rosmaita has quit IRC | 19:46 | |
| *** eharney has joined #openstack-infra | 19:49 | |
| *** lpetrut has joined #openstack-infra | 19:53 | |
| *** agopi|out has joined #openstack-infra | 19:53 | |
| *** florianf has quit IRC | 19:57 | |
| *** VW_ has joined #openstack-infra | 19:57 | |
| *** yamamoto has joined #openstack-infra | 19:58 | |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: add lower-constraints job https://review.openstack.org/555297 | 20:00 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: update parse test to use reliable comparison https://review.openstack.org/555911 | 20:00 |
| *** VW has quit IRC | 20:01 | |
| *** VW_ has quit IRC | 20:02 | |
| openstackgerrit | Borne Mace proposed openstack-infra/project-config master: kolla-cli https://review.openstack.org/555897 | 20:03 |
| openstackgerrit | Borne Mace proposed openstack-infra/project-config master: Add Zuul project for kolla-cli https://review.openstack.org/555898 | 20:03 |
| *** gouthamr has quit IRC | 20:03 | |
| *** patriciadomin has joined #openstack-infra | 20:03 | |
| *** kgiusti has left #openstack-infra | 20:04 | |
| *** yamamoto has quit IRC | 20:04 | |
| corvus | clarkb, fungi, mordred: i think we've got enough consensus on the 'alternate' version of the cgit hosting thing for me to break the dependency on the spec update and go ahead and land the implementation early. agree? | 20:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/jeepyb master: Support cgit alias sites https://review.openstack.org/555105 | 20:05 |
| fungi | corvus: yes please! | 20:05 |
| corvus | that's an updated version of the implementation which fixes clarkb's notes and drops depends-on | 20:05 |
| corvus | given the state of testing for jeepyb, i think i should check that out and run manually on one of the git farm hosts before we land it | 20:06 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: update parse test to use reliable comparison https://review.openstack.org/555808 | 20:07 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: add lower-constraints job https://review.openstack.org/555297 | 20:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-tempest-dsvm-cells job usage https://review.openstack.org/549785 | 20:07 |
| *** hemna_ has joined #openstack-infra | 20:07 | |
| *** jroll has quit IRC | 20:09 | |
| *** tesseract has quit IRC | 20:09 | |
| *** jroll has joined #openstack-infra | 20:10 | |
| *** r-daneel has quit IRC | 20:11 | |
| *** panda is now known as panda|off | 20:12 | |
| *** rosmaita has joined #openstack-infra | 20:18 | |
| *** dave-mccowan has joined #openstack-infra | 20:19 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Alias zuul git repos to git.zuul-ci.org https://review.openstack.org/555925 | 20:19 |
| *** bobh has quit IRC | 20:21 | |
| *** diablo_rojo has quit IRC | 20:26 | |
| *** diablo_rojo has joined #openstack-infra | 20:26 | |
| *** bobh has joined #openstack-infra | 20:28 | |
| *** r-daneel has joined #openstack-infra | 20:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Alias zuul git repos to git.zuul-ci.org https://review.openstack.org/555925 | 20:39 |
| corvus | fungi, mordred: if you want to approve ^ i can do manual testing of the jeepyb change on a git farm server later | 20:48 |
| fungi | looks sane | 20:49 |
| *** danpawlik has joined #openstack-infra | 20:52 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: Use utf8mb4 for MySQL database charset https://review.openstack.org/555806 | 20:54 |
| *** danpawlik has quit IRC | 20:54 | |
| *** yamamoto has joined #openstack-infra | 21:00 | |
| *** kjackal has quit IRC | 21:05 | |
| mordred | corvus: ++ | 21:05 |
| *** jesusaur has quit IRC | 21:05 | |
| *** yamamoto has quit IRC | 21:06 | |
| *** jesusaur has joined #openstack-infra | 21:07 | |
| openstackgerrit | Merged openstack-infra/project-config master: Alias zuul git repos to git.zuul-ci.org https://review.openstack.org/555925 | 21:09 |
| *** kjackal has joined #openstack-infra | 21:11 | |
| *** hongbin has quit IRC | 21:14 | |
| *** ldnunes has quit IRC | 21:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove empty sections from docs https://review.openstack.org/554315 | 21:14 |
| *** edmondsw has quit IRC | 21:17 | |
| *** jbadiapa has quit IRC | 21:19 | |
| fungi | in case anyone's on the edge of their seats for it, gerrit's online reindex doesn't seem to have completed yet | 21:20 |
| *** rfolco|ruck is now known as rfolco|off | 21:20 | |
| * dhellmann slides back to be more comfortable | 21:21 | |
| *** lpetrut has quit IRC | 21:26 | |
| *** eernst has quit IRC | 21:27 | |
| *** esberglu has quit IRC | 21:28 | |
| *** Swimingly has joined #openstack-infra | 21:28 | |
| *** lpetrut has joined #openstack-infra | 21:30 | |
| *** bnemec is now known as beekneemech | 21:32 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Remove unnecessary _route method from users https://review.openstack.org/387080 | 21:36 |
| *** jesusaur has quit IRC | 21:36 | |
| *** wolverineav has joined #openstack-infra | 21:36 | |
| *** boden has quit IRC | 21:37 | |
| corvus | diablo_rojo: i thought that looked familiar! :) | 21:48 |
| *** dave-mccowan has quit IRC | 21:57 | |
| *** jlabarre has quit IRC | 21:59 | |
| *** pblaho has quit IRC | 21:59 | |
| *** yamamoto has joined #openstack-infra | 22:02 | |
| *** gouthamr has joined #openstack-infra | 22:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/jeepyb master: Support cgit alias sites https://review.openstack.org/555105 | 22:04 |
| corvus | fungi, mordred, clarkb: ^ that's tested and working on git01.o.o; should be ready to merge | 22:04 |
| diablo_rojo | corvus, I hope you don't mind. I figured I would just fix that and then maybe this time we will get some py35 logs (it was one of the ones that was a casualty to the lost volumes a while bacl) | 22:05 |
| *** bobh has quit IRC | 22:05 | |
| *** e0ne has joined #openstack-infra | 22:05 | |
| corvus | diablo_rojo: of course not! i forgot about it :( | 22:06 |
| diablo_rojo | corvus, no worries :) Just circling back through open patches and trying to eek some things forward. | 22:07 |
| fungi | corvus: lgtm | 22:07 |
| *** yamamoto has quit IRC | 22:08 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: Use utf8mb4 for MySQL database charset https://review.openstack.org/555806 | 22:15 |
| *** Goneri has quit IRC | 22:25 | |
| mordred | corvus: cool. merging now | 22:25 |
| openstackgerrit | megan guiney proposed openstack-infra/subunit2sql master: Add ability to grab stream processing variables from kwargs https://review.openstack.org/543836 | 22:28 |
| *** eernst has joined #openstack-infra | 22:29 | |
| *** kjackal has quit IRC | 22:29 | |
| openstackgerrit | Merged openstack-infra/jeepyb master: Support cgit alias sites https://review.openstack.org/555105 | 22:33 |
| *** EmilienM has quit IRC | 22:34 | |
| *** EmilienM has joined #openstack-infra | 22:36 | |
| *** EmilienM has quit IRC | 22:36 | |
| *** EmilienM has joined #openstack-infra | 22:36 | |
| *** slagle has joined #openstack-infra | 22:37 | |
| *** hasharAway has quit IRC | 22:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade to webpack 4 https://review.openstack.org/551987 | 22:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 22:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 22:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use typescript for webpack config https://review.openstack.org/555948 | 22:40 |
| *** jamesmcarthur has joined #openstack-infra | 22:42 | |
| *** r-daneel has quit IRC | 22:43 | |
| *** rlandy has quit IRC | 22:43 | |
| dmsimard | Oh so there's a couple of folks in LA ? You guys have fun :D | 22:44 |
| dmsimard | mordred: woah, it's the first time I've seen "import openstack" in a python thing | 22:45 |
| dmsimard | That's... something | 22:45 |
| *** jamesmcarthur has quit IRC | 22:46 | |
| clarkb | and my flight is delayed (hopefully only slightly) | 22:47 |
| clarkb | mnaser so maybe dont try to stay awake :) | 22:47 |
| mnaser | clarkb: keep in touch | 22:50 |
| mrhillsman | I just got situated in hotel | 22:51 |
| *** mriedem is now known as mriedem_afk | 22:52 | |
| *** david-lyle has quit IRC | 22:52 | |
| *** e0ne has quit IRC | 23:02 | |
| *** gouthamr has quit IRC | 23:03 | |
| *** yamamoto has joined #openstack-infra | 23:04 | |
| *** lpetrut has quit IRC | 23:05 | |
| *** suhdood has joined #openstack-infra | 23:07 | |
| *** claudiub has quit IRC | 23:08 | |
| *** yamamoto has quit IRC | 23:10 | |
| *** yamahata has quit IRC | 23:16 | |
| *** tosky has quit IRC | 23:19 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Duplicating Users at Migration Time https://review.openstack.org/554726 | 23:24 |
| *** wolverineav has quit IRC | 23:24 | |
| *** jesusaur has joined #openstack-infra | 23:24 | |
| *** wolverineav has joined #openstack-infra | 23:24 | |
| *** wolverineav has quit IRC | 23:25 | |
| *** wolverineav has joined #openstack-infra | 23:25 | |
| *** Swimingly_ has joined #openstack-infra | 23:26 | |
| *** Swimingly has quit IRC | 23:26 | |
| *** wolverineav has quit IRC | 23:30 | |
| *** tpsilva has quit IRC | 23:31 | |
| *** wolverineav has joined #openstack-infra | 23:32 | |
| *** liusheng has quit IRC | 23:32 | |
| *** liusheng has joined #openstack-infra | 23:33 | |
| *** camunoz has quit IRC | 23:41 | |
| *** jesusaur has quit IRC | 23:42 | |
| *** jesusaur has joined #openstack-infra | 23:43 | |
| *** dkehn has quit IRC | 23:48 | |
| *** dkehn has joined #openstack-infra | 23:49 | |
| *** david-lyle has joined #openstack-infra | 23:50 | |
| *** felipemonteiro__ has quit IRC | 23:52 | |
| *** jgwentworth is now known as melwitt | 23:54 | |
| *** mriedem_afk has quit IRC | 23:56 | |
| *** jamesmcarthur has joined #openstack-infra | 23:57 | |
| *** eernst has quit IRC | 23:57 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Ironic to Storyboard https://review.openstack.org/554370 | 23:57 |
| *** chenying has quit IRC | 23:58 | |
| *** chenying has joined #openstack-infra | 23:58 | |
| *** Swami has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!