| *** jamesmcarthur has quit IRC | 00:00 | |
| *** rossella_s has quit IRC | 00:04 | |
| *** HeOS has quit IRC | 00:07 | |
| *** rossella_s has joined #openstack-infra | 00:08 | |
| *** odyssey4me has quit IRC | 00:09 | |
| *** odyssey4me has joined #openstack-infra | 00:09 | |
| *** xinliang has quit IRC | 00:11 | |
| *** dingyichen has joined #openstack-infra | 00:12 | |
| *** hongbin has joined #openstack-infra | 00:12 | |
| *** yamamoto has joined #openstack-infra | 00:13 | |
| *** rossella_s has quit IRC | 00:14 | |
| *** iyamahat has quit IRC | 00:16 | |
| *** yamamoto has quit IRC | 00:18 | |
| ianw | anyway, i've dropped a note to the afs list about the warning we are seeing : https://lists.openafs.org/pipermail/openafs-devel/2018-March/020473.html ... if we can play a role in getting the bionic packages more stable i think that's good | 00:19 |
|---|---|---|
| auristor | iamw: each rx connection has a limit of four simultaneous calls at a time. In 1.6 there is a limit of one rx connection per process authentication group (pag). In 1.8, there is a limit of three rx connections (or twelve calls) per PAG. That message is generated when the 12 call limit has been reached. | 00:21 |
| auristor | ianw: as I´ve mentioned previously, -512 is -ERESTARTSYS, which is the error returned when a syscall has been interrupted by a signal. I pointed you at the partial fix for -ERESTARTSYS errors that was not present in the version of OpenAFS 1.6 that you were running. | 00:23 |
| *** xinliang has joined #openstack-infra | 00:24 | |
| *** xinliang has quit IRC | 00:24 | |
| *** xinliang has joined #openstack-infra | 00:24 | |
| auristor | iamw: 110 (VBUSY) is the error returned from a file server when a client attempts to access a volume that is in use by the volserver. Most often due to volume moves or releases. | 00:25 |
| *** claudiub|2 has quit IRC | 00:27 | |
| ianw | auristor: yes, thanks, that was mostly why i switched to 1.8 series | 00:28 |
| ianw | it didn't seem worth us debugging 1.6 more and more when bionic is just around the corner with 1.8 | 00:28 |
| auristor | the concern you should have about 1.8 is that your fileservers are still configured to support only 23 calls at a time. If each PAG can issue 12 at a time, your fileservers cannot satisfy all of the requested by two PAGs on a single host let alone multiple hosts | 00:30 |
| *** jamesmcarthur has joined #openstack-infra | 00:37 | |
| ianw | auristor: thanks, as usual my afs knowledge is lagging a little behind your helpful comments :) | 00:40 |
| *** bobh has joined #openstack-infra | 00:44 | |
| ianw | auristor: so in our case, we only have the one writing host. but each mirror process, which starts under k5start with it's own keytab then runs it's stuff is effectively it's own PAG right? | 00:49 |
| ianw | to start with, i think us doing every mirror job at the top of the hour is not that great an idea; hence i proposed https://review.openstack.org/553894 to just offset the cron jobs | 00:49 |
| auristor | how about implementing the configuration changes we discussed a couple of months ago | 00:57 |
| ianw | auristor: if i hand edit those into the BosConfig, and then restart, is that the best way? | 01:01 |
| auristor | BosConfig should be considered a binary file that is owned by the bosserver. Updates must be made via bos commands | 01:02 |
| auristor | The bosserver maintains a copy of the contents in memory and writes them to BosConfig at shutdown. So modifying the file will result in the changes being lost. | 01:03 |
| *** gouthamr has quit IRC | 01:03 | |
| ianw | auristor: so to implement https://review.openstack.org/#/c/540198/2/doc/source/afs.rst ; i would bos stop dafileserver, then bos start it again with the new options? | 01:05 |
| ianw | (sorry, like i said, this is probably basic stuff but I'm playing catchup here :) | 01:05 |
| auristor | you would stop the dafs bnode. remove the bnode, add the new dafs bnode which would also start it | 01:05 |
| *** r-daneel has joined #openstack-infra | 01:06 | |
| *** cshastri has joined #openstack-infra | 01:13 | |
| *** yamamoto has joined #openstack-infra | 01:13 | |
| auristor | The commands you need are | 01:14 |
| auristor | bos stop http://doc.openafs.org/Reference/8/bos_stop.html | 01:14 |
| auristor | bos delete http://doc.openafs.org/Reference/8/bos_delete.html | 01:14 |
| auristor | bos create http://doc.openafs.org/Reference/8/bos_create.html | 01:14 |
| auristor | The bos instance you are modifying is ¨dafs¨ | 01:15 |
| auristor | I´m in London this week and its well past the time I should have gone to bed given when I need to wake up in the morning. | 01:15 |
| *** zhurong has joined #openstack-infra | 01:17 | |
| *** yamamoto has quit IRC | 01:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update AFS fileserver settings https://review.openstack.org/540198 | 01:22 |
| ianw | auristor: thanks, very helpful as always. I'll work on ^ to document what has to happen, and try it out | 01:22 |
| *** aeng has quit IRC | 01:22 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update AFS fileserver settings https://review.openstack.org/540198 | 01:24 |
| *** gouthamr has joined #openstack-infra | 01:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add allowed-triggers and allowed-reporters tenant settings https://review.openstack.org/554082 | 01:30 |
| *** r-daneel_ has joined #openstack-infra | 01:32 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update AFS fileserver settings https://review.openstack.org/540198 | 01:33 |
| *** r-daneel has quit IRC | 01:33 | |
| *** r-daneel_ is now known as r-daneel | 01:33 | |
| openstackgerrit | YumengBao proposed openstack-infra/project-config master: Set up cyborg-specs repository https://review.openstack.org/553975 | 01:35 |
| *** zhongjun_ has joined #openstack-infra | 01:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update AFS fileserver settings https://review.openstack.org/540198 | 01:36 |
| *** yamamoto has joined #openstack-infra | 01:41 | |
| *** rkukura has joined #openstack-infra | 01:41 | |
| openstackgerrit | YumengBao proposed openstack-infra/project-config master: Set up cyborg-specs repository https://review.openstack.org/553976 | 01:42 |
| openstackgerrit | YumengBao proposed openstack-infra/project-config master: Set up cyborg-specs repository https://review.openstack.org/553975 | 01:47 |
| *** VW has joined #openstack-infra | 01:59 | |
| *** rkukura has quit IRC | 02:04 | |
| *** niedbalski has joined #openstack-infra | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:20 | |
| *** jamesmca_ has joined #openstack-infra | 02:21 | |
| *** iyamahat has joined #openstack-infra | 02:24 | |
| *** annp has joined #openstack-infra | 02:27 | |
| *** zhurong has quit IRC | 02:36 | |
| *** Yumeng has joined #openstack-infra | 02:36 | |
| *** zhurong has joined #openstack-infra | 02:37 | |
| *** esberglu has joined #openstack-infra | 02:43 | |
| *** psachin has joined #openstack-infra | 02:50 | |
| *** salv-orl_ has joined #openstack-infra | 02:51 | |
| *** salv-orlando has quit IRC | 02:53 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add notes on reprepro _addreference https://review.openstack.org/554089 | 02:56 |
| *** esberglu has quit IRC | 02:58 | |
| *** VW_ has joined #openstack-infra | 03:01 | |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Add horizon to required-projects for solum-dashboard https://review.openstack.org/554091 | 03:02 |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Add horizon to required-projects for solum-dashboard https://review.openstack.org/554091 | 03:02 |
| *** VW has quit IRC | 03:03 | |
| *** jamesmca_ has quit IRC | 03:15 | |
| ianw | cryptography-2.2.tar.gz (md5, pgp) ... Source ... 2018-03-19 | 03:16 |
| *** germs has joined #openstack-infra | 03:17 | |
| *** germs has quit IRC | 03:17 | |
| *** germs has joined #openstack-infra | 03:17 | |
| ianw | hmm, might just be new release == no wheel | 03:17 |
| Jeffrey4l | got this error in ci, any idea? INFO:kolla.common.utils.tempest:[91mhttp://mirror.iad.rax.openstack.org:8080/rdo/centos7/46/5e/465e46159f59def9ba550cef0833f0e6469ac072_00a5f8f9/python2-oslotest-3.2.0-0.20180202131938.4f7f49a.el7.centos.noarch.rpm: [Errno 14] HTTP Error 403 - Forbidden | 03:20 |
| *** namnh has joined #openstack-infra | 03:21 | |
| *** germs has quit IRC | 03:22 | |
| *** rcernin_ has joined #openstack-infra | 03:23 | |
| *** jamesmcarthur has joined #openstack-infra | 03:25 | |
| *** rcernin has quit IRC | 03:25 | |
| ianw | Jeffrey4l: it's not there? that's a pass through for https://trunk.rdoproject.org/centos7/46/5e/465e46159f59def9ba550cef0833f0e6469ac072_00a5f8f9/ | 03:26 |
| Jeffrey4l | yeah. i just trid rdo, seems there are something wrong there. thanks. | 03:27 |
| *** rcernin_ has quit IRC | 03:29 | |
| *** rcernin has joined #openstack-infra | 03:29 | |
| ianw | ... /tmp/apply-ansible-env/bin/pip install ansible ... we just kind of assume this will work, without installing any binary dependencies | 03:29 |
| *** rcernin has quit IRC | 03:33 | |
| *** esberglu has joined #openstack-infra | 03:39 | |
| *** sree has joined #openstack-infra | 03:40 | |
| *** Kevin_Zheng has quit IRC | 03:40 | |
| *** esberglu has quit IRC | 03:43 | |
| *** hongbin has quit IRC | 03:44 | |
| *** udesale has joined #openstack-infra | 03:46 | |
| *** rcernin has joined #openstack-infra | 03:49 | |
| *** janki has joined #openstack-infra | 03:50 | |
| *** janki is now known as janki|office_ina | 03:51 | |
| *** wxy has quit IRC | 03:56 | |
| *** bobh has quit IRC | 04:11 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove cursive legacy jobs https://review.openstack.org/550999 | 04:16 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove neutron-dynamic-routing legacy jobs https://review.openstack.org/528988 | 04:20 |
| *** bobh has joined #openstack-infra | 04:20 | |
| *** ykarel has joined #openstack-infra | 04:20 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix ceph py35 job definition https://review.openstack.org/550497 | 04:21 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove migrated legacy puppet openstack libraries jobs https://review.openstack.org/550340 | 04:21 |
| *** rwsu has joined #openstack-infra | 04:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire 2 jenkins base ansible roles https://review.openstack.org/550185 | 04:24 |
| *** bobh has quit IRC | 04:25 | |
| *** rwsu has quit IRC | 04:25 | |
| *** sree_ has joined #openstack-infra | 04:29 | |
| *** sree_ is now known as Guest31421 | 04:29 | |
| *** ykarel_ has joined #openstack-infra | 04:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Make the task that remove the remote idempotent https://review.openstack.org/549886 | 04:30 |
| *** sree has quit IRC | 04:32 | |
| *** ykarel has quit IRC | 04:33 | |
| *** bobh has joined #openstack-infra | 04:34 | |
| *** bobh has quit IRC | 04:39 | |
| *** bobh has joined #openstack-infra | 04:43 | |
| *** bobh has quit IRC | 04:48 | |
| *** bobh has joined #openstack-infra | 04:51 | |
| *** claudiub has joined #openstack-infra | 04:52 | |
| *** gouthamr has quit IRC | 04:54 | |
| *** bobh has quit IRC | 04:57 | |
| *** bobh has joined #openstack-infra | 05:00 | |
| *** lpetrut has joined #openstack-infra | 05:01 | |
| *** zhurong has quit IRC | 05:04 | |
| *** bobh has quit IRC | 05:04 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add zookeeper driver and /{tenant}/labels controller https://review.openstack.org/553979 | 05:06 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add nodepool labels list page https://review.openstack.org/553980 | 05:06 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/nodes controller https://review.openstack.org/553998 | 05:06 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add nodepool nodes status page https://review.openstack.org/553999 | 05:06 |
| kazsh | Hi Infra root, could you kindly review this patch ? waiting for another +2 https://review.openstack.org/#/c/551978/ | 05:08 |
| *** bobh has joined #openstack-infra | 05:10 | |
| *** bobh has quit IRC | 05:10 | |
| *** jamesmcarthur has quit IRC | 05:13 | |
| *** imacdonn has quit IRC | 05:14 | |
| *** pgadiya has joined #openstack-infra | 05:14 | |
| *** imacdonn has joined #openstack-infra | 05:14 | |
| *** pahuang has quit IRC | 05:18 | |
| *** germs has joined #openstack-infra | 05:18 | |
| *** bobh has joined #openstack-infra | 05:21 | |
| *** pahuang has joined #openstack-infra | 05:22 | |
| *** germs has quit IRC | 05:23 | |
| *** bobh has quit IRC | 05:25 | |
| *** dhajare has joined #openstack-infra | 05:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add xstatic-** projecsts for heat-dashboard https://review.openstack.org/551978 | 05:31 |
| *** sree_ has joined #openstack-infra | 05:33 | |
| *** dsariel has joined #openstack-infra | 05:33 | |
| *** sree_ is now known as Guest5888 | 05:34 | |
| *** Guest31421 has quit IRC | 05:37 | |
| *** kjackal has joined #openstack-infra | 05:39 | |
| *** ykarel_ is now known as ykarel | 05:41 | |
| *** ccSnowCrash has joined #openstack-infra | 05:47 | |
| *** ccSnowCrash has quit IRC | 05:47 | |
| *** openstackgerrit has quit IRC | 05:49 | |
| *** iyamahat has quit IRC | 05:49 | |
| *** lpetrut has quit IRC | 05:50 | |
| *** udesale has quit IRC | 05:51 | |
| *** udesale has joined #openstack-infra | 05:52 | |
| *** bobh has joined #openstack-infra | 05:53 | |
| *** openstackgerrit has joined #openstack-infra | 05:54 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add Chef OpenStack meeting information https://review.openstack.org/554052 | 05:54 |
| *** bobh has quit IRC | 05:58 | |
| *** bobh has joined #openstack-infra | 06:00 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add Glance bug squad meeting https://review.openstack.org/553834 | 06:03 |
| *** gaurangt has quit IRC | 06:04 | |
| *** bobh has quit IRC | 06:05 | |
| *** bhavik1 has joined #openstack-infra | 06:10 | |
| *** bobh has joined #openstack-infra | 06:12 | |
| *** dhajare has quit IRC | 06:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/554132 | 06:17 |
| *** bobh has quit IRC | 06:17 | |
| *** bobh has joined #openstack-infra | 06:20 | |
| *** kiennt26 has joined #openstack-infra | 06:22 | |
| *** bobh has quit IRC | 06:25 | |
| *** ianychoi__ has joined #openstack-infra | 06:26 | |
| *** ianychoi_ has quit IRC | 06:29 | |
| *** bobh has joined #openstack-infra | 06:31 | |
| *** udesale_ has joined #openstack-infra | 06:34 | |
| *** bhavik1 has quit IRC | 06:35 | |
| *** bobh has quit IRC | 06:37 | |
| *** udesale has quit IRC | 06:38 | |
| *** alexchadin has joined #openstack-infra | 06:39 | |
| *** bobh has joined #openstack-infra | 06:41 | |
| *** bobh has quit IRC | 06:46 | |
| *** udesale_ is now known as udesale | 06:47 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add jobs of xstatic-** for heat-dashboard https://review.openstack.org/552735 | 06:50 |
| *** jcoufal has joined #openstack-infra | 06:57 | |
| *** d0ugal has joined #openstack-infra | 07:00 | |
| *** d0ugal has quit IRC | 07:00 | |
| *** d0ugal has joined #openstack-infra | 07:00 | |
| *** zhurong has joined #openstack-infra | 07:01 | |
| *** bobh has joined #openstack-infra | 07:03 | |
| *** rcernin has quit IRC | 07:07 | |
| *** bobh has quit IRC | 07:08 | |
| *** bobh has joined #openstack-infra | 07:11 | |
| *** armaan has quit IRC | 07:15 | |
| *** armaan has joined #openstack-infra | 07:15 | |
| *** bobh has quit IRC | 07:16 | |
| *** udesale_ has joined #openstack-infra | 07:18 | |
| *** udesale has quit IRC | 07:18 | |
| *** germs has joined #openstack-infra | 07:18 | |
| *** germs has quit IRC | 07:18 | |
| *** germs has joined #openstack-infra | 07:18 | |
| *** dhajare has joined #openstack-infra | 07:20 | |
| *** yolanda_ is now known as yolanda | 07:21 | |
| *** germs has quit IRC | 07:23 | |
| *** bobh has joined #openstack-infra | 07:23 | |
| *** bobh has quit IRC | 07:27 | |
| *** kjackal_ has joined #openstack-infra | 07:34 | |
| *** andreas_s has joined #openstack-infra | 07:38 | |
| *** pcaruana has joined #openstack-infra | 07:39 | |
| *** bobh has joined #openstack-infra | 07:42 | |
| *** bobh has quit IRC | 07:47 | |
| *** claudiub has quit IRC | 07:48 | |
| *** florianf has joined #openstack-infra | 07:49 | |
| *** e0ne has joined #openstack-infra | 07:52 | |
| *** AJaeger has quit IRC | 07:53 | |
| *** zhurong has quit IRC | 07:54 | |
| openstackgerrit | Zhao Chao proposed openstack-infra/project-config master: Use -python35-jobs-horizon for trove-dashboard https://review.openstack.org/553116 | 07:55 |
| *** bobh has joined #openstack-infra | 07:55 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects for solum-dashboard https://review.openstack.org/554091 | 07:56 |
| *** kjackal has quit IRC | 07:56 | |
| *** liujiong has joined #openstack-infra | 07:56 | |
| *** lpetrut has joined #openstack-infra | 07:57 | |
| *** bobh has quit IRC | 07:59 | |
| *** lpetrut_ has joined #openstack-infra | 08:00 | |
| *** lpetrut has quit IRC | 08:00 | |
| *** vivsoni_ has joined #openstack-infra | 08:01 | |
| *** aluria` is now known as aluria | 08:04 | |
| *** aluria has quit IRC | 08:05 | |
| *** aluria has joined #openstack-infra | 08:05 | |
| *** tesseract has joined #openstack-infra | 08:17 | |
| *** udesale_ has quit IRC | 08:23 | |
| *** janki|office_ina has quit IRC | 08:25 | |
| *** numans_ has joined #openstack-infra | 08:26 | |
| *** numans has quit IRC | 08:29 | |
| *** udesale_ has joined #openstack-infra | 08:30 | |
| *** numans has joined #openstack-infra | 08:38 | |
| *** slaweq has quit IRC | 08:38 | |
| *** jaosorior has joined #openstack-infra | 08:40 | |
| *** kjackal has joined #openstack-infra | 08:42 | |
| *** numans_ has quit IRC | 08:42 | |
| *** kjackal_ has quit IRC | 08:42 | |
| *** claudiub has joined #openstack-infra | 08:42 | |
| *** numans has quit IRC | 08:42 | |
| *** gfidente has joined #openstack-infra | 08:43 | |
| *** gfidente has joined #openstack-infra | 08:43 | |
| *** amoralej|off is now known as amoralej | 08:45 | |
| *** numans has joined #openstack-infra | 08:48 | |
| *** iyamahat has joined #openstack-infra | 08:50 | |
| *** bobh has joined #openstack-infra | 08:51 | |
| *** bobh has quit IRC | 08:56 | |
| *** dingyichen has quit IRC | 08:57 | |
| *** iyamahat has quit IRC | 08:57 | |
| *** yamahata has quit IRC | 08:58 | |
| *** jpena|off is now known as jpena | 08:59 | |
| *** shardy has joined #openstack-infra | 08:59 | |
| *** Guest5888 has quit IRC | 09:00 | |
| *** slaweq has joined #openstack-infra | 09:00 | |
| *** jtomasek has joined #openstack-infra | 09:02 | |
| *** sree_ has joined #openstack-infra | 09:04 | |
| *** sree_ is now known as Guest45662 | 09:04 | |
| *** rpittau has joined #openstack-infra | 09:07 | |
| *** rpittau has quit IRC | 09:08 | |
| *** Guest45662 has quit IRC | 09:08 | |
| *** salv-orl_ has quit IRC | 09:09 | |
| *** salv-orlando has joined #openstack-infra | 09:09 | |
| *** rpittau has joined #openstack-infra | 09:10 | |
| *** bobh has joined #openstack-infra | 09:14 | |
| *** salv-orlando has quit IRC | 09:14 | |
| *** psachin has quit IRC | 09:19 | |
| *** bobh has quit IRC | 09:19 | |
| *** chkumar|off is now known as chandankumare | 09:21 | |
| *** chandankumare is now known as chandankumar | 09:21 | |
| *** bobh has joined #openstack-infra | 09:21 | |
| *** psachin has joined #openstack-infra | 09:24 | |
| *** bobh has quit IRC | 09:26 | |
| *** danpawlik has joined #openstack-infra | 09:29 | |
| *** udesale_ has quit IRC | 09:30 | |
| *** gibi_ is now known as gibi | 09:30 | |
| *** bobh has joined #openstack-infra | 09:30 | |
| *** udesale has joined #openstack-infra | 09:32 | |
| *** hashar has joined #openstack-infra | 09:32 | |
| *** bobh has quit IRC | 09:35 | |
| *** hashar has quit IRC | 09:39 | |
| *** gus has quit IRC | 09:43 | |
| *** jamielennox has quit IRC | 09:43 | |
| *** gus has joined #openstack-infra | 09:44 | |
| *** jamielennox has joined #openstack-infra | 09:48 | |
| *** inc0 has quit IRC | 09:50 | |
| openstackgerrit | Zhao Chao proposed openstack-infra/project-config master: Use -python35-jobs-horizon for trove-dashboard https://review.openstack.org/553116 | 09:51 |
| openstackgerrit | Zhao Chao proposed openstack-infra/project-config master: Use -python35-jobs-horizon for trove-dashboard https://review.openstack.org/553116 | 09:52 |
| *** HeOS has joined #openstack-infra | 09:52 | |
| *** rosmaita has joined #openstack-infra | 09:57 | |
| *** bobh has joined #openstack-infra | 09:57 | |
| *** liujiong has quit IRC | 09:59 | |
| *** bobh has quit IRC | 10:01 | |
| *** inc0 has joined #openstack-infra | 10:03 | |
| *** hashar has joined #openstack-infra | 10:03 | |
| *** bobh has joined #openstack-infra | 10:05 | |
| *** namnh has quit IRC | 10:05 | |
| *** sambetts_ is now known as sambetts | 10:08 | |
| *** bobh has quit IRC | 10:10 | |
| *** kiennt26 has quit IRC | 10:10 | |
| *** bobh has joined #openstack-infra | 10:12 | |
| *** pgadiya has quit IRC | 10:14 | |
| *** tosky has joined #openstack-infra | 10:15 | |
| *** yamamoto has quit IRC | 10:16 | |
| *** bobh has quit IRC | 10:16 | |
| *** cshastri has quit IRC | 10:19 | |
| *** dtantsur|afk is now known as dtantsur | 10:19 | |
| *** e0ne has quit IRC | 10:19 | |
| *** pgadiya has joined #openstack-infra | 10:22 | |
| *** bobh has joined #openstack-infra | 10:22 | |
| *** e0ne has joined #openstack-infra | 10:25 | |
| *** bobh has quit IRC | 10:28 | |
| *** zoli is now known as zoli|lunch | 10:29 | |
| *** zoli|lunch is now known as zoli | 10:29 | |
| *** jcoufal_ has joined #openstack-infra | 10:32 | |
| *** jcoufal has quit IRC | 10:35 | |
| *** jcoufal has joined #openstack-infra | 10:35 | |
| *** annp has quit IRC | 10:37 | |
| *** jcoufal_ has quit IRC | 10:38 | |
| *** jcoufal_ has joined #openstack-infra | 10:40 | |
| *** jcoufal has quit IRC | 10:42 | |
| *** sree has joined #openstack-infra | 10:43 | |
| *** bobh has joined #openstack-infra | 10:45 | |
| *** sree has quit IRC | 10:48 | |
| *** bobh has quit IRC | 10:51 | |
| *** e0ne has quit IRC | 10:51 | |
| *** bobh has joined #openstack-infra | 10:51 | |
| *** bobh has quit IRC | 10:56 | |
| *** chason has quit IRC | 10:57 | |
| *** chason has joined #openstack-infra | 10:58 | |
| *** bobh has joined #openstack-infra | 11:01 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| *** salv-orlando has joined #openstack-infra | 11:05 | |
| *** bobh has quit IRC | 11:06 | |
| *** bobh has joined #openstack-infra | 11:13 | |
| *** edmondsw has joined #openstack-infra | 11:14 | |
| *** edmondsw has quit IRC | 11:18 | |
| *** bobh has quit IRC | 11:18 | |
| *** germs has joined #openstack-infra | 11:20 | |
| *** germs has quit IRC | 11:20 | |
| *** germs has joined #openstack-infra | 11:20 | |
| *** bobh has joined #openstack-infra | 11:20 | |
| *** alexchadin has quit IRC | 11:21 | |
| *** alexchadin has joined #openstack-infra | 11:21 | |
| *** alexchadin has quit IRC | 11:21 | |
| *** alexchadin has joined #openstack-infra | 11:22 | |
| *** pbourke_ has quit IRC | 11:22 | |
| *** alexchadin has quit IRC | 11:22 | |
| *** pbourke_ has joined #openstack-infra | 11:22 | |
| *** katkapilatova has joined #openstack-infra | 11:23 | |
| *** alexchadin has joined #openstack-infra | 11:23 | |
| *** alexchadin has quit IRC | 11:23 | |
| *** e0ne has joined #openstack-infra | 11:23 | |
| *** alexchadin has joined #openstack-infra | 11:24 | |
| *** alexchadin has quit IRC | 11:24 | |
| *** alexchadin has joined #openstack-infra | 11:24 | |
| *** alexchadin has quit IRC | 11:24 | |
| *** bobh has quit IRC | 11:25 | |
| *** germs has quit IRC | 11:25 | |
| *** pcichy has joined #openstack-infra | 11:27 | |
| *** bobh has joined #openstack-infra | 11:30 | |
| *** ldnunes has joined #openstack-infra | 11:33 | |
| *** pblaho has joined #openstack-infra | 11:33 | |
| *** chason has quit IRC | 11:35 | |
| *** alexchadin has joined #openstack-infra | 11:35 | |
| *** chason has joined #openstack-infra | 11:35 | |
| *** pgadiya has quit IRC | 11:35 | |
| *** pgadiya has joined #openstack-infra | 11:36 | |
| *** bobh has quit IRC | 11:36 | |
| *** e0ne has quit IRC | 11:40 | |
| *** bobh has joined #openstack-infra | 11:40 | |
| *** rfolco has joined #openstack-infra | 11:42 | |
| *** e0ne has joined #openstack-infra | 11:44 | |
| *** bobh has quit IRC | 11:45 | |
| *** alexchadin has quit IRC | 11:49 | |
| *** armaan has quit IRC | 11:51 | |
| *** logan- has quit IRC | 11:52 | |
| *** armaan has joined #openstack-infra | 11:52 | |
| *** logan- has joined #openstack-infra | 11:52 | |
| *** bobh has joined #openstack-infra | 11:53 | |
| *** armaan has quit IRC | 11:53 | |
| *** armaan has joined #openstack-infra | 11:53 | |
| *** ldnunes has quit IRC | 11:57 | |
| *** ldnunes has joined #openstack-infra | 11:57 | |
| *** beagles_biab is now known as beagesl | 12:00 | |
| *** beagesl is now known as beagles | 12:00 | |
| *** psachin has quit IRC | 12:00 | |
| *** VW_ has quit IRC | 12:01 | |
| *** efried has quit IRC | 12:03 | |
| *** ramishra has joined #openstack-infra | 12:04 | |
| *** pgadiya has quit IRC | 12:05 | |
| *** kashyap has joined #openstack-infra | 12:06 | |
| kashyap | Hi folks, I'm seeing something bizarre: I can't just fetch from: git.openstack.org :-( | 12:06 |
| kashyap | My 'origin' is this: | 12:06 |
| kashyap | origin https://git.openstack.org/openstack/nova.git (fetch) | 12:06 |
| kashyap | origin https://git.openstack.org/openstack/nova.git (push) | 12:06 |
| kashyap | And I tried: `GIT_CURL_VERBOSE=1 GIT_TRACE=1 git pull`, that too hung here: http://paste.openstack.org/show/704380/ | 12:07 |
| kashyap | Note that I get through the TLS auth phase | 12:07 |
| kashyap | Any hints? | 12:07 |
| kashyap | I tried 'git' / 'https'. Also 'github' fails too. | 12:08 |
| kashyap | (But note: I fetch from other non-OpenStack repositories.) | 12:08 |
| *** jlabarre has joined #openstack-infra | 12:08 | |
| kashyap | It's something surely local | 12:08 |
| *** psachin has joined #openstack-infra | 12:11 | |
| *** trown|outtypewww is now known as trown|ruck | 12:12 | |
| *** edmondsw has joined #openstack-infra | 12:15 | |
| frickler | could some suse folks look into this failure and check whether we have some mirror issue or some upstream breakage? http://logs.openstack.org/65/535365/1/check/devstack-platform-opensuse-tumbleweed/2e118b2/ara/result/db54aae7-068e-40d1-801a-17b0ab1b4adc/ | 12:15 |
| frickler | kashyap: so the git pull just hangs at that point? one thing to check for might be firewall/mtu issues | 12:17 |
| kashyap | frickler: Yeah, indeed it hangs. A fresh clone hangs / doesn't pull any detail | 12:18 |
| kashyap | I'm on my home n/w | 12:18 |
| kashyap | And I am able to fetch from other repos | 12:18 |
| kashyap | frickler: It's not an MTU issue because I can fetch other HTTPS websites & repos | 12:18 |
| *** chason has quit IRC | 12:19 | |
| *** chason has joined #openstack-infra | 12:19 | |
| *** gyankum has joined #openstack-infra | 12:22 | |
| *** stevebaker has quit IRC | 12:22 | |
| cmurphy | dirk: i wonder if our network issues are affecting the mirror problems frickler pointed out ^ | 12:23 |
| *** bobh has quit IRC | 12:23 | |
| *** tosky has quit IRC | 12:24 | |
| *** stevebaker has joined #openstack-infra | 12:24 | |
| *** dmellado has quit IRC | 12:24 | |
| kashyap | frickler: For instance, fresh clone doesn't fetch any refs: http://paste.openstack.org/show/704408/ | 12:24 |
| *** chason has quit IRC | 12:25 | |
| kashyap | cmurphy: Hi there; the network issues also affected GitHub? | 12:25 |
| *** chason has joined #openstack-infra | 12:25 | |
| cmurphy | kashyap: no, unrelated to your problem | 12:25 |
| kashyap | Ah, sorry. | 12:25 |
| *** dmellado has joined #openstack-infra | 12:27 | |
| *** tpsilva has joined #openstack-infra | 12:28 | |
| *** bhagyashris has joined #openstack-infra | 12:33 | |
| dirk | frickler: the mirror looks broken, essential files are missing | 12:34 |
| dirk | frickler: compare https://download.opensuse.org/tumbleweed/repo/oss/suse/setup/descr/ with http://mirror.gra1.ovh.openstack.org/opensuse/tumbleweed/repo/oss/suse/setup/descr/ | 12:35 |
| frickler | infra-root: can someone try to repair? /me is afk now for a bit | 12:38 |
| *** odyssey4me has quit IRC | 12:42 | |
| *** odyssey4me has joined #openstack-infra | 12:42 | |
| *** rlandy has joined #openstack-infra | 12:43 | |
| *** pas-ha has joined #openstack-infra | 12:49 | |
| pas-ha | hi all, I'm trying to reuse the base 'devstack' zuul3 job. I need to setup a very lightweight version of devstack with only keystone and neutron. Is there any way to override default enabled services completely? I've tried already 'devstack_services: base: False' but nova, cinder etc are still appended (as I presume they are defimed somewhere above and zuul just merges those dicts) | 12:52 |
| pas-ha | the patch I'm talking about is here https://review.openstack.org/#/c/553813/ | 12:52 |
| kashyap | Alright, it works now; some blip | 12:52 |
| *** VW has joined #openstack-infra | 12:53 | |
| *** armaan has quit IRC | 12:53 | |
| *** kgiusti has joined #openstack-infra | 12:54 | |
| *** jpena is now known as jpena|lunch | 12:56 | |
| *** dprince has joined #openstack-infra | 12:57 | |
| *** danpawlik has quit IRC | 12:57 | |
| *** tosky has joined #openstack-infra | 13:02 | |
| *** dhill_ has joined #openstack-infra | 13:04 | |
| *** mriedem has joined #openstack-infra | 13:06 | |
| *** amoralej is now known as amoralej|lunch | 13:08 | |
| *** yamamoto has quit IRC | 13:08 | |
| *** efried has joined #openstack-infra | 13:13 | |
| *** AJaeger has joined #openstack-infra | 13:14 | |
| *** germs has joined #openstack-infra | 13:21 | |
| *** germs has quit IRC | 13:21 | |
| *** germs has joined #openstack-infra | 13:21 | |
| *** psachin has quit IRC | 13:24 | |
| *** germs has quit IRC | 13:26 | |
| pas-ha | aha, I think I got this, I need to use the `devstack-base` job as a parent then.. | 13:27 |
| *** shardy is now known as shardy_mtg | 13:28 | |
| *** Goneri has joined #openstack-infra | 13:29 | |
| pas-ha | but would have to re-implement the 'devstack-tempest' ... sigh... | 13:30 |
| *** sshnaidm is now known as sshnaidm|afk | 13:31 | |
| *** dhajare has quit IRC | 13:34 | |
| tosky | pas-ha: or you override devstack-tempest and set all services to False (long list, but...) | 13:41 |
| *** eharney has joined #openstack-infra | 13:42 | |
| *** jroll has quit IRC | 13:42 | |
| tosky | pas-ha: also, you should remove dsvm from the name of the job | 13:43 |
| *** jroll has joined #openstack-infra | 13:43 | |
| fungi | might highlight a usecase for a separate tempest job with no services, as a building block? | 13:43 |
| *** hamzy__ is now known as hamzy | 13:43 | |
| tosky | or that | 13:43 |
| *** yamamoto has joined #openstack-infra | 13:45 | |
| *** psachin has joined #openstack-infra | 13:47 | |
| *** esberglu has joined #openstack-infra | 13:47 | |
| *** yamamoto_ has joined #openstack-infra | 13:48 | |
| *** kashyap has left #openstack-infra | 13:50 | |
| *** yamamoto has quit IRC | 13:52 | |
| *** dtantsur is now known as dtantsur|brb | 13:56 | |
| *** rkukura has joined #openstack-infra | 13:56 | |
| *** amoralej|lunch is now known as amoralej | 13:56 | |
| *** rkukura has quit IRC | 13:57 | |
| AJaeger | mordred: please check http://logs.openstack.org/22/554222/2/check/openstack-tox-pep8/700407f/job-output.txt.gz - the neutron install there looks wrong in the interaction with tox-siblings etc. See also the emails on openstack-dev with garyk | 13:58 |
| *** ramishra has quit IRC | 13:58 | |
| *** myoung|afk is now known as myoung|rover | 13:58 | |
| pas-ha | tosky, doing just that right now, although it is too explicit IMO | 13:58 |
| pas-ha | fungi: that would be perfect, a devstack-base-tempest job with parent of 'devstack-base', although there'd be duplication in the tempest's zuul.yaml then (but not too much) | 13:59 |
| tosky | pas-ha: I think that the basic tempest job with no services was mentioned during the PTG, but IIRC with no enough use cases; andreaf, do you think it could be added? | 14:00 |
| pas-ha | fungi: AFAIU zuul3 does not support multiple parents / mixins (yet? :-) ) | 14:00 |
| *** hongbin has joined #openstack-infra | 14:00 | |
| tosky | pas-ha: yep, but minimal (the playbooks are already there); did you see the other comment about job naming? | 14:00 |
| pas-ha | yes, will do | 14:01 |
| *** beekneemech is now known as bnemec | 14:02 | |
| pas-ha | but the doc you mentioned is actually inconsistent a bit. On one hand it proposes to set MAINPURPOSE to e.g. 'tempest', but then in the examples it says `legacy-tempest-dsvm-nova-lvm -> nova-lvm` | 14:03 |
| pas-ha | so I'll drop `dsvm` from the name but not sure if I have to drop `tempest` as well. advice? | 14:03 |
| fungi | do you expect there to be additional jobs whose primary purpose is to test nova with lvm? | 14:04 |
| fungi | the point is really just to keep the name short yet representative of what you want to achieve with it | 14:05 |
| fungi | you could consider the fact that it's using tempest to achieve that in a devstack environment to be mere implementation details | 14:05 |
| *** sree has joined #openstack-infra | 14:06 | |
| *** jpena|lunch is now known as jpena | 14:06 | |
| *** psachin has quit IRC | 14:06 | |
| *** sree has quit IRC | 14:10 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:12 | |
| *** bobh has joined #openstack-infra | 14:13 | |
| *** gouthamr has joined #openstack-infra | 14:13 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:14 | |
| *** psachin has joined #openstack-infra | 14:17 | |
| *** felipemonteiro_ has quit IRC | 14:18 | |
| pabelanger | dmanchad: ianw: it would be great if we can bring online linestone this week, anything I can do to help? | 14:20 |
| pabelanger | fungi: clarkb: maybe once everybody is caffeinated we can talk through the process of starting up review-dev.o.o with puppet. I tried debugging a few times last week, and wasn't able to properly launch it. Because it looked like gerrit was trying to init and maybe update the production database for review-dev.o.o | 14:21 |
| *** tellesnobrega has quit IRC | 14:22 | |
| *** psachin has quit IRC | 14:23 | |
| fungi | frickler: cmurphy: dirk: looking at our mirror update log, it seems our rsyncs from ftp.gwdg.de are not completing in the allotted 30 minutes (large update today?) but for whatever reason when the timeout wrapper kills the rsync process its signal handler seems to cause it to exit 0 (or at least that's the only explanation i can come up with for why it's not trapping on set -e and instead proceeding | 14:25 |
| fungi | to perform a vos release of incomplete data) | 14:25 |
| fungi | so anyway, the suse mirror issue is easily explained: we're downloading 30 minutes worth every 4 hours so it's still incomplete for the moment | 14:26 |
| dirk | fungi: well, tumbleweed updates are always "huge". | 14:27 |
| fungi | i can manually take the lock and initiate a sync with no timeout to get it back into good shape, but figuring out how to fix the script so this doesn't recur is a separate matter | 14:27 |
| pabelanger | yah, we might want to bump tha. I think it is just copypaste from other script | 14:27 |
| dirk | fungi: can you share the rsync statistics (e.g. download rate and so on)? | 14:27 |
| fungi | dirk: sure, i'll add some flags to rsync, we don't set stats on by default i don't think | 14:28 |
| *** dayou has quit IRC | 14:28 | |
| fungi | pabelanger: well, it would also be better to find a way to make sure we don't vos release if the rsync times out | 14:28 |
| fungi | because we designed this for atomic updating specifically to avoid this situation | 14:29 |
| pabelanger | I thought we didn't, we'd get error 1 from timeout and bash script would exit | 14:29 |
| fungi | rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(638) [generator=3.1.1] | 14:30 |
| fungi | rsync error: received SIGUSR1 (code 19) at main.c(1434) [receiver=3.1.1] | 14:30 |
| fungi | [...] | 14:30 |
| fungi | rsync completed successfully, running vos release. | 14:30 |
| *** yamamoto_ has quit IRC | 14:30 | |
| pabelanger | ouch | 14:30 |
| pabelanger | yah, we should see about fixing that | 14:30 |
| fungi | so seems something's eating the nonzero exit, if rsync is exiting nonzero | 14:30 |
| pabelanger | man page says we should be getting 124 exit code for timeout | 14:32 |
| *** salv-orlando has quit IRC | 14:32 | |
| fungi | we run timeout directly under k5start... maybe it's kerberos not plumbing through the exit code? | 14:32 |
| *** salv-orlando has joined #openstack-infra | 14:33 | |
| fungi | easy enough to test if someone gets time | 14:33 |
| fungi | in the interim, i'll work on getting this mirror to complete | 14:33 |
| *** felipemonteiro__ has quit IRC | 14:34 | |
| dirk | hmm, looks like k5start returns with exit 0 code then | 14:34 |
| *** kiennt26 has joined #openstack-infra | 14:34 | |
| dirk | maybe we want -x ? | 14:35 |
| dirk | (on k5start) | 14:35 |
| *** pas-ha has left #openstack-infra | 14:36 | |
| *** gyankum has quit IRC | 14:36 | |
| fungi | -x only talks about failures to refresh the ticket cache, but worth testing whether it considers nonzero termination of a child process as an error too | 14:36 |
| fungi | the return values section of the k5start manpage makes it sound like it should do the right thing normally | 14:37 |
| *** sree has joined #openstack-infra | 14:38 | |
| fungi | "If k5start runs aklog or some other program k5start returns the exit status of that program." | 14:38 |
| *** tellesnobrega has joined #openstack-infra | 14:40 | |
| *** boden has joined #openstack-infra | 14:40 | |
| fungi | the manually-initiated mirror run is chewing on the tumbleweed file list now | 14:41 |
| boden | AJaeger ping if you get a sec pls | 14:41 |
| fungi | i added --stats to the rsync invocations to get some more detail on performance | 14:41 |
| *** felipemonteiro_ has joined #openstack-infra | 14:42 | |
| *** VW has quit IRC | 14:43 | |
| *** VW has joined #openstack-infra | 14:43 | |
| *** yamamoto has joined #openstack-infra | 14:46 | |
| *** salv-orlando has quit IRC | 14:46 | |
| *** salv-orlando has joined #openstack-infra | 14:47 | |
| *** yamahata has joined #openstack-infra | 14:50 | |
| *** arxcruz|off is now known as arxcruz | 14:50 | |
| *** udesale has quit IRC | 14:50 | |
| *** salv-orlando has quit IRC | 14:51 | |
| *** yamamoto has quit IRC | 14:51 | |
| *** sree_ has joined #openstack-infra | 14:52 | |
| *** sree_ is now known as Guest82448 | 14:52 | |
| clarkb | pabelanger: part of the init step for gerrit is to do db migrations if necessary. It is probably just checking if it needs to do any migrations but should find that it does not need to | 14:52 |
| fungi | infra-root: reminder, if you manually add files (e.g., a README.txt) to the tarballs site, _please_ set permissions and ownership consistent with the other files our jobs are publishing. if you don't, it causes failures like this one: http://logs.openstack.org/5b/5b4b88cbe957d10549668f9b3fab3157f934880f/pre-release/release-openstack-python/1f8e9ca/ara/result/726635a5-6d60-43f2-89b9-c507d00cab84/ | 14:53 |
| corvus | fungi, pabelanger: this script seems to show the right thing happening: http://paste.openstack.org/show/704553/ | 14:54 |
| clarkb | fungi: might make that not recursive so that if the dir is fine the files in it won't affect it? | 14:54 |
| fungi | clarkb: yeah, i don't know what the ansible solution to it is, but this is not the first one i've had to repair | 14:55 |
| dmsimard | fungi: the mistake is unfortunate but I'm happy that at least the error is very obvious | 14:55 |
| *** sree has quit IRC | 14:55 | |
| * dmsimard wishes every error could be obvious like that one | 14:55 | |
| fungi | #status log manually set ownership of /srv/static/tarballs/kolla/images/README.txt from root:root to jenkins:jenkins so that release jobs no longer fail uploading | 14:56 |
| openstackstatus | fungi: finished logging | 14:56 |
| clarkb | fungi: in this particular case I assume the dir is being made so that we can't write a tarball to it, and the readme or any other file already in the dir shouldn't prevent that | 14:56 |
| clarkb | pabelanger: do you have more info on what gerrit is doing? log output maybe? | 14:58 |
| fungi | clarkb: nope, the directory is owned by jenkins:jenkins but if ansible sees a file in there for which it lacks write access it seems to fail that way | 14:58 |
| clarkb | fungi: http://logs.openstack.org/5b/5b4b88cbe957d10549668f9b3fab3157f934880f/pre-release/release-openstack-python/1f8e9ca/ara/file/afffbdb5-e4ba-4403-ba54-85c50e6f7f03/#line-1 because it is recursively setting the permissions | 15:00 |
| clarkb | its the recurse: yes and mode: 0775 together that do it | 15:00 |
| fungi | yeah. i suppose that's something we want it doing | 15:00 |
| fungi | though setting 0775 on all the uploaded files is sort of strange | 15:00 |
| fungi | would be better to only set the directories to 0775 and the files to something like 0664 | 15:01 |
| clarkb | in puppet land you set mode to 0664 then it automagically adds +x to that for dirs | 15:01 |
| * clarkb reads ansible docs | 15:01 | |
| fungi | could maybe use something like ug=rwX,o=rX | 15:02 |
| fungi | taking advantage of the magic X that sets executable bit on directories but not regular files | 15:03 |
| clarkb | it does accept mode strings in that form but doesn't specify if it honors the magic X | 15:03 |
| pabelanger | clarkb: yah, I'll run with --keep this time to start properly debugging | 15:03 |
| clarkb | looks like the recurse: yes is so that it will create all the directories necessary in that path | 15:04 |
| *** yamamoto has joined #openstack-infra | 15:04 | |
| corvus | pabelanger: what are you using --keep for? | 15:04 |
| *** felipemonteiro_ has quit IRC | 15:04 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:05 | |
| *** VW has quit IRC | 15:06 | |
| *** eernst has joined #openstack-infra | 15:07 | |
| *** ykarel has quit IRC | 15:09 | |
| *** dtantsur|brb is now known as dtantsur | 15:10 | |
| clarkb | corvus: the review-dev replacement I think | 15:11 |
| *** iyamahat has joined #openstack-infra | 15:11 | |
| fungi | yeah, the puppet-gerrit module isn't capable of bootstrapping gerrit sanely, so will fail to apply completely | 15:12 |
| corvus | we need to switch all the executors to --keep to debug that? | 15:12 |
| clarkb | corvus: no, launch-node with keep | 15:13 |
| fungi | corvus: different --keep | 15:13 |
| corvus | oh, i thought this was for the gerrit tests. gotcha. | 15:13 |
| fungi | past gerrits we've built, i believe, were initially puppeted bare and then puppet-gerrit added once they were up, and the installation fixed up in place manually | 15:13 |
| fungi | gerrit itself doesn't lend itself well to being bootstrapped automatically from scratch | 15:14 |
| *** hashar is now known as hasharAway | 15:14 | |
| fungi | what's the magic sauce for referring to a section of a sphinx doc from another sphinx document in the same tree? | 15:15 |
| fungi | :ref:section-name-string doesn't seem to be sufficient | 15:15 |
| *** kiennt26 has quit IRC | 15:15 | |
| pabelanger | corvus: to bring review-dev01.o.o online, puppet is failing | 15:15 |
| fungi | oh, or maybe i've just got the syntax for renaming the hyperlink wrong | 15:15 |
| *** eharney has quit IRC | 15:21 | |
| fungi | yep, that was it | 15:21 |
| *** germs has joined #openstack-infra | 15:22 | |
| *** germs has quit IRC | 15:22 | |
| *** germs has joined #openstack-infra | 15:22 | |
| clarkb | dmsimard: comments on https://review.openstack.org/#/c/513874/6 | 15:23 |
| clarkb | dmsimard: mostly its not compiling docs successfully due to some warning I don't fully understand yet | 15:24 |
| dmsimard | clarkb: yeah, I've seen it -- haven't had the time to address it yet. Thanks for the reminder | 15:24 |
| frickler | FYI, pip likes to further break stuff not only with major releases, but also patch releases https://github.com/pypa/pip/issues/5079 | 15:24 |
| *** eharney has joined #openstack-infra | 15:26 | |
| *** germs has quit IRC | 15:26 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Update Gerrit project renaming for Zuul v3 https://review.openstack.org/554261 | 15:27 |
| *** thingee has quit IRC | 15:27 | |
| *** VW has joined #openstack-infra | 15:28 | |
| fungi | frickler: cmurphy: dirk: manual suse mirror update has completed, looks like. as requested, here's the detail from adding --stats while pulling the remaining tumbleweed content: http://paste.openstack.org/show/704599/ | 15:30 |
| fungi | looks like we averaged around 1.6MB/sec from ftp.gwdg.de | 15:30 |
| *** thingee has joined #openstack-infra | 15:31 | |
| *** thingee has left #openstack-infra | 15:31 | |
| dirk | Which isn't bad I guess | 15:32 |
| *** eernst has quit IRC | 15:32 | |
| fungi | it's pretty reasonable | 15:32 |
| clarkb | especially if transatlantic | 15:32 |
| fungi | which i'm guessing it is | 15:32 |
| dirk | So the initial rsync filelist generation takes time? It only transfered for about a minute it seems | 15:32 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add a loading_errors info to {tenant}/info endpoint https://review.openstack.org/553873 | 15:33 |
| dirk | fungi: yeah, there's a mirror in US, we might want to switch to that | 15:33 |
| fungi | dirk: maybe you're off by a few orders of magnitude. my math says 3290926312 bytes at 1552952.22 per second is 2119 seconds or roughly 35 minutes | 15:34 |
| dirk | fungi: ah | 15:35 |
| fungi | and yeah, switching to a closer mirror, if it's reliable, might help the transfer rate a little | 15:35 |
| *** VW_ has joined #openstack-infra | 15:35 | |
| *** VW_ has quit IRC | 15:35 | |
| *** VW_ has joined #openstack-infra | 15:36 | |
| *** eernst has joined #openstack-infra | 15:38 | |
| *** Guest82448 has quit IRC | 15:38 | |
| dirk | Fungi: sorry, just a guess from looking at it without a calculator | 15:38 |
| *** VW has quit IRC | 15:38 | |
| *** VW__ has joined #openstack-infra | 15:38 | |
| *** VW_ has quit IRC | 15:38 | |
| *** sree has joined #openstack-infra | 15:38 | |
| dirk | Fungi: http://suse.mobile-central.org might be an alternative, can you check how the bw would compare? | 15:39 |
| fungi | not easily, i'm afraid, but as long as they stay sufficiently updated and don't themselves have frequent consistency issues we could just try switching and see what happens | 15:39 |
| fungi | if another infra-root wants to set up a scratch partial rsync from it on mirror-update01 and collect some stats, that may give us a better idea ahead of time but i'm already fairly booked | 15:41 |
| fungi | and switching it in production is _probably_ safe enough | 15:41 |
| *** ykarel has joined #openstack-infra | 15:42 | |
| *** sree has quit IRC | 15:43 | |
| clarkb | dmsimard: I see the doc compile issue I'm just going to fix it really quickly | 15:45 |
| *** VW has joined #openstack-infra | 15:45 | |
| dmsimard | clarkb: sure | 15:45 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Refactor "emit-ara-html" into a new role: "ara-report" https://review.openstack.org/513874 | 15:45 |
| clarkb | dmsimard: can you check my other comments on patchset 6 though? | 15:45 |
| *** VW__ has quit IRC | 15:45 | |
| dmsimard | clarkb: let me take a quick look | 15:45 |
| *** felipemonteiro_ has quit IRC | 15:46 | |
| *** armaan has joined #openstack-infra | 15:46 | |
| *** eernst has quit IRC | 15:52 | |
| clarkb | dmsimard: doing testing locally I think having extra leading /'s will be fine so file:/// and file://// both work. But it will allow the use of paths that appear to be relative (no leading /) as absolute paths whcih seems weird | 15:52 |
| clarkb | but probably not going to cause many problems so maybe it is fine as is | 15:52 |
| *** eernst has joined #openstack-infra | 15:54 | |
| dmsimard | {{ zuul.executor.work_root }} is not relative, right ? | 15:54 |
| * dmsimard double checks | 15:54 | |
| clarkb | dmsimard: correct, but its also a user settable value so they could put `foo` in there | 15:54 |
| *** r-daneel has quit IRC | 15:54 | |
| dmsimard | yeah, that's why the docs mention the path should be an absolute path, not a relative path | 15:55 |
| dmsimard | because the intent | 15:55 |
| dmsimard | is that, OSA, for example, could use this role to generate a report from their "nested" Ansible playbook runs | 15:55 |
| clarkb | yup I think its a fine intent, but if that is the case we shouldn't have three /'s in the url prefix | 15:55 |
| clarkb | should just be two because the file path being supplied starts with a / of its own | 15:56 |
| *** markvoelker_ has joined #openstack-infra | 15:56 | |
| dmsimard | tbh I never really took the time to understand why the connection string was like sqlite:////tmp/db.sqlite (4 slashes!) | 15:56 |
| *** markvoelker has quit IRC | 15:56 | |
| dmsimard | I just accepted it as a fact of life and went on my merry way | 15:56 |
| dmsimard | let me look up some docs real quick | 15:56 |
| clarkb | dmsimard: I think because in the path portion (so everything after file://) multiple /'s get treated as a single / | 15:57 |
| clarkb | `ls ////` for example | 15:57 |
| dmsimard | http://flask-sqlalchemy.pocoo.org/2.3/config/#connection-uri-format mentions the four leading slashes explicitely | 15:57 |
| *** sree has joined #openstack-infra | 15:57 | |
| dmsimard | "#Unix/Mac (note the four leading slashes)" | 15:58 |
| dmsimard | I never really dug to understand why that is | 15:58 |
| clarkb | huh if its sqlite specific that would be weird but ok | 15:58 |
| dmsimard | maybe zzzeek would know | 15:58 |
| clarkb | I don't think its that important particularly if sqlite says do it this way | 15:59 |
| dmsimard | clarkb: yeah /me shrugs | 15:59 |
| fungi | yeah, i'm not aware of any standard that says file:////foo is treated any differently from file:///fpp | 15:59 |
| clarkb | and it will work with file:// too bceause multiple /'s get trimmed own to a single one | 15:59 |
| clarkb | I was more concerned about someone providing a relative path and having it "work" because it is treated as an absolute path | 15:59 |
| *** gyankum has joined #openstack-infra | 15:59 | |
| *** markvoelker has joined #openstack-infra | 15:59 | |
| clarkb | rather than just failing directly for them | 15:59 |
| dmsimard | fungi, clarkb: It seems specific to the sqlalchemy driver for sqlalchemy/flask | 15:59 |
| fungi | bizarre | 16:00 |
| dmsimard | clarkb: well, I mean, the docs do mention that it's an absolute path and not a relative one | 16:00 |
| pabelanger | Mar 19 15:56:23 review-dev01 puppet-user[7875]: (/Stage[main]/Gerrit/Exec[gerrit-initial-index]/returns) org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /home/gerrit2/review_site/git/All-Users | 16:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add irrelevant-branches negative matcher https://review.openstack.org/552809 | 16:00 |
| *** gyankum has quit IRC | 16:00 | |
| fungi | file:///foo is absolute (url for /foo) | 16:00 |
| clarkb | dmsimard: yes, but if I as a user provide 'foo/bar/sqlite' as the path then that gets appended to file:///foo/bar/sqlite | 16:00 |
| pabelanger | yah, looks like the missing git repos is the reason review-dev01 isn't running puppet correctly | 16:00 |
| fungi | file://foo is relative | 16:00 |
| clarkb | dmsimard: so as a user I provided a relative path but it got used as an absolute path. Which may cause confusion | 16:00 |
| *** markvoelker_ has quit IRC | 16:01 | |
| dmsimard | fungi: yeah, but the sqlite connection string format doesn't seem to follow that convention (it's mentioned explicitely to use 4 slashes here but doesn't say why http://flask-sqlalchemy.pocoo.org/2.3/config/#connection-uri-format ) | 16:01 |
| *** auristor has quit IRC | 16:01 | |
| clarkb | ya if this is a weird case I'm willing to just gowith the docs and not worry about theoretical confusion too much | 16:01 |
| pabelanger | I wonder if we don't start gerrit if we can get by it | 16:01 |
| *** dtruong has joined #openstack-infra | 16:02 | |
| pabelanger | clarkb: fungi: what are your thoughts about making: http://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/init.pp#n724 a little smarter. Either a flag to disable it or first check if git directories exist? | 16:04 |
| dmsimard | clarkb: worst case it's going to fail, I dunno | 16:04 |
| dmsimard | clarkb: more importantly, it's going to fail in something other than the base job | 16:05 |
| *** salv-orlando has joined #openstack-infra | 16:05 | |
| clarkb | dmsimard: ya it should still fail but the error message might confuse people. But I don't think its a major problem if the docs say do it this way | 16:05 |
| zzzeek | dmsimard: the four slashes is because the DB url is "<dbtype://><user>@<host>/dbname" | 16:05 |
| zzzeek | dmsimard: for SQLite you are just giving it "dbname" so it needs the forward slash to know thats where it is in the URL | 16:05 |
| *** sree has quit IRC | 16:05 | |
| dmsimard | zzzeek: oh, that sort of makes sense | 16:05 |
| zzzeek | dmsimard: then within "dbname" we support relative file locations, so to make it absolute, another slash :) | 16:05 |
| dmsimard | and then the fourth slash is the beginning of the path | 16:05 |
| dmsimard | right, okay | 16:05 |
| zzzeek | dmsimard: yes. standard URLs there would be no "relative" vs. "absolute" but I added that in :) | 16:06 |
| dmsimard | zzzeek: thanks for chiming in, keep up the good work :D | 16:06 |
| corvus | (so if you had 3 slashes, it would be a relative path) | 16:06 |
| clarkb | pabelanger: and git repos aren't there because volume isn't there? this is the classic volumes don't work great in many cases problem | 16:06 |
| fungi | interesting (if somewhat confusing) implementation choice | 16:06 |
| fungi | clarkb: well, also git repos aren't there is a problem because database _is_ there | 16:06 |
| zzzeek | fungi: sqlalchemy is all that :) intersting if somewhat confusing choices :) | 16:06 |
| dmsimard | fungi: I suppose it's to keep the interface to all the different drivers as similar as possible? | 16:06 |
| zzzeek | because i am a freak | 16:06 |
| fungi | aren't we all? | 16:06 |
| zzzeek | fungi: so ive been told | 16:07 |
| *** pcaruana has quit IRC | 16:07 | |
| pabelanger | clarkb: right, we could maybe make launch-node.py a little smarter, if we used --mount-path, not to try and format said mount but allow for existing data to be used. Today mount_volume.sh will exit 1 if blkid is already found | 16:08 |
| clarkb | pabelanger: or just use keep to get 90% there, do the volume stuff and take it from there? | 16:09 |
| pabelanger | but, not sure that will work either since gerrit2 user isn't created and uid / gid might differ | 16:09 |
| pabelanger | clarkb: yah, I guess that is how we've done it in the past? | 16:09 |
| clarkb | I think in the past we've done a puppet run without the gerrit puppet run to start | 16:09 |
| clarkb | so its just adding in the root exim rules and firewall basically | 16:09 |
| *** salv-orlando has quit IRC | 16:09 | |
| clarkb | then you can set up volumes and run puppet for the full set. And ya thats a good point about the uids | 16:10 |
| pabelanger | do we create gerrit2 users? | 16:10 |
| *** lpetrut_ has quit IRC | 16:10 | |
| clarkb | puppet should | 16:10 |
| pabelanger | I think adding a flag to skip gerrit-initial then 2 step online might be a good option too | 16:11 |
| pabelanger | then everything minus init should be setup | 16:11 |
| pabelanger | let me try pushing up a patch for that | 16:13 |
| dirk | fungi: so I looked at it from an instance launched in ec2 us-east. ftp.gwdg.de isn't slower than us mirrors (compared it with the linux foundation one, 1m17 vs 1m20s) | 16:13 |
| dirk | so switching mirrors won't help | 16:14 |
| fungi | dirk: thanks for checking | 16:14 |
| dirk | ah, wait, found a different one | 16:15 |
| dirk | mirrors.rit.edu has 50MB/s transfer | 16:15 |
| dirk | that might be a good choice. posting a review.. | 16:15 |
| clarkb | dmsimard: I've +2'd the latest patchset | 16:17 |
| *** salv-orlando has joined #openstack-infra | 16:18 | |
| *** mriedem is now known as mriedem_away | 16:19 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Switch opensuse mirroring to mirrors.rit.edu https://review.openstack.org/554278 | 16:20 |
| dirk | fungi: frickler : ^^ | 16:20 |
| smcginnis | Just gotta share this one - https://www.bustedtees.com/t-shirt/thereisnoprincess | 16:21 |
| fungi | dirk: good find! | 16:22 |
| *** VW has quit IRC | 16:22 | |
| fungi | dirk: and supports rsync? | 16:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-gerrit master: Allow user to toggle gerrit-initial-init https://review.openstack.org/554279 | 16:22 |
| *** VW has joined #openstack-infra | 16:22 | |
| *** yamamoto has quit IRC | 16:23 | |
| *** VW has quit IRC | 16:23 | |
| *** VW has joined #openstack-infra | 16:24 | |
| AJaeger | infra-root, we still seem to have some race/bug (or zuul disabled?) with new project creation. See https://review.openstack.org/#/c/546683/ and https://review.openstack.org/#/c/552735/ - the repos have been created but zuul is not happy ;( | 16:24 |
| clarkb | smcginnis: I think rbergeron got corvus a copy of that tshirt | 16:24 |
| smcginnis | clarkb: Nice! | 16:25 |
| clarkb | AJaeger: :/ I guess I'll dig into that more today since I precipitated the first case of that | 16:25 |
| AJaeger | also, github sync is broken - https://github.com/openstack/xstatic-filesaver , https://github.com/openstack/xstatic-filesaver https://github.com/openstack/xstatic-filesaver are not set up | 16:25 |
| AJaeger | https://github.com/openstack/xstatic-vis is the third one | 16:25 |
| AJaeger | clarkb: thanks | 16:25 |
| clarkb | in a meeting now but will look into it after | 16:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Update README and add GPL license https://review.openstack.org/554280 | 16:26 |
| AJaeger | note https://git.openstack.org/cgit//openstack/xstatic-vis exists | 16:26 |
| *** andreas_s has quit IRC | 16:27 | |
| *** r-daneel has joined #openstack-infra | 16:27 | |
| AJaeger | mordred: are you around today? | 16:27 |
| dirk | fungi: yes, I tested via rsync | 16:28 |
| *** andreas_s has joined #openstack-infra | 16:28 | |
| dirk | smcginnis: https://www.redbubble.com/de/shop/zuul+t-shirts | 16:28 |
| *** VW has quit IRC | 16:28 | |
| corvus | smcginnis: i can confirm that t-shirt is awesome. | 16:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add gerrit_initial_init setting to gerrit https://review.openstack.org/554282 | 16:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Disable gerrit_initial_init on review-dev01.o.o https://review.openstack.org/554283 | 16:28 |
| pabelanger | clarkb: something like ^ for 2 step gerrit initialize | 16:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add gerrit_initial_init setting to gerrit https://review.openstack.org/554282 | 16:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Disable gerrit_initial_init on review-dev01.o.o https://review.openstack.org/554283 | 16:30 |
| smcginnis | dirk: Wow, that's pretty impressive. | 16:31 |
| *** dalgaaf_ has joined #openstack-infra | 16:32 | |
| *** samthetechieaa has joined #openstack-infra | 16:35 | |
| *** samthetechieaa has quit IRC | 16:35 | |
| mordred | AJaeger: yes! still booting my brain into a useable state and catching up on things | 16:35 |
| AJaeger | mordred: looking at https://review.openstack.org/554222, I wonder whether the problem comes from the way the tox_install.sh sets up constraints - and we need https://review.openstack.org/552865 to merge first? | 16:37 |
| *** rwsu has joined #openstack-infra | 16:38 | |
| *** dsariel has quit IRC | 16:38 | |
| AJaeger | mordred: thanks. I'm really puzzled with these and need help. | 16:38 |
| *** VW has joined #openstack-infra | 16:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/554132 | 16:39 |
| *** VW_ has joined #openstack-infra | 16:39 | |
| *** VW_ has joined #openstack-infra | 16:39 | |
| *** camunoz has joined #openstack-infra | 16:40 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 16:43 |
| *** VW has quit IRC | 16:43 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/543403 | 16:43 |
| *** jpena is now known as jpena|brb | 16:43 | |
| *** trown|ruck is now known as trown|lunch | 16:44 | |
| *** andreas_s has quit IRC | 16:45 | |
| *** armaan has quit IRC | 16:46 | |
| mordred | AJaeger: I'm not sure I fully grok what the problem is from that - what is not working? | 16:47 |
| *** VW_ has quit IRC | 16:47 | |
| fungi | config-core: https://review.openstack.org/551406 "Replace Queens signing key with Rocky" should be safe to merge today assuming it looks okay | 16:48 |
| *** VW has joined #openstack-infra | 16:48 | |
| dmsimard | That reminds me I /still/ haven't uploaded my key, I'll do that now. | 16:48 |
| *** myoung|rover is now known as myoung|food | 16:48 | |
| *** VW has quit IRC | 16:48 | |
| *** e0ne has quit IRC | 16:48 | |
| *** VW has joined #openstack-infra | 16:49 | |
| AJaeger | mordred: check the version of neutron installed in 554222 - it's not from git but the 12.0.0 from pypi | 16:49 |
| *** gyee has joined #openstack-infra | 16:49 | |
| AJaeger | mordred: my guess: the tools/tox_install*sh setup ;( | 16:50 |
| *** shardy_mtg has quit IRC | 16:52 | |
| *** hemna_ has joined #openstack-infra | 16:53 | |
| AJaeger | mordred: testing now in https://review.openstack.org/554292 | 16:53 |
| AJaeger | boden ^ | 16:53 |
| mordred | AJaeger: where are the required-projects for that being defined? | 16:53 |
| boden | AJaeger ack thanks | 16:54 |
| AJaeger | mordred: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n16605 | 16:54 |
| clarkb | AJaeger: did the original problem project get unstuck? | 16:54 |
| AJaeger | clarkb: no | 16:55 |
| *** jamesmcarthur has joined #openstack-infra | 16:55 | |
| AJaeger | clarkb: see https://review.openstack.org/#/c/546683/ | 16:55 |
| AJaeger | that was the first one | 16:55 |
| mordred | AJaeger: gotcha.so - fwiw, I only see neutron in the required-projects list for the docs job | 16:55 |
| AJaeger | mordred: http://logs.openstack.org/22/554222/2/check/openstack-tox-pep8/700407f/job-output.txt.gz#_2018-03-19_13_40_21_391523 is the list of required-projects | 16:56 |
| AJaeger | mordred: I'm checking tox-pep8 | 16:56 |
| AJaeger | boden: which job are you looking at? | 16:56 |
| mordred | I do not understand why neutron is in that list | 16:56 |
| AJaeger | mordred: openstack-python-jobs-neutron puts neutron in the list | 16:57 |
| mordred | ok. awesome. that at least explains that | 16:57 |
| boden | AJaeger: can you clarify your question pls? what job are you referring to | 16:57 |
| AJaeger | mordred: here everything is fine http://logs.openstack.org/22/554222/2/check/openstack-tox-pep8/700407f/job-output.txt.gz#_2018-03-19_13_46_32_360186 | 16:57 |
| clarkb | AJaeger: thanks. Reading the puppet logs on zuul01 we definitely update the main.yaml before the zuul reload is triggered. So I'm thinking either the project's git repo isn't being created before the zuul reload or the zuul reload is broken | 16:57 |
| AJaeger | boden: which job exactly is failing for you - where did you notice the problem? | 16:57 |
| *** jcoufal has joined #openstack-infra | 16:58 | |
| boden | AJaeger I’m testing with https://review.openstack.org/#/c/554245/ I think the commit msg describes its intent | 16:59 |
| boden | and its failing http://logs.openstack.org/45/554245/1/check/openstack-tox-py27/d71b20a/testr_results.html.gz | 17:00 |
| *** jcoufal_ has quit IRC | 17:00 | |
| clarkb | http://paste.openstack.org/show/704731/ from the jeepyb log shows why some of those projects are not created on github | 17:00 |
| clarkb | and jeepyb log ends before puppet log starts so we should create the git repos in gerrit before reloading zuul | 17:01 |
| corvus | clarkb: "Full reconfiguration" is the keyword to look for in the log | 17:01 |
| AJaeger | mordred: http://logs.openstack.org/22/554222/2/check/openstack-tox-pep8/700407f/job-output.txt.gz#_2018-03-19_13_47_11_511169 shows that neutron is at 12.0.0 again ;( | 17:01 |
| clarkb | corvus: thanks | 17:02 |
| mordred | AJaeger: yah. I see that now | 17:02 |
| mordred | also, http://logs.openstack.org/22/554222/2/check/openstack-tox-pep8/700407f/tox/pep8-5.log fwiw | 17:02 |
| corvus | clarkb: it should always come in pairs (beginning / complete) | 17:03 |
| corvus | unless there is an error | 17:03 |
| AJaeger | mordred: yep | 17:03 |
| mordred | AJaeger: oh- I think ... gah. | 17:04 |
| *** iyamahat has quit IRC | 17:04 | |
| clarkb | corvus: 2018-03-19 05:59:39,411 INFO zuul.Scheduler: Full reconfiguration beginning it definitely started but I don't see the matching complete | 17:04 |
| corvus | clarkb: we may want to look for the next exception after that line | 17:04 |
| *** weshay is now known as weshay_brb | 17:05 | |
| *** yamahata has quit IRC | 17:05 | |
| pabelanger | corvus: clarkb: do you happen to know how we fix rspec tests with zuul-cloner? puppet-gerrit is failing: http://logs.openstack.org/79/554279/1/check/legacy-puppet-beaker-rspec-infra/37f6f7c/job-output.txt.gz#_2018-03-19_16_40_19_740767 | 17:05 |
| mordred | AJaeger: so, when we install siblings, we create a pip install command that has all of the sibling repos in it - but it's still pip and is still going to do things in order, which means the transitive depend on neutron from the other networking modules is causing pip to believe it has already installed neutron in that installation | 17:05 |
| *** dtruong has quit IRC | 17:05 | |
| clarkb | Unknown project openstack-infra/puppet-apps_site | 17:06 |
| mordred | AJaeger: I'm not sure what the fix is yet - but I believe I should be able to reproduce the issue | 17:06 |
| clarkb | ok so another change that was approved on friday to deprecate puppet-apps_site must've been incomplete | 17:06 |
| clarkb | I'll push up a fix for ^ then we can try another reload | 17:06 |
| *** shardy_mtg has joined #openstack-infra | 17:08 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Remove puppet-apps_site https://review.openstack.org/554295 | 17:09 |
| clarkb | AJaeger: corvus ^ | 17:09 |
| AJaeger | clarkb: argh, thanks | 17:10 |
| clarkb | now to see why github is unhappy | 17:10 |
| clarkb | AJaeger: we should make sure the retiring doc calls out removing the project from all the places | 17:12 |
| *** lpetrut has joined #openstack-infra | 17:12 | |
| AJaeger | clarkb: yeah ;/ the person retiring was me ;( sorry | 17:12 |
| *** zoli is now known as zoli|gone | 17:13 | |
| *** zoli|gone is now known as zoli | 17:13 | |
| clarkb | AJaeger: it happens, new system to learn and all | 17:13 |
| mordred | AJaeger: aha! I think the problem is actually different | 17:14 |
| mordred | AJaeger: and I think much more easily fixed | 17:14 |
| mordred | AJaeger: basically, in siblings we uninstall the packages that were found to be a sibling, then we install the siblings all in one go | 17:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-gerrit master: Allow ZUUL_REF to be empty for rspec testing https://review.openstack.org/554296 | 17:15 |
| mordred | AJaeger: thatmeans that neutron gets uninstalled (the one that was installed originally by tox pip) - then when we say "pip install ../networking-l2gw ../neutron" - pip finds the neutron depend in networking-l2gw first and installs it, then since it's the same command it doesn't install neutron a second time since it's already installed | 17:15 |
| mordred | AJaeger: instead, I think what we should do is do a loop of "pip uninstall $sibling ; pip install ../$sibling_dir" - which will ensure correct sequencing wrt uninstall/install of sibling packages | 17:16 |
| AJaeger | oh, intersting | 17:17 |
| *** iyamahat has joined #openstack-infra | 17:17 | |
| clarkb | raeding the github api docs I think this edit should work. I wonder if there is a race between creating the project and being able to edit it | 17:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-gerrit master: Allow user to toggle gerrit-initial-init https://review.openstack.org/554279 | 17:19 |
| *** d0ugal has quit IRC | 17:21 | |
| clarkb | AJaeger: was there one you found that wasn't created in github at all? The one I am spot checking is created it just didn't get edited with the project details | 17:24 |
| clarkb | AJaeger: I see all three xstatic repos in github | 17:25 |
| *** shardy_mtg has quit IRC | 17:26 | |
| AJaeger | clarkb: all fixed now... | 17:27 |
| AJaeger | just double checked | 17:27 |
| clarkb | AJaeger: cool. I think that leaves us with a zuul reload once my fix merges | 17:27 |
| AJaeger | clarkb: indeed | 17:27 |
| *** auristor has joined #openstack-infra | 17:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Uninstall and reinstall siblings one at a time https://review.openstack.org/554297 | 17:28 |
| dmsimard | I'm going to grab a bite and then I'll start working on the limestone mirror | 17:28 |
| mordred | AJaeger: ^^ I think that should fix it | 17:28 |
| AJaeger | clarkb: we could merge https://review.openstack.org/#/c/552861 for the reload | 17:29 |
| mordred | AJaeger: but, obviously, since it's a changeto tox-siblings, we should heavily depends-on it | 17:29 |
| clarkb | AJaeger: I'll take a look | 17:29 |
| AJaeger | mordred: yes, please add some tests... | 17:29 |
| AJaeger | mordred: use boden's 554245 as one example | 17:30 |
| *** pcichy has quit IRC | 17:31 | |
| AJaeger | mordred: change does what you describe, hope it works fine;) | 17:32 |
| AJaeger | mordred: check https://review.openstack.org/#/q/topic:tox-siblings for some test repos to push a change for | 17:32 |
| *** esberglu_ has joined #openstack-infra | 17:32 | |
| *** eharney has quit IRC | 17:33 | |
| *** eernst has quit IRC | 17:33 | |
| mordred | AJaeger: ++ | 17:33 |
| *** eernst has joined #openstack-infra | 17:33 | |
| *** eernst has quit IRC | 17:33 | |
| *** eernst has joined #openstack-infra | 17:33 | |
| mordred | AJaeger: remote: https://review.openstack.org/554300 DNM Testing siblings <-- there's one to start with | 17:35 |
| mordred | AJaeger: I'll see how that goes before adding any more | 17:35 |
| AJaeger | mordred: thanks! | 17:35 |
| *** esberglu has quit IRC | 17:35 | |
| AJaeger | mordred: merge conflict? | 17:36 |
| AJaeger | mordred: also,wrong depends-on | 17:36 |
| mordred | AJaeger: boo | 17:37 |
| *** jpena|brb is now known as jpena | 17:37 | |
| AJaeger | you want https://review.openstack.org/554297 instaed of 292 | 17:37 |
| *** mriedem_away is now known as mriedem | 17:37 | |
| *** jbadiapa_ has joined #openstack-infra | 17:37 | |
| *** ykarel has quit IRC | 17:37 | |
| mordred | AJaeger: thanks | 17:38 |
| *** jbadiapa has quit IRC | 17:40 | |
| *** eernst has quit IRC | 17:43 | |
| *** eernst has joined #openstack-infra | 17:43 | |
| clarkb | any other config-core willing to review https://review.openstack.org/#/c/513874/ ? I think it is ready at this point and is the next step in testing the ara middleware | 17:43 |
| *** felipemonteiro_ has joined #openstack-infra | 17:43 | |
| *** VW_ has joined #openstack-infra | 17:43 | |
| AJaeger | bbl | 17:43 |
| *** VW_ has quit IRC | 17:44 | |
| *** VW_ has joined #openstack-infra | 17:44 | |
| *** eernst has quit IRC | 17:45 | |
| openstackgerrit | Merged openstack-infra/zuul master: Update README and add GPL license https://review.openstack.org/554280 | 17:46 |
| *** eernst has joined #openstack-infra | 17:46 | |
| *** VW has quit IRC | 17:46 | |
| *** eharney has joined #openstack-infra | 17:47 | |
| *** jamesdenton has joined #openstack-infra | 17:49 | |
| *** weshay_brb is now known as weshay | 17:50 | |
| pabelanger | clarkb: +2, if you want to approve and watch | 17:51 |
| clarkb | pabelanger: up to dmsimard probably | 17:52 |
| clarkb | (I just think it is valuable to be able toget ara on all the jobs again so wanted to help push it along) | 17:52 |
| clarkb | pabelanger: I'm looking at your rspec test fail now | 17:52 |
| pabelanger | while puppet-gerrit is waiting to test, I have a new mirror in vexxhost going but using v2 flavors. We can then see if that helps speed up jobs | 17:52 |
| dmsimard | That patch is just a role, it's noop and is safe to merge | 17:52 |
| pabelanger | dmsimard: k, feel free to +3, there are 2 +2 | 17:53 |
| dmsimard | So I'll +W it and then we can control the base-test update | 17:53 |
| clarkb | pabelanger: the z-c thing I want to say we just stopped supplying a zuul ref entirely? | 17:53 |
| dmsimard | I'll get limestone up and then we can land the base-test update | 17:53 |
| clarkb | pabelanger: so that might be a job update? fungi did a lot of the initial work in correcting that behavior I think | 17:53 |
| pabelanger | yah, I'm wondering if it might have failed on another project before | 17:54 |
| pabelanger | but couldn't see anything | 17:54 |
| dmsimard | clarkb, pabelanger: thanks for the reviews! | 17:54 |
| *** jcoufal has quit IRC | 17:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/550978 | 17:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/projects routes https://review.openstack.org/550979 | 17:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 17:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Rename javascript package to @zuul-ci/dashboard https://review.openstack.org/551999 | 17:55 |
| fungi | clarkb: pabelanger: oh, for using z-c in non-change-oriented pipelines? | 17:55 |
| fungi | i have the vaguest of memories on hacking through some of that | 17:55 |
| clarkb | fungi: in this case it is change oriented but it is passing --zuul-ref to z-c without an argument and failing | 17:56 |
| *** jcoufal has joined #openstack-infra | 17:56 | |
| clarkb | fungi: http://logs.openstack.org/79/554279/1/check/legacy-puppet-beaker-rspec-infra/37f6f7c/job-output.txt.gz#_2018-03-19_16_40_19_740767 | 17:56 |
| pabelanger | clarkb: fungi: I've proposed https://review.openstack.org/554296/ however unsure if valid syntax | 17:56 |
| pabelanger | waiting for test results | 17:56 |
| clarkb | oh ya I seem to recall fixing some of these related items in the common rspec lib thing we have | 17:57 |
| clarkb | pabelanger: I think most projects use the common rspec helper acceptance file? | 17:57 |
| clarkb | so the real fix may be to switch to that in puppet-gerrit? | 17:57 |
| *** myoung|food is now known as myoung|rover | 17:58 | |
| *** germs has joined #openstack-infra | 17:58 | |
| *** germs has quit IRC | 17:58 | |
| *** germs has joined #openstack-infra | 17:58 | |
| *** lpetrut has quit IRC | 17:59 | |
| pabelanger | clarkb: sure, remember where you fixed it? | 18:00 |
| clarkb | pabelanger: in openstack-infra/puppet-openstack_infra_spec_helper which is supposed to be used by all the projects so we don't have to fix them individually aiui | 18:01 |
| clarkb | pabelanger: and that gets pulled in via the Gemfile | 18:01 |
| *** VW_ is now known as VW | 18:03 | |
| pabelanger | k | 18:03 |
| pabelanger | let me see why puppet-gerrit didn't do that | 18:03 |
| *** germs has quit IRC | 18:03 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove puppet-apps_site https://review.openstack.org/554295 | 18:04 |
| clarkb | fungi: re discussion in #zuul https://docs.openstack.org/infra/system-config/stackforge.html the terminology that our docs uses is still "stackforge". I remember being -2'd when I tried to change it at one point too. If we want to move away from that as the official term maybe we need a doc update? | 18:04 |
| clarkb | pabelanger: there may even be a change open for it that just hasn't been reviewd :/ | 18:04 |
| *** jbadiapa_ is now known as jbadiapa | 18:05 | |
| fungi | clarkb: the only references to "stackforge" in at this point governance are in old resolutions | 18:06 |
| *** myoung|rover is now known as myoung|taxi | 18:06 | |
| *** lpetrut has joined #openstack-infra | 18:06 | |
| *** myoung|taxi is now known as myoung|biab | 18:06 | |
| *** iyamahat has quit IRC | 18:06 | |
| clarkb | fungi: sounds like I should update our docs then. | 18:07 |
| *** tesseract has quit IRC | 18:07 | |
| clarkb | fungi: "unofficial projects" being the new terminology? | 18:07 |
| fungi | clarkb: see https://governance.openstack.org/tc/resolutions/20160119-stackforge-retirement.html | 18:08 |
| fungi | This updated resolution reflects the TC’s revised intention to retire not only the “stackforge” namespace but also the term itself in favor of “unofficial project”. | 18:08 |
| clarkb | cool, thanks | 18:09 |
| *** dtantsur is now known as dtantsur|afk | 18:09 | |
| clarkb | AJaeger: I have approved the ansible systemd project additions | 18:10 |
| clarkb | AJaeger: so we should see zuul get reloaded sometime soonish | 18:10 |
| AJaeger | clarkb, ah - just wanted to ask about approving https://review.openstack.org/#/c/550491/ ;) | 18:10 |
| AJaeger | clarkb: let's do that later if everything worked | 18:10 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Refactor "emit-ara-html" into a new role: "ara-report" https://review.openstack.org/513874 | 18:10 |
| clarkb | AJaeger: I didn't want to do it until the puppet-apps_site change merged to avoid racing | 18:10 |
| *** dhill__ has joined #openstack-infra | 18:11 | |
| AJaeger | bnemec: I'm confused you gave +1 on https://review.openstack.org/#/c/550491/ but -1 on the governance change - I would expect the same vote on both. | 18:11 |
| AJaeger | bnemec: Shall we really merge 550491 | 18:12 |
| bnemec | AJaeger: I left a comment on https://review.openstack.org/#/c/550496 too | 18:12 |
| AJaeger | clarkb: yes, exactly. | 18:12 |
| *** eharney has quit IRC | 18:12 | |
| *** sambetts is now known as sambetts|afk | 18:12 | |
| bnemec | I think we want it tagged as follows-policy. | 18:12 |
| *** dhill__ has left #openstack-infra | 18:12 | |
| bnemec | But I'm +1 on adding the library. | 18:12 |
| AJaeger | bnemec: thanks for explanation - then we're fine... | 18:12 |
| AJaeger | bnemec, clarkb, then I'll +2 550491 and we can merge once zuul is really happy | 18:13 |
| *** esberglu_ is now known as esberglu | 18:16 | |
| vdrok | folks, seems like setuptools 39 breaks our unittests (and I guess not only them) http://logs.openstack.org/69/543569/4/check/openstack-tox-py35/c706b82/job-output.txt | 18:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Import systemd_mount and systemd_service to OSA https://review.openstack.org/552861 | 18:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Use unix sockets instead of TCP sockets https://review.openstack.org/542469 | 18:19 |
| fungi | vdrok: wow, that's an impressively long logfile | 18:19 |
| AJaeger | mordred: http://logs.openstack.org/00/554300/2/check/openstack-tox-pep8/1aad490/ looks wrong ;( | 18:19 |
| fungi | oh, nevermind, my browser was just being slow i think | 18:19 |
| vdrok | fungi: sorry, the exact line is http://logs.openstack.org/69/543569/4/check/openstack-tox-py35/c706b82/job-output.txt#_2018-03-19_16_23_53_920564 :) | 18:20 |
| AJaeger | mordred: "neutron==12.0.0" | 18:20 |
| AJaeger | in http://logs.openstack.org/00/554300/2/check/openstack-tox-pep8/1aad490/tox/pep8-3.log | 18:20 |
| vdrok | downgrading to latest 38 helps | 18:20 |
| AJaeger | mordred: your siblings change is broken, see http://logs.openstack.org/00/554300/2/check/openstack-tox-pep8/1aad490/job-output.txt.gz#_2018-03-19_18_08_33_710877 | 18:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-gerrit master: Allow user to toggle gerrit-initial-init https://review.openstack.org/554279 | 18:21 |
| AJaeger | mordred: "AttributeError: 'str' object has no attribute 'name'\n", | 18:21 |
| vdrok | my guess is this commit does something wrong https://github.com/pypa/setuptools/commit/929acc4e551448a68411968fb50336ad51ed4d3c | 18:21 |
| fungi | vdrok: have you been discussing it in #openstack-oslo yet? looks like it's oslo.utils raising the exeption there | 18:22 |
| vdrok | fungi: nope not yet | 18:22 |
| *** iyamahat has joined #openstack-infra | 18:23 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Properly deprecate stackforge https://review.openstack.org/554312 | 18:23 |
| clarkb | fungi: ^ fyi | 18:24 |
| clarkb | now to respond to electrofelix | 18:24 |
| openstackgerrit | Merged openstack-infra/project-config master: Change base-test to use "ara-report" role instead of "emit-ara-html" https://review.openstack.org/553659 | 18:24 |
| *** iyamahat has quit IRC | 18:24 | |
| *** iyamahat has joined #openstack-infra | 18:24 | |
| AJaeger | dmsimard: happy testing ^ | 18:24 |
| *** trown|lunch is now known as trown | 18:25 | |
| *** trown is now known as trown|compassing | 18:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add volume-size flag to launch-node https://review.openstack.org/554313 | 18:26 |
| *** eharney has joined #openstack-infra | 18:26 | |
| dmsimard | AJaeger: ty | 18:26 |
| pabelanger | infra-root: ^adds --volume-size flag to be used with --boot-from-volume flag, which is needed for vexxhost control plane servers now. V2 flavors will be using that | 18:27 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Do not merge: Re-parent integration jobs to base-test https://review.openstack.org/553663 | 18:27 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Start log receiver in executor https://review.openstack.org/546304 | 18:27 |
| clarkb | pabelanger: v2 flavors can only boot from volume I take it? | 18:27 |
| *** dave-mccowan has joined #openstack-infra | 18:28 | |
| pabelanger | opps, 1 issue | 18:28 |
| pabelanger | clarkb: yah | 18:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add volume-size flag to launch-node https://review.openstack.org/554313 | 18:29 |
| *** rlandy is now known as rlandy|brb | 18:31 | |
| *** armaan has joined #openstack-infra | 18:31 | |
| pabelanger | clarkb: fungi: I'm getting invalid credentials when setting up rackdns: http://paste.openstack.org/show/704865/ any ideas? | 18:31 |
| clarkb | pabelanger: yes the credentials were updated a little while back. I should've sent email to all rooters detailing the update | 18:31 |
| clarkb | pabelanger: tldr check the passwords file | 18:32 |
| *** wolverineav has joined #openstack-infra | 18:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove empty sections from docs https://review.openstack.org/554315 | 18:32 |
| pabelanger | ah, so our scripts likely have old ones | 18:32 |
| pabelanger | kk, let me see if I can figure it out | 18:32 |
| clarkb | oh hrm I thought I updated them? but maybe I missed some? | 18:32 |
| *** felipemonteiro__ has joined #openstack-infra | 18:32 | |
| clarkb | but ya new creds should be in file. Definitely possible an rc file or two need updating | 18:32 |
| pabelanger | maybe? | 18:32 |
| *** rkukura has joined #openstack-infra | 18:33 | |
| clarkb | oh you know what | 18:33 |
| *** wolverineav has quit IRC | 18:33 | |
| clarkb | I bet I checked clouds.yaml and saw we didn't use them | 18:33 |
| clarkb | but thats because dns isn't openstack | 18:33 |
| *** wolverineav has joined #openstack-infra | 18:33 | |
| clarkb | and I neglected to update the dns specfici rc file | 18:33 |
| pabelanger | ah | 18:33 |
| *** e0ne has joined #openstack-infra | 18:36 | |
| *** felipemonteiro_ has quit IRC | 18:37 | |
| pabelanger | okay, updated | 18:37 |
| clarkb | sorry about that | 18:37 |
| pabelanger | http://mirror02.ca-ymq-1.vexxhost.openstack.org/ | 18:37 |
| pabelanger | clarkb: np! | 18:37 |
| corvus | pabelanger: what provider is review-dev01 in? | 18:39 |
| pabelanger | corvus: rackspace | 18:39 |
| pabelanger | 104.239.149.179 is the IP, but I haven't updated dns yet | 18:40 |
| *** eernst has quit IRC | 18:40 | |
| corvus | pabelanger: it has 2 problems: 1) no reverse dns. 2) http://paste.openstack.org/show/704884/ | 18:40 |
| *** eernst has joined #openstack-infra | 18:40 | |
| corvus | pabelanger: so we need to talk to rackspace to get that ip whitelisted or something | 18:40 |
| corvus | pabelanger: unless google uses the spamhaus PBL? i dunno. | 18:41 |
| *** eernst has quit IRC | 18:41 | |
| *** Goneri has quit IRC | 18:41 | |
| *** eernst has joined #openstack-infra | 18:41 | |
| pabelanger | corvus: oh, hmm. I was going to delete it and try launch it again once puppet-gerrit patches land to try a 2 step initialization of gerrit: https://review.openstack.org/554283/ | 18:41 |
| corvus | stackoverflow says they *do* use the pbl | 18:42 |
| pabelanger | I can delete it now if spam is getting generated | 18:42 |
| corvus | so probably just need to fixup reverse dns and do the pbl. | 18:42 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Temporarily make legacy-tempest-dsvm-neutron-src NV https://review.openstack.org/554318 | 18:42 |
| corvus | pabelanger: if you're going to delete it anyway, yeah, sooner would be better for me :) | 18:42 |
| *** eernst has quit IRC | 18:42 | |
| pabelanger | k, will do that now | 18:42 |
| *** eernst_ has joined #openstack-infra | 18:42 | |
| pabelanger | sorry for the noise | 18:43 |
| pabelanger | corvus: deleted | 18:43 |
| *** iyamahat_ has joined #openstack-infra | 18:45 | |
| *** iyamahat has quit IRC | 18:45 | |
| *** yamahata has joined #openstack-infra | 18:46 | |
| fungi | i also keep meaning to mention, the zk servers seem to be sending cronspam without an fqdn, and that's getting rejected by some of our sysadmin e-mail addresses (so i'm also seeing the forwarded bounces) | 18:48 |
| *** jamesmcarthur has quit IRC | 18:48 | |
| *** openstackgerrit has quit IRC | 18:48 | |
| *** openstackgerrit has joined #openstack-infra | 18:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add mirror02.ca-ymq-1.vexxhost.openstack.org to cacti.o.o https://review.openstack.org/554321 | 18:49 |
| pabelanger | fungi: yah, that is a result of me updating /etc/hosts by hand. we talked a while back about adding back the proper host info, but haven't done so yet | 18:50 |
| pabelanger | I can look today | 18:50 |
| fungi | i don't think there's any hurry, i just keep seeing and deleting mis-filed cronspam e-mails and bounces for those every day and forgetting to bring it up | 18:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Bring vexxhost back online for testing https://review.openstack.org/554322 | 18:51 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Revert "Temporarily make legacy-tempest-dsvm-neutron-src NV" https://review.openstack.org/554323 | 18:51 |
| pabelanger | fungi: clarkb: mnaser: ^bring 10 nodes in vexxhost online to test new mirror | 18:51 |
| smcginnis | I need to make a test non-voting to unblock some requirements shuffling. | 18:52 |
| rosmaita | i have a job configuration question when someone has a minute ... i don't understand why neutron-grenade ran a check on https://review.openstack.org/#/c/358946/7 given this line in project-config: https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L6608 | 18:52 |
| smcginnis | Any help on the infra job changes here would be appreciated: https://review.openstack.org/#/q/status:open+topic:pip/5081 | 18:52 |
| smcginnis | rosmaita: I've seen that elsewhere. | 18:53 |
| mnaser | +2'd | 18:53 |
| smcginnis | rosmaita: I'm still not clear on when and where you can override what has already been defined elsewhere, and I think that's actually changed a couple times recently. | 18:54 |
| mnaser | actually | 18:54 |
| mnaser | pabelanger: left a comment | 18:55 |
| pabelanger | mnaser: sure | 18:57 |
| rosmaita | smcginnis: thanks | 18:57 |
| *** VW has quit IRC | 18:58 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Uninstall and reinstall siblings one at a time https://review.openstack.org/554297 | 18:58 |
| AJaeger | mordred: hope this fixes the problems ^ | 18:58 |
| smcginnis | rosmaita: Not really too helpful, but there's my 2 cents. ;) | 18:58 |
| smcginnis | rosmaita: Hopefully someone else here knows better than me. | 18:58 |
| fungi | smcginnis: sorry to do this to you, but see my comment on 554318 | 18:58 |
| *** VW has joined #openstack-infra | 18:58 | |
| fungi | happy to help expedite it though | 18:58 |
| *** krtaylor has joined #openstack-infra | 18:59 | |
| *** VW has quit IRC | 19:00 | |
| *** VW has joined #openstack-infra | 19:00 | |
| *** VW has quit IRC | 19:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Bring vexxhost back online for testing https://review.openstack.org/554322 | 19:01 |
| AJaeger | evrardjp: want to rebase https://review.openstack.org/552867 ? | 19:02 |
| pabelanger | mnaser: updated | 19:02 |
| *** VW has joined #openstack-infra | 19:02 | |
| evrardjp | AJaeger: will do | 19:02 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add systemd roles base jobs https://review.openstack.org/552867 | 19:02 |
| evrardjp | oh, no rebase conflict | 19:02 |
| evrardjp | I expected worse | 19:02 |
| AJaeger | clarkb: the new repos are created, I rechecked - and seems to look fine now (no immediate -1 from zuul) | 19:03 |
| *** dhill__ has joined #openstack-infra | 19:03 | |
| *** dhill__ is now known as amanjo | 19:03 | |
| clarkb | AJaeger: ya I see it just started a full reconfiguration | 19:03 |
| AJaeger | clarkb: great | 19:03 |
| clarkb | AJaeger: I don't see it syaing the reconfiguration is ended yet but looking good | 19:03 |
| *** pcichy has joined #openstack-infra | 19:05 | |
| openstackgerrit | megan guiney proposed openstack-infra/subunit2sql master: Add delete by uuid functions https://review.openstack.org/537775 | 19:05 |
| smcginnis | fungi: No problem. So you are saying with the check job being non-voting, you would rather it just be removed entirely from gate, correct? | 19:06 |
| fungi | smcginnis: yeah, and since it's the only gate job in that template, you can just remove the whole gate section | 19:06 |
| smcginnis | fungi: OK, update coming. | 19:07 |
| fungi | i'm on hand to review asap | 19:07 |
| clarkb | ok confirmed via logs that the reconfig completed | 19:08 |
| clarkb | so that should be the only current issue with the zuul configs fixed | 19:08 |
| *** myoung|biab is now known as myoung|rover | 19:08 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Temporarily make legacy-tempest-dsvm-neutron-src NV https://review.openstack.org/554318 | 19:09 |
| *** gfidente has quit IRC | 19:09 | |
| *** eernst_ has quit IRC | 19:11 | |
| fungi | smcginnis: lgtm, though you'll (less urgently) want to update the revert change too | 19:12 |
| *** rlandy|brb is now known as rlandy | 19:12 | |
| smcginnis | fungi: Yep, I'll get that one updated shortly. | 19:12 |
| fungi | config-core: can we get a second set of eyes on 554318 there to help un-wedge an urgent transition? | 19:12 |
| clarkb | fungi: on it | 19:12 |
| fungi | thanks! | 19:12 |
| smcginnis | fungi, clarkb: Thanks! | 19:13 |
| clarkb | done | 19:13 |
| *** ykarel has joined #openstack-infra | 19:13 | |
| AJaeger | clarkb: thanks | 19:15 |
| AJaeger | clarkb: let's merge the next new repo then ;) | 19:15 |
| *** jpena is now known as jpena|off | 19:15 | |
| fungi | smcginnis: clarkb: i've punted 554318 directly into the gate to get it into effect sooner | 19:16 |
| smcginnis | Awesome, thanks fungi. | 19:16 |
| *** amanjo has quit IRC | 19:18 | |
| *** dhill_ has quit IRC | 19:18 | |
| *** gary_perkins has quit IRC | 19:18 | |
| *** dhill__ has joined #openstack-infra | 19:19 | |
| *** pickle has joined #openstack-infra | 19:20 | |
| *** pcichy has quit IRC | 19:20 | |
| *** florianf has quit IRC | 19:21 | |
| *** pickle has quit IRC | 19:22 | |
| Shrews | corvus: would it be safe to move writeLoggingConfig() from _execute() to before startLogReceiver() in your log streaming change (line 623 here https://review.openstack.org/#/c/546304/3/zuul/executor/server.py)? | 19:22 |
| Shrews | gah, wrong channel | 19:22 |
| clarkb | fungi: is this setuptools issue the same one from early this morng (PDT)? | 19:22 |
| *** felipemonteiro_ has joined #openstack-infra | 19:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create new project for oslo.limit https://review.openstack.org/550491 | 19:24 |
| *** felipemonteiro_ has quit IRC | 19:25 | |
| AJaeger | clarkb: the three repo batches are all fine now - Zuul is aware of them. config-core, https://review.openstack.org/552735 , https://review.openstack.org/546683 , and https://review.openstack.org/552867 are ready for merge now. | 19:25 |
| *** felipemonteiro_ has joined #openstack-infra | 19:25 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Temporarily make legacy-tempest-dsvm-neutron-src NV https://review.openstack.org/554318 | 19:25 |
| *** gary_perkins has joined #openstack-infra | 19:26 | |
| AJaeger | pabelanger: shall I approve your vexxhost-mirror change https://review.openstack.org/#/c/554322 ? Or want to do it later yourslef? | 19:26 |
| dmsimard | infra-root: The base-test is broken after merging the ara role change btw. Good thing I went that route instead of going through the existing role used by base. I'm looking into it. | 19:27 |
| dmsimard | config-core ^ | 19:27 |
| *** felipemonteiro__ has quit IRC | 19:27 | |
| clarkb | AJaeger: evrardjp https://review.openstack.org/#/c/546683/3/zuul.d/projects.yaml does that need noop jobs for merging changes? | 19:27 |
| pabelanger | AJaeger: sure, I think we are ready | 19:27 |
| AJaeger | pabelanger: approved | 19:28 |
| pabelanger | great | 19:28 |
| pabelanger | infra-root: mind a review / +3 on https://review.openstack.org/554321/ to add new mirror to cacti.o.o | 19:29 |
| dmsimard | Interesting... "2018-03-19 18:53:42,350 DEBUG zuul.AnsibleJob: [build: 642b1191b9dd48c59174b3faa9303e5d] msg: 'There was an issue creating ara-report as requested: [Errno 30] Read-only" | 19:29 |
| pabelanger | dmsimard: trying to create on executor? | 19:29 |
| dmsimard | pabelanger: well that's what it is supposed to be doing anyway, the html report was always generated straight in the log directory | 19:30 |
| evrardjp | clarkb: to be honest I don't know I thought we could just add our in repo jobs | 19:30 |
| AJaeger | mordred: this looks still wrong with neutron=12.0.0 in http://logs.openstack.org/00/554300/2/check/openstack-tox-pep8/fb77b33/job-output.txt.gz#_2018-03-19_19_26_40_127537 | 19:30 |
| clarkb | evrardjp: oh that will work too, but you'll have to do that before anything else merges | 19:30 |
| clarkb | evrardjp: basically oyu have to have a check and a gate job of some sort but sounds like you have it planned out | 19:31 |
| pabelanger | dmsimard: must be writing to some directory location where bwrap is read-only | 19:32 |
| dmsimard | pabelanger: yeah, I'll double check that the paths are the right ones (i.e, relative vs absolute) | 19:32 |
| evrardjp | clarkb: yes, I removed the zuul.d folder from the repo of import | 19:33 |
| evrardjp | next step is rolling it back | 19:33 |
| fungi | clarkb: yes, same pip/setuptools/oslo.utils issue | 19:33 |
| evrardjp | that's what I did for the bazillion of repos we are now importing... Beginning of the cycle it is! | 19:33 |
| *** dprince has quit IRC | 19:33 | |
| clarkb | evrardjp: I have approved both changes for your new repos | 19:34 |
| evrardjp | woot! | 19:34 |
| evrardjp | thanks clarkb ! | 19:34 |
| *** e0ne has quit IRC | 19:34 | |
| fungi | smcginnis: the temp non-voting change merged 10 minutes ago, and zuul should hopefully have put it into effect immediately | 19:35 |
| fungi | smcginnis: do you now have other changes which were failing that job you need enqueued/promoted? | 19:36 |
| evrardjp | we'll have another one soon, it should be the last of the series. | 19:36 |
| smcginnis | fungi: Thanks, working through https://review.openstack.org/#/q/topic:pip/5081+(status:open+OR+status:merged) | 19:36 |
| smcginnis | fungi: I will ping the respective groups now to get the others going. | 19:36 |
| *** eernst has joined #openstack-infra | 19:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add jobs of xstatic-** for heat-dashboard https://review.openstack.org/552735 | 19:38 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: base-test: Remove unused parameter and move variable declaration https://review.openstack.org/554330 | 19:39 |
| dmsimard | config-core ^ | 19:39 |
| dmsimard | brb | 19:39 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Revert "Temporarily make legacy-tempest-dsvm-neutron-src NV" https://review.openstack.org/554323 | 19:39 |
| *** VW has quit IRC | 19:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Bring vexxhost back online for testing https://review.openstack.org/554322 | 19:40 |
| *** ociuhandu has joined #openstack-infra | 19:41 | |
| *** dalgaaf_ has quit IRC | 19:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add config_template repo base jobs https://review.openstack.org/546683 | 19:42 |
| openstackgerrit | Merged openstack-infra/project-config master: Add systemd roles base jobs https://review.openstack.org/552867 | 19:42 |
| clarkb | dmsimard: won't that change mean it will generate html reports? | 19:42 |
| *** eernst has quit IRC | 19:43 | |
| *** armaan has quit IRC | 19:43 | |
| *** armaan has joined #openstack-infra | 19:44 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-gerrit master: Allow user to toggle gerrit-initial-init https://review.openstack.org/554279 | 19:44 |
| *** hasharAway has quit IRC | 19:46 | |
| *** openstackgerrit has quit IRC | 19:48 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:48 | |
| *** HeOS has quit IRC | 19:50 | |
| *** felipemonteiro_ has quit IRC | 19:52 | |
| *** openstackgerrit has joined #openstack-infra | 19:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-gerrit master: Switch to using openstack-infra/puppet-openstack_infra_spec_helper https://review.openstack.org/554296 | 19:53 |
| *** slagle has quit IRC | 19:56 | |
| *** eernst has joined #openstack-infra | 19:58 | |
| *** slagle has joined #openstack-infra | 19:58 | |
| *** felipemonteiro__ has quit IRC | 19:59 | |
| *** germs has joined #openstack-infra | 20:00 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:00 | |
| *** felipemonteiro_ has quit IRC | 20:00 | |
| *** VW has joined #openstack-infra | 20:00 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:01 | |
| clarkb | pabelanger: ^^ lgtm | 20:01 |
| dmsimard | clarkb: I need to be able to test the different permutations easily from o-z-j -- if we're going through base-test instead I can't do that easily | 20:01 |
| clarkb | dmsimard: gotcha | 20:02 |
| * dmsimard starts working on limestone mirror | 20:02 | |
| *** iyamahat_ has quit IRC | 20:02 | |
| *** iyamahat has joined #openstack-infra | 20:03 | |
| *** germs has quit IRC | 20:04 | |
| dmsimard | dtroyer: just curious, would it be silly to implement a command like "openstack cloud list" / "openstack cloud show" which would basically print out the contents of clouds.yaml ? | 20:04 |
| dmsimard | we kinda have a lot of clouds and I don't always remember what we happened to call them in our clouds.yaml :D | 20:04 |
| clarkb | dmsimard: region listing too would be nice but at least it prints out the valid regions if you provide an invalid one iirc | 20:05 |
| pabelanger | clarkb: k, waiting on zuul to report testing | 20:06 |
| dmsimard | clarkb: yeah | 20:06 |
| *** felipemonteiro__ has joined #openstack-infra | 20:06 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Start log receiver in executor https://review.openstack.org/546304 | 20:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Start log receiver in executor https://review.openstack.org/546304 | 20:08 |
| *** felipemonteiro_ has quit IRC | 20:09 | |
| dmsimard | There's an env var/argument for OS_CLOUD, is there one to select a specific clouds.yaml file to load data from ? (feeling noob right now) | 20:10 |
| pabelanger | Mar 19 19:28:07 nl03 puppet-user[10942]: Could not find data item nodepool_linaro_password in any Hiera data file and no default supplied at /opt/system-config/production/manifests/site.pp:1052 on node nl03.openstack.org | 20:11 |
| pabelanger | dmsimard: guess hiera needs to be updated?^ | 20:11 |
| dtroyer | dmsimard: that's not crazy, just one of those things I never got around to doing. | 20:11 |
| pabelanger | looks like it has been that way for a few days | 20:11 |
| dmsimard | pabelanger: linaro is ianw | 20:12 |
| dmsimard | pabelanger: I mean, I haven't added limestone to nodepool yet | 20:12 |
| dmsimard | but I did commit credentials to the hiera file before submitting my initial patches | 20:12 |
| *** eernst has quit IRC | 20:13 | |
| dmsimard | dtroyer: meanwhile, openstack train list :p | 20:13 |
| *** eernst has joined #openstack-infra | 20:13 | |
| dmsimard | dtroyer: (just kidding) | 20:13 |
| dmsimard | dtroyer: ok, wondered if anyone had thought about it until now -- I can create a story or something. | 20:14 |
| pabelanger | dmsimard: Oh, hmm | 20:14 |
| pabelanger | so why is that breaking | 20:14 |
| dmsimard | pabelanger: can you grep since when it's doing that ? | 20:14 |
| dtroyer | dmsimard: You can't cound PTG-Friday evening as work, so osc-choochoo and osc-snopenstack are both in the exception zone :) | 20:14 |
| *** ociuhandu has quit IRC | 20:14 | |
| pabelanger | mar. 15 | 20:15 |
| pabelanger | guess they were never commited into hiera | 20:15 |
| pabelanger | and maybe cleared out | 20:15 |
| pabelanger | will add them back now | 20:16 |
| dmsimard | dtroyer: haha, please don't take what I said seriously, was just poking fun. | 20:16 |
| dmsimard | (those are awesome btw, you and dhellmann keep on doing those they're great) | 20:16 |
| dtroyer | dmsimard: of course! | 20:16 |
| *** jcoufal_ has joined #openstack-infra | 20:17 | |
| * dhellmann didn't realize there was a "namespace" for fun openstack plugins | 20:18 | |
| *** jcoufal has quit IRC | 20:18 | |
| dtroyer | dhellmann: I've seen one or two of those in the wild, but its mostly just in my personal github | 20:19 |
| dmsimard | "Easter eggs" like those are fun, they definitely deserve to be highlighted and collected somewhere. They make my life less depressing :D | 20:19 |
| dhellmann | dtroyer : I'll have to use osc- in the name the next time I make one | 20:19 |
| dmsimard | It's april's fools soon, gotta think of something. | 20:19 |
| dhellmann | yeah, we should link to them on the readme or something | 20:19 |
| fungi | the lighter side of open infrastructure | 20:20 |
| dmsimard | I'm still kind of bummed we didn't merge the patch I sent for last year's april fools :) | 20:20 |
| dhellmann | dtroyer : maybe we can convince dmsimard to add a list to the osc docs if we point out that you and I can approve the patch if he writes it | 20:20 |
| *** HeOS has joined #openstack-infra | 20:20 | |
| dtroyer | +2 | 20:20 |
| *** sree has joined #openstack-infra | 20:21 | |
| *** hashar has joined #openstack-infra | 20:21 | |
| *** jcoufal_ has quit IRC | 20:22 | |
| dmsimard | dhellmann: only if we can merge this patch: "Allow developers to recheck patches with emotion in the check pipeline" https://review.openstack.org/#/c/452247/ | 20:22 |
| ianw | pabelanger: prettty sure i committed them? weird | 20:22 |
| pabelanger | ianw: okay, just added them. I think it is correct, but if you don't mind confirming | 20:22 |
| dhellmann | dmsimard : heh | 20:22 |
| pabelanger | puppet ran on nl03 again | 20:22 |
| *** jcoufal has joined #openstack-infra | 20:23 | |
| *** kjackal has quit IRC | 20:23 | |
| pabelanger | clarkb: mnaser: nodes launching again: http://grafana.openstack.org/dashboard/db/nodepool-vexxhost | 20:24 |
| clarkb | pabelanger: this is our first use of boot from volume for nodepool nodes too | 20:24 |
| clarkb | probably want to watch that for any fallout cloud side | 20:24 |
| *** sree has quit IRC | 20:25 | |
| pabelanger | clarkb: Agree! Confident the code works, we test and RDO was using it. But agree might see something else | 20:26 |
| dmsimard | We've been using BFV (with nodepool v2) in RDO for a while and it's working pretty well | 20:26 |
| pabelanger | ha | 20:26 |
| clarkb | pabelanger: I'd be more concerned about making cinder have a sad in the cloud than nodepool breaking | 20:26 |
| pabelanger | Build of instance c7ee4999-0f4d-4e46-a3a8-3a360b646b6c aborted: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes | 20:26 |
| pabelanger | mnaser: ^ | 20:26 |
| clarkb | see :) | 20:26 |
| mnaser | woopsies | 20:26 |
| mnaser | does anyone have the tenant name or id handy | 20:26 |
| pabelanger | yah, 1 sec | 20:27 |
| mnaser | would it be okay to keep the limit low like 15-20 volumes just in case things spin out of control? | 20:27 |
| clarkb | mnaser: ya should be especially if instance quota is only 10 | 20:27 |
| pabelanger | mnaser: openstackjenkins | 20:27 |
| *** eharney has quit IRC | 20:28 | |
| dmsimard | we really need to move launch-env off of ansible 2.1.. | 20:28 |
| dmsimard | and shade 1.12 | 20:28 |
| ianw | pabelanger: lgtm, maybe i missed a git add in 29bdedbcd92d286e19e9d74fd763c4d6730c1585 | 20:28 |
| * dmsimard adds to todo list | 20:28 | |
| mnaser | clarkb, pabelanger: set to 20 volumes and 1600gb | 20:29 |
| pabelanger | ianw: oh, maybe | 20:29 |
| pabelanger | mnaser: perfect | 20:29 |
| pabelanger | will know shortly | 20:29 |
| fungi | anybody happen to know how to get sphinx's raw url linking to not think a ? in a url should end the url? | 20:30 |
| *** kjackal has joined #openstack-infra | 20:32 | |
| dmsimard | fungi: hmmm, I tend to wrap anything remotely complex into `some long string`_ (and then follow up with a .. _<link> thing) | 20:32 |
| *** jcoufal_ has joined #openstack-infra | 20:33 | |
| fungi | yeah, in this case i'd like to actually display the url too. i guess i can just put it in twice | 20:33 |
| *** iyamahat has quit IRC | 20:33 | |
| *** iyamahat has joined #openstack-infra | 20:33 | |
| pabelanger | mnaser: should we be using raw or qcow2 images when booting from volume? | 20:33 |
| mnaser | raw preferably | 20:33 |
| dmsimard | clarkb, fungi, pabelanger: do we have a preference for the limestone mirror dns ? I see that we're not really putting "regionone" as the name of the region in some of the other clouds. | 20:33 |
| pabelanger | k let me check | 20:33 |
| mnaser | avoids a long conversion on the host | 20:33 |
| *** VW has quit IRC | 20:34 | |
| clarkb | dmsimard: if the regionname is regionone I think that is what we should use? | 20:34 |
| dmsimard | mirror01.osa.limestone.openstack.org perhaps ? | 20:34 |
| pabelanger | dmsimard: yah, it should be built out of what openstack values are | 20:34 |
| clarkb | dmsimard: we configure the jobs based on the cloud data from nodepool so if those don't match it won't work | 20:34 |
| *** lpetrut has quit IRC | 20:34 | |
| dmsimard | hmm, for linaro we went with mirror01.cn1.linaro.openstack.org | 20:34 |
| clarkb | dmsimard: I think it ended up getting renamed in dns | 20:34 |
| dmsimard | but in clouds.yaml I see RegionOne | 20:34 |
| clarkb | dmsimard: bceause it broke with that name? | 20:34 |
| *** lpetrut has joined #openstack-infra | 20:34 | |
| *** VW has joined #openstack-infra | 20:35 | |
| *** VW has quit IRC | 20:35 | |
| pabelanger | yah, that was broken, and likely want to rebuild the server at some point. We did cname for dns | 20:35 |
| dmsimard | oh | 20:35 |
| dmsimard | so the name of the server is just not up to date then | 20:35 |
| *** VW has joined #openstack-infra | 20:35 | |
| pabelanger | yah, I believe so | 20:35 |
| dmsimard | ok, thanks, I'll check it out. | 20:35 |
| pabelanger | vexxhost mirror was just launched today | 20:35 |
| *** jcoufal has quit IRC | 20:35 | |
| pabelanger | that is the right values | 20:35 |
| fungi | dmsimard: aha, found the non-redundant solution. wrap in `< and >`_ and it still works even if you don't include an alternate link name in there | 20:36 |
| ianw | fungi: `<http://weird.url/this?is>`__ ? | 20:36 |
| fungi | ianw: heh. yep! | 20:36 |
| dmsimard | the angle brackets? | 20:36 |
| dmsimard | magic | 20:36 |
| pabelanger | Hmm, I don't see raw images on nodepool-buliders, let me see why | 20:37 |
| clarkb | pabelanger: probably no cloud with raw images? | 20:38 |
| pabelanger | clarkb: I think it is a result of split configuration | 20:38 |
| *** jcoufal_ has quit IRC | 20:38 | |
| dmsimard | I thought we used raw everywhere | 20:38 |
| dmsimard | in case people use ceph or something | 20:38 |
| pabelanger | no, we use what os-client-config defaults to | 20:39 |
| clarkb | dmsimard: no we tend to prefer qcow2 because they are less than half the size of raw images | 20:39 |
| dmsimard | clarkb: +1, I just thought we used raw anyway | 20:40 |
| pabelanger | http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/vendors/vexxhost.json | 20:40 |
| pabelanger | doesn't list raw | 20:40 |
| pabelanger | should we add it? | 20:40 |
| clarkb | pabelanger: or have mnaser do it so that we don't bikeshed it :) | 20:40 |
| pabelanger | :) | 20:40 |
| pabelanger | I can propose nodepool.yaml change however | 20:41 |
| pabelanger | oh, right. we removed it. We can update clouds.yaml | 20:42 |
| mnaser | https://review.openstack.org/554351 Add image_format for VEXXHOST profile | 20:42 |
| ianw | infra-root: i'd like to try rolling out afs server parameter updates per https://review.openstack.org/#/c/540198/6/doc/source/afs.rst ; i've added the procedure in there. any additional thoughts? | 20:43 |
| *** trown|compassing is now known as trown|ruck | 20:43 | |
| clarkb | ianw: will we need to do it on the RO server first, then fail over to that server then do the old RW server? | 20:43 |
| *** Goneri has joined #openstack-infra | 20:43 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul master: Add instructions for reporting vulnerabilities https://review.openstack.org/554352 | 20:44 |
| clarkb | er I guess RO is actually where most of our production connections go | 20:44 |
| clarkb | so probably doing RW first is safest | 20:44 |
| clarkb | and it is the thing suffering | 20:44 |
| fungi | i need to disappear to grab some dinner, hopefully back in time for the zuul meeting | 20:44 |
| *** ihrachys has joined #openstack-infra | 20:45 | |
| dmsimard | clarkb: so we don't have our own ubuntu images that we ship for our control plane ? I guess we take whatever the provider has ? | 20:45 |
| clarkb | dmsimard: correct | 20:45 |
| clarkb | dmsimard: if provider doesn't have them we tend to grab the usptream image since that is closest to what is in other clouds | 20:46 |
| dmsimard | ok, just making sure, thanks | 20:46 |
| *** kgiusti has left #openstack-infra | 20:46 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to add RSVP template question https://review.openstack.org/554353 | 20:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Update vexxhost to use raw images https://review.openstack.org/554354 | 20:46 |
| pabelanger | dmsimard: clarkb: mnaser: ^fixes our clouds.yaml for now | 20:47 |
| pabelanger | until os-client-config is merged and released | 20:47 |
| clarkb | pabelanger: does that need to go in the builder config instead? | 20:48 |
| clarkb | ohwait its clouds.yaml nevermind | 20:48 |
| ianw | clarkb: yeah, things have been stable with the backported 1.8 packages so far. i think it's stable enough and it's worth us trying to optimise things a bit more now | 20:48 |
| pabelanger | mnaser: mind checking logs, we are seeing: Error in creating the server: Build of instance 620e21a1-5de6-4425-867b-9102764566a8 aborted: Gateway Time-out (HTTP 504) | 20:48 |
| mnaser | pabelanger: can we suspend creation or speed up that clouds.yaml fix | 20:48 |
| pabelanger | we are using qcow2 | 20:48 |
| mnaser | yeah, the api servers are being nailed because its doing a ton of conversions on the hosts | 20:49 |
| pabelanger | yah | 20:49 |
| mnaser | so it's sucking up all iops on the controllers | 20:49 |
| pabelanger | let me do it in emergency | 20:49 |
| pabelanger | we'll need to build images and upload too | 20:49 |
| mnaser | we'll probably wait for a little bit till things settle down | 20:49 |
| clarkb | ianw: the -p argument in particular seems like a good update based on the conversation wtih auristor so I think we should probably take the plunge | 20:49 |
| clarkb | mordred: corvus you may be particularly interested in ianw's afs stuff | 20:50 |
| pabelanger | mnaser: should be stopping now | 20:50 |
| mnaser | pabelanger: thank you | 20:50 |
| *** gouthamr has quit IRC | 20:51 | |
| mnaser | i'm tempted to disable all qcow image uploads | 20:51 |
| pabelanger | #status log nl03.o.o added to emergency file and max-server to 0 for vexxhost until https://review.openstack.org/554354/ land and new raw images built / uploaded | 20:51 |
| openstackstatus | pabelanger: finished logging | 20:51 |
| pabelanger | mnaser: might be worth it | 20:51 |
| mnaser | this jsut pretty much destroyed a controller | 20:51 |
| openstackgerrit | Merged openstack-infra/project-config master: base-test: Remove unused parameter and move variable declaration https://review.openstack.org/554330 | 20:51 |
| pabelanger | dmsimard: fungi: mind a +3 on 554354^ to help not destroy vexxhost | 20:52 |
| dmsimard | not destroying vexxhost is nice | 20:52 |
| pabelanger | mnaser: :( | 20:52 |
| mnaser | i dunno who is to blame here | 20:52 |
| mnaser | cinder probably should have some sort of | 20:52 |
| mnaser | do X conversions at max or something | 20:52 |
| dmsimard | pabelanger: are you sure it works like that in clouds.yaml ? | 20:53 |
| clarkb | mnaser: controller in this case is ceph? | 20:53 |
| clarkb | in the non volume case the compute nodes do the conversion locally prior to boot and then cache the result | 20:53 |
| pabelanger | dmsimard: it should, we no longer support setting it in nodepool.yaml | 20:53 |
| mnaser | clarkb: well, cinder manages ceph and because we have qcow2 images, it would downloda them from the ceph cluster onto the local machine, convert to raw, and upload them again | 20:53 |
| clarkb | mnaser: ah | 20:53 |
| mnaser | now imagine max-servers: 10 so that happening 10 times and each machine is probably ~40g after conversion or so | 20:53 |
| dmsimard | pabelanger: huh, so nodepool relies 100% on os client config for that ? TIL | 20:53 |
| clarkb | mnaser: ya some sort of deduplication of conversions and caching of results would probably help | 20:54 |
| dmsimard | mnaser: doesn't that happen on the compute nodes ? | 20:54 |
| mnaser | dmsimard: not when using bfv with cinder | 20:54 |
| mnaser | cause cinder has to create the volume | 20:54 |
| clarkb | dmsimard: in the non volume case it defintiely does | 20:54 |
| *** jcoufal has joined #openstack-infra | 20:54 | |
| pabelanger | dmsimard: yah, we removed a lot of the duplicate settings from nodepool.yaml during feature/zuulv3 | 20:54 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/config.py#n111 | 20:54 |
| dmsimard | clarkb: ah, yeah -- that's what I meant, in the case where you boot ephemeral from a qcow2 image when your ephemeral storage is backed by ceph | 20:55 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to add RSVP template question https://review.openstack.org/554353 | 20:55 |
| *** amoralej is now known as amoralej|off | 20:55 | |
| *** zerick has quit IRC | 20:56 | |
| *** zerick has joined #openstack-infra | 20:56 | |
| *** dtruong has joined #openstack-infra | 20:59 | |
| pabelanger | dmsimard: clarkb: however, clouds.yaml changes are not detected by nodepool. So we'll need to restart services to pick up that change. | 20:59 |
| *** jtomasek has quit IRC | 21:00 | |
| dmsimard | pabelanger: is that a nodepool issue or a shade issue ? | 21:00 |
| clarkb | nodepool issue | 21:01 |
| *** trown|ruck is now known as trown|outtypewww | 21:01 | |
| pabelanger | yah, I think mordred explained how to fix it a while back | 21:01 |
| pabelanger | but we only load clouds.yaml once | 21:01 |
| smcginnis | mnaser: I didn't catch all of that conversation, but are you using image caching? | 21:04 |
| smcginnis | mnaser: https://docs.openstack.org/cinder/latest/admin/blockstorage-image-volume-cache.html | 21:05 |
| smcginnis | mnaser: Though I thought there was another option for ceph that optimized that even more to not have to download/convert/upload to the same data source. | 21:05 |
| mnaser | smcginnis: nope, not enabled, that looks like a good solution, im sure that option exists but probably not if the backend is qemu | 21:06 |
| mnaser | so it downloads qemu and gets it to raw and and then pushes it up afaik | 21:06 |
| smcginnis | The generic image caching should work with all backends. | 21:06 |
| smcginnis | It will still download an image the first time, but then after that it should be much quicker. | 21:06 |
| *** ldnunes has quit IRC | 21:07 | |
| *** ldnunes has joined #openstack-infra | 21:07 | |
| *** hashar has quit IRC | 21:08 | |
| smcginnis | bnemec: If you're around, the blocker fix backported to stable/ocata should be good to go - https://review.openstack.org/#/c/554338/ | 21:09 |
| *** e0ne has joined #openstack-infra | 21:09 | |
| smcginnis | Last job running is the one that's currently non-voting. | 21:09 |
| bnemec | Weirdly, the pike backport hasn't merged yet. It's waiting on the pep8 job. o.O | 21:11 |
| smcginnis | I noticed that. Hopefully that final one gets picked up soon. | 21:11 |
| smcginnis | At least it should be fairly quick once it does. | 21:12 |
| pabelanger | bnemec: it likely tried to use vexxhost, which we recently brought back online. We've since disabled it | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/irc-meetings master: Update Ironic meeting https://review.openstack.org/554361 | 21:15 |
| logan- | That’s pretty cool smcginnis I had no idea that existed | 21:15 |
| mnaser | TIL volumes have status 'downloading' | 21:16 |
| pabelanger | stepping away for food, I'll check back on vexxhost clouds.yaml change shortly | 21:16 |
| smcginnis | logan-: Cool, glad I was able to spread awareness about that. I'd actually like to make it the default, but probably not really safe to do that. | 21:17 |
| *** jcoufal_ has joined #openstack-infra | 21:18 | |
| *** jcoufal has quit IRC | 21:18 | |
| smcginnis | Any way to push that last job along on 554053? | 21:22 |
| openstackgerrit | Merged openstack-infra/system-config master: Update vexxhost to use raw images https://review.openstack.org/554354 | 21:22 |
| *** jcoufal_ has quit IRC | 21:28 | |
| *** jcoufal has joined #openstack-infra | 21:29 | |
| *** felipemonteiro__ has quit IRC | 21:32 | |
| *** kjackal has quit IRC | 21:33 | |
| *** jcoufal_ has joined #openstack-infra | 21:34 | |
| *** lpetrut has quit IRC | 21:34 | |
| *** jcoufal has quit IRC | 21:35 | |
| *** jcoufal has joined #openstack-infra | 21:37 | |
| *** jcoufal_ has quit IRC | 21:39 | |
| *** jcoufal has quit IRC | 21:41 | |
| smcginnis | So, I notice new jobs entering the gate queue are getting picked up, but the pep8 job we're waiting on for oslo.utils is still queued. When will that get picked up now? | 21:42 |
| *** jcoufal has joined #openstack-infra | 21:43 | |
| *** ykarel has quit IRC | 21:43 | |
| *** greghayn1 is now known as greghaynes | 21:46 | |
| clarkb | smcginnis: whenever nodepool finishes booting the instance for the job | 21:48 |
| clarkb | its mich less fifo now with the ditributed nodepool builders | 21:48 |
| *** gyee has quit IRC | 21:50 | |
| dansmith | clarkb: do you happen to know what makes loading logs from logs.o.o so slow? they come in at like 90kb/s.. is it the server side coloring or something? | 21:50 |
| bnemec | smcginnis: I think I'm just going to pull the trigger on the ocata backport. It's not like we're going to forget to merge the pike one when it's breaking so much stuff. | 21:52 |
| clarkb | dansmith: wget and curl will bypass the coloring by default as they don't send the correct accept encodings. YOu can try really quick with one of them to see if it is still slow | 21:52 |
| smcginnis | bnemec: pep8 did just pass on the pike one. \o/ | 21:52 |
| bnemec | And unless the pep8 job somehow fails it should merge whenever it gets a vm anyway. | 21:52 |
| bnemec | smcginnis: I should have made this decision a long time ago then. :-) | 21:53 |
| smcginnis | bnemec: Haha, no doubt. ;) | 21:53 |
| dansmith | clarkb: yeah, slow as fsck with curl too | 21:53 |
| clarkb | dansmith: ok then its not the coloring. Looking at cacti it isn't saturating its link either. http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all | 21:53 |
| clarkb | if I had to guess based on that I would probably blame the disk and the number of reads in aggregate? | 21:54 |
| dansmith | clarkb: is it not slow for you? | 21:54 |
| clarkb | dansmith: I've not checked recently | 21:54 |
| clarkb | I can try pulling something as I'm guessing I have a different isp than you | 21:54 |
| clarkb | now to find a file big enough to show me the slowness | 21:55 |
| dansmith | clarkb: well, I just pulled one a second time and it did go way quicker | 21:55 |
| dansmith | and it does appear to go faster as the file comes down, | 21:55 |
| clarkb | dansmith: I appear to get closer to 1MBps | 21:56 |
| dansmith | so maybe that's latency to pull it from disk and then prefetch kicks in or something | 21:56 |
| dansmith | clarkb: that's what I get summarized at the end, but it's slower to begin with and slowly ramps up to about that | 21:56 |
| dansmith | first pull took 35s, second pull took 4s | 21:56 |
| jhesketh | Morning | 21:56 |
| clarkb | this is first pull on http://logs.openstack.org/55/554155/1/check/tempest-full/2cedc84/controller/logs/screen-q-agt.txt.gz for me | 21:57 |
| clarkb | (its a larger file) | 21:57 |
| clarkb | 34.78MB in 37 seconds | 21:57 |
| dansmith | yeah, so I just pulled that in 3s | 21:57 |
| dansmith | because you got it into the cache I guess | 21:57 |
| dansmith | so I guess it's just that | 21:58 |
| clarkb | dansmith: also if you are on comcast their ipv6 can be flaky you might try forcing ipv4 to see if there is any difference | 21:58 |
| clarkb | otherwise I'm guessing its largly io bound to disk | 21:58 |
| dansmith | clarkb: I'm on ipv4 only right now yeah | 21:58 |
| *** boden has quit IRC | 21:58 | |
| dansmith | clarkb: I guess logs.o is using a big volume, so probably iscsi, latency, and low memory on the instance serving this which means very little cache | 21:58 |
| *** boden has joined #openstack-infra | 21:58 | |
| *** boden has quit IRC | 21:58 | |
| clarkb | dansmith: also with the number of files and the number of times we tend to hit them I'm not sure any amount of cache would help terribly if backend disk is slow | 22:00 |
| dansmith | that's what I mean, | 22:00 |
| dansmith | low memory compared to the working set | 22:00 |
| *** germs has joined #openstack-infra | 22:00 | |
| *** germs has quit IRC | 22:00 | |
| *** germs has joined #openstack-infra | 22:00 | |
| clarkb | ya | 22:01 |
| clarkb | 12TB to 8GB | 22:01 |
| dansmith | yeah | 22:01 |
| *** jcoufal has quit IRC | 22:01 | |
| dansmith | just.. sucks | 22:01 |
| dansmith | if it was a local disk it should yank out faster than my internet connection so I don't notice, but.. | 22:01 |
| clarkb | but even if we cached 1TB of it by the time we cached it we would no longer need it | 22:01 |
| dansmith | yeah | 22:02 |
| corvus | it's zuul meeting time in #openstack-meeting-alt | 22:02 |
| *** dangers_` is now known as dangers_away | 22:03 | |
| *** germs has quit IRC | 22:05 | |
| *** e0ne has quit IRC | 22:12 | |
| *** liusheng has quit IRC | 22:14 | |
| *** liusheng has joined #openstack-infra | 22:14 | |
| *** VW has quit IRC | 22:15 | |
| *** VW has joined #openstack-infra | 22:16 | |
| *** VW_ has joined #openstack-infra | 22:18 | |
| *** bobh has quit IRC | 22:19 | |
| *** VW has quit IRC | 22:21 | |
| *** aeng has joined #openstack-infra | 22:22 | |
| *** VW_ has quit IRC | 22:23 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config master: Ironic to Storyboard https://review.openstack.org/554370 | 22:26 |
| *** rcernin has joined #openstack-infra | 22:34 | |
| *** katkapilatova has left #openstack-infra | 22:36 | |
| *** katkapilatova has quit IRC | 22:36 | |
| *** rosmaita has quit IRC | 22:36 | |
| *** Goneri has quit IRC | 22:37 | |
| *** edmondsw has quit IRC | 22:49 | |
| *** hongbin has quit IRC | 22:49 | |
| dmsimard | I have to step away for now but I can't seem to get launch-node to use the right IP for the mirror node -- it tries to use the 10.x private network IP | 22:52 |
| dmsimard | I've sort of been trying to pattern my clouds.yaml off of what we had for OSIC since it's a similar deployment | 22:53 |
| dmsimard | ex: http://git.openstack.org/cgit/openstack-infra/system-config/commit/modules/openstack_project/templates/puppetmaster?id=45e93933006815f687c70d55b4171e58e926d597 | 22:54 |
| clarkb | dmsimard: you shouldn't need any of that | 22:55 |
| clarkb | the only reason osic needed it was we were using ipv6 networks which werne't the default | 22:55 |
| dmsimard | clarkb: the cloud is dual stack | 22:55 |
| dmsimard | clarkb: has ipv6 | 22:55 |
| clarkb | thats fine, I still don't think you need the osic like config in that case | 22:55 |
| clarkb | you'd only need it if the networks were separte | 22:55 |
| dmsimard | there's two public networks, one v4 and one v6 | 22:56 |
| dmsimard | and then a private v4 network | 22:56 |
| fungi | yeah, osic wasn't really proper dual stack, so we needed workarounds | 22:56 |
| clarkb | gotcha so its not a single network with different subnets? | 22:56 |
| *** rosmaita has joined #openstack-infra | 22:56 | |
| fungi | oh, and you get a different nic for global v4 vs v6? | 22:57 |
| fungi | so sort of like osic after all | 22:57 |
| dmsimard | There's "Internal v4" which is that 10.x network and then there's a "Public IPv6" network that are scoped to the tenant | 22:57 |
| dmsimard | logan- says we should use that public ipv6 network for nodepool | 22:58 |
| fungi | does the internal v4 network have egress through an overload nat? | 22:59 |
| logan- | yes there is snat enabled on the private v4 | 22:59 |
| clarkb | so ya this is basically osic in network setup | 23:00 |
| logan- | also floating ips available | 23:00 |
| logan- | for the mirror public v4 | 23:00 |
| dmsimard | stepping away, I'll read scrollback later and catch up | 23:00 |
| logan- | yup very similar to osic | 23:00 |
| fungi | sounds basically like the same network model we had there, agreed | 23:00 |
| clarkb | dmsimard: my hunch is if you are telling it to not use floating ips (whcih was in our osic config) that is why it is failing for you booting the mirror | 23:01 |
| clarkb | the mirror we want to boot with a floating ip | 23:01 |
| dmsimard | clarkb: probable | 23:01 |
| *** david-lyle has joined #openstack-infra | 23:01 | |
| clarkb | for the noepool nodes we'll v6 primary them and not floating ip | 23:02 |
| ianw | corvus: i was going to ask if we had free time in mtg, but https://review.openstack.org/#/c/553999/ with the nodepool web stuff in zuul. that's the direction we're going? i wasn't sure on the coupling there | 23:02 |
| fungi | or on some provider network that has an available global v4 address | 23:02 |
| logan- | clarkb: yup thats what I was thinking, ideally no public v4 on the nodepool nodes just internal v4 + nat | 23:02 |
| logan- | snat* | 23:02 |
| fungi | (does afs worth through nat?) | 23:02 |
| clarkb | fungi: it does just less well | 23:02 |
| fungi | k | 23:02 |
| clarkb | er less well with snat. floating ip nat should be fine | 23:03 |
| clarkb | since its 1:1 | 23:03 |
| fungi | ahh, okay | 23:03 |
| fungi | so not in the ways that ftp or sip break with nat | 23:03 |
| ianw | pabelanger: if you wouldn't mind a read of https://review.openstack.org/#/c/554089/ for typos, i think we can just merge. nothing great but might save someone some time pulling apart bits of reprepro in future! | 23:03 |
| *** hemna_ has quit IRC | 23:03 | |
| fungi | afs isn't embedding the client address in the payload or anything | 23:03 |
| clarkb | but ya I've definitely used it at home through nat | 23:04 |
| logan- | if you need a network with fixed public v4 for infra that can be arranged too | 23:04 |
| clarkb | logan-: I don' | 23:04 |
| clarkb | er | 23:04 |
| clarkb | I don't think it will be required | 23:04 |
| logan- | ok | 23:04 |
| fungi | yeah, sounds like it'll be fine to have a fip for that one server, but we can keep that idea in reserve just in case | 23:05 |
| logan- | wfm | 23:05 |
| *** aeng has quit IRC | 23:07 | |
| clarkb | the osic setup was essentially this. mirror on fip and on v6 network. Test nodes only on private ipv4 and on public v6. This meant that test nodes could talk to the mirror over v6 without nat but mirror could talk v4 to afs (because afs doesn't v6) | 23:08 |
| logan- | will be afk for an hour or so, but in case it helps any this is the whole network layout on that cloud: https://github.com/continuouspw/openstack-ci-cloud/blob/master/network_bootstrap/bootstrap-neutron.yml | 23:08 |
| clarkb | it also allowed us to v4 for network overlays because you can't vxlan over v6 using ovs | 23:08 |
| fungi | config-core reviewers: it would be great to get https://review.openstack.org/551406 (Replace Queens signing key with Rocky) approved in the next hour or so. otherwise i'll need to adjust the effective date on the corresponding release documentation change | 23:10 |
| clarkb | fungi: done | 23:11 |
| fungi | thanks! | 23:12 |
| * ianw starts up key procedure to sign it | 23:12 | |
| clarkb | mnaser: your review was basically the same as my review. I did not attempt to decrypt the data | 23:12 |
| mnaser | clarkb: :D | 23:13 |
| fungi | and thanks mnaser! | 23:13 |
| clarkb | we have a tool to do the encrypting so generally trust the output | 23:13 |
| *** myoung|rover is now known as myoung|bbl | 23:13 | |
| fungi | the litmus test will be when the release team approves something after that merges. i'll give them a heads up and can validate the resulting signatures once there are some | 23:13 |
| *** aeng has joined #openstack-infra | 23:14 | |
| logan- | clarkb: yep that makes sense. with the way its set up now you can dual stack the mirror dns if desired using the floating v4 + fixed v6 enabling nodepool to access it over either stack (snat v4 or fixed v6 from nodepool nodes) | 23:14 |
| fungi | they may want to trigger a tag for the release-test repo to exercise it | 23:14 |
| *** tpsilva has quit IRC | 23:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Replace Queens signing key with Rocky https://review.openstack.org/551406 | 23:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Use -python35-jobs-horizon for trove-dashboard https://review.openstack.org/553116 | 23:21 |
| *** masber has joined #openstack-infra | 23:27 | |
| *** yamamoto has joined #openstack-infra | 23:27 | |
| *** claudiub has quit IRC | 23:44 | |
| *** witek has quit IRC | 23:47 | |
| *** witek_ has joined #openstack-infra | 23:47 | |
| *** r-daneel has quit IRC | 23:49 | |
| *** mriedem has quit IRC | 23:52 | |
| *** aeng has quit IRC | 23:56 | |
| *** HeOS has quit IRC | 23:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update AFS fileserver settings https://review.openstack.org/540198 | 23:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Uninstall and reinstall siblings one at a time https://review.openstack.org/554297 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!