| *** hamzy has joined #openstack-infra | 00:00 | |
| *** aaronsheffield has quit IRC | 00:03 | |
| *** gyee has quit IRC | 00:18 | |
| *** yamamoto has quit IRC | 00:19 | |
| ianw | clarkb: i was reading https://opendev.org/opendev/puppet-diskimage_builder/src/branch/master/manifests/init.pp#L24 wrong ; it does write it to a file then specify that file as the path ... so weird it's not picked up | 00:20 |
|---|---|---|
| *** jamesmcarthur has joined #openstack-infra | 00:22 | |
| *** betherly has joined #openstack-infra | 00:23 | |
| *** betherly has quit IRC | 00:27 | |
| *** dklyle has joined #openstack-infra | 00:30 | |
| *** mriedem has quit IRC | 00:41 | |
| *** jamesmcarthur has quit IRC | 00:46 | |
| *** jamesmcarthur has joined #openstack-infra | 00:47 | |
| *** ricolin has joined #openstack-infra | 00:48 | |
| *** jamesmcarthur has quit IRC | 00:52 | |
| *** mattw4 has quit IRC | 00:56 | |
| *** Weifan has quit IRC | 01:04 | |
| *** Weifan has joined #openstack-infra | 01:04 | |
| *** sarob_ has quit IRC | 01:09 | |
| *** jamesmcarthur has joined #openstack-infra | 01:11 | |
| *** SpamapS has quit IRC | 01:16 | |
| *** rcernin has quit IRC | 01:16 | |
| *** SpamapS has joined #openstack-infra | 01:16 | |
| *** rcernin has joined #openstack-infra | 01:17 | |
| *** rcernin has quit IRC | 01:18 | |
| *** rcernin has joined #openstack-infra | 01:19 | |
| openstackgerrit | Ian Wienand proposed opendev/puppet-diskimage_builder master: Install backported debootstrap https://review.opendev.org/659031 | 01:32 |
| *** whoami-rajat has joined #openstack-infra | 01:34 | |
| *** betherly has joined #openstack-infra | 01:34 | |
| *** michael-beaver has quit IRC | 01:38 | |
| *** betherly has quit IRC | 01:40 | |
| *** sshnaidm|afk has quit IRC | 01:42 | |
| *** nicolasbock has quit IRC | 01:42 | |
| *** ykatabam has joined #openstack-infra | 01:46 | |
| *** Weifan has quit IRC | 01:49 | |
| *** SpamapS has quit IRC | 01:52 | |
| *** ykatabam has quit IRC | 01:52 | |
| *** SpamapS has joined #openstack-infra | 01:53 | |
| *** apetrich has quit IRC | 01:57 | |
| *** betherly has joined #openstack-infra | 02:06 | |
| *** betherly has quit IRC | 02:10 | |
| *** rlandy|bbl is now known as rlandy | 02:12 | |
| *** jamesmcarthur has quit IRC | 02:14 | |
| openstackgerrit | Colleen Murphy proposed openstack/openstack-zuul-jobs master: Make puppet-4 beaker jobs voting https://review.opendev.org/659203 | 02:30 |
| *** jamesmcarthur has joined #openstack-infra | 02:32 | |
| *** betherly has joined #openstack-infra | 02:37 | |
| *** hongbin has joined #openstack-infra | 02:39 | |
| *** rlandy has quit IRC | 02:40 | |
| *** betherly has quit IRC | 02:41 | |
| *** tkajinam has quit IRC | 02:55 | |
| *** tkajinam has joined #openstack-infra | 02:56 | |
| *** betherly has joined #openstack-infra | 02:57 | |
| *** xinranwang has joined #openstack-infra | 02:58 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:00 | |
| *** betherly has quit IRC | 03:02 | |
| *** ykarel|away has joined #openstack-infra | 03:04 | |
| *** yamamoto has joined #openstack-infra | 03:08 | |
| *** SpamapS has quit IRC | 03:08 | |
| *** SpamapS has joined #openstack-infra | 03:09 | |
| *** yamamoto has quit IRC | 03:17 | |
| *** hongbin has quit IRC | 03:18 | |
| *** gregoryo has joined #openstack-infra | 03:19 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: enable using a single host with different user or port https://review.opendev.org/659209 | 03:21 |
| *** ykarel|away is now known as ykarel | 03:21 | |
| *** armax has quit IRC | 03:24 | |
| *** betherly has joined #openstack-infra | 03:28 | |
| *** SpamapS has quit IRC | 03:30 | |
| *** SpamapS has joined #openstack-infra | 03:32 | |
| *** betherly has quit IRC | 03:33 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use megabyte granularity for image extra space https://review.opendev.org/657654 | 03:38 |
| *** SpamapS has quit IRC | 03:40 | |
| *** jamesmcarthur has quit IRC | 03:40 | |
| *** SpamapS has joined #openstack-infra | 03:41 | |
| *** yamamoto has joined #openstack-infra | 03:54 | |
| *** ykarel is now known as ykarel|afk | 03:54 | |
| *** psachin has joined #openstack-infra | 03:56 | |
| *** udesale has joined #openstack-infra | 03:58 | |
| *** ykarel|afk has quit IRC | 03:58 | |
| *** jamesmcarthur has joined #openstack-infra | 03:58 | |
| *** betherly has joined #openstack-infra | 03:59 | |
| *** jamesmcarthur has quit IRC | 04:03 | |
| *** betherly has quit IRC | 04:04 | |
| AJaeger | config-core, could you review https://review.opendev.org/#/c/658682/ and https://review.opendev.org/#/c/659001/ , please? | 04:05 |
| *** ykarel|afk has joined #openstack-infra | 04:15 | |
| *** diablo_rojo has quit IRC | 04:15 | |
| *** yamamoto has quit IRC | 04:18 | |
| *** mattw4 has joined #openstack-infra | 04:24 | |
| *** ykarel|afk is now known as ykarel | 04:27 | |
| openstackgerrit | Logan V proposed openstack/project-config master: Increase nodepool image journal size https://review.opendev.org/659215 | 04:31 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove legacy ansible-lint job https://review.opendev.org/658682 | 04:32 |
| *** yamamoto has joined #openstack-infra | 04:35 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove unused legacy-rally-dsvm-zaqar-zaqar https://review.opendev.org/659001 | 04:36 |
| *** yamamoto has quit IRC | 04:38 | |
| *** yamamoto has joined #openstack-infra | 04:38 | |
| *** mattw4 has quit IRC | 04:39 | |
| *** janki has joined #openstack-infra | 04:43 | |
| *** betherly has joined #openstack-infra | 04:51 | |
| johnsom | Are the DNS issues at vexxhost a known issue? http://logs.openstack.org/95/659195/1/check/octavia-v2-dsvm-scenario/3700090/controller/logs/unbound_log.txt.gz and http://logs.openstack.org/95/659195/1/check/octavia-v2-dsvm-scenario/3700090/controller/logs/dib-build/amphora-x64-haproxy.qcow2_log.txt.gz#_2019-05-15_01_15_37_132 | 04:53 |
| johnsom | curl: (6) Could not resolve host: bootstrap.pypa.io | 04:54 |
| *** betherly has quit IRC | 04:56 | |
| *** e0ne has joined #openstack-infra | 05:05 | |
| *** e0ne has quit IRC | 05:08 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 05:12 | |
| *** bhavikdbavishi has quit IRC | 05:14 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 05:14 | |
| *** betherly has joined #openstack-infra | 05:32 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: bubblewrap: bind mount /etc/subuid https://review.opendev.org/659218 | 05:35 |
| *** betherly has quit IRC | 05:37 | |
| tristanC | Did skopeo got updated recently on the zuul-executor hosts? It seems like the push-to-intermediate-registry task is failing because of a missing subuid file | 05:37 |
| *** yamamoto has quit IRC | 05:39 | |
| *** kjackal has joined #openstack-infra | 05:40 | |
| *** yamamoto has joined #openstack-infra | 05:44 | |
| tristanC | which would have been introduced by https://launchpadlibrarian.net/420875087/skopeo_0.1.36-1~dev~ubuntu18.04~ppa16_0.1.36-1~dev~ubuntu18.04~ppa17.diff.gz | 05:47 |
| *** spsurya has joined #openstack-infra | 05:49 | |
| *** jtomasek has joined #openstack-infra | 05:52 | |
| *** betherly has joined #openstack-infra | 05:53 | |
| *** psachin has quit IRC | 05:54 | |
| *** yboaron_ has joined #openstack-infra | 05:55 | |
| *** psachin has joined #openstack-infra | 05:57 | |
| AJaeger | thanks, ianw for reviewing | 05:57 |
| *** betherly has quit IRC | 05:58 | |
| AJaeger | config-core, a couple of new repos: https://review.opendev.org/657764 https://review.opendev.org/656657 https://review.opendev.org/657862 , please review | 05:58 |
| *** dciabrin has quit IRC | 06:02 | |
| *** pgaxatte has joined #openstack-infra | 06:06 | |
| *** xinranwang has quit IRC | 06:08 | |
| *** dpawlik has joined #openstack-infra | 06:23 | |
| *** kaisers has joined #openstack-infra | 06:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 06:33 |
| ianw | cmurphy: ^ for when you're around ... just because I know you'll be waking up thinking "I hope today I can be debugging an external site making a wide range of lightly documented moves of key packages" | 06:35 |
| *** e0ne has joined #openstack-infra | 06:35 | |
| *** auristor has quit IRC | 06:37 | |
| *** jaosorior has quit IRC | 06:42 | |
| *** apetrich has joined #openstack-infra | 06:47 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 06:49 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.0 https://review.opendev.org/610436 | 06:49 |
| *** yamamoto has quit IRC | 06:50 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Fix scheduler breakage when using periodic jobs with artifacts https://review.opendev.org/659222 | 06:50 |
| openstackgerrit | Merged openstack/diskimage-builder master: Use megabyte granularity for image extra space https://review.opendev.org/657654 | 06:51 |
| *** jaosorior has joined #openstack-infra | 06:51 | |
| *** bhavikdbavishi has quit IRC | 06:51 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:51 | |
| *** auristor has joined #openstack-infra | 06:51 | |
| *** yamamoto has joined #openstack-infra | 06:52 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 06:55 | |
| *** bhavikdbavishi has quit IRC | 06:56 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:56 | |
| *** yamamoto has quit IRC | 07:00 | |
| *** rpittau|afk is now known as rpittau | 07:07 | |
| *** ginopc has joined #openstack-infra | 07:10 | |
| *** tesseract has joined #openstack-infra | 07:13 | |
| *** yamamoto has joined #openstack-infra | 07:16 | |
| *** rcernin has quit IRC | 07:25 | |
| *** serhatd__ has joined #openstack-infra | 07:26 | |
| *** tosky has joined #openstack-infra | 07:27 | |
| *** zigo has quit IRC | 07:28 | |
| *** serhatd_ has quit IRC | 07:28 | |
| *** serhatd_ has joined #openstack-infra | 07:31 | |
| *** serhatd__ has quit IRC | 07:33 | |
| *** iurygregory has joined #openstack-infra | 07:36 | |
| *** yamamoto has quit IRC | 07:38 | |
| *** yboaron_ has quit IRC | 07:39 | |
| dangtrinhnt | Hi fungi, please have a look at this patch for moving Telemetry to storyboard #link https://review.opendev.org/#/c/651963/ | 07:43 |
| dangtrinhnt | I've updated the launchpad pages mentioning the migration. | 07:43 |
| dangtrinhnt | Many thanks | 07:43 |
| *** corvus has quit IRC | 07:47 | |
| *** imacdonn has quit IRC | 07:47 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 07:51 | |
| *** imacdonn has joined #openstack-infra | 07:51 | |
| *** corvus has joined #openstack-infra | 07:52 | |
| *** bhavikdbavishi has quit IRC | 07:52 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 07:52 | |
| ianw | cmurphy / clarkb: i've pretty much run out of time for today on the puppet install bits for 659221 but in a comment i've linked to an upstream bug i've filed, because i don't think the release packages should have disappeared too. | 07:53 |
| ianw | feel free to update of course, otherwise i can look again tomorrow | 07:54 |
| *** ykarel is now known as ykarel|lunch | 07:55 | |
| *** jpich has joined #openstack-infra | 07:55 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: remove SafeLoader left-over from ZuulJSONEncoder https://review.opendev.org/659026 | 07:59 |
| *** kopecmartin|off is now known as kopecmartin | 08:01 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.opendev.org/643731 | 08:02 |
| *** tkajinam has quit IRC | 08:08 | |
| *** yamamoto has joined #openstack-infra | 08:13 | |
| *** bhavikdbavishi has quit IRC | 08:13 | |
| *** lucasagomes has joined #openstack-infra | 08:15 | |
| *** lennyb has joined #openstack-infra | 08:16 | |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Deprecate rhel7 in favor of rhel https://review.opendev.org/647710 | 08:17 |
| *** sean-k-mooney has joined #openstack-infra | 08:19 | |
| sean-k-mooney | o/ | 08:19 |
| sean-k-mooney | i think we broke the redicts to opendev in the last 12 hours http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Could%20not%20install%20packages%20due%20to%20an%20EnvironmentError:%20HTTPSConnectionPool(host%3D'git.openstack.org',%20port%3D443)%5C%22 | 08:20 |
| sean-k-mooney | actully no im reading that wrong | 08:20 |
| sean-k-mooney | but we are seeing failure in the gate with tempest not being able to install deps form git.openstack.org with this error http://logs.openstack.org/97/652197/17/check/grenade-py3/7b3cac4/logs/grenade.sh.txt.gz#_2019-05-15_02_34_39_701 | 08:21 |
| *** yamamoto has quit IRC | 08:23 | |
| *** jpich has quit IRC | 08:28 | |
| *** jpich has joined #openstack-infra | 08:29 | |
| *** dtantsur|afk is now known as dtantsur | 08:35 | |
| *** gfidente has joined #openstack-infra | 08:35 | |
| *** calbers has quit IRC | 08:40 | |
| *** yamamoto has joined #openstack-infra | 08:43 | |
| *** ralonsoh has joined #openstack-infra | 08:46 | |
| sean-k-mooney | i have filed a bug for ^ https://bugs.launchpad.net/nova/+bug/1829161 and ill submit an elastic recheck query for it shortly | 08:50 |
| openstack | Launchpad bug 1829161 in OpenStack Compute (nova) "Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='git.openstack.org', port=443)" [Critical,Triaged] | 08:50 |
| *** pcaruana has joined #openstack-infra | 08:51 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:51 | |
| *** ykarel|lunch is now known as ykarel | 08:54 | |
| *** derekh has joined #openstack-infra | 08:57 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: bubblewrap: bind mount /etc/subuid https://review.opendev.org/659218 | 08:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Make image build jobs voting again https://review.opendev.org/659242 | 08:57 |
| *** gregoryo has quit IRC | 08:59 | |
| corvus | infra-root: ^ as tristanC mentioned earlier, it looks like a new version of skopeo has broken the docker jobs -- we need to merge the first change and restart the executors with it, then we can merge the second one | 08:59 |
| *** yboaron_ has joined #openstack-infra | 09:02 | |
| mordred | corvus: ++ | 09:03 |
| sean-k-mooney | corvus: mordred have ye seen the issue above with installing deps form git.openstack.org | 09:04 |
| sean-k-mooney | looks like it started on the 6th of may | 09:04 |
| mordred | sean-k-mooney: just waking up (at a time I wish I weren't) ... I shall now read scrollback | 09:04 |
| *** pkopec has joined #openstack-infra | 09:05 | |
| sean-k-mooney | mordred: no worries i normally would not be awake at this time or jsut waiking up too but i got up at 2am today | 09:05 |
| corvus | sean-k-mooney: istr some talk about the gitea mirrors sometimes returning 404s recently -- clarkb was wondering if one of the servers was bad (or our one bad server somehow was put back in rotation) | 09:06 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket build status reporter https://review.opendev.org/658335 | 09:06 |
| corvus | sean-k-mooney: i wonder if that's related | 09:06 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source https://review.opendev.org/658835 | 09:06 |
| sean-k-mooney | my sleep pattern is non existent after teh summit | 09:06 |
| sean-k-mooney | corvus: thats what it looks like to me too in that this only somethimes fails | 09:06 |
| *** jangutter has quit IRC | 09:06 | |
| corvus | sean-k-mooney, mordred: i'll check the lb config and see if gitea06 slipped back in | 09:07 |
| sean-k-mooney | that said the base error is "Temporary failure in name resolution" | 09:07 |
| corvus | sean-k-mooney: oh, that's almost certainly different then... :/ | 09:08 |
| sean-k-mooney | this is what im seeing in the logs http://logs.openstack.org/97/652197/17/check/grenade-py3/7b3cac4/logs/grenade.sh.txt.gz#_2019-05-15_02_34_39_701 | 09:08 |
| corvus | gitea06 is not in the lb config (which is good and correct -- that's a known bad server) | 09:09 |
| sean-k-mooney | it could just be a bad dns server | 09:10 |
| corvus | sean-k-mooney: have you observed any patterns in the providers for this error? | 09:11 |
| sean-k-mooney | is there a way in logstash to see if all the hits where in teh same cloud provider | 09:11 |
| sean-k-mooney | vexxhost-sjc1 for that one ill check on of the others | 09:12 |
| sean-k-mooney | oh there is a node_provider tick box | 09:13 |
| sean-k-mooney | ya most if are from vexhost | 09:13 |
| sean-k-mooney | mnaser: ^ | 09:13 |
| mordred | I think i remember something in scrollback from clarkb about dns | 09:13 |
| mordred | https://review.opendev.org/#/c/659181/ | 09:14 |
| sean-k-mooney | actully all but one hit are from vexhost the other was form limestone | 09:14 |
| mordred | scrollback from around 22:27:02 | 09:14 |
| sean-k-mooney | maybe alther the instance i checked had an ipv4 address | 09:16 |
| mordred | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-05-14.log.html#t2019-05-14T22:28:46 | 09:16 |
| sean-k-mooney | that said we do configre the ipv6 names servers http://logs.openstack.org/97/652197/17/check/grenade-py3/7b3cac4/job-output.txt.gz#_2019-05-15_02_09_26_673747 | 09:16 |
| sean-k-mooney | ah | 09:16 |
| sean-k-mooney | ok so ya that is likely the issue. | 09:17 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Prevent Zuul scheduler to crash at startup if gerrit down https://review.opendev.org/576192 | 09:18 |
| corvus | mordred, sean-k-mooney +3 | 09:19 |
| *** sshnaidm has joined #openstack-infra | 09:21 | |
| sean-k-mooney | its always nice to find someone has already fixed the issue you just hit :) | 09:22 |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.opendev.org/643731 | 09:25 |
| *** zigo has joined #openstack-infra | 09:35 | |
| aspiers | just had an idea - could we set up some simple IRC meeting subscription mechanism, so that people automatically get emailed a link to the minutes of meetings they choose to subscribe to? | 09:40 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Remove unneeded extras definition https://review.opendev.org/659255 | 09:40 |
| *** bhavikdbavishi1 has joined #openstack-infra | 09:40 | |
| *** yamamoto has quit IRC | 09:41 | |
| *** bhavikdbavishi has quit IRC | 09:41 | |
| openstackgerrit | Merged opendev/base-jobs master: Only use ipv6 dns if nodepool uses ipv6 https://review.opendev.org/659181 | 09:45 |
| *** bhavikdbavishi1 has quit IRC | 09:45 | |
| corvus | aspiers: almost certainly, though nothing in our tooling immediately suggests a mechanism to implement that. you could do it as a periodic zuul job (running on a repo which contains the subscription info?) | 09:46 |
| aspiers | corvus: ah, nice idea | 09:46 |
| aspiers | corvus: so people would subscribe by adding themselves to a yaml file? | 09:47 |
| corvus | aspiers: yep | 09:48 |
| aspiers | hrm | 09:48 |
| aspiers | sounds easy to implement but not ideal in terms of self-service workflow | 09:48 |
| *** yamamoto has joined #openstack-infra | 09:48 | |
| aspiers | requires going through the Gerrit review process | 09:49 |
| corvus | aspiers: oh i just asusmped you would approve all the changes :) | 09:49 |
| *** sshnaidm is now known as sshnaidm|afk | 09:49 | |
| aspiers | haha ;) | 09:49 |
| aspiers | something closer to the mailman sanity checks on email addresses would be more ideal | 09:49 |
| aspiers | I guess another option is to send all meeting links to a new "meetings" ML, and then have people set up their own filters | 09:49 |
| corvus | aspiers: depending on how far down the rabbit hole you want to go... technically you could probably create a job which auto-approved changes to the subscription if they only altered the email address of the change submitter.... | 09:50 |
| aspiers | that would deal with the email address validation part | 09:50 |
| corvus | aspiers: yes, one (or multiple) lists is another option | 09:50 |
| aspiers | yeah, except some people use a different address for Gerrit | 09:50 |
| corvus | could have one list per meeting and auto-create them... | 09:50 |
| aspiers | yikes | 09:50 |
| corvus | just brainstorming :) | 09:51 |
| aspiers | or just email openstack-discuss with [minutes] badge | 09:51 |
| aspiers | actually I like that one most | 09:51 |
| aspiers | clean, simple, and people immediately get the benefit through their existing email setup/filters | 09:51 |
| mordred | the nice thing about [minutes] is that peopel who are unaware of various irc meetings have discoverabilit | 09:52 |
| corvus | so -- [minutes][nova] ? | 09:52 |
| *** bhavikdbavishi has joined #openstack-infra | 09:52 | |
| aspiers | mordred: exactly | 09:52 |
| aspiers | corvus: yup | 09:52 |
| aspiers | it's a max of 1 or 2 per week per topic, which doesn't seem very spammy relative to other traffic per topic | 09:53 |
| *** yamamoto has quit IRC | 09:53 | |
| corvus | and it's *relevant* | 09:53 |
| aspiers | right | 09:53 |
| corvus | it's probably worth making sure we consider opendev here -- so airship meeting minutes go to airship-discuss (if they want that) | 09:54 |
| aspiers | in fact, if it's the *only* email on that topic during a given week, then it's even more valuable to send it | 09:54 |
| mordred | yeah. that's probably the hardest part of the game | 09:54 |
| mordred | figuring out the meeting/ml mapping | 09:54 |
| corvus | maybe we could add it as metadata to the irc-meetings repo | 09:55 |
| aspiers | that's just what I was typing :) | 09:55 |
| mordred | ++ | 09:55 |
| mordred | speaking of - we should probably update the bot to be opendev | 09:55 |
| corvus | oh you know, we have all the pieces in place to do the limnoria stuff now | 09:56 |
| mordred | oh neat! | 09:56 |
| corvus | so maybe we can do that together | 09:56 |
| aspiers | oh interesting | 09:56 |
| aspiers | in Denver notmyname was showing ttx and I how much harder we were making our lives by using python's irc module for ptgbot | 09:57 |
| *** dpawlik has quit IRC | 09:58 | |
| corvus | yeah, we have a spec about consolodating things: http://specs.openstack.org/openstack-infra/infra-specs/specs/irc.html | 09:59 |
| corvus | we should be able to do all the bots as limnoria plugins | 09:59 |
| aspiers | sounds nice | 09:59 |
| corvus | should mostly still be relevant, just add 'ptgbot' to the list, and s/openstack/opendev/' | 10:00 |
| corvus | apparently i should go get on another plane now. ttyl. | 10:03 |
| *** dpawlik has joined #openstack-infra | 10:03 | |
| *** faizy98 has joined #openstack-infra | 10:06 | |
| *** tesseract has quit IRC | 10:12 | |
| *** tesseract has joined #openstack-infra | 10:12 | |
| *** bhavikdbavishi has quit IRC | 10:17 | |
| *** pkopec has quit IRC | 10:24 | |
| *** pkopec has joined #openstack-infra | 10:24 | |
| aspiers | mordred (since corvus is on a plane now): Is there any point proposing an infra-spec for auto-emailing IRC meeting minutes to -discuss if I'm gonna struggle to find the time to actually implement it? | 10:25 |
| aspiers | Strikes me this could maybe be handled by a zuul periodic job, or by the meetbot sending immediately on #endmeeting (assuming it has access to an MDA), or by the meetbot adding the minutes to a queue which a zuul periodic job processes asynchronously | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Fix missing check if logger is None https://review.opendev.org/659262 | 10:27 |
| *** zigo has quit IRC | 10:30 | |
| *** yolanda_ has joined #openstack-infra | 10:39 | |
| *** zigo has joined #openstack-infra | 10:41 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:41 | |
| *** udesale has quit IRC | 11:08 | |
| *** udesale has joined #openstack-infra | 11:09 | |
| *** roman_g has quit IRC | 11:29 | |
| *** roman_g has joined #openstack-infra | 11:33 | |
| *** calbers has joined #openstack-infra | 11:35 | |
| *** bhavikdbavishi has quit IRC | 11:50 | |
| *** nicolasbock has joined #openstack-infra | 11:54 | |
| openstackgerrit | Adam Spiers proposed opendev/infra-specs master: Add spec for sending IRC meeting minutes via emails https://review.opendev.org/659273 | 11:58 |
| *** rh-jelabarre has joined #openstack-infra | 11:59 | |
| *** dpawlik has quit IRC | 11:59 | |
| *** panda is now known as panda|launch | 12:01 | |
| *** janki has quit IRC | 12:04 | |
| *** spsurya has quit IRC | 12:08 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 12:17 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id https://review.opendev.org/658895 | 12:18 |
| *** bhavikdbavishi has joined #openstack-infra | 12:19 | |
| *** jamesmcarthur has joined #openstack-infra | 12:19 | |
| *** jpich has quit IRC | 12:20 | |
| *** jpich has joined #openstack-infra | 12:21 | |
| *** dpawlik has joined #openstack-infra | 12:21 | |
| *** dpawlik has quit IRC | 12:21 | |
| *** mriedem has joined #openstack-infra | 12:23 | |
| *** rh-jelabarre has quit IRC | 12:28 | |
| *** jamesmcarthur has quit IRC | 12:29 | |
| *** dpawlik has joined #openstack-infra | 12:30 | |
| *** _erlon_ has joined #openstack-infra | 12:35 | |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** tdasilva has joined #openstack-infra | 12:37 | |
| *** rh-jelabarre has joined #openstack-infra | 12:38 | |
| *** rh-jelabarre has quit IRC | 12:42 | |
| *** rh-jelabarre has joined #openstack-infra | 12:44 | |
| *** janki has joined #openstack-infra | 12:44 | |
| openstackgerrit | Adam Spiers proposed opendev/infra-specs master: Fix reference to StoryBoard project https://review.opendev.org/659285 | 12:46 |
| *** jamesmcarthur has joined #openstack-infra | 12:50 | |
| *** aaronsheffield has joined #openstack-infra | 12:51 | |
| aspiers | infra-root: Ah, there's an issue with tox.ini in infra-specs where it's trying to install Sphinx 2 with Python 2.7.15. I recently saw this in os-resource-classes and fixed it: https://review.opendev.org/#/c/657815/ | 12:55 |
| aspiers | I wonder how many other projects are affected | 12:55 |
| aspiers | stephenfin: ^^^ | 12:56 |
| aspiers | Here's the error http://logs.openstack.org/73/659273/1/check/openstack-tox-docs/9e941cc/tox/docs-1.log | 12:56 |
| *** rfarr_ has joined #openstack-infra | 13:00 | |
| smcginnis | aspiers: The requirements.txt for that repo probably needs to be updated to match the latest global-requirements.txt in openstack/requirements. | 13:03 |
| smcginnis | Or rather, the doc/requirements.txt | 13:03 |
| smcginnis | But odd that it would still be running under py2. All docs jobs should be running under py3 now. | 13:03 |
| *** panda|launch is now known as panda | 13:05 | |
| smcginnis | Ah, opendev so not using upper-constraints or the global requirements. Also still using the deprecated oslosphinx and not using py3 for basepython. | 13:08 |
| *** bhavikdbavishi has quit IRC | 13:10 | |
| openstackgerrit | Sean McGinnis proposed opendev/infra-specs master: Move from oslosphinx to openstackdocstheme https://review.opendev.org/659290 | 13:15 |
| *** dangtrinhnt has quit IRC | 13:15 | |
| *** jchhatbar has joined #openstack-infra | 13:15 | |
| *** dangtrinhnt has joined #openstack-infra | 13:16 | |
| *** janki has quit IRC | 13:17 | |
| *** lseki has joined #openstack-infra | 13:19 | |
| *** _Cyclone_ has quit IRC | 13:26 | |
| *** yamamoto has joined #openstack-infra | 13:27 | |
| *** _Cyclone_ has joined #openstack-infra | 13:29 | |
| *** Goneri has joined #openstack-infra | 13:30 | |
| AJaeger | clarkb: could you review https://review.opendev.org/659290 for infra-specs, please? | 13:38 |
| AJaeger | smcginnis: thanks! | 13:38 |
| smcginnis | np :) | 13:38 |
| *** hwoarang has quit IRC | 13:44 | |
| mordred | AJaeger: we should probably make an opendevdocstheme - but we should probably figure out publishing opendev related docs to an opendev.org domain first :) | 13:47 |
| fungi | johnsom: yes, we're trying to work out the dns query failures for nodes in vexxhost-sjc1. we suspect it's related to forwarding queries over ipv6 so are coming up with a solution to force them over v4 there to see if it makes any difference (the issue cropped up around the time ipv6 was added to that region) | 13:49 |
| fungi | dangtrinhnt: i had it flagged to catch up with you on the sb migration, i was just on post-summit vacation until yesterday but did see that you finally got control of the corresponding lp projects so that we can move forward. do you need to schedule a time to switch over or should i just start importing the old lp bugs into sb as soon as possible? | 13:51 |
| AJaeger | mordred: yes, nice idea... | 13:51 |
| johnsom | fungi Thanks for the followup, yes, it looks like it was querying over IPv6 | 13:51 |
| mordred | fungi: corvus and I landed https://review.opendev.org/659181 this morning - so hopefully that helps the dns | 13:51 |
| AJaeger | mordred: or a subtheme like we have 'starlingxdocs' as option already for the package openstackdocstheme | 13:52 |
| mordred | AJaeger: yah. impl details can be sorted out :) | 13:52 |
| fungi | sean-k-mooney: i think mriedem already had a bug for grenade jobs failing to get the constraints file from git.o.o (which i think also points to another problem, it ought to use a copy from disk so that gating and depends-on work correctly, not tell pip to retrieve it over http) | 13:52 |
| mriedem | fungi: http://status.openstack.org/elastic-recheck/#1829024 ? | 13:53 |
| sean-k-mooney | i closed teh one i opened this morning | 13:54 |
| sean-k-mooney | assuming the ip change fixed it | 13:54 |
| *** jcoufal has joined #openstack-infra | 13:55 | |
| sean-k-mooney | looks like there have been no new hits since 10:30 this morning | 13:55 |
| sean-k-mooney | fungi: the failure that was happen was in the tempest config gernation | 13:56 |
| sean-k-mooney | we do that via a tox env | 13:56 |
| sean-k-mooney | and we dont override the tox env to use a local copy of the file | 13:57 |
| fungi | mordred: oh, great! | 13:57 |
| sean-k-mooney | we could be we just dont | 13:57 |
| *** Wasaac_ has joined #openstack-infra | 13:58 | |
| *** Wasaac_ is now known as Wasaac | 13:59 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove legacy-python-barbicanclient-dsvm https://review.opendev.org/659295 | 14:02 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: WIP Remove legacy-python-barbicanclient-* https://review.opendev.org/659295 | 14:06 |
| fungi | gerrit 3.0.0 released today https://groups.google.com/forum/#!topic/repo-discuss/27X1IFSgmo4 | 14:08 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Pin skopeo to unbreak skopeo+bubblewrap https://review.opendev.org/659296 | 14:08 |
| mordred | fungi: neat! | 14:08 |
| mordred | fungi: does that patch ^^ look like correct puppet for pinning to a specific version to you? | 14:09 |
| *** ykarel is now known as ykarel|away | 14:12 | |
| *** jbadiapa has quit IRC | 14:14 | |
| fungi | not sure, i'll have to hunt down an example. usually distros don't have more than one version of the same package name available so it's not something we've necessarily leveraged | 14:14 |
| mordred | fungi: yah. in this case I've verified that the older versions are in that ppa | 14:15 |
| *** bobh has joined #openstack-infra | 14:15 | |
| mordred | fungi: https://stackoverflow.com/questions/11614413/puppet-specifying-a-version-of-a-package-to-install seems to indicate that ensure should work with a specific package version | 14:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Pin skopeo to unbreak skopeo+bubblewrap https://review.opendev.org/659296 | 14:16 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Pin skopeo to unbreak skopeo+bubblewrap https://review.opendev.org/659296 | 14:17 |
| fungi | mordred: perfect, i agree that looks correct | 14:17 |
| mordred | yay! | 14:17 |
| mordred | the version is now hopefully correct too | 14:17 |
| mordred | tristanC, fungi: ^^ patch updated | 14:17 |
| mordred | fungi: the thinking here is that we shoudl really file a bug with skopeo and get upstream fixed, so pinning to the working skopeo for now unbreaks the image build jobs. if they decide not to fix the issue, the fallback would be to have zuul add some capabilities to the bwrap command | 14:18 |
| *** bobh has quit IRC | 14:19 | |
| fungi | makes sense, yes | 14:19 |
| AJaeger | Is zuul running amok? I see 782 jobs in periodic on http://zuul.openstack.org/status | 14:23 |
| AJaeger | and 264 in release-post | 14:24 |
| AJaeger | infra-root ^ | 14:24 |
| AJaeger | why is openstack/project-config in release-post? That looks wrong | 14:25 |
| *** jamesmcarthur has quit IRC | 14:25 | |
| mordred | AJaeger: that does look wrong - and I do not know why | 14:27 |
| fungi | grafana says we've been running ~1kjph since 09:00z | 14:27 |
| AJaeger | and some jobs are there since 24 hours? | 14:28 |
| *** Wasaac has quit IRC | 14:28 | |
| fungi | looks like we may not have had any node availability for today's periodic jobs to get assignments yet | 14:29 |
| fungi | based on the graph | 14:29 |
| clarkb | re release post could that be related to release work the openstack releaseteam was talking about yesterday? | 14:29 |
| fungi | likely, seems to have started trending upward ~6 hours ago | 14:29 |
| fungi | looks like we had a fair amount of node availability from 2200 to 0800 | 14:31 |
| *** dpawlik has quit IRC | 14:32 | |
| fungi | does seem like there's just a handful of tags still awaiting nodes in the release pipeline so hopefully release-post will start catching up | 14:33 |
| AJaeger | openstack/project-config should not be in release-post at all... | 14:33 |
| *** jchhatbar has quit IRC | 14:34 | |
| mordred | is this that thing where the project shows up there temporarily while it figures out if it should be there? | 14:34 |
| *** sarob has joined #openstack-infra | 14:35 | |
| AJaeger | mordred: project-config is there for 23 hours. | 14:35 |
| AJaeger | that's a long time of figuring out... | 14:36 |
| AJaeger | openstack/releases is top of release-post and jobs are succeeded, but Zuul is not noticiing. | 14:36 |
| AJaeger | Guess you might be right mordred that zuul is not processing the queue _at all_ ;( | 14:37 |
| AJaeger | logfiles for openstack/release jobs are all 24 hours old, so jobs finished but somehow Zuul is ignoring it ;( | 14:37 |
| *** armax has joined #openstack-infra | 14:37 | |
| fungi | yeah, confirmed, the logs from that openstack/release job are a day old | 14:37 |
| fungi | was just noticing that | 14:38 |
| *** pgaxatte has quit IRC | 14:38 | |
| mordred | oh - well that's not awesome | 14:39 |
| fungi | other pipelines are being processed though, seems like | 14:39 |
| * AJaeger just informed release team to not release anything right now.... | 14:40 | |
| AJaeger | yes, check, gate, post are - periodic seems not | 14:40 |
| clarkb | could it be waiting on a specific bode type? | 14:41 |
| clarkb | *node | 14:41 |
| *** sarob has quit IRC | 14:41 | |
| AJaeger | clarkb: for openstack/release all jobs are finished since 24 hours, it should not wait... | 14:42 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove legacy-python-barbicanclient-* https://review.opendev.org/659295 | 14:45 |
| *** dpawlik has joined #openstack-infra | 14:45 | |
| *** ykarel|away is now known as ykarel | 14:48 | |
| *** icarusfactor has joined #openstack-infra | 14:48 | |
| *** dpawlik has quit IRC | 14:49 | |
| *** factor has quit IRC | 14:50 | |
| *** bobh has joined #openstack-infra | 14:52 | |
| clarkb | AJaeger: infra-root the lack of jobs under the chagnes in release-post imply that they haven't been merged yet I think | 14:53 |
| clarkb | it is the merge step that updates their configs and checks if there is any work to do | 14:53 |
| clarkb | grafana says our merger queues look fine though | 14:56 |
| fungi | maybe they represent unmergeable states? | 14:57 |
| fungi | though zuul should just report in that case | 14:57 |
| *** iurygregory has quit IRC | 14:57 | |
| *** iurygregory_ has joined #openstack-infra | 14:57 | |
| fungi | so unlikely | 14:57 |
| clarkb | http://paste.openstack.org/show/751427/ possible that is a clue? (from scheduelr debug log) | 14:57 |
| AJaeger | and tripleo-ui is top of release pipeline... | 14:59 |
| clarkb | https://review.opendev.org/#/c/642857/ change that updated the code that is breaking is from late march | 14:59 |
| AJaeger | yes, https://opendev.org/openstack/tripleo-ui/commit/2620fa03c24a0114a144ad01f550f757c59b8d76 is the tag for 7.4.10 | 14:59 |
| clarkb | so we've run with it for a bit | 14:59 |
| *** hwoarang has joined #openstack-infra | 15:00 | |
| mordred | clarkb: that does seem like the type of thing that could cause something to get unhappy - but I would have expected it to break before now? | 15:00 |
| clarkb | mordred: the code only trips if there are artifacts | 15:01 |
| clarkb | could be that tripleo has added artifacts to their non change pipelines | 15:01 |
| mordred | oh - yeah- like using the same job in a change and a periodic | 15:02 |
| fungi | (and release) | 15:03 |
| fungi | (and release-post) | 15:03 |
| mordred | yeah | 15:03 |
| fungi | i wonder if the pipeline handler is indefinitely fouled now, or whether we could just dequeue those refs | 15:04 |
| clarkb | fungi: it is persistently trying to process that one ref | 15:04 |
| mordred | maybe we could dequeue that one ref and unstick it? | 15:05 |
| clarkb | we should probably try to figure out what artifact has been zuul_returned | 15:05 |
| clarkb | if this is in the release jobs now we'll just trip on the next one | 15:06 |
| mordred | yeah. that's a very good point | 15:07 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-javascript-output/tasks/main.yaml#L144-L150 this is it for that item | 15:11 |
| clarkb | https://opendev.org/zuul/zuul-jobs/commit/1814e8b316bf32226b874f25195d0ff2916bf016 from that change | 15:11 |
| *** sarob has joined #openstack-infra | 15:13 | |
| *** ociuhandu has joined #openstack-infra | 15:13 | |
| fungi | okay, so likely impacts any project generatnig js packages? | 15:14 |
| clarkb | and maybe sphinx tarballs? https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-sphinx-tarball/tasks/html.yaml#L31-L43 | 15:15 |
| clarkb | unrelated: I've confirmed we are using ipv4 dns in sjc1 now | 15:15 |
| *** tosky has quit IRC | 15:16 | |
| openstackgerrit | Adam Spiers proposed opendev/infra-specs master: Add spec for sending IRC meeting minutes via emails https://review.opendev.org/659273 | 15:18 |
| *** tosky has joined #openstack-infra | 15:18 | |
| openstackgerrit | Adam Spiers proposed opendev/infra-specs master: Fix reference to StoryBoard project https://review.opendev.org/659285 | 15:18 |
| clarkb | mordred: fungi so ya I think we can deuqueue that one item | 15:21 |
| clarkb | and see if that unsticks things | 15:21 |
| *** liuyulong has quit IRC | 15:21 | |
| *** rfolco|ruck is now known as rfolco | 15:22 | |
| *** serhatd_ has quit IRC | 15:23 | |
| *** panda is now known as panda|ruck | 15:26 | |
| *** ykarel is now known as ykarel|away | 15:28 | |
| clarkb | I've fixed virtualenv on lists.o.o | 15:28 |
| fungi | on a positive note, it seems https://github.com/go-gitea/gitea/pull/6758 "Ignore non-standard refs in git push" merged yesterday so will likely wind up in the next release | 15:28 |
| mordred | \o/ | 15:29 |
| *** dpawlik has joined #openstack-infra | 15:30 | |
| clarkb | `zuul dequeue --tenant openstack --pipeline release --project openstack/tripleo-ui --ref refs/tags/7.4.10` how does that look? | 15:31 |
| fungi | looks like how i've done it in the past | 15:32 |
| clarkb | mordred: fungi AJaeger ^ do we think that is still a useful thing to try? If so I will run it momentarily | 15:33 |
| AJaeger | clarkb: it can't make it worse ;) Alternative is restarting Zuul, isn't it? | 15:33 |
| clarkb | yes I think that would be the alternative | 15:33 |
| fungi | it can't hurt. the only other thing i can think of to try is to restart the scheduler and then reenqueue everything, right | 15:33 |
| clarkb | ok I'll try it now | 15:33 |
| *** dpawlik has quit IRC | 15:34 | |
| clarkb | I'll probably work on a change to zuul to fix that bug | 15:35 |
| clarkb | once I've settled into my day a bit more | 15:35 |
| *** tesseract has quit IRC | 15:35 | |
| *** yboaron_ has quit IRC | 15:35 | |
| *** ykarel|away has quit IRC | 15:36 | |
| AJaeger | thanks, clarkb | 15:37 |
| AJaeger | release-post is nearly empty, openstack/releases still in there | 15:38 |
| AJaeger | tag only one entry - so, things are moving forward... | 15:38 |
| clarkb | yup seems to be processing things | 15:39 |
| clarkb | we should watch it to check if it gets stuck again | 15:39 |
| fungi | i'll try to keep it up on screen here | 15:39 |
| AJaeger | openstack/releases in release-post is not moving forward - that might need some dequeue as well | 15:40 |
| AJaeger | rest looks fine | 15:40 |
| *** mattmceuen has left #openstack-infra | 15:40 | |
| AJaeger | periodic is moving forward again... | 15:42 |
| AJaeger | config-core, part of my spring cleanup to remove some unused jobs: https://review.opendev.org/659295 - reviews, welcome, please | 15:42 |
| AJaeger | oh, release-post is moving forward now... | 15:43 |
| AJaeger | no action needed for openstack/releases | 15:43 |
| fungi | looks like we've got periodic jobs enqueuing now too | 15:43 |
| fungi | and yeah, the remaining job for openstack/releases in release-post is running | 15:44 |
| *** dpawlik has joined #openstack-infra | 15:46 | |
| fungi | i should duck out and find some lunch before something else breaks (or before i manage to break anything myself) | 15:47 |
| *** sarob has quit IRC | 15:48 | |
| AJaeger | ;) | 15:49 |
| fungi | looks like the openstack/releases job just finished | 15:49 |
| fungi | so now just waiting for the periodic pipeline to burn down | 15:49 |
| *** iurygregory_ has quit IRC | 15:50 | |
| *** e0ne has quit IRC | 15:50 | |
| *** dpawlik has quit IRC | 15:50 | |
| *** e0ne has joined #openstack-infra | 15:50 | |
| AJaeger | fungi: don#t wait for it - it runs for over 3hours ... | 15:51 |
| fungi | oh, i'm not waiting for it, i just mean that's all we still seem to have outstanding | 15:51 |
| prometheanfire | proposal bot race? | 15:53 |
| prometheanfire | https://review.opendev.org/658636 | 15:53 |
| prometheanfire | I renamed the topic about a min before it pushed a new change | 15:53 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Handle artifacts on non change types. https://review.opendev.org/659329 | 15:54 |
| AJaeger | might be - we check for old change first, then do some processing... | 15:54 |
| prometheanfire | it also redid the topic :D | 15:54 |
| AJaeger | prometheanfire: it pushes with the topic... | 15:55 |
| prometheanfire | ya, imagined | 15:55 |
| AJaeger | prometheanfire: you need to look at the log file for the proposal job and see timing... | 15:55 |
| prometheanfire | I just need to figure out how to roll it back to patchset 4 :| | 15:56 |
| AJaeger | prometheanfire: git review -d X,4;git review | 15:56 |
| prometheanfire | ah, cool | 15:56 |
| AJaeger | where X is the change - 658636 | 15:56 |
| *** ccamacho has quit IRC | 15:56 | |
| prometheanfire | right | 15:56 |
| AJaeger | tell me whether it worked ;) | 15:57 |
| clarkb | lists.o.o is happy now with puppet (no more complaining about that fact) | 15:57 |
| cmurphy | clarkb: what was the fix? reinstalling python-virtualenv? | 15:59 |
| prometheanfire | fatal: couldn't find remote ref refs/changes/36/658636/4 | 15:59 |
| clarkb | cmurphy: I installed it from pypi but ya | 15:59 |
| cmurphy | ah | 16:00 |
| cmurphy | k so the last one is ask.o.o :) | 16:00 |
| clarkb | yup was going to sync with ianw on that one as he volunteered to do the server upgrade on it | 16:00 |
| *** pkopec has quit IRC | 16:01 | |
| clarkb | wanted to make sure we don't step on toes | 16:01 |
| *** imacdonn has quit IRC | 16:01 | |
| AJaeger | prometheanfire: works for me... | 16:01 |
| AJaeger | prometheanfire: shall I push it up? | 16:02 |
| prometheanfire | sure | 16:02 |
| prometheanfire | you ran the git review command from a branch off master (or master)? | 16:02 |
| AJaeger | no new changes ;( Will update commit message... | 16:03 |
| prometheanfire | that's what I was getting at first | 16:03 |
| AJaeger | prometheanfire: should be fine | 16:03 |
| AJaeger | prometheanfire: my change is pushed and comparing with patch set 4, only the ":" added to commit message | 16:04 |
| *** pkopec has joined #openstack-infra | 16:04 | |
| openstackgerrit | Colleen Murphy proposed opendev/system-config master: Update ask.openstack.org to puppet 4 https://review.opendev.org/647877 | 16:04 |
| *** kopecmartin is now known as kopecmartin|off | 16:05 | |
| *** rpittau is now known as rpittau|afk | 16:06 | |
| prometheanfire | AJaeger: thanks, I merged some of the stuuff from 5 into 6 as well (might as well use it since it ran) | 16:09 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 16:11 |
| AJaeger | prometheanfire: your playground ;) | 16:11 |
| prometheanfire | yarp | 16:11 |
| mordred | infra-root: I'm AFK for the rest of the day | 16:13 |
| *** tosky has quit IRC | 16:13 | |
| *** imacdonn has joined #openstack-infra | 16:13 | |
| *** xek has quit IRC | 16:14 | |
| *** sarob has joined #openstack-infra | 16:16 | |
| *** ijw has joined #openstack-infra | 16:18 | |
| *** diablo_rojo has joined #openstack-infra | 16:22 | |
| *** yamamoto has quit IRC | 16:22 | |
| *** tesseract has joined #openstack-infra | 16:24 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 16:25 |
| *** gfidente has quit IRC | 16:26 | |
| *** gfidente has joined #openstack-infra | 16:28 | |
| *** ginopc has quit IRC | 16:31 | |
| *** dklyle has quit IRC | 16:31 | |
| *** lucasagomes has quit IRC | 16:36 | |
| *** dklyle has joined #openstack-infra | 16:37 | |
| *** jpich has quit IRC | 16:38 | |
| *** gyee has joined #openstack-infra | 16:38 | |
| *** sarob has quit IRC | 16:49 | |
| *** diga has joined #openstack-infra | 16:53 | |
| diga | Hello Everyone | 16:53 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused jobs https://review.opendev.org/659340 | 16:56 |
| *** ociuhandu has quit IRC | 16:57 | |
| clarkb | diga: hello | 16:58 |
| diga | clarkb: Hi | 16:58 |
| diga | clarkb: I am planning to start new project in OpenStack | 16:59 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused jobs https://review.opendev.org/659340 | 16:59 |
| diga | clarkb: Can you let me know how I can get required infra from OpenSTack like github, zuul , gerrit, storyboard etc | 16:59 |
| *** ociuhandu has joined #openstack-infra | 16:59 | |
| clarkb | diga: https://docs.openstack.org/infra/manual/creators.html that is the documentation for the process | 17:00 |
| diga | clarkb: let me go through it | 17:00 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused jobs https://review.opendev.org/659340 | 17:00 |
| AJaeger | diga: what exactly do you mean with "new project in OpenStack"? There's some process for new projects outside of the docs, so please tell us a bit more first... | 17:01 |
| *** dtantsur is now known as dtantsur|afk | 17:01 | |
| diga | AJaeger: Hey | 17:02 |
| diga | AJaeger: wait a min, Prakash is joining | 17:02 |
| *** derekh has quit IRC | 17:02 | |
| *** ijw has quit IRC | 17:03 | |
| *** pramchan has joined #openstack-infra | 17:04 | |
| diga | AJaeger: clarkb : pramchan has joined | 17:04 |
| diga | let's discuss here | 17:04 |
| pramchan | Sure I am in thanks | 17:05 |
| diga | clarkb: AJaeger: Prakash and I are planning to start a new project | 17:05 |
| diga | AJaeger: Can you let us know what process we have to follow ? | 17:06 |
| AJaeger | diga: so, you can start as part of our infrastructure outside the "openstack" namespace. | 17:06 |
| diga | AJaeger: ok | 17:07 |
| AJaeger | diga, the rest of https://docs.openstack.org/infra/manual/creators.html applies | 17:07 |
| *** ricolin has quit IRC | 17:07 | |
| *** e0ne has quit IRC | 17:07 | |
| AJaeger | question is which namespace to take - that is a new thing, so might need an iteration or two. | 17:07 |
| pramchan | We would like to have it as part of Open Infrastructure project with Zuul as Project gating under name sapce OTF | 17:08 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused jobs https://review.opendev.org/659340 | 17:08 |
| diga | AJaeger: As pramchan said above, we would like to follow that process | 17:09 |
| pramchan | our two projects will be OTF/pilot for testing and later OTF/medhavi | 17:09 |
| diga | AJaeger: These two project we want to start with under the OTF namespace | 17:11 |
| pramchan | basic question is how do I create OTF namespace in opendev.og as an Organization | 17:14 |
| *** psachin has quit IRC | 17:15 | |
| diga | clarkb: AJaeger: as pramchan asked above, how we can achive it under opendev.org | 17:15 |
| pramchan | https://opendev.org/explore/organizations | 17:15 |
| pramchan | link https://opendev.org/explore/organizations | 17:16 |
| *** udesale has quit IRC | 17:21 | |
| *** bobh has quit IRC | 17:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused jobs https://review.opendev.org/659340 | 17:21 |
| AJaeger | diga: OTF? | 17:22 |
| diga | AJaeger: Yes I would like to create namespace as OTF | 17:22 |
| AJaeger | so, follow the doc but use "otf" - I don't see we use upper case and give a detailed commit message. | 17:23 |
| pramchan | thanks | 17:23 |
| pramchan | 'otf' should be fine | 17:23 |
| diga | AJaeger: Sure | 17:23 |
| AJaeger | We can discuss more in the review. Merging might take some time, we have not yet a process for a new namespace | 17:24 |
| *** ykarel|away has joined #openstack-infra | 17:24 | |
| diga | AJaeger: Yes. Sure | 17:24 |
| *** icarusfactor has quit IRC | 17:25 | |
| diga | AJaeger: I dont see option to create new namespace | 17:25 |
| fungi | in theory simply referencing the namespace in the project name when adding it to the projects.yaml should be sufficient, but i don't know whether we might still be missing something in manage-projects so we'll want to keep an eye on it | 17:26 |
| fungi | clarkb: ^ you may know more on that front | 17:26 |
| pramchan | Will creating a project 'otf' not create a new namespace to work with? | 17:26 |
| clarkb | fungi: I think the last update I sent said hold off on creating new namespaces | 17:27 |
| clarkb | because ya we have to make sure creating them works as expected | 17:27 |
| diga | pramchan: will create 'otf' namespace, will create medhavi under 'otf' namespace | 17:27 |
| fungi | clarkb: right, but we don't know for sure of any missing glue, i guess? | 17:27 |
| diga | AJaeger: clarkb: can you help us create 'otf' namespace, do you want me to push gerrit patch for this ? | 17:28 |
| clarkb | fungi: correct, someone needs to think it over and decide if we are ready or not | 17:28 |
| clarkb | fungi: I think the biggest issues are likley on the gitea side | 17:28 |
| clarkb | (if they still exist) | 17:28 |
| *** dklyle has quit IRC | 17:29 | |
| AJaeger | diga: just submit a review for the repos - and we check it ;) Nothing for you todo right now for the namespace | 17:29 |
| diga | AJaeger: Okay | 17:29 |
| diga | AJaeger: do you have any reference patch which I can refer for creating new project | 17:30 |
| AJaeger | diga: as you see clarkb and fungi discussing above, there's a couple of places to check, so expect that it takes some time... | 17:30 |
| clarkb | fungi: we stopped puppeting cgit so those should be fine | 17:30 |
| fungi | clarkb: before the opendev migration we created new projects in the inaugust namespace and i don't remember anything breaking, but my memory is far from infallible | 17:30 |
| AJaeger | diga: follow the document ;) | 17:30 |
| diga | AJaeger: okay | 17:30 |
| clarkb | fungi: correct, but then in the migration we found we didn't have support for namespace renaming all over | 17:30 |
| clarkb | fungi: so I think it is mostly an issue of ensuring we have support for that before we add on more work in that space | 17:31 |
| diga | clarkb: fungi: AJaeger: What's issue here creating new namespace in gitia | 17:31 |
| clarkb | diga: I think it may mostly come down to ensuring that https://review.opendev.org/#/c/655476/ is ready and merged | 17:32 |
| *** ykarel|away has quit IRC | 17:32 | |
| diga | clarkb: okay | 17:32 |
| *** gfidente is now known as gfidente|afk | 17:32 | |
| pramchan | gitea is sefhosted git and adding one namespace to add one or two projects should not impact the space availability I believe as I had spoken to Monty | 17:32 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused jobs https://review.opendev.org/659340 | 17:32 |
| clarkb | pramchan: it isn't a space concern | 17:33 |
| clarkb | pramchan: it is a management concern where we discovered we don't properly manager namespaces yet | 17:33 |
| clarkb | and we have changes to fix that, its just a matter of ensuring they are correct and merged | 17:34 |
| clarkb | that said | 17:34 |
| clarkb | we do have space concerns in general because we built the gitea nodes with only 40GB disks | 17:34 |
| clarkb | so in the near future we'll need to rebuild the cluster with bigger disks | 17:34 |
| pramchan | ok lets diga try out creating using templates you have pointed and see if it works for namspace plus pilot project to test the same | 17:35 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused jobs https://review.opendev.org/659340 | 17:36 |
| diga | clarkb: AJaeger: Thank you for your support and providing a fruitful info to us getting started | 17:37 |
| diga | clarkb: AJaeger I will work on the patch, will try to push new project patch tomorrow | 17:37 |
| diga | pramchan: Sure, will work it | 17:38 |
| pramchan | thanks let me drop out | 17:38 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Handle artifacts on non change types. https://review.opendev.org/659329 | 17:38 |
| *** mattw4 has joined #openstack-infra | 17:40 | |
| *** pramchan has quit IRC | 17:40 | |
| *** yamamoto has joined #openstack-infra | 17:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused jobs https://review.opendev.org/659340 | 17:43 |
| *** diga has quit IRC | 17:44 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Handle artifacts on non change types. https://review.opendev.org/659329 | 17:45 |
| *** mattw4 has quit IRC | 17:45 | |
| *** dpawlik has joined #openstack-infra | 17:47 | |
| AJaeger | clarkb, want to make the puppet-4 jobs voting? https://review.opendev.org/659203 | 17:47 |
| AJaeger | config-core, a couple of new repos: https://review.opendev.org/657764 https://review.opendev.org/656657 https://review.opendev.org/657862 , please review | 17:48 |
| clarkb | AJaeger: yes! | 17:48 |
| AJaeger | ;) | 17:48 |
| *** dklyle has joined #openstack-infra | 17:49 | |
| AJaeger | infra-root, octavia created in gerrit some new ACL experiment - using "Backport-Candidate". Change is https://review.opendev.org/657657 is anybody wants to review or veto... | 17:49 |
| *** yamamoto has quit IRC | 17:50 | |
| johnsom | AJaeger Thanks! | 17:51 |
| *** dpawlik has quit IRC | 17:51 | |
| clarkb | cmurphy: http://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb is a 404 now and is breaking our jobs? http://logs.openstack.org/96/659296/3/check/infra-puppet-apply-4-ubuntu-xenial/2014bbb/job-output.txt.gz#_2019-05-15_14_30_47_201871 | 17:53 |
| AJaeger | config-core, next change in spring cleanup for ozj: Please review https://review.opendev.org/659340 and https://review.opendev.org/659295 | 17:54 |
| * clarkb throws hands in the air | 17:54 | |
| clarkb | they really did remove that | 17:54 |
| *** jcoufal has quit IRC | 17:55 | |
| AJaeger | clarkb, cmurphy , did you see https://review.opendev.org/659221 ? Is that related? | 17:55 |
| clarkb | AJaeger: yes looks like it | 17:55 |
| cmurphy | yes ianw was working on it | 17:55 |
| cmurphy | other people seem to be having the same issue | 17:55 |
| clarkb | well ya because you aren't supposed to delete things like that | 17:56 |
| * clarkb goes to complain at puppet employees | 17:56 | |
| cmurphy | oh looks like they did it on purpose :( | 17:56 |
| cmurphy | https://tickets.puppetlabs.com/browse/CPR-680?focusedCommentId=657453&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-657453 | 17:56 |
| *** dklyle has quit IRC | 17:56 | |
| cmurphy | but ianw's fix to use the release package under pool/ should have worked i think :/ | 17:57 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Make puppet-4 beaker jobs voting https://review.opendev.org/659203 | 17:58 |
| AJaeger | should we revert this ? ^ | 17:58 |
| clarkb | cmurphy: http://logs.openstack.org/21/659221/1/check/system-config-zuul-role-integration-trusty/d623113/job-output.txt.gz#_2019-05-15_06_39_48_278853 | 17:58 |
| clarkb | cmurphy: I think what is happening is those archived packages point to the old repo | 17:58 |
| clarkb | AJaeger: I don't think we need to immediately since we are kind stuck in the water with a breeze until we fix it | 17:58 |
| cmurphy | ugh | 17:59 |
| clarkb | cmurphy: but maybe there is also more places to update | 17:59 |
| AJaeger | ok | 17:59 |
| clarkb | http://logs.openstack.org/21/659221/1/check/system-config-zuul-role-integration-trusty/d623113/ara-report/file/a2d9fdf7-2193-4442-90d1-9eda944547e1/#line-45 that seems like it did pull from the archive | 18:00 |
| cmurphy | maybe we need to explicitly remove the old apt repo | 18:01 |
| *** tesseract has quit IRC | 18:01 | |
| clarkb | cmurphy: in those test jobs there shouldn't be an old apt repo | 18:02 |
| clarkb | cmurphy: on our servers maybe | 18:02 |
| cmurphy | oh right | 18:02 |
| *** dpawlik has joined #openstack-infra | 18:02 | |
| *** dpawlik has quit IRC | 18:07 | |
| cmurphy | hmm and xenial only has puppet 3 so we can't just switch to the distro package | 18:08 |
| clarkb | thinking out loud here if puppet doesn't fix the archive package or restore the previous package locations: what do we do? I guess we could go back to puppet 3 (heh on the day where we finally get to puppet 4 across the baord) and ansible harder | 18:09 |
| clarkb | we could possibly make our own package archive and run sed over the ar files? | 18:09 |
| clarkb | we could upgrade to puppet 5? | 18:10 |
| fungi | odds are we're not running apt clean and could copy the old packages from one of the servers which was already upgraded | 18:10 |
| *** e0ne has joined #openstack-infra | 18:10 | |
| fungi | and... stick them somewhere | 18:10 |
| clarkb | I think our daily package updates does do a clean? or maybe its just autoremove | 18:11 |
| cmurphy | we could not use the release package and manually install the gpg key and apt source file | 18:11 |
| cmurphy | or does the repository not even work | 18:11 |
| *** rlandy is now known as rlandy|brb | 18:12 | |
| clarkb | fungi: is /var/cache/apt/archives the place to look? | 18:12 |
| *** e0ne has quit IRC | 18:13 | |
| fungi | yes | 18:13 |
| clarkb | If so I grabbed puppet-agent_1.10.14-1xenial_amd64.deb on ze01 and stuck it in my homedir | 18:14 |
| *** Goneri has quit IRC | 18:14 | |
| clarkb | I think it depends on other packages like facter and hiera though and I don't see packages for those on ze01 | 18:14 |
| fungi | hrm... | 18:14 |
| fungi | "The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, hiera and mcollective." | 18:14 |
| fungi | looks like they're bundled | 18:14 |
| clarkb | oh cool ok | 18:15 |
| fungi | i don't see separate facter or hiera packages installed on ze01 | 18:15 |
| clarkb | so ze01.openstack.org:/home/clarkb/puppet-agent_1.10.14-1xenial_amd64.deb may tide us over | 18:15 |
| fungi | so hopefully that one is sufficient | 18:15 |
| clarkb | 16MB so not tiny, but we can stick that on tarballs or somewhere and use that potentially | 18:15 |
| fungi | i know modernish versions of apt let you apt install a local file path and will automatically resolve package dependencies for you | 18:16 |
| fungi | it *may* also allow you to specify a url to a package, though i have no idea (never tried) | 18:16 |
| clarkb | and if thereare no deps dpkg -i to great success | 18:16 |
| fungi | worst case we curl and then apt install or dpkg -i | 18:16 |
| clarkb | yup | 18:17 |
| * clarkb makes a xenial container to install that package in | 18:17 | |
| *** dklyle has joined #openstack-infra | 18:20 | |
| clarkb | dpkg -i didn't complain in a xenial container and facter works | 18:20 |
| clarkb | I think that means crises is largely averted, just have t ofigure out how we want ot plumb that into places | 18:20 |
| clarkb | so I'm going to step away for a bit | 18:20 |
| *** jcoufal has joined #openstack-infra | 18:24 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for cinder DetachedInstanceError bug 1640920 https://review.opendev.org/659359 | 18:37 |
| openstack | bug 1640920 in Cinder "list manageable snapshots fails with lazy load error on snapshot model attr" [Undecided,New] https://launchpad.net/bugs/1640920 | 18:37 |
| *** sarob has joined #openstack-infra | 18:40 | |
| *** ab-a has quit IRC | 18:40 | |
| *** ianychoi has quit IRC | 18:42 | |
| *** wolke has quit IRC | 18:42 | |
| *** tosky has joined #openstack-infra | 18:43 | |
| *** rlandy|brb is now known as rlandy | 18:45 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Use pyyaml safe_load to avoid deprecation warning https://review.opendev.org/654454 | 18:46 |
| *** hwoarang has quit IRC | 18:50 | |
| *** hwoarang has joined #openstack-infra | 18:51 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for cinder DetachedInstanceError bug 1640920 https://review.opendev.org/659359 | 18:54 |
| openstack | bug 1640920 in Cinder "list manageable snapshots fails with lazy load error on snapshot model attr" [Undecided,New] https://launchpad.net/bugs/1640920 - Assigned to Gorka Eguileor (gorka) | 18:54 |
| *** jcoufal has quit IRC | 18:54 | |
| clarkb | fungi: in ianws change do you think I should stick my .deb in tarballs then we can install it directly from.there or find it on puppets archives aite and do it from there? | 19:00 |
| *** dklyle has quit IRC | 19:01 | |
| clarkb | hrm maybe wait for ianw to wake and we can have a brainstorm | 19:01 |
| fungi | yeah, can wait a bit | 19:02 |
| fungi | but if puppet still publishes it in their archives site that's reasonable | 19:02 |
| *** mattw4 has joined #openstack-infra | 19:09 | |
| *** mattw4 has quit IRC | 19:13 | |
| *** e0ne has joined #openstack-infra | 19:17 | |
| *** Goneri has joined #openstack-infra | 19:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Move sqlalchemy-migrate jobs in-tree https://review.opendev.org/659365 | 19:26 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 19:37 |
| clarkb | ianw: fungi cmurphy ^ that is a stab at installing the packages directly | 19:37 |
| clarkb | we'll have to sort out what to do with puppet 3 on trusty as that doesn't do the all in one install iirc | 19:37 |
| clarkb | (and maybe at this point we just drop that since we won't be building new trusty nodes (we hope) | 19:37 |
| clarkb | ianw: if you hate this workaround feel free to propose others, I just wanted to get test results so pushed a ps | 19:38 |
| clarkb | ianw: fungi cmurphy I think we have to update install_puppet.sh too (I'm a bit confused about what gets used when) | 19:41 |
| *** Weifan has joined #openstack-infra | 19:44 | |
| clarkb | ya test jobs continue to fail due to the install_puppet.sh problem | 19:47 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 19:48 |
| clarkb | that disables update_puppet_version.yaml since well we've updated puppet versions on all but ask | 19:48 |
| fungi | seems safe enough | 19:51 |
| clarkb | that was the only real use of install_puppet.sh that I could see | 19:52 |
| clarkb | so I think we will have effectively replaced install_puppet.sh with the install-puppet role at this point | 19:52 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source https://review.opendev.org/658835 | 19:56 |
| *** e0ne has quit IRC | 19:56 | |
| clarkb | ah ok the puppet noop apply tests also use install_puppet.sh | 19:56 |
| *** e0ne has joined #openstack-infra | 19:59 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 20:03 |
| clarkb | now trying to use ansible to do the puppet install for the apply tests | 20:03 |
| *** dpawlik has joined #openstack-infra | 20:03 | |
| *** pkopec has quit IRC | 20:07 | |
| cmurphy | clarkb: lgtm if zuul is happy with it | 20:07 |
| clarkb | I think this ps will fail beacuse I didn't use the path to ansible | 20:08 |
| clarkb | but I want it to prove that to me before pushing the next ps | 20:08 |
| *** dpawlik has quit IRC | 20:08 | |
| clarkb | then we'll likely have to drop trusty puppet 3 testing | 20:08 |
| clarkb | then I think we'll be close | 20:08 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 20:10 |
| clarkb | yup it didn't find the ansible executable, it should with latest ps | 20:10 |
| *** pcaruana has quit IRC | 20:14 | |
| clarkb | thinking about the skopeo problem is that something we might want to manually mitigate for now so that zuul isn't dead in the water? | 20:14 |
| clarkb | fungi: ^ were you around for that debugging? I don't fully have it paged in yet other than broken puppet is preventing fix for skopeo from getting in | 20:15 |
| clarkb | http://logs.openstack.org/21/659221/5/check/system-config-zuul-role-integration-trusty/6e8eadc/job-output.txt.gz#_2019-05-15_20_13_31_524305 exciting | 20:15 |
| *** rlandy has quit IRC | 20:15 | |
| clarkb | (I used https instead of http because there isn't a way to confirm the hash via the apt module) | 20:16 |
| *** rlandy has joined #openstack-infra | 20:16 | |
| *** dpawlik has joined #openstack-infra | 20:21 | |
| clarkb | pabelanger: dmsimard http://logs.openstack.org/21/659221/5/check/infra-puppet-apply-4-ubuntu-xenial/6ca1b0b/job-output.txt.gz#_2019-05-15_20_18_40_491614 any idea how I can set the ANSIBLE_ROLES_PATH on the command line while avoiding that error? | 20:23 |
| dmsimard | clarkb: I think include_rule is a typo | 20:23 |
| dmsimard | vs include_role | 20:23 |
| clarkb | oh that may be it thanks | 20:23 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 20:24 |
| *** whoami-rajat has quit IRC | 20:24 | |
| dmsimard | \o/ | 20:24 |
| clarkb | and now I need to eat tamales for lunch | 20:24 |
| *** dpawlik has quit IRC | 20:25 | |
| *** e0ne has quit IRC | 20:28 | |
| fungi | clarkb: yeah, back, just split between several things but can take a look now | 20:30 |
| *** diablo_rojo has quit IRC | 20:31 | |
| fungi | but sounds like dmsimard to the rescue anyway! ;) | 20:31 |
| clarkb | I got the role name wrong :/ | 20:31 |
| clarkb | it is puppet-install not install-puppet if you want to push a new ps update for tools/prep-apply.sh | 20:32 |
| clarkb | I can do it after lunch | 20:32 |
| dmsimard | sometimes all you need is another pair of eyes :D | 20:32 |
| *** sarob has quit IRC | 20:34 | |
| mriedem | clarkb: looks like the latest ipv4 thing for vexxhost-sjc1 is making the dns failure go away | 20:37 |
| clarkb | mriedem: ok that is data we can take back to mnaser (which is good) | 20:37 |
| *** dklyle has joined #openstack-infra | 20:39 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 20:41 |
| clarkb | mriedem: what that implies is that somewhere between sjc1 and google/cloudflare dns is an ipv6 problem | 20:41 |
| ianw | clarkb: hey ... just catching up | 20:44 |
| clarkb | ianw: sorry I kinda took things over there and stuff went weird | 20:45 |
| *** kjackal has quit IRC | 20:45 | |
| ianw | so like literally the day we finished migrating to puppet4, they EOL'd it :) | 20:45 |
| clarkb | ianw: but basically word on the puppet bug is you have to do a direct install of the package because the archives are not actual repos | 20:45 |
| clarkb | ianw: yup | 20:45 |
| clarkb | ianw: so I've updated your change to install the deb/rpm for puppet-agent (4) directly instead of using the repo package installer thing | 20:45 |
| clarkb | and have been iterating on that (had to pull install_puppet.sh out of our bootstrapping of test jobs for example) | 20:45 |
| ianw | yeah ... i don't know if they're open to discussion about that ... i mean yum is still working? | 20:46 |
| clarkb | ianw: is it? we don't test centos anymore and I have n't tested it manually | 20:46 |
| clarkb | but ya I'm open to other ideas on that change up to and including going back to the original ps | 20:46 |
| clarkb | (I left trusty puppet 3 alone so that will confirm if apt is able to work at all or not too) | 20:47 |
| ianw | one thought; if this is EOL and never updating; do we make our own little repo in afs? | 20:48 |
| clarkb | ianw: ya another idea I had similar to that is stick the puppet-agent pacakges we need on tarballs and dpkg -i from there | 20:49 |
| clarkb | for puppet 4 at least the puppet-agent package should contain everything we need in it (and the tests should confirm if I can ever get them to install puppet successfully in the first place) | 20:49 |
| ianw | i'm just thinking that is closer to status quo to s/apt.puppetlabs.com/tarballs.openstack.org/ than try to add dpkg calls | 20:50 |
| *** ijw has joined #openstack-infra | 20:50 | |
| clarkb | ya | 20:51 |
| ianw | although, we don't have the release package which was doing all this | 20:51 |
| ianw | i mean installing the repo files | 20:51 |
| clarkb | and ansible's apt module makes dpkg -i easy | 20:51 |
| *** ijw has quit IRC | 20:52 | |
| *** ijw has joined #openstack-infra | 20:52 | |
| clarkb | dmsimard: http://logs.openstack.org/21/659221/7/check/infra-puppet-apply-4-ubuntu-xenial/fc67363/job-output.txt.gz#_2019-05-15_20_51_32_619856 is there a way to have the ansible command gather facts before running the listed module? | 20:54 |
| *** Weifan has quit IRC | 20:57 | |
| clarkb | I may need a playbook for this afterall | 20:57 |
| ianw | hmm, does a "-m setup -m include_role" work ... i.e. multiple -m's? | 20:58 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 20:59 |
| clarkb | ianw: I couldn't find any evidence of that | 20:59 |
| clarkb | playbook is simple enough | 21:00 |
| ianw | yeah, local testing says "no" :) | 21:00 |
| *** Weifan has joined #openstack-infra | 21:01 | |
| *** diablo_rojo has joined #openstack-infra | 21:01 | |
| notmyname | aspiers: just saw IRC buffer playback. the IRC bot framework I use (the one I was showing in Denver) is https://sopel.chat | 21:03 |
| fungi | it's easy enough to generate our own releases file with apt-ftparchive (i have a half-dozen-line shell recipe i use to make very simple apt repositories), but we'd need to so our own signing and switch the key we're trusting | 21:03 |
| aspiers | notmyname: yeah, I managed to recall that in an earlier conversation with ttx. It looks nice although http://specs.openstack.org/openstack-infra/infra-specs/specs/irc.html suggests a different direction | 21:04 |
| ianw | fungi: yeah and to be a drop-in replacement, we'd need to build our own -release.deb that installs the repos files too | 21:04 |
| ianw | fungi: and, with that done, i bet people would find it and we'd become a defacto repository for all sorts of interesting users ;) | 21:05 |
| fungi | aspiers: notmyname: the main reason for the irc spec pushing limnoria is that we don't want to have to rewrite the meetbot plugin for supybot from scratch, and it should work in limnoria (as a continuation of the old supybot codebase) | 21:05 |
| *** Weifan has quit IRC | 21:06 | |
| clarkb | fungi: aspiers and it supports working around channel limits | 21:06 |
| ianw | upstream probably should have 1) removed the release files on archive site 2) just dumped everything in a directory that doesn't look confusingly like a apt repo but isn't | 21:06 |
| clarkb | bah if you use ansible then localhost just works but if you use ansible-playbook you have to specify a thing for that? | 21:06 |
| * clarkb works on a new patch | 21:06 | |
| *** stevebaker is now known as steveb | 21:07 | |
| ianw | clarkb: -i 'localhost,' ? | 21:07 |
| *** Goneri has quit IRC | 21:07 | |
| *** Goneri has joined #openstack-infra | 21:07 | |
| *** ralonsoh has quit IRC | 21:07 | |
| *** steveb is now known as stevebaker | 21:07 | |
| notmyname | aspiers: fungi: I'm not pushing for one or the other. only sharing what's worked really well for me. the entirety of my bot is https://github.com/notmyname/Patches2/blob/master/patchbot.py. It's insanely simple | 21:07 |
| fungi | notmyname: yep, i didn't mean to suggest you were pushing anything. i agree it's an amazing framework if you want to write a bot from scratch | 21:08 |
| clarkb | ianw: will that work? | 21:08 |
| clarkb | ianw: I was making a tmpfile with explicit local connection | 21:09 |
| ianw | clarkb: yeah, that's how i usually test -- /ansible-env/bin/ansible-playbook -i 'localhost,' test.yaml | 21:09 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 21:10 |
| clarkb | ianw: k can clean that up in followup. Mostly I want to see that this puppet-agent install is viable or not | 21:10 |
| ianw | you gotta get that comma though, otherwise it thinks it's a file | 21:10 |
| *** panda|ruck has quit IRC | 21:10 | |
| clarkb | oh wait there is a bug in that ps anyway | 21:10 |
| clarkb | so ya I'll use your thing | 21:10 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 21:11 |
| *** gfidente|afk has quit IRC | 21:12 | |
| *** panda has joined #openstack-infra | 21:14 | |
| clarkb | ianw: I like how trusty fails due to sni | 21:14 |
| *** dklyle has quit IRC | 21:14 | |
| clarkb | ianw: now it fails because it tries to ssh :) | 21:16 |
| clarkb | I think we have to do the annotation that says local connection | 21:16 |
| ianw | hrm, i don't have ssh disabled, i wonder if i did something in a config a long time ago | 21:18 |
| ianw | sshd enabled i mean | 21:18 |
| ianw | on localhost | 21:18 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 21:21 |
| clarkb | back to inventory file without previous issue | 21:21 |
| *** Lucas_Gray has joined #openstack-infra | 21:22 | |
| ianw | i think -i 'localhost,' --connection=local would work | 21:24 |
| clarkb | heh now it says unsupported puppet version on this platform | 21:26 |
| ianw | yeah that does work with "hosts: all" ... anyway, not important | 21:27 |
| clarkb | (I'm guessing all this pain is why we were still using install_puppet.sh for this) | 21:27 |
| *** dklyle has joined #openstack-infra | 21:29 | |
| clarkb | wow ok I think this is happening beacuse I supplied puppet_install_version on the command line so the value 4 is a string but we check against ints | 21:29 |
| clarkb | I can apparently use json to address this | 21:31 |
| fungi | and who doesn't love a bit of json every now and then? | 21:31 |
| fungi | it's so handy on the command line | 21:31 |
| ianw | oh wow ... <headslap emjoii> | 21:31 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 21:33 |
| clarkb | it is not pretty to read but maybe finally this will work | 21:33 |
| cmurphy | dear lord | 21:34 |
| clarkb | at least it managed to run ansible against the correct distribution type | 21:36 |
| clarkb | so that was progress | 21:36 |
| *** Weifan has joined #openstack-infra | 21:36 | |
| clarkb | (however still have to fix the jobs so that we can merge the skopeo fix so that we can merge the zuul fix for the release issue) | 21:38 |
| *** Weifan has quit IRC | 21:40 | |
| clarkb | grr I'm missing a comma somewhere | 21:43 |
| clarkb | but I thought json doesn't do trailing commas | 21:43 |
| clarkb | I see the issue | 21:46 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 21:46 |
| *** yamamoto has joined #openstack-infra | 21:49 | |
| *** yamamoto has quit IRC | 21:54 | |
| *** mattw4 has joined #openstack-infra | 21:56 | |
| clarkb | I think it is working now | 21:56 |
| clarkb | I'm going to take a break for a bit since it seems happier. I think people should review the change now and think about whether or not we can remove the trusty tests that fail due to sni | 21:58 |
| clarkb | hrm the beaker-rspec jobs also try to curl the pc1 release deb and install the repo that way | 21:59 |
| clarkb | is that via the helper repo? | 22:00 |
| cmurphy | it is probably part of the install_puppet helper or whatever it is | 22:00 |
| cmurphy | oh no actually we changed it to do install_puppet.sh | 22:00 |
| *** dklyle has quit IRC | 22:00 | |
| clarkb | ya lib/puppet-openstack_infra_spec_helper/spec_helper_acceptance.rb does install_puppet.sh exec | 22:01 |
| clarkb | so maybe we should consider just updating that script too? | 22:01 |
| clarkb | I was trying to avoid that but there is a lot ot untangle (as evidenced by my ~12 patchsets) | 22:01 |
| cmurphy | we could change spec_helper_acceptance to run your ansible command instead of the script | 22:02 |
| cmurphy | but either way same outcome | 22:02 |
| ianw | i gotta run kids to school but can look at that in a bit | 22:03 |
| clarkb | ya but then we also have to install ansible | 22:03 |
| cmurphy | the spec helper already does install ansible | 22:03 |
| cmurphy | because mordred | 22:03 |
| *** mattw4 has quit IRC | 22:03 | |
| clarkb | oh in that case maybe we should run ansible there too | 22:04 |
| cmurphy | https://opendev.org/opendev/puppet-openstack_infra_spec_helper/src/branch/master/lib/puppet-openstack_infra_spec_helper/spec_helper_acceptance.rb#L5 | 22:04 |
| *** xek has joined #openstack-infra | 22:07 | |
| *** stevebaker has quit IRC | 22:11 | |
| openstackgerrit | Clark Boylan proposed opendev/puppet-openstack_infra_spec_helper master: Install puppet with ansible https://review.opendev.org/659384 | 22:11 |
| clarkb | cmurphy: ^ something like that maybe | 22:11 |
| *** stevebaker has joined #openstack-infra | 22:12 | |
| cmurphy | clarkb: is that missing a depends-on? | 22:12 |
| clarkb | cmurphy: I think we need both changes in at the same time :/ | 22:13 |
| cmurphy | oh | 22:13 |
| clarkb | what we probably want to do is have system-config depends on the helper change and if that fixes the problem force merge the helper change then we can merge the system-config change | 22:13 |
| clarkb | I'm curious to see how it fails though. That should tell us if it is trying to do the right thing or not | 22:14 |
| *** Goneri has quit IRC | 22:14 | |
| *** rfarr_ has quit IRC | 22:16 | |
| *** rlandy is now known as rlandy|bbl | 22:18 | |
| clarkb | heh it doesn't actually run ansible | 22:18 |
| clarkb | is that beacuse there are no spec tests in the helper itself/ | 22:18 |
| * clarkb makes a depends on entry | 22:18 | |
| cmurphy | huh i guess not | 22:19 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 22:20 |
| clarkb | in theory that will install puppet with ansible as elsewhere then beaker jobs will work on puppet 4 and 3 | 22:21 |
| clarkb | then we can delete trusty jobs which we don't want to bother with and/or make trusty work too | 22:21 |
| *** dpawlik has joined #openstack-infra | 22:21 | |
| fungi | yeah, i'm fine bypassing ci on whichever one is simpler | 22:22 |
| *** sarob has joined #openstack-infra | 22:22 | |
| clarkb | openstackci-beaker-puppet-4 and puppet-beaker-rspec-puppet-4-infra-system-config are the jobs to watch | 22:24 |
| *** dpawlik has quit IRC | 22:26 | |
| clarkb | we install a ton of gems | 22:27 |
| cmurphy | the fog gem is a beast | 22:28 |
| clarkb | heh so we run base.yaml from the helper in beaker jobs and I think that will call puppet-install? | 22:29 |
| clarkb | bah I made the same json bug as before | 22:29 |
| clarkb | oh it won't beacuse of groups | 22:29 |
| clarkb | ok /me fixes | 22:30 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-openstack_infra_spec_helper master: Install puppet with ansible https://review.opendev.org/659384 | 22:30 |
| clarkb | ok infra spec help should have valid json now and I rechecked the other change | 22:31 |
| *** dpawlik has joined #openstack-infra | 22:37 | |
| *** aaronsheffield has quit IRC | 22:40 | |
| *** dpawlik has quit IRC | 22:42 | |
| *** tdasilva has quit IRC | 22:43 | |
| clarkb | I think it is working | 22:43 |
| fungi | so is it just 659384 and 659221 we need for this? | 22:45 |
| clarkb | yes though 659221 will need to be updated to drop the trusty tests that break or we need to further update puppet-install role to make those work | 22:45 |
| clarkb | so far this has been get it to work with puppet-4 on xenail to prove this option works | 22:46 |
| clarkb | and I think we may have finally gotten to that point | 22:46 |
| clarkb | yup 1 of 2 puppet 4 beaker jobs have passed | 22:49 |
| clarkb | so now the quetion becomes do we need system-config-zuul-role-integration-trusty and infra-puppet-apply-3-ubuntu-trusty ? | 22:50 |
| fungi | i vote no | 22:50 |
| clarkb | if yes we need to update roles/puppet-install to make trusty installs work. If no I think we just remove those jobs and cruise with broken trusty in puppet-install and remove trusty from puppet-install once all trusty nodes are gone | 22:50 |
| *** mriedem has quit IRC | 22:51 | |
| *** pkopec has joined #openstack-infra | 22:52 | |
| clarkb | and now 2/2 on puppet-4 beaker jobs | 22:52 |
| clarkb | going to let the last voting job finish | 22:53 |
| *** Weifan has joined #openstack-infra | 22:54 | |
| fungi | i don't see us getting much more use out of the trusty jobs anyway, so overhauling them at this late stage of life is going to be mostly wasted effort | 22:54 |
| clarkb | ya | 22:54 |
| clarkb | I can push a new patchset once this last job finishes that drops the trusty jobs | 22:54 |
| fungi | would rather let them walk off into the sunset | 22:54 |
| ianw | ++ on removing trusty, i don't think we have the bandwidth to bother with it | 22:55 |
| *** notmyname has quit IRC | 22:56 | |
| *** pkopec has quit IRC | 22:57 | |
| clarkb | does make me wonder why/how http://logs.openstack.org/21/659221/14/check/openstackci-beaker-ubuntu-trusty/6e221b6/job-output.txt.gz that worked | 22:57 |
| clarkb | something updated python/pip such that sni worked in that context but not elsewhere? | 22:58 |
| clarkb | I'm going to remove it | 22:58 |
| *** notmyname has joined #openstack-infra | 22:59 | |
| ianw | clarkb: hrm, it runs the base playbook first | 23:00 |
| *** tosky has quit IRC | 23:00 | |
| ianw | maybe we fix something in there | 23:00 |
| *** mattw4 has joined #openstack-infra | 23:01 | |
| openstackgerrit | Clark Boylan proposed openstack/openstack-zuul-jobs master: Remove infra trusty puppet jobs https://review.opendev.org/659395 | 23:01 |
| clarkb | ianw: ya I'm guessing it udpates pip or crpytography or something that makes sni work | 23:01 |
| ianw | ok this bumps the priority of ask upgrade (i mean it was before too) so i'll get onto that | 23:02 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Handle moved puppet repos https://review.opendev.org/659221 | 23:03 |
| clarkb | ok ^ removes the trusty jobs defined in that repo now and depends on the o-z-j change above to remove those from the template defined elsewhere | 23:03 |
| ianw | clarkb: can probably remove the "trusty" from Node-OS: in the manifest too? | 23:05 |
| clarkb | ianw: ya I think there are other jobs we can clean up too | 23:05 |
| *** tkajinam has joined #openstack-infra | 23:05 | |
| clarkb | this is more minimal to where things will work again change | 23:05 |
| clarkb | (part of that is I'm running out of steam after fighting fires all day) | 23:05 |
| clarkb | heh I updated .zuul.yaml so now way more jobs run | 23:06 |
| clarkb | lets hope those still work too :) | 23:06 |
| *** Adri2000 has quit IRC | 23:09 | |
| openstackgerrit | Merged opendev/puppet-openstack_infra_spec_helper master: Install puppet with ansible https://review.opendev.org/659384 | 23:09 |
| clarkb | while that churns I am going to take a break. ianw fungi how confident are we applying a change like that to all hosts late in the day (for fungi and myself)? | 23:10 |
| clarkb | it should mostly be a noop due to the check for /opt/puppetlabs | 23:10 |
| clarkb | and i think that for ask it is already broken? | 23:10 |
| *** sarob has quit IRC | 23:11 | |
| *** Lucas_Gray has quit IRC | 23:11 | |
| fungi | i will readily admit my bandwidth for troubleshooting this evening is minimal. christine came down with travel crud so i've been trying to juggle all the things and really need to get some yardwork knocked out with what little daylight remains here | 23:11 |
| clarkb | oh hrm ok so we have an issue | 23:11 |
| *** Adri2000 has joined #openstack-infra | 23:12 | |
| clarkb | because I updated .zuul.yaml it is running the docker image builds which need working skopeo | 23:12 |
| clarkb | so I think we will have to manually apply the skopeo fix to all executors, then merge these changes | 23:12 |
| clarkb | and considering that I feel like I'm running out of daylight | 23:12 |
| fungi | and we come full circle! ;) | 23:12 |
| fungi | i mean, we could merge it by temporarily disabling the docker image build jobs i guess | 23:13 |
| *** rcernin has joined #openstack-infra | 23:14 | |
| fungi | or just live with bypassing those and merging the change with the other jobs running | 23:14 |
| clarkb | we could also force merge the skopeo fix | 23:14 |
| fungi | fair | 23:14 |
| clarkb | considering that is already broken I'm a little more comfortable possibly breaking it harder :) | 23:14 |
| fungi | that one *was* working until puppet went away | 23:14 |
| clarkb | https://review.opendev.org/#/c/659296/ | 23:14 |
| fungi | i'm already +2 on it and okay with shoving it through | 23:15 |
| clarkb | fungi: are you able to push it in or should I? | 23:15 |
| fungi | i can, just a moment | 23:17 |
| ianw | ... not too up on the skopeo bits ... lmn if i can help | 23:18 |
| clarkb | ianw: basically skopeo updated which requires privileges to run but we run it unprivileged. Fix is to pin skopeo to older version while we either make zuul handle skopeo's new behavior or get upstream to fix skopeo | 23:18 |
| clarkb | ianw: issue is that we can't do the image copy/move dance without it aiui | 23:19 |
| clarkb | (its possible check will actjually work but not gate) | 23:19 |
| clarkb | its prevented zuul changes from merging all day and I found the puppet issues trying to fix the skopeo issues because I fixed a zuul issue that plugged up our release pripelines that I want to get merged so we can fix that issue | 23:20 |
| clarkb | https://i.imgur.com/t0XHtgJ.gif was my day | 23:21 |
| clarkb | so if anyone asks if we can fix the zuul bug that is totally what I'm doing lois | 23:21 |
| *** mattw4 has quit IRC | 23:21 | |
| fungi | oh! before i mash the big red button, there's probably one more option at our disposal | 23:21 |
| fungi | squash them | 23:21 |
| fungi | they *are* in the same repo, right? | 23:21 |
| clarkb | fungi: they are but we need the actual product executors to function | 23:22 |
| clarkb | which means squashing won't help I don't think | 23:22 |
| fungi | ahh, nevermind. so not actually self-testing in that regard | 23:22 |
| clarkb | skopeo is used to shuffle the images around | 23:22 |
| ianw | so to be clear; merge 659296 ... apply & restart executors? | 23:22 |
| clarkb | hrm the gitea job did succeed though | 23:22 |
| clarkb | so maybe this isn't strictly necessary? | 23:23 |
| clarkb | ianw: it would be merge change, make sure puppet installs the right version of skopeo then that is it | 23:23 |
| clarkb | (we don't need to restart the executors for this because skopeo is a forked process) | 23:23 |
| clarkb | fungi: so ya maybe we hold off and try to merge the system-config fix? | 23:23 |
| clarkb | and if that fails (like I'm worried then do the merge of skopeo fix?) | 23:24 |
| fungi | wfm | 23:24 |
| fungi | i'll leave my verify +2 staged so i can do it quickly if needed | 23:24 |
| clarkb | https://review.opendev.org/#/c/659221/15 +1'd so ya maybe we can just push things through per normal (and I must misunderstand some detail of the skopeo issue) | 23:25 |
| *** yamamoto has joined #openstack-infra | 23:26 | |
| clarkb | (or maybe skopeo has already fixed the issue upstream and we've installed that?) | 23:26 |
| clarkb | skopeo version 0.1.36-dev is what we have installed | 23:26 |
| *** ekultails has quit IRC | 23:27 | |
| clarkb | the ppa did build skopeo 2 hours ago | 23:27 |
| fungi | heh, automation solving our problems in the background | 23:27 |
| ianw | * autobuilt 30b0a17 | 23:28 |
| clarkb | ya apt history log shows it updated about 2.5 hours ago | 23:28 |
| clarkb | so maybe the skopeo issue is a non issue. I will recheck zuul change to see | 23:28 |
| fungi | https://github.com/containers/skopeo/issues/649 doesn't seem to reflect any fix though | 23:29 |
| clarkb | https://review.opendev.org/#/c/659329/3 has been rechecked as canary | 23:29 |
| ianw | i can't see anything suggesting a related fix in skoepo checkins | 23:29 |
| *** panda has quit IRC | 23:30 | |
| fungi | "Fix typo on the main man page" | 23:30 |
| ianw | yeah, just a man page fix https://github.com/containers/skopeo/commit/30b0a1741e0a5ecb97339c31ab32e74b72ceaa1e | 23:30 |
| fungi | that's the only thing committed this week | 23:30 |
| clarkb | huh | 23:30 |
| clarkb | ok so maybe the update is just noise | 23:31 |
| *** panda has joined #openstack-infra | 23:31 | |
| *** Weifan has quit IRC | 23:39 | |
| *** cloudnull has joined #openstack-infra | 23:42 | |
| *** d34dh0r53 has joined #openstack-infra | 23:43 | |
| clarkb | fungi: maybe you can review https://review.opendev.org/#/c/659395/ really quick before the end of the day then we can get 659221 in tomorrow? | 23:56 |
| clarkb | ianw: ^ your review on the second chagne there is much appreciated since you first ran into this stuff | 23:56 |
| fungi | lookin | 23:57 |
| *** cloudnull has quit IRC | 23:58 | |
| *** d34dh0r53 has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
