| *** ryohayakawa has joined #opendev | 00:02 | |
| *** spotz has joined #opendev | 00:25 | |
| ianw | kevinz: i think we might have some launch issues in arm64 cloud | 00:56 |
|---|---|---|
| ianw | kevinz: http://paste.openstack.org/show/795041/ | 01:10 |
| ianw | openstack.exceptions.SDKException: Error in creating the server (no further information available) | 01:10 |
| ianw | not very helpful | 01:10 |
| ianw | although then there's | 01:11 |
| ianw | 2020-06-22 01:07:16,656 ERROR nodepool.NodeLauncher: [e: 6db383fbde354429a12e752df7aec7e9] [node_request: 900-0009569244] [node: 0017298127] Detailed node error: MessagingTimeout | 01:11 |
| kevinz | Hi ianw | 01:32 |
| kevinz | Yes I see, I will dig to see the problem | 01:32 |
| *** DSpider has quit IRC | 01:36 | |
| ianw | kevinz: cool, LMN, it looks like it's pretty busy launching nodes | 01:48 |
| *** xiaolin has joined #opendev | 01:52 | |
| kevinz | ianw: looks recovered | 02:22 |
| ianw | kevinz: excellent :) thanks, was it a backend issue? | 02:27 |
| kevinz | ianw: yes, I see there is a problem with rabbitmq timeout, and I restart all the nova services and rabbitmq | 02:28 |
| kevinz | then it was recovered...Looks Restarting is the most effiective way :-D | 02:29 |
| *** sgw has quit IRC | 03:21 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Acutally run system-config arm64 test on an arm64 node https://review.opendev.org/735281 | 03:30 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: Use PPA for Xenial ARM64 https://review.opendev.org/735055 | 03:30 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: Use PPA for Xenial ARM64 https://review.opendev.org/735055 | 04:02 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Acutally run system-config arm64 test on an arm64 node https://review.opendev.org/735281 | 04:02 |
| *** sgw has joined #opendev | 04:58 | |
| *** sgw has quit IRC | 05:05 | |
| *** rpittau|afk is now known as rpittau | 06:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Fedora 32 support https://review.opendev.org/737217 | 06:36 |
| *** DSpider has joined #opendev | 06:47 | |
| *** hashar has joined #opendev | 07:08 | |
| *** bhagyashris is now known as bhagyashris|lunc | 07:27 | |
| *** tosky has joined #opendev | 07:29 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 07:29 | |
| *** lpetrut has joined #opendev | 07:40 | |
| *** factor has quit IRC | 07:45 | |
| *** raukadah is now known as chandankumar | 07:58 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:03 | |
| *** dpawlik6 has quit IRC | 08:18 | |
| *** ykarel is now known as ykarel|lunch | 08:25 | |
| *** bhagyashris|lunc is now known as bhagyashris | 08:34 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 08:35 |
| *** dpawlik6 has joined #opendev | 08:36 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Add support for CentOS 8 Stream https://review.opendev.org/734083 | 08:37 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed zuul/zuul-jobs master: [ensure-python] Remove debian-only note https://review.opendev.org/737231 | 08:44 |
| *** ykarel|lunch is now known as ykarel | 08:58 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Download latest CentOS cloud image https://review.opendev.org/737237 | 09:07 |
| *** ykarel is now known as ykarel|mtg | 09:07 | |
| *** bolg has joined #opendev | 09:15 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 09:17 |
| *** tkajinam has quit IRC | 09:21 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Add support for CentOS 8 Stream cloud image https://review.opendev.org/737245 | 09:42 |
| *** ykarel|mtg is now known as ykarel | 09:42 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 09:58 |
| *** rpittau is now known as rpittau|bbl | 10:15 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 10:17 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 10:49 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 11:06 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 11:08 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 11:21 |
| *** rpittau|bbl is now known as rpittau | 12:22 | |
| *** ryohayakawa has quit IRC | 12:27 | |
| *** ysandeep|away is now known as ysandeep|PTO | 12:38 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 12:38 |
| mnaser | infra-core: got a ptl+1 https://review.opendev.org/#/c/737133/4 on -- would be nice to land that! | 13:10 |
| fungi | mnaser: approved, but highlighting config-core in #openstack-infra might be more effective in such cases | 13:15 |
| mordred | you're a config-core in #openstack-infra | 13:15 |
| * mordred is enjoying a lovely morning thunderstorm | 13:18 | |
| mordred | fungi: a couple of months ago we made the mistake of thinking "wow, it's been uncommonly dry" | 13:18 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 13:18 |
| mordred | this is no longer true | 13:19 |
| fungi | yeah, it's nice and wet here. has become a struggle to find a dry enough day where there's no standing water in the yard (at least on the side of the seawall where there's not supposed to be water) to be able to push the mower through it | 13:21 |
| *** hashar is now known as hasharAway | 13:21 | |
| fungi | i really should do it at least weekly, but recently that's not been entirely possible | 13:21 |
| openstackgerrit | Merged openstack/project-config master: Retire the puppet-congress project - Step 1 End Project Gating https://review.opendev.org/737133 | 13:26 |
| mnaser | fungi: yeah i should figure out slowly where to ping more, i always go back and forth :p | 13:35 |
| fungi | heh. me too ;) | 13:36 |
| mordred | fungi: have you considered just getting a scythe? | 13:42 |
| *** redrobot has joined #opendev | 13:46 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 13:52 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add tests for upload-docker-image https://review.opendev.org/735402 | 13:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix and test multiarch docker builds in a release pipeline https://review.opendev.org/737059 | 13:55 |
| openstackgerrit | Merged openstack/project-config master: Retire the puppet-congress project - Step 3 Remove project https://review.opendev.org/737144 | 13:58 |
| *** hasharAway is now known as hashar | 13:58 | |
| fungi | mordred: my reel mower is basically like a steampunk scyth | 14:03 |
| fungi | scythe | 14:03 |
| fungi | clockwork rotary scythe | 14:05 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 14:19 |
| *** mlavalle has joined #opendev | 14:22 | |
| corvus | mordred, clarkb, fungi: the zk tls change deployed automatically without incident. there was about a 4 minute outage -- we expected something like that because the quorum was switching to tls, so it couldn't do a rolling restart. | 14:35 |
| corvus | oh sorry a 14m outage | 14:35 |
| corvus | (because i think our playbook does attempt a rolling restart) | 14:36 |
| fungi | nice! | 14:37 |
| clarkb | yes it restarts them in order | 14:37 |
| fungi | so zk-zk communication is tls now but client communication is still over plaintext? | 14:37 |
| clarkb | nirmally its zero downtime as a result | 14:37 |
| corvus | right, but our tls port is open, so we can write some test scripts to track down that bug | 14:38 |
| fungi | yep, that's where i thought we were, just making sure | 14:38 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66633&rra_id=all continues to look good. I think we can land https://review.opendev.org/#/c/737098/ and its parent to upgradegitea to 1.12 | 14:40 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: Make the base role and playbook compatible with CentOS https://review.opendev.org/737043 | 14:40 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 14:41 |
| clarkb | I'll remove gitea01 from the emergency file one those land | 14:41 |
| fungi | clarkb: cool, i was just about to check up on that | 14:42 |
| corvus | +w | 14:43 |
| clarkb | tyty | 14:43 |
| mordred | corvus: sweet! | 14:43 |
| fungi | and yeah, all the cacti graphs for it look reasonable and not newly resource-constrained | 14:44 |
| mordred | corvus: and - I'm guessing we're _not_ getting errors with just zk talking to zk | 14:44 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 14:45 |
| corvus | mordred: correct; not now, and not during the previous maint, which is why we figured that part was okay. plus, the bug seemed to be in kazoo. | 14:45 |
| *** ykarel is now known as ykarel|away | 14:46 | |
| *** sgw1 has joined #opendev | 14:48 | |
| corvus | perfect, i reproduced the error with a simple script :) cc: tobiash | 14:48 |
| corvus | i just walked the nodepool tree | 14:48 |
| mordred | corvus: EXCELLENT | 14:49 |
| mordred | corvus: and doing so blows your kazoo connection yeah? | 14:49 |
| corvus | yep | 14:49 |
| mordred | neat | 14:49 |
| corvus | i'll try to characterize this a bit more | 14:50 |
| mordred | so it's an easy test to run without breaking things | 14:50 |
| corvus | yep | 14:50 |
| mordred | that's excellent | 14:50 |
| corvus | as soon as i do, i'll start reporting on the github issue | 14:50 |
| clarkb | it is always a relief when reproducing a bug like that is easy | 14:51 |
| tobiash | cool | 14:51 |
| mordred | note to self - when using dib to create an image manually- make sure to include openssh-server element | 14:52 |
| clarkb | mordred: I include infra-package-needs from project-config as it pulls in a few useful things like that | 14:53 |
| mordred | clarkb: yeah - but this is an image I was building not for opendev | 14:55 |
| *** hashar is now known as hasharAway | 15:18 | |
| AJaeger | I have two small infra-manual changes to improve retirement docs - please review https://review.opendev.org/737134 and https://review.opendev.org/736732 | 15:18 |
| openstackgerrit | Rafael Folco proposed openstack/diskimage-builder master: DNM: Debug py3 on dib 7 https://review.opendev.org/736421 | 15:18 |
| *** hrw has quit IRC | 15:31 | |
| *** hasharAway is now known as hashar | 15:53 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add tests for upload-docker-image https://review.opendev.org/735402 | 15:55 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix and test multiarch docker builds in a release pipeline https://review.opendev.org/737059 | 15:55 |
| *** rpittau is now known as rpittau|afk | 16:01 | |
| *** aannuusshhkkaa has joined #opendev | 16:04 | |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 16:06 |
| clarkb | I've removed gitea01 from the emergency file as the two updates there should merge momentarily | 16:12 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 16:13 | |
| openstackgerrit | Merged opendev/system-config master: Update to gitea 1.12.0 https://review.opendev.org/729659 | 16:14 |
| openstackgerrit | Merged opendev/system-config master: Small repo template cleanups in Gitea https://review.opendev.org/737098 | 16:14 |
| *** shtepanie has joined #opendev | 16:27 | |
| mordred | clarkb: I see 1.12 when I go to opendev.org | 16:35 |
| clarkb | mordred: ya and gitea01 switched to the prod image | 16:35 |
| mordred | clarkb: nova is loading in 1 second for me | 16:35 |
| clarkb | awesome thats down from about 4 seconds before iirc | 16:37 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 16:39 |
| clarkb | the first set of updates is done | 16:44 |
| clarkb | the second is about to begin | 16:44 |
| clarkb | the second basically just cleans up the activity menu as its not quite what we want | 16:44 |
| * clarkb finds breakfast | 16:44 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 16:45 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 16:46 |
| *** hashar is now known as hasharAway | 16:47 | |
| mnaser | ouu | 16:50 |
| mnaser | upgraded gitea, looks nice | 16:50 |
| *** lpetrut has quit IRC | 16:50 | |
| mnaser | does indeed seem more responsive too | 16:51 |
| openstackgerrit | Javier Peña proposed opendev/system-config master: [WIP] Support CentOS for AFS mirror https://review.opendev.org/736996 | 16:51 |
| clarkb | mnaser: I did non scientific measurements and ya nova root is significantly faster as is neutron | 16:53 |
| mnaser | \o/ | 16:53 |
| * mnaser will see if search has had improvements | 16:53 | |
| fungi | yes, much faster | 16:56 |
| fungi | mnaser: i'm not sure we expect search to work yet, which is why we've still got hound (codesearch service) up | 16:57 |
| *** diablo_rojo has joined #opendev | 16:57 | |
| openstackgerrit | Mohammed Naser proposed opendev/project-config master: gate: enqueue if we get Verified+1 https://review.opendev.org/737319 | 16:58 |
| openstackgerrit | Mohammed Naser proposed opendev/project-config master: recheck: allow enqueue-directly-to-gate https://review.opendev.org/737320 | 16:59 |
| mnaser | fungi: yeah, i was thinking more on in-repo search which seems to struggle a tad | 17:00 |
| mnaser | infra-core: i made one not-very-controversal change (auto enqueue with verified+1) and one that might spark more discussion 'recheck' going straight to gate. also, i don't know how zuul will decide which pipeline wins when doing a recheck, so there's that to understand .. i think its ok but i split them as to not hinder the other patches progress | 17:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 17:00 |
| *** xiaolin has quit IRC | 17:02 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 17:03 |
| clarkb | mnaser: zuul will enqueue to both pipelines | 17:04 |
| clarkb | there is no winning an event. each pipeline that matches runs | 17:04 |
| clarkb | though I guess it says it supercedes check | 17:06 |
| clarkb | which may force check back out again if gate enqueues | 17:06 |
| fungi | mnaser: has gitea search ever worked for you? for example, https://opendev.org/openstack/nova/search?q=bindep returns zero matches for me while `git grep bindep` in current master branch of that repo gives me 10 matches | 17:06 |
| clarkb | I've confirmed the activity bar is gone now | 17:07 |
| clarkb | #status log Upgraded gitea farm to gitea 1.12.0 with minor local edits | 17:08 |
| openstackstatus | clarkb: finished logging | 17:08 |
| fungi | mnaser: also hound can limit searches to specific repos, though you have to use its advanced search options: http://codesearch.openstack.org/?repos=openstack/nova&q=bindep | 17:10 |
| clarkb | mnaser: ya check should be evaluated first because it is first in the config file. The change will be enqueued there. Then gate is evaluated and enqueued and becuase it supercedes check the change will then be removed from check | 17:10 |
| clarkb | what isn't clear to me is where do we require a positive verified value before gating | 17:10 |
| clarkb | because that would be the thing that interacts with this in unexpected ways | 17:11 |
| mnaser | yeah because there isn't really a lock in place unlike the openstack tenant | 17:11 |
| mnaser | fungi: it has worked _sometimes_, i think it really depends on how the string matches and if its followed by something or if its on its own | 17:12 |
| clarkb | oh this is opendev project config then ya I think its all fine. Did you look at how the zuul tenant does it? | 17:12 |
| clarkb | I would set it up the same as zuul and if this is how zuul has done it we should be good | 17:12 |
| mnaser | i thought zuul ran the same project-config as opendev, let me double check | 17:12 |
| openstackgerrit | Matthew Thode proposed opendev/glean master: switch glean.sh path in gentoo openrc init https://review.opendev.org/737325 | 17:15 |
| corvus | mnaser: drop the first patch, keep the second, and it will match the zuul tenant and do what you want in all cases | 17:22 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 17:25 |
| corvus | or, actually, i have a suggested revision on the second | 17:25 |
| openstackgerrit | Mohammed Naser proposed opendev/project-config master: recheck: allow enqueue-directly-to-gate https://review.opendev.org/737320 | 17:26 |
| mnaser | oh, one second | 17:26 |
| openstackgerrit | Mohammed Naser proposed opendev/project-config master: recheck: allow enqueue-directly-to-gate https://review.opendev.org/737320 | 17:26 |
| mnaser | corvus: sorry missed your comment by a sec, that should be it | 17:27 |
| openstackgerrit | Mohammed Naser proposed opendev/project-config master: Update promote pipeline precedence https://review.opendev.org/737333 | 17:29 |
| mnaser | and another small thing i just saw ^ | 17:29 |
| corvus | mnaser: cool thanks! :) | 17:33 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 17:39 |
| *** mlavalle has quit IRC | 17:40 | |
| openstackgerrit | Merged opendev/project-config master: recheck: allow enqueue-directly-to-gate https://review.opendev.org/737320 | 17:40 |
| openstackgerrit | Merged opendev/project-config master: Update promote pipeline precedence https://review.opendev.org/737333 | 17:40 |
| *** DSpider has quit IRC | 17:44 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 17:46 |
| openstackgerrit | Matthew Thode proposed opendev/glean master: switch glean.sh path in gentoo openrc init https://review.opendev.org/737325 | 17:57 |
| mnaser | hrm | 18:03 |
| mnaser | http://grafana.openstack.org/d/nuvIH5Imk/nodepool-vexxhost | 18:03 |
| mnaser | ok i can see the data inside graphite | 18:08 |
| fungi | neat, did we merge somethnig to break that dashboard's queries? | 18:09 |
| mnaser | i wonder if we just need to repoint to the right hostname (aka graphite01.opendev.org) | 18:09 |
| mnaser | ah, graphite.openstack.org is a CNAME to graphite01.opendev.org | 18:09 |
| mnaser | so that shouldn't be ok | 18:09 |
| fungi | yeah, its been graphite01.opendev.org for over a year | 18:10 |
| mnaser | grafana looks pretty new though | 18:10 |
| fungi | so i wouldn't expect that to have suddenly broken queries | 18:10 |
| mnaser | it lists version 7.0.3 | 18:10 |
| mnaser | which was released june 3rd 2020 | 18:11 |
| fungi | the grafana server itself hasn't been rebuilt in over a year, but yeah we do seem to be periodically or continuously deploying updates for it | 18:11 |
| mnaser | http://grafana.openstack.org/d/MYvSHcSiz/git-load-balancer?orgId=1 these other graphs are working for what it's worth | 18:11 |
| mnaser | fungi: ooooh i know, we started using ca-ymq-1 instead of sjc1 | 18:12 |
| mnaser | and the 'region' option is empty, so maybe that's it | 18:12 |
| fungi | aha! uep | 18:12 |
| fungi | er, yep | 18:13 |
| mnaser | but, the rackspace one is broken too | 18:13 |
| mnaser | the selection is empty, so it might be related to that | 18:13 |
| mnaser | so sounds like the query bit is the non-functional part | 18:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 18:14 |
| fungi | ovh is another where we have multiple regions | 18:15 |
| mnaser | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 and is broken too, so yeah, likely that is the culprit | 18:15 |
| fungi | oh, though some single-region nodepool dashboards are also broken | 18:17 |
| fungi | yeah, i have yet to find a working nodepool dashboard | 18:17 |
| fungi | in fact, the git lb dashboard you linked has quite a few graphs with "no data" too | 18:19 |
| *** hasharAway has quit IRC | 18:23 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul-jobs master: prepare-workspace: Add Role Variable in README.rst https://review.opendev.org/737352 | 18:36 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 18:40 |
| *** DSpider has joined #opendev | 18:45 | |
| fungi | mnaser: looks like 7.0.0 was released a month ago, though maybe puppet-grafana started pulling 7.x more recently than that | 18:50 |
| mnaser | fungi: i think if someone actually logs into the grafana instance, it might actually spit out some warnings.. | 18:50 |
| fungi | except our modules.env pins puppet-grafana at v6.0.0 which is over a year old | 18:51 |
| clarkb | the grafana logs actually aren't really helpful | 18:57 |
| clarkb | its just logging people trying to find urls in there for cgi/php vulns | 18:58 |
| clarkb | I think where we want to start is ask grafana for the dashboard json and from that see what graphite queries it is generating | 18:58 |
| clarkb | we can probably work backwards from that to see what is breaking | 18:58 |
| AJaeger | fungi, could you put this small infra-manual change on your review queue, please? https://review.opendev.org/737134 | 18:59 |
| fungi | clarkb: yeah, i was trying to figure out how to see what queries grafana thinks it's been asked to perform | 18:59 |
| clarkb | fungi: you click on the share button on a dashboard next to the name, then switch to export and json option from there | 19:00 |
| clarkb | "target": "sumSeries(stats.gauges.nodepool.provider.$region.nodes.building)" | 19:01 |
| *** hashar has joined #opendev | 19:02 | |
| fungi | aha, that was not easy to find, thanks ;) | 19:02 |
| clarkb | I expect that we need to provide provider info there along with the region variable? | 19:02 |
| clarkb | though I'm not super positive of that | 19:02 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 19:04 |
| clarkb | query: stats.gauges.nodepool.provider.ovh-* is what the grafana config says | 19:04 |
| clarkb | ah ok that shows up in the json under different headings | 19:05 |
| clarkb | aha that query is how we get the regions | 19:06 |
| clarkb | "datasource": null <- that is the issue I think | 19:08 |
| clarkb | looking at grafana docs that should be a non null value so that we can lookup the templated variable | 19:09 |
| AJaeger | htanks, fungi | 19:12 |
| AJaeger | thanks, I mean | 19:13 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 19:13 |
| fungi | clarkb: so this is something which changed with your grafyaml is creating the dashboards, or changed in new grafana? | 19:13 |
| fungi | i see we're passing $version = 'latest' in our puppet manifest | 19:14 |
| clarkb | fungi: I don't think we've changed how we create the dashboards, I think this is grafyaml not producing templated values properly | 19:16 |
| clarkb | looking at grafyaml we have a single datasource and we set it to be the default | 19:16 |
| clarkb | which I would expect to fix our problem | 19:16 |
| clarkb | but maybe with templates we have to be explicit about datasources now | 19:16 |
| clarkb | we can try setting a datasource and see if that fixes it | 19:16 |
| fungi | interestnig that it just seems to have broken recently though. we haven't altered grafyaml's source code since the beginning of last year | 19:17 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul-jobs master: prepare-workspace: Add Role Variable in README.rst https://review.opendev.org/737352 | 19:18 |
| fungi | or are you saying grafana has gotten more picky about it recently? | 19:18 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Try set explicit datasource on the OVH dashboard https://review.opendev.org/737361 | 19:19 |
| clarkb | fungi: that is my assumption | 19:19 |
| clarkb | we can see if ^ makes any difference for that dashboard | 19:19 |
| mordred | clarkb: seems reasonable | 19:21 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 19:25 |
| mordred | infra-root: could I interest anyone in reviewing https://review.opendev.org/#/c/737023/ ? I'd like to run the playbook to get review-test populated ... so it should probably be carefully reviewed | 19:30 |
| openstackgerrit | Merged openstack/project-config master: Try set explicit datasource on the OVH dashboard https://review.opendev.org/737361 | 19:40 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 19:44 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 19:56 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 20:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 20:41 |
| clarkb | ianw: mordred: because I don't quite always get how ansible works, does the condition at https://review.opendev.org/#/c/735055/5/roles/openafs-client/tasks/openafs-client/Debian.yaml check the arch of the target host of the host running ansible? | 21:11 |
| mordred | clarkb: before I look at it - I'm going to put money on "host running ansible" | 21:12 |
| mordred | nope. that would be target host | 21:12 |
| mordred | ansible_architecture should be a fact discovered about the remote host | 21:12 |
| fungi | ianw: not sure if you saw my ping earlier in #zuul (wrong channel sorry) but i peeked in the rsync mirror logs and vos release for mirror.fedora is running around 6-8 seconds. there's still a fairly sizeable, albeit brief, outbound spike from 01.dfw every two hours on the cacti graph though, and i'm not entirely sure what's causing it: | 21:13 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2362&rra_id=all | 21:13 |
| mordred | clarkb: I left a comment there | 21:16 |
| *** factor has joined #opendev | 21:17 | |
| clarkb | mordred: I like that suggestion, thanks | 21:17 |
| mordred | clarkb, ianw: I think we can do that in a followup - and it might be worth a larger patch | 21:17 |
| mordred | or maybe just a thing we all make a mental note to do as we do things | 21:17 |
| mordred | clarkb: oh good - turns out we're already using that form in many places | 21:19 |
| clarkb | fungi: you reviewed https://review.opendev.org/#/c/735284/ but not its parent depends on. Care to review the depends on too? /me is reviewing both now | 21:19 |
| openstackgerrit | Merged openstack/project-config master: [Grafana] Update label of neutron-functional-with-uwsgi job https://review.opendev.org/736168 | 21:22 |
| fungi | clarkb: yeah, i wasn't entirely clear on what the answers were to AJaeger's questions there, so thought it better to hold off reviewing until ianw could respond to them | 21:22 |
| clarkb | ah I gave a couple thoughts to those ideas | 21:23 |
| clarkb | but ya ianw's response would be good too | 21:23 |
| fungi | oh, indeed you just did, gertty hadn't synced them for me yet | 21:24 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Prefix several ansible_ variables with ansible_facts https://review.opendev.org/737381 | 21:24 |
| mordred | clarkb, ianw: ^^ | 21:25 |
| mordred | fungi, ianw: while we're fishing for reviews ... https://review.opendev.org/#/c/737023/ could use eyeballs | 21:26 |
| *** hashar has quit IRC | 21:27 | |
| openstackgerrit | Merged openstack/project-config master: Add Backport-Candidate label for oslo deliverables https://review.opendev.org/734096 | 21:28 |
| openstackgerrit | Merged openstack/project-config master: Add os_adjutant zuul project https://review.opendev.org/736145 | 21:28 |
| clarkb | re grafana I think we are waiting on a puppet pulse now which happens daily? | 21:29 |
| fungi | clarkb: yeah, i've been watching, there's a prod-hourly buildset in progress | 21:30 |
| clarkb | ah yup that runs puppet-else | 21:30 |
| fungi | i think that needs to run infra-prod-remote-puppet-else | 21:31 |
| fungi | right, that | 21:31 |
| fungi | since this is still deployed with puppetry | 21:31 |
| openstackgerrit | Merged opendev/system-config master: openafs-client: Use PPA for Xenial ARM64 https://review.opendev.org/735055 | 21:46 |
| fungi | clarkb: 737361 merged over 1.5 hours ago so the last prod-hourly run of infra-prod-remote-puppet-else should have incorporated it, but /etc/grafyaml/config/nodepool-ovh.yaml on grafana.o.o is still showing last modified several months ago | 21:49 |
| fungi | actually merged 2 hours ago (which is still "over 1.5 hours ago" so i wasn't wrong!) | 21:50 |
| clarkb | fungi: there is actually no puppet in syslog | 21:50 |
| clarkb | I wonder if wearen't puppeting like we expect | 21:50 |
| fungi | i too am wondering this | 21:52 |
| fungi | though my capacity for wondering about such things is rapidly drawing to a close for today | 21:52 |
| ianw | hey looking | 22:03 |
| ianw | fungi: re the spike -- the actual rsync process is still probably causing that? just the metadata transfer though | 22:05 |
| fungi | maybe? that's an awful lot of "metadata" though | 22:09 |
| * fungi checks the actual amount of data transferred | 22:09 | |
| ianw | fungi: a fedora run looks like ~ 10-15mb : http://paste.openstack.org/show/795062/ | 22:12 |
| ianw | the others are probably similar ; so ballpark i'd say if it's < 100mb that would explain it | 22:13 |
| fungi | the most recent outbound spike ran from roughly 20:25-20:45 and averaged 44.55Mbps according to cacti though i don't buy the math looking at the data points... still even if it's correct that's still 6.2GiB/sec | 22:14 |
| fungi | er, not /sec | 22:15 |
| fungi | 6.2GiB total data transferred over the course of that 20 minute period | 22:15 |
| fungi | looking at the data points though i think the average was more like 63.75Mbps | 22:16 |
| fungi | er, more like 60-65Mbps. my eyeballs aren't that precise | 22:17 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=2362&rra_id=5&view_type=&graph_start=1592857482&graph_end=1592858713&graph_height=120&graph_width=500&title_font_size=12 | 22:18 |
| fungi | it says Average: 44.55M | 22:19 |
| fungi | bit even the lowest data point in that timeframe seems to be higher than its reported average | 22:19 |
| ianw | fungi: i guess we have to multiply out any changes by the number of mirrors, until their cache updates? | 22:23 |
| fungi | likely | 22:27 |
| fungi | well, divided by the number of read-only volumes i guess | 22:28 |
| fungi | er, read-only replicas | 22:28 |
| fungi | assuming they're evenly balanced across the servers, which they likely aren't | 22:28 |
| ianw | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=3209&rra_id=5&view_type=&graph_start=1592850333&graph_end=1592864733 does look similar in the other diretion | 22:28 |
| openstackgerrit | Merged opendev/system-config master: Acutally run system-config arm64 test on an arm64 node https://review.opendev.org/735281 | 22:31 |
| openstackgerrit | Ian Wienand proposed opendev/bindep master: Cull the test bindep file https://review.opendev.org/735282 | 22:39 |
| openstackgerrit | Ian Wienand proposed opendev/bindep master: Add centos 8 and focal testing https://review.opendev.org/735269 | 22:39 |
| ianw | fungi: ^ that adds back zip/unzip to test the non-match path as suggested | 22:39 |
| ianw | probably be good to review that stack as i think the gate is broken | 22:40 |
| ianw | for bindep | 22:40 |
| fungi | awesome thanks. i'm +2 across the board. if you want to approve with just one reviewer i don't object | 22:44 |
| clarkb | I can look | 22:45 |
| ianw | fungi: https://review.opendev.org/#/c/703055/ to handle different LC_*'s also looks worthwhile | 22:46 |
| ianw | it's probably worth a release too, last one as in august | 22:49 |
| *** tkajinam has joined #opendev | 22:51 | |
| openstackgerrit | Merged opendev/bindep master: Use abstracted virtualenv_command from ensure-pip https://review.opendev.org/735267 | 22:56 |
| openstackgerrit | Merged opendev/bindep master: Cull the test bindep file https://review.opendev.org/735282 | 22:56 |
| *** tosky has quit IRC | 22:59 | |
| openstackgerrit | Merged opendev/bindep master: Add centos 8 and focal testing https://review.opendev.org/735269 | 23:08 |
| *** pmacdonnell has joined #opendev | 23:32 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: add musl profile to gentoo https://review.opendev.org/737394 | 23:38 |
| *** ryohayakawa has joined #opendev | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!