| *** mlavalle has quit IRC | 00:12 | |
| *** mailingsam has quit IRC | 00:19 | |
| *** jralbert has quit IRC | 00:23 | |
| *** lbragstad_ has joined #opendev | 00:43 | |
| *** lbragstad has quit IRC | 00:45 | |
| ianw | ok, the unreachable mirror node has made the base job fail :/ | 01:04 |
|---|---|---|
| kevinz | ianw: Just saw your email, I will take a look at this now | 01:08 |
| ianw | thanks :) | 01:13 |
| *** hamalq has quit IRC | 01:19 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add planet.openstack.org redirect to static https://review.opendev.org/c/opendev/system-config/+/785993 | 01:19 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add planet.openstack.org redirect to static https://review.opendev.org/c/opendev/system-config/+/785993 | 01:21 |
| kevinz | ianw: recovered, | 01:24 |
| kevinz | Looks the l3_agent is not recovered well after restart. Now it is back | 01:24 |
| ianw | kevinz: excellent, thanks; it can ping ipv4 again | 01:25 |
| ianw | and afs is back too | 01:26 |
| kevinz | ianw: Np :-) | 01:27 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Stop managing planet01.openstack.org https://review.opendev.org/c/opendev/system-config/+/785994 | 01:29 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: all-clouds: correct typo in OSU configuration https://review.opendev.org/c/opendev/system-config/+/785997 | 01:36 |
| openstackgerrit | Merged opendev/system-config master: all-clouds: correct typo in OSU configuration https://review.opendev.org/c/opendev/system-config/+/785997 | 02:39 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: all-clouds: add OSU OSL project_id as well https://review.opendev.org/c/opendev/system-config/+/786002 | 03:34 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add planet.openstack.org redirect to static https://review.opendev.org/c/opendev/system-config/+/785993 | 04:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Stop managing planet01.openstack.org https://review.opendev.org/c/opendev/system-config/+/785994 | 04:15 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add OSUOSL mirror node https://review.opendev.org/c/opendev/zone-opendev.org/+/786003 | 04:22 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add OSUOSL mirror https://review.opendev.org/c/opendev/system-config/+/786004 | 04:23 |
| *** d34dh0r53 has quit IRC | 04:30 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add OSUOSL mirror node https://review.opendev.org/c/opendev/zone-opendev.org/+/786003 | 04:39 |
| ianw | i've rebooted review02 with the netplan pin | 04:53 |
| ianw | kevinz: it still seems like we're getting a lot of node_failure results | 04:58 |
| ianw | 2021-04-13 04:38:56,102 ERROR nodepool.NodeLauncher: [e: bc285eb9006841dcad6f0f9a512c2032] [node_request: 300-0013667983] [node: 0023969930] Detailed node error: No valid host was found. There are not enough hosts available. | 05:00 |
| ianw | same issue as yesterday; i'll send you a mail with some id's | 05:01 |
| *** akahat has quit IRC | 05:02 | |
| *** marios has joined #opendev | 05:03 | |
| *** fbo has quit IRC | 05:03 | |
| openstackgerrit | Merged opendev/system-config master: all-clouds: add OSU OSL project_id as well https://review.opendev.org/c/opendev/system-config/+/786002 | 05:04 |
| *** fbo has joined #opendev | 05:07 | |
| *** akahat has joined #opendev | 05:14 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add planet.openstack.org redirect to static https://review.opendev.org/c/opendev/system-config/+/785993 | 05:18 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Stop managing planet01.openstack.org https://review.opendev.org/c/opendev/system-config/+/785994 | 05:18 |
| *** ralonsoh has joined #opendev | 05:29 | |
| *** ralonsoh has quit IRC | 05:29 | |
| *** ralonsoh has joined #opendev | 05:30 | |
| *** slaweq has joined #opendev | 05:52 | |
| *** sboyron has joined #opendev | 06:08 | |
| openstackgerrit | Merged opendev/system-config master: Add OSUOSL mirror https://review.opendev.org/c/opendev/system-config/+/786004 | 06:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add planet.openstack.org redirect to static https://review.opendev.org/c/opendev/system-config/+/785993 | 06:17 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Stop managing planet01.openstack.org https://review.opendev.org/c/opendev/system-config/+/785994 | 06:17 |
| *** amoralej|off is now known as amoralej | 06:23 | |
| ianw | the osu osl cloud mirror should deploy now; i'll check on it later. i'm off for now | 06:32 |
| *** dpawlik has joined #opendev | 06:33 | |
| *** dpawlik has quit IRC | 06:37 | |
| *** dpawlik has joined #opendev | 06:40 | |
| *** dpawlik has quit IRC | 06:42 | |
| *** eolivare has joined #opendev | 06:46 | |
| *** fressi has joined #opendev | 06:51 | |
| *** dpawlik has joined #opendev | 06:54 | |
| *** dpawlik has quit IRC | 06:55 | |
| *** dpawlik has joined #opendev | 07:12 | |
| *** dpawlik has joined #opendev | 07:12 | |
| *** dpawlik has quit IRC | 07:13 | |
| *** dpawlik has joined #opendev | 07:14 | |
| *** andrewbonney has joined #opendev | 07:14 | |
| kevinz | ianw: I wil check, so just arrived home | 07:23 |
| openstackgerrit | Bernard Cafarelli proposed openstack/project-config master: Update neutron stable grafana dashboards https://review.opendev.org/c/openstack/project-config/+/786008 | 07:23 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix OSU OSL mirror host variable file typo https://review.opendev.org/c/opendev/system-config/+/786009 | 07:23 |
| kevinz | ianw: sorry just arrived home, I will check it now | 07:23 |
| ianw | kevinz: no problems; it seems *mostly* ok, but a few failures popping up | 07:24 |
| kevinz | ianw: I suppose it should be the resource is used up | 07:24 |
| ianw | i don't think it's near capacity ... | 07:25 |
| ianw | at least, we're not at our capacity ... but someone else in the cloud? | 07:25 |
| kevinz | ianw: It looks that it did not reach the capacity | 07:26 |
| *** jpena|off is now known as jpena | 07:34 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: host_vars : add .yaml extension https://review.opendev.org/c/opendev/system-config/+/786011 | 07:35 |
| *** sboyron has quit IRC | 07:37 | |
| *** sboyron has joined #opendev | 07:38 | |
| *** sboyron has quit IRC | 07:38 | |
| *** sboyron has joined #opendev | 07:39 | |
| *** rpittau|afk is now known as rpittau | 07:43 | |
| kevinz | ianw: Just double check the total scheduler policy and capacity, I calculate the totally memories quota for os-jobs is 732G, and cpus is 320 * 2(cpu_overcommit) = 640 | 07:45 |
| openstackgerrit | Merged openstack/project-config master: Update neutron stable grafana dashboards https://review.opendev.org/c/openstack/project-config/+/786008 | 07:46 |
| hrw | morning | 07:47 |
| hrw | nice, new aarch64 nodes ;) | 07:47 |
| kevinz | hrw: morning! | 07:48 |
| *** tosky has joined #opendev | 07:49 | |
| kevinz | ianw: it looks that the os-jobs is not reaching the limit anyway. And I think we can also use some small vms to run UT for nova/neutron/cinder, rather than the os.large with 8C8G vms. | 07:49 |
| openstackgerrit | Merged opendev/system-config master: Fix OSU OSL mirror host variable file typo https://review.opendev.org/c/opendev/system-config/+/786009 | 07:57 |
| *** hberaud has left #opendev | 08:25 | |
| *** dtantsur|afk is now known as dtantsur | 08:31 | |
| *** janders8 is now known as janders | 08:32 | |
| *** hrw has quit IRC | 08:47 | |
| *** hrw has joined #opendev | 08:49 | |
| *** dtantsur is now known as dtantsur|brb | 08:57 | |
| *** snapdeal has joined #opendev | 09:07 | |
| *** dpawlik has quit IRC | 09:28 | |
| *** akahat has quit IRC | 09:28 | |
| *** fbo has quit IRC | 09:29 | |
| *** kopecmartin has joined #opendev | 09:33 | |
| *** snapdeal has quit IRC | 09:43 | |
| ianw | kevinz: we've been pretty consistent in keeping our nodes at 8gb environments. we could have smaller variants, but it's something we'd want to consider i guess | 09:47 |
| *** fbo has joined #opendev | 09:48 | |
| *** dpawlik has joined #opendev | 09:52 | |
| *** akahat has joined #opendev | 09:53 | |
| *** rpittau is now known as rpittau|bbl | 09:54 | |
| *** sboyron has quit IRC | 09:54 | |
| *** sboyron has joined #opendev | 09:55 | |
| ianw | kevinz: at our current limit of 40 nodes i wouldn't think we'd be having issues with 732g ram and 640-odd cpus | 09:58 |
| *** dtantsur|brb is now known as dtantsur | 10:00 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: OSU OSL: fix typo in certificate name https://review.opendev.org/c/opendev/system-config/+/786031 | 10:09 |
| ianw | osuosl and osusol are hard to tell apart! | 10:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: OSU OSL: fix typo in certificate name https://review.opendev.org/c/opendev/system-config/+/786031 | 10:18 |
| kevinz | ianw: Np, I will check if there is bug in our nova-scheduler | 10:18 |
| *** dpawlik has quit IRC | 10:45 | |
| *** dpawlik9 has joined #opendev | 10:52 | |
| openstackgerrit | Merged opendev/system-config master: OSU OSL: fix typo in certificate name https://review.opendev.org/c/opendev/system-config/+/786031 | 11:00 |
| *** tkajinam has quit IRC | 11:13 | |
| *** Alex_Gaynor has joined #opendev | 11:15 | |
| Alex_Gaynor | 👋 is there an issue with the arm64 provider? Those jobs don't appear to be starting. | 11:15 |
| ianw | Alex_Gaynor: probably, we've been having issues recently :/ | 11:21 |
| ianw | we've got a lot of active nodes currently | 11:22 |
| kevinz | ianw: I see several new vm are launched | 11:22 |
| ianw | kevinz: yeah, i might be being a bit pessimistic | 11:23 |
| Alex_Gaynor | Ok, looks like they are starting. Just a bit slow this morning. I empathasize, I too have not had my tea yet. | 11:23 |
| ianw | we've got 24 nodes active, 13 building | 11:23 |
| kevinz | ianw: But launching is slow due to parallelly creation | 11:23 |
| ianw | so we're close to capacity | 11:23 |
| kevinz | ianw: yes, I see | 11:24 |
| ianw | Alex_Gaynor: working on getting about an extra 15 nodes from OSU online soon | 11:24 |
| Alex_Gaynor | oooh, that's exciting! | 11:24 |
| ianw | of course, interest in using it tends to expand with capacity as well :) | 11:25 |
| kevinz | I advice we do not need to run some architecture non-related UT jobs | 11:25 |
| kevinz | ianw: I saw neutron ut is requesting quite a lot times.. | 11:26 |
| ianw | yeah, in one way, that's probably not likely to break on arm64; but it also has a lot of binary dependencies going on under the hood which are good to expose to alternative platforms | 11:27 |
| kevinz | ianw: OK, that's a good point. | 11:29 |
| ianw | it is a fair point though, i mean given the resources we always have to consider the usage/reward pay-off | 11:30 |
| ianw | https://us.linaro.cloud:8774/v2.1/servers, Quota exceeded for instances: Requested 1, but already used 41 of 40 instances | 11:31 |
| ianw | i wonder why we're exceeding the quota | 11:31 |
| *** jpena is now known as jpena|lunch | 11:31 | |
| ianw | ok, i'm out for now, ttyl | 11:32 |
| *** artom has joined #opendev | 12:15 | |
| *** amoralej is now known as amoralej|lunch | 12:29 | |
| *** rpittau|bbl is now known as rpittau | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:37 | |
| *** klonn has joined #opendev | 12:42 | |
| *** d34dh0r53 has joined #opendev | 12:53 | |
| *** mailingsam has joined #opendev | 13:15 | |
| *** tosky has quit IRC | 13:19 | |
| *** amoralej|lunch is now known as amoralej | 13:22 | |
| *** tosky has joined #opendev | 13:25 | |
| *** tosky has quit IRC | 13:29 | |
| *** tosky has joined #opendev | 13:29 | |
| *** mlavalle has joined #opendev | 13:40 | |
| *** lpetrut has joined #opendev | 13:56 | |
| *** klonn has quit IRC | 14:02 | |
| *** lpetrut has quit IRC | 14:20 | |
| *** lbragstad_ is now known as lbragstad | 14:31 | |
| *** klonn has joined #opendev | 14:52 | |
| *** amoralej is now known as amoralej|off | 15:05 | |
| *** DSpider has joined #opendev | 15:36 | |
| *** klonn has quit IRC | 15:54 | |
| *** marios is now known as marios|out | 16:09 | |
| clarkb | fungi: I'm going to put together PTG info today and send it to openstack-discuss with [tact-sig][dev][infra][qa] subject prefix and a separate email to service-discuss. I'm going to position it as office hours/help/debug/etc time and will ask people to throw their topics on an etherpad (to be made) as well to help scheduling | 16:13 |
| clarkb | fungi: do you think that is sufficient for getting this info in front of the necessary individuals? | 16:13 |
| fungi | yes, that sounds perfect. thanks! | 16:15 |
| *** hamalq has joined #opendev | 16:20 | |
| *** rpittau is now known as rpittau|afk | 16:24 | |
| *** marios|out has quit IRC | 16:38 | |
| *** jpena is now known as jpena|off | 16:43 | |
| openstackgerrit | Merged opendev/base-jobs master: Revert "Temporarily disable log uploads to OVH BHS Swift" https://review.opendev.org/c/opendev/base-jobs/+/785019 | 16:44 |
| clarkb | emails sent to service-discuss@lists.opendev.org and openstack-discuss lists. | 16:52 |
| JayF | what is a TaCT sig? | 17:04 |
| *** DSpider has quit IRC | 17:05 | |
| clarkb | JayF: https://governance.openstack.org/sigs/tact-sig.html basically the group of people within the openstack project that tie in closely with opendev making use of zuul, maintaining configs for zuul, etc | 17:05 |
| JayF | ah, makes sense. Thanks! | 17:06 |
| *** DSpider has joined #opendev | 17:06 | |
| *** dtantsur is now known as dtantsur|afk | 17:11 | |
| openstackgerrit | Merged opendev/system-config master: Add zuul keystore password https://review.opendev.org/c/opendev/system-config/+/785980 | 17:15 |
| *** cenne is now known as cenne-away | 17:16 | |
| *** klonn has joined #opendev | 17:16 | |
| fungi | JayF: when the opendev collaboratory became independent of the openstack project, the tact sig is more or less the openstack-specific bits the old infra team took care of | 17:19 |
| JayF | When you have an "infra IRC peanut gallery"-sig, I'll attend ;) | 17:20 |
| fungi | so that opendev wouldn't be burdened with openstack-specific responsibilities | 17:20 |
| JayF | That makes a lot of sense | 17:20 |
| fungi | (ultimately it's a lot of the same people, just helps us draw clearer boundaries between what's opendev and what's openstack) | 17:20 |
| *** sshnaidm has joined #opendev | 17:28 | |
| *** sshnaidm is now known as sshnaidm|pto | 17:28 | |
| *** eolivare has quit IRC | 18:00 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Add an empty project for an OpenStack base ACL https://review.opendev.org/c/openstack/project-config/+/782830 | 18:33 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Allow delete permissions in Gerrit ACLs https://review.opendev.org/c/openstack/project-config/+/786088 | 18:33 |
| *** mailingsam has quit IRC | 18:51 | |
| *** andrewbonney has quit IRC | 18:54 | |
| *** auristor has quit IRC | 19:01 | |
| *** auristor has joined #opendev | 19:04 | |
| *** auristor has quit IRC | 19:05 | |
| ianw | anyone ever seen something like | 19:07 |
| ianw | BadRequestException: 400: Client Error for url: http://arm-openstack.osuosl.org:9696/v2.0/security-group-rules, Unrecognized attribute(s) 'remote_address_group_id' | 19:07 |
| ianw | out of the cloud launcher? | 19:07 |
| fungi | nope, maybe some neutron folks can read those tea leaves? | 19:08 |
| ianw | https://storyboard.openstack.org/#!/story/2008577 ... looks like i'm not alone :/ | 19:12 |
| *** ralonsoh has quit IRC | 19:13 | |
| ianw | it seems the field was added with https://review.opendev.org/c/openstack/openstacksdk/+/755644 but unconditionally | 19:20 |
| *** beef_parotta has joined #opendev | 19:23 | |
| fungi | oh neat | 19:24 |
| fungi | and the version of neutron deployed there lacks support for it? | 19:24 |
| ianw | i guess so, that's my working theory. always nice to find my yak nice and early in the day :) | 19:25 |
| *** auristor has joined #opendev | 19:25 | |
| fungi | gives you time to become friends before the shavnig begins | 19:26 |
| *** beef_parotta has quit IRC | 19:30 | |
| JayF | I gave hangyang a heads up about that bug, since he committed the blamed code (he is a downstream co-worker) | 19:30 |
| ianw | JayF: thanks :) | 19:34 |
| openstackgerrit | Jeremy Stanley proposed opendev/jeepyb master: Update Gerrit hook command-line parameters https://review.opendev.org/c/opendev/jeepyb/+/786095 | 19:53 |
| fungi | it's also become apparent that we haven't restarted gerrit since the earlier jeepyb fix to stop trying to assign bugs merged, so we should get that ^ landed too and then plan for a restart sometime when it won't be too disruptive | 19:54 |
| fungi | melwitt: yoctozepto: ^ progress! | 19:54 |
| melwitt | \o/ | 19:55 |
| *** avass has quit IRC | 19:55 | |
| fungi | i keep forgetting we need gerrit restarts for jeepyb updates now, since it's baked into our container images | 19:57 |
| *** klonn has quit IRC | 20:02 | |
| ianw | fungi: does that need to be backwards compatible? | 20:02 |
| *** avass has joined #opendev | 20:02 | |
| clarkb | fungi: note I don't think that is true for jeepyb commands outside of the hooks (as they do docker run but gerrit execs the hooks directly so they are special) | 20:02 |
| clarkb | ianw: fungi: it is probably friendliest to be backward compat but I'm not sure if anyone is using jeepyb other than ourselves at this point | 20:03 |
| fungi | ianw: i didn't keep backward compatibility in the previous change to stop assigning bugs | 20:03 |
| fungi | my expectation is that if anyone is continuously deploying jeepyb they're also upgrading gerrit | 20:04 |
| ianw | ok, lgtm then | 20:04 |
| fungi | also i should have commented or included a link to the hooks documentation, but that's where i got the full set of parameters from | 20:05 |
| clarkb | ianw I abandoned my 3.2.8 change to avoid any confusion and put pointers to your plugin fix | 20:06 |
| fungi | i've hand-patched those two jeepyb changes (the merged one and the latest one) into the container and am continuing to watch my ad hoc debug log | 20:09 |
| fungi | i need to switch to cooking dinner, but will work on a test change/bug afterward if i don't spot one in the log | 20:09 |
| clarkb | should we land ianw's plugin fix, then the jeepyb fix? that way we won't end up with a broken image should we need to restart? | 20:09 |
| clarkb | but then we can sit on a restart for a bit | 20:09 |
| fungi | yeah, that sounds fine | 20:10 |
| clarkb | I've +2'd both of them if fungi wants to be second reviewers on the plugin fix and ianw on the jeepyb change I think we can +a them in that order | 20:11 |
| clarkb | but also eat dinner :) | 20:11 |
| openstackgerrit | Jeremy Stanley proposed opendev/jeepyb master: Correct set_in_progress parameters https://review.opendev.org/c/opendev/jeepyb/+/786120 | 20:15 |
| fungi | clarkb: ianw: also ^ (if it's not tested it's broken!) | 20:15 |
| clarkb | fungi: ianw: another one on the cleanup list should probably be firehose? I also think that one may be able to just go away? | 20:27 |
| fungi | yeah | 20:28 |
| ianw | yeah, seems likely | 20:28 |
| clarkb | cool, I can probably work on that since ya'll have others you are looking at | 20:29 |
| clarkb | I'm double checking what injects into mqtt and it seems that we stopped doing ansible at some point, germqtt is on firehose and lpmqtt is one firehose. This means i need to disable mqtt on the subunit and logstash workers | 20:38 |
| clarkb | we should also likely cleanup the inbox that lpmqtt uses? fungi do you know what it uses? | 20:38 |
| clarkb | logstash workers already disable mqtt | 20:39 |
| mtreinish | I'll have to ping andreaf, I think he was still playing with https://github.com/mtreinish/ciml a little bit which depends on firehose | 20:44 |
| *** sboyron has quit IRC | 20:46 | |
| clarkb | mtreinish: ok, the data can be retrieved through other channels too if necessary (like the gerrit event stream for gerrit events etc) | 20:47 |
| mtreinish | but it's not critical, there are alternatives (the firehose bit was just for demoing real time reporting) | 20:47 |
| fungi | clarkb: there is a cyrus-imap service running on the firehose server, we simply subscribed its address to some projects on lp | 20:47 |
| clarkb | fungi: oh so we need to remove email addrs from the lp side | 20:47 |
| clarkb | mtreinish: at this point all the config management needs rewriting and it was never really used in production for much so it is easier to turn if off if you need to pass along the reasoning | 20:48 |
| fungi | also we can't upgrade the underlying operating system because the existing configuration management for it would break | 20:48 |
| mtreinish | oh, I don't think it really needs a reason. It was a fun experiment, but it never really caught on | 20:49 |
| fungi | yeah, it was a worthwhile endeavor | 20:50 |
| fungi | i certainly learned a lot from that project | 20:50 |
| mtreinish | yeah, me too | 20:51 |
| mtreinish | heh, and it was the first time I wrote haskell (for the docs) :P | 20:52 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Stop publishing subunit worker data to mqtt https://review.opendev.org/c/opendev/system-config/+/786125 | 20:52 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove firehose.openstack.org https://review.opendev.org/c/opendev/system-config/+/786126 | 20:52 |
| clarkb | the first change there should be pretty safe to land (though maybe that is what andreaf was using? I don't mind waiting a bit before landing any of that) | 20:52 |
| clarkb | fungi: any objections to approving the git-review --no-thin change https://review.opendev.org/c/opendev/git-review/+/785723 ? | 20:53 |
| clarkb | ianw reviewed it too but needs a +A | 20:53 |
| ianw | i'm happy for +A after our discussion yesterday | 20:54 |
| clarkb | cool, want to do it or should I? | 20:54 |
| ianw | done :) | 20:54 |
| clarkb | thanks! | 20:55 |
| clarkb | now to figure out what lp account sends email to firehose | 20:55 |
| mtreinish | clarkb: nah, there's no real need to wait. I pinged him about it and I don't think he's actively using that component of ciml for anything important | 20:56 |
| clarkb | mtreinish: thanks! | 20:56 |
| mtreinish | it was listening to the subunit publish events though to trigger running predictions against the trained model based on new subunit2sql records, but it was never for anything more than a proof of concept | 20:57 |
| clarkb | gotcha | 20:57 |
| mtreinish | but, the real meat of that project was building the ML models and getting it to predict things based on the logs (never got it to predict why a failure occured, but we could identify which cloud vendor a job ran in from it's dstat data) | 20:58 |
| clarkb | I can see what email address is being delivered to in the exim logs, but I don't appear to be able to reverse lookup on lp for the user that way | 21:00 |
| mtreinish | IIRC we put the lp creds in the secrets store with the other shared root creds (I mainly just remember being chastized for sharing the password out loud at a midcycle when we first set it up) | 21:02 |
| clarkb | mtreinish: ah thanks I can look there next | 21:03 |
| clarkb | yup found it, now to see if I can disable email alerts | 21:05 |
| clarkb | ianw: I notice that the change to the zuul results plugin implies that the enablement is intended to go in refs/meta/config on a per project basis. Do you know if getFromProjectConfigWithInheritance(project.getNameKey(), pluginName) checks the global config like you set in your change? I assume it must since we use selenium to check this stuff now | 21:06 |
| ianw | clarkb: yeah, it gets defaulted on by the All-Projects update | 21:07 |
| ianw | as you say, the screenshots show it enabled | 21:07 |
| clarkb | ianw: all-projects doesn't help us in prod though | 21:08 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/778116/9/playbooks/roles/gerrit/templates/gerrit.config.j2 is what we should be applying in prod and in test, if that isn't sufficient then I think we'll need to manually update all-projects too? | 21:08 |
| clarkb | also if that isn't sufficient should we just remove it from gerrit.config.j2? | 21:08 |
| ianw | hrrrmm, now you've made me think :) | 21:10 |
| clarkb | (I'm also not sure I like changes like the upstream one. If you installed the plugin them maybe it should work by default rather than need to be enabled from there. I can see disabling it though) | 21:11 |
| ianw | actually I think yes it all comes from the global config. i'm thinking we should actually remove it from All-Projects | 21:11 |
| clarkb | ianw: cool, lets try that and see if it works and if so \o/ | 21:11 |
| ianw | i.e. remove setting it in https://review.opendev.org/c/opendev/system-config/+/778116/9/playbooks/test-review.yaml | 21:11 |
| clarkb | fungi: mtreinish: I'm in the lp account and it doesn't appear to be subscribed to any mailing lists and if I got to bugs it says there are no subscribed packages or bugs | 21:12 |
| clarkb | any idea what bit of lp configuration causes it to get emails? | 21:12 |
| mtreinish | it was a global thing for all of openstack. Maybe a group membership or something | 21:13 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Handle zuul-summary-results as .jar / per-project config https://review.opendev.org/c/opendev/system-config/+/778116 | 21:13 |
| mtreinish | it's been too long since I was last using lp regularly I don't remember the details of how to do that | 21:14 |
| clarkb | mtreinish: same here, but that helps thanks | 21:14 |
| ianw | clarkb: ^^ will keep an eye on | 21:14 |
| mtreinish | probably something related to: https://launchpad.net/openstack | 21:14 |
| mtreinish | clarkb: is just deleting the lp account an option? | 21:15 |
| mtreinish | that might be easier :) | 21:15 |
| clarkb | yes you can delete an account too | 21:16 |
| clarkb | I was thinking I would do that after firehose went away, but disable the email alerts before | 21:16 |
| clarkb | maybe it is ok for lp to fail to deliver email for a short time and delete after without stopping the emails first | 21:17 |
| *** tristanC_ is now known as tristanC | 21:19 | |
| clarkb | I'll punt on this for now. I think fungi groks this stuff pretty well and probably knows the anwer or how to find it | 21:21 |
| fungi | i'll catch up on all this after i'm done eating | 21:36 |
| fungi | but yeah, should just be able to auth as that account and unsubscribe it from https://bugs.launchpad.net/openstack | 21:39 |
| clarkb | hrm when I logged in and went to the susbcribed bug list page it said there were no subscriptions, but maybe going to that page directly would work | 22:02 |
| clarkb | and no rush I think I'll pick this back up tomorrow | 22:02 |
| fungi | or i'm happy to work on it once wallaby release jobs are in the clear | 22:07 |
| clarkb | if you want to do the unsub I won't complain :) | 22:07 |
| clarkb | once that happens and https://review.opendev.org/c/opendev/system-config/+/786125 lands and is applied my next plan was to stop services on the server and just double check eeverything remains happy | 22:07 |
| clarkb | then we can land the cleanup change | 22:08 |
| fungi | okay, 782603+786095+786120 have all been hand-patched into the jeepyb inside the gerrit container on review.o.o for the past hour, and i've seen no new backtraces from any of the hooks which call update-bug | 22:11 |
| fungi | so that's a good sign at least | 22:11 |
| clarkb | hrm I don't remember reviewing 3, just 2 /me looks | 22:12 |
| fungi | 782603 merged last week | 22:12 |
| fungi | we just haven't restarted gerrit | 22:12 |
| clarkb | ah ya and I reviewed it, got it | 22:13 |
| fungi | so i had to patch it into the volume tree for the currently running container, which is not the latest one pulled on the server | 22:13 |
| clarkb | fungi: I think we want to restart with https://review.opendev.org/c/opendev/system-config/+/778116/ in place too, once testing confirms the global config is sufficient | 22:13 |
| fungi | yeah, i figure we can plan to do a restart at some quiet time later this week | 22:14 |
| clarkb | yup sounds like friday is looking good for it | 22:15 |
| fungi | in theory, update-bug is fixed now with the hand-patched jeepyb in the container, i just haven't confirmed we've gotten any actual bug updates yet | 22:17 |
| *** tkajinam has joined #opendev | 22:23 | |
| clarkb | ianw: looks like it says "zuul integration is not enabled" | 22:40 |
| clarkb | Not sure I'm a fan of the upstream change :/ would really have preferred that installing the plugin implied on by default, disable if you want to rather than off by default if plugin is present | 22:40 |
| clarkb | that said updating our All-Projects config isn't terribly difficult, we would just need to plan to do it after we updated our image | 22:41 |
| ianw | clarkb: hrm, i feel like it should be able to read from the config file too | 22:41 |
| ianw | it probably needs more java | 22:42 |
| clarkb | that would be another good option, refer to the system config in addition to project configs | 22:43 |
| corvus | ianw: er, regarding your comment on https://gerrit-review.googlesource.com/c/plugins/zuul-results-summary/+/298465/ i can confirm there are more users than just opendev | 22:47 |
| corvus | ianw, clarkb: do you think we could switch the default real quick? :) | 22:48 |
| clarkb | corvus: that would probably be my preference. Since installing the plugin implies you want the functionality | 22:49 |
| clarkb | then if you don't want the functionality on a per project basis you can disable them individually | 22:49 |
| corvus | i also agree it seems counter-intuitive to have it off by default; it's just as easy for someone to set it to 'off' in all-projects then enable it per-project if they want the other behavior | 22:49 |
| corvus | ya | 22:49 |
| clarkb | but I don't know what the motivation behind that changw as in the first place | 22:49 |
| corvus | clarkb: i think it's because luca has some gerritforge projects using zuul but not all | 22:50 |
| clarkb | gotcha | 22:50 |
| corvus | so it's definitely a (friendly) behavior preference conflict between those of us in an all-zuul world, and those in a semi-zuul world | 22:50 |
| ianw | i should have thought more about it not reading the value from the global config file | 22:51 |
| ianw | i'm willing to take advice, we can revert, or propose basically making https://gerrit-review.googlesource.com/c/plugins/zuul-results-summary/+/298465/4/java/com/googlesource/gerrit/plugins/zuulresultssummary/GetConfig.java#44 true i guess | 22:52 |
| corvus | i was about to propose a change to do the latter | 22:53 |
| clarkb | ya I think making the default true there is a simple fix and gives behavior that makes sense to most users (I assume most users want the functionality to exist if they install it) | 22:53 |
| corvus | ianw, clarkb: remote: https://gerrit-review.googlesource.com/c/plugins/zuul-results-summary/+/303262 Enable the plugin by default [NEW] | 23:01 |
| ianw | as i've now found out, a single +2 in gerrit's workflow is indication anyone can submit. so i think i should probably +1 this and just check with luca/david they are ok with it | 23:02 |
| ianw | but lgtm | 23:04 |
| clarkb | I too just left a +1 based on ^ | 23:04 |
| clarkb | and double checking with them seems like a good idea | 23:04 |
| corvus | i added luca and david as reviewers (as that does seem to be the practice in the gerrit community) | 23:04 |
| ianw | we can make a retrospective tag before this change, if that would help | 23:05 |
| clarkb | maybe? I don't know what typical practice for plugin installs is. We have the flexibility to do branches, tags, refs, or shas I think | 23:07 |
| ianw | yeah, it would depend on what the other users are deploying with if a tag is easier. | 23:08 |
| clarkb | fungi: just saw that https://review.opendev.org/c/opendev/system-config/+/785675 (stop using our ubuntu ppa on debian for openafs) is passing now \o/ | 23:14 |
| clarkb | ianw: ^ that should be an easy review and a cleanup from the great openafs clock bug | 23:14 |
| corvus | ianw: i imagine the latest builds from gerritforge is a pretty common deployment method | 23:14 |
| corvus | i think that's what happens if you use the plugin manager too? | 23:15 |
| fungi | clarkb: yeah, earlier failures were just arm testing shenanigans, the change was working | 23:15 |
| clarkb | corvus: looks like the plugin manager tries to match gerrit versions to plugin versions, but this plugin doesn't do gerrit specific versioning so I suspect it is functionally equivalent to installing latest | 23:16 |
| corvus | ya | 23:23 |
| *** tosky has quit IRC | 23:34 | |
| openstackgerrit | Merged opendev/system-config master: Drop Debian PPA from openafs-client role https://review.opendev.org/c/opendev/system-config/+/785675 | 23:44 |
| ianw | https://review.opendev.org/c/openstack/openstacksdk/+/786148 is my guess at fixing the cloud-launcher security group issues with OSU OSL | 23:49 |
| ianw | i guess now i have some sort of solution i feel a bit better about either pinning openstacksdk or creating the rules by hand | 23:49 |
| *** DSpider has quit IRC | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!