| *** ryohayakawa has joined #opendev | 00:01 | |
| ianw | ok, old git servers should be gone | 00:05 |
|---|---|---|
| ianw | if anyone wants to point out anything else to remove i can before i close the window | 00:05 |
| *** Dmitrii-Sh has quit IRC | 00:20 | |
| clarkb | I still see the git servers | 00:25 |
| *** Dmitrii-Sh has joined #opendev | 00:25 | |
| clarkb | elasticsearch01.openstack.org can also be removed | 00:26 |
| ianw | hmm, yes, i wonder how i get rid of them, they're not in the host list any more | 00:30 |
| clarkb | maybe we need to rerun the generating script? | 00:32 |
| ianw | i think it's "graph trees" and then go in there and delete one by one | 00:36 |
| ianw | ok that seems to have got it | 00:40 |
| ianw | Silverblue/x86_64/os/images/install.img ... we probably want to cut this out | 00:50 |
| ianw | fedora is releasing now | 01:21 |
| ianw | i think the mit mirror is fine, but we should keep an eye on rawhide mirroring if it's timing out or causing problems | 01:21 |
| fungi | k, thanks for continuing with it! | 01:49 |
| fungi | i guess once the vos release completes we can recheck some failing dib changes | 01:50 |
| ianw | zuul-jobs too | 01:51 |
| ianw | ok, mirroring stopped, fedora jobs back to workin gin zuul-gate at least | 02:41 |
| *** zbr has quit IRC | 03:25 | |
| *** zbr has joined #opendev | 03:26 | |
| *** Eighth_Doctor has quit IRC | 03:29 | |
| *** mordred has quit IRC | 03:29 | |
| *** Eighth_Doctor has joined #opendev | 03:36 | |
| *** mordred has joined #opendev | 04:08 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add focal testing https://review.opendev.org/746629 | 04:19 |
| *** ysandeep|away is now known as ysandeep | 04:21 | |
| *** raukadah is now known as chkumar|rover | 04:25 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Makes EFI images bootable by bios https://review.opendev.org/743243 | 04:41 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: update gentoo to allow building arm64 images https://review.opendev.org/746000 | 04:50 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [dnm] trigger bridge jobs https://review.opendev.org/746631 | 04:59 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add focal testing https://review.opendev.org/746629 | 05:28 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [dnm] trigger bridge jobs https://review.opendev.org/746631 | 05:28 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add focal testing https://review.opendev.org/746629 | 05:35 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [dnm] trigger bridge jobs https://review.opendev.org/746631 | 05:35 |
| openstackgerrit | Merged openstack/diskimage-builder master: Handle NetworkManager for dhcp-all-interfaces https://review.opendev.org/745698 | 05:35 |
| openstackgerrit | Merged openstack/diskimage-builder master: source-repositories: git is a build-only dependency https://review.opendev.org/745678 | 06:07 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/746636 | 06:08 |
| ianw | clarkb: might be good to get focal testing in zuul-jobs with https://review.opendev.org/746629 before any more bitrot | 06:27 |
| *** ysandeep is now known as ysandeep|brb | 06:28 | |
| *** ysandeep|brb is now known as ysandeep | 06:48 | |
| *** Baragaki has joined #opendev | 07:08 | |
| *** Baragaki has left #opendev | 07:09 | |
| *** lpetrut has joined #opendev | 07:23 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/746636 | 07:37 |
| *** tosky has joined #opendev | 07:40 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fedora 32 support https://review.opendev.org/737217 | 07:43 |
| *** DSpider has joined #opendev | 07:49 | |
| *** andrewbonney has joined #opendev | 07:55 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Makes EFI images bootable by bios https://review.opendev.org/743243 | 07:59 |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Update name of ipa job https://review.opendev.org/743042 | 08:04 |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove glance-registry https://review.opendev.org/739796 | 08:12 |
| *** Eighth_Doctor has quit IRC | 08:12 | |
| *** mordred has quit IRC | 08:12 | |
| *** hashar has joined #opendev | 08:13 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add focal testing https://review.opendev.org/746629 | 08:21 |
| *** Eighth_Doctor has joined #opendev | 08:22 | |
| openstackgerrit | Denis proposed zuul/zuul-jobs master: terraform: Add parameter for plan file https://review.opendev.org/746657 | 08:26 |
| *** priteau has joined #opendev | 08:35 | |
| openstackgerrit | Denis proposed zuul/zuul-jobs master: terraform: Add parameter for plan file https://review.opendev.org/746657 | 08:37 |
| *** mordred has joined #opendev | 08:54 | |
| openstackgerrit | Denis proposed zuul/zuul-jobs master: terraform: Add parameter for plan file https://review.opendev.org/746657 | 08:59 |
| *** tkajinam has quit IRC | 09:58 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Do not install python2 packages in ubuntu focal https://review.opendev.org/745665 | 11:28 |
| *** hipr_c has joined #opendev | 11:39 | |
| *** hipr_c has joined #opendev | 11:43 | |
| *** hipr_c has joined #opendev | 11:43 | |
| auristor | clarkb: sorry I disappeared yesterday. The rxdebug output shows that at the time of the query, there were no active calls. | 11:46 |
| *** hipr_c has joined #opendev | 11:51 | |
| auristor | The history from 2020-07-20 that ianw linked to showed the fileserver being unable to access the callback service in response to receiving the first call on a new rx connection. the failure was most likely due to a nat port mapping timing out after 60s. | 11:56 |
| *** hipr_c has quit IRC | 12:00 | |
| *** hipr_c has joined #opendev | 12:25 | |
| *** hipr_c has joined #opendev | 12:28 | |
| *** ysandeep is now known as ysandeep|brb | 12:30 | |
| *** hipr_c has quit IRC | 12:34 | |
| *** hipr_c has joined #opendev | 12:35 | |
| *** ysandeep|brb is now known as ysandeep | 12:44 | |
| *** priteau has quit IRC | 13:06 | |
| *** ryohayakawa has quit IRC | 13:23 | |
| *** tkajinam has joined #opendev | 13:50 | |
| *** olaph has joined #opendev | 13:53 | |
| openstackgerrit | Vanou Ishii proposed opendev/puppet-openstackci master: Remove Empty Top Namespace from Puppet Classes https://review.opendev.org/746726 | 14:00 |
| *** mlavalle has joined #opendev | 14:01 | |
| corvus | clarkb: starlink launch in 1.5m | 14:29 |
| * clarkb has a call... | 14:30 | |
| corvus | also planet sat payload | 14:30 |
| *** lpetrut has quit IRC | 14:48 | |
| *** qchris has quit IRC | 14:56 | |
| *** chkumar|rover is now known as raukadah | 14:57 | |
| *** priteau has joined #opendev | 15:04 | |
| *** qchris has joined #opendev | 15:09 | |
| *** lpetrut has joined #opendev | 15:23 | |
| clarkb | and 6th launch for the booster | 15:31 |
| *** ysandeep is now known as ysandeep|away | 16:05 | |
| *** olaph has quit IRC | 16:10 | |
| openstackgerrit | Emilien Macchi proposed openstack/project-config master: Re-introduce puppet-tripleo-core group https://review.opendev.org/746759 | 16:21 |
| *** fbo|pto has joined #opendev | 16:53 | |
| clarkb | I upgraded my local test gerrit from 2.16 to 3.0 and change file diffs are all empty :/ Now I need to retest everything and make sure that isn't broken through 2.16 | 16:54 |
| clarkb | there was an exception on startup which could be related I guess | 16:55 |
| clarkb | also the db isn't auto cleaned when you go to 3.0 | 16:55 |
| openstackgerrit | Jeremy Stanley proposed opendev/infra-specs master: Central Authentication Service https://review.opendev.org/731838 | 16:56 |
| fbo|pto | Hi, any thoughs moving from Gitea to Pagure for opendev.org ? Just relaying the question from twitter: https://twitter.com/Det_Conan_Kudo/status/1295706502909562882 | 16:57 |
| clarkb | I'm not sure what the benefits would be? | 16:57 |
| clarkb | we don't connect zuul to gitea | 16:58 |
| clarkb | nor would we to pagure I imagine | 16:58 |
| clarkb | and if we did maybe I should stop working on this gerrit upgrade testing | 16:58 |
| fungi | fbo|pto: people wanted a "github-like" code browser instead of cgit, and gitea seemed to fit the bill at the time. replacing it at this point would be a lot of work when all we need is a read-only web display of git repos | 16:58 |
| fungi | the features he's suggesting there (issue tracking and documentation publishing) are features we already don't enable in gitea for good reason, not because gitea is bad at them but because they're not a fit for our service model | 17:01 |
| fbo|pto | clarkb: fungi thanks for the explanation. Would you like me to respond this one twitter. Or you manage a opendev account on twitter (I'm not aware of any btw) and you'd like to reply ? | 17:01 |
| fungi | fbo|pto: twitter is a proprietary service, so i don't think many of the regulars in here use it (not sure why someone would try to communicate with us on twitter to start with) | 17:02 |
| fungi | i think the osf marketing team may operate an opendev twitter handle but i wouldn't begin to know how to get access to it | 17:03 |
| fungi | if someone really wants to initiate a discussion about getting involved in helping us maintain services, we have this irc channel and also the service-discuss@lists.opendev.org mailing list they can reach out through... both linked from the https://opendev.org/ [age | 17:04 |
| fbo|pto | fungi: :) that's fair, I'll replay your explanation there on proprietary service and invite to join on #opendev for other questions. | 17:04 |
| fungi | er, page | 17:04 |
| clarkb | huh I restarted to change gerrit's plugin manager config since it was the thing throwing exceptions. Now the diffs render. But I still get plugin manager exceptions | 17:06 |
| fungi | what sort of exceptions? | 17:06 |
| clarkb | [2020-08-18 17:03:23,645] [plugin-manager-preloader] ERROR com.google.gerrit.pgm.Daemon : Thread plugin-manager-preloader threw exception com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException | 17:07 |
| clarkb | I don't think thats super critical as we manage our plugins via the docker images anyway | 17:07 |
| clarkb | can likely just drop the plugin manager entirely | 17:07 |
| clarkb | what I don't understand is why the diffs suddenly work | 17:07 |
| clarkb | there were no other exceptions and I waited for reindexing to complete before browsing | 17:07 |
| clarkb | I'm going to restart again and see if it flip flops | 17:08 |
| clarkb | but also I want to add some inline comments and run through the upgrades again anyway | 17:08 |
| clarkb | that way I can check diffs on the older versions as we go | 17:08 |
| fungi | infra-root: rackspace has notified us that they'll be performing disruptive block storage maintenance in october impacting 29 of our current volumes, and that we can avoid downtime by following these migration instructions: https://support.rackspace.com/how-to/minimizing-the-impact-of-cloud-block-storage-maintenance/ | 17:10 |
| clarkb | fungi: I wonder if we create new volumes now, attach, lvm mirror, then detach and clean up the old volumes if that would work too | 17:11 |
| fungi | clarkb: yep, they also note that any new volumes created are going to backends which won't be impacted by the maintenance | 17:11 |
| fungi | i wouldn't bother with lvm mirror, we can just attach a new volume and then pvmove to it | 17:12 |
| fungi | that can happen live, and is safely interruptable as long as you don't remove the volumes prematurely | 17:12 |
| clarkb | cool. If the es servers are in that list maybe we just turn them off. review.o.o's volume as well as eavesdrop should get the pvmove treatment likely | 17:13 |
| fungi | pvmove is how i've done a lot of our other volume replacements in the past | 17:13 |
| fungi | right, i figured there were some servers where if they choke it's fine, and others where we'll want to avoid an outage | 17:14 |
| fungi | unfortunately the list is all uuids, so i haven't yet checked to see which systems will be affected | 17:14 |
| *** hashar is now known as hasharDinner | 17:16 | |
| * clarkb does a new set of test gerrit upgrades from 2.13 with inline comments | 17:17 | |
| fungi | ugh, also they sent the list in html-only e-mail, making it hard to just open in a text editor and write a script around | 17:24 |
| * fungi loses faith in humanity | 17:24 | |
| clarkb | adding inline comments populates the accountPatchReviewDb DB | 17:25 |
| clarkb | thats one mystery solved at least | 17:25 |
| *** lpetrut has quit IRC | 17:29 | |
| *** andrewbonney has quit IRC | 17:36 | |
| *** lseki has joined #opendev | 17:54 | |
| *** ajitha has joined #opendev | 17:56 | |
| clarkb | ah the accountPatchReviewDb is there to track the little check mark next to files to say you've seen them or not | 17:58 |
| clarkb | now I GET IT | 17:58 |
| ajitha | Hi all,Today, I have login issue with https://review.opendev.org/.. | 17:59 |
| ajitha | ssh -p 29418 <username>@review.opendev.org | 17:59 |
| ajitha | Connection reset by 104.130.246.32 port 29418 | 17:59 |
| clarkb | ajitha: you may have hit our connection limit | 17:59 |
| clarkb | ajitha: is this for a CI system? | 18:00 |
| ajitha | yes | 18:00 |
| ajitha | No, its for updating a patch. | 18:00 |
| ajitha | clarkb | 18:00 |
| fungi | i can check gerrit's ssh log | 18:01 |
| clarkb | ajitha: is the account shared wtih a CI system? we often see this with CI systems ending up with stale connections | 18:01 |
| clarkb | and we allow 96 iirc? | 18:01 |
| clarkb | we also have a per ip limit but its great than that I think | 18:01 |
| fungi | 100 simultaneous established connections per ip address | 18:01 |
| ajitha | clarkb : how to check connection limit. I am in a personal laptop. I will try once by disabling VPN | 18:02 |
| clarkb | ajitha: fungi is checking | 18:02 |
| ajitha | ok | 18:02 |
| fungi | ajitha: if i knew your gerrit username, that would help. the server is very busy so quite a long log | 18:03 |
| ajitha | fungi:ajitharobert | 18:03 |
| fungi | huh, okay, i had already checked the log for "ajitha" and there were no matches, so it's not getting as far as exchanging credentials at least | 18:04 |
| fungi | i so see some successful logins from you yesterday (in an earlier rotated logfile) | 18:05 |
| fungi | er, i do see | 18:05 |
| fungi | last successful activity was at 06:11:39 utc yesterday | 18:05 |
| ajitha | yes. yesterday i could login | 18:06 |
| clarkb | It could be the ip restriction if on corp VPN and that goes through NAT? | 18:06 |
| fungi | i also don't see any activity from the same ip address as you were coming from yesterday | 18:06 |
| clarkb | its also possible the corp firewall isn't allowing 29418 out | 18:07 |
| ajitha | i disconnected the VPN now | 18:07 |
| clarkb | ajitha: is it still erroring or does it work after that? | 18:11 |
| fungi | also i don't think it's the connection limit because the error was "Connection reset" (so tcp rst packet) while we tell iptables to reject those with icmp port unreachable | 18:12 |
| ajitha | still not accessible. some updates happened from AD.. it is possible to block requests from laptop | 18:12 |
| ajitha | ? | 18:12 |
| zbr | clarkb: what is the next step regarding e-r? | 18:13 |
| fungi | anything's possible. for example if this is a work laptop with a mandatory network firewall installed on it and it specifies allowed egress destination ports/addresses then it could do what you're seeing | 18:13 |
| ajitha | yes right.. thanks.. i shall contact the respective admin | 18:14 |
| fungi | ajitha: but the short answer is that it doesn't look like your connection attempts are ever reaching our server | 18:14 |
| ajitha | yes got it. blocking from my side right? | 18:15 |
| fungi | ajitha: that's what it looks like to me at least | 18:15 |
| ajitha | fungi: thank you | 18:15 |
| ajitha | clarkb: thank you | 18:16 |
| fungi | you mentioned "updates from ad" so if this is a microsoft windows machine it's entirely possible network connectivity policy is being managed from your employer's active directory | 18:16 |
| ajitha | yes right | 18:16 |
| clarkb | zbr: did gmann ever respond? | 18:16 |
| clarkb | I think that was the last person we were waiting on? | 18:16 |
| gmann | clarkb: zbr on which one? | 18:17 |
| clarkb | gmann: e-r maintainership | 18:17 |
| clarkb | gmann: I think tripleo (and zbr) are offering to take that on | 18:17 |
| clarkb | I wanted to make sure the qa team didn't want to own it first | 18:17 |
| gmann | clarkb: i did respond on qa channel with OK. | 18:17 |
| gmann | I can do on ML also if i missed if there is any? | 18:18 |
| clarkb | gmann: ya there was an opendev service-discuss thread. If you are subscribed and can reply that would be great | 18:19 |
| clarkb | but if not thats probably fine. I can modify the core group | 18:19 |
| fungi | is anybody able to do `openstack volume list` from rackspace at the moment? i'm getting "Not Found (HTTP 404)" when i try, which is making it really hard to figure out which volumes are affected by the upcoming maintenance. also if i do `openstack volume show <some uuid>` i get "No volume with a name or ID of '...' exists." (with the uuid i specified instead of ...) | 18:20 |
| fungi | i wonder if they've broken their volume api for all existing volumes | 18:21 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove tmp gerrit plugins from our docker images https://review.opendev.org/746784 | 18:22 |
| clarkb | fungi: its an api version thing | 18:23 |
| clarkb | I thought I fixed it | 18:23 |
| clarkb | oh ya I remember now | 18:23 |
| clarkb | the original proposed fix was to pin to volumes v1 | 18:23 |
| clarkb | but mordred pointed out they actually have avolumes v2 its just not in the cinder catalog | 18:24 |
| clarkb | so we updated the clouda.yaml to override the volumes url to the v2 endpoint that isn't in the catalog | 18:24 |
| clarkb | fungi: I would try using v1 api instead | 18:24 |
| fungi | well, it's moot for the moment. i just pulled up dfw for our tenant in the rackspace dashboard and we have 29 volumes | 18:24 |
| fungi | so i'm going to assume the 29 uuids they put in the ticket represent every volume | 18:25 |
| clarkb | gmann: http://lists.opendev.org/pipermail/service-discuss/2020-August/000071.html is the thread | 18:25 |
| clarkb | gmann: just let me know if you think I should modify the core group without an email and I'll do that | 18:25 |
| gmann | clarkb: done, replied on ML also. | 18:25 |
| clarkb | excellent thanks | 18:25 |
| fungi | clarkb: which clouds.yaml did we tweak for that? i'm just doing `sudo openstack --os-cloud openstackci-rax --os-region-name DFW volume list` from bridge so it should be using the /etc/openstack/clouds.yaml we install there | 18:26 |
| clarkb | fungi: yes that one | 18:27 |
| clarkb | you'll probably need to make a copy, edit the contents and point osc at the copy to confirm v1 works | 18:27 |
| fungi | so v2 should be working with that one? | 18:27 |
| clarkb | fungi: yes, we updated to make v2 work despite it being in the catalog, but it is possible it wasn't in the catalog for a reason | 18:27 |
| clarkb | zbr: https://review.opendev.org/#/admin/groups/218,members has been updated | 18:27 |
| gmann | clarkb: sorry for missing the ML reply, i thought IRC confirmation is enough. | 18:27 |
| clarkb | #status log Updated elastic-recheck-core to include tripleo-ci-core as the tripleo-ci team intends to maintain the elastic-recheck tool | 18:28 |
| openstackstatus | clarkb: finished logging | 18:28 |
| zbr | thanks! clarkb i will bother you around switching to container use once I am ready with, ok? | 18:29 |
| clarkb | fungi: its also possible the openstacksdk release isn't new enough for that to work? it depends-on https://review.opendev.org/#/c/714624/ | 18:29 |
| clarkb | zbr: ok | 18:29 |
| fungi | the command-line help for osc is not forthcoming in how to override an api version. possible i'm looking in the wrong places | 18:30 |
| clarkb | ya its part of sdk | 18:30 |
| clarkb | I still think ist a bug and have to look in my history to figure it out | 18:30 |
| clarkb | fungi: look in clarkb's bridge history for openstack commands | 18:30 |
| clarkb | should be something like OS_CLIENT_CONFIG=/path | 18:30 |
| clarkb | fungi: https://review.opendev.org/#/c/714553/5/playbooks/templates/clouds/bridge_all_clouds.yaml.j2 that diff shows you how to use v1 | 18:31 |
| fungi | oh, you mean altering the clouds.yaml to use v1, got it. thought maybe there was a cli option | 18:31 |
| fungi | yeah, i know how to specify an alt config file | 18:32 |
| fungi | looks like we need python-openstackclient>=5.1.0 for that change | 18:32 |
| fungi | we're running 5.2.1 from the container | 18:33 |
| fungi | so change 714624 is there | 18:33 |
| *** hasharDinner has quit IRC | 18:40 | |
| fungi | clarkb: switching it to v1 doesn't seem to help either | 18:43 |
| fungi | either that or setting envvars doesn't percolate into the docker container? | 18:43 |
| clarkb | fungi: I would try adding --debug to the client call and see what it is trying to talk to | 18:43 |
| clarkb | oh ya if its in the containe I doubt that helps | 18:44 |
| clarkb | you'd have to put the env var in the invocation | 18:44 |
| clarkb | on the other side of the docker run/exec | 18:44 |
| fungi | maybe i'll just use a virtualenv | 18:45 |
| clarkb | I've got on in my homedir. ~clarkb/venv/bin/openstack iirc | 18:45 |
| fungi | i do too, but i had stopped using it because we had new osc installed | 18:46 |
| * clarkb needs to pop out and get a quick snack before the meeting. Back shortly for that | 18:46 | |
| fungi | unfortunately installed in a way that makes it hard to set behaviors with envvars | 18:46 |
| fungi | yep, i'm able to get to rackspace volume api by reverting 714553 in a copy of clouds.yaml and using osc from a virtualenv | 18:50 |
| fungi | annoyingly `openstack volume show -c Name` doesn't seem to work, returns blank entries even though `openstack volume show <id>` gives a name in its output | 18:58 |
| fungi | i guess i'll have to scrape these in a loop | 18:58 |
| *** diablo_rojo has joined #opendev | 18:59 | |
| *** ajitha has quit IRC | 19:01 | |
| *** hashar has joined #opendev | 19:09 | |
| *** priteau has quit IRC | 19:44 | |
| *** diablo_rojo has quit IRC | 19:48 | |
| clarkb | oh neat the /var/gerrit/tmp is mounted into the container so its actually becuse I'm doing upgrades that I see that I think | 20:05 |
| clarkb | so thats actuall a non issue (not great to have the noise but I can ignore it I think) | 20:05 |
| clarkb | next stop skip level upgrades | 20:05 |
| * clarkb finds lunch first | 20:06 | |
| openstackgerrit | Emilien Macchi proposed openstack/project-config master: Re-introduce puppet-tripleo-core group https://review.opendev.org/746759 | 21:04 |
| clarkb | testing seems to show we can do a "skip level" upgrade by running the init for 2.14, init for 2.15, init for 2.16, then reindex everything, then start | 21:15 |
| clarkb | I'm 99% certain we can't do online reindexing if we do that | 21:15 |
| fungi | might be interesting to try. the offline reindex is going to add an order of magnitude more downtime (at least) | 21:16 |
| clarkb | ya I'll give it a go I guess. There are a couple of things we have to reindex manually anyway, so will do those but then see if online reindexing works I guess | 21:16 |
| clarkb | though I think I'm going to take a berak and do some other things as I've basically spent all day doing gerrit stuff | 21:17 |
| fungi | i've got a one evening window to mow my lawn before the rains return, so trying to knock out stuff on my to do list when resting between laps | 21:20 |
| *** hashar has quit IRC | 21:37 | |
| clarkb | ok I've requested PTG space and signed us up for the same blocks of time. The one thing I need to double check is DST. I know here in the USA we don't switch until like november so we should be fine but that may not be true for europe | 21:52 |
| * clarkb tries to do timezone math and not get it wrong | 21:53 | |
| clarkb | ya looks like october 25th for europe dropping summer time | 21:53 |
| clarkb | oh and australia starts summer time first sunday in october | 21:55 |
| clarkb | more maths | 21:55 |
| clarkb | ianw: you'll be UTC+11 last week of october ya? | 22:03 |
| clarkb | doing maths I think we should consider moving the 04:00 - 0:600 block to 05:00-07:00. That will be slightly more EU friendly and its all bad for fungi :) | 22:11 |
| clarkb | but I think australia and china can handle that shift without much trouble | 22:11 |
| * clarkb makes that small edit to the ethercalc | 22:11 | |
| clarkb | the other times are probably fine as is | 22:12 |
| ianw | clarkb: yeep, seems we switch on 4th oct | 22:13 |
| clarkb | thanks for confirming. I've requested Monday the 26th 13:00 - 15:00UTC Monday 26th 23:00 - Tuesday 27th 01:00 UTC and Wednesday 05:00 - 07:00. The only change from last time is moving wednesday from 04:00-06:00 to 05:00-07:00 to be slightly more europe friendly | 22:15 |
| clarkb | their switch from UTC+2 to UTC+1 means 04:00-06:00 is incredibly early | 22:15 |
| fungi | 05:00-07:00 before usa dst ends? | 22:16 |
| fungi | or after? | 22:16 |
| fungi | i already got confused | 22:16 |
| clarkb | before usa dst ends after eu dst ends | 22:16 |
| fungi | ahh, before usa dst ends | 22:16 |
| fungi | yeah | 22:16 |
| clarkb | PTG happens in the week of limbo | 22:16 |
| clarkb | fungi: I don't really expect you to show up for that one | 22:17 |
| clarkb | but its "early" enough I should be able to make it | 22:17 |
| fungi | so 05:00-07:00 is basically 1-3am here. i'll survive ;) | 22:17 |
| fungi | maybe i'll skip. maybe i'll see it as an excuse to play video games until 1am. who knows? | 22:17 |
| *** olaph has joined #opendev | 22:18 | |
| *** tosky has quit IRC | 22:18 | |
| clarkb | fungi: https://www.gerritcodereview.com/2.16.html#reindex-for-new-projects-index-and-changed-group-index that notes you need to do an offline reindex of changes and accounts if not upgrading from 2.15 | 22:32 |
| clarkb | I've just tested the skip upgrade with online reindexing and it breaks beacuse I need to reindex changes | 22:32 |
| clarkb | I beleve that is the costly one | 22:33 |
| clarkb | I think that means we have to do the offlien reindex | 22:33 |
| *** mlavalle has quit IRC | 22:33 | |
| * clarkb tries to figure out what all the indexes are now | 22:34 | |
| fungi | ahh, yeah, too bad | 22:35 |
| fungi | if memory serves, offline reindexing nova's changes was somewhere around 10 hours last time we did it? | 22:35 |
| clarkb | accounts, groups, changes, projects | 22:35 |
| clarkb | I think it is the changes and projects reindexes that are slow? | 22:35 |
| fungi | yep | 22:35 |
| fungi | anyway, we should probably just plan for and announce a 24-hour outage to be safe | 22:36 |
| clarkb | ya so the alternative is to do shorter outages and step through 2.14 and 2.15 in order to get to 2.16 | 22:36 |
| clarkb | and basically let each one online reindex | 22:36 |
| clarkb | (which is slower iirc becuse it has to compete with doing the regular server duties) | 22:36 |
| fungi | and basically have two maintenances, the first where we get everything upgraded and start the offline reindex, the second where we try to bring gerrit back online the next day | 22:36 |
| clarkb | probably figure 2-3 days for each of those | 22:36 |
| fungi | so, yeah, several short-ish outages plus possible bugs from now-no-longer-fixable gerrit versions? or one huge outage? | 22:37 |
| clarkb | yup. I think I prefer the huge otuage | 22:37 |
| fungi | the big outage would be more downtime overall, but probably also safer | 22:38 |
| fungi | anyway, we *might* have a better idea of how long after we try a production dry-run on review-test | 22:39 |
| clarkb | ya that should give us a good ballpark. Anyway trying to write up the distilled version of all this at https://etherpad.opendev.org/p/gerrit-2.16-upgrade next then will respond to luca and see if he can look it over once I think its good | 22:39 |
| fungi | like i could be around for a late-night (my time) start maintenance and then a mid-morning (my time) fire-up of 2.16 after the reindex | 22:39 |
| fungi | so it could start in your evening and then finish the next morning once you're up, say | 22:40 |
| fungi | we might also want a temporary maintenance page for an outage of that duration. people will be surprised by it no matter how much communication we provide ahead of time | 22:41 |
| *** dviroel has quit IRC | 22:43 | |
| *** ildikov has quit IRC | 22:43 | |
| *** knikolla has quit IRC | 22:43 | |
| *** aannuusshhkkaa has quit IRC | 22:43 | |
| clarkb | ++ | 22:44 |
| clarkb | fungi: want to look over https://etherpad.opendev.org/p/gerrit-2.16-upgrade really quickly? I've got a couple other questions I'll send luca's way as well | 22:46 |
| *** olaph has quit IRC | 22:48 | |
| *** olaph has joined #opendev | 22:49 | |
| fungi | yeah, just a sec | 22:52 |
| *** aannuusshhkkaa has joined #opendev | 22:54 | |
| *** dviroel has joined #opendev | 22:55 | |
| *** ildikov has joined #opendev | 22:55 | |
| *** knikolla has joined #opendev | 22:55 | |
| fungi | any point in dropping db tables if the notedb migration is on the horizon anyway? we'll drop the whole db when that comes | 22:59 |
| clarkb | fungi: I think that doesn't happen until 3.0 so cleaning up early seems like good hygiene | 22:59 |
| clarkb | (I mean i want to get to 3.x as quickly asp ossible but..) | 23:00 |
| fungi | ahh, the sql db is still used for some things in 2.16 even after notedb migration? | 23:03 |
| clarkb | yes | 23:04 |
| fungi | neat | 23:04 |
| clarkb | at least that seemed to be what luca said in his response to my first email | 23:04 |
| clarkb | he says its 3.0 where you can drop the db | 23:04 |
| fungi | draft plan lgtm | 23:04 |
| clarkb | cool firing off another email to luca now then | 23:04 |
| fungi | thanks! | 23:04 |
| fungi | for some reason i read his previous reply as indicating that 3.0 no longer supported using the db, but not necessarily that 2.16 required it if using notedb | 23:05 |
| fungi | anyway, better safe than sorry. it doesn't really lose us anything | 23:06 |
| *** shtepanie has joined #opendev | 23:06 | |
| clarkb | fungi: one of my questions was why does the chagnes table remain and he said that you can drop the whole db after 3.x aiui | 23:06 |
| clarkb | ebcause in testing that was one of the things I wanted to check to ensure that notedb migration worked | 23:06 |
| clarkb | it does turn out that no new changes end up in the chagnes table after the notedb migration so tahts how I've been checking | 23:06 |
| fungi | ahh, okay. fair enough | 23:10 |
| fungi | that's ambiguous, but better to err on the side of caution | 23:10 |
| *** ryohayakawa has joined #opendev | 23:38 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!