| ianw | https://gerrit-review.googlesource.com/Documentation/rest-api-accounts.html#get-account-external-ids ... in theory i guess | 00:07 |
|---|---|---|
| fungi | yeah, i linked the response data structure doc section in the etherpad | 00:25 |
| fungi | in the ptg etherpad i mean | 00:25 |
| ianw | https://hub.docker.com/r/kristophjunge/test-saml-idp/ | 00:27 |
| fungi | https://simplesamlphp.org/ is the base component knikolla suggested in our earlier discussion, linked in the spec | 00:30 |
| *** hamalq has quit IRC | 00:30 | |
| *** qchris has quit IRC | 00:42 | |
| clarkb | mnaser: we've discovered it would be good to talk to you about running k8s for infra things. We are about at time today and our block "tomorrow" likely isn't in a good spot for your timezone. Is there some time tomorrow when you might be available to jump on meetpad and talk about that? | 00:43 |
| *** qchris has joined #opendev | 00:56 | |
| *** mugsie has quit IRC | 01:00 | |
| ianw | fungi: for when you have time, i think the reprepro mirrors to ansible is gtg : https://review.opendev.org/#/c/757660/ ... all i've done is turn off the cron job as suggested, and add some basic testinfra since you last looked | 01:02 |
| fungi | clarkb: so just to confirm, i'm showing it's 05:00-07:00 utc wednesday, which will be 9-11pm tuesday pacific daylight time | 01:03 |
| fungi | (1-3am wednesday eastern daylight for me) | 01:03 |
| fungi | oh, sorry, 10pm-midnight pacific | 01:04 |
| *** mugsie has joined #opendev | 01:04 | |
| fungi | ianw: awesome, i'll try to take a look before i pass out | 01:05 |
| clarkb | fungi: thabks | 01:06 |
| fungi | ianw: i guess it also got rebased at some point in the last six patch sets too | 01:31 |
| fungi | in ps24 apparently | 01:34 |
| fungi | ianw: lgtm! are you okay if i approve it now or do you think it needs more reviews first? i think it should be safe since it's just deploying unused files initially and not adding the cronjob | 01:38 |
| ianw | fungi: sorry, getting lunch. i can approve and watch it, and do a bit of testing | 01:52 |
| ianw | the other thing i might just do now as well is cleanup the debian-ceph mirrors https://review.opendev.org/#/c/758517/ | 01:53 |
| ianw | looking at the abuse response for google cloud it seems like you get a warning; https://support.google.com/cloud/answer/7002354?hl=en and also the abuse form has space to put free-form text in | 01:56 |
| ianw | i think i might just report the two IP's as we have pretty specific info; with the request that they get the account owner to contact us to work something out | 01:56 |
| ianw | i don't expect them to get blocked ... but if it continues, we can look at a layer 7 approach of blocking the queries | 01:57 |
| fungi | i went ahead and approved it | 02:02 |
| *** DSpider has quit IRC | 02:07 | |
| *** zbr has quit IRC | 02:51 | |
| openstackgerrit | Merged opendev/system-config master: reprepro: convert to Ansible https://review.opendev.org/757660 | 02:57 |
| openstackgerrit | Merged opendev/system-config master: Remove old debian-ceph mirrors https://review.opendev.org/758517 | 02:57 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ARM64 : run base test on Focal too https://review.opendev.org/756629 | 02:58 |
| *** zbr has joined #opendev | 03:12 | |
| openstackgerrit | likui proposed openstack/diskimage-builder master: Remove the unused coding style modules https://review.opendev.org/759791 | 03:38 |
| *** auristor has quit IRC | 03:43 | |
| *** auristor has joined #opendev | 03:47 | |
| ianw | #status log removed ceph h/j/l/m AFS volumes and mirroring jobs | 04:02 |
| openstackstatus | ianw: finished logging | 04:02 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: reprepro: fixup script name https://review.opendev.org/759803 | 04:10 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: reprepro: install keytab https://review.opendev.org/759804 | 04:17 |
| *** ykarel has joined #opendev | 04:31 | |
| *** ykarel has quit IRC | 04:35 | |
| *** ykarel has joined #opendev | 04:36 | |
| *** ykarel has quit IRC | 05:10 | |
| *** ykarel has joined #opendev | 05:16 | |
| *** ykarel_ has joined #opendev | 05:18 | |
| openstackgerrit | Merged opendev/system-config master: reprepro: fixup script name https://review.opendev.org/759803 | 05:19 |
| *** ykarel has quit IRC | 05:21 | |
| openstackgerrit | Merged opendev/system-config master: reprepro: install keytab https://review.opendev.org/759804 | 05:22 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: reprepro: run deploy job on role changes https://review.opendev.org/759810 | 05:30 |
| *** ysandeep|holiday is now known as ysandeep|ruck | 05:42 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: publish reprepro logs https://review.opendev.org/759811 | 05:45 |
| *** zbr1 has joined #opendev | 06:03 | |
| *** marios has joined #opendev | 06:03 | |
| *** zbr has quit IRC | 06:06 | |
| *** zbr1 is now known as zbr | 06:06 | |
| *** marios has quit IRC | 06:31 | |
| *** fressi has quit IRC | 06:32 | |
| *** marios has joined #opendev | 06:44 | |
| openstackgerrit | Rico Lin proposed opendev/system-config master: Remove TW User Group ML https://review.opendev.org/584035 | 07:03 |
| ttx | ianw: re: that crawler, I'll need a bit more information to narrow it down. Is it walking through all changes, or just a period of time / specific repositories? Also is there a pattern in the timing? | 07:04 |
| ianw | ttx: it's pretty much going as fast as it can | 07:05 |
| ttx | Re: Bitergia they are only gathering stats on StarlingX and Zuul | 07:05 |
| ttx | ianw: is it more like a daily cronjob? | 07:05 |
| ianw | the requests all look like "GET /changes/?q=status:merged&q=status:abandoned&o=ALL_REVISIONS&o=ALL_FILES&o=ALL_COMMITS&o=MESSAGES&o=DETAILED_ACCOUNTS&n=100&S=297400 HTTP/1.1" 200 209096 "-" "python-requests/2.23.0" | 07:05 |
| ttx | ah, that's a pretty wide net indeed | 07:06 |
| ianw | it walks every change | 07:06 |
| *** ralonsoh has joined #opendev | 07:07 | |
| ttx | OK, that's probably not them, if only because they have been around for a long time and we would have noticed earlier, but also because they should only walk very specific repos for very specific periods of time | 07:07 |
| openstackgerrit | Merged opendev/system-config master: reprepro: run deploy job on role changes https://review.opendev.org/759810 | 07:07 |
| ttx | But I'll ask, could be a bug | 07:07 |
| ttx | ianw: does it happen every day at the same time? If yes, what time is that? | 07:08 |
| *** sboyron has joined #opendev | 07:08 | |
| ianw | ttx: no, not regular, but the requests always look the same | 07:12 |
| openstackgerrit | Merged opendev/system-config master: mirror-update: publish reprepro logs https://review.opendev.org/759811 | 07:13 |
| *** marios has quit IRC | 07:19 | |
| *** eolivare has joined #opendev | 07:28 | |
| *** rpittau|afk is now known as rpittau | 07:42 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:46 | |
| *** ykarel_ is now known as ykarel | 07:56 | |
| *** andrewbonney has joined #opendev | 08:10 | |
| *** slaweq has joined #opendev | 08:11 | |
| *** lpetrut has joined #opendev | 08:11 | |
| *** ralonsoh has quit IRC | 08:13 | |
| *** ykarel has quit IRC | 08:13 | |
| *** ykarel has joined #opendev | 08:16 | |
| *** tosky has joined #opendev | 08:36 | |
| *** ralonsoh has joined #opendev | 08:37 | |
| *** DSpider has joined #opendev | 08:37 | |
| *** marios has joined #opendev | 08:43 | |
| *** manpreet has quit IRC | 09:01 | |
| *** ysandeep|lunch is now known as ysandeep|ruck | 09:01 | |
| *** fressi has joined #opendev | 09:04 | |
| *** webmariner has quit IRC | 09:06 | |
| *** hashar has joined #opendev | 09:31 | |
| frickler | infra-root: does anyone know about a recent change in focal images that could result in localhost getting resolved to ::1 now instead of 127.0.0.1 | 10:03 |
| * frickler is seeing designate jobs being broken because of that. there is an entry in /etc/hosts with "::1 localhost" but I'm not sure whether that might be new or why | 10:04 | |
| *** ykarel_ has joined #opendev | 10:06 | |
| *** ykarel has quit IRC | 10:09 | |
| *** ykarel_ is now known as ykarel | 11:17 | |
| *** sboyron has quit IRC | 11:45 | |
| fungi | do some jobs archive /etc/hosts? we could look back at earlier builds to compare them that way | 11:47 |
| fungi | my debian machines have included localhost as an alias on ::1 (in addition to 127.0.0.1) for ages | 11:48 |
| fungi | out of curiosity, why does that break jobs? | 11:50 |
| fungi | is it just that they're hard-coded to expect 127.0.0.1 or is there something only listening on 127.0.0.1 instead of ::1? | 11:55 |
| frickler | fungi: I think it is just this one https://review.opendev.org/759850 , memcached is listening on 127.0.0.1 explicitly. but I'm not sure why this is breaking only now and why it doesn't seem to affect any other service | 12:01 |
| *** lpetrut has quit IRC | 12:01 | |
| *** lpetrut has joined #opendev | 12:03 | |
| frickler | seems to have changed on 2020-10-23, not sure yet whether the change is in the focal image or in our setup phase | 12:07 |
| sean-k-mooney | fungi: fedora also has ::1 as an aias for local hosts | 12:48 |
| sean-k-mooney | i think its pretty common now | 12:48 |
| sean-k-mooney | frickler: the grenade job hit the current nova gate blocker | 12:50 |
| sean-k-mooney | TypeError: Parameterized generics cannot be used with class or instance checks | 12:51 |
| sean-k-mooney | https://review.opendev.org/#/c/759831/ should fix that | 12:51 |
| frickler | sean-k-mooney: yes, I've seen that issue, but I guess we need a stable backport of that fix in order to get grenade to work again? or does it only affect master? | 13:04 |
| frickler | I guess we'll see that on the grenade job for that patch itself | 13:06 |
| sean-k-mooney | is grenade using victoria for the base version | 13:08 |
| sean-k-mooney | i guess it is now | 13:08 |
| sean-k-mooney | so ya | 13:08 |
| fungi | master grenade will start with victoria | 13:08 |
| sean-k-mooney | i think this only affect 20.04 based jobs but that includes victoria so ya this need to be merged there first or grenade need to be made non voting | 13:09 |
| sean-k-mooney | actully | 13:10 |
| sean-k-mooney | i was going to say upper constiratign might help | 13:10 |
| sean-k-mooney | but no | 13:10 |
| sean-k-mooney | since its failing in that grenade job | 13:11 |
| frickler | fungi: how about we clean up our akick lists a bit, too? 6y old IPs likely aren't relevant any longer. fwiw I also added that haunted spammer to some more channels where I saw them | 13:26 |
| openstackgerrit | Michal Pryc proposed zuul/zuul-jobs master: Allow bindep role to install additional test requirements https://review.opendev.org/759868 | 13:48 |
| fungi | frickler: yeah, can't hurt to remove old entries there | 14:04 |
| *** sshnaidm|rover has quit IRC | 14:05 | |
| *** sshnaidm has joined #opendev | 14:05 | |
| *** sshnaidm is now known as sshnaidm|rover | 14:06 | |
| *** mlavalle has joined #opendev | 14:07 | |
| *** slaweq is now known as slaweq|ptg | 14:07 | |
| openstackgerrit | Michal Pryc proposed zuul/zuul-jobs master: Allow bindep role to install additional test requirements https://review.opendev.org/759868 | 14:24 |
| *** lpetrut has quit IRC | 14:32 | |
| *** lpetrut has joined #opendev | 14:33 | |
| *** Goneri has joined #opendev | 15:18 | |
| Goneri | Opendev is in Korean and it's anoying because I don't speak the language | 15:20 |
| Goneri | is there a way to change that? https://i.imgur.com/CqirrRh.png | 15:20 |
| frickler | Goneri: please try to clear your cookies. we have seen that before, but don't know exactly when or why this happens | 15:21 |
| clarkb | Goneri: yes there is a setting bottom of the page next to a glob looking icon | 15:21 |
| Goneri | oh it's indeed much better if I clean my cookies... :-) | 15:22 |
| Goneri | the settings button was in Korean too, so it don't really help. | 15:23 |
| Goneri | eheh :-) | 15:23 |
| frickler | is gerrit getting slowed down again? /me is in ptg session and can't check right now | 15:23 |
| *** lpetrut has quit IRC | 15:23 | |
| clarkb | fungi: ^ ? I'm not quite to a computer yet either. | 15:23 |
| fungi | i saw reports of it, but was bogged down on other things, will see if i can identify the new address | 15:29 |
| *** ykarel has quit IRC | 15:40 | |
| lourot | o/ just checking if I can get another +2 on this simple project-config change, thanks! https://review.opendev.org/#/c/758429/ | 15:43 |
| openstackgerrit | Masayuki Igawa proposed opendev/irc-meetings master: Update QA office hour https://review.opendev.org/759890 | 15:45 |
| *** sboyron has joined #opendev | 15:46 | |
| *** tosky_ has joined #opendev | 16:07 | |
| sshnaidm|rover | hi, folks, if someone is not in the middle of ptg meeting, can you please take a look why zuul can't "freeze" the graph in this patch: https://review.opendev.org/#/c/759892 ? I'd appreciate any ideas there | 16:07 |
| *** tosky has quit IRC | 16:08 | |
| *** ysandeep|ruck is now known as ysandeep|away | 16:08 | |
| *** slaweq|ptg is now known as slaweq | 16:08 | |
| fungi | that usually implies a configuration problem/conflict, but i'll see if i can spot it | 16:09 |
| *** fressi has quit IRC | 16:09 | |
| fungi | this'll take a bit. zuum is dragging my workstation to its knees | 16:11 |
| fungi | s/zuum/zoom/ | 16:11 |
| sshnaidm|rover | fungi, yeah, it's frustrating not to have detailed info, usually we try to play around by "try and see" effort | 16:11 |
| sshnaidm|rover | fungi, yeah, mine too :) | 16:11 |
| fungi | sshnaidm|rover: it says "Job tripleo-ci-centos-8-undercloud-containers depends on tripleo-ci-centos-8-content-provider which was not run." so i guess the question is why did it decide tripleo-ci-centos-8-undercloud-containers needed to run but not tripleo-ci-centos-8-content-provider | 16:17 |
| clarkb | https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml#L369 is why | 16:17 |
| fungi | aha, yep | 16:18 |
| clarkb | infra-root I'm going to try and refine https://etherpad.opendev.org/p/lAv5xqj0oNUjZKeV4vxg a bit more today so that that can go out | 16:18 |
| clarkb | having a slow start today which is probably good since it will be a late night :) | 16:19 |
| fungi | sshnaidm|rover: so the short story there is that having giant tangles of file exclusions in interdependent jobs quickly becomes impossible to reason about | 16:20 |
| sshnaidm|rover | fungi, I see, thanks, will try to fix it | 16:20 |
| fungi | it's one of the reasons we try to avoid irrelevant-files in our jobs | 16:20 |
| openstackgerrit | Merged opendev/irc-meetings master: Update QA office hour https://review.opendev.org/759890 | 16:24 |
| openstackgerrit | Merged openstack/project-config master: Mirror charm-neutron-api-plugin-ironic to GitHub https://review.opendev.org/758429 | 16:25 |
| clarkb | ok did some editing of https://etherpad.opendev.org/p/lAv5xqj0oNUjZKeV4vxg to capture thoughts during discussions yesterday | 16:36 |
| fungi | clarkb: you mention the summary table going away, but probably equally important to some users is the additional ci comment toggle | 16:39 |
| fungi | that goes away too, right? | 16:39 |
| clarkb | oh ya I'll add that | 16:40 |
| clarkb | it sort of goes away | 16:40 |
| clarkb | you can filter by vote category | 16:40 |
| clarkb | which gives a similar but not quite the same experience | 16:40 |
| fungi | oh, that's a good point, also you can filter by comment category right? | 16:41 |
| clarkb | not sure about comment category. Pretty sure I did vote and confirmed that worked at least | 16:41 |
| fungi | the comments can be flagged as added by automation, which zuul started doing sometime early in 3.x | 16:42 |
| fungi | but jenkins or older zuul-based 3pci comments likely won't do that | 16:42 |
| clarkb | oh ya there are bot comments but not sure if gerrit does anything with them yet | 16:42 |
| clarkb | I'm sure a plugin we write could if gerrit itself currently ignores them though | 16:43 |
| fungi | the "autogenerated" message tag | 16:43 |
| fungi | ahh, okay, so no current pg feature to filter on autogenerated | 16:43 |
| *** tosky_ is now known as tosky | 16:56 | |
| *** odyssey4me is now known as odyssey4me|PTO | 17:04 | |
| *** marios is now known as marios|out | 17:05 | |
| *** ricolin has quit IRC | 17:10 | |
| *** marios|out has quit IRC | 17:14 | |
| *** ralonsoh has quit IRC | 17:25 | |
| *** olaph has joined #opendev | 17:30 | |
| *** rpittau is now known as rpittau|afk | 17:34 | |
| clarkb | corvus: frickler ianw if you get a chance to look at https://etherpad.opendev.org/p/lAv5xqj0oNUjZKeV4vxg today I would love feedback. I'll try to send that out once ianw's day has started and has had a chance to look at it | 17:43 |
| *** eolivare has quit IRC | 17:49 | |
| fungi | clarkb: on the bit where it mentions the git v2 requirement, would it help to point out that centos/rhel 7 have a default git version which won't work with it? | 17:56 |
| clarkb | ya, though I'm trying to double check where I saw taht in the release notes and am not finding it so maybe this isn't true? | 17:58 |
| *** hashar has quit IRC | 17:59 | |
| clarkb | https://www.gerritcodereview.com/2.16.html#git-clients-older-than-2x-are-not-supported-anymore there it is | 17:59 |
| fungi | the oldest debian and ubuntu releases which still have packages listed on their respective package sites are all 2.x at least, so seems this will mostly be a problem for centos/rhel 7 users | 18:00 |
| clarkb | ok added a little note. | 18:01 |
| *** hashar has joined #opendev | 18:02 | |
| fungi | would it help to mention trying things out on revire-test? maybe that should be a followup once we've got it upgraded again | 18:05 |
| fungi | er, review-test | 18:05 |
| clarkb | ya I didn't want to mention that until we have a review-test up and running again | 18:05 |
| fungi | makes sense | 18:08 |
| corvus | clarkb: made some small changes at the top (mostly to get the most actionable info in the 1st pgraph) | 18:08 |
| clarkb | thanks | 18:08 |
| *** mlavalle has quit IRC | 18:11 | |
| *** mlavalle has joined #opendev | 18:14 | |
| zbr | is this git v2 requirement specific to gerrit only or gitea will continue to work? | 18:20 |
| clarkb | gerrit only and really only for the commit message hook | 18:21 |
| fungi | gerrit only | 18:21 |
| zbr | a requirement on git v2 would render maintenance branches useless for ancient clients | 18:21 |
| zbr | in that case, go for it! | 18:21 |
| *** andrewbonney has quit IRC | 18:21 | |
| fungi | yeah, you could roll your own commit hook or just manually generate commit ids | 18:21 |
| zbr | i doubt any developer is using a centos-7 box | 18:21 |
| fungi | or even reuse the old commit hook probably? | 18:21 |
| clarkb | yes the release notes suggest using the old hook as a workaround | 18:22 |
| zbr | if someone is affected, let them dig a way out of that pit ;) | 18:22 |
| fungi | right, if the question is "do we make fedora 33 users adjust their ssh configs or make centos 7 users install a third-party git package?" i think the latter wins | 18:22 |
| zbr | indeed | 18:23 |
| zbr | modern platforms should always take priority | 18:23 |
| *** Green_Bird has joined #opendev | 18:23 | |
| fungi | clarkb: draft message lgtm | 18:29 |
| *** hashar has quit IRC | 18:29 | |
| frickler | clarkb: fungi: how about the two workarounds mentioned in https://www.gerritcodereview.com/2.16.html#git-clients-older-than-2x-are-not-supported-anymore ? IIUC the requirement for new git is only via the commit-hook script, we could either keep serving the old script for some time, or at least test whether using the legacy script (option 2) does work for centos 7 | 18:50 |
| frickler | also I wonder whether the mail should somewhere mention the year 2020, just to be sure | 18:51 |
| clarkb | frickler: ++ to adding the year and maybe mention it is for the commit hook and if people complain we can host a version somewhere for them? | 18:53 |
| clarkb | I dont want to host it if no one needs it | 18:53 |
| clarkb | (maybe it can go inti git review packaging) | 18:54 |
| fungi | i've been tempted to integrate something like that into git-review anyway, since without ssh access people can't fetch the one gerrit serves, and we do support folks with only https client access after all | 18:55 |
| fungi | git review could ni theory embed both variants and install the correct one for the user's git version | 18:58 |
| clarkb | frickler: how about that? then if people ask about how they can use the commit hook on centos 7 we can figure out embedding it in git review or something | 18:59 |
| *** Green_Bird has quit IRC | 19:01 | |
| fungi | on the other hand, we'll likely want to drop python 2.7 support in git-review sometime soon as well, so folks on centos/rhel 7 would need to install the nonstandard python 3.6 which got added there when we do | 19:01 |
| clarkb | fungi: its standard now aiui | 19:01 |
| clarkb | it just doesn't come with a full set of libs (not really a problem for git review being pip installed) | 19:02 |
| fungi | "nonstandard" as in they need to expressly install it rather than it coming preinstalled like 2.7 does | 19:02 |
| frickler | clarkb: that sounds o.k. to me, waiting for feedback on how many people would actually be affected | 19:11 |
| * frickler heads away now and will try to be back at 5, might be half an hour late or so | 19:13 | |
| clarkb | no worries see you then | 19:14 |
| fungi | still not sure if i'll be awake | 19:17 |
| clarkb | the intent with the variety of times was that it would enable more people to interact and not to force people to stay up at odd hours :P | 19:22 |
| clarkb | its ok if you would rather sleep | 19:22 |
| fungi | huh, the magic "gerrit code review" account can't set group membership, but it can suexec as, for example, the openstack-project-creator account which can set group membership | 19:25 |
| fungi | i guess that's a reasonable workaround | 19:25 |
| fungi | i feel like i've already discovered this fact more than once | 19:27 |
| fungi | also gerrit doesn't allow / in usernames. it's (case-insensitive) alphanumeric plus ._- | 19:29 |
| *** webmariner has joined #opendev | 19:30 | |
| fungi | so i'm going with fungi.admin as my admin username | 19:30 |
| clarkb | that seems reasonable | 19:30 |
| fungi | anyway, turns out it can all be done in a single command on review.o.o: | 19:31 |
| fungi | sudo -u gerrit2 ssh -i ~gerrit2/review_site/etc/ssh_host_rsa_key -p 29418 -l 'Gerrit Code Review' localhost "suexec --as openstack-project-creator -- gerrit create-account --group 'Administrators' --ssh-key 'ssh-rsa AAAA...THBj fungi@bridge' fungi.admin" | 19:32 |
| fungi | i'm still trying to decide how i feel about hopping through bridge.o.o to access the account | 19:35 |
| fungi | vs directly from my workstation | 19:36 |
| fungi | ultimately i'll wind up securing the ssh key i use to log into bridge the same as i do the ssh key used to log into that gerrit account (and they could also just be the same key, i don't think that'e necessarily any less secure?) | 19:36 |
| clarkb | ya not sure about that one either | 19:38 |
| clarkb | I think it was corvus who mentioned it as an option? | 19:38 |
| fungi | either account could be used to gain control of the other anyway | 19:39 |
| corvus | yeah, just brainstorming. | 19:40 |
| fungi | no worries, just trying to think through whether it adds tangible security or merely complexity | 19:53 |
| *** melwitt has joined #opendev | 20:07 | |
| *** slaweq has quit IRC | 20:16 | |
| *** rpittau|afk is now known as rpittau | 20:18 | |
| *** slaweq has joined #opendev | 20:19 | |
| ianw | clarkb: thanks for writing; i put one suggestion there on the intro paragraph, feel free to take or leave, but otherwise LGTM | 20:38 |
| clarkb | ianw: that looks good I'll swap the content out for your suggestion | 20:57 |
| clarkb | and I'll send this email out shortly | 20:57 |
| *** gouthamr has quit IRC | 20:58 | |
| *** logan- has quit IRC | 20:58 | |
| *** gouthamr has joined #opendev | 20:58 | |
| *** gouthamr has quit IRC | 20:59 | |
| *** qchris has quit IRC | 20:59 | |
| *** gouthamr has joined #opendev | 20:59 | |
| *** logan- has joined #opendev | 21:01 | |
| *** rpittau is now known as rpittau|afk | 21:03 | |
| *** slaweq has quit IRC | 21:04 | |
| clarkb | alright, the email is queued up in my mua. I'll give it a few more minutes for last second edits otherwise sending it out | 21:06 |
| clarkb | I had on the etherpad that should go to service-discuss but I'm switching that to announce | 21:16 |
| clarkb | and sending now | 21:16 |
| fungi | ahh, yeah i wondered | 21:17 |
| fungi | we can still do followup minor additions to -discuss, like when review-test is up and available | 21:18 |
| clarkb | ++ | 21:18 |
| *** hamalq has joined #opendev | 21:20 | |
| *** fressi has joined #opendev | 21:21 | |
| *** slaweq has joined #opendev | 21:22 | |
| melwitt | clarkb: have you gotten a chance to try out the streaming log processing? I notice e-r indexing is behind 10 hrs at present | 21:27 |
| clarkb | melwitt: sorry no there are very many distractions right now. That may be something I can look at tomorrow since my main ptg commitments will be done by then | 21:28 |
| ianw | fungi: if you'd like to check anything on mirror-update.opendev.org, i think we're ready to cut-over reprepro to there. i've been running the deb-docker mirror overnight, exporting logs @ https://static.opendev.org/mirror/logs/reprepro/ and it's working | 21:28 |
| melwitt | clarkb: no worries, was just curious | 21:28 |
| TheJulia | are pep8 jobs timing out expected these days? | 21:45 |
| *** Goneri has quit IRC | 21:48 | |
| clarkb | TheJulia: I looked at that a few days ago and it appeared that our pip installs were really slow, but tox/pip don't record timestamps for installation times. More recently fungi pointed out that we've got reduced capacity due to taking a clodu out of rotation after ip address conflicts and arp fights. That may lead to more noisy neighbor problems? | 21:49 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 21:50 | |
| clarkb | looking at https://420bbb86771d88900a0a-0bd2acfdbec8f6316ef91de37dd7cea6.ssl.cf5.rackcdn.com/701410/1/gate/openstack-tox-pep8/9ef06ca/job-output.txt that appears to have had a slow apt-get installation | 21:51 |
| TheJulia | No idea, just seen it a couple times in the last few days on the pep8 jobs and that is understandably not great all things considered. :( | 21:51 |
| TheJulia | Fun :( | 21:51 |
| clarkb | then the job times out as it is installing pip things | 21:51 |
| fungi | ianw: oh! thanks, i meant to check it out earlier today but have been sidetracked. i should have some time between the oilabs and opendev ptg sessions though | 21:51 |
| clarkb | the mirror server there seems happy | 21:51 |
| clarkb | https://mirror.ord.rax.opendev.org/ubuntu/dists/focal/main/ is navigable, server load is low and there is plenty of free memory | 21:52 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68289&rra_id=all is curious though | 21:53 |
| clarkb | could we be maximizing network bw? | 21:53 |
| openstackgerrit | wes hayutin proposed zuul/zuul-jobs master: Allow rdo repos to be turned off for openvswitch install https://review.opendev.org/759107 | 21:54 |
| TheJulia | that is oddly consistent if jobs are evenly distributed | 21:59 |
| clarkb | apache logs look the way I'd expect. A lot of rackspace ips pulling pacakges that things like openstack need | 21:59 |
| clarkb | one oddity I notice (likely unrelated to the issue) is it seems some centos 8 pulls are via http and others are via https | 22:01 |
| clarkb | fungi: did we add latex back as a dep ? | 22:01 |
| clarkb | (seems like that may cause similar issues and figure we should rule it out quickly) | 22:01 |
| *** fressi has quit IRC | 22:05 | |
| *** sboyron has quit IRC | 22:06 | |
| clarkb | our cache miss to hit rate for pypi looks decent | 22:13 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: reprepo: enable cron jobs https://review.opendev.org/759965 | 22:15 |
| clarkb | ianw: fungi I notice we've got some html.tmp files in our wheel mirror that apache seems to complain about. I don't think that is related to the slowness in this region but it does fill our apache error logs and is distracting. https://mirror.ord.rax.opendev.org/wheel/ubuntu-18.04-x86_64/asyncio/ | 22:17 |
| clarkb | is that a side effect of when we were trying to write our own html files? we may want to clean those up? | 22:18 |
| ianw | hrm yes i think so. i can clear them out | 22:18 |
| *** Green_Bird has joined #opendev | 22:19 | |
| clarkb | rxtx factor that flavor is 1600.0 and I believe we're supposed to have a theoretical max for any one connection of ~1/2 that value | 22:23 |
| clarkb | we seem to be well below that currently | 22:23 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68280&rra_id=all seems to show the backup | 22:24 |
| clarkb | that looks a lot to me like we're getting throttled and so the requests are piling up | 22:25 |
| clarkb | I guess the next thing to check is if apache has free slots | 22:25 |
| fungi | 42gb used in /var/cache/openafs so i don't think we're overrunning our afs cache limit | 22:25 |
| clarkb | apache shows plenty of open slots | 22:28 |
| TheJulia | could it just be variable performance on rax with our timers being too low to account for it? | 22:28 |
| TheJulia | I ask because we've actually had to have our devstack plugin auto-extend some in-job timeouts on rax because of performance differences and variability | 22:29 |
| clarkb | it could be, though it seems network related as the other resources we're measuring don't seem to be having trouble | 22:29 |
| clarkb | possible we're just overwhelming a switch or router | 22:30 |
| clarkb | fetching https://mirror.ord.rax.opendev.org/fedora/atomic/stable/Fedora-29-updates-20190820.0/AtomicHost/x86_64/images/Fedora-AtomicHost-29-20190820.0.x86_64.qcow2 (a large file) is indeed very slow to my desktop | 22:31 |
| ianw | clarkb: ok, tmp files gone at least | 22:31 |
| clarkb | like KBps measurement slow | 22:31 |
| clarkb | but the server itself has plenty of memory to service connections as well as cpu which is making me suspect a network throttle somewhere | 22:32 |
| ianw | agree, same here | 22:32 |
| clarkb | pulling from dfw is much quicker | 22:33 |
| fungi | could just be there's a party going on in chicago | 22:33 |
| fungi | our ord max-servers is higher right? | 22:34 |
| clarkb | its about the same as iad | 22:34 |
| clarkb | then dfw is a bit lower | 22:34 |
| ianw | locally on the mirror itself : 2020-10-27 22:34:09 (39.7 MB/s) - ‘Fedora-AtomicHost-29-20190820.0.x86_64.qcow2’ saved [717020672/717020672] | 22:34 |
| fungi | ord is a good 33% higher than dfw/iad | 22:35 |
| clarkb | fungi: not for actual utilziation though | 22:35 |
| clarkb | I think the quotas may haev been tweaked down? | 22:35 |
| fungi | oh interesting | 22:35 |
| fungi | yeah i was merely looking at our config | 22:35 |
| clarkb | ianw: I think that observation points to a throttle off host | 22:36 |
| ianw | you wouldn't think it would throttle to internal rax ord hosts though? | 22:37 |
| clarkb | we use the public address not the internal one though, but also it could be an unintentional throttle | 22:37 |
| clarkb | some switch or router having a hard time for $reason | 22:37 |
| ianw | can they see the 10.29 address? | 22:37 |
| clarkb | I think so? It would depend on how glean configures it | 22:38 |
| clarkb | we could try it to see if it is faster via the 10 net | 22:38 |
| ianw | at about 150k to the backup server in rax ord | 22:39 |
| clarkb | might also want to try ipv6 vs ipv4, though I think the failed job example I had would've used ipv6 | 22:39 |
| openstackgerrit | melanie witt proposed opendev/elastic-recheck master: Add query for bug 1901739 https://review.opendev.org/759967 | 22:39 |
| openstack | bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [Undecided,New] https://launchpad.net/bugs/1901739 | 22:39 |
| clarkb | ianw: yay it is consistent :) | 22:40 |
| ianw | 12mb/s when using "wget http://10.209.128.57/" | 22:40 |
| ianw | from same mirror | 22:40 |
| ianw | s/mirror/backup server/ to mirror | 22:40 |
| ianw | same speed to public address with -4 & -6 | 22:41 |
| clarkb | not sure I got all that. The backup server to mirror is slow when and fast when? | 22:42 |
| clarkb | oh I think I get it. public v4 and v6 are both slow. private v4 is good | 22:43 |
| ianw | sorry; yeah stream of conciousness | 22:45 |
| ianw | from the backup server in the same region, it is fast to grab the iso over the private v4 network (10.209 address), but slow to get it from the public ip via both the server's public ipv4 and ipv6 address | 22:46 |
| ianw | we could probably setup /etc/hosts on test nodes to use the internal address for rax hosts? | 22:47 |
| ianw | as part of the mirror configuration | 22:47 |
| clarkb | or create a mirror-int record in dns and use that? | 22:49 |
| clarkb | that will likely be easier to understand 6 months from now | 22:49 |
| clarkb | but we have to check glean configures that interface and routes properly (I think it does) | 22:50 |
| ianw | clarkb: the thing with that is that i think we'll be back to https issues getting a cert for that? | 22:50 |
| clarkb | oh ya hrm | 22:51 |
| ianw | i jumped on a focal node currently doing something, and it can access the mirror on it's private address | 22:52 |
| ianw | and it's pulling that iso at 12mb/s | 22:52 |
| clarkb | we can add mirror-int to the altnames then reissue certs | 22:54 |
| ianw | umm, i guess we do dns validation so yes | 22:57 |
| clarkb | fungi: ^ thoughts? | 22:58 |
| fungi | seems like a fine experiment | 22:59 |
| ianw | it doesn't have a internal ipv6 address though? | 23:00 |
| ianw | i can add the records, just a tick | 23:00 |
| clarkb | ya just ipv4 for internal | 23:01 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: RAX ord mirror : add internal address https://review.opendev.org/759970 | 23:03 |
| clarkb | that looks fine to me but helpign with kids so not in a spot to leave avote | 23:04 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Generate internal certs for RAX ORD mirror https://review.opendev.org/759971 | 23:05 |
| *** slaweq has quit IRC | 23:10 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Use internal address for RAX ORD https://review.opendev.org/759972 | 23:14 |
| ianw | i think that's the chain then. i'm pretty confident ^ works because it was what we used to use to swtich between openstack.org/opendev.org servers when that was a thing | 23:15 |
| johnsom | It seems the StoryBoard/Gerrit bot is still not functional. | 23:19 |
| *** mlavalle has quit IRC | 23:20 | |
| fungi | oh, yep, we never did restart gerrit, thanks for the reminder | 23:25 |
| fungi | maybe before the 05:00 opendev ptg session would be a good opportunity, if i manage to be awake for it | 23:26 |
| ianw | fungi: i think i missed that? happy to manage if there's a tl;dr | 23:30 |
| ianw | i'll get the dns entries and certs for ord internal mirror, then we can test from a node | 23:31 |
| fungi | ianw: oh, so as part of the breach cleanup i blew away all the session keys and api tokens for production sb after deescalating an account and disabling it, but that all happened after gerrit was brought back up so the new api token i generated for the its-storyboard plugin to use won't be read from configuration until the next gerrit restart | 23:34 |
| fungi | which i meant to take a moment for over the weekend | 23:35 |
| ianw | ahh. yeah a few more hours is probably a good time for minimal impact | 23:36 |
| *** Goneri has joined #opendev | 23:38 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: RAX ord mirror : add internal address https://review.opendev.org/759970 | 23:46 |
| *** tosky has quit IRC | 23:52 | |
| *** Green_Bird has quit IRC | 23:58 | |
| *** DSpider has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
