| clarkb | tonyb: was it a rax node and added the 10 net address? | 00:06 |
|---|---|---|
| clarkb | I don't think we haev any floating IP clouds anymore but that can happe nthere too potentially | 00:06 |
| tonyb | clarkb: Correct. | 00:06 |
| clarkb | we probably need to work backwards from the zuul inventory to zuul to nodepool to openstack sdk to the cloud to figure that one out | 00:06 |
| clarkb | tonyb: that said I would expect the meetpad server to also be using the 10 net address in the configs if it was applied consistently then it should work because we'd connect from 10 net to 10 net | 00:08 |
| tonyb | I don't really want to change the rules template but, can we use `ansible_host` instead of `public_v4` ? | 00:08 |
| clarkb | potentially. Looking at https://zuul.opendev.org/t/openstack/build/d53662680ded4cb88102ea1f89e97adb/log/zuul-info/inventory.yaml the public address is the public non 10 net address for all nodes | 00:08 |
| clarkb | is our translation from zuul inventory to system-config bridge inventory at fault maybe? | 00:09 |
| tonyb | where does that happen? | 00:10 |
| clarkb | tonyb: playbooks/zuul/run-base.yaml runs write-inventory | 00:11 |
| clarkb | and that role lives in zuul-jobs | 00:12 |
| tonyb | https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/run-base.yaml#L82 | 00:13 |
| clarkb | ok that explains why we use that value. It does not explain why network connections weren't made over that network. Or maybe they are and it isn't routable? | 00:14 |
| clarkb | oh no its because we configure the xmpp target to be the public address | 00:14 |
| tonyb | Yeah | 00:14 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/905182/3/inventory/service/group_vars/jvb.yaml there | 00:15 |
| tonyb | I can update that to a better value | 00:15 |
| clarkb | ok cool we understand now. I think the fix here is to set meetpad_jvb_xmpp_server in the test group vars file | 00:15 |
| clarkb | that way we can address my comment there and also select public_v4 rather than ansible_host | 00:15 |
| clarkb | the prod value will continue as is unchagned pointed at meetpad01 (until we have a meetpad02) and the test side can talk to meetpad99 on the correct addr | 00:16 |
| tonyb | okay | 00:16 |
| clarkb | this is good I feel like I'm learning things. | 00:17 |
| tonyb | Next question .... I created an autohold with the webUI and set the trigger count to 5 ... because I figured I'd need it a more than once. So how do I release the autohold without deleteing the entry? | 00:18 |
| tonyb | or is that not what the trigger count is for? | 00:18 |
| clarkb | you know thats a really good question. I think the trigger count might be more applicable to casting a wide net to gather more datapoints for a single failure rather than iterative one after another captures | 00:19 |
| clarkb | you can delete the does in nodepool directly and leave the hold in place | 00:19 |
| clarkb | but I don't think you can do the cleanup you want via zuul directly | 00:19 |
| clarkb | corvus: ^ fyi on zuul functionality | 00:19 |
| clarkb | * you can delete the nodes | 00:20 |
| tonyb | clarkb: Thanks. I'll work on updateing the chnage and making a coffee to give corvus time to reply. | 00:20 |
| clarkb | I could go for a warm beverage. | 00:21 |
| clarkb | we had to take the long way around to get home from school today beacuse the kids don't have boots on and everything is swampy | 00:22 |
| tonyb | eeek | 00:22 |
| tonyb | clarkb: so the crux of the update would be: https://paste.opendev.org/show/bhzB7mSEjOMvZdHtm4Ui/ | 00:26 |
| clarkb | yes I think that should do it | 00:27 |
| corvus | yeah i think there's a gap there. :) i agree you should be able to do that thru the ui but can't. so what clarkb said. | 00:30 |
| corvus | also, as long as opendev isn't at capacity, we're not going to notice a few held nodes sitting around a bit too long | 00:31 |
| corvus | so for times like right now, probably okay to just ignore it | 00:31 |
| tonyb | okay | 00:39 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Switch meetpad servers to jammy https://review.opendev.org/c/opendev/system-config/+/905182 | 00:52 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: force jitsi_meet node failure for testing https://review.opendev.org/c/opendev/system-config/+/905183 | 00:52 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Switch meetpad servers to jammy https://review.opendev.org/c/opendev/system-config/+/905182 | 02:08 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: force jitsi_meet node failure for testing https://review.opendev.org/c/opendev/system-config/+/905183 | 02:08 |
| tonyb | Is there a better way to achieve: https://review.opendev.org/c/opendev/system-config/+/905182/5/playbooks/zuul/templates/group_vars/jvb.yaml.j2 | 02:09 |
| Clark[m] | tonyb wrap the section in raw tags | 02:09 |
| Clark[m] | Section == line in this case | 02:10 |
| tonyb | Gah! https://jinja.palletsprojects.com/en/3.0.x/templates/#escaping | 02:11 |
| tonyb | I searched and search for that | 02:11 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Switch meetpad servers to jammy https://review.opendev.org/c/opendev/system-config/+/905182 | 02:18 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: force jitsi_meet node failure for testing https://review.opendev.org/c/opendev/system-config/+/905183 | 02:18 |
| tonyb | That's much neater | 02:19 |
| tonyb | Okay, the servers are up and held. Things seem to work, but there are worrying outputs in the container logs. | 05:29 |
| tonyb | I'm going to finish up for today | 05:29 |
| tonyb | clarkb: are you able to look at the held node with me tomorrow to work out if the things I'm seeing are problems? | 05:31 |
| Clark[m] | tonyb: I should be able to | 06:05 |
| tonyb | Thanks | 06:05 |
| *** mmalchuk_ is now known as mmalchuk | 06:13 | |
| *** noonedeadpunk_ is now known as noonedeadpunk | 08:14 | |
| frickler | guilhermesp: mnaser: I'm having no connectivity to (review.)opendev.org via IPv6 once again. global reachability seems to be only via cogent, which is ... a questionable design choice. anything you could do about this? (both short and long term) cf. https://bgp.tools/prefix/2604:e100:1::/48#connectivity | 10:33 |
| SvenKieske | me too | 10:34 |
| SvenKieske | at first I was wondering if review.opendev.org was down oO | 10:35 |
| frickler | SvenKieske: you are also behind AS3320 iirc? | 10:36 |
| SvenKieske | yes | 10:39 |
| SvenKieske | at least it works once the IPv4 Version is cached. | 10:46 |
| SvenKieske | mhm, the WebUI is rather unresponsive and slow.. :/ | 10:50 |
| ihalomi | hey, not sure if this is right channel but i have problem with my account at opendev. I changed prefered email in my ubuntu one account and now I cannot login into opendev account. It sents me to this page https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator who should I contact about it? I think its because there is already account in opendev using my new prefered email but ubuntu one accout used to create it no | 10:50 |
| SvenKieske | "Error submitting review TypeError: NetworkError when attempting to fetch resource." | 10:56 |
| frickler | ihalomi: seems your message was cut off at the end after "... but ubuntu one accout used to create it no". I can't check right now, please wait for some other admin to show up | 11:51 |
| ihalomi | it was endding no longer exists, okay i will wait thanks for reply | 12:04 |
| fungi | ihalomi: changing your ubuntuone e-mail address creates a new ubuntuone id, so you end up trying to log into gerrit as a new id which creates a new gerrit account. if the address for the new account was also an address associated with your old gerrit account, gerrit will refuse to create the new account because multiple accounts aren't allowed to have any of the same e-mail addresses | 12:35 |
| ihalomi | fungi: okay but what can i do about it? | 12:36 |
| fungi | you'll need one of our sysadmins to deactivate your old account and clear the addresses out of it, which is a complicated and time-consuming process. i'm just waking up and don't have time to work on it right now, but may be able to later today | 12:37 |
| ihalomi | fungi: okay so there is nothing I can do otherwise than remind myself to sysadmins, please try to do it I would like to push some changes but i cant do it under old client address | 12:39 |
| fungi | ihalomi: if you can /msg me the e-mail address you're trying to use | 12:47 |
| fungi | that should help me find the old and new accounts | 12:47 |
| SvenKieske | mhm, seems there is some general trouble with the largest german ISP :/ routing problems after a botched network upgrade | 13:28 |
| opendevreview | Elod Illes proposed openstack/project-config master: Adapt make_branch script to new 'unmaintained/<series>' branch https://review.opendev.org/c/openstack/project-config/+/904837 | 14:49 |
| dpanech | Hi, this review is stuck. It has all the votes, but is not merging: https://review.opendev.org/c/starlingx/tools/+/904421 . Could someone help me investigate why? | 15:00 |
| fungi | dpanech: gerrit says the parent of that change got revised and merged but the change was never rebased onto that new state | 15:15 |
| fungi | dpanech: the parent commit of 904421 is patchset #1 of 904420, but patchset #2 of 904420 is what ended up merging | 15:16 |
| fungi | dpanech: if you rebase 904421 onto the current state of the master branch it should work | 15:16 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container https://review.opendev.org/c/opendev/system-config/+/905469 | 15:33 |
| fungi | Clark[m]: what's the order of operations for cleaning external-ids out of an account and retiring it? i do this so infrequently i can't recall which comes first, and the comments in those scripts don't say | 15:50 |
| fungi | also, what's the easiest way to look up the id if all i know is an e-mail address? put together a rest api query or is there an easier route? | 15:51 |
| Clark[m] | fungi: I believe it is removal of the external ids first. Then removal of preferred email and disabling the account happen together in the second step | 15:52 |
| fungi | thanks | 15:53 |
| Clark[m] | We should communicate that the user will end up with a new account | 15:53 |
| Clark[m] | Not sure if we did that. Currently on a school run | 15:53 |
| fungi | yeah, i did (see scrollback) | 15:53 |
| dpanech | fungi: ok thanks | 15:59 |
| fungi | infra-root: update on the matrix homeserver hosting situation... i got budget approval for openinfra foundation to cover the account upgrade cost and reached out to ems yesterday asking them to proceed. they responded today stating we're in the upgrade queue now and they'll get back to us with more details at the end of the month | 16:00 |
| clarkb | fungi: ack thanks for the update | 16:11 |
| fungi | i'm headed out to meet some friends for lunch on an adjacent island and run a quick errand, but should be able to get to ihalomi's old account retirement when i get back around 18:30 utc | 16:14 |
| clarkb | fungi: enjoy! | 16:14 |
| fungi | thanks! | 16:14 |
| clarkb | service coordinator election plan email sent making it official | 17:06 |
| fungi | thanks! | 18:34 |
| fungi | clarkb: what's the easiest way to look up the id if all i know is an e-mail address? put together a rest api query or is there an easier route? | 19:08 |
| clarkb | fungi: one sec | 19:09 |
| clarkb | fungi: In the web ui do a owner:emailaddr query. On the resulting page will be a view dashboard link. That link has the id in it | 19:10 |
| clarkb | thats the hack way I do it | 19:10 |
| fungi | oh, good trick | 19:11 |
| clarkb | fungi: I think the error log will also record the conflicting id for the email address on login attempts | 19:13 |
| clarkb | something liek couldn't create id foo because email addr xyz@com conflicts with id bar | 19:13 |
| clarkb | it might be a good idea to double check hyou see that error message before making the account changes | 19:13 |
| clarkb | just to confirm this is the issue | 19:13 |
| fungi | good call. it took about three clicks from the query to get to a version of the owner dashboard that embedded the id but i did find it. grepping the error log for that id number also turns up the "cannot create external id" error | 19:25 |
| fungi | would adding that new external id to the existing account be a cleaner solution than killing the old account? | 19:25 |
| fungi | that was what we mostly did in the pre-notedb days, but not sure if there's an equivalent solution nopw | 19:26 |
| clarkb | yes, but the only way to do that is to push the update under gerrit which can only safely be done while offline | 19:27 |
| clarkb | becuse gerrit will validate the entire db then complain about those unrelated errors we still haven't fixed and prevent this update from happening | 19:27 |
| clarkb | it is possible we might eb able to get away with an out of band push while gerrit is running and then do a reindex but I don't want to risk it given its the account database | 19:27 |
| clarkb | if we fix all the errors then we could do this properly. But I have a hard tiem prioritizing it because its pretty annoying drudge work that really only helps a few people | 19:28 |
| clarkb | we could also potentially declare bankruptcy and break all the remaining accounts (I think there are like 30 of them) on the assumption that most (all?) are not active users and if/when they do become active we can sort them out later | 19:29 |
| fungi | oh, got it. i keep forgetting we can't push updates for the all-users repo into running gerrit currently | 19:30 |
| mbeierl | Hello. I'm following this https://docs.opendev.org/opendev/infra-manual/latest/gettingstarted.html and when it comes to the `git review -s`, it fails due to 404 on "GET https://opendev.org/tools/hooks/commit-msg" | 19:39 |
| clarkb | mbeierl: it should be fetching that commit-msg hook from gerrit not gitea (gerrit is at https://review.opendev.org and gitea runs https://opendev.org) | 19:41 |
| clarkb | mbeierl: did you manually set your git remote for 'gerrit' ? | 19:41 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container https://review.opendev.org/c/opendev/system-config/+/905469 | 19:42 |
| mbeierl | No. I did the clone like so: "git clone https://opendev.org/<namespace>/<projectname" Should that be from https://review.opendev.org instead? | 19:42 |
| clarkb | mbeierl: no, you should clone from opendev.org to keep non code review load off of the code review server. | 19:43 |
| clarkb | mbeierl: so you cloned then ran `git review -s` without changing any git remotes? Can you share the project so that we can check the project config? | 19:44 |
| fungi | so it's definitely odd that git review -s isn't finding the .gitreview file in the repo | 19:44 |
| mbeierl | So I have a `origin` fetch and push like so: https://opendev.org/openstack/sunbeam-charms.git | 19:44 |
| clarkb | fungi: and that it is using https instead of ssh | 19:44 |
| fungi | oh, yes that's also strange. maybe a global git environment setting? | 19:44 |
| clarkb | https://opendev.org/openstack/sunbeam-charms/src/branch/main/.gitreview that config looks fine to me | 19:44 |
| clarkb | it should be using ssh too | 19:44 |
| clarkb | maybe run git review with the verbose flag so that we can see the decisions it is making and paste that? | 19:45 |
| fungi | to https://paste.opendev.org/ (not directly in the channel, it will be a lot of text) | 19:46 |
| clarkb | thats the -v or --verbose flag | 19:46 |
| clarkb | mbeierl: ya the origin remote should point there. Then when you run git review -s it should create a new 'gerrit' remote by default using ssh not https that points to gerrit for pushing of code changes. It also uses this ssh remote to scp the commit msg hook by default. I think what we're confused about is that it is choosing the wrong protocol and the wrong host | 19:48 |
| mbeierl | aha. I found I had an old .gitconfig [gitreview] entry in my home directory, with remote=origin in it. That certainly can mess things up | 19:49 |
| fungi | sounds like you'd configured something to try to avoid using ssh with a gerrit (somewhere else maybe?) | 19:50 |
| clarkb | maybe we should go ahead and vendor the hook in git-review | 19:52 |
| clarkb | add a flag to fetch it from gerrit using the existing code path that is disabled by default. Though that wouldn't help here because the next step would be pushing and gitea would reject that | 19:53 |
| fungi | #status log Retired Gerrit account 35105 at the request of ihalomi who is has changed their UbuntuOne login E-mail resulting in a new OpenID | 20:06 |
| opendevstatus | fungi: finished logging | 20:06 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container https://review.opendev.org/c/opendev/system-config/+/905469 | 20:25 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container https://review.opendev.org/c/opendev/system-config/+/905469 | 21:10 |
| tonyb | clarkb: Are you around to look at meetpad99? | 21:11 |
| tonyb | mbeierl: I *think* you're using the packaged version of git-review and you need the version from pypi | 21:17 |
| clarkb | tonyb: yup! | 21:21 |
| clarkb | sorry just finished up lunch | 21:21 |
| tonyb | np | 21:21 |
| clarkb | do you want ot use meetpad prod and voice chat or just use irc? | 21:21 |
| tonyb | Let's stick with IRC | 21:22 |
| clarkb | ok I'm looking up held nodes now | 21:23 |
| tonyb | that way we can use the test meetpad | 21:23 |
| tonyb | # Temp hosts for OpenDev testing | 21:23 |
| tonyb | 23.253.166.123 bridge99.opendev.org | 21:23 |
| tonyb | 23.253.166.151 jvb99.opendev.org | 21:23 |
| tonyb | 23.253.166.131 meetpad99.opendev.org | 21:23 |
| clarkb | thanks | 21:23 |
| clarkb | ok I'm sshing into meetpad and jvb | 21:23 |
| tonyb | Okay | 21:24 |
| clarkb | are you saying you want to try and do voice chat on them too? | 21:24 |
| tonyb | We will need some load | 21:24 |
| clarkb | ack let me update /etc/hosts | 21:24 |
| tonyb | for my testing I did just have a muted call running | 21:25 |
| clarkb | https://meetpad99.opendev.org/isitbroken I'm there and it seems to have loaded the etherpad successfully too | 21:26 |
| tonyb | Yup | 21:26 |
| tonyb | What I was seeing is a) meetpad isn't scheduling callin on jvb99, I don't know if we can force that | 21:27 |
| tonyb | ... I guess we could down the jvb containre on meetpad99 | 21:27 |
| clarkb | I don't know how to force it either. But usually I think it just round robins or maybe one after hte other | 21:27 |
| clarkb | oh ya that would be the way to try and force it | 21:27 |
| clarkb | I'll stop my call then we do that on meetpad99 and try again | 21:27 |
| clarkb | ok I stopped jvb on meetpad99 | 21:28 |
| tonyb | okay | 21:28 |
| clarkb | as soon as you joined the meeting jvb99's log exploded with content | 21:29 |
| clarkb | so I think it is working but I'll unmute to triple check | 21:29 |
| tonyb | jvb_1 | JVB 2024-01-12 21:29:31.551 SEVERE: [24] [confId=750e571cd2cb8973 conf_name=isitbroken@muc.localhost meeting_id=44f07352 epId=b2c414b3 stats_id=Odie-8XV] Endpoint.scheduleEndpointMessageTransportTimeout$lambda$30#702: EndpointMessageTransport still not connected. | 21:30 |
| clarkb | tonyb: ha this doesn't work beacuse chrome is verifying the https cert issuer | 21:44 |
| * clarkb tries firefox | 21:44 | |
| clarkb | or maybe it is trying to drop to http beacuse of some redirect in any case both ff and chrome don't like it when I edit /etc/hosts to make meetpad99 also meetpad | 21:45 |
| clarkb | "meetpad.opendev.org has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site." | 21:45 |
| tonyb | Yeah, and as there is an invalid cert it's refusing | 21:46 |
| clarkb | I guess this is about not downgrading to http | 21:46 |
| tonyb | Okay I'll rebuild and set the hostnames everywhere to meetpad99 | 21:46 |
| clarkb | I don't know why it would downgrade to http though | 21:46 |
| clarkb | ok | 21:46 |
| tonyb | That was super helpful | 21:47 |
| tonyb | Thanks | 21:47 |
| clarkb | I don't see my requests hitting it. Really wish browsers had a "I'm ok with shooting myself in the foot with certs and names" mode for testing | 21:49 |
| clarkb | intsead of "you cannot do this" | 21:49 |
| clarkb | anyway I think if we get the names right it should work | 21:49 |
| tonyb | Yeah me too | 21:49 |
| clarkb | for anyone at home wondering the web client was trying to connect to meetpad.opendev.org to set up a websocket to the jvb server. Problem is meetpad.opendev.org != meetpad99 which we used to set up the call so it failed | 21:52 |
| clarkb | need to get the test env configured to set the names properly | 21:52 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Switch meetpad test servers to jammy https://review.opendev.org/c/opendev/system-config/+/905182 | 22:00 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: force jitsi_meet node failure for testing https://review.opendev.org/c/opendev/system-config/+/905183 | 22:00 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Allow overriding of meetpad PUBLIC_URL https://review.opendev.org/c/opendev/system-config/+/905510 | 22:00 |
| tonyb | I don't know if 905510 is correct, it feels "hinky" because of mixing role and group vars | 22:02 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container https://review.opendev.org/c/opendev/system-config/+/905469 | 22:11 |
| fungi | clarkb: i think we saw that same websocket issue testing with held nodes in the past | 22:18 |
| clarkb | tonyb: I think that should work though but ya its a lot of plumbing in part because we have somewhat duplicated config between the jvb and meetpad proper | 22:22 |
| tonyb | thanks. I broke it somewhere. I might pick this back up on Monday | 22:23 |
| opendevreview | Merged opendev/irc-meetings master: Revitalize I18n meeting slot https://review.opendev.org/c/opendev/irc-meetings/+/905236 | 22:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!