| *** tosky has quit IRC | 00:02 | |
| *** ociuhandu has quit IRC | 00:03 | |
| *** iurygregory has quit IRC | 00:06 | |
| *** jamesmcarthur has quit IRC | 00:17 | |
| *** harlowja has quit IRC | 00:17 | |
| *** jamesmcarthur has joined #openstack-infra | 00:18 | |
| *** rcernin_ has joined #openstack-infra | 00:19 | |
| *** jamesmcarthur has quit IRC | 00:20 | |
| *** rcernin has quit IRC | 00:20 | |
| *** jamesmcarthur has joined #openstack-infra | 00:21 | |
| *** CrayZee has quit IRC | 00:35 | |
| *** snapiri has joined #openstack-infra | 00:35 | |
| *** Adri2000 has quit IRC | 00:43 | |
| *** iurygregory has joined #openstack-infra | 00:43 | |
| *** jamesmcarthur has quit IRC | 00:44 | |
| *** Adri2000 has joined #openstack-infra | 00:44 | |
| *** jamesmcarthur has joined #openstack-infra | 00:45 | |
| *** rlandy has quit IRC | 00:48 | |
| *** jamesmcarthur has quit IRC | 00:48 | |
| *** jamesmcarthur has joined #openstack-infra | 00:49 | |
| *** jamesden_ has joined #openstack-infra | 01:01 | |
| *** abhishekk has quit IRC | 01:02 | |
| *** stevebaker has quit IRC | 01:03 | |
| *** jamesdenton has quit IRC | 01:03 | |
| *** abhishekk has joined #openstack-infra | 01:04 | |
| *** hamalq has quit IRC | 01:26 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** jamesmcarthur has joined #openstack-infra | 01:35 | |
| *** jamesmcarthur has quit IRC | 01:40 | |
| *** armax has quit IRC | 01:47 | |
| *** dychen has joined #openstack-infra | 02:06 | |
| *** jamesmcarthur has joined #openstack-infra | 02:06 | |
| *** dychen has quit IRC | 02:11 | |
| *** dychen has joined #openstack-infra | 02:11 | |
| *** dchen has quit IRC | 02:12 | |
| *** jamesmcarthur has quit IRC | 02:14 | |
| *** jamesmcarthur has joined #openstack-infra | 02:15 | |
| *** stevebaker has joined #openstack-infra | 02:37 | |
| *** ajitha has joined #openstack-infra | 02:38 | |
| *** ricolin has joined #openstack-infra | 02:45 | |
| *** jamesmcarthur has quit IRC | 02:47 | |
| *** jamesmcarthur has joined #openstack-infra | 02:49 | |
| *** jamesmcarthur has quit IRC | 02:50 | |
| *** jamesmcarthur has joined #openstack-infra | 02:50 | |
| *** dklyle has quit IRC | 02:52 | |
| *** david-lyle has joined #openstack-infra | 02:52 | |
| *** david-lyle has quit IRC | 03:01 | |
| *** zzzeek has quit IRC | 03:04 | |
| *** zzzeek has joined #openstack-infra | 03:05 | |
| *** rcernin_ has quit IRC | 03:23 | |
| *** rcernin_ has joined #openstack-infra | 04:01 | |
| *** psachin has joined #openstack-infra | 04:11 | |
| *** ykarel|away has joined #openstack-infra | 05:03 | |
| *** rakhmerov has joined #openstack-infra | 05:17 | |
| *** iurygregory has quit IRC | 05:33 | |
| *** ricolin_ has joined #openstack-infra | 05:47 | |
| *** ricolin has quit IRC | 05:47 | |
| *** jamesmcarthur has quit IRC | 05:50 | |
| *** jamesmcarthur has joined #openstack-infra | 05:52 | |
| *** vishalmanchanda has joined #openstack-infra | 05:52 | |
| *** jamesmcarthur has quit IRC | 05:57 | |
| *** dingyichen has joined #openstack-infra | 05:58 | |
| *** gyee has quit IRC | 05:59 | |
| *** dychen has quit IRC | 06:00 | |
| *** jamesmcarthur has joined #openstack-infra | 06:00 | |
| *** zbr5 has joined #openstack-infra | 06:04 | |
| *** zbr has quit IRC | 06:06 | |
| *** zbr5 is now known as zbr | 06:06 | |
| *** dingyichen has quit IRC | 06:09 | |
| *** dychen has joined #openstack-infra | 06:09 | |
| *** matt_kosut has joined #openstack-infra | 06:13 | |
| *** ykarel_ has joined #openstack-infra | 06:17 | |
| *** ykarel|away has quit IRC | 06:19 | |
| *** ykarel_ is now known as ykarel | 06:26 | |
| *** jamesmcarthur has quit IRC | 06:37 | |
| *** nightmare_unreal has joined #openstack-infra | 06:45 | |
| *** slaweq has joined #openstack-infra | 07:04 | |
| *** rcernin_ has quit IRC | 07:28 | |
| *** slaweq has quit IRC | 07:30 | |
| *** openstackgerrit has joined #openstack-infra | 07:30 | |
| openstackgerrit | Rico Lin proposed openstack/project-config master: Add ubuntu-bionic-arm64-xlarge https://review.opendev.org/c/openstack/project-config/+/771565 | 07:30 |
|---|---|---|
| *** eolivare has joined #openstack-infra | 07:31 | |
| *** rcernin_ has joined #openstack-infra | 07:32 | |
| *** xek has joined #openstack-infra | 07:35 | |
| *** jcapitao has joined #openstack-infra | 07:51 | |
| *** dychen has quit IRC | 07:59 | |
| *** slaweq has joined #openstack-infra | 08:00 | |
| *** hashar has joined #openstack-infra | 08:03 | |
| *** dchen has joined #openstack-infra | 08:03 | |
| *** dchen has quit IRC | 08:05 | |
| *** amoralej|off is now known as amoralej | 08:06 | |
| *** andrewbonney has joined #openstack-infra | 08:09 | |
| *** sboyron_ has joined #openstack-infra | 08:12 | |
| *** rpittau|afk is now known as rpittau | 08:17 | |
| *** rcernin_ has quit IRC | 08:35 | |
| *** jamesmcarthur has joined #openstack-infra | 08:37 | |
| *** sboyron__ has joined #openstack-infra | 08:38 | |
| *** abhishekk has quit IRC | 08:40 | |
| *** sboyron_ has quit IRC | 08:41 | |
| *** psachin has quit IRC | 08:41 | |
| *** stevebaker has quit IRC | 08:41 | |
| *** abhishekk has joined #openstack-infra | 08:41 | |
| *** jamesmcarthur has quit IRC | 08:42 | |
| *** akahat|rover is now known as akahat|lunch | 08:46 | |
| *** tosky has joined #openstack-infra | 08:47 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** lucasagomes has joined #openstack-infra | 09:12 | |
| *** DinaBelova has quit IRC | 09:17 | |
| *** freerunner has quit IRC | 09:17 | |
| *** ramishra has quit IRC | 09:17 | |
| *** ramishra has joined #openstack-infra | 09:18 | |
| *** tristanC has quit IRC | 09:18 | |
| *** freerunner has joined #openstack-infra | 09:19 | |
| *** DinaBelova has joined #openstack-infra | 09:19 | |
| *** tristanC has joined #openstack-infra | 09:20 | |
| *** yamamoto has joined #openstack-infra | 09:21 | |
| *** yamamoto has quit IRC | 09:23 | |
| *** yamamoto has joined #openstack-infra | 09:30 | |
| *** yamamoto has quit IRC | 09:37 | |
| *** yamamoto has joined #openstack-infra | 09:45 | |
| *** ysandeep is now known as ysandeep|afk | 09:45 | |
| *** yamamoto has quit IRC | 09:45 | |
| *** yamamoto has joined #openstack-infra | 09:47 | |
| *** derekh has joined #openstack-infra | 09:49 | |
| *** yamamoto has quit IRC | 09:50 | |
| hberaud | o/ Can someone say me how to create a launchpad bug tracker page for a new project (etcd3gw)? I didn't find useful info https://docs.openstack.org/project-team-guide/bugs.html | 09:58 |
| hberaud | I guess this should be hosted by openstack/project-config or openstack/governance but I'm not sure | 09:59 |
| *** yamamoto has joined #openstack-infra | 10:03 | |
| *** gfidente|afk is now known as gfidente | 10:03 | |
| *** ociuhandu has joined #openstack-infra | 10:07 | |
| frickler | hberaud: IIUC you'd just create that page manually with the launchpad UI and then give the openstack user account the needed access rights | 10:07 |
| frickler | https://launchpad.net/projects/+new | 10:08 |
| *** akahat|lunch is now known as akahat|rover | 10:09 | |
| hberaud | Also, can someone reenque this job? https://zuul.opendev.org/t/openstack/build/aba1acc623e74cf08e82ffc4d73134aa/log/job-output.txt it failed previously and the monasca team ask to reenqueue the failing job http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019891.html | 10:09 |
| hberaud | frickler: thanks I'll take a look | 10:09 |
| *** ociuhandu has quit IRC | 10:12 | |
| hberaud | ah I missed that the bugtracker page already existed https://launchpad.net/python-etcd3gw so nothing new to create, frickler thank you anyway | 10:13 |
| frickler | hberaud: regarding monasco, it isn't obvious to me why the job wouldn't fail again with the issue you mention in your mail. I'll leave that to some other infra-root to look at | 10:13 |
| hberaud | frickler: ack | 10:14 |
| hberaud | apparently the team fixed the issue http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019928.html | 10:14 |
| hberaud | but I'm not a zuul expert too | 10:15 |
| *** ysandeep|afk is now known as ysandeep | 10:17 | |
| frickler | hberaud: hmm, then it still might make more sense to just push a new tag. I don't see any 2.x releases on https://hub.docker.com/r/monasca/tempest-tests/tags anyhow, so it shouldn't matter much | 10:19 |
| *** rpittau is now known as rpittau|bbl | 10:20 | |
| *** yamamoto has quit IRC | 10:20 | |
| hberaud | frickler: Do you mind replying directly to the ML thread? It will allow to the Monasca team to reply directly | 10:21 |
| hberaud | I asked them to join this IRC channel, but for now they aren't not there | 10:22 |
| hberaud | s/aren't not there/aren't there | 10:23 |
| frickler | hberaud: yes, I do mind, but I'll also wait for some other root to take a look at this anyway | 10:24 |
| hberaud | frickler: ok | 10:25 |
| *** ociuhandu has joined #openstack-infra | 10:26 | |
| *** priteau has joined #openstack-infra | 10:31 | |
| *** rcernin_ has joined #openstack-infra | 10:34 | |
| *** rcernin_ has quit IRC | 10:41 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 10:43 | |
| *** dtantsur|afk is now known as dtantsur | 10:44 | |
| *** hashar has quit IRC | 10:50 | |
| *** ociuhandu has quit IRC | 10:54 | |
| *** ociuhandu_ has joined #openstack-infra | 10:54 | |
| *** ociuhandu_ has quit IRC | 10:54 | |
| *** ociuhandu has joined #openstack-infra | 10:54 | |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| *** chaconpiza has joined #openstack-infra | 11:05 | |
| *** yamamoto has quit IRC | 11:07 | |
| *** ociuhandu has quit IRC | 11:14 | |
| *** ociuhandu_ has joined #openstack-infra | 11:14 | |
| *** jcapitao is now known as jcapitao_lunch | 11:24 | |
| *** rpittau|bbl is now known as rpittau | 11:24 | |
| *** iurygregory has joined #openstack-infra | 11:27 | |
| *** sboyron__ has quit IRC | 11:30 | |
| *** ociuhandu_ has quit IRC | 11:42 | |
| *** yamamoto has joined #openstack-infra | 11:42 | |
| *** rcernin_ has joined #openstack-infra | 11:42 | |
| *** yamamoto has quit IRC | 11:45 | |
| *** zzzeek has quit IRC | 12:18 | |
| *** rlandy has joined #openstack-infra | 12:20 | |
| *** yamamoto has joined #openstack-infra | 12:20 | |
| *** zzzeek has joined #openstack-infra | 12:21 | |
| *** jpena is now known as jpena|lunch | 12:29 | |
| *** yamamoto has quit IRC | 12:31 | |
| *** ociuhandu has joined #openstack-infra | 12:42 | |
| *** ociuhandu has quit IRC | 12:44 | |
| *** sboyron has joined #openstack-infra | 12:45 | |
| *** ociuhandu has joined #openstack-infra | 12:45 | |
| *** ociuhandu has quit IRC | 12:54 | |
| *** ociuhandu has joined #openstack-infra | 12:56 | |
| *** rcernin_ has quit IRC | 12:58 | |
| *** ociuhandu has quit IRC | 13:00 | |
| *** jcapitao_lunch is now known as jcapitao | 13:02 | |
| *** rcernin_ has joined #openstack-infra | 13:08 | |
| *** amoralej is now known as amoralej|lunch | 13:09 | |
| *** ociuhandu has joined #openstack-infra | 13:19 | |
| *** ysandeep is now known as ysandeep|afk | 13:24 | |
| *** ociuhandu has quit IRC | 13:26 | |
| *** jpena|lunch is now known as jpena | 13:28 | |
| *** ociuhandu has joined #openstack-infra | 13:38 | |
| *** ociuhandu has quit IRC | 13:45 | |
| *** ociuhandu has joined #openstack-infra | 13:47 | |
| *** sboyron has quit IRC | 13:48 | |
| *** ociuhandu has quit IRC | 13:51 | |
| *** sboyron has joined #openstack-infra | 13:51 | |
| *** derekh has quit IRC | 13:51 | |
| *** ociuhandu has joined #openstack-infra | 13:54 | |
| *** amoralej|lunch is now known as amoralej | 13:56 | |
| *** rcernin_ has quit IRC | 14:13 | |
| *** vishalmanchanda has quit IRC | 14:15 | |
| *** derekh has joined #openstack-infra | 14:17 | |
| *** Nisha_Agarwal has joined #openstack-infra | 14:25 | |
| Nisha_Agarwal | ianw, hi | 14:25 |
| *** irclogbot_0 has quit IRC | 14:27 | |
| Nisha_Agarwal | ianw, we were testing glean with ironic using CentOS8 image. but after glean apply the network data in network scripts, the networkmanager is not restarted | 14:29 |
| *** irclogbot_3 has joined #openstack-infra | 14:31 | |
| Nisha_Agarwal | ianw, and thats the reason the deploy fails as the interface doesnt gets the IP address provided as an input | 14:31 |
| *** gfidente has quit IRC | 14:31 | |
| fungi | Nisha_Agarwal: we pretty much only know that glean works reliably on our specific cloud images with the odd combinations of provider metadata choices and the network management frameworks we've picked (or distros have picked for us). if you find a combination which doesn't work, don't be surprised but please feel free to provide a patch to fix it | 14:35 |
| Nisha_Agarwal | fungi, i couldnt find how to patch glean...but manually when i just did "systemctl restart NetworkManager", the ip address got assigned | 14:36 |
| Nisha_Agarwal | and deploy passed | 14:36 |
| Nisha_Agarwal | fungi, specific cloud images mean what? | 14:37 |
| fungi | Nisha_Agarwal: the cloud images we build with diskimage-builder for nodepool to boot as our continuous integration test nodes in opendev | 14:38 |
| fungi | as for patches, you'd push them to gerrit for review just like any other openstack project | 14:39 |
| Nisha_Agarwal | fungi, the image is built using diskimage-builder | 14:39 |
| fungi | the repository is https://opendev.org/opendev/glean | 14:39 |
| *** rcernin_ has joined #openstack-infra | 14:40 | |
| Nisha_Agarwal | fungi, i know the repo :) | 14:40 |
| fungi | oh, got it, seemed like you were asking how to submit a patch | 14:41 |
| *** rcernin_ has quit IRC | 14:45 | |
| *** vishalmanchanda has joined #openstack-infra | 14:52 | |
| *** rcernin_ has joined #openstack-infra | 14:58 | |
| *** ricolin_ has quit IRC | 15:00 | |
| *** rcernin_ has quit IRC | 15:03 | |
| *** hashar has joined #openstack-infra | 15:04 | |
| *** rcernin_ has joined #openstack-infra | 15:07 | |
| *** rcernin_ has quit IRC | 15:12 | |
| *** d34dh0r53 has quit IRC | 15:12 | |
| *** d34dh0r53 has joined #openstack-infra | 15:19 | |
| *** slaweq has quit IRC | 15:21 | |
| *** slaweq has joined #openstack-infra | 15:23 | |
| *** dklyle has joined #openstack-infra | 15:31 | |
| *** ysandeep|afk is now known as ysandeep | 15:31 | |
| *** Nisha_Agarwal has quit IRC | 15:39 | |
| *** armax has joined #openstack-infra | 15:41 | |
| *** sboyron has quit IRC | 15:45 | |
| *** gfidente has joined #openstack-infra | 15:49 | |
| *** sboyron has joined #openstack-infra | 16:02 | |
| *** ykarel has quit IRC | 16:21 | |
| *** ricolin_ has joined #openstack-infra | 16:22 | |
| *** ociuhandu has quit IRC | 16:25 | |
| *** yamamoto has joined #openstack-infra | 16:30 | |
| *** yamamoto has quit IRC | 16:35 | |
| *** ociuhandu has joined #openstack-infra | 16:36 | |
| *** lamt has joined #openstack-infra | 16:42 | |
| *** gyee has joined #openstack-infra | 16:50 | |
| *** DannyMassa has joined #openstack-infra | 16:52 | |
| DannyMassa | Hello. I recently lost access to my account and there doesn't seem to be a way to recover it. I added a new email on review.opendev.org, made it the preferred email, and removed the old email. I added the new email to Ubuntu One, but logging in sends me to a new account. Is there any way to 'reconnect' my review.opendev.org account with my Ubuntu | 16:55 |
| DannyMassa | One account? | 16:55 |
| clarkb | that implies a new openid was created for you on the openid provider side when you made your changes there | 16:56 |
| clarkb | with the old sql setup we had a process for collapsing those together. There is probably a way to do that in notedb but I don't know what that process is | 16:57 |
| *** lucasagomes has quit IRC | 16:57 | |
| clarkb | the apis appear t oallow us to get and delete external ids but not set them? | 16:58 |
| DannyMassa | That's what I have read so far | 16:58 |
| clarkb | fungi: I think you looked at similar stuff when someone had an email address collision? do you know if you found any useful info around this? | 16:59 |
| clarkb | Ideally we'd avoid direct git repo manipulation since the notedb stuff is hashed in a number of complicated ways and gerrit does internal consistency checking (so I worry we'll get things half updated then it won't work at all) | 17:00 |
| fungi | i believe i saw that you can now add a new openid to your account in the version of gerrit we're using | 17:01 |
| fungi | however that doesn't help if you don't have access to the account any longer | 17:01 |
| clarkb | fungi: I think DannyMassa is saying they don't have access to the gerrit account becaues when they login they get a new account | 17:02 |
| fungi | as for e-mail address collisions, i think what i found was that now gerrit won't allow two accounts to have the same e-mail address associated with them even if one of them is set to inactive | 17:02 |
| clarkb | I only see get and delete. This might be worth an email to the gerrit ml | 17:03 |
| *** nightmare_unreal has quit IRC | 17:03 | |
| clarkb | to see if they've got any suggestions better than manually manipulate the git repo directly | 17:03 |
| *** amoralej is now known as amoralej|off | 17:03 | |
| *** jcapitao has quit IRC | 17:03 | |
| clarkb | DannyMassa: can you confirm you still have the same ubuntuone account? ordid that change too? | 17:04 |
| clarkb | (just to help us understand the way this happened) | 17:04 |
| DannyMassa | I (wrongly) assumed updating the email on both services would have me keep the same account. give me one sec I'll give you the rest of the details | 17:05 |
| *** rcernin_ has joined #openstack-infra | 17:08 | |
| *** jamesmcarthur has joined #openstack-infra | 17:10 | |
| DannyMassa | I added a new email to Gerrit, made it the preferred, removed the old email. For Ubuntu, I added the new email and made it the preferred. I cannot log in at all with Ubuntu using the new email, having to roll back to the old email. When signing on with the old email, I am given a new Gerrit account. | 17:11 |
| clarkb | thanks that helps. I think that implies we're likely experiencing the email conflict too | 17:12 |
| *** rcernin_ has quit IRC | 17:12 | |
| clarkb | "I cannot log in at all with Ubuntu using the new email, having to roll back to the old email." <- that is likely due to the email conflict | 17:13 |
| DannyMassa | The new empty account has my Ubuntu Identity and Old email. So presumably my regular account has no Ubuntu Identification, just the new email | 17:13 |
| clarkb | DannyMassa: they should both have your ubuntu openid, I think the problem is that you've got two of them | 17:14 |
| DannyMassa | oh OK | 17:14 |
| clarkb | that second new one is why you end up with a new gerrit account when logging in there | 17:14 |
| DannyMassa | Now I understand how the issues tie together | 17:14 |
| DannyMassa | Is there a next step I can take? | 17:18 |
| clarkb | DannyMassa: on your login.ubuntu.com side does it show you multiple openids? When this was all in launchpad it used to be that you could view your user page source to see them. Not sure how ubuntu one does it (and I'm in a meeting so not a great spot to check my own) | 17:18 |
| clarkb | DannyMassa: I think we'll need to engage upstream gerrit and see what they say. I think there are two related issues here that upstream gerrit should make simpler | 17:20 |
| clarkb | but I'm wary of hacking the external ids directly myself for fear of making it worse (through internal inconsistencies that the server dislikes) | 17:20 |
| DannyMassa | I don't see anything about OpenIDs on my account page. Just name, user-name, email addresses, password | 17:21 |
| DannyMassa | I just have the one username if that's what you're talking about. | 17:21 |
| clarkb | no, its a url that identifies your ubuntu one account with external services | 17:22 |
| clarkb | the problem historically is that in some situations ubuntu one will give you multiple of them | 17:22 |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| DannyMassa | I only see OpenID from the Gerrit side, I can't find anything on the Ubuntu side. | 17:25 |
| clarkb | ok | 17:25 |
| *** rpittau is now known as rpittau|afk | 17:26 | |
| clarkb | give me a bit to finish up meetings and I'll take a look at my side and then see what we can find from gerrit to dig in more | 17:26 |
| clarkb | I think we need to build a complete picture of what is going on and then figure out what our steps from there are | 17:26 |
| DannyMassa | Thank you so much for looking into this. | 17:26 |
| *** ysandeep is now known as ysandeep|away | 17:26 | |
| DannyMassa | well I'll be here when you're free | 17:26 |
| fungi | i'll try to catch up on the problem description once i'm out of my current meeting | 17:26 |
| *** ociuhandu_ has joined #openstack-infra | 17:38 | |
| *** ociuhandu has quit IRC | 17:42 | |
| *** ociuhandu_ has quit IRC | 17:42 | |
| zbr | fungi: easy https://review.opendev.org/c/opendev/gerritbot/+/764104 | 17:50 |
| zbr | btw, i see that default permissions do not allow us to edit hastags, but allow us to change topics. it should be both. | 17:51 |
| zbr | do we have any reasons for not allowing users to hashtags? if not, i can look into making a CR to address this. | 17:52 |
| *** sreejithp has joined #openstack-infra | 17:55 | |
| *** DannyMassa has quit IRC | 17:58 | |
| *** ralonsoh has quit IRC | 17:58 | |
| *** derekh has quit IRC | 18:01 | |
| *** DannyMassa has joined #openstack-infra | 18:03 | |
| fungi | it sounds fine, we can discuss in a bit. my meeting is running long | 18:05 |
| *** ricolin_ has quit IRC | 18:15 | |
| *** cloudnull has quit IRC | 18:19 | |
| *** cloudnull has joined #openstack-infra | 18:20 | |
| *** eolivare has quit IRC | 18:21 | |
| zbr | fungi: sure. ok to abandon https://review.opendev.org/c/opendev/git-review/+/681906 -- it personally do not like the idea of spaces and i doubt you have time to make it work. | 18:25 |
| *** cloudnull5 has joined #openstack-infra | 18:26 | |
| *** cloudnull has quit IRC | 18:27 | |
| *** cloudnull5 is now known as cloudnull | 18:27 | |
| *** jpena is now known as jpena|off | 18:32 | |
| clarkb | fungi: DannyMassa gave me identity details via pm which I can share if you are going to look too | 18:36 |
| *** gfidente is now known as gfidente|afk | 18:36 | |
| sreejithp | hello, newly added opendev to github mirroring job is failing with error "Load key "/tmp/ansible._wq583d2": invalid format". | 18:36 |
| sreejithp | i can use the ssh key to authenticate with github successfully | 18:36 |
| clarkb | sreejithp: could it be a problem with the secret encoding in zuul? | 18:36 |
| clarkb | double check no newlines were added maybe? (though I think those are fine at the end of key files) | 18:37 |
| sreejithp | i encoded with this command "zuul/tools/encrypt_secret.py --infile keys/airshipbot --tenant openstack https://zuul.openstack.org airship/vino --strip STRIP --outfile newsecret.yaml" | 18:37 |
| clarkb | fungi: I'm going to promote my user account to make api interaction more straightforward | 18:37 |
| sreejithp | clarkb: i see newline only at the end | 18:38 |
| *** vishalmanchanda has quit IRC | 18:40 | |
| clarkb | sreejithp: it definitelyseems like the issue is using the key at all on the client side based on that error message | 18:40 |
| clarkb | which is why I suspect somethign with the encoding of the key | 18:40 |
| clarkb | I wonder is base64 assumed maybe? | 18:40 |
| sreejithp | is there a way to validate the keys? | 18:41 |
| clarkb | I'm not sure since it is asymetric | 18:42 |
| *** sboyron has quit IRC | 18:44 | |
| fungi | sorry, scarfing down some lunch real quick but catching up on the extra openid problem now | 18:48 |
| fungi | clarkb: DannyMassa: so what's probably going to be the solution here is that we query the openid string from the new account, then remove it and set that account inactive; once that's done add that openid string as an external id for the old account | 18:53 |
| clarkb | fungi: yup its the last step that has me worried since I don't see an api method for it | 18:54 |
| clarkb | but working on concrete details now via read apis | 18:54 |
| clarkb | fungi: DannyMassa this is very curious though, the old account doesn't appear to have an openid external id at all | 18:56 |
| clarkb | the new one does | 18:56 |
| clarkb | I think that disproves the idea that ubuntu one is the problem by creating two openids? | 18:56 |
| clarkb | and that does make me wonder if gerrit saw a new preffered id from the provider and dieced to migrat DannyMassa to a new account | 18:57 |
| clarkb | also none of the email addresses seem to overlap if I list emails and external.ids | 18:57 |
| clarkb | so don't think we've collided tehre either | 18:58 |
| fungi | weird | 18:58 |
| *** DannyMassa has quit IRC | 18:59 | |
| clarkb | if you can confirm you see the same thing that would be great just ot make sure I'm not missing something obivous there | 18:59 |
| fungi | yeah, i'm getting to it, be another minute or two | 19:00 |
| *** hashar is now known as hasharAway | 19:00 | |
| fungi | need to switch computers now that i'm done lunching | 19:00 |
| clarkb | both accounts appear active so gerrit didn't go as far as to disable the old one | 19:01 |
| fungi | okay, i've added my web account to the admin group temporarily | 19:03 |
| *** DannyMassa has joined #openstack-infra | 19:04 | |
| DannyMassa | I'm in a meeting but I'll try to remain responsive here | 19:04 |
| clarkb | I've also cross checked the gerrit error log and that has some interesting clues | 19:08 |
| clarkb | it seems there was a second openid | 19:08 |
| clarkb | so at first it says "I cannot create the new account with new openid because the email address is already in use on the old account" | 19:08 |
| *** rcernin_ has joined #openstack-infra | 19:09 | |
| *** andrewbonney has quit IRC | 19:09 | |
| clarkb | then it switches to the openid on the new account and says it cannot create the openid there because of the email address conflict | 19:10 |
| clarkb | then later I suppose the email address conflict was resolved (the two accounts do have distinct email addrs) | 19:10 |
| clarkb | and that allowed the new account with the new openid to be created | 19:10 |
| clarkb | its almost like gerrit decided to move the external id from the old account to the new account because the email addr changed? | 19:11 |
| clarkb | I'm assuming the old account had the same openid previously which is maybe not a great assumption | 19:11 |
| fungi | clarkb: yeah, so i confirm that accounts/<id>/external.ids lists an openid for the new account and one e-mail address, for the old account it has no openids and two other e-mail addresses | 19:11 |
| clarkb | ya, I'm beginning to suspect this is Gerrit's way of resolving an exceptional situation. Unfortauntely it isn't helping :( | 19:12 |
| clarkb | and ya I think the way to fix this since there is no email conflict and no openid conflict is to add the openid(s) to the old account and remove them from the new | 19:12 |
| clarkb | but I think only the removal step is possible via the api? | 19:13 |
| clarkb | https://gerrit-review.googlesource.com/Documentation/config-accounts.html#external-ids I'm reading that next | 19:13 |
| *** rcernin_ has quit IRC | 19:14 | |
| DannyMassa | I still have access to all of the emails and the empty Gerrit account if there is anything that needs to be done on my end | 19:14 |
| clarkb | that document says that if we push via gerrit it should do a consistency check and reject things if we're out of sync | 19:15 |
| fungi | i agree the rest api docs indicate the available options for handling external ids are get and delete, no set/create/update | 19:15 |
| clarkb | hwoever, it doesn't say how it resolves a race between other people updating their external ids and us pushing over that | 19:15 |
| clarkb | since in theory we could pull the state, make our updates, then push and in that period of time someone could've created a new account and then break? | 19:16 |
| fungi | unlikely, but yeah probably something to keep an eye out for | 19:16 |
| fungi | oh! it's in a shared ref? not per-user? | 19:17 |
| fungi | that's... unfortunate | 19:17 |
| clarkb | yes | 19:17 |
| clarkb | I'm beginning to suspect there isn't a truly safe method for doing this | 19:19 |
| clarkb | we can stop gerrit and push directly but lose all checks | 19:19 |
| clarkb | or we can push to gerrit and race other accounts | 19:19 |
| fungi | probably something worth raising on the gerrit ml | 19:19 |
| clarkb | ya I'll work on a draft for that in an etherpad and describe the scenario and our concerns with brute forcing the fix | 19:20 |
| fungi | reading the details on how external ids are indexed in notedb, i suspect what happened is that the openid plugin overwrite the external id reference for that openid (the object identifier would have remained the same since it's a sha1 hash over the openid value alone) | 19:24 |
| clarkb | aha | 19:25 |
| *** rcernin_ has joined #openstack-infra | 19:25 | |
| clarkb | https://etherpad.opendev.org/p/N3E3V8APKHlBadW8IOsg fwiw | 19:25 |
| fungi | not sure why it would have done that, maybe because the id came in with a new e-mail address which wasn't one of the two associated with the old account? | 19:25 |
| clarkb | yes I think that is it | 19:27 |
| fungi | DannyMassa: to confirm, you have three e-mail addresses in gerrit. you associated all of them with your old gerrit account before logging into ubuntu.com with a new address? just trying to work out the exact steps which might be used to reproduce the problem | 19:27 |
| clarkb | gerrit updated the external id with email address that didn't map to any other accounts so then it created a new account | 19:27 |
| clarkb | and that new account now owns the existing openid | 19:27 |
| fungi | DannyMassa: or did you have two of them associated with the old account and then log into ubuntu.com with a third address? | 19:27 |
| clarkb | it is possible, though I don't think we're ready to test it that we can delete the exteranl id for the new account then have DannyMassa log in with ubuntu one updated back to the dsired value? that would then create a new openid and map it to the existing account? | 19:28 |
| clarkb | hrm no beacuse it will probably do the email check at that point and fail? | 19:29 |
| clarkb | ya I don't think that will actually work | 19:29 |
| *** rcernin_ has quit IRC | 19:30 | |
| fungi | i've also confirmed there's not, like, a third account which ended up getting created with only the name or something (only two accounts returned by a query for that name) | 19:33 |
| fungi | in theory the old account could still be accessible by http password, it still has a username associated with it | 19:34 |
| DannyMassa | sorry this meeting is relentless I'll need to catchup in a second | 19:34 |
| clarkb | fungi: yes and likely ssh too | 19:35 |
| fungi | DannyMassa: no need to apologize, we don't really have anything for you yet, still brainstorming | 19:36 |
| portdirect | DannyMassa: you've got a long way to go before you encounter a relentless meeting ;) | 19:36 |
| *** jamesmcarthur has quit IRC | 19:39 | |
| DannyMassa | portdirect Haha. I'm trying to understand these stories. 'Relentless' in the fact that I can't stop listening or I'll miss something. | 19:40 |
| fungi | clarkb: so one possible explanation for why pushing to refs/meta/external-ids is safe is that each external id is a separate object in git? | 19:40 |
| fungi | DannyMassa: so no zombies at least? | 19:40 |
| fungi | or maybe that's why it's important you listen closely | 19:40 |
| clarkb | fungi: they are but since they all live at a single ref and refs are version all their objects I don't think that helps? | 19:41 |
| clarkb | each ref represents an entire state of the world | 19:41 |
| fungi | ahh, yeah so maybe the fact that the ids are individual notes doesn't really help | 19:42 |
| fungi | unless a conflict would appear as a non-fast-forward update | 19:42 |
| fungi | so maybe safe as long as it's not push --force | 19:42 |
| DannyMassa | RE: so no zombies at least?:: not that I know of. All emails are under 1 Ubuntu account so there should only be the two accounts. I'll PM a third email a third email that's on the ubuntu account just to make sure nothing else got created | 19:43 |
| clarkb | ya we see three emails in 2 accounts | 19:43 |
| clarkb | as noted in the email draft this feels like a proper gerrit bug, it should've refused to create a new account with the old openid because that would orphan the old account | 19:47 |
| clarkb | I think my draft is largely complete | 19:48 |
| clarkb | fungi: DannyMassa maybe you can look that over and I'll send it out? | 19:48 |
| clarkb | if we can get away with it I think waiting a day or two to hear back from usptream would be nice | 19:48 |
| DannyMassa | Well one of these accounts is usable so I'll survive a couple days. | 19:49 |
| clarkb | well using the new account might make things more complicated for merging later | 19:49 |
| clarkb | because we'd end up orphaning the new account if we fix the new one | 19:50 |
| DannyMassa | Then I will stay away | 19:50 |
| clarkb | I mean, thats not the end of the world for us on the gerrit side of things I don't think | 19:50 |
| clarkb | but may irritate you :) | 19:50 |
| DannyMassa | yeah no worries. I'll figure something out. | 19:51 |
| clarkb | the simplest thing for us would actualyl be to just accept the old account is orphaned and move forawrd with the new account. But I'm not super happy with that as a user experience | 19:51 |
| clarkb | we are able to manage email addresses for an account so in theory could remove the conflicting email addresses from the old account and put them on the new account for you | 19:52 |
| clarkb | but thats still not a great answer :/ | 19:53 |
| DannyMassa | If my permissions are transferred, I don't mind. I don't have a ton of history in need of saving. | 19:53 |
| clarkb | ok maybe we keep that in our back pocket depending on what upstream says | 19:53 |
| *** jamesmcarthur has joined #openstack-infra | 19:53 | |
| clarkb | also sorry for the trouble | 19:54 |
| DannyMassa | no worries. | 19:57 |
| fungi | yeah, permissions transfer would be a challenge as far as change owner status on your prior changes, for group membership it's not hard (we can query for groups which include the old account and replace with the new one) | 19:58 |
| clarkb | made another small update to the etherpad, fungi if you're happy with it I can send it out? | 19:59 |
| fungi | but more for us, we'd like to figure out for sure 1. how this happens, and 2. what the "correct" way out is | 19:59 |
| fungi | clarkb: would it help to mention how we fixed similar external id problems pre-notedb? | 20:00 |
| clarkb | fungi: I briefly hit at that in the etherpad btu I can be mroe explicit. Might help people understand why this is so much more complicated (and consequently annoying) | 20:00 |
| fungi | your sql finger memory is better than mine | 20:02 |
| *** jamesmcarthur has quit IRC | 20:05 | |
| clarkb | fungi: you think that is ready enough to send out now? | 20:08 |
| fungi | yeah, should be good | 20:09 |
| openstackgerrit | Kendall Nelson proposed openstack/project-config master: Remove Karbor projects from infra https://review.opendev.org/c/openstack/project-config/+/767057 | 20:12 |
| *** rcernin_ has joined #openstack-infra | 20:12 | |
| clarkb | email sent and I've removed myself from admin | 20:16 |
| DannyMassa | (y) Thanks for all the help today. If there is any news and you don't see me here, you have all of my email addresses. | 20:18 |
| clarkb | DannyMassa: ha | 20:19 |
| clarkb | good point | 20:19 |
| DannyMassa | XD | 20:19 |
| fungi | sreejithp: were you able to sort out your account credentials for github mirroring? in your example encrypt_secret command, is keys/airshipbot an ssh private key file? it looks like the problem is ssh failing to read it due to formatting errors... there are some openssh versions which want pkcs#8 rather than pkcs#1/pem formatted keys... i wonder if that could be the problem | 20:19 |
| clarkb | https://groups.google.com/g/repo-discuss/c/4mbykY3oS1o if anyone wants to follow along and isn't subscribed I Think updates to the thread will show up there | 20:20 |
| clarkb | fungi: oh I hadn't considered that but ya that could be it because our zuul executors are xenial? | 20:20 |
| DannyMassa | awesome. bookmarked. | 20:20 |
| fungi | clarkb: well, that's the other interesting bit. i thought we ran that on the executors but this seems to be running on an ubuntu-bionic node? https://zuul.opendev.org/t/openstack/build/2fbfecc152684a77938d982dc535692e/console | 20:21 |
| clarkb | certainly looks like bionic there | 20:22 |
| *** rcernin_ has quit IRC | 20:23 | |
| fungi | here's another airship project doing it, and it's ubuntu-bionic there too: https://zuul.opendev.org/t/openstack/build/13d386428409486ebc4250cb9aad3ad1/console | 20:25 |
| fungi | openstack's jobs do just use the executor though: https://zuul.opendev.org/t/openstack/build/bd5be45baa9f4bd5aa856695c5e898dc/console | 20:27 |
| *** jamesmcarthur has joined #openstack-infra | 20:35 | |
| fungi | clarkb: also, not to pile on worries, but what happens when we upgrade the underlying operating system for the executors and suddenly the openssh on them no longer wants to read the old key format, i wonder? | 20:37 |
| fungi | i suppose we can decorate the jobs with some sort of sanity/version check and extra convert step | 20:37 |
| fungi | though the project keys on the scheduler will need to be mass updated too right? | 20:38 |
| fungi | (the ssh ones i mean) | 20:38 |
| sreejithp | fungi: yes its the ssh private key and it work well will github. i have tested it with this command "ssh -T git@github.com -i keys/airshipbot" | 20:38 |
| fungi | sreejithp: that makes me increasingly suspicious that the version of openssh on ubuntu 18.04 (where that job is configured to run) doesn't understand the same private key format. what openssh version are you using? | 20:39 |
| *** dave-mccowan has joined #openstack-infra | 20:39 | |
| sreejithp | ssh version is OpenSSH_8.2p1 | 20:41 |
| sreejithp | i am using ubuntu 20.04 to generate the secret | 20:41 |
| clarkb | fungi: new ssh reads the old format just fine aiui | 20:41 |
| clarkb | fungi: its only the other way around that is a problem | 20:41 |
| *** jamesmcarthur_ has joined #openstack-infra | 20:41 | |
| *** armax_ has joined #openstack-infra | 20:41 | |
| clarkb | I've got notes on how to convert a key, one sec | 20:41 |
| *** tosky has quit IRC | 20:41 | |
| *** armax has quit IRC | 20:42 | |
| *** armax_ is now known as armax | 20:42 | |
| *** tosky has joined #openstack-infra | 20:42 | |
| clarkb | `ssh-keygen -p -m PEM -f ./$FILE -N '' -P ''` note that assumes an empty passphrase | 20:42 |
| *** jamesmcarthur has quit IRC | 20:42 | |
| *** stevebaker has joined #openstack-infra | 20:42 | |
| clarkb | essentially what that command does is say please change the passphrase from empty to empty and a side effectof that is we can control the format versino wtih -m PEM | 20:42 |
| sreejithp | let convert the key and test this | 20:45 |
| fungi | yeah, so the openssh on ubuntu 18.04 is 7.6p1 | 20:46 |
| fungi | now to see when support for pkcs#8 format private keys was added | 20:46 |
| *** rcernin_ has joined #openstack-infra | 20:48 | |
| fungi | looks like rfc 4716 (new) format became the default for private keys in openssh 7.8: https://www.openssh.com/txt/release-7.8 | 20:56 |
| fungi | "ssh-keygen(1): write OpenSSH format private keys by default instead of using OpenSSL's PEM format. The OpenSSH format, supported in OpenSSH releases since 2014..." so, er, openssh should have already been supporting it since ages | 20:57 |
| fungi | i see where paramiko didn't add support until circa 2018 | 20:58 |
| *** jamesmcarthur_ has quit IRC | 21:00 | |
| *** jamesmcarthur has joined #openstack-infra | 21:11 | |
| *** DannyMassa has quit IRC | 21:12 | |
| *** jamesmcarthur has quit IRC | 21:16 | |
| *** sean-k-mooney has quit IRC | 21:16 | |
| *** jamesmcarthur has joined #openstack-infra | 21:17 | |
| *** jamesmcarthur has quit IRC | 21:20 | |
| *** jamesmcarthur has joined #openstack-infra | 21:20 | |
| sreejithp | clarkb, fungi, thanks converting the key and generating new secret worked. now mirroring is happening | 21:27 |
| *** jamesmcarthur has quit IRC | 21:29 | |
| *** ajitha has quit IRC | 21:47 | |
| *** rcernin_ has quit IRC | 21:49 | |
| *** rcernin has joined #openstack-infra | 21:50 | |
| *** jamesden_ has quit IRC | 21:51 | |
| *** jamesdenton has joined #openstack-infra | 21:51 | |
| *** matt_kosut has quit IRC | 21:53 | |
| *** matt_kosut has joined #openstack-infra | 21:54 | |
| *** matt_kosut has quit IRC | 21:58 | |
| fungi | sreejithp: thanks for trying that, we probably need to add something about that to https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#using-secrets | 22:07 |
| fungi | or somewhere | 22:07 |
| clarkb | fungi: maybe in the job details/info/docs | 22:07 |
| clarkb | since that is a bit of a job specific behavior? | 22:08 |
| fungi | mebbe | 22:09 |
| fungi | then again, it's also going to depend on the distro it's running from | 22:09 |
| fungi | will need to noodle on that | 22:09 |
| fungi | if nothing else, it's good to keep in mind that's a potential pitfall | 22:09 |
| clarkb | ++ | 22:09 |
| *** xek has quit IRC | 22:13 | |
| *** hasharAway has quit IRC | 22:14 | |
| *** yamamoto has joined #openstack-infra | 22:19 | |
| *** armax has quit IRC | 22:24 | |
| *** hamalq has joined #openstack-infra | 22:25 | |
| *** yamamoto has quit IRC | 22:30 | |
| *** yamamoto has joined #openstack-infra | 22:30 | |
| *** slaweq has quit IRC | 22:32 | |
| *** cloudnull8 has joined #openstack-infra | 22:44 | |
| *** cloudnull has quit IRC | 22:46 | |
| *** cloudnull8 is now known as cloudnull | 22:46 | |
| *** sreejithp has quit IRC | 23:03 | |
| *** lbragstad has quit IRC | 23:13 | |
| *** lbragstad_ has joined #openstack-infra | 23:13 | |
| dansmith | fungi: clarkb: do you have a gut feeling for which of ovh or vexx is faster for devstack bootstrap? | 23:24 |
| clarkb | dansmith: vexx | 23:25 |
| dansmith | too varied? about the same? one better than the other? no freaking? | 23:25 |
| dansmith | okay I thought so | 23:25 |
| clarkb | vexx is running on new amds now iirc | 23:25 |
| dansmith | so, my patch | 23:25 |
| clarkb | ovh has had iops limitations in the past | 23:25 |
| dansmith | which doesn't have every tweak in it locally, | 23:25 |
| dansmith | is 23% faster on ovh than the non-parallel one is on vexx | 23:25 |
| fungi | i'm too out of the loop on who has what hardware to place a wager | 23:25 |
| dansmith | I have another 5% gain locally I have to push up | 23:25 |
| clarkb | that sounds very promising | 23:26 |
| clarkb | dansmith: do you have a link to the faster ovh job logs? | 23:27 |
| dansmith | I'm also keeping the output from interleaving, which I think would be one of the primary complaints | 23:27 |
| dansmith | clarkb: https://d44b9fc81fe03a87c0e1-4e84cd8dd84809b3a32cd6523c3fd858.ssl.cf2.rackcdn.com/771505/4/check/devstack/8b7cae0/job-output.txt | 23:28 |
| dansmith | searching for "async job" in there should get you the highlights | 23:28 |
| *** dchen has joined #openstack-infra | 23:29 | |
| *** jamesmcarthur has joined #openstack-infra | 23:29 | |
| *** rlandy has quit IRC | 23:30 | |
| clarkb | we should be able to dig up a devstack job that ran on ovh recently too via logstash | 23:32 |
| clarkb | dansmith: double chekcing the job name for those logs is 'devstack' right? | 23:32 |
| dansmith | clarkb: yep, that's the bare "devstack" job | 23:33 |
| dansmith | which I think is just running devstack with all the bells on, no tempest | 23:33 |
| fungi | yep, it may also do smoketests | 23:33 |
| dansmith | doesn't look like it | 23:34 |
| *** jamesmcarthur has quit IRC | 23:34 | |
| fungi | oh neat, so a very good test for install/setup speedups in that case | 23:34 |
| dansmith | I have stacked locally more times in the last 72 hours than any one man should | 23:34 |
| clarkb | is 771505 your change? | 23:34 |
| dansmith | I feel like a nodepool worker | 23:34 |
| dansmith | https://review.opendev.org/c/openstack/devstack/+/771505 | 23:35 |
| dansmith | yup | 23:35 |
| clarkb | ok I've found it in logstash, now to find another recent example that doesn't have your improvements | 23:35 |
| dansmith | clarkb: I didn't mean to make work for you... | 23:35 |
| clarkb | dansmith: its ok, I think getting that data will be important for you to show the qa team if it shows a good improvement | 23:36 |
| clarkb | (also I'm waiting for nodepool to finish building an image so I can confirm some leaks on its disk) | 23:36 |
| dansmith | I guess I should collect some base numbers so I can compare new numbers regardless of what worker I land on | 23:37 |
| clarkb | dansmith: build_name:"devstack" AND node_provider:"ovh-bhs1" AND build_node:"ubuntu-focal" AND NOT build_change:"771505" is the logstash query I'm currently fiddling with if you want to take that and improve it | 23:37 |
| dansmith | cool thanks | 23:38 |
| clarkb | http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_61b/765149/3/check/devstack/61b8d61/job-output.txt is an example on bhs1 that did not run against your changeI think | 23:38 |
| clarkb | its runtime was 1263 seconds as self reported by devsatck, your example was 1911 | 23:38 |
| clarkb | almost a 6 minute improvement | 23:39 |
| clarkb | if that holds up thats a pretty decent improvment overall | 23:39 |
| dansmith | that's backwards | 23:39 |
| dansmith | 1263 is much faster than vexx at 2400 something | 23:39 |
| clarkb | oh interesting I didn't expect vexx to be so slow | 23:39 |
| clarkb | mnaser: may be interested in that too I guess | 23:40 |
| dansmith | the improvement I see from vexx to ovh with my change matches almost exactly my local findings, so it seemed legit | 23:40 |
| clarkb | er wait I can't do math I thought it was 2263 | 23:40 |
| dansmith | 1263 is faster than local bare metal | 23:40 |
| clarkb | so thats a curious result, but also may pointto ovh inconsistencies around io? | 23:41 |
| dansmith | I guess but that's like.. really fast | 23:41 |
| dansmith | maybe someone turned off a bunch of stuff in localconf for that? | 23:41 |
| clarkb | ya it could be change rlated | 23:41 |
| fungi | maybe they rubbed the servers with cheetah blood | 23:42 |
| dansmith | no, that was a cap-pip thing | 23:42 |
| dansmith | no changes to the config | 23:42 |
| clarkb | https://e9486e17b99c21dc6e90-b49d25c19e9456c92d7e602d475342ab.ssl.cf2.rackcdn.com/616416/12/check/devstack/3d23609/job-output.txt is a vexxhost example and it self reports 3890 | 23:42 |
| dansmith | yikes | 23:42 |
| dansmith | that's over an hour | 23:43 |
| clarkb | I changed ovh-bhs1 to vexxhost-ca-ymq-1 in my example query to find it | 23:43 |
| fungi | keep in mind that test runtimes have a wide variance to begin with. all it takes is a node getting scheduled to a host where some other instance (entirely likely it's also one of ours!) is starving it of resources and the build winds up taking 2x or longer | 23:44 |
| dansmith | that's the hostname right? | 23:44 |
| dansmith | because that's the same vexx I got 2400 from a couple days aog, | 23:44 |
| dansmith | so yeah may be too variable to compare single runs | 23:44 |
| clarkb | dansmith: thats the cloud name + region name | 23:44 |
| *** tosky has quit IRC | 23:44 | |
| dansmith | ah okay | 23:44 |
| fungi | i would caution against drawing too much conclusion from individual data points, yes | 23:44 |
| dansmith | fungi: yeah I thought they'd be closer than off by 25% just by luck of the draw | 23:45 |
| dansmith | but I guess not | 23:45 |
| dansmith | it's very consistent locally | 23:45 |
| fungi | you'll want a representative sample size to really have confidence | 23:45 |
| dansmith | (obviously) | 23:45 |
| fungi | your local runs are probably not sharing a machine with some tripleo and neutron jobs ;) | 23:45 |
| dansmith | presumably I could get a little more firmness by running once, unstack, restack, unstack, restack and then compare runs 2 and 3 | 23:46 |
| dansmith | could have a new neighbor in the middle, but that's as close a proximity as I can imagine | 23:46 |
| fungi | could be, though i also wouldn't trust that the performance of a single node is constant over time | 23:46 |
| dansmith | "more firmness" | 23:47 |
| dansmith | like, a stiff jello | 23:47 |
| clarkb | just don't call it salad | 23:47 |
| fungi | add some canned fruit, carrot shavings and little marshmallows | 23:47 |
| fungi | that'll firm it right up | 23:47 |
| dansmith | I'm from the south... lots of things are "salad" | 23:47 |
| fungi | we do call pretty much anything "salad" here, yep | 23:48 |
| dansmith | there has to be a "will it salad?" podcast.. | 23:48 |
| clarkb | even in the pnw people call things salad that I'm like uh wat | 23:49 |
| clarkb | and usually jello is involved | 23:49 |
| clarkb | also "dressing" is something you cook in a bird and not put on salad | 23:49 |
| fungi | in my part of the south, cooked greens are also colloquially referred to as "salad" | 23:49 |
| fungi | oh no, dressing is cooked around a bird. stuffing is cooked inside the bird | 23:50 |
| dansmith | fungi: I think it's like, if it's not something a red-blooded 'merican would order as a main dish, then it's probably a salad | 23:50 |
| fungi | macaroni and cheese? yeah that's a salad | 23:50 |
| dansmith | obvi | 23:50 |
| fungi | and it's also a vegetable if you're at the buffet | 23:50 |
| clarkb | lol | 23:50 |
| dansmith | alright, well, clarkb has sufficiently rained on my parade with his "data" | 23:50 |
| clarkb | sorry | 23:51 |
| dansmith | suffice it to say, it's much faster locally. Hopefully we can get it disabled by default and then running on a couple jobs to see | 23:51 |
| dansmith | man, even with the tripleo stuff removed, neutron has a bunch of >1h jobs and a few >2h ones | 23:53 |
| dansmith | still rechecking the patch to try to drop those, but it does not lottery well | 23:53 |
| fungi | i remember urging people to keep jobs below 45 minutes. and then it was please at least don't run jobs longer than an hour | 23:54 |
| dansmith | fungi: don't you control the job timeout? :) | 23:54 |
| fungi | a global one, yes | 23:55 |
| clarkb | if you ever want to institute chaos as a method for developing software: set the global timeout to 30 minutes | 23:55 |
| fungi | but we're hesitant to bring that particular hammer down | 23:55 |
| dansmith | it's been a struggle to get glance people to add tempest tests, so I really don't want it to be easier to write off those jobs | 23:55 |
| dansmith | clarkb: hah | 23:55 |
| dansmith | fungi: for sure, I'm only joking | 23:55 |
| fungi | i figured. we mostly use the global timeout as a backstop for for "yeah this job isn't ever gonna finish" | 23:56 |
| dansmith | yup | 23:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!