| *** diurnalist has quit IRC | 00:06 | |
| *** diurnalist has joined #openstack-keystone | 00:31 | |
| *** diurnalist has quit IRC | 00:51 | |
| *** diurnalist has joined #openstack-keystone | 00:53 | |
| *** diurnalist has quit IRC | 00:59 | |
| *** hemna has quit IRC | 01:01 | |
| *** hemna has joined #openstack-keystone | 01:01 | |
| *** diurnalist has joined #openstack-keystone | 01:11 | |
| *** hemna has quit IRC | 01:17 | |
| *** hemna has joined #openstack-keystone | 01:18 | |
| *** diurnalist has quit IRC | 01:47 | |
| *** diurnalist has joined #openstack-keystone | 02:10 | |
| *** gyee has quit IRC | 02:11 | |
| *** diurnalist has quit IRC | 02:14 | |
| *** rcernin has quit IRC | 02:37 | |
| *** rcernin has joined #openstack-keystone | 02:48 | |
| *** tobberydberg_ has quit IRC | 03:04 | |
| *** bnemec has quit IRC | 03:04 | |
| *** irclogbot_2 has quit IRC | 03:05 | |
| *** hemna has quit IRC | 03:05 | |
| *** bnemec has joined #openstack-keystone | 03:06 | |
| *** hemna has joined #openstack-keystone | 03:06 | |
| *** irclogbot_2 has joined #openstack-keystone | 03:07 | |
| *** tobberydberg has joined #openstack-keystone | 03:10 | |
| *** ayoung has quit IRC | 03:18 | |
| *** ayoung has joined #openstack-keystone | 03:22 | |
| *** diurnalist has joined #openstack-keystone | 03:23 | |
| *** ayoung has quit IRC | 03:29 | |
| *** ayoung has joined #openstack-keystone | 03:32 | |
| *** rcernin has quit IRC | 03:54 | |
| *** diurnalist has quit IRC | 04:02 | |
| *** ayoung has quit IRC | 04:03 | |
| *** ayoung has joined #openstack-keystone | 04:04 | |
| *** diurnalist has joined #openstack-keystone | 04:13 | |
| *** rcernin has joined #openstack-keystone | 04:21 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-keystone | 04:33 | |
| *** abdysn has joined #openstack-keystone | 05:21 | |
| *** vishakha has joined #openstack-keystone | 05:53 | |
| *** diurnalist has quit IRC | 06:14 | |
| *** Abdallahyas has joined #openstack-keystone | 06:56 | |
| *** abdysn has quit IRC | 06:59 | |
| *** bengates has joined #openstack-keystone | 07:02 | |
| *** Abdallahyas has quit IRC | 07:54 | |
| *** abdysn has joined #openstack-keystone | 07:55 | |
| *** rcernin has quit IRC | 09:11 | |
| *** rcernin has joined #openstack-keystone | 09:34 | |
| *** tkajinam has quit IRC | 09:36 | |
| *** markvoelker has joined #openstack-keystone | 09:45 | |
| *** markvoelker has quit IRC | 09:49 | |
| *** rcernin has quit IRC | 09:56 | |
| *** vishakha has quit IRC | 10:50 | |
| *** markvoelker has joined #openstack-keystone | 10:52 | |
| *** markvoelker has quit IRC | 10:56 | |
| *** rcernin has joined #openstack-keystone | 11:31 | |
| *** markvoelker has joined #openstack-keystone | 11:35 | |
| *** markvoelker has quit IRC | 11:40 | |
| mnaser | hrm - is there anyone who has played with using application credentials for openstack services (keystonemiddleware?) | 11:55 |
|---|---|---|
| *** raildo has joined #openstack-keystone | 11:58 | |
| knikolla | mnaser: i haven't, but it should be possible since keystonemiddleware supports using keystoneauth plugins by setting auth_type | 12:31 |
| mnaser | knikolla: ok, we're giving it a try right now and we'll see how that all pans out | 12:32 |
| frickler | mnaser: not yet, but it is on my todo list, so I'd be interested to learn something about it. my main motivation would be credential rotation, which is essentially impossible currently | 12:32 |
| knikolla | mnaser: awesome. let me know if there's something that's not working as it should. | 12:32 |
| mnaser | knikolla: i mean, right now there is something not working, we tried the 'provide username and application name/secret' and that did not work, resulting in service giving 403s | 12:33 |
| mnaser | knikolla: we're trying again by explicitly providing id and secret instead | 12:33 |
| mnaser | knikolla: i.e. this https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85e/745551/19/check/openstack-operator:functional/85ea9cb/controller/logs/etc/heat/heat_conf.txt | 12:34 |
| mnaser | frickler: yeah -- that's one of our big goal, right now stage 1 is doing long lived credentails and stage 2 is rotating them automatically :) | 12:34 |
| mnaser | fyi if you're looking at those logs you'll find services running in k8s so -https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85e/745551/19/check/openstack-operator:functional/85ea9cb/docker/ | 12:34 |
| mnaser | hmm, interesting.. "Could not find Application Credential: heat.: keystone.exception.ApplicationCredentialNotFound: Could not find Application Credential: heat" | 12:35 |
| mnaser | we're probably going to use id based things just because its going to be far more performant and avoid extra lookups | 12:36 |
| knikolla | so many pods! | 12:39 |
| *** Luzi has joined #openstack-keystone | 12:55 | |
| *** Luzi has quit IRC | 13:00 | |
| *** vishalmanchanda has joined #openstack-keystone | 13:04 | |
| *** dave-mccowan has joined #openstack-keystone | 13:11 | |
| knikolla | gagehugo, cmurphy, lbragstad: i won't be able to host/attend the weekly meeting as i have a medical appt. if either of you wants to run it please go ahead, otherwise we can cancel it for this week. sorry! | 13:17 |
| *** dave-mccowan has quit IRC | 13:29 | |
| *** abdysn has quit IRC | 14:03 | |
| *** rcernin has quit IRC | 14:08 | |
| *** rcernin has joined #openstack-keystone | 14:19 | |
| *** rcernin has quit IRC | 14:24 | |
| *** rcernin has joined #openstack-keystone | 14:27 | |
| *** rcernin has quit IRC | 14:31 | |
| *** flaviof has joined #openstack-keystone | 14:35 | |
| *** diurnalist has joined #openstack-keystone | 14:40 | |
| flaviof | hi frickler et all... grenade jobs in neutron (and nova) in Zuul master seem to be broken. The smoking gun on that failure seems to be related to this: https://zuul.opendev.org/t/openstack/build/5736faa004194fc6af092c63d96e76e3/log/compute1/logs/screen-n-cpu.txt#7724-7727 Any chance the folks here have an understanding on recent changes (after Aug/8/20) that could be related to that? | 14:43 |
| flaviof | ERROR nova.compute.manager [instance: 07888380-65b9-41cd-8e8d-6f55624e9204] neutronclient.common.exceptions.ServiceUnavailable: The server is currently unavailable. Please try again at a later time.<br /><br /> | 14:44 |
| *** vishakha has joined #openstack-keystone | 14:55 | |
| *** brtknr has left #openstack-keystone | 15:08 | |
| cmurphy | knikolla: i could but if there aren't any topics maybe just cancel it? | 15:19 |
| knikolla | cmurphy: ++ that works. sending an email out. | 15:20 |
| cmurphy | flaviof: looks like the keystone error is https://zuul.opendev.org/t/openstack/build/5736faa004194fc6af092c63d96e76e3/log/controller/logs/screen-keystone.txt#37999 I think I've seen that before but not consistently, is it persistent? there are no changes in keystone since august 5 | 15:20 |
| vishakha | cmurphy knikolla Could you please look #link https://review.opendev.org/#/c/744940/ | 15:23 |
| flaviof | cmurphy: yeas, seems very persistent now: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-grenade-multinode&project=openstack/neutron + https://zuul.opendev.org/t/openstack/builds?job_name=nova-grenade-multinode&project=openstack/nova | 15:24 |
| gagehugo | knikolla: wfm, I will be double booked today | 15:24 |
| vishakha | cmurphy knikolla gagehugo Its was blocking the requirements repo to update pymysql latest version | 15:25 |
| knikolla | vishakha: thanks! | 15:26 |
| flaviof | cmurphy: just another ref point: https://review.opendev.org/#/c/741303/ if you look at the builds or that gerrit change, the grenade jobs began failing after Aug/8, it seems. | 15:26 |
| vishakha | knikolla gagehugo Also adding the skipped test cases again #link https://review.opendev.org/#/c/745376/ | 15:27 |
| flaviof | cmurphy: sorry for the newbie question, but is there any chance that this modification: https://review.opendev.org/#/c/744753/1/lib/keystone could cause the error in keystone? | 15:36 |
| cmurphy | flaviof: that seems unlikely to me | 15:38 |
| *** rcernin has joined #openstack-keystone | 15:40 | |
| *** rcernin has quit IRC | 15:44 | |
| frickler | cmurphy: flaviof: the keystone error is in msgpack, which was updated here https://review.opendev.org/#/c/745437/2/upper-constraints.txt , I'd bet on that to be the culprit | 15:51 |
| frickler | prometheanfire: ^^ | 15:51 |
| cmurphy | frickler: ++ | 15:51 |
| prometheanfire | ya, that just updated a few days ago | 15:52 |
| *** bengates has quit IRC | 16:24 | |
| *** bengates has joined #openstack-keystone | 16:25 | |
| *** bengates has quit IRC | 16:30 | |
| openstackgerrit | Merged openstack/keystone master: Skip tests to update u-c for PyMySql to 0.10.0 https://review.opendev.org/744940 | 16:38 |
| *** zzzeek has quit IRC | 16:47 | |
| openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Add missing test case in test_backends.py for assignments https://review.opendev.org/745710 | 17:02 |
| openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Fix PyMySql on latest version 0.10.0 https://review.opendev.org/745376 | 17:13 |
| lbragstad | knikolla cmurphy gagehugo the only request i had was reviews on https://review.opendev.org/#/c/744251/ | 17:29 |
| flaviof | prometheanfire++ frickler++ please advice on what we should do to remedy this situation. | 17:43 |
| frickler | flaviof: prometheanfire: I'd propose for someone to propose a revert of the msgpack bump for now, then someone can look into making the update backwards compatible. from the release notes on pypi I fear that that may become difficult | 17:46 |
| prometheanfire | I'm unsure about the nature of the problem, is it something keystone needs to adapt to for msgpack-1.0.0 or a bug in msgpack (that's upstream reported/ack'd preferably)? | 17:46 |
| prometheanfire | so, normal major release bump issue? | 17:46 |
| frickler | prometheanfire: from https://pypi.org/project/msgpack/ I get that 1.0 is intentionally breaking things, yes | 17:47 |
| frickler | from the backtrace it looks like dogpile.cache will need to handle that somehow, not sure if there are also other consumers | 17:48 |
| frickler | I'm also not sure why only the grenade-multinode job seems to see this, does that test with only one of the nodes updated? | 17:49 |
| * prometheanfire shrugs, that I do not know | 17:52 | |
| *** diurnalist has quit IRC | 18:15 | |
| melwitt | has anyone noticed this error in the gate? "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte" https://zuul.opendev.org/t/openstack/build/316ecf4d0b69480e95f3c71ceed60690/log/logs/screen-keystone.txt#30623 | 18:57 |
| melwitt | oh guh sorry, I see that it's already being discussed | 18:57 |
| frickler | melwitt: yeah, but not much progress yet. it's eod for me, so if you could create a bug and possible a reqs patch to cap msgpack<1.0 that might be helpful | 19:22 |
| melwitt | ack | 19:25 |
| *** vishakha has quit IRC | 19:44 | |
| *** gyee has joined #openstack-keystone | 19:59 | |
| *** openstackgerrit has quit IRC | 20:52 | |
| -openstackstatus- NOTICE: The openstackgerrit IRC bot (gerritbot) will be offline for a short period while we redeploy it on a new server | 20:52 | |
| *** vishalmanchanda has quit IRC | 21:14 | |
| *** raildo has quit IRC | 21:15 | |
| *** zzzeek has joined #openstack-keystone | 21:38 | |
| *** ayoung has quit IRC | 21:48 | |
| *** ayoung has joined #openstack-keystone | 21:49 | |
| *** gregwork has joined #openstack-keystone | 22:45 | |
| *** rcernin has joined #openstack-keystone | 22:45 | |
| *** tkajinam has joined #openstack-keystone | 22:54 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!