Wednesday, 2020-09-23

openstackgerritClark Boylan proposed opendev/system-config master: Update gitea project descriptions  https://review.opendev.org/75346400:04
clarkbI think that may actually work now00:04
*** hamalq has quit IRC00:23
*** DSpider has quit IRC00:26
*** zigo has quit IRC00:53
*** kden has joined #opendev01:00
*** kden has quit IRC01:12
openstackgerritIan Wienand proposed opendev/system-config master: Update gitea project descriptions  https://review.opendev.org/75346401:19
*** tkajinam_ has joined #opendev02:17
*** jrosser_ has joined #opendev02:19
*** zigo has joined #opendev02:26
*** tkajinam has quit IRC02:27
*** johnsom has quit IRC02:27
*** auristor has quit IRC02:27
*** mordred has quit IRC02:27
*** jrosser has quit IRC02:27
*** dviroel has quit IRC02:27
*** ajya has quit IRC02:27
*** odyssey4me has quit IRC02:27
*** jrosser_ is now known as jrosser02:27
*** johnsom has joined #opendev02:33
*** dviroel has joined #opendev02:33
*** ajya has joined #opendev02:33
*** odyssey4me has joined #opendev02:33
*** auristor has joined #opendev02:39
openstackgerritIan Wienand proposed opendev/system-config master: gitea: upgrade to 1.12.4  https://review.opendev.org/75348503:00
*** diablo_rojo has quit IRC03:01
*** ysandeep|away is now known as ysandeep03:36
*** kden has joined #opendev03:53
ianwa lot of POST_FAILURES on ^04:10
ianwopenstack.exceptions.HttpException: HttpException: 503: Server Error for\04:12
ianw url: https://api.us-east.open-edge.io:8080/swift/04:12
ianwindeed https://api.us-east.open-edge.io:8080/ is giving me a 50304:13
ianwdonnyd: ^04:13
openstackgerritIan Wienand proposed opendev/base-jobs master: base: temporarily disable openedge log upload  https://review.opendev.org/75349804:19
openstackgerritIan Wienand proposed opendev/base-jobs master: Revert "base: temporarily disable openedge log upload"  https://review.opendev.org/75349904:22
ianwthere's a lot of failures so i think this is the time to merge ^^04:27
fricklerianw: just approved it04:27
ianwfrickler: thanks :)04:27
ianwif it gets unlucky i can force merge it04:28
ianwi wonder if you do a retry with with_random_choice if you get another random choice04:34
ianwi think it's probably better to actually know about upload issues anyway, rather than hide them04:34
*** kden has quit IRC04:35
*** ysandeep is now known as ysandeep|brb04:35
*** kden has joined #opendev04:35
openstackgerritMerged opendev/base-jobs master: base: temporarily disable openedge log upload  https://review.opendev.org/75349804:36
ianw#status log disabled openedge log upload due to a 503 error from API04:37
openstackstatusianw: finished logging04:37
ianwinfra-root: ^  reverted with https://review.opendev.org/753499 when we get to the bottom of it04:37
fricklerianw: maybe also do a status notice to let people know they can safely recheck now?04:40
ianw#status notice A failing log storage endpoint has been removed, you can recheck any recent jobs with POST_FAILURE where logs have failed to upload04:43
openstackstatusianw: sending notice04:43
-openstackstatus- NOTICE: A failing log storage endpoint has been removed, you can recheck any recent jobs with POST_FAILURE where logs have failed to upload04:43
openstackstatusianw: finished sending notice04:46
*** ysandeep|brb is now known as ysandeep05:10
openstackgerritIan Wienand proposed opendev/system-config master: gitea: upgrade to 1.12.4  https://review.opendev.org/75348505:31
*** kden has quit IRC05:33
ianwclarkb: ^ ohhh, i see what i've done ... i've merged the git master template files against our templates, which are different to the 1.12.4 templates.  will fix tomorrow05:48
*** eolivare has joined #opendev06:13
openstackgerritzhangboye proposed openstack/diskimage-builder master: Remove install unnecessary package  https://review.opendev.org/75351306:17
*** Tengu has quit IRC06:17
*** Tengu has joined #opendev06:19
*** qchris has quit IRC06:21
*** kden has joined #opendev06:23
*** qchris has joined #opendev06:34
*** hashar has joined #opendev06:47
*** andrewbonney has joined #opendev07:14
*** fressi has joined #opendev07:16
*** tosky has joined #opendev07:19
*** slaweq has joined #opendev07:27
*** priteau has joined #opendev07:50
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
*** dtantsur|afk is now known as dtantsur08:03
*** bhagyashris is now known as bhagyashri|rover08:04
*** ysandeep is now known as ysandeep|lunch08:10
openstackgerritMerged openstack/project-config master: Revert "Use only noop jobs for openstack/monasca-transform"  https://review.opendev.org/75220708:14
*** dtantsur has quit IRC08:18
*** dtantsur has joined #opendev08:20
*** DSpider has joined #opendev08:34
*** tkajinam_ is now known as tkajinam08:34
*** priteau has quit IRC08:56
*** marios has joined #opendev09:02
*** lpetrut has joined #opendev09:16
*** ysandeep|lunch is now known as ysandeep09:51
*** hashar has quit IRC09:53
donnydLooking to find out what went wrong now10:08
*** slaweq has quit IRC10:17
*** slaweq has joined #opendev10:27
*** marios has quit IRC10:40
donnydOk, so it looks like all 3 of my rgw's crashed with no logging whatsoever10:47
donnydI guess I forgot to add those nodes to the central logging service11:01
*** fressi has quit IRC11:15
*** marios has joined #opendev11:27
mariosjoin openstack-election11:27
donnydianw: clarkb fungi I am working on an issue with OE object storage where everything via api is saying nosuckbucket, but the openstack client returns there is such a bucket and the bucket does have data in it11:54
*** fressi has joined #opendev11:55
donnydfor instance - https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_1c7/11:55
*** tosky_ has joined #opendev12:02
*** tosky is now known as Guest7321912:03
*** tosky_ is now known as tosky12:03
*** Guest73219 has quit IRC12:05
*** redrobot has joined #opendev12:24
*** Topner has joined #opendev12:31
*** priteau has joined #opendev12:35
*** Topner has quit IRC12:37
*** Topner has joined #opendev12:38
*** tkajinam has quit IRC12:51
*** tkajinam has joined #opendev12:52
*** Topner has quit IRC12:58
*** Topner has joined #opendev12:58
donnydit seems to only be an issue with public buckets, I can list and see authenticated buckets just fine13:16
*** tosky_ has joined #opendev13:26
*** tosky is now known as Guest1333613:26
*** tosky_ is now known as tosky13:26
*** Guest13336 has quit IRC13:28
*** tosky_ has joined #opendev13:29
*** tosky is now known as Guest6102013:31
*** tosky_ is now known as tosky13:31
*** Guest61020 has quit IRC13:32
*** fressi has quit IRC13:34
*** elod is now known as elod_afk13:41
openstackgerritDmitry Tantsur proposed ttygroup/gertty master: tokenizer: do not try to decode strings on Python 3  https://review.opendev.org/75369713:47
donnydok, I have a fix in place - will continue to monitor the issue13:59
*** tkajinam has quit IRC14:05
donnydI think I have an RCA - my upgrade scripts ran last night and it auto-upgraded radosgw - well rgw versions 14.2.8 and later have a bug that prevents public listings.. still not sure why things crashed - but service has been restored14:09
donnyda test url is here - https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_0f7/753131/1/gate/openstack-ansible-deploy-aio_distro_metal-centos-8/0f7bab3/job-output.txt14:10
*** dmellado has quit IRC14:12
*** ysandeep is now known as ysandeep|away14:28
fungiheaded out to run errands, should be back in an hour or two tops14:34
*** Tengu has quit IRC14:41
*** dmellado has joined #opendev14:46
*** elod_afk is now known as elod14:53
sshnaidmhi all, can I force python 3.7 on tox jobs? Instead of python 3.814:59
clarkbsshnaidm: the tox jobs/roles accept a python version specifier and will attempt to install and use that version. The key thing being you should ensure the nodeset label is capable of installing that version of python15:00
clarkb3.7 is available on ubuntu bionic but not xenial or focal for example15:00
clarkbalso there should be existing 3.7 jobs?15:00
*** diablo_rojo has joined #opendev15:01
sshnaidmI see.. wanted to test if ansible-lint fails on them..15:01
*** dmellado has quit IRC15:03
*** kden has quit IRC15:03
*** dmellado has joined #opendev15:22
openstackgerritMartin Kopec proposed openstack/project-config master: Add ansible-role-refstack-client under x namespace  https://review.opendev.org/75377315:33
*** lpetrut has quit IRC15:45
*** tosky has quit IRC15:59
*** marios is now known as marios|out16:00
*** marios|out has quit IRC16:06
*** hashar has joined #opendev16:28
*** dtantsur is now known as dtantsur|afk16:43
*** priteau has quit IRC17:02
*** eolivare has quit IRC17:07
*** dmellado has quit IRC17:16
*** dmellado has joined #opendev17:17
*** hamalq has joined #opendev17:21
*** dmellado has quit IRC17:42
*** dmellado has joined #opendev17:42
openstackgerritMerged zuul/zuul-jobs master: explicit error when test_command is undef  https://review.opendev.org/75165917:49
spotzHey is the links to the project code on opendev.org supposed to be in I think Russian?18:02
spotz Get Started is still in English18:02
fungispotz: no, sounds like browser locale detection going haywire18:04
fungiit's always been in english for me anyway18:04
spotzIt's always been in English for me until today:)18:04
fungispotz: have a url to a page which is showing up in russian for you?18:05
spotz fungi - https://opendev.org/explore/repos?tab=&sort=recentupdate&q=upstream18:05
fungithat's all showing as english for me in a current firefox version18:06
fungispotz: look down on the very bottom row of the page18:06
spotzOk I'm in firefox too and it says preferences English language which is why I paniced18:07
fungifor me there's a globe and it says "english"18:07
fungiand if i click that it pops up a selection list where i can pick a language18:07
spotzOk mine wasn't but I fixed it there:)18:07
fungiwhat had it detected?18:07
spotzWell it wasn't english but started with a character that looked like a p18:08
fungiokay, yeah, that's cyrillic for the russian word for russian language18:09
fungii wonder why it guessed that for you18:09
spotzNo clue unless maybe I hit by accident on the page last time. That or it knew I was looking for the Stacky McStackface stuff and it was punishing me for that18:11
fungistacky mcstackface is entirely russian, you should have known! ;)18:12
fungibut no, this is not the first time i've heard folks report gitea guesses their language preference incorrectly18:13
spotzhehe, I'm trying to find my old screenshots as I don't want to create another OSF user for demo reasons18:13
fungiyour firefox preferences has a language section with (some sort of) english selected?18:14
spotzfungi yeah18:14
fungiweird for gitea to ignore that18:14
spotzBut we are talking me:)18:18
fungispotz: looks like it could be this: https://github.com/go-gitea/gitea/issues/780518:19
spotzLooks like a possible18:20
fungispotz: if you go to your firefox preferences general language and select alternatives, do you have more languages listed there?18:22
fungier, i guess it says "set alternatives"18:22
fungifor me only "english (united states)" is listed18:22
fungibut i think that's the list that it will pass as acceptable languages, which could trigger that gitea bug18:23
fungiif it's that bug, the problem is that it's just picking an arbitrary language from that list rather than the primary you've set18:23
openstackgerritSaul Wold proposed openstack/project-config master: starlingx: add meta-starlingx repo  https://review.opendev.org/75381618:25
dmsimardo/ I need to delete a mistaken bad tag I just created but gerrit isn't letting me due to lack of force push privileges, could anyone delete 1.5.0.1 for recordsansible/ara ?18:25
dmsimardI've re-tagged under 1.5.1 already18:26
spotzfungi: nope18:29
*** sgw has left #opendev18:38
*** sgw1 has joined #opendev18:42
*** sgw has joined #opendev18:43
*** andrewbonney has quit IRC18:53
*** sgw1 has quit IRC19:03
*** sgw1 has joined #opendev19:04
*** sgw has quit IRC19:12
*** tosky has joined #opendev19:15
*** slaweq has quit IRC19:25
fungidmsimard: git tag deletion is extremely unreliable. it doesn't propagate from a remote via pull/fetch, so if we delete it from gerrit we'll also need to clear the repo on our 20 zuul executors and mergers, and anyone else who has already done a remote update likely has a copy of that tag which will never go away unless they manually delete their copy of it or completely reclone the repo20:05
*** sgw1 has quit IRC20:06
openstackgerritMerged openstack/project-config master: Add noop jobs to os-loganalyze  https://review.opendev.org/75339820:09
dmsimardyeah I'm aware of the distributed nature of git :( I don't mind that the tag might continue to exist if it's already been checked out20:10
dmsimardno need to clean it up everywhere, deleting it off of gerrit would be good enough20:11
dmsimardI'll deal with the fallout, if any20:11
openstackgerritAndreas Jaeger proposed openstack/project-config master: Fix ansible-lint  https://review.opendev.org/75384320:13
*** slaweq has joined #opendev20:14
openstackgerritMerged openstack/project-config master: starlingx: add meta-starlingx repo  https://review.opendev.org/75381620:15
*** slaweq has quit IRC20:16
*** slaweq has joined #opendev20:17
clarkbdmsimard: the issue is more that we have to deal with the fallout :) all the zuul executors and mergers have to be cleaned up20:24
clarkboh fungi mentioned that20:24
*** sgw has joined #opendev20:28
dmsimardI would've signed up for cleaning my screw up but I guess I can't do that anymore :/20:38
dmsimardit's not immediately clear to me what kind of issues zuul might bump into as a result of deleting the tag -- there's certainly no ara job that refers explicitly to that tag20:38
dmsimardI don't mind too much and can live with it I suppose20:40
*** hillpd has quit IRC20:49
*** diablo_rojo_phon has quit IRC20:49
*** jbryce has quit IRC20:49
*** jrosser has quit IRC20:50
*** dviroel has quit IRC20:50
*** rpittau|afk has quit IRC20:50
*** gmann has quit IRC20:50
*** mnasiadka has quit IRC20:50
*** hemanth_n has quit IRC20:50
*** knikolla has quit IRC20:50
*** Open10K8S has quit IRC20:50
*** jentoio has quit IRC20:50
*** ildikov has quit IRC20:51
*** hillpd has joined #opendev20:51
*** jbryce has joined #opendev20:51
*** mwhahaha has quit IRC20:51
*** erbarr has quit IRC20:51
*** knikolla has joined #opendev20:52
*** mnasiadka has joined #opendev20:52
*** jrosser has joined #opendev20:52
*** jentoio has joined #opendev20:52
*** ildikov has joined #opendev20:52
*** Open10K8S has joined #opendev20:53
*** erbarr has joined #opendev20:53
*** hemanth_n has joined #opendev20:53
*** mwhahaha has joined #opendev20:53
*** rpittau|afk has joined #opendev20:53
*** gmann has joined #opendev20:53
sgwinfra-root:  I guess I need to be added to the new starlingx/meta-starlingx-core group20:54
*** hashar has quit IRC20:54
clarkbsgw: done20:55
clarkbdmsimard: the issues zuul can run into are that the tag can "pollute" your jobs20:56
sgwI posted the zuul.yaml and .gitreview just now, ,I will self approve them20:56
clarkbdmsimard: that could affect things like versioning potentially if using pbr20:56
fungior reno20:56
*** dviroel has joined #opendev20:57
sgwclarkb: thanks, is there a way to confirm a user name for someone?  I am trying to add a second person to the new group21:07
clarkbsgw: you can use email addresses instead21:08
clarkbif you find a change they've pushed previously you can work from that too21:08
sgwYeah, I tried 2 different email the person provide, maybe he does not have a proper gerrit account yet.21:09
*** mlavalle has joined #opendev21:20
openstackgerritClark Boylan proposed opendev/system-config master: Update gitea project descriptions  https://review.opendev.org/75346421:33
sgwIs there an ordering issue on getting the .zuul and .gitreview  changes in or is zuul processing slower today?21:34
clarkbsgw: you need zuul jobs to merge .gitreview changes so either they go in together or zuul config goes in first21:35
clarkbnote you don't haveto commit the .gitreview for git-review to read and use it21:35
sgwI guess it's waiting for Zuul gate then, but I did not see it on the zuul status page, I have not set up a repo from an upstream before.21:36
sgwI saw a failure on the project-config new-project commit21:36
clarkbsgw: can you link to the cahgnes?21:39
sgwclarkb: project-config change: https://review.opendev.org/#/c/753816/21:53
sgwadding zuul.yaml to meta-starlingx: https://review.opendev.org/#/c/753855/21:54
clarkbsgw: the issue is https://review.opendev.org/#/c/753854/1 is the first change but that chnage has no zuul jobs confogured21:56
clarkb.zuul.yaml needs to be first21:57
*** dviroel has quit IRC21:59
*** dviroel has joined #opendev22:00
sgwso, remove +W for 854?22:04
clarkbsgw: no you need to reorder them in the git tree22:05
sgwOh, I understand22:05
clarkbsince git is a dag you always need to merge the parent change first, but in this case the parent change has no CI jobs so cannot merge. If you flip the order you'll fix that22:05
ianwdonnyd / infra-root : so should we revert the OE log skipping?22:06
donnydI need to work on my logging and alerting for that service, so probably not yet. What i had setup didn't actually work to tell me it was out. I need to get something better to monitor my apis22:08
donnydThe service is critical to the function of the CI, so i want to be sure. I would be willing to bet people are getting mad at me at this point. I would be22:09
ianwdonnyd: heh, no, no omelettes without breaking some eggs :)22:10
donnydI think since we turned OE back on for logging we have been through a bakers dozen22:11
donnydMaybe I can put something together that would be useful for more than just OE22:11
*** slaweq has quit IRC22:13
ianwdonnyd: at one point i went through getting our grafana instance to report to irc -> https://review.opendev.org/#/c/573183/22:30
ianwpeople did not want it reporting to the main channel, but we never fully explored having a separate channel for alerts to be posted to22:31
ianwthere was some concern it would be a bit like "pager duty" for opendev admins, and that a lot of the time the problem is with the thing doing the alerting, not the actual thing22:33
ianw(e.g. disk space bumping to 91% setting off alarms, then dropping, rinse, repeat)22:33
ianwthere's probably a strong argument that "jobs failing to upload logs" is it's own alert anyway -- being notified via some other mechanism I doubt would have been that much faster anyway22:36
donnydOk, then maybe i will just wire it to myself for now then22:40
donnydBecause that makes sense, and pager duty is not much fun22:41
*** knikolla has quit IRC22:50
*** hemanth_n has quit IRC22:50
*** ildikov has quit IRC22:50
*** jbryce has quit IRC22:50
*** mwhahaha has quit IRC22:50
*** erbarr has quit IRC22:50
*** davidlenwell has quit IRC22:50
*** bbezak has quit IRC22:50
*** johnsom has quit IRC22:50
*** ShadowJonathan has quit IRC22:50
*** jentoio has quit IRC22:50
*** mnasiadka has quit IRC22:50
*** mnaser has quit IRC22:50
*** davidlenwell has joined #opendev22:51
*** knikolla has joined #opendev22:51
*** erbarr has joined #opendev22:51
*** ildikov has joined #opendev22:51
*** hemanth_n has joined #opendev22:51
*** bbezak has joined #opendev22:51
*** jbryce has joined #opendev22:51
*** ShadowJonathan has joined #opendev22:51
*** seongsoocho has quit IRC22:52
*** seongsoocho has joined #opendev22:52
*** rm_work has quit IRC22:52
*** mnaser has joined #opendev22:53
*** rpittau|afk has quit IRC22:54
*** jrosser has quit IRC22:54
*** mnasiadka has joined #opendev22:54
*** mwhahaha has joined #opendev22:54
*** rm_work has joined #opendev22:54
*** johnsom has joined #opendev22:55
*** jrosser has joined #opendev22:56
*** jentoio has joined #opendev22:56
*** rpittau|afk has joined #opendev22:56
*** hemanth_n has quit IRC22:56
*** davidlenwell has quit IRC22:56
*** davidlenwell has joined #opendev22:59
*** hemanth_n has joined #opendev22:59
*** tkajinam has joined #opendev23:03
fungisgw: if you're having someone to a group (or as a requested reviewer on a change) and think they have a gerrit account, it's possible they have more than one active account associated with the same e-mail address (this can happen due to corner cases with address changes at an openid provider). i can check the database to see if that might be the case, if you let me know (here or in private) who you're trying23:14
fungito add. we have some solutions, usually picking the user's least preferred account and setting it inactive23:14
fungier, i meant to say if you're having trouble adding someone23:15
*** elod has quit IRC23:32
*** elod has joined #opendev23:34
*** moppy has quit IRC23:35
*** moppy has joined #opendev23:36
*** hamalq has quit IRC23:37
*** kden has joined #opendev23:41
*** tosky has quit IRC23:49

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!