Thursday, 2021-06-03

*** dhellmann has quit IRC00:00
*** artom has quit IRC00:00
opendevreviewTim Burke proposed openstack/project-config master: pti-python-tarball: Add option to install bindep  https://review.opendev.org/c/openstack/project-config/+/79435100:01
opendevreviewIan Wienand proposed opendev/system-config master: Testing ansible unzip fixes  https://review.opendev.org/c/opendev/system-config/+/79435300:07
ianwhrm, the limestone mirror has a highish system load -> load average: 2.00, 2.08, 1.5400:09
ianwand i've noticed at least "system-config-run-letsencrypt (2. attempt)"00:10
ianwthe load average is now up to 5.  i will keep an eye, but it seems suspiciously like when we put a real load on the server it starts to fall over00:15
opendevreviewIan Wienand proposed opendev/system-config master: Add static eavesdrop.openstack.org site  https://review.opendev.org/c/opendev/system-config/+/79408900:20
*** artom has joined #opendev00:24
*** dhellmann has joined #opendev00:25
ianw794353 still doesn't appear to be in the gate queue.  it has a depends-on a github change which makes me suspicious00:26
opendevreviewMerged opendev/system-config master: Serve meetings.opendev.org  https://review.opendev.org/c/opendev/system-config/+/79408600:33
ianwload average: 106.95, 57.03, 25.8100:39
ianwi would say that the limestone mirror is unhappy00:39
opendevreviewIan Wienand proposed openstack/project-config master: Revert "Revert "Disable limestone due to mirror issues""  https://review.opendev.org/c/openstack/project-config/+/79435700:46
ianwi cna't seem to log into the ticketing system using any credentials00:50
*** ysandeep has joined #opendev00:52
opendevreviewMerged openstack/project-config master: Revert "Revert "Disable limestone due to mirror issues""  https://review.opendev.org/c/openstack/project-config/+/79435701:00
fungiianw: i didn't even know there was a ticketing system, just always pestered logan-01:13
fungihe's not in here, but still in our old channel on freenode, leaving me to suspect he might not be around much01:14
*** boistordu has joined #opendev02:11
*** boistordu_old has quit IRC02:17
*** gthiemon1e has joined #opendev02:54
*** gthiemonge has quit IRC02:59
*** brtknr has quit IRC03:00
*** brtknr has joined #opendev03:00
*** whoami-rajat has quit IRC03:06
opendevreviewIan Wienand proposed opendev/zone-opendev.org master: meetings.o.o : fix acme typo  https://review.opendev.org/c/opendev/zone-opendev.org/+/79436903:10
opendevreviewMerged opendev/zone-opendev.org master: meetings.o.o : fix acme typo  https://review.opendev.org/c/opendev/zone-opendev.org/+/79436903:20
*** ysandeep is now known as ysandeep|afk03:23
*** sledbarred has joined #opendev03:49
sledbarredhello, is there someone i can talk to about deleting my account on opendev?03:51
*** ysandeep|afk has quit IRC03:54
*** ykarel|away has joined #opendev03:56
*** sledbarred has quit IRC04:12
*** ykarel|away is now known as ykarel04:14
*** ysandeep|afk has joined #opendev04:30
*** ysandeep|afk is now known as ysandeep|ruck04:35
opendevreviewIan Wienand proposed opendev/system-config master: Add static eavesdrop.openstack.org site  https://review.opendev.org/c/opendev/system-config/+/79408904:36
*** ysandeep__ has joined #opendev04:38
opendevreviewyatin proposed openstack/diskimage-builder master: [DNM] Test oooq build image without CentOS preshipped repos  https://review.opendev.org/c/openstack/diskimage-builder/+/79439504:40
*** ysandeep|ruck has quit IRC04:44
*** marios has joined #opendev05:15
*** ysandeep__ is now known as ysandeep_afk05:26
*** ysandeep_afk is now known as ysandeep|ruck05:51
*** ralonsoh has joined #opendev06:01
*** slaweq has joined #opendev06:16
*** amoralej|off has joined #opendev06:28
*** amoralej|off is now known as amoralej06:30
*** mrunge_ is now known as mrunge06:41
*** whoami-rajat has joined #opendev06:43
opendevreviewMerged opendev/system-config master: Add static eavesdrop.openstack.org site  https://review.opendev.org/c/opendev/system-config/+/79408906:47
*** slaweq[m] has joined #opendev06:48
opendevreviewIan Wienand proposed openstack/project-config master: Revert "Revert "Revert "Disable limestone due to mirror issues"""  https://review.opendev.org/c/openstack/project-config/+/79440606:49
*** slaweq has quit IRC06:57
*** david-lyle has quit IRC06:57
opendevreviewNiklas Borg proposed zuul/zuul-jobs master: Add properties to upload-artifactory  https://review.opendev.org/c/zuul/zuul-jobs/+/78954307:09
*** slaweq[m] is now known as slaweq07:15
*** hashar has joined #opendev07:17
*** andrewbonney has joined #opendev07:18
*** jpena|off is now known as jpena07:20
*** tosky has joined #opendev07:20
*** lucasagomes has joined #opendev07:26
*** rpittau|afk is now known as rpittau07:34
*** lucasagomes has quit IRC07:38
*** noonedeadpunk has joined #opendev07:46
*** open10k8s_ has quit IRC08:03
*** lucasagomes has joined #opendev08:04
*** open10k8s_ has joined #opendev08:04
*** yasufum has quit IRC08:10
opendevreviewSankar Palanivel proposed zuul/zuul-jobs master: TEST: Do not merge  https://review.opendev.org/c/zuul/zuul-jobs/+/79426808:30
*** ysandeep|ruck is now known as ysandeep|lunch08:32
*** lourot has quit IRC08:32
*** lourot has joined #opendev08:32
*** ykarel is now known as ykarel|lunch08:48
zbrcorvus: what is the status of gertty? i see you worked on it recently while the only job running being "noop". See comment on https://review.opendev.org/c/ttygroup/gertty/+/79424109:02
opendevreviewSankar Palanivel proposed zuul/zuul-jobs master: TEST: Do not merge  https://review.opendev.org/c/zuul/zuul-jobs/+/79426809:23
*** cloudnull has quit IRC09:28
*** cloudnull has joined #opendev09:28
opendevreviewxinliang proposed opendev/system-config master: Enable openEuler mirroring  https://review.opendev.org/c/opendev/system-config/+/78487409:35
*** ysandeep|lunch is now known as ysandeep|ruck09:41
opendevreviewxinliang proposed opendev/system-config master: Enable openEuler mirroring  https://review.opendev.org/c/opendev/system-config/+/78487409:43
opendevreviewxinliang proposed opendev/system-config master: Enable openEuler mirroring  https://review.opendev.org/c/opendev/system-config/+/78487409:46
*** ykarel|lunch is now known as ykarel09:49
*** odyssey4me has quit IRC09:51
opendevreviewThierry Carrez proposed opendev/irc-meetings master: Move Large Scale SIG to #openstack-operators  https://review.opendev.org/c/opendev/irc-meetings/+/79455510:05
*** odyssey4me has joined #opendev10:10
*** osmanlic- has joined #opendev10:14
*** osmanlicilegi has quit IRC10:17
opendevreviewMerged opendev/irc-meetings master: Move Large Scale SIG to #openstack-operators  https://review.opendev.org/c/opendev/irc-meetings/+/79455510:26
opendevreviewxinliang proposed opendev/system-config master: Enable openEuler mirroring  https://review.opendev.org/c/opendev/system-config/+/78487410:36
*** osmanlic- has quit IRC11:20
*** osmanlicilegi has joined #opendev11:20
*** jpena is now known as jpena|lunch11:33
*** akahat has joined #opendev11:48
*** ysandeep|ruck is now known as ysandeep|debug_ssn12:01
*** amoralej is now known as amoralej|lunch12:12
*** jpena|lunch is now known as jpena12:23
mnaseri'm seeing some spam in #openinfra-summit / TheJulia is currently op'd there, perhaps fungi could help get that registered? it seems unregistered atm13:06
*** amoralej|lunch is now known as amoralej13:07
fungimnaser: #openinfra-summit isn't an official channel, you want #openinfra-events13:08
fungiwe moved from that on freenode months ago13:08
fungiTheJulia: ^13:08
mnaserahhhh, yes, the topic in freenode says the same13:09
fungii guess some people were still configured to autojoin it anyway13:09
mnaseryeah i haven't really parted13:09
* mnaser is trying to stay in as much channels as possible13:09
mnaserthanks and sorry for the noise :)13:10
fungithough that also demonstrates that we probably can't drop the +s flag from our official oftc channels, if channels without it are still getting spammed13:10
fungiwas it more libera bashing spam?13:10
mnaserfungi: yeah, #vexxhost was hit too13:11
fungii'm hoping once the war between freenode and libera calms down, that spam will dry up and we can consider dropping the +s flag on our channels13:12
TheJuliamnaser: topic set and departed13:12
fungithanks TheJulia!13:12
mnaserfungi: by the way, any reason why we didnt do +M ?13:13
mnaserwhich in combination of a 'join message' could be a little user friendly as to why someone cannot speak (i did some mode learning today bc of the spam in #vexxhost)13:13
fungimnaser: mostly because we don't want users to have to learn how to do nick registration and auth if possible13:14
fungimnaser: if you're doing +M then you might also consider +z and leaving some folks opped in the channel, so that they can see whatever the unidentified nicks are trying to say13:15
mnaserfungi: ohhhh i see!13:15
mnaseri thought +s was what +R is13:15
mnaserok, that is a much better solution then!13:15
* mnaser will drop +M and swithc to +s then13:16
fungibut yes, +Mz is our current fallback option if +s doesn't hold (though so far it seems to be working)13:16
mnaserfungi: thank you!13:16
fungiyw13:16
opendevreviewMerged ttygroup/gertty master: Bump python version.  https://review.opendev.org/c/ttygroup/gertty/+/79424113:18
*** open10k8s_ has quit IRC13:43
*** open10k8s has joined #opendev13:43
*** odyssey4me has quit IRC13:49
*** odyssey4me has joined #opendev14:00
*** lowercase has joined #opendev14:01
lowercaseHere i am14:04
lowercase@jrosser14:04
jrosserlowercase: hi there - i have a meeting starting right now (sorry!)14:05
jrossernoonedeadpunk: are you around to look at this python_venv_build trouble? ^^14:05
*** ysandeep|debug_ssn is now known as ysandeep14:05
*** ysandeep is now known as ysandeep|ruck14:05
opendevreviewMerged zuul/zuul-jobs master: Add properties to upload-artifactory  https://review.opendev.org/c/zuul/zuul-jobs/+/78954314:05
noonedeadpunkI am14:05
noonedeadpunklowercase: can you kindly provide some more details about what has been done during upgrade? And eventually what osa version do you run?14:06
noonedeadpunkafaik you was adding just new hosts to swift or smth like that? So it was not like osa version upgrade among computes?14:08
*** dklyle has joined #opendev14:11
*** ykarel is now known as ykarel|away14:37
lowercaseI was upgraded a dozen hypervisors from ussuri to victoria. The default variable for the os-nova-install playbook is to pull the git repo from opendev.org/openstack/nova. That step did not succeed due to the service being offline. I jumped into this irc and confirmed the issue was indeed remotely, but was caused due to large git clones of afromentioned git repos. I modified the14:44
lowercasedefault nova_git_repo variable to github.com/openstack/nova and was able to complete my upgrade.14:44
funginoonedeadpunk: ^14:44
lowercaseThe ceph stuff you mentioned is happening in tandem, but not related.14:46
noonedeadpunklowercase: hm, gotcha. and do you happen to have overwritten `venv_wheel_build_enable`?14:59
noonedeadpunkalso, are you running same OS on the controllers and hypervisors?15:00
*** jentoio has quit IRC15:08
lowercaseI checked, and no, venv_wheel_build_enable is not overwritten.15:15
lowercaseIn this case, I am not running the same OS version on the controllers and hypervisors. For Victoria, nova requires libvirt 5.0+ which in turn requires ubuntu 20.04. So I moved only the hypervisors to 20.0415:16
*** hashar has quit IRC15:20
clarkbsledbarred isn't here anymore but if they show up again and others are able to respond but not delete accuonts certainly an email to the list to start a conversation makes sense (we can then take it off list if we need to share details)15:22
clarkbfungi: have you tried pinging logan on freenode? if not I can do that. Is the TLDR that load on the mirror climbs to the point that it is no longer functional and we suspect disk io?15:23
*** ykarel|away has quit IRC15:24
*** hashar has joined #opendev15:25
fungii have not, no15:32
opendevreviewAde Lee proposed zuul/zuul-jobs master: Add role to enable FIPS on a node  https://review.opendev.org/c/zuul/zuul-jobs/+/78877815:33
clarkbfungi: is that tldr of the issue the way you understand it?15:33
noonedeadpunklowercase: that's the reason why it failed15:34
noonedeadpunkyou can't do that :)15:34
noonedeadpunkyou need to also upgrade at least one of the repo containers15:35
clarkbfungi: ianw: ok I PM'd over on freenode and offered to move to whatever platform works best for logan15:35
noonedeadpunkotherwise all upgrades will take "number of nodes"*time15:35
noonedeadpunklowercase: moreover, ubuntu 18.04 is supported on Victoria and does have required libvirt15:35
noonedeadpunkbut actaully good to know that one of our thoughs has been proved15:36
fungiclarkb: yeah, sorry, slow to respond, i'm literally in three meetings at the same time15:36
noonedeadpunkthen we can think about the way to protect from that15:36
clarkbfungi: no worries, today has been a slow start for me for other reasons and trying to kick start things so jumping in :)15:37
noonedeadpunklowercase: UCA http://ubuntu-cloud.archive.canonical.com/ provides required libvirt15:38
*** hashar is now known as Guest74015:55
*** hashar has joined #opendev15:55
*** Guest740 has quit IRC16:02
*** marios is now known as marios|out16:08
*** rpittau is now known as rpittau|afk16:12
lowercasenoonedeadpunk: if you go down the links, you will see that victoria is not supported by bionic16:20
lowercasehttp://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/16:20
*** gthiemon1e is now known as gthiemonge16:20
*** amoralej is now known as amoralej|off16:23
*** ralonsoh has quit IRC16:24
*** lucasagomes has quit IRC16:25
jrosserlowercase: thats not relevant, because for a victoria OSA deployment on bionic, the 'ussuri' UCA repo is used, see https://github.com/openstack/openstack-ansible-openstack_hosts/blob/stable/victoria/vars/ubuntu-18.04.yml#L9516:26
jrosserand as we deploy the services themselves from source code, not from .deb in UCA that is a valid approach to get the required libvirt16:26
*** jpena is now known as jpena|off16:27
*** ysandeep|ruck is now known as ysandeep|away16:27
*** sshnaidm is now known as sshnaidm|afk16:32
jrosserfungi: good news on the base-test patch https://review.opendev.org/c/openstack/openstack-ansible/+/79433216:39
jrosseri had to adjust it a little to remove our existing workaround, it's passed everything with that removed16:39
fungioh, cool. the bullseye and non-bullseye jobs both, hopefully (exercising both branches of logic there)16:40
*** marios|out has quit IRC16:40
fungiin that case i'll go ahead and propose the production change to that role16:41
fungiin unrelated news, this is somewhat unfortunate: https://discuss.python.org/t/887016:42
fungiapparently matrix was semi-represented as the "gitter" entry there. calls to add open source options like zulip seem to have gone ignored16:44
clarkbinfra-root ok both test listservs have been upgraded. I took notes on the process and recorded that all here https://etherpad.opendev.org/p/listserv-inplace-upgrade-testing-202116:44
clarkbif you can take a look at those two servers and see if anything seems compeltely broken that would be helpful (I'm not the biggest mailman expert)16:44
clarkbNote on lists.o.o only the opendev vhost is running though it should be fine to start the other vhosts if you like16:44
clarkbI'm not feeling super confident in the upgrade based on using those test nodes because our test images seem odd enough that we are unlikely to run itno the same package update questsions in prod. For this reason I think we probably should find some time to do a prod snapshot and test an upgrade on that.16:45
clarkbThat said this has been useful for sorting out things like the systemd units for mailman and general process16:45
clarkbI think major issues will be cuaght here and so far I haven't seen any. Its the samll issues like what do we do fro $config update during the upgrade where the snapshot will be helpful16:46
fungiinfra-root: i pushed a series of irc doc changes for system-config yesterday under topic:oftc, including the process change we talked about in the meeting for channel renames17:10
clarkbfungi: thanks, I'll review them now so I don't forget17:11
fungibroke the edits up by section to make discussion easier, they're each fairly trivial on their own17:11
clarkbhttps://review.opendev.org/c/opendev/system-config/+/794313 appears to be the base change17:12
fungiyes, that's the one discussed in the meeting so i put it first in hopes we could merge it sooner, as there are outstanding changes hingingon solidifying the process change we talked about17:12
clarkbfungi: any thoughts on why we should choose to make someone an admin vs an op?17:14
clarkbre https://review.opendev.org/c/opendev/system-config/+/794314/1/doc/source/irc.rst17:14
fungiwe could stand to add some documentation explaining when one would be more useful than the other. more generally it's that you really only need to add ops for basic things like banning disruptive users and updating the channel topic, but in cases where the team running the channel wants more direct control in collaboration with us having admins accomplishes that17:16
fungii'd probably add that explanation in a separate change17:17
*** hashar has quit IRC17:19
fungialso i'm trying to avoid us getting too prescriptive about how teams manage their channels. we already require they grant us admin access to the channel if they want us to run bots in it, so beyond that i think if they want additional channel rights for some of their team that's fine17:21
fungii certainly don't want it to seem like we're saying they're not allowed more control over their channels17:22
clarkbfor sure. Its more that I expect many won't understand the difference so we might need to give them that guidance17:23
clarkbnot to say "you can't have admin" but "if you want to do X Y and Z then admin is necessary"17:24
fungialso this was a stopgap for the migration, where a lot of channels had various rights delegated to specific users, and replicating that to oftc would have been intractable, so this allows them to re-bootstrap some semblance of that while still being tracked in git17:26
fungiand hopefully reduces the burden on us in the longer term, since we don't need to manually add people to channels any longer17:27
clarkbfungi: that stack lgtm17:28
fungithanks@17:28
fungis/@/!/17:28
*** ysandeep|away has quit IRC17:31
corvusfungi: your perl is showing17:31
fungioops!17:31
fungiat least nobody say my pascal17:32
fungisaw17:32
opendevreviewJeremy Stanley proposed opendev/base-jobs master: Set a fallback VERSION_ID in the mirror-info role  https://review.opendev.org/c/opendev/base-jobs/+/79117717:34
fungijrosser: ^17:34
clarkbinfra-root can you take a look at https://review.opendev.org/c/opendev/system-config/+/794049 and tell me if you think that is too dangerous? I just noticed we run puppet testing a lot but more and more we don't need/use puppet so wanted to flip around the decision making process from irrelevant files to files on that job17:50
clarkbfungi: https://review.opendev.org/c/opendev/base-jobs/+/791177 is tested via the base-test job?17:51
fungiclarkb: yes, see my comments on it17:51
clarkbaha ya I see your comment now. Thanks17:51
funginp17:51
opendevreviewJeremy Stanley proposed opendev/system-config master: Forward openstack-security ML to openstack-discuss  https://review.opendev.org/c/opendev/system-config/+/79465917:58
*** amoralej|off has quit IRC18:21
*** amoralej|off has joined #opendev18:30
*** hashar has joined #opendev18:32
*** jentoio has joined #opendev18:35
*** whoami-rajat has quit IRC19:10
*** dmsimard4 has joined #opendev19:14
*** ralonsoh has joined #opendev19:16
*** dmsimard has quit IRC19:20
*** dmsimard4 is now known as dmsimard19:20
opendevreviewJeremy Stanley proposed opendev/system-config master: Explain when to add admins and ops with accessbot  https://review.opendev.org/c/opendev/system-config/+/79467119:20
*** lowercase has quit IRC19:26
*** lowercase has joined #opendev19:29
*** amoralej|off has quit IRC19:37
*** hashar has quit IRC19:47
*** dhellmann has quit IRC20:15
*** dhellmann has joined #opendev20:16
*** ralonsoh has quit IRC20:28
*** lowercase has quit IRC20:28
*** hashar has joined #opendev20:34
*** hashar has quit IRC20:41
*** andrewbonney has quit IRC21:14
ianwhttps://meetings.opendev.org/ is up and appears to be proxying correctly to eavesdrop0121:49
ianw(i.e. https://meetings.opendev.org/irclogs and meetings.opendev.org/meetings)21:50
ianwstatic is setup to forward all eavesdrop.o.o there; if someone has time to put it in hosts and test it, i think we can switch21:52
ianwi think we can bringup eavesdrop01.opendev.org and maybe run a parallel meetbot in sandbox channel just to confirm everything21:55
ianwthen cut over to it, bring the other bots, kill all the puppet and move on21:56
*** whoami-rajat has joined #opendev21:56
fungiawesome! i'll check it out in a few minutes22:05
fungiianw: yep, working for me, including the redirect22:53
fungithat plan sounds great22:54
ianwfungi: excellent, thanks for checking.22:56
fungithanks for getting it going22:56
fungiin other ircbot news, statusbot is still running smoothly... i hate heisenbugs22:57
ianwas dicussed in #zuul, i'm going to try creating an API token for our zuul.  i think this just needs to go into the secrets dict; but i guess we'll have to run ansible and restart the scheduler to pick it up22:59
*** tosky has quit IRC23:00
*** mrunge has quit IRC23:01
corvusianw: i think that's XOR with app installation23:02
ianwcorvus: yeah, it seems that if the app is authenticated to the project, then graphql queries can run23:07
ianwmy theory is to add a developer api key to https://github.com/opendevadmin and put that in the api_token field of the github connection in our secrets23:07
ianwalright, i've done that; i can run the zuul graphql query using "Authorization bearer: <token>"23:30
ianwi've added the details in the usual places, and updated secrets with "api_token"23:30
ianwi guess we need a scheduler restart to see if it works; i'm not sure what the appetite for doing that is23:31
*** mrunge has joined #opendev23:34

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!