Thursday, 2020-04-09

calcmandando any auto manufacturers use cobol for the computers in the cars these days?00:28
fungii highly doubt it. did they ever? most people i know working on automotive software use very low-level languages00:43
ianwsomehow i doubt cobol and CAN is a popular combination00:44
fungican? i'd need an entire six pack before i write cobol00:45
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-pip: Add role  https://review.opendev.org/71763901:32
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-pip: export ensure_pip_virtualenv_command  https://review.opendev.org/71822401:32
openstackgerritIan Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: use ensure-pip  https://review.opendev.org/71788201:32
openstackgerritIan Wienand proposed zuul/zuul-jobs master: fetch-subunit-output test: use ensure-pip  https://review.opendev.org/71822501:32
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-tox: use ensure-pip role  https://review.opendev.org/71766301:32
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Update Fedora to 31  https://review.opendev.org/71765701:32
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-pip: export ensure_pip_virtualenv_command  https://review.opendev.org/71822401:45
openstackgerritIan Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: use ensure-pip  https://review.opendev.org/71788201:45
openstackgerritIan Wienand proposed zuul/zuul-jobs master: fetch-subunit-output test: use ensure-pip  https://review.opendev.org/71822501:45
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-tox: use ensure-pip role  https://review.opendev.org/71766301:45
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Update Fedora to 31  https://review.opendev.org/71765701:45
*** corvus has quit IRC01:48
*** corvus has joined #opendev01:49
*** ysandeep|away is now known as ysandeep|rover01:52
*** mrunge_ has joined #opendev02:02
*** mrunge has quit IRC02:03
ianw[repo-update|http://mirror.bhs1.ovh.openstack.org/opensuse/update/leap/15.1/oss/] Valid metadata not found at specified URL02:28
ianwhrm suse builds failing02:28
ianwthere seem to be a lot of vos releases in progress02:32
ianwnothing actually looks ridiculously over time02:37
*** ysandeep|rover is now known as ysandeep|afk03:09
*** ysandeep|afk is now known as ysandeep04:00
*** logan- has quit IRC04:45
*** ykarel|away is now known as ykarel04:47
*** logan_ has joined #opendev04:47
*** logan_ is now known as logan-04:48
openstackgerritIan Wienand proposed opendev/system-config master: mirror-update: stats for vos release of mirrors  https://review.opendev.org/71860004:50
*** _mlavalle_1 has joined #opendev04:50
*** mlavalle has quit IRC04:53
*** ysandeep is now known as ysandeep|rover04:56
*** sgw has quit IRC05:01
ianwAJaeger: i believe our suse mirror is up-to-date, but we're seeing errors in jobs https://zuul.opendev.org/t/zuul/build/25c9843d1d7f41b68d2c962cf4274cf8/log/job-output.txt#27905:22
ianw"Valid metadata not found at specified URL"05:22
ianwwhich leads me to think it's related to the upstream mirror05:22
AJaegerlet's ask dirk - but he's only on #openstack-infra, so I pinged him there05:23
openstackgerritMerged zuul/zuul-jobs master: tox: Don't inline python warnings  https://review.opendev.org/71855405:39
*** jaicaa has quit IRC05:52
*** jaicaa has joined #opendev05:55
*** dpawlik has joined #opendev06:07
*** dpawlik has quit IRC06:37
*** dpawlik has joined #opendev06:39
*** roman_g has quit IRC06:41
*** _mlavalle_1 has quit IRC06:41
*** sshnaidm|afk is now known as sshnaidm06:55
*** rpittau|afk is now known as rpittau07:18
*** dirk has joined #opendev07:39
*** tosky has joined #opendev07:42
*** ralonsoh has joined #opendev07:45
*** zbr has quit IRC07:46
*** zbr has joined #opendev07:46
*** ysandeep|rover is now known as ysandeep|lunch08:10
openstackgerritIvan Kolodyazhny proposed openstack/project-config master: Add publish-to-pypi jobs for new Vitrage xstatic-* projects  https://review.opendev.org/71663009:05
*** ykarel is now known as ykarel|lunch09:09
*** dpawlik has quit IRC09:10
*** ysandeep|lunch is now known as ysandeep|rover09:18
*** hashar has joined #opendev09:20
*** dpawlik has joined #opendev09:49
*** ykarel|lunch is now known as ykarel09:51
*** rpittau is now known as rpittau|bbl10:13
dirkfrickler: so, continuing the discussion here..10:20
dirkfrickler: the situation is that there was an issue with how the root certificates are stored which are used for validating the update channels. this is fixed for many weeks, but the image hasn't been rebuilt sicne then, so it has the old certificates still10:21
dirkfrickler: the image is not rebuilding properly due to an unrelated reason, which is already fixed in dib via https://review.opendev.org/#/c/716437/210:21
dirkhowever there was no dib release since then, so the fix is n ot effective10:21
dirkjust getting a dib release would make nodepool images build again and then the problem above should be gone10:22
dirkI've been lobbying for a new dib release already previously, it seems now I have even better reasons for getting it ;)10:22
dirkianw: ^^10:22
fricklerdirk: ah, thanks for double checking. I'll have to defer on ian for releasing dib, though10:32
*** hashar has quit IRC10:50
*** hashar has joined #opendev11:08
*** hashar is now known as hasharLunch11:09
*** hasharLunch is now known as hashar11:39
*** ysandeep|rover is now known as ysandeep|coffee11:47
openstackgerritBernard Cafarelli proposed openstack/project-config master: Update Grafana dashboards for stable Neutron releases  https://review.opendev.org/71867611:49
*** dpawlik has quit IRC11:56
*** rpittau|bbl is now known as rpittau11:57
*** ysandeep|coffee is now known as ysandeep|rover12:00
*** dpawlik has joined #opendev12:08
*** roman_g has joined #opendev12:13
dirkfrickler: a recheck succeeded - https://zuul.opendev.org/t/zuul/build/4c71706b3aa349ffb2be314c3246d56d - WTF ?!12:21
openstackgerritSean McGinnis proposed openstack/project-config master: Extract only series name in release job template  https://review.opendev.org/71868912:30
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job  https://review.opendev.org/71869413:01
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job  https://review.opendev.org/71869413:11
*** ykarel is now known as ykarel|afk13:11
*** mrunge_ is now known as mrunge13:14
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job  https://review.opendev.org/71869413:22
*** vblando has joined #opendev13:24
*** cmurphy has joined #opendev13:25
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job  https://review.opendev.org/71869413:29
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job  https://review.opendev.org/71869413:34
*** sgw has joined #opendev13:44
openstackgerritThierry Carrez proposed openstack/project-config master: Introduce job for granular GitHub mirroring  https://review.opendev.org/71847913:45
ttxclark, fungi, AJaeger ^ I fixed the comments. Hopefully got the syntax right for release-test13:46
AJaegerttx, let's try it ;)13:47
openstackgerritJames E. Blair proposed opendev/system-config master: Use SafeLoader in irc_checks  https://review.opendev.org/71855813:52
corvusfungi: mind landing this?  https://review.opendev.org/71412613:53
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job  https://review.opendev.org/71869414:00
*** ykarel|afk is now known as ykarel14:02
*** lpetrut has joined #opendev14:19
openstackgerritMerged openstack/project-config master: Extract only series name in release job template  https://review.opendev.org/71868914:24
openstackgerritMerged opendev/system-config master: Add a note about rename files to project renames doc  https://review.opendev.org/71412614:30
clarkbdirk: AJaeger ianw I can push a dib release today. Maybe I'll wait to see if ianw can ack that plan, but I think it is a holiday weekend already in ianw's corner of the world14:32
openstackgerritMerged openstack/project-config master: Introduce job for granular GitHub mirroring  https://review.opendev.org/71847914:47
AJaegerconfig-core, a few small reviews, please: https://review.opendev.org/716630 https://review.opendev.org/717775 https://review.opendev.org/71823214:49
*** ysandeep|rover is now known as ysandeep|afk14:51
clarkbfungi: following up from yesterday is there a key rotation docs update I can review?14:56
openstackgerritPaul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles  https://review.opendev.org/69351314:57
fungiclarkb: there is not but i'll see if i can piece together the earlier bits others did with the bits i did to come up with something14:58
clarkbfungi: ok, I can help too though I'm not really caught up on the expiration side of things14:59
fungithat was a fairly easy bit to do, just a hard bit to discover15:00
fungibind maintains a persistent key database, in combination with the keyfiles15:01
fungiso their entries have to be removed from the database before they're removed from disk15:01
fungiand the simplest way to do that is to update them with an immediate expiration (or one coincident with the ttl of the records you're validating with it at least) and reload those15:02
fungithen when they reach the expiration time bind removes them from its database15:02
fungiafter which you're free to also delete them from disk15:02
openstackgerritMerged openstack/project-config master: Add publish-to-pypi jobs for new Vitrage xstatic-* projects  https://review.opendev.org/71663015:02
clarkbfungi: if you did an expiration matching record ttls can you remove them from disk as soon as the expiration is set or do you have to wait for the expiration to occur?15:04
clarkb(maybe we don't know becaues that wasn'ttested)15:05
fungii think it's probably safest to wait, plus deleting them from disk sooner doesn't help, and deleting them long after their expired (or not at all) is probably also fine15:05
fungis/their/they're/15:06
openstackgerritMerged openstack/project-config master: Retire OSA repo_build and pip_install projects  https://review.opendev.org/71777515:08
openstackgerritMerged openstack/project-config master: Update grafana for ceph to reflect current jobs  https://review.opendev.org/71823215:08
mordredclarkb: yeah - with expiration, it seems that removing from disk is really more of just a cleanup, not as important15:18
mordredclarkb, fungi : https://review.opendev.org/#/c/718559/15:18
clarkbmordred: is new etherpad all sorted at this point?15:23
clarkbwe can do meetpad testing before we do the cut over too15:23
mordredclarkb: yes, should be15:24
mordredclarkb: and yes, I think it's a great idea to point meetpad at it15:25
clarkbI'm updating the config now15:25
fungiawesome!15:26
openstackgerritClark Boylan proposed opendev/system-config master: Switch meetpad to the opendev etherpad  https://review.opendev.org/71873015:27
clarkbmordred: fungi ^15:27
mordredclarkb, fungi: also - https://review.opendev.org/#/c/717056/ and https://review.opendev.org/#/c/717057/ need re-reviews (and everything else there is actually working nicely, so I think we're ready to move on those)15:29
fungii suppose i should also do an /etc/hosts hack on my workstation to make sure the etherpad.oppenstack.org name will still work and get redirected normally when we turn it into a cname15:29
clarkbfungi: ya we should do that before updating dns15:29
mordredi checked the hourly and periodic runs and everything is green now15:29
mordred++15:29
fungii assume the image it's using is etherpad-lite 1.8.015:29
mordredyeah15:30
mordredfungi: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/etherpad/templates/docker-compose.yaml.j2#L2015:31
fungioh, yep, indeed15:31
fungi1.8.1(or .3?) may be up any day now too from the sound of things15:31
mordredcool15:32
mordredshould be an easy update :)15:32
fungiglorious15:32
clarkbmordred: sorry this just occured to me. I think the hiera data in system-config should be in the files list for https://review.opendev.org/#/c/717057/7/.zuul.yaml15:32
mordredclarkb: totally agree - fixing15:32
mordredclarkb: or - why don't I do that as a followup15:33
mordredso I don't have to push the rebased stack15:33
clarkbmordred: k15:33
*** ykarel is now known as ykarel|away15:33
openstackgerritMonty Taylor proposed opendev/system-config master: Trigger remote-puppet-else on hiera changes  https://review.opendev.org/71873315:34
mordredclarkb: thereyago15:34
clarkbmordred: note on https://review.opendev.org/#/c/717058/8 for a cleanup item we shouldn't forget15:41
openstackgerritJan Kundr├ít proposed zuul/zuul-jobs master: docs: Document CORS requirements of Zuul's web dashboard  https://review.opendev.org/71874015:46
mordredclarkb: ++15:49
mordredclarkb: speaking of cleanup: https://review.opendev.org/#/c/717112/ when you get a moment15:51
mordredfungi: ^^15:51
clarkbmordred: strings are so much fun15:53
*** diablo_rojo has joined #opendev15:57
*** rpittau is now known as rpittau|afk15:59
openstackgerritSean McGinnis proposed openstack/project-config master: Release cycle test template file cleanup  https://review.opendev.org/71874315:59
*** dpawlik has quit IRC16:06
fungiclarkb: the extra \ was due to puppet adding an extra level of escaping, right?16:11
clarkbfungi: yes16:11
fungicool16:12
fungiand ansible/jinja2 does not suffer that i guess16:12
openstackgerritMatthew Thode proposed opendev/glean master: write one resolv config  https://review.opendev.org/71733916:13
clarkbfungi: nope, because ansible will replace {{ }} variables in '' or "" quoted strings16:15
fungigood deal16:15
clarkbfungi: with puppet the issue was we had to use "" in order to do var replacement but then that treats \ as an escape16:15
clarkbansible + yaml doens't have that issue as {{ }} is always replaced16:15
fungiright16:15
openstackgerritVladimir Blando proposed openstack/project-config master: Add vexxhost/documentation  https://review.opendev.org/71874716:24
*** lpetrut has quit IRC16:26
clarkbthe logstash queue has remained mostly stable over the last couple days16:27
clarkbits still somewhat highe rthan I would prefer but it isn't growing (that just means we've got a delay in indexing but are keeping up overall)16:27
mordredfungi: I think I'm just going to go remove the cron jobs on review now so we can land the change, yeah?16:28
mordred(we can live without them for a couple of hours)16:28
clarkbthat should be fine16:30
openstackgerritMatthew Thode proposed opendev/glean master: write one resolv config  https://review.opendev.org/71733916:31
openstackgerritMerged opendev/system-config master: Switch meetpad to the opendev etherpad  https://review.opendev.org/71873016:34
fungimordred: sounds good, yep16:38
fungii just didn't want to approve that change until that had been done so they didn't get doubled up16:38
openstackgerritVladimir Blando proposed openstack/project-config master: Add vexxhost/documentation  https://review.opendev.org/71874716:43
*** ysandeep|afk is now known as ysandeep|out16:46
openstackgerritVladimir Blando proposed openstack/project-config master: Add vexxhost/documentation  https://review.opendev.org/71874717:03
*** sshnaidm is now known as sshnaidm|afk17:12
openstackgerritMerged openstack/project-config master: Release cycle test template file cleanup  https://review.opendev.org/71874317:24
openstackgerritMerged opendev/system-config master: Add cron jobs that were managed by puppet  https://review.opendev.org/71711217:28
openstackgerritMatthew Thode proposed opendev/glean master: write one resolv config  https://review.opendev.org/71733917:33
clarkbAJaeger: was there ever a chnage to publish opendev/system-config docs to docs.opendev.org? I don't see it under https://docs.opendev.org/opendev17:48
*** mlavalle has joined #opendev17:52
clarkbopendevians https://etherpad.openstack.org/p/4mEmDBuwoY is a rough draft on governance next steps. I'm presenting that from my viewpoint but want to make sure it is agreeable before sending it more broadly. Let me know what you think17:54
AJaegerclarkb: I don't remember such a change17:57
clarkbAJaeger: thanks I wasn't sure if I had seen that or not. I guess thats a good task for me to figure out17:58
AJaegerclarkb: see what we did for infra-manual. Change the promote job and update openstack-manuals to add redirects17:59
AJaegerclarkb: etherpad LGTM, I added a few ","17:59
clarkbAJaeger: thanks17:59
*** sshnaidm|afk is now known as sshnaidm|off18:02
openstackgerritClark Boylan proposed opendev/system-config master: Publish docs updates to docs.opendev.org  https://review.opendev.org/71876218:08
clarkbAJaeger: ^ is it that simple?18:08
clarkbthat doesn't do the redirects though18:08
openstackgerritMonty Taylor proposed opendev/system-config master: Change Etherpad default intro text  https://review.opendev.org/71876318:10
openstackgerritMonty Taylor proposed opendev/system-config master: Change Etherpad default intro text  https://review.opendev.org/71876318:13
openstackgerritMonty Taylor proposed opendev/system-config master: Update link to etherpad from OpenDev  https://review.opendev.org/71876418:13
clarkbmordred: we should only land that last change after the cut over?18:14
mordredclarkb: yup18:14
*** roman_g has quit IRC18:14
mordredbecause data put into etherpad.opendev right now will be deleted18:14
mordredclarkb: I put a -2 on it18:15
mordredclarkb: I guess we need to do eavesdrop.opendev.org next :)18:15
mordredand paste18:15
clarkball the things18:16
*** melwitt has joined #opendev18:17
AJaegerclarkb: yes, that simple - time to make a template? ;)18:19
mordredcorvus: when you have a sec could I interest you in a quick re-revie of https://review.opendev.org/#/c/717056 and https://review.opendev.org/#/c/71705718:19
*** ralonsoh has quit IRC18:20
clarkbfungi: mordred following up on the tarballs LE cert renewall it seems that we don't have a new warning for that otday so the fixed up ansible from zuul must've corrected that for us18:21
mordredwoot18:22
mordredalso - don't know if you saw my comment from earlier - but I checked and all the ansibles seem to be running properly18:22
mordredso I thnik we're pretty solid now18:22
fungiclarkb: yep, i'll double check just to be sure18:27
fungi-rw-r----- 1 root letsencrypt 1923 Apr  8 16:21 /etc/letsencrypt-certs/tarballs.opendev.org/tarballs.opendev.org.cer18:28
fungiso it updated yesterday18:29
fungilgtm18:29
AJaegerclarkb: want to change the theme for system-config as well?18:30
clarkbAJaeger: that is a good idea18:30
clarkbI'm about to eat lunch but will continue to look at that docs stuff18:31
AJaegerclarkb: enjoy!18:32
fungihave we started publishing system-config to docs.opendev.org already?18:35
fungii guess that's still pending18:35
AJaegerfungi, https://review.opendev.org/718762 - that's what we're discussing18:36
fungiaha, 71876218:36
fungiyep, just took a closer look at scrollback18:36
fungibeen a little scattered today18:36
clarkbfungi: https://etherpad.openstack.org/p/4mEmDBuwoY is related if yo uwant to look at thattoo18:37
fungii'd already pulled it up, just not read yet18:37
fungiclarkb: this may also be a good opportunity for me to switch gears yet again and double-check the ml config for service-discuss to see what we've done on openstack-discuss and others for dmarc mitigation18:42
clarkbfungi: good idea18:43
clarkbthough I can probably do that after lunch if you are busy18:44
clarkbit fits under my todo item of getting that email out :)18:44
fungimeh, there are command-line tools and i can diff, so at least easy to generate the hitlist of what we need to tweak18:45
corvusmordred: +2 on those 2 (looks like it's basically just the ansible forks change)18:49
corvusmordred: i did not +w18:49
openstackgerritDavlet Panech proposed openstack/project-config master: Add kernel to StarlingX  https://review.opendev.org/71877218:58
mordredcorvus: thanks!19:04
*** diablo_rojo has quit IRC19:16
*** diablo_rojo has joined #opendev19:17
*** roman_g has joined #opendev19:26
fungiclarkb: so after diffing the ml configs and discarding the irrelevant differences, i think we need to change the following: clear out subject_prefix and msg_footer, we may also want to turn off bounce_processing and we should probably add a welcome_msg which is just a copy of the info field19:44
clarkbfungi: that sounds reasonable. Are you doing that or should I?19:48
fungii can do it now19:49
fungineed to let my afternoon tea steep a few anyhow19:49
fungichanges applied19:51
*** olaph has joined #opendev19:52
*** olaph has quit IRC19:52
*** olaph has joined #opendev19:53
mordredclarkb: it JUST occurred to me ... we can combine all-clouds.yaml and clouds.yaml on bridge now19:53
mordredclarkb: because we are not using the openstack dynamic inventory anymore (that's why we had them split)19:54
mordredwe can really just put all-clouds in as clouds.yaml and be done with it19:54
*** hashar is now known as idoine19:58
clarkbmordred: oh yup19:58
*** idoine is now known as quibble19:59
fungithat'll be nice19:59
corvusinfra-root: fyi: https://freenode.net/news/scheduled-maintenance-services-database-spring-cleaning   --  i think our nicks are used enough that shouldn't affect us?19:59
clarkband the cloud manager thing was already using all clouds right?19:59
*** quibble is now known as hashar19:59
clarkbcorvus: ya I've only got hte one registered nick and I use it every day:)19:59
corvusand i think our bot nicks are well used too?20:00
fungiought to be20:00
corvus(we don't have any bot nicks we're not using, right?)20:00
funginone that we care about, if we do20:01
openstackgerritMonty Taylor proposed opendev/system-config master: Get rid of all-clouds.yaml  https://review.opendev.org/71878820:01
clarkbopenstackstatus should be active enough but its not feed a constant stream of events/meetings so would be the one I worry about20:03
clarkbbut we #status log frequently enough that shouldn't be an issue20:03
mordredclarkb: yeah - everything we were doing started with "set OS_CLIENT_CONFIG_FILE"20:03
openstackgerritMerged opendev/system-config master: Run AFS in zuul  https://review.opendev.org/71705620:08
openstackgerritMerged opendev/system-config master: Run remote-puppet-else in zuul  https://review.opendev.org/71705720:10
clarkbinfra-root did anyone else want to weigh in on https://etherpad.openstack.org/p/4mEmDBuwoY or should I go ahead and send it to service-discuss?20:13
corvusclarkb: that looks swell20:17
fungioh, yep, read it, lgtm. thanks clarkb!20:19
openstackgerritClark Boylan proposed opendev/system-config master: Switch documentation to alabaster theme  https://review.opendev.org/71879020:23
clarkbthere is the doc retheme20:23
clarkbcool I'll get that email sent out now20:23
openstackgerritMonty Taylor proposed opendev/system-config master: Remove puppet and cron mentions from docs  https://review.opendev.org/71879120:27
mordred#status log All ansible has been migrated to Zuul jobs, nothing is running via run_all.sh any more20:28
openstackstatusmordred: finished logging20:28
clarkb\o/20:30
clarkbfungi: hrm my email hasn't gone through yet, I wonder if we have OOM issues again?20:30
clarkbmaybe I should try and trace that again if so20:31
clarkbwe have an OOM from 11 hours ago20:31
fungiclarkb: oh yep, there were oom issues, i noticed them while comparing configs, but then i restarted the services for all the sites20:32
fungimeant to #status log that but have been very distracted today20:32
clarkbfungi: hrm I wonder why my email hasn't shown up yet20:32
openstackgerritMerged opendev/system-config master: Remove run_all.sh and ansible cron job  https://review.opendev.org/71705820:34
fungii can take a look in a moment20:36
mordredclarkb: I have cleaned out the log rotate script related to run_all now20:43
mordredclarkb: this can go in now: https://review.opendev.org/#/c/71705920:46
mordredalso - corvus fungi https://review.opendev.org/#/c/717059/ - is a thing clarkb noticed earlier today that should add20:47
clarkbmordred: in 717059 we remove the setting of env var PATH for cron. Any idea if that is more global?20:48
*** hashar has quit IRC20:48
clarkb(possible we need to add that back in somewhere for things like the cloud management cron)20:48
clarkbI've approved the change though as it should be fine to fix that up later if we need it20:48
mordredclarkb: not sure - it's a weird task isn't it?20:49
clarkbya20:49
mordredclarkb: I'm going to convert cloud-launcher to a zuul job next, which will get us out of the cron business on bridge anyway20:49
clarkbthat works too20:49
fungi#status log restarted all mailman sites on lists.openstack.org following oom events around 09:00z20:51
openstackstatusfungi: finished logging20:51
fungithis time i confirmed they stopped properly before i started them again20:51
fungithere are 5 IncomingRunner processes, which matches our site count20:52
clarkbmy email went through20:53
fungiit was really my secret ploy to be able to catch up on e-mail ;)20:53
openstackgerritMonty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul  https://review.opendev.org/71879821:07
openstackgerritMonty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron  https://review.opendev.org/71879921:07
openstackgerritMerged opendev/system-config master: Remove ansible-cron role  https://review.opendev.org/71705921:15
openstackgerritMonty Taylor proposed opendev/system-config master: Get rid of all-clouds.yaml  https://review.opendev.org/71878821:44
openstackgerritMonty Taylor proposed opendev/system-config master: Remove puppet and cron mentions from docs  https://review.opendev.org/71879121:44
openstackgerritMonty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul  https://review.opendev.org/71879821:44
openstackgerritMonty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron  https://review.opendev.org/71879921:44
openstackgerritMerged opendev/system-config master: Trigger everything on inventory changes  https://review.opendev.org/71711421:50
openstackgerritMonty Taylor proposed opendev/system-config master: Remove puppet and cron mentions from docs  https://review.opendev.org/71879121:57
openstackgerritMonty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul  https://review.opendev.org/71879821:58
openstackgerritMonty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron  https://review.opendev.org/71879921:58
corvusmordred: heh, it looks like we would have gotten an inline comment from zuul duplicating my comment about the '-' but the job didn't quite finish in time.  but child changes got an invalid file comment (which is fine and expected)22:00
clarkbinfra-root alright I mentioned doing a dib release for dirk this morning. I think ianw is properly out on holiday today so won't be weighing in22:00
clarkbany concerns with doing that now?22:00
clarkbthis will apparnetly fix opensuse builds22:01
mordredcorvus: aw darn22:02
corvusclarkb: other than "i'm a casual dib maintainer unfamiliar with the current state and it's a holiday weekend", no :)22:02
mordredclarkb: I think it's fine - aiui there's one outstanding change22:02
fungiclarkb: yeah, from scrollback (mostly in #openstack-infra) it seems like ianw's hesitation was around wanting to get more proper pip-and-virtualenv element removal in for opensuse instead of releasing with those workarounds merged?22:02
corvusbut, i mean, we released recently, and not much has changed?22:02
mordredand it is "fix opensuse"22:02
clarkbfungi: oh I missed that22:03
clarkbfungi: more than 68bb43535e8f032890687be6a253c3505cb319e5 you mean?22:03
clarkbthere are 3 non merge changes and one is fix opensuse22:03
clarkbmaybe I should wait if there is more outstanding work22:03
corvusmordred: we're ready to do the etherpad switcheroo, right?  do we want to send an email and schedule that for... tomorrow?22:04
corvus(i think we were not worried about the outage, but did want to let people know about it in case of issues, so a 1-day advance notice for a friday maint seems okay to me....)22:04
clarkbcorvus: ya I think that should work.22:04
clarkbits well vetted at this point so should be fine, but warning people in case something comes up is a good idea22:05
openstackgerritMerged opendev/system-config master: Trigger remote-puppet-else on hiera changes  https://review.opendev.org/71873322:06
corvusmordred, clarkb: how about we put it in emergency and shut it down until the switch?  because if we're going to announce it, i don't want folks to start using it and then have their data wiped when we migrate.22:06
mordredcorvus, clarkb : yeah - I thnk doing it tomorrow sound great22:06
clarkbcorvus: that is also probably a good idea22:06
corvusmordred: want me to draft an email?22:06
mordredsure! I'll go put it in emergency and shut it down22:06
corvusmordred: ack, thx22:06
* fungi tests the redirection real quick, hadn't gotten around to that yet22:07
corvusclarkb: this should go to service-announce@lists.opendev -- should we also send it elsewhere since the list is new?22:08
clarkbcorvus: ya I think we still need to send to the other lists, but service-announce would be great too22:08
mordred++22:08
mordreddid we announce a gerrit restart for tomorrow too?22:08
clarkbcorvus: you'll get moderated there if we set that up properly (fungi and I are set up as moderators)22:08
clarkbmordred: I don't think so22:08
mordreddo we want to?22:08
fungiseems to be working on the apache side of things anyway, though leaves me at a "service unavailable" error on etherpad.opendev.org22:08
mordredfungi: I just turned it off22:08
* fungi calls it "good enough"22:09
mordredfungi: want me to turn it back on?22:09
corvuswhat time do we want to do it?  nowish?  2200 utc?  or some other time?22:09
clarkbcorvus: I would say earlier in the day like 1600-2000 UTC22:09
clarkbmordred: ++ for gerrit email too22:09
fungimordred: nah. i was just trying to jump in and test the redirect before you turned it off, but seems i was too slow. no biggie. we can always test later when we're ready to cut over22:09
fungiand i can test apache anyway22:09
corvusmordred, clarkb: k, i'll write an omnibus email for both etherpad and gerrit?22:09
mordredcorvus: yeah - I thnik that's a good idea22:09
clarkbcorvus: ++ thanks22:10
corvuswhen gerrit restart?22:10
corvus(let's not do it at the same time :)22:10
mordredand I figure we can likely do both around the same time - they both shoudl be quick22:10
mordredoh22:10
mordredlet's not do it at the same time then :)22:10
corvusmordred: 1 hour apart okay?22:10
mordred++22:10
clarkbI would say do them both earlier in the day and split them about na hour22:10
clarkbya22:10
* mordred has no plans tomorrow22:10
clarkbyaI can be around all day tomorrow22:10
clarkbno school stuff for kids22:10
fungii *had* no plans tomorrow, so this can be my plans ;)22:10
corvus1600 gerrit; 1700 etherpad?22:11
clarkbcorvus: wfm22:11
mordred++22:11
fungisounds great22:11
corvuswas there a version upgrade for etherpad?22:16
corvusmordred: and how long does the db import take?22:16
clarkbcorvus: yes up to 1.8.022:16
clarkb(most recent release)22:16
fungialso we'll want to keep an eye out for 1.8.1 (or .3, it's not clear from their discussions) any day22:18
corvusclarkb, fungi, mordred: https://etherpad.openstack.org/p/tpdNLYKbYn22:20
mordredcorvus: I was just thinking I should go do another one real quick22:20
mordredlet me go do that and time it22:20
corvusmordred: cool, i'll wait for that before sending (i put 15m in as a placeholder)22:21
corvushave we thought about subscribing the foo-discuss lists to service announce?22:21
clarkbcorvus: lgtm other than updating the placeholder time for db migrations22:22
corvusclarkb: have a list of *-discuss lists handy?22:23
clarkbya I'll add to the etherpad22:24
fungicorvus: lgtm, thanks!22:25
corvusok, there's the headers copy/pasted from my mail buffer :)22:26
openstackgerritOleksandr Kozachenko proposed openstack/project-config master: Add vexxhost/rbac-helm  https://review.opendev.org/71882422:27
fungii'm wary of rebroadcasting service-announce messages on other lists indefinitely. folks who are admins of those lists can of course choose to subscribe them, i suppose22:27
corvusk.  i don't feel strongly about it22:28
fungicrossing the streams makes me a little twitchy, not sure why22:28
corvusseems like it might be an okay way to reach more people and probably within scope of all those lists22:29
mordredso - interestingly enogh - I cannot run mysqldump directly from etherpad.opendev.org22:31
mordredso I'm having to do the dump on etherpad.openstack.org - then we'll have to copy the dump over, then run it in22:31
clarkbmordred: is that because the my.cnf isn't copied over? or network access?22:32
mordredclarkb: mysqldump: Got error: 1045: Access denied for user 'eplite'@'10.208.198.148' (using password: YES) when trying to connect22:32
mordredclarkb: the same creds from etherpad.openstack.org work22:32
clarkboh this sounds familiar actually22:32
clarkbthe ssl issue22:33
clarkbmaybe22:33
fungimysql "localhost" problem?22:33
mordredyeah? cause it would be quicker to do a pipeline dump | mysql22:33
fungihave we been dumping via socket?22:33
mordredit's not that - in neither case are localhost involved22:33
fungiahh22:33
mordrednah - I can dump over tcp from etherpad.openstack.org22:33
clarkbya I want to say we ran into similar and it comes across as auth error but really its like you arne't using ssl but it wants to by default or something22:33
fungioh, right, current etherpad db is in trove22:33
clarkbmordred: ya we set ssl=false in the my.cnf on old server22:34
mordredlemme try that22:34
fungioho22:34
fungigood memory22:35
fungi(and opaque/misleading error)22:35
mordredclarkb: yeah - that did it22:36
mordredlet me restart dump/restore test22:37
clarkbI think I only remember it because the error was so obscure compared to the fix22:37
mordredk. running dump/restore in a pipeline22:38
fungiobviously we can drop that hackaround except for the dump from trove22:38
mordredyeah22:38
fungiyay progress!22:38
mordredftr, I did an exec into the mariadb service on etherpad.opendev.org and am running: mysqldump --single-transaction -h0e39d6e1213c22be004888d63f24ba61822b46fb.rackspaceclouddb.com -ueplite -p etherpad-lite | mysql -p$MYSQL_ROOT_PASSWORD etherpad-lite22:39
mordredwith the password pasted to stdin for the first commant22:39
mordredcommand22:39
mordred(tomorrow, if etherpad is shut down, --single-transaction can be left off because nobody will be using it - but if we left that off today it would lock all the tables :) )22:41
mnasermordred: can i +w https://review.opendev.org/#/c/717136/ ?22:42
mordredmnaser: ++22:44
mnaserok cool, done22:44
mordred\o/22:46
mordredmnaser: look at how magical we are now!@22:46
mnasermordred: im loving the post jobs that run project manages :)22:46
mordred\o/22:47
mordredoh - that reminds me - we should turn log collection back on for manage_projects22:47
mordredwe've vetted that one22:47
mnasermordred: if i can borrow some reviews on https://review.opendev.org/#/c/718824/ and https://review.opendev.org/#/c/718747/ :>22:48
openstackgerritMonty Taylor proposed opendev/system-config master: Collect logs from manage-projects runs  https://review.opendev.org/71882722:48
mordredclarkb: ^^22:48
mordredmnaser: done22:49
mnasermordred: thanks!22:49
* mnaser is trying github mirroring for the first time too22:50
corvusmordred: how's it going?22:51
openstackgerritMerged openstack/project-config master: Run remote-puppet-else when zuul/main.yaml changes  https://review.opendev.org/71713622:53
mordredcorvus: still dumping/restoring22:57
mordredprocess started at Thu Apr  9 22:38:13 UTC 202022:58
mordredso that's about 20 minutes so far22:58
openstackgerritMerged openstack/project-config master: Add vexxhost/rbac-helm  https://review.opendev.org/71882422:59
openstackgerritMerged openstack/project-config master: Add vexxhost/documentation  https://review.opendev.org/71874722:59
mnaseri imported the wrong ssh key (apparently macos uses some new openssh key format that our build nodes don't understand) for upload-git-mirror23:00
mnaserwould someone be able to run? zuul enqueue --tenant vexxhost --pipeline post --project vexxhost/libvirtd_exporter --change 718826 -- i could just land "noop" changes but that seems, eh.23:01
mnaserthe secret was changed in vexxhost/base-jobs so that's why it doesn't really result in another CI run (but hopefully once this works, it just works for everything)23:01
clarkbmnaser: out of curiousity what was the key? because bionic should have a fairly recent openssh23:02
corvusmordred: i'm going to take a break, i'll check back23:02
clarkbmnaser: also post doesn't operate relative to changes. Can you link to the job that ran and failed? We can get the enqueue-ref parameters from there23:03
mnaserclarkb: the header of the key had "-----BEGIN OPENSSH PRIVATE KEY-----"23:03
mnasersome context seems to be here - https://serverfault.com/questions/939909/ssh-keygen-does-not-create-rsa-private-key23:03
mnaserhttp://zuul.opendev.org/t/vexxhost/build/c9f8f3ff93064198a57e7e640abf85a7 this is the exact build that failed23:04
fungiwe had to deal with similar issues on bionic, if memory serves23:06
mnaseri mean, i'm _guessing_ that key probably confused it23:06
mordreddump/restore finished23:06
clarkbmnaser: `sudo zuul enqueue-ref --tenant vexxhost --trigger gerrit --pipeline post --project vexxhost/libvirtd_exporter --ref refs/heads/master --newrev 25ddaad5ba01d702bf3421d6f3a37f41d7ef854d --oldrev 9f73c0f66a396a5faadd71a9fff202dff96c1122` has been run23:06
mordredcorvus: so that was 27 minutes - so probably an even half-hour23:06
mnaserclarkb: thank you, i'll watch and see what happens..23:07
*** tosky has quit IRC23:08
mnaserok cool, we got beyond that -- remote: error: GH006: Protected branch update failed for refs/heads/master.23:10
mnaseri guess i can just get rid of branch protection and manually mirror it for now until the next time we land something23:11
*** diablo_rojo has quit IRC23:11
openstackgerritMonty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul  https://review.opendev.org/71879823:12
openstackgerritMonty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron  https://review.opendev.org/71879923:12
fungimnaser: i suppose the issue was something like https://federicofr.wordpress.com/2019/01/02/how-to-convert-openssh-private-keys-to-rsa-pem/23:12
mnaserfungi: yep, exactly what i ended up doing and it seems to have resolved it23:13
fungimnaser: yeah, seems my ssh keys on debian/sid have been upgraded to the newer format as well23:14
fungibut you're right, bionic is too old to use them directly23:14
fungittx: next time you're about, you may want to make sure whatever platform you generated the github ssh key for openstack on exported it in a format bionic can use ^23:15
mnaseri assume there is a plan of moving to mirroring via post in openstack fungi ?23:16
fungimnaser: yeah, it's related to the thread ttx started about cleaning up the github mirror23:18
fungibasically we can't easily selectively mirror from gerrit to github with the current legacy configuration (gerrit replication plugin) so the idea is to turn that off and replicate via zuul job23:19
fungihowever i think your idea of working out how to make that doable from an executor-only/nodeless job is an important piece of that23:20
corvusmordred: great, thanks!  updating and sending23:25
corvusmnaser, fungi: i think i only have the outline of what you're talking about, but i think either opendev base-jobs or any of the individual tenants with a trusted project could add a job that ran the upload-git-mirror role, then projects could add a secret with pass-to-parent set to true23:33
corvusfungi, clarkb: my service-announce message is moderated23:35
clarkbcorvus: k will get to it momentarily23:36
mnasercorvus: arguably I would even say that we would have that job in base jobs, list allowed projects in there with a var that matches the project name23:36
mnaserThe only issue is controlling if someone decides to override the project and and override another repo23:37
mnaserso many we need to make it in trusted repo only and define those jobs not in repo23:38
corvusmnaser: i don't think allowed-projects is the right answer; instead you just configure the job to only mirror exactly the triggering repo23:38
corvusmnaser: it's like tho doc promote jobs23:38
corvus(you can't promote someone elses docs, but any repo can add the single doc promote job and it runs on the executor)23:38
clarkbcorvus: should be accepted now23:38
clarkband I see a copy now23:39
corvusmnaser: i'm writing up a slightly more detailed proposal as a response to ttx's thread.23:48
corvusmnaser, fungi: how's this?  http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014038.html23:53
fungisorry, back now, looks like clarkb beat me to the moderation queue23:55
fungicorvus: i believe there's already a job proposed, just a sec...23:56
fungicorvus: nope, merged, looks like i approved it earlier today (have i said i'm scattered?) https://review.opendev.org/71847923:57
fungieasy to iterate on though, nothing important is using it yet23:57
mnasercorvus, fungi: I like that approach, nodeless would help a lot with efficiency honestly23:57
fungithe idea was to have one central secret in a trusted config repo so that we don't have to reencrypt the same account credentials over and over for every repo in the openstack namespace23:57
fungichecking out the proposal on the ml now23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!