| clarkb | tonyb[m]: what I did was git remote update ; git checkout master ; git pull origin master ; git remote rm gerrit ; git review -s | 00:00 |
|---|---|---|
| clarkb | that assumes the origin remote is git.openstack.org or similar | 00:00 |
| clarkb | I guess that doesnt update origin either, just gerrit | 00:00 |
| tonyb[m] | clarkb: Okay, I was trying soemthing similar but I'm getting: | 00:01 |
| tonyb[m] | fatal: unable to connect to git.openstack.org: | 00:01 |
| tonyb[m] | git.openstack.org[0: 23.253.125.17]: errno=No route to host | 00:01 |
| tonyb[m] | git.openstack.org[1: 2001:4800:7817:103:be76:4eff:fe04:e3e3]: errno=Network is unreachable | 00:01 |
| clarkb | ah you have git:// remotes? | 00:01 |
| clarkb | git remote -v will confirm | 00:01 |
| tonyb[m] | Yeah | 00:01 |
| clarkb | for that you may just want to git remote rm then git remote add | 00:02 |
| clarkb | git remote rm origin ; git remote add origin https://opendev.org/project/path | 00:02 |
| tonyb[m] | okay | 00:03 |
| tonyb[m] | is the a map for things that changed org? | 00:03 |
| clarkb | We have one on our bastion (maybe we should publish that) | 00:03 |
| clarkb | otherwise you can lookup the old name via https and see where it redirects you | 00:04 |
| tonyb[m] | okay. | 00:04 |
| tonyb[m] | I'll do that | 00:04 |
| clarkb | so s/git/https/ in your remote then see where it redirects tou | 00:04 |
| tonyb[m] | I'll hack something up and drop it on the list | 00:05 |
| paladox | you could do sed -i -e 's/review.openstack.org/review.opendev.org/' .git/config | 00:05 |
| tonyb[m] | paladox: Thanks, but that's only part of the problem | 00:06 |
| fungi | yeah, the .gitreview files are updated in all branches of all repos, but you need to be able to pull them (which won't be possible if you were using git:// remotes) | 00:10 |
| fungi | using sed to convert them to https:// (or http://) first will solve that | 00:11 |
| tonyb[m] | Cool, I was just being lazy and hoping y'all had done the hard work ;P | 00:11 |
| fungi | we definitely did the hard work, this is just what's left for you ;) | 00:11 |
| tonyb[m] | LOL | 00:12 |
| tonyb[m] | fair point ;P | 00:12 |
| fungi | the really lazy but entirely effective alternative is just to delete and re-clone (though that really isn't necessary) | 00:12 |
| *** rascasoft has quit IRC | 00:13 | |
| tonyb[m] | True but being on the wrong side of the wet string that isn't super appealing. | 00:17 |
| tonyb[m] | fungi: it takes me an hour just to update ~24hours changes | 00:17 |
| fungi | i hear you | 00:18 |
| tonyb[m] | fungi: I suspect fresh clones would be much longer ;P | 00:18 |
| fungi | indeed | 00:18 |
| *** whoami-rajat has quit IRC | 00:22 | |
| *** zhurong has quit IRC | 00:32 | |
| *** yamamoto has joined #openstack-infra | 00:33 | |
| *** yamamoto has quit IRC | 00:36 | |
| *** yamamoto has joined #openstack-infra | 00:36 | |
| *** rascasoft has joined #openstack-infra | 00:40 | |
| *** yamamoto has quit IRC | 00:47 | |
| *** rascasoft has quit IRC | 00:52 | |
| *** yamamoto has joined #openstack-infra | 00:52 | |
| *** jamesmcarthur has joined #openstack-infra | 01:06 | |
| *** d34dh0r53 has quit IRC | 01:09 | |
| *** eglute has quit IRC | 01:09 | |
| *** cloudnull has quit IRC | 01:09 | |
| jamesmcarthur | :mordred | 01:09 |
| *** Goneri has joined #openstack-infra | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:16 | |
| *** rascasoft has joined #openstack-infra | 01:18 | |
| prometheanfire | do we have a reason rechecks aren't working for the reqs project? (see https://review.openstack.org/653887 ) | 01:21 |
| clarkb | I dont see a recheck comment on that change | 01:22 |
| *** Goneri has quit IRC | 01:22 | |
| clarkb | prometheanfire: I left a recheck comment and it is in the check queue now | 01:24 |
| paladox | clarkb that's another thing i added to the ui :) (a recheck button) | 01:30 |
| paladox | so now all the user has to do is click the button and watch zuul-status light up. (automatically gets the data without refreshing the page) | 01:31 |
| clarkb | nice | 01:31 |
| prometheanfire | ok | 01:34 |
| prometheanfire | maybe my gertty is sad | 01:34 |
| prometheanfire | oh, migration break gertty? | 01:34 |
| *** rascasoft has quit IRC | 01:35 | |
| clarkb | Im not currently a gertty user so not sure. I believe corvus got it working though | 01:38 |
| *** Goneri has joined #openstack-infra | 01:39 | |
| *** jamesmcarthur has joined #openstack-infra | 01:42 | |
| prometheanfire | ya, doesn't look like there are fields in the DB for it, just after the domain, seems like it's hardcoded | 01:42 |
| *** rascasoft has joined #openstack-infra | 01:43 | |
| prometheanfire | last release july 30 2017, and that's the version I have | 01:43 |
| prometheanfire | probably about due :P | 01:44 |
| *** bhavikdbavishi has joined #openstack-infra | 01:47 | |
| *** jamesmcarthur has quit IRC | 01:54 | |
| prometheanfire | it also is still py2 only | 01:55 |
| *** ykarel|away has joined #openstack-infra | 01:56 | |
| *** ykarel|away is now known as ykarel | 01:56 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 01:59 | |
| dtroyer | FIWI I threw together a little git remote updater, tuned to stx so it doesn't get all of the namespace changes, like to x/: https://raw.githubusercontent.com/starlingx-staging/tools-contrib/master/misc/stx-remote-fix.sh. | 01:59 |
| *** bhavikdbavishi has quit IRC | 01:59 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 01:59 | |
| prometheanfire | cool | 01:59 |
| *** rascasoft has quit IRC | 02:01 | |
| prometheanfire | ya, installed git master gertty, guess those changes need updating | 02:05 |
| *** Goneri has quit IRC | 02:05 | |
| *** rascasoft has joined #openstack-infra | 02:08 | |
| *** yamamoto has quit IRC | 02:20 | |
| *** yamamoto has joined #openstack-infra | 02:25 | |
| *** rascasoft has quit IRC | 02:29 | |
| prometheanfire | tonyb[m]: you use that script you just linked in the ML ( opendev-git-update ) to migrate openstack / openstack-dev / openstack-infra? | 02:29 |
| *** Goneri has joined #openstack-infra | 02:31 | |
| *** cloudnull has joined #openstack-infra | 02:43 | |
| *** eglute has joined #openstack-infra | 02:44 | |
| tonyb[m] | prometheanfire: Yup | 02:45 |
| *** d34dh0r53 has joined #openstack-infra | 02:46 | |
| tonyb[m] | prometheanfire: more specifically from git:// based orgins to new https://opendev based origins | 02:46 |
| tonyb[m] | prometheanfire: I'm still running it but that's the idea certainly ;P | 02:46 |
| prometheanfire | right | 02:49 |
| prometheanfire | tonyb[m]: let me know how it goes, I'm away this week but still need to migrate | 02:49 |
| tonyb[m] | prometheanfire: I'll update the list when I'm done ;P | 02:51 |
| tonyb[m] | prometheanfire: I'll try to ping you also | 02:51 |
| prometheanfire | :D | 02:51 |
| prometheanfire | thanks | 02:51 |
| ykarel | hmm we also fixed our third party playbooks to switch git:// to https:// in RDO | 02:54 |
| prometheanfire | for me it's just local | 02:55 |
| *** yamamoto has quit IRC | 03:00 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add ansible-lint job https://review.opendev.org/532083 | 03:02 |
| *** yamamoto has joined #openstack-infra | 03:02 | |
| *** ramishra has joined #openstack-infra | 03:15 | |
| ykarel | Looks like gitweb is still not working | 03:21 |
| ykarel | example https://review.opendev.org/#/c/651904/ | 03:22 |
| ykarel | gitweb url not working | 03:22 |
| ykarel | returns 404 | 03:22 |
| *** psachin has joined #openstack-infra | 03:22 | |
| ykarel | infra-root ^^ | 03:22 |
| mordred | ykarel: good catch. the cause is that we're not currently replicating refs/changes due to a gitea bug. we'll clearly need to sort out a solution and I'm not 100% sure which thing will be the right choice right now | 03:32 |
| ykarel | mordred, ack, Thanks for the info, so can you confirm https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-fedora-28-master-containers-build-push/81f1910/job-output.txt.gz#_2019-04-22_03_05_56_660514 also related to ^^ | 03:35 |
| clarkb | mordred: I think we might just enable the embedded gitweb? | 03:36 |
| clarkb | mordred: that seems simple for now | 03:36 |
| mordred | clarkb: yeah | 03:36 |
| ykarel | hmm i think we need to use review.opendev.org | 03:37 |
| mordred | ykarel: yes - fetching refs/chages from opendev.org/openstack/kolla is not going to work until we can fix the gitea bug | 03:38 |
| mordred | or someone else can | 03:38 |
| paladox | What is the bug? | 03:39 |
| clarkb | if using zuul zuul should provide all the refs? | 03:39 |
| clarkb | I thought rdo used software factory zuul | 03:39 |
| clarkb | (and zuul gets refs from gerrit) | 03:39 |
| mordred | paladox: the bug is - if we push refs/changes/* to gitea, it gets grumpy | 03:40 |
| paladox | Oh, you mean like github? | 03:41 |
| ykarel | mordred, Okk Thanks for the bug, any link to that gitea bug? so i can link it in our fix | 03:42 |
| ykarel | to use review.opendev.org instead of opendev.org | 03:42 |
| mordred | ykarel: no - we haven't filed one yet, because we're not sure what to file | 03:42 |
| ykarel | mordred, ack | 03:42 |
| openstackgerrit | Ghanshyam Mann proposed openstack/hacking master: Dropping the py35 testing https://review.opendev.org/654287 | 03:42 |
| mordred | paladox: pushing refs/changes/* to github actually works | 03:44 |
| paladox | I didn’t with NoteDB last year :) | 03:44 |
| paladox | *it | 03:44 |
| paladox | One of the gerrit maintainers has to file a bug against GitHub issue tracker to get that fixed | 03:45 |
| mordred | ah - awesome. we haven't tried with newer gerrit / notedb - but we currently replicate refs/changes/ and refs/notes there | 03:45 |
| mordred | paladox: *excellent* | 03:45 |
| paladox | (We hit that at the wmf too) | 03:45 |
| paladox | It’s now fixed :) | 03:45 |
| mordred | well - luckily that's less important to us overall :) | 03:45 |
| mordred | ah - cool! | 03:45 |
| mordred | hopefully it's not too hard to fix in gitea | 03:45 |
| paladox | To replicate mediawiki/core refs/changes/ to GitHub required a script :P | 03:46 |
| paladox | To work around the bug | 03:46 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-performance-tools master: Dropping the py35 testing https://review.opendev.org/654289 | 03:49 |
| tonyb[m] | I have git notes enabled on a few repos, it looks like those refs aren't available on opendev is that a variation of the same issue with refs/changes? | 03:50 |
| paladox | mordred: https://github.com/isaacs/github/issues/1253 | 03:51 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr master: Dropping the py35 testing https://review.opendev.org/654290 | 03:55 |
| *** udesale has joined #openstack-infra | 03:56 | |
| clarkb | tonyb[m]: yes I think so | 04:03 |
| *** hwoarang has quit IRC | 04:10 | |
| *** bobh has quit IRC | 04:10 | |
| *** bobh has joined #openstack-infra | 04:12 | |
| *** hwoarang has joined #openstack-infra | 04:15 | |
| *** bobh has quit IRC | 04:16 | |
| tonyb[m] | clarkb: Thanks. I'll just avoid them then | 04:32 |
| *** ricolin has joined #openstack-infra | 04:38 | |
| *** Goneri has quit IRC | 04:42 | |
| *** tonyb has joined #openstack-infra | 04:46 | |
| *** zhurong has joined #openstack-infra | 04:52 | |
| *** sajauddin has joined #openstack-infra | 05:08 | |
| *** ykarel is now known as ykarel|afk | 05:57 | |
| *** ykarel|afk has quit IRC | 06:01 | |
| *** jbadiapa has joined #openstack-infra | 06:04 | |
| openstackgerrit | zhongshengping proposed openstack/os-performance-tools master: Update python 3 unit test job https://review.opendev.org/637759 | 06:06 |
| *** quiquell has joined #openstack-infra | 06:09 | |
| *** quiquell is now known as quiquell|rover | 06:09 | |
| *** tonyb has quit IRC | 06:13 | |
| *** tonyb has joined #openstack-infra | 06:14 | |
| *** ykarel|afk has joined #openstack-infra | 06:16 | |
| AJaeger | infra-root, https://opendev.org/openstack/openstack is still broken - even after my changes merged ;( | 06:18 |
| *** ykarel_ has joined #openstack-infra | 06:19 | |
| *** ykarel_ is now known as ykarel | 06:19 | |
| *** amansi26 has joined #openstack-infra | 06:21 | |
| *** ykarel|afk has quit IRC | 06:21 | |
| *** yamamoto has quit IRC | 06:24 | |
| AJaeger | is gitea confused with some of the renames? | 06:26 |
| *** yamamoto has joined #openstack-infra | 06:30 | |
| amansi26 | I am facing this issue http://paste.openstack.org/show/749582/ . Can someone help in resolving this? | 06:31 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 06:33 |
| AJaeger | infra-root, do we need a "git submodule update" or something like that on gitea for openstack/openstack? | 06:35 |
| AJaeger | amansi26: looks like a tempest issue, better ask on #openstack-qa | 06:37 |
| *** quiquell|rover is now known as quique|rover|brb | 06:46 | |
| *** ccamacho has joined #openstack-infra | 06:58 | |
| *** sshnaidm|off is now known as sshnaidm | 07:00 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:02 | |
| *** ralonsoh has joined #openstack-infra | 07:16 | |
| *** yolanda_ has joined #openstack-infra | 07:18 | |
| *** pcaruana has joined #openstack-infra | 07:19 | |
| *** ykarel is now known as ykarel|lunch | 07:23 | |
| *** slaweq has joined #openstack-infra | 07:30 | |
| *** dikonoor has quit IRC | 07:31 | |
| *** amansi26 has quit IRC | 07:36 | |
| *** yolanda_ has quit IRC | 07:52 | |
| *** bobh has joined #openstack-infra | 08:01 | |
| *** bobh has quit IRC | 08:06 | |
| *** kjackal has joined #openstack-infra | 08:09 | |
| *** ykarel|lunch is now known as ykarel | 08:19 | |
| *** tkajinam has quit IRC | 08:24 | |
| *** dciabrin__ has quit IRC | 08:34 | |
| *** rossella_s has joined #openstack-infra | 08:40 | |
| *** yikun has joined #openstack-infra | 08:41 | |
| *** pcaruana has quit IRC | 08:45 | |
| *** dave-mccowan has joined #openstack-infra | 08:45 | |
| *** zxiiro-away is now known as zxiiro | 08:48 | |
| *** dave-mccowan has quit IRC | 08:53 | |
| *** rossella_s has quit IRC | 08:53 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr master: Deprecate ostestr command https://review.opendev.org/573636 | 09:02 |
| *** rcernin has joined #openstack-infra | 09:03 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr master: Deprecate ostestr command https://review.opendev.org/573636 | 09:04 |
| *** slaweq has quit IRC | 09:17 | |
| *** rossella_s has joined #openstack-infra | 09:20 | |
| *** slaweq has joined #openstack-infra | 09:35 | |
| *** yamamoto has quit IRC | 09:36 | |
| *** yamamoto has joined #openstack-infra | 09:36 | |
| *** slaweq has quit IRC | 09:40 | |
| *** e0ne has joined #openstack-infra | 09:42 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** rossella_s has quit IRC | 09:52 | |
| *** xinranwang has joined #openstack-infra | 09:53 | |
| xinranwang | Hi guys, i can not pull the lastest code to local after we change to OpenDev. Is there any configuration I need to do? | 09:54 |
| *** hwoarang has quit IRC | 10:03 | |
| *** hwoarang has joined #openstack-infra | 10:04 | |
| xinranwang | The error is "gnutls_handshake() failed: The TLS connection was non-properly terminated." | 10:17 |
| *** quiquell|rover is now known as quiquell|ruck | 10:21 | |
| AJaeger | xinranwang: it works here. Do you have any firewalls in between that need to know about opendev.org? | 10:22 |
| AJaeger | infra-root, regarding openstack/openstack, I see that github.com is also confused - last update for project-config was three days ago. Did the org move confuse github and gitea? | 10:24 |
| *** yamamoto has joined #openstack-infra | 10:27 | |
| xinranwang | AJaeger: thanks. I was working behind the proxy, I switched to another network, and it works :) | 10:28 |
| *** yamamoto has quit IRC | 10:32 | |
| AJaeger | xinranwang: glad to hear that - so work with your proxy folks... | 10:34 |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** yamamoto has quit IRC | 10:57 | |
| *** pcaruana has joined #openstack-infra | 10:58 | |
| mordred | AJaeger: github/openstack/openstack seems ok | 11:01 |
| mordred | but I agree, gitea is still borked | 11:01 |
| *** yamamoto has joined #openstack-infra | 11:08 | |
| *** yamamoto has quit IRC | 11:10 | |
| *** yamamoto has joined #openstack-infra | 11:11 | |
| *** yamamoto has quit IRC | 11:12 | |
| *** Lucas_Gray has joined #openstack-infra | 11:14 | |
| mordred | 2019/04/22 11:17:21 [...routers/repo/view.go:55 renderDirectory()] [E] GetCommitsInfo: signal: segmentation fault (core dumped) | 11:18 |
| *** Lucas_Gray has quit IRC | 11:21 | |
| *** yamamoto has joined #openstack-infra | 11:22 | |
| dtroyer | Are there plans for the github mirror repos that moved namespaces and not transferred to other github organizations? ie, push empty contents and archive? | 11:24 |
| mordred | dtroyer: probably so for projects that didn't transfer to other github orgs and are not interested in mirroring to a github org - but there's also dmsimard's mirror to github job for people who want to do that - so I think we're waiting a bit for the owners of the repos to communicate their intent | 11:28 |
| mordred | like, if the new-org repos _do_ want to replicate using the job, we can probably go in and do some org transfers | 11:29 |
| *** udesale has quit IRC | 11:34 | |
| *** slaweq has joined #openstack-infra | 11:43 | |
| *** yamamoto has quit IRC | 11:44 | |
| *** yamamoto has joined #openstack-infra | 11:45 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename cgit_file to git_file https://review.opendev.org/654055 | 11:47 |
| *** lpetrut has joined #openstack-infra | 11:47 | |
| *** slaweq has quit IRC | 11:47 | |
| *** ykarel is now known as ykarel|afk | 11:51 | |
| *** rh-jelabarre has joined #openstack-infra | 12:03 | |
| *** rlandy has joined #openstack-infra | 12:06 | |
| *** rlandy is now known as rlandy|ruck | 12:06 | |
| *** yamamoto has quit IRC | 12:10 | |
| *** quiquell|ruck is now known as quiquell|rover | 12:12 | |
| *** edmondsw_ has joined #openstack-infra | 12:14 | |
| *** yamamoto has joined #openstack-infra | 12:19 | |
| AJaeger | mordred: thanks for checking | 12:20 |
| *** cmurpheus is now known as cmurphy | 12:21 | |
| *** quiquell|rover is now known as quique|rover|eat | 12:22 | |
| *** kgiusti has joined #openstack-infra | 12:26 | |
| openstackgerrit | boden proposed openstack/project-config master: update vmware-nsx related jobs https://review.opendev.org/652680 | 12:29 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove reviewstats unit test job config https://review.opendev.org/653150 | 12:33 |
| *** bhavikdbavishi1 has joined #openstack-infra | 12:33 | |
| *** bhavikdbavishi has quit IRC | 12:34 | |
| AJaeger | config-core, could you review https://review.opendev.org/653150 and https://review.opendev.org/652680 , please? | 12:34 |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 12:34 | |
| AJaeger | as well as https://review.opendev.org/#/c/654241/ | 12:34 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 12:36 | |
| *** yamamoto has quit IRC | 12:37 | |
| *** sajauddin has quit IRC | 12:38 | |
| *** _erlon_ has joined #openstack-infra | 12:41 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 12:42 | |
| lpetrut | Hi, I have some questions about the opendev transition, is there anyone I could talk to? | 12:42 |
| *** jamesmcarthur has joined #openstack-infra | 12:42 | |
| openstackgerrit | Merged openstack/project-config master: Remove reviewstats unit test job config https://review.opendev.org/653150 | 12:45 |
| mordred | lpetrut: all of the relevant people will be in here at various times - it's best to just ask away | 12:45 |
| openstackgerrit | Merged openstack/project-config master: update vmware-nsx related jobs https://review.opendev.org/652680 | 12:47 |
| *** bobh has joined #openstack-infra | 12:48 | |
| *** mriedem has joined #openstack-infra | 12:50 | |
| lpetrut | I work on the compute-hyperv Nova driver, which is out of tree. I've noticed that the github repo is no longer mirrored after the opendev transition: https://github.com/openstack/compute-hyperv | 12:51 |
| lpetrut | also, I saw that compute-hyperv is now under the "x" organization | 12:51 |
| lpetrut | I assume that's because compute-hyperv was not included in the governance projects.yaml file, thus not considered an official project | 12:51 |
| sshnaidm | cores, can you please check that it's configure correct: https://review.opendev.org/654447 ? <- clarkb mordred pabelanger | 12:52 |
| *** quique|rover|eat is now known as quiquell|rover | 12:52 | |
| *** bobh has quit IRC | 12:53 | |
| *** aaronsheffield has joined #openstack-infra | 12:53 | |
| mordred | lpetrut: yes - that's right. there is a zuul job that can be used to replicate to github if that's a thing you desire (although obviously it's not a requirement) - unofficial projects that didn't get claimed before the transition got put into the x/ namespace. I'm not sure what the overall plan is regarding renaming those to a new namespace for people who would want a new namepsace | 12:55 |
| lpetrut | I'd say the IBM PowerVM driver is in a really similar situation, having both an in-tree driver and an out of tree driver, yet the out of tree driver is marked as a deliverable, inc: https://github.com/openstack/governance/blob/61886e29771460bb1de7797a5d22a6f590521191/reference/projects.yaml#L2945-L2949 | 12:56 |
| lpetrut | so, we intend to have the compute-hyperv driver recognized as an official project as well. is there anything else required from our side, other than proposing a patch against the governance project? | 12:57 |
| mordred | no, that should be it - and then we can work through the necessary followup steps | 12:57 |
| lpetrut | that's great, thanks | 12:58 |
| mordred | sure thing! | 13:00 |
| *** ykarel|afk is now known as ykarel | 13:01 | |
| *** rf0lc0 is now known as rfolco | 13:02 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:08 | |
| openstackgerrit | Matt Riedemann proposed openstack/openstack-zuul-jobs master: Drop legacy-grenade-dsvm-neutron-multinode-live-migration https://review.opendev.org/640209 | 13:08 |
| openstackgerrit | Merged zuul/zuul-preview master: Use splice instead of erase/push_front https://review.opendev.org/652148 | 13:09 |
| *** yamamoto has joined #openstack-infra | 13:11 | |
| *** jamesmcarthur has quit IRC | 13:14 | |
| fungi | dtroyer: what to do with repositories still in the openstack namespace on github is more of a question for the openstack tc now, or whoever they want to delegate curation of that org to. in the near term we might want to wait a bit for the dust to settle. longer term my suggestion is to use the "archive" feature i've been told github has, so that if the maintainers of one of those repos wants to have | 13:17 |
| fungi | it transferred to a different org we can unarchive it and do that | 13:17 |
| *** sthussey has joined #openstack-infra | 13:18 | |
| mriedem | clarkb: when you're about, i've noticed that the e-r graph has "Unable to get launchpad data" for all of the bugs, which looks like it's a result of a request exception to launchpad - i don't hit it locally, but it should dump a trace in case the graph generation is logged | 13:18 |
| *** rpioso|afk is now known as rpioso | 13:25 | |
| dtroyer | fungi: thanks. Would it be possible to set the description line to the new location and indicate it is a stale mirror while the rest is sorted? | 13:25 |
| fungi | dtroyer: it might be. is that urgent? | 13:28 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Use pyyaml safe_load to avoid deprecation warning https://review.opendev.org/654454 | 13:28 |
| fungi | dtroyer: if it's urgent that something is done about stale github repos, i say we just hurry up and archive them or something rather than just making more work for ourselves | 13:29 |
| *** psachin has quit IRC | 13:39 | |
| dtroyer | fungi: no, not urgent, I was just asked to find out | 13:40 |
| *** smarcet has joined #openstack-infra | 13:40 | |
| dtroyer | I would be OK with archiving them now. I think this is another example situation of where many people only notice things when they break... | 13:41 |
| fungi | yes, pretty sure we all figured there would be vast swathes of folks who don't pay attention to the myriad of places we discussed/announced these changes in the service | 13:44 |
| fungi | first of many such changes, but this was the hardest to engineer and has the most expected lingering impact on folks | 13:46 |
| fungi | so figured it was the best place to start... it's all down hill from here | 13:46 |
| *** eharney has joined #openstack-infra | 13:49 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire oslosphinx https://review.opendev.org/649072 | 13:49 |
| cmurphy | mordred: i didn't catch up on scrollback, is the beaker job still broken? | 13:50 |
| *** kranthikirang has joined #openstack-infra | 13:52 | |
| *** eernst has quit IRC | 13:56 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 13:58 | |
| *** kranthikirang1 has joined #openstack-infra | 13:59 | |
| *** xinranwang has quit IRC | 14:02 | |
| *** lseki has joined #openstack-infra | 14:02 | |
| *** kranthikirang has quit IRC | 14:03 | |
| *** jamesmcarthur has joined #openstack-infra | 14:03 | |
| *** michael-beaver has joined #openstack-infra | 14:04 | |
| *** eernst has joined #openstack-infra | 14:08 | |
| smcginnis | https://media.giphy.com/media/FwpecpDvcu7vO/giphy.gif | 14:10 |
| *** amansi26 has joined #openstack-infra | 14:11 | |
| *** rfarr has joined #openstack-infra | 14:12 | |
| clarkb | cmurphy: I believe all system-config tests that vote are expected to work now | 14:13 |
| fungi | professor bunsen honeydew must have fixed him back up | 14:14 |
| cmurphy | clarkb: okay great | 14:17 |
| clarkb | infra-root When properly at a keyboard I expect I'll look at mriedem's e-r issue then work to fix ansipuppet on those hosts we noticed saturday just to be sure there arent rename related errors hiding behind the current errors | 14:18 |
| openstackgerrit | David Shrewsbury proposed opendev/puppet-nodepool master: Install yappi package https://review.opendev.org/654462 | 14:19 |
| openstackgerrit | Merged openstack/project-config master: Retire oslosphinx https://review.opendev.org/649072 | 14:19 |
| fungi | i also have a few to do items stacked up if anybody wants one... renaming projects on storyboard-dev to get puppet happy again, writing a change to reduce the db backup retention on paste.o.o from 30 days to something like 10 so that we don't fill up its rootfs in the middle of the summit, and running the fixup playbook to correct storyboard urls in gitea issues links in the wake of repository moves | 14:21 |
| clarkb | fungi: I can do the paste db change aspartof getting ansipuppet happy | 14:23 |
| clarkb | corvus: mordred we also need updates to our rename playbook based on fixes we did friday right? | 14:23 |
| *** Lucas_Gray has joined #openstack-infra | 14:26 | |
| *** mjblack has left #openstack-infra | 14:27 | |
| *** openstackgerrit has quit IRC | 14:28 | |
| *** bobh has joined #openstack-infra | 14:40 | |
| *** quiquell|rover is now known as quiquell|off | 14:42 | |
| *** openstackgerrit has joined #openstack-infra | 14:42 | |
| openstackgerrit | Drew Walters proposed openstack/project-config master: Update Airship projects for gerritbot https://review.opendev.org/654470 | 14:42 |
| AJaeger | Here's an OpenDev rename change for infra-manual. https://review.opendev.org/654222 could somebody queue that on their review queue, please? | 14:44 |
| *** yamamoto has quit IRC | 14:46 | |
| clarkb | AJaeger: looking now | 14:46 |
| AJaeger | thanks | 14:46 |
| fungi | can you make sure someone adds it to https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames ? | 14:47 |
| *** yamamoto has joined #openstack-infra | 14:47 | |
| *** yamamoto has quit IRC | 14:48 | |
| fungi | oh, nevermind, i misunderstood what you were saying | 14:48 |
| prometheanfire | so, who has gertty working after the migration? | 14:52 |
| fungi | i do | 14:52 |
| fungi | running master branch tip under python 3 | 14:53 |
| fungi | it should notice when projects disappear/reappear on restart | 14:53 |
| fungi | you may need to go into the full project list and subscribe to any which were renamed | 14:53 |
| prometheanfire | hmm, it seems like it's having probems for me, running master under py27 (setup.cfg doesn't state py3 :) | 14:54 |
| fungi | oh, and i updated the ~/.gertty.yaml for the new hostname | 14:54 |
| *** jcoufal has joined #openstack-infra | 14:54 | |
| prometheanfire | ah, that'd do it | 14:54 |
| prometheanfire | review.opendev.org ? | 14:54 |
| fungi | yep | 14:54 |
| prometheanfire | looks like it's working :D | 14:55 |
| prometheanfire | I checked the DB, but not the config | 14:55 |
| fungi | though in my case i took it as an opportunity to clear out a lot of cruft... blew away my ~/.gertty.db and the existing git cache, fired it up, went through the full list of projects subscribing to the ones i normally follow, and then letting it sync overnight | 14:55 |
| *** rosmaita has joined #openstack-infra | 14:56 | |
| fungi | but i run gertty in a somewhat resource-constrained vm where it regularly balloons in memory usage to the point where i need to restart it before it begins thrashing swap hard and/or the oom killer gets it | 14:56 |
| prometheanfire | heh | 14:57 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-lodgeit master: Reduce paste db local backup retention to 10 days https://review.opendev.org/654474 | 14:57 |
| prometheanfire | I'll end up using tony's remote updating script | 14:57 |
| clarkb | fungi: ^ fyi | 14:57 |
| openstackgerrit | Merged openstack/infra-manual master: Update for OpenDev Gerrit changes https://review.opendev.org/654222 | 14:58 |
| *** josephrsandoval has joined #openstack-infra | 14:58 | |
| fungi | prometheanfire: vm qith 1gb ram and 2gb swap and a handful of other long-term processes running... since cleaning it up i seem to be running much leaner so i think that may have helped (it's been going for a day and is still only using half as much memory as my irc client) | 14:58 |
| fungi | 97036res, 524260virt | 14:59 |
| prometheanfire | ya... | 14:59 |
| prometheanfire | running on the local laptop works | 15:00 |
| *** lpetrut has quit IRC | 15:00 | |
| clarkb | on pbx I am gzipping the asterisk logfiles messages.2 .3 and .4 | 15:00 |
| clarkb | that should give us enough space to have ansipuppet run successfully and then figureo ut proper logrotation | 15:02 |
| *** amansi26 has quit IRC | 15:03 | |
| *** dklyle has joined #openstack-infra | 15:03 | |
| *** diablo_rojo has joined #openstack-infra | 15:05 | |
| *** ykarel is now known as ykarel|away | 15:06 | |
| clarkb | mriedem: the e-r issue is realted to the rename. e-r runs on an older host and isn't doing sni with ssl so it fails the ssl check saying review.openstack.org doesn't match review.opendev.org | 15:08 |
| clarkb | mriedem: I think the fix there is just going to be updating the config to point at the new locations. I'll get a traceback up as soon as they stop scrolling paste | 15:08 |
| clarkb | mriedem: http://paste.openstack.org/show/749601/ | 15:09 |
| AJaeger | config-core, the airship irc notifications are broken due to the OpenDev change - could you review https://review.opendev.org/654470 , please? | 15:10 |
| *** gyee has joined #openstack-infra | 15:11 | |
| *** bhavikdbavishi has quit IRC | 15:11 | |
| clarkb | looks like fungi got ^ | 15:13 |
| clarkb | for asterisk logging we have an /etc/asterisk/logger.conf that says rotate and gzip but I don't see the gzip happening and the rotate seems to happen via logrotate instead | 15:13 |
| clarkb | I wonder if xenial asterisk doesn't load the logger.conf and instead relies on logrotate which rotates weekly without compression | 15:14 |
| AJaeger | yeah, thanks, fungi and clarkb | 15:15 |
| clarkb | pabelanger: ^ any idea if asterisk logging behavior around rotation and config files has chagned? | 15:15 |
| Shrews | corvus: i cannot find where objgraph is installed on the zuul hosts (though i did find yappi install in puppet-zuul). It *is* installed though. Maybe this was installed by hand and we just forgot to automate it? Or maybe I'm just blind. | 15:16 |
| corvus | Shrews: quite possibly by hand since things were on fire at the time | 15:17 |
| *** imacdonn has joined #openstack-infra | 15:17 | |
| Shrews | corvus: ack, just wanted to confirm | 15:18 |
| mriedem | clarkb: ack. i wonder why generating the graph didn't fail for me locally since i didn't change default config. | 15:20 |
| openstackgerrit | David Shrewsbury proposed opendev/puppet-nodepool master: Install yappi and objgraph packages https://review.opendev.org/654462 | 15:20 |
| mriedem | oh i'm not using ssl either | 15:21 |
| *** boden has joined #openstack-infra | 15:21 | |
| openstackgerrit | Clark Boylan proposed opendev/puppet-reviewday master: Use opendev git url https://review.opendev.org/654482 | 15:21 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Include zookeeper.hosts in ZFS example zuul.conf https://review.opendev.org/654483 | 15:22 |
| mriedem | clarkb: i'll push a change | 15:22 |
| boden | hi, has anyone run into the following error since the migration to opendev SystemError: b"fatal: not a git repository: '/home/zuul/src/git.openstack.org/openstack/requirements/.git'\n" | 15:22 |
| boden | for example: http://logs.openstack.org/09/653009/5/check/requirements-check/cb0e2ad/job-output.txt.gz#_2019-04-22_15_02_24_958181 | 15:22 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-bugdaystats master: Use opendev url for bugdaystats source https://review.opendev.org/654484 | 15:22 |
| clarkb | mriedem: k | 15:22 |
| clarkb | infra-root ^ fyi debugging 654483 and 654484 I've found that puppet vcsrepo will not update an existing origin | 15:23 |
| clarkb | those two vcsrepos are currently pointing at git:// urls depsite being updated to https at some point. My plan there is to get these two changes in then I'll delete the source dirs on the host and have puppet recloen them updating the remotes | 15:23 |
| openstackgerrit | Merged openstack/project-config master: Update Airship projects for gerritbot https://review.opendev.org/654470 | 15:24 |
| *** yamamoto has joined #openstack-infra | 15:24 | |
| clarkb | boden: then path is actually /home/zuul/src/opendev.org/openstack/requirements.git now. We'll have to look at that job and see what needs fixing | 15:24 |
| boden | clarkb should that path be used for upper constraints in tox.ini files? | 15:24 |
| boden | e.g. should I be updating my tox.ini files for the new url | 15:25 |
| AJaeger | mordred: do you know why your system-config change for gitea is failing? I think we need this one, don't we? https://review.opendev.org/654034 | 15:25 |
| clarkb | boden: we might have to update the requirements check script. http://logs.openstack.org/09/653009/5/check/requirements-check/cb0e2ad/ara-report/file/2483cec0-d822-404e-8948-37139f6dbc28/#line-3 the python script there | 15:26 |
| AJaeger | boden: the redirects will work. So, you can but don't have to. | 15:26 |
| clarkb | boden: https://opendev.org/openstack/requirements/src/branch/master/playbooks/files/project-requirements-change.py#L44-L45 that specifically | 15:26 |
| boden | clarkb AJaeger okay thanks... guess I'll wait for a fix to that script | 15:26 |
| clarkb | boden: do you want to write the change to fix ^ | 15:26 |
| clarkb | I think you just have to replace git.openstack.org with opendev.org in that path | 15:27 |
| AJaeger | clarkb: I'll do it if boden won't... | 15:29 |
| boden | AJaeger working on it | 15:29 |
| boden | 1 sec | 15:29 |
| AJaeger | boden: great! | 15:30 |
| AJaeger | boden: happy to wait ;) | 15:30 |
| *** eernst has quit IRC | 15:31 | |
| clarkb | cmurphy: infra-root https://review.opendev.org/#/c/654474/ can I get a review on that as part of the fixing things for paste work? | 15:32 |
| openstackgerrit | Merged zuul/zuul-website master: Update review.openstack.org URLs https://review.opendev.org/654223 | 15:32 |
| *** yamamoto has quit IRC | 15:32 | |
| cmurphy | i'm too slow | 15:33 |
| boden | AJaeger clarkb https://review.opendev.org/#/c/654487/ | 15:33 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Log an error when command socket is missing https://review.opendev.org/654488 | 15:33 |
| clarkb | boden: +2 lets see if maybe prometheanfire or dhellmann or another requirements core can do a second review | 15:34 |
| corvus | clarkb: https://review.opendev.org/654065 | 15:35 |
| clarkb | cmurphy: https://review.opendev.org/#/c/654482/ and https://review.opendev.org/#/c/654484/ are other puppet module updates to fix status.o.o | 15:35 |
| prometheanfire | looking | 15:36 |
| prometheanfire | done | 15:36 |
| clarkb | corvus: +3 | 15:37 |
| corvus | what recent changes happened regarding oslosphinx? | 15:37 |
| corvus | they seem to have caused some errors: http://zuul.openstack.org/config-errors | 15:37 |
| clarkb | from a consumption side I think oslosphinx has been retired in favor of openstackdocstheme | 15:38 |
| AJaeger | corvus: it was retired | 15:38 |
| fungi | corvus: oslosphinx is being retired because the maintainers don't feel like supporting sphinx 2.x | 15:38 |
| clarkb | maybe we just need to remove it from the zuul main.yaml? | 15:38 |
| AJaeger | we just removed it an hour or so ago from main.yaml | 15:38 |
| corvus | ok. we have cleanup to do. i'll do it. | 15:38 |
| AJaeger | what did we miss? | 15:38 |
| corvus | AJaeger: http://zuul.openstack.org/config-errors | 15:39 |
| AJaeger | corvus: ah, see it. thanks for fixing | 15:39 |
| openstackgerrit | James E. Blair proposed openstack/openstack-zuul-jobs master: Remove oslosphinx https://review.opendev.org/654537 | 15:40 |
| fungi | yeah, from the openstack project's perspective, oslosphinx was deprecated for a while in favor of openstackdocstheme, so when it broke with the release of sphinx 2.0.0 they decided it wasn't worth fixing | 15:40 |
| corvus | AJaeger, fungi: ^ can you instapprove that? | 15:40 |
| fungi | on it | 15:40 |
| *** kaspars__ has joined #openstack-infra | 15:40 | |
| *** kaspars__ has left #openstack-infra | 15:40 | |
| AJaeger | corvus: +2, thanks | 15:40 |
| *** ijw has quit IRC | 15:41 | |
| fungi | and approved | 15:41 |
| cmurphy | how come reviewday is openstack/reviewday but puppet-reviewday is opendev/puppet-reviewday ? | 15:41 |
| clarkb | cmurphy: most(all?) of the config management stuff went into opendev because it is our tooling for oeprating the services. But then services liek zuul or reviewday might actually be built by another org | 15:42 |
| cmurphy | gotcha | 15:42 |
| clarkb | it is the difference between building the software and running the service | 15:42 |
| cmurphy | clarkb: we're gonna have to fix all of these Gemfiles, they are pointing to /home/zuul/src/git.openstack.org/openstack-infra/puppet-openstack_infra_spec_helper and if that file is not found it's going directly to git.o.o and not zuul | 15:43 |
| cmurphy | i guess that's what mordred was talking about over the weekend? | 15:43 |
| clarkb | ya that sounds like it might be related | 15:43 |
| corvus | cmurphy: agreed | 15:43 |
| clarkb | the git.o.o fallback should mostly work except for where we need self testing of the infra spec helper | 15:43 |
| clarkb | definitely something to update | 15:43 |
| cmurphy | yeah | 15:44 |
| corvus | i thought we removed the centos7 jobs from system-config | 15:44 |
| clarkb | corvus: I think my change must've failed | 15:44 |
| clarkb | corvus: let me check on it | 15:44 |
| clarkb | hrm I may not have sufficiently removed the infra puppet apply centos7 jobs | 15:45 |
| clarkb | I will fix | 15:45 |
| corvus | we removed them from something, right? | 15:46 |
| corvus | i distinctly remember a change which couldn't land because the job was running and there are no centos7 hosts | 15:46 |
| clarkb | I thought I had removed the template that included that but there is another second template with those specific centos7 jobs | 15:47 |
| *** e0ne has quit IRC | 15:47 | |
| openstackgerrit | Clark Boylan proposed openstack/openstack-zuul-jobs master: Stop running centos7 puppet apply for infra https://review.opendev.org/654549 | 15:49 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Stop ansipuppeting the old cgit farm https://review.opendev.org/654053 | 15:50 |
| clarkb | corvus: ^ I think that should do it | 15:50 |
| *** jamesmcarthur has quit IRC | 15:50 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Switch gerrit URL to review.opendev.org https://review.opendev.org/654550 | 15:51 |
| *** Lucas_Gray has quit IRC | 15:51 | |
| *** josephrsandoval has quit IRC | 15:51 | |
| clarkb | mriedem: ^ one small thing on that | 15:53 |
| mriedem | ack | 15:55 |
| *** Lucas_Gray has joined #openstack-infra | 15:55 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Switch gerrit URL to review.opendev.org https://review.opendev.org/654550 | 15:56 |
| openstackgerrit | Merged opendev/puppet-lodgeit master: Reduce paste db local backup retention to 10 days https://review.opendev.org/654474 | 15:56 |
| mriedem | clarkb: will that require a restart of the e-s worker? | 15:56 |
| mriedem | *er e-r worker | 15:56 |
| clarkb | mriedem: yes that host is also unhappy for other reasons (which I'm fixing currently) I'll be sure to restart e-r as part of that | 15:57 |
| clarkb | the cron jobs should just start working but I expect the bot is also unhappy | 15:58 |
| *** openstackgerrit has quit IRC | 15:58 | |
| corvus | clarkb: the docker issue is pretty high-priority (we're getting to the point where we're having trouble landing zuul changes); is that ready to go? | 15:59 |
| *** ykarel|away has quit IRC | 15:59 | |
| *** openstackgerrit has joined #openstack-infra | 16:00 | |
| openstackgerrit | Merged opendev/system-config master: Add some more legacy cgit h= redirects https://review.opendev.org/654065 | 16:00 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove centos-7 puppet apply infra jobs https://review.opendev.org/654554 | 16:00 |
| AJaeger | clarkb: followup to your change ^ | 16:00 |
| clarkb | corvus: not quite. I need to update changes with your change and then we need mordreds ssh tunnel idea and the cert update | 16:00 |
| clarkb | corvus: let me start with that first bit | 16:00 |
| corvus | my change is the cert update | 16:00 |
| mriedem | clarkb: ok - fwiw i haven't seen the bot in forever anyway | 16:00 |
| clarkb | ah ok then we'll just need teh ssh tunnel | 16:00 |
| corvus | did anyone write that yet? | 16:01 |
| clarkb | corvus: I don't think so | 16:01 |
| *** e0ne has joined #openstack-infra | 16:01 | |
| corvus | so what was the idea? set up an ssh tunnel and then use that when using skopeo? | 16:02 |
| fungi | i think i missed what the tunnel idea was for docker | 16:02 |
| fungi | ahh, to work around the skopeo/docker raw v6 address url issue | 16:02 |
| corvus | ya | 16:02 |
| clarkb | corvus: yes exactly then skopeo can speak ipv4 | 16:03 |
| clarkb | over ipv6 | 16:03 |
| corvus | okay, i'll give that a shot | 16:03 |
| fungi | i suppose that's easier than setting up a 6-over-4 tunnel or something | 16:04 |
| fungi | er, i guess this would be 4-over-6 | 16:04 |
| AJaeger | amorin: is ovh-bhs1 ok again? We disabled it on the 19th due to network problems | 16:05 |
| fungi | clarkb: as best i can piece together from history, you just ran https://opendev.org/opendev/system-config/src/branch/master/playbooks/sync-gitea-projects.yaml to correct issues links last time? | 16:06 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Only alias docker registries on test nodes https://review.opendev.org/654555 | 16:08 |
| clarkb | corvus: ^ I think that is everything but the skopeo tunneling | 16:08 |
| clarkb | fungi: ya it will take a few hours. We should maybe sanity check that the force option on gitea-repos isn't going to undo things like our redirects | 16:09 |
| fungi | clarkb: should we be able to just use a custom copy of that role which drops the force option instead? | 16:10 |
| clarkb | fungi: the force option is necessary to make chagnes to repos that already exist | 16:10 |
| clarkb | fungi: otherwise it skips | 16:10 |
| fungi | ahh | 16:10 |
| clarkb | fungi: I expect it is fine | 16:11 |
| clarkb | but gitea is also new to me so my expectations may be wrong :) | 16:11 |
| fungi | how a about a stripped-down https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gitea-git-repos/tasks/setup-repo.yaml which just does the sb bits? | 16:11 |
| openstackgerrit | Merged opendev/puppet-gerrit master: Fix opendev urls https://review.opendev.org/654031 | 16:11 |
| openstackgerrit | Merged opendev/puppet-gerrit master: Add support for gitea links https://review.opendev.org/654033 | 16:11 |
| clarkb | fungi: that may be a good option as it will run faster too | 16:12 |
| clarkb | mordred: may have throughts since he wrote teh original iirc | 16:12 |
| corvus | clarkb, fungi: for the ssh tunnel, we're going to need a random unused port on the executor, right? | 16:12 |
| fungi | like dropping the create repo and repo settings and default branch tasks | 16:12 |
| corvus | fungi: the repo settings is what does the storyboard link | 16:13 |
| corvus | but you can drop set default branch | 16:13 |
| fungi | corvus: i expect so on the tunnel... and unless openssh has an option for that, allocating one could be racy? | 16:13 |
| corvus | fungi: yeah, that's what i'm worried about... | 16:13 |
| fungi | corvus: oh, i guess the storuboard tasks are just setting facts which the repo settings task then applies | 16:14 |
| fungi | makes sense | 16:14 |
| *** eernst has joined #openstack-infra | 16:14 | |
| clarkb | corvus: I wonder if 0 works with ssh | 16:19 |
| clarkb | then we can ask netstat/ss for the number | 16:19 |
| corvus | clarkb: no, just tried it. it does with -R but not -L | 16:19 |
| clarkb | is bwrap running in its own network namespace? | 16:20 |
| clarkb | if so the concern over collisions is much smaller | 16:20 |
| *** jamesmcarthur has joined #openstack-infra | 16:21 | |
| corvus | clarkb: not by default | 16:21 |
| corvus | it has that option | 16:21 |
| *** sshnaidm is now known as sshnaidm|afk | 16:22 | |
| corvus | if we do that, we just get a loopback device... what then? | 16:22 |
| *** tosky has joined #openstack-infra | 16:22 | |
| clarkb | probably have to bring your own network routing and setup in that case :/ | 16:23 |
| clarkb | we may be able to use nc | 16:24 |
| clarkb | and if not nc then socat | 16:24 |
| corvus | i also have an idea, but let's hear the nc/socat idea first | 16:25 |
| clarkb | well I thought nc supported more proxy formats than https connect and socks. But in general set up a proxythat goes from localhost:0 to registry:5000 and not worry about encryption | 16:25 |
| clarkb | we really only need to translate the ip layer headers | 16:26 |
| clarkb | I'm fairly certain there is a socat incantation that would do that for us but its manpage is large and complicated so may take a few minutes to sort out | 16:26 |
| corvus | i just realized my idea won't work so nevermind | 16:26 |
| clarkb | socat localhost:0 registry:5000 (maybe? that seems far too simple) | 16:27 |
| fungi | yeah, having nc listen on localhost as a proxy would be just as good as an ssh tunnel, i expect | 16:27 |
| fungi | oh, except that the listener on the remote end could close the socket i suppose? | 16:28 |
| clarkb | ya socat and nc can both be told to do forks on connection | 16:29 |
| clarkb | so that you don't kill the process | 16:29 |
| fungi | ahh | 16:29 |
| openstackgerrit | Merged opendev/elastic-recheck master: Switch gerrit URL to review.opendev.org https://review.opendev.org/654550 | 16:30 |
| tosky | up to which stable branch should we backport the opendev fixes? ocata? | 16:31 |
| fungi | tosky: which opendev fixes? | 16:31 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove oslosphinx https://review.opendev.org/654537 | 16:31 |
| corvus | clarkb: something like: socat TCP-LISTEN:0,fork TCP:zuul.openstack.org:80 | 16:32 |
| fungi | tosky: and it really depends on the nature of the fixes probably. if just cleanup i wouldn't backport them, but if required to be able to run tests then probably any on which you expect to do that (so presumably all open branches?) | 16:32 |
| clarkb | corvus: ya I'm working through it locally too and that looks similar to what I have | 16:32 |
| corvus | would love a "print the port on stdout" option :) | 16:32 |
| clarkb | corvus: also TCP4-LISTEN to force tcp4 on the local side | 16:32 |
| fungi | i suppose the listening port could be obtained from /proc/pid/something | 16:34 |
| fungi | or lsof | 16:34 |
| clarkb | corvus: yup I just got `socat TCP4-LISTEN:0,fork TCP:127.0.0.1:1337` to pass data between two nc's | 16:35 |
| rajinir | We are seeing git review errors, can anyone help? | 16:35 |
| rajinir | https://www.irccloud.com/pastebin/ngKTA71B/ | 16:35 |
| clarkb | and did lsof | grep socat | grep TCP to find the port (which is probably not robust enough for our purposes but does give us an option) | 16:35 |
| clarkb | rajinir: can you show us what git remote -v looks like? From that we'll be able to recommend steps to update the remotes for the renamed repos | 16:36 |
| clarkb | rajinir: in general though you cannot use git:// anymore and the path component of the repo will need to be updated to whatever the new path is for any ssh:// remotes | 16:36 |
| *** e0ne has quit IRC | 16:36 | |
| clarkb | you can find the new path by opening the old http(s) path in your browser and seeing where it leads you | 16:36 |
| corvus | clarkb: socat -d -d TCP-LISTEN:0,fork TCP:zuul.openstack.org:80 | 16:37 |
| corvus | clarkb: 2019/04/22 09:36:55 socat[16220] N listening on AF=2 0.0.0.0:50033 | 16:37 |
| clarkb | debug output to the rescue | 16:37 |
| *** smarcet has quit IRC | 16:40 | |
| tosky | fungi: so are fixes which only affects the README or other documentation file probably not worth backporting so back in time? | 16:41 |
| mwhahaha | is the github sync broken since the rename? | 16:41 |
| fungi | tosky: yeah, up to you but i wouldn't personally make updates to old branches if it's simply to avoid using the redirects | 16:42 |
| corvus | clarkb: i'll try to ansible that and come up with a change; thanks! | 16:42 |
| tosky | fungi: ack, thanks, I will ask the other cores too | 16:43 |
| mwhahaha | nm i got confused by the git log | 16:43 |
| fungi | mwhahaha: depends on the repo. any repos which were in the openstack namespace and remained there are still actively mirrored to the openstack org on github. repositories in other namespaces (including those which used to be in openstack.*) are no longer and need a zuul job added to sync commits after merge to whatever gh org you want | 16:43 |
| *** smarcet has joined #openstack-infra | 16:44 | |
| AJaeger | corvus: zuul config-config errors are fixed ;) Thanks! | 16:46 |
| *** jamesmcarthur_ has joined #openstack-infra | 16:47 | |
| *** smarcet has quit IRC | 16:48 | |
| *** lpetrut has joined #openstack-infra | 16:49 | |
| *** smarcet has joined #openstack-infra | 16:50 | |
| *** jamesmcarthur has quit IRC | 16:50 | |
| corvus | clarkb: have a link to your bug handy? | 16:52 |
| corvus | docker bug | 16:52 |
| clarkb | one swc | 16:52 |
| clarkb | https://github.com/moby/moby/issues/39033 | 16:53 |
| corvus | thx | 16:53 |
| fungi | custom playbook with stripped-down role now updating issues links. https://opendev.org/airship/armada goes to the correct sb project and https://opendev.org/openstack/airship-armada is still correctly redirected | 16:56 |
| *** chandankumar is now known as raukadah | 16:56 | |
| fungi | seems to be churning through them fairly quickly... already up to opendev/lpmqtt | 16:57 |
| fungi | probably worth thinking about how to do this correctly in the rename_repos playbook | 16:57 |
| *** amansi26 has joined #openstack-infra | 16:58 | |
| clarkb | fungi: aweaome thanks! | 16:58 |
| corvus | clarkb, fungi: i verified that "socat -d -d TCP-LISTEN:0,fork TCP:[2001:4800:7819:103:be76:4eff:fe04:5870]:80" works inside a zuul-bwrap on an executor | 16:59 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Use py36 instead of py35 for periodic master jobs https://review.opendev.org/654571 | 16:59 |
| corvus | we will need to install socat on the executors, but that's easy | 16:59 |
| *** eharney has quit IRC | 17:00 | |
| *** ijw has joined #openstack-infra | 17:01 | |
| clarkb | corvus: exciting | 17:02 |
| openstackgerrit | Merged opendev/puppet-bugdaystats master: Use opendev url for bugdaystats source https://review.opendev.org/654484 | 17:09 |
| corvus | clarkb: does https://review.opendev.org/654555 replace https://review.opendev.org/651334 ? | 17:09 |
| clarkb | corvus: yes when you were on vacation I put up 651334 and child as they were easier to review split like that but when we merged them they went in squashed together. I've done similar this time. | 17:10 |
| clarkb | then we had to revert the squashed change when skopeo was no better at ipv6 | 17:10 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use py36 instead of py35 for periodic master jobs https://review.opendev.org/654574 | 17:10 |
| *** Lucas_Gray has quit IRC | 17:11 | |
| fungi | no better, and indicated they wished to remain bug-compatible with docker's behaviors there | 17:11 |
| corvus | clarkb: mind if i update your change to remove the whitespace changes which conflict with my socat change? | 17:11 |
| clarkb | corvus: not at all | 17:11 |
| *** Lucas_Gray has joined #openstack-infra | 17:12 | |
| *** Weifan has joined #openstack-infra | 17:12 | |
| corvus | i think i ended up removing the double newlines anyway | 17:12 |
| corvus | clarkb: actually, i should squash my change into yours | 17:12 |
| clarkb | corvus: that also works | 17:12 |
| corvus | we don't actually need an ssh tunnel, right? | 17:13 |
| corvus | the buildset registry is expected to be remotely accessible | 17:14 |
| clarkb | corvus: nope no ssh necessary | 17:14 |
| clarkb | and the buildset registry does ssl | 17:14 |
| *** harlowja has joined #openstack-infra | 17:15 | |
| corvus | ya | 17:15 |
| *** ijw has quit IRC | 17:15 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Only alias docker registries on test nodes https://review.opendev.org/654555 | 17:15 |
| corvus | clarkb, fungi: ^ new omnibus patch | 17:15 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Install socat on zuul executors https://review.opendev.org/654577 | 17:17 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Only alias docker registries on test nodes https://review.opendev.org/654555 | 17:18 |
| clarkb | corvus: for the port discovery log file and ansible var do we want to use different values so that we can run both playbooks on the same executor context? | 17:18 |
| corvus | readme update | 17:18 |
| clarkb | > socat_port_pull and socat_port_push etc | 17:19 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Use py36 instead of py35 for periodic master jobs https://review.opendev.org/654571 | 17:19 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove periodic-py35 jobs https://review.opendev.org/654580 | 17:19 |
| corvus | clarkb: we could just for extra tidiness, but i don't think the current setup would be a problem since we don't need the variable once we hit the end of that set of tasks | 17:19 |
| clarkb | k | 17:20 |
| clarkb | mriedem: the e-r graphs seem to be fixed now fyi | 17:22 |
| mriedem | \o/ | 17:23 |
| *** Lucas_Gray has quit IRC | 17:23 | |
| fungi | clarkb: corvus: do we have a depends-on change to exercise 654555? | 17:25 |
| corvus | fungi: no, unfortunately we can't because it's used in trusted playbooks | 17:26 |
| corvus | fungi: our options there are (a) land it and quick-revert if it goes sideways; (b) land alternate "test-" roles and set up new jobs to use those; (c) make full-scale integration tests for those | 17:27 |
| fungi | ohh | 17:27 |
| corvus | fungi: my iniclination is (a) today and (c) in a month | 17:27 |
| fungi | it's already partly broken anyway depending on where it runs, so not a huge risk | 17:27 |
| clarkb | we've been doing a) | 17:27 |
| corvus | yeah, that's what's driving my thinking on this | 17:27 |
| clarkb | rechecking a zuul or nodepool change has worked well as a checker | 17:28 |
| fungi | right. anyway it's on its way to the gate now | 17:28 |
| corvus | i think i'll direct-enqueue it | 17:29 |
| fungi | sgtm | 17:29 |
| *** ramishra has quit IRC | 17:30 | |
| *** lpetrut has quit IRC | 17:32 | |
| fungi | i've added a note at the end of https://ethercalc.openstack.org/opendev-transition so people hopefully don't think they should add follow-up moves/renames there | 17:34 |
| SpamapS | FYI, https://opendev.org/openstack/openstack 500's pretty reliably. | 17:34 |
| SpamapS | Guessing gitea wasn't designed to display all of openstack in one repo. ;) | 17:35 |
| corvus | SpamapS: we think there's something wrong with the submodules there; folks have been working through trying to fix that | 17:35 |
| fungi | SpamapS: yeah, we're still trying to work out if gitea needs some additional jolt to fix the subrepos change | 17:35 |
| fungi | SpamapS: though the incorrect issues links you spotted yesterday should hopefully be corrected shortly... we're up around openstack/m repos already | 17:37 |
| SpamapS | fungi: sweet, thanks! | 17:37 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Only alias docker registries on test nodes https://review.opendev.org/654555 | 17:37 |
| clarkb | fwiw I did double check that the submoduels in opendev/gerrit render fine and they do | 17:37 |
| clarkb | so it isn't just submodules being broken | 17:37 |
| corvus | fungi, clarkb: ^ can you re-approve ps4 of that | 17:38 |
| clarkb | looking | 17:38 |
| fungi | ahh, command vs shell task | 17:38 |
| fungi | approved | 17:38 |
| corvus | i switched it for a change i didn't end up making and forgot to switch it back | 17:38 |
| corvus | (that was back when i thought i would be doing ssh tunnels) | 17:39 |
| fungi | i'm going to go snag a late lunch and then start in on project renames in the storyboard-dev database when i get back | 17:39 |
| fungi | bbiaw | 17:39 |
| *** eernst has quit IRC | 17:41 | |
| fungi | (well, once the diversity wg meeting wraps up, anyway) | 17:41 |
| clarkb | I've removed /var/lib/bugdaystats/bugdaystats on status.o.o. Puppet should put it back in place with the new origin | 17:42 |
| clarkb | will do the same with reviewday when that fix is merged | 17:44 |
| corvus | i don't see anything useful in the logs about the openstack/openstack 500 | 17:44 |
| corvus | we may not be logging right yet :/ | 17:44 |
| clarkb | corvus: that from docker logs gitea ? | 17:44 |
| corvus | clarkb: yes and also /var/gitea/logs | 17:44 |
| corvus | (i think we should have docker logs empty and actual logs on disk) | 17:45 |
| corvus | (also, rotation) | 17:45 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Stop running centos7 puppet apply for infra https://review.opendev.org/654549 | 17:45 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove centos-7 puppet apply infra jobs https://review.opendev.org/654554 | 17:45 |
| clarkb | corvus: that sounds good to me | 17:45 |
| *** otherwiseguy has quit IRC | 17:48 | |
| *** mhu has quit IRC | 17:48 | |
| *** mhu has joined #openstack-infra | 17:48 | |
| corvus | clarkb: http://paste.openstack.org/show/749606/ | 17:50 |
| *** mattw4 has joined #openstack-infra | 17:51 | |
| clarkb | I'm cloning from gerrit to fsck | 17:52 |
| *** otherwiseguy has joined #openstack-infra | 17:52 | |
| clarkb | git fsck --strict --full passes in that repo | 17:53 |
| clarkb | and git symbolic-ref head tells me head is not a symbolic ref | 17:54 |
| corvus | i don't see those commands in the gitea source, so that must be from a library? | 17:55 |
| clarkb | the fsck was something I did, but the symbolic ref command si from your paste | 17:55 |
| corvus | oh sorry i meant the commands in the log | 17:55 |
| clarkb | got it | 17:56 |
| *** zbr is now known as zbr|pto | 17:56 | |
| *** ykarel|away has joined #openstack-infra | 17:56 | |
| *** eharney has joined #openstack-infra | 17:57 | |
| openstackgerrit | David Shrewsbury proposed opendev/puppet-nodepool master: Install yappi and objgraph packages https://review.opendev.org/654462 | 17:57 |
| openstackgerrit | Merged opendev/puppet-reviewday master: Use opendev git url https://review.opendev.org/654482 | 17:58 |
| *** e0ne has joined #openstack-infra | 17:58 | |
| corvus | clarkb: the "looking for beginning of value" error may come from the json lib | 17:59 |
| clarkb | oh interesting | 17:59 |
| corvus | clarkb: i ran it locally in debug mode and got: An error has occurred : signal: segmentation fault (core dumped) | 18:07 |
| clarkb | do you know what segfaulted? | 18:07 |
| corvus | there is, in fact, a core file in the openstack repo on the gitea server | 18:07 |
| corvus | clarkb: git | 18:07 |
| corvus | [14864.599480] git[25510]: segfault at 7ffccdf96ff8 ip 00007fcf6f98ccdc sp 00007ffccdf97068 error 6 in libz.so.1.2.11[7fcf6f983000+1c000] | 18:08 |
| clarkb | wow | 18:08 |
| corvus | from my local machine | 18:08 |
| corvus | Apr 22 17:35:05 gitea01 kernel: [4592809.533995] git[25309]: segfault at 7ffee3950fd0 ip 00007f127edf91ea sp 00007ffee3950fc0 error 6 in libz.so.1.2.11[7f127edeb000+1c000] | 18:08 |
| corvus | from gitea01 ^ | 18:08 |
| Shrews | I'm not sure why, but the infra-puppet-apply-3-ubuntu-trusty job doesn't like that I'm trying to define a package in puppet-nodepool that is already defined in puppet-zuul. Other jobs seem ok with it (and I admit, I don't understand why one is using the other). Any puppet experts have an explanation for me? | 18:08 |
| *** e0ne has quit IRC | 18:08 | |
| clarkb | Shrews: puppet requires all resource definitions be unique which means youcan't redefine a resource (this is a really annoying behavior in puppet) | 18:09 |
| clarkb | Shrews: the fix is to wrap it in a conditional that checks if it is already defined first. Let me find you an example | 18:09 |
| Shrews | clarkb: ah ha | 18:09 |
| Shrews | clarkb: no need. i did that with the latest PS | 18:09 |
| clarkb | Shrews: k | 18:09 |
| Shrews | clarkb: but thx for the explanation | 18:09 |
| openstackgerrit | Merged zuul/zuul-jobs master: Only alias docker registries on test nodes https://review.opendev.org/654555 | 18:10 |
| clarkb | corvus: for ^ ithink we can recheck a zuul or nodepool change and see how it does | 18:10 |
| Shrews | clarkb: curious that the other jobs didn't complain though | 18:10 |
| clarkb | Shrews: it requires we install the two modules on the same host so its possible the others don't do that | 18:10 |
| *** ricolin has quit IRC | 18:10 | |
| *** ykarel|away has quit IRC | 18:11 | |
| *** rnoriega has quit IRC | 18:11 | |
| *** rnoriega has joined #openstack-infra | 18:12 | |
| clarkb | or depending on evaluation order if the other side already checks ! defined but the new side doesn't then it dpeends on ordering (and puppet ordering is often not sane) | 18:13 |
| *** e0ne has joined #openstack-infra | 18:14 | |
| corvus | https://review.opendev.org/654577 hasn't landed | 18:14 |
| corvus | i will manually install socat on all executors real quick | 18:14 |
| clarkb | k | 18:14 |
| corvus | i rechecked https://review.opendev.org/654238 | 18:15 |
| corvus | k, socat is installed | 18:15 |
| clarkb | corvus: any idea what git command is segfaulting? (I'm guessing the coredump has that in argv somewhere?) | 18:16 |
| openstackgerrit | Merged opendev/system-config master: Stop ansipuppeting the old cgit farm https://review.opendev.org/654053 | 18:16 |
| openstackgerrit | David Shrewsbury proposed opendev/puppet-zuul master: Install objgraph package https://review.opendev.org/654599 | 18:18 |
| corvus | clarkb: working on it | 18:20 |
| corvus | Core was generated by `git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filt'. | 18:21 |
| openstackgerrit | David Shrewsbury proposed opendev/puppet-zuul master: Install objgraph package https://review.opendev.org/654599 | 18:22 |
| corvus | i think that's truncated | 18:22 |
| corvus | the last command i have in my log is this: | 18:22 |
| corvus | web_1 | [git-module] /data/git/repositories/openstack/openstack.git: git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= name-rev 923e72d5ab3f7d3c8064ec471a5d946e64b92658 | 18:22 |
| clarkb | alright bugdaystats seems happy on status now. I've also removed reviewstats repo on status.o.o as that fix merged | 18:23 |
| corvus | but that works | 18:23 |
| clarkb | git-name-rev - Find symbolic names for given revs | 18:23 |
| clarkb | (I had to look that one up) | 18:25 |
| mnaser | just gonna barge in but | 18:26 |
| mnaser | http://zuul.openstack.org/config-errors | 18:26 |
| mnaser | looks like we have a few things right now | 18:26 |
| corvus | mnaser: i think we just cleared those; do you see any on reload? | 18:27 |
| mnaser | nope, I don't. I thought that autorefreshed (and when I clicked the little bell, I thought it pulled it from the API again) | 18:27 |
| mnaser | indeed, I don't see anything, sorry for the noise! | 18:27 |
| *** amansi26 has quit IRC | 18:28 | |
| mwhahaha | so i noticed that old Depends-On no longer get listed in zuul changes if they still reference openstack.org | 18:29 |
| clarkb | mwhahaha: yup I sent email warning about that | 18:30 |
| clarkb | mwhahaha: zuul sees the gerrit as review.opendev.org now so the old urls are no longer valid from its perspective | 18:30 |
| mwhahaha | meh i missed that one | 18:30 |
| corvus | clarkb: i have reproduced it | 18:31 |
| corvus | clarkb: i think this is the actual command: git -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= name-rev 05ca984219b8fb6d6fed708061d81d82e8c77a0f | 18:31 |
| corvus | (i expect that's the command that's run after the last one logged) | 18:32 |
| clarkb | corvus: that segfaults for me too | 18:32 |
| clarkb | in my gerrit cloned repo | 18:32 |
| corvus | clarkb: it will succeed in a repo with refs/changes but segfault otherwise | 18:32 |
| clarkb | hrm | 18:32 |
| corvus | still, pretty sure segfaulting isn't the right behavior | 18:33 |
| clarkb | ++ | 18:33 |
| *** smarcet has quit IRC | 18:34 | |
| clarkb | corvus: that commit does show up in the master log | 18:34 |
| AJaeger | interesting, 05ca984219b8fb6d6fed708061d81d82e8c77a0f is a change for openstack/openstack/README.rst from 2012 | 18:35 |
| corvus | clarkb: oops i don't think we fixed the underscore issue in your change | 18:36 |
| clarkb | corvus: rgr I'll get a patch up | 18:36 |
| corvus | clarkb: i'll do that | 18:36 |
| clarkb | ok I'll get out of the way then | 18:36 |
| clarkb | :) | 18:37 |
| clarkb | name-rev does have an --execlude flag but I can't get it to not segfault with that | 18:37 |
| clarkb | probably need to examine the core dump | 18:37 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't use underscore in buildset_registry_alias https://review.opendev.org/654603 | 18:40 |
| clarkb | corvus: I went ahead and fast approved ^ since it was something we had talked through previously | 18:40 |
| corvus | clarkb: thanks; i've direct-enqueued it | 18:41 |
| *** jamesmcarthur_ has quit IRC | 18:44 | |
| clarkb | I'm learning how to install debug symbol'd packages on tumbleweed now | 18:48 |
| *** jamesmcarthur has joined #openstack-infra | 18:48 | |
| corvus | clarkb: do you need to do that on ubuntu? | 18:48 |
| corvus | same versions, etc | 18:48 |
| clarkb | possibly. Though I observe the same behavior here on tumbleweed | 18:49 |
| corvus | oh, yeah, i guess you could view your own core file :) | 18:49 |
| *** jamesmcarthur has quit IRC | 18:50 | |
| *** jamesmcarthur has joined #openstack-infra | 18:50 | |
| clarkb | ya I figure understanding how to do this on the current distro I run is worthwile too. I'm failing to get the symbols despite installing what I expected are the packages with such things | 18:52 |
| openstackgerrit | Merged zuul/zuul-jobs master: Don't use underscore in buildset_registry_alias https://review.opendev.org/654603 | 18:53 |
| clarkb | I need to update git is why I htink | 18:55 |
| *** Weifan has quit IRC | 19:04 | |
| AJaeger | config-core, here's a change to use py36 for some periodic jobs - please put on your review queue: https://review.opendev.org/654571 | 19:05 |
| *** slaweq has joined #openstack-infra | 19:08 | |
| *** nicolasbock has joined #openstack-infra | 19:08 | |
| clarkb | corvus: it is segfaulting in libz inflating what I assume is a packfile | 19:13 |
| clarkb | weird that fsck wouldn't catch that | 19:14 |
| smcginnis | clarkb, fungi: We just got another ensure-twine failure looking like it was somehow run in a venv again. | 19:19 |
| smcginnis | Rerunning the job last time that happened passed the second time. | 19:20 |
| smcginnis | Can we reenqueue this one? http://logs.openstack.org/93/93665aaf9bfaec15c5b045118398fc8b339c39ea/release/release-openstack-python/a86df25/job-output.txt.gz#_2019-04-22_19_15_01_719230 | 19:20 |
| smcginnis | Unless anyone has any ideas for troubleshooting how that ended up in that state in the first place. | 19:20 |
| *** Weifan has joined #openstack-infra | 19:21 | |
| clarkb | smcginnis: hrm that runs on localhost? | 19:21 |
| clarkb | smcginnis: I wonder if this is an interaction with the ansible venvs that zuul creats | 19:21 |
| smcginnis | clarkb: I believe so. | 19:21 |
| clarkb | like maybe sometimes we install into that venv | 19:21 |
| clarkb | or try to rather | 19:21 |
| smcginnis | There aren't any jobs or roles that I'm aware of that would even be setting up a venv. | 19:22 |
| smcginnis | Unless somehow this gets invoked somewhere: https://opendev.org/openstack/horizon/src/branch/stable/pike/tools/with_venv.sh | 19:22 |
| smcginnis | I see it being copied, but not run anywhere. | 19:22 |
| *** e0ne has quit IRC | 19:23 | |
| smcginnis | Looks like that is removed in stable/queens, so pike might be the last place we hit it if that is the root cause. Or involved in whatever the root cause is. | 19:23 |
| smcginnis | Dropped here: https://opendev.org/openstack/horizon/commit/6529cd1eb8aef16b7ded96779c684644ed5a9fe0 | 19:24 |
| corvus | clarkb: remote: https://review.opendev.org/654613 Touch the README | 19:26 |
| smcginnis | with_venv.sh and run_tests.sh not present in the python-solumclient release that failed last week (and passed after reenqueing) | 19:28 |
| clarkb | corvus: neat hack if it works | 19:29 |
| corvus | clarkb: i'm trying it locally now | 19:29 |
| clarkb | fwiw I think I've extracted the broken chunk of the packfile but gzip says its not valid so maybe pebcak | 19:29 |
| clarkb | 05ca984219b8fb6d6fed708061d81d82e8c77a0f commit 289 202 54832784 is what verify-pack says | 19:29 |
| clarkb | SHA-1 type size size-in-packfile offset-in-packfile are the columns | 19:30 |
| fungi | okay, back and getting ready to work out how to fix the storyboard-dev database. the gitea issues linking playbook completed in 2 hours 15 minutes, for the record | 19:31 |
| fungi | still catching up, but... mwhahaha: you can query gerrit for changes matching message:"depends-on: https://review.openstack" and then update their commit messages with the correct domain | 19:33 |
| mwhahaha | yea just a giant pain that's all | 19:34 |
| mwhahaha | if you're not paying attention you can end up hanging stuff | 19:34 |
| smcginnis | A lot of those tags out there. | 19:34 |
| mwhahaha | this is why i liked using the change-id instead :D | 19:34 |
| mwhahaha | pros and cons of both | 19:34 |
| corvus | clarkb: there are other commits that will trigger it; da2b65dadfda6e6551ef4114a7ae362faaa9ba57 for ex | 19:35 |
| *** smarcet has joined #openstack-infra | 19:40 | |
| clarkb | assuming I've extracted the right bytes from the packfile pigz and python zlib both are mad about the data | 19:41 |
| corvus | clarkb: i've tried variations of fsck, gc, and repack with no change | 19:45 |
| clarkb | I'm unpacking into a just git init'd repo then will see what I get for loose objects | 19:46 |
| clarkb | `git unpack-objects < ../openstack/.git/objects/pack/pack-cc39805ca42cb2cdabf0ba776d157f15524c500f.pack` sort of thing | 19:47 |
| clarkb | puppet on status is happy again now. I'm going to restart the e-r bot to pick up mriedem's fix | 19:48 |
| *** josephrsandoval has joined #openstack-infra | 19:49 | |
| clarkb | corvus: in my new repo with unpacked object name-rev does not segfault and returns "undefined" | 19:52 |
| clarkb | corvus: I'm kinda thinking maybe the packfile is valid but tickles a zlib or git bug? | 19:52 |
| clarkb | and for whatever reason doing a full conversion doesn't trip over that | 19:53 |
| clarkb | git cat-file works in both cases | 19:53 |
| corvus | clarkb: i cloned the repo into my homedir on review.o.o, verified it did not segfault, tar'd it and copied it to my workstation, untar'd and it does segfault | 19:54 |
| corvus | so it seems that git/zlib on review.o.o doesn't have this bug? | 19:54 |
| clarkb | corvus: that would make sense to me given the behavior we are seeing | 19:55 |
| clarkb | I can also pigz -d -c < $objectfile and that works | 19:55 |
| clarkb | so I think my earlier thing is pebkac somewhere | 19:56 |
| *** slaweq has quit IRC | 19:56 | |
| clarkb | I'm goign to repack in my test repo and see if that results in broken again | 19:56 |
| *** jjohnson42 has joined #openstack-infra | 19:58 | |
| jjohnson42 | I have a question, is it currently expected that github mirror of project is not getting updates after opendev change? | 19:59 |
| clarkb | corvus: I can't convince git repack to actually repack because it says "Nothing new to pack." | 19:59 |
| fungi | jjohnson42: it depends entirely on the project. is it an official openstack deliverable, or an unofficial project which was moved out of openstack and a stale repository left behind in the "openstack" namespace on github? | 20:00 |
| fungi | the latter is entirely expected, it's up to the openstack tc to decide how, when and who will take care of those | 20:00 |
| jjohnson42 | fungi: https://github.com/openstack/pyghmi | 20:01 |
| AJaeger | jjohnson42: So, an unofficial project that was moved out of namespace | 20:01 |
| jjohnson42 | fungi: So who should I talk to about either getting that alive again or moving it elsewhere? | 20:01 |
| clarkb | corvus: hrm I moved .git/objects aside in the repo I cloned and moved the .git/objects from my hacked up repo into the original repo (so no packfile) and now it segfaults again | 20:02 |
| fungi | jjohnson42: yep, that's not an official part of openstack and it will be removed from https://github.com/openstack/ at some point in the future by a method as of yet undetermined. we can arrange to transfer it to a different github organization and point you to a zuul job you'll want to run in the post pipeline to start mirroring there | 20:02 |
| clarkb | corvus: so it is some interaction between the data and the repo state I think | 20:02 |
| fungi | jjohnson42: in the meantime, https://opendev.org/x/pyghmi should be entirely up to date | 20:02 |
| jjohnson42 | fungi: right, I have that as my remote, just most others come across through github | 20:03 |
| jjohnson42 | and have been worried about how dead-looking it would be | 20:03 |
| fungi | i understand the worry. we tried to communicate that this would be the case, but there was a lot for people to read | 20:03 |
| corvus | clarkb: i'm going to grab food | 20:03 |
| jjohnson42 | Any rough guesses as to when the openstack github plans to clean house? | 20:04 |
| jjohnson42 | If it's just a few weeks, will just sit tight and let nature take its course | 20:05 |
| clarkb | jjohnson42: I think that is mostly up to the openstack TC? | 20:05 |
| *** e0ne has joined #openstack-infra | 20:05 | |
| smcginnis | A maintainer could choose what they want to happen to unofficial projects, right? | 20:06 |
| smcginnis | TC can give a recommendation, but being unofficial, it's really just that. | 20:06 |
| jjohnson42 | ok, I don't know how rare my situation is, but the project evolved such that I'm pretty oblivious to openstack governance and haven't paid attention, hence my obliviousness | 20:08 |
| jjohnson42 | anyway, I have another github remote and I'll steer things that way, and I guess I'll go asking about the right person if it doesn't naturally disappear from github | 20:09 |
| jjohnson42 | thanks for the clarification | 20:09 |
| smcginnis | jjohnson42: Would having it be archived on GitHub meet your needs? | 20:09 |
| fungi | jjohnson42: no worries. regardless of what the tc decides we should be able to simply transfer it to a different org under your control and show you how to set up replication if that's preferred. then github sets up redirection naturally for folks who still have the old remote | 20:10 |
| jjohnson42 | smcginnis: so from my perspective, the ideal would be a visitor to that url gets sent to the 'correct' url, even if the correct url is something that I (the maintainer) designate | 20:10 |
| *** josephrsandoval has quit IRC | 20:10 | |
| jjohnson42 | for example: https://github.com/lenovo/pyghmi | 20:13 |
| jjohnson42 | I'll of course settle for whatever is most convenient, a url change wouldn't be the worst thing in the world since it's just the git repository not the enduser download experience or anything | 20:13 |
| fungi | yeah, if you control the lenovo org, you could delete the existing pyghmi repository there temporarily and we could transfer openstack/pyghmi to lenovo/pygmhi, the main complexity is that we need an account which is an administrator in both the old and new gh org at the time the transfer is initiated | 20:14 |
| fungi | but after doing that, the account can be removed from the lenovo org | 20:14 |
| *** ralonsoh has quit IRC | 20:14 | |
| fungi | and anyone going to https://github.com/openstack/pyghmi (browsing or cloning/pulling) will be redirected to https://github.com/lenovo/pyghmi | 20:15 |
| jjohnson42 | fungi: but that account can be demoted after transfer? | 20:15 |
| Shrews | infra-root: Just FYI, in order to start tracking down the nodepool mem leak, we're going to need https://review.opendev.org/654462 merged. I can restart the launchers after that. | 20:16 |
| Shrews | (assuming i did the puppeting things right) | 20:16 |
| *** sshnaidm|afk has quit IRC | 20:16 | |
| jjohnson42 | fungi: hmm, if it is still feasible, then could move to my personal and then I could coordinate with the lenovo owners to do it once more, if doing it twice makes sense.. | 20:17 |
| fungi | jjohnson42: yes, it can be removed from the org permissions as soon as the transfer request is completed | 20:17 |
| jjohnson42 | fungi: ok, again sorry from coming in cold, but who do I talk to to get openstack to set up the repository redirect? | 20:17 |
| fungi | also, as i noted earlier, in combination with this or separately, there is a job in the zuul-jobs standard library which can be added to your project to mirror it to any external service via git push, you just need to provide it with encrypted credentials for an account to use for that purpose | 20:18 |
| fungi | jjohnson42: no apologies needed. you're effectively the first organic request we've had for a transfer (we only just implemented the changes on friday) but we were prepared for the likelihood. dmsimard already did one before the manitenance for the openstack/ara repo to test out the gh api scripting he wrote for it | 20:20 |
| jjohnson42 | fungi: yeah... I might do that through another git hook offsite in this case due to the org I'd probably target, but thanks for the thought | 20:20 |
| clarkb | https://github.com/madler/zlib/blob/master/inftrees.c#L20-L39 is where zlib seems to segfault | 20:20 |
| fungi | jjohnson42: i'll see if i can dig up where we landed the transfer script and work with you on the details | 20:21 |
| *** josephrsandoval has joined #openstack-infra | 20:21 | |
| *** kgiusti has left #openstack-infra | 20:22 | |
| jjohnson42 | fungi: thanks. I just have been keenly aware that I was living in the infrastructure without paying appropriate attention and so I want to be extra clear that I'm not being rude or anything ;) | 20:23 |
| fungi | not rude at all, though we do at least try to communicate upcoming changes in our services on the openstack-infra@lists.openstack.org (we're aware it's still a very "openstacky" mailing list name for the moment) | 20:24 |
| jjohnson42 | fungi: going to have to run, but I'll check the irc logs around about this when I get back if you post anything, thanks. | 20:26 |
| fungi | jjohnson42: cool, and i found it by the way, still under review at https://review.opendev.org/644937 | 20:26 |
| *** rlandy|ruck is now known as rlandy|ruck|biab | 20:26 | |
| fungi | i'm approving it now, and we can always improve it if we discover any bugs | 20:27 |
| corvus | fungi: when you have a sec, https://review.opendev.org/654577 could use a +3 (though i already installed it manually) | 20:28 |
| jjohnson42 | fungi: great, thanks for the pointer, I'll try my luck sometime this week | 20:28 |
| fungi | thanks corvus, taking a look | 20:29 |
| clarkb | corvus: also I hoped on my xenial fileserver and it too segfaults | 20:29 |
| *** lpetrut has joined #openstack-infra | 20:29 | |
| corvus | clarkb: weird that review doesn't -- it's xenial | 20:29 |
| fungi | i was about to say the same | 20:29 |
| fungi | compare zlib1g package versions? | 20:29 |
| clarkb | corvus: fungi I think it goes back to refs/changes | 20:29 |
| clarkb | I've got one clone on my xenail fileserver with refs/changes mirrored in and one that doesn't | 20:30 |
| corvus | clarkb: right, but my clone doesn't have refs/changes | 20:30 |
| clarkb | it segfaults on the one that doesnt but does not on the one without refs/changes | 20:30 |
| clarkb | hrm | 20:30 |
| clarkb | maybe it is something else in a git cloen --mirror then | 20:30 |
| fungi | cuold something be referring to object ids under refs/changes and then breaking when those are excluded? | 20:30 |
| clarkb | http://paste.openstack.org/show/749613/ is a bt by the way | 20:31 |
| clarkb | https://github.com/madler/zlib/blob/master/inftrees.c#L88-L91 is the sort of thing that may cause this I expect | 20:32 |
| *** jjohnson42 has quit IRC | 20:33 | |
| *** lpetrut has quit IRC | 20:33 | |
| clarkb | codes = 19 and lens[0:19] area all smaller than MAX_BITS so that rules that out | 20:37 |
| fungi | was that bt with symbols installed for zlib? | 20:38 |
| clarkb | fungi: yup | 20:39 |
| clarkb | and git | 20:39 |
| clarkb | https://marc.info/?l=git&m=139526849815131&w=2 fwiw that may be related? | 20:39 |
| fungi | yeah, i guess so, it has function names and line numbers | 20:39 |
| *** kjackal has quit IRC | 20:39 | |
| fungi | i got lost in the lengthy data segments | 20:40 |
| clarkb | yup doubling stacksize fixes it for me | 20:41 |
| clarkb | corvus: ^ fyi | 20:41 |
| corvus | clarkb: i'll try that with gitea | 20:41 |
| clarkb | 05ca984219b8fb6d6fed708061d81d82e8c77a0f master~124884^2 is the answer it emits | 20:41 |
| fungi | so this may be a stack overrun? | 20:41 |
| clarkb | fungi: ya | 20:41 |
| clarkb | which eplains why the segfault happens on the function line and not internally | 20:41 |
| fungi | right | 20:42 |
| clarkb | I think if you have refs/changes/foo that it doesn't need as many recursive calls to find the name-rev | 20:42 |
| clarkb | because the name-rev is refs/changes/something but without them it has to go all the way back to end of master and fine it | 20:42 |
| corvus | clarkb: yep that works | 20:42 |
| clarkb | we might be able to workaround this by tagging things periodically on that repo | 20:42 |
| clarkb | so that it finds the tgas | 20:43 |
| clarkb | corvus: woot. I am going to find lunch myself now | 20:43 |
| fungi | that git ml thread is definitely interesting | 20:43 |
| clarkb | before I go the openstack discuss list has a thread on codesearch needing updates | 20:43 |
| clarkb | anyone know what we need to do to make it reindex things? | 20:44 |
| fungi | unfortunately it's over 5 years old and ends with "I'll look into it and hopefully have a patch by the end of the weekend." | 20:44 |
| fungi | did dragos patch this? or lose interest? | 20:44 |
| clarkb | I don't know | 20:44 |
| clarkb | but I do know my stomach demands food. Back in a bit | 20:44 |
| fungi | yeah, following up to the ml would have been swell | 20:44 |
| corvus | doubling the stack size should be sufficient | 20:45 |
| corvus | so something in the 16M neighborhood | 20:45 |
| *** josephrsandoval has quit IRC | 20:45 | |
| *** pcaruana has quit IRC | 20:46 | |
| fungi | SpamapS: ^ mystery solved by the way | 20:46 |
| fungi | a testament to openstack that its git activity is massive enough to trigger this sort of bug | 20:47 |
| corvus | if anyone asks why the cncf activity thingy hasn't updated, we'll have a *really* good answer | 20:48 |
| corvus | ETOOMUCHACTIVITY | 20:48 |
| fungi | also yet another argument against monorepos for large projects, i guess | 20:48 |
| corvus | i'll work on a change | 20:49 |
| *** seyeongkim has quit IRC | 20:51 | |
| *** guilhermesp has quit IRC | 20:52 | |
| *** davidlenwell has quit IRC | 20:53 | |
| *** icey has quit IRC | 20:53 | |
| *** yikun has quit IRC | 20:53 | |
| *** michael-beaver has quit IRC | 20:54 | |
| *** jungleboyj has quit IRC | 20:54 | |
| *** dwalt has quit IRC | 20:55 | |
| *** sthussey has quit IRC | 20:55 | |
| *** jungleboyj has joined #openstack-infra | 20:55 | |
| *** lsell has quit IRC | 20:56 | |
| *** seyeongkim has joined #openstack-infra | 20:56 | |
| *** michael-beaver has joined #openstack-infra | 20:57 | |
| *** lseki has quit IRC | 20:57 | |
| *** icey has joined #openstack-infra | 20:57 | |
| *** davidlenwell has joined #openstack-infra | 20:57 | |
| *** dwalt has joined #openstack-infra | 20:57 | |
| *** seyeongkim has quit IRC | 20:58 | |
| *** e0ne has quit IRC | 20:58 | |
| *** sweston has quit IRC | 20:58 | |
| SpamapS | fungi:s/for large projects// ... it's hard to know which projects will be too large for monorepo when you set up your repo structure. ;) | 20:59 |
| *** gmann has quit IRC | 20:59 | |
| fungi | this is very true | 20:59 |
| *** smarcet has quit IRC | 21:00 | |
| *** dougwig has quit IRC | 21:00 | |
| fungi | granted, you *can* split things out as the project grows | 21:00 |
| *** beisner has quit IRC | 21:00 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Double stack size on gitea https://review.opendev.org/654629 | 21:01 |
| corvus | fungi, clarkb, SpamapS: ^ should be the fix | 21:01 |
| corvus | rather, "a" fix :) | 21:01 |
| *** josephrsandoval has joined #openstack-infra | 21:01 | |
| *** e0ne has joined #openstack-infra | 21:02 | |
| *** bobh has quit IRC | 21:02 | |
| *** sthussey has joined #openstack-infra | 21:02 | |
| *** dwalt has quit IRC | 21:02 | |
| *** davidlenwell has quit IRC | 21:03 | |
| *** zbr|pto has quit IRC | 21:03 | |
| *** icey has quit IRC | 21:04 | |
| *** jungleboyj has quit IRC | 21:04 | |
| *** mnaser has quit IRC | 21:04 | |
| *** msmarcal has quit IRC | 21:04 | |
| SpamapS | corvus:neat | 21:04 |
| clarkb | corvus: will that apply to the proceses running in the container? | 21:06 |
| *** e0ne has quit IRC | 21:06 | |
| corvus | clarkb: probably? :) | 21:06 |
| clarkb | SpamapS may know. Do the ulimits defined in the host system or the container image apply to processes in a container run by docker? | 21:07 |
| *** sthussey has quit IRC | 21:07 | |
| clarkb | corvus: https://mtyurt.net/post/docker-how-to-increase-number-of-open-files-limit.html it might be something docker compose exposes for us | 21:07 |
| *** msmarcal has joined #openstack-infra | 21:07 | |
| *** jungleboyj has joined #openstack-infra | 21:07 | |
| *** icey has joined #openstack-infra | 21:07 | |
| *** kranthikirang1 has quit IRC | 21:07 | |
| corvus | clarkb: i'll check out the docker-compose idea | 21:07 |
| *** gmann has joined #openstack-infra | 21:07 | |
| *** dwalt has joined #openstack-infra | 21:08 | |
| *** mnaser has joined #openstack-infra | 21:08 | |
| *** lseki has joined #openstack-infra | 21:08 | |
| *** sweston has joined #openstack-infra | 21:08 | |
| *** beisner has joined #openstack-infra | 21:08 | |
| *** davidlenwell has joined #openstack-infra | 21:08 | |
| *** lsell has joined #openstack-infra | 21:08 | |
| *** seyeongkim has joined #openstack-infra | 21:08 | |
| clarkb | reading more I think docker gives containers the limits set in the host so your change will probably work as is | 21:08 |
| *** dougwig has joined #openstack-infra | 21:08 | |
| *** dwalt has left #openstack-infra | 21:09 | |
| *** sthussey has joined #openstack-infra | 21:09 | |
| *** yikun has joined #openstack-infra | 21:09 | |
| clarkb | (fwiw I like the idea of capturing that in docker-compose if possible because then it is self contained) | 21:10 |
| corvus | any idea how to tell docker something is unlimited? | 21:11 |
| clarkb | corvus: https://docs.docker.com/engine/reference/commandline/run/ says they pass the value on the command line straight to the syscall | 21:11 |
| corvus | services.web.ulimits.stack.hard contains "unlimited", which is an invalid type, it should be an integer | 21:12 |
| corvus | ERROR: Service 'web' has invalid ulimit '{'stack': {'hard': 0, 'soft': 16384}}'. 'soft' value can not be greater than 'hard' value | 21:12 |
| corvus | services.web.ulimits.stack u'hard' is a required property | 21:12 |
| *** guilhermesp has joined #openstack-infra | 21:12 | |
| corvus | i'm 0/3 | 21:12 |
| clarkb | -1 maybe? | 21:12 |
| corvus | ERROR: Service 'web' has invalid ulimit '{'stack': {'hard': -1, 'soft': 16384}}'. 'soft' value can not be greater than 'hard' value | 21:12 |
| clarkb | "To specify the unlimited or infinity limit (RLIM_INFINITY), the -1 or 'unlimited' string can be passed." from the prlimit man page | 21:15 |
| clarkb | I'm guessing docker doesn't translate either -1 or unlimited to the defined constant | 21:15 |
| corvus | yeah. it's almost like the docker folks are unfamiliar with unix. | 21:15 |
| corvus | it's failed to support basic unix resource control and networking so far. | 21:16 |
| clarkb | can you pass RLIM_INFINITY ? | 21:16 |
| clarkb | (I doubt that will work but maybe they translate that?) | 21:16 |
| corvus | clarkb: https://github.com/docker/compose/blob/master/compose/config/config_schema_v3.7.json#L258-L276 | 21:17 |
| corvus | services.web.ulimits.stack.hard contains "RLIM_INFINITY", which is an invalid type, it should be an integer | 21:17 |
| *** fmount has quit IRC | 21:18 | |
| clarkb | welp another case of overzealous input checking that prevents people from using their system | 21:18 |
| clarkb | are you wanting to set stack size to unlimited? | 21:18 |
| corvus | i will file a bug for them to ignore | 21:18 |
| clarkb | oh the hard limit I expect you want to set to unlimited | 21:18 |
| clarkb | maybe we set hard to 8x the size of soft and double soft? | 21:18 |
| corvus | yeah, if gitea is smart enough to do a ulimit change i don't want to stop it | 21:19 |
| *** josephrsandoval has quit IRC | 21:20 | |
| corvus | docker itself has the same problem: | 21:21 |
| corvus | invalid argument "stack=16123:unlimited" for "--ulimit" flag: strconv.ParseInt: parsing "unlimited": invalid syntax | 21:21 |
| clarkb | not surprising | 21:22 |
| *** fmount has joined #openstack-infra | 21:22 | |
| smcginnis | Did python-storyboard client end up getting moved somewhere other than openstack/ ? | 21:23 |
| smcginnis | I get a 404 on opendev, but the paste that had the list of renames said it should be there. | 21:23 |
| corvus | clarkb: ugh -- https://github.com/moby/moby/issues/12515 | 21:24 |
| clarkb | smcginnis: https://opendev.org/opendev/python-storyboardclient works for me | 21:24 |
| clarkb | corvus: so we need compose to accept -1? | 21:24 |
| smcginnis | clarkb: So that's a "yes" then. :) | 21:24 |
| corvus | clarkb: nope, compose behaves the same as docker | 21:24 |
| corvus | clarkb: it accepts -1, but still requires that hard be > soft | 21:24 |
| clarkb | corvus: oh wow | 21:25 |
| clarkb | smcginnis: well you said it 404s | 21:25 |
| clarkb | smcginnis: I'm saying it doesn't and yes it moved to opendev/ | 21:25 |
| corvus | clarkb: you can set unlimited/unlimited, but that's the only thing you can do with unlimited | 21:25 |
| smcginnis | clarkb: Yes, the openstack namespace 404's and that was what was listed in the migration list. So yes, it was moved from somewhere other than openstack/ | 21:25 |
| corvus | smcginnis: what migration list? | 21:26 |
| smcginnis | http://paste.openstack.org/show/749402/ | 21:26 |
| corvus | smcginnis: where did that come from? | 21:26 |
| clarkb | corvus: I think that was one of the early lists jroll and fungi shared | 21:27 |
| smcginnis | corvus: It was what was published to openstack-infra ML. | 21:27 |
| clarkb | smcginnis: fwiw the easiest way to find where something moved to is to use its old url and let the redirects show you | 21:27 |
| corvus | smcginnis: that was probably a draft published for feedback | 21:28 |
| corvus | we got lots of feedback and many changes happened | 21:28 |
| clarkb | https://git.openstack.org/openstack-infra/python-storyboardclient -> https://opendev.org/opendev/python-storyboardclient for example | 21:28 |
| smcginnis | clarkb: Oh, good idea. I think this was the only rename I had an issue with, but that will be a quick way if I run into anything else. | 21:28 |
| clarkb | as an alternative you can click to explore button in gitea and type in python-storyboardclient and you should get results that way too | 21:28 |
| clarkb | corvus: http://logs.openstack.org/38/654238/1/check/zuul-build-image/9b08032/job-output.txt.gz#_2019-04-22_21_05_54_075128 looks like the cert isn't working for docker proxied through socat, but it seems that ipv6 shouldn't be an issue once we fix that | 21:31 |
| corvus | clarkb: oh i think i see the problem | 21:35 |
| clarkb | good I wasn't seeing it yet :) | 21:35 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix localhost cert location for push-to-intermediate-registry https://review.opendev.org/654632 | 21:36 |
| corvus | clarkb: ^ | 21:36 |
| corvus | clarkb: also https://github.com/moby/moby/issues/39125 | 21:36 |
| clarkb | corvus: the error I linked seem to be with ssl validation | 21:37 |
| clarkb | the config issue may be an issue too but do we expect it to fix ^ | 21:38 |
| corvus | clarkb: yeah, i think it didn't find the CA cert because it didn't match the "hostname" | 21:38 |
| clarkb | oh I see | 21:38 |
| clarkb | though you only updated the json? | 21:38 |
| clarkb | we also need to update filepaths? | 21:38 |
| corvus | hrm | 21:39 |
| corvus | well, that was the only localhost instance i found | 21:39 |
| corvus | yeah, i guess i only updated the auth config | 21:40 |
| clarkb | grep for certs | 21:40 |
| corvus | that doesn't seem like it should cause the 509 error | 21:40 |
| clarkb | I think we use buildset_registry.host | ipwrap currently there and that needs to become 127.0.0.1 | 21:40 |
| corvus | clarkb: yep, thx | 21:40 |
| corvus | will update that patch | 21:40 |
| *** boden has quit IRC | 21:41 | |
| clarkb | and re ulimits I think we can just set a really big hard value then | 21:42 |
| clarkb | should have roughly the same effect | 21:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Fix localhost cert location for push-to-intermediate-registry https://review.opendev.org/654632 | 21:43 |
| clarkb | fungi: ^ have a quick moment to review that too? | 21:45 |
| *** bobh has joined #openstack-infra | 21:46 | |
| clarkb | corvus: ^ it is approved now if you want to direct enqueue. I need to step away for a little bit again | 21:48 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Double stack size on gitea https://review.opendev.org/654634 | 21:54 |
| corvus | clarkb, fungi: ^ alternate to https://review.opendev.org/654629 | 21:54 |
| fungi | oh, cool, should i assume i don't yet need to dig into the system-config-run-gitea failure on 654629? | 21:55 |
| clarkb | fungi: not if you prefer the alternate :) | 21:56 |
| corvus | fungi: i dug into it enough to see that it looks like a real failure, so there's probably a bug in the change or the test; so yeah, it's moot if we go with 654634 | 21:56 |
| clarkb | (I prefer the alternate) | 21:56 |
| clarkb | ok in my todo list I scribbled down the last remaining big item I wanted to followup on from friday was the update of the rename playbook | 21:57 |
| clarkb | has anyone done that? | 21:57 |
| clarkb | if not I can take that on | 21:57 |
| corvus | well, there are two problems with the first change: 1) there's a bug in the test which is causing it not to work; and 2) the output shows that the test would fail if it did work | 21:57 |
| clarkb | (I'm sure there are other todos to add to my list too) | 21:57 |
| *** bauzas_ has joined #openstack-infra | 21:58 | |
| *** manjeets_ has joined #openstack-infra | 21:58 | |
| *** irclogbot_0 has quit IRC | 21:58 | |
| *** niedbalski has quit IRC | 21:58 | |
| *** bauzas has quit IRC | 21:58 | |
| *** tonyb has quit IRC | 21:58 | |
| *** bauzas_ is now known as bauzas | 21:58 | |
| *** diablo_rojo has quit IRC | 21:58 | |
| *** panda has quit IRC | 21:58 | |
| *** odyssey4me has quit IRC | 21:58 | |
| *** zhurong has quit IRC | 21:59 | |
| *** ddurst has quit IRC | 21:59 | |
| *** diablo_rojo has joined #openstack-infra | 21:59 | |
| *** manjeets has quit IRC | 21:59 | |
| *** dansmith has quit IRC | 21:59 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix localhost cert location for push-to-intermediate-registry https://review.opendev.org/654632 | 21:59 |
| *** imacdonn has quit IRC | 21:59 | |
| *** panda has joined #openstack-infra | 22:00 | |
| clarkb | I recheced 654238 to test ^ | 22:00 |
| *** odyssey4me has joined #openstack-infra | 22:00 | |
| *** anteaya has quit IRC | 22:00 | |
| *** rlandy|ruck|biab is now known as rlandy|ruck | 22:00 | |
| *** ddurst has joined #openstack-infra | 22:00 | |
| *** seyeongkim has quit IRC | 22:01 | |
| *** bgmccollum has quit IRC | 22:01 | |
| *** dansmith has joined #openstack-infra | 22:01 | |
| *** imacdonn has joined #openstack-infra | 22:01 | |
| clarkb | gitea 1.8.0 released too apparently | 22:01 |
| fungi | yeah, i like the new version | 22:01 |
| *** mandre has quit IRC | 22:02 | |
| *** irclogbot_1 has joined #openstack-infra | 22:02 | |
| *** seyeongkim has joined #openstack-infra | 22:03 | |
| *** mandre has joined #openstack-infra | 22:04 | |
| fungi | i had to math for a moment to confirm that number was what i thought it was | 22:05 |
| *** sthussey_ has joined #openstack-infra | 22:05 | |
| fungi | this was so much easier when we only had to think about 32-bit systems | 22:06 |
| clarkb | it was really big and I figured the test would tell us if there was something wrong with its size | 22:06 |
| fungi | or, heck, 16-bit ;) | 22:06 |
| fungi | it's 2^63-1 | 22:06 |
| clarkb | so max signed int | 22:06 |
| fungi | so max size for a signed 64-bit int, right | 22:06 |
| *** sthussey has quit IRC | 22:07 | |
| *** jungleboyj has quit IRC | 22:07 | |
| *** gmann has quit IRC | 22:07 | |
| *** sthussey_ is now known as sthussey | 22:07 | |
| *** nicolasbock has quit IRC | 22:07 | |
| *** nicolasbock has joined #openstack-infra | 22:07 | |
| clarkb | http://zuul.openstack.org/stream/9ef300ffdfac48cea619b5c62695d4c4?logfile=console.log hopefullyshows working docker | 22:07 |
| *** gmann has joined #openstack-infra | 22:07 | |
| *** bgmccollum has joined #openstack-infra | 22:07 | |
| *** bobh has quit IRC | 22:08 | |
| *** tjgresha_nope has joined #openstack-infra | 22:10 | |
| *** gmann has quit IRC | 22:11 | |
| *** irclogbot_1 has quit IRC | 22:11 | |
| *** amotoki_ has joined #openstack-infra | 22:11 | |
| *** gmann has joined #openstack-infra | 22:11 | |
| *** kopecmartin has joined #openstack-infra | 22:11 | |
| *** defionscode_ has joined #openstack-infra | 22:12 | |
| *** kinrui has joined #openstack-infra | 22:13 | |
| *** armax_ has joined #openstack-infra | 22:13 | |
| *** jrist- has joined #openstack-infra | 22:13 | |
| *** rpioso_ has joined #openstack-infra | 22:15 | |
| fungi | i did a double-take because i'm not used to looking at numbers that large outside of initialization vectors for cryptographic algorithms | 22:16 |
| *** freerunner has quit IRC | 22:16 | |
| *** amotoki has quit IRC | 22:16 | |
| *** kopecmartin|off has quit IRC | 22:16 | |
| *** rpioso has quit IRC | 22:16 | |
| *** jbadiapa has quit IRC | 22:16 | |
| *** strigazi has quit IRC | 22:16 | |
| *** Qiming has quit IRC | 22:16 | |
| *** gagehugo has quit IRC | 22:16 | |
| *** brwyatt has quit IRC | 22:16 | |
| *** dansmith has quit IRC | 22:16 | |
| *** nhicher has quit IRC | 22:16 | |
| *** defionscode has quit IRC | 22:16 | |
| *** fungi has quit IRC | 22:16 | |
| *** slittle1 has quit IRC | 22:16 | |
| *** tjgresha has quit IRC | 22:16 | |
| *** SpamapS has quit IRC | 22:16 | |
| *** jrist has quit IRC | 22:16 | |
| *** pots has quit IRC | 22:16 | |
| *** armax has quit IRC | 22:16 | |
| *** armax_ is now known as armax | 22:16 | |
| *** defionscode_ is now known as defionscode | 22:16 | |
| *** rpioso_ is now known as rpioso | 22:16 | |
| *** pots has joined #openstack-infra | 22:16 | |
| *** dansmith has joined #openstack-infra | 22:17 | |
| *** brwyatt has joined #openstack-infra | 22:17 | |
| *** freerunner has joined #openstack-infra | 22:18 | |
| *** irclogbot_2 has joined #openstack-infra | 22:18 | |
| *** kinrui is now known as fungi | 22:19 | |
| *** SpamapS has joined #openstack-infra | 22:19 | |
| *** irclogbot_2 has quit IRC | 22:19 | |
| fungi | whee, freenode asplodey | 22:19 |
| corvus | clarkb: there is a bug in that change which will cause it to fail zuul-quick-start, but we can still learn what we need to about the registry changes, so i'm holding a new patchset to fix it until the current run finishes | 22:19 |
| clarkb | corvus: ok | 22:20 |
| corvus | (missed a zuul-ci.org -> opendev.org change) | 22:20 |
| *** irclogbot_2 has joined #openstack-infra | 22:20 | |
| *** Qiming has joined #openstack-infra | 22:21 | |
| corvus | my gitea change to add .git support is failing tests; i will try to get to it this week, but if anyone wants to hack go, https://github.com/go-gitea/gitea/pull/6688 | 22:21 |
| corvus | (BTW, that should really be two different PRs with a dependency, but github doesn't support that) | 22:22 |
| *** sshnaidm has joined #openstack-infra | 22:24 | |
| *** sshnaidm is now known as sshnaidm|afk | 22:24 | |
| clarkb | corvus: so I think the build image job worked this time and now we are waiting on a test node for the quickstart job | 22:25 |
| clarkb | if quickstart pulls images then I htink we are set | 22:25 |
| Weifan | excuse me, our repo has been moved from openstack namespace to x, will the package still be uploaded to pypi automatically when a new tag is pushed, or should we manually do it? | 22:29 |
| clarkb | Weifan: the package name on pypi should be independent of the namespace and things should continue to work as before | 22:29 |
| Weifan | thanks :) | 22:29 |
| corvus | clarkb: the push and pull aspects of this are under-tested because there isn't a change before or after this one; we'll test the push-to-intermediate in the second half of the build-image job, but not the pull-from-intermediate | 22:31 |
| corvus | clarkb: so actually, how about i push up my change on its own and have it depends-on mordred's change | 22:31 |
| corvus | then we'll see the pull exercised | 22:31 |
| clarkb | corvus: wfm | 22:32 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix another opendev rename https://review.opendev.org/654643 | 22:33 |
| corvus | so 654238 should exercise push-to-intermediate, 654643 should exercise pull-from-intermediate; they should both exercise use-buildset-registry and the /etc/hosts stuff, and the second should fix all known errors and be green | 22:34 |
| *** ijw has joined #openstack-infra | 22:34 | |
| *** rfarr has quit IRC | 22:36 | |
| ijw | Hey, I might be a bit late with a daft question, but I'll ask anyway. I see there's a redirect on the review.openstack URLs, but the filters I have in my bookmarks are screwy because project names have changed from openstack/... to x/... - is that a permanent change? I'll update my bookmarks if so. | 22:37 |
| clarkb | ijw: it is permanent as long as those projects don't ask to get migrated into new namespaces | 22:37 |
| corvus | as permanent as any project repo url :) | 22:38 |
| ijw | OK. The redirector would probebly be more useful to the next people along if it made a comment in passing, if that's not too difficult a request | 22:40 |
| clarkb | ijw: I'm not sure I understand what you mean by that? we did redirect everything on the public mirror side | 22:40 |
| corvus | clarkb: push-to-intermediate on 654238 looks good | 22:43 |
| ijw | clarkb: You redirected a URL which used to work (review.openstack.org, project:openstack/*) to one that didn't without rewriting the querystring to fix it | 22:43 |
| clarkb | ijw: can you share examples of the original url and the resulting url? I don't think we rewrote any review.openstack.org urls | 22:44 |
| clarkb | (just the domain) | 22:44 |
| *** rcernin has quit IRC | 22:45 | |
| ijw | clarkb: https://review.openstack.org/#/q/project:openstack/networking-vpp+status:open -> https://review.opendev.org/#/q/project:openstack/networking-vpp+status:open | 22:45 |
| *** rcernin has joined #openstack-infra | 22:45 | |
| clarkb | ya ok that was expected. gerrit doesn't have proper renaming tools so we aren't able to sanely redirect everything in gerrit. This is one of many reasons we push people towards other tools for browsing and fetching | 22:46 |
| clarkb | we expect that our developers will be able to manage if it happens infrequently | 22:46 |
| clarkb | (and hopefully we can eventually write that gerrit rename plugin and have it do a sane thing around this) | 22:47 |
| paladox | clarkb i think there's a plugin to rename projects in gerrit (but it's not future proof). | 22:47 |
| paladox | https://gerrit-review.googlesource.com/admin/repos/plugins/rename-project | 22:48 |
| clarkb | paladox: there is but its broken :P | 22:48 |
| clarkb | paladox: it deletes the project and creats a new one | 22:48 |
| clarkb | so you lose all the existing changes and stuff | 22:48 |
| paladox | oh | 22:48 |
| paladox | in notedb you can easily rename, just change the name of the repo then reindex. | 22:48 |
| ijw | All I was thinking is maybve you could somehow flash up a message during the redirect or on the resulting page | 22:49 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Double stack size on gitea https://review.opendev.org/654634 | 22:49 |
| ijw | Plus you get to tell people at the same time 'the project names have changed' | 22:50 |
| clarkb | corvus: ^ small fix that will hopefully make the test pass | 22:50 |
| clarkb | ijw: got it, ya wecould possibly do that | 22:50 |
| *** jcoufal has quit IRC | 22:50 | |
| clarkb | paladox: that is excellent news | 22:50 |
| clarkb | paladox: and more motivation to upgrade I guess | 22:51 |
| paladox | :) | 22:51 |
| fungi | paladox: does changing the repo name in notedb migrate all user preferences/watches for it or is that stored separately still? we currently do update queries during renames to switch those to the new project name | 22:51 |
| paladox | Oh, that im not sure about. (i haven't personally tryed that). | 22:51 |
| fungi | though also, the "doesn't have proper renaming tools" in this case seemed to be more about the fact that we don't have a mechanism inside gerrit to magically redirect requests for an old name to a new one. doesn't sound like the new situation is any better | 22:53 |
| fungi | in that regard i mean | 22:54 |
| *** ijw has quit IRC | 22:55 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't log setting up creds for the intermediate registry https://review.opendev.org/654644 | 22:55 |
| corvus | clarkb, fungi: ^ let's er, merge that and rekey | 22:55 |
| fungi | oops ;) | 22:55 |
| corvus | yep. i don't think it's a big deal as long as we don't build and publish any images until the rekey | 22:56 |
| corvus | or rather, "publish". building is fine. | 22:56 |
| fungi | single-core approved that one for obvious reasons | 22:56 |
| clarkb | I +2'd too | 22:56 |
| *** tkajinam has joined #openstack-infra | 22:57 | |
| corvus | after it lands, we should put a new password in hostvars and once that's applied, restart the intermediate registry | 22:57 |
| paladox | fungi just tested, nope doesn't automatically rename preferences/watches. I also found you have to reindex offline. (it hit a null pointer when running the online index) | 22:57 |
| fungi | ouch | 22:58 |
| *** yamamoto has joined #openstack-infra | 22:58 | |
| fungi | at least our current solution with offline renaming can be done with an outage on the order of a minute or two, and then take advantage of online reindexing thereafter | 22:58 |
| fungi | though we'll obviously have to revisit a good chunk of it when we upgrade and switch database implementations | 22:59 |
| fungi | so no idea how many of those properties will still hold once that's done | 23:00 |
| paladox | fungi oh | 23:01 |
| paladox | aha | 23:01 |
| paladox | if you copy the project first it should work | 23:01 |
| *** yamamoto has quit IRC | 23:01 | |
| paladox | So ie: cp -R test test2 (then ssh -p 29418 <host> gerrit index start projects --force) | 23:02 |
| corvus | clarkb: pull-from ran without error | 23:02 |
| clarkb | corvus: \o/ | 23:02 |
| fungi | paladox: cp instead of mv? do you rm the original afterward? | 23:03 |
| * paladox test rm | 23:03 | |
| corvus | clarkb: so i think all the ipv6 related changes have been exercised; the only unknown is whether that's the final fix for quickstart | 23:03 |
| paladox | ah | 23:03 |
| paladox | fungi rm after yp | 23:03 |
| paladox | *yup | 23:03 |
| paladox | that works | 23:03 |
| fungi | neat! so there's hope that we can do online renames, perhaps with some caveats around user preferences | 23:04 |
| *** threestrands has joined #openstack-infra | 23:04 | |
| paladox | Yeh, user preferences are stored in All-Users now | 23:04 |
| fungi | we could probably push updates to that | 23:05 |
| paladox | yup | 23:05 |
| fungi | sounds like we have at least a semblance of a plan for better renames with notedb in that case | 23:05 |
| corvus | paladox: if there are 2 copies of the project, won't there be duplicate change numbers? | 23:05 |
| fungi | does notedb still use change numbers? | 23:06 |
| corvus | istr change numbers are stored as refs in notedb | 23:06 |
| paladox | i doin't think so | 23:06 |
| paladox | corvus gerrit now includes the project name in the url | 23:06 |
| corvus | i last looked > 6 months ago, so that could be new since then | 23:07 |
| fungi | got it, so change numbers are no longer assumed to be globally unique | 23:07 |
| fungi | i wonder how they kept that backward-compatible with old change urls | 23:07 |
| corvus | ah like this? https://gerrit-review.googlesource.com/c/gerrit/+/221245 | 23:08 |
| fungi | maybe non-unique change numbers require the project in the url, and a projectless change url gives you a change list like a query result for a duplicatyed change-id does | 23:08 |
| paladox | fungi corvus from what i just tested, it won't show the changes from the new repo until you delete the old repo. | 23:09 |
| *** tosky has quit IRC | 23:09 | |
| corvus | fungi: it works somehow: https://gerrit-review.googlesource.com/153111/ | 23:09 |
| paladox | corvus it looks up the project name internally | 23:09 |
| fungi | nifty | 23:11 |
| *** ijw has joined #openstack-infra | 23:11 | |
| corvus | clarkb: some more quick-start errors, i'll fix them and squash everything into mordred's change | 23:11 |
| clarkb | corvus: ok let me know if I can help otehrwise I'm finally catching up on email related stuff | 23:11 |
| clarkb | sending out the infra meeting agenda etc | 23:11 |
| corvus | clarkb: but i think we're good on the job changes | 23:11 |
| clarkb | this is excellent news | 23:12 |
| paladox | ah | 23:13 |
| paladox | i see what it does now | 23:13 |
| paladox | fungi corvus, apprently gerrit's smart and will only show one of the repos with the same change number (for example i have rename and rename2, it will only show rename due to it being higher then rename2). | 23:14 |
| fungi | "higher" as being older, or appearing earlier alphabetically, or...? | 23:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Update references for opendev https://review.opendev.org/654238 | 23:15 |
| *** rcernin has quit IRC | 23:15 | |
| paladox | fungi order | 23:15 |
| *** rcernin has joined #openstack-infra | 23:16 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Don't log setting up creds for the intermediate registry https://review.opendev.org/654644 | 23:18 |
| clarkb | corvus: ^ you driving the rotation? | 23:19 |
| *** aaronsheffield has quit IRC | 23:19 | |
| corvus | clarkb: i'll update the creds now | 23:19 |
| corvus | and i'll kick intermediate-ci-registry | 23:21 |
| corvus | er, insecure-ci-registry | 23:21 |
| corvus | whatever we called it | 23:21 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Double stack size on gitea https://review.opendev.org/654634 | 23:23 |
| clarkb | another fix for the test. Apparently testinfra doesn't expose stdout_lines just stdout | 23:24 |
| clarkb | I think the check should still work though | 23:24 |
| corvus | clarkb: should i remove all the data from the registry since i'm restarting it anyway | 23:24 |
| corvus | at 89% | 23:24 |
| clarkb | corvus: ++ that will give us breathing room | 23:24 |
| clarkb | and ensures we won't publish something there | 23:24 |
| *** mriedem has quit IRC | 23:24 | |
| *** mriedem has joined #openstack-infra | 23:25 | |
| fungi | yeah, good thinking | 23:26 |
| *** sthussey has quit IRC | 23:28 | |
| corvus | okay, cleared and restarted | 23:29 |
| clarkb | tyty | 23:29 |
| corvus | issued a recheck on my swift patch (which may have been failing due to issues since fixed, or could be broken) | 23:30 |
| paladox | fungi clarkb though when we first migrated to NoteDB, it almost caused a db outage for phabricator (and possibly gerrit) due to writes. | 23:30 |
| clarkb | corvus: you may want to review my small edits on https://review.opendev.org/654634 too | 23:30 |
| paladox | We had to hotfix our phab install to ignore refs/changes/**/**/meta | 23:30 |
| paladox | And oh man the repo's grew huge due to just refs/changes/**/**/meta. | 23:31 |
| corvus | clarkb: why change stdout_lines to stdout? | 23:31 |
| corvus | clarkb: (that was one of the things i fixed from my first patch) | 23:31 |
| clarkb | corvus: the test failed because the return object doesn't have stdout_lines | 23:31 |
| clarkb | https://github.com/philpep/testinfra/blob/master/testinfra/backend/base.py#L31 is the code and http://logs.openstack.org/34/654634/2/check/system-config-run-gitea/84b8900/job-output.txt.gz#_2019-04-22_23_12_06_415508 the error | 23:31 |
| corvus | clarkb: well.. the first change failed because stdout isn't iterable | 23:31 |
| clarkb | oh hrm | 23:32 |
| corvus | clarkb: i guess we need a third way :) | 23:32 |
| clarkb | sounds like it | 23:32 |
| fungi | paladox: good to know, thanks for the warning | 23:32 |
| paladox | https://phabricator.wikimedia.org/T196840 <-- that was task | 23:32 |
| paladox | and your welcome :) | 23:32 |
| corvus | clarkb: apparently we should do cmd.stdout.split('\n'): | 23:33 |
| clarkb | corvus: that seems right | 23:33 |
| clarkb | or at least not wrong | 23:33 |
| corvus | i'll push up a new ps with that? | 23:33 |
| clarkb | ++ | 23:34 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Double stack size on gitea https://review.opendev.org/654634 | 23:34 |
| clarkb | fungi: ^ one last time hopefully | 23:35 |
| fungi | i'm curious why that didn't just match the substring in the splatted output stream | 23:36 |
| clarkb | type difference maybe? | 23:37 |
| fungi | they aren't both str? | 23:37 |
| clarkb | they definitely appear to be | 23:38 |
| *** yamamoto has joined #openstack-infra | 23:38 | |
| fungi | "foo" in "foobarbaz" evaluates true | 23:39 |
| fungi | as does "bar" in "foobarbaz" | 23:40 |
| clarkb | maybe the whitespace counts were off before? | 23:40 |
| fungi | i mean, the expected value is declared as a str explicitly, and for out.split('\n') to work it presumably is too | 23:41 |
| clarkb | http://logs.openstack.org/29/654629/1/check/system-config-run-gitea/905340d/job-output.txt.gz#_2019-04-22_21_24_39_434931 is the old error | 23:42 |
| clarkb | I think it is a quoting issue | 23:42 |
| clarkb | need a closing quote in the first line of the assert | 23:42 |
| clarkb | so its possible that both pevious ps and current one will work. Current one is maybe a bit more explicit that this is a separate line | 23:42 |
| fungi | TypeError: argument of type 'CommandResult' is not iterable | 23:45 |
| fungi | ahh, maybe it simply needs to be coerced with str() | 23:45 |
| clarkb | logan-: I know you were following along on the docker + ipv6 stuff before. Wanted to point out that we ended up using socat to rewrite ipv4 to ipv6 connections in front of docker as the fix | 23:52 |
| *** lseki has quit IRC | 23:52 | |
| clarkb | logan-: not sure if you have similar problems in your infrastructure but that may be useful info :) | 23:52 |
| fungi | the case of needing to use a docker registry with a raw ipv6 address and no dns is presumably a very unusual corner case, given the amount of head-scratching it has caused | 23:53 |
| clarkb | fungi: corvus now failing on not enough whitespace I think | 23:56 |
| clarkb | should I update it | 23:56 |
| clarkb | ? | 23:56 |
| fungi | may as well | 23:57 |
| clarkb | ok one moment | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
