Friday, 2019-05-03

openstackgerritPaul Belanger proposed openstack/diskimage-builder master: Use fedora-release-common for fedora 30+  https://review.opendev.org/65690500:00
pabelangerlets see if it is that simple00:00
*** Weifan has quit IRC00:00
*** Weifan has joined #openstack-infra00:02
*** mriedem has quit IRC00:03
*** dciabrin has quit IRC00:03
openstackgerritPaul Belanger proposed opendev/system-config master: Mirror fedora 30 for jobs  https://review.opendev.org/65690600:05
*** eumel8 has quit IRC00:07
ianwpabelanger: urgh, i'd say fedora's full ... http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1 ... i'll look into that in a big00:08
ianwbit00:08
pabelangerack00:09
*** hamzy has joined #openstack-infra00:10
pabelangerianw:so, I uncapped DIB today, but still get the following install problem with fedora-29: http://paste.openstack.org/show/750347/00:10
pabelangerI'm not sure why we don't see that issue here00:10
*** Weifan has quit IRC00:10
ianwi think *sometimes* we do ... i can't understand why it's non-deterministic00:11
pabelangeroh, so eventually you build a good image?00:11
*** slaweq has joined #openstack-infra00:13
ianwi think i've just seen it in our CI00:13
pabelangerianw: maybe we should check to see if dbus-daemon is enabled, and skip?00:14
pabelangercause right now, I can't get a fedora-29 image to build00:15
ianwi'm trying to remember what it's about ... there's dbus-broker and dbus-daemon right?  one's on demand or something and the other, well a daemon00:15
pabelangeryah, I think that is right00:16
pabelangersomething is still pulling in dbus-daemon, from what I see in logs00:16
ianwbut dbus-broker is the new default?00:16
pabelangeryah00:16
pabelangerdbus still pulls in dbus-daemon00:16
pabelangerianw:but, I don't know when dbus-broker became the default00:17
ianwpabelanger: so how are we getting dbus-daemon?  https://opendev.org/openstack/diskimage-builder/commit/7cb5916a7600b0bd4cd26c9f254cab71778b170200:18
ianwmy thought is that dbus-broker is pulled in for initial chroot, and then that should satisfy any dbus requirements00:19
pabelangerianw:http://paste.openstack.org/show/750348/00:20
pabelangerianw: Oh, so I think we install dbus before dbus-broker00:21
pabelangerand for some reason, that still depends on dbus-daemon00:21
pabelangerbut, why doesn't nodepool.o.o hit this00:22
*** e0ne has quit IRC00:22
ianwdbus-daemon does come in ... https://nb02.openstack.org/fedora-29-0000038820.log00:23
*** slaweq has quit IRC00:23
*** igordc has quit IRC00:24
pabelangeryah00:24
pabelangerystemctl --no-reload enable dbus-broker.service works for nodepool.o.o00:24
pabelangerbut me fails00:24
pabelangerlet me find the spec file for dbus-daemon00:25
openstackgerritIan Wienand proposed zuul/zuul-jobs master: tox siblings install: handle no metadata in setup.cfg  https://review.opendev.org/65690800:27
*** bhavikdbavishi has quit IRC00:28
openstackgerritIan Wienand proposed opendev/system-config master: [wip] Use testinfra from master in devel job  https://review.opendev.org/65687700:28
openstackgerritIan Wienand proposed opendev/system-config master: [wip] Use testinfra from master in devel job  https://review.opendev.org/65687700:29
*** gyee has quit IRC00:30
pabelangerianw: http://paste.openstack.org/show/750349/ that is the install scripts for dbus-daemon rpm00:31
ianwpreset ... that's not quite enable is it?00:35
*** Lucas_Gray has quit IRC00:37
pabelangerianw: if I understand, preset falls back to distro policy to then enable / disable said service00:40
pabelangerso, while dbus-daemon could be preset, it might not actually start before of policy00:40
pabelanger90-default.preset:enable dbus-broker.service00:41
pabelangerthat is on local fedora-3000:41
pabelanger/lib/systemd/system-preset00:41
pabelangerso, I wonder if what we are doing in: https://opendev.org/openstack/diskimage-builder/commit/7cb5916a7600b0bd4cd26c9f254cab71778b1702 is the right patch here00:42
pabelangerwe should maybe make sure that /lib/systemd/system-preset/90-default.preset has enable-dbus-broker00:42
pabelangerianw: that should be in fedora-release00:44
ianwyeah, fedora-release-common on my f30 system ... i wonder if the two issues are related ...00:45
pabelangerianw: https://src.fedoraproject.org/rpms/fedora-release/blob/f29/f/90-default.preset#_1600:45
pabelangerso, I think we might have jumped the gun on dbus-broker on f2900:46
pabelangerhttps://src.fedoraproject.org/rpms/fedora-release/blob/f30/f/90-default.preset#_1600:46
pabelangerlooks like dbus-broker is fedora-30+00:47
ianwhrm, the problem was we had no dbus starting which resulted in no networking and no login00:48
pabelangerianw: Hmm, I have no issue building fedora-29 on 2.19.0, which doesn't have that patch00:49
*** _erlon_ has quit IRC00:49
pabelangerso maybe it was something else?00:49
pabelangeror now fixed?00:49
pabelangerI'm just reverting again locally, and confirming00:50
ianwmaybe it was fixed?00:50
pabelangermaybe00:50
pabelangerif we revert, nodepool dsvm test on fedora should confirm00:51
pabelangerotherwise, we'll have broken network00:51
openstackgerritIan Wienand proposed opendev/system-config master: [wip] Use testinfra from master in devel job  https://review.opendev.org/65687700:52
ianwyeah, i'd have to go back and remember what was failing, i assume it was that00:54
openstackgerritPaul Belanger proposed openstack/diskimage-builder master: Revert "Enable dbus-broker for Fedora 29"  https://review.opendev.org/65690900:54
pabelangerianw: k, lets see what testing says00:55
pabelangerianw: okay cool, https://review.opendev.org/656838/ is green. That gets us glean under python3 for fedora00:57
pabelangercentos-7 just did pip install, fedora-29 pip3 install00:58
*** Weifan has joined #openstack-infra01:01
*** lseki has quit IRC01:02
*** Weifan has quit IRC01:10
ianw#status log bumped fedora.mirror quota to 800000000 k01:15
openstackstatusianw: finished logging01:15
*** jbadiapa has joined #openstack-infra01:16
ianw28 being kind of special given rhel8 branching we might keep that for a bit01:16
ianwrunning update manually since it's been hitting quota for a while01:18
*** bobh has joined #openstack-infra01:21
pabelangerack01:23
*** Weifan has joined #openstack-infra01:25
*** Weifan has quit IRC01:26
*** Weifan has joined #openstack-infra01:27
*** bobh has quit IRC01:31
*** Weifan has quit IRC01:31
*** tjgresha has joined #openstack-infra01:45
*** tjgresha has quit IRC01:50
*** apetrich has quit IRC01:57
*** whoami-rajat has quit IRC01:58
*** michael-beaver has quit IRC01:58
pabelangerianw: so we build a fedora-30 image, it just doesn't boot. Looks grub related: http://logs.openstack.org/05/656905/2/check/nodepool-functional-py35-redhat-src/2b93322/controller/logs/screen-nodepool-launcher.txt.gz#_May_03_00_56_07_39219202:01
openstackgerritPaul Belanger proposed openstack/diskimage-builder master: Use fedora-release-common for fedora 30+  https://review.opendev.org/65690502:02
pabelangerI'll do the depends on the other way, so we can stack patches on ^02:03
ianwpabelanger: there was something in the faq i hit the other day about grub boot menus ...02:06
ianwhttps://fedoraproject.org/wiki/Common_F30_bugs#GRUB_boot_menu_is_not_populated_after_an_upgrade02:07
ianwwe might be fiddling grub the wrong way now :/02:07
*** jbadiapa has quit IRC02:08
openstackgerritIan Wienand proposed opendev/system-config master: Add NO_TIMEOUT for mirror update scripts  https://review.opendev.org/65691202:20
*** rh-jelabarre has quit IRC02:23
*** ykarel|away has joined #openstack-infra02:23
*** dklyle has joined #openstack-infra02:30
*** dklyle has quit IRC02:36
*** gmann has quit IRC02:43
*** yamamoto has joined #openstack-infra02:52
*** jbadiapa has joined #openstack-infra02:53
*** jbadiapa has quit IRC02:58
*** yamamoto has quit IRC03:04
*** otherwiseguy has quit IRC03:08
*** bhavikdbavishi has joined #openstack-infra03:09
*** otherwiseguy has joined #openstack-infra03:10
*** hongbin has joined #openstack-infra03:17
*** yamamoto has joined #openstack-infra03:17
*** yamamoto has quit IRC03:17
*** yamamoto has joined #openstack-infra03:21
*** yamamoto has quit IRC03:21
*** yamamoto has joined #openstack-infra03:21
*** efried has joined #openstack-infra03:25
*** gmann has joined #openstack-infra03:28
*** psachin has joined #openstack-infra03:33
*** sarob has joined #openstack-infra03:34
*** ykarel|away has quit IRC03:34
*** psachin has quit IRC03:35
*** sarob has quit IRC03:36
*** psachin has joined #openstack-infra03:36
*** sarob has joined #openstack-infra03:36
*** yamamoto has quit IRC03:41
*** sshnaidm has joined #openstack-infra03:43
*** sarob has quit IRC03:45
*** yamamoto has joined #openstack-infra03:46
*** armax has quit IRC03:50
*** hongbin has quit IRC03:54
*** e0ne has joined #openstack-infra04:06
*** pcaruana has joined #openstack-infra04:12
*** ykarel has joined #openstack-infra04:17
*** ykarel_ has joined #openstack-infra04:20
*** ykarel has quit IRC04:23
*** pcaruana has quit IRC04:32
*** udesale has joined #openstack-infra04:35
*** raukadah is now known as chandankumar04:46
*** samueldmq has quit IRC04:46
*** Lucas_Gray has joined #openstack-infra04:51
*** janki has joined #openstack-infra04:54
*** yamamoto has quit IRC05:03
*** hwoarang has quit IRC05:05
*** hwoarang has joined #openstack-infra05:07
*** e0ne has quit IRC05:15
*** yamamoto has joined #openstack-infra05:18
*** Lucas_Gray has quit IRC05:22
*** yamamoto has quit IRC05:28
*** yamamoto has joined #openstack-infra05:31
*** yamamoto has quit IRC05:36
fungiianychoi: i've added you to the chanserv access list for #openstack-i18n so you should now be able to set the channel topic with something like `/msg chanserv topic #openstack-i18n This is our new topic`05:40
ianychoifungi, thanks a lot - sorry that I couldn't join in this PTG physically, and enjoy PTG :)05:41
fungithanks! your presence is missed, and i hope to see you sometime soon!05:42
ianychoiThanks too! sure :)05:44
*** lpetrut has joined #openstack-infra05:46
*** yamamoto has joined #openstack-infra05:53
*** bradm has joined #openstack-infra05:53
*** yamamoto has quit IRC05:58
*** kjackal has joined #openstack-infra06:03
*** yamamoto has joined #openstack-infra06:13
*** kjackal has quit IRC06:14
*** lpetrut has quit IRC06:18
*** markvoelker has quit IRC06:20
*** yamamoto has quit IRC06:21
*** yamamoto has joined #openstack-infra06:21
*** yamamoto has quit IRC06:24
*** yamamoto has joined #openstack-infra06:24
*** jtomasek has joined #openstack-infra06:31
*** pcaruana has joined #openstack-infra06:36
openstackgerritTim Burke proposed zuul/zuul-jobs master: Better handle packages without [metadata]name option  https://review.opendev.org/65692506:42
*** yamamoto has quit IRC06:43
*** yamamoto has joined #openstack-infra06:43
*** yamamoto has quit IRC06:44
*** yamamoto has joined #openstack-infra06:53
*** ginopc has joined #openstack-infra07:01
*** kopecmartin|off is now known as kopecmartin07:05
*** shardy has joined #openstack-infra07:05
*** tesseract has joined #openstack-infra07:06
AJaegerianw, timburke , you both fix the same problem with https://review.opendev.org/#/c/656925/1 and https://review.opendev.org/#/c/656908 . Could you review each other's changes and decide on how to move forward, please?07:12
*** kjackal has joined #openstack-infra07:19
*** pcaruana has quit IRC07:22
openstackgerritAndreas Jaeger proposed openstack/project-config master: Check groups entry in gerrit/projects  https://review.opendev.org/65692607:23
openstackgerritAndreas Jaeger proposed openstack/project-config master: DNM: Test groups list check  https://review.opendev.org/65692707:23
AJaegerfungi, this should avoid groups errors coming like those you patched yesterday ^07:23
*** markvoelker has joined #openstack-infra07:25
*** lpetrut has joined #openstack-infra07:30
*** bhavikdbavishi has quit IRC07:35
*** rcernin has quit IRC07:36
*** jpich has joined #openstack-infra07:37
*** roman_g has quit IRC07:43
*** apetrich has joined #openstack-infra07:44
*** tosky has joined #openstack-infra07:46
AJaegerstephenfin: Is it spring cleaning time? Thanks!07:47
*** jpena|off is now known as jpena07:48
*** kjackal_v2 has joined #openstack-infra07:54
*** kjackal has quit IRC07:54
*** bhavikdbavishi has joined #openstack-infra07:57
*** ykarel_ is now known as ykarel08:00
*** ykarel is now known as ykarel|lunch08:01
*** altlogbot_0 has quit IRC08:03
*** altlogbot_0 has joined #openstack-infra08:04
*** kjackal has joined #openstack-infra08:25
*** kjackal_v2 has quit IRC08:25
*** rpittau|afk is now known as rpittau08:39
*** ykarel|lunch is now known as ykarel08:49
*** pcaruana has joined #openstack-infra08:56
*** gmann has quit IRC08:57
*** kjackal has quit IRC09:01
*** kjackal has joined #openstack-infra09:01
*** whoami-rajat has joined #openstack-infra09:09
*** gthiemonge has joined #openstack-infra09:13
*** jbadiapa has joined #openstack-infra09:15
*** Lucas_Gray has joined #openstack-infra09:16
*** Lucas_Gray has quit IRC09:31
*** kjackal has quit IRC09:36
*** kjackal has joined #openstack-infra09:36
*** Lucas_Gray has joined #openstack-infra09:37
*** ginux has joined #openstack-infra09:49
*** ginopc has quit IRC09:52
*** ginux is now known as ginopc09:52
*** yolanda_ has quit IRC09:54
*** kjackal has quit IRC10:10
*** roman_g has joined #openstack-infra10:16
*** jbadiapa has quit IRC10:21
*** pcaruana has quit IRC10:22
*** jbadiapa has joined #openstack-infra10:26
*** yamamoto has quit IRC10:33
*** lpetrut has quit IRC10:51
*** d34dh0r53 has quit IRC10:52
*** cloudnull has quit IRC10:53
*** eglute has quit IRC10:54
*** ginopc has quit IRC10:57
*** ginopc has joined #openstack-infra10:57
*** pcaruana has joined #openstack-infra11:01
*** lpetrut has joined #openstack-infra11:05
*** yamamoto has joined #openstack-infra11:09
*** yamamoto has quit IRC11:14
*** kaisers has quit IRC11:25
*** bhavikdbavishi has quit IRC11:30
*** rascasoft has quit IRC11:32
*** kaisers has joined #openstack-infra11:34
*** kjackal has joined #openstack-infra11:34
*** jpena is now known as jpena|lunch11:35
*** rascasoft has joined #openstack-infra11:38
openstackgerritIan Wienand proposed opendev/system-config master: [wip] Use testinfra from master in devel job  https://review.opendev.org/65687711:46
*** shardy has quit IRC11:48
*** kjackal has quit IRC11:50
*** kjackal has joined #openstack-infra11:50
*** yamamoto has joined #openstack-infra11:54
*** e0ne has joined #openstack-infra11:55
*** imacdonn has quit IRC11:57
*** imacdonn has joined #openstack-infra11:57
*** xek has quit IRC12:01
*** yamamoto has quit IRC12:02
gthiemongeHi folks, I have random 404 errors on opendev.org: http://logs.openstack.org/00/656300/1/check/octavia-v1-dsvm-scenario/f932553/job-output.txt.gz#_2019-05-03_07_38_10_29847112:10
*** rh-jelabarre has joined #openstack-infra12:11
gthiemongehttp://paste.openstack.org/show/750358/12:11
pabelangerianw: thanks, will look today to see if I find anything obvious12:11
ianwpabelanger: thanks12:13
ianwpabelanger: i *just* got a good ansible-runner+testinfra run : http://logs.openstack.org/77/656877/5/check/system-config-run-base-ansible-devel/a50c4ad/12:13
ianwi believe this to now be on par with the direct calls, although it might be slower12:13
*** sthussey has joined #openstack-infra12:28
*** udesale has quit IRC12:32
*** jpena|lunch is now known as jpena12:33
*** udesale has joined #openstack-infra12:33
fricklerinfra-root: I can reproduce gthiemonge's errors. I do not see any errors when I curl from gitea0[1-8] directly. so possibly the haproxy is having issues. I don't have time to dig more, maybe someone can pick this up12:34
*** whoami-rajat has quit IRC12:38
*** rlandy has joined #openstack-infra12:41
roman_gclarkb: Hello Clark. Could we have repos on GitHub marked as abandoned with information on where to find new repositories? / Airship12:44
*** rfarr_ has joined #openstack-infra12:46
AJaegerroman_g: can you give an example, please?12:47
*** jbadiapa has quit IRC12:50
*** yamamoto has joined #openstack-infra12:55
*** whoami-rajat has joined #openstack-infra12:58
*** yamamoto has quit IRC13:02
*** rosmaita has joined #openstack-infra13:02
*** rosmaita has left #openstack-infra13:02
roman_gAJaeger: https://github.com/openstack/?q=airship13:03
roman_gSyncing repos to GitHub no longer works, as repositories are renamed13:03
roman_gAll of them have last update date over 2 weeks13:04
roman_gBut in fact the work continues here: https://opendev.org/airship/13:04
AJaegerroman_g: I see, you might need to ask again next week due to the summit - or send an email to openstack-infra mailing list. I agree, that is one of the followup tasks to be done and I don't know the plan for that13:05
roman_gExample of an "Archived" repository on GitHub: https://github.com/att-comdev/treasuremap - note the yellow notice line and description saying that project has moved13:06
roman_gAJaeger: all right, thank you.13:06
*** yamamoto has joined #openstack-infra13:07
Shrewsclarkb: infra-root: looks like the builder on nb2 was killed due to memory usage. I see the other builders are pretty maxed out right now too. I'm concerned by this memory issue suddenly showing up on both launchers and builders.13:10
*** bdodd has joined #openstack-infra13:12
clarkbfrickler: gthiemonge I wouldnt expect http 404s from an unhappy proxy. Maybe ~1-2 backends are unhappy13:14
clarkbroman_g: AJaeger I think we are waiting for the tc to decide that archiving those is appropriate13:15
AJaegerclarkb: those are airship repos and outside of TC control, aren't they?13:15
clarkbAJaeger: they arein the openstack org on github though13:16
AJaegerwhich is owned by TC - I see...13:16
clarkbif airship wantsto move them out and set up replication jobs they can do that13:16
fricklerclarkb: that was my first idea, but checking them individually with e.g. https://gitea01.opendev.org:3000/... doesn't give any errors for me on any of them13:17
jrollAJaeger: clarkb: roman_g: there's been no objections to archiving these repos, so we're going to do it. I just didn't want to try to talk people into scripting that until after PTG13:17
jrollso probably next week13:17
jrollhere's the ML thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005629.html13:18
*** Goneri has joined #openstack-infra13:19
*** markvoelker has quit IRC13:21
*** janki has quit IRC13:22
*** e0ne has quit IRC13:22
AJaegerjroll: I like roman_g 's idea to adjust the project description to point to the new location, e.g say for project "x/y": "The repo now lives at opendev.org/x/y". roman_g want to propose that as followup to jroll's email?13:23
jrollsure, that'd be fine13:24
*** _erlon_ has joined #openstack-infra13:26
*** kjackal has quit IRC13:30
*** kjackal has joined #openstack-infra13:33
clarkbfrickler: infra-root looks like at least gitea06 404s on that url13:35
clarkbya I think gitea06 is unhappy. the others return 200 for that url13:36
*** yamamoto has quit IRC13:36
clarkbI've got to get ready for ptg'ing but imagine looking into ^ is something we can do there13:37
*** markvoelker has joined #openstack-infra13:40
clarkblooks like the stable stein branch for openstack requiremeents is a 404 so not just that one file13:41
*** gmann has joined #openstack-infra13:51
*** yamamoto has joined #openstack-infra14:02
*** lpetrut has quit IRC14:04
*** e0ne has joined #openstack-infra14:05
mordredclarkb: gitea-web_1  | 2019/05/03 14:05:42 [...ules/context/repo.go:602 func1()] [E] RepoRef invalid repo: branch or tag not exist:14:06
mordredgitea-web_1  | [Macaron] 2019-05-03 14:05:42: Completed HEAD /openstack/requirements/raw/branch/stable/stein/upper-constraints.txt 404 Not Found in 15.88504ms14:06
mordredclarkb: I agree - that's logs from gitea on 06 when trying to get openstack/requirements14:06
fricklermordred: how did you create/look at at that log?14:09
mordredfrickler: there's two ways - you can cd /etc/gitea-docker and run 'docker-compose logs --tail=X' (where X is a number of lines ... without X it'll give you logs since the beginning of time)14:10
mordredfrickler: second, you can run docker ps to find the containers that are running, then run docker logs on one of them - so like "docker logs --since=5m giteadocker_gitea-web_1"14:11
*** efried has quit IRC14:12
fricklermordred: nice, thx14:14
clarkbcan we trigger replication on openstack/requirements to see if that fixes it14:16
*** jpich has quit IRC14:18
*** psachin has quit IRC14:19
*** jpich has joined #openstack-infra14:19
ykarelnow the gitweb link is not visible in gerrit, earlier it was visible but not working, and it was said after restarting(as config changed to use internal gitweb) it will work14:19
ykarel#infra-root is ^^ known14:19
pabelangerclarkb: do you mind reviewing: https://review.opendev.org/656838/ that updates glean to use python3, when possible. In my case downstream, trying to build images with no python2.14:21
clarkbykarel: ues the restart ended up removing the links due to a second missong config14:22
clarkbwe have since added that config and another restart should actually fix the links rather than removong them14:22
ykarelclarkb, okk Thanks for confirming14:22
clarkbI'm going to trigger replication on openstack/requirements now14:27
clarkbalso I've camped at the coffee shop as you enter the convention center14:28
clarkbthe lighting is a bit nicer and there is some power14:28
*** sshnaidm has quit IRC14:29
clarkbreplication has not fixed the 404 issue14:30
*** efried has joined #openstack-infra14:31
clarkbif I clone from gitea06 I don't get a remotes/origin/stable/stein14:33
*** mriedem has joined #openstack-infra14:34
clarkbgitea06:/var/gitea/data/git/repositories/openstack/requirements.git/refs/heads/stable/stein looks like binary junk data14:34
clarkbI imagine we might be able to copy that file from another host14:35
clarkbhrm dmesg shows a really unhappy fs14:36
clarkbI think that is likly the underlying cause. We might want ot consider removing this host from the load balancer and replacing it14:36
clarkbmnaser: ^ is that of interest to you before we delete the host? I believe it is a booted from volume ubuntu bionic node running ext4 on vexxhost scj114:38
*** ricolin has joined #openstack-infra14:39
*** jpich has quit IRC14:39
*** jpich has joined #openstack-infra14:39
clarkbpabelanger: is glean known to work under python3?14:40
*** e0ne has quit IRC14:41
clarkbpabelanger: I think check experimental runs the nodepool integration with dib and glean so I've added that comment14:42
*** armax has joined #openstack-infra14:43
mordredclarkb: I can verify that the gerrit config on disk is now correct for gitweb14:45
clarkbinfra-root I'm working to remove gitea06 from our lb pool and socat isn't installed on our loadbalancer node. Any objections to me installing socat there?14:46
clarkbI hear no objections I'm installing it now14:48
*** iurygregory_ has joined #openstack-infra14:49
*** iurygregory has quit IRC14:51
mriedemat some point today i'd like to talk to some infra/qa people about some issues i'm having in enabling a pre-run playbook in this change https://review.opendev.org/#/c/656656/ - i was hitting issues in PS2 so removed a thing, but now hitting different failures in PS3, just wondering what i'm screwing up. it seems that i'm hitting http://status.openstack.org/elastic-recheck/#1806912 consistently in PS214:52
mriedembecause of the pre-run using the orchestrate-devstack role14:52
clarkb#status log Disabled gitea06 backends in gitea-lb01 haproxy because gitea06 has a sad filesystem according to dmesg and openstack/requirements stable/stein was affected.14:53
*** slaweq has joined #openstack-infra14:53
openstackstatusclarkb: finished logging14:53
clarkbfrickler: gthiemonge ^ fyi I ran the 100 curls loop and all are 200s now14:53
clarkbmriedem: it is my undestanding that orchestrate devstack will run devstack on all the nodes it knows about in the inventory14:55
clarkbmriedem: so your suspicion in the ps2 comments is correct aiui14:55
clarkbmriedem: you want to run your setup multi cell policy after devstack runs I take it? There are a set of jobs that have devstack setup in pre-run which means your pre run would happen after that14:56
clarkblet me see if I can find that parent job14:56
*** dklyle has joined #openstack-infra14:56
mriedemclarkb: yeah i need to modify the policy file after devstack is stacked on the controller14:56
mriedembtw i copied that orchestrate-devstack role / play from devstack-tempest which is the parent job (sort of, grandparent) of my job14:57
mriedemso i'm not sure why it blows up for me, but not the devstack-tempest job14:57
clarkbmriedem: because the devstack-tempest job is also going to run it for you I think14:57
clarkbyou don't want it to run twice14:57
gmannclarkb: this is ready. what we discussed to test stable on d-g - https://review.opendev.org/#/c/647912/14:58
*** e0ne has joined #openstack-infra14:58
clarkbmriedem: it iwll run in your pre-run then it will run in the devstack run14:58
clarkbmriedem: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L585-L607 parenting on that might help since it runs devstack in pre run then your pre run will run after and can just set up the multi cell config (orchestrate-devstack already handled for you)15:00
gthiemongeclarkb: thanks for fixing this issue!15:00
mriedemclarkb: ok so i guess the devstack-tempest run will override parent devstack-base run which also does devstack-orchestrate15:00
mriedemfor some reason i thought devstack-tempest was letting its parent setup devstack, but i see now it's overriding the parent run so it has to run devstack and then also run tempest15:01
*** ykarel is now known as ykarel|away15:01
*** dklyle has quit IRC15:01
*** kjackal has quit IRC15:03
*** lucasagomes has joined #openstack-infra15:03
*** lucasagomes has joined #openstack-infra15:04
gmannmriedem: clarkb other clean way but little duplicate is to override all playbook and then insert the policy-override role before run-tempest15:04
mriedemgmann: yeah i was worried about that - my PS3 is failing because my pre-run is running before devstack-tempest runs devstack15:04
*** rosmaita has joined #openstack-infra15:05
mriedemi'd like to avoid overriding the devstack-tempest playbook in case that changes later for some reason15:05
*** diablo_rojo has joined #openstack-infra15:05
gmannyeah that is one thing.15:06
mriedemmy alternative way to do this is a simple env var flag in devstack which my job can set in local.conf and just have devstack setup the policy for me - the legacy way15:06
*** rosmaita has left #openstack-infra15:06
*** eharney has joined #openstack-infra15:07
mriedemis there a way we could put some conditional task in the tempest run such that a child job could set a variable which if set will push the policy variable to the defined file?15:07
gmannyeah that is ok for now and as we discussed in onboarding , we can check for hook in devstack-tempest job for pre-run-tempest things15:07
corvusmriedem: fwiw, we've concieved of roles as the base unit of reusability here; the playbooks involved should be fairly simple and not change very much; the roles are the thing.  so if you can conveniently avoid making a new playbook, that's great, but if not, i wouldn't recommend going out of your way.  these playbooks are simple lists of roles for exactly this reason.15:07
corvus(i haven't dug into the issue under discussion -- that's just a general observation.  mileage may vary)15:08
pabelangerclarkb: yah, glean boot for fedora nodepool test as python3, so I think we are safe here15:09
pabelangerclarkb: let me reply15:09
mriedemi wonder if i could just run the setup-stack-user role on the controller before my policy task, before devstack sets up15:12
mriedemassuming that's all fairly idempotent15:12
mriedemsince all i need to do is write a file to the host15:12
*** rlandy is now known as rlandy|brb15:14
openstackgerritClark Boylan proposed opendev/system-config master: Remove gitea06 from the gitea lb  https://review.opendev.org/65696015:16
*** Lucas_Gray has quit IRC15:17
*** eumel8 has joined #openstack-infra15:17
*** sshnaidm|ptg has joined #openstack-infra15:17
*** ralonsoh has joined #openstack-infra15:17
openstackgerritClark Boylan proposed opendev/system-config master: Install socat where we install haproxy  https://review.opendev.org/65696215:18
clarkbgmann: one thing on that chnage (I left a comment on it)15:21
*** Lucas_Gray has joined #openstack-infra15:21
*** gyee has joined #openstack-infra15:21
*** bgmccollum has joined #openstack-infra15:23
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687115:29
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687115:31
*** ykarel|away has quit IRC15:32
*** sdatko has joined #openstack-infra15:33
*** iurygregory_ has quit IRC15:35
*** rpittau is now known as rpittau|afk15:36
*** rlandy|brb is now known as rlandy15:38
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687115:39
mriedemgmann: i'm going to see if i can get the stack user setup and the policy file written in my pre-run before the tempest job sets up devstack https://review.opendev.org/65665615:39
mriedemif that fails i'll go the hacky devstack variable local.conf route15:39
SotKI'm thinking about IRC bots, where can I find the source for the bot which expands launchpad links ooi?15:44
*** jamesmcarthur has joined #openstack-infra15:45
clarkb#164001 I want to see which nick responds (so that I can find the source for ^)15:45
openstackgerritMonty Taylor proposed openstack/openstack-zuul-jobs master: Set bindep_fallback in legacy-base  https://review.opendev.org/65696915:45
clarkbapparently you have to use a link maybe?15:45
fungiSotK: which launchpad links? have a concrete example?15:45
fungii can also just walk over there ;)15:45
corvusSotK, fungi: https://opendev.org/opendev/meetbot i think15:45
*** ijw has joined #openstack-infra15:45
fungioh, that, yes15:45
clarkbhttps://bugs.launchpad.net/bzr/+bug/7774415:46
openstackLaunchpad bug 77744 in Breezy "win32 bzr is confused by filename case changes" [High,Triaged] - Assigned to Jelmer Vernooij (jelmer)15:46
clarkbyup openstack is meetbot15:46
corvusbug 115:46
openstackbug 1 in Ubuntu Malaysia LoCo Team "Microsoft has a majority market share" [Critical,In progress] https://launchpad.net/bugs/1 - Assigned to MFauzilkamil Zainuddin (apogee)15:46
*** ijw has quit IRC15:46
SotKthanks15:46
* SotK tries to work out where the LP integration is15:47
*** ijw has joined #openstack-infra15:47
mordredSotK: java15:47
SotKfun15:47
corvusignore mordred15:47
corvus /ignore mordred15:47
* mordred throws some perl at SotK15:47
openstackgerritMonty Taylor proposed openstack/project-config master: Make bindep fallback only happen in openstack  https://review.opendev.org/65619515:48
openstackgerritAdam Spiers proposed openstack/ptgbot master: Fix crash after #unsubscribe  https://review.opendev.org/65697015:48
*** jbadiapa has joined #openstack-infra15:48
*** ccamacho has joined #openstack-infra15:49
*** ijw has quit IRC15:49
*** ijw has joined #openstack-infra15:49
*** ccamacho has quit IRC15:50
*** jamesmcarthur has quit IRC15:51
*** ccamacho has joined #openstack-infra15:52
*** ykarel|away has joined #openstack-infra15:55
*** jbadiapa has quit IRC15:55
*** jamesmcarthur has joined #openstack-infra15:57
*** ccamacho has quit IRC15:58
openstackgerritMonty Taylor proposed opendev/system-config master: Fix cgit references in docstrings  https://review.opendev.org/65697416:00
openstackgerritMerged openstack/ptgbot master: Fix crash after #unsubscribe  https://review.opendev.org/65697016:02
*** Lucas_Gray has quit IRC16:02
*** diablo_rojo has quit IRC16:03
openstackgerritMerged opendev/system-config master: Remove gitea06 from the gitea lb  https://review.opendev.org/65696016:03
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687116:04
openstackgerritMerged openstack/openstack-zuul-jobs master: Set bindep_fallback in legacy-base  https://review.opendev.org/65696916:06
*** markvoelker has quit IRC16:13
*** efried has quit IRC16:13
*** diablo_rojo has joined #openstack-infra16:14
openstackgerritMerged opendev/system-config master: Install socat where we install haproxy  https://review.opendev.org/65696216:15
*** _erlon_ has quit IRC16:16
rm_workdhellmann: is tooz ... "working"? or is it dead?16:16
*** efried has joined #openstack-infra16:16
rm_workyou were the last real committer that I see16:16
*** jpich has quit IRC16:17
*** igordc has joined #openstack-infra16:18
fungicorvus: the guidelines i mentioned are published at https://governance.openstack.org/tc/resolutions/20170530-binary-artifacts.html16:20
*** dklyle has joined #openstack-infra16:20
openstackgerritVitaliy Lotorev proposed zuul/zuul master: Replace /bin/bash with /bin/sh shebang  https://review.opendev.org/65697916:20
openstackgerritVitaliy Lotorev proposed zuul/zuul master: Don't use full path while calling ansible-playbook  https://review.opendev.org/65698016:20
fungicorvus: it's two years old now, so if any of that is unreasonable or there are now better alternative solutions i'm happy to draft a new resolution16:21
openstackgerritVitaliy Lotorev proposed zuul/zuul master: Replace /bin/bash with /bin/sh shebang  https://review.opendev.org/65697916:22
openstackgerritVitaliy Lotorev proposed zuul/zuul master: Don't use full path while calling ansible-playbook  https://review.opendev.org/65698016:22
openstackgerritMerged opendev/system-config master: Add cache-control headers for service-types.openstack.org  https://review.opendev.org/48416316:25
openstackgerritMerged opendev/system-config master: Rename cgit_file to git_file  https://review.opendev.org/65405516:25
*** jamesmcarthur has quit IRC16:28
*** eernst has joined #openstack-infra16:29
*** ccamacho has joined #openstack-infra16:29
*** mattw4 has joined #openstack-infra16:31
*** shardy has joined #openstack-infra16:33
*** quiquell has joined #openstack-infra16:33
AJaegerconfig-core, a small (not urgent) cleanup change for review please: https://review.opendev.org/#/c/654580/16:33
*** bgmccollum has quit IRC16:34
mordredmattoliverau: https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L77-L15516:34
*** ricolin has quit IRC16:34
mordredmattoliverau: https://opendev.org/zuul/zuul/src/branch/master/Dockerfile16:34
*** mpjetta has quit IRC16:34
*** ricolin has joined #openstack-infra16:34
corvusmattoliverau: https://zuul-ci.org/docs/zuul/user/encryption.html16:34
corvusmattoliverau: https://opendev.org/zuul/zuul/raw/branch/master/tools/encrypt_secret.py16:35
*** tosky has quit IRC16:35
AJaegermordred: looking at https://review.opendev.org/#/c/484163 - the process for the proposal bot is: Push to service-types page, then download directly. If we cache the site, a new content will not be available directly, won't it?16:36
*** tosky has joined #openstack-infra16:36
AJaegermordred: Wait, we run curl and threre no proxies involved - so that should not affect us . Correct?16:37
mattoliveraucorvus, mordred: sweet, thanks heaps guys!16:37
*** mattw4 has quit IRC16:37
*** yamamoto has quit IRC16:37
*** sdatko has quit IRC16:38
fungimattoliverau: please also review https://governance.openstack.org/tc/resolutions/20170530-binary-artifacts.html and keep those guidelines in mind16:38
mordredAJaeger: yeah - I think we should be fine16:38
*** sdatko has joined #openstack-infra16:39
AJaegermordred: ok...16:40
*** lifeless has quit IRC16:40
*** tdasilva has joined #openstack-infra16:41
fungiclarkb: SotK: AJaeger's https://review.opendev.org/656926 change should help prevent us merging new broken storyboard project groups entries like we saw earlier16:41
*** e0ne has quit IRC16:41
*** mattw4 has joined #openstack-infra16:41
*** jamesmcarthur has joined #openstack-infra16:41
*** lucasagomes has quit IRC16:43
*** sdatko has quit IRC16:44
clarkbfungi: I approved it, thanks for pointing it out16:45
*** ijw has quit IRC16:47
openstackgerritMerged opendev/system-config master: Fix cgit references in docstrings  https://review.opendev.org/65697416:48
*** sdatko has joined #openstack-infra16:49
clarkbmordred: fungi corvus http://logs.openstack.org/80/656880/1/check/system-config-run-docker-registry/ea474d6/job-output.txt.gz#_2019-05-03_16_23_11_780378 bad credentials for the registry?16:52
clarkbis that a thing that has been debugged yet?16:53
*** hwoarang has quit IRC16:53
corvusmordred: http://logs.openstack.org/71/656871/6/check/system-config-run-base/a784d0f/bridge.openstack.org/ara-report/16:53
*** betherly has joined #openstack-infra16:53
*** hwoarang has joined #openstack-infra16:54
openstackgerritMerged openstack/project-config master: Check groups entry in gerrit/projects  https://review.opendev.org/65692616:57
*** betherly has quit IRC16:58
*** jpena is now known as jpena|off16:58
*** bgmccollum has joined #openstack-infra16:59
*** jamesmcarthur has quit IRC17:00
*** dklyle has quit IRC17:02
*** e0ne has joined #openstack-infra17:02
*** Lucas_Gray has joined #openstack-infra17:03
*** udesale has quit IRC17:05
*** jamesmcarthur has joined #openstack-infra17:05
*** diablo_rojo has quit IRC17:06
spotzHey what url does our git config need to be now?17:06
clarkbspotz: if you open the old http(s) urls you will get redirected to the correct url17:07
fungithe old urls should still work, as long as they're not git://17:07
clarkbthat is probably the easiest way to determine what it is17:07
spotzok I'll switch to https:17:07
*** diablo_rojo has joined #openstack-infra17:07
*** lucasagomes has joined #openstack-infra17:07
*** ginopc has quit IRC17:08
spotzand no git. but https://opendev.org worked:)17:08
mordred\o/17:08
*** yamamoto has joined #openstack-infra17:08
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687117:09
*** Weifan has joined #openstack-infra17:09
*** kopecmartin is now known as kopecmartin|off17:10
*** Weifan has quit IRC17:12
*** jamesmcarthur has quit IRC17:13
*** yamamoto has quit IRC17:13
*** ralonsoh has quit IRC17:14
*** bnemec has quit IRC17:14
*** Lucas_Gray has quit IRC17:15
*** gyee has quit IRC17:16
openstackgerritDavid Shrewsbury proposed opendev/puppet-nodepool master: Install jemalloc on nodepool servers  https://review.opendev.org/65698617:19
corvusMay  3 17:17:53 static puppet-user[12900]: (/Stage[main]/Httpd/Service[httpd]) AH00526: Syntax error on line 38 of /etc/apache2/sites-enabled/50-service-types.openstack.org.conf:17:19
corvusMay  3 17:17:53 static puppet-user[12900]: (/Stage[main]/Httpd/Service[httpd]) ExpiresByType takes two arguments, a MIME type followed by an expiry date code17:19
corvusthat has caused static.o.o to die17:19
openstackgerritMonty Taylor proposed opendev/system-config master: Revert "Add cache-control headers for service-types.openstack.org"  https://review.opendev.org/65698717:20
*** zbr has joined #openstack-infra17:21
Shrewsinfra-root: hoping https://review.opendev.org/656986 helps a bit with the nodepool mem issues. please double check my puppeting17:21
mordred/etc/apache2/sites-enabled/50-service-types.openstack.org.conf:    ExpiresActive On17:21
mordred/etc/apache2/sites-enabled/50-service-types.openstack.org.conf:    ExpiresByType  ""17:21
*** mattw4 has quit IRC17:22
*** bnemec has joined #openstack-infra17:22
*** Weifan has joined #openstack-infra17:23
*** dklyle has joined #openstack-infra17:23
*** bnemec is now known as beekneemech17:24
*** beekneemech is now known as bnemec17:24
*** bnemec is now known as beekneemech17:24
*** eernst has quit IRC17:24
openstackgerritMerged opendev/storyboard-webclient master: Add page size of 1000 to preferences  https://review.opendev.org/60443717:25
*** ykarel|away has quit IRC17:26
*** ralonsoh has joined #openstack-infra17:27
*** mattw4 has joined #openstack-infra17:28
*** tesseract has quit IRC17:28
*** shardy has quit IRC17:28
*** dklyle has quit IRC17:29
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687117:30
openstackgerritMonty Taylor proposed opendev/system-config master: Revert "Revert "Add cache-control headers for service-types.openstack.org""  https://review.opendev.org/65699017:34
spotzfungi clarkb any changes for gerrit?17:41
clarkbspotz: gerrit is not canonically at review.opendev.org and if a project was renamed you'll need to change the path (we can't redirect there)17:41
*** chandankumar is now known as raukadah17:42
openstackgerritDirk Mueller proposed zuul/zuul-website master: Correct company spelling  https://review.opendev.org/65699217:42
openstackgerritVitaliy Lotorev proposed zuul/zuul master: Replace /bin/bash with /bin/sh shebang  https://review.opendev.org/65697917:42
openstackgerritVitaliy Lotorev proposed zuul/zuul master: Don't use full path while calling ansible-playbook  https://review.opendev.org/65698017:42
spotzclarkb: ok Tobias is trying to publish a meeting update we'll look further17:42
fungispotz: need to pull the latest state of the repository and then delete your gerrit remote. after that, git review should readd the gerrit remote for you with the corrected hostname and project name17:43
*** EmilienM is now known as EvilienM17:43
spotzfungi: ok thanks17:43
fungianother alternative is to delete the local repository and clone it again17:44
*** michael-beaver has joined #openstack-infra17:45
spotzWe'll know in a second if we got it:)17:45
openstackgerritTobias Rydberg proposed opendev/irc-meetings master: Removing unused meeting time for the Public Cloud WG.  https://review.opendev.org/65699317:46
openstackgerritAmy Marrich (spotz) proposed opendev/irc-meetings master: Removing unused meeting time for the Public Cloud WG.  https://review.opendev.org/65699317:46
*** efried has quit IRC17:52
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687117:52
*** ralonsoh has quit IRC17:53
*** sdatko has quit IRC17:55
*** lucasagomes has quit IRC17:55
*** efried has joined #openstack-infra17:57
*** sdatko has joined #openstack-infra17:58
*** bdodd has quit IRC17:58
*** bdodd has joined #openstack-infra18:00
*** slaweq has quit IRC18:00
*** kjackal has joined #openstack-infra18:03
logan-logs.openstack.org seems down18:06
clarkblogan-: should be fixed18:06
fungilogan-: yeah, the manual revert got undone while waiting for the actual revert to merge18:07
*** bgmccollum has quit IRC18:07
clarkbwe can all buy monty a beer for both breaking and fixingg it18:07
openstackgerritMerged zuul/zuul-website master: Correct company spelling  https://review.opendev.org/65699218:07
logan-:) thanks18:07
*** eernst has joined #openstack-infra18:08
openstackgerritMerged opendev/system-config master: Revert "Add cache-control headers for service-types.openstack.org"  https://review.opendev.org/65698718:09
*** e0ne has quit IRC18:12
*** ricolin has quit IRC18:17
*** eharney has quit IRC18:17
*** quiquell has quit IRC18:20
openstackgerritMonty Taylor proposed opendev/puppet-openstack_infra_spec_helper master: Make hosts file in inventory directory  https://review.opendev.org/65700218:20
openstackgerritMonty Taylor proposed opendev/puppet-openstack_infra_spec_helper master: Make hosts file in inventory directory  https://review.opendev.org/65700218:20
*** sshnaidm|ptg has quit IRC18:20
mordredcmurphy: ^^ wanna make sure i rubyd correctly?18:21
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687118:22
*** ccamacho has quit IRC18:23
*** dklyle has joined #openstack-infra18:28
cmurphymordred: that looks indeed like ruby18:30
*** eernst has quit IRC18:31
*** jamesmcarthur has joined #openstack-infra18:33
*** dklyle has quit IRC18:35
*** beekneemech has quit IRC18:36
*** bobh has joined #openstack-infra18:37
*** jamesmcarthur has quit IRC18:38
*** imacdonn has quit IRC18:38
*** imacdonn has joined #openstack-infra18:39
*** jamesmcarthur has joined #openstack-infra18:39
rm_workclarkb: so it seems like I added those IPs correctly to the br-infra interface, and it didn't appear to help18:44
*** diablo_rojo has quit IRC18:46
*** jamesmcarthur has quit IRC18:47
roman_gAJaeger, jroll, clarkb, replied to that e-mail with suggestion (to the mailing list). Sorry, have just subscribed, so mesage is actually out of thread.18:51
clarkbroman_g: I rhink it ie a good idea and something we can do but likely not before we get home from the summit18:52
*** jamesmcarthur has joined #openstack-infra18:52
*** lseki has joined #openstack-infra18:52
*** ricolin has joined #openstack-infra18:53
*** dklyle has joined #openstack-infra18:53
roman_gclarkb: no problem at all. Thank you for all the work you are doing for the Infra.18:54
roman_gclarkb: the only drawback I see from not archieving repos earlier, is that Github repos are indexed and ranked high by search engines, and people git clone from github and complain about software not working, whereas we are many commits ahead already and bugs people complain about are fixed already18:55
roman_ge.g. https://www.google.com/search?q=airship+armada - first hist would be github repo18:57
roman_g*hit18:57
roman_g>> Airship Armada is the second level of World Star in Super Mario Run.18:58
roman_glol18:58
*** bnemec has joined #openstack-infra18:59
*** efried has quit IRC19:02
*** efried has joined #openstack-infra19:02
*** pcaruana has quit IRC19:02
*** betherly has joined #openstack-infra19:16
*** ricolin has quit IRC19:20
*** dklyle has quit IRC19:20
*** jamesmcarthur has quit IRC19:21
*** bnemec has quit IRC19:21
*** betherly has quit IRC19:21
*** rosmaita has joined #openstack-infra19:22
*** eharney has joined #openstack-infra19:26
*** bobh has quit IRC19:27
*** bnemec has joined #openstack-infra19:28
*** sshnaidm|ptg has joined #openstack-infra19:28
*** quiquell has joined #openstack-infra19:30
*** jtomasek has quit IRC19:33
*** e0ne has joined #openstack-infra19:34
*** ijw has joined #openstack-infra19:34
*** betherly has joined #openstack-infra19:37
*** bgmccollum has joined #openstack-infra19:37
*** ralonsoh has joined #openstack-infra19:39
openstackgerritTobias Henkel proposed zuul/zuul master: Restore posargs when running stestr  https://review.opendev.org/65702019:41
*** jamesmcarthur has joined #openstack-infra19:41
*** betherly has quit IRC19:42
*** rosmaita has left #openstack-infra19:42
*** jamesmcarthur has quit IRC19:45
*** slaweq has joined #openstack-infra19:45
*** ijw has quit IRC19:45
*** ijw has joined #openstack-infra19:46
*** ccamacho has joined #openstack-infra19:47
*** slaweq has quit IRC19:51
*** dklyle has joined #openstack-infra19:52
rm_workclarkb: is there a good time to catch you for maybe setting up a hold? this is the patch i've been using to test it: https://review.opendev.org/#/c/656852/19:54
*** slaweq has joined #openstack-infra19:55
*** bgmccollum has quit IRC19:56
rm_workclarkb: http://logs.openstack.org/52/656852/9/check/octavia-v2-dsvm-scenario-two-node/aa76969/job-output.txt.gz (search in page for "inet6 2001:db8::"19:57
rm_work)19:57
*** bgmccollum has joined #openstack-infra19:58
*** devananda has joined #openstack-infra19:59
*** e0ne has quit IRC20:00
openstackgerritTobias Henkel proposed zuul/zuul master: DNM: try out mitogen for zuul jobs  https://review.opendev.org/65702420:02
clarkbrm_work: hold is in place feel free to recheck20:04
*** e0ne has joined #openstack-infra20:06
*** kjackal has quit IRC20:07
*** ijw has quit IRC20:08
rm_workok20:08
*** pkopec has quit IRC20:08
*** betherly has joined #openstack-infra20:09
*** ricolin has joined #openstack-infra20:09
*** ccamacho has quit IRC20:09
rm_workclarkb: i guess soon i'll be asking you how to actually... use the held nodes :D20:10
*** lifeless has joined #openstack-infra20:11
clarkbrm_work: when the jobs fail we'll get a record in the nodepool db saying the nodes are heald and we add your ssh key to the nodes and then you ssh in20:11
clarkbrm_work: so the next step is recheck jobs and wait for them to fail. Then give us your ssh keys20:12
rm_workok cool :)20:12
rm_worki did so20:12
rm_workwaiting on the 656852 `octavia-v2-dsvm-scenario-two-node` job to fail20:12
*** betherly has quit IRC20:13
*** ccamacho has joined #openstack-infra20:16
*** devananda has quit IRC20:22
*** ijw has joined #openstack-infra20:25
*** tjgresha has joined #openstack-infra20:39
openstackgerritMatthew Thode proposed openstack/project-config master: add version map for python3.6 to 3.7  https://review.opendev.org/65704420:52
*** ijw has quit IRC20:55
*** ijw has joined #openstack-infra20:56
openstackgerritStephen Finucane proposed openstack/project-config master: Retire bilean, python-bileanclient  https://review.opendev.org/65685621:00
*** betherly has joined #openstack-infra21:01
gmannmriedem: you might need to take all things base job's pre.yaml does and then extend it to policy role for host: controller21:02
gmannthis one - https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L29121:03
gmannas you are overriding  the complete pre-run21:04
openstackgerritStephen Finucane proposed openstack/project-config master: Retire hurricane  https://review.opendev.org/65686221:05
*** lucasagomes has joined #openstack-infra21:05
*** betherly has quit IRC21:06
openstackgerritVitaliy Lotorev proposed zuul/zuul master: Add proxy settings for Zuul scheduler  https://review.opendev.org/65704821:07
mriedemgmann: i'll cut my right arm off before doing that21:07
*** jamesmcarthur has joined #openstack-infra21:07
openstackgerritVitaliy Lotorev proposed zuul/zuul master: Add proxy settings for Zuul scheduler  https://review.opendev.org/65704821:08
aspiersinfra-root: https://cfp.all-systems-go.io/ASG2018/talk/207/ is the talk I just mentioned (sorry for those not at PTG)21:08
fungithanks aspiers!21:09
*** bnemec has quit IRC21:11
*** ricolin has quit IRC21:12
openstackgerritStephen Finucane proposed openstack/project-config master: Retire aeromancer  https://review.opendev.org/65705421:16
*** e0ne has quit IRC21:18
gmannmriedem: i am worried about log files but waiting for gate results21:19
*** slaweq has quit IRC21:21
*** bnemec has joined #openstack-infra21:21
*** ijw has quit IRC21:22
*** ijw has joined #openstack-infra21:22
openstackgerritDirk Mueller proposed opendev/system-config master: Add mirroring for Stein packages  https://review.opendev.org/65568621:24
openstackgerritDirk Mueller proposed opendev/system-config master: Properly mirror the lxc container images as well  https://review.opendev.org/65589521:24
*** e0ne has joined #openstack-infra21:25
*** ccamacho has quit IRC21:25
*** sdatko has quit IRC21:26
*** slaweq has joined #openstack-infra21:27
*** ijw has quit IRC21:29
*** ijw has joined #openstack-infra21:29
*** ijw has quit IRC21:30
*** ijw has joined #openstack-infra21:30
openstackgerritStephen Finucane proposed openstack/project-config master: Retire aeromancer  https://review.opendev.org/65705421:31
openstackgerritStephen Finucane proposed openstack/project-config master: Initiate retirement of aeromancer  https://review.opendev.org/65705921:31
*** rfolco|ruck has quit IRC21:32
*** ralonsoh has quit IRC21:32
*** betherly has joined #openstack-infra21:33
*** Weifan has quit IRC21:34
openstackgerritStephen Finucane proposed openstack/project-config master: Retire mors  https://review.opendev.org/65706121:36
*** betherly has quit IRC21:37
*** uberjay has quit IRC21:39
*** lewo has quit IRC21:39
*** csatari has quit IRC21:39
*** dancek has quit IRC21:39
*** gary_perkins has quit IRC21:39
*** woojay has quit IRC21:39
*** mgagne has quit IRC21:39
*** mugsie has quit IRC21:39
*** ricolin has joined #openstack-infra21:40
*** freerunner has quit IRC21:42
corvusmordred: http://logs.openstack.org/71/656871/10/check/system-config-run-eavesdrop/71bb94f/ara-report/result/a4da2b8a-ba30-4312-a465-7948fa7d878e/21:42
openstackgerritStephen Finucane proposed openstack/project-config master: Initiate retirement of hurricane  https://review.opendev.org/65706521:42
openstackgerritStephen Finucane proposed openstack/project-config master: Retire alexandria  https://review.opendev.org/65706621:42
mordredcorvus: looking21:42
corvusmordred: so we're back at the missing puppet module mystery21:42
*** MarkMaglana has quit IRC21:43
*** jamespage has quit IRC21:43
*** jroll has quit IRC21:43
*** dustinc has quit IRC21:43
*** dtantsur has quit IRC21:43
*** cjloader has quit IRC21:43
*** persia has quit IRC21:43
*** gema has quit IRC21:43
*** stevebaker has quit IRC21:43
*** ericyoung has quit IRC21:43
*** freerunner has joined #openstack-infra21:44
*** cgoncalves has quit IRC21:44
aspiersinfra-root: https://dirkmueller.github.io/presentation-berlin-log-classify/#/721:45
*** ijw has quit IRC21:45
*** ralonsoh has joined #openstack-infra21:45
*** ijw has joined #openstack-infra21:46
*** cgoncalves has joined #openstack-infra21:47
stephenfinAJaeger: I think that's enough project retirement emails for one day21:47
*** betherly has joined #openstack-infra21:53
*** bdodd has quit IRC21:54
*** mattw4 has quit IRC21:55
*** mattw4 has joined #openstack-infra21:55
*** betherly has quit IRC21:58
openstackgerritJames E. Blair proposed opendev/system-config master: Split the base playbook into services  https://review.opendev.org/65687121:58
*** jbadiapa has joined #openstack-infra21:58
*** jamesmcarthur has quit IRC21:59
*** jamesmcarthur has joined #openstack-infra22:01
*** jamesmcarthur has quit IRC22:02
*** yamamoto has joined #openstack-infra22:03
*** gary_perkins has joined #openstack-infra22:07
*** uberjay has joined #openstack-infra22:07
*** dancek has joined #openstack-infra22:07
*** yamamoto has quit IRC22:07
*** csatari has joined #openstack-infra22:07
*** persia has joined #openstack-infra22:07
*** stevebaker has joined #openstack-infra22:08
*** jamespage has joined #openstack-infra22:08
*** dustinc has joined #openstack-infra22:08
*** ericyoung has joined #openstack-infra22:08
*** jroll has joined #openstack-infra22:08
*** MarkMaglana has joined #openstack-infra22:08
*** pfallenop has quit IRC22:08
smcginnisstephenfin: Love seeing those!22:09
*** dtantsur has joined #openstack-infra22:10
*** mugsie has joined #openstack-infra22:10
*** mgagne has joined #openstack-infra22:10
rm_workclarkb: ok so they've failed, I need to send you ssh pubkeys?22:10
clarkbrm_work: ya if you can give me a pubkey I'll add it to the host22:11
clarkbhosts*22:11
clarkbpm is fine22:12
*** rfarr__ has joined #openstack-infra22:13
rm_worki mean, it's pubkeys22:13
rm_workbut sure :D22:13
clarkbya it just helpskeeps noise out of the channel22:13
fungithough usually it's just a url to a pubkey somewhere else ;)22:14
*** pfallenop has joined #openstack-infra22:14
*** rlandy has quit IRC22:15
clarkbrm_work: ssh root@104.239.174.153 and ssh root@104.239.174.14322:16
clarkbfungi: ya rm_work gave me a paste link but often people splat them straight into irc22:16
*** rfarr_ has quit IRC22:16
clarkbrm_work: I don't know which is which in your test so oyu'll have to cross check agains the test job inventory22:16
rm_workok22:17
rm_workyeah we can see22:17
*** rfarr_ has joined #openstack-infra22:18
rm_work153=controller, 143=controller222:18
*** ralonsoh has quit IRC22:19
*** rfarr__ has quit IRC22:21
*** yamamoto has joined #openstack-infra22:21
*** sshnaidm|ptg has quit IRC22:21
*** quiquell has quit IRC22:22
*** lucasagomes has quit IRC22:22
*** rfarr_ has quit IRC22:23
*** niceplace has quit IRC22:27
*** slaweq has quit IRC22:29
rm_workclarkb: so if we do changes and it triggers another check, will those failed nodes also hold, or not?22:32
rm_work(did you do a limit-1 hold, or indefinite?)22:32
clarkbI think I did the one22:33
*** niceplace has joined #openstack-infra22:33
clarkbyup no existing holds22:33
clarkber autoholds in zuul22:33
*** betherly has joined #openstack-infra22:35
rm_workkk that's fine, i'll ping you again if i need another :)22:36
*** ijw has quit IRC22:38
*** ijw has joined #openstack-infra22:39
*** ijw_ has joined #openstack-infra22:40
*** betherly has quit IRC22:40
*** ricolin has quit IRC22:40
*** ijw_ has quit IRC22:41
*** jbryce has quit IRC22:42
*** jbryce has joined #openstack-infra22:42
*** ijw_ has joined #openstack-infra22:42
*** ijw has quit IRC22:47
*** ijw_ has quit IRC22:47
*** ijw has joined #openstack-infra22:47
*** cjloader has joined #openstack-infra22:47
*** ijw has quit IRC22:47
*** sshnaidm|ptg has joined #openstack-infra22:47
*** openstackstatus has quit IRC22:47
*** ijw has joined #openstack-infra22:47
*** e0ne has quit IRC22:47
*** ijw has quit IRC22:49
*** Ng has quit IRC22:49
*** eumel8 has quit IRC22:49
*** Ng has joined #openstack-infra22:50
*** egonzalez has quit IRC22:50
*** egonzalez has joined #openstack-infra22:50
*** e0ne has joined #openstack-infra22:50
*** e0ne has quit IRC22:50
*** e0ne has joined #openstack-infra22:51
*** dklyle has quit IRC22:51
*** e0ne has quit IRC22:51
*** e0ne has joined #openstack-infra22:52
*** bgmccollum has quit IRC22:53
openstackgerritMerged zuul/zuul master: Restore posargs when running stestr  https://review.opendev.org/65702022:53
*** wendar has joined #openstack-infra22:55
*** ricolin has joined #openstack-infra22:55
*** bgmccollum has joined #openstack-infra22:56
rm_workclarkb: so after adding the ip to br-infra it still doesn't work, BUT it looks like there's a duplicate route on br-ex to that ipv6 subnet, and when we remove THAT one, it works22:57
*** e0ne has quit IRC22:57
rm_worki am assuming neutron adds that one22:57
clarkbya br-ex is nwutron22:58
*** cgoncalves has quit IRC22:58
*** ccamacho has joined #openstack-infra22:58
clarkbI wonder if neutron filters it on br-ex so VMs cant talk to the control plane22:58
*** cgoncalves has joined #openstack-infra22:58
*** ricolin has quit IRC23:00
*** cgoncalves has quit IRC23:02
*** e0ne has joined #openstack-infra23:03
*** cgoncalves has joined #openstack-infra23:04
*** e0ne has quit IRC23:05
rm_workyeah, *I* have no idea how I'd fix that, but hopefully johnsom is on it ;)23:05
*** e0ne has joined #openstack-infra23:05
*** mriedem has quit IRC23:07
*** e0ne has quit IRC23:08
*** bnemec has quit IRC23:11
*** tosky has quit IRC23:13
*** bgmccollum has quit IRC23:13
fungimordred: corvus: looking at the new run failures on 656871 it seems like maybe the job isn't finding the openstack_project module in system-config when installing puppet modules and so later the puppet run breaks on global site manifest references to our server classes? http://logs.openstack.org/71/656871/11/check/system-config-run-base/5b12895/trusty/syslog.txt.gz#_May__3_22_22_1123:14
*** aaronsheffield has quit IRC23:14
*** Lucas_Gray has joined #openstack-infra23:15
*** mattw4 has quit IRC23:16
*** betherly has joined #openstack-infra23:18
*** eharney has quit IRC23:18
*** bgmccollum has joined #openstack-infra23:18
*** yamamoto has quit IRC23:19
*** bnemec has joined #openstack-infra23:19
*** Goneri has quit IRC23:22
*** betherly has quit IRC23:23
*** jbadiapa has quit IRC23:25
*** mriedem has joined #openstack-infra23:29
*** tdasilva has quit IRC23:31
*** cloudnull has joined #openstack-infra23:36
*** eglute has joined #openstack-infra23:37
*** d34dh0r53 has joined #openstack-infra23:37
*** efried has quit IRC23:38
*** betherly has joined #openstack-infra23:39
*** Weifan has joined #openstack-infra23:39
*** betherly has quit IRC23:43
*** jbadiapa has joined #openstack-infra23:49
*** Weifan has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!