| clarkb | ok my day is about over now. Sorry I wasn't very useful beyond doing some reviews and cleaning out the cgit cluster | 00:01 |
|---|---|---|
| clarkb | mordred: do you intend on replacing status.o.o on xenial node still? If not I may try to pick thatup this week | 00:02 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Create opendev mirrors https://review.opendev.org/658281 | 00:02 |
| *** jamesmcarthur has joined #openstack-infra | 00:04 | |
| *** sshnaidm|afk has quit IRC | 00:04 | |
| *** sshnaidm has joined #openstack-infra | 00:06 | |
| *** panda|rover has quit IRC | 00:10 | |
| *** panda has joined #openstack-infra | 00:11 | |
| mordred | clarkb: yes - I do - but if you run out of things to do before I get to it, I won't be offended | 00:11 |
| clarkb | mordred: k I may focus on gitea06 replacement first then | 00:12 |
| mordred | ++ | 00:12 |
| clarkb | ianw: opendev mirrors change looks great | 00:12 |
| ianw | clarkb: one other slightly related one is the updates in zone file for ssl proposed by fungi -> https://review.opendev.org/#/c/652280/ | 00:13 |
| ianw | if that's all ok, will follow that for mirror nodes too from start | 00:14 |
| fungi | and https://review.opendev.org/652281 to turn on hsts | 00:14 |
| *** smarcet has joined #openstack-infra | 00:16 | |
| ianw | fungi: https://review.opendev.org/#/c/658281/30/playbooks/roles/mirror/templates/mirror.vhost.j2 should contain the ssl config you proposed as standard too | 00:16 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add RFC 6844 CAA RR for graphite01 https://review.opendev.org/652280 | 00:17 |
| clarkb | fungi: ianw for https://review.opendev.org/#/c/652281/1 considering the validity period of the ssl cert itself is only 90 days should we use a more conservative period for insisting on ssl? | 00:17 |
| clarkb | I guess reducing the period increases the attack surface as it has to renew that ttl? | 00:18 |
| *** sreejithp has joined #openstack-infra | 00:18 | |
| openstackgerrit | Jimmy McArthur proposed zuul/zuul-website master: Revert "Revert promotional message banner and event list" https://review.opendev.org/660232 | 00:19 |
| ianw | clarkb: yeah i think the less you check that flag, the better | 00:20 |
| ianw | although now i think about it, i wonder if with the mirrors maybe we don't want it, if we get weird interactions where we're still specifying http:// and then it's saying only use https:// | 00:21 |
| fungi | clarkb: the idea with setting it to a year is that it only matters you can age out that setting if you decide that you want to switch the site from https to http later | 00:21 |
| ianw | not sure if apt etc even obey it | 00:21 |
| clarkb | fungi: yup which you might do if you cannot renew the cert | 00:21 |
| fungi | sure | 00:21 |
| fungi | conversely, the longer you make it, the more it reduces the opportunity for downgrade attacks | 00:22 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Create opendev mirrors https://review.opendev.org/658281 | 00:24 |
| ianw | clarkb: i'll bring up, say, a rax dfw node for an initial opendev test mirror? i guess the process is launch.py it, add to opendev.org zone, add to inventory.yaml | 00:26 |
| fungi | anyway, hsts for graphite is not really critical, i was using that change more as an example | 00:27 |
| clarkb | yup shouldn't be any different from older launches except for the dns management | 00:27 |
| fungi | for services we know we won't ever want to serve over http for safety reasons, it could be a nice little extra bit of insurance | 00:27 |
| clarkb | followup ps adds the file that I should've checked was added before I +2'd the last ps :) | 00:30 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0"" https://review.opendev.org/659991 | 00:32 |
| *** sreejithp has quit IRC | 00:32 | |
| *** jamesmcarthur has quit IRC | 00:35 | |
| clarkb | and with that I need to call it a day. ianw thank you for pushing on this stuff | 00:37 |
| ianw | clarkb: thanks! later | 00:37 |
| openstackgerrit | Merged opendev/puppet-graphite master: Enable RFC 6797 HSTS signaling https://review.opendev.org/652281 | 00:39 |
| *** smarcet has quit IRC | 00:39 | |
| *** markvoelker has joined #openstack-infra | 00:41 | |
| ianw | May 21 00:41:06 mirror01 systemd-resolved[988]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP | 00:41 |
| ianw | i dunno what that means, during launching a node | 00:41 |
| clarkb | maybe it tried to resolve itself and failed? | 00:42 |
| *** markvoelker has quit IRC | 00:45 | |
| *** mattw4 has quit IRC | 00:46 | |
| fungi | https://github.com/systemd/systemd/issues/9833 | 00:47 |
| fungi | which also mentions https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1822416 | 00:48 |
| openstack | Launchpad bug 1818527 in systemd (Ubuntu) "duplicate for #1822416 Stub resolver cache is corrupted" [Undecided,Confirmed] | 00:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Add mirror.dfw.rax.opendev.org https://review.opendev.org/660235 | 00:52 |
| *** zhurong has joined #openstack-infra | 00:52 | |
| ianw | http://logs.openstack.org/13/660013/1/gate/openstackci-beaker-puppet-4/acc41ed/job-output.txt.gz#_2019-05-20_21_46_46_358317 gpg failure again; not sure if the long key patch was merged though | 00:59 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Bringup mirror01.dfw.rax.opendev.org https://review.opendev.org/660237 | 01:00 |
| *** jamesmcarthur has joined #openstack-infra | 01:05 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Create opendev mirrors https://review.opendev.org/658281 | 01:09 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Bringup mirror01.dfw.rax.opendev.org https://review.opendev.org/660237 | 01:09 |
| *** jamesmcarthur_ has joined #openstack-infra | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:11 | |
| ianw | one thing with running the playbooks in a loop is that i don't think we get streaming output during the testing | 01:26 |
| *** rh-jelabarre has quit IRC | 01:31 | |
| *** roman_g has quit IRC | 01:38 | |
| openstackgerrit | Merged opendev/system-config master: launch.py : fix typo calling legacy dns print function https://review.opendev.org/660013 | 01:38 |
| *** gregoryo has joined #openstack-infra | 01:50 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0"" https://review.opendev.org/659991 | 01:53 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0"" https://review.opendev.org/659991 | 01:58 |
| *** whoami-rajat has joined #openstack-infra | 02:00 | |
| *** hwoarang has quit IRC | 02:04 | |
| *** hwoarang has joined #openstack-infra | 02:05 | |
| *** sreejithp has joined #openstack-infra | 02:13 | |
| *** sreejithp_ has joined #openstack-infra | 02:15 | |
| *** nicolasbock has quit IRC | 02:16 | |
| *** sreejithp has quit IRC | 02:19 | |
| *** ykarel|away has joined #openstack-infra | 02:29 | |
| *** ricolin has joined #openstack-infra | 02:44 | |
| *** ykarel|away is now known as ykarel | 02:44 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: DNM: this should cause a React exception https://review.opendev.org/660231 | 02:51 |
| *** dave-mccowan has quit IRC | 02:55 | |
| *** hongbin has joined #openstack-infra | 03:00 | |
| *** tkajinam has quit IRC | 03:03 | |
| *** tkajinam has joined #openstack-infra | 03:04 | |
| *** kaisers has quit IRC | 03:17 | |
| *** tkajinam has quit IRC | 03:20 | |
| *** jamesmcarthur_ has quit IRC | 03:23 | |
| *** psachin has joined #openstack-infra | 03:28 | |
| *** tkajinam has joined #openstack-infra | 03:28 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:42 | |
| *** udesale has joined #openstack-infra | 03:47 | |
| *** yolanda_ has quit IRC | 03:49 | |
| *** jamesmcarthur has joined #openstack-infra | 03:54 | |
| *** armax has quit IRC | 03:54 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| *** sreejithp_ has quit IRC | 04:03 | |
| *** zerick has quit IRC | 04:03 | |
| *** zerick has joined #openstack-infra | 04:04 | |
| *** hongbin has quit IRC | 04:21 | |
| *** markvoelker has joined #openstack-infra | 04:35 | |
| *** janki has joined #openstack-infra | 04:37 | |
| *** markvoelker has quit IRC | 04:39 | |
| *** ramishra has joined #openstack-infra | 04:40 | |
| *** slaweq has joined #openstack-infra | 04:49 | |
| *** yboaron_ has joined #openstack-infra | 04:57 | |
| *** slaweq has quit IRC | 04:59 | |
| *** jamesmcarthur has joined #openstack-infra | 05:01 | |
| *** pcaruana has joined #openstack-infra | 05:02 | |
| *** ykarel is now known as ykarel|away | 05:03 | |
| *** raukadah is now known as chandankumar | 05:06 | |
| *** jamesmcarthur has quit IRC | 05:07 | |
| *** ykarel|away has quit IRC | 05:08 | |
| *** pcaruana has quit IRC | 05:11 | |
| *** slaweq has joined #openstack-infra | 05:16 | |
| AJaeger | config-core, please review https://review.opendev.org/659963 and https://review.opendev.org/659978 https://review.opendev.org/660022 | 05:18 |
| *** slaweq has quit IRC | 05:20 | |
| *** ykarel|away has joined #openstack-infra | 05:20 | |
| *** Lucas_Gray has joined #openstack-infra | 05:23 | |
| *** e0ne has joined #openstack-infra | 05:23 | |
| *** e0ne has quit IRC | 05:24 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 05:28 | |
| *** bhavikdbavishi has quit IRC | 05:28 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 05:28 | |
| *** ykarel|away is now known as ykarel | 05:28 | |
| *** quiquell has joined #openstack-infra | 05:38 | |
| *** quiquell has quit IRC | 05:38 | |
| *** quiquell has joined #openstack-infra | 05:39 | |
| *** imacdonn has quit IRC | 05:40 | |
| *** imacdonn has joined #openstack-infra | 05:41 | |
| *** ykarel_ has joined #openstack-infra | 05:42 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Fix broken setRefs whith missing objects https://review.opendev.org/621667 | 05:43 |
| *** udesale has quit IRC | 05:44 | |
| *** ykarel has quit IRC | 05:44 | |
| *** udesale has joined #openstack-infra | 05:44 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Match tag items against containing branches https://review.opendev.org/578557 | 05:47 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches https://review.opendev.org/640272 | 05:47 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Update cached repo during job startup only if needed https://review.opendev.org/648229 | 05:53 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add support for smart reconfigurations https://review.opendev.org/652114 | 05:57 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add --check-config option to zuul scheduler https://review.opendev.org/542160 | 05:57 |
| *** ykarel__ has joined #openstack-infra | 06:01 | |
| *** ykarel_ has quit IRC | 06:03 | |
| *** ykarel__ is now known as ykarel | 06:04 | |
| *** dangtrinhnt has quit IRC | 06:04 | |
| *** zbr_ has quit IRC | 06:05 | |
| *** dangtrinhnt has joined #openstack-infra | 06:06 | |
| *** bhavikdbavishi has quit IRC | 06:10 | |
| *** bhavikdbavishi has joined #openstack-infra | 06:12 | |
| *** slaweq has joined #openstack-infra | 06:12 | |
| *** lpetrut has joined #openstack-infra | 06:15 | |
| *** Lucas_Gray has quit IRC | 06:18 | |
| *** ramishra has quit IRC | 06:19 | |
| *** jistr is now known as jistr|mtg | 06:20 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Match tag items against containing branches https://review.opendev.org/578557 | 06:20 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches https://review.opendev.org/640272 | 06:20 |
| *** yboaron_ has quit IRC | 06:23 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Update cached repo during job startup only if needed https://review.opendev.org/648229 | 06:26 |
| *** kaisers has joined #openstack-infra | 06:34 | |
| *** jamesmcarthur has joined #openstack-infra | 06:35 | |
| *** markvoelker has joined #openstack-infra | 06:36 | |
| *** jamesmcarthur has quit IRC | 06:39 | |
| ianw | #status log ask.openstack.org migrated to new xenial server | 06:39 |
| openstackstatus | ianw: finished logging | 06:39 |
| ianw | i have had to manually redo the solr indexes; it doesn't seem puppet handles this (maybe a periodic update might have got it?) | 06:40 |
| ianw | anyway, it seems to be working enough ... | 06:40 |
| *** ramishra has joined #openstack-infra | 06:51 | |
| *** pgaxatte has joined #openstack-infra | 06:55 | |
| *** ccamacho has joined #openstack-infra | 06:56 | |
| *** sreejithp has joined #openstack-infra | 07:03 | |
| *** dpawlik has joined #openstack-infra | 07:03 | |
| *** trident has quit IRC | 07:03 | |
| *** trident has joined #openstack-infra | 07:05 | |
| *** tesseract has joined #openstack-infra | 07:06 | |
| *** sreejithp has quit IRC | 07:07 | |
| *** ginopc has joined #openstack-infra | 07:08 | |
| *** markvoelker has quit IRC | 07:09 | |
| openstackgerrit | Merged openstack/project-config master: Remove unused job publish-openstack-sphinx-docs-direct https://review.opendev.org/659963 | 07:09 |
| *** kopecmartin|off is now known as kopecmartin | 07:12 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint https://review.opendev.org/621344 | 07:18 |
| *** roman_g has joined #openstack-infra | 07:18 | |
| *** rcernin has quit IRC | 07:19 | |
| *** ralonsoh has joined #openstack-infra | 07:19 | |
| *** pcaruana has joined #openstack-infra | 07:21 | |
| *** jamesmcarthur has joined #openstack-infra | 07:25 | |
| *** piotrowskim has joined #openstack-infra | 07:26 | |
| *** apetrich has joined #openstack-infra | 07:28 | |
| *** jamesmcarthur has quit IRC | 07:30 | |
| *** ramishra has quit IRC | 07:33 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ask.openstack.org: switch backup user to ask01-bup https://review.opendev.org/660269 | 07:34 |
| *** ramishra has joined #openstack-infra | 07:34 | |
| *** amoralej|off is now known as amoralej | 07:35 | |
| *** ykarel is now known as ykarel|lunch | 07:35 | |
| *** udesale has quit IRC | 07:35 | |
| *** udesale has joined #openstack-infra | 07:36 | |
| *** pcaruana has quit IRC | 07:39 | |
| *** yikun_ has joined #openstack-infra | 07:40 | |
| *** tosky has joined #openstack-infra | 07:41 | |
| *** rpittau|afk is now known as rpittau | 07:41 | |
| *** e0ne has joined #openstack-infra | 07:42 | |
| *** bhavikdbavishi has quit IRC | 07:51 | |
| *** jpena|off is now known as jpena | 07:56 | |
| *** e0ne has quit IRC | 07:57 | |
| *** happyhemant has joined #openstack-infra | 07:58 | |
| *** e0ne has joined #openstack-infra | 08:01 | |
| *** markvoelker has joined #openstack-infra | 08:06 | |
| *** ramishra_ has joined #openstack-infra | 08:08 | |
| *** lucasagomes has joined #openstack-infra | 08:08 | |
| *** ramishra has quit IRC | 08:11 | |
| *** tkajinam has quit IRC | 08:12 | |
| *** ramishra_ is now known as ramishra | 08:15 | |
| *** diablo_rojo has joined #openstack-infra | 08:17 | |
| openstackgerrit | Merged opendev/system-config master: Add #starlingx to statusbot channels https://review.opendev.org/659652 | 08:19 |
| openstackgerrit | Merged opendev/system-config master: ask.openstack.org: switch backup user to ask01-bup https://review.opendev.org/660269 | 08:19 |
| *** dtantsur|afk is now known as dtantsur | 08:20 | |
| *** priteau has joined #openstack-infra | 08:20 | |
| *** priteau has quit IRC | 08:24 | |
| *** betherly has joined #openstack-infra | 08:25 | |
| *** priteau has joined #openstack-infra | 08:28 | |
| *** ykarel|lunch is now known as ykarel | 08:30 | |
| *** xek has joined #openstack-infra | 08:36 | |
| *** xek has quit IRC | 08:38 | |
| *** markvoelker has quit IRC | 08:39 | |
| openstackgerrit | Merged opendev/system-config master: Remove ask-staging* from disabled list https://review.opendev.org/659421 | 08:39 |
| openstackgerrit | Merged opendev/system-config master: ask.o.o : workaround old puppet-solr package https://review.opendev.org/659473 | 08:39 |
| *** dabukalam has joined #openstack-infra | 08:41 | |
| zigo | clarkb: Oh, great, thanks ! :) | 08:42 |
| *** derekh has joined #openstack-infra | 08:43 | |
| *** jistr|mtg is now known as jistr | 08:46 | |
| *** xek has joined #openstack-infra | 08:48 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:56 | |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Add openstack-tox-py37 job https://review.opendev.org/610659 | 08:56 |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Resolve some issue with tox.ini, setup.cfg https://review.opendev.org/660297 | 08:56 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 09:02 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 09:02 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 09:02 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 09:02 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 09:02 |
| *** diablo_rojo has quit IRC | 09:03 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 09:03 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 09:03 |
| *** sreejithp has joined #openstack-infra | 09:03 | |
| *** dtantsur is now known as dtantsur|brb | 09:06 | |
| *** sreejithp has quit IRC | 09:07 | |
| *** openstackstatus has quit IRC | 09:13 | |
| *** openstackstatus has joined #openstack-infra | 09:14 | |
| *** ChanServ sets mode: +v openstackstatus | 09:14 | |
| *** gregoryo has quit IRC | 09:15 | |
| *** pkopec has joined #openstack-infra | 09:21 | |
| *** jamesmcarthur has joined #openstack-infra | 09:26 | |
| *** electrofelix has joined #openstack-infra | 09:30 | |
| *** jamesmcarthur has quit IRC | 09:31 | |
| *** markvoelker has joined #openstack-infra | 09:35 | |
| *** diablo_rojo has joined #openstack-infra | 09:38 | |
| *** ramishra has quit IRC | 09:46 | |
| diablo_rojo | Maybe a dumb question/ situation, but bear with me. I used to have ssh access to storyboard-dev. I *thought* that I had transferred my ssh key from my old laptop to my new one, but I legit cant find the file in system-config where that info had been kept. Am I blind? Or did that all change? | 09:48 |
| *** ramishra has joined #openstack-infra | 09:49 | |
| *** pkopec has quit IRC | 10:08 | |
| *** dpawlik has quit IRC | 10:09 | |
| *** markvoelker has quit IRC | 10:09 | |
| *** dpawlik has joined #openstack-infra | 10:09 | |
| *** pkopec has joined #openstack-infra | 10:10 | |
| *** Lucas_Gray has joined #openstack-infra | 10:11 | |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Use more verbose logging for test failure https://review.opendev.org/660339 | 10:14 |
| openstackgerrit | Mark Goddard proposed opendev/irc-meetings master: Update Kolla meeting chair to mgoddard https://review.opendev.org/660344 | 10:26 |
| *** dtantsur|brb is now known as dtantsur | 10:30 | |
| *** udesale has quit IRC | 10:46 | |
| *** amoralej is now known as amoralej|lunch | 11:01 | |
| *** snapiri has quit IRC | 11:03 | |
| *** markvoelker has joined #openstack-infra | 11:06 | |
| *** panda is now known as panda|rover|eat | 11:13 | |
| *** nicolasbock has joined #openstack-infra | 11:15 | |
| *** sreejithp has joined #openstack-infra | 11:20 | |
| *** dave-mccowan has joined #openstack-infra | 11:26 | |
| *** jamesmcarthur has joined #openstack-infra | 11:27 | |
| *** jamesmcarthur has quit IRC | 11:33 | |
| *** markvoelker has quit IRC | 11:39 | |
| *** sreejithp has quit IRC | 11:40 | |
| *** sreejithp has joined #openstack-infra | 11:40 | |
| *** jamesmcarthur has joined #openstack-infra | 11:43 | |
| *** sreejithp has quit IRC | 11:44 | |
| *** bobh has joined #openstack-infra | 11:45 | |
| *** bobh has quit IRC | 11:46 | |
| openstackgerrit | Thomas Goirand proposed openstack/project-config master: Add a Debian Buster image. https://review.opendev.org/645574 | 11:47 |
| *** rascasoft has quit IRC | 11:52 | |
| *** janki has quit IRC | 11:54 | |
| *** jamesmcarthur has quit IRC | 11:55 | |
| *** jamesmcarthur has joined #openstack-infra | 12:00 | |
| *** jpena is now known as jpena|lunch | 12:00 | |
| mordred | diablo_rojo: https://opendev.org/opendev/system-config/src/branch/master/playbooks/group_vars/all.yaml#L153-L157 | 12:02 |
| mordred | diablo_rojo: and https://opendev.org/opendev/system-config/src/branch/master/playbooks/group_vars/storyboard-dev.yaml#L4 | 12:03 |
| mordred | diablo_rojo: so I agree - you should have access to storyboard-dev, assuming you have the ssh key - but if you don't, there's where you can update the public key | 12:03 |
| *** sreejithp has joined #openstack-infra | 12:06 | |
| *** sreejithp has quit IRC | 12:12 | |
| diablo_rojo | mordred, thank you! I will get that updated. | 12:15 |
| *** jamesmcarthur has quit IRC | 12:16 | |
| *** jamesmcarthur has joined #openstack-infra | 12:16 | |
| *** udesale has joined #openstack-infra | 12:16 | |
| *** bhavikdbavishi has quit IRC | 12:23 | |
| *** amoralej|lunch is now known as amoralej | 12:25 | |
| pabelanger | clarkb: re kata / github yesterday, I can confirm I was seeing delayed events from github yesterday, sometime upwards of 20mins. | 12:27 |
| pabelanger | so, maybe zuul.o.o was seeing the same thing from kata PRs | 12:27 |
| *** panda|rover|eat is now known as panda|rover | 12:27 | |
| openstackgerrit | Merged opendev/storyboard master: Clarify the rationale for StoryBoard's unique design https://review.opendev.org/657633 | 12:30 |
| *** pkopec has quit IRC | 12:31 | |
| *** jamesmcarthur has quit IRC | 12:32 | |
| *** diablo_rojo has quit IRC | 12:33 | |
| *** lseki has joined #openstack-infra | 12:34 | |
| *** psachin has quit IRC | 12:34 | |
| *** priteau has quit IRC | 12:36 | |
| *** rlandy has joined #openstack-infra | 12:37 | |
| *** priteau has joined #openstack-infra | 12:39 | |
| *** jamesmcarthur has joined #openstack-infra | 12:42 | |
| *** Goneri has joined #openstack-infra | 12:43 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml https://review.opendev.org/640027 | 12:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files https://review.opendev.org/660374 | 12:44 |
| mordred | clarkb: ^^ I updated the control plane patch to match current reality - then made a followup patch to clean up the unused clouds from everywhere | 12:45 |
| *** rfarr has joined #openstack-infra | 12:45 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml https://review.opendev.org/640027 | 12:49 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files https://review.opendev.org/660374 | 12:49 |
| *** rascasoft has joined #openstack-infra | 12:50 | |
| *** priteau has quit IRC | 12:51 | |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Build ubuntu bionic minimal images for control plane https://review.opendev.org/640044 | 12:51 |
| mordred | clarkb, AJaeger: ^^ I also updated the nodepool config file patch that depends on it to rename the cloud entries. "openstackci" seems like an antiquated way of saying "control plane" - so I thought maybe just saying "control-plane" would be better | 12:52 |
| *** jpena|lunch is now known as jpena | 12:52 | |
| *** rascasoft has quit IRC | 12:56 | |
| *** tobias-urdin has quit IRC | 13:03 | |
| *** pkopec has joined #openstack-infra | 13:05 | |
| *** mriedem has joined #openstack-infra | 13:07 | |
| *** cgoncalves has quit IRC | 13:08 | |
| *** eharney has quit IRC | 13:11 | |
| *** janki has joined #openstack-infra | 13:11 | |
| *** hwoarang has quit IRC | 13:21 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files https://review.opendev.org/660374 | 13:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add the mirror-only cloud regions to nodepool builder https://review.opendev.org/660384 | 13:22 |
| *** hwoarang has joined #openstack-infra | 13:23 | |
| *** cgoncalves has joined #openstack-infra | 13:23 | |
| *** sreejithp has joined #openstack-infra | 13:26 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files https://review.opendev.org/660374 | 13:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add the mirror-only cloud regions to nodepool builder https://review.opendev.org/660384 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add control plane mirror-only cloud regions https://review.opendev.org/660388 | 13:30 |
| *** rkukura has quit IRC | 13:31 | |
| *** rossella_s has quit IRC | 13:34 | |
| *** ramishra has quit IRC | 13:37 | |
| *** Lucas_Gray has quit IRC | 13:38 | |
| *** jamesmcarthur has quit IRC | 13:39 | |
| *** Lucas_Gray has joined #openstack-infra | 13:39 | |
| *** ramishra has joined #openstack-infra | 13:42 | |
| *** bhavikdbavishi has joined #openstack-infra | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:48 | |
| *** jchhatbar has joined #openstack-infra | 13:57 | |
| *** rf0lc0 has joined #openstack-infra | 13:57 | |
| *** kmarc has joined #openstack-infra | 13:57 | |
| *** mnencia_ has joined #openstack-infra | 13:58 | |
| *** calbers_ has joined #openstack-infra | 13:58 | |
| *** jamesmcarthur_ has joined #openstack-infra | 13:59 | |
| *** Wryhder has joined #openstack-infra | 13:59 | |
| *** rh-jelabarre has joined #openstack-infra | 14:00 | |
| *** ekultails has joined #openstack-infra | 14:00 | |
| *** gary_perkins_ has joined #openstack-infra | 14:00 | |
| *** gfidente|afk is now known as gfidente | 14:00 | |
| *** lifeless_ has joined #openstack-infra | 14:00 | |
| *** tobiash_ has joined #openstack-infra | 14:01 | |
| *** Anticime1 has joined #openstack-infra | 14:01 | |
| *** pgaxatte1 has joined #openstack-infra | 14:01 | |
| *** ktsuyuzaki has joined #openstack-infra | 14:04 | |
| *** pgaxatte1 has quit IRC | 14:05 | |
| *** ddurst- has joined #openstack-infra | 14:05 | |
| *** aspiers_ has joined #openstack-infra | 14:05 | |
| *** pgaxatte1 has joined #openstack-infra | 14:05 | |
| *** tosky__ has joined #openstack-infra | 14:05 | |
| *** Lucas_Gray has quit IRC | 14:06 | |
| *** janki has quit IRC | 14:06 | |
| *** mriedem has quit IRC | 14:06 | |
| *** derekh has quit IRC | 14:06 | |
| *** tosky has quit IRC | 14:06 | |
| *** zhurong has quit IRC | 14:06 | |
| *** jamesdenton has quit IRC | 14:06 | |
| *** kota_ has quit IRC | 14:06 | |
| *** zer0c00l has quit IRC | 14:06 | |
| *** SpamapS has quit IRC | 14:06 | |
| *** rfolco has quit IRC | 14:06 | |
| *** markmcclain has quit IRC | 14:06 | |
| *** lifeless has quit IRC | 14:06 | |
| *** dosaboy has quit IRC | 14:06 | |
| *** amoralej has quit IRC | 14:06 | |
| *** andreykurilin has quit IRC | 14:06 | |
| *** mnencia has quit IRC | 14:06 | |
| *** bradm has quit IRC | 14:06 | |
| *** Wryhder is now known as Lucas_Gray | 14:06 | |
| *** mnencia_ is now known as mnencia | 14:06 | |
| *** jamesmcarthur has quit IRC | 14:06 | |
| *** udesale has quit IRC | 14:06 | |
| *** electrofelix has quit IRC | 14:06 | |
| *** ralonsoh has quit IRC | 14:06 | |
| *** pgaxatte has quit IRC | 14:06 | |
| *** sshnaidm has quit IRC | 14:06 | |
| *** toabctl has quit IRC | 14:06 | |
| *** harlowja has quit IRC | 14:06 | |
| *** calbers has quit IRC | 14:06 | |
| *** dayou_ has quit IRC | 14:06 | |
| *** aspiers has quit IRC | 14:06 | |
| *** smcginnis has quit IRC | 14:06 | |
| *** ddurst has quit IRC | 14:06 | |
| *** gary_perkins has quit IRC | 14:06 | |
| *** Anticimex has quit IRC | 14:06 | |
| *** strigazi has quit IRC | 14:06 | |
| *** edmondsw_ has quit IRC | 14:06 | |
| *** tobiash has quit IRC | 14:06 | |
| *** kmarc_ has quit IRC | 14:06 | |
| *** calbers_ is now known as calbers | 14:06 | |
| *** aspiers_ is now known as aspiers | 14:06 | |
| *** rpittau is now known as rpittau|afk | 14:06 | |
| *** udesale has joined #openstack-infra | 14:07 | |
| *** electrofelix has joined #openstack-infra | 14:07 | |
| *** mriedem has joined #openstack-infra | 14:08 | |
| *** rf0lc0 is now known as rfolco | 14:12 | |
| *** mattw4 has joined #openstack-infra | 14:12 | |
| *** aaronsheffield has joined #openstack-infra | 14:12 | |
| *** eharney has joined #openstack-infra | 14:12 | |
| *** sshnaidm has joined #openstack-infra | 14:12 | |
| *** ralonsoh has joined #openstack-infra | 14:12 | |
| *** dayou_ has joined #openstack-infra | 14:13 | |
| *** SpamapS has joined #openstack-infra | 14:13 | |
| *** harlowja has joined #openstack-infra | 14:14 | |
| *** armax has joined #openstack-infra | 14:26 | |
| *** lpetrut has quit IRC | 14:27 | |
| *** _erlon_ has joined #openstack-infra | 14:27 | |
| *** ykarel is now known as ykarel|afk | 14:27 | |
| *** bhavikdbavishi has quit IRC | 14:28 | |
| *** rascasoft has joined #openstack-infra | 14:33 | |
| *** mattw4 has quit IRC | 14:35 | |
| *** liuyulong has joined #openstack-infra | 14:39 | |
| zigo | clarkb: mordred: fungi: ianw: https://review.opendev.org/#/c/645574/ <--- Hopefully, this will work after debootstrap is fixed! | 14:43 |
| zigo | Please review. | 14:43 |
| *** rascasoft has quit IRC | 14:44 | |
| *** chandankumar is now known as raukadah | 14:46 | |
| clarkb | zigo: ianw did most of the work. I just helped ensure it got in happily at the end | 14:55 |
| clarkb | mordred: cool I'll take a look at those changes after my morning meeting | 14:55 |
| zigo | clarkb: Thanks a lot. :P | 14:56 |
| clarkb | pabelanger: it happens due to needing to lookup PRs via commit hashes and there is no direct api for that so zuul ends up scanning all the PRs. In most cases this is actually a one time cost because zuul scans the PRs and then caches the results. The problems arise with post merge testing which ansible uses extensively because the cache is out of data if multiple non fast forwardable PRs merge in | 14:57 |
| clarkb | sequence (as only the first can have a cached hash, subsequent PRs all get new hashes that are uncached) | 14:57 |
| clarkb | s/out of data/out of date/ | 14:58 |
| *** tosky__ is now known as tosky | 14:59 | |
| *** jchhatbar is now known as janki | 15:02 | |
| *** hamzy has quit IRC | 15:07 | |
| *** LordGT has joined #openstack-infra | 15:08 | |
| fungi | skopeo (0.1.37-1~dev~ubuntu16.04.2~ppa2) xenial; urgency=medium * autobuilt 0c90e57 -- Lokesh Mandvekar (Bot) <...> Tue, 21 May 2019 03:44:55 +0000 | 15:23 |
| fungi | https://github.com/containers/skopeo/commit/0c90e57 | 15:23 |
| fungi | that's what we've been waiting for | 15:23 |
| *** priteau has joined #openstack-infra | 15:24 | |
| * fungi un-wips and approves his revert | 15:24 | |
| clarkb | \o/ | 15:24 |
| * fungi checks apt cache show on ze01 to be sure | 15:24 | |
| clarkb | I think we'll be ready to approve https://review.opendev.org/#/c/647877/3 today too (though we should probably double check new ask is happy with ianw first) | 15:25 |
| fungi | apt show skopeo|grep ^Version: | 15:26 |
| fungi | Version: 0.1.37-1~dev~ubuntu16.04.2~ppa2 | 15:26 |
| fungi | looks like we're ready | 15:26 |
| clarkb | what is the current version? | 15:26 |
| clarkb | (that version looks a lot like the one I remember from yseterday) | 15:26 |
| fungi | adding -a to the apt show command reveals 0.1.36-1~dev~ubuntu16.04.2~ppa19.1 | 15:26 |
| fungi | which jives with the installed version reported by dpkg -l too | 15:27 |
| fungi | the .1 in the package revision is what differentiates my hand-built package from the one in the ppa | 15:28 |
| *** sreejithp_ has joined #openstack-infra | 15:28 | |
| fungi | so that was the source package for 0.1.36-1~dev~ubuntu16.04.2~ppa19 patched with tristanC's pr and then built in a local xenial chroot | 15:28 |
| clarkb | got it | 15:28 |
| *** sreejithp__ has joined #openstack-infra | 15:30 | |
| clarkb | mordred: https://review.opendev.org/#/c/640027/11 requires an update to the secret ansible vars right? for the control plane clouds group? | 15:31 |
| *** sreejithp has quit IRC | 15:31 | |
| *** sreejithp_ has quit IRC | 15:32 | |
| clarkb | fungi: you've been more up to date on the arm and linaro cloud situation iirc care to look at https://review.opendev.org/#/c/660374/4 just to double check we are doing the right things there? | 15:33 |
| dmsimard | I have a weird issue when trying to use "virtualenv_command: /usr/bin/python3 -m venv" instead of "virtualenv_python: python3", has anyone else see something like it ? Looks like pip gives up if it can't find a wheel for a package. | 15:34 |
| dmsimard | With virtualenv_python it works: http://logs.openstack.org/14/660414/1/check/ansible-role-ara-api-ubuntu/e4dbc27/ara-report/result/e061c0a3-3ca0-47fe-8aa7-c9825f84de0b/ | 15:34 |
| dmsimard | With virtualenv_command it doesn't: http://logs.openstack.org/20/660420/1/check/ansible-role-ara-api-ubuntu/fae8f2f/ara-report/result/1d5255ec-83bc-4a87-9aa3-7590daa19a9b/ | 15:34 |
| dmsimard | FWIW the virtualenv_command works on Fedora but not on Ubuntu | 15:35 |
| dmsimard | using virtualenv_command on Fedora: http://logs.openstack.org/20/660420/1/check/ansible-role-ara-api-fedora/7071797/ara-report/result/a35a86bf-fb73-4447-b03b-5dcb2e72f2a2/ | 15:35 |
| *** hamzy has joined #openstack-infra | 15:35 | |
| fungi | dmsimard: part of the problem may be that python3 -m venv is going to use the venv module shipped with the python stdlib, while virtualenv is a separate python package versioned independently of the interpreter | 15:36 |
| clarkb | worht noting that python -m venv != virtualenv | 15:36 |
| dmsimard | yeah python3-venv is installed on ubuntu prior to trying that | 15:36 |
| dmsimard | I'll try reproducing without Ansible at all | 15:36 |
| fungi | right, so the venv module will have features contemporary with the python release from which it's shipped | 15:37 |
| clarkb | fungi: dmsimard ya I think that is likely it. the venv module is going to be ancient (whatever shipped with the distro) whereas virtualenv should be up to date on our images | 15:37 |
| clarkb | in particular this may be the issue I managed to get pip to fix | 15:37 |
| clarkb | where it bails out early rather than trying additional indexes | 15:37 |
| fungi | ahh, right, it'll also be using the pip version vendored with the python interpreter rather than the version vendored into latest virtualenv package | 15:38 |
| clarkb | debian (via zigo) fixed this in their pip packaging | 15:38 |
| clarkb | but I don't know that ubuntu has | 15:38 |
| fungi | so less likely this is actually a venv vs virtualenv behavior issue, and more likely it's just a question of which version of pip is being invoked | 15:38 |
| *** pgaxatte1 has quit IRC | 15:40 | |
| *** LordGT has quit IRC | 15:41 | |
| fungi | https://metadata.ftp-master.debian.org/changelogs//main/p/python-pip/python-pip_18.1-5_changelog shows clarkb's fix in the version for sid/buster | 15:41 |
| fungi | the job in question ran on our ubuntu-bionic image | 15:42 |
| dmsimard | I'm unable to reproduce on a local 18.04 vm, I'll hold a node to troubleshoot | 15:43 |
| *** michael-beaver has joined #openstack-infra | 15:43 | |
| fungi | looks like that version has been pulled into ubuntu starting in 19.04 | 15:44 |
| clarkb | mordred: comments on https://review.opendev.org/#/c/640044/3 I think our minimal image is still a bit too minimal | 15:44 |
| fungi | the version in bionic-updates has this changelog, which doesn't indicate any backport of that fix (or any update at all in over a year): http://changelogs.ubuntu.com/changelogs/pool/universe/p/python-pip/python-pip_9.0.1-2.3~ubuntu1/changelog | 15:45 |
| fungi | we could file an sru request if that hasn't been done already | 15:45 |
| *** ykarel|afk is now known as ykarel|away | 15:45 | |
| fungi | though i have no idea how easily that patch can be backported to the pip 9.0.1 codebase | 15:46 |
| *** kskels has joined #openstack-infra | 15:46 | |
| *** smarcet has joined #openstack-infra | 15:46 | |
| mordred | clarkb: I hear you on python and haveged - I'd argue that the other things are better if we express them in the base role. BUT - I don't feel *strongly* about that | 15:48 |
| mordred | clarkb: and yes - 640027 still requires an update to secret ansible vars | 15:48 |
| mordred | clarkb: maybe what we shoudl do is do that move in 2 parts ... copy to the new file now, then land the change, then delete the old file | 15:49 |
| clarkb | mordred: ya that should work safely | 15:49 |
| clarkb | mordred: ya I don't mean to make that list an absolute suggestion. Was more here is all the stuff we are likely to want on our servers in production that we don't currently get | 15:50 |
| clarkb | haveged is importand as is python for base functionality though | 15:50 |
| fungi | i think you could get a newer pip by using python3.7 in bionic, though there doesn't seem to be a distinct python3.7-pip package (there is a python3.7-venv package but neither it nor libpython3.7-stdlib seem to include the pip module) | 15:50 |
| fungi | and anyway, the source package was last updated in october | 15:50 |
| *** Lucas_Gray has quit IRC | 15:51 | |
| clarkb | for example we use iptables rules on all servers | 15:51 |
| clarkb | maybe we should have the base package for that in the image | 15:51 |
| clarkb | ntp is in that boat too | 15:51 |
| dmsimard | fungi: let me get back to you on that once I have more details and a way to reproduce it | 15:51 |
| fungi | cool | 15:51 |
| *** ykarel|away has quit IRC | 15:51 | |
| *** raissa has joined #openstack-infra | 15:51 | |
| fungi | g'luck! | 15:51 |
| dmsimard | fungi: yeah, it definitely reproduces in the nodepool vm but doesn't in my local vm | 15:52 |
| fungi | dmsimard: may want to confirm which version of pip is being invoked | 15:53 |
| clarkb | dmsimard: does your local vm talk to our mirrors? | 15:53 |
| openstackgerrit | Merged opendev/system-config master: Revert "Pin skopeo to unbreak skopeo+bubblewrap" https://review.opendev.org/659908 | 15:53 |
| dmsimard | clarkb: does not, it's a vanilla ubuntu server 18.04 | 15:53 |
| clarkb | dmsimard: the issue here is there is no wheel package for that package on our mirror (this is expected). Pip is supposed to look at all theo ther indexes like pypi for that package; hwoever pip has had broken versions (that I helped fix since) where it won't do that | 15:53 |
| clarkb | dmsimard: ok thats why you aren't reproducing locally then | 15:53 |
| dmsimard | yeah if I move /etc/pip.conf elsewhere it works | 15:54 |
| *** e0ne has quit IRC | 15:55 | |
| *** janki has quit IRC | 15:55 | |
| clarkb | the fix is to either use up to date pip (with my fixing) or don't use package mirrors that don't have all the packages you need | 15:55 |
| fungi | specifically it's our wheel caches which are the issue, right? | 15:56 |
| fungi | not our pypi proxy? | 15:56 |
| dmsimard | can you specify a fallback index ? | 15:56 |
| clarkb | fungi: our wheel caches not being a complete mirror of pypi trigger the bug in pypi | 15:56 |
| clarkb | s/pypi/pip/ | 15:56 |
| dmsimard | or, well, more than one index | 15:56 |
| clarkb | dmsimard: the bug is that pip doesn't use those alternative indexes | 15:56 |
| clarkb | dmsimard: it is supposed to (and I helped fix it so that it does) | 15:56 |
| *** smcginnis has joined #openstack-infra | 15:57 | |
| *** ccamacho has quit IRC | 15:57 | |
| fungi | clarkb: right, but point being a job could work around this by avoiding use of our wheel cache while still benefiting from our pypi proxy | 15:57 |
| clarkb | fungi: yes, or using an up to date pip | 15:57 |
| *** kskels has left #openstack-infra | 15:57 | |
| fungi | sure | 15:57 |
| *** ginopc has quit IRC | 15:59 | |
| dmsimard | so the pip.conf as set up by infra has "index-url = http://mirror.iad.rax.openstack.org/pypi/simple" and "extra-index-url = http://mirror.iad.rax.openstack.org/wheel/ubuntu-18.04-x86_64" | 16:00 |
| dmsimard | I tried using "index-url = http://mirror.iad.rax.openstack.org/pypi/simple" and "extra-index-url = https://pypi.python.org/simple/" and that works | 16:01 |
| clarkb | dmsimard: yup beacuse both are complete mirrors of pypi | 16:01 |
| clarkb | (they don't lack any packages like our wheel mirror) | 16:01 |
| dmsimard | clarkb: pip still pulls the wheels from our mirror though | 16:01 |
| clarkb | dmsimard: not from our wheel mirror | 16:01 |
| clarkb | (because you removed it) | 16:02 |
| dmsimard | clarkb: am I reading this wrong? http://paste.openstack.org/show/751889/ | 16:02 |
| fungi | that's the pypi proxy, not the wheel cache | 16:02 |
| *** rascasoft has joined #openstack-infra | 16:02 | |
| clarkb | dmsimard: no those are wheels that are put on pypi by those packages | 16:03 |
| clarkb | dmsimard: our wheel mirror builds wheels for all packages on specific platforms | 16:03 |
| clarkb | dmsimard: because many packages on pypi don't supply wheels | 16:03 |
| clarkb | libvirt is one example | 16:03 |
| fungi | well, more that many packages on pypi can't provide generic wheels | 16:03 |
| fungi | pure python packages can easily, some of the more simple non-pure python packages can build manylinux1 compliant wheels | 16:04 |
| dmsimard | ok, I understand now :D | 16:04 |
| fungi | but other more complex packages need to build platform-specific wheels | 16:04 |
| clarkb | fungi: which fedora 30 has broken btw due to changes in libcrypt | 16:04 |
| fungi | and we pre-build platform-specific wheels to save every job from having to build those locally from the sdist on the fly, which can take a very long time in some cases | 16:05 |
| fungi | which is what's being served at the /wheel/ubuntu-18.04-x86_64 paths | 16:05 |
| fungi | (and similar for other platforms) | 16:05 |
| roman_g | Hello dear Infra team. How could we get e-mail notification for failed post-merge jobs? | 16:06 |
| clarkb | the underlying bug in distro pip was due to broken monkey patching done by pip for distros so that they could unvendor requests. This caused pip to try and catch the wrong exception which led to uncaught exceptions from requests when the lookup fails because package doesn't exist (and no other indexes were checked as a result) | 16:06 |
| *** rascasoft has quit IRC | 16:07 | |
| openstackgerrit | Nate Johnston proposed opendev/irc-meetings master: Update neutron-ci agenda and chairs https://review.opendev.org/660443 | 16:07 |
| clarkb | roman_g: zuul has an email reporter, but that has to be added to the pipeline (so typically isn't great if different jobs need to report to different email addrs) | 16:07 |
| clarkb | roman_g: openstack health has rss/atom feeds for openstack jobs. I think this has been the suggested method for subscribing to those events | 16:07 |
| clarkb | we may want to have zuul provide that directly in a generic manner | 16:08 |
| dmsimard | fungi, clarkb: I'll work around this for now by avoiding the wheel index -- thanks for your help :D | 16:09 |
| *** panda|rover has quit IRC | 16:09 | |
| roman_g | clarkb: good, thanks. Could we may be get openstackgerrit bot notify us in the channel? | 16:09 |
| roman_g | or other bot | 16:10 |
| *** panda has joined #openstack-infra | 16:10 | |
| *** smarcet has quit IRC | 16:11 | |
| clarkb | roman_g: I think openstack's gerritbot can actually do that | 16:11 |
| clarkb | it can trigger on verified -1 and -2 votes iirc | 16:11 |
| clarkb | oh except in post there is no -1 or -2 hrm | 16:11 |
| roman_g | yes, yes. | 16:11 |
| *** smarcet has joined #openstack-infra | 16:12 | |
| roman_g | it's Zuul's responsibility to report something | 16:12 |
| clarkb | roman_g: also do you know about the zuul builds dashboard? | 16:12 |
| roman_g | clarkb: yes, I do. But prefer to have either e-mail, or something in IRC, or better both. | 16:13 |
| roman_g | specifically for post-merge | 16:14 |
| roman_g | because once we WF+1 - no one cares if those jobs were successful or not. No one would get back to Zuul dashboard and manually check post-merge jobs. | 16:15 |
| *** gyee has joined #openstack-infra | 16:15 | |
| clarkb | yup unfortunately this is true even when we sent a ton of email to people on those failures (the openstack-qa list for years got all the periodic and post test job failure reports iirc) | 16:15 |
| clarkb | which is why I think we haven't put a lot of effort into improving the feature | 16:15 |
| clarkb | that said I think a subscription model like rss or atom would be a great reporter addition | 16:16 |
| clarkb | because then anyone could subscribe as necessary and add their own filters | 16:16 |
| roman_g | if we get e-mail with failure report, then we could set up a filter in e-mail client and watch for failures | 16:16 |
| roman_g | btw, where is the RSS? | 16:17 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile https://review.opendev.org/660191 | 16:17 |
| *** rosmaita has quit IRC | 16:18 | |
| clarkb | roman_g: rss is currently provided by http://status.openstack.org/openstack-health/#/ | 16:18 |
| clarkb | roman_g: the problem with email is in determining the targets | 16:18 |
| clarkb | roman_g: we probably don't want anyone to arbitrarily decide where email should be sent (because spam) | 16:18 |
| clarkb | roman_g: so currently that is a per pipeline setting (which is config with a bit more oversight) | 16:19 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile https://review.opendev.org/660191 | 16:19 |
| clarkb | the benefit of rss/atom is that it is subscription based so zuul could publish to an endpoint and people can opt in to subscribe | 16:19 |
| *** igordc has joined #openstack-infra | 16:20 | |
| roman_g | clarkb: we are planning to have mailing list for failures & automated reports (e.g. for periodic jobs) | 16:20 |
| fungi | yikes, wiki.o.o has a load average of 90 righth now | 16:20 |
| *** smarcet has quit IRC | 16:21 | |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Retire ansible-role-k8s-* repos https://review.opendev.org/660446 | 16:21 |
| clarkb | roman_g: so that will work as long as you have your own pipelines (which I think isn't a major issue if we get people split into tenants, but prior to that we'd probably not want to add a ton of project specific pipelines) | 16:21 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile https://review.opendev.org/660191 | 16:21 |
| roman_g | clarkb: yes, that is a bit problematic (i'd need to add param to common jobs) | 16:22 |
| roman_g | but still doable | 16:22 |
| clarkb | roman_g: it isn't a job parameter, it is a piepline parameter | 16:22 |
| fungi | oh yeah, the wiki server's being flooded with requests right now | 16:22 |
| clarkb | (pipelines determine what reports are made) | 16:22 |
| *** smarcet has joined #openstack-infra | 16:22 | |
| roman_g | ahha, reported is a pipeline param... good | 16:23 |
| clarkb | fungi: new wave of spam? | 16:23 |
| roman_g | thank you for the consultation, clarkb. I need to digest it, and will then start working. | 16:23 |
| fungi | looks like getting flooded with random attempts at php script compromises | 16:23 |
| fungi | i see tons of requests for things like /wiki/info/infoShowAction.do?method=getAccessoryByManager&id=../../../../../../../../etc/passwd%00.jpg&accessory=1 | 16:24 |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: Install more dependencies for integration testing https://review.opendev.org/660449 | 16:24 |
| dmsimard | clarkb, fungi: just confirming that taking out the extra-index-url from pip.conf resolved my issue, thanks <3 | 16:26 |
| clarkb | dmsimard: fwiw I would use virtualenv over -m venv due to issues like this | 16:27 |
| clarkb | rather than editing the pip configs | 16:27 |
| clarkb | (because you get up to date pip that way) | 16:27 |
| *** ricolin has quit IRC | 16:27 | |
| dmsimard | clarkb: unfortunately doing the other way around brings other problems so I'm trying to find something that works for both :D | 16:27 |
| fungi | #status log blocked all traffic to wiki.openstack.org from 61.164.47.194 using an iptables drop rule to quell a denial of service condition | 16:29 |
| openstackstatus | fungi: finished logging | 16:29 |
| fungi | load average for wiki.o.o is back under 1 again now | 16:32 |
| fungi | i'll keep an eye on the access log for a while in case it starts back up again | 16:32 |
| *** smarcet has quit IRC | 16:36 | |
| *** mattw4 has joined #openstack-infra | 16:36 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Drop jobs for devstack-plugins-sheepdog https://review.opendev.org/660452 | 16:36 |
| *** raissa has quit IRC | 16:36 | |
| *** raissa has joined #openstack-infra | 16:37 | |
| *** raissa has quit IRC | 16:37 | |
| *** raissa has joined #openstack-infra | 16:38 | |
| *** raissa has quit IRC | 16:38 | |
| *** raissa has joined #openstack-infra | 16:39 | |
| *** raissa has quit IRC | 16:39 | |
| *** raissa has joined #openstack-infra | 16:40 | |
| *** raissa has quit IRC | 16:40 | |
| *** ramishra has quit IRC | 16:42 | |
| clarkb | mwhahaha: fedora 28 is the version of fedora that tripleo is using to test python3 transition work right? And ya'll expect to need that until we have centos 8 images? | 16:43 |
| * clarkb trying to figure out which images we can clean up with the impedning removal of trusty | 16:43 | |
| mordred | clarkb: I'll update the control plane images patches in just a sec - debugging the gitea 1.8.0 patch atm ... | 16:45 |
| mordred | I'm getting this: | 16:45 |
| mordred | 2019/05/21 16:44:17 ...s/setting/setting.go:738:NewContext() [F] error saving generating JWT secret to custom config: open /custom/conf/app.ini: permission denied | 16:46 |
| mordred | so apparently in 1.8.0 they're wanting to do that thing where they write something to a config file :( | 16:46 |
| clarkb | ugh | 16:46 |
| clarkb | mordred: I'm guessing the jwt secret is something they want to generate and not somethign we can stick in our images ;) | 16:47 |
| clarkb | mordred: maybe the simplest fix is to change the perms on the file? | 16:47 |
| *** smarcet has joined #openstack-infra | 16:47 | |
| *** lucasagomes has quit IRC | 16:48 | |
| *** dtantsur is now known as dtantsur|afk | 16:48 | |
| mordred | clarkb: well - I'm investigating to see if it's one of those things where maybe we can set the setting in our app.ini ourselves first | 16:48 |
| mordred | like - maybe they generate if you dont have one already | 16:48 |
| Shrews | mordred: 'chmod -R o+rw /' will fix it | 16:48 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Remove devstack-plugin-sheepdog https://review.opendev.org/660454 | 16:49 |
| mordred | Shrews: that's a great suggestion | 16:49 |
| Shrews | \o/ | 16:49 |
| clarkb | mordred: ya but its also a secret right? | 16:49 |
| *** whoami-rajat has quit IRC | 16:49 | |
| clarkb | I guess we can bind mount that in then have our init thing sed it into the file? | 16:49 |
| clarkb | (or just bind mount the entire file I suppose) | 16:50 |
| *** armax has quit IRC | 16:52 | |
| *** udesale has quit IRC | 16:53 | |
| *** smarcet has quit IRC | 16:54 | |
| * clarkb finds breakfast | 16:55 | |
| mordred | clarkb: ok. turns out the issue is that what's in our file isn't a base64 encoded secret | 16:55 |
| mordred | clarkb: we just have a random string of characters | 16:55 |
| mordred | this is also true of our real secret on bridge | 16:55 |
| mordred | actually - it turns out this is also not the actual issue | 16:58 |
| mwhahaha | clarkb: yes but we're trying to get an alternative. Might want to check with weshay | 16:59 |
| weshay | what's the context? | 17:01 |
| clarkb | weshay: I'm taking stock of the images we may be able to clean up and wanted to check if fedora 28 was still used by tripleo | 17:01 |
| fungi | keeping f28 images until centos8 is out | 17:01 |
| clarkb | in particular as the python3 test bed until centos 8 happens | 17:01 |
| fungi | (rather than switching to f29 or f30 i suppose) | 17:02 |
| clarkb | part of the motiviation here is zigo wants to add buster images and monty wants to add a control plane bionic image and we already can't support all the images we have when we lose a builder due to disk sizes | 17:02 |
| *** kashyap has joined #openstack-infra | 17:05 | |
| weshay | ya.. so yes we are still using it atm. However I wouldn't stand in your way if you needed to kill it. We're waiting on some items for rhel-8 in rdo zuul but hopefully it won't take too long, few more week. That should help bootstrap centos-8 when it becomes a thing | 17:05 |
| kashyap | Hi folks, can someone remind me -- where do we capture guest boot log, if at all, during CI run? | 17:05 |
| weshay | so if you guys can hold out a few more weeks that would be super, but if we can't, that is ok too | 17:06 |
| clarkb | I don't think we haev to kill it. Was mostly checking if there was an easy win there | 17:06 |
| kashyap | E.g. should I find it somewhere here? -- http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/ | 17:06 |
| clarkb | kashyap: do you have an example job run (it will depend on the job itself) | 17:06 |
| kashyap | clarkb: Check above ^ | 17:06 |
| clarkb | kashyap: http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/libvirt/qemu/ there | 17:06 |
| weshay | k, thanks when we have 8 I'll definitely come back and say NUKE IT | 17:06 |
| clarkb | kashyap: oh the boot log | 17:06 |
| clarkb | kashyap: for that I think it is captured in the tempest log | 17:06 |
| kashyap | clarkb: It doesn't contain the full serial console log of the guest boot | 17:06 |
| kashyap | (Yeah; boot) | 17:06 |
| clarkb | kashyap: it will if ssh conenctivity failed iirc | 17:07 |
| kashyap | clarkb: And where can I find it? | 17:07 |
| clarkb | it may not do it unless that condition is tripped | 17:07 |
| *** mattw4 has quit IRC | 17:07 | |
| kashyap | Ah, I see. Trying to figure out why on earth guest fails to shutdown cleanly: | 17:07 |
| kashyap | http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/screen-n-cpu.txt.gz#_May_16_19_47_26_017509 | 17:08 |
| kashyap | Damned if I could put a finger on the cause | 17:08 |
| clarkb | kashyap: did you check syslog too? | 17:08 |
| kashyap | The return code from libvirt's shutdown request is '0'; that's "boring" (as in, expected) | 17:08 |
| kashyap | clarkb: This one? -- http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/syslog.txt.gz | 17:08 |
| clarkb | ya or even http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/devstack.journal.xz.gz | 17:08 |
| *** dpawlik has quit IRC | 17:08 | |
| clarkb | (we capture the systemwide journal too | 17:08 |
| kashyap | (I filtered the first one by ERROR; and didn't get anything.) | 17:09 |
| * kashyap clicks on the second | 17:09 | |
| clarkb | http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/devstack.journal.README.txt.gz is the notes on how to deal with that exported journal | 17:09 |
| *** quiquell has quit IRC | 17:10 | |
| clarkb | https://wiki.centos.org/About/Building_8 looks like there isn't anything for us to do on our end with centos 8 yet | 17:10 |
| kashyap | clarkb: Thanks; useful notes | 17:10 |
| clarkb | mordred: | ac2c4d17-8d6f-4e3c-a9eb-57c155f0a949 | v2-standard-2 | 8192 | 0 | 0 | 2 | True | is the flavor that I think new gitea servers can run under with 80GB volumes instead of 40 GB | 17:12 |
| *** igordc has quit IRC | 17:12 | |
| *** igordc has joined #openstack-infra | 17:13 | |
| kashyap | clarkb: Do you know WTF is this: "Bad message": | 17:13 |
| kashyap | $> journalctl --file ./devstack.journal -u 'devstack@n-api' | 17:13 |
| kashyap | Failed to open files: Bad message | 17:13 |
| kashyap | $> file devstack.journal | 17:13 |
| kashyap | devstack.journal: data | 17:13 |
| kashyap | I can `less` it, fine, though. | 17:13 |
| kashyap | s/it,/it/ | 17:13 |
| clarkb | kashyap: did you run /lib/systemd/systemd-journal-remote <(xzcat ./devstack.journal.xz) -o output.journal ? | 17:13 |
| kashyap | Bad me | 17:14 |
| clarkb | you need to do that format conversion for journalctl --file to work (and you do --file on the output of ^) | 17:14 |
| kashyap | I'm insufferably blind | 17:14 |
| kashyap | Thx | 17:14 |
| *** whoami-rajat has joined #openstack-infra | 17:15 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use local fork of gitea and upgrade to 1.8.0 https://review.opendev.org/655522 | 17:18 |
| mordred | clarkb: ok. I *think* that will fix it | 17:18 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects https://review.opendev.org/660461 | 17:18 |
| clarkb | silly new features | 17:19 |
| *** dpawlik has joined #openstack-infra | 17:20 | |
| fungi | the executors are still on the old skopeo packages... i wonder if this is because we ensure=>present instead of ensure=>latest and so wouldn't normally see it happen until unattended-upgrades runs later? | 17:20 |
| *** mattw4 has joined #openstack-infra | 17:20 | |
| clarkb | fungi: ya I think we ran into that with the debootstrap fix yesterday | 17:21 |
| clarkb | apt-get install debootstrap was enough for apt to figure out it could update | 17:21 |
| *** jpena is now known as jpena|off | 17:24 | |
| *** dpawlik has quit IRC | 17:24 | |
| clarkb | as a sanity check nb01 hasn't leaked any images. All the images in /opt/nodepool_dib are active images according to nodepool dib-image-list | 17:26 |
| clarkb | I think we'll be ok to add more images but it will start getting tight | 17:26 |
| *** armax has joined #openstack-infra | 17:27 | |
| clarkb | /opt is an lvm'd cinder volume so we can add another 1TB volume, add it to lvm vg then extend the filesystem if we need to | 17:27 |
| *** eharney has quit IRC | 17:27 | |
| fungi | wondering if i should go ahead and manually trigger the skopeo upgrades now while more folks are around rather than wait for them to upgrade at random times over the next 24 hours | 17:29 |
| clarkb | fungi: ++ | 17:29 |
| fungi | clarkb: nb03 will be the tricky one since we don't have cinder in that cloud so need to rebuild on a custom flavor with the desired rootfs size | 17:30 |
| clarkb | thinking out loud here on the gitea06 plan. I think we get mordred's control plane clouds change in, then add ubuntu-bionic-minimal image to control planes, then add journalctl fix from logan-'s change to those images, then we rebuild gitea06 on flavor v2-standard-2 with an 80GB disk | 17:30 |
| clarkb | fungi: nb03 is ok because it only builds like 2 images | 17:31 |
| fungi | k | 17:31 |
| *** kopecmartin is now known as kopecmartin|off | 17:31 | |
| clarkb | then if in the process of adding these new images we run out of disk we add 1TB extra disk to lvm | 17:31 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add example statsd_exporter mapping https://review.opendev.org/660472 | 17:33 |
| mordred | clarkb: I think that's a good plan | 17:34 |
| *** tobiash_ is now known as tobiash | 17:34 | |
| mordred | corvus: I just pushed up a gitea PR - watch out, I might catch up to you! | 17:35 |
| clarkb | mordred I want to double check with ianw today before approving logan-'s journal size fix (note not journalctl as mentioned previously) since he had some concern about its interaction with growroot | 17:35 |
| clarkb | but I expect we'll get that in today and it will start making test nodes happier | 17:35 |
| *** electrofelix has quit IRC | 17:36 | |
| clarkb | I also wanted to make sure ianw is ready for us to puppet-4 ask.o.o, but I expect we'll get that in today too completing our puppet-4 upgrades | 17:36 |
| clarkb | slow but steady progress | 17:36 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Add statsd_exporter mapping https://review.opendev.org/660473 | 17:37 |
| *** ralonsoh has quit IRC | 17:37 | |
| *** smarcet has joined #openstack-infra | 17:37 | |
| *** ykarel|away has joined #openstack-infra | 17:37 | |
| mordred | clarkb: working on the element for the packages ... | 17:41 |
| *** jamesmcarthur_ has quit IRC | 17:42 | |
| *** priteau has quit IRC | 17:42 | |
| mordred | clarkb: in the infra-package-needs element we enable acpid under openrc but not under systemd (we enable ntpd under systemd) | 17:42 |
| mordred | this seems weird to me | 17:42 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Build ubuntu bionic minimal images for control plane https://review.opendev.org/640044 | 17:43 |
| *** rkukura has joined #openstack-infra | 17:44 | |
| mordred | clarkb: how does that look? ^^ | 17:44 |
| clarkb | ya I think we want acpid across the board (it makes things like nova server restarts work) | 17:44 |
| clarkb | mordred: I'm guessing acpid is enabled by default when installed on other platforms? | 17:45 |
| *** igordc has quit IRC | 17:46 | |
| clarkb | mordred: that looks great | 17:46 |
| mordred | clarkb: awesome | 17:47 |
| clarkb | mordred: oh except one thing, I don't think we should use our mirrors to build them | 17:47 |
| mordred | clarkb: yeah - also - I don't think we currently have any intention to use this for anything other than ubuntu or anything <bionic | 17:47 |
| mordred | clarkb: totally. fix coming | 17:48 |
| clarkb | mordred: left a comment to make sure we capture that too, but other tahn that I think this is about ready | 17:48 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Build ubuntu bionic minimal images for control plane https://review.opendev.org/640044 | 17:49 |
| *** smarcet has quit IRC | 17:49 | |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add control plane mirror-only cloud regions https://review.opendev.org/660388 | 17:49 |
| clarkb | +2'd | 17:50 |
| mordred | \o/ | 17:50 |
| clarkb | one thing I'll have to remind myself is those control plane images should be quite small | 17:53 |
| clarkb | probably only a couple gigabytes | 17:53 |
| clarkb | so their cost is low compared to the test node images | 17:53 |
| mordred | yeah | 17:53 |
| mordred | clarkb: now - the _fun_ part is going to be using them from launch-node | 17:53 |
| mordred | since the image name in glance will change daily | 17:53 |
| mordred | clarkb: we'll probably want to make launch-node smarter | 17:54 |
| clarkb | ya though I personally set image name every time I use laucnh node anyways | 17:54 |
| clarkb | too many times I've been bitten by the default not being what I want | 17:54 |
| clarkb | that it was easier to always set it and know I got the right thing | 17:54 |
| mordred | oh - totally. | 17:54 |
| mordred | I was more thinking we coudl set IMAGE='ubuntu-bionic-minimal' and have launch-node do the list and find the newer of the two images prefixed with that | 17:55 |
| mordred | mostly because it'll allow us to interact with launch-node as humans - and launch-node can find the correct image pretty easily I'd imagine | 17:55 |
| mordred | (although, that said - if we add that, we could set the default to "ubuntu-bionic-minimal" - and it would be correct on all of our clouds!) | 17:56 |
| clarkb | mordred: fungi http://logs.openstack.org/22/655522/2/check/system-config-build-image-gitea/677bc01/job-output.txt.gz#_2019-05-21_17_31_19_348141 is that possibly related to updating skopeo? | 17:57 |
| mordred | clarkb: uh | 17:57 |
| *** jamesmcarthur has joined #openstack-infra | 17:58 | |
| fungi | i haven't upgraded it yet, so hopefully not? | 17:59 |
| clarkb | ah ok | 17:59 |
| fungi | was eating a lateish lunch first | 18:00 |
| *** rossella_s has joined #openstack-infra | 18:00 | |
| mordred | oh yeah. lunch. I'mma gonna do that | 18:01 |
| clarkb | oh right /var/registry/data is empty because we use swift now | 18:01 |
| *** jamesmcarthur has quit IRC | 18:02 | |
| clarkb | what is odd there is prior to that we push the image itself which seemed to work | 18:03 |
| clarkb | I'm going to try a recheck to see if this is one of our many network related issues and inconsistent or if it is something else | 18:03 |
| fungi | infra-root: heads up, the culprit i blocked from wiki.o.o earlier also hit www.openstack.org hard in the last few minutes... keep an eye out for other servers misbehaving i guess | 18:04 |
| clarkb | whois says it is a chinese clothing company. I'm guessing they have been compromised | 18:05 |
| clarkb | they have a phone number listed if someone wants to warn them | 18:06 |
| openstackgerrit | Ildiko Vancsa proposed opendev/system-config master: [WIP] Adding new ML for the NBMP group https://review.opendev.org/660485 | 18:08 |
| *** jamesmcarthur has joined #openstack-infra | 18:12 | |
| *** panda has quit IRC | 18:14 | |
| *** panda has joined #openstack-infra | 18:15 | |
| *** igordc has joined #openstack-infra | 18:18 | |
| *** eharney has joined #openstack-infra | 18:20 | |
| *** rosmaita has joined #openstack-infra | 18:21 | |
| *** xek_ has joined #openstack-infra | 18:21 | |
| *** jamesmcarthur_ has joined #openstack-infra | 18:22 | |
| clarkb | that intermediate registry push for gitea images seem to happen consistently on limestone at least | 18:24 |
| *** xek has quit IRC | 18:24 | |
| clarkb | odd that it affects the tag push and not the image push | 18:24 |
| *** jamesmcarthur has quit IRC | 18:24 | |
| *** mattw4 has quit IRC | 18:24 | |
| clarkb | makes me wonder if there is something wrong with the intermediate registry | 18:24 |
| fungi | perhaps in ways which only occur when being accessed over ipv6? what could that be though? | 18:25 |
| clarkb | could imply our socat tunnel is broken | 18:26 |
| clarkb | I've asked docker for the registry logs and they have been scrolling for quite some time | 18:26 |
| fungi | but it uses that for the image push too, right? | 18:26 |
| *** hwoarang has quit IRC | 18:27 | |
| fungi | is it a persistent socket connection or started on-demand and terminated again? | 18:27 |
| *** hwoarang has joined #openstack-infra | 18:28 | |
| *** mattw4 has joined #openstack-infra | 18:28 | |
| clarkb | correct should be used for the image push too. Should be persistent forking socat | 18:28 |
| clarkb | There are no logs for gitea accesses when the two job runs failed | 18:29 |
| clarkb | time="2019-05-21T14:50:01.506433721Z" is most recently logged gitea access by the intermediate registry | 18:29 |
| clarkb | so maybe it is never getting to the intermedaite registry and some problem between limestone and there exists | 18:30 |
| *** xek__ has joined #openstack-infra | 18:30 | |
| fungi | the intermediate registry is our persistent server, right? | 18:31 |
| fungi | what's the hostname? | 18:31 |
| clarkb | insecure-ci-registry.opendev.org | 18:31 |
| clarkb | ok now I notice that the image build itself failed | 18:31 |
| clarkb | for the incorrect username or password issue | 18:32 |
| fungi | ahh, so it's not just affecting tag pushes then | 18:32 |
| clarkb | possible that the image push task (which is no_log=true) isn't actually pushing the image as a result but then does try to tag it and fails because it wasn't pushed? | 18:32 |
| clarkb | I can look into that more after our meeting as it may be that the error we get is just confusing us due to procedural mistakes | 18:32 |
| *** xek_ has quit IRC | 18:33 | |
| fungi | yeah, that's an interesting possibility | 18:34 |
| clarkb | I'm beginning to suspect something like that is happening | 18:35 |
| logan- | i've been watching the conntrack table size and it looks like it hovers between 8k-12k active connections, which doesn't seem like enough to trigger snat port contention. nothing in dmesg indicating full conntrack tables or that sort of thing | 18:35 |
| clarkb | logan-: another possibility I hadn't considered yet/before was docker registry proxy could be trying to talk via ipv6 because it doesn't like the 10/8 address or something | 18:36 |
| clarkb | logan-: and just failing outright | 18:36 |
| clarkb | I need to pop out for a bit before the infra meeting but can help look into this more after the meeting | 18:37 |
| fungi | at this point i'll save the skopeo upgrades for immediately following the meeting so we don't have to deal with it breaking something during | 18:38 |
| *** bhavikdbavishi has joined #openstack-infra | 18:42 | |
| *** xek has joined #openstack-infra | 18:49 | |
| *** xek__ has quit IRC | 18:50 | |
| clarkb | so I don't forget the other item to consider is firewalls | 18:56 |
| *** sshnaidm is now known as sshnaidm|afk | 18:56 | |
| clarkb | I thought we had punched all the holes necessary but we may be missing some still | 18:57 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects https://review.opendev.org/660461 | 18:59 |
| fungi | clarkb: ooh, yeah maybe we forgot v6 equivalent of some v4 rule(s)? | 18:59 |
| fungi | that could explain why we see it where we do | 19:00 |
| *** gyee has quit IRC | 19:07 | |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Retire ansible-role-k8s-* repos https://review.opendev.org/660446 | 19:07 |
| fungi | #thanks cmurphy for driving our massive puppet upgrade | 19:12 |
| openstackstatus | fungi: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 19:12 |
| *** gyee has joined #openstack-infra | 19:19 | |
| *** bhavikdbavishi has quit IRC | 19:27 | |
| *** whoami-rajat has quit IRC | 19:29 | |
| *** pkopec has quit IRC | 19:34 | |
| openstackgerrit | Merged opendev/system-config master: Update ask.openstack.org to puppet 4 https://review.opendev.org/647877 | 19:35 |
| *** ykarel|away has quit IRC | 19:36 | |
| AJaeger | config-core, please review https://review.opendev.org/#/c/659977 to update a few docs jobs | 19:36 |
| mordred | fungi: re: mediawiki + sqlite - a lot of the projects that make Dockefiles and images seem focused on enabling simple use, rather than production use | 19:42 |
| mordred | fungi: I thnik that's one of the times when we might look at what they did, go - "thanks for the starting point" - and then make our own | 19:42 |
| fungi | yeah, it's... very short (which is a great demonstration i guess): https://github.com/wikimedia/mediawiki-docker/blob/master/1.32/Dockerfile | 19:44 |
| *** jamesmcarthur_ has quit IRC | 19:44 | |
| mordred | fungi: oh - there's another thing - the dockerfiles we've been writing make use of a "modern" technique - which is multi-stage dockerfiles | 19:45 |
| mordred | fungi: this allows you to have one "stage" do build-type things, and then copy artifacts from that stage into the final image - so that you don't need to install -dev packages and whatnot in your actual application image | 19:46 |
| fungi | ahh, so simplifies cleanup | 19:47 |
| mordred | yeah | 19:47 |
| clarkb | right part of the issue is docker images would balloon and get huge due to accumulated intermediate artifacts | 19:48 |
| *** Goneri has quit IRC | 19:48 | |
| *** logan- has quit IRC | 19:55 | |
| openstackgerrit | Merged openstack/project-config master: Drop jobs for devstack-plugins-sheepdog https://review.opendev.org/660452 | 19:55 |
| Shrews | mordred: question for you on https://review.opendev.org/640027 | 20:00 |
| fungi | continuing the ask.o.o discussion, i guess a major point in the services we run as alternatives to other popular services is that they're built on free software... is the source code for stackexchange open? if so then i see zero benefit in competing with it | 20:01 |
| mordred | clarkb, fungi, ianw: I think I was trying to get at 2 different things - one is that various of our opendev tenants might, from time to time, want to run a specific service that is not generally opendev - and we've got several openstack examples that exist via history - but we should figure out what, if any, part in running such services in the future we want to have as opendev | 20:01 |
| mordred | and the second is that, since ask is not a general-pupose opendev tool, I thnik it's reasonable to ask that folks who care about it existing show up with resources to make that happen | 20:02 |
| *** Goneri has joined #openstack-infra | 20:02 | |
| mordred | Shrews: good catch | 20:02 |
| *** jamesmcarthur has joined #openstack-infra | 20:02 | |
| clarkb | fungi: I don't think it is. askbot was the answer to that problem | 20:02 |
| ianw | fungi: the source isn't, but all comments are creative common licensed and i think you can bulk download them | 20:03 |
| fungi | yeah, in that case there could be a reason to host an open source alternative to stackexchange, unless there's someone out there who already is doing that we can recommend instead | 20:03 |
| fungi | okay, preparing to upgrade skopeo | 20:05 |
| clarkb | fungi: https://askbot.com/ | 20:05 |
| clarkb | btw I think I know what the docker image problem is | 20:05 |
| clarkb | will have patch as soon as I figure out syntax | 20:05 |
| fungi | clarkb: doesn't look like askbot.com is providing free hosting of a general ask site | 20:06 |
| *** rlandy is now known as rlandy|biab | 20:06 | |
| fungi | instead it looks like you can do a 30-day free trial of a paid ask site specific to your project | 20:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml https://review.opendev.org/640027 | 20:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused clouds from clouds.yaml files https://review.opendev.org/660374 | 20:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add the mirror-only cloud regions to nodepool builder https://review.opendev.org/660384 | 20:06 |
| mordred | clarkb: ^^ respun due to review from Shrews | 20:06 |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Open port 5001 for the registry proxy https://review.opendev.org/660504 | 20:08 |
| fungi | okay, preparing to upgrade skopeo | 20:08 |
| mordred | fungi: yah - I agree, I could see a reason to run one on that reason - but if we did, I'd think we'd want to either make a general purpose one (ask.opendev.org) and/or be willing to generally provide white-labeled ones for more than just openstack | 20:08 |
| clarkb | fungi: mordred logan- Shrews ^ I think 660504 does the firewall stuff we need | 20:08 |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Remove legacy sheepdog tests https://review.opendev.org/660505 | 20:08 |
| mordred | clarkb: FOR THE LOVE OF | 20:08 |
| clarkb | fungi: ya its paid | 20:09 |
| fungi | mordred: i would probably not even want to consider white-labeled ones, just a generic opendev one where people can ask questions about anything related to projects hosted in opendev | 20:09 |
| clarkb | fungi: but it is oepn source | 20:09 |
| fungi | sure | 20:09 |
| ianw | fungi: seems similar to the translate discussion. it might be a better use of resources to pay a company providing hosting & support of an opensource product for services, rather than allocate developer time to self-hosting | 20:09 |
| ianw | if it's a situation where paying for that hosting is supporting development of the project, it seems win-win | 20:10 |
| fungi | ianw: yep, in that case it's a matter of getting osf to work out contract details and long-term financial commitments for those i guess | 20:10 |
| clarkb | fwiw we had a similar setup with them in the past | 20:10 |
| clarkb | I don't know what happened to move away from that | 20:10 |
| clarkb | reed probably knows | 20:11 |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Fix typo in tempest-dsvm-full-hdfs https://review.opendev.org/660507 | 20:11 |
| clarkb | thank you #openstack-infra for being my rubber ducky while I talked myself through this docker thing | 20:12 |
| openstackgerrit | Merged openstack/project-config master: Switch some repos to PTI docs jobs https://review.opendev.org/659977 | 20:12 |
| clarkb | ianw: you mentioend wanted to check growroot with https://review.opendev.org/#/c/659215/ any reason to not approve that now? | 20:13 |
| mordred | clarkb: well - I think our original deal with the ask folks was that they'd run it but it would run on our infrastructur | 20:13 |
| ianw | oh i was just wondering what happens with growing the disk and journals | 20:13 |
| clarkb | fungi: fwiw I see our mailing list(s) as the better venue for that type of discussion. Ask suffers from lack of moderation in addition to service side problems | 20:13 |
| openstackgerrit | Merged openstack/project-config master: Remove broken docs jobs from monasca-agent https://review.opendev.org/659978 | 20:14 |
| fungi | clarkb: yep, i agree. maybe mailman 3 is the answer | 20:14 |
| clarkb | ianw: wondering if it is a noop because growroot will also grow the journal? | 20:14 |
| mordred | while I think with the translate suggetsion it's just "let someone else completely run a hosted service for us" | 20:14 |
| mordred | fungi: mailman3 is always the answer | 20:14 |
| fungi | #status upgraded skopeo on all zuul executors to 0.1.37-1~dev~ubuntu16.04.2~ppa2 | 20:14 |
| openstackstatus | fungi: unknown command | 20:14 |
| clarkb | mordred: yup they did that for a bit then at some point we ended up hosting it afterall | 20:14 |
| fungi | #status log upgraded skopeo on all zuul executors to 0.1.37-1~dev~ubuntu16.04.2~ppa2 | 20:14 |
| openstackstatus | fungi: finished logging | 20:14 |
| mordred | clarkb: yah. | 20:14 |
| fungi | clarkb: so... 660504 firewall rules ftw? | 20:15 |
| ianw | clarkb: yeah, i mean that's the way it works now right, grows up to the 80gb of the vm at boot time. does the journal grow with that too? | 20:16 |
| clarkb | ianw: May 21 20:04:09 ask01 ansible-apt: Invoked with dpkg_options=force-confdef,force-confold autoremove=False force=False force_apt_get=False install_recommends=None package=None autoclean=False purge=False allow_unauthenticated=False state=present upgrade=None update_cache=None default_release=None only_upgrade=False | 20:16 |
| clarkb | deb=https://release-archives.puppet.com/apt/pool/xenial/PC1/p/puppet-agent/puppet-agent_1.10.14-1xenial_amd64.deb cache_valid_time=0 | 20:16 |
| clarkb | ianw: I think that implies that we didn't actually upgrade puppet | 20:16 |
| clarkb | (this may be why the upgrade process was relying on install.sh | 20:17 |
| openstackgerrit | Merged opendev/puppet-mediawiki master: Replace transitional package names for Xenial https://review.opendev.org/651352 | 20:17 |
| ianw | i did a bit of research yesterday, but i couldn't find anything on what the best practice is for growing disks and journals | 20:17 |
| ianw | clarkb: hrm ... did they end up re-creating a repo? | 20:17 |
| mordred | fwiw - I pushed up https://github.com/go-gitea/gitea/pull/7011 earlier to correct a piece of documentation - just in case someone wants to check the docs when looking at the gitea 1.8.0 patch | 20:17 |
| clarkb | ianw: re growroot we should be able to interrogate a test node for its journal size (I imagine) | 20:18 |
| clarkb | ianw: re ask puppet 4 install_puppet.sh was a bit more sophisticated than just install this deb | 20:18 |
| clarkb | ianw: its possible that puppet doesn't want to update because of some missing option | 20:18 |
| ianw | ok, to my repo question : https://tickets.puppetlabs.com/browse/CPR-685 is the answer an it's "in progress" | 20:19 |
| clarkb | I would've expected a dpkg -i equivalent to result in just install this package please | 20:20 |
| clarkb | but maybe it won't because puppet 3 is a conflict or something? | 20:20 |
| fungi | if there are unmet dependencies it won't | 20:21 |
| clarkb | fungi: there aren't any that I know of and this was tested by our test jobs | 20:21 |
| clarkb | (we install puppet 4 via the same manner on test jobs where there is no preinstalled puppet) | 20:22 |
| clarkb | so maybe one thing we can try is uninstall puppet-3 by hand and have ansible install puppet 4 | 20:22 |
| fungi | you can apt install a local file path to the downloaded package if you want to make sure that declared dependencies get fetched and installed | 20:22 |
| fungi | oh, also if the package name is different, they could conflict | 20:22 |
| *** diablo_rojo_phon has joined #openstack-infra | 20:22 | |
| clarkb | I think it is puppet from distro for puppet-3 and puppet-agent for puppet-4 deb | 20:23 |
| clarkb | so that may be it | 20:23 |
| fungi | btw, for anybody who hasn't yet seen a demo of the checks plugin for gerrit, go to https://gerrit-review.googlesource.com/c/gerrit/+/224831 and then select the "checks" tab where the list of changed files is displayed | 20:23 |
| diablo_rojo_phon | clarkb: fungi sorry am at dinner and missed the meeting, yes we do want to start doing releases. We talked about it at the PTG. | 20:23 |
| fungi | diablo_rojo_phon: cool! thanks for confirming | 20:24 |
| clarkb | fungi: ianw so what do we think about uninstalling the puppet package supplied by the distro? | 20:24 |
| mtreinish | fungi: is that the new gerrit interface we have to look forward to at some point in the future? | 20:25 |
| clarkb | May 21 19:37:15 ask01 puppet-user[12470]: Invalid parameter state on Package[jetty] at /opt/system-config/production/modules/openstack_project/manifests/ask.pp:34 on node ask01.openstack.org | 20:25 |
| fungi | i think we probably have to before we can install the newer differently-named one. unless it provides the appropriate metadata to indicate it replaces the distro package, in which case we still need something smarter than dpkg -i (like apt install of the local file path maybe) | 20:25 |
| clarkb | puppet is unhappy either way so probably not urgent (though getting this done would be nice) | 20:25 |
| ianw | clarkb: that seems reasonable ... | 20:25 |
| clarkb | ok I'll start with that uninstal | 20:26 |
| * mtreinish doesn't like it (but that's not really a surprise, still miss the old view in gerrit) | 20:26 | |
| openstackgerrit | Merged opendev/base-jobs master: Open port 5001 for the registry proxy https://review.opendev.org/660504 | 20:26 |
| fungi | mtreinish: yes, that's their new polyglot-based "polygerrit" ui | 20:26 |
| mtreinish | the checks tab is cool though | 20:26 |
| clarkb | I don't like that the checks tab is a tab you have to click on | 20:26 |
| clarkb | people will wonder why can't my change merge | 20:27 |
| clarkb | I ran `apt-get purge puppet` to remove puppet then `apt-get autoremove` to remove augeas-lenses facter hiera libaugeas0 puppet-common ruby-augeas ruby-deep-merge ruby-json ruby-nokogiri ruby-rgen ruby-safe-yaml ruby-selinux ruby-shadow virt-what | 20:28 |
| fungi | clarkb: also there's no grouping mechanism, so we'd need to prepend all the job names for every different ci system, and i don't think there's currently a way to prevent one ci system from replacing the results of another because they don't consider the possibility you don't trust the ci systems reporting on your change or that there would be more than one | 20:28 |
| ianw | clarkb: hrm ... "root@ask01:/tmp# dpkg -i puppet-agent_1.10.14-1xenial_amd64.deb" seemed like it wanted to work ... | 20:28 |
| clarkb | ianw: was this after I removed things? | 20:28 |
| ianw | ohhh, right | 20:29 |
| clarkb | fungi: I think the spec for it does allow for different systems at least | 20:29 |
| clarkb | fungi: it has been a while since I read the spec though | 20:29 |
| ianw | clarkb: should i run a kick.sh? | 20:29 |
| clarkb | maybe, I think kick.sh may not do what we want anymore iwth the base.yaml split? | 20:30 |
| ianw | it runs remote_puppet_adhoc, which should do the puppeting? | 20:31 |
| clarkb | ianw: but that doesn't do the install | 20:31 |
| clarkb | oh it does | 20:31 |
| clarkb | I take that back | 20:31 |
| clarkb | so ya I think a kick.sh should work | 20:31 |
| ianw | yeah, i think it installs and copies hiera and all that | 20:31 |
| ianw | running now | 20:32 |
| ianw | TASK [puppet : Sanity check puppet version] ********************************************************************************************************************************** | 20:33 |
| ianw | fatal: [ask01.openstack.org]: FAILED! => {"changed": false, "msg": "Failed as requested from task"} | 20:33 |
| clarkb | ianw: /opt/puppetlabs exists now | 20:33 |
| fungi | so i guess it was the package name difference | 20:34 |
| fungi | most likely they tried to both provide at least one file at an identical path and dpkg refused to install the new one because it would overwrite a file it knew belonged to the existing one | 20:35 |
| clarkb | /opt/puppetlabs/bin is empty though | 20:36 |
| clarkb | /opt/puppetlabs/puppet/bin/puppet.dpkg-new exists though | 20:36 |
| ianw | yeah, it failed in the "Sanity check puppet version" check per above ... but didn't really say why | 20:36 |
| clarkb | did the halfway dpkg -i get us into a halfway state? | 20:36 |
| clarkb | ianw: one sec I'll link you the code | 20:36 |
| ianw | hrm, yeah just ran a dpkg --configure -a , maybe i messed it up. | 20:37 |
| clarkb | ianw: https://opendev.org/opendev/ansible-role-puppet/src/branch/master/tasks/main.yaml#L37-L39 is the check and https://opendev.org/opendev/ansible-role-puppet/src/branch/master/tasks/main.yaml#L25-L29 is what seems to fail | 20:37 |
| ianw | let's try again | 20:37 |
| clarkb | if you run that puppet version command it fails because the code isn't there and in the other bin dir is all the dpkg.new files | 20:38 |
| ianw | fatal: [ask01.openstack.org]: FAILED! => {"changed": false, "msg": "E:The package puppet-agent needs to be reinstalled, but I can't find an archive for it." | 20:38 |
| ianw | base-server : Install base packages | 20:38 |
| fungi | puppet-agent package is in iHR state | 20:39 |
| clarkb | I need to eat lunch now but suspect the issue is the package isn't fully installed and once it is it'll be happy | 20:39 |
| *** e0ne has joined #openstack-infra | 20:39 | |
| clarkb | I also rechecked the gitea change after the iptables port opening merged | 20:39 |
| *** rlandy|biab is now known as rlandy | 20:39 | |
| ianw | ok, i'm just dpkg -i ing it manually | 20:40 |
| ianw | ok, puppet failed, but it ran | 20:41 |
| fungi | seems it's in ii state now, which is what we want | 20:42 |
| ianw | no parameter named 'state' at /opt/system-config/production/modules/openstack_project/manifests/ask.pp:34 on Package[jetty] | 20:42 |
| fungi | the previous iHR state expands to Desired=Install,Status=Half-inst,Err?=Reinst-required | 20:44 |
| ianw | i guess that should be "ensure" | 20:45 |
| *** hamzy has quit IRC | 20:45 | |
| fungi | yeah, ensure=>present is probably what we want there? | 20:46 |
| fungi | i'm unfamiliar with puppet's dpkg package provider though | 20:46 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ask.o.o : fix jetty package install https://review.opendev.org/660517 | 20:46 |
| ianw | fungi: ^ :) | 20:46 |
| clarkb | fungi: this is ansibles dpkg via apt module | 20:47 |
| fungi | really? | 20:47 |
| fungi | it looks like a puppet resource to me | 20:47 |
| clarkb | oh for jetty | 20:49 |
| *** xek has quit IRC | 20:49 | |
| clarkb | sorry thought you were refferring to the puppet dpkg install | 20:49 |
| *** dave-mccowan has quit IRC | 20:50 | |
| fungi | nah, we moved on whilst you sammiched | 20:50 |
| fungi | (or whatever lunch was) | 20:50 |
| clarkb | still sammiching :) | 20:50 |
| fungi | s/was/is/ then ;) | 20:50 |
| ianw | clarkb / mordred: so if we can get https://review.opendev.org/#/q/status:open+topic:letsencrypt in (clarkb just one syntax fix since your last +2 on the base change) we can have mirror.dfw.rax.opendev.org up pretty much now (modulo fixing unforseen issues) ... then i'm happy to move forward based on the nodepool built images as that comes online | 20:52 |
| ianw | speaking of food ... breakfast ... bib :) | 20:53 |
| *** rfarr_ has joined #openstack-infra | 20:54 | |
| *** e0ne has quit IRC | 20:56 | |
| *** rfarr has quit IRC | 20:56 | |
| *** gfidente has quit IRC | 21:02 | |
| *** gfidente has joined #openstack-infra | 21:03 | |
| fungi | ianw: i like the macro approach in 658281, i'd never noticed that feature of apache before | 21:07 |
| clarkb | the gitea image build job succeeded but the jobs ran on ovh not limestone so not confirmation the port issue was the thing | 21:08 |
| clarkb | but is confirmation that mordred fixed the gitea problem | 21:08 |
| clarkb | er no need the run-gitea job to run I guess | 21:09 |
| *** jamesmcarthur has quit IRC | 21:11 | |
| mordred | fungi, ianw: WOAH | 21:12 |
| mordred | clarkb: yay fixing gitea problem! | 21:13 |
| mordred | clarkb: we can finally land the "make the gitea front page text in the present tense" patch :) | 21:13 |
| fungi | yay for being able to land patches again | 21:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update opendev website to be more present tense https://review.opendev.org/653946 | 21:13 |
| clarkb | mordred: yup | 21:13 |
| mordred | rebased | 21:13 |
| *** logan- has joined #openstack-infra | 21:13 | |
| clarkb | mordred: also I think sparkycollier has some edits he wants to propose (I was assisting with git set up yesterday) | 21:14 |
| mordred | clarkb: cool! I'm excited about getting edits from sparkycollier | 21:14 |
| cmurphy | ask.o.o puppet is unhappy? | 21:14 |
| clarkb | cmurphy: yes but we think we've got it under control https://review.opendev.org/#/c/660517/1 | 21:14 |
| clarkb | that was a preexisting issue not puppet-4 related | 21:14 |
| cmurphy | lol | 21:14 |
| mordred | clarkb: like - he's going to submit a new patch from scratch? so we don't have to wait on this one, yeah? | 21:15 |
| fungi | more of a nobody-looks-at-this-thing-like-ever issue | 21:15 |
| *** kjackal has joined #openstack-infra | 21:15 | |
| fungi | the ask.o.o puppet issue i mean | 21:15 |
| clarkb | mordred: correct | 21:15 |
| cmurphy | i'm about to be driving for a while so hopefully nothing puppetty explodes too badly | 21:16 |
| *** diablo_rojo has joined #openstack-infra | 21:19 | |
| clarkb | I've rechecked a few zuul changes that seemed to have hit the registry proxy issue | 21:20 |
| fungi | looks like we're 2x+2 on the letsencrypt changes, and the earliest in the sequence is in the gate now | 21:25 |
| fungi | holding off on approving the second since i don't think it shares a change queue with the others | 21:25 |
| *** rfarr__ has joined #openstack-infra | 21:26 | |
| openstackgerrit | Merged openstack/project-config master: Finish moving python-jenkins jobs in-tree https://review.opendev.org/660022 | 21:27 |
| *** slaweq has quit IRC | 21:27 | |
| *** Lucas_Gray has joined #openstack-infra | 21:28 | |
| *** rfarr_ has quit IRC | 21:28 | |
| fungi | i'm going to try and knock out some yardwork but will be in and out | 21:30 |
| zigo | fungi: Hi there! Any chance you have time to review adding a Buster image? https://review.opendev.org/#/c/645574/ | 21:35 |
| zigo | :) | 21:35 |
| openstackgerrit | Merged opendev/system-config master: ask.o.o : fix jetty package install https://review.opendev.org/660517 | 21:43 |
| *** Lucas_Gray has quit IRC | 21:47 | |
| clarkb | ^ should apply in about 30 minutes | 21:49 |
| clarkb | not seeing any buildset registry proxies on limestone since the fix went in. At least it doesn't seem to break anything for the other regions | 21:49 |
| logan- | my bnc dropped for a bit. what was the fix? | 21:50 |
| clarkb | logan-: https://review.opendev.org/#/c/660504/ is the current suspect | 21:51 |
| logan- | ah | 21:51 |
| *** mriedem has quit IRC | 21:51 | |
| logan- | docker really does hate ipv6 heh | 21:52 |
| *** Lucas_Gray has joined #openstack-infra | 21:52 | |
| weshay | https://review.opendev.org/660529 warlock again | 21:57 |
| *** sreejithp__ has quit IRC | 21:59 | |
| *** sreejithp has joined #openstack-infra | 21:59 | |
| clarkb | ianw: May 21 22:00:21 ask01 puppet-user[9397]: (/Stage[main]/Openstack_project::Ask/File[/root/jetty_1.0_all.deb]) Could not evaluate: Could not retrieve file metadata for puppet://modules/openstack_project/ask/jetty_1.0_all.deb: getaddrinfo: Name or service not known | 22:01 |
| clarkb | I think you need three slashes on the puppet:/// | 22:01 |
| openstackgerrit | Kendall Nelson proposed opendev/system-config master: Adds new key for diablo_rojo https://review.opendev.org/660530 | 22:01 |
| diablo_rojo | clarkb, fungi, corvus whoever is around, if its not too much trouble to give that a look ^^ I wanted to try to get the swift test migration done before their meeting tomorrow (I think its tomorrow lol) | 22:02 |
| ianw | clarkb: ok ... i'm not even going to think about why that used to work but doesn't now | 22:02 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add more /s to puppet:/// url for jetty shim deb https://review.opendev.org/660531 | 22:03 |
| clarkb | ianw: ^ like that | 22:03 |
| notmyname | diablo_rojo: yes. tomorrow at 2100utc (2pm pacific) | 22:03 |
| ianw | thanks :) | 22:04 |
| clarkb | logan-: fungi mordred ok we caught one on limestone | 22:04 |
| *** sreejithp has quit IRC | 22:04 | |
| clarkb | the tests for https://review.opendev.org/636197 | 22:04 |
| diablo_rojo | notmyname, ....I'll figure out what time it is where I am lol | 22:05 |
| diablo_rojo | clarkb, thanks! | 22:05 |
| *** rcernin has joined #openstack-infra | 22:05 | |
| *** gfidente is now known as gfidente|afk | 22:05 | |
| clarkb | diablo_rojo: 11pm for you I think | 22:05 |
| clarkb | diablo_rojo: an hour ago | 22:05 |
| clarkb | but tomorrow | 22:05 |
| *** tesseract has quit IRC | 22:07 | |
| mordred | clarkb: woot! | 22:08 |
| clarkb | ouch linux 5.1 fstrim reportedly discarding the wrong blocks | 22:09 |
| *** dciabrin_ has quit IRC | 22:10 | |
| *** rfarr__ has quit IRC | 22:11 | |
| *** dciabrin_ has joined #openstack-infra | 22:11 | |
| diablo_rojo | Oh wow okay got it. | 22:13 |
| diablo_rojo | thanks for the conversion clarkb :) | 22:13 |
| clarkb | diablo_rojo: well it was 2200UTC | 22:23 |
| clarkb | and cest is +2 so was relatively easy for me to do the maths after looking at my irc buffer | 22:23 |
| clarkb | I'm cloning the linux kernel now to figure out if the current kernel I am running is going to destroy my disk | 22:23 |
| *** kjackal has quit IRC | 22:23 | |
| clarkb | maybe I should just turn it off and go fishing and hope things are better when I return on thursday | 22:24 |
| mordred | clarkb: I think that's the best plan | 22:26 |
| openstackgerrit | Merged openstack/project-config master: Add a Debian Buster image. https://review.opendev.org/645574 | 22:30 |
| clarkb | I think it trimmed yesterday due to fstrim.timer | 22:32 |
| clarkb | and that was after I rebooted into the new kernel | 22:32 |
| clarkb | so maybe I'm already broken and I just don't know it yet | 22:32 |
| *** sarob has joined #openstack-infra | 22:32 | |
| clarkb | or it isn't a universal error with btrfs + dm + lvm + linux 5.1 | 22:33 |
| fungi | debian/sid being toward the end of a lengthy release freeze for buster has luckily kept me on 4.19 for a bit | 22:34 |
| *** sarob has quit IRC | 22:35 | |
| clarkb | http://logs.openstack.org/97/636197/20/check/zuul-build-image/e75436b/ara-report/result/0d0ebdbc-93b6-4d06-9e2d-30724fbfca1a/ is the iptables hole punch on limestone and that job as well as the consuming job succeeded | 22:36 |
| clarkb | hopefully that deals with a major issue in the system | 22:36 |
| * fungi cheers | 22:37 | |
| * mordred hands everyone baby hippos | 22:37 | |
| clarkb | woo ya the bisected change culprit was in 5.0-rc6 | 22:39 |
| clarkb | so my 5.1.2 almost certainly is running with it | 22:39 |
| clarkb | exciting! | 22:39 |
| clarkb | it should be known that my fileserver runs a stable ubuntu release | 22:40 |
| *** rcernin has quit IRC | 22:40 | |
| clarkb | my desktop and laptop are more like thinclients that I won't cry too much over if fstrim eats the disk | 22:40 |
| clarkb | I'll just have to reinstall | 22:40 |
| *** rcernin has joined #openstack-infra | 22:41 | |
| clarkb | https://www.redhat.com/archives/dm-devel/2019-May/msg00102.html is the fix | 22:41 |
| *** tosky has quit IRC | 22:42 | |
| fungi | i rely on backups | 22:48 |
| fungi | but yeah, not having to restore data from a backup is definitely preferable | 22:49 |
| clarkb | fungi: ya I have those too, now with offsites | 22:49 |
| ianw | clarkb: sorry, right, back to the ext4 thing ... it's unclear to me if resize2fs ultimately changes the journal size on growth | 22:49 |
| clarkb | ianw: can we inspect the fs on a running post growroot'd instance? | 22:49 |
| fungi | i basically have that insofar as i stick encrypted backups of my local systems on my remote cloud servers and vice cersa | 22:49 |
| fungi | versa | 22:49 |
| clarkb | http://blog.dailystuff.nl/2012/07/getting-ext34-journal-size/ oh wow you have to find the inode where the journal starts? | 22:50 |
| *** armax has quit IRC | 22:50 | |
| clarkb | dumpe2fs is easy mode apparently | 22:50 |
| ianw | User: 0 Group: 0 Size: 67108864 | 22:51 |
| *** tjgresha has joined #openstack-infra | 22:51 | |
| ianw | that's on one of our nodes /dev/xvda1 38G 5.3G 31G 15% / | 22:52 |
| clarkb | dumpe2fs on an ovh bhs1 node says 64M for journal after growing to 75G | 22:53 |
| clarkb | I think that implies we don't grow the journal when we grow the fs | 22:53 |
| clarkb | that blog post also says never change its size unless it is a clean unmounted fs | 22:54 |
| *** auristor has quit IRC | 22:55 | |
| clarkb | also 128MB seems to be a common journal size | 22:55 |
| clarkb | we might want to change the dib default to that? | 22:56 |
| clarkb | that does make me wonder if 512M is huge | 22:57 |
| *** auristor has joined #openstack-infra | 22:57 | |
| ianw | per https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n340 256 is probably as big as our fs gets | 22:59 |
| *** tkajinam has joined #openstack-infra | 22:59 | |
| clarkb | we are in between 64GB and 128GB | 23:00 |
| clarkb | so 512 is indeed the default | 23:01 |
| openstackgerrit | Merged opendev/system-config master: Create opendev mirrors https://review.opendev.org/658281 | 23:01 |
| openstackgerrit | Merged opendev/system-config master: Adds new key for diablo_rojo https://review.opendev.org/660530 | 23:01 |
| clarkb | I say we go for the change as is, it fits into the defaults and should make the journal on ext4 happier | 23:01 |
| clarkb | we can always tune it back down if the balloons the image size | 23:01 |
| clarkb | I'm guessing this is why dib makes it so small | 23:01 |
| fungi | i have to assume it compresses well when freshly initialized | 23:02 |
| diablo_rojo | thanks fungi and clarkb! | 23:02 |
| clarkb | fungi: ya but we can't compress or trim the vhd images | 23:03 |
| clarkb | fungi: we compress qcow2 so should be fine there and we trim raw images so also fine there | 23:03 |
| clarkb | ianw: if no objections feel free to approve the change (I would but I'm about to afk for a day so figure I shouldn't be responsible) | 23:04 |
| clarkb | https://review.opendev.org/#/c/659215/ is the change for anyone following along | 23:05 |
| ianw | ok, i approved it. there's some numbers out there on things like moving the journal to a faster ssd, etc, but not really much i can see on sizing | 23:06 |
| ianw | you'd think people dealing with cloud-init and growing root file systems would have discussed it before, but if they have, i can't find it | 23:07 |
| *** calbers has quit IRC | 23:07 | |
| clarkb | ianw: we are trailblazers! | 23:07 |
| clarkb | lets hope we don't get swept by the warriors | 23:07 |
| clarkb | :( | 23:07 |
| *** jklare has quit IRC | 23:09 | |
| *** sreejithp has joined #openstack-infra | 23:09 | |
| *** sreejithp_ has joined #openstack-infra | 23:11 | |
| *** mattw4 has quit IRC | 23:11 | |
| *** mattw4 has joined #openstack-infra | 23:12 | |
| *** sreejithp has quit IRC | 23:14 | |
| fungi | sports joke, i guess | 23:16 |
| clarkb | yes the san francisco warriors just destroyed the portland trailblazers | 23:16 |
| fungi | which of our clouds still rely on vhd images, btw... just rax? | 23:16 |
| clarkb | fungi: correct | 23:16 |
| *** _erlon_ has quit IRC | 23:17 | |
| openstackgerrit | Merged openstack/project-config master: Increase nodepool image journal size https://review.opendev.org/659215 | 23:17 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add mirror.dfw.rax.opendev.org https://review.opendev.org/660235 | 23:17 |
| *** Lucas_Gray has quit IRC | 23:20 | |
| fungi | okay, slightly too real... i ripped a variety of sound effects discs along with the rest of my cd collection when building my home jukebox/radio station... taking a break from pushing around the reel mower, catching up on e-mail and listening to said radio... what comes up in rotation? sound clip of a power lawn mower | 23:21 |
| *** jklare has joined #openstack-infra | 23:24 | |
| *** aaronsheffield has quit IRC | 23:25 | |
| *** calbers has joined #openstack-infra | 23:26 | |
| ianw | so you just have random sound effects in your radio shuffle? | 23:27 |
| ianw | i don't know if the humor goes outside an australian context but as a sound effects fan you might like sizzletown https://www.podbean.com/site/EpisodeDownload/PB8F29CFECIHY | 23:29 |
| fungi | heh, yes random clips of sound effects are salted throughout my random shuffle. it gets entertaining at times, as evidenced | 23:32 |
| openstackgerrit | Merged opendev/system-config master: Add more /s to puppet:/// url for jetty shim deb https://review.opendev.org/660531 | 23:32 |
| fungi | also occasionally artist interviews, sound clips from films (john carpenter seemed to do this on a number of the soundtracks he composed for his films) and short episodes from some radio serials | 23:33 |
| *** icarusfactor has joined #openstack-infra | 23:34 | |
| fungi | but the collection is approaching a solid 5 weeks of uninterrupted audio though, so this stuff accounts for less than 1% probably. it's *almost* entirely music (depending on how you define music at least) | 23:34 |
| fungi | i have no idea what tony martin's sizzletown is, but going to find out once i'm done with the yard ;) | 23:36 |
| *** factor has quit IRC | 23:36 | |
| *** icarusfactor is now known as factor | 23:37 | |
| * tonyb tried to work out why irssi needed to tell me y'all are talking sound effects .. but it did | 23:38 | |
| tonyb | Oh .... I see now nm | 23:39 |
| fungi | now i get it, "tony martin" is tonyb's stage name! | 23:42 |
| tonyb | fungi: for sure ;P | 23:42 |
| openstackgerrit | Merged opendev/system-config master: Bringup mirror01.dfw.rax.opendev.org https://review.opendev.org/660237 | 23:42 |
| *** lseki has quit IRC | 23:43 | |
| *** Goneri has quit IRC | 23:52 | |
| *** rlandy has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!