| *** xarses_ has quit IRC | 00:01 | |
| *** jamesmcarthur has joined #openstack-infra | 00:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Use the SQLAlchemy ORM https://review.openstack.org/620426 | 00:04 |
|---|---|---|
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add artifact table https://review.openstack.org/620427 | 00:04 |
| *** jaosorior has quit IRC | 00:05 | |
| *** jamesmcarthur has quit IRC | 00:07 | |
| *** diablo_rojo has joined #openstack-infra | 00:07 | |
| ianw | clarkb: heh, yeah the comments already say "when:" as that's how i'd started writing it. i changed when it seemed more logical with the env-var-not-defined case, but i agree now when: works better | 00:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add artifact table https://review.openstack.org/620427 | 00:11 |
| *** tosky has quit IRC | 00:13 | |
| *** armax has quit IRC | 00:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Capture ramdisk test run logs https://review.openstack.org/619400 | 00:25 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix unit tests for elements https://review.openstack.org/619387 | 00:25 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Revert "Make tripleo-buildimage-overcloud-full-centos-7 non-voting" https://review.openstack.org/620201 | 00:25 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var https://review.openstack.org/619119 | 00:25 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support https://review.openstack.org/619120 | 00:25 |
| *** yamamoto has quit IRC | 00:34 | |
| clarkb | ianw: on https://review.openstack.org/#/c/619119/9..10/diskimage_builder/elements/package-installs/bin/package-installs-squash the two return Falses should still be return True's right? | 00:36 |
| clarkb | ianw: docstring says return True if package should be installed and if var = val then we want to return True | 00:36 |
| clarkb | ianw: I left some comments on the change directly. I think I got that right | 00:39 |
| ianw | hrm | 00:44 |
| ianw | did i mess up the test case then? | 00:44 |
| clarkb | I think so | 00:44 |
| ianw | test-driven development only works when the test cases are right :) | 00:45 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var https://review.openstack.org/619119 | 00:49 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support https://review.openstack.org/619120 | 00:49 |
| ianw | i think that's right, thanks! | 00:49 |
| ianw | ok, the pypi volume is disappearing, slowly. i didn't really consider it would take about as long to remove as to create | 00:53 |
| *** sthussey has quit IRC | 00:55 | |
| ianw | ok, it's sped up considerably | 00:57 |
| *** wolverin_ has quit IRC | 00:59 | |
| *** wolverineav has joined #openstack-infra | 01:00 | |
| *** wolverineav has quit IRC | 01:02 | |
| *** wolverineav has joined #openstack-infra | 01:02 | |
| ianw | #status pypi volume removed from afs : afs01.dfw much happier -> /dev/mapper/main-vicepa 4.0T 2.1T 1.9T 53% /vicepa | 01:03 |
| openstackstatus | ianw: unknown command | 01:03 |
| ianw | #status log pypi volume removed from afs : afs01.dfw much happier -> /dev/mapper/main-vicepa 4.0T 2.1T 1.9T 53% /vicepa | 01:04 |
| openstackstatus | ianw: finished logging | 01:04 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove Fedora 27 mirroring https://review.openstack.org/618416 | 01:12 |
| *** ramishra has quit IRC | 01:13 | |
| *** annp has joined #openstack-infra | 01:14 | |
| *** bawz has quit IRC | 01:15 | |
| *** bawz has joined #openstack-infra | 01:17 | |
| *** diablo_rojo has quit IRC | 01:24 | |
| *** bawz has quit IRC | 01:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:34 | |
| *** jamesmcarthur has quit IRC | 01:54 | |
| *** diablo_rojo has joined #openstack-infra | 02:03 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Rework zuul nodepool stats reporting https://review.openstack.org/620285 | 02:23 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Add a statsd check for clashing keys https://review.openstack.org/620436 | 02:23 |
| *** dhineshb has quit IRC | 02:24 | |
| *** mrsoul has quit IRC | 02:25 | |
| *** xarses_ has joined #openstack-infra | 02:28 | |
| *** andreww has quit IRC | 02:31 | |
| *** ramishra has joined #openstack-infra | 02:35 | |
| *** hongbin has joined #openstack-infra | 02:36 | |
| *** bobh has joined #openstack-infra | 02:36 | |
| ianw | #status log removed f27 directories after https://review.openstack.org/618416 | 02:37 |
| openstackstatus | ianw: finished logging | 02:37 |
| *** psachin has joined #openstack-infra | 02:41 | |
| *** bobh has quit IRC | 02:41 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:51 | |
| gmann | clarkb: thanks, we are waiting on project side testing till Dec 10th. | 02:56 |
| gmann | frickler: i commented on your keystone patch, it needs to be Depends-On on tempest patch instead of devstack. currently jobs are running on xenial | 02:57 |
| *** rh-jelabarre has quit IRC | 03:03 | |
| *** bobh has joined #openstack-infra | 03:08 | |
| *** bobh has quit IRC | 03:12 | |
| *** wolverineav has quit IRC | 03:12 | |
| *** ramishra has quit IRC | 03:15 | |
| *** wolverineav has joined #openstack-infra | 03:17 | |
| *** jamesmcarthur has joined #openstack-infra | 03:17 | |
| *** wolverineav has quit IRC | 03:24 | |
| *** bhavikdbavishi has quit IRC | 03:30 | |
| *** ramishra has joined #openstack-infra | 03:35 | |
| *** jamesmcarthur has quit IRC | 03:36 | |
| *** diablo_rojo has quit IRC | 03:41 | |
| *** jamesmcarthur has joined #openstack-infra | 03:42 | |
| *** wolverineav has joined #openstack-infra | 03:53 | |
| *** ykarel|away has joined #openstack-infra | 03:55 | |
| *** jamesmcarthur has quit IRC | 03:55 | |
| *** wolverineav has quit IRC | 03:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Add a statsd check for clashing keys https://review.openstack.org/620436 | 04:08 |
| *** hongbin_ has joined #openstack-infra | 04:23 | |
| *** hongbin has quit IRC | 04:24 | |
| *** hongbin has joined #openstack-infra | 04:37 | |
| *** hongbin_ has quit IRC | 04:38 | |
| *** hongbin has quit IRC | 04:40 | |
| *** janki has joined #openstack-infra | 04:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Cleanup temporary directories https://review.openstack.org/592340 | 04:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Make indexer more generic https://review.openstack.org/592852 | 04:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Stub out dry run in the uploader https://review.openstack.org/592929 | 04:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Create a download script https://review.openstack.org/592341 | 04:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.openstack.org/592853 | 04:48 |
| *** lpetrut has joined #openstack-infra | 04:51 | |
| *** udesale has joined #openstack-infra | 04:54 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Capture ramdisk test run logs https://review.openstack.org/619400 | 04:57 |
| *** yamamoto has joined #openstack-infra | 04:58 | |
| *** xarses_ has quit IRC | 04:59 | |
| *** pcaruana has joined #openstack-infra | 05:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: Add NetworkManager distro plugin support https://review.openstack.org/618964 | 05:10 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: A systemd skip for Debuntu systems https://review.openstack.org/620420 | 05:10 |
| *** yboaron_ has joined #openstack-infra | 05:10 | |
| *** ykarel|away has quit IRC | 05:16 | |
| *** lpetrut has quit IRC | 05:28 | |
| *** ykarel|away has joined #openstack-infra | 05:31 | |
| *** ykarel|away is now known as ykarel | 05:31 | |
| *** e0ne has joined #openstack-infra | 05:37 | |
| ianw | rax dfw mirror seeing a ton of | 05:38 |
| ianw | https://review.openstack.org/#/q/topic:bulk-log-download+(status:open+OR+status:merged) | 05:38 |
| ianw | "(104)Connection reset by peer: [client 2001:4800:7818:104:be76:4eff:fe04:3d22:57888] AH01102: error reading status line from remote server pypi.org:443" | 05:38 |
| *** tdasilva has quit IRC | 05:39 | |
| *** e0ne has quit IRC | 05:42 | |
| *** e0ne has joined #openstack-infra | 05:44 | |
| *** e0ne has quit IRC | 05:45 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:54 | |
| *** jamesmcarthur has joined #openstack-infra | 05:56 | |
| *** jamesmcarthur has quit IRC | 06:01 | |
| *** diablo_rojo has joined #openstack-infra | 06:06 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix test race in test_hold_expiration_higher_than_default https://review.openstack.org/620222 | 06:20 |
| *** gyee has quit IRC | 06:20 | |
| *** ahosam has joined #openstack-infra | 06:29 | |
| *** jaosorior has joined #openstack-infra | 06:36 | |
| *** diablo_rojo has quit IRC | 06:40 | |
| *** hjensas has quit IRC | 06:40 | |
| *** dhill_ has quit IRC | 06:40 | |
| *** udesale has quit IRC | 06:43 | |
| *** udesale has joined #openstack-infra | 06:44 | |
| *** udesale has quit IRC | 06:45 | |
| *** udesale has joined #openstack-infra | 06:46 | |
| *** psachin has quit IRC | 06:47 | |
| *** dhill_ has joined #openstack-infra | 06:48 | |
| *** ramishra has quit IRC | 06:48 | |
| *** dhill_ has quit IRC | 06:54 | |
| *** roman_g has joined #openstack-infra | 06:57 | |
| *** udesale has quit IRC | 06:57 | |
| *** rcernin has quit IRC | 06:57 | |
| *** chkumar|away is now known as chkumar|ruck | 06:59 | |
| *** wolverineav has joined #openstack-infra | 07:01 | |
| *** slaweq has joined #openstack-infra | 07:04 | |
| *** ramishra has joined #openstack-infra | 07:05 | |
| *** wolverineav has quit IRC | 07:06 | |
| *** kjackal has joined #openstack-infra | 07:08 | |
| *** quiquell|off is now known as quiquell | 07:09 | |
| chkumar|ruck | ianw: Hello | 07:16 |
| *** dpawlik has joined #openstack-infra | 07:16 | |
| chkumar|ruck | ianw: http://logs.openstack.org/68/619968/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/7236dfd/job-output.txt.gz#_2018-11-28_05_50_32_997945 | 07:16 |
| chkumar|ruck | ianw: pypi mirrors are timing out | 07:16 |
| chkumar|ruck | ianw: http://mirror.dfw.rax.openstack.org/ | 07:16 |
| chkumar|ruck | ianw: please have a look | 07:16 |
| *** ramishra has quit IRC | 07:18 | |
| *** dpawlik has quit IRC | 07:20 | |
| *** ramishra has joined #openstack-infra | 07:21 | |
| openstackgerrit | Merged openstack-infra/system-config master: Retire bandersnatch mirroring https://review.openstack.org/618326 | 07:22 |
| *** dpawlik has joined #openstack-infra | 07:24 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix test race in test_hold_expiration_higher_than_default https://review.openstack.org/620222 | 07:24 |
| *** aojea has joined #openstack-infra | 07:28 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix test race in test_hold_expiration_higher_than_default https://review.openstack.org/620222 | 07:29 |
| *** dhill_ has joined #openstack-infra | 07:29 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Remove uneeded if statement https://review.openstack.org/617984 | 07:38 |
| *** ykarel is now known as ykarel|lunch | 07:41 | |
| *** quiquell is now known as quiquell|brb | 07:43 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix stuck job caused by exception during repo update https://review.openstack.org/590697 | 07:50 |
| *** udesale has joined #openstack-infra | 07:53 | |
| *** e0ne has joined #openstack-infra | 07:54 | |
| *** udesale has quit IRC | 07:58 | |
| *** kjackal has quit IRC | 08:01 | |
| *** kjackal_v2 has joined #openstack-infra | 08:01 | |
| *** jtomasek has joined #openstack-infra | 08:07 | |
| *** simon-AS559 has joined #openstack-infra | 08:09 | |
| *** ginopc has joined #openstack-infra | 08:09 | |
| *** quiquell|brb is now known as quiquell | 08:10 | |
| *** udesale has joined #openstack-infra | 08:11 | |
| *** roman_g has quit IRC | 08:12 | |
| *** xek_ has joined #openstack-infra | 08:22 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix test race in test_hold_expiration_higher_than_default https://review.openstack.org/620222 | 08:30 |
| *** florianf|afk is now known as florianf | 08:30 | |
| *** ahosam has quit IRC | 08:31 | |
| *** ahosam has joined #openstack-infra | 08:31 | |
| *** ralonsoh has joined #openstack-infra | 08:33 | |
| *** e0ne has quit IRC | 08:37 | |
| *** tosky has joined #openstack-infra | 08:51 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** rossella_s has joined #openstack-infra | 08:59 | |
| chkumar|ruck | tristanC: Hello | 09:01 |
| chkumar|ruck | tristanC: in some of the gate jobs http://mirror.dfw.rax.openstack.org/ is timing out on tripleo and openstack-ansible project side | 09:01 |
| chkumar|ruck | tristanC: http://logs.openstack.org/68/619968/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/7236dfd/job-output.txt.gz#_2018-11-28_05_50_26_092860 | 09:01 |
| chkumar|ruck | tristanC: please have a look | 09:02 |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** yboaron_ has quit IRC | 09:04 | |
| *** yboaron_ has joined #openstack-infra | 09:04 | |
| tristanC | chkumar|ruck: Hello, it seems like that mirror produce time-out, but i don't have access to it. Perhaps infra-core knows what's going on? | 09:06 |
| chkumar|ruck | tristanC: thanks :-) | 09:07 |
| *** shardy has joined #openstack-infra | 09:10 | |
| *** ahosam has quit IRC | 09:10 | |
| *** ahosam has joined #openstack-infra | 09:10 | |
| *** ginopc has quit IRC | 09:12 | |
| *** owalsh_ is now known as owalsh | 09:12 | |
| *** yboaron_ has quit IRC | 09:13 | |
| frickler | infra-root: running "gpg-agent --daemon emacs ..." will start a gpg-agent and an emacs process, and the gpg-agent will automatically terminate when emacs does | 09:21 |
| frickler | thanks go to prometheanfire for forwarding that solution to us, see http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-11-15.log.html#t2018-11-15T16:21:20 | 09:23 |
| *** yboaron_ has joined #openstack-infra | 09:25 | |
| *** jamesmcarthur has joined #openstack-infra | 09:26 | |
| *** kjackal_v2 has quit IRC | 09:27 | |
| *** kjackal has joined #openstack-infra | 09:27 | |
| *** jamesmcarthur has quit IRC | 09:30 | |
| *** roman_g has joined #openstack-infra | 09:32 | |
| *** ykarel|lunch is now known as ykarel | 09:39 | |
| *** yamamoto has joined #openstack-infra | 09:43 | |
| *** ahosam has quit IRC | 09:49 | |
| *** yamamoto has quit IRC | 10:02 | |
| *** bhavikdbavishi has quit IRC | 10:06 | |
| *** yamamoto has joined #openstack-infra | 10:07 | |
| *** tobias-urdin has quit IRC | 10:10 | |
| *** derekh has joined #openstack-infra | 10:12 | |
| *** e0ne has joined #openstack-infra | 10:14 | |
| *** tobias-urdin has joined #openstack-infra | 10:17 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/project-config master: Remove ansible-role-redhat-subscription from central repo https://review.openstack.org/617974 | 10:18 |
| *** ahosam has joined #openstack-infra | 10:19 | |
| evrardjp | is there anyone using the zuul-jobs role install-docker under ubuntu xenial? It seems I have trouble getting the docker service running in infra. | 10:21 |
| evrardjp | works perfectly fine in xenial | 10:22 |
| *** jistr is now known as jistr|mtg | 10:34 | |
| *** wolverineav has joined #openstack-infra | 10:37 | |
| *** electrofelix has joined #openstack-infra | 10:39 | |
| *** wolverineav has quit IRC | 10:42 | |
| *** ginopc has joined #openstack-infra | 10:43 | |
| *** ginopc has quit IRC | 10:43 | |
| *** ginopc has joined #openstack-infra | 10:44 | |
| AJaeger_ | evrardjp: http://codesearch.openstack.org/?q=install-docker&i=nope&files=&repos= | 10:53 |
| AJaeger_ | evrardjp: pbrx seems to use it | 10:53 |
| *** gtema has joined #openstack-infra | 10:53 | |
| AJaeger_ | evrardjp: as does zuul - and a few others. the link above should get you started to look into job config | 10:53 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Whitelist command processor thread in tests https://review.openstack.org/620559 | 10:57 |
| *** ahosam has quit IRC | 10:57 | |
| openstackgerrit | Benoît Bayszczak proposed openstack-infra/zuul master: add fetch_vault_secrets Ansible module https://review.openstack.org/620311 | 11:01 |
| evrardjp | AJaeger_: interesting. pbrx seem indeed to use bionic for pbrx-build-container-images . I have just noticed that it works fine under bionic if I am using upstream docker. So I will dig deeper if I was the only one to use system packages + bionic | 11:04 |
| evrardjp | and of pbrx is using upstream docker : ) | 11:04 |
| evrardjp | ofc* | 11:04 |
| evrardjp | and zuul seem to be using xenial , that all makes sense. | 11:06 |
| evrardjp | If I have time I will check on the reasons the docker daemon doesn't start. Probably a configuration thing. | 11:07 |
| tobiash | infra-root: are there some mirror problems? I saw several job timeouts today where all seem to be using the same mirror: e.g. | 11:08 |
| tobiash | http://logs.openstack.org/69/619069/4/check/zuul-quick-start/6a6ecb4/job-output.txt.gz#_2018-11-28_08_04_31_642473 | 11:08 |
| tobiash | http://logs.openstack.org/69/619069/4/check/zuul-quick-start/6a6ecb4/job-output.txt.gz#_2018-11-28_08_04_31_642473 | 11:08 |
| tobiash | http://logs.openstack.org/22/620222/5/check/pbrx-build-container-images/572a359/job-output.txt.gz#_2018-11-28_09_16_39_987812 | 11:08 |
| gtema | also openstacksdk jobs there are affected | 11:10 |
| tobiash | oh sorry, didn't read enough backlog | 11:10 |
| frickler | infra-root: mirror.dfw.rax fails to connect to pypi.org via IPv6. reaching other sites works, as does reaching pypi.org from other locations. so I'm assuming some IPv6 routing issue within rackspace. | 11:20 |
| *** bhavikdbavishi has joined #openstack-infra | 11:22 | |
| frickler | infra-root: created https://portal.rackspace.com/506635/tickets/list - waiting for feedback from rackspace now | 11:29 |
| *** chason has quit IRC | 11:31 | |
| *** chason has joined #openstack-infra | 11:36 | |
| *** dave-mccowan has joined #openstack-infra | 11:43 | |
| *** tpsilva has joined #openstack-infra | 11:45 | |
| *** yamamoto has quit IRC | 11:57 | |
| *** jpena is now known as jpena|lunch | 12:02 | |
| *** bhavikdbavishi has quit IRC | 12:03 | |
| *** gtema has quit IRC | 12:05 | |
| *** e0ne has quit IRC | 12:09 | |
| *** yamamoto has joined #openstack-infra | 12:11 | |
| *** janki has quit IRC | 12:20 | |
| *** witek has quit IRC | 12:23 | |
| *** witek has joined #openstack-infra | 12:23 | |
| *** ahosam has joined #openstack-infra | 12:33 | |
| *** takamatsu has quit IRC | 12:37 | |
| *** takamatsu has joined #openstack-infra | 12:43 | |
| ttx | frickler: though we did have a network-related fail from limestone on a release job... let me see | 12:53 |
| ttx | http://logs.openstack.org/ff/ff34e6541baa539de36c80c6f0ce8d687129644d/release/release-openstack-python/0658165/job-output.txt.gz#_2018-11-28_12_30_12_096183 | 12:53 |
| ttx | so I was wondering if it really was RAX-specific | 12:54 |
| ttx | (other two fails we got were from RAX though) | 12:54 |
| *** beagles is now known as beagles_mtgs | 12:55 | |
| frickler | ttx: that job tries to connect to pypi directly, not using our mirror, so it is susceptible to general pypi errors that may happen occasionally. probably not related to the other issue, I'd say | 12:55 |
| *** rh-jelabarre has joined #openstack-infra | 12:56 | |
| *** gtema has joined #openstack-infra | 12:58 | |
| *** rlandy has joined #openstack-infra | 13:00 | |
| *** jpena|lunch is now known as jpena | 13:02 | |
| *** e0ne has joined #openstack-infra | 13:05 | |
| *** trown|outtypewww is now known as trown | 13:12 | |
| *** ahosam has quit IRC | 13:13 | |
| *** rosmaita has joined #openstack-infra | 13:20 | |
| Dobroslaw | hello zuul magicians | 13:20 |
| Dobroslaw | I have some problem with my ingredients and are looking for logs what step stopped my job from running in `post` step | 13:20 |
| Dobroslaw | I know logs from `check` and `gate` runs are accessible from gerrit review, but what about `post` step? | 13:20 |
| Dobroslaw | Some more background: I have job that should push docker image to the registry | 13:21 |
| Dobroslaw | https://github.com/openstack/monasca-common/blob/master/.zuul.yaml#L49 | 13:21 |
| Dobroslaw | and it already run some months ago http://zuul.openstack.org/builds?job_name=docker-publish-monasca-base | 13:21 |
| Dobroslaw | but somehow it did not run for change merged last week: https://review.openstack.org/#/c/616861/ | 13:21 |
| *** jistr|mtg is now known as jistr | 13:21 | |
| Dobroslaw | so I would like to see logs from zuul did it decided to not run this job last time | 13:21 |
| Dobroslaw | * why did it decided to not run this job last time? | 13:22 |
| *** ykarel has quit IRC | 13:29 | |
| frickler | Dobroslaw: I'm guessing the combination of "files" and "irrelevant-files" is invalid here https://github.com/openstack/monasca-common/commit/9fd9a01e748d0aa81abc634d6bd6c5c480e45c77#diff-7415f5ff7beee2cdf9ffe31e12e4c086 | 13:29 |
| *** takamatsu has quit IRC | 13:29 | |
| *** takamatsu has joined #openstack-infra | 13:30 | |
| frickler | Dobroslaw: maybe zuul experts will know more details. corvus clarkb ^^ | 13:30 |
| Dobroslaw | frickler: but this job was run ok | 13:30 |
| Dobroslaw | https://review.openstack.org/#/c/616861/ | 13:30 |
| Dobroslaw | `docker-build-monasca-base` | 13:31 |
| Dobroslaw | `docker-publish-monasca-base` is child of it | 13:31 |
| frickler | Dobroslaw: hmm, true. checking zuul logs now | 13:34 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Remove the legacy-tempest-dsvm-nova-v20-api https://review.openstack.org/620598 | 13:36 |
| fungi | frickler: ttx: if we're seeing what could be pypi connectivity issues in rax-dfw and limestone, note that both are in texas so it's possible one of the local fastly cdn endpoints there is misbehaving (pypi.org uses fastly to frontend their service) | 13:40 |
| *** jamesmcarthur has joined #openstack-infra | 13:41 | |
| *** simon-AS559 has quit IRC | 13:44 | |
| *** takamatsu has quit IRC | 13:44 | |
| *** yamamoto has quit IRC | 13:45 | |
| *** ykarel has joined #openstack-infra | 13:46 | |
| fungi | huh, my ipv6 connectivity to both of those mirrors is cutting in and out | 13:46 |
| fungi | i wonder if there's a peering problem upstream from them both | 13:46 |
| fungi | when my ssh connection to them is working, they can both reach pypi.org | 13:47 |
| *** jcoufal has joined #openstack-infra | 13:47 | |
| frickler | Dobroslaw: ah, wait, I think the issue is that what is enqueued in the post queue is the merge commit, so gerrit doesn't list any changed files for it, thus the "files" match fails there | 13:48 |
| frickler | Dobroslaw: the log says "Job <Job docker-publish-monasca-base branches: {MatchAny:{BranchMatcher:master}} source: openstack-infra/project-config/zuul.d/jobs.yaml@master#26> did not match files in <Branch 0x7ff7c80388d0 openstack/monasca-common refs/heads/master updated 279769276f874c0ac6c41c5d94a6e902182a7b2f..d77a54053e993d0d49d1f74e15ed76376a0c5fb7>" | 13:48 |
| fungi | yes, gerrit has "files" for change objects, not for commit objects | 13:48 |
| *** takamatsu has joined #openstack-infra | 13:48 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: fix ref column value for newrev https://review.openstack.org/620602 | 13:49 |
| Dobroslaw | hmmm | 13:49 |
| Dobroslaw | so `files` for `check` and `gate` works differently than `post` step? I'm not sure I'm getting this | 13:50 |
| fungi | the "post" pipeline operates on a branch state, not a gerrit change | 13:51 |
| Dobroslaw | frickler: are this logs accessible somewhere online? | 13:51 |
| *** dhill_ has quit IRC | 13:51 | |
| frickler | Dobroslaw: sadly not | 13:51 |
| Dobroslaw | ok | 13:51 |
| *** kgiusti has joined #openstack-infra | 13:51 | |
| fungi | Dobroslaw: not the service logs (unfortunately zuul's public build reporting is all about what it did run, not about what it didn't run) | 13:52 |
| Dobroslaw | so in `post` state checking for files is useless if I understand correctly? | 13:53 |
| Dobroslaw | it would be possible to filter on them if I somehow get last commit from specific branch and check for changed files | 13:54 |
| *** yboaron_ has quit IRC | 13:54 | |
| *** janki has joined #openstack-infra | 13:55 | |
| Dobroslaw | but then I can't use `docker-build-monasca-base` as a parrent... | 13:55 |
| *** eharney has joined #openstack-infra | 13:56 | |
| *** sthussey has joined #openstack-infra | 13:56 | |
| frickler | Dobroslaw: maybe create "docker-build-monasca" with parent "docker-build-monasca-base" and move the "files" stanza to that. then use the new job for check+gate | 13:59 |
| Dobroslaw | frickler: oh, that's good idea, will try it, thank you | 14:00 |
| Dobroslaw | fungi: | 14:00 |
| Dobroslaw | fungi: thank you | 14:00 |
| fungi | Dobroslaw: filtering on files in post, if it were possible, is dangerous anyway since the post pipeline uses a supercedent pipeline manager which can skip commits if multiples merge close together in time | 14:05 |
| fungi | basically if a change merges zuul will start running jobs for it, and then if several more merge after that for the same project+branch zuul will drop all but whatever the last one was and hold it until jobs for the first one complete | 14:06 |
| fungi | it's assumed that projects will run a consistent set of jobs for all builds in post | 14:07 |
| *** yamamoto has joined #openstack-infra | 14:07 | |
| frickler | fungi: so likely we should have zuul generate a warning in that scenario, to avoid users having to find this out the hard way like in this case? | 14:10 |
| *** janki has quit IRC | 14:11 | |
| fungi | i wonder where it would warn about that | 14:11 |
| frickler | in the config parsing phase, make a warning or maybe an error even when a post job has a files attribute? | 14:12 |
| *** janki has joined #openstack-infra | 14:12 | |
| frickler | corvus: ^^ maybe you have some other idea for that | 14:13 |
| *** wolverineav has joined #openstack-infra | 14:14 | |
| Dobroslaw | hmmm, but this also mean that now I will be pushing new docker image on every commit, by adding `files` I wanted to avoid that | 14:15 |
| Dobroslaw | I only want to push new images if there are changes in `docker` folder | 14:16 |
| Dobroslaw | we don't need to have new image for any other change in monasca-common repo | 14:17 |
| *** wolverineav has quit IRC | 14:19 | |
| fungi | Dobroslaw: this sounds more like a use for the promote pipeline corvus was working on, which triggers on change-merged gerrit events so is per change and should be able to filter on files to selectively do the sort of thing you describe, i think | 14:23 |
| *** mriedem has joined #openstack-infra | 14:23 | |
| fungi | Dobroslaw: or for making (pre-)releases with regularity and only publishing images for those? | 14:23 |
| Dobroslaw | fungi: that would be fabulous | 14:23 |
| *** udesale has quit IRC | 14:24 | |
| *** udesale has joined #openstack-infra | 14:25 | |
| *** jamesmcarthur has quit IRC | 14:29 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use publish-zuul-python-branch-tarball job https://review.openstack.org/618634 | 14:29 |
| *** dhill_ has joined #openstack-infra | 14:30 | |
| *** jamesmcarthur has joined #openstack-infra | 14:31 | |
| TheJulia | Out of curiosity, did some logs being published get lost in the last 24 hours? | 14:35 |
| *** jamesmcarthur has quit IRC | 14:37 | |
| *** jamesmcarthur has joined #openstack-infra | 14:39 | |
| *** jamesmcarthur has quit IRC | 14:46 | |
| *** jamesmcarthur has joined #openstack-infra | 14:46 | |
| *** dklyle has quit IRC | 14:47 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Add openstack-virtual-baremetal to openstack https://review.openstack.org/620613 | 14:47 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Add openstack-virtual-baremetal to openstack https://review.openstack.org/620613 | 14:48 |
| fungi | TheJulia: logs of what? | 14:50 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/irc-meetings master: rpm-packaging: Adjust meeting time https://review.openstack.org/620616 | 14:51 |
| TheJulia | fungi: ran yesterday http://logs.openstack.org/26/591826/2/check/networking-generic-switch-tempest-dsvm-dlm/82bd49a/ | 14:51 |
| frickler | TheJulia: looking at gerrit it ran 2 months ago | 14:53 |
| TheJulia | soryr to bother you guys, clearly I'm getting pulled in too many different directions htis morning to focus on any single thing :( | 14:54 |
| fungi | no worries! | 14:55 |
| fungi | september and november are easily confused, take it from me | 14:55 |
| frickler | TheJulia: np, it's nice to have some easy questions to solve, too ;) | 14:55 |
| TheJulia | I have two simillarly named patches, guess I should abandon the old one | 14:57 |
| *** eernst has joined #openstack-infra | 15:00 | |
| fungi | i love when i push up a patch for a bug, then notice a similar patch listed as a conflict, and realize i'd already pushed up the same fix a year before but nobody's reviewed it | 15:01 |
| *** dhill_ has quit IRC | 15:02 | |
| *** bobh has joined #openstack-infra | 15:03 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:04 | |
| *** timrc has quit IRC | 15:06 | |
| smcginnis | That's funny a really sad at the same time. | 15:10 |
| TheJulia | well, it was more along the lines of people wanted refactoring work done first :\ | 15:12 |
| *** graphene has quit IRC | 15:14 | |
| *** graphene has joined #openstack-infra | 15:15 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/puppet-pip master: Use official pypi.python.org as fallback with pip https://review.openstack.org/620626 | 15:15 |
| *** dpawlik has quit IRC | 15:16 | |
| *** chkumar|ruck is now known as chkumar|away | 15:17 | |
| *** dhill_ has joined #openstack-infra | 15:17 | |
| *** jamesmcarthur has quit IRC | 15:18 | |
| *** jamesmcarthur has joined #openstack-infra | 15:22 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create yarn_lock_file_path variable for install-yarn https://review.openstack.org/620628 | 15:24 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Use official pypi.python.org as fallback with pip https://review.openstack.org/620630 | 15:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Set yarn_lock_file_path for zuul-tarball jobs https://review.openstack.org/620631 | 15:28 |
| pabelanger | mordred: tobiash: config-core: ^ could use a review please to fix a bug with zuul tarball publish jobs | 15:29 |
| *** pbourke has quit IRC | 15:37 | |
| *** beagles_mtgs is now known as beagles | 15:37 | |
| *** pbourke has joined #openstack-infra | 15:38 | |
| *** ykarel is now known as ykarel|away | 15:41 | |
| *** quiquell is now known as quiquell|off | 15:41 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: executor: harden add_host usage https://review.openstack.org/620635 | 15:42 |
| *** udesale has quit IRC | 15:43 | |
| *** udesale has joined #openstack-infra | 15:43 | |
| *** panda|rover is now known as panda|pto | 15:45 | |
| *** ramishra has quit IRC | 15:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Use the SQLAlchemy ORM https://review.openstack.org/620426 | 15:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add artifact table https://review.openstack.org/620427 | 15:53 |
| *** armax has joined #openstack-infra | 15:54 | |
| corvus | infra-root: https://dnssec-analyzer.verisignlabs.com/opendev.org is green | 15:54 |
| clarkb | \o/ | 15:54 |
| *** rlandy is now known as rlandy|brb | 15:55 | |
| fungi | awesome! | 15:55 |
| corvus | i'll push up patches to create a website | 15:55 |
| fungi | thanks!!! | 15:56 |
| mordred | corvus: \o/ | 15:56 |
| pabelanger | w00t | 15:57 |
| *** eernst has quit IRC | 16:00 | |
| *** diablo_rojo has joined #openstack-infra | 16:05 | |
| ccamacho | hi fungi | 16:06 |
| ccamacho | man quick question | 16:06 |
| ccamacho | we might have a problem in instack-undercloud taking changes from master instead of stable/rocky | 16:06 |
| fungi | fire | 16:06 |
| ccamacho | would you mind to check last comment from bnemec in https://review.openstack.org/#/c/619467/ ? | 16:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Create opendev-website project https://review.openstack.org/620643 | 16:06 |
| ccamacho | in this case for the build-openstack-releasenotes job | 16:07 |
| fungi | ccamacho: i'm in a meeting at the moment but can take a peek around 17:00 utc (or sooner if it ends early) | 16:07 |
| ccamacho | sure thing man thanks! | 16:07 |
| fungi | ccamacho: oh! you might want to ask in #openstack-release, but the releasenotes jobs all operate on the master branch regardless of which branch triggers the job | 16:07 |
| fungi | the reason has to do with making sure release notes are gathered from all available branches each time it's run | 16:07 |
| mwhahaha | ccamacho: we need to drop the release no jobs from instakc-udnercloud | 16:08 |
| mwhahaha | s/no/on | 16:08 |
| mwhahaha | since we nuked the master branch they will no longer function | 16:08 |
| ccamacho | aaa mwhahaah ack | 16:08 |
| ccamacho | yeahp | 16:08 |
| ccamacho | I see now | 16:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Create opendev-website project https://review.openstack.org/620643 | 16:09 |
| ccamacho | mwhahaha just to confirm, shoulb be on the zuul/project.yaml right? | 16:10 |
| ccamacho | in instack-undercloud | 16:10 |
| mwhahaha | not sure where those live, might still be in the .zuul.d/ in instack-undercloud | 16:10 |
| mwhahaha | ccamacho: yea i think the release job definition is in project config | 16:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add opendev-website project to Zuul https://review.openstack.org/620646 | 16:11 |
| corvus | mwhahaha, ccamacho: http://git.openstack.org/cgit/openstack/instack-undercloud/tree/zuul.d/project.yaml?h=stable/rocky has 'release-notes-jobs' template | 16:15 |
| mwhahaha | ah i was looking in the wrong file | 16:15 |
| mwhahaha | ccamacho: so yea just drop the releases-notes-job from the stable branches | 16:16 |
| fungi | makes sense. in theory a retired project which only has stable branches maintained shouldn't be adding anything to stable worthy of a release note anyway | 16:17 |
| *** agopi is now known as agopi|food | 16:20 | |
| ccamacho | corvus mwhahaha fungi ack Ill drop it here https://review.openstack.org/#/c/619467/ and then Ill push it to older branches | 16:24 |
| *** dpawlik has joined #openstack-infra | 16:24 | |
| ccamacho | thanks for info! | 16:24 |
| *** gyee has joined #openstack-infra | 16:27 | |
| jamesmcarthur | clarkb: fungi: corvus: It appears that CSC finally came through with all of our requests. | 16:29 |
| jamesmcarthur | Would you all mind giving it a look to make sure things are working as expected? | 16:29 |
| fungi | jamesmcarthur: we noticed! corvus is working on getting a minimal website up on that domain next | 16:29 |
| fungi | it all looks good to us now | 16:29 |
| clarkb | jamesmcarthur: corvus confirmed a few minutes ago that it looked good. Thank you for all the help with that | 16:29 |
| jamesmcarthur | awesome | 16:29 |
| jamesmcarthur | for sure. Sorry it took so long. | 16:29 |
| fungi | not your fault | 16:29 |
| jamesmcarthur | It's nice that we all got to collectively bang our heads against the wall. | 16:30 |
| *** jamesmcarthur has quit IRC | 16:30 | |
| *** rlandy|brb is now known as rlandy | 16:34 | |
| *** udesale has quit IRC | 16:34 | |
| *** jamesmcarthur has joined #openstack-infra | 16:41 | |
| *** dpawlik has quit IRC | 16:42 | |
| *** hamerins has joined #openstack-infra | 16:45 | |
| *** jamesmcarthur has quit IRC | 16:46 | |
| *** hamerins has quit IRC | 16:47 | |
| *** hamerins has joined #openstack-infra | 16:48 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/puppet-pip master: Use official pypi.python.org as fallback with pip https://review.openstack.org/620626 | 16:51 |
| *** eernst has joined #openstack-infra | 16:57 | |
| smcginnis | Am I remembering correctly that IRC channel topics are applied or refreshed automatically out of a config file somewhere? | 16:59 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Use official pypi.python.org as fallback with pip https://review.openstack.org/620630 | 17:00 |
| clarkb | smcginnis: they are not. The status bot does its best to restore topics it changes, but channel ops are given the ability to /topic | 17:00 |
| *** lbragstad has quit IRC | 17:02 | |
| *** e0ne has quit IRC | 17:02 | |
| frickler | clarkb: can you check whether there was a response from rackspace in the meantime for the v6 issue? I'm not at my workstation anymore. we might need to consider a different workaround if it lasts | 17:02 |
| smcginnis | clarkb: Any idea who has ops in #openstack-operators? Someone pointed out its topic points to the old mailing list. | 17:02 |
| clarkb | frickler: ya I can take a look in a bit | 17:02 |
| clarkb | finishing up a meeting | 17:02 |
| *** dpawlik has joined #openstack-infra | 17:03 | |
| *** tdasilva has joined #openstack-infra | 17:03 | |
| *** pcaruana has quit IRC | 17:03 | |
| frickler | smcginnis: clarkb: I can take a look at the channel topic in a moment | 17:03 |
| *** eernst has quit IRC | 17:03 | |
| clarkb | smcginnis: /msg chanserv access #openstack-operators list | 17:03 |
| smcginnis | Thanks! | 17:04 |
| *** lbragstad has joined #openstack-infra | 17:05 | |
| smcginnis | Also willing to take care of it if someone can make me an operator | 17:06 |
| *** dpawlik has quit IRC | 17:07 | |
| corvus | smcginnis: one of the existing channel ops can add you and that would be best; but if they've all gone awol, we can do it | 17:08 |
| smcginnis | Yeah, looking at the list, I think it's all infra folks. | 17:09 |
| clarkb | frickler: the ticket was escalated to rax engineers ~2 hours ago, but no update since | 17:09 |
| *** aojea has quit IRC | 17:09 | |
| clarkb | cloudnull: ^ not sure if that interests you. 181128-ord-0000464 is ticket number | 17:09 |
| * cloudnull looking | 17:10 | |
| * cloudnull reading back | 17:10 | |
| frickler | smcginnis: you should be op now | 17:11 |
| *** bobh has quit IRC | 17:13 | |
| smcginnis | Thanks frickler! | 17:14 |
| *** gtema has quit IRC | 17:16 | |
| *** ssbarnea|bkp2 has quit IRC | 17:20 | |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/system-config master: Blackhole messages to openstack-ko-owner@l.o.o https://review.openstack.org/620661 | 17:24 |
| *** hamerins has quit IRC | 17:28 | |
| *** hamerins has joined #openstack-infra | 17:29 | |
| *** ssbarnea has joined #openstack-infra | 17:29 | |
| hogepodge | AJaeger_: since you've been making the most contributions to the project since it launched, and there's no team to speak of yet, do you want to be promoted to core on Four Opens? | 17:32 |
| *** ginopc has quit IRC | 17:33 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Add 'Review-Priority' for Cinder repos https://review.openstack.org/620664 | 17:38 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack-infra/bindep master: Add OracleLinux support https://review.openstack.org/536355 | 17:38 |
| *** jpich has quit IRC | 17:40 | |
| *** janki has quit IRC | 17:41 | |
| frickler | fungi: what do you think about notifying people still mailing to the old lists about the migration by direct mail? of course I don't propose this without also volunteering to perform that task | 17:43 |
| fungi | frickler: they get moderation queue notices from openstack-discuss already each time they send to one of the other lists | 17:44 |
| fungi | i've been watching and this seems to already have prompted lots of people to subscribe immediately after sending to one of the old lists | 17:44 |
| frickler | fungi: oh, o.k., that might be good enough, then | 17:45 |
| fungi | seems to be working so far | 17:46 |
| fungi | last night i ran a comparison of the addresses seen posting to the old mailing lists this year and the addresses subscribed to the new one. >50% of people who sent more than 10 messages to the old lists have subscribed already | 17:47 |
| *** graphene has quit IRC | 17:48 | |
| *** trown is now known as trown|lunch | 17:48 | |
| *** graphene has joined #openstack-infra | 17:49 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Add openstack-virtual-baremetal to openstack https://review.openstack.org/620613 | 17:50 |
| mriedem | i assume the apt/pypi mirrors is a known issue right now? | 17:50 |
| *** florianf is now known as florianf|afk | 17:50 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add allowed-labels tenant setting https://review.openstack.org/617740 | 17:52 |
| *** jamesmcarthur has joined #openstack-infra | 17:53 | |
| clarkb | mriedem: yes, rax is looking at it | 17:53 |
| clarkb | mriedem: aiui ipv6 is flaky. I think if it isn't fixed soon we might either disable the dfw region or disable ipv6 there? | 17:54 |
| mriedem | clarkb: ok just wanted to make sure it was known | 17:56 |
| mriedem | thanks | 17:56 |
| *** jamesmcarthur has quit IRC | 17:58 | |
| fungi | we saw evidence of it impacting both rax-dfw and limestone-regionone so it could be a regional peering problem impacting them both | 17:59 |
| fungi | if it persists we may want to take those regions offline until we can confirm it's no longer happening | 17:59 |
| fungi | (they're both in texas) | 17:59 |
| *** sshnaidm is now known as sshnaidm|afk | 18:01 | |
| fungi | my v6 routing to them was up and down earlier today too. looks like both providers peer with hurricane electric in dallas which is how i'm reaching them | 18:01 |
| *** hamerins has quit IRC | 18:01 | |
| fungi | so that _could_ be the common point of impact | 18:01 |
| *** armax has quit IRC | 18:01 | |
| fungi | but i haven't tried running reverse traceroutes to see whether it's symmetrical | 18:01 |
| *** hamerins has joined #openstack-infra | 18:02 | |
| fungi | right now i can `wget -O /dev/null https://pypi.org/` from both mirror hosts and it successfully retrieves content over ipv6 so it's obviously not 100% of the time | 18:03 |
| *** armax_ has joined #openstack-infra | 18:04 | |
| *** jpena is now known as jpena|off | 18:09 | |
| AJaeger_ | hogepodge: regarding four-opens - sorry, I don't plan to contribute many more changes to it. My main focus was helping you started for a good set up. But if you need another reviewer, I can help... | 18:09 |
| hogepodge | AJaeger_: cool, thanks | 18:09 |
| AJaeger_ | hogepodge: ok, will add it to my watchlist... | 18:11 |
| *** shardy has quit IRC | 18:11 | |
| *** hamerins has quit IRC | 18:12 | |
| *** hamerins has joined #openstack-infra | 18:12 | |
| *** rtjure has joined #openstack-infra | 18:13 | |
| fungi | infra-root: noting that we have a buuuuunch of projects with the openstack-dev ml address in their package config files (even changing those in git doesn't change them for old releases on pypi) as well as threads on old lists which even subscribers to the new list are still likely to attempt to reply on from time to time, what do you think about configuring exim on lists.o.o to alias the old list | 18:16 |
| fungi | posting addresses to openstack-discuss? i've tested it manually and seems to work | 18:16 |
| fungi | obviously we wouldn't do that until monday when we're ready to retire the old lists | 18:17 |
| AJaeger_ | fungi: what about an autoresponder "Use -discuss instead"? | 18:17 |
| corvus | fungi: no objections here; though also someone should propose changes to all the projects to change it; maybe they will beat dhellmann at stackalytics this round | 18:17 |
| fungi | AJaeger_: autoresponders create backscatter, these lists get hundreds of spam messages from non-subscribers in their moderation queues already | 18:17 |
| AJaeger_ | I see | 18:18 |
| fungi | so an autoresponder would basically just give most of the internet a reason to blacklist messages from the server | 18:18 |
| clarkb | fungi: the forward would force -discuss to filter that spam? Though I expect it will get a fair amount on its own so probably not a major concern | 18:19 |
| fungi | it already is getting a fair amount of spam, and yes i suppose that would be a downside | 18:19 |
| clarkb | based on the -infra spam seems like blocking qq.com would avoid 80% of the spam. But unsure if qq.com is used legitemetely too | 18:19 |
| *** bhavikdbavishi has quit IRC | 18:20 | |
| corvus | qq is still a real thing | 18:20 |
| fungi | qq.com is spoofed for most of that spam specifically *because* it's the most popular e-mail provider in mainland china | 18:20 |
| clarkb | ah | 18:20 |
| clarkb | TIL | 18:20 |
| fungi | blocking qq.com would be the eastern equivalent of blocking gmail.com | 18:20 |
| fungi | we had legitimate posts to the old mls from 6 different qq.com addresses in 2018 | 18:22 |
| fungi | so while not a ton, still not something i'd want to block | 18:22 |
| *** wolverineav has joined #openstack-infra | 18:23 | |
| *** ralonsoh has quit IRC | 18:25 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix test race in test_hold_expiration_higher_than_default https://review.openstack.org/620222 | 18:30 |
| openstackgerrit | Merged openstack-infra/nodepool master: Cache node request zNodes https://review.openstack.org/618806 | 18:30 |
| *** agopi|food is now known as agopi | 18:33 | |
| *** electrofelix has quit IRC | 18:45 | |
| *** wolverineav has quit IRC | 18:50 | |
| *** wolverineav has joined #openstack-infra | 18:51 | |
| frickler | corvus: clarkb: this is what I made out of the ksc functional job we discussed yesterday: https://review.openstack.org/620553 is my understanding correct that I cannot reference a playbook from devstack, but need to copy it to the local repo like I did? | 18:53 |
| *** graphene has quit IRC | 18:55 | |
| *** graphene has joined #openstack-infra | 18:56 | |
| frickler | is there a way for us to recover an abandoned, ownerless #openstack-* channel other than contacting freenode admins? I'd like to set "+i" on #openstack-designate so that the forward to #openstack-dns actually takes effect | 18:57 |
| logan- | fungi: o/ saw the note about ipv6 issues in dfw? happy to help debug if needed | 18:58 |
| fungi | logan-: i don't know if they're still ongoing | 18:58 |
| fungi | lemme check | 18:58 |
| *** trown|lunch is now known as trown | 18:59 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create yarn_lock_file_path variable for install-yarn https://review.openstack.org/620628 | 18:59 |
| *** hamerins has quit IRC | 19:00 | |
| pabelanger | fungi: clarkb: corvus: mind a review on https://review.openstack.org/620631/ to fix zuul tarball jobs | 19:01 |
| *** hamerins has joined #openstack-infra | 19:01 | |
| pabelanger | the jobs weren't installing yarn properly | 19:01 |
| fungi | logan-: http://status.openstack.org/elastic-recheck/ suggests we're still seeing signs of pip download failures... i'll see if i can spot whether they're still particular providers | 19:02 |
| *** ykarel|away has quit IRC | 19:03 | |
| *** dklyle has joined #openstack-infra | 19:03 | |
| fungi | logan-: looks like they're all rax-dfw in the past 6 hours, so i guess not | 19:04 |
| logan- | i checked the 3 pypi/pip ones (1708686, 1326813, and 1449136) ... looks like all rax-dfw | 19:04 |
| logan- | yeah agreed | 19:04 |
| corvus | frickler: i can take care of the #openstack-designate issue; i'll let you know when it's done | 19:04 |
| corvus | frickler: https://review.openstack.org/620553 looks fine unless you think that devstack-tox-functional-consumer should just be changed to drop the test-setup role. (cc mordred) | 19:07 |
| *** e0ne has joined #openstack-infra | 19:07 | |
| *** lbragstad has quit IRC | 19:08 | |
| frickler | corvus: well, there's more to it in the meantime, like using devstack-minimal in order to reduce the footprint. also using the -consumer variant was wrong IMO when the job also runs against keystone patches that may break devstack setup | 19:10 |
| corvus | frickler: ah ok | 19:11 |
| *** lbragstad has joined #openstack-infra | 19:12 | |
| *** lbragstad has quit IRC | 19:12 | |
| *** rosmaita has left #openstack-infra | 19:14 | |
| *** lbragstad has joined #openstack-infra | 19:14 | |
| *** lbragstad has quit IRC | 19:14 | |
| *** e0ne has quit IRC | 19:15 | |
| *** lbragstad has joined #openstack-infra | 19:16 | |
| *** lbragstad has quit IRC | 19:16 | |
| *** e0ne has joined #openstack-infra | 19:17 | |
| *** e0ne has quit IRC | 19:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Remove summit event notices https://review.openstack.org/620681 | 19:21 |
| fungi | corvus: thanks! i had that on my to do list for this week. one less thing to worry about | 19:22 |
| corvus | i just commented it out so it'll be easier to add back | 19:23 |
| openstackgerrit | Merged openstack-infra/zuul master: executor: harden add_host usage https://review.openstack.org/620635 | 19:24 |
| openstackgerrit | Merged openstack-infra/project-config master: Set yarn_lock_file_path for zuul-tarball jobs https://review.openstack.org/620631 | 19:24 |
| fungi | corvus: that's what i was thinking of doing too, so... perfect! | 19:25 |
| fungi | as soon as the draft render posts i'll review | 19:25 |
| *** dklyle has quit IRC | 19:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create opendev-website project https://review.openstack.org/620643 | 19:29 |
| pabelanger | corvus: mordred: what are your thoughts about cleaning up https://github.com/openstack-infra/puppet-zuul/blob/master/manifests/web.pp#L219 once we confirm publishing jobs are working properly with yarn. I'm unsure if we have zuul use the published wheel from tarballs.o.o or we also include yarn bits on production server | 19:30 |
| pabelanger | sorry, I should have linked git.o.o there | 19:31 |
| pabelanger | I've adapted to github.com faster them I hoped | 19:31 |
| clarkb | pabelanger: the infra deployment still uses the tarball on tarballs iirc | 19:32 |
| clarkb | pabelanger: we should switch to using the in sdist/whl contents as soonas that is reliable | 19:32 |
| pabelanger | Yah, i think that will be helpful. That is one reason why we didn't see the issue sooner | 19:33 |
| pabelanger | I can see yarn bits now in http://logs.openstack.org/27/2728e5d4adac81b3c79c6a453676565fc10fda9d/post/publish-zuul-python-branch-tarball/2bab6ed/job-output.txt.gz | 19:34 |
| pabelanger | and both tarball and wheel look to have web bits too | 19:35 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/irc-meetings master: Update Meeting Chairs https://review.openstack.org/620685 | 19:39 |
| fungi | "the tarball" isn't an sdist already then? | 19:41 |
| clarkb | fungi: no we produced an archive of the web bits separately aiui. The reason for this is they weren't going into the sdist, but as of a few minutes ago we now publish them as part of the sdist/whl too | 19:42 |
| fungi | oh, so there's a job building a separate tarball of the static web content | 19:43 |
| pabelanger | yah, that job can go away now | 19:44 |
| pabelanger | once we update puppet-zulu | 19:44 |
| pabelanger | puppet-zuul* | 19:44 |
| *** mrhillsman is now known as mrhillsman|lunch | 19:46 | |
| mordred | I could go either way ... | 19:52 |
| mordred | but just to toss a few thoughts into the bucket - we're currently serving the static files built in the javascript tarball directly via apache. we could also just let zuul-web serve them and pick them up through the apache proxy and just stop serving them directly as static files | 19:53 |
| mordred | we weren't producing the web bits separately because they weren't in the sdist - they *were* going in to the sdist and wheel - but then we broke that and didn't notice. :) | 19:54 |
| mordred | but - in all, I don't think we need the static file offloading - and serving them via zuul-web would certainly help us notice that our sdist/wheels are broken | 19:55 |
| clarkb | mordred: and if performance is a concern we can have apache cache them | 19:57 |
| pabelanger | yah, I'm mostly thinking, what is the right way for a non openstack-infra deploy to do it. Cause, I haven't actually done that part yet in my local testing. I figured it was just use the static bits in the wheel now | 19:57 |
| clarkb | (reality is the json status file is likely the biggest artifact and it changes often so a few tertiary things likely aren't a major concern) | 19:57 |
| mordred | yeah. I mean - I think we already have apache caching them based on headers | 19:57 |
| mordred | pabelanger: just having zuul-web serve the content is probably what most people should do until/unless there is a reason to do it otherwise - just cause it's easier | 19:58 |
| mordred | so yeah - I think most people should deploy it via the wheel we're publishing - and at this point I'm not sure there's a compelling reason for openstack-infra to deploy it differently | 19:58 |
| *** eharney has quit IRC | 19:59 | |
| pabelanger | k | 19:59 |
| mordred | clarkb, pabelanger: also, the javascript build process produces minified content in hashed filenames, so the javascript bundles can be _aggressively_ cached ... each one will literally never change | 20:00 |
| mordred | and when a change is pushd, it's a new hashed filename - with the tiny index.html file being updated to reference the new hashes | 20:00 |
| clarkb | ah perfect | 20:00 |
| clarkb | though what happens when someone requests the old file? | 20:01 |
| clarkb | they have to hard refresh I guess | 20:01 |
| *** wolverineav has quit IRC | 20:05 | |
| mordred | well, they should only ever request an old file if they've gotten an old copy of the index.html - that one should be served/cached differently - like with an etag or something else | 20:05 |
| mordred | it *mostly* should just work | 20:06 |
| *** wolverineav has joined #openstack-infra | 20:07 | |
| fungi | if they really want apache to serve those files instead, they can always just stick the paths to them on the fs into their vhost configf | 20:11 |
| fungi | and then the requests won't hit zuul-web at all | 20:11 |
| *** eharney has joined #openstack-infra | 20:11 | |
| mordred | indeed | 20:13 |
| mordred | but we should be providing all the right headers from zuul-web so that an apache proxy will appropriately cache at the apache layer | 20:13 |
| mordred | without even needing the complexity | 20:13 |
| *** wolverineav has quit IRC | 20:16 | |
| *** e0ne has joined #openstack-infra | 20:19 | |
| *** e0ne has quit IRC | 20:20 | |
| *** rossella_s has quit IRC | 20:23 | |
| *** kjackal has quit IRC | 20:27 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add arbitrary node metdata config option https://review.openstack.org/620691 | 20:27 |
| corvus | infra-root: i'm going to hard-restart the zuul executors to pick up a security fix | 20:28 |
| clarkb | corvus: ++ let me know if I can help. Though I'm currently trying to do administrivia tasks (just set up u2f multifactor auth with fastmail) | 20:29 |
| *** lbragstad has joined #openstack-infra | 20:30 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add arbitrary node metdata config option https://review.openstack.org/620691 | 20:30 |
| corvus | oh i'll clean up the builds directories while they're stopped | 20:32 |
| Shrews | did we never land that patch to do it for us? <-- corvus | 20:32 |
| Shrews | or am i thinking of something else.... | 20:32 |
| corvus | Shrews: hrm, i think it did land... | 20:33 |
| clarkb | ya I thought that was a thing | 20:33 |
| corvus | maybe i should just save a list of build dirs, start them, and then check whether it worked | 20:33 |
| corvus | also, maybe we should have the executor stats keep running until the process exits | 20:35 |
| corvus | (would be nice to see the graph wind down the number of jobs, disk usage, etc) | 20:35 |
| *** mrhillsman|lunch is now known as mrhillsman | 20:35 | |
| clarkb | I approximate that with ps -elf | grep zuul | wc -l | 20:35 |
| clarkb | but ya specific info would be nice | 20:35 |
| *** openstackgerrit has quit IRC | 20:36 | |
| pabelanger | +1 to keep statsd running on stop | 20:36 |
| *** e0ne has joined #openstack-infra | 20:40 | |
| *** armax_ has quit IRC | 20:41 | |
| corvus | okay, they've all stopped and i saved a list of build dirs | 20:43 |
| corvus | and restarted | 20:44 |
| *** ahosam has joined #openstack-infra | 20:48 | |
| clarkb | fwiw u2f is becoming more common and there are cheap open source hardware tokens for u2f. I've now got it working with fastmail, google, and github | 20:48 |
| *** wolverineav has joined #openstack-infra | 20:48 | |
| AJaeger_ | corvus: I fear you need to split up https://review.openstack.org/#/c/620646/ - see my comment there | 20:49 |
| *** e0ne has quit IRC | 20:49 | |
| clarkb | with firefox too | 20:49 |
| corvus | AJaeger_: ah yep | 20:50 |
| *** wolverineav has quit IRC | 21:02 | |
| *** wolverineav has joined #openstack-infra | 21:06 | |
| *** openstackgerrit has joined #openstack-infra | 21:06 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix deletion of stale build dirs on startup https://review.openstack.org/620697 | 21:06 |
| corvus | clarkb, Shrews: ^ *cough* | 21:06 |
| mordred | corvus: yay! | 21:07 |
| clarkb | that particular method of bug has shown up in other places | 21:09 |
| clarkb | seems like that is the sort of rule that maybe hacking should look for | 21:09 |
| clarkb | meh | 21:09 |
| openstackgerrit | Merged openstack-infra/zuul master: web: fix ref column value for newrev https://review.openstack.org/620602 | 21:14 |
| *** eharney has quit IRC | 21:20 | |
| *** e0ne has joined #openstack-infra | 21:21 | |
| *** e0ne has quit IRC | 21:21 | |
| *** wolverineav has quit IRC | 21:22 | |
| *** wolverineav has joined #openstack-infra | 21:23 | |
| *** wolverineav has quit IRC | 21:23 | |
| *** wolverineav has joined #openstack-infra | 21:23 | |
| *** e0ne has joined #openstack-infra | 21:26 | |
| *** graphene has quit IRC | 21:27 | |
| *** e0ne has quit IRC | 21:27 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use the SQLAlchemy ORM https://review.openstack.org/620426 | 21:28 |
| ianw | sorry, just catching up after a plugin-to-docking-station-oh-everythings-crashed-hrm-there's-a-new-firmware-i'll-do-that-oh-i-need-a-usb-key-find-one-oh-doesn't-work-hrm-let's-try-the-usb-port-on-other-side-oh-now-it-does-ok-now-i-can-boot-again dance | 21:29 |
| ianw | did the rax ipv6 pypi issue clear up? | 21:29 |
| corvus | that was one character wider than my window | 21:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Require sysadmin github accounts use 2fa https://review.openstack.org/620702 | 21:33 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Stop requiring separate github accounts https://review.openstack.org/620703 | 21:33 |
| clarkb | infra-root ^ this is more of a conversation starter than anything else. I'd appreciate feedback/thoughts on both of those | 21:33 |
| clarkb | fwiw I've been using u2f hardware tokens with github on my account for about a year now and haven't found it to be too problematic. There are also cheap keys available with open source hardware/software (and if cost is prohibitive I think we can likely get that expensed) | 21:34 |
| ianw | clarkb: http://logs.openstack.org/71/618671/14/experimental/nodepool-functional-py35-redhat-src/facfefe/ ... this is rather disappointing. using what seem to be exactly the same service dependencies as cloud init, we do not seem to get a good network bringup with glean+nm on centos | 21:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add opendev-website jobs https://review.openstack.org/620646 | 21:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add opendev-website project to Zuul https://review.openstack.org/620704 | 21:34 |
| clarkb | the second thing is an item that occuured to me writing the first. I think we mostly avoided that as an issue by not using github outside of mirroring openstack, but more and more we have infra-root that do use github for day to day work related thigns | 21:35 |
| corvus | clarkb: oh the idea was to avoid permissions leakage, not security | 21:35 |
| clarkb | so we should reconsider if the rule is necessary and if so maybe enforce it | 21:35 |
| clarkb | corvus: ya I think there are two different things going on there. One is security (2fa) the other is accidental push of code where you don't want it | 21:35 |
| clarkb | corvus: I'm happy to start enforcing both items (I don't think we've really enforced either), but wanted to bring them up for discussion | 21:36 |
| clarkb | ianw: that is disappointing. Maybe just go back to the after local-fs? | 21:36 |
| *** armax has joined #openstack-infra | 21:36 | |
| ianw | clarkb: yeah ... i don't think anyone really understands all this; it's a good idea that seems to have got too complex | 21:37 |
| *** lbragstad has quit IRC | 21:37 | |
| *** lbragstad has joined #openstack-infra | 21:37 | |
| ianw | clarkb: i'm not really sure how i got in openstack admin accounts on github, i think i was added at some point, i'd be happy to make a separate account because it really pollutes your notifications etc | 21:38 |
| clarkb | ianw: ya I think we've just been bad at enforcing that particular rule. I don't expect anyone to object if we asked everyone to make a second account and do a switch over | 21:38 |
| clarkb | but considering we hadn'y really enforced it I thought it was worht discussing if it is worth enforcing (being cautious/paranoid I'm inclined to start enforcing the rule, but someone like mordred who uses github more may have other insights, and for fungi I expect his answer is he only uses github for this purpose so already meets the criteria) | 21:39 |
| ianw | on the 2fa one, i presume you mean you'll flip the switch on github that requires 2fa? i think it has one (or maybe i'm getting confused with google's admin panels) | 21:43 |
| clarkb | ianw: reading the gentoo doc it appears that that is a org by org setting | 21:43 |
| clarkb | so yes would toggle it there then expect people to set it up. We can wait for a specific day if people need to sort out how they want to do 2fa first | 21:44 |
| corvus | i would probably seriously consider relinquishing my github privileges; i'm not sure it's worth the hassle for me | 21:44 |
| ianw | totp is fine, if we want keys i've never tried getting a key into my work vm via virtualbox, i presume i'd have to install the proprietary bits to get usb tunneling | 21:45 |
| clarkb | ianw: ya totp is prbably fine. I personally prefer u2f because it doesn't have shared secrets but in this case the benefit of totp over nothing is still worthwhile | 21:46 |
| ianw | corvus: yeah, i don't think any of us *want* it? but we can't really do 2fa with a shared account (i.e. one global admin account and password in the usual place)? | 21:47 |
| clarkb | ianw: you could with totp actually since it is shared secret | 21:47 |
| ianw | maybe could put the totp key in there somehow, but not sure how easy it is to regenerate from a token with usual apps | 21:47 |
| corvus | or maybe i'd get around to setting up that second account so it can have admin access and 2fa, and i never use it | 21:48 |
| corvus | er, actually, i did set up the other account, i just didn't switch to it | 21:48 |
| ianw | clarkb: heh, yeah we're thinking the same thing :) authy has an ENTER KEY MANUALLY option | 21:48 |
| corvus | this is not terrible and doesn't require more stuff in my pockets: https://play.google.com/store/apps/details?id=org.fedorahosted.freeotp&hl=en_US | 21:49 |
| clarkb | corvus: yup Ithink something like that would be fine | 21:49 |
| *** rcernin has joined #openstack-infra | 21:50 | |
| ianw | i'm sure there's something to generate totp tokens from the command-line? we could have it on bridge, with a shared account, and just a note on how to get the token | 21:50 |
| clarkb | oathtool says google | 21:51 |
| * clarkb checks if that is a debuntu package | 21:51 | |
| clarkb | it is (in universe) | 21:51 |
| clarkb | so that could be another option, switch to shared account instead of using our individual accounts | 21:52 |
| clarkb | fungi: ^curious if you have opinions on that | 21:52 |
| clarkb | in any case I don't expect everyone to go change how they use github right this instance. Mostly wanted to discuss it and see if we thought these sort of changes would be worthwhile | 21:54 |
| corvus | fungi: i abandoned my zuul-website patch because of your comment -- i think we need to revert more stuff. can you put that back on your todo list? | 21:57 |
| *** rcernin has quit IRC | 22:03 | |
| *** jcoufal has quit IRC | 22:06 | |
| *** jcoufal has joined #openstack-infra | 22:11 | |
| *** rcernin has joined #openstack-infra | 22:12 | |
| *** trown is now known as trown|outtypewww | 22:14 | |
| mordred | clarkb: oh my. so - I don't want to turn on 2fa for the github account I use for things | 22:16 |
| mordred | I should really set up an admin only account | 22:16 |
| clarkb | mordred: fwiw if you ssh (which is how I mostly use github) the number of times you have to enter a token is quite low | 22:16 |
| clarkb | but ya maybe the solution here is to enforce the second account rule and then 2fa there so no one notices | 22:17 |
| mordred | oh - I guess I have 2fa enabled | 22:17 |
| clarkb | ha | 22:17 |
| mordred | with text messages | 22:17 |
| mordred | so - I guess there's at least that | 22:17 |
| mordred | and I guess it's not a very high burden, as I'd forgotten it was enabled | 22:18 |
| corvus | clarkb, pabelanger: i'm spelunking into the history of our bind mount | 22:18 |
| clarkb | corvus: thanks! | 22:18 |
| pabelanger | kk | 22:18 |
| clarkb | mordred: ya I mean I've used it this way for about a year now and it isn't that bad. The worst part is setting it up and if you hardware token having that on your keychain everywhere and needing a safe place to store a backup | 22:19 |
| *** xek_ has quit IRC | 22:19 | |
| clarkb | using ssh to talk git reoves a lot of the pain I think | 22:19 |
| mordred | yah. that and not really using github for a ton of things :) | 22:19 |
| *** kgiusti has left #openstack-infra | 22:20 | |
| corvus | clarkb, pabelanger: this is why it's there: https://review.openstack.org/478274 | 22:21 |
| clarkb | hrm so we may not be able to just remove it then | 22:21 |
| corvus | yeah, we have 3 keys in that dir | 22:22 |
| clarkb | however we could limit it to mounting just that subset of keys? | 22:22 |
| *** tpsilva has quit IRC | 22:22 | |
| pabelanger | we should be able to move them into secrets now I think | 22:22 |
| corvus | pabelanger: yes... if we haven't already | 22:22 |
| clarkb | ah ya and that would give the added protection of requiring review before being accessed | 22:23 |
| corvus | i don't think we need to rotate those keys though, since it would only have been mounted for trusted playbooks | 22:23 |
| pabelanger | I agree with that | 22:23 |
| clarkb | corvus: ++ | 22:23 |
| clarkb | that also likely makes this a non urgent change but something that we should add to the todo list anyway | 22:24 |
| mordred | ++ | 22:24 |
| *** rh-jelabarre has quit IRC | 22:24 | |
| pabelanger | wfm | 22:24 |
| corvus | infra-root: is there a way for me to log into horizon on vexxhost? | 22:31 |
| corvus | i don't see likely credentials in the password file for https://secure.vexxhost.com/billing/clientarea.php | 22:31 |
| corvus | nor do i see any other urls | 22:31 |
| clarkb | corvus: mnaser set our accounts up by giving us keystone credentials which we then reset passwords on. I'm not sure we ever got web ui credentials. | 22:32 |
| clarkb | corvus: the catalog may point at a horizon url directly? /me looks | 22:33 |
| mordred | corvus: you can log in to horizon with the openstack creds | 22:33 |
| corvus | mordred: do you know the url for that? | 22:34 |
| mnaser | mordred: is correct. You can just use that username and psssword to login to horizon | 22:34 |
| mnaser | http://dashboard.vexxhost.net | 22:34 |
| mordred | https://dashboard-ca-ymq-1.vexxhost.net/ | 22:34 |
| mordred | yeah | 22:34 |
| mnaser | Or that. They’re aliases | 22:34 |
| clarkb | http://secure.vexxhost.com/console/api says the catalog | 22:34 |
| mnaser | (Region based dashboard is silly I’ve learned :)) | 22:34 |
| mordred | mnaser: :) | 22:35 |
| corvus | thanks. i've added that to our file :) | 22:35 |
| corvus | next question: is there a web ui for magnum? | 22:35 |
| clarkb | mnaser: fwiw the link in the catalog doesn't seem to point to what I would expect | 22:35 |
| corvus | oh there it is: "container infra" | 22:35 |
| clarkb | but maybe dashboard != horizon | 22:36 |
| *** diablo_rojo has quit IRC | 22:36 | |
| corvus | it very much looks like horizon | 22:36 |
| fungi | had a minor internet outage, catching back up now | 22:36 |
| clarkb | corvus: http://secure.vexxhost.com/console/api is different than the url that mnaser and mordred gave, but is what the catalog returns for "dashboard" | 22:36 |
| mordred | clarkb: well - I mean - there isn't really a defined use for dashboard in the catalog | 22:37 |
| clarkb | mordred: sure, but it is there | 22:37 |
| mordred | indeed | 22:37 |
| clarkb | and the url doesn't seem to give you dashboard is my point | 22:37 |
| *** diablo_rojo has joined #openstack-infra | 22:38 | |
| mnaser | clarkb: I don’t want to talk about that embarrassing catalog entry | 22:38 |
| clarkb | mnaser: ok :) mostly wanted to point it out if it was somethign that needed fixing | 22:38 |
| mnaser | It was in a time where we experimented with our own dashbord and using keystone for endpoint discovery | 22:38 |
| mnaser | Speaking of, mordred it would be interesting to have some sort of best practice document for using keystone for endpoint discovery for own services | 22:39 |
| mnaser | Prefix or what not to not collide with the OpenStack namespace | 22:39 |
| mordred | mnaser: I agree - that would be a very good document | 22:39 |
| *** jcoufal has quit IRC | 22:41 | |
| fungi | clarkb: my preference is to stop managing github and leave that to people in the community (perhaps not even infra-root folk) who care about it | 22:41 |
| clarkb | fungi: I agree, but also don't expect that transition to happen soon | 22:42 |
| clarkb | and if we can set up reasonable rules for how to do that then maybe those who take over will inherit some of that? | 22:42 |
| fungi | corvus: sure, i'll play around with partial/complete reverting of the change which added the events banner. we can always pop that in and out i suppose until the site design changes substantially | 22:42 |
| corvus | fungi, clarkb: one new wrinkle is if we want to continue having zuul do things on github, we need an app configured in some org on github. | 22:43 |
| corvus | so even if we're not mirring, if third-party-check is going to continue to be an opendev service (i think it should be), we'll still need at least that. | 22:43 |
| *** derekh has quit IRC | 22:43 | |
| *** derekh has joined #openstack-infra | 22:44 | |
| fungi | oh, good point | 22:44 |
| fungi | i guess the app doesn't have it's own account | 22:45 |
| *** slaweq has quit IRC | 22:46 | |
| clarkb | corvus ++ | 22:46 |
| mordred | ++ | 22:46 |
| *** ahosam has quit IRC | 22:51 | |
| *** trown|outtypewww has quit IRC | 22:54 | |
| *** slaweq has joined #openstack-infra | 22:54 | |
| *** trown has joined #openstack-infra | 22:55 | |
| *** slaweq has quit IRC | 22:59 | |
| *** jamesmcarthur has joined #openstack-infra | 22:59 | |
| *** wolverineav has quit IRC | 23:01 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: Add NetworkManager distro plugin support https://review.openstack.org/618964 | 23:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: A systemd skip for Debuntu systems https://review.openstack.org/620420 | 23:01 |
| *** wolverineav has joined #openstack-infra | 23:02 | |
| *** wolverineav has quit IRC | 23:03 | |
| *** wolverineav has joined #openstack-infra | 23:03 | |
| *** jamesmcarthur has quit IRC | 23:04 | |
| *** auristor has quit IRC | 23:05 | |
| *** auristor has joined #openstack-infra | 23:14 | |
| ianw | clarkb: ^ trying without defaultdependencies=no which i notice cloud-init only sets for debuntu | 23:17 |
| ianw | if anyone can tell me wtf "For example, target units will complement all configured dependencies of type Wants= or Requires= with dependencies of type After= unless DefaultDependencies=no is set in the specified units." means ... well good luck | 23:17 |
| *** auristor has quit IRC | 23:20 | |
| clarkb | ianw: wants means if you start foo then start bar, but doesn't mean bar happens after foo. The after will set up that additional relationship | 23:25 |
| clarkb | ianw: I think that defaultdependencies=no must mean we have to explicitly set the after ourselves if we want it? | 23:25 |
| clarkb | ianw: I think that must be a noop if you set both wants and after to the same value | 23:26 |
| *** auristor has joined #openstack-infra | 23:26 | |
| *** pbourke has quit IRC | 23:30 | |
| clarkb | ianw: do you know how you/we were tracking the rax dfw ipv6 errors? I was going to see if it is still persisting and if so start pondering work arounds | 23:32 |
| clarkb | (disable the cloud regions or switch to ipv4 in them) | 23:32 |
| ianw | clarkb: hrm, what's quite annoying is that in the logs (http://logs.openstack.org/71/618671/14/experimental/nodepool-functional-py35-redhat-src/facfefe/controller/logs/screen-nodepool-launcher.txt.gz#_Nov_28_06_15_23_694411) it really looks like glean starts first, sets things up and then networkmanager starts | 23:33 |
| ianw | but that host never responds on the network | 23:33 |
| *** rlandy is now known as rlandy|bbl | 23:33 | |
| clarkb | ianw: maybe nm + /e/n/i is broken on xenial? | 23:33 |
| clarkb | ianw: have you gotten it to work at all on that platform? | 23:33 |
| clarkb | oh nevermind its fedora | 23:34 |
| clarkb | the test node is xenial | 23:34 |
| ianw | yeah, this is just booting the centos test node | 23:34 |
| ianw | clarkb: no, sorry it was late and i was just poking at some very weird failures on my jobs ... let me see | 23:34 |
| ianw | frickler had something in scrollback? | 23:34 |
| ianw | https://portal.rackspace.com/506635/tickets/list | 23:35 |
| clarkb | ianw: ya they respodned and said they were looking at it and cloudnull took a look too. That said fungi said it could be an isp issue as he was having trouble to that part of the world too? | 23:35 |
| clarkb | ianw: I agree it seems glean runs first, then nm | 23:36 |
| clarkb | ianw: we don't know what /etc/sysconfig/network-scripts/ifcfg-ens3 looks like though right? | 23:36 |
| ianw | clarkb: well, the only thing that changed between working and not work was me copying the systemd dependencies from the cloud-init service file | 23:37 |
| clarkb | ah | 23:37 |
| clarkb | ianw: crazy idea, maybe the file isnt synced to disk? | 23:38 |
| clarkb | I actually ran into this with btrfs and zuuls test suite a few months ago | 23:38 |
| clarkb | in the db usage tests we fork a du and it wasn't seeing data that had been "written" | 23:38 |
| clarkb | maybe add an explicit sync/flush to glean | 23:39 |
| ianw | hrm, that was what i was thinking with https://review.openstack.org/#/c/585574/ | 23:39 |
| clarkb | I expect the older unit file would run glean quite a bit before nm whereas now its immediately after | 23:39 |
| ianw | i think the close implies it's been flushed | 23:40 |
| ianw | (assuming it's working ...) | 23:40 |
| clarkb | apparently it is for python3 | 23:40 |
| clarkb | explicitly so, but on python2 it was magically so | 23:40 |
| clarkb | ianw: I see logs for restoring selinux so I expect that code path is exercised | 23:41 |
| clarkb | ya I think that idea is a bad one. This should be safely flushed | 23:41 |
| ianw | i think pabelanger was also seeing some weirdness, perhaps with suse? that we could never pin down | 23:42 |
| corvus | mnaser: re magnum: clicking "Show Certificate" in horizon gives me a file with the string "undefined" as the only contents. however 'openstack coe cluster config ...' appears to work | 23:42 |
| *** wolverineav has quit IRC | 23:42 | |
| ianw | i actually took the .qcow2 dib built in one of these ci tests, moved it into my cloud and it booted just fine | 23:42 |
| clarkb | ianw: rereading the logs the starting glean for ens3 happens after the log output of glean.sh | 23:43 |
| clarkb | ianw: maybe the ordering isn't as strict as implied by the linear log file | 23:43 |
| *** wolverineav has joined #openstack-infra | 23:44 | |
| ianw | yeah, i could believe that | 23:45 |
| ianw | clarkb: back to the ipv6 issues, the file that was filling up with bad connections was mirror02.dfw.rax.openstack.org_error.log and it looks like it's stopped | 23:46 |
| *** pbourke has joined #openstack-infra | 23:46 | |
| clarkb | ianw: thank you for checking on that. In that case I won't figure out workarounds right now | 23:46 |
| clarkb | ianw: fwiw type oneshot units mean process must exit before follow on units are started. networkmanager.service is set up to happen after glean-nm so that should all work. I'm stumped | 23:47 |
| ianw | the last ipv6 error was @ 18:19:47.705761 and it's now 23:47:16 UTC 2018 | 23:47 |
| ianw | clarkb: well, it's nice to not be alone in the stumped club :) | 23:48 |
| clarkb | ianw: what if its the multiple before listings | 23:51 |
| *** wolverineav has quit IRC | 23:51 | |
| clarkb | ianw: the docs say it should be comman separated. any chance it is overridding with the second value and not appending? | 23:51 |
| clarkb | *comma separated | 23:51 |
| clarkb | oh sorry space separated | 23:52 |
| ianw | hrm, then you'd think someone would have noticed with https://git.launchpad.net/cloud-init/tree/systemd/cloud-init-local.service.tmpl ? | 23:53 |
| clarkb | "This option may be specified more than once, in which case ordering dependencies for all listed names are created." | 23:54 |
| clarkb | ok probably not that | 23:54 |
| *** slaweq has joined #openstack-infra | 23:54 | |
| clarkb | systemd-analyze dot may be useful here? | 23:56 |
| clarkb | ianw: is that qcow2 available somewhere? | 23:57 |
| ianw | clarkb: the one i have is from when f29 wouldn't boot, but it's the same symptoms | 23:58 |
| ianw | the problem is it *does* boot correctly in my cloud, but never did in the gate | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!