| *** agopi has joined #openstack-infra | 00:00 | |
| *** jamesmcarthur has quit IRC | 00:01 | |
| *** jcoufal has quit IRC | 00:03 | |
| *** hwoarang has quit IRC | 00:04 | |
| *** wolverineav has quit IRC | 00:05 | |
| *** hwoarang has joined #openstack-infra | 00:05 | |
| *** wolverineav has joined #openstack-infra | 00:06 | |
| *** wolverineav has quit IRC | 00:11 | |
| *** agopi has quit IRC | 00:12 | |
| *** mattw4 has joined #openstack-infra | 00:16 | |
| *** markvoelker has joined #openstack-infra | 00:19 | |
| *** tosky has quit IRC | 00:19 | |
| *** wolverineav has joined #openstack-infra | 00:37 | |
| *** hrubi has quit IRC | 00:38 | |
| *** wolverineav has quit IRC | 00:42 | |
| *** rascasoft has joined #openstack-infra | 00:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 00:47 |
|---|---|---|
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 00:47 |
| *** rascasoft has quit IRC | 00:50 | |
| *** mattw4 has quit IRC | 00:50 | |
| *** slaweq has quit IRC | 00:52 | |
| paladox | corvus did you see https://i.imgur.com/l0y7OVw.png ? (zuul status in gerrit's ui) :) | 00:57 |
| corvus | paladox: no -- is that using the verify status plugin? | 01:12 |
| paladox | Nope, that's using the zuul endpoint | 01:12 |
| *** whoami-rajat has joined #openstack-infra | 01:12 | |
| corvus | paladox: so javascript added to gerrit to query zuul api? | 01:13 |
| paladox | yeh, it's a PolyGerrit plugin | 01:13 |
| paladox | I built one for the wmf. | 01:13 |
| corvus | ah, neat | 01:13 |
| paladox | corvus His one was much better then mine :) (so i updated it to match his one) | 01:14 |
| paladox | https://gerrit-review.googlesource.com/admin/repos/plugins/zuul-status | 01:14 |
| paladox | that's the one i built for the wmf ^^ :) | 01:14 |
| paladox | that supports zuul v2 (and v3) | 01:15 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Add documentation for private stories https://review.openstack.org/636235 | 01:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 01:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 01:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: use-buildset-registry: configure as a pull-through proxy https://review.openstack.org/638312 | 01:18 |
| corvus | fungi, AJaeger, mordred, clarkb: https://review.openstack.org/638312 needs to merge before i can proceed with further registry testing | 01:20 |
| *** logan- has quit IRC | 01:22 | |
| paladox | corvus this is what it looks like https://phabricator.wikimedia.org/F28247287 :) | 01:22 |
| *** logan- has joined #openstack-infra | 01:23 | |
| *** wolverineav has joined #openstack-infra | 01:24 | |
| *** agopi has joined #openstack-infra | 01:26 | |
| *** wolverineav has quit IRC | 01:28 | |
| *** logan- has quit IRC | 01:31 | |
| *** logan- has joined #openstack-infra | 01:31 | |
| *** mriedem has quit IRC | 01:35 | |
| *** rascasoft has joined #openstack-infra | 01:36 | |
| *** rascasoft has quit IRC | 01:42 | |
| *** sdake has quit IRC | 01:44 | |
| *** rh-jelabarre has quit IRC | 01:44 | |
| *** hwoarang has quit IRC | 01:47 | |
| *** sdake has joined #openstack-infra | 01:47 | |
| *** hwoarang has joined #openstack-infra | 01:48 | |
| *** wolverineav has joined #openstack-infra | 01:50 | |
| *** hwoarang has quit IRC | 01:55 | |
| *** hwoarang has joined #openstack-infra | 01:56 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Imported Translations from Zanata https://review.openstack.org/625393 | 02:04 |
| *** anteaya has quit IRC | 02:04 | |
| *** gyee has quit IRC | 02:06 | |
| *** jamesmcarthur has joined #openstack-infra | 02:10 | |
| *** wolverineav has quit IRC | 02:20 | |
| *** wolverineav has joined #openstack-infra | 02:21 | |
| *** wolverineav has quit IRC | 02:22 | |
| *** wolverineav has joined #openstack-infra | 02:26 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Use python3 for basepython in tox.ini https://review.openstack.org/633352 | 02:30 |
| openstackgerrit | Merged openstack-infra/storyboard master: Remove upper limit on PyMySQL version https://review.openstack.org/633355 | 02:30 |
| *** wolverineav has quit IRC | 02:31 | |
| *** jamesmcarthur has quit IRC | 02:37 | |
| *** jamesmcarthur has joined #openstack-infra | 02:42 | |
| *** efried_pto is now known as efried | 02:43 | |
| *** hongbin has joined #openstack-infra | 02:48 | |
| *** sdake has quit IRC | 02:48 | |
| *** sdake has joined #openstack-infra | 02:50 | |
| *** psachin has joined #openstack-infra | 02:52 | |
| *** sdake has quit IRC | 02:57 | |
| *** sdake has joined #openstack-infra | 02:59 | |
| *** dklyle has quit IRC | 03:03 | |
| *** wolverineav has joined #openstack-infra | 03:06 | |
| *** jamesmcarthur has quit IRC | 03:08 | |
| *** wolverineav has quit IRC | 03:10 | |
| *** apetrich has quit IRC | 03:14 | |
| *** roman_g has quit IRC | 03:16 | |
| *** dklyle has joined #openstack-infra | 03:17 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: add support for report only connection https://review.openstack.org/568216 | 03:18 |
| *** markvoelker has quit IRC | 03:20 | |
| *** jamesmcarthur has joined #openstack-infra | 03:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildsets page https://review.openstack.org/630041 | 03:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 03:32 |
| *** spsurya has joined #openstack-infra | 03:34 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add nodepool-dib dashboard https://review.openstack.org/638325 | 03:36 |
| *** ramishra has joined #openstack-infra | 03:43 | |
| *** wolverineav has joined #openstack-infra | 03:48 | |
| *** udesale has joined #openstack-infra | 03:51 | |
| *** jamesmcarthur has quit IRC | 04:00 | |
| *** jamesmcarthur has joined #openstack-infra | 04:01 | |
| *** jamesmcarthur has quit IRC | 04:05 | |
| *** rascasoft has joined #openstack-infra | 04:06 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: add support for report only connection https://review.openstack.org/568216 | 04:08 |
| *** hwoarang has quit IRC | 04:12 | |
| *** hwoarang has joined #openstack-infra | 04:14 | |
| *** rascasoft has quit IRC | 04:18 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 04:19 |
| *** markvoelker has joined #openstack-infra | 04:21 | |
| *** wolverineav has quit IRC | 04:29 | |
| *** gagehugo has joined #openstack-infra | 04:29 | |
| *** ianychoi has quit IRC | 04:31 | |
| *** jbadiapa has quit IRC | 04:38 | |
| *** jbadiapa has joined #openstack-infra | 04:38 | |
| *** sdake has quit IRC | 04:50 | |
| *** sdake has joined #openstack-infra | 04:54 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 04:54 |
| *** markvoelker has quit IRC | 04:55 | |
| *** hongbin has quit IRC | 05:04 | |
| *** diablo_rojo has quit IRC | 05:05 | |
| *** Dobroslaw has quit IRC | 05:07 | |
| *** emccormick has quit IRC | 05:07 | |
| *** jtomasek has joined #openstack-infra | 05:09 | |
| *** sdake has quit IRC | 05:14 | |
| *** sdake has joined #openstack-infra | 05:16 | |
| *** hwoarang has quit IRC | 05:16 | |
| *** hwoarang has joined #openstack-infra | 05:18 | |
| kota_ | hello, can I have a way to pin a package version lower than openstack upper constraint in requirements.txt? | 05:22 |
| kota_ | when installing the package/dependencies with `-r requirements.txt -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt`, the version pinned at the upper constraint even we have `package == specific version` in requirments.txt | 05:23 |
| kota_ | the specific version is lower than upper constraints so it should be pinned at the lower one, is it correct? | 05:24 |
| *** hwoarang has quit IRC | 05:39 | |
| *** hwoarang has joined #openstack-infra | 05:41 | |
| *** yamamoto has quit IRC | 05:44 | |
| *** yamamoto has joined #openstack-infra | 05:48 | |
| *** fdegir has quit IRC | 05:48 | |
| *** sdake has quit IRC | 05:48 | |
| *** fdegir has joined #openstack-infra | 05:48 | |
| *** agopi has quit IRC | 05:49 | |
| *** sdake has joined #openstack-infra | 05:49 | |
| *** markvoelker has joined #openstack-infra | 05:52 | |
| *** wolverineav has joined #openstack-infra | 05:55 | |
| *** wolverineav has quit IRC | 05:59 | |
| *** kmrchdn is now known as chandankumar | 06:01 | |
| *** sdake has quit IRC | 06:03 | |
| *** sdake has joined #openstack-infra | 06:05 | |
| *** auristor has quit IRC | 06:07 | |
| *** auristor has joined #openstack-infra | 06:10 | |
| *** kjackal has joined #openstack-infra | 06:15 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add python-path option to label https://review.openstack.org/637338 | 06:22 |
| *** markvoelker has quit IRC | 06:25 | |
| *** spsurya has quit IRC | 06:25 | |
| *** sdake has quit IRC | 06:37 | |
| *** stakeda has joined #openstack-infra | 06:46 | |
| *** quiquell|off is now known as quiquell | 06:50 | |
| *** rkukura_ has joined #openstack-infra | 06:50 | |
| *** rkukura has quit IRC | 06:54 | |
| *** rkukura_ is now known as rkukura | 06:54 | |
| *** yamamoto has quit IRC | 06:59 | |
| *** hrubi has joined #openstack-infra | 07:02 | |
| *** rascasoft has joined #openstack-infra | 07:05 | |
| *** slaweq has joined #openstack-infra | 07:08 | |
| *** rascasoft has quit IRC | 07:09 | |
| *** psachin has quit IRC | 07:10 | |
| *** egonzalez has quit IRC | 07:10 | |
| *** egonzalez has joined #openstack-infra | 07:11 | |
| *** rcernin has quit IRC | 07:13 | |
| *** aojea has joined #openstack-infra | 07:14 | |
| *** spsurya has joined #openstack-infra | 07:17 | |
| *** wolverineav has joined #openstack-infra | 07:18 | |
| *** kopecmartin|off is now known as kopecmartin | 07:19 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:19 | |
| *** pcaruana has joined #openstack-infra | 07:19 | |
| *** markvoelker has joined #openstack-infra | 07:22 | |
| *** Dobroslaw has joined #openstack-infra | 07:29 | |
| *** quiquell is now known as quiquiell|brb | 07:30 | |
| *** psachin has joined #openstack-infra | 07:32 | |
| *** apetrich has joined #openstack-infra | 07:33 | |
| *** yamamoto has joined #openstack-infra | 07:38 | |
| *** rlandy|bbl is now known as rlandy | 07:44 | |
| *** wolverineav has quit IRC | 07:52 | |
| *** markvoelker has quit IRC | 07:55 | |
| *** bhavikdbavishi has quit IRC | 07:56 | |
| *** quiquiell|brb is now known as quiquell | 07:59 | |
| *** rascasoft has joined #openstack-infra | 08:05 | |
| *** jaosorior has quit IRC | 08:08 | |
| *** ccamacho has quit IRC | 08:18 | |
| *** tkajinam has quit IRC | 08:23 | |
| *** jpich has joined #openstack-infra | 08:23 | |
| *** ginopc has joined #openstack-infra | 08:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add nodepool-dib dashboard https://review.openstack.org/638325 | 08:26 |
| *** tosky has joined #openstack-infra | 08:26 | |
| *** wolverineav has joined #openstack-infra | 08:28 | |
| *** jpena|off is now known as jpena | 08:29 | |
| openstackgerrit | Merged openstack-infra/grafyaml master: Add valueMaps and rangeMaps support https://review.openstack.org/638288 | 08:40 |
| *** markvoelker has joined #openstack-infra | 08:52 | |
| *** dtantsur|afk is now known as dtantsur\ | 08:53 | |
| *** dtantsur\ is now known as dtantsur | 08:53 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** jento has quit IRC | 08:59 | |
| *** gfidente has joined #openstack-infra | 08:59 | |
| *** wolverineav has quit IRC | 09:00 | |
| *** jento has joined #openstack-infra | 09:01 | |
| *** yamamoto has joined #openstack-infra | 09:04 | |
| *** ociuhandu has quit IRC | 09:04 | |
| *** electrofelix has joined #openstack-infra | 09:08 | |
| *** ccamacho has joined #openstack-infra | 09:09 | |
| *** roman_g has joined #openstack-infra | 09:10 | |
| *** ccamacho has quit IRC | 09:10 | |
| *** ccamacho has joined #openstack-infra | 09:11 | |
| *** janki has joined #openstack-infra | 09:12 | |
| *** e0ne has joined #openstack-infra | 09:14 | |
| *** aojea has quit IRC | 09:14 | |
| *** ociuhandu has joined #openstack-infra | 09:24 | |
| *** markvoelker has quit IRC | 09:25 | |
| *** ociuhandu has quit IRC | 09:25 | |
| *** ociuhandu has joined #openstack-infra | 09:26 | |
| *** derekh has joined #openstack-infra | 09:31 | |
| *** panda|off is now known as panda | 09:54 | |
| *** wolverineav has joined #openstack-infra | 09:57 | |
| *** jaosorior has joined #openstack-infra | 09:57 | |
| *** yamamoto has quit IRC | 09:58 | |
| *** iurygregory has quit IRC | 10:01 | |
| *** markvoelker has joined #openstack-infra | 10:17 | |
| *** spsurya has quit IRC | 10:22 | |
| stephenfin | coreycb: Quick question: why are there two different types of py37 patch, "Change python3.5 job to python3.7 job on Stein+" and "add python 3.7 unit test job", and do we plan to switch the latter to the former? | 10:27 |
| *** wolverineav has quit IRC | 10:30 | |
| *** yamamoto has joined #openstack-infra | 10:34 | |
| *** ChosSimbaOne_Lap has quit IRC | 10:34 | |
| *** kjackal has quit IRC | 10:38 | |
| *** psachin has quit IRC | 10:38 | |
| *** kjackal has joined #openstack-infra | 10:38 | |
| *** yamamoto has quit IRC | 10:39 | |
| *** luizbag has joined #openstack-infra | 10:41 | |
| *** ChosSimbaOne_Lap has joined #openstack-infra | 10:41 | |
| *** psachin has joined #openstack-infra | 10:41 | |
| *** ChosSimbaOne_Lap has quit IRC | 10:45 | |
| *** stakeda has quit IRC | 10:49 | |
| *** jaosorior has quit IRC | 10:55 | |
| quiquell | hello | 10:56 |
| quiquell | when I ping review.openstack.org it rediects me to review01.openstack.org is this all good ? | 10:56 |
| *** lpetrut has joined #openstack-infra | 11:03 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:10 | |
| Shrews | quiquell: yes. that's a common pattern we follow so that we can upgrade/replace instances more easily | 11:12 |
| quiquell | Shrews: thanks I was following a red herring, it's all ok | 11:13 |
| *** jaosorior has joined #openstack-infra | 11:15 | |
| *** psachin has quit IRC | 11:22 | |
| *** udesale has quit IRC | 11:26 | |
| *** wolverineav has joined #openstack-infra | 11:27 | |
| *** yamamoto has joined #openstack-infra | 11:29 | |
| *** yamamoto has quit IRC | 11:34 | |
| geguileo | dhellmann: hi, do we need tonyb's +1 or is it enough with mriedem's? https://review.openstack.org/#/c/637614/ | 11:43 |
| *** ldnunes has joined #openstack-infra | 11:46 | |
| *** FlorianFa has joined #openstack-infra | 11:47 | |
| *** FlorianFa has quit IRC | 11:51 | |
| *** FlorianFa has joined #openstack-infra | 11:54 | |
| *** FlorianFa has quit IRC | 11:56 | |
| *** jpena is now known as jpena|lunch | 11:56 | |
| *** FlorianFa has joined #openstack-infra | 11:56 | |
| *** bhavikdbavishi has quit IRC | 11:59 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:59 | |
| *** wolverineav has quit IRC | 12:00 | |
| *** sdake has joined #openstack-infra | 12:02 | |
| *** quiquell is now known as quiquell|lunch | 12:05 | |
| *** dayou has quit IRC | 12:19 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 12:20 | |
| *** bhavikdbavishi has quit IRC | 12:20 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 12:20 | |
| *** dayou has joined #openstack-infra | 12:23 | |
| *** e0ne has quit IRC | 12:24 | |
| *** yamamoto has joined #openstack-infra | 12:28 | |
| *** e0ne has joined #openstack-infra | 12:31 | |
| *** yamamoto has quit IRC | 12:34 | |
| *** kgiusti has quit IRC | 12:37 | |
| *** kaiokmo has quit IRC | 12:44 | |
| *** jesusaur has quit IRC | 12:50 | |
| coreycb | stephenfin: the original approach to adding the py37 unit tests was "Change python3.5 job to python3.7 job on Stein+" and that received a lot of pushback | 12:54 |
| coreycb | stephenfin: i think dropping py35 has since been approved but that's not my focus anyway so i figured the patches should just focus on py37. | 12:55 |
| *** jesusaur has joined #openstack-infra | 12:56 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Assure iptables is installed inside multi-node-firewall role https://review.openstack.org/638414 | 12:58 |
| *** wolverineav has joined #openstack-infra | 12:58 | |
| coreycb | so the only ones that should land are titled "add python 3.7 unit test job". i've made an effort to override the old subject patches with the new one (using the same change-id), however I haven't heard back from Filippo Inzaghi about his abandoned patches which i can't override so i've submitted new ones for those. | 12:59 |
| coreycb | stephenfin: ^ | 12:59 |
| aspiers | is it just me or has the post job for https://review.openstack.org/#/c/638396/ *not* run, plus it's not in the zuul queue? | 12:59 |
| stephenfin | coreycb: Ahh, fair enough. I'll got back and re-review, in that case | 12:59 |
| coreycb | stephenfin: thanks! always good when someone reviews things and doesn't give a blind +1 :) | 13:00 |
| *** kaiokmo has joined #openstack-infra | 13:00 | |
| AJaeger | aspiers: check http://zuul.openstack.org/builds | 13:01 |
| aspiers | AJaeger: not there either | 13:02 |
| *** yamamoto has joined #openstack-infra | 13:03 | |
| dhellmann | geguileo : Matt's is enough for that one | 13:03 |
| *** trown|outtypewww is now known as trown | 13:06 | |
| *** rh-jelabarre has joined #openstack-infra | 13:06 | |
| aspiers | AJaeger: definitely looks to me like something went wrong | 13:08 |
| aspiers | the job should either be in the queue, or running, or done - right? | 13:08 |
| aspiers | no other cracks for it to fall down | 13:08 |
| *** yamamoto has quit IRC | 13:08 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Assure iptables is installed inside multi-node-firewall role https://review.openstack.org/638414 | 13:08 |
| *** quiquell|lunch is now known as quiquell | 13:10 | |
| *** jcoufal has joined #openstack-infra | 13:13 | |
| *** jpena|lunch is now known as jpena | 13:15 | |
| *** markvoelker has quit IRC | 13:17 | |
| *** markvoelker has joined #openstack-infra | 13:18 | |
| AJaeger | aspiers: yeah, should be... | 13:20 |
| AJaeger | aspiers: check again: http://zuul.openstack.org/builds?project=openstack%2Fopenstack-manuals&pipeline=post | 13:21 |
| AJaeger | aspiers: which points to http://zuul.openstack.org/build/ac6653406146418099af9c9e652ff0df | 13:21 |
| AJaeger | So, that one run | 13:21 |
| *** markvoelker has quit IRC | 13:22 | |
| *** sdake has quit IRC | 13:23 | |
| *** sdake has joined #openstack-infra | 13:24 | |
| aspiers | Oh OK, so it's not a substring search AND I missed that you have to switch to filter by project | 13:24 |
| geguileo | dhellmann: thanks | 13:24 |
| aspiers | nor does it give you access to the job logs | 13:26 |
| aspiers | and the publish job is still not shown at https://review.openstack.org/#/c/638396/ | 13:26 |
| aspiers | but apparently it finally ran | 13:26 |
| aspiers | I'm not sure why it's a black box though | 13:27 |
| AJaeger | aspiers: it gives access to logs, see second link | 13:27 |
| AJaeger | click on the "SUCCESS" for the log file | 13:27 |
| aspiers | I tried that, nothing happened | 13:27 |
| AJaeger | aspiers: the publish job will never be shown at 638396 | 13:27 |
| aspiers | really? I could have sworn I saw it previously | 13:28 |
| AJaeger | aspiers: http://logs.openstack.org/20/206a92cf3593340295692a78a97463fb6291072e/post/publish-openstack-manuals-lang/ac66534/ | 13:28 |
| aspiers | OK, somehow that page hung in my browser | 13:28 |
| AJaeger | to get there: Go to http://zuul.openstack.org/build/ac6653406146418099af9c9e652ff0df and then click on log URL | 13:28 |
| aspiers | after reloading it now works | 13:28 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: use-buildset-registry: configure as a pull-through proxy https://review.openstack.org/638312 | 13:28 |
| aspiers | Murphy is doing everything possible today :-/ | 13:28 |
| AJaeger | ah, that was the lang job - you want another one... | 13:28 |
| AJaeger | aspiers: in a meeting - can't help further - hope this bit helps... | 13:29 |
| aspiers | AJaeger: it does indeed, many thanks! | 13:29 |
| aspiers | although I'm still puzzled why it didn't show up in zuul's post queue | 13:30 |
| *** wolverineav has quit IRC | 13:30 | |
| *** bhavikdbavishi has quit IRC | 13:32 | |
| *** mriedem has joined #openstack-infra | 13:34 | |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| AJaeger | aspiers: did you have some filter running on zuul.openstack.org ? E.g filter for 638396 ? | 13:36 |
| *** sdake has quit IRC | 13:36 | |
| *** yamamoto has quit IRC | 13:36 | |
| *** sdake has joined #openstack-infra | 13:38 | |
| aspiers | I searched with and without a filter | 13:38 |
| *** rlandy has joined #openstack-infra | 13:40 | |
| AJaeger | aspiers: with filter you would not see it since post jobs do not have the 638396 but the sha1 of the change! | 13:42 |
| AJaeger | so, you need to search for sha1 or for openstack-manuals for post jobs | 13:42 |
| AJaeger | bbl | 13:43 |
| *** jamesmcarthur has joined #openstack-infra | 13:43 | |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| *** kgiusti has joined #openstack-infra | 13:47 | |
| *** kgiusti has quit IRC | 13:52 | |
| *** jamesmcarthur has quit IRC | 13:58 | |
| *** jamesmcarthur has joined #openstack-infra | 13:58 | |
| *** jamesmcarthur has quit IRC | 14:03 | |
| *** kgiusti has joined #openstack-infra | 14:08 | |
| *** jaosorior has quit IRC | 14:08 | |
| fungi | yeah, and unless the change could be pushed down as a fast-forward, the sha you'd need to look for would be the merge commit gerrit creates when zuul calls the submit api after the gate jobs succeed | 14:10 |
| fungi | easier to go to the builds table and filter there for the job name (and maybe also project and pipeline depending on how much that same job is reused in other places) | 14:12 |
| *** yamamoto has quit IRC | 14:13 | |
| *** sdake has quit IRC | 14:14 | |
| *** yamamoto has joined #openstack-infra | 14:14 | |
| *** yamamoto has quit IRC | 14:14 | |
| *** yamamoto has joined #openstack-infra | 14:15 | |
| *** panda is now known as panda|ruck | 14:16 | |
| *** markvoelker has joined #openstack-infra | 14:18 | |
| *** yamamoto has quit IRC | 14:19 | |
| *** udesale has joined #openstack-infra | 14:22 | |
| *** eharney has joined #openstack-infra | 14:25 | |
| *** wolverineav has joined #openstack-infra | 14:28 | |
| *** jamesmcarthur has joined #openstack-infra | 14:29 | |
| *** jaosorior has joined #openstack-infra | 14:30 | |
| *** sdake has joined #openstack-infra | 14:31 | |
| *** jamesmcarthur has quit IRC | 14:34 | |
| *** ramishra has quit IRC | 14:40 | |
| *** dave-mccowan has joined #openstack-infra | 14:47 | |
| *** sdake has quit IRC | 14:51 | |
| *** markvoelker has quit IRC | 14:53 | |
| *** ekultails has joined #openstack-infra | 14:53 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated Filter available summit class https://review.openstack.org/638430 | 14:56 |
| *** yamamoto has joined #openstack-infra | 14:57 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated Filter available summit class https://review.openstack.org/638430 | 14:57 |
| *** sdake has joined #openstack-infra | 15:01 | |
| *** wolverineav has quit IRC | 15:01 | |
| *** yamamoto has quit IRC | 15:01 | |
| *** anteaya has joined #openstack-infra | 15:04 | |
| *** jaosorior has quit IRC | 15:08 | |
| *** anteaya has quit IRC | 15:09 | |
| *** jamesmcarthur has joined #openstack-infra | 15:10 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Revert "Update API code to work with Presentation Moderators collection (+N)" https://review.openstack.org/638436 | 15:11 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Revert "Speakers/Moderators Refactoring" https://review.openstack.org/638437 | 15:11 |
| *** gfidente has quit IRC | 15:14 | |
| *** jamesmcarthur has quit IRC | 15:17 | |
| *** anteaya has joined #openstack-infra | 15:19 | |
| *** dave-mccowan has quit IRC | 15:21 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Revert "Update API code to work with Presentation Moderators collection (+N)" https://review.openstack.org/638436 | 15:22 |
| *** jistr is now known as jistr|afk | 15:22 | |
| *** markmcd has joined #openstack-infra | 15:22 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Revert "Update API code to work with Presentation Moderators collection (+N)" https://review.openstack.org/638436 | 15:23 |
| *** markmcd has left #openstack-infra | 15:24 | |
| dmsimard | infra-root: nb03 is out of disk space on the /opt partition | 15:25 |
| fungi | dmsimard: thanks for the heads up | 15:26 |
| pabelanger | dmsimard: what is the size of gentoo cache? That caused the last HDD space issues last time on other builder | 15:26 |
| fungi | dmsimard: are you working on cleaning it up? if not, i'll dig into that after i get done sorting out the config change we need to fix e-mail notifications for storyboard | 15:26 |
| fungi | (which will be after i see the light at the end of the current meeting tunnel) | 15:27 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Revert "Speakers/Moderators Refactoring" https://review.openstack.org/638437 | 15:27 |
| dmsimard | fungi: I found it "by accident" when looking for another problem -- I can get to it eventually but need to take care of the first issue before rabbitholing :) | 15:28 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Revert "Speakers/Moderators Refactoring" https://review.openstack.org/638437 | 15:29 |
| fungi | dmsimard: got it, thanks. sounds familiar ;) | 15:30 |
| *** ramishra has joined #openstack-infra | 15:30 | |
| *** jistr|afk is now known as jistr | 15:32 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed merge conflict https://review.openstack.org/638441 | 15:34 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed merge conflict https://review.openstack.org/638441 | 15:34 |
| *** kjackal has quit IRC | 15:35 | |
| *** apetrich has quit IRC | 15:36 | |
| *** apetrich has joined #openstack-infra | 15:41 | |
| *** dave-mccowan has joined #openstack-infra | 15:41 | |
| *** sc has joined #openstack-infra | 15:43 | |
| *** dave-mccowan has quit IRC | 15:45 | |
| *** jaosorior has joined #openstack-infra | 15:46 | |
| chandankumar | Hey Infra! | 15:47 |
| *** sdake has quit IRC | 15:48 | |
| chandankumar | Why stackviz has two tarballs published on tarballs.openstack.org? | 15:48 |
| chandankumar | http://tarballs.openstack.org/stackviz/dist/ and http://tarballs.openstack.org/package-stackviz-element/ | 15:48 |
| *** jamesmcarthur has joined #openstack-infra | 15:48 | |
| *** markvoelker has joined #openstack-infra | 15:50 | |
| *** sdake has joined #openstack-infra | 15:50 | |
| *** janki has quit IRC | 15:53 | |
| *** jamesmcarthur has quit IRC | 15:53 | |
| *** tosky has quit IRC | 15:53 | |
| *** tosky has joined #openstack-infra | 15:54 | |
| *** jamesmcarthur has joined #openstack-infra | 15:58 | |
| *** jamesmcarthur has quit IRC | 15:58 | |
| *** jamesmcarthur has joined #openstack-infra | 15:58 | |
| *** wolverineav has joined #openstack-infra | 15:58 | |
| *** diablo_rojo has joined #openstack-infra | 16:00 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add site name to Exim local_domains on SB server https://review.openstack.org/638448 | 16:01 |
| fungi | infra-root: ^ brown bag fix for e-mail notification on storyboard.o.o | 16:02 |
| dmsimard | fungi: ok, looking at nb03. | 16:02 |
| dmsimard | may I stop nodepool-builder and put nb03 in the emergency file so it doesn't get in my way ? | 16:04 |
| *** ekultails has quit IRC | 16:04 | |
| fungi | dmsimard: i say go for it | 16:04 |
| fungi | all that's going to do, at worst, is not build more images | 16:04 |
| fungi | which it probably isn't doing anyway | 16:04 |
| dmsimard | yeah, just validating :) | 16:04 |
| *** ramishra has quit IRC | 16:05 | |
| *** NCLanceman has joined #openstack-infra | 16:06 | |
| *** kjackal has joined #openstack-infra | 16:06 | |
| clarkb | pabelanger: the gentoo cache was not the cause of the disk problems | 16:07 |
| *** rfolco is now known as rfolco|rover | 16:07 | |
| clarkb | pabelanger: it just looked that way because of lvm disk accounting | 16:07 |
| dmsimard | I was discussing an odd behavior of nodepool with pabelanger in another channel | 16:07 |
| dmsimard | and noticed it upstream as well | 16:07 |
| clarkb | the actual problem was not being able to upload images reliably to some clouds so that odd images were sticking around (effectively we doubled the number of images we kept) as well as some leaked builds | 16:08 |
| dmsimard | should we ever only build and upload raw images ? we have some qcow, raw and even vhd images | 16:08 |
| clarkb | dmsimard: different clouds need different formats so that is normal on our nb01 and nb02 builders. I do not know if we should be doing that for the arm64 clouds (nb03 is the arm64 dedicated builder) | 16:10 |
| dmsimard | yeah nb03 is all qcow -- I saw the vhd's in nb01 | 16:11 |
| *** sthussey has joined #openstack-infra | 16:11 | |
| *** udesale has quit IRC | 16:11 | |
| pabelanger | rackspace needs vhds | 16:11 |
| pabelanger | vhd* | 16:11 |
| dmsimard | I was trying to correlate with an issue we're seeing in RDO's nodepool -- it's building both qcow2 and raw while we specify "formats: raw" | 16:12 |
| *** quiquell is now known as quiquell|off | 16:13 | |
| *** jamesmcarthur has quit IRC | 16:13 | |
| *** NCLanceman has quit IRC | 16:13 | |
| clarkb | dmsimard: newer nodepool uses the cloud.yaml data to determine format. Possible that is set to qcow2 and overriding? | 16:13 |
| pabelanger | dmsimard: do you have more then 1 provider for the image that be being used? | 16:13 |
| *** hamzy has quit IRC | 16:15 | |
| dmsimard | clarkb: nothing specific in clouds.yaml and it's a cloud that isn't mapped in openstacksdk | 16:15 |
| dmsimard | pabelanger: not as far as I can tell | 16:16 |
| dmsimard | anyway, I'll take care of nb03 and I'll go back to that after -- it's two different things :p | 16:16 |
| *** ekultails has joined #openstack-infra | 16:19 | |
| clarkb | dmsimard: looks like most of it is leaked images again in /opt/nodepool_dib | 16:21 |
| clarkb | dmsimard: when I ran into that before I do nodepool image listings to find which images are still valid then deleted those that werent | 16:21 |
| clarkb | I wonder if zk connections issues might explain that (possibly worth looking in the logs for evidence of that) | 16:22 |
| corvus | clarkb: can you remind me which is the current docker v2 mirror address? | 16:22 |
| corvus | is that 8082? | 16:22 |
| dmsimard | clarkb: nodepool@nb03:~$ 2019-02-21 16:19:44,912 WARNING kazoo.client: Connection dropped: socket connection error: Network is unreachable | 16:22 |
| dmsimard | that was when doing a "nodepool dib-image-list" | 16:22 |
| *** e0ne has quit IRC | 16:23 | |
| *** markvoelker has quit IRC | 16:23 | |
| clarkb | corvus: yes the higher numbered port. 8081 is the old protocol iirc | 16:23 |
| clarkb | though it might be 8080 and 8081 /me checks apache conf | 16:24 |
| clarkb | corvus: 8081 is old 8082 is new | 16:24 |
| clarkb | dmsimard: I think that is "normal" | 16:24 |
| corvus | clarkb: how sure are we that it's working? this is what i get with 'docker pull alpine' -- http://paste.openstack.org/show/745628/ | 16:26 |
| dmsimard | clarkb: do you have a script or a one liner to easily compare the image list against the image directory ? | 16:26 |
| clarkb | dmsimard: no I did it by hand beacuse I was trying to understand what might have caushed it (so was debugging concurrently) | 16:27 |
| clarkb | unfortunately I didn't find anything | 16:27 |
| dmsimard | ok | 16:27 |
| clarkb | corvus: hrm maybe dockerhub moved their backend data again | 16:27 |
| *** jamesmcarthur has joined #openstack-infra | 16:27 | |
| clarkb | corvus: the cachign there is weird because dockerhub redirects you to a cdn/object store and that has changed in the past | 16:28 |
| clarkb | wow and you can't do anonymous fetches? | 16:30 |
| *** defionscode has quit IRC | 16:30 | |
| *** wolverineav has quit IRC | 16:31 | |
| clarkb | corvus: `https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/ca/caf27325b298a6730837023a8a342699c8b7b388b8d878966b064a1320043019/data?verify=1550769259-lsxHynSfq%2Bt7BBVW3nPZCE8Q60Q%3D` works from here | 16:32 |
| corvus | clarkb: i'm attempting to reverse the rewrite for the first 404 there, and i come up with " https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/ca/caf27325b298a6730837023a8a342699c8b7b388b8d878966b064a1320043019/data?verify=1550769259-lsxHynSfq%2Bt7BBVW3nPZCE8Q60Q%3D" which seems to work.... | 16:32 |
| clarkb | ya | 16:32 |
| clarkb | corvus: possibly hitting a bad cdn node from $cloud? | 16:33 |
| *** kopecmartin is now known as kopecmartin|off | 16:33 | |
| corvus | clarkb: it works from mirror.iad.rax.openstack.org too | 16:33 |
| corvus | clarkb: maybe it doesn't like our user agent? | 16:33 |
| clarkb | maybe? or possibly apache is rewriting the url in a subtly different way and beraking it? | 16:34 |
| clarkb | corvus: will apache use a different user agent than the client it is proxying for? Maybe we can have it use the client's user agent | 16:37 |
| corvus | i'm going to try to get some debug logging from apache | 16:39 |
| clarkb | ok | 16:39 |
| *** hamzy has joined #openstack-infra | 16:40 | |
| *** emccormick has joined #openstack-infra | 16:40 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Add publish-to-pypi to instack-undercloud https://review.openstack.org/638459 | 16:41 |
| *** ekultails has quit IRC | 16:43 | |
| *** ekultails has joined #openstack-infra | 16:45 | |
| *** yamamoto has joined #openstack-infra | 16:45 | |
| corvus | clarkb: debug level logging doesn't seem to provide any more information, it just repeats the urls more often. :( | 16:46 |
| *** lpetrut has quit IRC | 16:46 | |
| clarkb | corvus: what about wget'ing the url through the mirror now. Is it a consistent failure? | 16:47 |
| * clarkb contructs that url | 16:47 | |
| *** defionscode has joined #openstack-infra | 16:47 | |
| *** dave-mccowan has joined #openstack-infra | 16:47 | |
| clarkb | `http://mirror.iad.rax.openstack.org:8082/cloudflare/registry-v2/docker/registry/v2/blobs/sha256/ca/caf27325b298a6730837023a8a342699c8b7b388b8d878966b064a1320043019/data?verify=1550769259-lsxHynSfq%2Bt7BBVW3nPZCE8Q60Q%3D` is a 404 | 16:48 |
| *** gyee has joined #openstack-infra | 16:48 | |
| clarkb | so does seem consistent | 16:48 |
| *** lpetrut has joined #openstack-infra | 16:49 | |
| *** yamamoto has quit IRC | 16:50 | |
| *** defionscode has quit IRC | 16:51 | |
| *** defionscode has joined #openstack-infra | 16:53 | |
| *** mlavalle has joined #openstack-infra | 16:53 | |
| clarkb | corvus: RequestHeader set User-Agent "docker/18.09.2 go/go1.10.6 git-commit/6247962 kernel/4.15.0-45-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.2 \\(linux\\))" to force that through? | 16:54 |
| dmsimard | That's a lot of leaked images on nb03... http://paste.openstack.org/show/745633/ | 16:55 |
| clarkb | dmsimard: hrm ya. It seems more and more liekly this is a regression in nodepool we'll need to address | 16:56 |
| *** ginopc has quit IRC | 16:57 | |
| *** gfidente has joined #openstack-infra | 16:57 | |
| dmsimard | let me do the same exercise on the other nbs out of curiosity | 16:57 |
| clarkb | fwiw those files can be safely deleted if nodepool has lost track of them. (not sure if that will make debugging harder though) | 16:58 |
| dmsimard | no leaks on nb01 apparently | 16:58 |
| fungi | clarkb: dmsimard: pabelanger: https://review.openstack.org/638448 is a one-liner change to get e-mail notifications working for storyboard again (it's been broken since friday's maintenance to replace the old trusty server with a xenial one, since we switched to using a different hostname from the service name) | 16:58 |
| fungi | a concerned user e-mailed me directly about it this morning, worried their mail host was blocking notifications | 16:59 |
| clarkb | fungi: approved | 16:59 |
| dmsimard | no leaks on nb02 either | 16:59 |
| fungi | thanks! | 16:59 |
| corvus | i'm restarting apache on mirror.iax to load mod_log_forensic | 17:01 |
| clarkb | rgr | 17:01 |
| corvus | that didn't help either | 17:03 |
| Shrews | dmsimard: clarkb: nb03 was what i was using for some manual testing a couple weeks back for the build timeout. possible some of those are from that? | 17:03 |
| dmsimard | nb03 now has 120GB free, up from ... 0 | 17:04 |
| dmsimard | going to restart nodepool builder and remove from the emergency file | 17:04 |
| *** mlavalle has left #openstack-infra | 17:06 | |
| *** jpich has quit IRC | 17:06 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Proposed spec: tenant-scoped admin web API https://review.openstack.org/562321 | 17:06 |
| dmsimard | #status log (dmsimard) nb03 was found out of disk space on /opt, there is now 120GB available after cleaning up leaked images | 17:06 |
| openstackstatus | dmsimard: finished logging | 17:06 |
| *** pcaruana has quit IRC | 17:07 | |
| clarkb | corvus: terrible idea: looks like cloudflare does http in addition to https. We could flip to http to capture the network transfer more easily | 17:07 |
| corvus | clarkb: i like it. will do. | 17:07 |
| dmsimard | Shrews: don't know, let's wait a bit and check if things seem to leak still. | 17:09 |
| openstackgerrit | Merged openstack-infra/infra-specs master: Update priority effort Gerrit topics https://review.openstack.org/638024 | 17:13 |
| *** yamamoto has joined #openstack-infra | 17:20 | |
| *** markvoelker has joined #openstack-infra | 17:20 | |
| *** luizbag_ has joined #openstack-infra | 17:21 | |
| corvus | clarkb: hrm, i can't figure out how to get apache to do that right now; i have to afk for a few, server is all yours if you want it. | 17:22 |
| *** jamesmcarthur_ has joined #openstack-infra | 17:24 | |
| *** luizbag has quit IRC | 17:24 | |
| *** erikm has joined #openstack-infra | 17:25 | |
| *** yamamoto has quit IRC | 17:25 | |
| *** luizbag_ has quit IRC | 17:26 | |
| *** wolverineav has joined #openstack-infra | 17:26 | |
| *** jamesmcarthur has quit IRC | 17:27 | |
| clarkb | ok curre tly juggling a sick kid but will look in a bit | 17:27 |
| *** dtantsur is now known as dtantsur|afk | 17:29 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add site name to Exim local_domains on SB server https://review.openstack.org/638448 | 17:31 |
| *** mattw4 has joined #openstack-infra | 17:35 | |
| *** ccamacho has quit IRC | 17:36 | |
| *** trown is now known as trown|lunch | 17:38 | |
| *** mattw4 has quit IRC | 17:42 | |
| *** mattw4 has joined #openstack-infra | 17:43 | |
| *** zzzeek has quit IRC | 17:44 | |
| clarkb | corvus: bah on closer inspection http is a 301 to https | 17:46 |
| clarkb | also my old urls now return 403 forbidden so I think the timer on whatever token we had has expired | 17:47 |
| *** yamamoto has joined #openstack-infra | 17:47 | |
| *** ociuhandu has quit IRC | 17:48 | |
| *** panda|ruck is now known as panda|ruck|off | 17:48 | |
| *** wolverineav has quit IRC | 17:48 | |
| clarkb | I'm going to put my thinking cap on while drinking this tea and ponder other approaches | 17:48 |
| clarkb | fungi: ^ may have ideas too. I'd like to avoid sorting out https pcap but maybe that is what we are left with | 17:49 |
| *** zzzeek has joined #openstack-infra | 17:49 | |
| *** shardy has quit IRC | 17:49 | |
| *** ociuhandu has joined #openstack-infra | 17:50 | |
| fungi | we want to be able to get plaintext packet captures of ssl/tls streams? | 17:50 |
| *** Vadmacs has joined #openstack-infra | 17:51 | |
| clarkb | fungi: more generally debug why the 404s in http://paste.openstack.org/show/745628/ are 404s when they are 200s when requested to the backend locally | 17:52 |
| clarkb | fungi: plaintext packet capture of the tls stream between apache and dockerhubs CDN may be on path there | 17:53 |
| *** markvoelker has quit IRC | 17:53 | |
| *** ociuhandu has quit IRC | 17:54 | |
| *** lpetrut has quit IRC | 17:54 | |
| clarkb | note that those urls seem to be 403s for me locally now, I think they must only be valid for an hour? | 17:56 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: runc: Allow overlayfs mounts for container's rootfs https://review.openstack.org/638469 | 17:57 |
| fungi | https://medium.com/@tonywooster/docker-in-docker-in-gitlab-runners-220caeb708ca seems to describe an implementation where a docker registry is used as a central dockerhub image cache. not as straightforward as a caching web proxy unfortunately, but worth mentioning since i happened to stumble across it looking for solutions to the problem we're seeing | 18:02 |
| fungi | similar use case to ours though, i think | 18:03 |
| *** sdake has quit IRC | 18:05 | |
| *** sdake has joined #openstack-infra | 18:06 | |
| clarkb | https://docs.docker.com/registry/recipes/mirror/ is the docs for that. So far we've avoided it because in theory this protocol is http and should just work (tm). Unfortunately they try really ahrd to not make that the case | 18:06 |
| *** wolverineav has joined #openstack-infra | 18:06 | |
| clarkb | its silly to think we'd need application specific http caches, but maybe that is the reality we live in | 18:07 |
| *** roman_g has quit IRC | 18:13 | |
| clarkb | we likely could replace the docker vhost config in our mirrors with a copy of the insecure docker registry but configured to be a pass through mirror instead. I don't know how it manages disk usage though | 18:15 |
| clarkb | (like will it have a sad not being the only cache on the system) | 18:15 |
| *** mattw4 has quit IRC | 18:17 | |
| *** mattw4 has joined #openstack-infra | 18:17 | |
| fungi | well, back to sniffing the https communication, did we consider https://httpd.apache.org/docs/2.4/mod/mod_dumpio.html yet? | 18:24 |
| clarkb | fungi: no I don't think so | 18:24 |
| clarkb | but that looks like a good option. I'm currently trying to set up a local docker daemon to talk to our mirror | 18:25 |
| clarkb | an interesting experience | 18:26 |
| fungi | i'm digging around for more info on whether mod_dumpio works with mod_proxy in the way we're hoping | 18:26 |
| *** jpena is now known as jpena|off | 18:29 | |
| fungi | we tried upping the mod_proxy log level to trace? | 18:29 |
| fungi | i.e., "LogLevel error proxy:trace5" or similar? | 18:30 |
| fungi | i do see a few anecdotes of people using mod_dumpio and mod_proxy together for similar troubleshooting, though it's not very clear how we'll differentiate the client-facing traffic from the server-facing traffic (i guess it dumps both) | 18:32 |
| clarkb | the headers might make that apparent? | 18:36 |
| clarkb | I think I've almost got my local docker daemon talking to mirror.iad so hopefully I can use that to learn something useful | 18:36 |
| *** electrofelix has quit IRC | 18:37 | |
| clarkb | this gets weirder. I set up local docker to talk to mirror.iad. I see my docker pull ubuntu in there and it succeeded locally. However apache logs shows the same 404s so did docker go and ask some other registry I hadn't configured for this image | 18:38 |
| fungi | a packet capture just looking at outbound network traffoc could maybe tell you that | 18:39 |
| clarkb | ya the -D debug flag tells me nothing extra | 18:41 |
| mordred | clarkb: the -D debug flag to the docker daemon? | 18:45 |
| clarkb | mordred: no to docker pull | 18:45 |
| clarkb | let me see if the daemon will log things better | 18:45 |
| mordred | there's also a debug setting for the docker daemon I believe that will spit more crap into the docker daemon logs about the http interactions with the registry | 18:45 |
| clarkb | (seriously though it has to be a security issue to fallback to unspecified registries) | 18:45 |
| clarkb | since people use names to pull stuff you could get the entirely wrong image pulled down this way | 18:46 |
| corvus | clarkb: yes, docker daemon will fall back to upstream docker | 18:46 |
| corvus | * docker hub | 18:47 |
| corvus | clarkb: there is an important thing to keep in mind, which is that docker is explicitly designed to work only with dockerhub. so if you use "registry-mirrors" you're saying "these are mirrors of dockerhub" | 18:47 |
| corvus | private registries, in dockerland, always have their hostnames as part of the repository name | 18:48 |
| corvus | it's really a mess and makes no sense | 18:48 |
| fungi | best summary yet | 18:48 |
| clarkb | I see so a private registry is a different config item and ya daemon debug shows that it tries our mirror, fails and then talks to upstream dockerhub | 18:48 |
| corvus | but that's why docker doesn't consider it to be erroneous behavior to try upstream dockerhub when the mirror doesn't work. | 18:49 |
| fungi | i guess that's sort of like an extra_index_url for pip | 18:49 |
| mordred | yeah. so a zuul image in registry.opendev.org would actually be registry.opendev.org/zuul/zuul and would not necessarily have a hub.docker.com/zuul/zuul counterpart | 18:49 |
| mordred | basically, docker pull zuul/zuul is shorthand for docker pull hub.docker.com/zuul/zuul | 18:49 |
| clarkb | kind of amazing the client debug flag doesn't help you debug anything either but that is probably because the daemon is doing all the work and not communicating debug info back over the socket | 18:50 |
| mordred | clarkb: yah. the client is almost meaningless | 18:50 |
| clarkb | in any case I can now reliably reproduce this problem and generate urls that should be valid to test with | 18:50 |
| *** markvoelker has joined #openstack-infra | 18:50 | |
| mordred | woot! | 18:51 |
| clarkb | now to try the mirror in france and see if we get the same behavior (that should rule out any cdn weirdness) | 18:51 |
| *** trown|lunch is now known as trown | 18:52 | |
| clarkb | happens in ovh gra1 as well. So ya likely not a cdn region specific behavior (which we already mostly ruled out via local wget) | 18:54 |
| *** wolverineav has quit IRC | 18:54 | |
| clarkb | corvus: did you see fungi's suggestion of mod_dumpio? | 19:01 |
| clarkb | corvus: that may be our next best bet | 19:01 |
| fungi | or did we try loglevel proxy:trace5? | 19:02 |
| fungi | i couldn't tell from scrollback | 19:02 |
| corvus | fungi: i have not tried that | 19:02 |
| corvus | clarkb: have an iad url handy? | 19:03 |
| corvus | i'll try trace5 now | 19:03 |
| fungi | again i don't know exactly what's provided at that loglevel, just saw anecdotes of using that to check headers on proxied requests | 19:04 |
| corvus | fungi: it didn't seem to help | 19:04 |
| clarkb | corvus: /cloudflare/registry-v2/docker/registry/v2/blobs/sha256/7b/7b722c1070cdf5188f1f9e43b8413157f8dfb2b4fe84db3c03cb492379a42fcc/data?verify=1550777451-F0Z9Fan80qywgPCyFRGqEkd0jdc%3D | 19:04 |
| fungi | :/ | 19:05 |
| clarkb | not a url, but the 404ing path | 19:05 |
| corvus | fungi: mod_dumpio doesn't seem to be available | 19:07 |
| fungi | infra-root: just to follow up, e-mail notifications from storyboard.openstack.org do seem to be getting delivered now that the configuration change merged, though it did need me to manually reload the exim4 service before it would take effect | 19:08 |
| mordred | fungi: nod | 19:08 |
| corvus | weird /usr/lib/apache2/modules/mod_dumpio.so exists | 19:10 |
| fungi | yeah, i just got so far as confirming that on one of our mirror servers | 19:10 |
| fungi | the apache2-bin package provides it | 19:11 |
| fungi | maybe ot | 19:11 |
| fungi | grr | 19:11 |
| corvus | fungi: a2enmod dump_io | 19:11 |
| corvus | that's really weird | 19:11 |
| fungi | maybe it's just lacking a config snippet in configs_available | 19:11 |
| corvus | note the extra "_" | 19:11 |
| mordred | yay for _s | 19:11 |
| fungi | oh, hah | 19:11 |
| fungi | mods-available i meant | 19:11 |
| fungi | and yeah, /etc/apache2/mods-available/dump_io.load so i guess that was why | 19:12 |
| corvus | restarting to pick up module | 19:12 |
| fungi | ooh, infra-root just got spam advertising "npm enterprise" | 19:13 |
| mordred | SWEET WHEN CAN WE BUY IT | 19:13 |
| corvus | fungi, clarkb: http://paste.openstack.org/show/745645/ | 19:13 |
| *** manjeets has quit IRC | 19:14 | |
| fungi | wow, that's pretty verbose | 19:14 |
| *** manjeets has joined #openstack-infra | 19:14 | |
| corvus | are we requesting "/cloudflare/..." from the cloudflare host? | 19:15 |
| fungi | looks like the "[remote 52.22.67.152:443] mod_dumpio: dumpio_out" lines are what we're sending to dockerhub | 19:16 |
| clarkb | corvus: it almost reads that way | 19:16 |
| fungi | and yeah, that does look to be the case | 19:16 |
| clarkb | corvus: also that seems to confirm that we are passing the user agent through so that shouldn't be an issue. One thing we might want to do is disable the x forwarded for headers | 19:17 |
| clarkb | to appear less like a proxy to the remote | 19:17 |
| *** alishamohanty_ has joined #openstack-infra | 19:17 | |
| * clarkb rereads vhost config | 19:17 | |
| fungi | i wouldn't disable X-Forwarded-For if we can get by with it. i mean, it's polite of us to include it | 19:17 |
| clarkb | ProxyPass "/cloudflare/" "https://production.cloudflare.docker.com/" ttl=120 keepalive=On retry=0 | 19:17 |
| corvus | does order matter? | 19:18 |
| clarkb | fungi: ya we should leave it if that isn't related to the problem | 19:18 |
| *** wolverineav has joined #openstack-infra | 19:18 | |
| corvus | i just put the / proxy at the end and i think it worked. | 19:18 |
| corvus | let me try with real docker | 19:18 |
| fungi | huh | 19:18 |
| clarkb | oh ya if it isn't doing a longest match but instead first match that may matter | 19:19 |
| *** wolverineav has quit IRC | 19:19 | |
| corvus | here are the logs from 2 'docker pull's of the same image: http://paste.openstack.org/show/745648/ | 19:20 |
| corvus | first is a cache miss, second is a cache hit. both are now response 200 | 19:20 |
| corvus | i'll work on a patch | 19:20 |
| mordred | corvus: that seems much betterer | 19:20 |
| clarkb | and 8081 doesn't exhibit this problem because we suffixed the root | 19:21 |
| clarkb | I can't help but feel that the reason docker deprecated url paths in the pull api is to make us sad | 19:22 |
| mordred | clarkb: are we sure we still need 8081? like - I thought I read that dockerhub should be able to handle both apis | 19:22 |
| clarkb | (seriously what point does it serve? this is basically just http) | 19:22 |
| clarkb | mordred: I think its more there for our clients that use it | 19:23 |
| fungi | well, if nothing else, today i learned that mod_dumpio is way cool | 19:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix docker registry proxy https://review.openstack.org/638490 | 19:23 |
| clarkb | mordred: if you move all of those jobs off of 8081 to 8082 when it is working then we can drop 8081 I think | 19:23 |
| corvus | fungi: ++ | 19:23 |
| *** markvoelker has quit IRC | 19:23 | |
| corvus | i, um, kept the SamYaple comment, but i don't understand it. | 19:24 |
| mordred | clarkb: ya - I was thinking it might be nice to test that old v1 clients actually can use 8082 - because it would make docker registry mirror settings a little easier to wrap the head around | 19:24 |
| clarkb | corvus: it is explaining the difference between 8081 and 8082 | 19:24 |
| clarkb | corvus: with 8081 we did /docker-registry to the docker registry | 19:24 |
| clarkb | corvus: but then docker stopped talking to a non / root | 19:24 |
| clarkb | mordred: that should be self testing if you propose a change to say the tripleo or kolla jobs | 19:25 |
| clarkb | then you just have to watch out for the wonderful fallback behavior I just discovered | 19:25 |
| mordred | clarkb: not really - they'd just fallback ... yeah | 19:25 |
| mordred | clarkb: I think I'd rather test it manually first :) | 19:25 |
| fungi | are we sure they're even still using the old one? | 19:26 |
| clarkb | fungi: fairly certain as basically nothing uses 8082 at this point | 19:26 |
| fungi | oh | 19:26 |
| clarkb | (I think only mordred and corvus' work is attempting to use 8082) | 19:26 |
| fungi | for some reason i thought this was required for newer docker versions anyway | 19:26 |
| mordred | fungi: it is | 19:27 |
| clarkb | it is, but very newer versions | 19:27 |
| clarkb | but also tripleo uses other tools to pull | 19:27 |
| clarkb | and those tools aren't so particular | 19:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Improve comments in docker proxy config https://review.openstack.org/638493 | 19:27 |
| *** wolverineav has joined #openstack-infra | 19:28 | |
| corvus | all right, i'm going to afk until after lunch, then, um, i guess i'm going to docker some more docker. | 19:29 |
| clarkb | I'm going to context switch to trying to confirm my comments about github status events over in #zuul | 19:30 |
| clarkb | (basically i think that if a project moves quickly enough we won't be able to cache the merge commit shas accurately) | 19:30 |
| mordred | corvus: docker docker the docker dockers with the docker in the docker | 19:31 |
| fungi | i'm going to start looking at the story attachments patches for storyboard | 19:31 |
| fungi | but on a related note, we should probably brainstorm some ideas for where to stick the datastore for those | 19:31 |
| *** ccamacho has joined #openstack-infra | 19:31 | |
| fungi | how far did we get on looking into hosted swift options for zuul logs? | 19:32 |
| mordred | fungi: we looked at a few different clouds and I think there are more than one good option | 19:35 |
| mordred | fungi: I think for starters we were just using vexxhost and deferring doing multi-swift til later | 19:35 |
| *** ldnunes has quit IRC | 19:35 | |
| fungi | makes sense | 19:37 |
| clarkb | hrm though most of the api load early this morning may have been the confluence of uncached merged commit sha and many many unlabel events | 19:38 |
| fungi | mordred: so in theory i can just create a swift container in vexxhost and configure storyboard to point there for uploading and serving story attachments? | 19:38 |
| fungi | any odd caveats with their swift deployment? | 19:39 |
| *** Vadmacs has quit IRC | 19:42 | |
| clarkb | ya reading ansible's github merge logs I think in general our cache is helping a ton. They've merged a bunch of commits in the last half hour all with status results happening after merge and our github queue is back down to 0 now | 19:43 |
| mordred | fungi: a few - but I believe we've captured all of them in openstacksdk so far | 19:43 |
| clarkb | we only went up into the high teens at peak too | 19:43 |
| clarkb | so I think the confluence of many events being sent around that time and possibly not having been able to cache the merge_commit_sha because either it moved in a way we didn't see or we just evicted the entries from the cache did us in early this monring | 19:44 |
| clarkb | given that I think I won't spend any extra time debugging this | 19:44 |
| mordred | ++ | 19:44 |
| * clarkb fixes local docker to speak to the normal registry | 19:45 | |
| fungi | clarkb: given where the lru stuff ended up and the impact we've seen from it, do you see any use in https://review.openstack.org/636404 or the suggestion there to track per-event time to process? | 19:46 |
| clarkb | fungi: I think that data may still be helpful to expose to users. Then ttx or gundalow or whoever can quickly check "why is github slow with my PR" without pinging us to query logs directly | 19:47 |
| clarkb | fungi: similar to how we expose zuuls internal event queue and the results queue | 19:47 |
| clarkb | maybe ^ implies we just want to expose it via the status page rather than via statsd? though I think statsd has the nice benefit of showing data over time | 19:48 |
| clarkb | fwiw I think the derivative data would be nice too but even just exposing the simpler count is probably fine | 19:49 |
| clarkb | re processing delay we peaked at 7 minutes during the great gundalow merging spree of 45 minutse ago | 19:50 |
| clarkb | which isn't bad compared to what it was before but still likely long enough to be noticed by users | 19:51 |
| fungi | cool, i'll pick it back up in that case | 19:52 |
| *** jamesmcarthur_ has quit IRC | 19:56 | |
| *** jamesmcarthur has joined #openstack-infra | 19:57 | |
| *** wolverineav has quit IRC | 20:01 | |
| *** emccormick has quit IRC | 20:01 | |
| *** jamesmcarthur has quit IRC | 20:01 | |
| *** whoami-rajat has quit IRC | 20:02 | |
| *** emccormick has joined #openstack-infra | 20:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add nodepool-dib dashboard https://review.openstack.org/638325 | 20:14 |
| *** e0ne has joined #openstack-infra | 20:14 | |
| *** jamesmcarthur has joined #openstack-infra | 20:18 | |
| ianw | dmsimard: i'm not sure if you got to the bottom of it, but there were changes put in for defaulting to .qcow2, and it sounds like maybe it's a bit aggressive and always builds it, even if you don't want it? | 20:19 |
| ianw | AJaeger: thanks ... if you're interested there's a test version of it @ http://grafana.openstack.org/d/ZGHytDriz/nodepool-dib?orgId=1 | 20:20 |
| ianw | i'll delete this of course in due course | 20:21 |
| *** markvoelker has joined #openstack-infra | 20:21 | |
| AJaeger | ianw: nice, thanks! I just fixed the titles, rest looked fine - and that graph confirms it ;) | 20:24 |
| ianw | i'm not sure about the way we put the stats in, though; we seem to miss a few which a pipeline would help with (https://review.openstack.org/#/c/638265/) | 20:27 |
| ianw | and the return code for each image type is going to be the same, as it's all done in the same build | 20:28 |
| *** wolverineav has joined #openstack-infra | 20:33 | |
| *** ijw has joined #openstack-infra | 20:37 | |
| *** e0ne has quit IRC | 20:37 | |
| *** wolverineav has quit IRC | 20:38 | |
| clarkb | ianw: ya though the biggest cost in building the image is the build itself last I looked. The image conversions do take time but not nearly as much as the rest of the process | 20:39 |
| *** jamesmcarthur has quit IRC | 20:40 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix docker registry proxy https://review.openstack.org/638490 | 20:40 |
| *** jamesmcarthur has joined #openstack-infra | 20:40 | |
| clarkb | ianw: what sets the test image size to 4TB? | 20:40 |
| clarkb | that seems a bit excessive :) | 20:41 |
| clarkb | hrm looks like we actually write "fake-data" to the image file | 20:42 |
| clarkb | ah g is for gauge not gb | 20:43 |
| clarkb | ms is for timing | 20:43 |
| clarkb | so 4096 is block size whcih is largely correct in places | 20:43 |
| *** jamesmcarthur has quit IRC | 20:45 | |
| ianw | yeah that's in bytes :) | 20:45 |
| ianw | it's this loop that will only ever return the same thing -> https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/builder.py#n888 | 20:47 |
| clarkb | ianw: those graphs seem to show our mysteriously large qcow2s getting smaller again? | 20:47 |
| clarkb | I wonder if that was a regression in qemu-img or something that has been fixed | 20:47 |
| ianw | clarkb: but others show it jump back up too | 20:47 |
| ianw | i'm not convinced it's not a graph artifact, yet | 20:47 |
| clarkb | hrm ya | 20:48 |
| clarkb | re that loop returning the same value I think it is technically correct. Since we convert the image types at the end of the single build. But probably confusing from a stats perspective | 20:48 |
| ianw | the conversion happens within dib though via the -t args, so it's all the one process | 20:49 |
| clarkb | ya | 20:50 |
| ianw | the other thing is that a single timer event every ~24 hours is hard to work with for the duration | 20:51 |
| ianw | you can "keeplastvalue", but that only works if there is a last value in your view | 20:51 |
| ianw | so essentially any zoom < 24 hours is likely to now show anything, which can be confusing | 20:51 |
| ianw | s/now/not | 20:51 |
| clarkb | ya I zoomed out to 7 days | 20:52 |
| clarkb | but probably not clear that doing so is probably required | 20:52 |
| *** gfidente is now known as gfidente|afk | 20:52 | |
| clarkb | can we set the default time range to say 24 hours by default for that one dashboard? | 20:52 |
| ianw | yeah, or make the duration a gauge | 20:52 |
| *** jamesmcarthur has joined #openstack-infra | 20:53 | |
| *** markvoelker has quit IRC | 20:55 | |
| openstackgerrit | Merged openstack-infra/system-config master: Improve comments in docker proxy config https://review.openstack.org/638493 | 20:58 |
| *** wolverineav has joined #openstack-infra | 21:06 | |
| *** kgiusti has left #openstack-infra | 21:08 | |
| clarkb | infra-root can I get a second reviewer on https://review.openstack.org/#/c/629662/2 I am happy to babysit that today | 21:08 |
| mordred | clarkb: on it | 21:09 |
| clarkb | if you can review the child of that (enable future parser for kerberos servers) I'll likely approve that one once afs is done and happy | 21:09 |
| mordred | clarkb: +2'd - +A at will | 21:09 |
| clarkb | tyty | 21:09 |
| clarkb | mordred: can you get the child too? | 21:09 |
| mordred | clarkb: done | 21:09 |
| *** wolverineav has quit IRC | 21:09 | |
| clarkb | woot thanks again | 21:09 |
| *** wolverineav has joined #openstack-infra | 21:10 | |
| fungi | 637388 (the storyboard cleanup change) touches some of the same files if anybody's in the mood | 21:10 |
| fungi | no idea if they're going to merge-confict | 21:10 |
| fungi | or merge-confLict | 21:10 |
| clarkb | fungi: conflicts with says the child does conflict | 21:10 |
| clarkb | mordred: if you want to approve fungi's change I'm happy to rebase the kerbeeros child change | 21:11 |
| *** rcernin has joined #openstack-infra | 21:11 | |
| mordred | clarkb: on it | 21:12 |
| *** ijw has quit IRC | 21:12 | |
| mordred | I approved fungi's change | 21:12 |
| *** ijw has joined #openstack-infra | 21:14 | |
| corvus | mmm. merge confit. | 21:18 |
| mordred | the best part of working from a coffee shop is when the snippet of conversation you overhear is "mmm. mini bundt cakes" | 21:20 |
| corvus | i don't know if the're good (how is it i don't know this?) but i love their name: https://www.nothingbundtcakes.com/ | 21:21 |
| fungi | and now i hunger | 21:22 |
| *** ijw has quit IRC | 21:22 | |
| clarkb | I just had lunch but now I want desert | 21:22 |
| clarkb | *dessert | 21:22 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Proposed spec: tenant-scoped admin web API https://review.openstack.org/562321 | 21:26 |
| *** ijw has joined #openstack-infra | 21:27 | |
| *** ijw has quit IRC | 21:32 | |
| *** ijw has joined #openstack-infra | 21:32 | |
| *** wolverineav has quit IRC | 21:34 | |
| mordred | corvus: they are actually delicious. I've gotten a red velvet cake from nothing bundt cakes the last 2 years | 21:35 |
| *** eharney has quit IRC | 21:35 | |
| mordred | corvus: and most days I'm sad that I do not have one - but also happy, because if I had one I would eat it | 21:35 |
| *** slaweq has quit IRC | 21:35 | |
| *** alishamohanty_ has quit IRC | 21:36 | |
| fungi | and then you would be sad again because you don't have one | 21:38 |
| fungi | a vicious cycle | 21:38 |
| *** wolverineav has joined #openstack-infra | 21:39 | |
| *** wolverineav has quit IRC | 21:40 | |
| *** wolverineav has joined #openstack-infra | 21:40 | |
| *** wolverineav has quit IRC | 21:41 | |
| clarkb | fwiw github event queue is quite large right now :/ | 21:41 |
| *** wolverineav has joined #openstack-infra | 21:41 | |
| clarkb | hrm it seems that one of the changes that merged when I thought things were ok may have been the kick off event? | 21:43 |
| clarkb | liek we had some delay in getting to processing that? | 21:43 |
| mordred | clarkb: it's your fault for thinking it was ok | 21:44 |
| clarkb | indeed | 21:45 |
| fungi | i need to go wet-sand some drywall mud. back soon | 21:46 |
| *** slaweq has joined #openstack-infra | 21:48 | |
| mordred | fungi: it might not be dry when you're finished | 21:51 |
| *** markvoelker has joined #openstack-infra | 21:53 | |
| *** ociuhandu has joined #openstack-infra | 21:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: run-buildset-registry: run a dual registry https://review.openstack.org/638514 | 21:54 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on future parser for afs servers https://review.openstack.org/629662 | 21:55 |
| *** jamesmcarthur has quit IRC | 21:57 | |
| mordred | corvus: I love that there is more content in the commit message than the actual patch there | 22:00 |
| corvus | mordred: yeah... i mean, at least docker makes "run a daemon twice on the same host on different ports because it's authors have an ongoing polite disagreement with reality" easy. | 22:01 |
| mordred | corvus: ++ | 22:02 |
| *** erikm has left #openstack-infra | 22:04 | |
| openstackgerrit | Merged openstack-infra/system-config master: Clean up after StoryBoard replacement maintenance https://review.openstack.org/637388 | 22:05 |
| *** jamesmcarthur has joined #openstack-infra | 22:05 | |
| *** jamesmcarthur has quit IRC | 22:07 | |
| *** trown is now known as trown|outtypewww | 22:07 | |
| *** ociuhandu has quit IRC | 22:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 22:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 22:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Use buildset registry push endpoint https://review.openstack.org/638520 | 22:12 |
| corvus | mordred, clarkb, fungi: can you +3 the stack of 4 changes which starts at https://review.openstack.org/638514 ? | 22:13 |
| corvus | about half of that is used in project-config, and half isn't, but it'll probably be easiest to review and merge all at once | 22:13 |
| *** ijw has quit IRC | 22:14 | |
| *** kjackal has quit IRC | 22:14 | |
| *** ijw has joined #openstack-infra | 22:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Use buildset registry push endpoint https://review.openstack.org/638520 | 22:15 |
| clarkb | corvus: yup I'll take a look as sono as I get these changes rebased now that storyboard cleanup merged | 22:17 |
| clarkb | hrm gerrit doesn't say that change conflicts anymore. I never understand how this conflicts checking works | 22:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/grafyaml master: Update valueName values https://review.openstack.org/638522 | 22:18 |
| corvus | clarkb: i'm starting to suspect it's "touches the same file" | 22:18 |
| *** jcoufal has quit IRC | 22:19 | |
| corvus | but even if it isn't, it does seem very loose. however, i've come to view that as an asset, in that it helps find potentially related changes, even if they don't truly textually conflict. | 22:20 |
| *** ijw has quit IRC | 22:20 | |
| clarkb | ya | 22:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use host networking for gitea https://review.openstack.org/637334 | 22:21 |
| *** hamzy has quit IRC | 22:21 | |
| mordred | corvus, clarkb: don't know if you saw, but the gitea folks have already put an indexer abstraction in front of issues, and I think the same dude is working on adding one for repos | 22:23 |
| corvus | mordred: i saw! fingers crossed :) | 22:23 |
| mordred | and has elasticsearch and redis drivers planned. and has implemented a "use the db" driver - which is a bad idea but is there for tiny installs | 22:23 |
| mordred | corvus: \o/ | 22:24 |
| clarkb | mordred: nice | 22:24 |
| *** markvoelker has quit IRC | 22:25 | |
| clarkb | afs01.dfw.o.o and afsdb01.o.o have had puppet runs under puppet4 and the only oddity (on afs01) was present before the futureparser change | 22:26 |
| clarkb | going to approve kerberos change now and if it conflicts rebase then | 22:26 |
| clarkb | corvus: also now I know why you know so much about mirror and private registries :) | 22:26 |
| corvus | nope. i don't know a thing. no one ask me any questions about docker. docker what? | 22:27 |
| *** sdake has quit IRC | 22:27 | |
| mordred | corvus: docker docker registry docker docker mirror docker | 22:29 |
| mordred | corvus: container container kuberenetes docker docker skopeo! | 22:29 |
| clarkb | that shared fs hack is so good (yay for sha256 addressed data blobs I guess | 22:29 |
| * mordred is cloud native now | 22:29 | |
| corvus | clarkb: that was mordred's idea :) and yeah, it's a good one | 22:30 |
| * mordred is useful for coming up with ways to subvert people's intentions | 22:30 | |
| *** wolverineav has quit IRC | 22:30 | |
| clarkb | corvus: have you checked yet if the main in region mirror proxies have the fix ? | 22:31 |
| corvus | clarkb: no -- however, none of this uses those proxies right now | 22:31 |
| *** wolverineav has joined #openstack-infra | 22:31 | |
| corvus | (we will be able to use them, i think, but i'd like to do it as a follow-up, because this has enough moving parts as-is) | 22:32 |
| clarkb | corvus: reviewing https://review.openstack.org/#/c/638180/8/roles/use-buildset-registry/tasks/main.yaml and noticing the daemon.json content is being decoded from base 64? isn't it just normal human readable ascii/utf8? | 22:33 |
| *** ijw has joined #openstack-infra | 22:34 | |
| corvus | (weirdly, if they *had* been working, we might not have ended up with the dual-registry idea, because i think we would just fall through. however, this is better because it makes the system work for folks without region-local mirrors) | 22:34 |
| corvus | clarkb: it is, ansible slurp module base64 encodes data | 22:34 |
| clarkb | ah | 22:34 |
| corvus | for, what i assume, must be excellent raisons. :) | 22:34 |
| *** wolverineav has quit IRC | 22:36 | |
| *** jtomasek has quit IRC | 22:37 | |
| clarkb | corvus: https://review.openstack.org/#/c/638180/8/roles/use-buildset-registry/tasks/user-config.yaml sets the username and passwd for the main docker registry which won't workl right? is that a bug? | 22:38 |
| mordred | clarkb: you're going to love the explanation of that one | 22:38 |
| corvus | i guess that should probably be in the commit message... | 22:38 |
| corvus | because docker assumes that any mirror is a mirror of *the* docker.io registry, whenever it communicates with a mirror, it uses, exclusively, the credentials for *the* docker.io registry. regardless of the actual network address of the mirror. | 22:39 |
| *** ijw has quit IRC | 22:39 | |
| * corvus ducks for cover | 22:39 | |
| *** dave-mccowan has quit IRC | 22:40 | |
| clarkb | so we can remove the credentials for the non push instance? | 22:40 |
| clarkb | I suppose that too would be confusing. That is amazing btw | 22:40 |
| corvus | yes, we could theoretically remove them | 22:40 |
| mordred | clarkb: right? I thought you'd like that | 22:40 |
| *** dave-mccowan has joined #openstack-infra | 22:40 | |
| *** gfidente|afk has quit IRC | 22:40 | |
| corvus | (they only way they would be used is if the non-push buildset registry endpoint were directly addressed (like "docker pull 1.2.3.4:5000/foo/bar:latest") and we don't do that) | 22:41 |
| corvus | and yes, this has been declared "not a bug" | 22:42 |
| clarkb | does it fail when it falls back to the upstream source? | 22:43 |
| clarkb | because the creds are wrong? | 22:43 |
| corvus | if it did, it would | 22:43 |
| *** slaweq has quit IRC | 22:43 | |
| corvus | but it doesn't in this case because of the pull-through proxy, which is configured with it's own set of empty credentials to talk to upstream | 22:44 |
| corvus | if the pull-through proxy fails (like, not because it doesn't have an image, but, say, there's a network partition or something), it will talk to upstream and fail because of the credentials. | 22:44 |
| clarkb | right if that failed for some reason (network blip) we'd see a failure in auth when talking upstream in the fallback (assuming network started working by then) | 22:45 |
| mordred | taht's awesome | 22:45 |
| clarkb | ok I think I understand | 22:45 |
| clarkb | or rather get the behavior | 22:45 |
| corvus | in this situation, i actually think that's good behavior for us. it may cause a job to fail rather that to accidentally use an upstream non-speculative image. | 22:45 |
| mordred | so we also get a hard fail if the local mirror usage stops working | 22:45 |
| clarkb | not sure I understand why that behavior is desirable but don't expect you would argue for it either | 22:45 |
| clarkb | ya I mean if it were me not falling back by default and hard failing as default behavior makes way moe sense | 22:45 |
| clarkb | but I didn't design the thing | 22:45 |
| clarkb | and ya adding more words to things might be a good idea (though I've approved the cnage, maybe a readme aside section?) | 22:46 |
| *** dave-mccowan has quit IRC | 22:46 | |
| corvus | clarkb: yeah, you *clearly* didn't design it. i am certain we would have a much better system if you had. :) | 22:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run an haproxy load balancer for gitea https://review.openstack.org/638033 | 22:49 |
| *** wolverineav has joined #openstack-infra | 22:49 | |
| *** rh-jelabarre has quit IRC | 22:52 | |
| openstackgerrit | Merged openstack-infra/grafyaml master: Update valueName values https://review.openstack.org/638522 | 22:52 |
| fungi | still catching up... any reason we're holding off approval of 638195 with three (soon to be four) +2 votes? | 22:54 |
| fungi | presumably those are just inherited from earlier patchsets on rebasing | 22:54 |
| corvus | fungi: nope, and we've both approved it now :) | 22:55 |
| *** slaweq has joined #openstack-infra | 22:56 | |
| fungi | heh | 22:56 |
| corvus | also, i'm not sure that "tox-py35-on-zuul" is necessarily the best thing to be running in zuul-jobs now... | 22:56 |
| fungi | because of waiting for xenial nodes? | 22:56 |
| clarkb | no reason I was just reading it and continuing to try and udnerstand this docker stuff | 22:57 |
| corvus | because it's a little flakey, and now that we have more robust usage of tox in zuul-jobs' own tox jobs, it's probably not adding anything | 22:57 |
| mordred | corvus: yeah. I agree | 22:57 |
| fungi | oh, i see now there's also a plain tox-py35 | 22:58 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on future parser for kerberos servers https://review.openstack.org/629663 | 22:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Remove tox-py35-on-zuul from zuul-jobs https://review.openstack.org/638525 | 22:59 |
| *** slaweq has quit IRC | 23:00 | |
| *** tkajinam has joined #openstack-infra | 23:01 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: [wip] rhel8 beta support https://review.openstack.org/623137 | 23:05 |
| clarkb | infra-root in checking if puppet futureparser is happy with the kdc servers I've discovered that kdc04 seems to be unhappy | 23:07 |
| mordred | uhonh | 23:07 |
| clarkb | I don't see it running a krb process | 23:08 |
| clarkb | and puppet can't install krb5-admin-server | 23:08 |
| clarkb | I think the package install failure might be a package bug? | 23:08 |
| clarkb | the --configure process returns exit code 10 | 23:08 |
| clarkb | and if I run it by hand I get a notice saying "hey this package won't configure this for you you need to do it yourself" | 23:09 |
| *** ijw has joined #openstack-infra | 23:09 | |
| clarkb | so I guess my first question is kdc04 listed as a valid kerberos server in dns | 23:09 |
| * clarkb looks at our docs | 23:09 | |
| *** rascasoft has quit IRC | 23:10 | |
| clarkb | kdc01 is primary but kdc04 is listed as a valid server in the SRV record | 23:11 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** tosky has quit IRC | 23:11 | |
| clarkb | my hunch is if we fix the package install weirdness that puppet will be happy and configure things the way we want | 23:12 |
| clarkb | then we can make kdc04 primary and upgrade kdc01 to xenial from trusty | 23:12 |
| clarkb | unless we aren't supposed to run the processes on kdc04 since it isn't primary? is this active active or active with standby? | 23:12 |
| clarkb | mordred: ^ | 23:12 |
| *** sdake has joined #openstack-infra | 23:13 | |
| clarkb | oh docs say it is a standby so maybe the service side of things is fine and it is just apt and puppet unhappy | 23:13 |
| corvus | clarkb: k believe krb5 should be running on both, and kadmind only running on 01 | 23:14 |
| corvus | er, i believe. i don't know what agent k believes. | 23:14 |
| fungi | clarkb: are you seeing the error in syslog or somewhere else? | 23:14 |
| *** mriedem has quit IRC | 23:14 | |
| clarkb | fungi: yes syslog but also running apt-get install -o DPkg::Options::=--force-confold krb5-admin-server in the foreground | 23:15 |
| clarkb | fungi: grep puppet-user to see puppet logs | 23:15 |
| fungi | i might be able to play the role of "the apt whisperer" | 23:15 |
| tonyb | Would y'll object if I made a new (apache) puppet template that was a mash-up of static-https-redirect.vhost.erb and static-http-and-https.vhost.erb | 23:15 |
| *** slaweq has quit IRC | 23:16 | |
| fungi | tonyb: mashup in what way? to what purpose? maybe we can just fix one (or both) of those? | 23:16 |
| tonyb | one that redirected http -> https but enabled Redirect on the https server? | 23:16 |
| tonyb | or would it be better to just do the latter in static-http-and-https.vhost.erb? | 23:16 |
| fungi | https-redirect is supposed to be used to redirect http to https, http-and-https is intended to serve the same content over both http and https | 23:17 |
| tonyb | fungi: basically I want Redirect on releases.o.o and I think we also want that content only available over https because signatures? | 23:17 |
| fungi | seems like we should just start redirecting http to https for releases.o.o | 23:17 |
| tonyb | fungi: we do | 23:17 |
| tonyb | fungi: BUT that template doesn't have AllowOverrideList Redirect RedirectMatch | 23:18 |
| tonyb | so I can't add a .htaccess file with Redirect statements in it | 23:18 |
| tonyb | fungi: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002771.html is the context | 23:18 |
| fungi | oh, got it. releases.o.o today redirects http to https but doesn't give you the ability to also put a redirect in a .htaccess file | 23:19 |
| tonyb | fungi: Yup that one | 23:19 |
| fungi | i would just update static-https-redirect.vhost.erb to allow that, personally. it doesn't seem super unsafe but feedback from others is warranted | 23:19 |
| tonyb | fungi: so we can either a) add AllowOverrideList... to the exsisting template ;or b) create a new one just for rleases.o.o | 23:19 |
| tonyb | fungi: OK, I'll propose a change to do the simple thing and we can discuss it there | 23:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: add support for report only connection https://review.openstack.org/568216 | 23:20 |
| fungi | tonyb: to clarify, the goal with those templates is to reduce the amount of duplication and risk of divergence between vhost definitions for sites on that server | 23:21 |
| fungi | so in keeping with that, improving the existing templates to be usable for the sites we're serving there rather than splitting more out is my preference | 23:22 |
| *** markvoelker has joined #openstack-infra | 23:22 | |
| tonyb | fungi: Yup I thought it might be, but there is a small risk in added this ALlowOverrideList for all static sites | 23:23 |
| fungi | the primary trigger for switching to those was when i was updating our mod_ssl settings and didn't want them to end up wildly divergent | 23:23 |
| clarkb | fungi: https://git.launchpad.net/ubuntu/+source/krb5/tree/debian/krb5-admin-server.postinst?h=ubuntu/xenial-updates#n17 we do change the config with puppet | 23:23 |
| tonyb | so I thought the trade off there might come down the other way | 23:23 |
| clarkb | but we are setting the confold flag so I wouldn't expect it to complain about it there | 23:23 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 23:25 |
| fungi | i guess the postinstall script doesn't provide any actual output to explain its concern | 23:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use host networking for gitea https://review.openstack.org/637334 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run an haproxy load balancer for gitea https://review.openstack.org/638033 | 23:28 |
| fungi | subprocess installed post-installation script returned error exit status 10 | 23:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-runner: add quick-start integration test https://review.openstack.org/635701 | 23:28 |
| clarkb | fungi: ya | 23:29 |
| fungi | any objection to me trying it without --force-confold to see what it wants to do? | 23:29 |
| clarkb | the postinst script runs set -e and doesn't explicitly exit 10 so it must be a command failing | 23:29 |
| clarkb | fungi: I guess not? it is probably going to want to update the config and it should show you diffs right? | 23:30 |
| fungi | entirely possible it's also nothing to do with the config though | 23:30 |
| clarkb | ya | 23:30 |
| openstackgerrit | Tony Breeds proposed openstack-infra/system-config master: Add Redirect options to static https vhosts https://review.openstack.org/638527 | 23:31 |
| tonyb | fungi: ^^^ | 23:31 |
| fungi | clarkb: actually, i'll first try adding -x in /var/lib/dpkg/info/krb5-admin-server.postinst | 23:31 |
| clarkb | fungi: ok | 23:31 |
| *** ijw has quit IRC | 23:32 | |
| *** ijw has joined #openstack-infra | 23:32 | |
| fungi | the last command it runs is: | 23:32 |
| fungi | exec /usr/share/debconf/frontend /var/lib/dpkg/info/krb5-admin-server.postinst configure 1.13.2+dfsg-5ubuntu2 | 23:32 |
| *** sdake has quit IRC | 23:33 | |
| fungi | it does try to start the krb5-admin-server service | 23:34 |
| clarkb | that is from the #DEBHELPER# substitued content of postinst I think | 23:35 |
| clarkb | so maybe service logs will give us the answer | 23:35 |
| *** rh-jelabarre has joined #openstack-infra | 23:35 | |
| fungi | so i suspect it's attempting to start kadmind, failing, and that's the cause | 23:35 |
| clarkb | `sudo journalctl -u krb5-admin-server` has no entries | 23:36 |
| *** sdake has joined #openstack-infra | 23:38 | |
| *** gyee has quit IRC | 23:38 | |
| fungi | i don't suppose we can tell perl to verbosely echo everything it runs to stdout? | 23:38 |
| *** mattw4 has quit IRC | 23:39 | |
| clarkb | /var/crash/krb5-admin-server.0.crash says this started february 1 | 23:39 |
| clarkb | assuming it hasn't rolled over the instances of this logging | 23:39 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: run-buildset-registry: run a dual registry https://review.openstack.org/638514 | 23:39 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: use-buildset-registry: support running before docker installed https://review.openstack.org/638180 | 23:39 |
| fungi | oh, i bet we can set debconf to noninteractive | 23:39 |
| clarkb | I assume not given the 0 in that file | 23:39 |
| clarkb | fungi: maybe try start the server manaully | 23:39 |
| clarkb | and see if systemd/journalctl give us anything | 23:39 |
| fungi | first gonna try: | 23:40 |
| fungi | sudo DEBIAN_FRONTEND=noninteractive apt-get install -o DPkg::Options::=--force-confold krb5-admin-server | 23:40 |
| fungi | yeah, still same inscrutible error | 23:40 |
| fungi | you wanna try starting the service? see where that leads up | 23:41 |
| fungi | er, leads us | 23:41 |
| clarkb | sure I'll give that a go | 23:41 |
| clarkb | heh systemctl seems to think it started but ps says no | 23:42 |
| clarkb | but Feb 21 23:41:53 kdc04 kadmind[28609]: kadmind: kadmind: Can not fetch master key (error: No such file or directory). while initializing, aborting | 23:42 |
| clarkb | is our answer I think? | 23:42 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Split docker mirror config into its own role https://review.openstack.org/638195 | 23:43 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use buildset registry push endpoint https://review.openstack.org/638520 | 23:43 |
| fungi | sounds quite likely | 23:43 |
| fungi | i've undone my edit to the postinst script now that we've gone down that path | 23:44 |
| clarkb | do we need to do a sync from kdc01 to kdc04 via kprod? | 23:45 |
| clarkb | *kpropd | 23:46 |
| clarkb | I wonder if this step was just never completed when the new 04 server was built | 23:46 |
| *** sdake has quit IRC | 23:46 | |
| *** sdake has joined #openstack-infra | 23:47 | |
| *** rh-jelabarre has quit IRC | 23:48 | |
| clarkb | we don't seem to have incremental propagation enabled on kdc04 | 23:48 |
| fungi | i guess the new server was to upgrade from trusty to xenial? | 23:49 |
| clarkb | however there is a kpropd running on kdc04 so maybe we can force it? | 23:49 |
| clarkb | fungi: ya I think pabelanger did the upgrade last summer | 23:49 |
| clarkb | should I go ahead and run /usr/local/bin/run-kprop.sh to see if that propagates the db info like we need it to? | 23:50 |
| clarkb | corvus: mordred ^ you may know off the top of your head? | 23:51 |
| *** sdake has quit IRC | 23:51 | |
| *** sdake_ has joined #openstack-infra | 23:51 | |
| pabelanger | clarkb: kdc04 was the replacement for kdc02 | 23:52 |
| *** gyee has joined #openstack-infra | 23:52 | |
| pabelanger | so, it should be slave to kdc01 | 23:52 |
| clarkb | pabelanger: yup | 23:53 |
| clarkb | pabelanger: the problem is it isn't running krb5-admin-server. I think possibly because we never propagated the database info from kdc01 to kdc04 | 23:53 |
| clarkb | I think the script above will do that for us | 23:53 |
| clarkb | so I'm tempted to just give it a go | 23:53 |
| clarkb | any objections? | 23:53 |
| corvus | clarkb: i don't know off the top of my head. no objections. | 23:54 |
| clarkb | that says it succeeded but I don't see a stash or keytab file in the /etc/krb5kdc dir | 23:55 |
| clarkb | so maybe this doesn't do that | 23:55 |
| *** markvoelker has quit IRC | 23:55 | |
| pabelanger | clarkb: I think I followed directions at: https://docs.openstack.org/infra/system-config/kerberos.html#realm-creation | 23:57 |
| *** wolverineav has quit IRC | 23:57 | |
| *** ekultails has quit IRC | 23:57 | |
| pabelanger | but been some time, so cannot fully remember | 23:57 |
| clarkb | https://web.mit.edu/kerberos/krb5-1.12/doc/admin/install_kdc.html#configure-slave-kdcs might be our answer | 23:59 |
| clarkb | I'm guessing we haven't puppeted that | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
