| openstackgerrit | Merged openstack-infra/system-config master: Rework intro https://review.openstack.org/452289 | 00:02 |
|---|---|---|
| *** yamahata has joined #openstack-infra | 00:02 | |
| ianw | pabelanger: while it's quiet do you want me to make the volumes for 437100? | 00:03 |
| *** jkilpatr has quit IRC | 00:04 | |
| pabelanger | ianw: sure! | 00:04 |
| *** masber has joined #openstack-infra | 00:05 | |
| ianw | pabelanger: i assume "opensuse" (as opposed, to say, suse?) | 00:06 |
| pabelanger | ianw: ya, that's what I have been using | 00:07 |
| *** aeng has quit IRC | 00:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add missing DEVSTACK_LOCAL_CONFIG+= for opensuse nodepool job https://review.openstack.org/452576 | 00:39 |
| *** erlon has quit IRC | 00:45 | |
| *** aeng has joined #openstack-infra | 00:51 | |
| *** baoli has quit IRC | 00:52 | |
| *** s-shiono has joined #openstack-infra | 00:54 | |
| *** baoli has joined #openstack-infra | 00:58 | |
| ianw | hmm POST_FAILURE on http://logs.openstack.org/00/437100/1/gate/gate-infra-puppet-apply-3-ubuntu-precise/e5491f5/ | 00:58 |
| ianw | odd | 00:58 |
| *** yamahata has quit IRC | 01:03 | |
| *** s-shiono_ has joined #openstack-infra | 01:04 | |
| *** kien-ha has quit IRC | 01:05 | |
| *** jamielennox|away is now known as jamielennox | 01:06 | |
| *** s-shiono has quit IRC | 01:07 | |
| openstackgerrit | Merged openstack-infra/system-config master: Mirror openSUSE Leap 42.2 repositories https://review.openstack.org/437100 | 01:09 |
| *** gcb has joined #openstack-infra | 01:14 | |
| *** aeng has quit IRC | 01:16 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Mirror rdo centos7 repo to AFS https://review.openstack.org/451972 | 01:17 |
| *** baoli has quit IRC | 01:19 | |
| *** Julien-zte has joined #openstack-infra | 01:27 | |
| *** baoli has joined #openstack-infra | 01:29 | |
| *** aeng has joined #openstack-infra | 01:30 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 01:33 |
| *** tuanluong has joined #openstack-infra | 01:35 | |
| *** thorst has joined #openstack-infra | 01:37 | |
| *** thorst has quit IRC | 01:42 | |
| *** baoli has quit IRC | 01:43 | |
| *** namnh has joined #openstack-infra | 01:48 | |
| *** rwsu has quit IRC | 01:54 | |
| *** fandi has joined #openstack-infra | 01:57 | |
| *** rcernin has joined #openstack-infra | 01:58 | |
| *** Guest72255 has joined #openstack-infra | 01:59 | |
| *** rwsu has joined #openstack-infra | 02:00 | |
| *** Hal has quit IRC | 02:11 | |
| *** Guest72255 has quit IRC | 02:11 | |
| *** Hal has joined #openstack-infra | 02:13 | |
| *** lucas-afk has quit IRC | 02:23 | |
| *** lucasagomes has joined #openstack-infra | 02:25 | |
| *** Julien-zte has quit IRC | 02:26 | |
| *** baoli has joined #openstack-infra | 02:28 | |
| *** yamamoto has joined #openstack-infra | 02:29 | |
| *** Hal has quit IRC | 02:29 | |
| *** baoli_ has joined #openstack-infra | 02:30 | |
| *** baoli has quit IRC | 02:33 | |
| *** thorst has joined #openstack-infra | 02:38 | |
| *** amotoki has joined #openstack-infra | 02:39 | |
| *** gouthamr has quit IRC | 02:41 | |
| *** rcernin has quit IRC | 02:45 | |
| *** cody-somerville has joined #openstack-infra | 02:46 | |
| *** Hal has joined #openstack-infra | 02:48 | |
| *** csomerville has quit IRC | 02:49 | |
| *** baoli_ has quit IRC | 02:50 | |
| *** baoli has joined #openstack-infra | 02:51 | |
| *** dave-mccowan has joined #openstack-infra | 02:55 | |
| *** thorst has quit IRC | 02:57 | |
| *** dave-mcc_ has quit IRC | 02:57 | |
| *** sekelso has joined #openstack-infra | 02:58 | |
| *** john-davidge has joined #openstack-infra | 02:58 | |
| *** amotoki has quit IRC | 03:00 | |
| *** ramishra has joined #openstack-infra | 03:00 | |
| *** john-davidge has quit IRC | 03:03 | |
| *** dave-mccowan has quit IRC | 03:05 | |
| *** namnh has quit IRC | 03:06 | |
| *** tjones has joined #openstack-infra | 03:08 | |
| *** tjones has quit IRC | 03:08 | |
| *** baoli has quit IRC | 03:16 | |
| *** sree has joined #openstack-infra | 03:20 | |
| *** amotoki has joined #openstack-infra | 03:20 | |
| *** mriedem has quit IRC | 03:21 | |
| *** vikrant has joined #openstack-infra | 03:23 | |
| *** sree has quit IRC | 03:28 | |
| *** sree has joined #openstack-infra | 03:28 | |
| *** amotoki has quit IRC | 03:29 | |
| *** aeng has quit IRC | 03:31 | |
| *** sree has quit IRC | 03:33 | |
| *** sekelso has quit IRC | 03:35 | |
| *** yamamoto has quit IRC | 03:37 | |
| *** sbezverk has quit IRC | 03:38 | |
| *** amotoki has joined #openstack-infra | 03:38 | |
| *** aeng has joined #openstack-infra | 03:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-planet master: Rotate planet log file https://review.openstack.org/452589 | 03:50 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-planet master: Up cron to 15 minute runs https://review.openstack.org/452590 | 03:50 |
| *** thorst has joined #openstack-infra | 03:54 | |
| *** dharinic has joined #openstack-infra | 03:54 | |
| *** aeng has quit IRC | 03:54 | |
| openstackgerrit | Hironori Shiina proposed openstack/diskimage-builder master: Fix code-block in README of rhel7 element https://review.openstack.org/452591 | 03:57 |
| *** thorst has quit IRC | 03:58 | |
| *** Dinesh_Bhor has joined #openstack-infra | 04:01 | |
| *** amotoki has quit IRC | 04:01 | |
| *** kaisers has joined #openstack-infra | 04:04 | |
| *** aeng has joined #openstack-infra | 04:07 | |
| *** amotoki has joined #openstack-infra | 04:08 | |
| *** yamamoto has joined #openstack-infra | 04:16 | |
| *** persia has quit IRC | 04:17 | |
| *** psachin has joined #openstack-infra | 04:26 | |
| *** fandi__ has joined #openstack-infra | 04:41 | |
| *** dhajare has joined #openstack-infra | 04:43 | |
| *** fandi has quit IRC | 04:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: openSUSE mirror : fix pathes https://review.openstack.org/452594 | 04:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: openSUSE mirror : fix paths https://review.openstack.org/452594 | 04:48 |
| *** kaisers has quit IRC | 04:49 | |
| *** cody-somerville has quit IRC | 04:51 | |
| *** thorst has joined #openstack-infra | 04:54 | |
| *** rcernin has joined #openstack-infra | 04:56 | |
| *** thorst has quit IRC | 04:59 | |
| *** kaisers1 has joined #openstack-infra | 05:04 | |
| *** flepied has quit IRC | 05:06 | |
| *** knangia has joined #openstack-infra | 05:07 | |
| *** bexelbie has quit IRC | 05:09 | |
| *** bexelbie has joined #openstack-infra | 05:10 | |
| *** hrubi has quit IRC | 05:10 | |
| *** pgadiya has joined #openstack-infra | 05:11 | |
| *** dharinic has quit IRC | 05:12 | |
| ianw | #success planet.openstack.org migrated to xenial and running on vexxhost | 05:12 |
| openstackstatus | ianw: Added success to Success page | 05:12 |
| *** kaisers1 has quit IRC | 05:15 | |
| *** hrubi has joined #openstack-infra | 05:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update to mention planet01 https://review.openstack.org/452600 | 05:17 |
| *** claudiub has joined #openstack-infra | 05:17 | |
| *** rcernin has quit IRC | 05:18 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack-infra/project-config master: Storlets: Use python-jobs instead of parts indivisually https://review.openstack.org/452601 | 05:18 |
| *** ihrachys has joined #openstack-infra | 05:21 | |
| *** asselin has joined #openstack-infra | 05:21 | |
| *** ihrachys has quit IRC | 05:22 | |
| *** ihrachys has joined #openstack-infra | 05:22 | |
| *** asselin_ has quit IRC | 05:23 | |
| *** dimtruck is now known as zz_dimtruck | 05:24 | |
| *** zz_dimtruck is now known as dimtruck | 05:24 | |
| *** prateek has joined #openstack-infra | 05:29 | |
| *** rcernin has joined #openstack-infra | 05:31 | |
| *** kaisers has joined #openstack-infra | 05:32 | |
| *** dimtruck is now known as zz_dimtruck | 05:33 | |
| *** IRCFrEAK has joined #openstack-infra | 05:36 | |
| *** IRCFrEAK has left #openstack-infra | 05:37 | |
| *** rbuzatu has quit IRC | 05:39 | |
| *** rbuzatu has joined #openstack-infra | 05:39 | |
| *** rbuzatu has quit IRC | 05:39 | |
| *** rbuzatu has joined #openstack-infra | 05:39 | |
| *** kaisers has quit IRC | 05:41 | |
| *** IRCFrEAK has joined #openstack-infra | 05:43 | |
| *** IRCFrEAK has left #openstack-infra | 05:44 | |
| *** e0ne has joined #openstack-infra | 05:45 | |
| openstackgerrit | Lenny Verkhovsky proposed openstack-infra/puppet-openstackci master: added Single Use Slave plugin https://review.openstack.org/450296 | 05:47 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Change mutex to counting semaphore https://review.openstack.org/442563 | 05:51 |
| *** e0ne has quit IRC | 05:52 | |
| *** adriant has quit IRC | 05:55 | |
| *** thorst has joined #openstack-infra | 05:55 | |
| *** yamahata has joined #openstack-infra | 05:56 | |
| *** kaisers has joined #openstack-infra | 05:56 | |
| *** rbuzatu_ has joined #openstack-infra | 05:57 | |
| *** florianf has joined #openstack-infra | 05:57 | |
| *** thorst has quit IRC | 05:59 | |
| *** rbuzatu has quit IRC | 06:00 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update to mention planet01 https://review.openstack.org/452600 | 06:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove planet.openstack.org https://review.openstack.org/452604 | 06:01 |
| *** rbuzatu_ has quit IRC | 06:01 | |
| *** kaisers has quit IRC | 06:03 | |
| *** oomichi has quit IRC | 06:03 | |
| *** oomichi has joined #openstack-infra | 06:04 | |
| *** pgadiya has quit IRC | 06:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Thats what I'm trying to tell you, kid. It ain't there. It's been totally blown away. https://review.openstack.org/452604 | 06:04 |
| openstackgerrit | Hong Hui Xiao proposed openstack-infra/project-config master: [WIP] Add bgp to dragonflow fullstack job https://review.openstack.org/452605 | 06:04 |
| *** makowals_ has quit IRC | 06:05 | |
| *** sree has joined #openstack-infra | 06:07 | |
| *** gcb has quit IRC | 06:07 | |
| *** yuanying has quit IRC | 06:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Link to Infra Manual for CI problems https://review.openstack.org/452287 | 06:08 |
| *** AJaeger has quit IRC | 06:10 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Explain OpenStack CI Status https://review.openstack.org/452285 | 06:12 |
| *** yuanying_ has joined #openstack-infra | 06:13 | |
| *** AJaeger has joined #openstack-infra | 06:14 | |
| *** mkoderer has joined #openstack-infra | 06:15 | |
| AJaeger | ianw, could you review https://review.openstack.org/#/q/topic:unpuppet-zuul-workers , please? This moves packages around a bit for dib. | 06:15 |
| *** kaisers1 has joined #openstack-infra | 06:19 | |
| Dinesh_Bhor | AJaeger: Hi, do you have some time? | 06:23 |
| *** e0ne has joined #openstack-infra | 06:23 | |
| AJaeger | Dinesh_Bhor: what's up? | 06:23 |
| *** isaacb has joined #openstack-infra | 06:24 | |
| Dinesh_Bhor | AJaeger: About this: https://review.openstack.org/#/c/451668/ After this patch and it's child https://review.openstack.org/#/c/451669/ got merged I can see patches submitted by bot job to masakari, python-masakariclient. | 06:24 |
| Dinesh_Bhor | AJaeger: but I don't see any patch for masakari-monitors | 06:24 |
| *** e0ne has quit IRC | 06:24 | |
| Dinesh_Bhor | AJaeger: could you please help me to debug this? | 06:25 |
| *** cshastri has joined #openstack-infra | 06:25 | |
| AJaeger | Dinesh_Bhor: you have to look at the requirements post job. | 06:25 |
| *** rbuzatu has joined #openstack-infra | 06:26 | |
| AJaeger | Last change merged as hash of fffff013eeee72038a29c3c3d4c4ec1b31b7b324, so using the formula logs.o.o/first two chars of hash/full hash/ gives us http://logs.openstack.org/ff/fffff013eeee72038a29c3c3d4c4ec1b31b7b324/ | 06:26 |
| openstackgerrit | Sreeram Vancheeswaran proposed openstack-infra/project-config master: Enable functional test jobs for nova-dpm https://review.openstack.org/450700 | 06:27 |
| AJaeger | Dinesh_Bhor: and that leads to http://logs.openstack.org/ff/fffff013eeee72038a29c3c3d4c4ec1b31b7b324/post/propose-requirements-updates/6716124/console.html#_2017-04-02_11_26_55_758643 | 06:27 |
| AJaeger | Dinesh_Bhor: your error then is http://logs.openstack.org/ff/fffff013eeee72038a29c3c3d4c4ec1b31b7b324/post/propose-requirements-updates/6716124/console.html#_2017-04-02_11_26_57_355816 | 06:27 |
| *** vikrant has quit IRC | 06:28 | |
| *** nadya has joined #openstack-infra | 06:28 | |
| *** dhajare has quit IRC | 06:28 | |
| *** vikrant has joined #openstack-infra | 06:28 | |
| AJaeger | Dinesh_Bhor: there's an #openstack-requirements channel if you have further questions | 06:28 |
| Dinesh_Bhor | AJaeger: ok, thanks a lot for the info | 06:29 |
| Dinesh_Bhor | AJaeger: it explains a lot | 06:29 |
| *** oomichi has quit IRC | 06:29 | |
| *** oomichi has joined #openstack-infra | 06:30 | |
| *** makowals has joined #openstack-infra | 06:30 | |
| *** sputnik13 has quit IRC | 06:32 | |
| *** kaisers1 has quit IRC | 06:33 | |
| *** nadya has quit IRC | 06:33 | |
| *** sputnik13 has joined #openstack-infra | 06:34 | |
| *** tesseract has joined #openstack-infra | 06:34 | |
| *** ralonsoh has joined #openstack-infra | 06:43 | |
| *** dhajare has joined #openstack-infra | 06:44 | |
| *** chem_gon` is now known as chem | 06:45 | |
| *** chem has quit IRC | 06:45 | |
| *** chem has joined #openstack-infra | 06:45 | |
| *** jcoufal has joined #openstack-infra | 06:46 | |
| *** flepied has joined #openstack-infra | 06:49 | |
| *** andreas_s has joined #openstack-infra | 06:50 | |
| *** kaisers1 has joined #openstack-infra | 06:50 | |
| *** martinkopec has joined #openstack-infra | 06:51 | |
| *** thorst has joined #openstack-infra | 06:56 | |
| *** pcaruana has joined #openstack-infra | 06:57 | |
| *** jaosorior has joined #openstack-infra | 06:57 | |
| *** markus_z has joined #openstack-infra | 06:58 | |
| *** yatin_away is now known as yatinkarel | 06:58 | |
| *** bhavik1 has joined #openstack-infra | 07:00 | |
| *** aeng has quit IRC | 07:00 | |
| *** ccamacho has joined #openstack-infra | 07:00 | |
| *** thorst has quit IRC | 07:00 | |
| *** zz_dimtruck is now known as dimtruck | 07:01 | |
| *** kaisers1 has quit IRC | 07:02 | |
| *** bhavik1 has quit IRC | 07:02 | |
| *** neiljerram has joined #openstack-infra | 07:03 | |
| *** jcoufal has quit IRC | 07:04 | |
| *** shardy has joined #openstack-infra | 07:07 | |
| *** dimtruck is now known as zz_dimtruck | 07:11 | |
| *** matrohon has joined #openstack-infra | 07:11 | |
| *** kaisers1 has joined #openstack-infra | 07:17 | |
| *** markmcd has quit IRC | 07:18 | |
| *** neiljerram has quit IRC | 07:19 | |
| *** amoralej|off is now known as amoralej | 07:20 | |
| *** knangia has quit IRC | 07:31 | |
| *** hashar has joined #openstack-infra | 07:31 | |
| *** kaisers1 has quit IRC | 07:33 | |
| *** hashar has quit IRC | 07:36 | |
| *** stakeda has joined #openstack-infra | 07:37 | |
| *** amotoki_ has joined #openstack-infra | 07:41 | |
| *** jascott1_ has quit IRC | 07:42 | |
| *** dizquierdo has joined #openstack-infra | 07:42 | |
| *** hashar has joined #openstack-infra | 07:42 | |
| *** dhajare has quit IRC | 07:42 | |
| *** amotoki has quit IRC | 07:44 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Start building opensuse based nodepool images https://review.openstack.org/412065 | 07:48 |
| *** kaisers1 has joined #openstack-infra | 07:49 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/nodepool master: Remove link to modindex https://review.openstack.org/415496 | 07:50 |
| ianw | dirk: does http://mirror.ord.rax.openstack.org/opensuse/ current layout work? | 07:50 |
| *** sshnaidm is now known as sshnaidm|afk | 07:50 | |
| dirk | ianw: yep, current layout is also fine | 07:51 |
| *** jascott1_ has joined #openstack-infra | 07:52 | |
| ianw | dirk: ok, i'm going to merge the changes to do that and drop my lock when they're in. if it needs to change, just have to run a manual sync first i'd say | 07:53 |
| openstackgerrit | Merged openstack-infra/project-config master: [magnum] Add ocata to gerritbot https://review.openstack.org/450745 | 07:55 |
| openstackgerrit | Merged openstack-infra/project-config master: [magnum] Fix dib elements path https://review.openstack.org/451728 | 07:56 |
| *** thorst has joined #openstack-infra | 07:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove gate-novaclient-dsvm-functional-mitaka-ubuntu-trusty-nv https://review.openstack.org/452574 | 07:57 |
| openstackgerrit | Merged openstack-infra/project-config master: Add missing DEVSTACK_LOCAL_CONFIG+= for opensuse nodepool job https://review.openstack.org/452576 | 07:57 |
| openstackgerrit | Merged openstack-infra/system-config master: openSUSE mirror : fix paths https://review.openstack.org/452594 | 07:57 |
| *** jascott1_ has quit IRC | 07:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Tweak hooks for golang dsvm job https://review.openstack.org/452544 | 07:57 |
| openstackgerrit | Merged openstack-infra/project-config master: Publish LOCI tarball to tarballs.o.o https://review.openstack.org/452299 | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** jpich has joined #openstack-infra | 08:00 | |
| *** flepied has quit IRC | 08:01 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix code-block in README of rhel7 element https://review.openstack.org/452591 | 08:01 |
| *** zz_dimtruck is now known as dimtruck | 08:02 | |
| *** bhavik1 has joined #openstack-infra | 08:02 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** dhajare has joined #openstack-infra | 08:03 | |
| *** openstackgerrit has joined #openstack-infra | 08:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Enable multinode topology for slow Patrole tests. https://review.openstack.org/450278 | 08:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Add experimental murano functional test gate. https://review.openstack.org/451996 | 08:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Add separate ACL/group for castellan https://review.openstack.org/449138 | 08:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Skip pre-caching /deb- repositories in nodepool images https://review.openstack.org/437126 | 08:06 |
| openstackgerrit | Merged openstack-infra/project-config master: Uninstall postfix on nodes if it's there https://review.openstack.org/446094 | 08:06 |
| openstackgerrit | Merged openstack-infra/project-config master: Add a new project sushy-tools https://review.openstack.org/450685 | 08:06 |
| openstackgerrit | Merged openstack-infra/project-config master: [Manila] Add 2 jobs for manila-image-elements project https://review.openstack.org/444339 | 08:06 |
| openstackgerrit | Merged openstack-infra/project-config master: Run check-requirements on remaining powervm projects https://review.openstack.org/452224 | 08:06 |
| *** jordanP has joined #openstack-infra | 08:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update ARA's Ansible version with releases for 2.1.x, 2.2.x and 2.3.x https://review.openstack.org/450847 | 08:06 |
| *** e0ne has joined #openstack-infra | 08:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add install-guide-jobs for murano installation guide. https://review.openstack.org/452272 | 08:09 |
| openstackgerrit | Merged openstack-infra/project-config master: add coverage-jobs to congress https://review.openstack.org/452016 | 08:09 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove tripleo-test-cloud-rh2 from nodepool https://review.openstack.org/451115 | 08:09 |
| *** shardy has quit IRC | 08:09 | |
| *** rossella_s has joined #openstack-infra | 08:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Made ironic-dsvm-standalone voting https://review.openstack.org/450822 | 08:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Enable Octavia gate with Centos amphora https://review.openstack.org/449741 | 08:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Enable LXD test on Pike+ https://review.openstack.org/451935 | 08:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Add ocata and pike UCA repos https://review.openstack.org/451491 | 08:11 |
| *** dimtruck is now known as zz_dimtruck | 08:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Allow installing proliant tools from an OS package https://review.openstack.org/344214 | 08:12 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Manage NTP on zuul workers in DIB https://review.openstack.org/450971 | 08:13 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Install more packages with DIB https://review.openstack.org/449955 | 08:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Allow installing proliant tools from an OS package https://review.openstack.org/344214 | 08:14 |
| *** alexchadin has joined #openstack-infra | 08:15 | |
| *** thorst has quit IRC | 08:16 | |
| *** dingyichen has quit IRC | 08:18 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire os-cloud-config https://review.openstack.org/451093 | 08:18 |
| *** jtomasek has joined #openstack-infra | 08:22 | |
| *** nmathew has joined #openstack-infra | 08:24 | |
| *** alexchadin has quit IRC | 08:29 | |
| ianw | dirk / pabelanger : ok ... whatever the opensuse mirror is doing now, it's doing on it's own. so please keep an eye on it for a while to make sure it's working :) | 08:30 |
| *** arxcruz|pto is now known as arxcruz | 08:31 | |
| *** shardy has joined #openstack-infra | 08:31 | |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config master: Update Blazar's gate jobs https://review.openstack.org/452635 | 08:32 |
| *** s-shiono_ has quit IRC | 08:33 | |
| *** openstackgerrit has quit IRC | 08:33 | |
| *** s-shiono has joined #openstack-infra | 08:33 | |
| *** s-shiono has quit IRC | 08:33 | |
| *** mjblack has quit IRC | 08:33 | |
| *** openstackgerrit has joined #openstack-infra | 08:36 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/storyboard-webclient master: Change maxlength to 100 for projects and project_groups name https://review.openstack.org/452637 | 08:36 |
| openstackgerrit | Debayan Ray proposed openstack-infra/project-config master: Add coverage job to sushy https://review.openstack.org/452638 | 08:37 |
| *** rbuzatu has quit IRC | 08:37 | |
| *** mjblack has joined #openstack-infra | 08:38 | |
| *** bhavik1 has quit IRC | 08:39 | |
| *** derekh has joined #openstack-infra | 08:42 | |
| *** shardy has quit IRC | 08:46 | |
| *** yamahata has quit IRC | 08:48 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/storyboard master: Increase project and project group name length limitation https://review.openstack.org/445965 | 08:48 |
| *** shardy has joined #openstack-infra | 08:48 | |
| *** rbuzatu has joined #openstack-infra | 08:50 | |
| *** john-davidge has joined #openstack-infra | 08:51 | |
| *** ssbarnea has joined #openstack-infra | 08:51 | |
| *** felipemonteiro_ has joined #openstack-infra | 08:52 | |
| *** hashar_ has joined #openstack-infra | 08:52 | |
| *** mtreinish has quit IRC | 08:53 | |
| *** neiljerram has joined #openstack-infra | 08:54 | |
| *** hashar has quit IRC | 08:55 | |
| *** felipemonteiro has quit IRC | 08:55 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/storyboard master: Decrease projects and project_groups name MIN length limitation https://review.openstack.org/452641 | 08:55 |
| *** hashar_ is now known as hashar | 08:56 | |
| *** mtreinish has joined #openstack-infra | 08:57 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing TLS everywhere https://review.openstack.org/452645 | 09:00 |
| *** dizquierdo has quit IRC | 09:02 | |
| *** zz_dimtruck is now known as dimtruck | 09:02 | |
| *** neiljerram has quit IRC | 09:04 | |
| odyssey4me | hmm, I see that logs.openstack.org/periodic/ is still returning a 404 page - is that a known issue? | 09:06 |
| openstackgerrit | Masayuki Igawa proposed openstack/boartty master: Fix inconsistency name https://review.openstack.org/452648 | 09:08 |
| frickler | odyssey4me: seems to be working fine for me, maybe some caching issue? | 09:10 |
| *** alexchadin has joined #openstack-infra | 09:11 | |
| *** dimtruck is now known as zz_dimtruck | 09:12 | |
| *** thorst has joined #openstack-infra | 09:13 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder master: Add option to update all packages early on https://review.openstack.org/452653 | 09:13 |
| *** sshnaidm|afk is now known as sshnaidm | 09:14 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/devstack-gate master: Add more varibales to the reproduce.sh script https://review.openstack.org/452654 | 09:14 |
| *** thorst has quit IRC | 09:17 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: testing tempest for oooq https://review.openstack.org/452655 | 09:19 |
| *** electrofelix has joined #openstack-infra | 09:21 | |
| *** szaher has quit IRC | 09:25 | |
| *** amotoki_ has quit IRC | 09:25 | |
| *** sambetts|afk is now known as sambetts | 09:25 | |
| *** cuongnv has joined #openstack-infra | 09:25 | |
| openstackgerrit | Roman Podoliaka proposed openstack-infra/project-config master: osc-placement: experimental job for running of functional tests https://review.openstack.org/452210 | 09:27 |
| *** tosky has joined #openstack-infra | 09:27 | |
| cuongnv | hi guys, I have a quesion related to jenkins jobs, how can I know which OS version jenkins is running for my test? | 09:27 |
| cuongnv | this is output of my test http://logs.openstack.org/11/437311/8/check/gate-neutron-fwaas-python27-ubuntu-xenial/681ac11/console.html.gz | 09:28 |
| cuongnv | I can guess it's running xenial, but how can I know exact which version or where jenkins pulls ubuntu from? | 09:28 |
| odyssey4me | cuongnv you'd have to add some sort of patch which shows whatever you're looking for | 09:29 |
| AJaeger | cuongnv: our tests use a local mirror of the ubuntu repos | 09:29 |
| odyssey4me | but the repo used is an up to date ubuntu mirror which has been mirrored in each provider | 09:30 |
| *** szaher has joined #openstack-infra | 09:30 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder master: Add option to update all packages early on https://review.openstack.org/452653 | 09:30 |
| odyssey4me | the OS images are built regularly - I think perhaps weekly, but not sure | 09:30 |
| AJaeger | cuongnv: the log files even point to the mirror so you can check them. | 09:30 |
| AJaeger | odyssey4me: daily builds | 09:30 |
| AJaeger | cuongnv: what is your real problem? Why do you ask? | 09:31 |
| odyssey4me | infra-root I see that logs.openstack.org/periodic/ is still returning a 404 page - is that a known issue? | 09:31 |
| cuongnv | AJaeger, I'm having a problem related to a shared library with the OS, just to make sure my local env is the same with OS image which jenkins running | 09:32 |
| AJaeger | odyssey4me: I have the same experience as frickler above: It just works for me. | 09:32 |
| *** szaher has quit IRC | 09:33 | |
| *** prateek has quit IRC | 09:33 | |
| odyssey4me | oh really? hmm, I must have something stuck in my browser cache then | 09:33 |
| odyssey4me | I appear to be missing messages - I didn't see frickler's response. :/ | 09:33 |
| cuongnv | AJaeger, do you mean this mirror? http://logs.openstack.org/11/437311/8/check/gate-neutron-fwaas-python27-ubuntu-xenial/681ac11/console.html.gz#_2017-03-31_10_36_52_184518 | 09:34 |
| AJaeger | cuongnv: yes | 09:34 |
| AJaeger | odyssey4me: frickler answered 4 minutes after your first question | 09:35 |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config master: Enable ironic service in congress gate job https://review.openstack.org/452110 | 09:36 |
| odyssey4me | ah yes, after clearing my cache it's there - thanks frickler & AJaeger | 09:37 |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder master: Add option to update all packages early on https://review.openstack.org/452653 | 09:37 |
| *** szaher has joined #openstack-infra | 09:39 | |
| AJaeger | glad to hear, odyssey4me | 09:39 |
| cuongnv | AJaeger, odyssey4me ok thank you, it seems my vagrant box is quite same with the one used by jenkins | 09:41 |
| *** prateek has joined #openstack-infra | 09:47 | |
| *** prateek has quit IRC | 09:47 | |
| *** prateek has joined #openstack-infra | 09:47 | |
| *** alexchadin has quit IRC | 09:51 | |
| *** stakeda has quit IRC | 09:52 | |
| chem | hi, I've got a strange message from Jenkins in https://review.openstack.org/#/c/452643/1 and https://review.openstack.org/#/c/452663/1 : "This change depends on a change that failed to merge.", how can I debug that ? | 09:52 |
| chem | currently I blindly recheck it | 09:53 |
| jordanP | chem, check that the patch(s) your patch depends on are mergeable | 09:53 |
| *** jascott1_ has joined #openstack-infra | 09:53 | |
| chem | jordanP: well .. but it is https://review.openstack.org/#/c/452630/ | 09:53 |
| *** alexchadin has joined #openstack-infra | 09:54 | |
| chem | jordanP: hi, by the way. Thanks for replying :) | 09:55 |
| jordanP | hi :), yeah that's weird | 09:56 |
| *** prateek has quit IRC | 09:56 | |
| *** prateek_ has joined #openstack-infra | 09:56 | |
| jordanP | what I usually do is isn"t of rechecking, I'd click the "rebase" button | 09:56 |
| AJaeger | jordanP: rebase will not help, recheck was correct | 09:56 |
| jordanP | rebase will recheck, so what's wrong ? | 09:57 |
| AJaeger | chem: this happens when you have a depends-on on another change that has not finished the jobs. | 09:57 |
| AJaeger | jordanP: you might be already completely rebased | 09:58 |
| jordanP | true | 09:58 |
| chem | AJaeger: ah, so it needs jenkins +1 before being able to run, right ? | 09:58 |
| AJaeger | chem: I think so - +1 on the job marked as depending on. But don't remember those details | 09:59 |
| *** jascott1_ has quit IRC | 09:59 | |
| chem | AJaeger: ack, oki, kinda make sense, but the error message could be better :) | 09:59 |
| chem | AJaeger: by the way, in the last commit I only update the commit message and it blanks the Jenkins+1, retriggering a "useless" test, can we fix that in some way ? | 10:00 |
| *** prateek has joined #openstack-infra | 10:00 | |
| *** szaher has quit IRC | 10:01 | |
| *** prateek_ has quit IRC | 10:03 | |
| *** alexchad_ has joined #openstack-infra | 10:03 | |
| *** zz_dimtruck is now known as dimtruck | 10:03 | |
| *** szaher has joined #openstack-infra | 10:04 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack-infra/project-config master: Storlets: Use python-jobs instead of parts indivisually https://review.openstack.org/452601 | 10:05 |
| *** kuromagi^ has quit IRC | 10:05 | |
| *** rcernin has quit IRC | 10:05 | |
| *** fandi__ has quit IRC | 10:05 | |
| *** alexchadin has quit IRC | 10:06 | |
| *** prateek has quit IRC | 10:09 | |
| *** szaher has quit IRC | 10:10 | |
| *** dizquierdo has joined #openstack-infra | 10:11 | |
| *** szaher has joined #openstack-infra | 10:11 | |
| *** dimtruck is now known as zz_dimtruck | 10:13 | |
| *** prateek has joined #openstack-infra | 10:15 | |
| *** kuromagi has joined #openstack-infra | 10:16 | |
| chem | AJaeger: sorry to ping you over again ... just wondering if you could point me to the right direction to avoid triggering a recheck when only the commit message is updated ? | 10:19 |
| openstackgerrit | Debayan Ray proposed openstack-infra/project-config master: Add coverage job to sushy https://review.openstack.org/452638 | 10:22 |
| *** sdague has joined #openstack-infra | 10:23 | |
| *** nicolasbock has joined #openstack-infra | 10:23 | |
| *** rcernin has joined #openstack-infra | 10:24 | |
| *** cuongnv has quit IRC | 10:26 | |
| odyssey4me | chem you can't | 10:27 |
| odyssey4me | a commit message change is a patch change, so it will trigger the jobs | 10:27 |
| chem | odyssey4me: ah, hum ... that's too bad. If only we could detect only depends-on change in the commit message then we could avoid useless jobs. | 10:28 |
| *** wolverineav has joined #openstack-infra | 10:31 | |
| *** rbuzatu has quit IRC | 10:31 | |
| *** wolverineav has quit IRC | 10:31 | |
| *** rbuzatu has joined #openstack-infra | 10:31 | |
| *** wolverineav has joined #openstack-infra | 10:32 | |
| AJaeger | chem: and you might add/remove a depends-on and that changes the environment your job runs in. So, we have to test again. | 10:33 |
| *** psachin has quit IRC | 10:34 | |
| chem | AJaeger: ^ yeap, aggreed, but see my above comment to odyssey4me, it's only for those one that a re-test is necessary AFAICT. | 10:35 |
| *** pbourke has quit IRC | 10:35 | |
| *** rbuzatu has quit IRC | 10:35 | |
| odyssey4me | chem any change to a patch, no matter how minor, could change the validity of the patch | 10:36 |
| openstackgerrit | Debayan Ray proposed openstack-infra/project-config master: Add coverage job to proliantutils https://review.openstack.org/452686 | 10:36 |
| odyssey4me | the tests are essential | 10:36 |
| chem | odyssey4me: would be if anything was testing the validity of the commit message, so I guess that would make sense to have it configurable. | 10:37 |
| *** pbourke has joined #openstack-infra | 10:37 | |
| *** sree has quit IRC | 10:41 | |
| *** sree has joined #openstack-infra | 10:42 | |
| AJaeger | chem: a commit change is a new revision in gerrit and that's what we test. | 10:45 |
| AJaeger | chem: Yes, you could not run tests but then it might become complicated. This is not as trivial as it sounds. | 10:46 |
| *** john-davidge has quit IRC | 10:46 | |
| chem | AJaeger: odyssey4me ack, it's oki, I just wanted to make sure that I was not missing something here. Thanks for the discussion. | 10:48 |
| *** psachin has joined #openstack-infra | 10:51 | |
| *** sree has quit IRC | 10:57 | |
| *** alexchad_ has quit IRC | 10:58 | |
| *** sree has joined #openstack-infra | 10:58 | |
| *** alexchadin has joined #openstack-infra | 10:58 | |
| *** alexchad_ has joined #openstack-infra | 10:58 | |
| *** alexchadin has quit IRC | 10:58 | |
| *** jkilpatr has joined #openstack-infra | 11:01 | |
| *** sree has quit IRC | 11:02 | |
| *** hashar is now known as hasharLunch | 11:04 | |
| *** zz_dimtruck is now known as dimtruck | 11:04 | |
| *** martinkopec has quit IRC | 11:05 | |
| *** arxcruz has quit IRC | 11:07 | |
| *** asselin has quit IRC | 11:07 | |
| *** asselin has joined #openstack-infra | 11:07 | |
| *** adarazs is now known as adarazs_lunch | 11:10 | |
| *** arxcruz has joined #openstack-infra | 11:12 | |
| *** mat128 has joined #openstack-infra | 11:12 | |
| *** mat128 has quit IRC | 11:13 | |
| *** dimtruck is now known as zz_dimtruck | 11:14 | |
| *** mat128 has joined #openstack-infra | 11:15 | |
| *** rbuzatu has joined #openstack-infra | 11:15 | |
| *** neiljerram has joined #openstack-infra | 11:20 | |
| *** lucasagomes is now known as lucas-hungry | 11:20 | |
| *** john-davidge has joined #openstack-infra | 11:22 | |
| *** ccamacho is now known as ccamacho|lunch | 11:24 | |
| *** rbuzatu has quit IRC | 11:24 | |
| *** rbuzatu has joined #openstack-infra | 11:25 | |
| *** john-davidge has quit IRC | 11:27 | |
| *** shaiton_ has joined #openstack-infra | 11:31 | |
| shaiton_ | Hi there, I would lik | 11:32 |
| shaiton_ | Hi there, I would like to signe the CLA on gerrit but I get a server Error "Cannot store contact information" | 11:33 |
| shaiton_ | any info on your side? | 11:33 |
| shaiton_ | I use a lot git-review and would like to give back some patches | 11:33 |
| *** thorst has joined #openstack-infra | 11:34 | |
| AJaeger | shaiton_: This needs your gerrit preferred e-mail address to match a primary e-mail address for a foundation individual member account. | 11:34 |
| AJaeger | shaiton_: If you already followed the instructions at http://docs.openstack.org/infra/manual/developers.html#account-setup and still get that, see https://ask.openstack.org/question/56720 for additional troubleshooting tips. | 11:34 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config master: gnocchi: remove unexisting jobs https://review.openstack.org/452721 | 11:35 |
| shaiton_ | how thank you AJaeger I missed the "fondation member" part.. | 11:36 |
| *** thorst has quit IRC | 11:36 | |
| AJaeger | Ah... Just register there, please. Thanks for contributing! | 11:36 |
| *** thingee has quit IRC | 11:36 | |
| *** ttx has quit IRC | 11:38 | |
| *** ttx has joined #openstack-infra | 11:38 | |
| *** shardy is now known as shardy_lunch | 11:38 | |
| *** dhellmann has quit IRC | 11:42 | |
| *** dhellmann has joined #openstack-infra | 11:42 | |
| *** tuanluong has quit IRC | 11:42 | |
| *** adam_g has quit IRC | 11:45 | |
| *** asselin_ has joined #openstack-infra | 11:46 | |
| *** adam_g has joined #openstack-infra | 11:47 | |
| *** neiljerram has quit IRC | 11:47 | |
| *** asselin__ has joined #openstack-infra | 11:48 | |
| *** thingee has joined #openstack-infra | 11:49 | |
| *** asselin has quit IRC | 11:49 | |
| *** jgr is now known as jgrassler | 11:49 | |
| *** asselin_ has quit IRC | 11:51 | |
| *** hasharLunch is now known as hashar | 11:52 | |
| *** jascott1_ has joined #openstack-infra | 11:56 | |
| *** ccamacho|lunch is now known as ccamacho | 11:56 | |
| *** psachin has quit IRC | 11:58 | |
| *** dprince has joined #openstack-infra | 12:00 | |
| *** thorst has joined #openstack-infra | 12:00 | |
| *** jascott1_ has quit IRC | 12:01 | |
| *** signed8bit has joined #openstack-infra | 12:03 | |
| *** hrubi has quit IRC | 12:04 | |
| *** signed8bit has quit IRC | 12:04 | |
| *** signed8bit has joined #openstack-infra | 12:04 | |
| *** zz_dimtruck is now known as dimtruck | 12:05 | |
| *** jaypipes has joined #openstack-infra | 12:05 | |
| *** trown|outtypewww is now known as trown | 12:05 | |
| *** lucas-hungry is now known as lucasagomes | 12:07 | |
| *** HeOS has joined #openstack-infra | 12:07 | |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config master: Promote Upgrade jobs to voting for OpenStack-Ansible https://review.openstack.org/450198 | 12:07 |
| *** adarazs_lunch is now known as adarazs | 12:08 | |
| shaiton_ | it's all good now AJaeger thanks. | 12:10 |
| *** prateek has quit IRC | 12:10 | |
| *** psachin has joined #openstack-infra | 12:11 | |
| *** ricardoas has left #openstack-infra | 12:12 | |
| *** rhallisey has joined #openstack-infra | 12:13 | |
| *** ldnunes has joined #openstack-infra | 12:13 | |
| *** mat128 has quit IRC | 12:14 | |
| *** dimtruck is now known as zz_dimtruck | 12:15 | |
| *** rbuzatu has quit IRC | 12:16 | |
| *** jcoufal has joined #openstack-infra | 12:19 | |
| *** gouthamr has joined #openstack-infra | 12:21 | |
| *** ssbarnea has quit IRC | 12:23 | |
| jkilpatr | anyone else seeing this? | 12:27 |
| jkilpatr | http://logs.openstack.org/99/445599/17/gate/gate-browbeat-python27-ubuntu-xenial/77894aa/console.html | 12:27 |
| *** thingee has quit IRC | 12:27 | |
| jkilpatr | 2017-04-03 11:51:53.677984 | error: Download error for http://mirror.regionone.infracloud-vanilla.openstack.org/pypi/packages/b6/61/7b374462d5b6b1d824977182db287758d549d8680444bad8d530195acba2/requests-2.12.5.tar.gz#md5=360bb3af97881a7056ed5bb3d0a3e8a3: timed out | 12:27 |
| jkilpatr | did it twice in a row (recheck) | 12:27 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Change mogan tempest job to use ipmitool driver https://review.openstack.org/452735 | 12:28 |
| *** ssbarnea has joined #openstack-infra | 12:29 | |
| pabelanger | jkilpatr: looks like an issue with pypi mirrors | 12:31 |
| pabelanger | I can take a quick look | 12:31 |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=4965&rra_id=all | 12:32 |
| pabelanger | network traffic looks odd | 12:32 |
| *** rlandy has joined #openstack-infra | 12:32 | |
| pabelanger | http://mirror.regionone.osic-cloud1.openstack.org | 12:33 |
| pabelanger | look a while to load for me too | 12:33 |
| *** adam_g has quit IRC | 12:35 | |
| pabelanger | load average: 0.49, 12.34, 27.78 | 12:35 |
| pabelanger | ya, server was underload | 12:35 |
| pabelanger | but coming back down | 12:35 |
| pabelanger | I wonder if apache2 was waiting on afs again | 12:35 |
| *** jaosorior is now known as jaosorior_brb | 12:36 | |
| pabelanger | Apr 3 12:28:14 mirror kernel: [15062560.448790] afs: Waiting for busy volume 536870932 () in cell openstack.org | 12:36 |
| pabelanger | yup | 12:36 |
| pabelanger | jkilpatr: recheck again | 12:36 |
| *** adam_g has joined #openstack-infra | 12:36 | |
| jkilpatr | pabelanger, this was on a gate, so unworkflow or reworkflow or does "recheck ci" work? | 12:37 |
| *** aysyd has joined #openstack-infra | 12:37 | |
| jkilpatr | and reworkflow* | 12:37 |
| * dims peeks | 12:37 | |
| pabelanger | jkilpatr: recheck should be enough | 12:37 |
| jkilpatr | pabelanger, cool, thanks for the help! | 12:37 |
| dims | pabelanger : same problem on rax too? | 12:38 |
| dims | pabelanger : http://logs.openstack.org/90/452390/1/gate/gate-futurist-python27-ubuntu-xenial/3001dc6/console.html#_2017-04-03_11_55_49_304980 | 12:38 |
| pabelanger | dims: ya, I think we are ddos'ng afs | 12:38 |
| dims | ack :) | 12:38 |
| pabelanger | this happened last week too | 12:38 |
| *** mdrabe has joined #openstack-infra | 12:39 | |
| *** neiljerram has joined #openstack-infra | 12:39 | |
| dims | ack thanks for confirming pabelanger | 12:39 |
| *** signed8bit is now known as signed8bit_Zzz | 12:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add redirect for shade docs https://review.openstack.org/452737 | 12:40 |
| *** tonytan4ever has quit IRC | 12:41 | |
| *** rbuzatu has joined #openstack-infra | 12:42 | |
| *** Julien-zte has joined #openstack-infra | 12:42 | |
| *** thingee has joined #openstack-infra | 12:44 | |
| AJaeger | mordred: we have all other redirects in openstack-manuals, what about adding your changes there as well? ^ | 12:45 |
| pabelanger | infra-root: I have an appointment with customs / immigration this morning to complete my NEXUS pass. As such, I'll be offline for most of the morning. | 12:45 |
| *** shardy_lunch is now known as shardy | 12:45 | |
| mordred | AJaeger: oh - neat - I will go look | 12:47 |
| *** rbuzatu has quit IRC | 12:50 | |
| mordred | AJaeger: remote: https://review.openstack.org/452740 Add redirect for shade docs <-- how's that look? | 12:51 |
| AJaeger | mordred: reviewed and suggested a better way | 12:52 |
| *** mjblack has quit IRC | 12:52 | |
| *** vikrant has quit IRC | 12:53 | |
| * mordred learns this morning | 12:53 | |
| mordred | whoops! | 12:53 |
| *** signed8bit_Zzz is now known as signed8bit | 12:53 | |
| mordred | AJaeger: hopefully that one looks better now | 12:54 |
| *** amoralej is now known as amoralej|lunch | 12:54 | |
| *** mjblack has joined #openstack-infra | 12:55 | |
| *** smarcet has joined #openstack-infra | 12:56 | |
| *** smarcet has quit IRC | 12:56 | |
| *** smarcet has joined #openstack-infra | 12:56 | |
| *** smarcet has quit IRC | 12:57 | |
| *** smarcet has joined #openstack-infra | 12:57 | |
| *** kgiusti has joined #openstack-infra | 12:57 | |
| *** smarcet has quit IRC | 12:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Re-add magnumclient to shade's magnum job https://review.openstack.org/452744 | 12:58 |
| *** smarcet has joined #openstack-infra | 12:58 | |
| *** smarcet has quit IRC | 12:58 | |
| AJaeger | mordred: add some '()'s | 12:58 |
| *** smarcet has joined #openstack-infra | 12:58 | |
| *** smarcet has quit IRC | 12:59 | |
| *** smarcet has joined #openstack-infra | 12:59 | |
| *** smarcet has quit IRC | 12:59 | |
| mordred | AJaeger: wow. I really am on first coffee aren't I? | 12:59 |
| *** smarcet_ has joined #openstack-infra | 12:59 | |
| *** baoli has joined #openstack-infra | 13:00 | |
| *** askb has quit IRC | 13:00 | |
| *** smarcet_ has quit IRC | 13:00 | |
| *** smarcet_ has joined #openstack-infra | 13:00 | |
| dhellmann | hi, folks! I wanted to check in and make sure things are running smoothly, as far as we know, before I start approving releases. Are we all clear this morning? | 13:00 |
| *** smarcet_ has quit IRC | 13:00 | |
| *** smarcet_ has joined #openstack-infra | 13:01 | |
| *** florianf has quit IRC | 13:02 | |
| *** tosky has quit IRC | 13:02 | |
| *** tosky has joined #openstack-infra | 13:02 | |
| *** smarcet_ has quit IRC | 13:02 | |
| AJaeger | mordred: no comments ;) | 13:03 |
| *** smarcet_ has joined #openstack-infra | 13:03 | |
| *** smarcet_ has quit IRC | 13:03 | |
| *** smarcet_ has joined #openstack-infra | 13:03 | |
| *** zz_dimtruck is now known as dimtruck | 13:06 | |
| *** john-davidge has joined #openstack-infra | 13:06 | |
| *** smarcet_ has quit IRC | 13:08 | |
| *** rfolco has joined #openstack-infra | 13:09 | |
| *** john-davidge has quit IRC | 13:11 | |
| mordred | dhellmann: there was a pypi mirror issue a little earlier, but I tihnk it's passed ... pabelanger ? | 13:11 |
| *** florianf has joined #openstack-infra | 13:12 | |
| *** sekelso has joined #openstack-infra | 13:13 | |
| *** pradk has quit IRC | 13:14 | |
| *** esberglu has joined #openstack-infra | 13:15 | |
| *** dimtruck is now known as zz_dimtruck | 13:15 | |
| *** skelso has joined #openstack-infra | 13:16 | |
| pabelanger | mordred: dhellmann: yes, we had a busy AFS volume that was causing issues for serving things over apache | 13:16 |
| pabelanger | looks to be better now | 13:16 |
| pabelanger | however, this is about the 2nd / 3rd time it has happened recently | 13:16 |
| dhellmann | pabelanger : ok, it sounds like it's safe to release, but maybe go slowly so we don't end up with a bunch of failures if we do start having issues | 13:17 |
| *** sekelso has quit IRC | 13:17 | |
| pabelanger | dhellmann: seems like a good plan | 13:18 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Support built packages to be installed in image https://review.openstack.org/452755 | 13:18 |
| *** srobert has joined #openstack-infra | 13:18 | |
| *** mat128 has joined #openstack-infra | 13:19 | |
| *** rook|tower is now known as rook | 13:19 | |
| *** thingee has quit IRC | 13:23 | |
| *** dhellmann has quit IRC | 13:24 | |
| pabelanger | mordred: we could try bumping out timeout in pip.conf to see if that helps. I am also not sure if we can control the retry attempts either | 13:24 |
| electrofelix | zaro: wonder if you might be able to give https://review.openstack.org/202543 a review, I did some work on helping fix it up so not sure I should be approving it myself | 13:24 |
| *** thingee has joined #openstack-infra | 13:25 | |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze master: Remove swift support https://review.openstack.org/452316 | 13:25 |
| *** dhellmann has joined #openstack-infra | 13:25 | |
| mordred | sdague: awesome. one minor nit, while you're in there | 13:28 |
| sdague | mordred: sure, I've got the patch up now, so will be easy to rev | 13:28 |
| *** makowals_ has joined #openstack-infra | 13:28 | |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze master: Remove swift support https://review.openstack.org/452316 | 13:29 |
| sdague | done | 13:29 |
| *** baoli has quit IRC | 13:29 | |
| mordred | sdague: +2 | 13:29 |
| *** makowals has quit IRC | 13:30 | |
| *** baoli has joined #openstack-infra | 13:30 | |
| *** eharney has joined #openstack-infra | 13:31 | |
| *** beekneemech is now known as bnemec | 13:31 | |
| *** jaosorior_brb is now known as jaosorior | 13:33 | |
| *** nt has quit IRC | 13:34 | |
| *** asselin has joined #openstack-infra | 13:35 | |
| *** asselin_ has joined #openstack-infra | 13:37 | |
| *** asselin__ has quit IRC | 13:38 | |
| mnaser | has anyone seen any issues with rdo/centos mirrors today? | 13:39 |
| *** mattmceuen has joined #openstack-infra | 13:39 | |
| *** sekelso has joined #openstack-infra | 13:39 | |
| mnaser | i just saw 2 jobs timeout in kolla | 13:39 |
| mnaser | oraclelinux and centos which both share the rdo repos | 13:39 |
| mordred | mnaser: we had a bit of a slowdown/timeout issue about an hour ago? | 13:39 |
| mnaser | mordred sounds like around the same time these failed - https://review.openstack.org/#/c/452207/ | 13:40 |
| mnaser | recheck is ok or are things still iffy? | 13:40 |
| _ari_ | hashar: is you could review this when you have some cycles that would be great https://review.openstack.org/#/c/451871/5 | 13:41 |
| *** gcb has joined #openstack-infra | 13:41 | |
| *** asselin has quit IRC | 13:41 | |
| *** skelso has quit IRC | 13:41 | |
| mordred | mnaser: recheck should be ok - it seems to be resolved | 13:42 |
| *** dtantsur is now known as dtantsur|brb | 13:42 | |
| mnaser | thanks mordred | 13:42 |
| *** mat128 has quit IRC | 13:45 | |
| *** rossella_s has quit IRC | 13:46 | |
| *** rossella_s has joined #openstack-infra | 13:47 | |
| *** jcoufal has quit IRC | 13:52 | |
| *** hongbin has joined #openstack-infra | 13:53 | |
| *** lazyPower is now known as lp|swap | 13:56 | |
| *** knangia has joined #openstack-infra | 13:56 | |
| *** Goneri has joined #openstack-infra | 13:56 | |
| *** dizquierdo has quit IRC | 13:57 | |
| *** sree has joined #openstack-infra | 13:58 | |
| *** makowals_ has quit IRC | 13:58 | |
| *** sree_ has joined #openstack-infra | 14:00 | |
| *** sree_ is now known as Guest64432 | 14:01 | |
| *** isaacb has quit IRC | 14:01 | |
| *** makowals has joined #openstack-infra | 14:01 | |
| *** ihrachys has quit IRC | 14:02 | |
| *** rbrndt has joined #openstack-infra | 14:02 | |
| *** jkilpatr has quit IRC | 14:02 | |
| *** nmathew has quit IRC | 14:02 | |
| *** mrunge_ has joined #openstack-infra | 14:03 | |
| *** kgiusti has quit IRC | 14:04 | |
| *** sree has quit IRC | 14:04 | |
| *** amotoki has joined #openstack-infra | 14:05 | |
| *** kgiusti has joined #openstack-infra | 14:05 | |
| fungi | pabelanger: mordred: looks like it's a very large sync from afs01.dfw to afs02.dfw which started at ~13:10 and is ongoing? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2362&rra_id=all http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=6405&rra_id=all | 14:05 |
| *** Guest64432 has quit IRC | 14:06 | |
| fungi | outbound for afs01.dfw:eth0 and inbound for afs02.dfw:eth0 look to match up precisely | 14:06 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fix debian-jessie settings for devstack https://review.openstack.org/452491 | 14:06 |
| *** zz_dimtruck is now known as dimtruck | 14:06 | |
| fungi | interestingly, that seems to have started just as the impact cleared up? | 14:07 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove unused timing constants https://review.openstack.org/450770 | 14:07 |
| *** Julien-zte has quit IRC | 14:09 | |
| AJaeger | fungi, ianw set up an opensuse mirror today - not sure what time that was finished. Looked more like 8:30 if I interpret his comment correctly. | 14:09 |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: networking-bagpipe: add an experimental fullstack job https://review.openstack.org/451897 | 14:09 |
| fungi | ohh, i wonder if we're riding memory utilization too close on afs01.dfw... we've been running with about 1.5gb paged into swap, and there was a memory bump around 12:00 which seems to have resulted in a sizable swap activity spike | 14:09 |
| *** florianf has quit IRC | 14:09 | |
| *** sree has joined #openstack-infra | 14:10 | |
| fungi | ahh, right, we had this conversation the last time we were looking at that | 14:10 |
| *** mdrabe has quit IRC | 14:11 | |
| *** Julien-zte has joined #openstack-infra | 14:12 | |
| fungi | definite spike in load average and swapping, with the memory bump lasting from around 11:35 to 13:10 (coincidentally ending when the large volume of sync traffic began) | 14:13 |
| *** smarcet has joined #openstack-infra | 14:13 | |
| *** EricGonczer_ has joined #openstack-infra | 14:13 | |
| *** sree has quit IRC | 14:14 | |
| *** mdrabe has joined #openstack-infra | 14:17 | |
| *** pradk has joined #openstack-infra | 14:18 | |
| *** dimtruck is now known as zz_dimtruck | 14:19 | |
| *** amoralej|lunch is now known as amoralej | 14:20 | |
| *** Julien-zte has quit IRC | 14:23 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Set REMAINING_TIME for local run https://review.openstack.org/452542 | 14:23 |
| *** ricardoas has joined #openstack-infra | 14:24 | |
| *** florianf has joined #openstack-infra | 14:24 | |
| *** amotoki has quit IRC | 14:25 | |
| *** Julien-zte has joined #openstack-infra | 14:27 | |
| *** Julien-zte has quit IRC | 14:27 | |
| *** Julien-zte has joined #openstack-infra | 14:27 | |
| *** Julien-zte has quit IRC | 14:27 | |
| *** gordc has joined #openstack-infra | 14:29 | |
| *** EricGonc_ has joined #openstack-infra | 14:30 | |
| *** EricGonczer_ has quit IRC | 14:30 | |
| *** beslemon has joined #openstack-infra | 14:32 | |
| *** cshastri has quit IRC | 14:33 | |
| *** zz_dimtruck is now known as dimtruck | 14:33 | |
| smarcet | fungi: morning , how are u doing ? already sent u email with hiera variables to update for openstackid :) cheers | 14:35 |
| *** rbuzatu has joined #openstack-infra | 14:38 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Support built packages to be installed in image https://review.openstack.org/452755 | 14:38 |
| *** nicolasbock has quit IRC | 14:41 | |
| *** chlong has joined #openstack-infra | 14:45 | |
| *** ZZelle_ has quit IRC | 14:46 | |
| *** nicolasbock has joined #openstack-infra | 14:48 | |
| *** baoli has quit IRC | 14:49 | |
| *** marst has quit IRC | 14:49 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove link to modindex https://review.openstack.org/415496 | 14:50 |
| *** dtantsur|brb is now known as dtantsur | 14:54 | |
| *** marst has joined #openstack-infra | 14:56 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/irc-meetings master: Removed korzen from the list of neutron upgrades meeting chairs https://review.openstack.org/452782 | 14:56 |
| *** mriedem has joined #openstack-infra | 14:57 | |
| *** ihrachys has joined #openstack-infra | 14:58 | |
| *** amotoki has joined #openstack-infra | 14:59 | |
| *** sekelso has quit IRC | 15:01 | |
| *** sekelso has joined #openstack-infra | 15:01 | |
| fungi | smarcet: received... one concern though | 15:01 |
| *** xarses has joined #openstack-infra | 15:02 | |
| smarcet | fungi; sure tell me | 15:02 |
| fungi | smarcet: the reason we enter the trove db hostnames into hiera (and the reason rackspace makes them so random-looking in the first place i expect) is that any tenant with an instance in that region can reach any trove db for any other tenant--there is no network access control there | 15:03 |
| *** xarses_ has joined #openstack-infra | 15:03 | |
| fungi | smarcet: so if we instead move that to a dns cname we risk disclosing the ip address of the db and make it easier for attackers to target it with pre-authentication mysql vulnerabilities (which do crop up from time to time) | 15:04 |
| *** makowals has quit IRC | 15:04 | |
| smarcet | fungi: but inst ip address private one ? | 15:05 |
| fungi | smarcet: yes, a "private" address in the rfc 1918 sense, but on a network shared by every instance for every tenant in that rackspace region | 15:06 |
| fungi | so the potential set of attackers is of course reduces from "everyone on the internet" to "everyone with a credit card" | 15:06 |
| *** makowals has joined #openstack-infra | 15:06 | |
| fungi | er, reduced | 15:06 |
| *** xarses has quit IRC | 15:07 | |
| smarcet | fungi: i got your point | 15:07 |
| *** john-davidge has joined #openstack-infra | 15:07 | |
| smarcet | fungi;: i will speak with jp about it , thx for let me know :) | 15:07 |
| *** dizquierdo has joined #openstack-infra | 15:09 | |
| *** sree has joined #openstack-infra | 15:10 | |
| gcb | fungi: we moved castellan to oslo recently in https://review.openstack.org/449137. and add separate ACL/group for castellan in https://review.openstack.org/449138 | 15:11 |
| *** john-davidge has quit IRC | 15:12 | |
| gcb | fugi: we decided to both oslo and barbican core reviewers wll be that group, can you add me in https://review.openstack.org/#/admin/groups/1742,members then I can add others there | 15:13 |
| gcb | fungi | 15:13 |
| *** makowals has quit IRC | 15:13 | |
| *** skelso has joined #openstack-infra | 15:13 | |
| *** sree has quit IRC | 15:14 | |
| *** felipemonteiro has joined #openstack-infra | 15:16 | |
| *** hashar has quit IRC | 15:19 | |
| *** jamesdenton has joined #openstack-infra | 15:20 | |
| *** dave-mccowan has joined #openstack-infra | 15:20 | |
| *** baoli has joined #openstack-infra | 15:21 | |
| *** baoli has quit IRC | 15:24 | |
| *** jascott1_ has joined #openstack-infra | 15:25 | |
| *** adrian_otto has joined #openstack-infra | 15:26 | |
| EmilienM | pabelanger: re: puppetlabs mirror. The packages take more than 2GB, would it be ok? | 15:26 |
| gcb | fungi: this is my info "ChangBo Guo(gcb) glongwave@gmail.com", thanks | 15:28 |
| *** annegentle has joined #openstack-infra | 15:28 | |
| clarkb | gcb: I can do that in a moment | 15:28 |
| gcb | clarkb, thanks | 15:28 |
| clarkb | gcb: done | 15:29 |
| *** jascott1_ has quit IRC | 15:29 | |
| gcb | clarkb, cool :-) | 15:30 |
| *** nt has joined #openstack-infra | 15:30 | |
| *** tesseract has quit IRC | 15:31 | |
| *** baoli has joined #openstack-infra | 15:31 | |
| fungi | gcb: sorry, in a meeting right now but any of our ~dozen gerrit admins can do that if it's urgent | 15:32 |
| *** skelso has quit IRC | 15:32 | |
| *** HoloIRCUser has joined #openstack-infra | 15:32 | |
| gcb | fungi, it doesn't matter , clarkb have done for me , thank all the same :-) | 15:33 |
| *** thorst is now known as thorst_afk | 15:33 | |
| gcb | yeah, I should check infra-core firstly | 15:34 |
| jpmaxman | Fungi we can make it more random if it helps | 15:35 |
| *** rbuzatu has quit IRC | 15:35 | |
| jpmaxman | Given how often these maintenance events come up just seems easier to be able to control with DNS | 15:35 |
| fungi | jpmaxman: yeah, that's probably a fine compromise | 15:36 |
| rajinir | I'm new to using zookeeper with nodepool, how do I clean/delete the images so I can rebuilt them? | 15:38 |
| clarkb | rajinir: same way as pre zookeeper. Just nodepool image-delete them | 15:38 |
| clarkb | rajinir: as an alternative, don't explicitly delete them and instead explicitly build them and the will replace existing onces automatically | 15:39 |
| *** amotoki has quit IRC | 15:39 | |
| Shrews | rajinir: also, https://docs.openstack.org/infra/system-config/nodepool.html#bad-images | 15:40 |
| *** HoloIRCUser2 has joined #openstack-infra | 15:41 | |
| *** HoloIRCUser has quit IRC | 15:41 | |
| *** tonytan4ever has joined #openstack-infra | 15:43 | |
| *** mat128 has joined #openstack-infra | 15:43 | |
| jeblair | well, that's our sysadmin proceduces specific to openstack-infra, and while potentially useful, it doesn't apply to all cases. if the nodepool docs themselves don't say how to delete or build an image, we should fix that. | 15:43 |
| rajinir | <Shrews,clarkb, thank you will try | 15:43 |
| *** jordanP has quit IRC | 15:43 | |
| jeblair | (the point of that documentation is so that when someone screams at us that an image is broken, any infra-root knows what to do to fix that) | 15:44 |
| clarkb | https://docs.openstack.org/infra/nodepool/operation.html#image-delete the command is documented but not general use case stuff | 15:45 |
| Shrews | jeblair: yes, but i think the info likely would be helpful to rajinir since it's mostly generic in how the builder operates | 15:45 |
| clarkb | (important to note that this hasn't changed as part of zk work, that interface is the same) | 15:45 |
| jeblair | Shrews, clarkb: yes. i think the nodepool docs need improvement to actually narratively describe some of this. but i think it's important to note that the infra docs linked are specifically geared toward solving one particular problem and only incidentally describe the system. they recommend doing several steps that may not be necessary for rajinir unless they're running into exactly that situation. | 15:48 |
| rajinir | Clarkb,Shrews: I have issued the deletes, the status changed to deleteing.., but I don't see any sign of delete happening , I restarted nodepool, no sign of nodepool moving along, | 15:48 |
| clarkb | rajinir: do you have a nodepool builder process running? | 15:51 |
| *** ijw has joined #openstack-infra | 15:51 | |
| clarkb | rajinir: it is responsible for monitoring the state of images and taking action on them, building deleting being the big ones | 15:52 |
| *** ihrachys has quit IRC | 15:52 | |
| *** pcaruana has quit IRC | 15:53 | |
| *** ihrachys has joined #openstack-infra | 15:53 | |
| rajinir | clarkb: nodepool 10608 1 0 10:50 ? 00:00:00 /usr/bin/python /usr/local/bin/nodepoold --no-builder -c /etc/nodepool/nodepool.yaml -l /etc/nodepool/logging.conf -p | 15:53 |
| *** ihrachys has quit IRC | 15:53 | |
| rajinir | clarkb it is running, but no activity in nodepool.log, debug.log is saying no image found | 15:54 |
| rajinir | someone it looks hosed | 15:54 |
| *** ihrachys has joined #openstack-infra | 15:54 | |
| SamYaple | where might i find a history of jobs (specifically logs) from the 'post' queue? | 15:54 |
| clarkb | rajinir: thats --no-builder so there is no builder thread there. You need a separate process running nodepool-builder iirc | 15:54 |
| *** HoloIRCUser2 has quit IRC | 15:55 | |
| smarcet | fungi: i just sent u another email with the final settings, sorry for back and forwards :/ | 15:55 |
| rajinir | clarkb: I do this warning: WARNING nodepool.cmd.nodepoold: --no-builder is deprecated and will be removed in the near future. Update your service scripts to avoid a brea | 15:55 |
| rajinir | kage. | 15:55 |
| jeblair | clarkb, rajinir: yes, you need to start the "nodepool-builder" process as well. | 15:55 |
| *** andreas_s has quit IRC | 15:56 | |
| *** e0ne has quit IRC | 15:57 | |
| *** ihrachys has quit IRC | 15:58 | |
| openstackgerrit | Edgar Magana proposed openstack-infra/irc-meetings master: Update for the UC IRC meetings https://review.openstack.org/452812 | 15:58 |
| rajinir | clarkb: jebliar: Thanks, restarted both nodepool and nodepool-builder services. Will check if that helped | 15:59 |
| mordred | SamYaple: I feel like someone wrote a little utility to print out log urls for a given change - sdague do you remember that utility? | 15:59 |
| openstackgerrit | Edgar Magana proposed openstack-infra/irc-meetings master: Update for the UC IRC meetings https://review.openstack.org/452812 | 16:00 |
| sdague | mordred: I don't off the top of my head | 16:01 |
| *** cody-somerville has joined #openstack-infra | 16:02 | |
| jeblair | was it dhellmann? | 16:02 |
| SamYaple | hmmm ok well then a more direct question, is tehre a way to know if a post job failed? | 16:02 |
| *** matrohon has quit IRC | 16:03 | |
| jeblair | SamYaple: find the git sha for the commit in question (the merge commit if it made one), then its logs.openstack.org/AB/ABCD... (where AB is the first two chars of the git sha, and ABCD... is the entirety) | 16:03 |
| *** gcb has quit IRC | 16:03 | |
| clarkb | or use openstack health | 16:03 |
| *** mat128 has quit IRC | 16:04 | |
| clarkb | http://status.openstack.org/openstack-health/#/ | 16:04 |
| fungi | smarcet: jpmaxman: received! will set those now if you feel it's safe to do so | 16:04 |
| smarcet | fungi: thx a lot :) | 16:05 |
| *** yamahata has joined #openstack-infra | 16:05 | |
| mordred | SamYaple: there's also a build_queue tag in logstash | 16:05 |
| *** adrian_otto has quit IRC | 16:05 | |
| mordred | SamYaple: so like: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_queue%3A%5C%22post%5C%22 | 16:06 |
| mordred | jeblair: I tried to find a docs page with that log structure documented but couldn't. do you think it's something we should add to the infra-manual? | 16:08 |
| *** trown is now known as trown|lunch | 16:08 | |
| clarkb | mordred: its there https://docs.openstack.org/infra/manual/developers.html#automated-testing maybe just not organized well? | 16:09 |
| jeblair | i'd really like to switch post to change-merged, but we need the commit sha. :| | 16:10 |
| *** psachin has quit IRC | 16:10 | |
| mordred | jeblair: ++ | 16:11 |
| SamYaple | jeblair: thanks, that worked | 16:11 |
| fungi | smarcet: jpmaxman: it's updated on our puppetmaster and will likely be applied on openstackid-dev.openstack.org and openstackid.org in the next 15-30 minutes | 16:11 |
| smarcet | ok got it | 16:11 |
| smarcet | fungi: thx so much :) | 16:11 |
| mordred | clarkb: cool - let me make a patch that breaks that bit out a little ibt | 16:11 |
| jpmaxman | great thanks fungi :) | 16:13 |
| *** markus_z has quit IRC | 16:14 | |
| fungi | pabelanger: ianw: not sure which of you added opensuse_keytab to production/fqdn/mirror-update.openstack.org.yaml in the private yaml on puppetmaster.o.o, but it's not committed yet | 16:14 |
| fungi | er, private hiera | 16:14 |
| rajinir | clarkb, jeblair: something is still wrong, the images are deleted from the list, but images rebuild is not happening.. | 16:16 |
| *** mat128 has joined #openstack-infra | 16:18 | |
| jeblair | rajinir: is there any information in the nodepool-builder log? | 16:18 |
| *** jkilpatr has joined #openstack-infra | 16:18 | |
| *** dizquierdo has quit IRC | 16:19 | |
| *** d0ugal has quit IRC | 16:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Split out section about finding post-queue logs https://review.openstack.org/452824 | 16:20 |
| mordred | clarkb: how does that look? ^^ (also SamYaple) | 16:20 |
| dhellmann | sdague, jeblair, SamYaple : https://pypi.python.org/pypi?name=git-os-job&version=1.0.1&:action=display | 16:21 |
| mordred | dhellmann: that's it! | 16:21 |
| *** rbuzatu_ has joined #openstack-infra | 16:22 | |
| *** rbuzatu_ has quit IRC | 16:22 | |
| *** rbuzatu has joined #openstack-infra | 16:22 | |
| *** jpich has quit IRC | 16:24 | |
| *** mat128 has quit IRC | 16:27 | |
| rajinir | jebliar: The last message is delete image one, Deleting image build dpc-0000000009 from local_01 | 16:29 |
| rajinir | Jebliar: on issuing the command manually to build, I see this exception Exception: Skipping build request for image dpc; paused | 16:30 |
| rajinir | https://www.irccloud.com/pastebin/Z3FWOh4J/Image%20rebuild | 16:30 |
| clarkb | rajinir: that means you are configured to not build that image, it is "paused" | 16:30 |
| clarkb | rajinir: you can remove the pause from your config then it should build | 16:30 |
| *** mat128 has joined #openstack-infra | 16:31 | |
| rajinir | clarb: I'm trying to build the xenial image for first time, which config file has this pause? | 16:32 |
| rajinir | This is my image config details from the nodepool.yaml | 16:32 |
| rajinir | https://www.irccloud.com/pastebin/Mjh8hEIM/xenial | 16:32 |
| clarkb | rajinir: the nodepool.yaml | 16:32 |
| rajinir | clarkb: I see the pause is set to True | 16:33 |
| *** alexchad_ has quit IRC | 16:33 | |
| clarkb | rajinir: yup so its "paused" no new builds will happen while that is set | 16:33 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config master: [Manila] Fix building of custom image https://review.openstack.org/452826 | 16:33 |
| jeblair | rajinir: did you set it to True earlier? | 16:34 |
| rajinir | clarkb: Is there a standard config , that's documented for zenial? | 16:34 |
| rajinir | clarkb: xenial, | 16:34 |
| clarkb | rajinir: we document the way we build them, but if there is any standard its probably set by ubuntu itself ( and we definiotely don't do all the things they do) | 16:35 |
| rajinir | clarb: I removed the pause and I can see some errors now | 16:35 |
| *** derekh has quit IRC | 16:35 | |
| clarkb | s/document/publish our configs/ | 16:35 |
| *** smarcet has quit IRC | 16:41 | |
| *** smarcet has joined #openstack-infra | 16:42 | |
| *** thorst_afk is now known as thorst | 16:42 | |
| rajinir | clarkb: Thanks a bunch | 16:43 |
| *** mat128 has quit IRC | 16:43 | |
| *** ihrachys has joined #openstack-infra | 16:46 | |
| *** smarcet has quit IRC | 16:46 | |
| openstackgerrit | Merged openstack/os-client-config master: Docs: add a note about rackspace API keys https://review.openstack.org/451563 | 16:46 |
| *** smarcet has joined #openstack-infra | 16:47 | |
| *** rossella_s has quit IRC | 16:47 | |
| *** sambetts is now known as sambetts|afk | 16:50 | |
| *** beslemon has quit IRC | 16:50 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_queue_rate_limiting https://review.openstack.org/452835 | 16:51 |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/project-config master: Add job to Neutron DVR check queue panel https://review.openstack.org/452294 | 16:52 |
| openstackgerrit | Luz Cazares proposed openstack-infra/project-config master: Add Refstack jobs to publish documentation https://review.openstack.org/446613 | 16:52 |
| SamYaple | my jjb job templates do not have 'gerrit-git-prep', is this the builder responsible for cloning the project to the WORKDIR? | 16:52 |
| *** annegentle has quit IRC | 16:53 | |
| *** e0ne has joined #openstack-infra | 16:55 | |
| *** aarefiev_afk is now known as aarefiev | 16:56 | |
| *** lucasagomes is now known as lucas-afk | 16:56 | |
| SamYaple | ah yea. it is | 16:56 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Add scenario005 job (4nodes) https://review.openstack.org/418098 | 16:56 |
| zaro | electrofelix: will take a look tomorrow | 16:57 |
| *** mat128 has joined #openstack-infra | 16:59 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Fix LOCI jobs https://review.openstack.org/452846 | 17:00 |
| *** rbuzatu has quit IRC | 17:03 | |
| electrofelix | zaro: cheers | 17:03 |
| *** rlandy is now known as rlandy|biab | 17:03 | |
| *** rbuzatu has joined #openstack-infra | 17:03 | |
| *** dhajare has quit IRC | 17:04 | |
| *** jkilpatr has quit IRC | 17:05 | |
| *** chlong has quit IRC | 17:05 | |
| *** rbuzatu has quit IRC | 17:07 | |
| *** john-davidge has joined #openstack-infra | 17:08 | |
| *** adrian_otto has joined #openstack-infra | 17:10 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_queue_rate_limiting https://review.openstack.org/452835 | 17:10 |
| *** Apoorva has joined #openstack-infra | 17:11 | |
| cloudnull | o/ all, qq do we know of a difference between the images found in the osic for centos7 then other clouds? Asking because in OSA we're seeing erasurecode issue with swift on cent when gating on not the OSIC. | 17:12 |
| cloudnull | passing gate http://logs.openstack.org/38/451138/3/check/gate-openstack-ansible-os_swift-ansible-func-centos-7/4e21f15/ | 17:12 |
| cloudnull | failing gate http://logs.openstack.org/38/451138/3/check/gate-openstack-ansible-os_swift-ansible-func-centos-7/38dc4f2/ | 17:12 |
| clarkb | cloudnull: they should be the same with the difference being format | 17:13 |
| cloudnull | and it looks to be consistent rax fails, osic passes. | 17:13 |
| *** john-davidge has quit IRC | 17:13 | |
| clarkb | rackspace runs xen and accepts vhd images, osic is kvm and accepts qcow2, but dib produces both from the same "master" image | 17:13 |
| notmyname | cloudnull: please feel free to ask in -swift if there's something we need to look at | 17:13 |
| odyssey4me | it could possibly be an inconsistent mirror too, unless those packages are pre-installed into the image? | 17:13 |
| cloudnull | notmyname: IDK if it's a swift issues, seems to gate and run just fine on swift just rax cent7 is being angry | 17:14 |
| clarkb | odyssey4me: the mirrors should all be the same too since we use AFS to host them | 17:14 |
| *** d0ugal has joined #openstack-infra | 17:14 | |
| * cloudnull wow, total English failure. | 17:15 | |
| *** mat128 has quit IRC | 17:15 | |
| cloudnull | IDK if it's an issue with swift, seems to gate and run just fine locally, and on the osic for that matter, just rax cent7 is being angry. | 17:16 |
| AJaeger | mordred, fungi, speaking about docs, could you review https://review.openstack.org/#/q/status:open+++topic:failures , please? | 17:16 |
| clarkb | cloudnull: the fails I see don't seem to be related to erasurecoding? they are bad status lines on acl check tests | 17:16 |
| fungi | cloudnull: i wonder if it's a concurrency difference exposing something racy in some tests? | 17:17 |
| clarkb | cloudnull: which is possibly a https proxy fail? we had a similar issue with apache and neutron api | 17:17 |
| *** dprince has quit IRC | 17:17 | |
| *** mat128 has joined #openstack-infra | 17:17 | |
| clarkb | due to timing on reused connections being closed as client made requests | 17:17 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_queue_rate_limiting_dependent https://review.openstack.org/452864 | 17:17 |
| _ari_ | zaro: if you have cycles to review https://review.openstack.org/#/c/451871/5 | 17:17 |
| clarkb | (python requests doesn't handle http persistent connection closes cleanly) | 17:18 |
| fungi | cloudnull: one of the biggest differences between our providers is the numbers of vcpus, so tests get parallelized into different groupings depending on how many processors are available | 17:18 |
| fungi | which can make tests run in different orders, or result in different tests running at the same time, exposing unintended test dependencies and races | 17:19 |
| *** sekelso has quit IRC | 17:19 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Fix LOCI jobs https://review.openstack.org/452846 | 17:20 |
| cloudnull | ah, I've been thinking this was due to ec traps because its failing on cross-policy writes (which is because our second policy is an ec policy) but maybe it is just a race issue in tempest. | 17:21 |
| *** chlong has joined #openstack-infra | 17:21 | |
| _ari_ | waynr: could you review this change for me when you have the cycles https://review.openstack.org/#/c/451871/5 | 17:21 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Remove enabled_services=tempest from patrole gate jobs. https://review.openstack.org/452865 | 17:22 |
| *** tosky has quit IRC | 17:23 | |
| *** esberglu_ has joined #openstack-infra | 17:23 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Remove enabled_services=tempest from patrole gate jobs. https://review.openstack.org/452865 | 17:24 |
| *** esberglu has quit IRC | 17:24 | |
| *** armax has joined #openstack-infra | 17:25 | |
| pabelanger | and back online | 17:25 |
| *** jerryz has joined #openstack-infra | 17:25 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Remove ovb updates job from promotion pipeline" https://review.openstack.org/452866 | 17:27 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Remove ovb updates job from promotion pipeline" https://review.openstack.org/452866 | 17:27 |
| *** jaosorior is now known as jaosorior_away | 17:29 | |
| *** yamahata has quit IRC | 17:32 | |
| *** rbuzatu has joined #openstack-infra | 17:34 | |
| pabelanger | EmilienM: 2GB for puppetlabs mirror? Should be okay | 17:34 |
| *** nicolasbock has quit IRC | 17:35 | |
| EmilienM | pabelanger: I'll try to find time to write the patches | 17:35 |
| clarkb | pabelanger: apache cache is between 648MB and 43MB on the various mirror hosts | 17:35 |
| clarkb | (osic is still largest) | 17:36 |
| pabelanger | clarkb: ya, I checked osic this morning, was 500MB | 17:36 |
| pabelanger | sub 1GB is great IMO | 17:36 |
| pabelanger | clarkb: I'm also going to start on RDO rsync, going to create AFS volumes | 17:36 |
| clarkb | pabelanger: ya, but I am not sure its properly expiring old data | 17:37 |
| clarkb | -rw------- 1 www-data www-data 27488 Mar 30 22:26 @@@UGoA.data | 17:37 |
| pabelanger | clarkb: oh? | 17:37 |
| clarkb | that is from more than 24 hours ago | 17:37 |
| pabelanger | what is the expire value? | 17:37 |
| clarkb | apache should have a default max expiration of 24 hours, thinking maybe we need to explicitly set the value | 17:37 |
| clarkb | (and our default is different because debuntu config or something) | 17:37 |
| *** rlandy|biab is now known as rlandy | 17:38 | |
| pabelanger | CacheMaxExpire ? | 17:38 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: s/gerrit-git-prep/zuul-git-prep/g https://review.openstack.org/452879 | 17:38 |
| AJaeger | fungi, jeblair, clarkb, pabelanger: I think we can retire gerrit-git-prep now, what do you think? ^ | 17:38 |
| clarkb | pabelanger: ya | 17:38 |
| clarkb | pabelanger: docs say default is one day but wodnering if thats overridden somewhere by debuntu | 17:39 |
| clarkb | AJaeger: we only needed it for the post and release jobs right? | 17:39 |
| clarkb | AJaeger: but now we don't need it there anymore right? | 17:39 |
| clarkb | pabelanger: oh wait I see what it is that checks the origin server | 17:40 |
| AJaeger | clarkb: that's my understanding. We fixed the problems we had there. | 17:40 |
| clarkb | pabelanger: so its possible that we aren't actually rotating out data like I expected | 17:40 |
| clarkb | pabelanger: https://httpd.apache.org/docs/2.4/mod/mod_cache.html#cachestoreexpired | 17:41 |
| *** beslemon has joined #openstack-infra | 17:41 | |
| *** chem is now known as chem_gone | 17:41 | |
| clarkb | pabelanger: that seems to only affect a resposne if it has already expired | 17:42 |
| clarkb | for some reason I had thought it was general pruning | 17:42 |
| pabelanger | clarkb: https://httpd.apache.org/docs/2.4/programs/htcacheclean.html | 17:42 |
| pabelanger | guess we need that? | 17:42 |
| clarkb | ya | 17:43 |
| *** smarcet has quit IRC | 17:43 | |
| pabelanger | k | 17:44 |
| pabelanger | I'll give it a test shortly | 17:44 |
| clarkb | I think based on the size of current caches, 2GB should be plenty to start | 17:44 |
| pabelanger | ack | 17:44 |
| *** sekelso has joined #openstack-infra | 17:45 | |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: add /usr/local/lib to library cache https://review.openstack.org/452886 | 17:46 |
| clarkb | htcacheclean -n -p /opt/apache_cache -t -l 2048M | 17:46 |
| clarkb | pabelanger: ^ something like that in an hourly cron? | 17:46 |
| pabelanger | clarkb: ya, think so. Just seeing that the interwebs suggests | 17:47 |
| *** shardy is now known as shardy_afk | 17:47 | |
| clarkb | I'm going to manually install apache2-utils on osic mirror to see if ubuntu install it as a service | 17:48 |
| pabelanger | sure | 17:48 |
| clarkb | doesn't look like it so cron seems like good easy mode | 17:48 |
| pabelanger | k | 17:49 |
| clarkb | I think we want the -t because these urls change based on git sha1 | 17:50 |
| *** smarcet has joined #openstack-infra | 17:50 | |
| clarkb | so we won't be reusing directories often once cleaned out | 17:50 |
| *** jamielennox is now known as jamielennox|away | 17:50 | |
| pabelanger | ya | 17:50 |
| *** ijw has quit IRC | 17:51 | |
| *** sekelso has quit IRC | 17:51 | |
| *** nicolasbock has joined #openstack-infra | 17:51 | |
| clarkb | pabelanger: do you want me to write the patch or are you already working on it? | 17:51 |
| clarkb | (but ya assuming that gets cache size under control I think we can open this up to all the jobs_ | 17:51 |
| pabelanger | clarkb: if you want to sure | 17:52 |
| pabelanger | already started on AFS volumes | 17:52 |
| clarkb | ok will get that up shortly | 17:52 |
| *** armax has quit IRC | 17:53 | |
| clarkb | I'm going to run the above command on osic mirror just to test, and will then reduce down to 512M and see if it cleans | 17:53 |
| *** jmorgan1 has joined #openstack-infra | 17:54 | |
| *** trown|lunch is now known as trown | 17:56 | |
| *** mat128 has quit IRC | 17:57 | |
| *** amoralej is now known as amoralej|off | 17:57 | |
| *** baoli has quit IRC | 17:57 | |
| *** Sukhdev has joined #openstack-infra | 17:57 | |
| *** mat128 has joined #openstack-infra | 17:57 | |
| *** Sihan has joined #openstack-infra | 17:58 | |
| *** d0ugal has quit IRC | 17:59 | |
| *** xyang1 has joined #openstack-infra | 17:59 | |
| clarkb | pabelanger: http://paste.openstack.org/show/605280/ so it seemed to work nicely | 17:59 |
| pabelanger | clarkb: nice | 18:00 |
| *** aarefiev is now known as aarefiev_afk | 18:00 | |
| *** Sihan has quit IRC | 18:00 | |
| *** ZZelle has joined #openstack-infra | 18:01 | |
| *** electrofelix has quit IRC | 18:01 | |
| mordred | woot | 18:01 |
| *** baoli has joined #openstack-infra | 18:01 | |
| *** aNuposic has quit IRC | 18:01 | |
| *** mat128 has quit IRC | 18:01 | |
| *** dprince has joined #openstack-infra | 18:02 | |
| *** alexchadin has joined #openstack-infra | 18:03 | |
| *** ssbarnea has quit IRC | 18:04 | |
| pabelanger | clarkb: mordred: if you have a moment, https://review.openstack.org/#/c/451972 for the RDO mirror | 18:05 |
| *** ssbarnea has joined #openstack-infra | 18:05 | |
| pabelanger | will take about 300GB of space | 18:05 |
| mordred | pabelanger: on it | 18:05 |
| *** markmcclain has quit IRC | 18:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 18:05 |
| mordred | pabelanger: oh - speaking of mirrors ... hongbin is working on adding consistent support for getting docker installed properly to devstack | 18:05 |
| hongbin | o/ | 18:06 |
| mordred | pabelanger: the script he's starting from ultimately installs an APT repo - although sdague and I suggested that perhaps having a mirror of that in the infra mirror system would be a better way to go | 18:06 |
| pabelanger | mordred: hongbin okay cool, ya. Happy to help with that | 18:06 |
| mordred | especially if this will be the devstack plugin that everyone using docker uses to get their docker | 18:06 |
| *** markmcclain has joined #openstack-infra | 18:07 | |
| pabelanger | shouldn't be had to mirror dockers apt repos | 18:07 |
| *** reed has quit IRC | 18:07 | |
| *** armax has joined #openstack-infra | 18:07 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run htcacheclean to clean mirror caches https://review.openstack.org/452887 | 18:07 |
| mordred | pabelanger: no - it shouldn't be - I think it's a pretty small repo | 18:07 |
| *** pvaneck has joined #openstack-infra | 18:07 | |
| pabelanger | k, I'll add it to the list | 18:08 |
| *** reed has joined #openstack-infra | 18:08 | |
| pabelanger | clarkb: we should try hub.docker.com later this week, and see how image caching works | 18:08 |
| mordred | pabelanger: yah - it's tiny: https://apt.dockerproject.org/ | 18:09 |
| mordred | pabelanger: I'll be excited if the hub.docker.com idea works out! | 18:09 |
| clarkb | pabelanger: ya, probably easiest thing is to just test it locally through a mirror | 18:09 |
| *** mat128 has joined #openstack-infra | 18:09 | |
| clarkb | pabelanger: rather than set up a job to do it | 18:09 |
| clarkb | then if that works we can push it out to jobs and worry about disk space a lot more :) | 18:09 |
| pabelanger | clarkb: indeed | 18:10 |
| *** dhellmann has quit IRC | 18:10 | |
| *** dhellmann has joined #openstack-infra | 18:11 | |
| hongbin | pabelanger: thanks for helping out about that, just let me know once the mirror is ready :) | 18:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove enabled_services=tempest from patrole gate jobs. https://review.openstack.org/452865 | 18:11 |
| pabelanger | hongbin: sure, do you have a patchset going now? | 18:11 |
| hongbin | pabelanger: https://review.openstack.org/#/c/452575/ | 18:11 |
| *** Swami has joined #openstack-infra | 18:12 | |
| *** harlowja has joined #openstack-infra | 18:12 | |
| *** florianf has quit IRC | 18:12 | |
| *** ssbarnea has quit IRC | 18:12 | |
| pabelanger | hongbin: thanks | 18:14 |
| *** tosky has joined #openstack-infra | 18:14 | |
| *** aNuposic has joined #openstack-infra | 18:15 | |
| *** florianf has joined #openstack-infra | 18:15 | |
| *** dtantsur is now known as dtantsur|afk | 18:15 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Populate logs exclude list with infra requirements https://review.openstack.org/446374 | 18:18 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Remove unused variables for Upgrades https://review.openstack.org/452170 | 18:19 |
| fungi | i'm probably jumping at shadows, but the mailing lists seem really quiet today | 18:19 |
| SamYaple | quiet fungi! | 18:20 |
| *** d0ugal has joined #openstack-infra | 18:20 | |
| * bkero knocks on wood. | 18:20 | |
| fungi | tailing the mailman posts log seems we're getting messages, and they do all seem to get delivered to me if i'm a subscriber, so nothing to worry about i guess | 18:20 |
| mordred | fungi: maybe we shoudl just blame the lovely spring weather april has brought? | 18:21 |
| fungi | i should certainly be out catching up on yardwork | 18:21 |
| mordred | fungi: I did that on saturday. I'm still very aware that I did that on saturday | 18:22 |
| fungi | it's the homeowner exercise program | 18:22 |
| *** rbrndt has quit IRC | 18:22 | |
| *** ralonsoh has quit IRC | 18:23 | |
| mordred | it's very effective | 18:23 |
| *** HeOS has quit IRC | 18:23 | |
| openstackgerrit | Merged openstack-infra/system-config master: Mirror rdo centos7 repo to AFS https://review.openstack.org/451972 | 18:24 |
| *** annegentle has joined #openstack-infra | 18:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:25 | |
| *** mat128 has quit IRC | 18:26 | |
| *** mat128 has joined #openstack-infra | 18:26 | |
| clarkb | I dug up a bunch of bamboo | 18:28 |
| clarkb | and there is still so much remainig | 18:28 |
| *** ssbarnea has joined #openstack-infra | 18:28 | |
| *** e0ne has quit IRC | 18:28 | |
| mordred | clarkb: by the time you dig up more, the stuff you already dug up will have regrown | 18:28 |
| fungi | give in and start harvesting it as a timber crop? | 18:30 |
| clarkb | its the miniature stuff so its not great timber | 18:30 |
| fungi | it's artisnal portland craft bamboo | 18:30 |
| clarkb | its also really splintery wood, not fun to work with | 18:30 |
| fungi | finest on the planet | 18:30 |
| clarkb | once I poke through my finger with a sliver of it | 18:31 |
| fungi | yikes | 18:31 |
| *** HoloIRCUser has joined #openstack-infra | 18:31 | |
| *** ssbarnea has quit IRC | 18:33 | |
| *** hrubi has joined #openstack-infra | 18:34 | |
| clarkb | hello infra-root, /me is going to ask again for reviews on https://review.openstack.org/#/c/449835/ so that we can continue to make progress on the gerrit upgrade | 18:38 |
| clarkb | that change is a prerequisite for deploying new gerrit on review-dev then doing testing there | 18:39 |
| *** cody-somerville has quit IRC | 18:39 | |
| *** cody-somerville has joined #openstack-infra | 18:39 | |
| *** andreww has quit IRC | 18:41 | |
| *** ihrachys has quit IRC | 18:41 | |
| *** xarses has joined #openstack-infra | 18:41 | |
| *** ihrachys has joined #openstack-infra | 18:41 | |
| *** felipemonteiro has quit IRC | 18:41 | |
| *** xarses has quit IRC | 18:41 | |
| clarkb | but note that it affects review.o.o too, hence not single approving it | 18:41 |
| *** xarses has joined #openstack-infra | 18:42 | |
| *** felipemonteiro has joined #openstack-infra | 18:42 | |
| jeblair | clarkb, zaro: is it the case that 1.52 won't work < 2.12? | 18:42 |
| mtreinish | fungi: https://review.openstack.org/#/q/status:open+topic:more-firehose are there any other services you think would be good/easy to add mqtt support to in the short term? | 18:43 |
| *** signed8bit is now known as signed8bit_Zzz | 18:43 | |
| mtreinish | fungi: I'm trying to think about what else we should flush out before our talk at the summit | 18:43 |
| clarkb | jeblair: that was my understanding, I asked that in an earlier patchset | 18:44 |
| mordred | clarkb: you have 3 +2s now | 18:44 |
| clarkb | jeblair: however it may just be that that is what things are tested with so people don't want to change it? | 18:44 |
| *** makowals has joined #openstack-infra | 18:44 | |
| mordred | it's sad that 1.52 can't work with 2.11 :( | 18:44 |
| clarkb | we can have zaro double confirm, and then I can approve and watch things | 18:45 |
| *** signed8bit_Zzz is now known as signed8bit | 18:46 | |
| *** dimtruck is now known as zz_dimtruck | 18:46 | |
| mordred | I mean - the patch is fine as is | 18:46 |
| mordred | it could just be simpler if it did work with 2.11 | 18:46 |
| mordred | but I don't think you need to wait for the double confirm | 18:46 |
| clarkb | ok | 18:47 |
| clarkb | I'm about to consume lunch but will approve after and keep an eye on review.o.o | 18:47 |
| clarkb | thanks everyone for reviewing | 18:47 |
| jeblair | yeah, that was definitely a +2 comment from me. :) | 18:47 |
| *** rbrndt has joined #openstack-infra | 18:49 | |
| *** john-davidge has joined #openstack-infra | 18:50 | |
| *** rbuzatu has quit IRC | 18:50 | |
| *** rbuzatu has joined #openstack-infra | 18:51 | |
| *** rbuzatu has quit IRC | 18:51 | |
| *** rbuzatu has joined #openstack-infra | 18:51 | |
| *** rbuzatu_ has joined #openstack-infra | 18:52 | |
| *** zz_dimtruck is now known as dimtruck | 18:53 | |
| *** john-davidge has quit IRC | 18:55 | |
| *** rbuzatu has quit IRC | 18:56 | |
| *** rbuzatu_ has quit IRC | 18:57 | |
| *** armax has quit IRC | 19:00 | |
| *** mat128 has quit IRC | 19:00 | |
| mtreinish | infra-root: if someone has time I'm curious if running "mosquitto_sub -h localhost --topic '$SYS/broker/bytes/received'" returns anything. I'm trying it from my laptop and not getting any data | 19:02 |
| mtreinish | I checked the config and we should be returning stats every 10secs (the default value) | 19:02 |
| mtreinish | I'm think it might be restricted to localhost (or I might need to add an acl to read that topic) | 19:02 |
| fungi | mtreinish: running now | 19:03 |
| fungi | so far... silence | 19:03 |
| fungi | what is $SYS? | 19:03 |
| mtreinish | it's a special topic var for mosquitto for stats to be published: https://mosquitto.org/man/mosquitto-8.html (the "Broker Status" section) | 19:04 |
| *** adrian_otto has quit IRC | 19:04 | |
| *** csomerville has joined #openstack-infra | 19:05 | |
| mtreinish | I'm looking into how we can collect metrics on how firehose is being used (if at all :) ) and how much data we're pumping into it | 19:05 |
| *** florianf has quit IRC | 19:07 | |
| *** csomerville has quit IRC | 19:07 | |
| *** csomerville has joined #openstack-infra | 19:07 | |
| _ari_ | dara | 19:07 |
| fungi | yeah, it's dead jim | 19:07 |
| *** cody-somerville has quit IRC | 19:08 | |
| mtreinish | hmm, I'll have to figure out what I setup wrong. I'm guess it's an acl or something like that then | 19:08 |
| * mtreinish reads more docs | 19:09 | |
| mordred | silly acls | 19:09 |
| mordred | mtreinish: have you tried turning it off and then back on again? | 19:09 |
| *** hashar has joined #openstack-infra | 19:09 | |
| mtreinish | mordred: heh, no. That actually works a lot for mosquitto in our experience :) | 19:11 |
| *** sree has joined #openstack-infra | 19:11 | |
| mordred | :) | 19:12 |
| *** alexchadin has quit IRC | 19:13 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-mosquitto master: Add system stats topics to public readonly acl https://review.openstack.org/452902 | 19:13 |
| mtreinish | fungi, mordred: ^^^ I think that'll do it. BUt I don't have a vm setup right now with mosquitto to test it | 19:13 |
| openstackgerrit | mark mcclain proposed openstack-infra/project-config master: Enable Python3.5 gating for networking-arista https://review.openstack.org/451572 | 19:14 |
| *** beslemon has quit IRC | 19:15 | |
| *** sree has quit IRC | 19:15 | |
| mtreinish | fungi, mordred: if we can get that working I'm planning on writing a little script to dump that data into graphite so we can make graphs | 19:17 |
| fungi | mtreinish: manually applied that config addition and restarted mosquitto. working over localhost for me now | 19:17 |
| fungi | give it a try? | 19:17 |
| mtreinish | fungi: it's working for me remotely too | 19:17 |
| mtreinish | \o/ | 19:18 |
| fungi | awesome. approving | 19:18 |
| mtreinish | oh, looks like someone already wrote that in go: https://github.com/latencia/mosquitto-stats/blob/master/mosquitto-stats.go | 19:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add rdo_keytab to mirror-update.o.o https://review.openstack.org/452904 | 19:19 |
| fungi | olaph: i noticed you ended up with two gerrit accounts for the same e-mail address... the ids are 5545 and 6726. i'm guessing just looking in the db that you're not actually using 6726 so happy to mark that account inactive if you're good with that | 19:20 |
| openstackgerrit | mark mcclain proposed openstack-infra/project-config master: Add check-requirements to networkings-arista https://review.openstack.org/452906 | 19:22 |
| olaph | fungi: sounds good | 19:23 |
| *** d0ugal has quit IRC | 19:23 | |
| fungi | olaph: thanks, i've set 6726 inactive in gerrit now. there was no openid associated with it, so i don't think there's any way you've been logging into it for ages. shouldn't cause you any problems | 19:25 |
| *** d0ugal has joined #openstack-infra | 19:25 | |
| mtreinish | fungi: heh, it failed puppet-lint. I didn't even change any puppet code | 19:27 |
| olaph | if it does, I know where you live! (well, kinda) | 19:28 |
| *** makowals has quit IRC | 19:32 | |
| clarkb | mtreinish: http://logs.openstack.org/02/452902/1/check/gate-puppet-mosquitto-puppet-lint/754eb88/console.html#_2017-04-03_19_17_25_732130 maybe tool got more strict? | 19:32 |
| *** liusheng has quit IRC | 19:32 | |
| *** marst_ has joined #openstack-infra | 19:32 | |
| pabelanger | clarkb: fungi: do you mind: https://review.openstack.org/#/c/452904/ adds missing rdo keytab file | 19:32 |
| *** liusheng has joined #openstack-infra | 19:32 | |
| mtreinish | clarkb: hmm, I guess. I'll push a fix for that right now | 19:34 |
| *** marst has quit IRC | 19:35 | |
| *** d0ugal has quit IRC | 19:36 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-mosquitto master: Add system stats topics to public readonly acl https://review.openstack.org/452902 | 19:36 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-mosquitto master: Fix puppet-lint issue https://review.openstack.org/452911 | 19:36 |
| mtreinish | clarkb, fungi: ^^ | 19:36 |
| *** chlong has quit IRC | 19:37 | |
| *** harlowja has quit IRC | 19:37 | |
| *** xyang1 has quit IRC | 19:39 | |
| *** HeOS has joined #openstack-infra | 19:39 | |
| *** yamahata has joined #openstack-infra | 19:40 | |
| *** xyang1 has joined #openstack-infra | 19:40 | |
| *** dimtruck is now known as zz_dimtruck | 19:45 | |
| mordred | olaph: I know where you live | 19:46 |
| *** armax has joined #openstack-infra | 19:47 | |
| mordred | olaph: you and your cow, too | 19:47 |
| cmurphy | clarkb: fungi while you're at it want to push https://review.openstack.org/#/c/452461/ and https://review.openstack.org/#/c/452487/ through so zaro and asselin_ can more forward? | 19:48 |
| openstackgerrit | Merged openstack-infra/system-config master: Add rdo_keytab to mirror-update.o.o https://review.openstack.org/452904 | 19:49 |
| *** rbuzatu has joined #openstack-infra | 19:50 | |
| *** john-davidge has joined #openstack-infra | 19:51 | |
| *** chlong has joined #openstack-infra | 19:51 | |
| pabelanger | okay, RDO rsync going | 19:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Migrate create_image_snapshot to REST https://review.openstack.org/452566 | 19:54 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: jjb: fix number of nodes for tripleo scenario005 https://review.openstack.org/452913 | 19:55 |
| *** john-davidge has quit IRC | 19:55 | |
| cmurphy | thanks fungi | 20:00 |
| fungi | thanks for fixing! | 20:00 |
| *** rbrndt has quit IRC | 20:00 | |
| *** rbrndt has joined #openstack-infra | 20:00 | |
| pabelanger | fungi: mind a +3 on https://review.openstack.org/#/c/452887/ | 20:00 |
| *** rbrndt has quit IRC | 20:01 | |
| pabelanger | cleans up the mirror caches for reverse proxy | 20:01 |
| pabelanger | clarkb: once we land ^, I think we can have EmilienM update tripleo jobs to start using the proxy | 20:01 |
| clarkb | yup | 20:01 |
| clarkb | and kolla and whoever else | 20:02 |
| *** shardy_afk has quit IRC | 20:02 | |
| pabelanger | ya | 20:02 |
| *** HoloIRCUser is now known as skelso | 20:02 | |
| EmilienM | I'm ready like never | 20:02 |
| EmilienM | tripleo patch is ready to use all repos from mirrors, except buildlogs, but no mirror there | 20:03 |
| openstackgerrit | Merged openstack-infra/puppet-gerrit master: Fix puppet-lint error https://review.openstack.org/452461 | 20:03 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Fix puppet-lint errors https://review.openstack.org/452487 | 20:03 |
| openstackgerrit | Merged openstack-infra/puppet-mosquitto master: Fix puppet-lint issue https://review.openstack.org/452911 | 20:04 |
| *** shardy_afk has joined #openstack-infra | 20:04 | |
| *** jamielennox|away is now known as jamielennox | 20:05 | |
| openstackgerrit | Merged openstack-infra/puppet-mosquitto master: Add system stats topics to public readonly acl https://review.openstack.org/452902 | 20:06 |
| *** skelso has quit IRC | 20:07 | |
| *** adrian_otto has joined #openstack-infra | 20:08 | |
| *** e0ne has joined #openstack-infra | 20:08 | |
| cmurphy | btw puppet changes would be more fun to review if https://review.openstack.org/#/c/447240/ went through | 20:08 |
| openstackgerrit | Merged openstack-infra/system-config master: Run htcacheclean to clean mirror caches https://review.openstack.org/452887 | 20:08 |
| openstackgerrit | Merged openstack-infra/system-config master: Add an example not using websockets https://review.openstack.org/449711 | 20:09 |
| *** e0ne has quit IRC | 20:09 | |
| fungi | mtreinish: did you see SamYaple's haskell indentation question on 449773? | 20:10 |
| *** e0ne has joined #openstack-infra | 20:10 | |
| mtreinish | fungi: I did not, let me look at that review | 20:10 |
| clarkb | cmurphy: did we just bump ahead to xenial without keeping trusty? I guess at this point probably just make that work then get updated to xenial | 20:10 |
| *** Goneri has quit IRC | 20:11 | |
| *** sekelso has joined #openstack-infra | 20:11 | |
| *** rwsu has quit IRC | 20:11 | |
| cmurphy | clarkb: i wasn't really paying attention when it happened but at some point it changed | 20:11 |
| cmurphy | clarkb: but it seems like it only changed for the puppet-openstackci beaker job and not for all the other ones | 20:12 |
| fungi | seems fine to me | 20:12 |
| mtreinish | fungi: responded, it's actually an rst indent question :) | 20:12 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add ironic-multinode-job to networking-generic-switch https://review.openstack.org/452921 | 20:13 |
| fungi | mtreinish: aha, it ended up in the quoted haskell, so i assumed the latter. i guess he was asking because it appears in the codeblock in the rendered copy? http://docs-draft.openstack.org/73/449773/2/check/gate-system-config-docs-ubuntu-xenial/1c1e133//doc/build/html/firehose.html#haskell | 20:13 |
| *** rbrndt has joined #openstack-infra | 20:13 | |
| mtreinish | fungi: that's part of the haskell, at least it didn't work unless I had that line at the top of the file | 20:14 |
| fungi | mtreinish: okay, then i guess i have no idea why he was asking about it | 20:14 |
| *** rwsu has joined #openstack-infra | 20:15 | |
| fungi | i agree it's rendering fine if that's part of the haskell source code | 20:15 |
| clarkb | mtreinish: fungi right thats like a compiler hint | 20:15 |
| clarkb | it modifies how thinsg are compiled | 20:15 |
| fungi | maybe he thought it was intended to be rst formatting metadata and didn't realize it was supposed to be part of the quotes source | 20:16 |
| fungi | er, quoted source | 20:16 |
| fungi | since clarkb likes haskell, i'll just +2 and give him an opportunity to review | 20:17 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Fix beaker-rspec tests https://review.openstack.org/447240 | 20:18 |
| openstackgerrit | Merged openstack-infra/puppet-gerrit master: Add a known_hosts file https://review.openstack.org/399821 | 20:19 |
| *** jamesdenton has quit IRC | 20:20 | |
| *** flepied has joined #openstack-infra | 20:22 | |
| clarkb | mtreinish: fungi the only thing I would change glancing at it is qosGranted should be anonymous "_" since its not used anywhere else | 20:23 |
| clarkb | but this is more verbose so maybe its ebtter | 20:23 |
| clarkb | ok brb then approving gerrit change | 20:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: (WIP) Run puppet 4 in beaker tests https://review.openstack.org/447243 | 20:24 |
| *** sekelso is now known as skelso | 20:24 | |
| *** zz_dimtruck is now known as dimtruck | 20:25 | |
| *** baoli has quit IRC | 20:25 | |
| *** jkilpatr has joined #openstack-infra | 20:25 | |
| *** megm has quit IRC | 20:27 | |
| *** e0ne has quit IRC | 20:28 | |
| *** rbrndt has quit IRC | 20:29 | |
| openstackgerrit | Merged openstack-infra/system-config master: Start firehose doc section for client on port 1883 https://review.openstack.org/449773 | 20:29 |
| *** Sukhdev has quit IRC | 20:30 | |
| *** megm has joined #openstack-infra | 20:30 | |
| *** baoli has joined #openstack-infra | 20:32 | |
| *** armax has quit IRC | 20:34 | |
| *** armax has joined #openstack-infra | 20:34 | |
| *** armax has quit IRC | 20:35 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit master: Fix placement of ssh public key https://review.openstack.org/400067 | 20:35 |
| *** rbrndt has joined #openstack-infra | 20:40 | |
| *** rbuzatu has quit IRC | 20:44 | |
| *** beslemon has joined #openstack-infra | 20:45 | |
| ianw | fungi: argh that's me sorry on the key. did you commit? | 20:46 |
| clarkb | I have approved the puppet-gerrit change | 20:46 |
| *** dimtruck is now known as zz_dimtruck | 20:48 | |
| *** zz_dimtruck is now known as dimtruck | 20:48 | |
| fungi | ianw: i did not since no idea who to attribute to nor what you wanted for commit subject | 20:48 |
| *** armax has joined #openstack-infra | 20:48 | |
| fungi | i usually commit specific files when i make hiera edits or use -p if necessary and leave edits i didn't make un-committed while hunting down those who know how they should be annotated | 20:49 |
| *** aeng has joined #openstack-infra | 20:50 | |
| *** david-lyle has quit IRC | 20:51 | |
| *** esberglu_ is now known as esberglu | 20:51 | |
| *** david-lyle has joined #openstack-infra | 20:51 | |
| *** beslemon has quit IRC | 20:51 | |
| ianw | oh cool, pabelanger did. if it's any consolation, i got distracted updating the docs on how to add the key (will push after a reread today) | 20:52 |
| *** annegentle has quit IRC | 20:52 | |
| *** harlowja has joined #openstack-infra | 20:53 | |
| pabelanger | ya, added it with rdo keytab | 20:54 |
| clarkb | which key? | 20:54 |
| pabelanger | opensuse | 20:54 |
| *** trown is now known as trown|outtypewww | 20:55 | |
| *** rbuzatu has joined #openstack-infra | 20:55 | |
| *** rbuzatu has quit IRC | 20:55 | |
| *** rbuzatu has joined #openstack-infra | 20:55 | |
| *** esberglu has quit IRC | 20:56 | |
| *** esberglu has joined #openstack-infra | 20:56 | |
| *** rbuzatu_ has joined #openstack-infra | 20:57 | |
| *** rockyg has joined #openstack-infra | 20:57 | |
| mtreinish | fungi: https://github.com/mtreinish/mqtt_statsd I think that should enable us to get graphs on mqtt usage | 20:58 |
| *** rbuzatu__ has joined #openstack-infra | 20:58 | |
| mtreinish | fungi: heh, of course after I test it. I'll spin up a vm with mosquitto and statsd when I'm back home later today | 20:58 |
| *** rbuzat___ has joined #openstack-infra | 20:59 | |
| *** thorst has quit IRC | 20:59 | |
| *** rbuzatu has quit IRC | 20:59 | |
| waynr | _ari_: okay, i've reviewed it. | 21:00 |
| *** dprince has quit IRC | 21:00 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 21:00 |
| *** thorst has joined #openstack-infra | 21:00 | |
| *** d0ugal has joined #openstack-infra | 21:00 | |
| *** rderose has joined #openstack-infra | 21:01 | |
| *** esberglu has quit IRC | 21:01 | |
| *** rbuzatu has joined #openstack-infra | 21:01 | |
| *** umbSublime has joined #openstack-infra | 21:01 | |
| *** rbuzatu_ has quit IRC | 21:01 | |
| *** dprince has joined #openstack-infra | 21:01 | |
| *** rbuzatu_ has joined #openstack-infra | 21:02 | |
| *** rbuzatu__ has quit IRC | 21:02 | |
| *** rbuzatu__ has joined #openstack-infra | 21:03 | |
| *** tonytan4ever has quit IRC | 21:04 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit master: Update bouncy castle libraries for Gerrit 2.13 https://review.openstack.org/449835 | 21:04 |
| *** rbuzat___ has quit IRC | 21:04 | |
| *** thorst has quit IRC | 21:04 | |
| *** rbuzatu has quit IRC | 21:05 | |
| *** rbuzatu_ has quit IRC | 21:06 | |
| openstackgerrit | dane-fichter proposed openstack-infra/project-config master: Add barbican-tempest-plugin tests to Cursive https://review.openstack.org/452950 | 21:07 |
| *** rbuzatu__ has quit IRC | 21:08 | |
| *** shardy_afk has quit IRC | 21:08 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Make tox-linters run the tox/linters play https://review.openstack.org/452953 | 21:11 |
| *** ihrachys has quit IRC | 21:13 | |
| *** dprince has quit IRC | 21:14 | |
| *** annegentle has joined #openstack-infra | 21:15 | |
| *** ldnunes has quit IRC | 21:16 | |
| fungi | mtreinish: i also haven't reviewed the log processor patch yet until we make sure things go smoothly with the subunit worker (since the log processor code is more heavily used downstream) | 21:16 |
| *** rfolco has quit IRC | 21:18 | |
| *** srobert has quit IRC | 21:18 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info role https://review.openstack.org/441617 | 21:19 |
| *** mattmceuen has quit IRC | 21:20 | |
| *** eharney has quit IRC | 21:20 | |
| *** skelso has quit IRC | 21:21 | |
| *** smarcet has quit IRC | 21:21 | |
| *** mattoliverau has quit IRC | 21:24 | |
| *** darrenc has quit IRC | 21:24 | |
| *** darrenc has joined #openstack-infra | 21:24 | |
| *** gouthamr has quit IRC | 21:24 | |
| *** matt6434 has joined #openstack-infra | 21:25 | |
| *** ijw has joined #openstack-infra | 21:25 | |
| *** thorst has joined #openstack-infra | 21:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info role https://review.openstack.org/441617 | 21:28 |
| *** smarcet has joined #openstack-infra | 21:28 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Cleanup docs builds and error on warnings https://review.openstack.org/452964 | 21:30 |
| *** thorst has quit IRC | 21:30 | |
| openstackgerrit | Merged openstack-infra/puppet-nodepool master: Rename nodepoold to nodepool-launcher for zuulv3 https://review.openstack.org/450894 | 21:31 |
| *** beslemon has joined #openstack-infra | 21:31 | |
| SamYaple | ive seen alot of zuulv3 stuff in the past few days... how close is that? is it about to land? | 21:32 |
| clarkb | SamYaple: its running jobs against itself, but I think there is still a fair bit of work to be done before we deploy it for all jobs | 21:33 |
| clarkb | in part just to do the transition itself but also sandboxing | 21:33 |
| *** Goneri has joined #openstack-infra | 21:33 | |
| *** esberglu has joined #openstack-infra | 21:35 | |
| SamYaple | ah ok cool | 21:35 |
| *** smarcet has quit IRC | 21:36 | |
| *** smarcet has joined #openstack-infra | 21:36 | |
| pabelanger | http://zuulv3-dev.openstack.org too | 21:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info role https://review.openstack.org/441617 | 21:38 |
| *** aysyd has quit IRC | 21:40 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make tox-linters run the tox/linters play https://review.openstack.org/452953 | 21:40 |
| *** beslemon has quit IRC | 21:41 | |
| *** ramishra has quit IRC | 21:42 | |
| *** kzaitsev_mb has quit IRC | 21:44 | |
| *** EricGonc_ has quit IRC | 21:46 | |
| *** EricGonczer_ has joined #openstack-infra | 21:46 | |
| *** felipemonteiro has quit IRC | 21:47 | |
| *** thorst has joined #openstack-infra | 21:47 | |
| *** Sukhdev_ has joined #openstack-infra | 21:47 | |
| *** hashar has quit IRC | 21:51 | |
| *** armax has quit IRC | 21:51 | |
| *** smarcet has quit IRC | 21:51 | |
| *** dizquierdo has joined #openstack-infra | 21:52 | |
| clarkb | pabelanger: reading up on docker registries, we'll have to whitelist it as insecure (due to no https), but otherwise forward proxies seem to be expected to work so I kidna expect the reverse to work too | 21:52 |
| *** wolverineav has quit IRC | 21:53 | |
| SamYaple | clarkb: whats the issue with docker registries and https? | 21:53 |
| pabelanger | clarkb: nice! Surprising about https, guess we should open a github issue on this | 21:53 |
| SamYaple | i run a registry with https.... | 21:54 |
| *** wolverineav has joined #openstack-infra | 21:54 | |
| clarkb | SamYaple: just that you have to explicitly allow non https (because you don't provide a proper url for downloading) | 21:54 |
| SamYaple | oh. thats on the client side though. is that what you are refering to? | 21:54 |
| pabelanger | Oh, ya. that makes sense | 21:55 |
| pabelanger | we do the same with APT | 21:55 |
| SamYaple | yea if thats the thing thats perfectly fine | 21:55 |
| *** rhallisey has quit IRC | 21:55 | |
| clarkb | ya that | 21:55 |
| fungi | though for lack of index signing on apt repos, not lack of ssl | 21:55 |
| fungi | apt repos are almost never served over https anyway | 21:56 |
| clarkb | just trying to reason through if a reverse proxy like we did for rdo would work | 21:56 |
| clarkb | and I think it would given what I can read. I guess next step is set it up and test it | 21:56 |
| SamYaple | clarkb: there is a docker specific caching thing fyi | 21:56 |
| clarkb | SamYaple: ya trying to avoid having 50 difference $thing specific daemons for caching | 21:56 |
| clarkb | because we already have one | 21:56 |
| fungi | the alternative is to set up a cert for the proxy and explicitly add it to the trust on the workers | 21:57 |
| pabelanger | ++ | 21:57 |
| *** armax has joined #openstack-infra | 21:57 | |
| SamYaple | fungi: i think dockerd may actually ignore the system ca-certs | 21:58 |
| inc0 | SamYaple: talking about registry in infra? | 21:59 |
| SamYaple | inc0: registry proxy | 21:59 |
| SamYaple | kolla gates will need to be modified, but i can submit a patch for that | 22:00 |
| *** rderose has quit IRC | 22:00 | |
| inc0 | well correct me if I'm wrong but we can also push stuff to registry proxy right? | 22:00 |
| fungi | the proxy would be caching requests to dockerhub itself | 22:00 |
| SamYaple | right, the base layers for things inc0 | 22:00 |
| *** wolverineav has quit IRC | 22:01 | |
| SamYaple | ubuntu/centos:latest kind of thing | 22:01 |
| *** asselin_ has quit IRC | 22:01 | |
| inc0 | that's cool, but it's also possible to setup some proxy that we can push to as well, as in, if not found in registry, pull from dockerhub | 22:01 |
| pabelanger | yes, we have a spec going for that | 22:01 |
| SamYaple | yea | 22:01 |
| SamYaple | inc0: https://review.openstack.org/#/c/447524/ | 22:01 |
| fungi | it's a potential stop-gap for things hosted externally which we don't have a good mirroring solution for yet | 22:01 |
| fungi | stuff we would push into would be a hosted registry | 22:02 |
| fungi | not a caching proxy | 22:02 |
| jeblair | zuul meeting time in #openstack-meeting-alt | 22:02 |
| *** armax has quit IRC | 22:02 | |
| inc0 | SamYaple: yeah I know this one, but what I'm thinking about is having docker-registry.openstack.org | 22:03 |
| *** asselin has joined #openstack-infra | 22:03 | |
| inc0 | that will act as both cache and pushable place to replace tarballs hack | 22:03 |
| SamYaple | right thats not what we were tlaking about just now | 22:03 |
| inc0 | we can kill 2 birds with one stone tho | 22:04 |
| *** Swami has quit IRC | 22:04 | |
| fungi | inc0: right, that's a separate need for things we generate. the current discussion is about improving access to things outside our control | 22:04 |
| pabelanger | inc0: we'd want to write to AFS, once then have a regional registry read from it and serve the image | 22:04 |
| SamYaple | its a bigger discussion though, this will allow us to stop pulling from dockerhub at all | 22:04 |
| fungi | inc0: except we can't kill those two with one stone | 22:04 |
| inc0 | what I'm saying, if my memory works well, proxy can do both | 22:04 |
| fungi | inc0: they're only related by the words "docker image" | 22:04 |
| inc0 | no, hold on | 22:04 |
| *** rbuzatu has joined #openstack-infra | 22:05 | |
| inc0 | caching registry is squid-easy | 22:05 |
| inc0 | as dockerhub is just https | 22:05 |
| fungi | does dockerhub maintain an index? not seeing how to magically inject things into the cached index which aren't actually on dockerhub if so | 22:05 |
| *** EricGonczer_ has quit IRC | 22:05 | |
| *** ssbarnea has joined #openstack-infra | 22:06 | |
| fungi | sure we could host local files and do 404 fallback to the proxy or vice versa, but that doesn't address indexing | 22:06 |
| inc0 | fungi: easy enough, if exists locally - serve, else check with dockerhub | 22:06 |
| SamYaple | what i think we are refering to is starting dockerd --registry-mirror=http://infrahost | 22:06 |
| SamYaple | which has nothing to do with pushing | 22:06 |
| fungi | inc0: so you're saying there's no index maintained by dockerhub we'd need to augment/substitute to include locally hosted files? | 22:07 |
| SamYaple | https://docs.docker.com/registry/recipes/mirror/#running-a-registry-as-a-pull-through-cache | 22:07 |
| inc0 | fungi: not sure about that, frankly I don't know docker registry internals that well | 22:07 |
| SamYaple | there it is, this cache is pull only. pushing still goes directly to dockerhub | 22:07 |
| fungi | this is orthogonal to running an apache caching proxy, which we already needed to add for other things. if you _don't_ want to use it for dockerhub access, then don't | 22:08 |
| mordred | yah - we started looking at that link late last week | 22:09 |
| fungi | it is being suggested as a (possibly temporary until we work out something better) solution to some of your download instability in jobs | 22:09 |
| inc0 | what I'm saying is if we're running registry, we can configure it to make it pushable too | 22:09 |
| inc0 | as we *really* need pushable registry | 22:10 |
| fungi | inc0: sure, completely separate discussion from this | 22:10 |
| inc0 | don't see why sepeare thoi | 22:10 |
| fungi | this is just apache mod_proxy with caching enabled | 22:10 |
| fungi | it's not "running a registry" in any sense of the word | 22:10 |
| inc0 | ok | 22:11 |
| jhesketh | Morning | 22:11 |
| fungi | it's something we already have, which we can easily give you access to leverage if you want to do so | 22:11 |
| SamYaple | fungi: oh if you are using mod_proxy, that is not what I am refering to at all | 22:11 |
| inc0 | well we didn't have much issues with dockerhub not working well, it's mostly about conservation of infra networking | 22:11 |
| mordred | inc0: yah - and setting up a registry is enough additional work that so far it has died on the vine every time it's come up for the last 6-9 months | 22:11 |
| fungi | clarkb: pabelanger: this reminds me, what do you think of setting it up to also proxy for tarballs.o.o? i bet that could relieve a lot of load on that server from ci jobs | 22:12 |
| SamYaple | fungi: but this could end up with the same affect i believe | 22:12 |
| mordred | we _want_ to do it, and we want it ot be good | 22:12 |
| clarkb | fungi: ++ | 22:12 |
| mordred | fungi: ++ | 22:12 |
| clarkb | fungi: in fact that would be such a simple thing to do we should just do it | 22:12 |
| mordred | yah | 22:12 |
| inc0 | mordred: does it still need to be puppet?;) | 22:12 |
| mordred | inc0: yes | 22:12 |
| mtreinish | fungi: ok sure, that's a good way to handle it | 22:12 |
| pabelanger | fungi: ++ | 22:12 |
| *** rcernin has quit IRC | 22:12 | |
| *** marst_ has quit IRC | 22:13 | |
| *** mdrabe has quit IRC | 22:13 | |
| mtreinish | fungi: also for the log_processor patch I wasn't sure what we wanted in the message payload. We've got a lot of flexibility there | 22:13 |
| mordred | inc0: and it still needs to not run in a docker container itself - which is totally easy and doable - it just needs to be done - the hardest part is going to be hooking it into the multi-region AFS backend infrastructure - BUT... | 22:13 |
| inc0 | I mean last time it died because there is half-functional puppet module for registry somewhere and we didn't want to reimplement it | 22:13 |
| inc0 | if you can just trust me on getting this done in infra, I'll get this done.. | 22:13 |
| SamYaple | mordred: im not sure multi-region AFS backend is needed | 22:13 |
| mordred | if apache proxying works for docker hubs at all (which this may show us) | 22:13 |
| SamYaple | if everything pulls through a registry mirror (see link above) you could have one per region/cloud/whatever | 22:13 |
| mordred | then we might be able to just have a single-region registry | 22:14 |
| mordred | right | 22:14 |
| inc0 | mordred: if I can have access to one shared dir, that's all I need | 22:14 |
| mtreinish | fungi: we basically have access to all the metadata from zuul, and other stuff too I think | 22:14 |
| pabelanger | inc0: you can get it done today by proposing a patch to system-config. No special access is needed | 22:14 |
| inc0 | https://review.openstack.org/#/c/399221/ <- just need to pick it back up | 22:14 |
| mordred | that's why I think great steps are: a) verify apache proxying b) maybe add pull-through registry cache c) implement pushable central registry | 22:14 |
| inc0 | I'll do it then | 22:14 |
| pabelanger | Once I finish up the AFS mirror for rdo, docker. My next task is to try a docker registry with afs | 22:14 |
| pabelanger | see how that all works and stuff | 22:15 |
| fungi | inc0: the puppet-docker_registry module got blocked because the puppet team expressed concerns we were creating a new module which duplicates one in puppetforge rather than working with the authors of that existing module to cover our use case | 22:15 |
| SamYaple | pabelanger: what i was saying above is im not sure AFS is needed | 22:15 |
| inc0 | pabelanger: so if we can have single writer and mirror readers across regions, would that be easier? | 22:15 |
| *** ZZelle has quit IRC | 22:15 | |
| *** Goneri has quit IRC | 22:15 | |
| pabelanger | SamYaple: we want AFS, so we can write images once, then release them to all our cloud regions | 22:15 |
| SamYaple | pabelanger: singler writer globally and read-only pull caches per environment | 22:15 |
| SamYaple | pabelanger: yup thats what im saying | 22:16 |
| *** rfolco has joined #openstack-infra | 22:16 | |
| *** rfolco has quit IRC | 22:16 | |
| mordred | two different things - although I agree it's possible we may _not_ need an AFS backend for the pushable registry | 22:16 |
| pabelanger | right, so we need to validate we can write to AFS, trigger a 'vos release', and ensure other registry can download image from | 22:16 |
| pabelanger | the issue is, what happens to the remote registry when the filesystem changes below | 22:17 |
| inc0 | pabelanger: nothing | 22:17 |
| inc0 | it just uses fs | 22:17 |
| mordred | pabelanger: right - I _think_ the pull-through registries per regoin may solve this for us differently | 22:17 |
| SamYaple | pabelanger: no, thats not what im refering to. | 22:17 |
| *** mat128 has joined #openstack-infra | 22:17 | |
| inc0 | mirroring files is all we need | 22:17 |
| pabelanger | mordred: agree | 22:17 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Manage unbound on zuul workers in DIB https://review.openstack.org/452976 | 22:18 |
| SamYaple | pabelanger: single writer globally. no file mirroring directly. image pushes to single writer registry. all other regions have local registries with independant storage. everything pulls through the local registries which mirrors things as needed | 22:18 |
| mordred | SamYaple: right. I think that's an excellent option we shoudl consider | 22:18 |
| pabelanger | SamYaple: right, we are experimenting with that now | 22:18 |
| pabelanger | until last week, we didn't have a proxy cache | 22:19 |
| SamYaple | cool. then i think everything is laid out and we are all aware of the options. now just to figure out the best path forward | 22:19 |
| mordred | and the best part is - we can consider its siutability by just having independent per-region registries without a writable registry | 22:19 |
| SamYaple | mordred: ++ | 22:19 |
| mordred | if the per-region cache registry approach _works_ - then we do not have to engineer how to make our central registry integrate with mutli-regoin replication | 22:19 |
| mordred | if it _doensn't_ - then we do | 22:20 |
| *** jascott1_ has joined #openstack-infra | 22:20 | |
| inc0 | ok, so can we do it instead of apache mod_proxy? | 22:20 |
| *** askb has joined #openstack-infra | 22:20 | |
| SamYaple | still have to write the registry code in puppet inc0 | 22:20 |
| SamYaple | thats the first stuf | 22:20 |
| SamYaple | step* | 22:21 |
| inc0 | I know, I also had volunteer to help in PTG | 22:21 |
| fungi | inc0: you can do it in addition to or instead of apache mod_proxy potentially, just pointing out that we'll have the apache mod_proxy solution anyway since we needed it for other stuff | 22:21 |
| *** marst_ has joined #openstack-infra | 22:21 | |
| clarkb | right its 3 lines of vhost config at this point to get the mod_proxy solution working | 22:21 |
| mordred | inc0: well, step 1 is to poke a little more at mod_proxy, since it exists and is being used for other things already - so it can be explored in parallel to poking at puppetting docker registries | 22:21 |
| inc0 | fungi: if we're doing registry, since I think we agreed it's good thing to do | 22:21 |
| fungi | if you don't want to use the mod_proxy setup we have, it's as easy as ignoring it | 22:21 |
| clarkb | since we've sorted it out generally otherwise | 22:21 |
| inc0 | we might as well just do it | 22:22 |
| *** matt6434 is now known as mattoliverau | 22:22 | |
| inc0 | will save us changes to gates of all kolla deliverables | 22:22 |
| *** jascott1_ has quit IRC | 22:22 | |
| inc0 | 2 lines for you, a bit more for us, not much more but still | 22:22 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Move unbound to openstack_project::server https://review.openstack.org/452979 | 22:22 |
| inc0 | SamYaple: how about you and me get cracking on puppet module then? | 22:22 |
| inc0 | and just get this one done? | 22:23 |
| SamYaple | inc0: i dont do puppet or i would have done it years ago | 22:23 |
| SamYaple | mnaser said he would be working on it | 22:23 |
| mordred | works for me - fwiw, I'm not convinced we _need_ a whole puppet module - we need to apt-get install one package, and we'll need to install a config file | 22:23 |
| inc0 | mordred: right... | 22:23 |
| inc0 | thing is, how do we do it in infra?;) | 22:24 |
| inc0 | without puppet? | 22:24 |
| fungi | SamYaple: i hadn't seen mnaser talk about working on that, he's got a spec up for publishing release artifacts to dockerhub | 22:24 |
| mordred | nono - it must be in puppet | 22:24 |
| inc0 | because it's literally that easy | 22:24 |
| SamYaple | fungi: talked ot him in a PM | 22:24 |
| fungi | oh, cool | 22:24 |
| mordred | I'm saying we don't necessarily need a whole independent _module_ in its own repository | 22:24 |
| clarkb | inc0: its literally 3 lines of config... because I already did all the work for you... | 22:24 |
| *** baoli has quit IRC | 22:24 | |
| inc0 | that and setup docker engines in our gates to actually use it... | 22:25 |
| clarkb | inc0: but thats cost regardless | 22:25 |
| inc0 | as opposed to local registry which is config change | 22:25 |
| SamYaple | alright fine people. ill own this. ill talk with mnaser and get it done so we can stop going round and round | 22:25 |
| clarkb | its fixed, you can't avoid that | 22:25 |
| pabelanger | we likely need a server running xenial | 22:25 |
| * SamYaple grumbles | 22:25 | |
| inc0 | yeah, what I'm saying it's 2 changes instead of 1 and since we agree we want registry | 22:25 |
| pabelanger | and decided if we are using apt.docker.com or OS package | 22:25 |
| inc0 | let's just do registy... | 22:25 |
| mordred | inc0: we have _not_ agreed we want a registry running in each mirror host - it's entirely possible that's unneccesary | 22:26 |
| clarkb | inc0: no one is stopping you from doing the registry, we had to do the other thing anyways. So its there and a possible option. If you don't want to use it then don't | 22:26 |
| inc0 | thanks SamYaple let me know if there is there is anything I can help with | 22:26 |
| mordred | inc0: but I do agree we _do_ want a central registry at some point - so the puppet work to be able to spin up a registry is not wasted work | 22:26 |
| mordred | so all the work that has been suggested here is valuable | 22:26 |
| fungi | yeah, the last discussion i'd seen was set up a registry pushing updates onto an afs filesystem, and then presenting that from all our per-provider mirrors (direct copies served via http out of read-only afs) | 22:27 |
| inc0 | yeah technically mod_proxy apache will work just fine with single writer registry | 22:27 |
| mordred | inc0: ++ | 22:27 |
| inc0 | yeah, you're right, just central writer is ok | 22:27 |
| inc0 | SamYaple: we have this https://review.openstack.org/#/c/399221/2 | 22:28 |
| SamYaple | yea im on that review i believe | 22:28 |
| mordred | that said - if mod_proxy that's on the mirror hosts for some reason winds up not working well and we need to supplement with local pull-through-cache registries in each region, that should not be a hard step to make | 22:28 |
| SamYaple | or rather ive seen it | 22:28 |
| inc0 | it needs reviving | 22:29 |
| SamYaple | thanks, gotta go guys! | 22:29 |
| mordred | SamYaple, inc0: thanks! | 22:29 |
| inc0 | reviving as a revive not review:P | 22:29 |
| inc0 | I'll go call some favors | 22:29 |
| fungi | but it was also pointed out that afs might be a poor fit for caching very large files, so caching http proxy of some centrally-served registry makes sense as an alternative | 22:29 |
| *** aunnam has quit IRC | 22:30 | |
| inc0 | yeah, caching proxy+single pusher registry is elegant imho | 22:30 |
| *** wolverineav has joined #openstack-infra | 22:30 | |
| mordred | ++ | 22:30 |
| fungi | what are the largest files you expect to retrieve? | 22:30 |
| fungi | curious how big those image layers get | 22:30 |
| *** gordc has quit IRC | 22:32 | |
| inc0 | fungi: that'd be single base image, think 600mb | 22:32 |
| fungi | okay, we may need to bump the max file size in our current config in that case | 22:32 |
| inc0 | I threw this on #tripleo channel to as they might be interested in helping | 22:33 |
| inc0 | and are good with puppet | 22:33 |
| *** andreww has joined #openstack-infra | 22:35 | |
| *** rockyg has quit IRC | 22:35 | |
| *** andreww has quit IRC | 22:36 | |
| *** andreww has joined #openstack-infra | 22:36 | |
| *** chlong has quit IRC | 22:38 | |
| *** xarses_ has quit IRC | 22:38 | |
| *** aunnam has joined #openstack-infra | 22:39 | |
| *** armax has joined #openstack-infra | 22:39 | |
| *** Goneri has joined #openstack-infra | 22:41 | |
| *** jascott1_ has joined #openstack-infra | 22:42 | |
| pabelanger | I can start on the central registry | 22:43 |
| *** jascott1_ has quit IRC | 22:43 | |
| pabelanger | we need a xenial host | 22:43 |
| inc0 | pabelanger: thank you good sir | 22:44 |
| *** ijw has quit IRC | 22:45 | |
| inc0 | oh one thing tho, we'll need secure access there | 22:45 |
| inc0 | that was one of issues afair | 22:45 |
| *** rbrndt has quit IRC | 22:45 | |
| *** jascott1_ has joined #openstack-infra | 22:45 | |
| pabelanger | why? | 22:45 |
| inc0 | I mean it's not hard to setup | 22:45 |
| *** ijw has joined #openstack-infra | 22:45 | |
| inc0 | just how to give access to these secrets to gates | 22:45 |
| pabelanger | inc0: it is in the spec | 22:46 |
| pabelanger | and is a good usecase for zuulv3 | 22:46 |
| *** Goneri has quit IRC | 22:46 | |
| clarkb | pushes you mean? | 22:46 |
| clarkb | the reads shouldn't matter | 22:46 |
| pabelanger | ya | 22:47 |
| *** ijw has quit IRC | 22:47 | |
| inc0 | yeah pushes | 22:47 |
| pabelanger | to publish we'd do it how it is down today with tarballs | 22:47 |
| pabelanger | upload to tarballs.o.o from unsecure node, | 22:47 |
| pabelanger | then secure node will download and publish | 22:47 |
| *** ijw has joined #openstack-infra | 22:47 | |
| mordred | yup. pushes will be much better with zuul v3 | 22:47 |
| pabelanger | this is why zuulv3 is appealing, it would flex the new secrets system | 22:47 |
| fungi | the secure worker thing was for if you wanted to push releases up to dockerhub (a la mnaser's spec) | 22:48 |
| *** sshnaidm is now known as sshnaidm|off | 22:48 | |
| fungi | a central registry, if it's also an http server, doesn't need special credentials unless i'm missing something | 22:49 |
| mordred | it needs credentials to publish to | 22:49 |
| fungi | even if it publishes to itself? | 22:49 |
| pabelanger | does it have an ACL, where we can disable from localhost lets say? | 22:49 |
| mordred | it wouldn't publish to itself | 22:49 |
| mordred | I mean - something needs to publish to it | 22:49 |
| fungi | oh, i'm picturing a docker registry as being like an apt repo | 22:49 |
| mordred | no - it's a thing with a rest api that you push things to | 22:50 |
| inc0 | well technically we could solve it on networking layer | 22:50 |
| *** xyang1 has quit IRC | 22:50 | |
| inc0 | ...maybe | 22:50 |
| jeblair | https://review.openstack.org/401003 is almost a change to add a docker server in the way that mordred suggested earlier. | 22:50 |
| fungi | could that rest api be on localhost, with a job we trigger on that node from zuul after any job pushes an image to tarballs.o.o it needs to incorporate? | 22:50 |
| mordred | fungi: quite possibly, yes | 22:51 |
| pabelanger | jeblair: danke! | 22:51 |
| *** pradk has quit IRC | 22:51 | |
| NobodyCam | Good Afternoon Infra folk. Just a off the wall question (as always) is there a order to restart G(errit)/J(enkins)/Z(uul) | 22:51 |
| NobodyCam | I ask because I restarted Jenkins / zuul and now have a bunch of errors like: | 22:51 |
| NobodyCam | ERROR zuul.Gearman: Job <gear.Job 0x7fb69c116450 handle: None name: build:zulu-cross-python-check-changes-no-build unique: f1016cdc1b5545d9b59cee663267c307> is not registered with Gearman | 22:51 |
| mordred | fungi: but ultimately I think we wind up doing that much more inefficiently than if we figure out how to allow things to publish to a local docker registry using docker - because otherwise we're re-copying a ton of stuff over and over again | 22:51 |
| * mnaser sees a lot of talk about the docker repo | 22:52 | |
| *** ijw has quit IRC | 22:52 | |
| mnaser | i haven't had a chance to reupdate the spec because of a major concern -- "this can be solved by zuulv3" | 22:52 |
| jeblair | NobodyCam: once nodepool nodes come online and jenkins registers that it can run those jobs, those errors will dissipate | 22:52 |
| mnaser | if we ignore that, we could start moving forward with that. unless another solution is being proposed right now | 22:53 |
| mordred | fungi: I think the thing you're saying is definitely the thing we can do for now | 22:53 |
| fungi | got it. so it's not just some files you stick in a directory, run a script to generate indices, and then publish with a webserver? | 22:53 |
| mordred | fungi: nope. not even a little bit | 22:53 |
| NobodyCam | jeblair: Thank you.. Will keep an eye on things :) | 22:53 |
| NobodyCam | hey hey g'Afternoon mordred :) | 22:53 |
| * fungi wonders when/why simple file distribution became so complicated | 22:53 | |
| mordred | fungi: because fancy? | 22:54 |
| mordred | fungi: it's hard to build a business model to become a unicorn if you use existing technology in a way that's immediately interoperable with things that existed before | 22:54 |
| mordred | hey NobodyCam ! | 22:54 |
| mnaser | if you dont mind, can i get caught up with if we're trying to run mirrors or a repo to push things to (ala tarballs.o.o) | 22:54 |
| mordred | mnaser: yes! I can help with that | 22:55 |
| fungi | ahh, makes sense | 22:55 |
| NobodyCam | :) | 22:55 |
| mordred | mnaser: there are a few parallel things - so stick with me for a sec | 22:55 |
| mordred | mnaser: for the effort "make consuming docker images on build slaves faster/more reliable" - clarkb has a mod_proxy entry on each of the webservers in our per-region mirrors to do a caching web proxy | 22:56 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Revert "Remove ovb updates job from promotion pipeline" https://review.openstack.org/452866 | 22:56 |
| mordred | mnaser: so the work there should be "make sure docker points to the per-region mirror URL instead of the dockerhub URL" and that should be good to go | 22:56 |
| clarkb | (assuming its actually just http) | 22:57 |
| mnaser | (i'll keep my thoughts for the end about these things) | 22:57 |
| mordred | yah. if the mod_proxy approach does not work, then the fallback there is to run a registry in pull-through-cache mode on each per-region mirror | 22:57 |
| mordred | we're starting with mod_proxy because we're already using that for some other things | 22:57 |
| *** sdague has quit IRC | 22:57 | |
| inc0 | clarkb: https | 22:58 |
| mordred | mnaser: for the "we want a registry we can push to" - we're reviving the puppet-docker_registry module and some combinatoin of SamYaple, you and folks from #tripleo are going to work on getting the registry module ready | 22:58 |
| *** sree has joined #openstack-infra | 22:58 | |
| mordred | so that we can install a docker registry machine to be a place we can publish images to | 22:59 |
| mordred | we still need to sort out exact workflow around actually publishing images to it - but it's most likely that the first versoin of that is going to be VERY similar to how we do other things today | 22:59 |
| *** marst_ has quit IRC | 23:00 | |
| pabelanger | just left comments on 401003 | 23:00 |
| mordred | which is that jobs that build images will upload them to tarballs.o.o and then we will have a trusted job that will download those and inject them into the registry | 23:00 |
| fungi | i gather that would mostly be for things like updating branch-tip images for in-developent versions of stuff | 23:00 |
| mordred | fungi: yah | 23:00 |
| fungi | not for final release artifacts you'd want to be more globally useful | 23:00 |
| inc0 | fungi: ideally on merge on stable branches and master | 23:01 |
| fungi | yup | 23:01 |
| mordred | finally there is "publish to dockerhub" - which can be also done as part of the same system that downloads form tarballs.o.o and uploads to the local registry - it can just as easily upload to dockerhub | 23:01 |
| inc0 | totally awesome: also push tagged stuff and allow people of world around to pull from it | 23:01 |
| mordred | but also we expect that process to work better once zuul v3 is out | 23:01 |
| fungi | tagged stuff you want the world to consume would presumably go to dockerhub anyway | 23:01 |
| *** adriant has joined #openstack-infra | 23:01 | |
| mordred | inc0: I believe we want to publish actual tagged releases to dockerhub and quay.io ratherthan running a 'production' registry that people's deployments might point at | 23:02 |
| mordred | fungi: yah. that | 23:02 |
| inc0 | either way works | 23:02 |
| mordred | I don't want to be on the hook for running a dockerhub that might directly break mnaser's cloud :) | 23:02 |
| mnaser | :D | 23:02 |
| inc0 | just throwing it out there, I don't have big incentive to use our infra for it | 23:02 |
| inc0 | if we can get autopushed done properly | 23:02 |
| mordred | yup. I think we can get pushes to dockerhub and quay.io done in the fulllness of time - and our caching/proxy approach should allow those to be consumed | 23:03 |
| *** sree has quit IRC | 23:03 | |
| *** annegentle has quit IRC | 23:03 | |
| mordred | now - I _do_ think we should run a registry and publish to it just for completeness- just like we publish tarball releases to tarballs.o.o - even though we also publish them to pypi | 23:03 |
| *** ijw has joined #openstack-infra | 23:03 | |
| *** ijw has quit IRC | 23:03 | |
| mordred | just because having more than one publication point is good for allowing people to double check content validity and stuff | 23:04 |
| fungi | (speaking of the fullness of time, please review mnaser's spec about automated publishing to dockerhub if you have thoughts on the matter) | 23:04 |
| inc0 | and have backup source if dockerhub dies in fire for various reasons | 23:04 |
| *** ijw has joined #openstack-infra | 23:04 | |
| mnaser | so to summarize work items of what mordred explained: 1) infra sets up some sort of per-region reverse cache for dockerhub to reduce internet pulls for things from dockerhub. b) infra+consumers of docker registries work together to build something that can stand up a docker registry, 3) the spec regarding publishing ( https://review.openstack.org/#/c/447524/ ) could be revised to include 'publish to local' and 'publish to dockerhub' | 23:05 |
| jeblair | btw, i'm not planning on updating https://review.openstack.org/401003 i just wrote the first patchset for inc0 to fill in the blanks; if we use it, i expect someone else to update it | 23:05 |
| fungi | right, like with sdists/wheels and pypi.p.o vs tarballs.o.o | 23:05 |
| pabelanger | jeblair: I was planning on pushing something up | 23:05 |
| *** rbuzatu has quit IRC | 23:05 | |
| jeblair | cool | 23:05 |
| mnaser | that's a nice start | 23:05 |
| fungi | mnaser: that sounds like what i got from the discussion, yes | 23:06 |
| mnaser | i would be happy to finish that stuff up as i've done too much puppet for my own good | 23:06 |
| inc0 | mnaser: if we get local registry in infra | 23:06 |
| mnaser | id just have to ask if there's a simple way to test these on a local vms (or something infra has used to easily bootstrap things locally in a vm) | 23:06 |
| inc0 | I think we can table autopusher to dockerhub for zuulv3 | 23:06 |
| mnaser | quite honestly, i've been working on our defcore stuff all day and i cant stand looking at tempest a second more | 23:07 |
| mnaser | so i dont mind toying around with this for now | 23:07 |
| mnaser | i dont expect it to take very long | 23:07 |
| jeblair | mnaser: yeah, https://docs.openstack.org/infra/system-config/sysadmin.html#making-a-change-in-puppet has a quick how-to local test infra puppet changes | 23:07 |
| inc0 | mnaser: right, if you make docker registry work in infra, I owe you beer | 23:07 |
| *** mat128 has quit IRC | 23:07 | |
| mnaser | on a super side note, would it be okay to install docker + run registry in docker? | 23:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Collect test-results for tox jobs https://review.openstack.org/452991 | 23:07 |
| mnaser | in my experience, its a bit easier to work with and much better tested overall (and we can easily stay on newer releases) | 23:08 |
| mnaser | rather than rely on os deployed ones | 23:08 |
| inc0 | well registry is glorified http server | 23:08 |
| mnaser | thank you jeblair and inc0 i'll get on this today | 23:08 |
| inc0 | so not sure if we want to add complexity of installind and configuring docker daemon | 23:08 |
| mnaser | mmkay | 23:09 |
| mnaser | ill take your word for it inc0 | 23:09 |
| *** ijw has quit IRC | 23:09 | |
| inc0 | yeah...it's a golang bin which we can, quite literally, curl down and run | 23:09 |
| inc0 | or do apt-get install | 23:09 |
| pabelanger | mnaser: I am going to push up an update to 401003 shortly, we can iterate on that if you'd like | 23:09 |
| bkero | I don't think golang advocates using distro packages. | 23:09 |
| mnaser | lets see what release its running, i know docker registry performance improved a lot lately so we'll have to check | 23:10 |
| mnaser | please pabelanger -- in the meantime i'll do the little work needed here to stand up a local env | 23:10 |
| inc0 | mnaser: registry 2 is better....a lot better... | 23:10 |
| mnaser | and i think distros still dont ship 2 afaik | 23:10 |
| inc0 | for what its worth, project already exists in project.config https://review.openstack.org/#/q/project:openstack-infra/puppet-docker_registry | 23:11 |
| mordred | bkero: I do not care | 23:11 |
| inc0 | in this case I'd go for curled binary | 23:11 |
| inc0 | mordred :D | 23:11 |
| mordred | in this case I would go for apt-get installing it | 23:11 |
| mordred | since it's in ubuntu | 23:11 |
| mordred | and there is no reason not to use it | 23:11 |
| bkero | mordred: I'd care if we needed to call on them to support one day and they told us to pound sand. | 23:11 |
| inc0 | mordred: if apt doesnt have v2 | 23:11 |
| mnaser | mordred id do that, but i think its best we verify the version available | 23:12 |
| inc0 | then we really want v2... | 23:12 |
| mnaser | the old registry was very bad and slow | 23:12 |
| mordred | bkero: I have zero expectatoins that they will ever support it anyway | 23:12 |
| mordred | and ubuntu xenial has v2 | 23:12 |
| mordred | so we're fine | 23:12 |
| inc0 | indeed we are | 23:12 |
| mnaser | oh cool | 23:12 |
| inc0 | cool mnaser thanks for help with this on | 23:13 |
| inc0 | e | 23:13 |
| inc0 | I'll get to full release upgrade gates when we have registry... | 23:13 |
| *** marst has joined #openstack-infra | 23:13 | |
| inc0 | (as in...gates that will deploy stable and upgrae to master...how cool is that?:D) | 23:13 |
| mordred | Version: 2.3.0~ds1-1 | 23:13 |
| mordred | is what's in ubuntu | 23:13 |
| mnaser | inc0 awesome. np. ill do my best to take this as far as i can | 23:13 |
| inc0 | mnaser: let me know if you need any help | 23:14 |
| * bkero remembers this whole thread about it https://twitter.com/bketelsen/status/843935584708182019 | 23:14 | |
| mnaser | if i can get feedback circled quickly, it would make it easier so hopefully ill base my work on what small things jeblair and pabelanger started off | 23:14 |
| *** andreww has quit IRC | 23:14 | |
| mordred | bkero: that thread is FUD | 23:14 |
| inc0 | bkero: after spending last week + weekend with golang | 23:15 |
| inc0 | I don't care at all what they do or do not propse | 23:15 |
| inc0 | because this is bag of bull... | 23:15 |
| bkero | mordred: about the only thing I took away is that person claiming that distro packages change paths | 23:15 |
| bkero | Which I wouldn't doubt, but it's likely not to matter for our use. | 23:15 |
| mordred | bkero: yah. that's what I get too | 23:15 |
| mordred | also - distro package managers don't change paths when upstream paths are in the right place :) | 23:16 |
| bkero | Haha really | 23:16 |
| mordred | we've been putting files in places for a LONG TIME - it's not rocket science | 23:16 |
| bkero | Let's talk about the LSB | 23:16 |
| mordred | unless you just flat don't care | 23:16 |
| *** ssbarnea has quit IRC | 23:16 | |
| clarkb | also thats about installing Go, not about installing things that are compiled from Go | 23:16 |
| * bkero peers into /usr/local | 23:16 | |
| mordred | yah | 23:17 |
| *** aunnam has quit IRC | 23:17 | |
| mordred | to be fair - I have go installed not-from-packages - but I have it installed in ~/.local/bin/go too | 23:17 |
| bkero | If distro-ifying other package managers is anything to go by, it'll also disable auto-update mechanisms for itself and whatever it considers $GOBIN/$GOHOME and accompanying libraries installed system-wide. | 23:17 |
| mnaser | for sometime | 23:17 |
| mnaser | i had to work a lot with smartos, stuff created of joyent from whatever was left of sun | 23:17 |
| mordred | oh wow | 23:18 |
| mnaser | if you like things going in the right place, you'd love it | 23:18 |
| clarkb | mnaser: you are reminding me of my days using blitz | 23:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: [WIP] Create docker01.openstack.org https://review.openstack.org/401003 | 23:18 |
| pabelanger | mnaser: untested ^ | 23:18 |
| pabelanger | should get us started | 23:18 |
| mordred | bkero: at least go binaries only need GOHOME at build time and not at runtime | 23:18 |
| mordred | woot! | 23:18 |
| mnaser | pabelanger sweet, i'll cherry pick and see where i go | 23:18 |
| pabelanger | things like HTTPs need to be solved | 23:19 |
| pabelanger | and user auth | 23:19 |
| pabelanger | need to read up more on it | 23:19 |
| pabelanger | for now, just added htaccess username / password | 23:19 |
| NobodyCam | jeblair: Fyi- I had to edit a slave to get the jobs reregistered with gearman | 23:19 |
| *** arxcruz has quit IRC | 23:19 | |
| *** aunnam has joined #openstack-infra | 23:19 | |
| mordred | fungi: fwiw, reading the docs, I believe we _could_ write a thing to just write image layers to filesystem locations and serve them with apache | 23:19 |
| mordred | fungi: it would be completely custom development on our part | 23:19 |
| mordred | fungi: but it does not appear that there are any things in the docker-registry API that cannot be handled by just putting files on disk in the right layout | 23:20 |
| inc0 | well yes, we can make docker registry from a http server | 23:20 |
| inc0 | that just navigates filesystem and serves content | 23:20 |
| mordred | inc0: do you know if there is already a tool to explode a docker image into the right files on filesystem? | 23:20 |
| inc0 | well, we do it today with tarballs | 23:21 |
| *** yamahata has quit IRC | 23:21 | |
| inc0 | we simply run registry in gate, push to localhost and tarball what comes out on the other end | 23:21 |
| *** rbuzatu has joined #openstack-infra | 23:21 | |
| *** arxcruz has joined #openstack-infra | 23:21 | |
| bkero | hehe oh man /usr/lib/go/bin/go | 23:21 |
| inc0 | go bin go | 23:21 |
| mordred | bkero: I can't IMAGINE why distros would move that | 23:21 |
| inc0 | run/forest/run | 23:21 |
| bkero | mordred: actually that's an ubuntuism | 23:22 |
| mordred | bkero: wat? | 23:22 |
| bkero | update-alternatives --remove go /usr/lib/go/bin/go | 23:22 |
| bkero | from the preinst rules file | 23:22 |
| mordred | oh. alternatives | 23:22 |
| mordred | of course | 23:22 |
| fungi | might more accurately be a debianism | 23:22 |
| * bkero is down the rabbit hole of apt-get source golang-go | 23:22 | |
| mordred | inc0: ok - I'm going to look at that and learn a little bit more | 23:22 |
| *** doffm has left #openstack-infra | 23:22 | |
| bkero | Could be. It's a debian rules script | 23:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Updates to adding mirror documentation https://review.openstack.org/452992 | 23:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Collect test-results for tox jobs https://review.openstack.org/452991 | 23:23 |
| mordred | inc0: might not be worth anything - but it would be good to konw all th ethings | 23:23 |
| *** wolverineav has quit IRC | 23:23 | |
| inc0 | mordred: I'd still be more comfortable with having registry running | 23:23 |
| mordred | totoally | 23:23 |
| mordred | inc0: I just like to know what the fallback options are | 23:23 |
| clarkb | zaro: ok, so next step is pushing a change to upgrade review-dev, and we'll have to follow our rough steps for doing a normal upgrade. Any chance you have time some time this week to run through that with me? | 23:23 |
| bkero | fungi: Although the changelog is @canonical.com addresses | 23:23 |
| inc0 | mordred: sure, good hacking project | 23:23 |
| *** signed8bit is now known as signed8bit_Zzz | 23:23 | |
| inc0 | as far as I know it's just glorified file server | 23:23 |
| mordred | inc0: you know, for when docker's VC folks tell them to remove their source code form the internets and turn it all proprietary | 23:23 |
| fungi | bkero: even on -1 package revs? | 23:23 |
| mnaser | alright almost getting that local machine running and then the puppet apply fun will start soon | 23:23 |
| inc0 | lol | 23:24 |
| ianw | nobody reviewed my comedy changelog yet | 23:24 |
| inc0 | well, if that happends I'm starting project called vac | 23:24 |
| bkero | fungi: yes, and control says 'Maintainer: Ubuntu Developers' | 23:24 |
| inc0 | and build rockets there | 23:24 |
| *** signed8bit_Zzz is now known as signed8bit | 23:25 | |
| *** rbuzatu has quit IRC | 23:25 | |
| fungi | bkero: for the golang compiler? wacky that they aren't importing that from debian | 23:26 |
| fungi | but i've learned to stop being surprised there | 23:26 |
| mnaser | pabelanger: if that does install things successfully (still getting a sample machine setup). what would be the next step? | 23:26 |
| clarkb | bkero: is there something that dictates the binary path? I guess I don't grok what the issue is | 23:26 |
| clarkb | then again I had an nfs mounted /usr/local/bin/ghc for things so maybe I am one of thoes terrible people the golangers don't like | 23:27 |
| pabelanger | mordred: I'd be okay with mirroring docker infrastructure without using docker things. Hosting a registry, I have hopes :) | 23:27 |
| mordred | inc0: can you point me to the place where you do the "run local docker registry, push to it, then tar what comes out hte other end" part? | 23:27 |
| mordred | pabelanger: ++ | 23:27 |
| pabelanger | mnaser: we'd bring a server online for the control plane | 23:28 |
| pabelanger | mnaser: actually, -1 on the review. We need to make sure afs is also installed | 23:28 |
| pabelanger | I think server.pp has a flag | 23:28 |
| mnaser | yep | 23:29 |
| mnaser | afs => true | 23:29 |
| *** reed_ has joined #openstack-infra | 23:29 | |
| pabelanger | yup | 23:30 |
| bkero | https://tracker.debian.org/media/packages/g/golang/rules-2%3A1.6.1-2 has a few gems though. "Remove Plan9 rc(1) scripts" | 23:30 |
| bkero | Doesn't look like there's anything nefarious in there though. | 23:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: [WIP] Create docker01.openstack.org https://review.openstack.org/401003 | 23:31 |
| bkero | lots of touching | 23:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Collect test-results for tox jobs https://review.openstack.org/452991 | 23:32 |
| mnaser | pabelanger are we okay with storing the encrypted value in hiera for auth or should we encrypt the plain value in puppet so that the password is visible in hiera | 23:32 |
| pabelanger | mnaser: we have encrypted things in hiera already | 23:33 |
| bkero | and the S390 port-in-a-patch, and the passing -no-pie to the linker | 23:33 |
| pabelanger | so, that should be fine | 23:33 |
| mnaser | okay cool | 23:33 |
| pabelanger | mnaser: I think the better question is, is htpasswd the right auth to use | 23:33 |
| pabelanger | I see token support | 23:34 |
| mnaser | pabelanger token is a bit more complicated, it pretty much relies on an external api for auth so it involves running another piece of software | 23:34 |
| *** dizquierdo has quit IRC | 23:34 | |
| mnaser | kinda like keystone so to speak | 23:34 |
| pabelanger | ah, ya. we don't want that | 23:34 |
| mnaser | you can setup things like this: https://github.com/cesanta/docker_auth | 23:34 |
| *** reed_ has quit IRC | 23:35 | |
| pabelanger | we should also consider wrapping the registry behind apache | 23:35 |
| clarkb | I get /usr/bin/go on tumbleweed | 23:36 |
| clarkb | which is symlink for alternatives to /usr/lib64/go/bin/go | 23:36 |
| bkero | No GOVER in there? | 23:38 |
| *** adrian_otto has quit IRC | 23:38 | |
| *** armax has quit IRC | 23:38 | |
| bkero | Ubuntu definitely involves /usr/lib/go-${GOVER}/ | 23:38 |
| bkero | Which is apparently the GOROOT | 23:38 |
| *** dingyichen has joined #openstack-infra | 23:39 | |
| clarkb | no /usr/lib64 doesn't hve a bunch of go versions | 23:41 |
| clarkb | not sure why they use alternatives | 23:41 |
| clarkb | probably because as a distro /usr/bin/ is in your path and they don't want to force users to add /usr/lib64/go/bin to their paths | 23:41 |
| *** mat128 has joined #openstack-infra | 23:43 | |
| mnaser | pabelanger do we want to do the https inside apache then? | 23:43 |
| *** signed8bit is now known as signed8bit_Zzz | 23:44 | |
| pabelanger | mnaser: ya, lets bind docker to 8080 localhost, then we can https apache on 80 / 443 for now | 23:45 |
| *** s-shiono has joined #openstack-infra | 23:45 | |
| mnaser | pabelanger cool, on it | 23:45 |
| pabelanger | then we can copy pasta vhost configuration for ssl | 23:46 |
| *** Hal has quit IRC | 23:46 | |
| *** asselin has quit IRC | 23:46 | |
| *** asselin has joined #openstack-infra | 23:47 | |
| *** tosky has quit IRC | 23:47 | |
| inc0 | mordred: https://github.com/openstack/kolla/blob/master/tools/start-registry <- this is registry starting | 23:48 |
| inc0 | pushing is just kolla-build push | 23:48 |
| inc0 | ahh | 23:48 |
| inc0 | sorry, wrong place | 23:48 |
| inc0 | https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh#L181-L185 | 23:48 |
| *** s-shiono_ has joined #openstack-infra | 23:49 | |
| inc0 | and then we tar contents of /tmp/kolla_registry | 23:49 |
| *** s-shiono has quit IRC | 23:51 | |
| *** asselin_ has joined #openstack-infra | 23:54 | |
| *** asselin has quit IRC | 23:57 | |
| *** mat128 has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!