| *** gouthamr has joined #openstack-infra | 00:02 | |
| *** dklyle has quit IRC | 00:03 | |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Change include_tasks back to include https://review.openstack.org/592220 | 00:03 |
|---|---|---|
| *** dingyichen has joined #openstack-infra | 00:05 | |
| *** jamesmcarthur has joined #openstack-infra | 00:13 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Increase releasenotes build timeout https://review.openstack.org/592159 | 00:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add 'build' method to API https://review.openstack.org/592225 | 00:23 |
| corvus | clarkb: can you review https://review.openstack.org/592206 for swift logs? | 00:24 |
| clarkb | corvus: do we have the base-test results of that somewhere? or just planning to get this in then iterate? | 00:26 |
| clarkb | (I want to see them if they already exist) | 00:27 |
| clarkb | oh wait base test requires this merge first | 00:27 |
| *** zzzeek has quit IRC | 00:31 | |
| *** zzzeek has joined #openstack-infra | 00:31 | |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Remove mariadb link from mirrors" https://review.openstack.org/577944 | 00:33 |
| *** bobh has quit IRC | 00:33 | |
| mordred | clarkb: puppet seems to be running fine | 00:34 |
| mordred | with the new patch | 00:34 |
| mordred | so I'm going to go eat dinner now | 00:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Swift logs: htmlify logs https://review.openstack.org/592206 | 00:34 |
| mordred | clarkb: however: | 00:37 |
| mordred | mirror02.us-west-1.packethost.openstack.org : ok=0 changed=0 unreachable=1 failed=0 | 00:37 |
| mordred | host key has changed on that host | 00:37 |
| mordred | clarkb: I'm not in a position to investigate whether that's valid/expected or not | 00:38 |
| mordred | also mirror01.sto2.citycloud.openstack.org : ok=0 changed=0 unreachable=1 failed=0 | 00:38 |
| fungi | that packethost mirror has been rebuilt so many times recently, i'm unsurprised | 00:38 |
| mordred | kk. cool. thought I'd report to channel in case anyone wants to sort it out | 00:38 |
| fungi | and are we actually using citycloud sto2? | 00:39 |
| ianw | yeah ++ packethost one is expected | 00:41 |
| clarkb | ya that may be my fail the fip remained the same bot hostkey wouldve changed | 00:42 |
| clarkb | fungi not currently | 00:42 |
| clarkb | I did email them asking if we canturn it back on but no response yet | 00:42 |
| clarkb | daniel had autoreply for vacation set | 00:42 |
| *** jamesmcarthur has quit IRC | 00:43 | |
| *** jamesmcarthur has joined #openstack-infra | 00:43 | |
| *** longkb has joined #openstack-infra | 00:55 | |
| *** dave-mccowan has quit IRC | 01:02 | |
| *** dave-mccowan has joined #openstack-infra | 01:03 | |
| *** r-daneel has joined #openstack-infra | 01:13 | |
| *** bobh has joined #openstack-infra | 01:14 | |
| *** zzzeek has quit IRC | 01:14 | |
| *** zzzeek has joined #openstack-infra | 01:16 | |
| *** bobh has quit IRC | 01:23 | |
| *** r-daneel has quit IRC | 01:29 | |
| *** jiapei has joined #openstack-infra | 01:37 | |
| *** hongbin_ has joined #openstack-infra | 01:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add 'build' method to API https://review.openstack.org/592225 | 01:43 |
| *** yamahata has quit IRC | 01:43 | |
| *** larainema has joined #openstack-infra | 01:43 | |
| *** graphene has quit IRC | 01:45 | |
| *** graphene has joined #openstack-infra | 01:46 | |
| kei-ichi | Hi, infra-team. Could you plz add me as member of this. https://review.openstack.org/#/admin/groups/1933,members | 02:01 |
| ianw | kei-ichi: do you have two accounts? | 02:10 |
| *** tinwood has quit IRC | 02:10 | |
| kei-ichi | Yes, but plz only add keiichi.hikita@gmail.com. Sorry for confusing. | 02:10 |
| ianw | ok done | 02:10 |
| *** tinwood has joined #openstack-infra | 02:11 | |
| *** jamesmcarthur has quit IRC | 02:13 | |
| kei-ichi | ianw thanks! | 02:14 |
| *** bobh has joined #openstack-infra | 02:15 | |
| *** anteaya has quit IRC | 02:18 | |
| *** jamesmcarthur has joined #openstack-infra | 02:18 | |
| openstackgerrit | neilsun proposed openstack-infra/zuul master: Add type check for zuul conf https://review.openstack.org/591917 | 02:22 |
| *** bobh has quit IRC | 02:24 | |
| *** armax has quit IRC | 02:38 | |
| *** ltomasbo has quit IRC | 02:42 | |
| *** dalvarez has quit IRC | 02:42 | |
| *** rlandy|afk is now known as rlandy | 02:44 | |
| openstackgerrit | Keiichi Hikita proposed openstack-infra/project-config master: Adding jobs for qinling-dashboard https://review.openstack.org/592251 | 02:44 |
| *** rlandy has quit IRC | 02:44 | |
| *** jpena|off has quit IRC | 02:45 | |
| *** bobh has joined #openstack-infra | 02:46 | |
| *** radez has quit IRC | 02:46 | |
| *** amoralej|off has quit IRC | 02:47 | |
| *** bobh has quit IRC | 02:47 | |
| *** psachin has joined #openstack-infra | 02:48 | |
| *** bobh has joined #openstack-infra | 02:48 | |
| *** bobh has quit IRC | 02:48 | |
| *** bobh has joined #openstack-infra | 02:49 | |
| *** bobh has quit IRC | 02:51 | |
| *** bobh has joined #openstack-infra | 02:55 | |
| *** jamesmcarthur has quit IRC | 03:00 | |
| *** bobh has quit IRC | 03:00 | |
| *** jamesmcarthur has joined #openstack-infra | 03:01 | |
| *** yamahata has joined #openstack-infra | 03:06 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Add treasuremap noop-jobs to project-config https://review.openstack.org/592255 | 03:09 |
| *** jamesmcarthur has quit IRC | 03:09 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Add treasuremap noop-jobs to project-config https://review.openstack.org/592255 | 03:10 |
| *** jamesmcarthur has joined #openstack-infra | 03:14 | |
| *** NeilSun has joined #openstack-infra | 03:16 | |
| *** rosmaita has quit IRC | 03:22 | |
| *** ykarel has joined #openstack-infra | 03:27 | |
| *** hongbin_ has quit IRC | 03:29 | |
| *** udesale has joined #openstack-infra | 03:36 | |
| *** Bhujay has quit IRC | 03:43 | |
| *** dave-mccowan has quit IRC | 04:04 | |
| AJaeger | config-core, could you review https://review.openstack.org/592158 , please? We merged https://review.openstack.org/#/c/592159/ to increase the building timeout, should increase the publish one as well... | 04:13 |
| mnaser | AJaeger: can’t help as much :( | 04:14 |
| ianw | sorry i thought i looked at those | 04:16 |
| *** jamesmcarthur has quit IRC | 04:17 | |
| AJaeger | thanks, mnaser and ianw | 04:19 |
| *** jamesmcarthur has joined #openstack-infra | 04:20 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Set timeout for job 'openstack-tox-py36' https://review.openstack.org/592192 | 04:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Increase releasenotes publish timeout https://review.openstack.org/592158 | 04:23 |
| *** jamesmcarthur has quit IRC | 04:25 | |
| *** gcb_ has quit IRC | 04:31 | |
| *** janki has joined #openstack-infra | 04:34 | |
| *** jamesmcarthur has joined #openstack-infra | 04:34 | |
| *** Bhujay has joined #openstack-infra | 04:38 | |
| *** jamesmcarthur has quit IRC | 04:43 | |
| *** gcb_ has joined #openstack-infra | 04:47 | |
| *** ramishra has joined #openstack-infra | 04:48 | |
| *** jamesmcarthur has joined #openstack-infra | 04:59 | |
| *** graphene has quit IRC | 05:04 | |
| *** graphene has joined #openstack-infra | 05:05 | |
| *** apetrich has joined #openstack-infra | 05:06 | |
| *** Bhujay has quit IRC | 05:07 | |
| *** Bhujay has joined #openstack-infra | 05:08 | |
| *** ramishra has quit IRC | 05:12 | |
| *** lewo has quit IRC | 05:13 | |
| *** lewo has joined #openstack-infra | 05:14 | |
| *** ramishra has joined #openstack-infra | 05:16 | |
| *** e0ne has joined #openstack-infra | 05:18 | |
| *** ramishra has quit IRC | 05:22 | |
| *** ramishra has joined #openstack-infra | 05:24 | |
| *** jamesmcarthur has quit IRC | 05:25 | |
| *** pcaruana has quit IRC | 05:26 | |
| *** ramishra has quit IRC | 05:29 | |
| *** jamesmcarthur has joined #openstack-infra | 05:37 | |
| *** jamesmcarthur has quit IRC | 05:41 | |
| *** ramishra has joined #openstack-infra | 05:50 | |
| *** dpawlik has joined #openstack-infra | 06:01 | |
| *** Bhujay has quit IRC | 06:02 | |
| *** Bhujay has joined #openstack-infra | 06:03 | |
| *** e0ne has quit IRC | 06:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/592278 | 06:04 |
| *** jamesmcarthur has joined #openstack-infra | 06:08 | |
| *** e0ne has joined #openstack-infra | 06:12 | |
| *** jamesmcarthur has quit IRC | 06:13 | |
| openstackgerrit | Keiichi Hikita proposed openstack-infra/project-config master: Adding jobs for qinling-dashboard https://review.openstack.org/592251 | 06:15 |
| *** ykarel is now known as ykarel|afk | 06:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 06:34 |
| *** ykarel|afk has quit IRC | 06:34 | |
| *** jiapei has quit IRC | 06:37 | |
| *** jamesmcarthur has joined #openstack-infra | 06:37 | |
| *** Dmitrii-Sh has quit IRC | 06:40 | |
| *** beisner has quit IRC | 06:40 | |
| *** Dmitrii-Sh has joined #openstack-infra | 06:40 | |
| *** beisner has joined #openstack-infra | 06:40 | |
| *** jamesmcarthur has quit IRC | 06:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/592278 | 06:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Adding jobs for qinling-dashboard https://review.openstack.org/592251 | 06:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Adding jobs for qinling-dashboard https://review.openstack.org/592251 | 06:46 |
| *** jd_ has joined #openstack-infra | 06:48 | |
| *** pcaruana has joined #openstack-infra | 06:48 | |
| *** jamesmcarthur has joined #openstack-infra | 06:51 | |
| *** slaweq_ has joined #openstack-infra | 06:51 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for etherpad.o.o https://review.openstack.org/590029 | 06:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for ethercalc.o.o https://review.openstack.org/590030 | 06:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for eavesdrop.o.o https://review.openstack.org/590048 | 06:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for paste.o.o https://review.openstack.org/590492 | 06:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for planet.o.o https://review.openstack.org/590493 | 06:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for logstash-worker01 https://review.openstack.org/590499 | 06:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for all logstash workers https://review.openstack.org/590500 | 06:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for firehose.o.o https://review.openstack.org/591500 | 06:51 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for review and review01 https://review.openstack.org/591516 | 06:51 |
| *** ykarel|afk has joined #openstack-infra | 06:53 | |
| *** ykarel|afk is now known as ykarel | 06:54 | |
| *** e0ne has quit IRC | 06:56 | |
| *** e0ne has joined #openstack-infra | 06:57 | |
| *** e0ne has quit IRC | 06:59 | |
| *** jamesmcarthur has quit IRC | 07:00 | |
| *** slaweq_ has quit IRC | 07:00 | |
| *** slaweq_ has joined #openstack-infra | 07:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Retire rst2bash (step 2) https://review.openstack.org/592289 | 07:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Retire rst2bash (step 4) https://review.openstack.org/592290 | 07:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Retire rst2bash: Remove unused jobs https://review.openstack.org/592292 | 07:14 |
| *** rcernin has quit IRC | 07:20 | |
| *** shardy has joined #openstack-infra | 07:21 | |
| *** jpena has joined #openstack-infra | 07:31 | |
| *** jpich has joined #openstack-infra | 07:34 | |
| *** jamesmcarthur has joined #openstack-infra | 07:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Build releasenotes when tox.ini changes https://review.openstack.org/592298 | 07:35 |
| AJaeger | cmurphy: could you review https://review.openstack.org/#/c/591740 and https://review.openstack.org/#/c/591786 to help finish docs python3 goal, please? | 07:37 |
| cmurphy | AJaeger: done | 07:39 |
| AJaeger | thanks, cmurphy ! | 07:41 |
| *** olivierbourdon38 has joined #openstack-infra | 07:48 | |
| *** e0ne has joined #openstack-infra | 07:48 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Move sphinx + deps to doc/requirements.txt https://review.openstack.org/591565 | 07:58 |
| *** dhellmann[m] has quit IRC | 08:11 | |
| *** aspiers[m] has quit IRC | 08:11 | |
| *** warlord77[m] has quit IRC | 08:11 | |
| *** linshuicheng[m] has quit IRC | 08:11 | |
| *** electrofelix has joined #openstack-infra | 08:20 | |
| *** alexchadin has joined #openstack-infra | 08:22 | |
| *** dalvarez has joined #openstack-infra | 08:24 | |
| *** rpittau has joined #openstack-infra | 08:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 08:38 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: use BrowserRouter https://review.openstack.org/592321 | 08:38 |
| *** vivsoni has joined #openstack-infra | 08:40 | |
| *** ianychoi_ has quit IRC | 08:42 | |
| *** derekh has joined #openstack-infra | 08:43 | |
| *** roman_g has joined #openstack-infra | 08:52 | |
| *** panda|off is now known as panda | 08:56 | |
| *** alexchadin has quit IRC | 09:00 | |
| *** dingyichen has quit IRC | 09:00 | |
| *** dingyichen has joined #openstack-infra | 09:01 | |
| *** dingyichen has quit IRC | 09:03 | |
| *** jaosorior has quit IRC | 09:10 | |
| *** alexchadin has joined #openstack-infra | 09:11 | |
| *** e0ne has quit IRC | 09:14 | |
| *** d0ugal has quit IRC | 09:14 | |
| *** d0ugal has joined #openstack-infra | 09:18 | |
| AJaeger | infra-root, I wonder why zuul did not run any jobs on https://review.openstack.org/#/c/592248/ . I noticed that releasenotes job is not run due to files section (fixed with https://review.openstack.org/592298 - please review) but the others should run AFAIK. Anything in the logs for 592248? | 09:23 |
| AJaeger | I would expect the openstack-tox-pep8 job to run for sure ^ | 09:26 |
| *** dpawlik has quit IRC | 09:28 | |
| *** dpawlik has joined #openstack-infra | 09:28 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 09:37 |
| *** alexchadin has quit IRC | 09:41 | |
| *** olivierbourdon38 has quit IRC | 09:52 | |
| *** olivierbourdon38 has joined #openstack-infra | 09:53 | |
| *** sambetts|afk is now known as sambetts | 09:53 | |
| *** markmcd has quit IRC | 09:59 | |
| *** ykarel is now known as ykarel|lunch | 10:01 | |
| *** longkb has quit IRC | 10:04 | |
| *** d0ugal has quit IRC | 10:07 | |
| *** d0ugal has joined #openstack-infra | 10:13 | |
| *** rf0lc0 has quit IRC | 10:15 | |
| *** rf0lc0 has joined #openstack-infra | 10:16 | |
| *** dbecker has joined #openstack-infra | 10:19 | |
| *** dbecker has quit IRC | 10:19 | |
| ianw | 2018-08-16 07:38:42,893 DEBUG zuul.Pipeline.openstack.check: <QueueItem 0x7f0900a05358 for <Change 0x7f09206500b8 openstack/qinling-dashboard 592248,8> in check> is a failing item because ['it has an invalid configuration'] | 10:23 |
| ianw | AJaeger: ^ ? | 10:23 |
| AJaeger | thanks, ianw - let's see what corvus makes out of that one ^ | 10:24 |
| *** jaosorior has joined #openstack-infra | 10:28 | |
| AJaeger | I see one problem - using a job name as template. But I would expect Zuul to report that. | 10:28 |
| *** mnasiadka has joined #openstack-infra | 10:28 | |
| *** alexchadin has joined #openstack-infra | 10:31 | |
| *** dpawlik has quit IRC | 10:31 | |
| *** Sigyn has quit IRC | 10:31 | |
| *** dpawlik has joined #openstack-infra | 10:32 | |
| *** vivsoni_ has joined #openstack-infra | 10:37 | |
| *** vivsoni has quit IRC | 10:39 | |
| *** olivierbourdon38 has quit IRC | 10:40 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Cleanup temporary directories https://review.openstack.org/592340 | 10:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Create a download script https://review.openstack.org/592341 | 10:42 |
| *** Bhujay has quit IRC | 10:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Cleanup temporary directories https://review.openstack.org/592340 | 10:47 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: upload-logs-swift: Create a download script https://review.openstack.org/592341 | 10:47 |
| *** olivierbourdon38 has joined #openstack-infra | 10:48 | |
| *** panda is now known as panda|lunch | 11:02 | |
| *** jpena is now known as jpena|lunch | 11:08 | |
| *** ykarel|lunch is now known as ykarel | 11:09 | |
| *** dave-mccowan has joined #openstack-infra | 11:15 | |
| *** gcb_ has quit IRC | 11:27 | |
| *** udesale has quit IRC | 11:30 | |
| *** e0ne has joined #openstack-infra | 11:32 | |
| *** Bhujay has joined #openstack-infra | 11:35 | |
| *** olivierbourdon38 has quit IRC | 11:38 | |
| *** olivierbourdon38 has joined #openstack-infra | 11:39 | |
| *** e0ne has quit IRC | 11:47 | |
| *** dbecker has joined #openstack-infra | 11:48 | |
| *** boden has joined #openstack-infra | 11:49 | |
| *** boden has quit IRC | 11:49 | |
| *** e0ne has joined #openstack-infra | 11:50 | |
| *** boden has joined #openstack-infra | 11:52 | |
| *** dpawlik has quit IRC | 11:53 | |
| *** dpawlik has joined #openstack-infra | 11:55 | |
| *** dpawlik has quit IRC | 11:55 | |
| *** dpawlik has joined #openstack-infra | 11:55 | |
| *** jcoufal has joined #openstack-infra | 11:57 | |
| *** dbecker has quit IRC | 12:00 | |
| *** dbecker has joined #openstack-infra | 12:02 | |
| *** jcoufal has quit IRC | 12:06 | |
| *** jcoufal has joined #openstack-infra | 12:07 | |
| *** zul has joined #openstack-infra | 12:08 | |
| *** jamesmcarthur has quit IRC | 12:09 | |
| *** jamesmcarthur has joined #openstack-infra | 12:09 | |
| *** xingzhang has joined #openstack-infra | 12:10 | |
| *** xingzhang_ has joined #openstack-infra | 12:13 | |
| *** jpena|lunch is now known as jpena | 12:14 | |
| *** tpsilva has joined #openstack-infra | 12:14 | |
| *** jcoufal_ has joined #openstack-infra | 12:15 | |
| *** jcoufal has quit IRC | 12:15 | |
| *** xingzhang has quit IRC | 12:16 | |
| *** jcoufal_ has quit IRC | 12:21 | |
| *** jamesmcarthur has quit IRC | 12:22 | |
| *** agopi has quit IRC | 12:22 | |
| *** jcoufal has joined #openstack-infra | 12:23 | |
| openstackgerrit | Xing Zhang proposed openstack-infra/puppet-cgit master: Fix cgit highlight https://review.openstack.org/592357 | 12:24 |
| openstackgerrit | Xing Zhang proposed openstack-infra/system-config master: Remove old highlight css https://review.openstack.org/592358 | 12:24 |
| *** rosmaita has joined #openstack-infra | 12:24 | |
| *** jaosorior has quit IRC | 12:25 | |
| *** jaosorior has joined #openstack-infra | 12:25 | |
| xingzhang_ | ^^ anybody have a look about the two patch about cgit highlighting | 12:25 |
| *** larainema has quit IRC | 12:25 | |
| *** xingzhang has joined #openstack-infra | 12:27 | |
| *** xingzhang_ has quit IRC | 12:27 | |
| *** xingzhang has quit IRC | 12:28 | |
| *** bobh has joined #openstack-infra | 12:32 | |
| *** eharney has quit IRC | 12:33 | |
| smcginnis | fungi: Morning. Since I saw your name on some old ML thread - I could have sworn we had a policy documented somewhere stating not to add IDE and OS (DS_Store) files to gitignore. I can't find anything though. Do you know if that is a policy we have set somewhere? | 12:36 |
| *** rlandy has joined #openstack-infra | 12:41 | |
| *** agopi has joined #openstack-infra | 12:50 | |
| *** agopi_ has joined #openstack-infra | 12:51 | |
| mordred | smcginnis: to my knowledge we have never set an openstack project-wide policy on that | 12:52 |
| mordred | some projects think it's fine, others dislike it | 12:52 |
| *** owalsh has quit IRC | 12:53 | |
| mordred | smcginnis: I've got IDE files and editor file things in the openstacksdk gitignore, fwiw | 12:54 |
| *** NeilSun has quit IRC | 12:54 | |
| smcginnis | I noticed that. Could have sworn I saw a policy documented somewhere. Oh well. | 12:54 |
| *** agopi has quit IRC | 12:55 | |
| *** agopi_ is now known as agopi | 12:56 | |
| *** janki has quit IRC | 12:59 | |
| *** janki has joined #openstack-infra | 13:00 | |
| *** r-mibu has joined #openstack-infra | 13:03 | |
| *** jcoufal has quit IRC | 13:04 | |
| AJaeger | smcginnis: I refuse to review those changes in general - unless I give a -2 for *.pyc on a docs only project ;) | 13:05 |
| *** e0ne has quit IRC | 13:05 | |
| AJaeger | There are things that belong IMHO in a global gitconfig | 13:06 |
| *** owalsh has joined #openstack-infra | 13:06 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add list of metrics provided to statsd https://review.openstack.org/590233 | 13:09 |
| *** shardy has quit IRC | 13:11 | |
| *** shardy has joined #openstack-infra | 13:11 | |
| *** bobh has quit IRC | 13:12 | |
| *** udesale has joined #openstack-infra | 13:12 | |
| *** bobh has joined #openstack-infra | 13:13 | |
| *** bobh has quit IRC | 13:14 | |
| *** jaosorior has quit IRC | 13:14 | |
| *** jaosorior has joined #openstack-infra | 13:15 | |
| *** olivierbourdon38 has quit IRC | 13:15 | |
| *** eharney has joined #openstack-infra | 13:17 | |
| *** bobh has joined #openstack-infra | 13:18 | |
| *** jaosorior has quit IRC | 13:20 | |
| *** shardy has quit IRC | 13:20 | |
| *** eharney has quit IRC | 13:21 | |
| *** eharney has joined #openstack-infra | 13:22 | |
| *** bobh has quit IRC | 13:23 | |
| *** rh-jelabarre has joined #openstack-infra | 13:23 | |
| openstackgerrit | Derek Waldner proposed openstack-infra/git-review master: Update default gerrit namespace https://review.openstack.org/584607 | 13:24 |
| *** kei-ichi has quit IRC | 13:24 | |
| *** shardy has joined #openstack-infra | 13:25 | |
| *** graphene has quit IRC | 13:26 | |
| smcginnis | AJaeger: :) | 13:26 |
| smcginnis | AJaeger: I agree. | 13:26 |
| *** graphene has joined #openstack-infra | 13:27 | |
| *** bobh has joined #openstack-infra | 13:31 | |
| *** jcoufal has joined #openstack-infra | 13:31 | |
| *** nicolasbock has joined #openstack-infra | 13:32 | |
| *** e0ne has joined #openstack-infra | 13:32 | |
| *** alexchadin has quit IRC | 13:33 | |
| *** alexchadin has joined #openstack-infra | 13:34 | |
| AJaeger | config-core, python3 goal for docs is finished (thanks, dhellmann !) and we can approve https://review.openstack.org/#/c/591760 anytime now - please review | 13:35 |
| *** bobh has quit IRC | 13:35 | |
| mnaser | #thanks dhellmann working with doc team to help hit python3 goal | 13:35 |
| openstackstatus | mnaser: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 13:35 |
| *** larainema has joined #openstack-infra | 13:36 | |
| * AJaeger thanks mnaser for remembering the bot ;) | 13:36 | |
| mnaser | i want to bring it back :> | 13:36 |
| AJaeger | ;) | 13:37 |
| *** e0ne has quit IRC | 13:38 | |
| *** e0ne has joined #openstack-infra | 13:39 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/592298/ , please? I think we should run releasenotes when tox.ini changes... | 13:42 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for Documentation repositories https://review.openstack.org/591760 | 13:43 |
| *** bobh has joined #openstack-infra | 13:44 | |
| *** radez has joined #openstack-infra | 13:45 | |
| *** bobh_ has joined #openstack-infra | 13:48 | |
| *** bobh has quit IRC | 13:48 | |
| *** ramishra has quit IRC | 13:50 | |
| *** ramishra has joined #openstack-infra | 13:52 | |
| *** bobh_ has quit IRC | 13:53 | |
| *** e0ne has quit IRC | 13:56 | |
| *** pbourke has quit IRC | 13:57 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for ironic repositories https://review.openstack.org/592472 | 13:58 |
| *** pbourke has joined #openstack-infra | 13:59 | |
| *** bobh has joined #openstack-infra | 13:59 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: announce patches proposed to oslo stable branches https://review.openstack.org/592474 | 14:00 |
| *** janki has quit IRC | 14:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-opensuse-423 nodeset https://review.openstack.org/591781 | 14:01 |
| *** dpawlik has quit IRC | 14:01 | |
| *** rpioso|afk is now known as rpioso | 14:02 | |
| *** bobh has quit IRC | 14:03 | |
| *** Qiming has quit IRC | 14:04 | |
| *** Qiming has joined #openstack-infra | 14:08 | |
| *** Bhujay has quit IRC | 14:12 | |
| *** slaweq_ is now known as slaweq | 14:14 | |
| *** hamerins has joined #openstack-infra | 14:18 | |
| *** bobh has joined #openstack-infra | 14:18 | |
| fungi | smcginnis: the only reply i recall making on the subject (years ago?) was suggesting people should take advantage of system-wide solutions to filter their editor.ide files from git rather than committing random tool-specific .gitignore patterns in every repo they ever touch. set once and forget (it's what i do) | 14:18 |
| smcginnis | fungi: Yeah, that's the one I came across. | 14:18 |
| smcginnis | fungi: There was some mention of concerns with tarballs or something, so wasn't sure if there was a technical reason not to do it in-repo/ | 14:18 |
| fungi | no idea. i don't recall that drawback, whatever it was | 14:19 |
| corvus | AJaeger, ianw: 2018-08-16 07:38:42,892 DEBUG zuul.Pipeline.openstack.check: Project openstack/qinling-dashboard not in pipeline <Pipeline check> for change <Change 0x7f09206500b8 | 14:19 |
| corvus | openstack/qinling-dashboard 592248,8> | 14:19 |
| fungi | smcginnis: anyway, certainly not a community-wide policy, just my personal policy (and the alternative recommendation i make to anyone who asks me why i'm not reviewing their random .gitignore additions for such things) | 14:20 |
| smcginnis | fungi: ++ | 14:21 |
| fungi | i set core.excludesfile to '~/.gitignore' in my ~/.gitconfig and then put my personal filtering preferences there | 14:22 |
| *** bobh has quit IRC | 14:23 | |
| AJaeger | corvus: sorry, I don't understand what this means | 14:24 |
| AJaeger | corvus: I see line 4 in https://review.openstack.org/#/c/592248/8/.zuul.yaml is wrong, it'S a job, not a template. But there'S no error by zuul | 14:26 |
| *** jcoufal_ has joined #openstack-infra | 14:27 | |
| *** jcoufal has quit IRC | 14:28 | |
| *** bobh has joined #openstack-infra | 14:30 | |
| fungi | #status log manually deleted the stable/rocky branch previously at 90dfca5dfc60e48544ff25f63c3fa59cb88fc521 from openstack/ovsdbapp at the request of amoralej and smcginnis | 14:30 |
| openstackstatus | fungi: finished logging | 14:30 |
| *** jaosorior has joined #openstack-infra | 14:32 | |
| corvus | AJaeger: it didn't report because it's not in the check pipeline | 14:34 |
| *** bobh has quit IRC | 14:35 | |
| corvus | AJaeger: that's why we have folks add something to project-config for new projects | 14:35 |
| AJaeger | corvus, then clarkb's latest doc update is wrong or I am confused... | 14:35 |
| AJaeger | corvus: That change adds a .zuul.yaml with jobs in the check pipeline. That should work, shouldn't it? | 14:36 |
| corvus | AJaeger: we should at least add "check: jobs: []" for every openstack project | 14:36 |
| AJaeger | corvus: the templates in there have check pipeline | 14:36 |
| corvus | AJaeger: that change has an invalid config, so that change didn't add it to check. if it isn't already in check, it won't report the error | 14:37 |
| AJaeger | corvus: AH! Now I get it... | 14:37 |
| corvus | AJaeger: the solution is to add every project to check, even if there are no jobs. we could add "check: jobs: []" to the system-required template and i think that would be sufficient. | 14:38 |
| *** d0ugal has quit IRC | 14:39 | |
| *** bobh has joined #openstack-infra | 14:39 | |
| *** d0ugal has joined #openstack-infra | 14:41 | |
| mnaser | any infra-root's around to test connectivity from our cloud to both nodepool and google | 14:41 |
| mnaser | to see if we can bring our infra back up again | 14:41 |
| fungi | mnaser: i can try. just a sec and i'll jump into the mirror host | 14:42 |
| *** ramishra has quit IRC | 14:42 | |
| mnaser | i think at the time there was either problems reaching rax nodepool or google | 14:42 |
| *** bobh has quit IRC | 14:43 | |
| fungi | from mirror.ca-ymq-1.vexxhost.openstack.org i can successfully ping6 nl01.openstack.org | 14:44 |
| *** alexchadin has quit IRC | 14:44 | |
| AJaeger | corvus: let'S try that, change coming | 14:45 |
| fungi | any idea what google ip address it was failing to reach? | 14:45 |
| clarkb | fungi: the google ipv6 dns server | 14:45 |
| fungi | that's not an ip address ;) | 14:45 |
| fungi | does it have a hostname? | 14:45 |
| mnaser | uh i cant remember it off hand | 14:45 |
| mnaser | let me search | 14:46 |
| *** e0ne has joined #openstack-infra | 14:46 | |
| fungi | if nobody knows i can dig it up from our image configs | 14:46 |
| clarkb | Unsure, its in our unbound config though | 14:46 |
| mnaser | fungi: 2001:4860:4860::8888 | 14:46 |
| fungi | yeah, that responds from the mirror too | 14:46 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Revert "Revert "Revert "Temporarily stop booting nodes in vexxhost""" https://review.openstack.org/592514 | 14:46 |
| fungi | and i'm able to perform dis queries through it too | 14:47 |
| mnaser | clarkb, fungi ^ | 14:47 |
| fungi | er, dns queries | 14:47 |
| fungi | so looks good | 14:47 |
| fungi | no packet loss reported for either | 14:47 |
| mnaser | great, lets keep tracking it, i think it should be ok now | 14:47 |
| mnaser | i am also a bit more free to dig into issues | 14:47 |
| mnaser | given the big size of the check queue, it'd be nice to bump it up to the gate to help clearing the check queue | 14:48 |
| *** bobh has joined #openstack-infra | 14:49 | |
| fungi | awesome, thanks! | 14:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add empty check queue to system-required https://review.openstack.org/592515 | 14:51 |
| AJaeger | corvus: ^ | 14:52 |
| *** bobh has quit IRC | 14:53 | |
| *** ramishra has joined #openstack-infra | 14:54 | |
| openstackgerrit | Jeremy Stanley proposed openstack-dev/cookiecutter master: Clean up .gitignore references to personal tools https://review.openstack.org/592520 | 14:57 |
| fungi | smcginnis: ^ a straw man | 14:57 |
| smcginnis | fungi: I like it. | 14:58 |
| smcginnis | fungi: I wonder if it would be worth adding a comment in there instructing IDE and OS-specific files to be added to a global gitignore to try to head off some attempts to add new ones. | 14:59 |
| fungi | smcginnis: great idea. i'll stuff a comment in there | 15:00 |
| *** bobh has joined #openstack-infra | 15:00 | |
| AJaeger | great idea! | 15:02 |
| *** bobh has quit IRC | 15:04 | |
| *** armax has joined #openstack-infra | 15:07 | |
| *** r-daneel has joined #openstack-infra | 15:07 | |
| *** jamesmcarthur has joined #openstack-infra | 15:09 | |
| *** ykarel is now known as ykarel|away | 15:09 | |
| *** bobh has joined #openstack-infra | 15:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack-dev/cookiecutter master: Clean up .gitignore references to personal tools https://review.openstack.org/592520 | 15:14 |
| *** bobh has quit IRC | 15:15 | |
| *** yamahata has quit IRC | 15:22 | |
| *** hemna_ has joined #openstack-infra | 15:25 | |
| *** dave-mccowan has quit IRC | 15:26 | |
| *** psachin has quit IRC | 15:27 | |
| *** florianf has joined #openstack-infra | 15:28 | |
| *** florianf has left #openstack-infra | 15:28 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add empty check queue to system-required https://review.openstack.org/592515 | 15:30 |
| *** bobh has joined #openstack-infra | 15:30 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: IGNORE testing pbrx https://review.openstack.org/592550 | 15:33 |
| *** bobh has quit IRC | 15:34 | |
| AJaeger | corvus: It works! See https://review.openstack.org/#/c/592248/10 ;) Thanks! | 15:35 |
| *** verdurin has quit IRC | 15:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add pbrx patch announcements to #zuul https://review.openstack.org/592554 | 15:37 |
| *** efried is now known as efried_afk | 15:37 | |
| *** verdurin has joined #openstack-infra | 15:37 | |
| *** bobh has joined #openstack-infra | 15:40 | |
| *** bobh has quit IRC | 15:44 | |
| corvus | mordred: what's the game plan with ansible/bridge? | 15:45 |
| clarkb | I'm almost around to help with that. Minor local networking hardware changes to complete | 15:46 |
| clarkb | adding a wireless nic and converting from suse netconfig to networkmanager is an interesting experience | 15:47 |
| *** d0ugal has quit IRC | 15:47 | |
| mordred | corvus: I was waiting for clarkb to be awake and up | 15:49 |
| mordred | clarkb, corvus: I think the gameplan is - land the "stop running puppet from puppetmaster patch" and then wait for puppetmaster to stop running things | 15:50 |
| mordred | then, do some manual trial-runs of running remote_puppet from bridge to hosts - so we can catch any major issues | 15:50 |
| *** e0ne has quit IRC | 15:50 | |
| clarkb | we can probably proceed with that first step now | 15:50 |
| clarkb | we disable puppet all the time :) | 15:51 |
| mordred | including, butnot limited to, manually running lists.* | 15:51 |
| mordred | clarkb: good point - I'll go click that one now | 15:51 |
| *** ykarel|away has quit IRC | 15:51 | |
| corvus | and its 2 parents | 15:51 |
| *** bobh has joined #openstack-infra | 15:51 | |
| mordred | done | 15:52 |
| *** gyee has joined #openstack-infra | 15:53 | |
| corvus | for folks without a gertty handy: http://paste.openstack.org/show/728215/ | 15:53 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Revert "Temporarily stop booting nodes in vexxhost""" https://review.openstack.org/592514 | 15:53 |
| *** bobh has quit IRC | 15:56 | |
| mordred | I think we should also land https://review.openstack.org/#/c/591145/ - clarkb when you get a sec if you wantto review it | 15:56 |
| *** d0ugal has joined #openstack-infra | 15:59 | |
| mordred | corvus, cmurphy, clarkb: http://logs.openstack.org/36/585836/16/check/infra-puppet-apply-4-ubuntu-xenial/072becb/applytest/puppetapplytest25.final.out.FAILED | 16:00 |
| mordred | the error there makes NO SENSE to me at all | 16:00 |
| *** bobh has joined #openstack-infra | 16:02 | |
| clarkb | mordred: its trying to index an undef value and not a array value | 16:02 |
| clarkb | mordred: I think maybe you need to pass empty array instead of undef to that class? | 16:02 |
| mordred | but we're ot passing anything | 16:02 |
| mordred | the calling line is: | 16:02 |
| mordred | apt::ppa { 'ppa:mosquitto-dev/mosquitto-ppa': } | 16:02 |
| clarkb | mordred: https://github.com/puppetlabs/puppetlabs-apt/blob/2.4.0/manifests/ppa.pp#L5 is the index that fails | 16:04 |
| mordred | yah | 16:04 |
| fungi | so it's passing an associative array with an undefined value for its only key | 16:04 |
| *** ramishra has quit IRC | 16:05 | |
| mordred | and that is what's on puppetmaster so isn't new | 16:05 |
| clarkb | (trying to figure out what xfacts is) | 16:05 |
| mordred | puppet3 says: | 16:05 |
| mordred | 2018-08-15 22:28:28 +0000 Puppet (err): ::apt::xfacts is not a hash or array when accessing it with lsbdistcodename at /etc/puppet/modules/apt/manifests/ppa.pp:5 on node ubuntu-xenial-rax-ord-0001345945.openstack.org | 16:05 |
| mordred | http://logs.openstack.org/36/585836/16/check/infra-puppet-apply-3-ubuntu-xenial/986945c/applytest/puppetapplytest25.final.out.FAILED | 16:05 |
| mordred | which also fails | 16:05 |
| *** udesale has quit IRC | 16:05 | |
| clarkb | https://github.com/puppetlabs/puppetlabs-apt/blob/2.4.0/manifests/params.pp#L7-L31 | 16:05 |
| fungi | a web search only turns up https://pastebin.com/SDmSnJEZ | 16:06 |
| * fungi continues to question why pastebin has crawlers index its contents | 16:06 | |
| mordred | "prior to puppet 3.5.0, defined() couldn't test if a variable was defined." | 16:06 |
| mordred | that's an amazing comment | 16:06 |
| *** bobh has quit IRC | 16:06 | |
| fungi | um | 16:07 |
| *** ramishra has joined #openstack-infra | 16:07 | |
| fungi | i can't even | 16:07 |
| clarkb | considering its an if else and bothblock define xfacts to be a hash I don't know why it would be undef other than if the params module isn't evaluated for some reason | 16:08 |
| clarkb | mordred: ^ you can probably test that by adding a include apt::params prior to our usage of the ppa define? | 16:08 |
| *** bobh has joined #openstack-infra | 16:11 | |
| *** shardy has quit IRC | 16:12 | |
| *** bobh has quit IRC | 16:15 | |
| *** jcoufal has joined #openstack-infra | 16:18 | |
| *** jpena is now known as jpena|off | 16:19 | |
| *** bobh has joined #openstack-infra | 16:19 | |
| *** jcoufal_ has quit IRC | 16:19 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:20 | |
| mordred | clarkb: ok. I'll give it a shot | 16:20 |
| smcginnis | We have a release validate job that keeps finding a stable branch that was manually deleted. I don't see it on git.o.o or anywhere else. Do we have a different locally cached instance used by the gate? | 16:22 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Allow setting puppet_version explicitly https://review.openstack.org/591145 | 16:22 |
| clarkb | smcginnis: we have 8 zuul mergers and each of our 11 zuul executors maintain their own git repo caches | 16:22 |
| smcginnis | clarkb: Ah, that could explain it. | 16:22 |
| smcginnis | clarkb: Any idea how long I would need to wait for those caches to be invalidated? | 16:22 |
| clarkb | ref deletes are not pulled and must be pushed or manualy made locally | 16:23 |
| clarkb | smcginnis: so unless a human intervenes I think the only thing to chagne it is to update the ref | 16:23 |
| smcginnis | Ah ha. Can you be a human intervener and remove the stable/rocky branch on ovsdbapp in those? | 16:23 |
| *** bobh has quit IRC | 16:24 | |
| clarkb | smcginnis: what is the objective here? or put another way, why can't we live with it as is until the new branch ref is created? | 16:24 |
| clarkb | (I'm assuming that new branch ref will be created in the next few minutes/) | 16:25 |
| smcginnis | clarkb: We can't create the new branch ref because it sees there is one already out there at a different commit. | 16:25 |
| fungi | we have more than 8 zuul mergers if you count the built-in mergers for the 10 or 11 executors, right? | 16:25 |
| clarkb | fungi: yes it is 8 + 11 that need to be updated | 16:25 |
| fungi | oh, yep, i see you said that now | 16:25 |
| * fungi clearly tried to read too fast | 16:26 | |
| clarkb | corvus: can we safely delete the entire ovdbapp repo in /var/lib/zuul/git (or whatever the path actually is) | 16:26 |
| corvus | clarkb: yes | 16:26 |
| clarkb | smcginnis: that is probably a bug you will want to accomodate | 16:26 |
| *** ramishra has quit IRC | 16:26 | |
| smcginnis | clarkb: That really is more of a feature than a bug. | 16:26 |
| clarkb | smcginnis: except you intentionally deleted the branch | 16:27 |
| clarkb | smcginnis: and branch deletion doesn't work the way you think it does in git | 16:27 |
| clarkb | (same situation with tags really) | 16:27 |
| fungi | well, almost | 16:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Add pbrx patch announcements to #zuul https://review.openstack.org/592554 | 16:27 |
| corvus | smcginnis: have a log link? | 16:27 |
| fungi | branches can be updated and that will propagate over pull. tags cannot | 16:28 |
| smcginnis | We need to protect against someone modifying a branch at a different point than where it is already branched. We can't really do that if we ignore the fact that we see the branch out there from another commit. | 16:28 |
| *** panda|lunch is now known as panda|off | 16:28 | |
| smcginnis | corvus: http://logs.openstack.org/00/592500/1/check/openstack-tox-validate/3458b23/job-output.txt.gz#_2018-08-16_16_12_47_052421 | 16:28 |
| smcginnis | Not sure why this is an issue now. When we've had to do this before we did not run into this situation. | 16:28 |
| clarkb | smcginnis: then you cannot delete branches | 16:29 |
| corvus | smcginnis: when was it deleted? | 16:29 |
| smcginnis | Ideally we wouldn't, but mistakes happen. | 16:29 |
| *** derekh has quit IRC | 16:29 | |
| smcginnis | corvus: A few hours ago. | 16:29 |
| fungi | corvus: ~1429z | 16:29 |
| fungi | i status logged it | 16:29 |
| corvus | i don't think this is zuul | 16:29 |
| corvus | i *think* zuul would handle this case | 16:30 |
| corvus | i think this is using the image cache | 16:30 |
| clarkb | corvus: oh interesting | 16:30 |
| mordred | oh - yeah | 16:30 |
| corvus | since this is one of the release jobs which doesn't use required-projects (because it would have to require *all* the projects) | 16:30 |
| *** onovy has quit IRC | 16:30 | |
| clarkb | similar issue though where you have to be explicit about it so the cache isn't updated there | 16:30 |
| corvus | i think it just does stuff manually | 16:30 |
| smcginnis | Yeah, it clones what it needs. | 16:31 |
| fungi | it likely needs to include a step to prune? | 16:31 |
| corvus | yes, or wait until the next image build | 16:31 |
| mordred | this is a situation where shifting from in-image cache to local-cloud-region-git-mirror would be helpful | 16:31 |
| smcginnis | How frequently are the image builds? | 16:31 |
| clarkb | for the record on mergers it is /var/lib/zuul/git/git.openstack.org/openstack/ovsdbapp and on executors it is /var/lib/zuul/executor-git/git.openstack.org/openstack/ovsdbapp (I've nto deleted any of them) | 16:31 |
| clarkb | smcginnis: daily | 16:32 |
| corvus | mordred: yes | 16:32 |
| smcginnis | OK, so this would work if we waited until tomorrow. We can do that if we need to. | 16:32 |
| fungi | but yeah, sounds like the job itself needs to account for branches which have disappeared from the remote relative to its current local state | 16:32 |
| clarkb | smcginnis: or update your scripts to manually check for branch deletions and clean them up | 16:32 |
| clarkb | smcginnis: which is sort of what I was trying to suggest before | 16:32 |
| smcginnis | clarkb: Do you know what the call would be to do that off the top of your head? | 16:33 |
| clarkb | smcginnis: git remote update; for branch in local branches: if not in remote branches delete ; done | 16:33 |
| *** rlandy is now known as rlandy|brb | 16:33 | |
| corvus | smcginnis: fwiw, the opposite danger may be present as well -- if the repo isn't correctly updated to fetch *new* branches, you could end up creating a branch twice because the check didn't see an existing branch in the cache | 16:34 |
| smcginnis | That doesn't seem as safe then. | 16:34 |
| fungi | should be `git fetch --prune1 maybe? | 16:34 |
| smcginnis | I would rather have to wait a period after manual branch deletion than risk messing more things up. | 16:34 |
| fungi | er, `git fetch --prune` | 16:34 |
| *** jpich has quit IRC | 16:34 | |
| *** pcaruana has quit IRC | 16:34 | |
| clarkb | fungi: ya looks like that is the git does what I describe for you invocation | 16:35 |
| fungi | "Before fetching, remove any remote-tracking references that no longer exist on the remote." | 16:35 |
| *** sshnaidm is now known as sshnaidm|bbl | 16:35 | |
| smcginnis | I'll just make a note that we need to hold off a recheck tomorrow. | 16:36 |
| fungi | fingers crossed our image builds aren't broken ;) | 16:37 |
| smcginnis | Just odd that we had at least two cases where we've had to do this this cycle and this is the first time there's been a problem. | 16:37 |
| smcginnis | fungi: Hah | 16:37 |
| clarkb | fungi: the ssh config update showed they were working as of a couple days ago | 16:37 |
| fungi | smcginnis: i expect the previous incidents straddled a day between deletion and creation | 16:37 |
| smcginnis | Perhaps | 16:37 |
| clarkb | fungi: smcginnis if we want to speed ti up we can trigger a rebuild now | 16:38 |
| clarkb | what image does the job run on? xenial? | 16:38 |
| smcginnis | clarkb: Is it an expensive operation? Really no strong urgency here. If we have to wait we can wait. | 16:39 |
| *** rf0lc0 is now known as rfolco | 16:39 | |
| clarkb | smcginnis: its mostly byte copying IO so expensive in network but otherwise is fine | 16:39 |
| corvus | smcginnis: yeah, we generally make no promises about image builds -- sometimes they're broken for days at a time. so waiting might work, but if we're at all in a hurry, fetch-prune would be a good idea | 16:39 |
| clarkb | smcginnis: will probably take a couple hours to complete | 16:40 |
| *** dbecker has quit IRC | 16:40 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add lists exim config to ansible https://review.openstack.org/591494 | 16:40 |
| openstackgerrit | Merged openstack-infra/system-config master: Add exim config for firehose and storyboard https://review.openstack.org/591495 | 16:40 |
| openstackgerrit | Merged openstack-infra/system-config master: Stop running puppet from puppetmaster https://review.openstack.org/591151 | 16:40 |
| mordred | well - here comes the fun | 16:40 |
| smcginnis | clarkb, corvus: Thanks, I'll cross my fingers about the image build and wait. | 16:41 |
| *** dpawlik has joined #openstack-infra | 16:41 | |
| corvus | clarkb, mordred: here's the current state of swift-logs: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/logs_78/587178/1/check/tox-py35/84d70ff/ | 16:43 |
| corvus | you can see the htmlified job log there | 16:44 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: Drop noop job of ui-cookiecutter https://review.openstack.org/592576 | 16:44 |
| clarkb | https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/logs_78/587178/1/check/tox-py35/84d70ff/job-output.html#l215 is works | 16:44 |
| corvus | i'll push up a dnm devstack change to see what it looks like | 16:45 |
| clarkb | also somehow the html file is smaller than the txt file | 16:45 |
| *** dpawlik has quit IRC | 16:46 | |
| corvus | i guess we should make sure all the lines are there? | 16:46 |
| cmurphy | mordred: did you figure it out? | 16:47 |
| mordred | cmurphy: not necessarily - I did push up a test patch to see if it solves things | 16:48 |
| *** openstackgerrit has quit IRC | 16:49 | |
| *** bobh has joined #openstack-infra | 16:49 | |
| *** openstackgerrit has joined #openstack-infra | 16:49 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 16:49 |
| mordred | cmurphy: by "push up" I meant "ran git review but forgot to type yes" | 16:49 |
| corvus | mordred: push down enter key to push up patch | 16:51 |
| clarkb | tough crowd today mordred | 16:51 |
| mordred | clarkb: ikr? | 16:52 |
| *** bobh has quit IRC | 16:54 | |
| * clarkb grabs breakfast | 16:55 | |
| *** dayou has quit IRC | 16:56 | |
| corvus | mordred: is now a good time for me to afk for 30m? | 16:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: test swift logs https://review.openstack.org/592581 | 16:58 |
| corvus | remote: https://review.openstack.org/592582 DNM: test swift logs is the devstack change | 17:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: test swift logs https://review.openstack.org/592581 | 17:01 |
| *** bobh has joined #openstack-infra | 17:01 | |
| *** bobh has quit IRC | 17:06 | |
| *** jamesmcarthur has quit IRC | 17:06 | |
| *** bobh has joined #openstack-infra | 17:08 | |
| *** r-mibu has quit IRC | 17:08 | |
| *** jamesmcarthur has joined #openstack-infra | 17:09 | |
| *** rlandy|brb is now known as rlandy | 17:11 | |
| *** weshay is now known as weshay|ruck | 17:12 | |
| *** bobh has quit IRC | 17:12 | |
| mordred | corvus: yes - I think it's a great time to afk - and we can take next steps when we're all back | 17:13 |
| *** electrofelix has quit IRC | 17:15 | |
| * fungi just got done afk'ing... christine came home from the marina with 10 lbs of fresh shrimp we needed to weigh out and freeze | 17:17 | |
| clarkb | I'm back to computer | 17:19 |
| mordred | I have verified that the run puppet cronjobs are no longer on puppetmsater | 17:21 |
| mordred | and ansible is not running | 17:21 |
| *** dbecker has joined #openstack-infra | 17:26 | |
| *** bobh has joined #openstack-infra | 17:28 | |
| *** bobh has quit IRC | 17:33 | |
| *** graphene has quit IRC | 17:35 | |
| *** graphene has joined #openstack-infra | 17:36 | |
| *** jamesmcarthur has quit IRC | 17:43 | |
| *** jamesmcarthur has joined #openstack-infra | 17:43 | |
| *** slaweq has quit IRC | 17:44 | |
| corvus | i am back but have no shrimp | 17:46 |
| clarkb | shrimp do sound tasty | 17:46 |
| *** onovy has joined #openstack-infra | 17:46 | |
| fungi | a lb of them are going into shrimp fried rice later this afternoon | 17:47 |
| corvus | mordred: next is careful running of ansible from bridge on lists.* ? | 17:47 |
| *** jamesmcarthur has quit IRC | 17:48 | |
| *** bobh has joined #openstack-infra | 17:53 | |
| clarkb | do we want to start with lists or do something like say etherpad first? | 17:55 |
| clarkb | etherpad-dev rather | 17:55 |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/592289/ to start retiring rst2bash, please? | 17:55 |
| mordred | corvus: yes | 17:55 |
| mordred | I also have less shrimp than fungi | 17:56 |
| mordred | clarkb, corvus, fungi: I'm first going to run update-system-config.yaml on bridge, which should get the latest system-config, ansible roles and puppet modules | 17:57 |
| *** bobh has quit IRC | 17:57 | |
| *** owalsh has quit IRC | 17:58 | |
| *** bobh has joined #openstack-infra | 18:00 | |
| fungi | sounds good to me | 18:00 |
| *** efried_afk is now known as efried | 18:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 18:02 |
| mordred | ok. I have run that on bridge and all seems happy | 18:02 |
| clarkb | mordred: did you find anything new on ^ with the ppa issue? | 18:03 |
| mordred | clarkb: well, the previous patch seems to have maybe fixed that, but a different issue emerged | 18:03 |
| mordred | clarkb: so I just pushed up anew one - I think this one is going to work | 18:04 |
| mordred | I'm now going to run base.yaml with --limit=bridge.openstack.org and bridge.yaml to get bridge up to date | 18:04 |
| clarkb | woot | 18:04 |
| *** bobh has quit IRC | 18:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: haproxy: don't manage rsyslog service https://review.openstack.org/592637 | 18:08 |
| corvus | mordred: ^ that implements my comment-suggestion that we were undecided on | 18:09 |
| mordred | corvus: ++ - I'm in favor of it if others are | 18:09 |
| *** yamahata has joined #openstack-infra | 18:10 | |
| clarkb | considering that part of our launch node process includes a reboot after all the config management happens I think we can get away with the downside to that approach | 18:10 |
| clarkb | we don't ever really change that config either | 18:10 |
| *** bobh has joined #openstack-infra | 18:11 | |
| *** owalsh has joined #openstack-infra | 18:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use ansible group vars for futureparser flag https://review.openstack.org/592638 | 18:12 |
| mordred | just noticed that in looking at the kick script | 18:13 |
| *** mmedvede has quit IRC | 18:13 | |
| mordred | ok - I think we're in a good state to start running puppet against individual hosts - I think we should run base.yaml and then puppet against them, yeah? | 18:13 |
| clarkb | mordred: ya | 18:14 |
| clarkb | and probably work our way up to lists/review/zuul ? | 18:14 |
| mordred | yah | 18:14 |
| mordred | I was thinking of starting with review-dev ? | 18:14 |
| clarkb | review-dev and etherpad-dev and logstash workers are good first candidates | 18:14 |
| mordred | k. I'm going to do that ... hrm, should I start a screen session? | 18:15 |
| clarkb | ++ to screen | 18:15 |
| clarkb | I'll attach and follow along at least | 18:15 |
| mordred | ok. I have started a screen session | 18:15 |
| *** bobh has quit IRC | 18:15 | |
| clarkb | I see it | 18:16 |
| mordred | running base on review-dev01 | 18:16 |
| *** mmedvede has joined #openstack-infra | 18:17 | |
| clarkb | thats curious why did it add our keys | 18:17 |
| mordred | I can still log in to review-dev | 18:17 |
| clarkb | maybe it appended to the file? | 18:17 |
| mordred | ansible does keys a little different than puppet | 18:17 |
| mordred | no, it's actually the opposite - it's set up to do a full replace | 18:17 |
| clarkb | it didn't append to the file but ya I can log in | 18:17 |
| clarkb | and file looks correct from here | 18:18 |
| clarkb | mordred: it also updated the exim config, not sure if that was expected | 18:18 |
| mordred | exim files also changed - corvus you wanna see if those make sense to you? /etc/aliases looks good | 18:18 |
| mordred | yeah - the exim4.conf has some whitespace differences | 18:18 |
| clarkb | attempt to sudo as non sudo user on review-dev and see if it emails us? | 18:20 |
| *** r-daneel has quit IRC | 18:20 | |
| clarkb | there are probably better ways to test that, but that should confirm aliases work too | 18:20 |
| corvus | mordred: yep i'll take a look | 18:21 |
| corvus | mordred: should i be looking in your screen session? | 18:21 |
| clarkb | mordred: when puppet applies we will revert the exim config back as well as the ssh keys right? at least until https://review.openstack.org/#/c/585836/18 gets in | 18:22 |
| clarkb | corvus: the screen session is just the ansible output so far | 18:22 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for grafana.o.o https://review.openstack.org/592640 | 18:22 |
| corvus | where should i diff the exim config? | 18:22 |
| clarkb | whcih doesn't have many details on what was changed just that the exim tasks report change | 18:22 |
| clarkb | mordred: ^ | 18:23 |
| mordred | maybe we shold run with --verbose on? | 18:23 |
| corvus | oh, perhaps the question is "does the new thing look right?" not "do the differences look right?" :) | 18:23 |
| clarkb | if that will show us diffs I'm on board | 18:23 |
| mordred | yah | 18:23 |
| corvus | "exim -bt root" on review-dev01 makes sense | 18:23 |
| corvus | that incidentally tests all of the things we expect exim to be able to know how to do on review-dev01 | 18:24 |
| corvus | that's safe for anyone to run (even as their own user) if you want | 18:24 |
| mordred | neat | 18:24 |
| mordred | so - it seems we're happy with this - let's run remote puppet now? | 18:25 |
| corvus | +1 from me | 18:25 |
| clarkb | ++ | 18:25 |
| mordred | this is going to be a back-and-forth until we land the remove-stuff-from-puppet patch | 18:25 |
| mordred | but should still be safe in general | 18:25 |
| corvus | i've tuned into the screen too | 18:26 |
| mordred | it's an exciting screen isn't it? | 18:26 |
| corvus | in living color! | 18:27 |
| clarkb | sudo grep puppet-user /var/log/syslog will show you want puppet did | 18:28 |
| clarkb | and as expected it replaced all our keys beacuse it added a header saying puppet is in charge and also updated exim | 18:28 |
| fungi | "puppet wuz here" | 18:28 |
| clarkb | corvus: probably the biggest potential issue with this is restarting exim often, do you expect that to causetrouble with email delivery on more active hosts like review and lists? | 18:28 |
| clarkb | it should spool right? | 18:29 |
| clarkb | and if you try sending email to it and fail you retry again later | 18:29 |
| fungi | i'm curious, why would it restart exim "often"? | 18:29 |
| *** onovy has quit IRC | 18:29 | |
| clarkb | fungi: beacuse ansible will change the config and restart it, then a minute later puppet will change the config and restart it | 18:29 |
| clarkb | fungi: and we do that every half hour | 18:29 |
| fungi | oh, you mean until 585836 | 18:30 |
| clarkb | until https://review.openstack.org/#/c/585836/18 is working and merges | 18:30 |
| mordred | yah - and the 'remove ansible things from puppet' patch is still unhappy in the testing | 18:30 |
| clarkb | yup | 18:30 |
| mordred | yah | 18:30 |
| fungi | so not a long-term thing | 18:30 |
| fungi | sorry | 18:30 |
| corvus | clarkb: it should be fine unless someone is sending mail from exchange (which does not know how to retry on connection failures) | 18:30 |
| fungi | obviously it's not ideal, but it's probably fine for a little bit? | 18:30 |
| corvus | but the connection downtime should be very short | 18:31 |
| fungi | i would argue that exchange is broken (actually, i've argued that for years, even when i was an exchange admin *shudder*) | 18:31 |
| corvus | clarkb: exim does support sighup for reload, perhaps our init scripts do that? | 18:31 |
| clarkb | corvus: based on ps output I think it properly restarts it | 18:32 |
| fungi | oh, that's true. there are only a few things (replacing smtps/starttls certs?) which need a restart, i think | 18:32 |
| clarkb | if we end up in this situation for longer than expected we can update that | 18:32 |
| corvus | bummer. the init scripts *do* support reloading with hup | 18:32 |
| fungi | altering the listening addreses/ports may also require a restart | 18:32 |
| corvus | so it's probably puppet doing the wrong thing there | 18:32 |
| clarkb | corvus: ya puppet by default does restart to the init script iirc | 18:32 |
| *** rlandy has quit IRC | 18:32 | |
| clarkb | there is a way to tell it to reload though | 18:32 |
| corvus | hasreload=>true iirc | 18:33 |
| *** bobh has joined #openstack-infra | 18:33 | |
| corvus | or somethin like that | 18:33 |
| fungi | seems like it's a confusingly-named option. something like "has_restart"? | 18:33 |
| *** diablo_rojo has joined #openstack-infra | 18:33 | |
| corvus | fungi: you tell puppet the service has a 'reload' option and it will try the 'reload' init command instead of 'restart' | 18:33 |
| clarkb | I'm going to look at the failures on 585836 to see if I can help get that merged sooner or later | 18:35 |
| *** rlandy_ has joined #openstack-infra | 18:35 | |
| corvus | i assume in the future cfgmgmt systems would interrogate systemd and dtrt automatically | 18:35 |
| fungi | yeah, but for some reason i thought they'd given it a backwards name... | 18:35 |
| clarkb | oh! mordred I think I grok the ppa issue better | 18:35 |
| *** dbecker has quit IRC | 18:35 | |
| clarkb | mordred: http://logs.openstack.org/36/585836/18/check/infra-puppet-apply-3-ubuntu-xenial/63be4d9/applytest/puppetapplytest25.final.out.FAILED says class ::apt has not been evaluated I think we need to include ::apt | 18:36 |
| corvus | oh, was something down the line doing that, and it got removed? | 18:36 |
| mordred | clarkb: I don't understand why this is failing all of a sudden from this patch though | 18:36 |
| mordred | also - http://logs.openstack.org/36/585836/18/check/infra-puppet-apply-4-centos-7/2e947aa/applytest/puppetapplytest26.final.out.FAILED | 18:36 |
| clarkb | mordred: I'm guessing corvus is onto that | 18:37 |
| mordred | is just weird | 18:37 |
| clarkb | I'm checking server.pp now to see if we removed that include | 18:37 |
| mordred | AH - that would make sense | 18:37 |
| *** bobh has quit IRC | 18:37 | |
| clarkb | yup we delete an include ::apt in there | 18:37 |
| clarkb | we could keep it in server.pp for now if we want or just fix it when it shows up | 18:37 |
| clarkb | mordred: fix would be to replace include ::apt::params with just include ::apt I think | 18:38 |
| mordred | yeah - let's just put ::apt back in server.pp | 18:38 |
| *** hemna_ has quit IRC | 18:38 | |
| mordred | cause I don't think this patch should be about tracking down and updating various puppet we're wanting to remove anyway | 18:38 |
| clarkb | mordred: for the centos thing it is looking for system-config/hiera/common.yaml to be in hiera and it isn't | 18:39 |
| clarkb | mordred: ++ | 18:39 |
| clarkb | mordred: we need to make sure we plumb the public hieradata into hieras pathing | 18:39 |
| *** bobh has joined #openstack-infra | 18:40 | |
| clarkb | server.pp can just be the puppet constructor function essentially | 18:40 |
| clarkb | and not make real changes to the system | 18:40 |
| mordred | yah | 18:41 |
| mordred | clarkb: the integration test harness should be plumbing through public heiradata yeah? | 18:41 |
| clarkb | mordred: I think you updated it so that ansible configures the hiera paths right? which would expect system config at /opt/system-config/production iirc and we may not have that path set on the test nodes? | 18:41 |
| clarkb | mordred: its possible we need to smlink /opt/system-config/production to /home/zuul/src/git.openstack.org/openstack-infra/system-config ? | 18:42 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn the future parser on for health.o.o https://review.openstack.org/592647 | 18:42 |
| fungi | aha, so that | 18:42 |
| fungi | 's what it was | 18:42 |
| mordred | clarkb: hrm. where should we do that? | 18:42 |
| mordred | in the test code perhaps? | 18:42 |
| fungi | puppet used to call stop and start and used hasrestart to indicate it could call restart instead of explicit stop/start | 18:42 |
| fungi | hasreload wasn't added until later, so you used to have to set hasrestart=>true and then set the "restart" command to actually call a reload | 18:43 |
| *** sshnaidm|bbl is now known as sshnaidm | 18:43 | |
| mordred | clarkb: actually - I think we should dig slighlty more on the hiera thing - that patch shoudlt' be touching how the integratoin tests set up hiera | 18:44 |
| mordred | it does rename the update_puppet_version playbook to update-puppet - but nothing, that I know of, calls that | 18:44 |
| clarkb | mordred: ya looking at system-config/tools/prep-apply.sh I think it is already doing what I thought we needed | 18:44 |
| *** bobh has quit IRC | 18:44 | |
| mordred | cmurphy: ^^ any thoughts/ideas/magic? | 18:45 |
| clarkb | mordred: maybe rerun and hold the node then check the hiera config and paths manually | 18:45 |
| corvus | i had been thinking along the same lines -- i think a hold is in order | 18:46 |
| cmurphy | mordred: context? | 18:46 |
| corvus | want me to take care of that? | 18:46 |
| cmurphy | nothing should be running that update playbook yet | 18:46 |
| clarkb | cmurphy: http://logs.openstack.org/36/585836/18/check/infra-puppet-apply-4-centos-7/2e947aa/applytest/puppetapplytest26.final.out.FAILED | 18:46 |
| clarkb | cmurphy: it can't find the public hiera key implying we haven't configured public hiera properly | 18:46 |
| cmurphy | hmm | 18:46 |
| clarkb | corvus: I would say go for it, can always remove it if cmurphy understands it | 18:47 |
| mordred | clarkb: ++ | 18:47 |
| cmurphy | i got nothing atm | 18:48 |
| mordred | yah - cause also - honestly, this cutover would probably be nicer to validate if we could validate new behavior without puppet rather than the ansible and puppet are going to fight over who owns a thing | 18:48 |
| mordred | ? | 18:48 |
| mordred | corvus: I've got an update to the patch for the ::apt thing- so if you have a hold, we shuld be able to trigger it | 18:49 |
| corvus | mordred: hold is ready, go for it | 18:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 18:49 |
| mordred | bombs away | 18:49 |
| corvus | i used the new expire setting too. so it should auto-delete after 8 hours | 18:49 |
| clarkb | mordred: ya I think validating that will be nice, that said I think the existing resulst are sane neough we can keep moving foward if necessary | 18:50 |
| mordred | clarkb: kk. so you want I should try another server? maybe etherpad-dev? | 18:51 |
| *** hemna_ has joined #openstack-infra | 18:51 | |
| *** harlowja has joined #openstack-infra | 18:51 | |
| clarkb | ya I think we do one or two more just to see if there are any weird interaction that one server wouldn't hit | 18:51 |
| clarkb | etherpad-dev seems reasonable | 18:52 |
| clarkb | then we can try the "important" servers then turn on the cron? | 18:52 |
| mordred | yah | 18:52 |
| *** bobh has joined #openstack-infra | 18:53 | |
| clarkb | and if we get the puppet cleanup in we can turn off cron (ifalready on and try a server or two again and turn cron back on) | 18:53 |
| mordred | ++ | 18:54 |
| mordred | ok. that seemed to do things | 18:54 |
| mordred | I can still log in | 18:54 |
| clarkb | looks similar to review-dev01 | 18:54 |
| mordred | yah | 18:55 |
| mordred | exim -bt root seems to be reasonable | 18:56 |
| *** hemna_ has quit IRC | 18:56 | |
| *** r-daneel has joined #openstack-infra | 18:56 | |
| mordred | anybody want to check anything else before I puppet? | 18:56 |
| *** SteffanW has joined #openstack-infra | 18:57 | |
| clarkb | I think we can puppet, but next time add --verbose? | 18:57 |
| *** bobh has quit IRC | 18:57 | |
| mordred | ok. verbose seems less happy :) | 18:58 |
| mordred | we need nologs on those rsyncs | 18:58 |
| clarkb | etherpad-dev has/had futureparser enabled but is not futureparsing here it seems like | 18:59 |
| mordred | clarkb: future parser seemed to turn off | 18:59 |
| mordred | yeah | 18:59 |
| clarkb | ya | 18:59 |
| mordred | we shoudl investigate that ... looking in code | 18:59 |
| clarkb | thats not critical but we should fix it | 18:59 |
| clarkb | (like puppet works on old and new parser on that host, but lets keep using future if we've transitioned) | 18:59 |
| mordred | yah | 18:59 |
| *** bobh has joined #openstack-infra | 19:00 | |
| mordred | group_names | 19:00 |
| mordred | what sets group_names ? | 19:00 |
| clarkb | mordred: thats ansible | 19:01 |
| clarkb | mordred: iirc its the list of all groups current node is part of | 19:01 |
| mordred | ok | 19:01 |
| mordred | clarkb: we could also rebase the futureparser patch I just pushedup and land it | 19:02 |
| clarkb | so I don't forget: we can run this against elasticsearch02.openstack.org if we weant to double check the public hiera lookup thing against real node | 19:02 |
| clarkb | mordred: that wfm if it is easier to debug | 19:02 |
| cmurphy | commented my guess on https://review.openstack.org/585836 | 19:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use ansible group vars for futureparser flag https://review.openstack.org/592638 | 19:03 |
| clarkb | cmurphy: that would also explain the complaint about hiera versions in the output too | 19:03 |
| mordred | clarkb: AHA | 19:03 |
| mordred | yeah | 19:03 |
| mordred | let's take that out - as well as the change to update_puppet_version | 19:03 |
| mordred | and circle back to it | 19:03 |
| clarkb | wfm | 19:03 |
| *** slaweq has joined #openstack-infra | 19:04 | |
| *** bobh has quit IRC | 19:04 | |
| cmurphy | actually the complaint about hiera version will remain, we'll need to rewrite our hiera data to be compatible with this new format | 19:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 19:05 |
| cmurphy | whatever that is | 19:05 |
| clarkb | cmurphy: right but puppet 3 hiera wouldn't complain will it? just puppet 4? | 19:05 |
| cmurphy | clarkb: right | 19:05 |
| mordred | I think it's just the hiera config file- not the hiera data - that it's complaining about | 19:05 |
| cmurphy | oh that might be it | 19:06 |
| mordred | which sohuld be much easier to deal with | 19:06 |
| clarkb | corvus: fungi: https://review.openstack.org/#/c/592638/ can we review that change to work on fixing the future parser support on bridge? | 19:06 |
| fungi | looking | 19:06 |
| clarkb | mordred: is ^ something you'll manually put in place and rerun? | 19:06 |
| *** jamesmcarthur has joined #openstack-infra | 19:06 | |
| mordred | clarkb: if we land it I can just re-run update-system-config | 19:07 |
| mordred | or if you want I could pull it manually and we could re-run etherpad before we land it | 19:07 |
| clarkb | mordred: mostly wondering how quickly we'll land it. /me tries to be patient :) | 19:07 |
| mordred | let's pull it - that way if it borks we can fix it more quicker | 19:08 |
| *** bobh has joined #openstack-infra | 19:08 | |
| *** kevinshort has joined #openstack-infra | 19:08 | |
| clarkb | that said that chagne shouldn't affect rgroup membership just the var value for members of futureparser group | 19:08 |
| mordred | clarkb: running again | 19:08 |
| mordred | yah | 19:08 |
| clarkb | so the bug must be in the futureparser group membership | 19:08 |
| mordred | gah. what's the scrollback key in screen again? | 19:09 |
| clarkb | mordred: ^A esc | 19:09 |
| clarkb | then esc again to go back | 19:09 |
| mordred | yeah- no hosts in futureparser | 19:09 |
| cmurphy | :( | 19:09 |
| *** apetrich has quit IRC | 19:10 | |
| clarkb | mordred: its the only new style group with |s to regex | 19:10 |
| clarkb | perhaps ansible is interpreting those ? | 19:10 |
| mordred | ok. that worked- let's try more simple regexmatch | 19:11 |
| cmurphy | might be worth mentioning that there's a bunch of crap in http://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/tasks/main.yaml that tries to deal with the possibility that there are different versions of puppet running on the ansible server and the host and it's probably likely that it doesn't deal well with the possibility that there is no puppet running on the ansible server | 19:11 |
| mordred | cmurphy: we should have fixed that by now ... | 19:12 |
| cmurphy | oh i might have missed it | 19:12 |
| *** bobh has quit IRC | 19:12 | |
| clarkb | mordred: I didn't quite catch what the cahgne to the groups.yaml file was | 19:14 |
| * clarkb expects we'll get a patch shortly | 19:14 | |
| *** onovy has joined #openstack-infra | 19:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use jinja match filter instead of regex_match https://review.openstack.org/592655 | 19:15 |
| mordred | clarkb: thereyago | 19:16 |
| mordred | fungi, corvus: ^^ | 19:16 |
| corvus | clarkb, mordred: i have to step out for an errand an lunch; i'll be back asap | 19:17 |
| mordred | woot. and now we have future back | 19:17 |
| mordred | clarkb: I'm going to pull 592655 locally and update things | 19:17 |
| mordred | clarkb: incidentally, I have a patch I want to write to shift a few things WRT inventory once we're in place on the new server | 19:18 |
| clarkb | mordred: ok, I'm reading up on match now | 19:18 |
| clarkb | mordred: you'll rerun against etherpad-dev to check that futureparser is reapplied? | 19:19 |
| mordred | most notable is instead of installing the groups.yaml file into /etc/ansible/hosts - I think we shoudl just point to it in a directory in system-config | 19:19 |
| mordred | clarkb: I already did - and it was | 19:19 |
| clarkb | oh cool and docs say match expects a full match regex | 19:19 |
| clarkb | so its ^'d | 19:19 |
| mordred | clarkb: that way we can have changes to group membership apply immediately with a change, rather than needing the inventory file to be installed irst | 19:19 |
| *** slaweq has quit IRC | 19:19 | |
| clarkb | which is what we want so yay | 19:20 |
| mordred | \o/ | 19:20 |
| mordred | oh - heh. those patches are parent on starting to run the cron - which is thankfully broken | 19:20 |
| clarkb | oh did it try to cron? | 19:21 |
| *** e0ne has joined #openstack-infra | 19:21 | |
| clarkb | mordred: the first one wasn't but the followup to fix groups.yaml is | 19:21 |
| *** slaweq has joined #openstack-infra | 19:22 | |
| clarkb | mordred: if there isn't any cron related fallotu to be concerned about I think we can work on testing the next things? you probably want to rebase the fixes so we can get those in first? | 19:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use jinja match filter instead of regex_match https://review.openstack.org/592655 | 19:24 |
| clarkb | mordred: https://review.openstack.org/#/c/592655/1..2/playbooks/roles/install-ansible/files/groups.yaml shows three groups being removed? is that intentional | 19:24 |
| mordred | yes. done. and those groups are added in the "remove puppet" patch | 19:25 |
| mordred | so yeah | 19:25 |
| clarkb | gotcha | 19:25 |
| mordred | we'll need a rebase of the remove puppet patch now | 19:25 |
| mordred | and yes - thereare no cron fallout things | 19:25 |
| clarkb | mordred: in that same diff I think you need to remove the ~ from the git-* groups | 19:25 |
| mordred | ++ | 19:25 |
| clarkb | because it already expects a regex | 19:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use jinja match filter instead of regex_match https://review.openstack.org/592655 | 19:26 |
| mordred | yup | 19:26 |
| clarkb | +2 | 19:26 |
| *** e0ne has quit IRC | 19:27 | |
| mordred | clarkb: ok - I re-ran the bridge playbok - am re-running puppet | 19:29 |
| clarkb | puppet nooped which I think was expected (because we didn't run base first) | 19:30 |
| clarkb | (when corvus returns I'll step out for my lunch) | 19:30 |
| AJaeger | corvus, just noticed a broken .zuul.yaml file for our CI - it was added to support running in third-party CI, see https://review.openstack.org/#/c/580561/ . This fails running in our CI since the nodes do not exist. Do we have a policy about this? | 19:30 |
| clarkb | AJaeger: maybe not policy but I want to say I explained how to set this up on thei nfra list | 19:31 |
| clarkb | AJaeger: my suggestion was that it would be ok that jobs be defined in our git repos but the check, etc pipeline config needed to be deployment specific and not in the upstream repos | 19:32 |
| AJaeger | clarkb: the check pipeline is in our CI - so shall I send a patch to remove it? | 19:32 |
| clarkb | ya just line 34 needs to be removed I think | 19:33 |
| clarkb | the idea was it would help people understand what the third party ci is doing if we have as much config as possible upstream, but then don't confuse zuul by trying to run those jobs in our pipelines | 19:33 |
| AJaeger | yep | 19:33 |
| clarkb | (so pipeline config had to be third party ci local) | 19:33 |
| AJaeger | agreed | 19:33 |
| AJaeger | clarkb: https://review.openstack.org/592664 | 19:36 |
| clarkb | mordred: the futureparser fixes have +2s if you want to approve them. I'm looking at next round of fails on the server.pp cleanup | 19:36 |
| mordred | cool | 19:36 |
| clarkb | mordred: http://logs.openstack.org/36/585836/20/check/puppet-beaker-rspec-infra-system-config/0cee700/job-output.txt.gz#_2018-08-16_19_24_54_138122 those should be an easy fix I'll just remove those expectations | 19:37 |
| AJaeger | infra-root, could you approve clarkb's infra-manual change to document the in-repo move, please? I would like to link to the new pages.. | 19:37 |
| clarkb | mordred: assuming its ok if I push up a new patch | 19:37 |
| AJaeger | infra-root, I'll approve myself - not sure why ianw didn't... | 19:37 |
| clarkb | AJaeger: thanks | 19:38 |
| cmurphy | clarkb: we could probably remove those tests entirely since it's now up to ansible and not puppet | 19:38 |
| mordred | cmurphy: please do | 19:38 |
| clarkb | cmurphy: good point | 19:38 |
| cmurphy | i just wrote those to make sure that puppet 4 wasn't breaking openstack_project::server | 19:38 |
| mordred | well... | 19:38 |
| mordred | actually- switch the tests | 19:38 |
| *** slaweq has quit IRC | 19:38 | |
| mordred | it's kind of great that we have coverage of the state from the exisitng puppet tests | 19:38 |
| mordred | so maybe just rmeove the yolanda/ricky bits - but the testing of the state overall has been super helpful to validate that the ansible is no-op change | 19:39 |
| clarkb | mordred: ok I'll update tests | 19:39 |
| clarkb | and thats a good point that its checking we don't have a delta | 19:39 |
| clarkb | (and we do have a delta but it is intentioanl here) | 19:40 |
| *** rlandy_ is now known as rlandy | 19:40 | |
| *** jrist has quit IRC | 19:40 | |
| clarkb | mordred: if I push that up you ok to rebase everything else on top of it later? | 19:40 |
| clarkb | (I'm kind of lost at where the stack goes now) | 19:40 |
| mordred | yah | 19:40 |
| mordred | in fact, push up that fix and then I'll do the rebase real quick so we're back on track | 19:41 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 19:42 |
| clarkb | that should do it | 19:42 |
| smcginnis | Just to circle back on the conversation earlier - I did add git fetch --purge to our validation job and that passed. | 19:42 |
| smcginnis | But then resulted in - http://logs.openstack.org/86/86fafe61e7566a182e9fc761390435997e1bb934/release-post/tag-releases/120175b/job-output.txt.gz#_2018-08-16_19_36_46_287431 | 19:42 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Push PTI job config into projects https://review.openstack.org/571520 | 19:43 |
| smcginnis | So I guess if we do it that way, we need to add the --purge everywhere, or we just wait until tomorrow. | 19:43 |
| clarkb | smcginnis: ya anywhere you want to create branches you either need to allow for changing them or clean them up first to get around this I think | 19:43 |
| *** bobh has joined #openstack-infra | 19:44 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: prune dead branches when syncing with origin https://review.openstack.org/592667 | 19:46 |
| mnaser | infra-root: can we do a spot check and find out if adding our role again hasnt broken the world | 19:46 |
| clarkb | mnaser: adding which role? | 19:46 |
| mnaser | sorry. brain too stuck in ansible | 19:47 |
| mnaser | our infra* | 19:47 |
| clarkb | mnaser: ya I can write a lgostash query for all th failed jobs in vexxhost then spot check them | 19:47 |
| clarkb | mordred: next step on rbdige is to test review/lists/etc or are you waiting for the groups changes to merge first? | 19:48 |
| *** bobh has quit IRC | 19:48 | |
| clarkb | I guess probably a good idea to have that fixed | 19:48 |
| *** jamesmcarthur has quit IRC | 19:49 | |
| mordred | clarkb: actually - amusingly enough ... | 19:49 |
| mordred | I think yolanda and ricky not being in the user list is a bug :) | 19:49 |
| *** jamesmcarthur has joined #openstack-infra | 19:49 | |
| *** bobh has joined #openstack-infra | 19:50 | |
| clarkb | mordred: oh | 19:50 |
| clarkb | well then | 19:50 |
| clarkb | mnaser: node_provider:"vexxhost-ca-ymq-1" AND message:"git.openstack.org/openstack-infra/project-config/playbooks/base/post-logs.yaml@master" AND build_status:"FAILURE" is my query string | 19:51 |
| cmurphy | another win for tests | 19:51 |
| clarkb | I've pciked http://logs.openstack.org/91/590491/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/31b8133/job-output.txt http://logs.openstack.org/39/592639/1/check/ipa-tempest-dsvm-ironic-inspector-src/a2b89c6/job-output.txt http://logs.openstack.org/85/592585/1/check/karbor-dsvm-fullstack/fdac401/job-output.txt to spot check | 19:51 |
| mordred | clarkb: fixing | 19:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Improve Central Config Exception description https://review.openstack.org/592669 | 19:51 |
| AJaeger | clarkb: small update to your change ^ | 19:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 19:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 19:53 |
| mordred | clarkb: ok - rebased - and also added yolanda and ricky back | 19:53 |
| mordred | clarkb: (they were in the original list of puppet users but somehow not in the ansible one) | 19:53 |
| clarkb | mnaser: tripleo timed out which isn't surprising, the ironic test failed to start its fake baremetal quickly enough but it only seemed to wait about 10 seconds whcih seems exceedingly fast, and karbor failed on tempest tests | 19:54 |
| corvus | mordred, clarkb: back -- if there's anything i should be doing, can you tldr me? | 19:54 |
| clarkb | mnaser: I don't think any of those are indicating unhappy clouds | 19:54 |
| mordred | corvus: we found bugs | 19:54 |
| *** bobh has quit IRC | 19:54 | |
| mordred | corvus: https://review.openstack.org/585836 should be good to go | 19:54 |
| mordred | corvus: more importantly, we discovered that our use of |regex_match in the groups.yaml file was ... not working | 19:54 |
| mordred | corvus: it has now been replaced with 'is match' - which does work | 19:55 |
| clarkb | with corvus back I'm going to go eat. But I think we are in good shape to test manually on some of the more complicated hosts next and then turn on cron and merge the server.pp cleanup | 19:55 |
| mordred | the relevant fix has been pulled directly on to bridge and is also winding through the gate | 19:55 |
| cmurphy | mordred: it's going to fail | 19:55 |
| mordred | \o/ | 19:55 |
| mordred | that's exciting | 19:56 |
| corvus | i gave it a +2 at the same time cmurphy said that | 19:56 |
| cmurphy | undo clarkb's change | 19:56 |
| mnaser | Thanks for checking it out clarkb | 19:56 |
| mordred | cmurphy: oh - piddle. because I fixed them missing, right? | 19:56 |
| cmurphy | right | 19:56 |
| mordred | corvus: https://review.openstack.org/#/c/592638/2 and https://review.openstack.org/#/c/592655/3 if you wanna see the two things going in | 19:56 |
| openstackgerrit | Merged openstack-infra/system-config master: Use ansible group vars for futureparser flag https://review.openstack.org/592638 | 19:57 |
| *** rosmaita has quit IRC | 19:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 19:57 |
| mordred | corvus: ok. NOW that ^^ should be good, right cmurphy ? | 19:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/afsmon master: Fix zuul.yaml https://review.openstack.org/592670 | 19:57 |
| *** eharney has quit IRC | 19:58 | |
| fungi | wow that's a massive change (585836) | 19:58 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move afsmon publish-to-pypi here https://review.openstack.org/592671 | 19:58 |
| mordred | fungi: yeah. it's the real fun one | 19:59 |
| mordred | fungi: but it's mostly deletions, which means in a bugs-per-sloc perspective it should have a negative amount of bugs | 19:59 |
| clarkb | and we just fixed 2 of them | 20:00 |
| fungi | right, yay for tests! | 20:00 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Improve Central Config Exception description https://review.openstack.org/592669 | 20:00 |
| *** jcoufal_ has joined #openstack-infra | 20:00 | |
| *** bobh has joined #openstack-infra | 20:01 | |
| mordred | corvus: so - how brave are you feeling? ready to try lists.o.o ? or should we start with firehose.o.o - which also has ansible exim things but isn't as noticable of a service as lists? | 20:01 |
| corvus | mordred: let's start with firehose, then lists | 20:04 |
| *** jcoufal has quit IRC | 20:04 | |
| corvus | if lists goes wrong, we can stand by to shut it down quickly | 20:04 |
| mordred | ++ | 20:04 |
| mordred | oh - I should mention to folks | 20:04 |
| mordred | because we're using the new inventory plugin framework ... ansible automatically does cache invalidation when inventory files get touched | 20:04 |
| *** e0ne has joined #openstack-infra | 20:05 | |
| *** bobh has quit IRC | 20:05 | |
| mordred | this is a good thing - but if you wonder why sometimes a command might take a minute ... | 20:05 |
| mordred | corvus: ok - I'm going to run base on firehose | 20:06 |
| mordred | firehose run done - I can still log in | 20:08 |
| corvus | mordred: the word 'cyrus' should show up in the firehose exim4.conf, right? | 20:09 |
| mordred | corvus: yah. I think something there was unhappy | 20:09 |
| openstackgerrit | Merged openstack-infra/system-config master: Use jinja match filter instead of regex_match https://review.openstack.org/592655 | 20:10 |
| mordred | oh! | 20:10 |
| mordred | I see it | 20:10 |
| mordred | firehose.openstack.org.yaml is an incorrect filename | 20:10 |
| corvus | mordred: does it need to be firehose01 or else a group? | 20:11 |
| mordred | yup | 20:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Make a firehose group with firehose01 in it https://review.openstack.org/592675 | 20:12 |
| openstackgerrit | Saul Wold proposed openstack-infra/project-config master: stx-specs: Add new repo for specifications https://review.openstack.org/592676 | 20:12 |
| *** SteffanW has quit IRC | 20:13 | |
| corvus | mordred: lgtm, and i believe lists.* is already taken care of | 20:13 |
| mordred | yah. lists.o.o is the actual fqdn | 20:13 |
| *** eharney has joined #openstack-infra | 20:13 | |
| *** pcaruana has joined #openstack-infra | 20:13 | |
| mordred | this is a thing I think we're probably going to want to think about / be careful with as we roll out more numbered servers | 20:13 |
| corvus | mordred: lists.* is in a group called mailman | 20:13 |
| mordred | oh - yeah. that too | 20:13 |
| mordred | it's in a mailman group- but also its fqdn is lists.opentack.org - so host-specific host_vars should also be correct | 20:14 |
| corvus | mordred: yeah. it's awkward right now, but i think considering that we want to number hosts, groups for each is the right way. maybe we can automate it or something later. | 20:14 |
| mordred | ++ | 20:14 |
| mordred | corvus: we can make 'templated' groups with teh constructed plugin as well | 20:15 |
| corvus | assuming that means what i imagine it means, that sounds great! | 20:15 |
| mordred | so we could have it automatically make groups for hosts that have [a-z]+[0-9]+.openstack.org names | 20:15 |
| openstackgerrit | Saul Wold proposed openstack-infra/project-config master: stx-specs: Add zuul job https://review.openstack.org/592677 | 20:16 |
| mordred | corvus: https://docs.ansible.com/ansible/2.5/plugins/inventory/constructed.html "keyed groups" | 20:16 |
| corvus | mordred: so you can have it make group(\$1) for (.*?)\d+.openstack.org ? | 20:16 |
| mordred | corvus: or it that doesn'tdo exactly what we want - we can write a plugin that does | 20:16 |
| *** jamesmcarthur has quit IRC | 20:17 | |
| mordred | corvus: yah. reading the docs I think we might need our own plugin - or add a feature to that one | 20:17 |
| *** rosmaita has joined #openstack-infra | 20:18 | |
| openstackgerrit | Saul Wold proposed openstack-infra/project-config master: stx-specs: Add new repo for specifications https://review.openstack.org/592676 | 20:18 |
| openstackgerrit | Saul Wold proposed openstack-infra/project-config master: stx-specs: Add zuul job https://review.openstack.org/592677 | 20:18 |
| mordred | corvus: I'm going to go ahead and pull that patch onto bridge | 20:18 |
| mordred | corvus: and now run base on firehose again - see if we get a better exim config this time | 20:20 |
| mordred | corvus: yay! there's a cyrus in them thar exim | 20:21 |
| mordred | does that look right? | 20:22 |
| corvus | lookin | 20:22 |
| corvus | mordred: i don't suppose we have the old one around? | 20:23 |
| *** eharney has quit IRC | 20:24 | |
| mordred | corvus: no- but we could run puppet and get the old one back | 20:24 |
| mordred | (that's our next step anyway) | 20:24 |
| corvus | mordred: let's do that... i have a "hrm..." | 20:24 |
| mordred | kk. doing | 20:24 |
| corvus | exim -bt lpmqtt did not say what i expected it to | 20:25 |
| corvus | is all that module rsync noise normal? | 20:26 |
| mordred | yah | 20:26 |
| mordred | or, rather - I have seen it before | 20:26 |
| corvus | indeed there was a problem, 1 sec while i look | 20:26 |
| mordred | \o/ | 20:26 |
| *** e0ne has quit IRC | 20:27 | |
| corvus | okay. in puppet, we have the cyrus router first, followed by localuser. in ansible we have the localuser router first followed by cyrus. we want the puppet order (but, also, that's actually slightly broken and needs a check added to it. but that can be for another day) | 20:29 |
| mordred | corvus: oh - so, yesterday when we were working on lists - we moved the routers section below the localuser section | 20:29 |
| corvus | how does firehose get its config, does it just have its own exim file? | 20:29 |
| mordred | you originally had it the opposite order in the exim template | 20:29 |
| mordred | corvus: it has exim_routers and exim_templates in playbooks/group_vars/firehose.yaml | 20:30 |
| corvus | mordred: i mean in puppet | 20:30 |
| mordred | oh - no - I believe it has routers and templates there too | 20:30 |
| mordred | corvus: yah - routers and transports in modules/openstack_project/manifests/firehose.pp | 20:31 |
| corvus | default_localuser_router => false, | 20:31 |
| corvus | that's the missing piece | 20:32 |
| mordred | aha | 20:32 |
| corvus | the ansible template is in the right order for lists. but the wrong order for firehose. let me confuse things with 2 choices: | 20:32 |
| fungi | yeah, i had to add that toggle to the puppet-exim module to juggle the localuser delivery behavior differences we want between the cyrus delivery on firehose and the owner address blacklisting for mailman on lists | 20:32 |
| * mordred agrees with all of the choices corvus presents | 20:32 | |
| corvus | 1) do what we did in puppet and add a flag which is basically "turn off the 'default' routers". 2) add the 'default' routers as the default value in the ansible exim module and explicitly re-add them for every host where we set routers | 20:33 |
| *** kevinshort has quit IRC | 20:33 | |
| fungi | basically, cyrus needs to match before localuser since the mailbox to which we're delivering lpmqtt messages matches a local shell account | 20:33 |
| corvus | (so with #2 we'd add localuser to both lists and firehose, but do so in a different order) | 20:34 |
| mordred | corvus: I think I like 2 | 20:34 |
| corvus | i think i do too. fungi? | 20:34 |
| fungi | but localuser needs to match before mailman on lists.o.o and friends so that we can blacklist owner addresses with localuser before exim would end up getting them | 20:34 |
| fungi | yep | 20:34 |
| corvus | mordred: cool, want me to type this one in? | 20:34 |
| *** slaweq has joined #openstack-infra | 20:35 | |
| mordred | corvus: yah - wanna do it on top of https://review.openstack.org/#/c/592675/ | 20:35 |
| corvus | wilco | 20:35 |
| mordred | also - corvus, clarkb, fungi, cmurphy: https://review.openstack.org/#/c/585836/ PASSED TESTS!!! | 20:35 |
| fungi | NICE | 20:35 |
| fungi | surprised that didn't still require more hammering | 20:36 |
| *** r-daneel has quit IRC | 20:36 | |
| cmurphy | \o/ | 20:36 |
| corvus | okay now that i said that... do we maybe not want to do that for the dnslookup and system_aliases routers? | 20:37 |
| corvus | hrm... i have an idea... 1 sec. | 20:38 |
| *** jrist has joined #openstack-infra | 20:40 | |
| EmilienM | do we mirror https://cbs.centos.org/repos/ in infra? can't find it in system-config | 20:41 |
| EmilienM | (oh and hi :-)) | 20:41 |
| mordred | EmilienM: heya - and no, doesn't look like it | 20:43 |
| mordred | EmilienM: modules/openstack_project/templates/mirror.vhost.erb is the file you want | 20:43 |
| EmilienM | mordred: ok, that's what I thought. Thanks :-) | 20:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP: template all exim routers https://review.openstack.org/592682 | 20:43 |
| mordred | EmilienM: we've got buildlogs.centos.org and buildlogs.cdn.centos.org but no cbs | 20:43 |
| corvus | mordred: ^ crazy idea -- is that going to work? | 20:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP: template all exim routers https://review.openstack.org/592682 | 20:44 |
| mordred | corvus: maybe? | 20:45 |
| mordred | corvus: I *think* the jinja stuff gets late-bound | 20:45 |
| *** ssbarnea has quit IRC | 20:45 | |
| corvus | yeah, that was my question, and my thought... | 20:45 |
| corvus | mordred: is it easy to pull patchset 2 onto bridge and try it? | 20:45 |
| corvus | (ps1 had a bug) | 20:45 |
| mordred | corvus: totally | 20:45 |
| *** apetrich has joined #openstack-infra | 20:46 | |
| corvus | mordred: i will write a real commit message while you do that, and also update the lists and storyboard configs (since they'll need it). | 20:46 |
| mordred | corvus: shall I hit go? | 20:46 |
| corvus | mordred: ++ | 20:46 |
| *** jamesmcarthur has joined #openstack-infra | 20:47 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config master: Enable cbs.centos.org mirror https://review.openstack.org/592695 | 20:47 |
| mordred | corvus: it seems to have worked | 20:48 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config master: Enable cbs.centos.org mirror https://review.openstack.org/592695 | 20:48 |
| mordred | corvus: but I will let you validate that to be true | 20:48 |
| corvus | lpmqtt@firehose01.openstack.org | 20:49 |
| corvus | router = cyrus, transport = cyrus | 20:49 |
| corvus | mordred: lgtm | 20:49 |
| mordred | \o/ | 20:49 |
| *** e0ne has joined #openstack-infra | 20:49 | |
| clarkb | EmilienM left a comment on ps1, can you take a look? would've been a -1 if ps2 didn't show up so fast | 20:49 |
| clarkb | mordred: corvus catching up we had exim bugs, anything I should review yet? | 20:50 |
| mordred | clarkb: https://review.openstack.org/#/c/585836/ passed tests | 20:50 |
| mordred | clarkb: corvus will have a followup to https://review.openstack.org/#/c/592682/ in just a sec, but it worked for firehose | 20:50 |
| EmilienM | clarkb: of course | 20:50 |
| *** jamesmcarthur has quit IRC | 20:50 | |
| *** jamesmcarthur_ has joined #openstack-infra | 20:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Template all exim routers https://review.openstack.org/592682 | 20:51 |
| corvus | mordred, clarkb: i think that's it | 20:51 |
| mordred | clarkb: since 585836 passed, we could land it and do the rest of our testing with it in place ... maybe go back through the hosts we've done already | 20:51 |
| mordred | if we do, I thnk we should rebase so that 'enable cron' comes after 'remove things from puppet' | 20:52 |
| clarkb | mordred: seems reasonable, I'd have to do a rereview of it though. let me start with the exim fix | 20:52 |
| mordred | ++ | 20:52 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config master: Enable cbs.centos.org mirror https://review.openstack.org/592695 | 20:52 |
| EmilienM | clarkb: let me know if it's not good. Thanks again for the quick review. | 20:52 |
| *** r-daneel has joined #openstack-infra | 20:53 | |
| *** pcaruana has quit IRC | 20:53 | |
| mordred | corvus: +2 from me - I think that looks really good | 20:53 |
| mordred | corvus: it feels like a good balance between not rewriting the config in cfg-mgmt dsl - and having clean cfg-mgmt building blocks | 20:54 |
| clarkb | mordred: ya its neat once I realized how we were reusing the default blocks | 20:56 |
| clarkb | mordred: corvus do we need similar for transports? | 20:56 |
| clarkb | or is that already the case beacuse we don't have defaults there? | 20:56 |
| clarkb | I approved the firehose group change as that is straightforward but left approval on the exim change to people that have been following that better than me | 20:58 |
| clarkb | I'm going to review the server.pp cleanup now | 20:58 |
| mordred | clarkb: cool - actually, I've got it rebased locally with the enable-cron after the server.pp ... | 20:59 |
| mordred | want me to push that up? | 20:59 |
| clarkb | sure | 20:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 21:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 21:00 |
| mordred | corvus: I've pulled PS3 of exim onto bridge - if you're ready we can try it on storyboard and then hit the list servers | 21:01 |
| clarkb | mordred: we may need to keep https://review.openstack.org/#/c/585836/24/modules/openstack_project/files/puppet.default around to prevent pupept agent from auto starting | 21:02 |
| clarkb | (I grepped for a replacement and didn't see one) | 21:02 |
| *** rh-jelabarre has quit IRC | 21:02 | |
| mordred | nod | 21:04 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a promotional message banner and events list https://review.openstack.org/591870 | 21:05 |
| mordred | clarkb: I've got that locally - but will wait to push it up to see if there are any other things | 21:06 |
| *** bobh has joined #openstack-infra | 21:07 | |
| *** jcoufal_ has quit IRC | 21:07 | |
| clarkb | mordred: ok I'm almost through it | 21:07 |
| clarkb | mordred: ok comments posted you may want to quickly skim them | 21:10 |
| *** bobh has quit IRC | 21:11 | |
| corvus | clarkb: let's leave transports for another day :) | 21:11 |
| clarkb | corvus: wfm | 21:11 |
| corvus | mordred: ready | 21:12 |
| mnaser | is it possible to get run the publish-to-pypi release on a project that it never ran on | 21:14 |
| mnaser | i added ceilometer a few days ago, but i'd like to see if we can get the latest release (11.0.0) up on pypi | 21:15 |
| clarkb | mnaser: we can but it may replace the artifacts that already exist which may confuse some artifact consumers | 21:15 |
| mnaser | https://pypi.org/project/ceilometer/ it has been abanonded since 2012 | 21:16 |
| mnaser | abandoned* | 21:16 |
| clarkb | mnaser: on tarballs.openstack.org I mean | 21:16 |
| mnaser | ah | 21:16 |
| clarkb | the set of jobs will produce new artifacts then upload them. I don't think we can skip to just upload them | 21:17 |
| mnaser | so no way of manually running that, okay :( | 21:17 |
| mnaser | maybe i can ask the ceilometer team to release 11.0.1 | 21:17 |
| clarkb | an owner can also manually upload the prebuilt artifacts | 21:17 |
| *** e0ne has quit IRC | 21:18 | |
| corvus | that's probably the thing to do in this case | 21:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 21:20 |
| *** agopi is now known as agopi|out | 21:20 | |
| mordred | clarkb: ok - updated per your comments - (good catches) | 21:20 |
| openstackgerrit | Merged openstack-infra/system-config master: Make a firehose group with firehose01 in it https://review.openstack.org/592675 | 21:20 |
| mordred | corvus: ^^ updated since last you looked | 21:20 |
| corvus | looking again | 21:20 |
| mordred | corvus: and I am also ready when you are | 21:21 |
| corvus | mordred: go for it | 21:21 |
| *** bobh has joined #openstack-infra | 21:21 | |
| mordred | corvus: ok.I'm going to do storyboard - which will do storyboard and storyboard-dev | 21:22 |
| *** agopi|out has quit IRC | 21:24 | |
| corvus | i'm looking into the error | 21:25 |
| mordred | corvus: storyboard exim templating == unhappy | 21:25 |
| *** bobh has quit IRC | 21:25 | |
| clarkb | mordred: one more comment related to disabling puppet (but not urgent if debugging exim on storyboard) | 21:26 |
| corvus | mordred: i assume it's the {{ but i didn't change that | 21:26 |
| mordred | corvus: yah. we're missing a {% raw %} - fix coming | 21:27 |
| corvus | mordred: did we not try storyboard yet? | 21:27 |
| mordred | corvus: not yet | 21:27 |
| corvus | aha! that makes sense then. :) | 21:27 |
| * corvus awaits mordred's fix | 21:28 | |
| *** bobh has joined #openstack-infra | 21:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add raw blocks to storyboard exim routers https://review.openstack.org/592739 | 21:29 |
| mordred | corvus: ^^ | 21:29 |
| clarkb | everyone read "raw" like Gordon Ramsey | 21:30 |
| mordred | I've pulled it on to bridge so we can run it whenever y'all are good with it | 21:30 |
| clarkb | +2 from me | 21:30 |
| corvus | ++ | 21:32 |
| mordred | also - maybe we should configure fact caching | 21:32 |
| corvus | i guess we need to read the error message more carefully | 21:33 |
| mordred | wel - that did not fix it! :) | 21:33 |
| corvus | to be fair, it's really hard when it's all on one line | 21:33 |
| *** bobh has quit IRC | 21:33 | |
| mordred | thta print did not help | 21:34 |
| clarkb | I read that as fact checking | 21:34 |
| corvus | i guess we need to double double unescape | 21:34 |
| *** graphene has quit IRC | 21:36 | |
| mordred | corvus: maybe I should actually pull the patch | 21:37 |
| *** graphene has joined #openstack-infra | 21:38 | |
| corvus | mordred: if you want to do it *that* way :) | 21:38 |
| mordred | TIL that flat-earthers are making a comeback | 21:39 |
| *** jamesmcarthur_ has quit IRC | 21:41 | |
| *** jamesmcarthur has joined #openstack-infra | 21:41 | |
| mordred | corvus: ok. that looks MUCH better, yes? | 21:41 |
| *** jamesmcarthur has quit IRC | 21:41 | |
| *** diablo_rojo has quit IRC | 21:44 | |
| corvus | mordred: lgtm! | 21:46 |
| corvus | exim -bt storyboard: mail to storyboard@storyboard.openstack.org is discarded | 21:47 |
| corvus | that's correct | 21:47 |
| mordred | \o/ | 21:48 |
| *** bobh has joined #openstack-infra | 21:49 | |
| *** rcernin has joined #openstack-infra | 21:49 | |
| *** slaweq has quit IRC | 21:49 | |
| mordred | clarkb: oh- on your comment on the remove patch - yes - thanks, I forgot about that | 21:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 21:50 |
| mordred | corvus, clarkb ^^ one more change | 21:51 |
| mordred | corvus: wanna do lists? | 21:51 |
| *** bobh has quit IRC | 21:53 | |
| clarkb | mordred: 585836 lgtm now | 21:54 |
| mordred | woot | 21:54 |
| *** jrist has quit IRC | 21:55 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a promotional message banner and events list https://review.openstack.org/591870 | 21:55 |
| corvus | mordred: let's do lists | 21:56 |
| mordred | corvus: ok. | 21:56 |
| *** kei-ichi has joined #openstack-infra | 21:56 | |
| mordred | corvus: maybe ssh in to lists and have a copy of exim4.conf ready to put back in place in case something goes horribly wrong? | 21:57 |
| fungi | that sounds like an excellent idea | 21:57 |
| corvus | mordred: done and ready | 21:57 |
| mordred | corvus: woot. ok. here we go | 21:57 |
| mordred | fungi: don't know if you saw earlier- but there is a screen on bridge going | 21:58 |
| corvus | it's call the "main viewer" | 21:58 |
| *** dbecker has joined #openstack-infra | 21:58 | |
| clarkb | On Screen | 21:58 |
| fungi | thanks, attached now | 21:58 |
| *** boden has quit IRC | 21:58 | |
| fungi | i'm just watching from the science station | 21:59 |
| corvus | mordred: may i have the conn? | 21:59 |
| mordred | yes | 21:59 |
| *** graphene has quit IRC | 21:59 | |
| *** bobh has joined #openstack-infra | 22:00 | |
| corvus | mordred: diff looks good, "exim -bt openstack-dev" checks out, as does the -owner aliases | 22:00 |
| corvus | mordred: you have the conn | 22:00 |
| *** graphene has joined #openstack-infra | 22:01 | |
| mordred | woot! | 22:01 |
| *** apetrich has quit IRC | 22:01 | |
| mordred | shall we try lists.katacontainers.io ? I thnk that'll be the last one on the special list | 22:01 |
| corvus | gimme 1 min to watch the exim log | 22:02 |
| mordred | kk | 22:02 |
| corvus | mordred: looks reasonable; ready for kata | 22:03 |
| mordred | k. here goes | 22:04 |
| *** bobh has quit IRC | 22:04 | |
| corvus | mordred: mind if i drive again? | 22:05 |
| mordred | corvus: all yours | 22:07 |
| corvus | mordred: no longer necessary :) | 22:07 |
| mordred | \o/ | 22:07 |
| corvus | things look good to me. there's not as much traffic there so harder to tell | 22:07 |
| mordred | yah- but the file looks solid | 22:08 |
| corvus | yep, and -bt answers as expected | 22:08 |
| corvus | let me go ahead and do a fake smtp session for this one though | 22:08 |
| *** bobh has joined #openstack-infra | 22:08 | |
| corvus | okay that looks good too | 22:10 |
| corvus | that's everything short of sending a real message over the transport | 22:10 |
| clarkb | what is involved in a fake smtp session? can you tell exim via smtp to not treat this as a real message? | 22:10 |
| corvus | clarkb: here's the transcript: http://paste.openstack.org/show/728238/ | 22:11 |
| corvus | (the ip address is one of mine -- just to tell exim it's a fake connection from a random non-special ip) | 22:12 |
| mordred | I think any time anyone wants to argue with corvus about email we need to just show them that paste :) | 22:13 |
| *** bobh has quit IRC | 22:13 | |
| corvus | oh, i guess input/output is hard to tell. it's mostly output -- i just typed the "ehlo", "mail from", "rcpt to", "data", and "quit" bits | 22:13 |
| corvus | and the message | 22:13 |
| *** harlowja has quit IRC | 22:14 | |
| mordred | corvus: I'm goingto choose to believe that you typed all of the lines | 22:14 |
| fungi | it's the dvorak keyboard that makes him so fast | 22:14 |
| clarkb | looks like we have special rules for jenkins@openstack.org ? | 22:14 |
| clarkb | can probably clean those up if so | 22:14 |
| fungi | doesn't surprise me | 22:14 |
| corvus | # In case people cargo-cult our Zuul configuration, only allow | 22:15 |
| corvus | # messages with an envelope sender of jenkins from OpenStack's Zuul. | 22:15 |
| corvus | or update that to zuul i guess? | 22:15 |
| clarkb | aha | 22:15 |
| clarkb | ya should probably send as zuul now | 22:15 |
| clarkb | mordred: is next step getting 585836 in then running manual tests again? | 22:16 |
| mordred | clarkb, corvus, fungi: ok - what should we take on as our next step? run base as it currently is against all hosts? or land 585836 then do specific spot hosts? | 22:16 |
| clarkb | fwiw I probably only have about 45 more minute today before I've got to run off and entertain guests | 22:16 |
| corvus | yeah, we send as "zuul@zuul.openstack.org" -- so we just don't have any protection from other folks doing that | 22:17 |
| mordred | yeah. I'm similarly around EOD | 22:17 |
| clarkb | mordred: I kinda like the diea of landing 585836 and getting that working | 22:17 |
| mordred | ++ | 22:17 |
| clarkb | mordred: since that will reduce a lot of noise when we run against everything | 22:17 |
| mordred | ++ | 22:17 |
| fungi | are we leaving things in an unfortunate state without 585836? | 22:17 |
| *** dbecker has quit IRC | 22:17 | |
| corvus | mordred: 836++ | 22:17 |
| clarkb | fungi: we are leaving them in the restart exim constantly state | 22:17 |
| fungi | right that | 22:17 |
| clarkb | fungi: which will be noisy and hard to tell if we are nooping as existing | 22:17 |
| clarkb | er | 22:17 |
| clarkb | s/existing/expected | 22:17 |
| fungi | also i can see it would be a bear to keep rebasing | 22:17 |
| mordred | well - currently puppet is not running automatically | 22:17 |
| clarkb | right, we can leave it as is for now if we want | 22:18 |
| mordred | so if we EOD we'll be in a mode of statis | 22:18 |
| mordred | stasis | 22:18 |
| clarkb | though I'm good to try with 585836 and test on some things for a bit too | 22:18 |
| mordred | I think that means new-projects want to not be merged | 22:18 |
| mordred | ++ | 22:18 |
| fungi | cool with me | 22:18 |
| fungi | either way | 22:18 |
| clarkb | and before we wrap up a status log notification of where we are at for the group would be great | 22:18 |
| mordred | let me cherry-pick 585836 onto the local stack we've got | 22:18 |
| mordred | and we can run some more runs | 22:19 |
| *** bobh has joined #openstack-infra | 22:19 | |
| *** gema has quit IRC | 22:19 | |
| openstackgerrit | Merged openstack-infra/system-config master: Template all exim routers https://review.openstack.org/592682 | 22:19 |
| *** edmondsw has quit IRC | 22:20 | |
| mordred | k. I'm going to run bridge.yaml to pick up the updated groups.yaml file- then base.yaml against bridge | 22:20 |
| clarkb | mordred: which groups.yaml update? | 22:21 |
| mordred | clarkb: there are a few additional groups defined in 585836 | 22:21 |
| clarkb | oh got it | 22:22 |
| mordred | mostly important to make sure defined group vars are good | 22:22 |
| mordred | shall we re-try review-dev with the ansible then puppet sequence? | 22:22 |
| clarkb | ++ | 22:22 |
| mordred | it should be less flappy this time | 22:22 |
| *** bobh has quit IRC | 22:23 | |
| mordred | oh no! 585836 failed this time :( | 22:24 |
| clarkb | bah | 22:24 |
| clarkb | Could not find or access 'puppet.default | 22:24 |
| mordred | oh piddle. Ileft the puppet.default reference in base.server | 22:24 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a promotional message banner and events list https://review.openstack.org/591870 | 22:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 22:25 |
| *** edmondsw has joined #openstack-infra | 22:25 | |
| corvus | yay i thought i wasn't going to get to +2 that again! | 22:25 |
| mordred | well - it also failed in prod - so I guess that means our tests are good :) | 22:26 |
| clarkb | failed the same way too | 22:26 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Add a promotional message banner and events list https://review.openstack.org/591870 | 22:26 |
| clarkb | new patch looks good, do we want to wait for zuul to +1 before continuing? | 22:26 |
| mordred | nah | 22:27 |
| mordred | I made the same edit locally on the box and it passed this time | 22:27 |
| clarkb | oh gotcha that is why it doesn't say failed=0 | 22:27 |
| clarkb | er failed=1 | 22:27 |
| mordred | yah | 22:27 |
| fungi | i even said it passed tests in my +2. it so made a liar out of me | 22:27 |
| mordred | it did the first time I ran | 22:27 |
| mordred | fungi: it did pass some tests in the past | 22:27 |
| clarkb | then I did a more thorough review. | 22:28 |
| clarkb | and pointed out things that should be changed, sorry | 22:28 |
| * mordred blames clarkb | 22:28 | |
| clarkb | Aug 16 22:28:48 review-dev01 puppet-user[19395]: (/Stage[main]/Gerrit/File[/home/gerrit2/review_site/etc/gerrit.config]/content) content changed '{md5}0197a098b6e268b5642d0746dc1f80a3' to '{md5}0b7c90ca906a5f97b93e53d63a4037dd' | 22:29 |
| clarkb | now why did that happen | 22:29 |
| * clarkb goes to find where puppet stores those diffs | 22:29 | |
| corvus | clarkb: could be whitespace changes | 22:29 |
| corvus | gerrit writes those files out itself, overwriting our cfgmgt. but usually just changes tabs and spaces around. | 22:30 |
| mordred | I'm going to run ansible again on review-dev - expecting less changes | 22:30 |
| clarkb | corvus: http://paste.openstack.org/show/728239/ | 22:31 |
| clarkb | corvus: seems to use tab indentation on old and new, but the escapes of regexes changed | 22:32 |
| corvus | clarkb: think it's the same kind of thing? gerrit "fixing" it for us? | 22:33 |
| mordred | ok. "Set ssh key for management" seems like it's going to report changed=True every time | 22:33 |
| mordred | corvus: probably | 22:33 |
| clarkb | corvus: maybe, looking at syslog it did the same switch in reverse back when we did the first passes | 22:33 |
| *** dklyle has joined #openstack-infra | 22:33 | |
| mordred | review-dev looks good from an ansible/puppet perspective - I'm going to do etherpad-dev again | 22:34 |
| *** bobh has joined #openstack-infra | 22:34 | |
| clarkb | corvus: Aug 16 18:27:07 review-dev01 puppet-user[577]: (/Stage[main]/Gerrit/File[/home/gerrit2/review_site/etc/gerrit.config]/content) content changed '{md5}0b7c90ca906a5f97b93e53d63a4037dd' to '{md5}0197a098b6e268b5642d0746dc1f80a3' | 22:34 |
| clarkb | then Aug 16 22:28:48 review-dev01 puppet-user[19395]: (/Stage[main]/Gerrit/File[/home/gerrit2/review_site/etc/gerrit.config]/content) content changed '{md5}0197a098b6e268b5642d0746dc1f80a3' to '{md5}0b7c90ca906a5f97b93e53d63a4037dd' | 22:34 |
| mordred | as a followup - I suggest we remove multiverse from our sources.list files | 22:35 |
| clarkb | the current version is the one I believe is correct | 22:35 |
| openstackgerrit | Merged openstack-infra/system-config master: Add raw blocks to storyboard exim routers https://review.openstack.org/592739 | 22:36 |
| clarkb | I'm not super concerned about it since its just the regexes for comment links | 22:36 |
| *** bobh has quit IRC | 22:38 | |
| mordred | clarkb, corvus, fungi: for both review-dev and etherpad-dev - running base, then puppet, then base again is resulting in only 1 changed=True task in the second base run | 22:38 |
| mordred | and the puppet run seems happy | 22:38 |
| mordred | so that seems good | 22:39 |
| *** bobh has joined #openstack-infra | 22:39 | |
| fungi | what was the changed task? | 22:40 |
| mordred | "Set ssh key for management" | 22:40 |
| mordred | it seems to consistently always return changed=True | 22:40 |
| clarkb | mordred: following up on the cron derp from before I see a few Aug 16 19:29:41 etherpad-dev puppet-user[9609]: Finished catalog run in 4.10 seconds runs like that which I am guessing were related to cron? | 22:40 |
| mordred | I *think* it's because of the extra comment | 22:40 |
| clarkb | they don't change ssh keys making me think we didn't run base at all (so was effectively a noop) | 22:41 |
| clarkb | pointing it out because A) we should maybe double check it didn't do anything we didn't want and B) from what I see it acted appropriately and nooped implying the puppet piece of all this works | 22:41 |
| mordred | clarkb: the cron derp should have been a complete noop - it didn't like the cron task | 22:41 |
| clarkb | mordred: hrm I wonder if you manually ran puppe tthen | 22:41 |
| mordred | but we did run against etherpad-dev before | 22:41 |
| clarkb | I can check on a different server | 22:41 |
| clarkb | ya | 22:42 |
| mordred | checka different one I think | 22:42 |
| clarkb | ok etherpad.o.o doesn't have puppet around that time so I think you are correct | 22:42 |
| mordred | sweet | 22:42 |
| clarkb | this must've just been repeating tests | 22:42 |
| mordred | so - before we next tested firehose and storyboard - should we do those as the next two? | 22:43 |
| clarkb | ++ | 22:43 |
| mordred | this second time around is working more smoother | 22:43 |
| *** bobh has quit IRC | 22:43 | |
| clarkb | mordred: not having the deltas and fixing the bugs first helps I'm sure :) | 22:43 |
| mordred | bah. sounds liek round-earth talk to me | 22:44 |
| *** sshnaidm is now known as sshnaidm|off | 22:45 | |
| fungi | makes sense | 22:45 |
| corvus | infra-root: i'm going to stop all zuul executors for a reboot to pick up a new kernel version | 22:46 |
| mordred | corvus: ok | 22:46 |
| mordred | k. firehose looked good- moving on to storyboard | 22:46 |
| fungi | thanks corvus! | 22:48 |
| *** tpsilva has quit IRC | 22:51 | |
| clarkb | Ok I'm about to go do the entertaining, I hear people downstairs. mordred assuming you EOD soon too maybe we leave it in this state, get any oustanding changes merged, reset to master rather than local fork, then work on running cron in the morning? | 22:52 |
| clarkb | mordred: and maybe send email or status log summarizing ^ in a useful way? | 22:52 |
| mordred | clarkb: ++ | 22:53 |
| corvus | #status log restarted all zuul executors with linux 4.15.0-32-generic | 22:56 |
| openstackstatus | corvus: finished logging | 22:56 |
| mordred | corvus: you comfortable with me doing the base/puppet/base dance with the two mailman servers? | 22:56 |
| mordred | (in fact, using the mailman group for it and doing them in paraelle) | 22:57 |
| * fungi is still around, just going to wind down a bit but can still help if something goes sideways with this | 22:57 | |
| * mordred is also about at EOD - mostly just repeating the earlier experiments with 585836 applied | 22:58 | |
| fungi | cool | 22:58 |
| *** jrist has joined #openstack-infra | 23:01 | |
| *** jamesmcarthur has joined #openstack-infra | 23:01 | |
| *** bobh has joined #openstack-infra | 23:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add base.yaml to kick.sh https://review.openstack.org/592760 | 23:03 |
| clarkb | https://review.openstack.org/#/c/592669/2 is an easy infra manual review before people call it a day | 23:04 |
| * mordred stepping away for a few - will check back in before EOD | 23:05 | |
| clarkb | mordred: will you have time to write an update summary? | 23:05 |
| clarkb | (or should someone else attempt it? I can probably do a status log) | 23:05 |
| corvus | mordred: er, yes, but i'm about to eod too... | 23:06 |
| clarkb | how about #status log Puppetmaster is no longer running puppet for us. bridge.openstack.org is now our cfg mgmt control. It is currently in a state of transition while we test things and puppet is not being automatically executed. | 23:06 |
| *** bobh has quit IRC | 23:07 | |
| *** jamesmcarthur has quit IRC | 23:07 | |
| *** rpioso is now known as rpioso|afk | 23:07 | |
| fungi | so this means configuration changes and software updates aren't getting continuously deployed without some manual intervention, right? | 23:08 |
| clarkb | correct | 23:08 |
| fungi | infra-root: ^ important to make note | 23:08 |
| clarkb | likely we'll just manually trigger what mordred has been doing and incorporate whatever that is into the testing :) but we should have the cron running soon | 23:08 |
| clarkb | I'm hoping tomorrow morning we get it going again once we are satisfied it works | 23:08 |
| *** bobh has joined #openstack-infra | 23:09 | |
| * clarkb goes ahead with that status log and we can add more later if appropriate | 23:10 | |
| clarkb | #status log Puppetmaster is no longer running puppet for us. bridge.openstack.org is now our cfg mgmt control. It is currently in a state of transition while we test things and puppet is not being automatically executed. | 23:10 |
| openstackstatus | clarkb: finished logging | 23:10 |
| *** slaweq has joined #openstack-infra | 23:10 | |
| clarkb | #status log This means that config changes will need to be manually applied while we work to get the puppet cron running on bridge.o.o. New projects won't be created for example. | 23:11 |
| openstackstatus | clarkb: finished logging | 23:11 |
| fungi | out of curiosity, does the current state of affairs mean we can now launch new (very boring) nodes without any installed puppet? just wondering whether this is a prime opportunity for me to do a rebuild of my mailman3 poc (since it needs bionic or newer, but doesn't really need any of our service-specific puppet modules) | 23:11 |
| clarkb | fungi: yes if you use the launch-node-ansible script | 23:12 |
| fungi | neat, thanks! | 23:12 |
| clarkb | a followup we need to do too is make ^ our launch node script with the option of also running puppet | 23:13 |
| clarkb | once 585836 merges old launch node won't do much useful things (you won't get a use to log in on the new host for example) | 23:13 |
| *** bobh has quit IRC | 23:13 | |
| clarkb | and with that I'm told babysitter is here so really do need to go visit | 23:13 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Improve Central Config Exception description https://review.openstack.org/592669 | 23:13 |
| *** slaweq has quit IRC | 23:15 | |
| *** bobh has joined #openstack-infra | 23:19 | |
| mordred | clarkb, fungi, corvus: back | 23:23 |
| *** bobh has quit IRC | 23:23 | |
| mordred | I also have an email ready to go | 23:24 |
| * fungi is still mostly around | 23:24 | |
| mordred | fungi: once this is live I'm going to finish updating launch-node | 23:25 |
| mordred | but yes - what clarkb said - I actually booted bridge with launch-node-ansible | 23:25 |
| *** bobh has joined #openstack-infra | 23:28 | |
| *** bobh has quit IRC | 23:33 | |
| fungi | right on | 23:49 |
| *** gyee has quit IRC | 23:53 | |
| *** rlandy is now known as rlandy|bbl | 23:53 | |
| *** bobh has joined #openstack-infra | 23:55 | |
| *** bobh has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!