| mtreinish | ocr it | 00:00 |
|---|---|---|
| mtreinish | I'm sure that it would have interesting results | 00:00 |
| fungi | us$839 (before inflation!) for a fancy 80-track mini-floppy drive capable of storing a whopping 405,000 bytes! | 00:02 |
| fungi | footnote: they didn't spring for a color ad | 00:03 |
| fungi | wow, an article about screen text layout planning, making reference to the classic 16x64 cell worksheet pads... i filled up so many of those | 00:05 |
| fungi | (paper) | 00:16 |
| *** auristor has quit IRC | 00:17 | |
| *** kaisers has quit IRC | 00:20 | |
| *** gyee has quit IRC | 00:21 | |
| *** yolanda_ has joined #openstack-infra | 00:24 | |
| jlvillal | fungi, I used to get 80 Micro. | 00:26 |
| jlvillal | I had a TRS-80 Model 1 with cassette deck as my first computer :) | 00:26 |
| jlvillal | And then a Model 3. Maybe a Model 4. Before switching to a PC. | 00:27 |
| *** esberglu has quit IRC | 00:27 | |
| *** yolanda__ has quit IRC | 00:27 | |
| *** jamesmcarthur has joined #openstack-infra | 00:28 | |
| *** auristor has joined #openstack-infra | 00:29 | |
| *** jlvillal has left #openstack-infra | 00:30 | |
| *** jlvillal has joined #openstack-infra | 00:30 | |
| *** Swimingly has quit IRC | 00:39 | |
| *** Swimingly has joined #openstack-infra | 00:41 | |
| *** Swimingly has quit IRC | 00:41 | |
| *** Swimingly has joined #openstack-infra | 00:41 | |
| *** kaisers has joined #openstack-infra | 00:42 | |
| *** zhangfei has joined #openstack-infra | 00:59 | |
| *** jamesmcarthur has quit IRC | 01:02 | |
| *** ianychoi has quit IRC | 01:11 | |
| *** jamesmcarthur has joined #openstack-infra | 01:12 | |
| *** ianychoi has joined #openstack-infra | 01:12 | |
| *** hemna_ has quit IRC | 01:15 | |
| *** jamesmcarthur has quit IRC | 01:16 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an Amazon EC2 driver https://review.openstack.org/535558 | 01:17 |
| *** slaweq has joined #openstack-infra | 01:17 | |
| *** caphrim007 has joined #openstack-infra | 01:17 | |
| *** yamahata has quit IRC | 01:18 | |
| *** lxkong has quit IRC | 01:18 | |
| *** iyamahat_ has quit IRC | 01:19 | |
| *** slaweq has quit IRC | 01:21 | |
| *** caphrim007 has quit IRC | 01:22 | |
| *** salv-orl_ has joined #openstack-infra | 01:26 | |
| *** hongbin_ has joined #openstack-infra | 01:26 | |
| *** salv-orlando has quit IRC | 01:29 | |
| *** auristor has quit IRC | 01:41 | |
| *** iyamahat has joined #openstack-infra | 01:45 | |
| *** jamesmcarthur has joined #openstack-infra | 01:48 | |
| *** jamesmcarthur has quit IRC | 01:48 | |
| *** auristor has joined #openstack-infra | 01:50 | |
| *** felipemonteiro__ has joined #openstack-infra | 01:51 | |
| *** yamahata has joined #openstack-infra | 01:58 | |
| *** xarses has quit IRC | 02:00 | |
| *** xarses has joined #openstack-infra | 02:00 | |
| *** xarses has quit IRC | 02:00 | |
| *** gongysh has joined #openstack-infra | 02:03 | |
| *** xarses has joined #openstack-infra | 02:04 | |
| *** janki has joined #openstack-infra | 02:09 | |
| *** janki has quit IRC | 02:10 | |
| *** tinwood has quit IRC | 02:10 | |
| *** janki has joined #openstack-infra | 02:11 | |
| *** tinwood has joined #openstack-infra | 02:11 | |
| *** janki has quit IRC | 02:11 | |
| *** janki has joined #openstack-infra | 02:12 | |
| *** janki has quit IRC | 02:13 | |
| *** janki has joined #openstack-infra | 02:14 | |
| *** janki has quit IRC | 02:14 | |
| *** janki has joined #openstack-infra | 02:15 | |
| *** janki has quit IRC | 02:16 | |
| *** janki has joined #openstack-infra | 02:17 | |
| *** janki has quit IRC | 02:17 | |
| *** janki has joined #openstack-infra | 02:18 | |
| *** agopi|zzz has quit IRC | 02:18 | |
| *** jchhatbar has joined #openstack-infra | 02:21 | |
| *** janki has quit IRC | 02:23 | |
| *** nicolasbock has quit IRC | 02:32 | |
| *** dhill_ has joined #openstack-infra | 02:34 | |
| *** agopi|zzz has joined #openstack-infra | 02:38 | |
| *** jchhatbar has quit IRC | 02:38 | |
| *** jchhatbar has joined #openstack-infra | 02:40 | |
| *** jchhatbar has quit IRC | 02:40 | |
| *** felipemonteiro__ has quit IRC | 02:41 | |
| *** jchhatbar has joined #openstack-infra | 02:41 | |
| *** jchhatbar has quit IRC | 02:42 | |
| *** jchhatbar has joined #openstack-infra | 02:43 | |
| *** jchhatbar has quit IRC | 02:43 | |
| *** jchhatbar has joined #openstack-infra | 02:44 | |
| *** jchhatbar is now known as janki | 02:45 | |
| *** iyamahat has quit IRC | 02:45 | |
| *** caphrim007 has joined #openstack-infra | 02:48 | |
| *** iyamahat has joined #openstack-infra | 02:49 | |
| *** VW has quit IRC | 02:52 | |
| *** VW has joined #openstack-infra | 02:53 | |
| *** ramishra has joined #openstack-infra | 03:01 | |
| *** lxkong has joined #openstack-infra | 03:02 | |
| *** psachin has joined #openstack-infra | 03:03 | |
| *** iyamahat has quit IRC | 03:11 | |
| *** slaweq has joined #openstack-infra | 03:18 | |
| *** rosmaita has quit IRC | 03:19 | |
| *** slaweq has quit IRC | 03:22 | |
| *** zhongjun_ has joined #openstack-infra | 03:28 | |
| *** bhavik1 has joined #openstack-infra | 03:37 | |
| *** CrayZee has quit IRC | 03:38 | |
| *** lpetrut has joined #openstack-infra | 03:39 | |
| *** caphrim007_ has joined #openstack-infra | 03:47 | |
| openstackgerrit | Merged openstack-infra/zuul master: Don't try to delete non-existing local refs https://review.openstack.org/563982 | 03:47 |
| *** caphrim007 has quit IRC | 03:50 | |
| *** rkukura_ has joined #openstack-infra | 03:52 | |
| *** VW has quit IRC | 03:56 | |
| *** rkukura has quit IRC | 03:56 | |
| *** rkukura_ is now known as rkukura | 03:56 | |
| *** VW has joined #openstack-infra | 03:56 | |
| *** VW has quit IRC | 03:56 | |
| *** vivsoni__ has joined #openstack-infra | 03:56 | |
| *** VW has joined #openstack-infra | 03:57 | |
| *** VW has quit IRC | 03:57 | |
| *** VW has joined #openstack-infra | 03:57 | |
| *** VW has quit IRC | 03:58 | |
| *** VW has joined #openstack-infra | 03:58 | |
| *** VW has quit IRC | 03:59 | |
| *** vivsoni_ has quit IRC | 03:59 | |
| *** VW has joined #openstack-infra | 04:03 | |
| *** hongbin_ has quit IRC | 04:04 | |
| *** jchhatbar has joined #openstack-infra | 04:10 | |
| *** lpetrut has quit IRC | 04:12 | |
| *** janki has quit IRC | 04:13 | |
| *** VW has quit IRC | 04:16 | |
| shu-mutow-AWAY | AJaeger: is the moving job settings from project-config to .zuul.yaml on own project recommended? | 04:17 |
| *** lpetrut has joined #openstack-infra | 04:19 | |
| *** jchhatba_ has joined #openstack-infra | 04:19 | |
| *** VW has joined #openstack-infra | 04:20 | |
| *** jchhatbar has quit IRC | 04:21 | |
| *** watanabe_isao has joined #openstack-infra | 04:28 | |
| *** yamamoto has quit IRC | 04:29 | |
| *** links has joined #openstack-infra | 04:29 | |
| *** watanabe_isao has quit IRC | 04:32 | |
| *** ociuhandu has joined #openstack-infra | 04:32 | |
| *** VW has quit IRC | 04:33 | |
| *** ociuhandu_ has quit IRC | 04:35 | |
| *** VW has joined #openstack-infra | 04:36 | |
| openstackgerrit | Shu Muto proposed openstack-infra/project-config master: Add required-projects config for docs job https://review.openstack.org/564097 | 04:37 |
| *** pgadiya has joined #openstack-infra | 04:38 | |
| *** pgadiya has quit IRC | 04:38 | |
| *** lpetrut has quit IRC | 04:39 | |
| *** wolverineav has quit IRC | 04:40 | |
| *** VW has quit IRC | 04:40 | |
| *** ianychoi has quit IRC | 04:41 | |
| *** VW has joined #openstack-infra | 04:43 | |
| *** ianychoi has joined #openstack-infra | 04:44 | |
| *** VW has quit IRC | 04:48 | |
| *** yamamoto has joined #openstack-infra | 04:50 | |
| *** wolverineav has joined #openstack-infra | 04:57 | |
| *** felipemonteiro__ has joined #openstack-infra | 04:57 | |
| *** iranzo has joined #openstack-infra | 04:58 | |
| *** VW has joined #openstack-infra | 04:59 | |
| AJaeger | shu-mutow-AWAY: looking at your changes: Please do not reinvent the wheel, let's follow what other projects do | 04:59 |
| *** ianychoi has quit IRC | 05:01 | |
| *** edmondsw has joined #openstack-infra | 05:02 | |
| *** ianychoi has joined #openstack-infra | 05:03 | |
| *** VW has quit IRC | 05:03 | |
| *** felipemonteiro__ has quit IRC | 05:04 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update Launchpad mentions to StoryBoard https://review.openstack.org/561753 | 05:07 |
| *** e0ne has joined #openstack-infra | 05:11 | |
| *** e0ne has quit IRC | 05:12 | |
| *** lpetrut has joined #openstack-infra | 05:15 | |
| *** e0ne has joined #openstack-infra | 05:19 | |
| *** e0ne has quit IRC | 05:21 | |
| *** garyk has joined #openstack-infra | 05:23 | |
| *** lpetrut has quit IRC | 05:25 | |
| *** bhavik1 has quit IRC | 05:26 | |
| *** VW has joined #openstack-infra | 05:31 | |
| *** wolverineav has quit IRC | 05:34 | |
| *** VW has quit IRC | 05:36 | |
| *** edmondsw has quit IRC | 05:36 | |
| *** ianychoi has quit IRC | 05:37 | |
| *** ianychoi has joined #openstack-infra | 05:39 | |
| *** ianychoi has quit IRC | 05:46 | |
| *** VW has joined #openstack-infra | 05:48 | |
| *** ianychoi has joined #openstack-infra | 05:49 | |
| *** armaan has joined #openstack-infra | 05:57 | |
| *** VW has quit IRC | 05:57 | |
| AJaeger | amotoki: I suggest you remove the WIP from https://review.openstack.org/#/c/563440 - let me ping a bit... | 06:00 |
| *** jtomasek has joined #openstack-infra | 06:05 | |
| amotoki | AJaeger: thanks for checking it. I removed the WIP. | 06:05 |
| amotoki | AJaeger: for horizon stuff, I will raise horizon release model (due to the infra change) in a horizon meeting. the meeting time was changed and I can join it now. | 06:06 |
| AJaeger | amotoki: please discuss with mordred about this as well, he drove many of the CI changes | 06:07 |
| amotoki | AJaeger: I persoanlly team would like to split horizon common stuffs to a separate module (like django-horizon) but the effort is beyond the bandwidth of the horizon team at the moment, so the only choice seems to change the rel model for infra change. | 06:07 |
| amotoki | I failed to find enough time to discuss it these couple of weeks..... | 06:08 |
| AJaeger | amotoki: Ah, good idea! I think dhellmann suggested this approach as well. | 06:10 |
| amotoki | but a big question is when we should bump the major version. operator/user visible changes need a major version bump but if we bump a major version for horizon plugin interface change. This sounds tricky from operator perspective.... | 06:11 |
| amotoki | neutron situation is a bit more complicated.... | 06:12 |
| AJaeger | fun ;( | 06:13 |
| *** AJaeger has quit IRC | 06:14 | |
| *** AJaeger has joined #openstack-infra | 06:17 | |
| *** VW has joined #openstack-infra | 06:19 | |
| *** VW has quit IRC | 06:23 | |
| *** iyamahat has joined #openstack-infra | 06:30 | |
| *** xinliang3 has quit IRC | 06:31 | |
| *** xinliang3 has joined #openstack-infra | 06:31 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add start and end timestamp to task result in zuul_json callback https://review.openstack.org/563888 | 06:34 |
| *** AJaeger has quit IRC | 06:35 | |
| *** AJaeger has joined #openstack-infra | 06:38 | |
| *** alexchadin has joined #openstack-infra | 06:44 | |
| *** lpetrut has joined #openstack-infra | 06:45 | |
| *** udesale has joined #openstack-infra | 06:51 | |
| *** ralonsoh has joined #openstack-infra | 06:52 | |
| *** slaweq has joined #openstack-infra | 06:54 | |
| *** tesseract has joined #openstack-infra | 07:01 | |
| *** gfidente has joined #openstack-infra | 07:05 | |
| *** gfidente has joined #openstack-infra | 07:05 | |
| *** pcaruana has joined #openstack-infra | 07:06 | |
| *** udesale has quit IRC | 07:09 | |
| *** lpetrut has quit IRC | 07:10 | |
| *** lpetrut has joined #openstack-infra | 07:10 | |
| mordred | amotoki, AJaeger: good morning! | 07:11 |
| amotoki | mordred: good afternoon :) | 07:11 |
| *** lpetrut has quit IRC | 07:11 | |
| *** xinliang3 has quit IRC | 07:11 | |
| *** lpetrut has joined #openstack-infra | 07:12 | |
| *** xinliang3 has joined #openstack-infra | 07:12 | |
| amotoki | we discussed horizon/neutron siblings stuff for non-tox environemtn like doc build. | 07:12 |
| mordred | infra-root: I'm finally back online - sorry for the unplanned absence - combo of travel flu, travel logistics and a conference really kicked my butt this time | 07:12 |
| *** xinliang3 has quit IRC | 07:13 | |
| mordred | amotoki: awesome. SO - I've owed AJaeger an email to the mailing list on this topic for too long now - I actually wrote one end of last week, then came down with travel flu while proofreading it - but since thinking about it I think I want to re-write it :) | 07:13 |
| *** xinliang3 has joined #openstack-infra | 07:14 | |
| AJaeger | good morning, mordred ! Where are you... | 07:19 |
| amotoki | mordred: welcome back. | 07:19 |
| AJaeger | mordred: happy to review... | 07:19 |
| amotoki | mordred: AJaeger: I rencetly noticed manila-ui introduced another workaround hack of tox-install.sh :( https://github.com/openstack/magnum-ui/blob/master/tox.ini#L37-L89 | 07:19 |
| AJaeger | mordred: just added you to https://review.openstack.org/564099 - your email should answer the problems that shu-mutow-AWAY has ;) | 07:19 |
| *** amoralej|off is now known as amoralej | 07:20 | |
| amotoki | I am worrying another variant of tox_install.sh to cope with this transient situation... | 07:20 |
| *** lpetrut has quit IRC | 07:20 | |
| AJaeger | amotoki: indeed - that why we need the email by mordred | 07:21 |
| amotoki | he introduced this to make local test easier and consume horizon master in CI | 07:21 |
| *** lpetrut has joined #openstack-infra | 07:21 | |
| amotoki | the motivation is understandable, but it should be avoided. we need to explore the right way :) | 07:21 |
| AJaeger | we need to solve this *together*, magnum-ui is not an island ;( | 07:21 |
| amotoki | AJaeger: totally agree. so I am excited mordred is now back | 07:22 |
| AJaeger | ;) | 07:22 |
| amotoki | I also noticed most 'npm' job calls tox internally. | 07:24 |
| amotoki | an example is https://github.com/openstack/octavia-dashboard/blob/master/package.json#L35-L37 and https://github.com/openstack/octavia-dashboard/blob/master/octavia_dashboard/post_install.sh | 07:24 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add start and end timestamp to task result in zuul_json callback https://review.openstack.org/563888 | 07:24 |
| amotoki | ironic-ui example https://github.com/openstack/ironic-ui/blob/master/package.json#L27 | 07:24 |
| amotoki | so I wonder we need something like tox-sibling for npm jobs (or refactor the way of running npm test in each repo) | 07:25 |
| *** udesale has joined #openstack-infra | 07:26 | |
| mordred | AJaeger: I'm currently on a ferry | 07:28 |
| mordred | AJaeger: (and thus dropped internet for 10 minutes - but I'm here :) ) | 07:28 |
| mordred | amotoki: yes - I would like to provide siblings like support for npm things | 07:30 |
| *** florianf has joined #openstack-infra | 07:30 | |
| mordred | amotoki: also - I've learned a BILLION things about javascript over the last 6 months working on the zuul dashboard, so I now have ... opinions :) | 07:30 |
| AJaeger | mordred: will you be next week in Copenhagen? | 07:31 |
| amotoki | mordred: that's nice :) | 07:32 |
| mordred | amotoki: in zuul now we have the javascript/html code all built with webpack natively, and some hooks in place to allow the python code to serve the webpack-built bundle | 07:32 |
| mordred | amotoki: I have not yet looked to see if we can apply the lessons learned there to make things better/easier to horizon | 07:32 |
| mordred | but I've been meaning to :) | 07:33 |
| *** dpawlik has joined #openstack-infra | 07:33 | |
| *** udesale has quit IRC | 07:33 | |
| *** ianychoi has quit IRC | 07:33 | |
| mordred | AJaeger: I'm not planning on it - are you going to be there? | 07:33 |
| mordred | AJaeger, amotoki: https://etherpad.openstack.org/p/6ChjKrKVuX is what I have so far | 07:33 |
| mordred | however, I think I want to restructure it - I'll put a quick second organization idea at the bottom | 07:34 |
| *** dpawlik has quit IRC | 07:34 | |
| *** dpawlik has joined #openstack-infra | 07:34 | |
| AJaeger | mordred: Yep, will be at KubeCon. Just around the corner (nearly ;). Just wanted to check since I missed you in December in Austin :/ | 07:34 |
| *** apetrich has quit IRC | 07:34 | |
| amotoki | I am not so familiar with JS area in horizon, but i know magnum-ui maitainers (who maintains various horizon plugins with angular) and would like to loop him into the discussion. | 07:35 |
| *** apetrich has joined #openstack-infra | 07:35 | |
| *** ianychoi has joined #openstack-infra | 07:35 | |
| *** szaher has quit IRC | 07:37 | |
| AJaeger | mordred: I'll add some quick comments - you see my color ;) | 07:38 |
| *** szaher has joined #openstack-infra | 07:38 | |
| amotoki | I am in a meeting. I will check it later. | 07:41 |
| *** jpena|off is now known as jpena | 07:42 | |
| AJaeger | mordred: tell me once you're done... | 07:42 |
| mordred | AJaeger: yah - let's just write near/over each other - I'm not sure I think I need to reoganize anymore | 07:43 |
| *** hwoarang has quit IRC | 07:45 | |
| *** hwoarang_ has joined #openstack-infra | 07:45 | |
| *** ianychoi has quit IRC | 07:46 | |
| *** ianychoi has joined #openstack-infra | 07:49 | |
| *** jpich has joined #openstack-infra | 07:50 | |
| *** ethfci has quit IRC | 07:50 | |
| *** ethfci has joined #openstack-infra | 07:51 | |
| *** gongysh has quit IRC | 07:51 | |
| AJaeger | mordred: have meetings for next two hours now, will check later | 07:52 |
| AJaeger | mordred: and feel free to send out without my review - it's a start of discussion, so we'll iterate anyhow ;) | 07:54 |
| *** hwoarang_ has quit IRC | 07:54 | |
| *** hwoarang has joined #openstack-infra | 07:54 | |
| *** alexchadin has quit IRC | 07:56 | |
| *** alexchadin has joined #openstack-infra | 07:57 | |
| *** dsariel has joined #openstack-infra | 07:57 | |
| *** electrofelix has joined #openstack-infra | 07:58 | |
| *** ssbarnea_ has joined #openstack-infra | 08:00 | |
| *** ifat_afek has joined #openstack-infra | 08:04 | |
| ifat_afek | Hi, seems like the meeting bot is not working in #openstack-meeting-4. Can someone please check it? | 08:04 |
| *** VW has joined #openstack-infra | 08:05 | |
| *** VW has quit IRC | 08:10 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: reporter: do not expect branch attribute in Tag object https://review.openstack.org/560700 | 08:10 |
| *** ralonsoh has quit IRC | 08:10 | |
| *** zoli is now known as zoli|wfh | 08:14 | |
| *** zoli|wfh is now known as zoli | 08:14 | |
| *** lucas-afk is now known as lucasagomes | 08:16 | |
| *** jaosorior has joined #openstack-infra | 08:16 | |
| *** armaan has quit IRC | 08:21 | |
| *** armaan has joined #openstack-infra | 08:22 | |
| *** gongysh has joined #openstack-infra | 08:30 | |
| *** iyamahat has quit IRC | 08:32 | |
| *** dhajare has joined #openstack-infra | 08:33 | |
| *** sambetts|afk is now known as sambetts | 08:37 | |
| *** derekh has joined #openstack-infra | 08:41 | |
| *** zhangfei has quit IRC | 08:42 | |
| *** alexchadin has quit IRC | 08:43 | |
| *** armaan has quit IRC | 08:48 | |
| *** armaan has joined #openstack-infra | 08:48 | |
| *** xinliang has quit IRC | 08:49 | |
| *** xinliang has joined #openstack-infra | 08:51 | |
| *** xinliang has quit IRC | 08:51 | |
| *** xinliang has joined #openstack-infra | 08:51 | |
| *** panda|ruck|off is now known as panda|ruck | 08:54 | |
| *** zhangfei has joined #openstack-infra | 08:54 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/os-loganalyze master: Support ANSI coloring in logs https://review.openstack.org/563947 | 09:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy job for glance https://review.openstack.org/526950 | 09:01 |
| *** dsariel has quit IRC | 09:02 | |
| *** zoli is now known as zoli|brb | 09:03 | |
| *** VW has joined #openstack-infra | 09:04 | |
| *** VW has quit IRC | 09:09 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change barbican meeting time https://review.openstack.org/564000 | 09:12 |
| *** priteau has joined #openstack-infra | 09:14 | |
| *** e0ne has joined #openstack-infra | 09:17 | |
| *** armaan has quit IRC | 09:26 | |
| *** armaan has joined #openstack-infra | 09:26 | |
| *** dtantsur|afk is now known as dtantsur | 09:28 | |
| *** ZZelle has joined #openstack-infra | 09:30 | |
| *** gongysh has quit IRC | 09:35 | |
| *** zoli|brb is now known as zoli | 09:36 | |
| *** zoli is now known as zoli|wfh | 09:36 | |
| *** alexchadin has joined #openstack-infra | 09:39 | |
| *** armaan has quit IRC | 09:40 | |
| *** ianychoi has quit IRC | 09:52 | |
| *** ianychoi has joined #openstack-infra | 09:54 | |
| *** ianychoi has quit IRC | 10:00 | |
| *** ianychoi has joined #openstack-infra | 10:03 | |
| mordred | AJaeger: I still have not sent the email if you want to look still | 10:05 |
| *** ociuhandu has quit IRC | 10:06 | |
| *** ianychoi has quit IRC | 10:12 | |
| *** ianychoi has joined #openstack-infra | 10:17 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/os-loganalyze master: Support ANSI coloring in logs https://review.openstack.org/563947 | 10:18 |
| *** neith has quit IRC | 10:20 | |
| *** neith has joined #openstack-infra | 10:20 | |
| *** camunoz has joined #openstack-infra | 10:22 | |
| *** evgenyl has quit IRC | 10:25 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/os-loganalyze master: Support ANSI coloring in logs https://review.openstack.org/563947 | 10:25 |
| *** evgenyl has joined #openstack-infra | 10:25 | |
| *** alexchadin has quit IRC | 10:30 | |
| *** zxiiro has quit IRC | 10:31 | |
| *** zxiiro has joined #openstack-infra | 10:32 | |
| *** nicolasbock has joined #openstack-infra | 10:33 | |
| *** VW has joined #openstack-infra | 10:34 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/os-loganalyze master: Support ANSI coloring in logs https://review.openstack.org/563947 | 10:35 |
| *** VW has quit IRC | 10:38 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** VW has joined #openstack-infra | 10:45 | |
| *** derekh has quit IRC | 10:48 | |
| *** dsariel has joined #openstack-infra | 10:49 | |
| *** VW has quit IRC | 10:50 | |
| *** simondodsley has quit IRC | 10:50 | |
| *** simondodsley has joined #openstack-infra | 10:51 | |
| *** udesale has joined #openstack-infra | 10:51 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove glance legacy job https://review.openstack.org/526952 | 10:53 |
| *** jpmaxman has quit IRC | 10:58 | |
| *** auggy has quit IRC | 10:58 | |
| *** jpmaxman has joined #openstack-infra | 10:58 | |
| *** udesale has quit IRC | 10:59 | |
| *** auggy has joined #openstack-infra | 10:59 | |
| *** dpawlik has quit IRC | 10:59 | |
| *** dpawlik has joined #openstack-infra | 11:01 | |
| *** dpawlik has quit IRC | 11:05 | |
| *** yamamoto has joined #openstack-infra | 11:07 | |
| *** rh-jelabarre has quit IRC | 11:07 | |
| *** rh-jelabarre has joined #openstack-infra | 11:08 | |
| *** dham1 has quit IRC | 11:10 | |
| *** dham1 has joined #openstack-infra | 11:10 | |
| *** armaan has joined #openstack-infra | 11:10 | |
| *** fmccrthy has quit IRC | 11:11 | |
| *** fmccrthy has joined #openstack-infra | 11:11 | |
| *** lucasagomes is now known as lucas-hungry | 11:14 | |
| *** VW has joined #openstack-infra | 11:15 | |
| *** zoli|wfh is now known as zoli|lunch | 11:19 | |
| *** garyk has quit IRC | 11:22 | |
| *** garyk has joined #openstack-infra | 11:22 | |
| *** rkukura has quit IRC | 11:26 | |
| *** stakeda has quit IRC | 11:27 | |
| *** VW has quit IRC | 11:27 | |
| *** zhangfei has quit IRC | 11:29 | |
| *** wolverineav has joined #openstack-infra | 11:34 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** ldnunes has joined #openstack-infra | 11:36 | |
| *** katkapilatova has joined #openstack-infra | 11:39 | |
| *** katkapilatova has left #openstack-infra | 11:39 | |
| *** udesale has joined #openstack-infra | 11:42 | |
| *** dpawlik has joined #openstack-infra | 11:45 | |
| *** amoralej is now known as amoralej|lunch | 11:48 | |
| *** armaan has quit IRC | 11:51 | |
| *** armaan has joined #openstack-infra | 11:51 | |
| *** udesale has quit IRC | 11:51 | |
| *** udesale has joined #openstack-infra | 11:52 | |
| *** kgiusti has joined #openstack-infra | 11:58 | |
| *** eharney has quit IRC | 12:01 | |
| *** yamamoto has quit IRC | 12:08 | |
| *** VW has joined #openstack-infra | 12:08 | |
| *** tosky has joined #openstack-infra | 12:09 | |
| *** jd__ has joined #openstack-infra | 12:17 | |
| *** raginbaj- has joined #openstack-infra | 12:18 | |
| *** jd_ has quit IRC | 12:23 | |
| *** raginbajin has quit IRC | 12:23 | |
| *** VW has quit IRC | 12:23 | |
| *** garyk has quit IRC | 12:23 | |
| *** jtomasek has quit IRC | 12:23 | |
| *** yolanda_ has quit IRC | 12:23 | |
| *** dklyle has quit IRC | 12:23 | |
| *** lbragstad has quit IRC | 12:23 | |
| *** Guest62751 has quit IRC | 12:23 | |
| *** jrist has quit IRC | 12:23 | |
| *** dayou has quit IRC | 12:23 | |
| *** krtaylor has quit IRC | 12:23 | |
| *** bramwelt has quit IRC | 12:23 | |
| *** frickler has quit IRC | 12:23 | |
| *** strigazi has quit IRC | 12:23 | |
| *** Jeffrey4l has quit IRC | 12:23 | |
| *** gothicmindfood has quit IRC | 12:23 | |
| *** bradm has quit IRC | 12:23 | |
| *** wznoinsk has quit IRC | 12:23 | |
| *** ddurst has quit IRC | 12:23 | |
| *** SamYaple has quit IRC | 12:23 | |
| *** antonym has quit IRC | 12:23 | |
| *** hemna has quit IRC | 12:23 | |
| *** kazsh has quit IRC | 12:23 | |
| *** jd__ is now known as jd_ | 12:23 | |
| *** raginbaj- is now known as raginbajin | 12:23 | |
| *** edmondsw has joined #openstack-infra | 12:24 | |
| *** jpena|lunch is now known as jpena|off | 12:25 | |
| *** garyk has joined #openstack-infra | 12:26 | |
| *** jtomasek has joined #openstack-infra | 12:26 | |
| *** yolanda_ has joined #openstack-infra | 12:26 | |
| *** dklyle has joined #openstack-infra | 12:26 | |
| *** lbragstad has joined #openstack-infra | 12:26 | |
| *** Guest62751 has joined #openstack-infra | 12:26 | |
| *** jrist has joined #openstack-infra | 12:26 | |
| *** dayou has joined #openstack-infra | 12:26 | |
| *** krtaylor has joined #openstack-infra | 12:26 | |
| *** bramwelt has joined #openstack-infra | 12:26 | |
| *** frickler has joined #openstack-infra | 12:26 | |
| *** strigazi has joined #openstack-infra | 12:26 | |
| *** Jeffrey4l has joined #openstack-infra | 12:26 | |
| *** gothicmindfood has joined #openstack-infra | 12:26 | |
| *** bradm has joined #openstack-infra | 12:26 | |
| *** wznoinsk has joined #openstack-infra | 12:26 | |
| *** ddurst has joined #openstack-infra | 12:26 | |
| *** SamYaple has joined #openstack-infra | 12:26 | |
| *** antonym has joined #openstack-infra | 12:26 | |
| *** hemna has joined #openstack-infra | 12:26 | |
| *** kazsh has joined #openstack-infra | 12:26 | |
| *** eventingmonkey has quit IRC | 12:26 | |
| *** mtreinish has quit IRC | 12:27 | |
| *** mtreinish has joined #openstack-infra | 12:28 | |
| *** edmondsw has quit IRC | 12:28 | |
| *** eventingmonkey has joined #openstack-infra | 12:28 | |
| *** sree has joined #openstack-infra | 12:30 | |
| *** zoli|lunch is now known as zoli | 12:31 | |
| *** zoli is now known as zoli|wfh | 12:31 | |
| *** zoli|wfh is now known as zoli | 12:31 | |
| *** lucas-hungry is now known as lucasagomes | 12:33 | |
| *** jcoufal has joined #openstack-infra | 12:38 | |
| *** panda|ruck is now known as panda|ruck|lunch | 12:38 | |
| *** amoralej|lunch is now known as amoralej | 12:40 | |
| *** jpena|off is now known as jpena | 12:41 | |
| *** edmondsw has joined #openstack-infra | 12:43 | |
| *** garyk has quit IRC | 12:44 | |
| *** edmondsw has quit IRC | 12:46 | |
| *** edmondsw has joined #openstack-infra | 12:47 | |
| *** eharney has joined #openstack-infra | 12:47 | |
| *** VW has joined #openstack-infra | 12:47 | |
| *** edmondsw has quit IRC | 12:50 | |
| *** VW has quit IRC | 12:52 | |
| *** Goneri has joined #openstack-infra | 12:55 | |
| *** hashar has joined #openstack-infra | 12:55 | |
| *** esberglu has joined #openstack-infra | 12:56 | |
| *** dizquierdo has joined #openstack-infra | 12:56 | |
| *** jamespage has quit IRC | 12:57 | |
| *** jamespage has joined #openstack-infra | 12:58 | |
| *** sree has quit IRC | 12:59 | |
| *** VW has joined #openstack-infra | 13:02 | |
| *** yamamoto has joined #openstack-infra | 13:04 | |
| *** VW has quit IRC | 13:06 | |
| *** yamamoto has quit IRC | 13:10 | |
| *** trown|outtypewww is now known as trown | 13:11 | |
| *** edmondsw has joined #openstack-infra | 13:13 | |
| *** hashar has quit IRC | 13:15 | |
| *** edmondsw has quit IRC | 13:15 | |
| *** pbourke has quit IRC | 13:15 | |
| *** edmondsw has joined #openstack-infra | 13:15 | |
| *** udesale has quit IRC | 13:19 | |
| *** panda|ruck|lunch is now known as panda|ruck | 13:20 | |
| *** zoli is now known as zli|afk | 13:21 | |
| *** zli|afk is now known as zoli|afk | 13:21 | |
| *** VW has joined #openstack-infra | 13:22 | |
| *** eernst has joined #openstack-infra | 13:23 | |
| fungi | jlvillal: yeah, the model 1 is the first machine i recall learning to write software on. we had a couple heathkits an an altair my dad built by subscription before that but i wasn't really old enough to express even simple algorithms until '79 or so | 13:24 |
| *** VW has quit IRC | 13:24 | |
| fungi | though i did like helping with the soldering station | 13:25 |
| *** VW has joined #openstack-infra | 13:25 | |
| fungi | and learning to sort resistors | 13:25 |
| *** pbourke has joined #openstack-infra | 13:25 | |
| *** dizquierdo has quit IRC | 13:26 | |
| *** zhangfei has joined #openstack-infra | 13:29 | |
| panda|ruck | is tehre any way to understand if networks in cloud provider are having some hiccups ? we are getting random failures related to network | 13:30 |
| *** psachin has quit IRC | 13:30 | |
| *** viks has quit IRC | 13:31 | |
| *** viks has joined #openstack-infra | 13:31 | |
| fungi | #status log logstash-worker20 was rebooted 2018-04-22 20:50z due to a provider incident (CSHD-AjJP61XQ2n5) but is back in working order | 13:31 |
| openstackstatus | fungi: finished logging | 13:31 |
| dhellmann | is there a way to use git review to download a version of a change other than the most recent? I can't figure out the syntax from the help output | 13:31 |
| dhellmann | gerrit used to show the commands to do this, but it doesn't seem to do that any more | 13:32 |
| Shrews | dhellmann: git review -d 555555,6 iirc | 13:32 |
| dhellmann | ah, thanks | 13:32 |
| fungi | #status log ze09 was rebooted 2018-04-25 01:39z due to a provider incident (CSHD-vwoxBJl5x7L) but is back in working order | 13:32 |
| openstackstatus | fungi: finished logging | 13:32 |
| Shrews | ,6 is the 6th patchset | 13:32 |
| dhellmann | yeah, I tried a / but not , | 13:32 |
| *** VW has quit IRC | 13:32 | |
| fungi | dhellmann: unsurprisingly, we picked the same notation for zuul's representation of change,patchset as well | 13:33 |
| *** dave-mccowan has joined #openstack-infra | 13:33 | |
| fungi | panda|ruck: most of the time if we can correlate issues to a specific provider/region (possibly via logstash or elastic-recheck tracking) then we check that provider's status site (if they have one) and/or file a trouble ticket with them | 13:34 |
| dhellmann | fungi : I guess you don't copy & paste from browser windows very often :-) | 13:34 |
| *** rosmaita has joined #openstack-infra | 13:34 | |
| fungi | dhellmann: i use tmux's clipboard to transfer strings between elinks and other console applications at least | 13:35 |
| fungi | panda|ruck: though some of our donor providers also helpfully have operators in channel, so in those cases we may also ping them if it's a time we think they might be at the computer | 13:35 |
| dhellmann | fungi : I was being a little snarky about the fact that the syntax doesn't match the URL | 13:36 |
| *** eernst has quit IRC | 13:38 | |
| *** kiennt26 has joined #openstack-infra | 13:39 | |
| fungi | ahh | 13:41 |
| *** mriedem has joined #openstack-infra | 13:43 | |
| *** owalsh is now known as owalsh_afk | 13:43 | |
| fungi | dhellmann: anyway, we _do_ have documentation (also as a manpage if you can work out how to get it installed into the manpath on your system) https://docs.openstack.org/infra/git-review/usage.html | 13:43 |
| fungi | you can see a couple examples of that syntax toward the end there | 13:43 |
| dhellmann | fungi : ok, I ran "git review -h" and the help there didn't include that syntax so maybe I'll try to put a patch together for that | 13:43 |
| dhellmann | it does show that syntax for one of the other options, I just didn't make the connection | 13:44 |
| fungi | dhellmann: cool, thanks! | 13:44 |
| fungi | if you manage to get the manpage in your manpath, `git review --help` would show it to you | 13:45 |
| fungi | but yeah, `git-review --help` just gives you the built-in argparse help strings | 13:46 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/git-review master: expand help for --download option https://review.openstack.org/564224 | 13:48 |
| *** dave-mcc_ has joined #openstack-infra | 13:48 | |
| dhellmann | fungi : ^^ | 13:48 |
| *** efried has joined #openstack-infra | 13:49 | |
| *** eernst has joined #openstack-infra | 13:50 | |
| *** dave-mccowan has quit IRC | 13:51 | |
| *** zhipeng has joined #openstack-infra | 13:53 | |
| *** dtantsur is now known as dtantsur|brb | 13:53 | |
| pabelanger | infra-root: I'd like to finish off the removal of debian-jessie from nodepool this morning, could I get a review on https://review.openstack.org/562870/ and https://review.openstack.org/563180/ (will need a recheck after first merges) | 13:58 |
| *** dklyle has quit IRC | 14:00 | |
| Shrews | w00t double approving | 14:01 |
| *** trown is now known as trown|brb | 14:03 | |
| AJaeger | mordred: sent it out ;) | 14:05 |
| corvus | AJaeger: sent or send? | 14:06 |
| *** yamamoto has joined #openstack-infra | 14:06 | |
| *** bobh has joined #openstack-infra | 14:06 | |
| AJaeger | corvus: indeed, send. | 14:06 |
| corvus | AJaeger: heh, maybe if he's not back soon we should just send it out for him :) | 14:07 |
| *** trown|brb is now known as trown | 14:07 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove debian-jessie nodesets https://review.openstack.org/562870 | 14:09 |
| *** hemna_ has joined #openstack-infra | 14:11 | |
| *** yamamoto has quit IRC | 14:11 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Enable warnings-as-errors for publish-openstack-sphinx-docs-python3 https://review.openstack.org/564232 | 14:13 |
| *** VW has joined #openstack-infra | 14:13 | |
| *** iyamahat has joined #openstack-infra | 14:19 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:22 | |
| *** andreas_s has joined #openstack-infra | 14:25 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:26 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Colors in job-output.txt https://review.openstack.org/564066 | 14:29 |
| *** felipemonteiro__ has quit IRC | 14:30 | |
| *** zoli|afk is now known as zli|afk | 14:30 | |
| *** zli|afk is now known as zoli|wfh | 14:30 | |
| *** caphrim007_ has quit IRC | 14:33 | |
| AJaeger | corvus: one option for sure ;) | 14:33 |
| *** caphrim007 has joined #openstack-infra | 14:33 | |
| *** caphrim007 has quit IRC | 14:33 | |
| AJaeger | corvus, mordred , could you review https://review.openstack.org/559348 - is this something where a deprecation message should go out? | 14:34 |
| *** caphrim007 has joined #openstack-infra | 14:34 | |
| AJaeger | this is for zuul-jobs and sphinx building | 14:34 |
| *** dhill_ has quit IRC | 14:34 | |
| *** rossella_s has quit IRC | 14:35 | |
| *** tushar has joined #openstack-infra | 14:35 | |
| *** lpetrut has quit IRC | 14:35 | |
| *** lpetrut has joined #openstack-infra | 14:36 | |
| mordred | AJaeger: awesome - sending | 14:36 |
| pabelanger | AJaeger: okay, I think I've stopped all projects on master branches from using slave_scripts/install-distro-packages.sh I haven't looked on stable branches, mostly because codesearch.o.o isn't setup to do that | 14:37 |
| mordred | AJaeger: "[requirements][horizon][neutron] plugins depending on services" is the subject I was going to use | 14:37 |
| *** felipemonteiro has joined #openstack-infra | 14:37 | |
| AJaeger | mordred: ok | 14:37 |
| pabelanger | AJaeger: possible we could create an elastic-recheck query to help track usage too | 14:37 |
| AJaeger | pabelanger: great, thanks! I would check the repos you fixed and check their stable branches | 14:38 |
| corvus | AJaeger: yeah, i think we should send out a notice about that. the role has an option to set warning-on-error, so it will be easy for folks to mitigate, but we're effectively changing the default. we should also update the docs for that role var. | 14:38 |
| pabelanger | AJaeger: yah, good point | 14:38 |
| fungi | corvus: 563781 is the followup we discussed to your laravel tarball fix from yesterday | 14:38 |
| *** felipemonteiro_ has quit IRC | 14:38 | |
| AJaeger | pabelanger: and check http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/batch_update_projects.py - might be something to leverage... | 14:38 |
| fungi | or monday, whenever that was | 14:39 |
| *** rkukura has joined #openstack-infra | 14:39 | |
| *** sree has joined #openstack-infra | 14:40 | |
| corvus | AJaeger: reviewed 559348; mordred you should still take a look i think. | 14:40 |
| AJaeger | thanks, corvus | 14:41 |
| mnaser | infra-root: until zuul gets quota support for volumes, do you want us to either setup a private flavor with certain amount of disk or disable new vms for now? we're pretty much just getting hit with non-stop new-vm requests that instantly fail | 14:41 |
| mnaser | but they still have to go through the scheduler and reach the compute node then fail because of quota | 14:41 |
| mordred | mnaser: I thought we had support for volumes already. don't we pabelanger ? | 14:41 |
| mordred | oh | 14:41 |
| mordred | duh | 14:41 |
| mordred | I'm stupid | 14:41 |
| mordred | QUOTA | 14:41 |
| mnaser | :< | 14:42 |
| fungi | mnaser: are we going over our volume quota? | 14:42 |
| *** zhipeng has quit IRC | 14:42 | |
| mnaser | well something is happening and volumes arent being erased at some point.. nova or cinder, not sure the culprit at this point | 14:42 |
| mnaser | but once quota limits hit, nodepool just spins and pretty much builds vms non stop that go to ERROR state | 14:42 |
| fungi | oh, got it, so there's a leak somewhere? | 14:42 |
| mordred | right. I thnk we were going to also add a leaked volume cleaner | 14:42 |
| mordred | mnaser: we're VERY good at being a spam bot in scenarios like that | 14:43 |
| mordred | mnaser: great way to make sure your API servers can handle a LOT of requests ;) | 14:43 |
| mnaser | # grep 'Requested volume or snapshot exceeds allowed gigabytes quota' /var/log/nova/nova-compute.log | wc -l => 1047 | 14:43 |
| mnaser | in a single compute node | 14:43 |
| pabelanger | let me finish my patch to nodepool for volume quota, started on it the other day | 14:43 |
| mordred | mnaser: and congrts - you've already survived longer than hp public cloud did | 14:43 |
| mnaser | hah :p | 14:44 |
| mordred | pabelanger: do we have the volume quota things you need in shade? | 14:44 |
| mnaser | can we temporarily disable until we get things fixed up? | 14:44 |
| mnaser | as much as i enjoy being benchmarked it's affecting a few things | 14:44 |
| AJaeger | mnaser: sure, want to propose a change? | 14:44 |
| mnaser | ok 1 second | 14:44 |
| pabelanger | mordred: not sure, I think I've just started plumbing changes into nodepool provider, let me check | 14:44 |
| mordred | pabelanger: kk. lemme know if you need anything | 14:45 |
| Shrews | what in nodepool is using volumes? | 14:45 |
| *** felipemonteiro has quit IRC | 14:45 | |
| *** zhipeng has joined #openstack-infra | 14:45 | |
| pabelanger | TODO(ghe): Manage volume and network quotas | 14:46 |
| pabelanger | mordred: ^ | 14:46 |
| pabelanger | oh, that is on set_compute_quotas | 14:46 |
| pabelanger | so, as long as get_compute_quotas() returns them I think we are okay. Testing now | 14:46 |
| mordred | corvus: I agree with your comment on 559348 | 14:47 |
| Shrews | oh, we use boot-from-volume on vexx | 14:48 |
| mordred | pabelanger: I'd expect it to be get_volume_quotas | 14:48 |
| Shrews | nm | 14:48 |
| *** zhangfei has quit IRC | 14:48 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Disable VEXXHOST in Nodepool https://review.openstack.org/564240 | 14:48 |
| mnaser | ^ | 14:48 |
| *** dhajare has quit IRC | 14:48 | |
| *** dhill_ has joined #openstack-infra | 14:48 | |
| Shrews | mnaser: approved | 14:49 |
| Shrews | nodepool should catch it after merge and puppeting things | 14:49 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Further fix laravel branch and master publish jobs https://review.openstack.org/563781 | 14:49 |
| openstackgerrit | David Ames proposed openstack-infra/project-config master: Add Neutron Dynamic Routing charm and BGP interface https://review.openstack.org/564074 | 14:50 |
| *** derekh has joined #openstack-infra | 14:51 | |
| *** dtantsur|brb is now known as dtantsur | 14:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove debian-jessie from nodepool https://review.openstack.org/563180 | 14:52 |
| *** iyamahat has quit IRC | 14:55 | |
| *** armaan has quit IRC | 14:57 | |
| *** armaan has joined #openstack-infra | 14:58 | |
| *** dsariel has quit IRC | 14:59 | |
| *** zhipeng has quit IRC | 15:01 | |
| *** VW has quit IRC | 15:01 | |
| *** pbourke has quit IRC | 15:01 | |
| *** edmondsw has quit IRC | 15:01 | |
| *** jtomasek has quit IRC | 15:01 | |
| *** yolanda_ has quit IRC | 15:01 | |
| *** lbragstad has quit IRC | 15:01 | |
| *** Guest62751 has quit IRC | 15:01 | |
| *** jrist has quit IRC | 15:01 | |
| *** dayou has quit IRC | 15:01 | |
| *** krtaylor has quit IRC | 15:01 | |
| *** bramwelt has quit IRC | 15:01 | |
| *** frickler has quit IRC | 15:01 | |
| *** strigazi has quit IRC | 15:01 | |
| *** Jeffrey4l has quit IRC | 15:01 | |
| *** gothicmindfood has quit IRC | 15:01 | |
| *** bradm has quit IRC | 15:01 | |
| *** wznoinsk has quit IRC | 15:01 | |
| *** ddurst has quit IRC | 15:01 | |
| *** SamYaple has quit IRC | 15:01 | |
| *** antonym has quit IRC | 15:01 | |
| *** hemna has quit IRC | 15:01 | |
| *** kazsh has quit IRC | 15:01 | |
| *** dmueller[m] has quit IRC | 15:01 | |
| *** jhesketh has quit IRC | 15:01 | |
| *** armaan has quit IRC | 15:01 | |
| *** bobh has quit IRC | 15:01 | |
| *** esberglu has quit IRC | 15:01 | |
| *** tosky has quit IRC | 15:01 | |
| *** kgiusti has quit IRC | 15:01 | |
| *** rh-jelabarre has quit IRC | 15:01 | |
| *** xinliang has quit IRC | 15:01 | |
| *** electrofelix has quit IRC | 15:01 | |
| *** iranzo has quit IRC | 15:01 | |
| *** jchhatba_ has quit IRC | 15:01 | |
| *** auristor has quit IRC | 15:01 | |
| *** jbadiapa has quit IRC | 15:01 | |
| *** sshnaidm|off has quit IRC | 15:01 | |
| *** hrubi has quit IRC | 15:01 | |
| *** markvoelker has quit IRC | 15:01 | |
| *** kjackal has quit IRC | 15:01 | |
| *** tonyb has quit IRC | 15:01 | |
| *** diablo_rojo_ has quit IRC | 15:01 | |
| *** slagle has quit IRC | 15:01 | |
| *** andreykurilin has quit IRC | 15:01 | |
| *** panda|ruck has quit IRC | 15:01 | |
| *** mpjetta has quit IRC | 15:01 | |
| *** Nil_ has quit IRC | 15:01 | |
| *** bdodd has quit IRC | 15:01 | |
| *** toabctl has quit IRC | 15:01 | |
| *** kiennt26 has quit IRC | 15:01 | |
| *** eharney has quit IRC | 15:01 | |
| *** vivsoni__ has quit IRC | 15:01 | |
| *** Douhet has quit IRC | 15:01 | |
| *** jogo has quit IRC | 15:01 | |
| *** numans has quit IRC | 15:01 | |
| *** rcarrillocruz has quit IRC | 15:01 | |
| *** jmorgan1 has quit IRC | 15:01 | |
| *** zigo has quit IRC | 15:01 | |
| *** snapiri has quit IRC | 15:01 | |
| *** adam_g has quit IRC | 15:01 | |
| *** adreznec has quit IRC | 15:01 | |
| *** johnthetubaguy_ has quit IRC | 15:01 | |
| *** kota_ has quit IRC | 15:01 | |
| *** darvon has quit IRC | 15:01 | |
| *** gtmanfred has quit IRC | 15:01 | |
| *** ericyoung has quit IRC | 15:01 | |
| *** igormarnat has quit IRC | 15:01 | |
| *** michaelxin has quit IRC | 15:01 | |
| *** jokke_ has quit IRC | 15:01 | |
| *** ChanServ has quit IRC | 15:01 | |
| *** eharney has joined #openstack-infra | 15:01 | |
| *** vivsoni__ has joined #openstack-infra | 15:01 | |
| *** Douhet has joined #openstack-infra | 15:01 | |
| *** jogo has joined #openstack-infra | 15:01 | |
| *** numans has joined #openstack-infra | 15:01 | |
| *** rcarrillocruz has joined #openstack-infra | 15:01 | |
| *** jmorgan1 has joined #openstack-infra | 15:01 | |
| *** zigo has joined #openstack-infra | 15:01 | |
| *** snapiri has joined #openstack-infra | 15:01 | |
| *** adam_g has joined #openstack-infra | 15:01 | |
| *** adreznec has joined #openstack-infra | 15:01 | |
| *** johnthetubaguy_ has joined #openstack-infra | 15:01 | |
| *** kota_ has joined #openstack-infra | 15:01 | |
| *** darvon has joined #openstack-infra | 15:01 | |
| *** gtmanfred has joined #openstack-infra | 15:01 | |
| *** ericyoung has joined #openstack-infra | 15:01 | |
| *** igormarnat has joined #openstack-infra | 15:01 | |
| *** michaelxin has joined #openstack-infra | 15:01 | |
| *** jokke_ has joined #openstack-infra | 15:01 | |
| *** ChanServ has joined #openstack-infra | 15:01 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 15:01 | |
| *** zhipeng has joined #openstack-infra | 15:02 | |
| *** dmueller[m] has joined #openstack-infra | 15:03 | |
| *** jhesketh has joined #openstack-infra | 15:03 | |
| *** VW has joined #openstack-infra | 15:03 | |
| *** pbourke has joined #openstack-infra | 15:03 | |
| *** edmondsw has joined #openstack-infra | 15:03 | |
| *** jtomasek has joined #openstack-infra | 15:03 | |
| *** yolanda_ has joined #openstack-infra | 15:03 | |
| *** lbragstad has joined #openstack-infra | 15:03 | |
| *** Guest62751 has joined #openstack-infra | 15:03 | |
| *** jrist has joined #openstack-infra | 15:03 | |
| *** dayou has joined #openstack-infra | 15:03 | |
| *** krtaylor has joined #openstack-infra | 15:03 | |
| *** bramwelt has joined #openstack-infra | 15:03 | |
| *** frickler has joined #openstack-infra | 15:03 | |
| *** strigazi has joined #openstack-infra | 15:03 | |
| *** Jeffrey4l has joined #openstack-infra | 15:03 | |
| *** gothicmindfood has joined #openstack-infra | 15:03 | |
| *** bradm has joined #openstack-infra | 15:03 | |
| *** wznoinsk has joined #openstack-infra | 15:03 | |
| *** ddurst has joined #openstack-infra | 15:03 | |
| *** SamYaple has joined #openstack-infra | 15:03 | |
| *** antonym has joined #openstack-infra | 15:03 | |
| *** hemna has joined #openstack-infra | 15:03 | |
| *** kazsh has joined #openstack-infra | 15:03 | |
| *** bandini has quit IRC | 15:03 | |
| *** armaan has joined #openstack-infra | 15:04 | |
| *** bobh has joined #openstack-infra | 15:04 | |
| *** esberglu has joined #openstack-infra | 15:04 | |
| *** tosky has joined #openstack-infra | 15:04 | |
| *** kgiusti has joined #openstack-infra | 15:04 | |
| *** rh-jelabarre has joined #openstack-infra | 15:04 | |
| *** xinliang has joined #openstack-infra | 15:04 | |
| *** electrofelix has joined #openstack-infra | 15:04 | |
| *** iranzo has joined #openstack-infra | 15:04 | |
| *** jchhatba_ has joined #openstack-infra | 15:04 | |
| *** auristor has joined #openstack-infra | 15:04 | |
| *** jbadiapa has joined #openstack-infra | 15:04 | |
| *** sshnaidm|off has joined #openstack-infra | 15:04 | |
| *** hrubi has joined #openstack-infra | 15:04 | |
| *** markvoelker has joined #openstack-infra | 15:04 | |
| *** kjackal has joined #openstack-infra | 15:04 | |
| *** tonyb has joined #openstack-infra | 15:04 | |
| *** diablo_rojo_ has joined #openstack-infra | 15:04 | |
| *** slagle has joined #openstack-infra | 15:04 | |
| *** andreykurilin has joined #openstack-infra | 15:04 | |
| *** panda|ruck has joined #openstack-infra | 15:04 | |
| *** mpjetta has joined #openstack-infra | 15:04 | |
| *** Nil_ has joined #openstack-infra | 15:04 | |
| *** bdodd has joined #openstack-infra | 15:04 | |
| *** toabctl has joined #openstack-infra | 15:04 | |
| *** aspiers[m] has quit IRC | 15:04 | |
| *** kbaikov[m] has quit IRC | 15:04 | |
| *** weshay has quit IRC | 15:04 | |
| *** warlord77[m] has quit IRC | 15:05 | |
| *** masayukig has quit IRC | 15:05 | |
| *** ilpianista_ has quit IRC | 15:05 | |
| *** dmueller[m] has quit IRC | 15:05 | |
| *** bandini has joined #openstack-infra | 15:05 | |
| *** weshay has joined #openstack-infra | 15:06 | |
| *** yamamoto has joined #openstack-infra | 15:08 | |
| *** bobh has quit IRC | 15:10 | |
| *** iyamahat has joined #openstack-infra | 15:10 | |
| *** bobh has joined #openstack-infra | 15:10 | |
| pabelanger | mordred: mnaser: https://review.openstack.org/564247 should give us volume limits for shade | 15:12 |
| *** yamamoto has quit IRC | 15:13 | |
| *** caphrim007 has quit IRC | 15:17 | |
| Shrews | pabelanger: lgtm assuming it passes tests | 15:17 |
| *** caphrim007 has joined #openstack-infra | 15:17 | |
| pabelanger | Shrews: yah, tested locally. But agree, lets see if passes testing in gate | 15:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable VEXXHOST in Nodepool https://review.openstack.org/564240 | 15:19 |
| pabelanger | basically copypasta from get_compute_limits | 15:19 |
| *** hemna_ has quit IRC | 15:20 | |
| *** ZZelle has quit IRC | 15:20 | |
| *** andreas_s has quit IRC | 15:21 | |
| *** ccamacho has joined #openstack-infra | 15:21 | |
| *** andreas_s has joined #openstack-infra | 15:21 | |
| *** caphrim007 has quit IRC | 15:22 | |
| fungi | and if we're hammering vexxhost to no benefit right now, we can zero the max-servers there while we work on the volume quota support | 15:24 |
| fungi | ahh, which 564240 is about to do | 15:24 |
| fungi | perfect! | 15:24 |
| *** felipemonteiro has joined #openstack-infra | 15:24 | |
| openstackgerrit | David Ames proposed openstack-infra/project-config master: Add the neutron dynamic routing charm to zuul https://review.openstack.org/564252 | 15:25 |
| *** ifat_afek has left #openstack-infra | 15:25 | |
| *** rlandy has joined #openstack-infra | 15:25 | |
| *** dpawlik has quit IRC | 15:25 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:28 | |
| *** zhipeng has quit IRC | 15:28 | |
| *** rajinir has joined #openstack-infra | 15:29 | |
| *** felipemonteiro__ has joined #openstack-infra | 15:29 | |
| *** felipemonteiro has quit IRC | 15:31 | |
| *** r-daneel has joined #openstack-infra | 15:31 | |
| *** felipemonteiro_ has quit IRC | 15:33 | |
| *** petems has quit IRC | 15:33 | |
| *** gfidente has quit IRC | 15:33 | |
| *** petems has joined #openstack-infra | 15:33 | |
| *** andreas_s has quit IRC | 15:36 | |
| *** dirk has quit IRC | 15:37 | |
| *** dirk has joined #openstack-infra | 15:37 | |
| *** iranzo has quit IRC | 15:38 | |
| *** gyee has joined #openstack-infra | 15:39 | |
| *** gfidente has joined #openstack-infra | 15:39 | |
| *** felipemonteiro__ has quit IRC | 15:40 | |
| *** Ng has quit IRC | 15:40 | |
| *** felipemonteiro__ has joined #openstack-infra | 15:40 | |
| *** Ng has joined #openstack-infra | 15:40 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/git-review master: expand help for --download option https://review.openstack.org/564224 | 15:41 |
| *** hongbin_ has joined #openstack-infra | 15:43 | |
| *** felipemonteiro__ has quit IRC | 15:45 | |
| *** SamYaple has quit IRC | 15:45 | |
| *** tpsilva has joined #openstack-infra | 15:45 | |
| *** mlavalle has joined #openstack-infra | 15:46 | |
| mlavalle | clarkb: you around? | 15:46 |
| clarkb | mlavalle: yes | 15:46 |
| mlavalle | Regarding this http://lists.openstack.org/pipermail/openstack-dev/2018-April/129835.html | 15:47 |
| mlavalle | I tried to help and added this person to the onos release team | 15:47 |
| mlavalle | but then Ihar raised a vlid issue in the mail thread | 15:47 |
| mlavalle | I saw in the audit log that you added Vikram a few weeks ago | 15:47 |
| mlavalle | I don't mind helping as long as I am not stepping on someone's toes | 15:48 |
| clarkb | mlavalle: ya I actually responded to this individual on the infra mailing list at that time and added vikram since it was never done before | 15:48 |
| clarkb | (my thread there seemed to be ignored and a new one was started on -dev) | 15:48 |
| clarkb | mlavalle: I basically said vikram should've been in the group initially so I'ev added them now and they are who to coordinate with to switch ownership. If that isn't possible we'll figure something out | 15:49 |
| mlavalle | clarkb: cool, thanks. I just didn't want this stuff to fall through the crack | 15:49 |
| clarkb | I think if people are going to work on it and vikram isn't interested we should definitely switch ownership | 15:50 |
| mlavalle | what do you suggest, wait a few days? | 15:50 |
| clarkb | has anyone reached out to vikram directly? I would do that ask nicely and if no response in a week or two just make teh switch | 15:51 |
| mlavalle | I can try that | 15:51 |
| mlavalle | see if we get a response | 15:51 |
| *** xgerman_ has quit IRC | 15:53 | |
| *** xgerman_ has joined #openstack-infra | 15:53 | |
| *** felipemonteiro has joined #openstack-infra | 15:54 | |
| *** lpetrut has quit IRC | 15:55 | |
| *** pcaruana has quit IRC | 15:57 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove debian-jessie from reprepro https://review.openstack.org/564263 | 15:57 |
| pabelanger | clarkb: fungi: mordred: Shrews: ^ removes debian-jessie from AFS mirrors | 15:58 |
| pabelanger | our last step | 15:58 |
| *** pcaruana has joined #openstack-infra | 15:59 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:59 | |
| *** patricku_ has joined #openstack-infra | 16:02 | |
| mlavalle | clarkb: sent him an email directly. copied you in the email address I found in Gerrit | 16:02 |
| clarkb | mlavalle: sounds good. I expect it isn't an issue but also don't want to assume that without first trying to do the right thing :) | 16:02 |
| mlavalle | yeap | 16:03 |
| ccamacho | hey guys | 16:03 |
| ccamacho | quick question, do you know if it's possible to query http://graphite.openstack.org/ and get info like, | 16:03 |
| ccamacho | number of machines created, flavors, time that those machines lived if it was not possible to allocate them... | 16:03 |
| ccamacho | I know we can query graphite i.e. http://graphite.openstack.org/render?target=stats.gauges.nodepool.provider.rax-dfw.nodes.building&format=json | 16:03 |
| ccamacho | but I wanted to know if there are specific metric info for things we do like, how many tenants are created, deleted, how much time they live | 16:03 |
| *** khappone_ has joined #openstack-infra | 16:03 | |
| *** ildikov_ has joined #openstack-infra | 16:03 | |
| *** felipemonteiro has quit IRC | 16:03 | |
| clarkb | ccamacho: grafana.openstack.org may answer a lot of those questions for you | 16:04 |
| clarkb | ccamacho: look for the nodepool* dashboards | 16:04 |
| clarkb | ccamacho: it is just presenting the data in graphite with some predefined graphs in dashboards. I think a lot of similar data to what you are asking for is exposed so should give you an idea of what data is there | 16:05 |
| *** hashar has joined #openstack-infra | 16:06 | |
| *** bobh has quit IRC | 16:07 | |
| *** florianf has quit IRC | 16:07 | |
| *** briancurtin has quit IRC | 16:07 | |
| *** briancurtin has joined #openstack-infra | 16:07 | |
| *** pcaruana has quit IRC | 16:08 | |
| *** bmjen has joined #openstack-infra | 16:09 | |
| *** andreaf_ has joined #openstack-infra | 16:09 | |
| *** jmccrory_ has joined #openstack-infra | 16:09 | |
| *** bgmccollum_ has joined #openstack-infra | 16:09 | |
| *** links has quit IRC | 16:09 | |
| *** kencjohnston has joined #openstack-infra | 16:09 | |
| *** vkmc- has joined #openstack-infra | 16:09 | |
| *** yamamoto has joined #openstack-infra | 16:10 | |
| *** SamYaple has joined #openstack-infra | 16:10 | |
| *** andreaf has quit IRC | 16:10 | |
| *** ildikov has quit IRC | 16:10 | |
| *** patricku has quit IRC | 16:10 | |
| *** vkmc has quit IRC | 16:10 | |
| *** _bmjen has quit IRC | 16:10 | |
| *** bgmccollum has quit IRC | 16:10 | |
| *** kencjohnston_ has quit IRC | 16:10 | |
| *** khappone has quit IRC | 16:10 | |
| *** jmccrory has quit IRC | 16:10 | |
| *** patricku_ is now known as patricku | 16:10 | |
| *** vkmc- is now known as vkmc | 16:10 | |
| *** vkmc has quit IRC | 16:10 | |
| *** vkmc has joined #openstack-infra | 16:10 | |
| *** vkmc has quit IRC | 16:10 | |
| *** vkmc has joined #openstack-infra | 16:10 | |
| *** ildikov_ is now known as ildikov | 16:10 | |
| *** SamYaple has quit IRC | 16:10 | |
| *** jmccrory_ is now known as jmccrory | 16:11 | |
| *** andreaf_ is now known as andreaf | 16:11 | |
| ccamacho | hey clarkb thanks a lot for the info, actually seeing the grafana dashboards it's quite easy to know what targets are used from graphite, thanks! | 16:11 |
| *** masayukig has joined #openstack-infra | 16:12 | |
| *** bandini has quit IRC | 16:12 | |
| dmsimard | Sorry for being largely unresponsive and missing yesterday's infra meeting, been sort of a mess between PTO and catching up | 16:13 |
| dmsimard | I was wondering if there was going to be some sort of infra/social thing on sunday prior to the summit ? | 16:14 |
| fungi | ccamacho: also we're using a tool called grafyaml to build those configurations from some simple structured data files (should be links to them from the interface there) | 16:14 |
| *** bandini has joined #openstack-infra | 16:15 | |
| *** yamamoto has quit IRC | 16:15 | |
| *** gfidente has quit IRC | 16:15 | |
| ccamacho | ack fungi thanks! | 16:16 |
| clarkb | dmsimard: I don't know if it will necessarily be Sunday but we tend to try and do something | 16:17 |
| clarkb | I guess I can look at options and polling people for availability | 16:17 |
| dmsimard | clarkb: yeah, I know there's usually an evening for social things -- it's alright if it's not sunday, just starting to prep schedule etc | 16:18 |
| fungi | dmsimard: sunday can be tight for some of us (particularly bod/tc/staff) as there tends to be a leadership mixer following the joint leadership meeting | 16:23 |
| dmsimard | fungi: yeah that's exactly the reason I was asking, in fact | 16:24 |
| fungi | but if people wanted to do an infra something later that evening i could certainly duck out after an hour or two | 16:24 |
| fungi | wouldn't be the first time | 16:25 |
| *** jaosorior has quit IRC | 16:25 | |
| *** felipemonteiro_ has quit IRC | 16:27 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:27 | |
| clarkb | something like https://ethercalc.openstack.org/7vm2xrsk1yju | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Switch to stestr https://review.openstack.org/536862 | 16:29 |
| dmsimard | clarkb: wfm | 16:29 |
| *** sree has quit IRC | 16:30 | |
| *** lucasagomes is now known as lucas-afk | 16:31 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul-jobs master: Default warning-is-error to True for non-legacy Sphinx projects https://review.openstack.org/559348 | 16:32 |
| clarkb | dmsimard: sent to the mailing list as well. As noted in the email I wasn't at last vancouver summit so if people have venue ideas please let me know | 16:33 |
| *** dpawlik has joined #openstack-infra | 16:33 | |
| dmsimard | I wasn't there either, EmilienM is local and I'm sure he has good suggestions for us :D | 16:33 |
| clarkb | I really liked the beer garden in denver because it meant we didn't have to figure out one large bill and it was super laid back when it came to handling a large group | 16:34 |
| clarkb | I'll probably look for something like that | 16:34 |
| *** trown is now known as trown|lunch | 16:36 | |
| *** pcichy has joined #openstack-infra | 16:37 | |
| *** dpawlik has quit IRC | 16:37 | |
| *** panda|ruck is now known as panda|ruck|off | 16:39 | |
| *** yujunz has quit IRC | 16:40 | |
| fungi | yes, i'd just spend the week in that biergarten if they'd let me | 16:40 |
| *** yujunz has joined #openstack-infra | 16:40 | |
| EmilienM | I'm "local" lol | 16:40 |
| EmilienM | aka yet another french guy being in Canada | 16:41 |
| EmilienM | are you folks looking for a bar? i'm definitely your guy then | 16:41 |
| EmilienM | I know a friend who go to bars :P | 16:41 |
| fungi | food and drink options in vancouver more generally, but yes | 16:42 |
| fungi | we hear you may have spent slightly more time in vancouver (and more recently) than most of the rest of us | 16:42 |
| EmilienM | I guess you want to stay around gastown, probably nearby your hotel (the venue) and good place to hang out, many options there | 16:42 |
| *** mrhillsman has quit IRC | 16:43 | |
| EmilienM | yeah and I've been trying a bunch of places recently | 16:43 |
| fungi | "research" on behalf of the community ;) | 16:43 |
| *** mrhillsman has joined #openstack-infra | 16:43 | |
| *** esarault has joined #openstack-infra | 16:45 | |
| *** slaweq has quit IRC | 16:48 | |
| *** diablo_rojo_ has quit IRC | 16:50 | |
| *** ilpianista_ has joined #openstack-infra | 16:50 | |
| *** warlord77[m] has joined #openstack-infra | 16:50 | |
| *** aspiers[m] has joined #openstack-infra | 16:50 | |
| *** kbaikov[m] has joined #openstack-infra | 16:50 | |
| *** dmueller[m] has joined #openstack-infra | 16:50 | |
| *** iyamahat has quit IRC | 16:51 | |
| *** amoralej is now known as amoralej|off | 16:51 | |
| *** zoli|wfh is now known as zoli|gone | 16:51 | |
| *** ramishra has quit IRC | 16:51 | |
| *** zoli|gone is now known as zoli | 16:51 | |
| *** yamahata has quit IRC | 16:54 | |
| *** rkukura has quit IRC | 16:55 | |
| *** coreycb has quit IRC | 16:57 | |
| *** coreycb has joined #openstack-infra | 16:58 | |
| *** slaweq has joined #openstack-infra | 16:58 | |
| *** e0ne has quit IRC | 17:01 | |
| clarkb | apparently vancouver lacks in the proper beer gardens | 17:01 |
| clarkb | I sympathize we haev the same problem here | 17:01 |
| *** lpetrut has joined #openstack-infra | 17:02 | |
| corvus | rain? | 17:02 |
| pabelanger | we could have a picnic in stanley park | 17:02 |
| *** r-daneel_ has joined #openstack-infra | 17:02 | |
| *** derekh has quit IRC | 17:03 | |
| clarkb | corvus: that and weird liquor laws I think | 17:03 |
| clarkb | your drinking must be private | 17:04 |
| esarault | FYI, it's a statutory Holiday on Monday in Canada | 17:04 |
| esarault | There won'T be a lot of things open | 17:04 |
| pabelanger | Stanley's Bar and Grill in stanley park too | 17:04 |
| pabelanger | a short walk from summit | 17:04 |
| *** r-daneel has quit IRC | 17:05 | |
| *** r-daneel_ is now known as r-daneel | 17:05 | |
| clarkb | victoria day says the internet | 17:05 |
| clarkb | on tuesday? | 17:05 |
| esarault | on Monday | 17:05 |
| clarkb | or wait google shows me 2017 | 17:05 |
| esarault | The first day to the Summit | 17:05 |
| *** jpich has quit IRC | 17:05 | |
| clarkb | silly google | 17:05 |
| esarault | it's a statutory holiday | 17:06 |
| *** dhajare has joined #openstack-infra | 17:06 | |
| esarault | and having talked with the marketing folks, they had no clue about the holiday | 17:06 |
| esarault | I just hope restaurants are going to be open | 17:06 |
| esarault | Otherwise it'll be a clusterfudge | 17:06 |
| clarkb | esarault: even in a major city things will shut down outside of government? | 17:06 |
| esarault | Yes | 17:06 |
| esarault | Here in Quebec, on June 24th, EVERYTHING is closed | 17:07 |
| esarault | even grocery stores | 17:07 |
| esarault | Most pharmacies too | 17:07 |
| *** efried has quit IRC | 17:07 | |
| esarault | So I don't know in B.C. how strict they are about it | 17:07 |
| pabelanger | Victoria Day, for us loyal subjects :) | 17:07 |
| esarault | but when you're opened during a statutory holiday in Quebec, you can get multi-thounsand dollar fines | 17:07 |
| esarault | Yeah in Quebec it's PAtrio Day :p | 17:07 |
| esarault | *patriot | 17:07 |
| *** iyamahat has joined #openstack-infra | 17:08 | |
| *** links has joined #openstack-infra | 17:08 | |
| *** ssbarnea_ has quit IRC | 17:08 | |
| *** iyamahat has quit IRC | 17:09 | |
| *** iyamahat has joined #openstack-infra | 17:09 | |
| clarkb | https://www.narcity.com/ca/bc/vancouver/news/whats-open-and-whats-closed-on-victoria-day-in-vancouver looks like we should survive | 17:09 |
| *** links has quit IRC | 17:09 | |
| esarault | Yeah it's the restaurants that's buggin' me | 17:09 |
| clarkb | worst case we can organize a supply run to bellingham | 17:11 |
| esarault | There's always the hotel's lobby restaurant for Monday night I guess but it does feel like a dud on the first night | 17:11 |
| *** yamamoto has joined #openstack-infra | 17:11 | |
| *** jpena is now known as jpena|off | 17:12 | |
| fungi | heh, smuggle food in from wa | 17:12 |
| fungi | and yeah, this may make the hotel lobby bar even more crowded than usual that night | 17:12 |
| clarkb | or to point roberts | 17:13 |
| *** hashar is now known as hasharAway | 17:13 | |
| clarkb | I'm sure they will be in worse shape for dealing with openstackers than vancouver proper though | 17:13 |
| fungi | as long as there's an abandoned overpass we can just hang out there with booze in paper bags, right? | 17:13 |
| clarkb | liquor store open til 6pm ! | 17:14 |
| fungi | would make for an amusing group photo op anyway | 17:14 |
| clarkb | fungi: cmurphy ok I've finally had my tea, should we commence the puppeting? | 17:14 |
| fungi | yes, lets | 17:15 |
| fungi | commence ta puppetin, yall | 17:15 |
| clarkb | fungi: step zero is disabling the puppet cron, I'll do that now I suppose | 17:15 |
| *** edmondsw has quit IRC | 17:15 | |
| *** yamamoto has quit IRC | 17:16 | |
| clarkb | I've commented out the infracloud and regular run_all.sh puppetmaster cron entries | 17:16 |
| clarkb | infra-root as a general heads up we are making updates to ansible role puppet to better support puppet4 as an option (I think also because limesurvey needs newer puppet?) | 17:17 |
| clarkb | process of doing that is disable ansible puppet globally, we'll merge the change then update the local copy and do a targeted run again a host or two before reenabling puppet globally | 17:18 |
| clarkb | https://review.openstack.org/#/c/562068/ is change in question | 17:18 |
| clarkb | fungi: looks like no ansible or puppet is running puppetmaster currently. You want to double check before I approve ^ ? | 17:18 |
| fungi | looking | 17:20 |
| fungi | yeah, looks all clear | 17:20 |
| *** pcaruana has joined #openstack-infra | 17:20 | |
| clarkb | I've approved the change | 17:20 |
| *** tosky has quit IRC | 17:25 | |
| *** rpioso is now known as rpioso|eat | 17:26 | |
| *** tosky has joined #openstack-infra | 17:26 | |
| *** yamahata has joined #openstack-infra | 17:27 | |
| *** diablo_rojo_ has joined #openstack-infra | 17:36 | |
| *** sambetts is now known as sambetts|afk | 17:43 | |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 17:43 |
| *** dtantsur is now known as dtantsur|afk | 17:43 | |
| clarkb | I'm going to manually run ansible-galaxy as root now to pick up ^ | 17:46 |
| clarkb | fungi: cmurphy ok ansible puppet role is updated | 17:48 |
| clarkb | let me know when you are ready to run an ad_hoc kick.sh against smething (I'm thinking etherpad-dev) | 17:48 |
| *** r-daneel has quit IRC | 17:49 | |
| *** r-daneel has joined #openstack-infra | 17:49 | |
| *** edmondsw has joined #openstack-infra | 17:51 | |
| fungi | how do we get ansible-role-puppet updates on puppetmaster? | 17:51 |
| fungi | oh, you just said | 17:51 |
| * fungi is blind | 17:51 | |
| clarkb | fungi: cmurphy pointed it out in the change comments but basically part of run_all.yaml runs a ansible-galaxy update, which I've done yup | 17:51 |
| clarkb | I've run it outside of run_all.yaml so we don't have to update everything to test this | 17:52 |
| fungi | perfect, and yeah etherpad-dev seems like a good target | 17:52 |
| *** hemna_ has joined #openstack-infra | 17:52 | |
| *** shardy has quit IRC | 17:52 | |
| clarkb | fungi: `tools/kick.sh etherpad-dev.openstack.org` out of /opt/system-config/production | 17:53 |
| clarkb | I guess I can run that in a root screen so others can watch it? | 17:53 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: Move environment create/cleanup to python https://review.openstack.org/555463 | 17:53 |
| clarkb | root screen started | 17:54 |
| *** edmondsw has quit IRC | 17:55 | |
| clarkb | fungi: does the queued up command look right ot you ^ | 17:55 |
| fungi | i've attached to your screen session, and yeah that looks like how i've run it | 17:56 |
| clarkb | ok running now | 17:56 |
| clarkb | and now we see what broke | 17:57 |
| fungi | mmm | 17:57 |
| clarkb | Could not find data item etherpad-dev_db_password in any Hiera data file and no default supplied at /opt/system-config/production/manifests/site.pp:474 on node etherpad-dev.openstack.org | 17:57 |
| *** esarault2 has joined #openstack-infra | 17:57 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add regex support to project stanzas https://review.openstack.org/535713 | 17:57 |
| fungi | i see that both in fqdn/etherpad-dev.openstack.org.yaml and group/etherpad-dev.yaml | 17:58 |
| fungi | same value both places | 17:58 |
| clarkb | yup I think it may be on the config side telling puppet where to find hiera data | 17:59 |
| *** esarault has quit IRC | 17:59 | |
| *** edmondsw has joined #openstack-infra | 17:59 | |
| *** edmondsw has quit IRC | 18:00 | |
| *** priteau has quit IRC | 18:00 | |
| fungi | on etherpad-dev.o.o, /etc/puppet/hieradataproduction/fqdn/ and /etc/puppet/hieradataproduction/group/ are both empty | 18:01 |
| clarkb | oh missing / ? | 18:01 |
| *** ssbarnea_ has joined #openstack-infra | 18:01 | |
| fungi | i think so | 18:01 |
| fungi | etherpad.o.o has /etc/puppet/hieradata/production instead of /etc/puppet/hieradataproduction | 18:01 |
| clarkb | fungi: -dev has the first path as well with the real content | 18:02 |
| *** eernst has quit IRC | 18:02 | |
| fungi | oh, yep | 18:02 |
| clarkb | so I'm guessing ya the role just isn't lining things up properly | 18:02 |
| fungi | hieradataproduction was last updated Apr 25 17:56 | 18:02 |
| fungi | so i think we've dropped a / with the most recent change | 18:03 |
| clarkb | https://review.openstack.org/#/c/562068/21/tasks/main.yml lines 30 and 35 those tasks | 18:03 |
| *** diablo_rojo_ has quit IRC | 18:04 | |
| *** edmondsw has joined #openstack-infra | 18:04 | |
| *** diablo_rojo_ has joined #openstack-infra | 18:05 | |
| fungi | agreed, it's string concatenation happening there, not path concatenation | 18:05 |
| fungi | so we should be able to just add a / on each of those lines | 18:06 |
| clarkb | ya | 18:06 |
| clarkb | fungi: do we want to test that with a manual edit in the role then push up the change if that fixes it? | 18:06 |
| clarkb | also now that I see it touches the hieradata on the puppetmaster side do we want to do a quick local backup of that first? | 18:06 |
| fungi | making a copy of /etc/puppet/hieradata is probably warranted, yes | 18:07 |
| fungi | just in case it somehow destroys .git | 18:07 |
| clarkb | fungi: ya, do you want to make that copy? I'm working on the role updates | 18:07 |
| cmurphy | clarkb: hi i am here now | 18:08 |
| clarkb | I've updated the role locally on the puppet master nad will push up a change now | 18:08 |
| fungi | i'm not sure how/where to tweak that task locally but since you seem to know it's all yours | 18:08 |
| fungi | i'll work on the hieradata copy | 18:08 |
| cmurphy | trying to catch up, was there a typo in my change? | 18:08 |
| fungi | cmurphy: minor ones | 18:08 |
| clarkb | cmurphy: ya couple missing /'s I'll have a fix up shortly | 18:08 |
| cmurphy | sorry about that | 18:08 |
| fungi | clarkb: /etc/puppet/hieradata.bak exists now with a duplicate just in case | 18:09 |
| openstackgerrit | Clark Boylan proposed openstack-infra/ansible-role-puppet master: Add some missing /'s to fix hieradata pathing https://review.openstack.org/564305 | 18:10 |
| clarkb | cmurphy: fungi if ^ looks good I've made that edit locally in the local role install and can rerun ansible | 18:10 |
| clarkb | (maybe double check I didn't miss anywhere else that needs updates) | 18:10 |
| *** ssbarnea_ has quit IRC | 18:12 | |
| cmurphy | one minor thing | 18:12 |
| cmurphy | it can live without it | 18:12 |
| *** yamamoto has joined #openstack-infra | 18:12 | |
| *** ssbarnea_ has joined #openstack-infra | 18:12 | |
| clarkb | hrm | 18:12 |
| fungi | doesn't hurt to also remove that while we're testing | 18:13 |
| clarkb | puppet environment doesn't end with a / though right? | 18:13 |
| fungi | if we're fairly sure it's no longer needed | 18:13 |
| clarkb | so in the case of not having a puppet environment we would need that | 18:13 |
| clarkb | *in the case of having an environmnet | 18:13 |
| cmurphy | i'm still unclear where the missing / is happening | 18:13 |
| *** efried has joined #openstack-infra | 18:13 | |
| clarkb | cmurphy: between hieradata and environment | 18:14 |
| clarkb | cmurphy: so we are getting /etc/puppet/hieradataproduction | 18:14 |
| clarkb | instead of /etc/puppet/hierdata/production | 18:14 |
| fungi | '/etc/puppet/hieradata' + puppet_environment | 18:14 |
| cmurphy | but in what config file? | 18:14 |
| fungi | tasks/main.yml | 18:14 |
| clarkb | cmurphy: on disk | 18:14 |
| cmurphy | oh i see | 18:14 |
| fungi | is creating a /etc/puppet/hieradataproduction directory on the target machine | 18:14 |
| fungi | and it's empty | 18:15 |
| fungi | (-ish) | 18:15 |
| clarkb | and copying the hieradata from there then presumably using the hieradata (that doesn't eixst) from there | 18:15 |
| *** kevinbenton has joined #openstack-infra | 18:16 | |
| fungi | right, that's at least the best explanation i have for why it wasn't able to find any hiera keys | 18:16 |
| clarkb | do we want to try and get the correct number of /s before we contineu or are we happy with the fact that linux does the right thing if you double them up? | 18:16 |
| cmurphy | i'm fine with it being doubled up | 18:17 |
| *** Guest62751 is now known as zeus | 18:17 | |
| fungi | i'm good either way | 18:17 |
| *** zeus has quit IRC | 18:17 | |
| *** zeus has joined #openstack-infra | 18:17 | |
| fungi | if we think we want to clean that up we should go ahead and test now that it still dtrt to save us some time | 18:17 |
| clarkb | ok I'm going to rerun it now then as is | 18:17 |
| fungi | but if we just want to leave it that way, i'm cool with it | 18:17 |
| clarkb | fungi: cmurphy reading really quickly I don't think it will be the simplest thing because ansible doesn't have a built in path joiner | 18:17 |
| clarkb | so we'd have to implement one ourselves or keep track of /s everywhere | 18:18 |
| *** yamamoto has quit IRC | 18:18 | |
| *** Goneri has quit IRC | 18:18 | |
| EmilienM | it would really help if we could import https://review.openstack.org/#/c/563197/ asap | 18:18 |
| fungi | EmilienM: looking | 18:19 |
| corvus | clarkb, fungi, cmurphy: is '/'.join() an option? https://github.com/pallets/jinja/issues/367 | 18:19 |
| *** diablo_rojo_ has quit IRC | 18:19 | |
| fungi | oh, you can just put python directives in there? | 18:19 |
| fungi | neat | 18:19 |
| clarkb | corvus: '/'.join('/etc/puppet/hieradata/', puppet_environment) ? | 18:20 |
| clarkb | er | 18:20 |
| clarkb | corvus: '/'.join('/etc/puppet/hieradata', puppet_environment) ? | 18:20 |
| cmurphy | i think my comment on the patch was wrong and the / is still necessary so i don't think it's actually being doubled up | 18:21 |
| clarkb | if puppet_environment is empty its the same problem | 18:21 |
| corvus | clarkb: join(['a', b]) add the [] | 18:21 |
| *** kevinbenton_ has quit IRC | 18:22 | |
| clarkb | so I don't think that helps solve this exact problem, good to keep in mind that is an option in the future though | 18:22 |
| cmurphy | if puppet_environment is empty then the hiera path isn't going to be valid so that's a different problem | 18:22 |
| EmilienM | fungi: thank you | 18:22 |
| clarkb | cmurphy: ah ok | 18:22 |
| clarkb | fungi: let me know when you are done helping EmilienM and are ready to watch a rerun on puppetmaster | 18:23 |
| *** esarault2 is now known as esarault | 18:24 | |
| fungi | yeah, ready | 18:27 |
| fungi | and watching | 18:27 |
| clarkb | it is running | 18:27 |
| *** rpioso|eat is now known as rpioso | 18:27 | |
| fungi | mmm | 18:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Import ansible-role-container-registry https://review.openstack.org/563197 | 18:29 |
| clarkb | missing a + | 18:29 |
| fungi | syslog does show it using path=/etc/puppet/hieradata/production//group | 18:29 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Configure ansible-role-container-registry zuul jobs https://review.openstack.org/563200 | 18:29 |
| fungi | so the double-/ | 18:29 |
| fungi | i guess the environment ends up being "production/" instead of just "production"? | 18:30 |
| fungi | not that it's causing a problem | 18:30 |
| clarkb | fungi: I've put the cursor over the + I just added in the screen I Think that is the issue | 18:31 |
| fungi | yes, i don't think "'/' item" would have been valid | 18:31 |
| *** e0ne has joined #openstack-infra | 18:31 | |
| cmurphy | wait i see it | 18:32 |
| *** ssbarnea_ has quit IRC | 18:32 | |
| clarkb | the last time it ran there were no contents in the hierdata | 18:32 |
| cmurphy | there is a literal // in the task | 18:32 |
| clarkb | so we skipped this but now the path is right and its unhappy | 18:32 |
| clarkb | cmurphy: ya I think that is a thing too | 18:32 |
| cmurphy | on line 42 of main.yml | 18:32 |
| fungi | ahh, yep | 18:32 |
| clarkb | let me push a new patchset then apply manually to pupeptmaster if we are happy with it | 18:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/ansible-role-puppet master: Add some missing /'s and +'s to fix hieradata pathing https://review.openstack.org/564305 | 18:33 |
| clarkb | I think that covers all identified issues so far | 18:33 |
| cmurphy | oh i didn't even see the + problem | 18:33 |
| clarkb | cmurphy: that is what ansibel actually yelled at us about | 18:34 |
| fungi | clarkb: thanks, let's give that a whirl | 18:35 |
| fungi | see if there's anything else it gets stuck on | 18:35 |
| clarkb | ok running now | 18:36 |
| fungi | still unhappy about hiera | 18:36 |
| *** rosmaita has quit IRC | 18:37 | |
| *** calebb has quit IRC | 18:37 | |
| cmurphy | what's the error now? | 18:37 |
| clarkb | can we see how puppet was invoked somewhere? | 18:37 |
| clarkb | cmurphy: Apr 25 18:36:38 etherpad-dev puppet-user[11419]: Could not find data item etherpad-dev_db_password in any Hiera data file and no default supplied at /opt/system-config/production/manifests/site.pp:474 on node etherpad-dev.openstack.org | 18:38 |
| clarkb | cmurphy: same as before, we thought the issue before was the bad pathing on the environment but I guess that isn't the only issue | 18:38 |
| *** calebb has joined #openstack-infra | 18:38 | |
| *** trown|lunch is now known as trown | 18:39 | |
| fungi | i'm going to blow away the /etc/puppet/hieradataproduction directory on etherpad-dev.o.o since that's cleared up at least | 18:40 |
| clarkb | fungi: probably should on puppetmaster too | 18:40 |
| clarkb | cmurphy: looking at /etc/puppet/hiera.yaml on etherpad-dev it looks correct ot me | 18:40 |
| fungi | oh, yep, will do | 18:41 |
| clarkb | "hieradata/%{::environment}/fqdn/%{::fqdn}" is the first hierarchy entry which should have the data | 18:41 |
| clarkb | cmurphy: thinking either we aren't loading that config for some reason or environment isn't set properly | 18:41 |
| fungi | worth noting, we have /etc/puppet/hieradata/production/fqdn/etherpad-dev.openstack.org.yaml but /etc/puppet/hieradata/production/group/ is empty even though we have a matching group on the master | 18:41 |
| fungi | but this also seems to match etherpad.o.o | 18:41 |
| clarkb | fungi: I think it may use the most specific file? | 18:42 |
| fungi | righy | 18:42 |
| clarkb | somethign to figure out too | 18:42 |
| fungi | right | 18:42 |
| notmyname | are feature branch docs builds published anywhere? | 18:43 |
| fungi | clarkb: cmurphy and yeah, there is a etherpad-dev_db_password key in /etc/puppet/hieradata/production/fqdn/etherpad-dev.openstack.org.yaml on etherpad-dev.o.o so it's not looking there i guess | 18:43 |
| clarkb | notmyname: yes should be at $project/$branchname if you have the job configured to run against the branch iirc | 18:44 |
| clarkb | Apr 25 18:36:38 puppetmaster ansible-puppet: Invoked with logdest=syslog execute=None facter_basename=ansible tags=None puppetmaster=None show_diff=False certname=None manifest=/opt/system-config/production/manifests/site.pp environment=production debug=False noop=False timeout=30m facts=None | 18:44 |
| fungi | notmyname: we relied on that for our feature/zuulv3 branches on zuul and nodepool docs | 18:44 |
| clarkb | it seems to have gotten the environment there | 18:44 |
| notmyname | https://docs.openstack.org/swift/feautre/deep is giving me a 404 | 18:44 |
| clarkb | Apr 25 18:36:31 etherpad-dev ansible-puppet: Invoked with logdest=syslog execute=None facter_basename=ansible tags=None puppetmaster=None show_diff=False certname=None manifest=/opt/system-config/production/manifests/site.pp environment=production debug=False noop=False timeout=30m facts=None and there | 18:45 |
| notmyname | heh, even if I speel it correctly | 18:45 |
| notmyname | clarkb: we've got a build-openstack-sphinx-docs job (see https://review.openstack.org/#/c/564277/). does that mean it's published at https://docs.openstack.org/swift/feature/deep | 18:47 |
| fungi | notmyname: agreed, i don't see it under http://files.openstack.org/docs/swift/ | 18:47 |
| fungi | no "feature" subdir there i mean | 18:47 |
| clarkb | notmyname: no, the build job is just a test to make sure you can build, there are separate publis jobs | 18:47 |
| notmyname | ah ok | 18:47 |
| clarkb | I don't see any post job logs at http://logs.openstack.org/74/74356fc3cc9e46597b8ae619ac0d9b8e8a32378b so I do't think that branch is running publsih jobs | 18:48 |
| clarkb | compare to http://logs.openstack.org/f3/f3194ce547be2a11cb0da2be17abdbfb9134ef76/post/ for tip of nova master | 18:48 |
| notmyname | fungi: clarkb: thanks. and thanks for chekcin | 18:48 |
| cmurphy | clarkb: fungi okay so there is a puppet_hiera_datadir set to /opt/system-config in hiera.yaml right? for some reason that's different from the hieradata variable which is being set to /etc/puppet/hieradata | 18:48 |
| clarkb | :datadir: "/opt/system-config/" | 18:50 |
| clarkb | cmurphy: I think we have that because we use "public" hiera in the system-config tree | 18:51 |
| clarkb | cmurphy: does that need to be /opt/system-config/production? | 18:51 |
| notmyname | how long are the docs job results kept? | 18:51 |
| clarkb | notmyname: forever | 18:51 |
| notmyname | oh? | 18:51 |
| notmyname | does the link work forever? | 18:52 |
| clarkb | notmyname: I mean in as much as we do our best. There was a decision made to stop deleting old docs semi recently | 18:52 |
| notmyname | cool | 18:52 |
| notmyname | thanks | 18:52 |
| cmurphy | clarkb: no it should be fine without /production since the hierarchy specifies the environment | 18:52 |
| clarkb | cmurphy: ya comparing to etherpad.o.o its the same | 18:52 |
| *** dpawlik has joined #openstack-infra | 18:53 | |
| fungi | notmyname: assuming you mean published documentation at docs.o.o, and not the draft renderings at logs.o.o? | 18:53 |
| notmyname | no, the draft ones at logs.o.o | 18:53 |
| fungi | the latter are retained for roughly a month, i think | 18:53 |
| clarkb | oh sorry I misread | 18:53 |
| fungi | we stopped retiring and removing old documentation, but draft documentation is still only kept for a limited time | 18:54 |
| fungi | its removal is now in sync with the time when we expire other job logs | 18:54 |
| fungi | which is roughly a month at the moment | 18:54 |
| clarkb | cmurphy: do you think it would help to rerun it with debug set to true? | 18:54 |
| *** sree has joined #openstack-infra | 18:55 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-openstackci master: Revert "Reduce log archive to 30 days" https://review.openstack.org/564313 | 18:55 |
| cmurphy | clarkb: probably wouldn't hurt | 18:55 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix race in test_hold_expiration_no_default https://review.openstack.org/564314 | 18:55 |
| clarkb | ok let me figureo ut how to do that | 18:55 |
| pabelanger | clarkb: fungi: speaking of logs ^ we should be able to support 45 days again on logs.o.o | 18:55 |
| pabelanger | currently at 50% capacity | 18:56 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 18:56 |
| fungi | good call | 18:56 |
| *** esarault2 has joined #openstack-infra | 18:56 | |
| *** dpawlik has quit IRC | 18:57 | |
| clarkb | fungi: I'm going to edit the adhoc playbook to set debug to true | 18:59 |
| clarkb | fungi: ^ any concerns with doing that? | 18:59 |
| *** esarault2 is now known as esarault_ | 18:59 | |
| fungi | nope, sounds fine | 18:59 |
| *** sree has quit IRC | 18:59 | |
| fungi | let's see what it tells us | 18:59 |
| *** esarault has quit IRC | 18:59 | |
| *** esarault has joined #openstack-infra | 19:00 | |
| *** bobh has joined #openstack-infra | 19:00 | |
| clarkb | rerunning now | 19:00 |
| *** Swimingly has quit IRC | 19:01 | |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/irc-meetings master: neutron: remove jlibosva from chairing Neutron meetings https://review.openstack.org/564316 | 19:01 |
| clarkb | that didn't work debug=False still in the logs | 19:01 |
| clarkb | maybe its true not True because yaml? | 19:01 |
| clarkb | I'll try that a gain | 19:01 |
| *** Swimingly has joined #openstack-infra | 19:01 | |
| *** Swimingly has quit IRC | 19:01 | |
| *** Swimingly has joined #openstack-infra | 19:01 | |
| clarkb | oh its actually puppet_debug gets changed to debug internally gah | 19:02 |
| clarkb | rerunning now | 19:02 |
| *** ssbarnea_ has joined #openstack-infra | 19:03 | |
| *** myoung is now known as myoung|dr | 19:03 | |
| clarkb | Looking for data source production/hiera/fqdn/etherpad-dev.openstack.org | 19:03 |
| *** esarault_ has quit IRC | 19:04 | |
| clarkb | fungi: cmurphy ^ from syslog on etherpad-dev | 19:04 |
| clarkb | I think maybe it isn't rooting it? | 19:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools master: handle reno 2.9.0 anchor references https://review.openstack.org/564317 | 19:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools master: stop if announce.sh fails to build the virtualenv https://review.openstack.org/564318 | 19:04 |
| clarkb | Cannot find datafile /opt/system-config/production/hiera/fqdn/etherpad-dev.openstack.org.yaml <- it is there though | 19:04 |
| clarkb | /etc/puppet/hieradata/production/fqdn is the path no additional hiera in there | 19:05 |
| clarkb | I think that may be the issue | 19:05 |
| clarkb | oh that may be a public vs private hiera thing | 19:06 |
| * clarkb keeps looing | 19:06 | |
| clarkb | it seems to only be looking it up relative to /opt/system-config | 19:07 |
| fungi | clarkb: the extra hiera | 19:07 |
| fungi | yeah, caught up now | 19:07 |
| *** jchhatba_ has quit IRC | 19:08 | |
| fungi | okay, so that's just for the public hiera file | 19:08 |
| clarkb | what it appears to be oding is looking it up with and without the hiera which si correct based on our hierarchy | 19:08 |
| clarkb | but it is apply that only relative to the datadir which is not where the private hierdata lives | 19:08 |
| clarkb | how did this work at all before? | 19:09 |
| cmurphy | i was just typing that | 19:09 |
| *** bhavik1 has joined #openstack-infra | 19:09 | |
| clarkb | https://review.openstack.org/#/c/562068/21/library/puppet_get_hiera_file_list related maybe? | 19:09 |
| cmurphy | in hiera.yaml.j2 it seems like it is only ever pointed at /opt/system-config | 19:10 |
| clarkb | I think the full_path in the puppet_ger_hiera_file_list may have been the magic | 19:10 |
| cmurphy | oh i think i see it | 19:11 |
| clarkb | hrm no that seems only sued for copying files | 19:11 |
| *** SamYaple has joined #openstack-infra | 19:11 | |
| cmurphy | hrm maybe not | 19:11 |
| cmurphy | that task delegates to localhost but it's using the variable {{ hieradata }} as the location input when it should be using mgmt_hieradata | 19:11 |
| cmurphy | but right now that should be the same so never mind | 19:12 |
| *** imacdonn has quit IRC | 19:14 | |
| *** yamamoto has joined #openstack-infra | 19:14 | |
| *** imacdonn has joined #openstack-infra | 19:14 | |
| clarkb | 'ensure hieradata manifest link is present' is the magic I think | 19:15 |
| clarkb | and its there /opt/system-config/hieradata is a link to /etc/puppet/hieradata/production | 19:15 |
| clarkb | oh! | 19:15 |
| clarkb | cmurphy: I think that link needs to be environment less so should be /opt/system-config/hierdata to /etc/puppet/hieradata | 19:16 |
| clarkb | Cannot find datafile /opt/system-config/hieradata/production/fqdn/etherpad-dev.openstack.org.yaml, skipping | 19:16 |
| cmurphy | clarkb: i think you're right | 19:17 |
| clarkb | which is correct because the actual path there is /etc/puppet/hieradata/production/production/fqdn/etc | 19:17 |
| clarkb | correct in not finding the things at that path | 19:17 |
| clarkb | cmurphy: should I push a new patchset with a fix for that? | 19:19 |
| cmurphy | clarkb: what's your fix? i think it's gonna need another variable like hieradata_environment and mgmt_hieradata_environment | 19:20 |
| *** yamamoto has quit IRC | 19:20 | |
| clarkb | cmurphy: puppet4 also complicates it | 19:20 |
| cmurphy | yeah i think that link is gonna be weird with the puppet 4 path | 19:20 |
| clarkb | ya we'll have to update our hiera.yaml too | 19:21 |
| cmurphy | if you want we can back this out and i can work out a better solution | 19:21 |
| clarkb | cmurphy: src: "{{ '/etc/puppet/hieradata' if puppet_version == '3' else '/etc/puppetlabs/code/environments' }}" something like that then modify our hiera.yaml to look in environment/hieradata in addition to just environment/ | 19:22 |
| fungi | yeah, if there's no quick way forward we probably want to temporarily revert so that we can continue puppeting our servers in the meantime | 19:22 |
| fungi | but i'm happy to press on if there are ideas on how to get past this | 19:22 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Set path to xstatic version checker https://review.openstack.org/564323 | 19:22 |
| clarkb | that works for me, we can revert, use ansible-galaxy to reapply the old role, then test everything is still happy on etherpad-dev with old role ebfore enabling globally again | 19:23 |
| clarkb | I too am happy to press on. where do we set the hiera.yaml? | 19:23 |
| cmurphy | it's set in config.yml | 19:24 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix race in test_hold_expiration_no_default https://review.openstack.org/564314 | 19:25 |
| cmurphy | i think let's not change hiera.yaml but let's have two different tasks to create the link, one for when: puppet_version == '3' and one for when: puppet_version == '4' | 19:25 |
| openstackgerrit | Clark Boylan proposed openstack-infra/ansible-role-puppet master: Fix hieradata lookup paths https://review.openstack.org/564324 | 19:27 |
| clarkb | cmurphy: ^ is the hiera.yaml option | 19:27 |
| clarkb | cmurphy: want ot write the other? I'm happy to have different tasks that may be simpler | 19:27 |
| clarkb | cmurphy: but I think it needs an updated hiera.yaml either way? | 19:27 |
| *** rkukura has joined #openstack-infra | 19:30 | |
| cmurphy | clarkb: with my way i was thinking we wouldn't have to change hiera.yaml at all but thinking it though it might be impossible to create that link and i like your way better | 19:33 |
| clarkb | cmurphy: ya I think the hieradata/environment becoming environment/hieradata makes this difficult | 19:34 |
| clarkb | fungi: do you want to look over https://review.openstack.org/#/c/564324/ and I can make those edits on puppetmaster and rerun? | 19:36 |
| *** lpetrut has quit IRC | 19:36 | |
| cmurphy | i'm looking at https://puppet.com/docs/puppet/4.10/hiera_hierarchy.html and they don't have environment in the path even though it seems to be aware of the environment | 19:36 |
| *** Goneri has joined #openstack-infra | 19:36 | |
| fungi | clarkb: sure, looking now | 19:37 |
| cmurphy | hrm it's for hiera version 5 and we have hiera version 3 that migration will probably make things complicated | 19:37 |
| openstackgerrit | Merged openstack-infra/release-tools master: handle reno 2.9.0 anchor references https://review.openstack.org/564317 | 19:37 |
| clarkb | cmurphy: the datadir appears to be relative to the environment in that case | 19:39 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: update the announce-release job to look in releases repo for tools https://review.openstack.org/564329 | 19:41 |
| fungi | clarkb: yeah, 564324 looks reasonable (i trust you and cmurphy wrt having researched what that new path actually is) | 19:42 |
| *** eernst has joined #openstack-infra | 19:42 | |
| clarkb | cmurphy: do you think explicit environment like that will work for puppet 4? | 19:43 |
| clarkb | cmurphy: fungi if the setup there isn't expect to work with puppet4 maybe we hold off since that was the entire point? | 19:43 |
| cmurphy | clarkb: yeah i am not positive it will work the same | 19:44 |
| clarkb | each environment has its own hiera.yaml now too :/ | 19:44 |
| fungi | which actually probably makes things more flexible but by default also means more symlinks i guess? | 19:45 |
| clarkb | fungi: I think more configs :/ | 19:46 |
| clarkb | we'll need a separate hiera.yaml for puppet4 | 19:46 |
| fungi | oh, we couldn't just symlink alternative environment hiera.yaml files to the one for the production environment? | 19:46 |
| *** eernst has quit IRC | 19:46 | |
| clarkb | fungi: no because the pathing has completely changed | 19:47 |
| clarkb | fungi: in puppet3 it appears to all be rooted, in puppet 4 its all relative to your environment and that is built in | 19:47 |
| clarkb | this also complicates the we want to have puppet3 and puppet4 at the same time cohabitating problem | 19:47 |
| *** sree has joined #openstack-infra | 19:48 | |
| cmurphy | I *think* using our current hiera setup with puppet 4 will still work and we can create symlinks similarly to what we do now but I haven't tested it thoroughly | 19:49 |
| cmurphy | I do know we can still use hiera version 3 with puppet 4 it just gives you deprecation warnings | 19:49 |
| clarkb | cmurphy: do you want to move forward in that case and continue to try getting this to work with puppet3? | 19:49 |
| cmurphy | clarkb: sure, i do at least feel confident that 564324 will fix it for puppet 3 | 19:50 |
| clarkb | ok I'll go ahead and manually apply that then give it another go | 19:51 |
| *** sree has quit IRC | 19:53 | |
| *** esarault_ has joined #openstack-infra | 19:54 | |
| *** dklyle has joined #openstack-infra | 19:54 | |
| clarkb | fungi: ok I'm ready if you are | 19:54 |
| *** dhill_ has quit IRC | 19:55 | |
| *** bobh has quit IRC | 19:55 | |
| *** dhajare has quit IRC | 19:56 | |
| clarkb | if this run isn't happy I need to take a break for lunch | 19:56 |
| fungi | yep, ready | 19:57 |
| fungi | sorry, split three ways between some election officiating stuff, the storyboard meeting and here | 19:57 |
| fungi | which is why i keep lagging on responses | 19:57 |
| *** esarault has quit IRC | 19:57 | |
| clarkb | ok here goes | 19:58 |
| *** dhill_ has joined #openstack-infra | 19:58 | |
| *** hemna_ has quit IRC | 19:58 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: update the announce-release job to look in releases repo for tools https://review.openstack.org/564329 | 19:58 |
| *** eernst has joined #openstack-infra | 19:59 | |
| clarkb | ok it says that worked | 19:59 |
| cmurphy | \o/ | 19:59 |
| *** wolverineav has quit IRC | 20:00 | |
| clarkb | infra-root can we get reviews on https://review.openstack.org/#/c/564324/1 and its parent? with those in I can manually update the ansible role using ansible-galaxy again then rerun just to double check then I guess we open the floodgates | 20:00 |
| *** wolverineav has joined #openstack-infra | 20:00 | |
| fungi | yeah, looks good! | 20:01 |
| *** esarault_ has quit IRC | 20:01 | |
| clarkb | it does say changed localhost on the puppetmaster | 20:01 |
| clarkb | so now I want to see what changed | 20:01 |
| clarkb | the debug logs make it kinda noisy | 20:02 |
| clarkb | nothing is jumping out at me, I'm going to remove the debug and rerun as is just to see if that helps | 20:04 |
| clarkb | oh it updated the remote adhoc role to not have debug in it so maybe that is what changed | 20:04 |
| corvus | clarkb: i feel like two tasks and a 'when' might be more ansibly, but that lgtm. | 20:05 |
| *** wolverineav has quit IRC | 20:05 | |
| corvus | (that lets you have task names like "ensure hieradata manifest link is present for puppet 3") | 20:05 |
| clarkb | corvus: good point | 20:06 |
| clarkb | I'm rerunning now fwiw | 20:06 |
| *** dklyle has quit IRC | 20:06 | |
| corvus | clarkb: +2 both of those, +W at will | 20:07 |
| clarkb | well that didn't work | 20:07 |
| clarkb | so rerunning it went back to broken puppet /me is looking at why | 20:08 |
| clarkb | hiera problems again | 20:08 |
| cmurphy | :( | 20:08 |
| *** iranzo has joined #openstack-infra | 20:09 | |
| clarkb | I think it reverted the ansible module | 20:09 |
| clarkb | which I did not expect, does puppet run ansible-galaxy? | 20:09 |
| *** bhavik1 has quit IRC | 20:10 | |
| clarkb | ya it toally did that so I think we approve these two changes, manually update via ansible-galaxy, then rerun twice | 20:10 |
| clarkb | and make sure it steady states | 20:10 |
| clarkb | then cleanup hieradataproduction again | 20:10 |
| fungi | sounds good to me | 20:11 |
| clarkb | it totally reverted the role contents, I don't know via what mechanism | 20:11 |
| cmurphy | puppet is running ansible-galaxy? i thought only the update_puppet.yaml playbook did that | 20:11 |
| clarkb | cmurphy: maybe? it definitely undid my changes to the ansible puppet role when it ran puppet | 20:12 |
| *** edmondsw has quit IRC | 20:12 | |
| clarkb | cmurphy: or at least I think that is how it happened | 20:12 |
| *** e0ne has quit IRC | 20:13 | |
| *** edmondsw has joined #openstack-infra | 20:15 | |
| clarkb | Apr 25 20:01:58 puppetmaster ansible-command: Invoked with warn=True executable=None _uses_shell=True _raw_params=ansible-galaxy install --force -r roles.yaml removes=None creates=None chdir=/opt/system-config/production | 20:15 |
| *** yamamoto has joined #openstack-infra | 20:15 | |
| clarkb | that is what did it | 20:15 |
| clarkb | I don't know why that ran though | 20:17 |
| clarkb | oh cron is reenabled? | 20:17 |
| clarkb | ugh | 20:17 |
| clarkb | infra-root ^ ftyi | 20:17 |
| clarkb | ok ansible is not running right now | 20:18 |
| clarkb | I've re disabled it | 20:18 |
| clarkb | do we have a better way of doing this? | 20:18 |
| clarkb | and did everything just break? | 20:19 |
| *** iranzo has quit IRC | 20:19 | |
| fungi | how did cron get reenabled? | 20:19 |
| fungi | did we somehow rerun puppet against the puppetmaster? | 20:19 |
| clarkb | fungi: because kick.sh modifies localhost | 20:19 |
| fungi | er, really? | 20:19 |
| clarkb | yup | 20:19 |
| *** iranzo has joined #openstack-infra | 20:19 | |
| clarkb | ok looking at puppet log most things failed. elasticsearch and mirror nodes did not | 20:20 |
| *** edmondsw has quit IRC | 20:20 | |
| fungi | i don't recall kick.sh undoing commented-out cronjobs on puppetmaster but maybe i've just not been paying attention | 20:20 |
| cmurphy | oof :( | 20:20 |
| fungi | clarkb: presumably elasticsearch and mirror nodes don't rely on hiera? | 20:20 |
| *** e0ne has joined #openstack-infra | 20:21 | |
| fungi | that would at least explain why they didn't fail | 20:21 |
| *** jcoufal has quit IRC | 20:21 | |
| *** yamamoto has quit IRC | 20:21 | |
| clarkb | fungi: ya, I think my biggest concern right now is anywwhere we have hiera defaults that won't work | 20:21 |
| clarkb | but will allow puppet to apply cleanly | 20:21 |
| *** tjgresha has joined #openstack-infra | 20:22 | |
| *** dhill_ has quit IRC | 20:22 | |
| clarkb | git.o.o backends failed so review.o.o wasn't touched | 20:23 |
| clarkb | I think largely things failed in a way that prevented puppet from doing anything bad | 20:23 |
| clarkb | but we'll have to cleanup a lot of hieradataproductions | 20:23 |
| fungi | at least that's easy to blast via a single ansible command | 20:23 |
| clarkb | afs nodes did not fail | 20:24 |
| *** puiterwijk is now known as puiterwijk39583 | 20:24 | |
| clarkb | but http://mirror.dfw.rax.openstack.org/ appears to still exist so Ithink we are ok? | 20:24 |
| fungi | hopefully whatever misconfiguration got applied via hiera defaults will get unrolled with the fixed role | 20:24 |
| clarkb | ya I guess thats the next step? | 20:24 |
| clarkb | my spot checking hasn't revealed any sad services | 20:25 |
| *** rosmaita has joined #openstack-infra | 20:25 | |
| *** diablo_rojo has joined #openstack-infra | 20:25 | |
| *** puiterwijk39583 is now known as x9e92puiterwijk | 20:26 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Don't run against localhost by default with kick.sh https://review.openstack.org/564338 | 20:27 |
| *** edmondsw has joined #openstack-infra | 20:28 | |
| *** x9e92puiterwijk is now known as puiterwijk | 20:32 | |
| fungi | yeah, i've so far spotted nothing out of the ordinary | 20:32 |
| fungi | i think we got off easy (for a change) | 20:32 |
| *** therve has quit IRC | 20:32 | |
| clarkb | ya I think puppet bails really early in this case | 20:32 |
| clarkb | so unless we have a hiera data'd config with a default that would take the place of private data we have been fine | 20:33 |
| *** edmondsw has quit IRC | 20:33 | |
| fungi | i agree | 20:33 |
| *** e0ne has quit IRC | 20:33 | |
| *** dhill_ has joined #openstack-infra | 20:33 | |
| clarkb | I think we wait for fixes to merge, retest it without updating localhost | 20:33 |
| clarkb | run it a coupel times to make sure it steady states then open the floodgates? | 20:33 |
| clarkb | that shoudl correct the symlink and hierdata situation except for deleting the bad hierdata dir | 20:34 |
| *** iranzo has quit IRC | 20:34 | |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Add some missing /'s and +'s to fix hieradata pathing https://review.openstack.org/564305 | 20:34 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet master: Fix hieradata lookup paths https://review.openstack.org/564324 | 20:34 |
| *** edmondsw has joined #openstack-infra | 20:34 | |
| clarkb | ok I'm updating with ansible-galaxy command now | 20:34 |
| *** eharney has quit IRC | 20:34 | |
| clarkb | fungi: using a different command to run ansible puppet now, can you double check this one should be safer? | 20:35 |
| fungi | i guess that's reverse-engineered from kick.sh? | 20:36 |
| clarkb | ya I cat'd it just above | 20:36 |
| clarkb | and dropped the :localhost | 20:37 |
| fungi | yeah, seems less likely to have side effects | 20:37 |
| clarkb | shall I run it? | 20:37 |
| fungi | go for it | 20:37 |
| *** hemna_ has joined #openstack-infra | 20:37 | |
| *** therve has joined #openstack-infra | 20:38 | |
| clarkb | ok that puppet run looked good | 20:38 |
| clarkb | I'm going to rerun now to see if it steady states | 20:38 |
| *** edmondsw has quit IRC | 20:39 | |
| *** felipemonteiro_ has quit IRC | 20:39 | |
| *** ediardo has joined #openstack-infra | 20:40 | |
| *** tesseract has quit IRC | 20:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:40 | |
| clarkb | fungi: what do you think? we open it to everything now? | 20:40 |
| clarkb | I really need to find lunch soon | 20:40 |
| fungi | yeah, let's get this into production | 20:40 |
| clarkb | ok I'll edit the crontab now | 20:40 |
| fungi | you should be able to go eat | 20:40 |
| *** myoung|dr is now known as myoung | 20:41 | |
| clarkb | fungi: you'll watch the puppet run all log? | 20:41 |
| clarkb | I've uncommented it so it should run in ~4 minutes | 20:41 |
| fungi | yep, i'm tailing it now | 20:42 |
| fungi | will keep an eye on it | 20:42 |
| *** jcoufal has joined #openstack-infra | 20:42 | |
| clarkb | cool I'll keep my eye on irc but there is leftover burger with my name on it | 20:42 |
| *** pcaruana has quit IRC | 20:42 | |
| fungi | i'll yell if i spot anything obviously broken | 20:43 |
| cmurphy | thanks clarkb and fungi | 20:44 |
| *** agopi|zzz has quit IRC | 20:45 | |
| fungi | thanks to you cmurphy! you did the work on this | 20:45 |
| fungi | okay, it's kicking off in the log now | 20:45 |
| fungi | fingers crossed | 20:45 |
| fungi | and all other superstitious gestures engaged | 20:45 |
| fungi | hrm, the initial run puppet task failed | 20:47 |
| fungi | fatal: [localhost]: FAILED! => {"changed": false, "disabled": false, "error": true, "failed": true, "msg": "puppet did not run", "rc": 1, "stderr": "", "stdout": "", "stdout_lines": []} | 20:47 |
| fungi | i guess that's against localhost? | 20:47 |
| corvus | fungi: yep | 20:48 |
| fungi | the remote tasks seem to be working so far | 20:48 |
| corvus | that should mean puppet exited with code 1 | 20:49 |
| fungi | i'll peer around in syslog on puppetmaster for details i guess | 20:49 |
| corvus | fungi: could it be: Apr 25 20:46:58 puppetmaster puppet-user[24505]: Could not find data item puppetmaster_root_rsa_key in any Hiera data file and no default supplied at /opt/system-config/production/manifests/site.pp:221 on node puppetmaster.openstack.org | 20:49 |
| fungi | yeah, that looks like it | 20:50 |
| fungi | and lines up with the hiera challenges we've noted as part of this | 20:50 |
| cmurphy | so the ansible role needs to be updated on puppetmaster? | 20:51 |
| clarkb | hieradata -> /etc/puppet/hieradataproduction still bad symlink on puppetmaster | 20:51 |
| fungi | so eventually consistent, or a deadlock we need to resolve to get it applying correctly? | 20:51 |
| clarkb | the role looks up todate | 20:51 |
| *** tjgresha has quit IRC | 20:51 | |
| clarkb | ya I'm wondering if it will correct itself or if we have to manually fix that everywhere? | 20:51 |
| clarkb | we didn't have to fix it on etherpad-dev at least | 20:52 |
| *** agopi|zzz has joined #openstack-infra | 20:52 | |
| fungi | i'm inclined to see if it recurs | 20:52 |
| fungi | and then switch to celebratory wednesday evening drinking if it does not | 20:52 |
| clarkb | symlink is at /opt/system-config/hieradata | 20:52 |
| clarkb | (if we have to manually correct it) | 20:53 |
| clarkb | oh wait | 20:53 |
| clarkb | how does this work with copying the files? | 20:53 |
| clarkb | is it just going to nop because nothing to copy? | 20:53 |
| *** owalsh_afk is now known as owalsh | 20:53 | |
| clarkb | I guess that is what it must've been doing? | 20:53 |
| *** tjgresha has joined #openstack-infra | 20:53 | |
| fungi | seems like it | 20:54 |
| fungi | and the servers already have their existing copies | 20:54 |
| *** trown is now known as trown|outtypewww | 20:54 | |
| *** edmondsw has joined #openstack-infra | 20:54 | |
| clarkb | ok that link is only made when copy_hieradata is set to true. The default in the role is to not copy hieradata | 20:55 |
| fungi | ahh | 20:55 |
| clarkb | I'm wondering if run_all.sh sets copy_hierdata to false for localhost (so we don't update the link) but kick.sh doesn't so we set it in the first place | 20:55 |
| clarkb | playbooks/remote_puppet_git.yaml ya that sets it to false for localhost | 20:57 |
| clarkb | fungi: so I think we likely have to fix this ourselves? | 20:57 |
| fungi | welp, easy enough | 20:58 |
| clarkb | fungi: that does make me wonder though, do we use a different hieradata setup by default on the puppetmaster? | 20:58 |
| clarkb | I guess all the usage of kick.sh likely makes that moot at this point though | 20:58 |
| fungi | just replace /opt/system-config/hieradata with a symlink to /etc/puppet/hieradata ? | 20:58 |
| clarkb | like even if we were trying to do somethign different after the first run of kick.sh we'd override it | 20:58 |
| clarkb | hieradata -> /etc/puppet/hieradata is what etherpad-dev has | 20:59 |
| clarkb | we might want to understand how this is supposed to work on the puppet master in the first place though | 20:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: ignore the series_status.yaml when processing release requests https://review.openstack.org/564342 | 20:59 |
| dhellmann | smcginnis : ^^ | 20:59 |
| *** edmondsw has quit IRC | 20:59 | |
| clarkb | fungi: puppetmaster has the same hiera.yaml so ya I think we just update the symlink? | 20:59 |
| clarkb | and its just going to avoid nop copying things | 21:00 |
| clarkb | ok back to making burger | 21:00 |
| *** edmondsw has joined #openstack-infra | 21:00 | |
| fungi | oh, maybe /opt/system-config/hieradata is supposed to be a symlink to /etc/puppet/hieradata/production | 21:02 |
| clarkb | fungi: no I think that was part of the problem before | 21:02 |
| clarkb | fungi: because our hiera.yaml explicitly lists the evnironment in it we need to symlink above the envs | 21:02 |
| fungi | aha, okay, so /opt/system-config/hieradata is supposed to be a symlink to /etc/puppet/hieradata then like i originally though | 21:03 |
| fungi | t | 21:03 |
| clarkb | ya I think so | 21:03 |
| fungi | command is queued up in that root screen session | 21:03 |
| fungi | if you're in a place where you can see it | 21:04 |
| fungi | rm /opt/system-config/hieradata;ln -s /etc/puppet/hieradata /opt/system-config/hieradata | 21:04 |
| clarkb | ya that looks right | 21:04 |
| *** edmondsw has quit IRC | 21:05 | |
| fungi | done | 21:05 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Enable breaks in the markdown parser to stop mangling pasted logs https://review.openstack.org/548342 | 21:05 |
| fungi | watching for the next run at 21:15z to make sure it's sane | 21:05 |
| *** edmondsw has joined #openstack-infra | 21:06 | |
| *** tjgresha has quit IRC | 21:08 | |
| *** edmondsw has quit IRC | 21:11 | |
| *** edmondsw has joined #openstack-infra | 21:12 | |
| *** ldnunes has quit IRC | 21:17 | |
| *** edmondsw has quit IRC | 21:17 | |
| cmurphy | fungi: how's it looking? | 21:17 |
| *** yamamoto has joined #openstack-infra | 21:17 | |
| *** edmondsw has joined #openstack-infra | 21:18 | |
| fungi | cmurphy: seems not to have failed this time | 21:20 |
| *** openstackgerrit has quit IRC | 21:20 | |
| cmurphy | \o/ | 21:20 |
| clarkb | cool I guess that leaves removing the bad hieradataproduction dirs | 21:20 |
| clarkb | also cleaning up the crontab, looks like puppet added new entries instead if reenabling the old ones | 21:21 |
| *** edmondsw has quit IRC | 21:23 | |
| *** esberglu has quit IRC | 21:23 | |
| fungi | it does that, yes | 21:23 |
| *** yamamoto has quit IRC | 21:23 | |
| *** edmondsw has joined #openstack-infra | 21:24 | |
| *** jtomasek has quit IRC | 21:28 | |
| *** edmondsw has quit IRC | 21:28 | |
| *** aeng has joined #openstack-infra | 21:29 | |
| *** edmondsw has joined #openstack-infra | 21:30 | |
| *** esberglu has joined #openstack-infra | 21:33 | |
| *** edmondsw has quit IRC | 21:35 | |
| *** agopi|zzz has quit IRC | 21:35 | |
| *** dave-mcc_ has quit IRC | 21:36 | |
| *** edmondsw has joined #openstack-infra | 21:36 | |
| *** esberglu has quit IRC | 21:38 | |
| *** jcoufal has quit IRC | 21:40 | |
| *** edmondsw has quit IRC | 21:41 | |
| clarkb | I've cleaned up the crontab | 21:45 |
| clarkb | infra-root something like `sudo ansible -f 10 'etherpad-dev.openstack.org' -m shell -a 'rm -r /etc/puppet/hieradataproduction'` to cleanup the bad hierdata dirs | 21:47 |
| clarkb | double check me on ^ | 21:48 |
| fungi | looks right | 21:49 |
| fungi | and then switch to '*' for the host if that dtrt | 21:49 |
| clarkb | we'll want to run it against * eventually but starting small by checking it on etherpad-dev | 21:49 |
| clarkb | ya | 21:49 |
| clarkb | ok I'll give that a go on etherpad-dev | 21:50 |
| *** rkukura has quit IRC | 21:50 | |
| corvus | ++ | 21:50 |
| *** edmondsw has joined #openstack-infra | 21:50 | |
| clarkb | seemed to do what I wanted on etherpad-dev | 21:51 |
| clarkb | fungi: corvus want to do anymore testing before I change it to sudo ansible -f 10 '*' ... ? | 21:51 |
| fungi | nope, have at it | 21:52 |
| clarkb | its running in the same root screen | 21:53 |
| clarkb | heh its into the infraclouds and timing out on ssh | 21:54 |
| clarkb | I think cmurphy had a change up to clean out infracloud, I guess we should do thatsooner than later | 21:54 |
| *** ssbarnea_ has quit IRC | 21:55 | |
| cmurphy | https://review.openstack.org/#/c/563634/ | 21:55 |
| *** edmondsw has quit IRC | 21:55 | |
| clarkb | do we play taps? | 21:56 |
| corvus | brr brr brr..... brr brr brr... brr brr brr brr brr brr brr brr brr... brr brr brr... brr brr brr... brr brr brrrrrrr | 21:56 |
| fungi | a proper tapout | 21:57 |
| fungi | approved | 21:58 |
| corvus | that makes me sad, but the removal of the line 'SOURCE_MODULES["$OPENSTACK_GIT_ROOT/openstack/puppet-nova"]="mitaka-eol"' can only be a good thing | 21:58 |
| fungi | farewell ancient and now offline/repurposed servers! | 21:59 |
| corvus | (also, i like that it makes it really clear what we were doing there. it's right there in the name) | 21:59 |
| clarkb | corvus: the good news is that while the puppet modules themselves may have been eol they used distro packages to install all the services so the distro was keeping the dream alive for us | 22:00 |
| *** rh-jelabarre has quit IRC | 22:01 | |
| clarkb | looks like we still have dns records too. I'll work on cleaning those up now | 22:01 |
| fungi | we should be able to just delete the whole ic.openstack.org zone | 22:01 |
| *** hasharAway has quit IRC | 22:02 | |
| *** edmondsw has joined #openstack-infra | 22:02 | |
| clarkb | fungi: I'm assumign I should delete the delegation for that first? /me looks how that is done in the rax dashboard | 22:03 |
| fungi | there likely isn't any separate delegation | 22:04 |
| *** Goneri has quit IRC | 22:04 | |
| fungi | the same nameservers are already authoritative for the parent zone | 22:04 |
| clarkb | ya there isn't | 22:04 |
| clarkb | at least not seeing any | 22:04 |
| clarkb | now double checking the records in ic.openstack.org to make sure we don't need to keep any | 22:04 |
| *** VW_ has joined #openstack-infra | 22:05 | |
| clarkb | ic.openstack.org has A records for baremetal00, controller nodes, and compute hosts and NS records for itself | 22:06 |
| clarkb | infra-root should I just go ahead and delete domain via the dashboard on ic.openstack.org? | 22:06 |
| fungi | i would, yes | 22:06 |
| *** edmondsw has quit IRC | 22:07 | |
| *** VW has quit IRC | 22:08 | |
| clarkb | "Successfully deleted domain ic.openstack.org" | 22:09 |
| *** VW_ has quit IRC | 22:09 | |
| clarkb | #status log ic.openstack.org domain deleted from dns management as part of infracloud cleanup" | 22:09 |
| openstackstatus | clarkb: finished logging | 22:09 |
| clarkb | #status log ansible-role-puppet updated with new support for Puppet 4 (backward compatible with puppet 3) | 22:10 |
| openstackstatus | clarkb: finished logging | 22:10 |
| * fungi lowers infra flag to half mast | 22:10 | |
| *** rlandy has quit IRC | 22:11 | |
| *** mriedem has quit IRC | 22:11 | |
| clarkb | next on the list, nodepool restarts? | 22:12 |
| *** isviridov_away has quit IRC | 22:13 | |
| *** vaidy has quit IRC | 22:13 | |
| clarkb | corvus: Shrews pabelanger for nodepool restarts I'm assuming we want to restart the launchers and builders and do all of them? | 22:13 |
| clarkb | Probably easiest to do the builders first then do the launchers? | 22:14 |
| * clarkb goes to vote | 22:14 | |
| *** rpioso is now known as rpioso|afk | 22:14 | |
| fungi | sounds good (then we can be more comfortable with tagging a new nodepool release) but i likely won't be much help as i'm winding down for the evening | 22:14 |
| *** dave-mccowan has joined #openstack-infra | 22:14 | |
| clarkb | ya I'm feelign wron out on the creating posible emergencies front. Maybe I do other things for the rest of the day and look at nodepool in the morning | 22:14 |
| fungi | also sounds fine | 22:15 |
| *** openstackgerrit has joined #openstack-infra | 22:15 | |
| openstackgerrit | Merged openstack-infra/system-config master: Stop managing infracloud things https://review.openstack.org/563634 | 22:15 |
| clarkb | shrews might want to be around for that too and its past his EOD now | 22:16 |
| fungi | yep, he and i share the same regional happy hour | 22:16 |
| clarkb | hrm its a bit early for me to claim it is happy hour here | 22:17 |
| clarkb | I did have a popsicle once puppet was happy again though | 22:17 |
| corvus | clarkb: yeah, i agree we should restart all components. the changes to builders are small, i think, but there are some | 22:17 |
| *** slaweq has quit IRC | 22:17 | |
| fungi | that sounds like a legitimate means of celebrating | 22:17 |
| corvus | (for when we're ready) | 22:17 |
| *** threestrands has joined #openstack-infra | 22:17 | |
| *** slaweq has joined #openstack-infra | 22:18 | |
| clarkb | we get ~3.5 days of proper warm weather then back to rain friday | 22:18 |
| clarkb | corvus: fungi https://review.openstack.org/#/c/563752/ is a fix for xstatic release building if you have a moement to review it | 22:19 |
| *** yamamoto has joined #openstack-infra | 22:19 | |
| *** mlavalle has quit IRC | 22:20 | |
| *** slaweq has quit IRC | 22:23 | |
| *** yamamoto has quit IRC | 22:24 | |
| *** vaidy has joined #openstack-infra | 22:29 | |
| *** isviridov_away has joined #openstack-infra | 22:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix xstatic-check-version https://review.openstack.org/563752 | 22:31 |
| *** dave-mccowan has quit IRC | 22:36 | |
| *** verdurin has quit IRC | 22:37 | |
| clarkb | thanks!@ | 22:38 |
| clarkb | still waiting for my ansible command to clean up those dirs to return | 22:40 |
| clarkb | I'm guessing an ssh connection is out to lunch | 22:40 |
| *** hongbin_ has quit IRC | 22:44 | |
| *** bobh has joined #openstack-infra | 22:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: [WIP] Cache configuration objects in addition to YAML dicts https://review.openstack.org/564061 | 22:45 |
| *** kaisers has quit IRC | 22:47 | |
| *** bobh has quit IRC | 22:47 | |
| *** gyee has quit IRC | 22:48 | |
| *** rcernin has joined #openstack-infra | 22:52 | |
| *** tpsilva has quit IRC | 23:00 | |
| *** edmondsw has joined #openstack-infra | 23:01 | |
| *** felipemonteiro_ has quit IRC | 23:03 | |
| *** kaisers has joined #openstack-infra | 23:03 | |
| *** yamamoto has joined #openstack-infra | 23:09 | |
| *** diablo_rojo has quit IRC | 23:10 | |
| *** edmondsw has quit IRC | 23:10 | |
| *** bobh has joined #openstack-infra | 23:17 | |
| *** bobh has quit IRC | 23:18 | |
| *** bobh has joined #openstack-infra | 23:18 | |
| *** caphrim007 has joined #openstack-infra | 23:19 | |
| *** caphrim007_ has joined #openstack-infra | 23:20 | |
| *** bobh has quit IRC | 23:23 | |
| *** caphrim007 has quit IRC | 23:23 | |
| clarkb | I need to pop out soon so I went ahead and ^C'd the ansible command that was running to remove the bad hieradataproduction dirs. I believe it managed to udpate all the servers that were reachable | 23:33 |
| *** threestrands_ has joined #openstack-infra | 23:35 | |
| *** agopi|zzz has joined #openstack-infra | 23:35 | |
| *** threestrands has quit IRC | 23:38 | |
| *** pcichy has quit IRC | 23:46 | |
| *** pcichy has joined #openstack-infra | 23:46 | |
| *** agopi|zzz has quit IRC | 23:46 | |
| *** efried has quit IRC | 23:47 | |
| *** s-shiono has joined #openstack-infra | 23:49 | |
| *** tosky has quit IRC | 23:50 | |
| *** caphrim007_ has quit IRC | 23:54 | |
| *** caphrim007 has joined #openstack-infra | 23:54 | |
| *** dingyichen has joined #openstack-infra | 23:59 | |
| *** dingyichen has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!