| *** EricGonczer_ has quit IRC | 00:01 | |
| *** sdake_ has quit IRC | 00:02 | |
| *** roxanaghe has quit IRC | 00:02 | |
| *** vhosakot has quit IRC | 00:03 | |
| *** asettle has quit IRC | 00:04 | |
| *** elo has joined #openstack-infra | 00:06 | |
| *** sdake has joined #openstack-infra | 00:06 | |
| *** salv-orlando has quit IRC | 00:06 | |
| *** asettle has joined #openstack-infra | 00:06 | |
| *** mylu has quit IRC | 00:07 | |
| sdake | mordred aorund? | 00:07 |
|---|---|---|
| sdake | mordred did you manage to push that fix into infra today? | 00:07 |
| sdake | if so, the gate is failing in the same way | 00:07 |
| *** elo has quit IRC | 00:08 | |
| *** kengo_sakai has quit IRC | 00:09 | |
| *** elo has joined #openstack-infra | 00:10 | |
| *** rbrndt_ has joined #openstack-infra | 00:13 | |
| *** mylu has joined #openstack-infra | 00:13 | |
| *** rbrndt_ has quit IRC | 00:14 | |
| *** Goneri has quit IRC | 00:16 | |
| *** asettle has quit IRC | 00:19 | |
| *** mylu has quit IRC | 00:19 | |
| *** mylu has joined #openstack-infra | 00:21 | |
| *** asettle has joined #openstack-infra | 00:21 | |
| *** nwkarste_ has quit IRC | 00:24 | |
| *** Jae00 has quit IRC | 00:25 | |
| *** hughhalf_ is now known as hughhalf | 00:27 | |
| *** elo has quit IRC | 00:28 | |
| *** elo has joined #openstack-infra | 00:29 | |
| *** ekarlso has quit IRC | 00:32 | |
| *** asettle has quit IRC | 00:32 | |
| *** asettle has joined #openstack-infra | 00:32 | |
| mordred | sdake: nope. I pushed it up for review - wanted to make sure I got clarkb's eyes on it | 00:34 |
| *** Goneri has joined #openstack-infra | 00:35 | |
| *** newjersey has joined #openstack-infra | 00:36 | |
| *** vhosakot has joined #openstack-infra | 00:40 | |
| *** aeng_ has joined #openstack-infra | 00:40 | |
| *** newjersey has joined #openstack-infra | 00:40 | |
| *** aeng has quit IRC | 00:43 | |
| *** mylu has quit IRC | 00:44 | |
| *** ekarlso has joined #openstack-infra | 00:44 | |
| *** mylu has joined #openstack-infra | 00:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:50 | |
| *** thorst has joined #openstack-infra | 00:52 | |
| *** kzaitsev_mb has quit IRC | 00:54 | |
| *** sflanigan has joined #openstack-infra | 00:58 | |
| *** sflanigan has joined #openstack-infra | 00:58 | |
| *** vhosakot has quit IRC | 01:00 | |
| *** thorst has quit IRC | 01:00 | |
| *** zhurong has joined #openstack-infra | 01:00 | |
| *** dkehn has joined #openstack-infra | 01:01 | |
| *** dkehn has quit IRC | 01:01 | |
| *** irdkehn has joined #openstack-infra | 01:02 | |
| *** thorst has joined #openstack-infra | 01:03 | |
| sdake | mordred cool thaanks | 01:03 |
| *** irdkehn is now known as dkehn | 01:05 | |
| *** aeng_ has quit IRC | 01:08 | |
| *** dkehn_ has joined #openstack-infra | 01:08 | |
| *** mnencia has quit IRC | 01:08 | |
| *** mnencia has joined #openstack-infra | 01:09 | |
| *** thorst_ has joined #openstack-infra | 01:10 | |
| *** asettle has quit IRC | 01:13 | |
| *** thorst has quit IRC | 01:13 | |
| *** sdake_ has joined #openstack-infra | 01:14 | |
| *** thorst has joined #openstack-infra | 01:14 | |
| *** thorst_ has quit IRC | 01:16 | |
| *** sdake has quit IRC | 01:18 | |
| *** thorst_ has joined #openstack-infra | 01:19 | |
| sdake_ | suprising how many talks are tagged with kolla that apparently have nothing to do with kolla :) | 01:19 |
| sdake_ | https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=kolla | 01:19 |
| sdake_ | oops wong channel | 01:19 |
| *** thorst has quit IRC | 01:20 | |
| *** aeng_ has joined #openstack-infra | 01:21 | |
| *** thorst_ has quit IRC | 01:22 | |
| *** vhosakot has joined #openstack-infra | 01:22 | |
| *** asettle has joined #openstack-infra | 01:25 | |
| *** tphummel has quit IRC | 01:29 | |
| *** nwkarsten has joined #openstack-infra | 01:36 | |
| *** mriedem has quit IRC | 01:39 | |
| *** nwkarsten has quit IRC | 01:41 | |
| *** thorst has joined #openstack-infra | 01:42 | |
| *** roxanaghe has joined #openstack-infra | 01:46 | |
| *** thorst has quit IRC | 01:47 | |
| *** gongysh has joined #openstack-infra | 01:48 | |
| *** asettle has quit IRC | 01:49 | |
| *** roxanaghe has quit IRC | 01:50 | |
| *** asettle has joined #openstack-infra | 01:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:50 | |
| *** kzaitsev_mb has quit IRC | 01:55 | |
| *** asettle has quit IRC | 02:04 | |
| *** asettle has joined #openstack-infra | 02:04 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 02:06 |
| *** _amrith_ is now known as amrith | 02:06 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 02:08 |
| *** r-mibu has quit IRC | 02:09 | |
| *** mylu has quit IRC | 02:10 | |
| *** nwkarsten has joined #openstack-infra | 02:13 | |
| *** r-mibu has joined #openstack-infra | 02:14 | |
| *** kengo_sakai has joined #openstack-infra | 02:14 | |
| *** Jeffrey4l has joined #openstack-infra | 02:15 | |
| *** nwkarsten has quit IRC | 02:17 | |
| *** alex_xu has quit IRC | 02:18 | |
| *** alex_xu has joined #openstack-infra | 02:22 | |
| *** eric_lopez has joined #openstack-infra | 02:29 | |
| *** elo has quit IRC | 02:30 | |
| *** asettle has quit IRC | 02:33 | |
| *** asettle has joined #openstack-infra | 02:34 | |
| *** eric_lopez has quit IRC | 02:37 | |
| *** nadya has joined #openstack-infra | 02:39 | |
| *** eric_lopez has joined #openstack-infra | 02:40 | |
| *** amrith is now known as _amrith_ | 02:43 | |
| *** nadya has quit IRC | 02:43 | |
| *** zhenguo_ has joined #openstack-infra | 02:47 | |
| *** yuanying_ has quit IRC | 02:51 | |
| *** dimtruck is now known as zz_dimtruck | 02:52 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove Telco Working Group meeting https://review.openstack.org/306577 | 02:54 |
| *** yamamoto_ has joined #openstack-infra | 02:59 | |
| *** BobBall-1wol has joined #openstack-infra | 03:01 | |
| *** BobBall-awol has quit IRC | 03:04 | |
| *** amotoki has joined #openstack-infra | 03:11 | |
| *** yamamoto_ has quit IRC | 03:11 | |
| *** Sukhdev has joined #openstack-infra | 03:14 | |
| *** asettle has quit IRC | 03:20 | |
| *** yamamoto_ has joined #openstack-infra | 03:21 | |
| *** binbincong has quit IRC | 03:25 | |
| *** hdd has joined #openstack-infra | 03:29 | |
| *** newjersey has quit IRC | 03:30 | |
| *** sdake_ has quit IRC | 03:30 | |
| *** hdd has quit IRC | 03:35 | |
| *** binbincong has joined #openstack-infra | 03:38 | |
| *** yamamoto_ has quit IRC | 03:43 | |
| *** yuanying has joined #openstack-infra | 03:46 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:52 | |
| *** piet has joined #openstack-infra | 03:54 | |
| *** kushal has joined #openstack-infra | 03:54 | |
| *** kzaitsev_mb has quit IRC | 03:57 | |
| *** asettle has joined #openstack-infra | 04:01 | |
| AJaeger | dims: will read... | 04:03 |
| openstackgerrit | Merged openstack-infra/project-config: translations: Do not merge untranslated pot files https://review.openstack.org/306837 | 04:07 |
| *** salv-orlando has joined #openstack-infra | 04:09 | |
| *** vhosakot has quit IRC | 04:10 | |
| *** amotoki has quit IRC | 04:11 | |
| *** salv-orlando has quit IRC | 04:13 | |
| openstackgerrit | Merged openstack/requirements: Bump neutron-lib to 0.1.0 https://review.openstack.org/306586 | 04:19 |
| *** amotoki has joined #openstack-infra | 04:22 | |
| *** yamamoto_ has joined #openstack-infra | 04:27 | |
| *** ekarlso has quit IRC | 04:27 | |
| *** sdake has joined #openstack-infra | 04:28 | |
| *** oanson has joined #openstack-infra | 04:29 | |
| *** amotoki has quit IRC | 04:30 | |
| *** amotoki has joined #openstack-infra | 04:38 | |
| *** ekarlso has joined #openstack-infra | 04:40 | |
| *** links has joined #openstack-infra | 04:41 | |
| *** salv-orlando has joined #openstack-infra | 04:48 | |
| *** oanson has quit IRC | 04:50 | |
| *** kengo_sa_ has joined #openstack-infra | 04:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:53 | |
| *** kzaitsev_mb has quit IRC | 04:58 | |
| *** amotoki has quit IRC | 05:01 | |
| *** mylu has joined #openstack-infra | 05:03 | |
| *** flwang1 has quit IRC | 05:04 | |
| *** rcernin has joined #openstack-infra | 05:04 | |
| *** kengo_sa_ has quit IRC | 05:04 | |
| *** arif-ali has joined #openstack-infra | 05:05 | |
| *** amotoki has joined #openstack-infra | 05:09 | |
| *** mylu has quit IRC | 05:11 | |
| *** elo has joined #openstack-infra | 05:11 | |
| *** eric_lopez has quit IRC | 05:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: strip locale archive https://review.openstack.org/304399 | 05:13 |
| *** e0ne has joined #openstack-infra | 05:15 | |
| *** nbloom has joined #openstack-infra | 05:16 | |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Fix script to include repos in PYTHONPATH https://review.openstack.org/306907 | 05:17 |
| *** e0ne has quit IRC | 05:18 | |
| jhesketh | AJaeger: any chance you're around to take a quick look at this simple one https://review.openstack.org/#/c/306907/1 | 05:18 |
| nbloom | Can anyone from infra approve workflow for https://review.openstack.org/#/c/304588/ ? thanks | 05:20 |
| *** abregman has joined #openstack-infra | 05:21 | |
| *** Nikolay_St has quit IRC | 05:21 | |
| *** ramishra has joined #openstack-infra | 05:21 | |
| *** roxanaghe has joined #openstack-infra | 05:22 | |
| *** mylu has joined #openstack-infra | 05:24 | |
| *** roxanaghe has quit IRC | 05:26 | |
| *** ramishra has quit IRC | 05:28 | |
| *** mylu has quit IRC | 05:29 | |
| *** ramishra has joined #openstack-infra | 05:30 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Capitalize first characters in param docs https://review.openstack.org/306908 | 05:30 |
| *** hdd has joined #openstack-infra | 05:32 | |
| *** oanson has joined #openstack-infra | 05:33 | |
| *** mkoderer has quit IRC | 05:36 | |
| *** mkoderer has joined #openstack-infra | 05:39 | |
| *** abregman has quit IRC | 05:39 | |
| *** abregman has joined #openstack-infra | 05:39 | |
| stevemar | jamielennox: lifeless: looks like https://review.openstack.org/#/c/303271/ is stuck | 05:44 |
| stevemar | oh nvm... depends on https://review.openstack.org/#/c/305552/1 | 05:45 |
| *** hrubi_ is now known as hrubi | 05:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Make gate-tempest-dsvm-ironic-pxe_ssh-dib non-voting https://review.openstack.org/306910 | 05:48 |
| *** davideagnello has joined #openstack-infra | 05:49 | |
| *** Jeffrey4l has quit IRC | 05:50 | |
| *** davideagnello has quit IRC | 05:51 | |
| *** ramishra has quit IRC | 05:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 05:54 | |
| *** ramishra has joined #openstack-infra | 05:55 | |
| *** hparekh has joined #openstack-infra | 05:55 | |
| *** infra-red has joined #openstack-infra | 05:56 | |
| hparekh | AJaeger: hi you here ? | 05:58 |
| *** kzaitsev_mb has quit IRC | 05:58 | |
| *** hdd has quit IRC | 06:00 | |
| *** kengo_sa_ has joined #openstack-infra | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/306914 | 06:02 |
| *** kengo_s__ has joined #openstack-infra | 06:03 | |
| *** kengo_sa_ has quit IRC | 06:03 | |
| *** Sukhdev_ has joined #openstack-infra | 06:04 | |
| *** Sukhdev has quit IRC | 06:06 | |
| *** kushal has quit IRC | 06:06 | |
| *** infra-re_ has joined #openstack-infra | 06:07 | |
| *** infra-red has quit IRC | 06:08 | |
| *** fedexo has joined #openstack-infra | 06:11 | |
| ianw | greghaynes: oh, wow, paperwork for releases. great | 06:12 |
| *** elo has quit IRC | 06:15 | |
| *** cody-somerville has quit IRC | 06:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/306798 | 06:21 |
| lifeless | stevemar: thanks for calling attention to that; +2'd. | 06:23 |
| *** jaosorior has joined #openstack-infra | 06:24 | |
| *** Qiming has joined #openstack-infra | 06:26 | |
| *** piet has quit IRC | 06:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/306914 | 06:29 |
| *** scheuran has joined #openstack-infra | 06:30 | |
| *** zeih has joined #openstack-infra | 06:31 | |
| *** jtomasek has joined #openstack-infra | 06:33 | |
| *** eharney has quit IRC | 06:36 | |
| ianw | greghaynes : 1.15.0 will fix the flooding of logs issue in nodepool -> https://review.openstack.org/#/c/306925/ | 06:39 |
| nbloom | Can anyone from infra look at https://review.openstack.org/#/c/304588/ ? thanks | 06:40 |
| *** alex_xu has quit IRC | 06:40 | |
| *** eliqiao has quit IRC | 06:40 | |
| openstackgerrit | Merged openstack/requirements: Bump ironic-lib to 1.3.0 https://review.openstack.org/305553 | 06:41 |
| hparekh | Hi all, my https://review.openstack.org/#/c/306252/ patch got merged two days ago still its change is not reflecting in zuul job. I added check experimental which is working but I have modified the script which need to be run for this job. But in gate previous script is running instead of new script | 06:41 |
| *** alex_xu has joined #openstack-infra | 06:41 | |
| *** eliqiao has joined #openstack-infra | 06:42 | |
| *** eharney has joined #openstack-infra | 06:43 | |
| *** infra-re_ has quit IRC | 06:47 | |
| *** kengo_s__ has quit IRC | 06:48 | |
| *** Nikolay_St has joined #openstack-infra | 06:50 | |
| *** claudiub has joined #openstack-infra | 06:51 | |
| *** ihrachys has joined #openstack-infra | 06:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Translations: Fix releasenotes https://review.openstack.org/306864 | 06:53 |
| *** kzaitsev_mb has joined #openstack-infra | 06:54 | |
| AJaeger | hparekh: Now I'm here... Do you have an example log file? | 06:59 |
| AJaeger | jhesketh: reviewed | 06:59 |
| *** kzaitsev_mb has quit IRC | 06:59 | |
| hparekh | AJaeger: http://logs.openstack.org/11/306911/1/experimental/gate-mistral-docker-buildimage/028e9eb/console.html#_2016-04-18_05_53_44_326 | 06:59 |
| jhesketh | AJaeger: thanks, appreciated :-) | 07:00 |
| hparekh | AJaeger: it should be https://github.com/openstack/mistral/blob/master/docker_image_build.sh#L4 | 07:00 |
| AJaeger | hparekh: The job was run on jenkins05, an admin needs to check whether that change was properly applied. | 07:02 |
| AJaeger | jhesketh: do you have time to check hparekh's problem - and run JJB manually if needed? | 07:03 |
| *** nmagnezi has joined #openstack-infra | 07:03 | |
| * jhesketh looks | 07:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix script to include repos in PYTHONPATH https://review.openstack.org/306907 | 07:06 |
| *** mrunge_ is now known as mrunge | 07:06 | |
| *** ttx_ is now known as ttx | 07:06 | |
| AJaeger | tonyb: you can approve stable/mitaka translation imports again if you want, those should be fine now. | 07:07 |
| *** arxcruz has joined #openstack-infra | 07:08 | |
| *** dizquierdo has joined #openstack-infra | 07:09 | |
| jhesketh | AJaeger, hparekh: puppet runs on the jenkins masters were disabled while other infra-root's were solving problems... I've re-enabled them so the jjb updates should be caught next run | 07:10 |
| jhesketh | I'll keep an eye around in case they don't | 07:10 |
| *** jaosorior has quit IRC | 07:11 | |
| AJaeger | thanks, jhesketh | 07:11 |
| nbloom | Ajaeger: From whom do I need to get additional approval for change https://review.openstack.org/#/c/304588/ to get merged? | 07:12 |
| hparekh | AJaeger, jhesketh Thanks !!! | 07:12 |
| AJaeger | nbloom: from any project-config core. Please have patience, reviews can take a week... | 07:13 |
| *** dalgaaf has joined #openstack-infra | 07:13 | |
| *** dtantsur|afk is now known as dtantsur | 07:14 | |
| *** dtardivel has joined #openstack-infra | 07:14 | |
| nbloom | AJaeger: alright, thank you :) sorry if I seem impatient, first time dealing with infra team.. | 07:14 |
| *** pcaruana has joined #openstack-infra | 07:15 | |
| *** jed56 has joined #openstack-infra | 07:16 | |
| *** e0ne has joined #openstack-infra | 07:16 | |
| *** acabot has joined #openstack-infra | 07:17 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout https://review.openstack.org/293194 | 07:20 |
| tonyb | AJaeger: Thanks. I did see your +1 I just haven't quite made it to that tab yet ;P | 07:20 |
| *** matrohon has joined #openstack-infra | 07:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Translations: Handle non-translated module https://review.openstack.org/306944 | 07:21 |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr: Fix docs typos https://review.openstack.org/306945 | 07:22 |
| AJaeger | hparekh: try another "check experimental" in an hour - that's how long the update should roughly take, if it still fails, please ask again showing the log file. | 07:22 |
| *** infra-red has joined #openstack-infra | 07:22 | |
| hparekh | AJaeger: Ok thanks | 07:23 |
| *** infra-re_ has joined #openstack-infra | 07:23 | |
| *** eezhova has quit IRC | 07:24 | |
| *** fhubik has joined #openstack-infra | 07:24 | |
| openstackgerrit | Ryo Tagami proposed openstack-infra/jenkins-job-builder: Adding support for Jenkins OpenStack Cloud Plugin https://review.openstack.org/306948 | 07:26 |
| *** infra-red has quit IRC | 07:27 | |
| *** vincentll has joined #openstack-infra | 07:28 | |
| AJaeger | jhesketh: could you check https://review.openstack.org/306944 to fix keystone translation proposal, please? | 07:31 |
| *** ifarkas has joined #openstack-infra | 07:32 | |
| *** nbloom has quit IRC | 07:35 | |
| *** jlanoux has joined #openstack-infra | 07:36 | |
| *** dizquierdo has quit IRC | 07:36 | |
| *** odyssey4me_ is now known as odyssey4me | 07:38 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Make gate-cue-python34 voting; add it as gate job https://review.openstack.org/306950 | 07:39 |
| *** lezbar has quit IRC | 07:39 | |
| jhesketh | AJaeger: lgtm, +W | 07:40 |
| AJaeger | thanks, jhesketh | 07:40 |
| *** esikachev has joined #openstack-infra | 07:44 | |
| *** mikelk has joined #openstack-infra | 07:44 | |
| *** Sukhdev_ has quit IRC | 07:48 | |
| *** nadya has joined #openstack-infra | 07:48 | |
| *** rossella_s has joined #openstack-infra | 07:48 | |
| *** alex_xu has quit IRC | 07:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Translations: Handle non-translated module https://review.openstack.org/306944 | 07:50 |
| *** alex_xu has joined #openstack-infra | 07:51 | |
| *** asettle has quit IRC | 07:52 | |
| yolanda | good morning | 07:53 |
| *** vgridnev has joined #openstack-infra | 07:54 | |
| *** jaosorior has joined #openstack-infra | 07:55 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:55 | |
| *** gildub has quit IRC | 07:55 | |
| *** vgridnev has quit IRC | 07:56 | |
| *** jaosorior has quit IRC | 07:56 | |
| *** scheuran has quit IRC | 07:57 | |
| *** HeOS has joined #openstack-infra | 07:57 | |
| AJaeger | morning, yolanda ! | 07:59 |
| *** kzaitsev_mb has quit IRC | 08:00 | |
| *** kengo_sa_ has joined #openstack-infra | 08:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:01 | |
| *** asettle has joined #openstack-infra | 08:02 | |
| *** kzaitsev_mb has quit IRC | 08:06 | |
| *** shardy has joined #openstack-infra | 08:06 | |
| *** asettle has quit IRC | 08:07 | |
| odyssey4me | morning all | 08:10 |
| openstackgerrit | hardik proposed openstack-infra/project-config: Fixed script path in mistral job https://review.openstack.org/306986 | 08:10 |
| *** scheuran has joined #openstack-infra | 08:11 | |
| *** sshnaidm has quit IRC | 08:12 | |
| *** fedexo has quit IRC | 08:12 | |
| *** skolekonov has joined #openstack-infra | 08:12 | |
| *** lezbar has joined #openstack-infra | 08:14 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Fix a bug that ofagent jobs don't work for stable branch https://review.openstack.org/306987 | 08:16 |
| *** hdd has joined #openstack-infra | 08:17 | |
| *** daemontool has quit IRC | 08:17 | |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-gerrit: Order of the classes parameters is refactored https://review.openstack.org/294728 | 08:19 |
| *** jistr has joined #openstack-infra | 08:21 | |
| *** hdd has quit IRC | 08:24 | |
| *** kushal has joined #openstack-infra | 08:26 | |
| *** pfalleno1 has quit IRC | 08:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Make gate-neutron-dsvm-api job non-voting" https://review.openstack.org/303621 | 08:30 |
| *** |-paul-| has joined #openstack-infra | 08:31 | |
| *** dingyichen has quit IRC | 08:36 | |
| openstackgerrit | Dong Ma proposed openstack-infra/subunit2sql: Add db api tests for subunit2sql (1/3) https://review.openstack.org/305030 | 08:37 |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** mpaolino has joined #openstack-infra | 08:41 | |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: email publisher: Clarify format of recipients list in docstring https://review.openstack.org/306992 | 08:43 |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: email_ext publisher: Improve docstring wording https://review.openstack.org/306993 | 08:43 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Move gate-grenade-dsvm-neutron-dvr-multinode into check queue https://review.openstack.org/299844 | 08:45 |
| openstackgerrit | Merged openstack-infra/project-config: Fixed script path in mistral job https://review.openstack.org/306986 | 08:46 |
| *** BobBall-1wol has quit IRC | 08:49 | |
| aspiers | hi all, is it possible to set IRC meeting times based on non-UTC timezone? we want ours to auto-adjust based on daylight savings | 08:49 |
| openstackgerrit | Merged openstack-infra/project-config: Fix a bug that ofagent jobs don't work for stable branch https://review.openstack.org/306987 | 08:51 |
| AJaeger | aspiers: talk with tonyb - I doubt that's possible, since it would give massive conflicts ;) | 08:51 |
| aspiers | AJaeger: why massive conflicts? | 08:51 |
| openstackgerrit | Merged openstack-infra/project-config: Add new project Coupler to openstack repo https://review.openstack.org/302791 | 08:53 |
| AJaeger | aspiers: your meeting would conflict with another one that would not auto-adjust. | 08:53 |
| aspiers | AJaeger: our meeting already conflicts with other meetings, and that doesn't matter | 08:55 |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-tempest-dsvm-ironic-pxe_ssh-dib non-voting https://review.openstack.org/306910 | 08:55 |
| aspiers | AJaeger: the time of our meeting needs to be best for us, not for members of some imaginary intersection of people in unrelated projects | 08:55 |
| aspiers | AJaeger: and we have already decided the time | 08:56 |
| tonyb | aspiers: No they need to be in UTC, but other teams just send an update 2tice a year to move with DST | 08:56 |
| *** claudiub|2 has joined #openstack-infra | 08:56 | |
| AJaeger | aspiers: I'm under the assumption that you use an official meeting channel and thus you would conflict on the *Room* with another meeting. | 08:56 |
| AJaeger | aspiers: Let's get rid of DST! | 08:57 |
| aspiers | tonyb: ok :-/ thanks for the info - please consider this a feature request | 08:57 |
| aspiers | AJaeger: your assumption is wrong | 08:57 |
| aspiers | AJaeger: half wrong, that is. we use #openstack-meeting but it is unused at the new time | 08:58 |
| AJaeger | aspiers: That's the usual assumption for the IRC meetings, if your's is different, better state it ;) | 08:58 |
| *** roxanaghe has joined #openstack-infra | 08:58 | |
| AJaeger | aspiers: so, you're lucky this time. Then send a change and ask tonyb nicely to merge it;) | 08:58 |
| aspiers | but anyway it would make more sense for us to use #openstack-ha | 08:58 |
| aspiers | I don't really understand the need for using an official room | 08:58 |
| openstackgerrit | yolanda.robla proposed openstack/requirements: Update dependency for betamax to 0.6.0 https://review.openstack.org/307000 | 08:58 |
| openstackgerrit | Merged openstack-infra/project-config: Vitrage - Zuul/Jenkins update https://review.openstack.org/304588 | 08:59 |
| AJaeger | aspiers: http://docs.openstack.org/project-team-guide/open-community.html#public-meetings-on-irc | 08:59 |
| *** claudiub has quit IRC | 08:59 | |
| aspiers | AJaeger: I see, thanks | 09:00 |
| bunting | One note is that daylight savings time starts on different dates around the world | 09:01 |
| *** electrofelix has joined #openstack-infra | 09:01 | |
| *** kengo_sa_ has quit IRC | 09:02 | |
| *** yamamoto_ has quit IRC | 09:02 | |
| openstackgerrit | Rob Cresswell proposed openstack/requirements: Add horizon-cisco-ui to projects.txt https://review.openstack.org/300062 | 09:02 |
| openstackgerrit | Merged openstack-infra/project-config: Move gate-grenade-dsvm-neutron-dvr-multinode into check queue https://review.openstack.org/299844 | 09:02 |
| *** yamamoto has joined #openstack-infra | 09:02 | |
| *** roxanaghe has quit IRC | 09:03 | |
| *** tesseract has joined #openstack-infra | 09:03 | |
| *** tesseract is now known as Guest30394 | 09:03 | |
| *** jistr has quit IRC | 09:05 | |
| *** vgridnev has joined #openstack-infra | 09:06 | |
| aspiers | AJaeger: https://wiki.openstack.org/w/index.php?title=Meetings/CreateaMeeting&diff=124075&oldid=122059 | 09:06 |
| *** sdake has quit IRC | 09:07 | |
| *** sdake has joined #openstack-infra | 09:07 | |
| openstackgerrit | Rob Cresswell proposed openstack/requirements: Add django debug toolbar and template profiler https://review.openstack.org/307001 | 09:08 |
| openstackgerrit | Adam Spiers proposed openstack-infra/irc-meetings: switch HA meetings to daylight savings https://review.openstack.org/307002 | 09:10 |
| *** jistr has joined #openstack-infra | 09:12 | |
| AJaeger | thanks, aspiers ! | 09:13 |
| *** watersoul has quit IRC | 09:16 | |
| *** nmagnezi has quit IRC | 09:19 | |
| *** watersoul has joined #openstack-infra | 09:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:22 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Make Mitaka Horizon dj19 test non-voting https://review.openstack.org/307010 | 09:24 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Disable Mitaka Horizon dj19 test https://review.openstack.org/307010 | 09:29 |
| robcresswell | Sorry about the title there AJaeger. | 09:29 |
| *** sshnaidm has joined #openstack-infra | 09:29 | |
| AJaeger | robcresswell: no problem. Just wanted to be sure that you did the right thing ;) | 09:29 |
| *** kushal has quit IRC | 09:32 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow filtering of worklists and boards by story or task https://review.openstack.org/288910 | 09:35 |
| *** asettle has joined #openstack-infra | 09:36 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add infracloud east to puppetmaster clouds https://review.openstack.org/282920 | 09:38 |
| *** gildub has joined #openstack-infra | 09:39 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add infracloud east to puppetmaster clouds https://review.openstack.org/282920 | 09:40 |
| *** SotK_ is now known as SotK | 09:40 | |
| *** claudiub has joined #openstack-infra | 09:42 | |
| *** kushal has joined #openstack-infra | 09:42 | |
| *** dizquierdo has joined #openstack-infra | 09:42 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Use infra-cloud east in nodepool https://review.openstack.org/283573 | 09:43 |
| *** Hal has joined #openstack-infra | 09:44 | |
| *** claudiub|2 has quit IRC | 09:44 | |
| *** zhurong has quit IRC | 09:44 | |
| *** Hal is now known as Guest12114 | 09:45 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add infracloud east to nodepool https://review.openstack.org/283576 | 09:45 |
| *** scheuran has quit IRC | 09:49 | |
| *** mpaolino has quit IRC | 09:49 | |
| electrofelix | waynr zxiiro zaro: if you're about this week, I should have a bit more time free so think it would useful to try and close out a chunk of existing reviews and cut a final 1.x release on Monday so that waynr can target his 2.x work directly onto master and we look to release that work in about a month | 09:50 |
| *** mpaolino has joined #openstack-infra | 09:50 | |
| *** esikachev has quit IRC | 09:51 | |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: email_ext publisher: Improve docstring wording https://review.openstack.org/306993 | 09:53 |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: email publisher: Clarify format of recipients list in docstring https://review.openstack.org/306992 | 09:53 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 09:53 |
| AJaeger | odyssey4me: thanks for your comments on the Install Guide specs! | 09:54 |
| odyssey4me | AJaeger yeah, it's a positive move - unfortunately I can't make the summit chat due to a conflict | 09:55 |
| odyssey4me | I wonder if we can't find a cross-project session room for that one actually? | 09:55 |
| *** esikachev has joined #openstack-infra | 09:55 | |
| odyssey4me | it's probably too late | 09:56 |
| AJaeger | odyssey4me: I agree, too late | 09:56 |
| odyssey4me | AJaeger I was also wondering what it takes to enable multi-lingual support in the project docs | 09:56 |
| AJaeger | odyssey4me: you mean: Translation?` | 09:56 |
| odyssey4me | I was thinking that perhaps we could try that for OpenStack-Ansible. | 09:56 |
| odyssey4me | yeah, translation | 09:56 |
| odyssey4me | I have no idea how it all works, and I know it's your pet project :) | 09:57 |
| AJaeger | Like http://docs.openstack.org/draft/ja/install-guide-obs/ ? | 09:57 |
| *** tosky has joined #openstack-infra | 09:57 | |
| odyssey4me | yep, something like that - I'd love to try it out on our install guide | 09:57 |
| AJaeger | odyssey4me: not that difficult to setup thanks to sphinx and babel. | 09:57 |
| odyssey4me | so the translation is all completely automatic? | 09:58 |
| AJaeger | problem for translations is that it's an always moving target | 09:58 |
| *** Nisha has joined #openstack-infra | 09:58 | |
| AJaeger | odyssey4me: for openstack-manuals: Yes - besides configuring it once | 09:58 |
| AJaeger | http://git.openstack.org/cgit/openstack/openstack-manuals/tree/doc-tools-check-languages.conf | 09:59 |
| AJaeger | But that could be done differently. | 09:59 |
| AJaeger | Let's first figure out how the guide will look ;) | 09:59 |
| AJaeger | The translators are interested in translating release notes. Once that's done (automatic!), we could handle documentation the same way. | 09:59 |
| odyssey4me | AJaeger so our guide is currently published at http://docs.openstack.org/developer/openstack-ansible/ and is sourced from http://git.openstack.org/cgit/openstack/openstack-ansible/tree/doc/source | 10:00 |
| AJaeger | automatic= pushing translations and publishing. | 10:00 |
| odyssey4me | ah, ok - that sounds promising - we can learn from the release notes option | 10:00 |
| openstackgerrit | Vasyl Saienko proposed openstack/requirements: Block broken netmiko release 0.4.3 https://review.openstack.org/306206 | 10:01 |
| AJaeger | odyssey4me: and right now we're only half-way done for release notes, we still need to figure out how to publish properly | 10:02 |
| AJaeger | odyssey4me: And translators need to translate as well ;) | 10:02 |
| *** asettle has quit IRC | 10:03 | |
| odyssey4me | AJaeger alright, when we're in a place where we can be your testers for project doc publishing, then feel free to ping me :) | 10:04 |
| *** salv-orl_ has joined #openstack-infra | 10:04 | |
| *** scheuran has joined #openstack-infra | 10:04 | |
| *** salv-orlando has quit IRC | 10:07 | |
| odyssey4me | AJaeger on another topic, should I fix the merge conflict for https://review.openstack.org/299192 now or wait until the action is scheduled? | 10:07 |
| *** Qiming has quit IRC | 10:11 | |
| *** Yarboa has joined #openstack-infra | 10:13 | |
| derekh | Hey, is this change to block access to the jenkins servers permanent ? http://git.openstack.org/cgit/openstack-infra/system-config/commit/?id=6d99934ff0e9a66531327c86c23e2cd88d5d2ccc | 10:14 |
| *** sdague has joined #openstack-infra | 10:15 | |
| tosky | derekh: http://lists.openstack.org/pipermail/openstack-dev/2016-April/092482.html | 10:18 |
| odyssey4me | heh, tosky beat me to it by a millisecond | 10:19 |
| derekh | tosky odyssey4me : thanks | 10:19 |
| openstackgerrit | Yanyan Hu proposed openstack-infra/project-config: Add rally job for Senlin service https://review.openstack.org/307050 | 10:20 |
| rcarrillocruz | doh | 10:23 |
| *** Yarboa has quit IRC | 10:24 | |
| rcarrillocruz | i'm seeing dns issues on ovh nodepool nodes | 10:24 |
| rcarrillocruz | https://review.openstack.org/#/c/306502/ | 10:24 |
| yolanda | one of my jobs failed as well, rcarrillocruz | 10:24 |
| *** kzaitsev_mb has quit IRC | 10:24 | |
| rcarrillocruz | :/ | 10:24 |
| yolanda | going to check mirror status | 10:25 |
| *** pfallenop has joined #openstack-infra | 10:28 | |
| AJaeger | odyssey4me: I would wait until it's scheduled... | 10:30 |
| yolanda | i cannot reach that mirror :( | 10:30 |
| odyssey4me | AJaeger thanks for confirming, I'll look out for a ping once the rename is scheduled but will leave it alone for now | 10:31 |
| yolanda | AJaeger, what do you think of setting of disabling ovh until we have it working? mirror seems to be not responsive | 10:31 |
| AJaeger | once it's scheduled, there should also be an announcement email for the downtime. But I'm sure we'll ping or one of us updates... | 10:31 |
| AJaeger | yolanda: seems to be the best short term workaround... | 10:32 |
| AJaeger | ... if we cannot fix the mirror again quickly | 10:32 |
| yolanda | AJaeger, i tried to connect by ssh, and i have no connection | 10:32 |
| *** |-paul-| has quit IRC | 10:33 | |
| AJaeger | Then let's disable it for now - and enable once somebody fixed the mirror | 10:34 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Temporarily disable ovh-bhs provider https://review.openstack.org/307079 | 10:34 |
| yolanda | AJaeger ^ | 10:35 |
| AJaeger | yolanda: +2A | 10:35 |
| yolanda | going to deal with the mirror , trying to access with nova | 10:35 |
| *** _degorenko|afk is now known as degorenko | 10:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Disable Mitaka Horizon dj19 test https://review.openstack.org/307010 | 10:36 |
| AJaeger | thanks, yolanda | 10:36 |
| AJaeger | yolanda, rcarrillocruz this is both ovh-bhs and ovh-gra, isn't it? | 10:38 |
| *** Yarboa has joined #openstack-infra | 10:38 | |
| yolanda | i saw in ovh-bhs , haven't checked gra | 10:38 |
| rcarrillocruz | good call | 10:38 |
| rcarrillocruz | i think i had also on gra | 10:38 |
| AJaeger | http://logs.openstack.org/02/306502/2/check/gate-shade-dsvm-functional-neutron/93403ab/console.html | 10:38 |
| AJaeger | that'S gra | 10:38 |
| *** daemontool has joined #openstack-infra | 10:38 | |
| rcarrillocruz | yep | 10:38 |
| rcarrillocruz | you may want to amend yolanda | 10:39 |
| yolanda | yes please, send another change for gra | 10:39 |
| yolanda | the other had +a | 10:39 |
| AJaeger | rcarrillocruz: do you want to do a change? | 10:40 |
| rcarrillocruz | i can sure | 10:40 |
| rcarrillocruz | sec | 10:40 |
| yolanda | weird, i try to connect to openstackci-ovh and get 401 errors, and i can access to other clouds | 10:41 |
| yolanda | ouch | 10:42 |
| yolanda | Making authentication request to https://auth.cloud.ovh.net/v3/auth/tokens | 10:42 |
| yolanda | Resetting dropped connection: auth.cloud.ovh.net | 10:42 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Temporarily disable OVH GRA, as we are seeing failures https://review.openstack.org/307083 | 10:43 |
| rcarrillocruz | AJaeger , yolanda : ^ | 10:43 |
| yolanda | thx | 10:43 |
| AJaeger | thanks! | 10:43 |
| *** hferenc has joined #openstack-infra | 10:44 | |
| hferenc | hi all! | 10:44 |
| yolanda | i'll need help of another infra-root for that. I am unable to communicate with OVH cloud | 10:47 |
| derekh | mordred: I'm seeing a lot of floating IP's being leaked by nodpool, I've had to delete them several times over the weekend so that the floating ip pool isn't exhausted, I'm also seeing these in the neutron logs | 10:49 |
| derekh | nova.api.openstack.compute.contrib.floating_ips NeutronClientException: 409-{u'NeutronError': {u'message': u'Cannot associate floating IP 66.187.229.97 (99426001-dd18-46f2-b6ea-3ea492fcdd3f) with port 7ac6c9da-01fa-4cf7-a6ba-644983aec94d using fixed IP 10.2.8.251, as that fixed IP already has a floating IP on external network d982c5c4-03f2-49f0-8f64-b4020e260ba2.', u'type': u'FloatingIPPortAlreadyAssociated', u'detail': u''}} | 10:49 |
| derekh | any idea if they're related to the nodepool networking stuff your doing? | 10:49 |
| *** asettle has joined #openstack-infra | 10:54 | |
| *** Nisha has quit IRC | 10:55 | |
| *** dizquierdo has quit IRC | 10:59 | |
| *** andreykurilin has joined #openstack-infra | 10:59 | |
| *** dizquierdo has joined #openstack-infra | 11:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:01 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: 'All Boards and Worklists' Page https://review.openstack.org/279753 | 11:01 |
| *** SlickN1k has joined #openstack-infra | 11:01 | |
| *** sheeprine_ has joined #openstack-infra | 11:01 | |
| *** markmcd__ has joined #openstack-infra | 11:01 | |
| *** monester_ has joined #openstack-infra | 11:01 | |
| *** aorourke__ has joined #openstack-infra | 11:02 | |
| *** mahatic_ has joined #openstack-infra | 11:02 | |
| *** marcusvrn__ has joined #openstack-infra | 11:02 | |
| *** lindsayk__ has joined #openstack-infra | 11:02 | |
| *** etoews_ has joined #openstack-infra | 11:02 | |
| *** zhiyan_ has joined #openstack-infra | 11:02 | |
| *** xgerman_ has joined #openstack-infra | 11:02 | |
| *** andreykurilin___ has joined #openstack-infra | 11:02 | |
| *** dham1_ has joined #openstack-infra | 11:02 | |
| *** serverascode_ has joined #openstack-infra | 11:02 | |
| *** rguillebert_ has joined #openstack-infra | 11:02 | |
| *** GheRivero_ has joined #openstack-infra | 11:02 | |
| *** vdrok_ has joined #openstack-infra | 11:03 | |
| *** mgkwill_ has joined #openstack-infra | 11:03 | |
| yolanda | i need to leave for an hour | 11:04 |
| *** igorbelikov_ has joined #openstack-infra | 11:04 | |
| *** beekneemech has joined #openstack-infra | 11:04 | |
| *** yonglihe_ has joined #openstack-infra | 11:04 | |
| *** hrybacki_ has joined #openstack-infra | 11:04 | |
| *** krotscheck_ has joined #openstack-infra | 11:04 | |
| *** jgriffith_ has joined #openstack-infra | 11:04 | |
| *** lbragstad_ has joined #openstack-infra | 11:04 | |
| *** Aegil_ has joined #openstack-infra | 11:04 | |
| *** tristanC_ has joined #openstack-infra | 11:04 | |
| *** Shalmazar has joined #openstack-infra | 11:04 | |
| *** a2hill has joined #openstack-infra | 11:04 | |
| *** rodrigods has quit IRC | 11:04 | |
| *** jaosorior has joined #openstack-infra | 11:05 | |
| *** rodrigods has joined #openstack-infra | 11:05 | |
| *** cinerama` has joined #openstack-infra | 11:05 | |
| *** dmellado_ has joined #openstack-infra | 11:05 | |
| *** jasondotstar_ has joined #openstack-infra | 11:05 | |
| *** cschwede_ has joined #openstack-infra | 11:05 | |
| *** elmiko_ has joined #openstack-infra | 11:05 | |
| *** clayton_ has joined #openstack-infra | 11:05 | |
| *** kevinshort has quit IRC | 11:05 | |
| *** xgerman has quit IRC | 11:05 | |
| *** hrybacki has quit IRC | 11:05 | |
| *** lindsayk_ has quit IRC | 11:05 | |
| *** Ng has quit IRC | 11:05 | |
| *** marcusvrn_ has quit IRC | 11:05 | |
| *** vdrok has quit IRC | 11:05 | |
| *** asilenkov has quit IRC | 11:05 | |
| *** andreykurilin__ has quit IRC | 11:05 | |
| *** rguillebert has quit IRC | 11:05 | |
| *** mgkwill has quit IRC | 11:05 | |
| *** melwitt has quit IRC | 11:05 | |
| *** yonglihe has quit IRC | 11:05 | |
| *** monester has quit IRC | 11:05 | |
| *** timburke has quit IRC | 11:05 | |
| *** katyafervent_awa has quit IRC | 11:05 | |
| *** SlickNik has quit IRC | 11:05 | |
| *** jgriffith has quit IRC | 11:05 | |
| *** tdasilva has quit IRC | 11:05 | |
| *** etoews has quit IRC | 11:05 | |
| *** jpmaxman has quit IRC | 11:05 | |
| *** igorbelikov has quit IRC | 11:05 | |
| *** dham1 has quit IRC | 11:05 | |
| *** dtroyer has quit IRC | 11:05 | |
| *** jzerebecki has quit IRC | 11:05 | |
| *** lbragstad has quit IRC | 11:05 | |
| *** lindsayk has quit IRC | 11:05 | |
| *** therve has quit IRC | 11:05 | |
| *** bookwar has quit IRC | 11:05 | |
| *** Madasi has quit IRC | 11:05 | |
| *** jasondotstar has quit IRC | 11:05 | |
| *** nt has quit IRC | 11:05 | |
| *** anthonyper has quit IRC | 11:05 | |
| *** mhayden has quit IRC | 11:06 | |
| *** kinrui has quit IRC | 11:06 | |
| *** serverascode has quit IRC | 11:06 | |
| *** zhiyan has quit IRC | 11:06 | |
| *** briancli1e has quit IRC | 11:06 | |
| *** harlowja has quit IRC | 11:06 | |
| *** krotscheck has quit IRC | 11:06 | |
| *** penguinolog__ has quit IRC | 11:06 | |
| *** markmcd1 has quit IRC | 11:06 | |
| *** dtantsur has quit IRC | 11:06 | |
| *** dmellado has quit IRC | 11:06 | |
| *** ianw has quit IRC | 11:06 | |
| *** Aegil has quit IRC | 11:06 | |
| *** cinerama has quit IRC | 11:06 | |
| *** gongysh has quit IRC | 11:06 | |
| *** xnox has quit IRC | 11:06 | |
| *** sheeprine has quit IRC | 11:06 | |
| *** gabriel-bezerra has quit IRC | 11:06 | |
| *** msuriar has quit IRC | 11:06 | |
| *** GheRivero has quit IRC | 11:06 | |
| *** bnemec has quit IRC | 11:06 | |
| *** markmcclain has quit IRC | 11:06 | |
| *** elmiko has quit IRC | 11:06 | |
| *** tristanC has quit IRC | 11:06 | |
| *** csaba has quit IRC | 11:06 | |
| *** cschwede has quit IRC | 11:06 | |
| *** ptoohill has quit IRC | 11:06 | |
| *** ZZelle has quit IRC | 11:06 | |
| *** mahatic has quit IRC | 11:06 | |
| *** jesusaur has quit IRC | 11:06 | |
| *** briancline has joined #openstack-infra | 11:06 | |
| *** clayton has quit IRC | 11:06 | |
| *** aorourke_ has quit IRC | 11:06 | |
| *** SlickN1k is now known as SlickNik | 11:06 | |
| *** Shalmazar is now known as Madasi | 11:06 | |
| *** jgriffith_ is now known as jgriffith | 11:06 | |
| *** cschwede_ is now known as cschwede | 11:06 | |
| *** vdrok_ is now known as vdrok | 11:06 | |
| *** markmcclain1 has joined #openstack-infra | 11:06 | |
| *** lindsayk__ is now known as lindsayk_ | 11:06 | |
| *** rguillebert_ is now known as rguillebert | 11:06 | |
| *** monester_ is now known as monester | 11:06 | |
| *** jasondotstar_ is now known as jasondotstar | 11:06 | |
| *** gabriel-bezerra has joined #openstack-infra | 11:06 | |
| *** therve has joined #openstack-infra | 11:06 | |
| *** jgriffith is now known as Guest72772 | 11:06 | |
| *** bookwar has joined #openstack-infra | 11:06 | |
| *** markmcclain1 is now known as markmcclain | 11:06 | |
| *** andreykurilin___ is now known as andreykurilin__ | 11:06 | |
| *** yonglihe_ is now known as yonglihe | 11:06 | |
| *** timburke has joined #openstack-infra | 11:06 | |
| *** andreww has quit IRC | 11:06 | |
| *** krotscheck_ is now known as krotscheck | 11:06 | |
| *** xgerman_ is now known as xgerman | 11:06 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job https://review.openstack.org/229789 | 11:06 |
| *** andreww has joined #openstack-infra | 11:07 | |
| *** clayton_ is now known as clayton | 11:07 | |
| *** marcusvrn__ is now known as marcusvrn_ | 11:07 | |
| *** asilenkov has joined #openstack-infra | 11:07 | |
| *** lunarlamp is now known as mariusv | 11:07 | |
| *** penguinolog has joined #openstack-infra | 11:07 | |
| *** dtroyer has joined #openstack-infra | 11:07 | |
| *** mgkwill_ is now known as mgkwill | 11:07 | |
| *** hrybacki_ is now known as hrybacki | 11:07 | |
| *** dham1_ is now known as dham1 | 11:07 | |
| *** igorbelikov_ is now known as igorbelikov | 11:07 | |
| *** kevinshort has joined #openstack-infra | 11:07 | |
| *** etoews_ is now known as etoews | 11:07 | |
| *** xnox has joined #openstack-infra | 11:08 | |
| *** katyafervent_awa has joined #openstack-infra | 11:08 | |
| *** ianw has joined #openstack-infra | 11:08 | |
| *** GheRivero_ is now known as GheRivero | 11:09 | |
| *** dougwig has quit IRC | 11:09 | |
| *** skath has quit IRC | 11:09 | |
| *** sweston has quit IRC | 11:09 | |
| *** jzerebecki has joined #openstack-infra | 11:09 | |
| *** dtantsur has joined #openstack-infra | 11:09 | |
| *** anthonyper has joined #openstack-infra | 11:10 | |
| *** tosky_ has joined #openstack-infra | 11:10 | |
| *** pblaho has quit IRC | 11:10 | |
| *** melwitt has joined #openstack-infra | 11:10 | |
| *** zhiyan_ is now known as zhiyan | 11:10 | |
| *** |-paul-| has joined #openstack-infra | 11:10 | |
| *** melwitt is now known as Guest64003 | 11:10 | |
| *** Ng has joined #openstack-infra | 11:11 | |
| *** Qiming has joined #openstack-infra | 11:11 | |
| *** pblaho has joined #openstack-infra | 11:11 | |
| *** csaba has joined #openstack-infra | 11:12 | |
| *** tosky has quit IRC | 11:12 | |
| *** fhubik has quit IRC | 11:12 | |
| *** rcernin has quit IRC | 11:12 | |
| *** nt has joined #openstack-infra | 11:12 | |
| *** jpmaxman has joined #openstack-infra | 11:12 | |
| *** serverascode_ is now known as serverascode | 11:12 | |
| AJaeger | rcarrillocruz, yolanda : The OVH changes are failing in the gate ;( | 11:13 |
| *** sheeprine_ has quit IRC | 11:13 | |
| *** sheeprine has joined #openstack-infra | 11:13 | |
| *** skath has joined #openstack-infra | 11:13 | |
| *** Kennan has quit IRC | 11:13 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 11:14 |
| *** sweston has joined #openstack-infra | 11:15 | |
| *** dougwig has joined #openstack-infra | 11:16 | |
| *** msuriar has joined #openstack-infra | 11:16 | |
| *** kinrui has joined #openstack-infra | 11:16 | |
| *** tdasilva has joined #openstack-infra | 11:18 | |
| *** mugsie_ is now known as mugsie | 11:19 | |
| *** mhayden has joined #openstack-infra | 11:20 | |
| openstackgerrit | Rob Cresswell proposed openstack/requirements: Add django debug toolbar and template profiler https://review.openstack.org/307001 | 11:21 |
| *** samueldm1 has quit IRC | 11:22 | |
| *** rhallisey has joined #openstack-infra | 11:22 | |
| *** rcernin has joined #openstack-infra | 11:23 | |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Decouple Shotgun project from Fuel https://review.openstack.org/298615 | 11:24 |
| dims | AJaeger : "OVH changes" failures show up like this? http://logs.openstack.org/38/306838/2/gate/gate-keystone-pep8/c3293de/console.html#_2016-04-18_10_48_25_080 | 11:25 |
| *** nt has quit IRC | 11:26 | |
| yolanda | AJaeger, sorry, i'm in and out taking care of children. Let me disable nodepool on puppet and set max-servers to 0 manually | 11:28 |
| yolanda | then you can recheck | 11:28 |
| *** nt has joined #openstack-infra | 11:28 | |
| *** dprince has joined #openstack-infra | 11:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily disable OVH GRA, as we are seeing failures https://review.openstack.org/307083 | 11:29 |
| mattymo | I'm getting some CI failures related to inability to fetch http://mirror.bhs1.ovh.openstack.org/ubuntu/dists/trusty/InRelease | 11:29 |
| mattymo | oh cool | 11:29 |
| yolanda | AJaeger, done, can you retry ovh bhs1 ? | 11:30 |
| hferenc | I might have an off topic problem, but since it seems like a cross-project issues let me ask it here. | 11:31 |
| hferenc | So, I just came across a problem (that might be a bug as well): there's an option called "Watchdog Action" on the "Update Flavor Metadata" page in Horizon, adding this option to any flavor adds a "hw_watchdog_action": "option" key-value pair to the extra specs of the given flavor. However, Nova uses the new namespace:key format i.e. "hw:watchdog_action" : "option", hence it fails to create instances based on a flavor that h | 11:31 |
| *** ldnunes has joined #openstack-infra | 11:31 | |
| *** Kennan has joined #openstack-infra | 11:32 | |
| *** dizquierdo has quit IRC | 11:33 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Don't allow cards to be dragged to/from automatic worklists https://review.openstack.org/306497 | 11:34 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Make the Add Lane button show the new worklist modal https://review.openstack.org/306399 | 11:34 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Remove pointless variable creation https://review.openstack.org/306401 | 11:34 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal when editing worklists in a board https://review.openstack.org/306400 | 11:34 |
| *** gordc has joined #openstack-infra | 11:35 | |
| *** Yarboa has quit IRC | 11:37 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 11:40 |
| *** ams_ has joined #openstack-infra | 11:40 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add support for Designate zones https://review.openstack.org/302770 | 11:43 |
| *** lucasagomes is now known as lucas-hungry | 11:43 | |
| *** baoli_ has joined #openstack-infra | 11:49 | |
| *** asettle has quit IRC | 11:50 | |
| *** tpsilva has joined #openstack-infra | 11:50 | |
| *** Yarboa has joined #openstack-infra | 11:50 | |
| *** asettle has joined #openstack-infra | 11:50 | |
| *** baoli has quit IRC | 11:52 | |
| *** thorst_ has joined #openstack-infra | 11:52 | |
| *** thorst_ has quit IRC | 11:53 | |
| *** thorst_ has joined #openstack-infra | 11:53 | |
| AJaeger | yolanda: will retry. TAke care of you r kids! Thanks | 11:55 |
| AJaeger | hferenc: Better use #openstack-dev | 11:55 |
| AJaeger | dims, yes exactly like that | 11:56 |
| *** flepied1 is now known as flepied | 11:56 | |
| AJaeger | dims: will you release oslo.i18n for the Babel change? | 11:56 |
| *** weshay has joined #openstack-infra | 11:56 | |
| andreykurilin | AJaeger: hi! I have a problems with pushign tag to gerrit. Could you help me? | 11:57 |
| andreykurilin | *problem | 11:57 |
| hferenc | AJaeger: ok, thanks! | 11:57 |
| AJaeger | andreykurilin: I doubt I can but let's see ;) Which repo? What's the problem? | 11:57 |
| *** kengo_sakai has quit IRC | 11:57 | |
| AJaeger | andreykurilin: did you read the manual? | 11:57 |
| dims | AJaeger : i'll let harlowja make the decision. it feels to me that other than i18n, none of the oslo projects should have babel | 11:58 |
| AJaeger | http://docs.openstack.org/infra/manual/drivers.html#tagging-a-release | 11:58 |
| AJaeger | dims: indeed | 11:59 |
| dims | andreykurilin : did you catch the discussion on the mailing list around https://review.openstack.org/#/c/298866/ ? | 11:59 |
| andreykurilin | AJaeger: https://github.com/openstack/rally/ . I did not read any manuals yet (just had a quick instruction), but I had no problem with pushing a month ago | 12:00 |
| *** esikachev has quit IRC | 12:00 | |
| andreykurilin | dims: no, I saw it first time now | 12:00 |
| dims | andreykurilin : http://lists.openstack.org/pipermail/openstack-dev/2016-March/thread.html#90737 | 12:01 |
| *** yamamoto has quit IRC | 12:01 | |
| AJaeger | dims, thanks! You've been faster than me ;) | 12:02 |
| dims | :) lol | 12:02 |
| AJaeger | andreykurilin: also http://lists.openstack.org/pipermail/openstack-dev/2016-April/092451.html | 12:02 |
| dims | ++ AJaeger | 12:02 |
| *** asettle has quit IRC | 12:03 | |
| *** _amrith_ is now known as amrith | 12:03 | |
| AJaeger | andreykurilin: and finally http://lists.openstack.org/pipermail/openstack-dev/2016-April/092391.html | 12:03 |
| *** mtreinish has joined #openstack-infra | 12:03 | |
| AJaeger | andreykurilin: so everything works as designed ;) | 12:03 |
| AJaeger | andreykurilin: dims might be able to guide you through the new process via the releases repository | 12:04 |
| *** rfolco has joined #openstack-infra | 12:04 | |
| andreykurilin | dims, AJaeger : so there is no ability to send tag manually and I need to use openstack/releases ? | 12:04 |
| AJaeger | andreykurilin: correct | 12:04 |
| jroll | hey infra, ironic gate started failing often late last week (friday?), ssh to the instance is timing out. locally we can reproduce it and looks like this polling code isn't working (if we force it to not use polling, it's all good): https://github.com/openstack/tempest-lib/blob/master/tempest_lib/common/ssh.py#L127-L160 | 12:04 |
| *** aysyd has joined #openstack-infra | 12:04 | |
| andreykurilin | ;( | 12:04 |
| AJaeger | andreykurilin: We now have Releases-as-a-service ;) | 12:05 |
| andreykurilin | heh | 12:05 |
| jroll | curious if there could be an image update thing or what | 12:05 |
| AJaeger | andreykurilin: I've gone through that with openstack-doc-tools last year and it's a smooth experience! | 12:05 |
| andreykurilin | AJaeger dims: as far as I understand, I need just send a patch to https://github.com/openstack/releases/blob/master/deliverables/_independent/rally.yaml and after it merged, everything will be done. | 12:07 |
| AJaeger | jroll: same problem as http://lists.openstack.org/pipermail/openstack-dev/2016-April/092498.html ? | 12:07 |
| AJaeger | andreykurilin: correct! | 12:07 |
| AJaeger | andreykurilin: including a nice release mail ;) | 12:07 |
| andreykurilin | AJaeger: we already merged release notes to our repo :) | 12:07 |
| andreykurilin | AJaeger: rally.yaml missed a lot of rally release, should I add them(What will happen for existing tags?!)? | 12:08 |
| *** trown|outtypewww is now known as trown | 12:09 | |
| AJaeger | andreykurilin: better ask dhellmann and dims those questions - in #openstack-release | 12:09 |
| jroll | AJaeger: I guess I don't understand the problem there, even after reading | 12:09 |
| andreykurilin | AJaeger: thanks a lot | 12:09 |
| *** andreykurilin has left #openstack-infra | 12:09 | |
| jroll | AJaeger: but sounds like this thread is about ssh'ing to the gate node itself? | 12:09 |
| *** andreykurilin has joined #openstack-infra | 12:09 | |
| jroll | AJaeger: and sounds like only multi-node jobs? | 12:10 |
| AJaeger | jroll: That email is the only problem I'm aware of right now regarding networking in our nodes. | 12:10 |
| AJaeger | jroll: but I cannot help further with debugging it. | 12:10 |
| AJaeger | jroll: better wait until other infra team members are awake and provide log files as well | 12:11 |
| jroll | AJaeger: yeah, I guess I'm asking for help with this | 12:11 |
| jroll | sure | 12:11 |
| jroll | one example log (see timestamps): http://logs.openstack.org/98/305498/1/check/gate-tempest-dsvm-ironic-pxe_ssh-dib/0f57bec/console.html#_2016-04-17_16_41_51_992 | 12:11 |
| AJaeger | jroll: we merged this morning https://review.openstack.org/306910 - see the referenced bug there as well | 12:13 |
| *** zz_dimtruck is now known as dimtruck | 12:13 | |
| jroll | AJaeger: yes, that's the bug I'm trying to track down | 12:13 |
| AJaeger | ah ;) | 12:14 |
| AJaeger | jroll: then you need to wait for others... | 12:14 |
| jroll | sure | 12:15 |
| *** mylu has joined #openstack-infra | 12:15 | |
| *** markvoelker has joined #openstack-infra | 12:19 | |
| yolanda | back | 12:21 |
| mattymo | I'm not sure why, but recheck and recheck no bug aren't working to retrigger jenkins on my patch :( | 12:22 |
| mattymo | https://review.openstack.org/#/c/305918/5 | 12:22 |
| AJaeger | mattymo: have patience, our systems are busy. | 12:22 |
| AJaeger | mattymo: check status.openstack.org/zuul - is your patch there? | 12:23 |
| mattymo | AJaeger, oh cool it's queued | 12:24 |
| *** mylu has quit IRC | 12:26 | |
| *** edmondsw has joined #openstack-infra | 12:26 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Don't allow cards to be dragged to/from automatic worklists https://review.openstack.org/306497 | 12:27 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Make the Add Lane button show the new worklist modal https://review.openstack.org/306399 | 12:27 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Remove pointless variable creation https://review.openstack.org/306401 | 12:27 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal when editing worklists in a board https://review.openstack.org/306400 | 12:27 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet/periodic: export ZUUL_REF https://review.openstack.org/307142 | 12:28 |
| *** ddieterly has joined #openstack-infra | 12:29 | |
| *** ddieterly is now known as ddieterly[away] | 12:30 | |
| *** julim has joined #openstack-infra | 12:30 | |
| *** keedya has joined #openstack-infra | 12:31 | |
| *** esikachev has joined #openstack-infra | 12:33 | |
| *** hferenc has quit IRC | 12:33 | |
| *** mylu has joined #openstack-infra | 12:33 | |
| *** roxanaghe has joined #openstack-infra | 12:34 | |
| *** zeih has quit IRC | 12:36 | |
| yolanda | infra-root, who is better point of contact for OVH failures? | 12:37 |
| *** zeih has joined #openstack-infra | 12:37 | |
| *** yamamoto has joined #openstack-infra | 12:38 | |
| *** esikachev has quit IRC | 12:38 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: run beaker jobs on puppet-ovn https://review.openstack.org/307148 | 12:38 |
| *** mylu has quit IRC | 12:39 | |
| *** Yarboa has quit IRC | 12:39 | |
| morgan | yolanda: o/ | 12:39 |
| *** roxanaghe has quit IRC | 12:39 | |
| yolanda | morgan, so mirror.bhs1.ovh.openstack.org and gra1 are not responsive. And when i try to connect with nova, to manage the server, i get that connection dropped problem | 12:39 |
| *** samueldmq has joined #openstack-infra | 12:39 | |
| yolanda | auth.cloud.ovh.net is reachable | 12:40 |
| morgan | yolanda: is this in a different region from the other tenant's servers? | 12:40 |
| openstackgerrit | Mathieu Rohon proposed openstack-infra/project-config: Add release notes jobs for networking-bgpvpn https://review.openstack.org/307150 | 12:40 |
| yolanda | no, same region | 12:40 |
| morgan | or is this failing *at* login? | 12:40 |
| yolanda | and i have that failure in the two regions | 12:40 |
| morgan | vs when talking to non-keystone endpoints? | 12:40 |
| yolanda | morgan, the ovh cloud that works could login to keystone successfully | 12:41 |
| samueldmq | yolanda: and continue using other services? | 12:41 |
| morgan | and the one you're having issue with you can't even get a token for? | 12:41 |
| *** newjersey has joined #openstack-infra | 12:41 | |
| yolanda | morgan, no, i cannot get a token | 12:41 |
| yolanda | let me paste you the full error | 12:41 |
| morgan | yo ok | 12:42 |
| openstackgerrit | amrith proposed openstack-infra/project-config: Add release notes jobs for trove-dashboard CI https://review.openstack.org/307152 | 12:42 |
| morgan | yolanda: * | 12:42 |
| *** lucas-hungry is now known as lucasagomes | 12:44 | |
| EmilienM | yolanda: same here, puppet jobs are currently failing (cc degorenko) | 12:44 |
| yolanda | morgan, samueldmq, http://paste.openstack.org/show/494407/ | 12:44 |
| yolanda | EmilienM, we disabled ovh temporarily from nodepool | 12:44 |
| *** piet has joined #openstack-infra | 12:45 | |
| *** yamamoto has quit IRC | 12:45 | |
| EmilienM | yolanda: ack - we can do recheck then | 12:45 |
| *** yamamoto has joined #openstack-infra | 12:45 | |
| anteaya | can a channel op remove patchbot from #openstack-meeting please? | 12:45 |
| samueldmq | yolanda: morgan: maybe password/role assignments? | 12:47 |
| samueldmq | maybe something simpler like that | 12:47 |
| *** rlandy has joined #openstack-infra | 12:49 | |
| mattymo | should zuul really hold the patch for 1h20mfor recheck before starting? https://review.openstack.org/#/c/305918/ | 12:49 |
| mattymo | it's never been so slow before | 12:50 |
| *** yamamoto has quit IRC | 12:50 | |
| morgan | anteaya: you could also ask notmyname, he's usually very responsive | 12:50 |
| morgan | anteaya: and it's his bot | 12:50 |
| *** nelsnelson has quit IRC | 12:51 | |
| AJaeger | mattymo: we're having a problem with one of our clouds and currently have a long queue. So, please have patience... | 12:51 |
| *** lonelywolf has joined #openstack-infra | 12:52 | |
| *** vincentfrancoise has joined #openstack-infra | 12:52 | |
| morgan | yolanda: looking at the past, it's just slow to load for me atm | 12:52 |
| *** ddieterly[away] is now known as ddieterly | 12:53 | |
| *** yamamoto has joined #openstack-infra | 12:54 | |
| lonelywolf | Hello, I shall report broken gates gate-networking-ovs-dpdk-docs, gate-networking-ovs-dpdk-pep8. It looks like some infrastructure issue, as apt-get update got crashed. Logs are http://logs.openstack.org/55/307055/1/check/gate-networking-ovs-dpdk-docs/298d9a0/console.html. Is this known problem ? Thanks | 12:55 |
| *** pradk has joined #openstack-infra | 12:55 | |
| AJaeger | lonelywolf: is this on ovh nodes? Then it's a known problem. | 12:55 |
| *** Guest12114 has quit IRC | 12:55 | |
| samueldmq | morgan: do you thnk it can be just a simple permission issue? | 12:56 |
| samueldmq | morgan: in keystone? | 12:56 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: add releasenotes & beaker jobs to puppet-ec2api https://review.openstack.org/307162 | 12:56 |
| samueldmq | looking at that paste .. | 12:56 |
| morgan | samueldmq: unknown. | 12:56 |
| *** dizquierdo has joined #openstack-infra | 12:56 | |
| morgan | samueldmq: it might be. | 12:56 |
| *** gildub has quit IRC | 12:56 | |
| morgan | looks like username/password is bad or the user doesn't have access to the tenant now. | 12:56 |
| lonelywolf | AJaeger: sorry, what's ovh ? | 12:56 |
| *** yamamoto has quit IRC | 12:56 | |
| AJaeger | lonelywolf: one of our nodes - one that has problems. | 12:57 |
| openstackgerrit | amrith proposed openstack-infra/project-config: Add release notes jobs for trove-dashboard CI https://review.openstack.org/307152 | 12:57 |
| *** yamamoto has joined #openstack-infra | 12:57 | |
| AJaeger | lonelywolf: line 3 of your log file says "Building remotely on ubuntu-trusty-ovh-bhs1-216687", so it's on ovsh - and therefore: Yes, known problem | 12:57 |
| morgan | since the post to the /v3/auth/tokens is failing | 12:57 |
| morgan | yolanda: ^ cc | 12:57 |
| *** dprince has quit IRC | 12:57 | |
| lonelywolf | AJaeger: I found this error - Failed to fetch http://mirror.bhs1.ovh.openstack.org/ubuntu/dists/trusty/InRelease, so it's confirmed I guess | 12:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: add releasenotes and beaker jobs on puppet-ovn https://review.openstack.org/307148 | 12:58 |
| AJaeger | lonelywolf: indeed | 12:58 |
| lonelywolf | AJaeger: should I just cross my fingers and wait, "recheck" in couple of hrs or can I help somehow ? | 12:58 |
| amrith | AJaeger, ping. please see two reviews I tagged you on :) | 12:58 |
| krotscheck | jeblair: Neat! Ok, so it sounds like I might be able to set this script up as a daemon rather than a cron job? | 12:59 |
| *** amotoki has quit IRC | 12:59 | |
| krotscheck | jeblair: What's teh status of the VOS release? | 12:59 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: add releasenotes and beaker jobs on puppet-ovn https://review.openstack.org/307148 | 12:59 |
| *** eharney has quit IRC | 13:01 | |
| *** cbader has joined #openstack-infra | 13:01 | |
| *** Hal has joined #openstack-infra | 13:02 | |
| *** yamamoto has quit IRC | 13:02 | |
| *** Hal is now known as Guest78460 | 13:02 | |
| *** tonytan4ever has joined #openstack-infra | 13:02 | |
| *** kgiusti has joined #openstack-infra | 13:04 | |
| AJaeger | amrith: thanks for updating docu as well! | 13:04 |
| amrith | np AJaeger, I hope that's the complete list | 13:04 |
| AJaeger | lonelywolf: you can recheck now - but it will be veeerry slow. We disabled ovh while we're trying to fix the issue | 13:04 |
| AJaeger | amrith: looked fine ;) | 13:05 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: add reno jobs to puppet-oslo https://review.openstack.org/307166 | 13:05 |
| *** pfallenop has quit IRC | 13:05 | |
| amrith | thanks AJaeger, wasn't sure there were other things that I may be able to add to the doc. | 13:06 |
| *** annegentle has joined #openstack-infra | 13:07 | |
| *** ddieterly has quit IRC | 13:07 | |
| pabelanger | morning | 13:08 |
| *** zz_ja has quit IRC | 13:08 | |
| matrohon | ttx, hi : I have a question related to reno | 13:08 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/release-tools: Fix ability to send notifications for multiple destinations https://review.openstack.org/307167 | 13:08 |
| openstackgerrit | Merged openstack-infra/shade: Pass specific cloud to openstack_clouds function https://review.openstack.org/294814 | 13:08 |
| AJaeger | morgan: pabelanger | 13:08 |
| *** bryan_att has joined #openstack-infra | 13:08 | |
| AJaeger | matrohon: best ask on #openstack-release | 13:08 |
| matrohon | AJaeger, ok thanks | 13:09 |
| *** vincentfrancoise has left #openstack-infra | 13:09 | |
| *** kinrui is now known as fungi | 13:09 | |
| * morgan lurks and sets another trap for AJaeger's <tab> miss :) | 13:09 | |
| zxiiro | electrofelix: sounds good, I'll try to get some reviews in this week. | 13:09 |
| AJaeger | sorry, morgan ! | 13:09 |
| *** matt-borland has joined #openstack-infra | 13:09 | |
| morgan | AJaeger: :) | 13:10 |
| AJaeger | morgan, pabelanger : Good morning! | 13:10 |
| morgan | AJaeger: hehe ^_^ | 13:10 |
| *** fhubik has joined #openstack-infra | 13:10 | |
| zxiiro | electrofelix: my intern should be starting next month too | 13:10 |
| *** nelsnelson has joined #openstack-infra | 13:11 | |
| *** elmiko_ is now known as elmiko | 13:11 | |
| *** lindsayk has joined #openstack-infra | 13:12 | |
| *** _ari_ has joined #openstack-infra | 13:13 | |
| *** yamamoto has joined #openstack-infra | 13:13 | |
| *** asselin__ has joined #openstack-infra | 13:15 | |
| *** annegentle has quit IRC | 13:15 | |
| *** tonytan_brb has joined #openstack-infra | 13:16 | |
| openstackgerrit | Mathieu Rohon proposed openstack-infra/project-config: Add release notes jobs for networking-bgpvpn https://review.openstack.org/307150 | 13:17 |
| *** eharney has joined #openstack-infra | 13:18 | |
| *** tonytan4ever has quit IRC | 13:19 | |
| *** amotoki has joined #openstack-infra | 13:19 | |
| *** dtantsur is now known as dtantsur|brb | 13:19 | |
| yolanda | hi pabelanger, do you know contact person from ovh? | 13:19 |
| *** nwkarsten has joined #openstack-infra | 13:20 | |
| *** zhurong has joined #openstack-infra | 13:20 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add domain_id param to project operations https://review.openstack.org/291701 | 13:21 |
| *** tosky_ is now known as tosky | 13:23 | |
| *** tonytan_brb has quit IRC | 13:24 | |
| openstackgerrit | Lajos Katona proposed openstack/requirements: Put an upper bound on testresources https://review.openstack.org/307174 | 13:24 |
| *** rfolco has quit IRC | 13:26 | |
| openstackgerrit | Sebastian Schuberth proposed openstack-infra/jenkins-job-builder: cmd: Add a config file fallback to the user's home directory https://review.openstack.org/237694 | 13:26 |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: Replace deprecated assertEquals with assertEqual https://review.openstack.org/307176 | 13:26 |
| *** xyang1 has joined #openstack-infra | 13:26 | |
| *** rfolco has joined #openstack-infra | 13:26 | |
| *** mriedem has joined #openstack-infra | 13:28 | |
| *** phschwartz_ is now known as pschwartz | 13:28 | |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: Replace deprecated assertEquals with assertEqual https://review.openstack.org/307176 | 13:28 |
| krotscheck | jeblair: Test build for dfw-rackspace https://review.openstack.org/#/c/307177/ | 13:28 |
| *** amitgandhinz has joined #openstack-infra | 13:29 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/release-tools: [WIP] Compare local and remote tarballs https://review.openstack.org/307178 | 13:31 |
| *** nwkarsten has quit IRC | 13:31 | |
| *** nwkarsten has joined #openstack-infra | 13:32 | |
| *** EricGonczer_ has joined #openstack-infra | 13:33 | |
| *** |-paul-| has quit IRC | 13:34 | |
| *** amotoki has quit IRC | 13:34 | |
| *** jrmyjohn has joined #openstack-infra | 13:35 | |
| *** f1ller has quit IRC | 13:35 | |
| *** nwkarsten has quit IRC | 13:36 | |
| *** nmagnezi has joined #openstack-infra | 13:36 | |
| *** lonelywolf has quit IRC | 13:37 | |
| *** f1ller has joined #openstack-infra | 13:37 | |
| *** Goneri has quit IRC | 13:37 | |
| pabelanger | yolanda: I cannot remember of the top of my head | 13:38 |
| *** jheroux has joined #openstack-infra | 13:38 | |
| fungi | i am around and catching up on e-mail, but am declaring bankruptcy on irc scrollback | 13:38 |
| fungi | i saw the bit about ovh mirror servers being unreachable | 13:38 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Don't allow cards to be dragged to/from automatic worklists https://review.openstack.org/306497 | 13:38 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Make the Add Lane button show the new worklist modal https://review.openstack.org/306399 | 13:38 |
| fungi | i wonder if someone deleted them/our account there? | 13:39 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Remove pointless variable creation https://review.openstack.org/306401 | 13:39 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal when editing worklists in a board https://review.openstack.org/306400 | 13:39 |
| yolanda | fungi yes, we have a problem with the mirror | 13:39 |
| pabelanger | fungi: Woah, that would not be good | 13:39 |
| yolanda | it's not reachable, and i cannot connect to openstackci-ovh | 13:39 |
| fungi | looks like both mirrors (bhs1 and gra1) | 13:39 |
| yolanda | openstackci-jenkins works | 13:39 |
| fungi | yeah, makes me wonder if they accidentally deleted our openstackci tenant | 13:39 |
| fungi | hpcloud used to do that on a regular basis, so it's not a foreign experience for us | 13:40 |
| *** amotoki has joined #openstack-infra | 13:40 | |
| yolanda | fungi, which is the contact person for ovh? | 13:40 |
| morgan | fungi: that is kindof what it sounds like - or something similar as the post to get a token fails now for ovh | 13:40 |
| fungi | yolanda: i'll check in our passwords list | 13:40 |
| morgan | fungi: according to yolanda's debug output | 13:41 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/release-tools: [WIP] Compare local and remote tarballs https://review.openstack.org/307178 | 13:41 |
| *** BigWillie has joined #openstack-infra | 13:41 | |
| *** nelsnelson has quit IRC | 13:41 | |
| *** zeih has quit IRC | 13:43 | |
| *** infra-re_ has quit IRC | 13:43 | |
| fungi | yolanda: looks like we probably log into https://www.ovh.com/manager/web/index.html and file a support ticket there. i can give that a try shortly | 13:43 |
| *** zeih has joined #openstack-infra | 13:43 | |
| yolanda | fungi i tried to log in, but these creds I had, didn't work | 13:43 |
| waynr | electrofelix: i can dedicate some review time after work this week, sorry i haven't been more useful with 1.x work--been helping with major redesign of our CI architecture at $dajob | 13:43 |
| yolanda | i used the openstackci-ovh creds | 13:44 |
| *** pc_m has quit IRC | 13:44 | |
| fungi | yolanda: looks like we have a separate client id and password listed for it | 13:44 |
| fungi | not our tenant-specific keystone creds | 13:44 |
| yolanda | ah, didn't check passwords | 13:44 |
| *** infra-red has joined #openstack-infra | 13:47 | |
| *** jaypipes has joined #openstack-infra | 13:47 | |
| *** nwkarsten has joined #openstack-infra | 13:48 | |
| *** pc_m has joined #openstack-infra | 13:49 | |
| *** amotoki has quit IRC | 13:49 | |
| *** Goneri has joined #openstack-infra | 13:49 | |
| *** abregman has quit IRC | 13:50 | |
| *** ddieterly has joined #openstack-infra | 13:51 | |
| *** ddieterly is now known as ddieterly[away] | 13:51 | |
| *** ddieterly[away] is now known as ddieterly | 13:51 | |
| sdague | ovh is wonky this morning - http://logs.openstack.org/43/307043/1/check/gate-nova-docs/021bac3/console.html ? | 13:52 |
| *** ddieterly is now known as ddieterly[away] | 13:52 | |
| *** nmagnezi has quit IRC | 13:52 | |
| AJaeger | sdague: ovh is disabled now - yolanda and fungi are on it. | 13:52 |
| *** derekh is now known as derekh_afk | 13:53 | |
| *** ddieterly[away] is now known as ddieterly | 13:54 | |
| *** oanson has quit IRC | 13:55 | |
| fungi | heh... "Your account has an outstanding balance of 135.81 € as of Apr 18, 2016 9:58:08 AM. Please go to the billing section of the website to settle the outstanding payment." | 13:55 |
| *** tonytan4ever has joined #openstack-infra | 13:55 | |
| yolanda | oh dear | 13:55 |
| *** psanchez has quit IRC | 13:55 | |
| *** psanchez has joined #openstack-infra | 13:56 | |
| fungi | there is only an openstackjenkins project listed for our account, no openstackci | 13:56 |
| AJaeger | 135.81? Let's do some quick fundraising here ;) | 13:57 |
| *** andrewbogott_ is now known as andrewbogott | 13:57 | |
| *** eharney has quit IRC | 13:57 | |
| *** andrewbogott has quit IRC | 13:57 | |
| *** andrewbogott has joined #openstack-infra | 13:57 | |
| fungi | i suspect that's just an artifact of how they do our free hosting. i'm ignoring that pop-up | 13:57 |
| *** links has quit IRC | 13:57 | |
| fungi | opening a support ticket now to inquire into the disappearance of our openstackci project | 13:58 |
| *** piet has quit IRC | 13:58 | |
| *** amotoki has joined #openstack-infra | 13:59 | |
| yolanda | fungi thx | 14:00 |
| fungi | judging from the dead space on our cacti graphs, this began right at 10:00 utc today | 14:00 |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily disable ovh-bhs provider https://review.openstack.org/307079 | 14:00 |
| *** bkeller` has quit IRC | 14:01 | |
| *** jmccrory has quit IRC | 14:01 | |
| yolanda | yes, we started to see the problems at that time | 14:01 |
| *** coreyob_ is now known as coreyob | 14:02 | |
| *** jmccrory has joined #openstack-infra | 14:02 | |
| pabelanger | Wow, 135.81 for all theses months, that's a pretty good deal | 14:02 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Don't allow users to interact with cards in automatic worklists https://review.openstack.org/306497 | 14:02 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Make the Add Lane button show the new worklist modal https://review.openstack.org/306399 | 14:02 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Remove pointless variable creation https://review.openstack.org/306401 | 14:02 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal when editing worklists in a board https://review.openstack.org/306400 | 14:02 |
| *** zeih has quit IRC | 14:03 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:03 | |
| fungi | this is now reported as ovh ticket 2016041819053688 | 14:04 |
| *** sc68cal has quit IRC | 14:04 | |
| *** bkeller` has joined #openstack-infra | 14:04 | |
| *** infra-red has quit IRC | 14:05 | |
| AJaeger | should we make some public announcement - here on IRC? - that we lost a cloud and have lower capacity? | 14:05 |
| *** avarner__ has joined #openstack-infra | 14:05 | |
| AJaeger | right now 325 jobs in check queue... | 14:05 |
| *** ddieterly is now known as ddieterly[away] | 14:06 | |
| fungi | i should also find out where rackspace got to with their issues in ord | 14:06 |
| fungi | assuming nobody else has followed up on that yet | 14:06 |
| pabelanger | I have not | 14:06 |
| AJaeger | pabelanger: please check your change https://review.openstack.org/306829 and self-approve if timing is fine | 14:07 |
| *** dstanek has joined #openstack-infra | 14:07 | |
| *** infra-red has joined #openstack-infra | 14:07 | |
| AJaeger | fungi: good idea | 14:07 |
| pabelanger | we also have 31 leaked FIP in bluebox, cleaning them up now | 14:07 |
| pabelanger | that will help | 14:07 |
| fungi | okay, rackspace support has thrown in the towel on providing stable networking to us | 14:08 |
| fungi | this is not good | 14:08 |
| yolanda | i'm in and out today because i have to take care of my children, i'm leaving now for another hour | 14:08 |
| mordred | fungi: oh good | 14:08 |
| yolanda | fungi, that sounds bad | 14:08 |
| AJaeger | fungi: that's unfortunate ;( | 14:08 |
| fungi | "I wish I was able to find something on the host servers, but the logs weren't able to provide any real insight. | 14:08 |
| *** tonytan4ever has quit IRC | 14:08 | |
| fungi | Please let us know if there is anything else that we can assist you with as we are here 24x7!" | 14:08 |
| *** sc68cal has joined #openstack-infra | 14:09 | |
| yolanda | infra-root, in other news, seems we are going to have servers for infra-cloud ready soon. They are finishing configuration, and waiting for some fixes to ILO ips DHCP | 14:09 |
| mordred | yolanda: woot! | 14:09 |
| pabelanger | AJaeger: sure, +A'd. We can revert easy | 14:09 |
| fungi | yolanda: that's excellent news indeed, thanks for the update | 14:09 |
| pabelanger | yolanda: \o/ | 14:09 |
| yolanda | i put a topic tomorrow for the meeting, to discuss what are we going to do in terms of naming, ip ranges, etc | 14:10 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: [WIP] Compare local and remote tarballs https://review.openstack.org/307178 | 14:10 |
| *** eharney has joined #openstack-infra | 14:11 | |
| amrith | AJaeger, some days ago you pinged me about mitaka translation job failures in Trove (https://review.openstack.org/#/c/303690/). The fix for this is https://review.openstack.org/#/c/307221/ and it is a clean cherry pick of a change that was just merged into master. | 14:12 |
| yolanda | leaving for a while again, today is a busy day | 14:12 |
| fungi | looks like rax has a maintenance scheduled for thursday in which they're upgrading their networking in ord, so maybe we should try it again after that | 14:13 |
| AJaeger | amrith: thanks for the info! | 14:13 |
| pabelanger | yolanda: ack | 14:15 |
| AJaeger | pabelanger: could you put two translation changes on your review queue for today, please? https://review.openstack.org/#/c/306864/ and https://review.openstack.org/#/c/306866 | 14:15 |
| *** nelsnelson has joined #openstack-infra | 14:15 | |
| pabelanger | AJaeger: checking | 14:15 |
| fungi | oh, nevermind, the ord maintenance later this week is moving first-gen servers to their next-gen environment, but we're not running first-gen instances there anyway | 14:15 |
| openstackgerrit | Merged openstack-infra/project-config: Switch to shade specific statsd for ServerTasks https://review.openstack.org/306829 | 14:16 |
| AJaeger | pabelanger: thanks, that was fast... | 14:17 |
| *** rbrndt_ has joined #openstack-infra | 14:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Translations: Fix releasenotes https://review.openstack.org/306864 | 14:22 |
| mordred | fungi: when you get a sec: https://review.openstack.org/#/c/306835/ is to fix multi-node issues | 14:23 |
| *** ddieterly[away] is now known as ddieterly | 14:23 | |
| mordred | Shrews: I think it's time to release shade since we're running current master in prod for nodepool ... any more outstanding patches you want to see landed before we do? | 14:23 |
| *** genunix has joined #openstack-infra | 14:24 | |
| *** ddieterly is now known as ddieterly[away] | 14:24 | |
| *** ajmiller has joined #openstack-infra | 14:25 | |
| *** rfolco has quit IRC | 14:26 | |
| *** rfolco has joined #openstack-infra | 14:26 | |
| *** tobiash has quit IRC | 14:26 | |
| mugsie | mordred: I would love the designate bits to be in the release :) | 14:26 |
| *** ddieterly[away] is now known as ddieterly | 14:27 | |
| *** tobiash has joined #openstack-infra | 14:27 | |
| *** tobiash has quit IRC | 14:27 | |
| tosky | mordred: hi, question on os-client-config; as it "honors all of the normal OS_* variables.", can I simply throw my OS_FOO_BAR in the mix? Is there some official standard for those variables, to avoid clashes? | 14:28 |
| openstackgerrit | Merged openstack-infra/project-config: Translations: Cleanup django modules https://review.openstack.org/306866 | 14:28 |
| *** rfolco has quit IRC | 14:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove swift credentials from zuul jobs https://review.openstack.org/306664 | 14:28 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add latest-deliverable-versions command https://review.openstack.org/307229 | 14:29 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add validate_tarballs.sh https://review.openstack.org/307230 | 14:29 |
| *** tobiash has joined #openstack-infra | 14:29 | |
| *** rfolco has joined #openstack-infra | 14:29 | |
| *** sridhar_ram has joined #openstack-infra | 14:30 | |
| Shrews | mordred: nothing pressing | 14:30 |
| *** lbragstad_ is now known as lbragstad | 14:31 | |
| mordred | mugsie, rcarrillocruz: I just -1d both designate patches, although I like them in general. however, before anybody changes anything, let's see if Shrews agrees with me | 14:31 |
| *** sridhar_ram has quit IRC | 14:31 | |
| mordred | tosky: os-client-config deals with OS variables by grabbing all env vars that start with OS_, stripping OS_ and lowercasing. so - yeah, you can totally just throw an OS_FOO_BAR into the mix | 14:32 |
| mordred | tosky: and you'll wind up with foo_var in the inner config dict of the CloudConfig object | 14:33 |
| tosky | mordred: and do you know about some standard for OS_ variables? Some of them are well known and used, but I wonder if there is some coordination | 14:33 |
| mordred | tosky: there are no standards for anything, no | 14:34 |
| openstackgerrit | Merged openstack-infra/nodepool: Write public IP into private when there's no private https://review.openstack.org/306835 | 14:34 |
| mordred | tosky: there are some common patterns though | 14:34 |
| tosky | or even some patterns; I would use the name of a specific project for some project specific value (OS_SAHARA_something) | 14:34 |
| jroll | "there are no standards for anything" too true | 14:34 |
| tosky | oh, which (other) common patterns? | 14:35 |
| *** ddieterly is now known as ddieterly[away] | 14:35 | |
| mordred | tosky: actually using the non-codename is more the pattern | 14:35 |
| tosky | so it would be OS_DATAPROCESSING_something | 14:35 |
| mordred | tosky: so, for instance, if you want to specify which API version to use - it's OS_COMPUTE_API_VERSION or OS_DNS_API_VERSION etc | 14:35 |
| mordred | yah | 14:35 |
| mordred | or maybe OS_DATA_PROCESSING_something | 14:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch to shade specific statsd for FloatingIPTasks https://review.openstack.org/307233 | 14:36 |
| mordred | I tink data-processing is two words, yeah? | 14:36 |
| tosky | weeeel, yes, should be | 14:36 |
| fungi | our standard is an anti-standard. anarchy prevails | 14:36 |
| mordred | tosky: a patch to os-client-config adding an entry for sahara in os_client_config/constructors.json would be nice | 14:37 |
| Shrews | mordred: i agree with the words that you typed | 14:37 |
| mordred | whatever key you use there is the closest you'll get to declaring a standard | 14:37 |
| *** sridhar_ram has joined #openstack-infra | 14:37 | |
| tosky | mordred: oh, it was for the sahara-scenario test runner, there is a proposed patch which replaces one of the yaml files, the one with credentials, with OS_* variables | 14:37 |
| mordred | oh - well | 14:38 |
| mordred | you should do neither of those things | 14:38 |
| mordred | you should replace the yaml file that has credentials with a clouds.yaml file | 14:38 |
| tosky | mordred: so I was trying to see if we could not step into anyone else toes | 14:38 |
| mordred | and you should rework sahara-scenario to use os-client-config to construct things | 14:38 |
| mordred | since devstack already writes out a clouds.yaml | 14:38 |
| mordred | can you link me the patch? I can go make some suggestoins | 14:39 |
| *** ddieterly[away] is now known as ddieterly | 14:39 | |
| tosky | mordred: sure, this one: https://review.openstack.org/#/c/297757/ | 14:39 |
| mordred | thanks | 14:39 |
| mordred | ~ | 14:39 |
| tosky | mordred: please note that some yaml files are required for the test specific configuration, but it's all documented in the proper README (or it should be :) | 14:39 |
| tosky | thank you! | 14:39 |
| *** Guest11043 is now known as redrobot | 14:40 | |
| mordred | tosky: WOW there is so much in this patch which has reimplemented things we do in os-client-config or shade | 14:42 |
| *** Qiming has quit IRC | 14:42 | |
| *** dmsimard has quit IRC | 14:42 | |
| *** Qiming has joined #openstack-infra | 14:42 | |
| mordred | tosky: which is not me picking on you guys - more that it's sad that openstack makes us all implement things like "auto-assign-floating-ip" | 14:42 |
| *** dmsimard has joined #openstack-infra | 14:42 | |
| mordred | in any case, current os-client-config should allow you to express all of the things you currently express in your credentials.yaml | 14:43 |
| tosky | mordred: auto-assign-floating-ip was there for a long time | 14:43 |
| mordred | yah | 14:43 |
| mordred | it's a sad need | 14:43 |
| *** zz_ja has joined #openstack-infra | 14:43 | |
| mordred | openstack should not make us know that | 14:43 |
| tosky | and I understand the pain of going back and rewriting code, removing old cruft | 14:43 |
| mordred | I'm more amused that your solution and our solution look so similar | 14:44 |
| mordred | it's almost lke the problem is real :) | 14:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Translations: Simplify https://review.openstack.org/307237 | 14:45 |
| tosky | eheh | 14:46 |
| clarkb | mordred no no everyone wants sdn so that they can hand cure their bacon I mean networks to achieve the perfect flavor | 14:46 |
| clarkb | now I want bacon for breakfast | 14:46 |
| mordred | clarkb: oh. right. I'm sorry. I forgot that nobody wants to just eat bacon | 14:46 |
| derekh_afk | mordred: I'm seeing a lot of unused floating ip's building up since friday on the tripleo rack, I've had to delete them several times, to stop the pool being exhausted, any idea if it related to something your doing? | 14:47 |
| *** mrunge has quit IRC | 14:47 | |
| *** erlon_ has joined #openstack-infra | 14:48 | |
| *** ajmiller_ has joined #openstack-infra | 14:48 | |
| mordred | derekh_afk: it's entirely possible that there is a bug in the new code - the tripleo cloud is on an interesting code path at the moment due to the neutron/nova interaction bug - but I would not expect a constant leak | 14:48 |
| mordred | derekh_afk: I'll look in to it today | 14:48 |
| *** jlanoux has quit IRC | 14:48 | |
| derekh_afk | mordred: thanks | 14:49 |
| derekh_afk | mordred: also I'm seeing errors like this in the logs nova.api.openstack.compute.contrib.floating_ips NeutronClientException: 409-{u'NeutronError': {u'message': u'Cannot associate floating IP 66.187.229.97 (99426001-dd18-46f2-b6ea-3ea492fcdd3f) with port 7ac6c9da-01fa-4cf7-a6ba-644983aec94d using fixed IP 10.2.8.251, as that fixed IP already has a floating IP on external network d982c5c4-03f2-49f0-8f64-b4020e260ba2.', u'type': u'FloatingIPPort | 14:49 |
| derekh_afk | AlreadyAssociated', u'detail': u''}} | 14:49 |
| mordred | derekh_afk: that should be fixed now - are you seeing those recently? | 14:50 |
| derekh_afk | mordred: looking | 14:50 |
| *** mrunge has joined #openstack-infra | 14:50 | |
| *** Guest64003 is now known as melwitt | 14:51 | |
| derekh_afk | mordred: yes, still seeing them | 14:51 |
| mordred | derekh_afk: hrm. yes - I see this: 2016-04-18 14:46:48,847 DEBUG nodepool.ProviderManager: Manager tripleo-test-cloud-rh1 running task NeutronFloatingIPCreate (queue: 11) | 14:51 |
| mordred | which I should not be seeing | 14:51 |
| *** ajmiller has quit IRC | 14:52 | |
| mordred | since creating floating ips on tripleo cloud with neutron is bad mmmkay? | 14:52 |
| jeblair | krotscheck: well, since we want to vos release only when the mirror is in a stable state, i think it's better if we can run the mirror script as a cron (that way we can test the return code to know whether it finished) | 14:52 |
| mordred | oh. heh | 14:53 |
| *** ajmiller has joined #openstack-infra | 14:53 | |
| mordred | derekh_afk: it's possible that I have not restarted nodepool since making the fix for that in shade | 14:53 |
| mordred | that would have been silly of me | 14:53 |
| derekh_afk | mordred: ;-) | 14:53 |
| krotscheck | jeblair: Got it. So we'll leave it as is. | 14:53 |
| krotscheck | jeblair: How's the latest vos-release doing? | 14:53 |
| derekh_afk | mordred: I'm deleting unused floating IP's again, when nodpool is restarted next just let me know and I'll see if IP's are still leaking | 14:54 |
| mordred | cool. | 14:54 |
| mordred | infra-root: anybody have any problems with me restarting nodepool to pick up the trieplo cloud change? | 14:54 |
| *** zhurong has quit IRC | 14:55 | |
| fungi | mordred: seems prudent to me | 14:55 |
| *** elo has joined #openstack-infra | 14:56 | |
| *** ajmiller_ has quit IRC | 14:56 | |
| mordred | k. I have done so and am tailing the debug log with: tail -f debug.log | grep -i tripleo.*float | 14:56 |
| *** sshnaidm has quit IRC | 14:58 | |
| anteaya | can someone remove patchbot from -meeting please? | 14:58 |
| *** ociuhandu has joined #openstack-infra | 14:59 | |
| jeblair | krotscheck: though, as is seems to be running as a daemon -- did whoever ran it not run it the way the cron does or what? | 14:59 |
| *** ddieterly is now known as ddieterly[away] | 15:00 | |
| jeblair | krotscheck: vos release is still in progress | 15:00 |
| krotscheck | jeblair: I'm pretty certain that the default execution for that package is as a daemon. let me confirm. | 15:00 |
| jeblair | krotscheck: the last one took 33 hours | 15:02 |
| krotscheck | jeblair: The one on... wednesday I think? | 15:03 |
| derekh_afk | mordred: number of unused floating IP's has gone from 4 up to 24 since your restart, is that expected (i.e. are you keeping a pool of them)? or are they leaking? | 15:03 |
| mordred | derekh_afk: nope. that means they're leaking :( | 15:03 |
| *** EricGonc_ has joined #openstack-infra | 15:04 | |
| *** oanson has joined #openstack-infra | 15:04 | |
| krotscheck | jeblair: Yep, the default way for it to run is as a daemon. There's no update-then-exit flag. | 15:04 |
| mordred | derekh_afk: ok. I'm seeing a weirdness in the logs but I think I know where to look | 15:04 |
| *** sridhar_ram has quit IRC | 15:05 | |
| derekh_afk | mordred: looking through the cloud logs also for clues | 15:05 |
| *** pfallenop has joined #openstack-infra | 15:05 | |
| mordred | derekh_afk: I tink it's a shade bug | 15:05 |
| mordred | derekh_afk: we should not be makin neutorn calls to you at all | 15:05 |
| *** oanson has quit IRC | 15:05 | |
| mordred | derekh_afk: but we're making nova and neutron call | 15:05 |
| derekh_afk | mordred: ahh ok | 15:06 |
| *** doug-fish has joined #openstack-infra | 15:06 | |
| mordred | so I don't know exactly what the issue is yet - but I know where to look | 15:06 |
| derekh_afk | ack | 15:06 |
| *** ddieterly[away] is now known as ddieterly | 15:06 | |
| *** cinerama` is now known as cinerama | 15:07 | |
| *** dtantsur|brb is now known as dtantsur | 15:07 | |
| jeblair | krotscheck: yeah. i'd estimate this one has a bit over 2 more days to run till it's caught up. | 15:07 |
| *** EricGonczer_ has quit IRC | 15:08 | |
| krotscheck | jeblair: Is that the final one, or will we have to run it again? | 15:08 |
| anteaya | can someone please remove patchbot from -meeting? | 15:08 |
| jeblair | krotscheck: this one is actually the bulk of the mirror at this point | 15:08 |
| *** asettle has joined #openstack-infra | 15:08 | |
| krotscheck | jeblair: So we'll have to run it again, but the last one's likely to be faster. | 15:09 |
| *** abregman has joined #openstack-infra | 15:09 | |
| jeblair | krotscheck: yeah, the next one will be syncing a few days worth of updates, then the one after, hopefully only a few hours | 15:09 |
| jeblair | krotscheck: in the mean time, i think we need to figure out how to get the update script to stop :/ | 15:10 |
| krotscheck | jeblair: Shouldn't the timeout be sufficient? | 15:10 |
| fungi | did anyone get an upper bound on the disk space requirement for a full npm mirror? | 15:10 |
| fungi | that's a lot of isInteger() | 15:11 |
| *** julim has quit IRC | 15:11 | |
| krotscheck | jeblair: I can talk to yahoo to see if I can get a --single-run flag in there. | 15:11 |
| jeblair | anteaya: done | 15:12 |
| anteaya | jeblair: thank you | 15:12 |
| jeblair | krotscheck: the timeout will leave the mirror in an inconsistent state -- we only want to release when it is consistent | 15:12 |
| krotscheck | So that's a no. | 15:12 |
| krotscheck | Alrightey, I'll put that on my afternoon docket. | 15:12 |
| jeblair | krotscheck: thanks! | 15:12 |
| *** tonytan4ever has joined #openstack-infra | 15:13 | |
| *** asettle has quit IRC | 15:13 | |
| tdurakov | hey folks, is it ok that https://jenkins02.openstack.org/ is not available, either jenkins01.openstack.org | 15:14 |
| *** julim has joined #openstack-infra | 15:15 | |
| anteaya | tdurakov: we made jenkins opaque | 15:16 |
| fungi | tdurakov: http://lists.openstack.org/pipermail/openstack-dev/2016-April/092482.html | 15:16 |
| anteaya | fungi: that's the one I was looking for, thanks | 15:16 |
| tdurakov | anteaya, fungi thank you for info | 15:17 |
| anteaya | welcome | 15:17 |
| *** armax has joined #openstack-infra | 15:17 | |
| *** amitgandhinz has quit IRC | 15:18 | |
| *** jlanoux has joined #openstack-infra | 15:18 | |
| *** abregman is now known as abregman|afk | 15:19 | |
| *** tonytan_brb has joined #openstack-infra | 15:19 | |
| *** tonytan_brb is now known as tonytan4ever_ | 15:19 | |
| *** links has joined #openstack-infra | 15:19 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add new virtualbmc project under ironic's umbrella https://review.openstack.org/307323 | 15:20 |
| *** cody-somerville has joined #openstack-infra | 15:22 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add new virtualbmc project under ironic's umbrella https://review.openstack.org/307323 | 15:22 |
| *** thiagop has joined #openstack-infra | 15:23 | |
| *** tonytan4ever has quit IRC | 15:23 | |
| *** inc0 has joined #openstack-infra | 15:23 | |
| *** _ody has joined #openstack-infra | 15:23 | |
| *** tonytan4ever_ has quit IRC | 15:24 | |
| *** EricGonc_ has quit IRC | 15:24 | |
| *** vhosakot has joined #openstack-infra | 15:25 | |
| *** mtanino has joined #openstack-infra | 15:25 | |
| *** armax has quit IRC | 15:26 | |
| *** EricGonczer_ has joined #openstack-infra | 15:27 | |
| *** Guest72772 is now known as jgriffith | 15:28 | |
| *** armax has joined #openstack-infra | 15:29 | |
| *** ddieterly is now known as ddieterly[away] | 15:29 | |
| *** abregman|afk is now known as abregman | 15:30 | |
| *** matrohon has quit IRC | 15:31 | |
| *** armax has quit IRC | 15:33 | |
| *** amrith is now known as _amrith_ | 15:33 | |
| *** reed has quit IRC | 15:34 | |
| *** reed has joined #openstack-infra | 15:34 | |
| *** woodster_ has joined #openstack-infra | 15:34 | |
| *** roxanaghe has joined #openstack-infra | 15:34 | |
| *** e0ne has quit IRC | 15:34 | |
| mordred | derekh_afk: the problem is that I cannot read git shas | 15:35 |
| fungi | oh wow, i see a rackspace ticket here stating that they rebooted the host where review.openstack.org was running on friday | 15:35 |
| anteaya | fungi: wow | 15:35 |
| *** Guest15381 is now known as medberry | 15:35 | |
| *** medberry is now known as med_ | 15:35 | |
| anteaya | so far in backscroll I don't see gerrit complaints | 15:35 |
| fungi | uptime of review.o.o is 3 days 7 hours 24 minutes | 15:36 |
| fungi | last says "reboot system boot Fri Apr 15 08:12 - 15:36 (3+07:24) 0.0.0.0" | 15:36 |
| *** EricGonczer_ has quit IRC | 15:36 | |
| anteaya | wow | 15:37 |
| mordred | derekh_afk: there were two different shas 2 commits appart that started with 59 - so when I checked to see if we were running the right one, I got a false-positive :) | 15:37 |
| *** infra-red has quit IRC | 15:37 | |
| clarkb | anteaya: fungi likely the cause of zuul and review not communicating. yolanda dealt with it | 15:37 |
| *** EricGonczer_ has joined #openstack-infra | 15:38 | |
| anteaya | yolanda is so on the ball that way | 15:38 |
| mordred | derekh_afk: I have restarted nodepool again _actually_ with the new change | 15:39 |
| mordred | derekh_afk: if we clear the leaked, I expect it to not leak anymore | 15:39 |
| *** dizquierdo has quit IRC | 15:39 | |
| *** jklare has quit IRC | 15:39 | |
| *** vincentll has quit IRC | 15:40 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add new virtualbmc project under ironic's umbrella https://review.openstack.org/307323 | 15:41 |
| derekh_afk | mordred: ok, clearing it now | 15:41 |
| derekh_afk | mordred: will let you know in a few minutes | 15:41 |
| *** jklare has joined #openstack-infra | 15:41 | |
| *** catherineD has left #openstack-infra | 15:43 | |
| *** gyee has joined #openstack-infra | 15:43 | |
| *** mrodden has quit IRC | 15:43 | |
| *** esikachev has joined #openstack-infra | 15:44 | |
| *** armax has joined #openstack-infra | 15:44 | |
| *** vgridnev has quit IRC | 15:44 | |
| *** mrodden has joined #openstack-infra | 15:44 | |
| *** vgridnev has joined #openstack-infra | 15:45 | |
| *** roxanaghe has quit IRC | 15:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Restore ability to run nodepoold with fakes https://review.openstack.org/231120 | 15:46 |
| *** ddieterly[away] is now known as ddieterly | 15:46 | |
| *** diablo_rojo has joined #openstack-infra | 15:46 | |
| jeblair | mordred, clarkb: ^ can you look at that now that the shade stuff has landed -- i'd like to be able to run it locally again :) | 15:46 |
| *** markmcclain has quit IRC | 15:47 | |
| *** vgridnev has quit IRC | 15:47 | |
| *** markmcclain has joined #openstack-infra | 15:47 | |
| *** ddieterly is now known as ddieterly[away] | 15:47 | |
| *** Sukhdev has joined #openstack-infra | 15:47 | |
| *** harlowja_at_home has joined #openstack-infra | 15:48 | |
| *** inc0 has quit IRC | 15:48 | |
| *** alhernan has joined #openstack-infra | 15:48 | |
| *** inc0 has joined #openstack-infra | 15:49 | |
| *** Nikolay_St has quit IRC | 15:49 | |
| mordred | jeblair: looks fine to me | 15:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Make stopping more reliable https://review.openstack.org/293605 | 15:50 |
| *** nwkarsten has quit IRC | 15:50 | |
| *** dprince has joined #openstack-infra | 15:50 | |
| jeblair | mordred, clarkb: ^ and that (which is related :) | 15:50 |
| *** nwkarsten has joined #openstack-infra | 15:50 | |
| *** lakshmiS has joined #openstack-infra | 15:50 | |
| *** arxcruz has quit IRC | 15:52 | |
| *** tonytan4ever has joined #openstack-infra | 15:53 | |
| *** sshnaidm has joined #openstack-infra | 15:53 | |
| *** apoorvad has joined #openstack-infra | 15:54 | |
| derekh_afk | mordred: floating ips no longer seem to be leaking | 15:55 |
| *** annegentle has joined #openstack-infra | 15:55 | |
| derekh_afk | mordred: thanks for looking into it | 15:56 |
| *** mikelk has quit IRC | 15:56 | |
| mordred | derekh_afk: thanks for pointing it out! I'm glad it was just a git derp :) | 15:56 |
| diablo_rojo | Hello, so I have been hitting an issue in gate-grenade-dsvm where it fails to pull stable/mitaka. I created a story for it against openstack-infra/zuul and posted the log failure in the description. Hope that helps and is correct? Here is the link: https://storyboard.openstack.org/#!/story/2000557 And here is the link to one of the patches where this is occurring: https://review.openstack.org/#/c/290885/ | 15:57 |
| nibalizer | fungi: neat! | 15:58 |
| *** nwkarste_ has joined #openstack-infra | 15:58 | |
| *** ddieterly[away] is now known as ddieterly | 15:59 | |
| *** bpokorny has joined #openstack-infra | 15:59 | |
| *** nwkarsten has quit IRC | 15:59 | |
| *** ddieterly has quit IRC | 15:59 | |
| *** tqtran has joined #openstack-infra | 16:00 | |
| *** ddieterly has joined #openstack-infra | 16:00 | |
| clarkb | diablo_rojo: can you update that bug with links to the jobs, it isn't failing to pull mitaka, it is failing to get the zuul ref which isn't always a problem | 16:00 |
| *** _amrith_ is now known as amrith | 16:00 | |
| clarkb | diablo_rojo: but without the entire job logs we will not know | 16:00 |
| *** ddieterly has quit IRC | 16:00 | |
| clarkb | diablo_rojo: ya, thats a change to cinder so the ceck queue will only have zuul refs for cinder there won't be any for zaqar so that isn't a bug | 16:01 |
| *** links has quit IRC | 16:01 | |
| diablo_rojo | clarkb: I added a link to the bottom of the description with the full logs. | 16:01 |
| *** alhernan has left #openstack-infra | 16:01 | |
| *** alhernan has joined #openstack-infra | 16:01 | |
| diablo_rojo | clarkb: Okay my bad. So I just need to recheck again? Or what do I need to do? | 16:01 |
| clarkb | diablo_rojo: you need to find out why the job failed | 16:02 |
| diablo_rojo | clarkb: Okay. | 16:02 |
| clarkb | diablo_rojo: http://logs.openstack.org/85/290885/5/check/gate-grenade-dsvm/042d35a/logs/grenade.sh.txt.gz#_2016-04-15_17_35_12_757 is the relevant portion of the logs | 16:02 |
| diablo_rojo | clarkb: Interesting. Okay thanks. | 16:03 |
| *** andreww has quit IRC | 16:03 | |
| *** andymaier has joined #openstack-infra | 16:04 | |
| *** salv-orlando has joined #openstack-infra | 16:04 | |
| *** Guest30394 has quit IRC | 16:05 | |
| *** raginbajin has quit IRC | 16:05 | |
| *** gyee has quit IRC | 16:06 | |
| *** andymaier has quit IRC | 16:06 | |
| *** salv-orl_ has quit IRC | 16:06 | |
| *** nwkarste_ has quit IRC | 16:07 | |
| clarkb | fungi: I wonder if we spin up a handful of VMs in rax-ord and start a continuous ping/traceroute (mtr) if we can see it fall over then point rax to the live packets | 16:07 |
| *** Swami has joined #openstack-infra | 16:08 | |
| clarkb | mordred: any idea why osic appears to be 100% fail to build right now? | 16:08 |
| *** tphummel has joined #openstack-infra | 16:08 | |
| clarkb | mordred: logs say because we have leaked floating IPs there too, | 16:08 |
| *** nwkarsten has joined #openstack-infra | 16:08 | |
| clarkb | mordred: any chance you are interested in cleaning those up and the ones in bluebox (but also I thought we were supposed to never leak anymore) | 16:08 |
| *** aysyd has quit IRC | 16:09 | |
| jeblair | clarkb: i'd probably write this first instance off to maybe all the stop/starts switching between the two code bases | 16:09 |
| *** skolekonov has quit IRC | 16:10 | |
| jeblair | clarkb: after we clean these up, if it happens again, then we know it's shade | 16:10 |
| *** nadya has quit IRC | 16:10 | |
| *** raginbajin has joined #openstack-infra | 16:10 | |
| openstackgerrit | Merged openstack/requirements: Bump python-openstacksdk to 0.8.5 https://review.openstack.org/305430 | 16:11 |
| * clarkb went ahead and gathered the first state of fips in osci | 16:12 | |
| clarkb | and bluebox | 16:13 |
| *** Qiming has quit IRC | 16:14 | |
| mordred | yah. I agree with jeblair | 16:14 |
| nibalizer | clarkb: howabout i do one of those | 16:14 |
| nibalizer | clarkb: that way i can learn | 16:14 |
| clarkb | nibalizer: sure, you want to do bluebox? | 16:14 |
| clarkb | nibalizer: venv/bin/neutron floatingip-list | grep -v '10\.220\.' | sed -e '1,3d' -e '$d' | cut -d'|' -f 2 > first-fips-bb.txt is what I run to get the first state of unattached fips | 16:15 |
| clarkb | not the 10.220 address prefix is unique to bluebox | 16:15 |
| clarkb | s/not/note/ | 16:15 |
| *** purp has quit IRC | 16:17 | |
| clarkb | nibalizer: after 5 minutes or so you want to rerun that command and save to a second file then look at the diff between the two listings. Any fips that show up in both are leaked | 16:17 |
| nibalizer | sure | 16:17 |
| nibalizer | so becuase this is neutron i shouldn't even be trying to openstackclient ya? | 16:17 |
| *** piet_ has joined #openstack-infra | 16:17 | |
| clarkb | you could update the thing to use openstackclient, the seds and cuts may change | 16:17 |
| *** purp has joined #openstack-infra | 16:17 | |
| mordred | also, fwiw, [cloud.delete_floating_ip(ip.id) for ip in cloud.list_floating_ips() if ip.attached==False] | 16:17 |
| mordred | that is a list comprehension that deletes unattached fips | 16:17 |
| mordred | in shade | 16:17 |
| *** infra-red has joined #openstack-infra | 16:17 | |
| clarkb | mordred: thta assumes that the fips won't attach in the future, which they may stil lwhile we transition | 16:17 |
| mordred | on osic and bluebox we do not need to do the 2-step dance | 16:17 |
| clarkb | (but probably not) | 16:17 |
| mordred | we don't attach unattached fips anymore | 16:17 |
| *** trown is now known as trown|lunch | 16:18 | |
| clarkb | oh in that case just do venv/bin/neutron floatingip-list | grep -v '10\.220\.' | sed -e '1,3d' -e '$d' | cut -d'|' -f 2 | xargs -n 1 -P 1 neutron floating-ip-delete | 16:18 |
| mordred | or that python above :) | 16:18 |
| *** Kiall has quit IRC | 16:18 | |
| rcarrillocruz | wow | 16:19 |
| clarkb | note some clients its floatingip-delete and others its floating-ip-delete | 16:19 |
| rcarrillocruz | my change from this morning passed tests now | 16:19 |
| clarkb | and I never know what is what | 16:19 |
| mordred | cloud = shade.openstack_cloud(cloud='bluebox-sjc1') | 16:19 |
| rcarrillocruz | is ovh back yet? | 16:19 |
| mordred | [cloud.delete_floating_ip(ip.id) for ip in cloud.list_floating_ips() if ip.attached==False] | 16:19 |
| mordred | cloud = shade.openstack_cloud(cloud='osic-cloud1') | 16:19 |
| mordred | [cloud.delete_floating_ip(ip.id) for ip in cloud.list_floating_ips() if ip.attached==False] | 16:19 |
| mordred | that will do the whole thing | 16:19 |
| rcarrillocruz | mugsie, mordred : thx for the reviews, i'll have a look in a bit | 16:19 |
| mordred | rcarrillocruz: if you can rework it that way just a little, we can get that landed and into today's release | 16:20 |
| rcarrillocruz | sweet | 16:21 |
| *** nwkarsten has quit IRC | 16:21 | |
| clarkb | mordred: you also need a sleep in there possibly (I have had clouds yell at me for too many requests) | 16:21 |
| *** fhubik has quit IRC | 16:21 | |
| *** pcaruana has quit IRC | 16:21 | |
| clarkb | whcih is why xargs does a -P 1 instead of -P bignumber | 16:22 |
| *** scheuran has quit IRC | 16:22 | |
| mordred | yah. I haven't had any issues - that's how I clean them. however, it does bring up the point that the nodepool rate-limited taskmanager is useful for more than nodepool | 16:22 |
| *** Kiall has joined #openstack-infra | 16:22 | |
| *** mariusv has quit IRC | 16:22 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/yaml2ical: Remove intermediate datetime.datetime().date() usage https://review.openstack.org/307355 | 16:22 |
| *** amrith is now known as _amrith_ | 16:23 | |
| *** aysyd has joined #openstack-infra | 16:23 | |
| *** mariusv has joined #openstack-infra | 16:23 | |
| clarkb | mordred: it used to happen a lot with hpcloud | 16:24 |
| clarkb | even with serialized delete requests that take a couple seconds each that was too much with nodepool running in the background | 16:24 |
| *** markmcclain has quit IRC | 16:25 | |
| clarkb | heh and now osic hosts build but fail resolving git.openstack.org | 16:25 |
| mordred | yah | 16:25 |
| *** markmcclain has joined #openstack-infra | 16:25 | |
| *** nwkarsten has joined #openstack-infra | 16:25 | |
| mordred | yah - I've been seeing that sporadically in osic builds as I watch logs | 16:26 |
| nibalizer | dns is hard | 16:26 |
| mordred | nibalizer: it's so not | 16:26 |
| clarkb | that is on centos-7, I wonder if sshd is up before unbound | 16:26 |
| *** harlowja_at_home has quit IRC | 16:26 | |
| clarkb | because systemd | 16:26 |
| *** piet_ has quit IRC | 16:26 | |
| clarkb | (systemd should bind the ssh socket and do socket activation for that service iirc, which menas nodepool can get into the host asap, and if unbound isn't running yet resolving to localhost will fail) | 16:27 |
| nibalizer | clarkb: ok saved the first list of bleubox | 16:27 |
| clarkb | but you would think unbound would also be socket activated in that case | 16:27 |
| nibalizer | will grab a second list in a few minutes | 16:27 |
| *** Kiall has quit IRC | 16:27 | |
| pabelanger | clarkb: nibalizer: Wow, bluebox leaked again? I ran the commands this morning | 16:29 |
| *** tosky has quit IRC | 16:29 | |
| clarkb | pabelanger: I think mordred has been surgerying | 16:29 |
| clarkb | mordred: that said, if everytime we restart nodepool we leak all the fips that would be unhappyness | 16:29 |
| *** vgridnev has joined #openstack-infra | 16:29 | |
| pabelanger | Ah | 16:29 |
| *** tonytan_brb has joined #openstack-infra | 16:29 | |
| *** Yarboa has joined #openstack-infra | 16:29 | |
| clarkb | ya these git.o.o resolution errors appear to only be on centos7 | 16:30 |
| *** abregman is now known as abregman|afk | 16:30 | |
| clarkb | implying image/boot issues not cloud issues | 16:30 |
| mordred | clarkb: yah. if we are still leaking around restarts, it's a thing we should fix | 16:30 |
| zaro | morning | 16:30 |
| mordred | clarkb: I've been avoiding fixing it so far just because I knew the fip story was changing with shade-nodepool | 16:31 |
| mordred | but if we're still leaking around restarts, it's definitely worth sorting out | 16:31 |
| *** andreww has joined #openstack-infra | 16:31 | |
| fungi | any idea if it's a national holiday in france or something? it's been more than a couple hours since i opened a support ticket with ovh about our little problem there, and it hasn't been triaged yet. how responsive are they usually? given we basically have our largest two providers offline right now, this is pretty crippling | 16:31 |
| *** genunix has left #openstack-infra | 16:31 | |
| jeblair | clarkb: my suggestion about restarting was mostly related to the fact that we kept changing codebases; i don't know that we leak on restart with shade | 16:31 |
| clarkb | fungi: they are typically active during GMT ish work hours in my experience | 16:32 |
| clarkb | jeblair: gotcha | 16:32 |
| fungi | clarkb: good to know. thankfully i got this in early in their afternoon | 16:32 |
| *** tonytan4ever has quit IRC | 16:32 | |
| fungi | still waiting to see whether we're going to need to rebuild the mirrors there from scratch or if they've just "disabled" us | 16:33 |
| therve | Sorry if I missed a notice, but is there something going on with the build queue? | 16:33 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Fix updating automatic worklists with no filters https://review.openstack.org/307362 | 16:33 |
| clarkb | fungi: I can dig up email addresses from previous contacts if you want them | 16:34 |
| mpaolino | Hello dtroyer, ianw, all, I am working to port an old devstack patch (https://review.openstack.org/#/c/108714/) to a devstack plugin. I am wondering how should I present this to the community e.g., send the new code on top of the abandoned change 108714, send everything as a brand new contribution, etc | 16:34 |
| fungi | therve: we're just low on resources today, between having rackspace ord offline because of network issues there and all our ovh regions offline because it looks like one of our accounts got deleted/suspended | 16:34 |
| therve | fungi, OK, thanks | 16:34 |
| anteaya | fungi: according to wikipedia they could be celebrating the 1797 – The Battle of Neuwied: French victory against the Austrians. | 16:35 |
| *** tosky has joined #openstack-infra | 16:35 | |
| odyssey4me | pabelanger if I wish to switch ansible-role-functional-jobs-centos7-nv to be an experimental job for OSA, how can I go about doing that? I could do it globally, but I think that'll affect other roles | 16:35 |
| pabelanger | odyssey4me: you can remove the ansible-role-functional-jobs-centos7-nv template from the project and just add an experimental pipleline for said project | 16:37 |
| *** abregman|afk has quit IRC | 16:37 | |
| therve | fungi, And it's not a national holiday :) | 16:37 |
| *** ihrachys has quit IRC | 16:37 | |
| fungi | therve: thanks! | 16:37 |
| odyssey4me | pabelanger so just remove the template and add 'gate-{name}-ansible-func-centos-7-nv' to the experimental pipeline for the project ? | 16:38 |
| pabelanger | odyssey4me: yup | 16:38 |
| odyssey4me | pabelanger ok cool, thanks - patch incoming | 16:38 |
| jeblair | fungi, clarkb: we have email contacts -- has ovh been emailed? | 16:38 |
| *** Kiall has joined #openstack-infra | 16:38 | |
| clarkb | jeblair: I just passed the email contacts on to fungi. I have not emailed them | 16:38 |
| jeblair | kk | 16:38 |
| *** jistr has quit IRC | 16:39 | |
| fungi | jeblair: oh, i missed those in the password list if that's where they are. i opened a support ticket instead | 16:39 |
| *** tqtran has quit IRC | 16:40 | |
| fungi | no idea whether they've been responsive to support tickets in the past. should i just go ahead and escalate via e-mail immediately, or give them some time to triage it? | 16:40 |
| *** derekh_afk is now known as derekh | 16:41 | |
| anteaya | +1 escalate | 16:42 |
| clarkb | fungi: for the ovh fails, it looks like the mirror hosts went away? were we catching that in the jobs or in the ready script? if not the ready script we should probably update it to check connectivity to the region local mirror | 16:43 |
| *** rcernin has quit IRC | 16:43 | |
| jeblair | fungi: i think i forgot to add them to the pwlist | 16:43 |
| *** Kiall has quit IRC | 16:43 | |
| *** infra-re_ has joined #openstack-infra | 16:43 | |
| clarkb | well thats interesting. At least on rax we are not using unbound + google | 16:44 |
| clarkb | /etc/resolve.conf is using local dns | 16:44 |
| jeblair | for crying out loud | 16:44 |
| *** jlk` is now known as jlk | 16:44 | |
| clarkb | jeblair: this was on a centos-7 host picked at semi random | 16:44 |
| fungi | clarkb: no idea how it manifested, i started from reports that the mirrors were unreachable and our keystone creds stopped working. in the ovh web dashboard that project is also completely missing now (except as an option in the "service" drop-down in their ticket creation form) | 16:44 |
| nibalizer | clarkb: cleaning them out now | 16:44 |
| *** Kiall has joined #openstack-infra | 16:44 | |
| jeblair | clarkb: by local dns, do you mean local recursive unbound or rax local dns? | 16:44 |
| clarkb | jeblair: the rax local dns | 16:45 |
| anteaya | clarkb: yes yolanda was unable to ssh into the ovh mirrors in both regions this morning | 16:45 |
| clarkb | eg nameserver 69.20.0.196 | 16:45 |
| yolanda | yes, mirror was down | 16:45 |
| yolanda | no ping, no ssh | 16:45 |
| jeblair | clarkb: is this still snapshot? | 16:45 |
| jeblair | clarkb: or dib? | 16:45 |
| clarkb | jeblair: no it is centos-minimal + dib | 16:45 |
| clarkb | yolanda: were jobs failing or the ready scripts? | 16:45 |
| yolanda | clarkb, the jobs | 16:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Move OpenStack-Ansible CentOS7 jobs to experimental queue https://review.openstack.org/307369 | 16:46 |
| jeblair | clarkb: what about ubuntu hosts? | 16:46 |
| *** infra-red has quit IRC | 16:46 | |
| odyssey4me | pabelanger ^ please review when you have a gap | 16:46 |
| *** Nikolay_St has joined #openstack-infra | 16:46 | |
| *** Yarboa has quit IRC | 16:47 | |
| clarkb | jeblair: same situation | 16:47 |
| clarkb | jeblair: a quick grep of the centos-7 build log shows unbound being installed | 16:47 |
| clarkb | jeblair: if I had to guess, glean is fighting us there | 16:47 |
| jeblair | mordred: can you suggest someone to track down that bug? | 16:48 |
| *** derekh has quit IRC | 16:48 | |
| clarkb | jeblair: on osic we appear to use the host local resolver | 16:48 |
| clarkb | (on ubuntu-trusty + osic) | 16:48 |
| clarkb | unbound is running on the rax host | 16:48 |
| clarkb | but resolv.conf isn't using it | 16:49 |
| mordred | jeblair: some combination of greghaynes, clarkb and I probably | 16:51 |
| clarkb | best I can tell the images are built properly with unbound configured and running on boot. We are just ignoring it and slapping rax config over /etc/resolv.conf | 16:51 |
| pabelanger | odyssey4me: looks right | 16:51 |
| jeblair | mordred: a quick look at glean suggests that it blindly writes whatever is in configdrive to /etc/resolv.conf | 16:52 |
| *** vgridnev_ has joined #openstack-infra | 16:52 | |
| jeblair | mordred: in our images, we never want anything to touch /etc/resolv.conf | 16:52 |
| zigo | Hi there! I'm back from a week of holidays. We seem to be stuck with the debian package build with this: http://logs.openstack.org/82/302382/5/check/pkgdeb-build-openstack-pkg-tools/4312526/console.html | 16:53 |
| *** pc_m has quit IRC | 16:53 | |
| mordred | jeblair: perhaps we should only overwrite resolv.conf if one is not there? | 16:53 |
| zigo | ie: 2016-04-18 09:52:34.852 | User jenkins is not currently a member of group sbuild, but is in the system group database | 16:53 |
| zigo | 2016-04-18 09:52:34.852 | You need to log in again to gain sbuild group privileges | 16:53 |
| odyssey4me | pabelanger ack, thanks | 16:53 |
| zigo | I tried a bunch of things, but none works. | 16:53 |
| zigo | Anyone got an idea on how to make the build process login again? | 16:53 |
| *** annegentle has quit IRC | 16:53 | |
| *** sridhar_ram has joined #openstack-infra | 16:54 | |
| clarkb | zigo: start a new login shell? | 16:54 |
| *** davideagnello has joined #openstack-infra | 16:54 | |
| zigo | clarkb: Yup, something like that. | 16:54 |
| clarkb | zigo: bash --login | 16:54 |
| zigo | clarkb: Before a command? | 16:54 |
| zigo | bash --login <something> ? | 16:54 |
| clarkb | ya, or feed it a heredoc or something | 16:55 |
| zigo | Ok, trying... | 16:55 |
| *** pc_m has joined #openstack-infra | 16:55 | |
| jeblair | mordred: i kind of suspect a system will always have a resolv.conf, one way or another... | 16:55 |
| fungi | clarkb: i played around with it a little too... not sure how to hand over the remaining commands to the new login as pretty much everything which can accomplish it results in a subshell, so probably the script will need to be broken up into more than one script | 16:55 |
| fungi | heredoc would certainly be one way to accomplish it | 16:55 |
| pabelanger | over the top idea, use ansible to run commands over localhost | 16:55 |
| fungi | yep, ansible would also accomplish that | 16:56 |
| zigo | clarkb: My other idea was to write an sbuild DIB element. Do you think that's a good idea? | 16:57 |
| clarkb | I am going to manually boot a centos-7 image in osic to see if I can find it racing in the logs or similar | 16:57 |
| zigo | This way, it'd be ready to go... | 16:57 |
| clarkb | zigo: no I do not htink we should rely on our images to set the correct groups for the jenkins user | 16:57 |
| *** sputnik13 has joined #openstack-infra | 16:57 | |
| fungi | here's an option, if slightly hacky... group names can share a gid | 16:58 |
| clarkb | jenkins should be the entrypoint to a fairly simple but useable system, that you then do fancy stuff on | 16:58 |
| zigo | clarkb: That's not *only* specific to this particular issue, I also would like sbuild to be ready by default. | 16:58 |
| fungi | why not make the sbuild gid == to the jenkins gid? | 16:58 |
| clarkb | this means packaging should work, root should be available, and any region local data provided | 16:58 |
| openstackgerrit | Vasyl Saienko proposed openstack/requirements: Block broken netmiko release 0.4.3 https://review.openstack.org/306206 | 16:59 |
| cloudnull | clarkb mordred -- idk if things are still being angry in the osic but if there's anything i can do to help let me know. | 16:59 |
| fungi | yeah, pre-caching things to make setup go more quickly in a job is encouraged, but we shouldn't move setup for jobs into the image building process | 16:59 |
| cloudnull | i have access to the env and i'd be happy to jump in if its something on the infra side that needs fixerating | 17:00 |
| clarkb | cloudnull: I think it is our images, from what I can tell dns resolution is working there on systems with different boot processes | 17:00 |
| cloudnull | ok | 17:00 |
| jeblair | mordred: can we pass options to glean? | 17:00 |
| zigo | fungi: So, it's your view that preinstalling sbuild is out of scope for the image? | 17:00 |
| cloudnull | just wanted to reach out just in case | 17:00 |
| mordred | jeblair: no, not currently | 17:00 |
| odyssey4me | clarkb the image worked before - or was there a change in the image recently? | 17:00 |
| anteaya | cloudnull: thank you for doing so | 17:01 |
| clarkb | cloudnull: we install a local caching resolver (unbound) and I have a feeling it is not starting fast enough to work for our ssh connections | 17:01 |
| *** psanchez has quit IRC | 17:01 | |
| jeblair | mordred: cause i think what we want is something like "in the context of our image builds, never write the resolv.conf file" | 17:01 |
| cloudnull | anteaya: anytime :) | 17:01 |
| mordred | yah. I think we're at the point where we need to do that | 17:01 |
| anteaya | cloudnull: thanks :) | 17:01 |
| *** psanchez has joined #openstack-infra | 17:01 | |
| fungi | zigo: preinstalling is out of scope, but caching packages onto the image so they can be installed more rapidly during the job seems fine | 17:01 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Ignore bash errors when posting metrics https://review.openstack.org/307374 | 17:01 |
| clarkb | odyssey4me: we rebuild images daily so could be new image, or similar | 17:01 |
| jeblair | mordred: this feels like a thing where we want one behavior in nodepool images, and another, in, say, infra-cloud images | 17:01 |
| mordred | jeblair: I cannot come up with a way to infer whether one should write or not write a resolv.conf when the cloud provides one | 17:01 |
| jeblair | yolanda: you've worked on glean -- do you want to add that feature? | 17:02 |
| zigo | fungi: How about the PGP key for jenkins, so that it can sign package? Any idea on how it shall be done? (ie: /home/jenkins/.gunpg/puring.gpg) | 17:02 |
| jeblair | yolanda: basically, we need to be able to tell glean that when it runs on our nodepool images, it should not update resolv.conf | 17:02 |
| zigo | fungi: Maybe we should get signing completely out of the build process, for security reasons? | 17:03 |
| fungi | zigo: that will need to be done on a special-purpose worker. we can't put sensitive key material onto single-use slaves right now | 17:03 |
| zigo | (and get it signed elsewhere...) | 17:03 |
| zigo | Right. | 17:03 |
| zigo | Exactly my thoughts too. | 17:03 |
| *** rossella_s has quit IRC | 17:03 | |
| fungi | zigo: right, that's how we do it now. one job to build things, another job which runs on a very constrained worker to sign them | 17:03 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Make the reviewday config file pluggable https://review.openstack.org/307377 | 17:03 |
| *** rossella_s has joined #openstack-infra | 17:04 | |
| zigo | fungi: So you have all of that in place already? | 17:04 |
| fungi | or to upload them to a publication channel (pypi, maven, npm) or whatever | 17:04 |
| *** baoli_ has quit IRC | 17:04 | |
| fungi | zigo: actually on the signing, no. i have a spec i didn't get time to implement this cycle but hopefully soon after the summit if things calm down at all http://specs.openstack.org/openstack-infra/infra-specs/specs/artifact-signing.html | 17:04 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Remove 'Subscriptions' button from header dropdown https://review.openstack.org/294579 | 17:05 |
| zigo | Cheers. | 17:05 |
| fungi | zigo: but it follows a similar pattern to how we do uploads to expertal services already | 17:05 |
| fungi | external | 17:05 |
| fungi | same separation of security concerns | 17:05 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add support for Designate zones https://review.openstack.org/302770 | 17:06 |
| *** alhernan has quit IRC | 17:07 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again https://review.openstack.org/288915 | 17:08 |
| clarkb | fungi: mordred is it possible that this glean issue would explain the rax-ord connectivity problems if rax dns in ord is less reliable | 17:09 |
| *** amitgandhinz has joined #openstack-infra | 17:09 | |
| *** tqtran has joined #openstack-infra | 17:10 | |
| fungi | clarkb: it was tcp timeouts and packet loss, so probably not no | 17:10 |
| *** madhu_ak has joined #openstack-infra | 17:10 | |
| *** tonytan_brb has quit IRC | 17:10 | |
| rcarrillocruz | doh | 17:11 |
| rcarrillocruz | long gate | 17:11 |
| clarkb | looks like centos 7 doesn't have working selinux for unbound http://paste.openstack.org/show/494473/ | 17:11 |
| *** harlowja has joined #openstack-infra | 17:11 | |
| anteaya | rcarrillocruz: long check | 17:12 |
| anteaya | short gate | 17:13 |
| rcarrillocruz | as in general, yeah | 17:13 |
| *** trown|lunch is now known as trown | 17:13 | |
| clarkb | do we need to reapply the selinux stuff globally? | 17:13 |
| greghaynes | mordred: jeblair we could set /etc/resolv.conf to read-only in our images | 17:13 |
| clarkb | pabelanger: ^ I seem to recall reviewing something similar where I suggested we do it to all the files instead of just one? | 17:13 |
| *** nwkarsten has quit IRC | 17:14 | |
| anteaya | well I'm just finding it confusing trying to understand what people are talking about when they call everything the gate | 17:14 |
| greghaynes | That's a pretty universal way to say 'don't write to this file' ;) | 17:14 |
| anteaya | or CI | 17:14 |
| rcarrillocruz | greghaynes: hah | 17:14 |
| greghaynes | I bet glean will explode in that case, but that's a bug | 17:14 |
| mordred | yah. putting in "if resolv.conf is not writable that's fine" would be pretty easy | 17:14 |
| mordred | and managing that more flexibly in puppet in an ongoing manner is easy | 17:15 |
| clarkb | mordred: note that as soon as you fix that we will no longer have working centos-7 images anywhere | 17:15 |
| pabelanger | clarkb: https://review.openstack.org/#/c/303430/ was my first thought | 17:15 |
| clarkb | mordred: so first step is fix the selinux thing above | 17:15 |
| mordred | kk | 17:15 |
| pabelanger | clarkb: but ianw added support into glean to fix that issue | 17:15 |
| clarkb | pabelanger: ok well it appears that /etc/unbound/forwarding.conf cannot be written to by unbound | 17:15 |
| *** Nisha has joined #openstack-infra | 17:15 | |
| clarkb | due to selinux contexts not matching | 17:16 |
| clarkb | seems similar | 17:16 |
| pabelanger | ya | 17:16 |
| clarkb | I am going to try running a restorecon on my test machine to see if that file is updated properly | 17:16 |
| pabelanger | clarkb: we should see how long touch /.autorelabel && reboot takes to restore default security contexts. | 17:17 |
| clarkb | pabelanger: http://paste.openstack.org/show/494478/ | 17:17 |
| *** vgridnev has quit IRC | 17:18 | |
| clarkb | I think ^ is a dib bug somewhere, either we need to explicitly restorecon after puppet runs or we need to have dib do it if selinux is enabled or something | 17:18 |
| pabelanger | looks like it | 17:18 |
| *** abregman has joined #openstack-infra | 17:18 | |
| *** abregman has quit IRC | 17:19 | |
| *** dizquierdo has joined #openstack-infra | 17:19 | |
| pabelanger | https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/finalise.d/99-unbound#n38 | 17:20 |
| pabelanger | so, DIB should be fine | 17:20 |
| clarkb | pabelanger: maybe we are skipping that because it isn't -x ? | 17:21 |
| clarkb | we should check the build logs | 17:21 |
| pabelanger | ya, that is what I am checking now | 17:21 |
| clarkb | and update that to be -R /etc/unbound | 17:21 |
| pabelanger | all 114MB of it | 17:21 |
| *** abregman has joined #openstack-infra | 17:21 | |
| *** nwkarsten has joined #openstack-infra | 17:22 | |
| pabelanger | 2016-04-17 16:11:12,053 INFO nodepool.image.build.centos-7: + restorecon -v /etc/unbound/forwarding.conf | 17:23 |
| pabelanger | possible we do something after dibs? | 17:23 |
| clarkb | maybe during boot? | 17:25 |
| pabelanger | checking glean to see | 17:25 |
| clarkb | I really don't think we would though, not to unbound config | 17:25 |
| pabelanger | ya, don't see anything | 17:25 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:26 | |
| *** jesusaur has joined #openstack-infra | 17:26 | |
| openstackgerrit | Cedric Brandily proposed openstack-infra/git-review: Support git without git credential https://review.openstack.org/307385 | 17:27 |
| Zara | https://storyboard.openstack.org/#!/story/2000487 ('git-review says submitting is pointless, but it's not') seems to be affecting a few people; someone's asked if work on it is planned; posting here as I'm subscribed to story but others may not be. | 17:27 |
| *** Guest30 has joined #openstack-infra | 17:27 | |
| clarkb | Zara: I just responded to that story, its not a bug | 17:27 |
| pabelanger | clarkb: I guess the issue with include files on centos7 default selinux policy. Going to google and see if they are supported by default | 17:28 |
| clarkb | Zara: but I want the people on the bug to confirm that for themselves and close it out | 17:28 |
| Zara | clarkb: heh, thanks, I missed that, then | 17:28 |
| *** Guest30 is now known as johnny1 | 17:28 | |
| clarkb | Zara: I just responded a minute or tw oago :) | 17:28 |
| *** kzaitsev_mb has quit IRC | 17:28 | |
| clarkb | pabelanger: Sorry I don't parse that | 17:28 |
| Zara | heh, I take ages to type, then! :) sorry for noise. | 17:28 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 17:28 | |
| *** stewie925 has joined #openstack-infra | 17:28 | |
| *** yamahata has joined #openstack-infra | 17:29 | |
| clarkb | pabelanger: you can ssh into 172.99.106.213 which is my test box that I ran the earlier restorecon on if it will help | 17:31 |
| clarkb | pabelanger: your user should be there and everything | 17:31 |
| *** pots has joined #openstack-infra | 17:32 | |
| *** sridhar_ram1 has joined #openstack-infra | 17:32 | |
| *** sridhar_ram has quit IRC | 17:33 | |
| pabelanger | clarkb: | 17:34 |
| pabelanger | clarkb: okay | 17:34 |
| *** Guest78460 has quit IRC | 17:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add option to assign nodes via Gearman https://review.openstack.org/307388 | 17:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add ansible launch server https://review.openstack.org/307389 | 17:34 |
| jeblair | clarkb, fungi, mordred: first Zuulv2.5 patches ^ | 17:34 |
| jeblair | that handles 'get nodes from nodepool to the zuul launcher'. does not yet handle anything past that like 'actually run jobs'. | 17:35 |
| *** avarner__ has quit IRC | 17:36 | |
| *** tosky has quit IRC | 17:36 | |
| clarkb | jeblair: it would be great if we could test that even if only by the integration test | 17:36 |
| *** dprince has quit IRC | 17:36 | |
| clarkb | should be self testing if you update the config for nodepool then add an assignment call at the end | 17:36 |
| pabelanger | clarkb: Actually, we're probably safer adding forwarding.conf into /etc/unbound/conf.d and using the default include statement over appending our own includes | 17:37 |
| jeblair | clarkb: it will wait forever for the job to complete, but that shouldn't matter in the integration test | 17:37 |
| *** EricGonczer_ has quit IRC | 17:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add option to assign nodes via Gearman https://review.openstack.org/307388 | 17:38 |
| yolanda | hi jeblair, i was out and missed the message | 17:38 |
| jeblair | clarkb: ^ | 17:38 |
| yolanda | i can work on that | 17:38 |
| clarkb | pabelanger: but that is unrelated to selinux right? | 17:39 |
| clarkb | pabelanger: I odn't think changing the config and changing the includes will fix selinux | 17:39 |
| pabelanger | clarkb: that is what I am checking. My thought is, the default selinux policy will just allow files in the conf.d directory in the default policy over us shoehorning our own configuration | 17:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Translations: Refactor https://review.openstack.org/307237 | 17:40 |
| pabelanger | I can't remember how to check the defaults ATM, just googling | 17:40 |
| clarkb | pabelanger: but the default policy also works for the existing file sas evidenced by my restorecon | 17:40 |
| pabelanger | that is true | 17:40 |
| dmsimard | Am I the only one no longer able to reach the jenkins instances ? I wanted to look at links provided by zuul as they were in progress, i.e, https://jenkins06.openstack.org/job/gate-packstack-integration-scenario002-tempest-centos-7/13/ | 17:40 |
| openstackgerrit | cathy proposed openstack-infra/irc-meetings: change service-chaining meeting time schedule https://review.openstack.org/306657 | 17:41 |
| anteaya | dmsimard: http://lists.openstack.org/pipermail/openstack-dev/2016-April/092482.html | 17:41 |
| clarkb | pabelanger: somewhere between image build and boot we aren't applying the policy to these files properly | 17:41 |
| dmsimard | anteaya: ah, thanks. | 17:41 |
| *** stevemar has quit IRC | 17:41 | |
| anteaya | welcome | 17:42 |
| pabelanger | clarkb: right, I do see unbound-control-setup generating keys and certs, but doesn't appear to touch forwarding.conf | 17:42 |
| *** stevemar has joined #openstack-infra | 17:44 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: make compare_tarball_diff.sh quieter https://review.openstack.org/307390 | 17:44 |
| pabelanger | clarkb: are we seeing the selinux failure in a specific cloud? | 17:44 |
| clarkb | pabelanger: osic in particular, I haven't checked to see if other clouds also have the same issue. Rax is not affected because we are not using unbound there currently | 17:44 |
| *** ifarkas has quit IRC | 17:44 | |
| clarkb | pabelanger: rather rax is affected but dns continues to work anyways so we don't notice | 17:45 |
| pabelanger | Apr 18 17:39:02 localhost unbound-checkconf: unbound-checkconf: no errors in /etc/unbound/unbound.conf | 17:45 |
| pabelanger | that's from a osic node just now | 17:45 |
| pabelanger | so, unbound started properly there | 17:45 |
| clarkb | thats just checkconf | 17:45 |
| *** andreww is now known as xarses | 17:45 | |
| pabelanger | which means a race I guess | 17:45 |
| clarkb | are you sure it started all the way up? | 17:45 |
| pabelanger | yup, job is running now | 17:45 |
| pabelanger | node is on hold atm | 17:46 |
| clarkb | pabelanger: and is that centos-7? | 17:46 |
| pabelanger | yes | 17:46 |
| clarkb | huh | 17:46 |
| pabelanger | clarkb: 172.99.106.111 | 17:46 |
| clarkb | seems to be a race as you say then. Possibly dns coming up before all the selinux policy data can be applied? | 17:47 |
| pabelanger | let me check that | 17:47 |
| clarkb | odd though that it didn't eventually get applied, maybe a race in applying the policy | 17:47 |
| *** jlanoux has quit IRC | 17:48 | |
| *** dtantsur is now known as dtantsur|afk | 17:49 | |
| yolanda | jeblair, so actually resolv.conf seems to have the same behaviour as it had with hostname. It was overwriting any previous content there. What i did is to merge the pre-existing content of /etc/hostname with the config from glean | 17:49 |
| yolanda | do we want to do the same for resolv.conf? | 17:49 |
| clarkb | yolanda: no, we don't want to merge the content. | 17:49 |
| clarkb | yolanda: we do not want to use cloud provider dns servers, instead we always want to use the host local caching forwarding resolver | 17:50 |
| yolanda | clarkb, so we need to totally stop glean to overwriting resolv.conf ? | 17:50 |
| clarkb | yes | 17:51 |
| clarkb | yolanda: greghaynes' suggestion was to not write to /etc/resolv.conf if it is read only | 17:51 |
| greghaynes | clarkb: ah, perfect | 17:51 |
| yolanda | clarkb cool, because it may be an use case only needed for nodepool | 17:51 |
| *** lucasagomes is now known as lucas-dinner | 17:51 | |
| yolanda | sounds like a good idea | 17:51 |
| greghaynes | Oh, you're relaying | 17:52 |
| *** dprince has joined #openstack-infra | 17:52 | |
| greghaynes | Ya | 17:52 |
| clarkb | yolanda: no I don't think it is a use case only for nodepool | 17:52 |
| clarkb | yolanda: all of our cloud servers do it too | 17:52 |
| clarkb | (and hoesntly if you are running cloud instances you should be too) | 17:52 |
| clarkb | (unless you like downtime) | 17:52 |
| jeblair | we've had trouble with the 'make resolv.conf read-only' idea in the past | 17:52 |
| jeblair | it would probably work for this | 17:52 |
| *** Hal has joined #openstack-infra | 17:53 | |
| clarkb | jeblair: ya the problem in the past was nova agent didn't respect the read only file and overwrote it anyways | 17:53 |
| clarkb | just to be helpful | 17:53 |
| *** Hal is now known as Guest30844 | 17:53 | |
| jeblair | ah, yeah that was it | 17:53 |
| *** thorst_ has quit IRC | 17:53 | |
| jeblair | anyway, we should at least handle that case, but do we think it's sufficient or should we add something more explicit on top of that? | 17:54 |
| clarkb | the way other things are handled in glean eg ssh host keys is that if the file is not there then it writes it | 17:54 |
| clarkb | if we wanted to be consistent with those things we should probably just if not file /etc/resolv.conf then write else pass | 17:55 |
| jeblair | clarkb: that assumes we can reasonably end up without an /etc/resolv.conf | 17:55 |
| jeblair | we probably can in the nodepool dib situtation | 17:56 |
| *** pvaneck has joined #openstack-infra | 17:56 | |
| yolanda | what about of the use case, of a default /etc/resolv.conf being generated, but people willing to use the glean config instead? | 17:56 |
| pabelanger | clarkb: Ya, SELinux policies look good for both images. They appear to be loaded before starting unbound | 17:57 |
| pabelanger | going to release the hold and look at another slave | 17:57 |
| clarkb | pabelanger: huh | 17:57 |
| *** EricGonczer_ has joined #openstack-infra | 17:57 | |
| clarkb | yolanda: the read only idea fits that use case better | 17:57 |
| jeblair | clarkb: however we want /etc/resolv.conf to exist in our images (we want it to point to unbound) | 17:57 |
| *** baoli has joined #openstack-infra | 17:58 | |
| jeblair | clarkb: so 'if not file' doesn't work for us | 17:58 |
| clarkb | jeblair: it would | 17:58 |
| clarkb | jeblair: the file would exist so glean would not touch it | 17:58 |
| yolanda | yep, read only will mean that you should have tweaked manually. While existing resolv.conf can be left by the operating system itself... | 17:58 |
| jeblair | clarkb: oh, sorry, i follow now | 17:58 |
| clarkb | this is how it handles every other "should I edit this file situation" | 17:58 |
| clarkb | the network interface files are the same way | 17:58 |
| jeblair | clarkb: then i think it would make glean less useful as a general cloud-init replacement | 17:58 |
| jeblair | clarkb: (i don't know how much we care about that) | 17:59 |
| clarkb | yes I personally dislike the current checks | 17:59 |
| *** electrofelix has quit IRC | 17:59 | |
| clarkb | I think the method cloud init uses is much better, it checks an instance uuid, and maintains state on disk | 17:59 |
| *** nwkarsten has quit IRC | 17:59 | |
| *** degorenko is now known as _Degorenko|afk | 18:00 | |
| clarkb | if current uuid hasn't been updated according to state file then update (granted this doesn't help the resolv.conf case, but as a general rule I like it) | 18:00 |
| *** _Degorenko|afk is now known as _degorenko|afk | 18:00 | |
| *** avarner__ has joined #openstack-infra | 18:00 | |
| *** nwkarsten has joined #openstack-infra | 18:00 | |
| *** esikachev has quit IRC | 18:01 | |
| clarkb | the current system means your image build prcess has to do a lot of work to preconfigure everything so that glean operates properly. Leading to not making new ssh host keys or not reseting dhcp on interfaces that should not dhc | 18:01 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add releasenotes jobs for networking-ovn https://review.openstack.org/307394 | 18:02 |
| yolanda | clarkb, the uuid checks idea look quite good, but this is a major change for glean | 18:03 |
| *** jaosorior has quit IRC | 18:03 | |
| *** pfallenop has quit IRC | 18:03 | |
| yolanda | i think that for the moment the read-only check is simpler, and we can do a bigger refactor on glean state checks with time | 18:03 |
| *** infra-re_ has quit IRC | 18:03 | |
| jeblair | clarkb, yolanda, mordred, greghaynes: okay, let's try to make a decision. a) ignore resolv.conf if read-only. b) do not write resolv.conf if it exists. c) add option to specifically ignore resolv.conf. | 18:03 |
| clarkb | yolanda: ya I am not sugesting we just go and do it, there would have to be a transition period. I am just pointing it out becaues the lack of robust checks like that has led to multiple debilitating bugs in glean | 18:03 |
| jeblair | clarkb, yolanda, mordred, greghaynes: that's a multiple choice question ^ | 18:03 |
| yolanda | clarkb yes, we faced several problems with that, such as hostname , ssh keys | 18:04 |
| jeblair | clarkb, yolanda, mordred, greghaynes: i like a+c. | 18:04 |
| yolanda | a | 18:04 |
| clarkb | jeblair: yes I think that combo is the most flexible | 18:04 |
| yolanda | jeblair, how would you pass that option to glean? on config drive? | 18:04 |
| clarkb | yolanda: at image build time I think | 18:04 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Move OpenStack-Ansible CentOS7 jobs to experimental queue https://review.openstack.org/307369 | 18:04 |
| jeblair | clarkb, yolanda: yeah -- so probably either a command-line option wherever glean is invoked, or some kind of config file like /etc/glean.conf | 18:05 |
| yolanda | i like the config file idea | 18:06 |
| yolanda | to avoid some problems if glean is called by some reason without parameters | 18:07 |
| *** gyee has joined #openstack-infra | 18:07 | |
| pabelanger | AJaeger: Ya, thanks for catching that. My brain didn't process that in 307369 | 18:07 |
| *** rhallisey has quit IRC | 18:08 | |
| *** Nisha_away has joined #openstack-infra | 18:08 | |
| mordred | I like config file more | 18:08 |
| mordred | easy for us to drop that in place at image build time | 18:09 |
| anteaya | zaro: pleia2 was just processing this request: http://lists.openstack.org/pipermail/openstack-infra/2016-April/004179.html | 18:09 |
| yolanda | so c option? or how do you think about a+c combo? | 18:09 |
| *** thorst_ has joined #openstack-infra | 18:09 | |
| *** Sukhdev has quit IRC | 18:09 | |
| mordred | we should also make sure that glean doesn't grow _requiring_ the config file | 18:09 |
| anteaya | zaro: both she and I hoped that deactivating an account would remove it from any gerrit groups of which it was a member | 18:09 |
| *** Sukhdev has joined #openstack-infra | 18:09 | |
| *** Sukhdev has quit IRC | 18:10 | |
| anteaya | zaro: she deactivated the account, but it was still a member of the neutron-ci gerrit group so then removed it | 18:10 |
| *** sridhar_ram1 is now known as sridhar_ram | 18:10 | |
| anteaya | zaro: is there some reason why a deactivated gerrit account should still be a member of gerrit groups? | 18:10 |
| zaro | anteaya: i believe deactivation only prevents the user from logging into gerrit. | 18:11 |
| *** exploreshaifali has joined #openstack-infra | 18:11 | |
| *** nwkarsten has quit IRC | 18:11 | |
| fungi | anteaya: probably in case it gets activated again in the future? more likely the question is whether inactive accounts should be displayed as group members | 18:11 |
| *** yamahata has quit IRC | 18:11 | |
| anteaya | zaro: oh | 18:11 |
| *** Nisha has quit IRC | 18:11 | |
| zaro | anteaya: : was just about to say what fungi said | 18:12 |
| anteaya | fungi: oh displayed as opposed to hidden | 18:12 |
| yolanda | so happy to work on that glean patch | 18:12 |
| *** ajmiller_ has joined #openstack-infra | 18:12 | |
| fungi | as an exmaple, you can't _select_ inactive accounts and add them to groups, so it does strike me as odd that they'd be displayed as group members | 18:12 |
| anteaya | zaro: okay is there a reason inactive accounts should be displayed as group members? | 18:12 |
| pleia2 | fungi: ah yes, since sometimes we deactivate and reactivate and we don't want to have to add them back to all groups | 18:12 |
| pleia2 | so it makes sense for us too | 18:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Translations: Fix sync of releasenotes https://review.openstack.org/307402 | 18:12 |
| *** nwkarsten has joined #openstack-infra | 18:12 | |
| *** salv-orlando has quit IRC | 18:13 | |
| *** MarkAtwood has joined #openstack-infra | 18:13 | |
| *** salv-orlando has joined #openstack-infra | 18:13 | |
| fungi | at least inactive accounts don't show up in the typeahead completion for group member additions and subscriber selections, though i have no idea if it would allow you to enter one directly | 18:13 |
| zaro | anteaya: yeah, the question is should user still be a member when reactivated. the two things are decoupled now so i think that makes the most sense | 18:14 |
| AJaeger | project-config cores, could you review 307402, please? I introduced a bug with releasenotes syncing that we should better fix | 18:14 |
| zaro | anteaya: i'm agreeing to what pleia2 said | 18:15 |
| *** ajmiller has quit IRC | 18:15 | |
| *** pots has quit IRC | 18:15 | |
| AJaeger | the good thing about today's long queue is that it's easy to spot translation sync failures in the post queue. I'm missing the jenkins "failing" job list ;( | 18:16 |
| EmilienM | pabelanger: our puppet periodic jobs are still failing, this is an attempt to fix them https://review.openstack.org/#/c/307142/ - ZUUL_REF was missing | 18:17 |
| jeblair | AJaeger: what's the 'jenkins "failing" job list' ? | 18:17 |
| *** salv-orlando has quit IRC | 18:18 | |
| samueldmq | hi all | 18:18 |
| AJaeger | jeblair: https://jenkins.openstack.org/ has a list of failed jobs | 18:18 |
| anteaya | zaro: okay so gerrit is working as expected? | 18:18 |
| AJaeger | jeblair: since translation jobs run all on jenkins.o.o, it was easy to see which ones failed... | 18:18 |
| samueldmq | there is a keystone change that had received +2+A and then the score was removed | 18:18 |
| samueldmq | jenkins still posted a +2 vote on the change | 18:18 |
| *** diablo_rojo has left #openstack-infra | 18:19 | |
| zaro | anteaya: i believe that is expected behavior | 18:19 |
| samueldmq | is this expected behavior or a bug ? | 18:19 |
| jeblair | AJaeger: ah | 18:19 |
| samueldmq | the change I am talking about is 298925 | 18:19 |
| clarkb | samueldmq: which vote was removed? | 18:19 |
| AJaeger | jeblair: do you know a good way now to figure out which jobs of pattern "*-upstream-translation-update" failed recently? | 18:20 |
| JayF | What do typical infra jobs do to avoid failing on distribution mirrors? We've been seeing more and more timeout failures in ipa source jobs (in docker build trying to pull down packages) in the last week and curious if there's a long-term fix. | 18:20 |
| samueldmq | clarkb: Brant had +2+W, then removed both | 18:20 |
| clarkb | samueldmq: the +A is what you are worried about? Zuul does not stop testing if the votes change, I am not sure if we have thought too hard about whether or not it should | 18:20 |
| anteaya | zaro: okay thanks | 18:20 |
| JayF | Along those lines, if there's any recent change in the last couple of weeks that would've caused those types of failures to be more frequent, I'm all ears as well | 18:20 |
| *** pots has joined #openstack-infra | 18:20 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-dev/pbr: Remember the insertion order for pbr.json https://review.openstack.org/307405 | 18:20 |
| clarkb | JayF: we run mirrors for them | 18:20 |
| clarkb | JayF: this is why everyone using containers because they are cool gets tricky | 18:21 |
| samueldmq | clarkb: if workflow has been removed, maybe it makes sense to stop running the gate jobs | 18:21 |
| clarkb | because the containers typically have their own (wrong) ideas about what dns and repos they should use | 18:21 |
| JayF | clarkb: I mean, I'm asking the question because I want to make our build more aware of this | 18:21 |
| pabelanger | EmilienM: mind posting a URL to the most recent log? | 18:21 |
| samueldmq | clarkb: I was just wondering if this was expected behavior, and wanted to share my observation here in -infra | 18:21 |
| clarkb | JayF: ya, you could inherit everything from the host | 18:21 |
| clarkb | JayF: dns, pip config, apt config, yum config, apt and yum package caches, git repo cache and so on | 18:22 |
| JayF | clarkb: well, given we have a debian container being built out of ubuntu, I can't exactly do it that way | 18:22 |
| clarkb | JayF: we put a lot of effort into making sure that our images our prebuilt to make success more likely then containers come through and break that | 18:22 |
| clarkb | JayF: you could builddebian on debian | 18:22 |
| clarkb | JayF: but we don't have a debian mirror yet | 18:22 |
| *** nwkarsten has quit IRC | 18:23 | |
| pabelanger | shouldn't be hard to add these days :) | 18:23 |
| *** nwkarsten has joined #openstack-infra | 18:23 | |
| fungi | not that the debian mirror is likely very far off at this point | 18:23 |
| JayF | What is needed to make the debian mirror work? That'll solve our immediate problem and I can integrate it into our ramdisk builds. | 18:23 |
| EmilienM | pabelanger: sure, http://logs.openstack.org/periodic/periodic-puppet-openstack-integration-4-scenario001-tempest-centos-7/86c8101/console.html.gz#_2016-04-15_06_02_01_182 | 18:23 |
| fungi | kinda expecting to see reviews for it pop up any day now, since it's just a little more copying the existing reprepro pattern we have | 18:23 |
| pabelanger | I hope to start RPM mirrors this week, just waiting for an email about validating REPO data | 18:24 |
| pabelanger | I'll be able to help on debian after that | 18:24 |
| JayF | If someone can point me in a direction; I don't mind doing the work myself | 18:24 |
| openstackgerrit | Merged openstack-infra/project-config: Add release notes jobs for trove-dashboard CI https://review.openstack.org/307152 | 18:24 |
| *** salv-orlando has joined #openstack-infra | 18:24 | |
| pabelanger | also, https://review.openstack.org/#/c/289520/ could use some reviews! RPM mirrors | 18:25 |
| JayF | right now I have three jobs that haven't gotten clean tests and I need to fix them, and the debian mirror sounds like the best path to getting there | 18:25 |
| pabelanger | minus the rsync check | 18:25 |
| fungi | JayF: out of curiosity, which debian mirrors are failing for you? hash sum mismatch or something else? 404? | 18:25 |
| openstackgerrit | Merged openstack-infra/project-config: add releasenotes & beaker jobs to puppet-ec2api https://review.openstack.org/307162 | 18:25 |
| openstackgerrit | Merged openstack-infra/release-tools: Fix ability to send notifications for multiple destinations https://review.openstack.org/307167 | 18:26 |
| *** abregman has quit IRC | 18:26 | |
| clarkb | fungi: dib builds tend to just fail on network connectivity to them | 18:26 |
| JayF | fungi: 2016-04-18 17:07:24.542 | [91mE: Failed to fetch http://httpredir.debian.org/debian/pool/main/p/python-urllib3/python-urllib3_1.9.1-3_all.deb Error reading from server. Remote end closed connection [IP: 5.153.231.35 80] | 18:26 |
| fungi | wow | 18:26 |
| pabelanger | JayF: mirror_update.pp in system-config | 18:26 |
| clarkb | ya ^ we see that too during dib builds | 18:26 |
| JayF | fungi: that's what we're seeing, over and over, during our ipa-src builds | 18:26 |
| fungi | httpredir may be a questionable choice | 18:26 |
| *** rhallisey has joined #openstack-infra | 18:26 | |
| pabelanger | clarkb: fungi: When we talked about debian mirrors zigo said he had contacts to help make them stable | 18:26 |
| pabelanger | since we randomize our mirror over pinning it | 18:27 |
| JayF | fungi: I'm all ears and willing to change it to something better if you know what is better | 18:27 |
| pabelanger | fungi: agreed | 18:27 |
| pabelanger | would rather see us pin to one | 18:27 |
| fungi | JayF: this sort of hints at why... http://httpredir.debian.org/ | 18:27 |
| JayF | httpredir is what ships in the container, fwiw, we don't touch it today | 18:27 |
| JayF | I'm very willing to if it fixes things though | 18:27 |
| mordred | pabelanger: I'm confused as to what "make them stable" means in this context | 18:28 |
| *** abregman has joined #openstack-infra | 18:30 | |
| openstackgerrit | Merged openstack-infra/project-config: add reno jobs to puppet-oslo https://review.openstack.org/307166 | 18:30 |
| fungi | i have a feeling just switching to something like http.us.debian.org would yield more consistency over httpredir.debian.org | 18:30 |
| clarkb | pabelanger: any ideas on further debugging centos-7 boot? | 18:30 |
| pabelanger | clarkb: was trying to find another node that was failing in the gate, | 18:31 |
| JayF | fungi: will give that a shot and let you know :) | 18:31 |
| *** sambetts is now known as sambetts|afk | 18:32 | |
| clarkb | pabelanger: they don't fail in the gate, they fail during the ready script | 18:32 |
| pabelanger | clarkb: can only think it is a race some place | 18:32 |
| pabelanger | clarkb: Oh, well that explains it | 18:32 |
| fungi | JayF: and as always, make sure you apt-get update before trying to apt-get install anything | 18:32 |
| JayF | ofc | 18:32 |
| clarkb | we intentionally resolve git.o.o in the ready script becaues DNS has broken so many times in the past | 18:32 |
| *** abregman has quit IRC | 18:33 | |
| pabelanger | mordred: I pointed out to zigo http://httpredir.debian.org might be a problem because of random mirror timeouts. And thought he offered to assisted with bad mirrors. Trying to find conversation, however I may have been dreaming this too | 18:33 |
| dstufft | I forget what the file system that y'all use is that doesn't support many directories, is a scheme like https://bpaste.net/show/b302989de8bd enough breaking up of directories and a short enough filename and whatever other limits you have? (Also you need to add the final filename to the end of that, like pip-8.1.0.tar.gz or w/e) | 18:34 |
| openstackgerrit | Merged openstack-infra/project-config: add releasenotes and beaker jobs on puppet-ovn https://review.openstack.org/307148 | 18:34 |
| dstufft | or do you need more sub dirs | 18:34 |
| fungi | i'm pretty sure httpredir performs some basic checks of mirrors to automatically remove them from its pool when they're "bad" but it may not take intermittent behavior issues into account | 18:35 |
| mordred | pabelanger: ah - well, really, just having out own debian mirror is the best bet for us | 18:35 |
| *** mpaolino has quit IRC | 18:35 | |
| nibalizer | some kind soul kicked off another vos release on the npm mirror | 18:35 |
| *** nwkarsten has quit IRC | 18:35 | |
| pabelanger | mordred: At this point, ya. We're knocking out mirrors at a good pace | 18:35 |
| *** nwkarsten has joined #openstack-infra | 18:36 | |
| nibalizer | but the npm mirror process is chugging along | 18:36 |
| clarkb | mordred: we are leaking fips | 18:36 |
| fungi | dstufft: likely depends on the length and number of that third part of the triplet. afs is limited to 64k directories at a given level if the directory names are short enough, but longer directory names consume additional slots from that limit | 18:36 |
| pabelanger | mordred: when you have spare cycles, 289520 was minimized to a basic rsync for centos7. Would be interested in your thoughts (still waiting on syntax for validation) | 18:36 |
| *** esikachev has joined #openstack-infra | 18:36 | |
| clarkb | mordred: at a very high rate looks like | 18:36 |
| *** Nisha_away has quit IRC | 18:36 | |
| clarkb | ~60 leaked already in osic | 18:37 |
| clarkb | dstufft: jeblair and mordred are going to be your best bets, I want to say I recall a dir name length consideration too, so while that nesting is good it is possible the deepest dirname would cause trouble? | 18:38 |
| clarkb | dstufft: definitely check with mordred and jeblair | 18:38 |
| dstufft | fungi: that's the length of the third part, 60 characters (though each sub dir i add reduces that by whatever number of characters I add). The _total_ number of those third parts (across all sub dirs) is going to be like, half a million | 18:38 |
| dstufft | and growing | 18:38 |
| fungi | dstufft: taken from jeblair's earlier explanation, "there are 64k slots -- each file <16chars takes 1 slot; 16<len<32 is 2 slots, etc" | 18:39 |
| fungi | so for all 60 character directory or file names within a given subdirectory at that level, afs would support 16k of them | 18:40 |
| openstackgerrit | Merged openstack-infra/system-config: Add playbook to manage power state of ironic servers https://review.openstack.org/284981 | 18:40 |
| dstufft | There should be 65536 of the top two levels, If you assume equal distribution you're looking at uh... 9 of those 60 character directories per sub dir? | 18:40 |
| nibalizer | we're up to 566.8G in npm | 18:41 |
| nibalizer | which feels pretty close to where we were on friday | 18:41 |
| nibalizer | so im wondering why it didn't finish over the weekend | 18:41 |
| fungi | dstufft: it's just the directory name which counts, not the full path length, luckily | 18:41 |
| *** ryanpetrello has left #openstack-infra | 18:41 | |
| *** flwang1 has joined #openstack-infra | 18:41 | |
| mordred | clarkb: ok. I'll look in to it, thanks! | 18:42 |
| *** MarkAtwood has quit IRC | 18:43 | |
| *** roxanagh_ has joined #openstack-infra | 18:43 | |
| fungi | dstufft: and yeah, 9 subdirectories is definitely well below the threshold for that | 18:43 |
| *** jed56 has quit IRC | 18:43 | |
| *** nadya has joined #openstack-infra | 18:43 | |
| fungi | dstufft: i assume that's based on present count of package names in pypi/warehouse? | 18:44 |
| dstufft | assuming my math is right yea | 18:44 |
| fungi | and so would expect it to increase, but, yeah, that's plenty of room for expansion | 18:44 |
| dstufft | and yea, current number of _files_ in PyPI atm | 18:44 |
| *** eil397 has joined #openstack-infra | 18:45 | |
| *** roxanagh_ has quit IRC | 18:45 | |
| *** baoli has quit IRC | 18:46 | |
| fungi | dstufft: as a point of comparison, right now we only hash out the first letter of a package name with our pypi mirrors, so for example the total number of packages starting with lower case p (14296) is still small enough to have not hit the limit but depending on how long some of those names are it mighht be getting close | 18:47 |
| *** ajmiller_ is now known as ajmiller | 18:48 | |
| *** dtardivel has quit IRC | 18:48 | |
| clarkb | mordred: based on the number of NeutronFloatingIPDelete tasks I don't think it is a 100% fail | 18:48 |
| clarkb | mordred: but appears to leak at a rate higher than the old code | 18:48 |
| dstufft | fungi: ok awesome | 18:49 |
| *** eil397 has quit IRC | 18:50 | |
| *** roxanagh_ has joined #openstack-infra | 18:50 | |
| dmsimard | Does a project need to be in the big tent to have the OpenStack bots in their IRC channels ? (i.e, gerrit, openstackstatus, eavesdrop) | 18:50 |
| fungi | dstufft: looks like we have 11345 using one slot, 2685 using two, 131 using 3 and 1 using four, for a total of 17247 (out of 65536 slots) | 18:50 |
| *** kzaitsev_mb has joined #openstack-infra | 18:50 | |
| fungi | under lower-case p | 18:50 |
| jeblair | nibalizer: see earlier conversation with krotscheck -- in short: the npm mirror script is a daemon and never exists but rather continues to keep things updated in real time. krotscheck is working on a solution. | 18:51 |
| dmsimard | We might be interested in having the OpenStack bots in #rdo. I'm familiar with the process of setting that up -- just don't know if you'd let us :) | 18:51 |
| dstufft | I'm going to be switching PyPI to using hashed names by default for the actual files (it's what I'm working on atm) and trying to sort out the best way forward. I want the hash in the URL because then I never have to worry about purging a file, ever :] but I don't want to shove half a million 64 character direcotires in one top level dir heh. | 18:51 |
| *** ociuhandu has quit IRC | 18:52 | |
| *** nwkarsten has quit IRC | 18:52 | |
| fungi | dmsimard: i don't believe we've ever required that in the past, however you do need to grant our statusbot account founder access so that it can add our other community channel operators to be able to help troubleshoot bot issues and provide general assistance in channel | 18:52 |
| dmsimard | fungi: aware of the access part, yes. Ok, thanks. | 18:53 |
| fungi | dmsimard: i think that came up in the past with rdo and was basically a conflict due to already having an existing community culture which might have prevented making us chanops | 18:53 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add optional start_date parameter to get_recent*() methods https://review.openstack.org/307416 | 18:53 |
| fungi | but i may be misremembering, or that may have changed | 18:53 |
| *** roxanagh_ has quit IRC | 18:54 | |
| *** infra-red has joined #openstack-infra | 18:54 | |
| *** infra-red has quit IRC | 18:54 | |
| *** baoli has joined #openstack-infra | 18:54 | |
| *** Rockyg has joined #openstack-infra | 18:54 | |
| *** infra-red has joined #openstack-infra | 18:55 | |
| dmsimard | fungi: RDO is closer to upstream than it was before but indeed, we have to reach out to the community to see if they have any problems with this | 18:55 |
| fungi | dstufft: makes sense, that's a reasonable concern. also if it's sane i'd only add those directories as needed rather than precreating them all | 18:56 |
| EmilienM | dmsimard: yeah, should not be a blocker I think | 18:56 |
| dstufft | fungi: I mean this is PyPI itself so it's all just in S3 so "directories" | 18:56 |
| jeblair | dstufft, fungi: yeah, that hashing scheme looks creat -- 2char*255 / 2char*255 / 60char*9. (9 to start, and then increasing as pypi grows) afs and sysadmins will both love that. | 18:56 |
| anteaya | fungi: it came up before and yes there were issues with putting bots in #rdo | 18:56 |
| jeblair | dstufft, fungi: er, 'great'. not 'creat'. | 18:57 |
| anteaya | as I don't think #rdo is just openstack | 18:57 |
| dstufft | bandersnatch should even work automatically with this | 18:57 |
| dstufft | and just in time upgrade old files to the new name anytime that package gets updated at all | 18:57 |
| jeblair | i mean, also o_creat. | 18:57 |
| dstufft | jeblair: awesome, thanks | 18:57 |
| jeblair | dstufft: how does pip fetch those files? | 18:57 |
| dstufft | jeblair: I'm not sure what specifically you're asking here, but they'll still be linked from /simple/<whatever>/ and then pip will just do a HTTP fetch on them | 18:58 |
| dstufft | this isn't going to change /simple/, just /packages/ | 18:58 |
| dstufft | (other than /simple/'s links to /packages/ needing to be updated) | 18:59 |
| fungi | ahh, my numbers were for simple rather than packages, but still an effective demo | 19:00 |
| *** nwkarsten has joined #openstack-infra | 19:00 | |
| jeblair | dstufft: gotcha. it's the /simple URLs that we do the apache mod_rewrite on (because they all appear in one dir). the existing package hashing is fine for us. it sounds like the new package hashing will be fine too, but we'll need to continue doing what we're doing with bandersnatch and mod_rewrite for /simple. | 19:00 |
| fungi | the additional version-specific subpaths in packages make the current scheme it's using smaller for our purposes anyway | 19:00 |
| fungi | for example, packages/2.7/p/ has only 1293 entries | 19:01 |
| dstufft | /simple/ is a harder problem to solve without some magic like the patches y'all have to bandersnatch or forcing pip to download a single big file with al of the package versions | 19:01 |
| nibalizer | jeblair: ah ok | 19:01 |
| dstufft | fungi: hilarously, there's apparently no validation on what goes in the 2.7 spot | 19:01 |
| jeblair | dstufft: yeah -- it at least would require a change to pip. | 19:02 |
| *** fernnest_ has joined #openstack-infra | 19:02 | |
| dstufft | https://s.caremad.io/rm5kdB2CGe/ some of these make sense, some of these... | 19:02 |
| fungi | my oh my | 19:03 |
| *** nwkarsten has quit IRC | 19:04 | |
| dstufft | jeblair: yea, best thing I could think of is just apply a similar sort of thing to bucket the names, so instead of /simple/whatever/ it's /simple/w/whatever/ or /simple/HASH(whatever)[:2]/whatever/ or something. But that's a protocol level change and needs a PEP and such | 19:05 |
| dstufft | something to keep in mind when I redesign the protocol API though | 19:05 |
| jeblair | dstufft: ++ | 19:06 |
| *** baoli has quit IRC | 19:07 | |
| pabelanger | clarkb: mind if I delete your test instances and relaunch it in osic? | 19:09 |
| openstackgerrit | Rodrigo Duarte proposed openstack-infra/project-config: Enable non-voting keystone tempest plugin tests https://review.openstack.org/298696 | 19:12 |
| clarkb | pabelanger: go for it | 19:12 |
| *** MarkAtwood has joined #openstack-infra | 19:13 | |
| mordred | dstufft: and since you've got packages/ already - honestly just having a simple.json file or endpoint for whatever and not having pip walk the simple index anymore seems like a nicer thing to do if you're going ot make new API things | 19:13 |
| dstufft | I'm not sure I want to force every pip install to download a big file everytime. /simple/ itself is already 3+MB and it doesn't even contain any file information, jsut a list of projects | 19:15 |
| dstufft | granted it's HTML and not JSON, but I don't think HTML is that much less efficient than JSON when it's just a list of links | 19:16 |
| *** raginbajin has quit IRC | 19:16 | |
| fungi | and also compressed otf | 19:16 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add api to get recent failed runs by run_metadata https://review.openstack.org/305495 | 19:16 |
| openstackgerrit | Merged openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/303416 | 19:16 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix add-apt-repository package for precise https://review.openstack.org/294760 | 19:17 |
| dstufft | gzip helps, that makes it 0.6MB instead, but /simple/requests/ is 113 bytes when compressed :] | 19:17 |
| fungi | granted, apt-get attempts to solve that issue with incremental index updates applied to a local cache of the list, but that's a ton of additional complication | 19:18 |
| fungi | and even then, they have separate commands for updating your index cache and installing a package | 19:18 |
| fungi | (and they have a lot fewer packages to index, but they also put a ton more metadata in their indices) | 19:18 |
| *** Yarboa has joined #openstack-infra | 19:19 | |
| dstufft | it's a bit of a question though whether one big HTTP request is better than more smaller ones. I sort of lean towards the second one, maybe that's wrong though I never actually attempted to science it (though one benefit is that the effective cache for the popular packages is basically 100% since say Django is released rarely but downloaded often) | 19:19 |
| stevemar | sdague: poke | 19:19 |
| dstufft | so /simple/Django/ gets cached a lot longer than /simple.json would | 19:19 |
| *** nmagnezi has joined #openstack-infra | 19:20 | |
| stevemar | sdague: wanted your input on https://review.openstack.org/#/c/306587/ (cc mtreinish too) | 19:20 |
| *** rfolco has quit IRC | 19:20 | |
| mordred | dstufft: ah. good point | 19:20 |
| stevemar | sdague: this release will remove the keystone CLI | 19:20 |
| fungi | need to pop out for an errand, but should brb | 19:20 |
| mtreinish | stevemar: that's gonna break people. Even though it's been announced everywhere for a while | 19:21 |
| sdague | stevemar: ... we currently have a 549 job check queue, and people prepping for summit. Can we wait until after summit? | 19:21 |
| *** raginbajin has joined #openstack-infra | 19:21 | |
| stevemar | sdague: we could, i was thinking pre-summit would be a good idea since people could yell at us at the summit | 19:22 |
| stevemar | mtreinish: grr | 19:22 |
| sdague | stevemar: well if it breaks all code merge .... | 19:22 |
| sdague | you could get a lot of yelling :) | 19:22 |
| AJaeger | stevemar: you would be popular ;) | 19:23 |
| sdague | mtreinish: any ideas who still gets broken by it | 19:23 |
| mtreinish | stevemar: heh, you know it's gonna break someone/something | 19:23 |
| *** esikachev has quit IRC | 19:23 | |
| sdague | I thought we got novaclient functional off of it | 19:23 |
| stevemar | looking at http://codesearch.openstack.org/?q=keystone%20user-list&i=nope&files=&repos= these things would get hit | 19:23 |
| sdague | devstack doesn't use it (hasn't for a long time) | 19:23 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: cmd: Add a config file fallback to the user's home directory https://review.openstack.org/237694 | 19:23 |
| stevemar | sdague: right, devstack and nova should be OK now | 19:23 |
| mtreinish | sdague: I htink most of the jobs stopped using it, but I was thinking maybe even more outside the gate | 19:23 |
| stevemar | lots of charms, but just in the readme? | 19:23 |
| *** bkeller` has quit IRC | 19:24 | |
| sdague | does codesearch do paging? | 19:24 |
| sdague | do we know if that's the complete list | 19:24 |
| stevemar | sdague: no | 19:24 |
| sdague | that's actually not that bad | 19:24 |
| stevemar | sdague: well that was just user-list :) | 19:24 |
| persia | dstufft: If you decide to science one-big-file vs. many-little-files, two two extreme behaviours are at low-bandwidth low-latency and high-bandwidth high-latency, as in the former case, parallelism consumes all the bandwidth, and in the latter case, TCP windows never expand all the way. | 19:24 |
| sdague | stevemar: yeh, but that's honestly likely to be the big one | 19:25 |
| stevemar | which i think is the most used command | 19:25 |
| clarkb | sdague: stevemar I would be most worried about things on stable branches, are we properly pinned onthose? | 19:25 |
| *** sputnik13 has quit IRC | 19:25 | |
| *** rfolco has joined #openstack-infra | 19:25 | |
| stevemar | clarkb: u-c should protect us there no? | 19:25 |
| sdague | clarkb: upper constraints | 19:25 |
| mtreinish | stevemar: user-create has some: http://codesearch.openstack.org/?q=keystone%20user-create&i=nope&files=&repos= | 19:25 |
| sdague | so... other than kilo | 19:25 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Fix the typo of docstring https://review.openstack.org/305941 | 19:25 |
| clarkb | stevemar: sdague no, thats not a complete help | 19:25 |
| clarkb | stevemar: sdague beacuse you can break u-c updates | 19:25 |
| erlon_ | guys, how do you push tags between gerrit/bitbucket projects? | 19:26 |
| sdague | clarkb: no, stable is not updating upper constraints | 19:26 |
| AJaeger | clarkb, sdague, mordred : could you review a change for translations, please? https://review.openstack.org/#/c/307402/ we should sync releasenotes only for master and if they exist - sorry, my last patch broke that. | 19:26 |
| clarkb | sdague: oh nevermind then I guess | 19:26 |
| stevemar | mtreinish: i just did that, lots of rst hits, and random shell scripts (barbican) | 19:26 |
| clarkb | erlon_: we don't use bitbucket at all | 19:26 |
| *** dizquierdo has quit IRC | 19:26 | |
| sdague | clarkb: it was decided uc would be manual update only | 19:26 |
| clarkb | sdague: thats unfortunate | 19:26 |
| erlon_ | clarkb: sorry, github | 19:26 |
| clarkb | (considering the whole point with automated updates was to get up to date things that work for security raisins) | 19:27 |
| fungi | erlon_: we don't mirror to bitbucket, we mirror to our own cgit farm and to github, and we use gerrit's replication plugin to push updates from gerrit to those mirrors | 19:27 |
| erlon_ | clarkb: we have a similar infra here, but we use bitbucket | 19:27 |
| mtreinish | stevemar: barbican, designate, and the murano devstack plugin | 19:27 |
| sdague | clarkb: not for stable, anyway, I'm not getting into that fight right now | 19:27 |
| stevemar | mtreinish: cue as well | 19:27 |
| fungi | erlon_: you probably want gerrit's replication plugin then | 19:27 |
| sdague | stevemar: you know what... honestly, I'd say the following | 19:27 |
| clarkb | sdague: ya I stopped caring long ago when no one listened to any of my opinions on deps :) | 19:27 |
| erlon_ | fungi: hmm, nice | 19:27 |
| stevemar | sdague: drop some wisdom on me | 19:27 |
| clarkb | sdague: stevemar it should work in that case on stable with constraints | 19:27 |
| sdague | send an email today, list the projects you found in codesearch as being an issue | 19:27 |
| sdague | saying that on thursday you are cutting the release with this | 19:28 |
| clarkb | erlon_: note it is one direction gerrit -> mirror, we don't sync the other direction | 19:28 |
| fungi | erlon_: it basically treats your mirrors as git push targets and pushes your updates to them more or less instantly | 19:28 |
| sdague | and everyone has 3 days to address this | 19:28 |
| sdague | then push it | 19:28 |
| *** esikachev has joined #openstack-infra | 19:28 | |
| erlon_ | fungi: is there a way to review the tags pushes? or that goes directly through pushes? | 19:28 |
| erlon_ | clarkb: mhm | 19:28 |
| *** sputnik13 has joined #openstack-infra | 19:28 | |
| sdague | because, honestly, I don't think any of those projects are getting fixed until they break | 19:28 |
| sdague | as they are pretty disconnected from paying attention | 19:29 |
| sdague | and this has been out there as coming for a while | 19:29 |
| mugsie | sdague: what do we have 3 days to fix? | 19:29 |
| sdague | mugsie: not using keystone cli | 19:29 |
| stevemar | sdague: i agree with your assessment | 19:29 |
| fungi | erlon_: reviewing proposed tags is a long-standing wishlist item for gerrit. we have some convoluted workarounds coalescing to give us some semblance of tag reviews instead | 19:29 |
| erlon_ | fungi: does it pushes not merged tags or only merged? | 19:29 |
| *** nwkarsten has joined #openstack-infra | 19:29 | |
| *** weshay has quit IRC | 19:29 | |
| fungi | erlon_: tags aren't merged or unmerged. tags are just tags | 19:29 |
| stevemar | sdague: i wrote http://lists.openstack.org/pipermail/openstack-dev/2016-April/092471.html on friday, but i'll write out something new today | 19:29 |
| stevemar | and use tags with the projects we found | 19:29 |
| erlon_ | fungi: that I mean for objects | 19:30 |
| sdague | stevemar: yeh, adding specific project names gives people more heads up | 19:30 |
| mugsie | sdague: crap, I thought that was fixed. | 19:30 |
| erlon_ | fungi: for example, we a user pushes a new patchset | 19:30 |
| stevemar | sdague: yep! cool - i'll whip something up | 19:30 |
| fungi | erlon_: all objects are replicated by default, including proposed changes | 19:30 |
| erlon_ | fungi: hmm, that's nice | 19:30 |
| clarkb | if you tag a comit that is not in gerrit and push the tag to gerrit that commit will magically live in gerrit at that point | 19:30 |
| clarkb | it isn't an error to do that | 19:30 |
| erlon_ | fungi: will help us with another issue | 19:31 |
| mtreinish | stevemar: how many commands were there in the keystoneclient cli? | 19:31 |
| sdague | mugsie: some projects still list things in docs, I don't know which project you are specifically on top of, but it's probably pretty easy if there are references there | 19:31 |
| *** Yarboa has quit IRC | 19:31 | |
| erlon_ | fungi: great! thanks!! | 19:31 |
| sdague | I think in almost all these cases it's a 5 minute fix | 19:31 |
| erlon_ | clarkb: thanks! | 19:31 |
| fungi | erlon_: you just have to know how to construct the url to a proposed patchset on the mirror, but it'll be there | 19:31 |
| stevemar | mtreinish: a handful, i was thinking of enumerating all of them | 19:31 |
| *** nmagnezi has quit IRC | 19:31 | |
| openstackgerrit | Merged openstack/diskimage-builder: Skip gentoo test https://review.openstack.org/302516 | 19:31 |
| stevemar | mtreinish: http://paste.openstack.org/show/494502/ | 19:31 |
| mordred | clarkb: I believe I've found the race bug | 19:32 |
| openstackgerrit | Merged openstack/diskimage-builder: Add qcow2 generation for better test coverage https://review.openstack.org/302517 | 19:32 |
| mordred | clarkb: fix coming | 19:32 |
| mtreinish | stevemar: heh, ok we're on the same page then :) | 19:32 |
| *** ociuhandu has joined #openstack-infra | 19:32 | |
| stevemar | mtreinish: overwhelmingly user-list / tenant-list / user-create were the most often used | 19:32 |
| mtreinish | stevemar: sure, that makes sense | 19:32 |
| stevemar | with role-list and token-get coming up | 19:32 |
| jeblair | mordred: the fip thing? | 19:32 |
| mordred | jeblair: yah | 19:32 |
| *** esikachev has quit IRC | 19:33 | |
| stevemar | mtreinish: token-get had only 4 hits: http://codesearch.openstack.org/?q=keystone%20token-get&i=nope&files=&repos= | 19:33 |
| zigo | clarkb: fungi: mordred: Using bash --login -c "gbp buildpackage" didn't help, I still get the message "You need to log in again to gain sbuild group privileges". I'm really out of ideas, except moving sbuild setup to DIB, which will for sure work. | 19:33 |
| clarkb | zigo: are you sure the user is being added to the group prior to running all of these things thta have failed? | 19:34 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: email publisher: Clarify format of recipients list in docstring https://review.openstack.org/306992 | 19:35 |
| clarkb | apparetnly su is another option | 19:35 |
| zigo | clarkb: look at the logs, and check for "sbuild-adduser" http://logs.openstack.org/82/302382/7/check/pkgdeb-build-openstack-pkg-tools/bdf98ad/console.html | 19:35 |
| mordred | jeblair: also, I think ew should do the server/port every-five-seconds trick with fips too | 19:35 |
| mordred | jeblair: we do list_floating_ips a lot on fip clouds | 19:35 |
| mtreinish | stevemar: heh, a doc, a security note (x2), and osops | 19:35 |
| clarkb | zigo: I would cat the passwd and group files just to be sure | 19:35 |
| zigo | clarkb: When doing such a setup with sbuild, I always close the ssh connection and reconnect, then it does work, just as the messages says ... | 19:36 |
| zigo | clarkb: I'm 100% that this would work. | 19:36 |
| *** kzaitsev_mb has quit IRC | 19:36 | |
| zigo | clarkb: I'm unsure of the sbuild internals though, but relogin does work, that I know. | 19:36 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix disk usage report https://review.openstack.org/302509 | 19:36 |
| openstackgerrit | Merged openstack/diskimage-builder: dib-run-parts: make cp to target root more robust https://review.openstack.org/219453 | 19:37 |
| openstackgerrit | Merged openstack/diskimage-builder: simple-init: Fix path for /etc/ssh test https://review.openstack.org/305498 | 19:37 |
| *** avarner__ has quit IRC | 19:37 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: email_ext publisher: Improve docstring wording https://review.openstack.org/306993 | 19:38 |
| stevemar | mtreinish: working on the note now... https://etherpad.openstack.org/p/keystonecliremovalnote | 19:39 |
| *** ihrachys has joined #openstack-infra | 19:39 | |
| *** yamamoto has quit IRC | 19:39 | |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Bump upper-constraints on novaclient to 4.0.0 https://review.openstack.org/307437 | 19:41 |
| *** flwang1 has quit IRC | 19:41 | |
| pabelanger | clarkb: okay, so, we are running restorecon in our dib element, but for some reason, restorecon doesn't detect an issue and just does a noop | 19:41 |
| pabelanger | clarkb: which is why the follow line after restorecon -v /etc/unbound/forwarding.config is not related to restorecon | 19:42 |
| clarkb | pabelanger: I wonder if the issue is in chroot vs out of chroot | 19:42 |
| pabelanger | maybe | 19:43 |
| clarkb | pabelanger: eg ubuntu doesn't have anything to restore for that file on centos so it noops | 19:43 |
| pabelanger | I am looking at older logs now, to see if there is a difference | 19:43 |
| pabelanger | clarkb: that would make sense | 19:43 |
| pabelanger | clarkb: looking at the old init.d script for unbound on centos6, it would restorecon the whole folder on start | 19:44 |
| *** avarner has joined #openstack-infra | 19:44 | |
| *** exploreshaifali has quit IRC | 19:44 | |
| pabelanger | I am testing touch /.autorelabel now | 19:45 |
| pabelanger | to see how long it takes | 19:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Wait for floating ip deletion as part of server deletion https://review.openstack.org/307439 | 19:46 |
| *** salv-orlando has quit IRC | 19:47 | |
| *** rhallisey_ has joined #openstack-infra | 19:47 | |
| mordred | clarkb: ^^ I'd like to try that and see if it makes osic stop leaking | 19:50 |
| *** Sukhdev has joined #openstack-infra | 19:50 | |
| *** rhallisey has quit IRC | 19:50 | |
| mordred | clarkb: I don't actually have a cause identified from logs, so I'm trying to find places where we could potentially not issue the call | 19:50 |
| mordred | if that doesn't work, I'll add some more logging in | 19:50 |
| mordred | (that it's only sometimes makes it harder) | 19:50 |
| mordred | jeblair, Shrews: ^^ you too | 19:51 |
| clarkb | mordred I am lunching but based on commit message seems like a thing to fix | 19:51 |
| mordred | ya. seems like it won't be worse at the least | 19:52 |
| pabelanger | clarkb: looks like relabel takes about 57 seconds on osic. | 19:52 |
| *** ddieterly has joined #openstack-infra | 19:52 | |
| pabelanger | and unbound started correctly after the reboot | 19:52 |
| mtreinish | stevemar: are you gonna mention osops hit on the mail? or send a separate one to the ops list? | 19:52 |
| *** e0ne has joined #openstack-infra | 19:52 | |
| stevemar | mtreinish: i think i'll go with a separate mail with that one | 19:53 |
| stevemar | i sent a similar note to ops on friday | 19:53 |
| stevemar | maybe it was lost in the weekend shuffle | 19:53 |
| *** yamamoto has joined #openstack-infra | 19:54 | |
| stevemar | mtreinish: sdague take a quick once-over? https://etherpad.openstack.org/p/keystonecliremovalnote | 19:54 |
| clarkb | pabelanger: so a whole minute extra to boot? hrm | 19:54 |
| Shrews | mordred: already looking | 19:54 |
| stevemar | i'm actually not entirely disappointed with the amount of projects it will affect | 19:55 |
| stevemar | i was pessimistically thinking it would be more | 19:56 |
| *** kgiusti has left #openstack-infra | 19:56 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:58 | |
| sdague | stevemar: seems good to me | 19:58 |
| stevemar | ack | 19:58 |
| *** lakshmiS has quit IRC | 19:59 | |
| sdague | I would also say people should do a git grep in their documentation | 19:59 |
| *** lakshmiS_ has joined #openstack-infra | 19:59 | |
| sdague | because there were plenty of examples there as well | 19:59 |
| pabelanger | clarkb: that was the whole filesystem, so we could be more specific some how on reboot | 19:59 |
| *** ddieterly is now known as ddieterly[away] | 20:00 | |
| mordred | stevemar: out of curiosity, has osc grown the ability to talk to keystone v2 clouds? | 20:02 |
| stevemar | mordred: has it not always had that ability? | 20:03 |
| dtroyer | mordred: ?? do you mean v3? | 20:03 |
| mordred | nope | 20:03 |
| mordred | I mean openstack catalog list | 20:03 |
| *** pfallenop has joined #openstack-infra | 20:03 | |
| mordred | it does not work if the cloud has v2 | 20:03 |
| mordred | you need to use keystone cli for that, and osc if the cloud has v3 | 20:03 |
| mordred | at least last time I tried | 20:03 |
| * mordred tries again | 20:03 | |
| stevemar | mordred: i think that was a keystone (server side) bug that has since been fixed | 20:04 |
| mordred | ah - neat | 20:04 |
| mordred | yes - it works | 20:04 |
| mordred | nevermind | 20:04 |
| stevemar | mordred: don't scare me like that! | 20:04 |
| * dtroyer wipes brow… | 20:04 | |
| stevemar | :) | 20:04 |
| *** alex_xu has quit IRC | 20:04 | |
| mordred | stevemar: what about nova? has osc grown the ability to talk to nova? :) | 20:04 |
| * mordred hands dtroyer a bunny | 20:04 | |
| dtroyer | that nova thing might just catch on someday, we'll think about it | 20:04 |
| stevemar | :) | 20:05 |
| *** Sukhdev has quit IRC | 20:05 | |
| *** ddieterly[away] is now known as ddieterly | 20:05 | |
| *** alex_xu has joined #openstack-infra | 20:06 | |
| *** sdake has quit IRC | 20:07 | |
| openstackgerrit | Pedro Alvarez Piedehierro proposed openstack-infra/storyboard-webclient: Fix dashboard template to show the right information https://review.openstack.org/307448 | 20:07 |
| *** e0ne has quit IRC | 20:08 | |
| *** nwkarsten has quit IRC | 20:09 | |
| *** Nikolay_St has quit IRC | 20:09 | |
| *** eil397 has joined #openstack-infra | 20:09 | |
| *** nwkarsten has joined #openstack-infra | 20:10 | |
| *** kzaitsev_mb has quit IRC | 20:10 | |
| openstackgerrit | Catherine Diep proposed openstack-infra/irc-meetings: Move RefStack weekly team meeting from Monday to Tuesday. https://review.openstack.org/307450 | 20:10 |
| anteaya | here is the local.pp that I'm using to try to get the puppet-mediawiki stood up locally on precise: http://paste.openstack.org/show/494508/ | 20:11 |
| anteaya | at the bottom of that paste I have included the apache error, which will not start | 20:11 |
| anteaya | as it is looking for a ssl cert in a certain place and can't find it | 20:11 |
| anteaya | does anyone have any suggestions of how I can get apache to start? | 20:11 |
| clarkb | anteaya: remove the ssl chain file parameter, you are setting it to be empty when it should just not exist I think | 20:12 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fix string formatting https://review.openstack.org/307454 | 20:12 |
| clarkb | (though usually we test for content before writing the file) | 20:12 |
| anteaya | clarkb: very good, not existing is a great solution for me | 20:12 |
| *** beekneemech is now known as bneme | 20:12 | |
| clarkb | anteaya: you may need to remove it manually too | 20:12 |
| *** bneme is now known as bnemec | 20:12 | |
| Shrews | mordred: ^^^ easy review | 20:12 |
| anteaya | clarkb: thanks, I'll look around | 20:12 |
| *** baoli has joined #openstack-infra | 20:13 | |
| mordred | Shrews: +A | 20:13 |
| *** ryanpetrello has joined #openstack-infra | 20:13 | |
| *** _amrith_ has quit IRC | 20:14 | |
| *** tonytan4ever has joined #openstack-infra | 20:14 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Redirect instack image build output to file https://review.openstack.org/306533 | 20:14 |
| *** amrith has joined #openstack-infra | 20:15 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:15 | |
| *** dprince has quit IRC | 20:16 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/puppet-mediawiki: [WIP] Remove chain file parameter https://review.openstack.org/307459 | 20:17 |
| openstackgerrit | Ryan Petrello proposed openstack-infra/irc-meetings: Add Ryan Petello as astara meeting chair https://review.openstack.org/307460 | 20:18 |
| anteaya | clarkb: like this? https://review.openstack.org/#/c/307459/1/manifests/init.pp | 20:18 |
| anteaya | clarkb: I just put it on gerrit for you to look at and for me to pull down onto a new vm | 20:18 |
| *** madhu_ak has quit IRC | 20:18 | |
| anteaya | clarkb: not for merging it | 20:18 |
| *** _ari_ has quit IRC | 20:19 | |
| pabelanger | clarkb: I think this is our issue: http://www.redhat.com/archives/fedora-selinux-list/2005-August/msg00112.html | 20:20 |
| pabelanger | 2nd paragraph | 20:20 |
| *** kzaitsev_mb has quit IRC | 20:22 | |
| *** ddieterly is now known as ddieterly[away] | 20:23 | |
| *** rhallisey_ is now known as rhallisey | 20:23 | |
| AJaeger | project-config cores, could you put this one on your review list, please? https://review.openstack.org/#/c/307402/ Thanks - and good night! | 20:25 |
| pabelanger | clarkb: I guess we can update our systemd script to add a ExecStartPre=restorecon step | 20:25 |
| *** abregman has joined #openstack-infra | 20:26 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:26 | |
| *** ddieterly[away] is now known as ddieterly | 20:26 | |
| *** matrohon has joined #openstack-infra | 20:27 | |
| *** tonytan_brb has joined #openstack-infra | 20:28 | |
| *** nwkarste_ has joined #openstack-infra | 20:28 | |
| *** nwkarsten has quit IRC | 20:28 | |
| *** _ody has quit IRC | 20:29 | |
| *** nwkarste_ has quit IRC | 20:29 | |
| *** nwkarsten has joined #openstack-infra | 20:30 | |
| *** _ody has joined #openstack-infra | 20:30 | |
| anteaya | stevemar: moar subject tags please | 20:31 |
| openstackgerrit | Catherine Diep proposed openstack-infra/irc-meetings: Re-schedule RefStack weekly team meeting from Monday to Tuesday. https://review.openstack.org/307450 | 20:31 |
| *** tonytan4ever has quit IRC | 20:31 | |
| anteaya | stevemar: you didn't use nearly enough | 20:31 |
| *** sridhar_ram has quit IRC | 20:31 | |
| *** yamahata has joined #openstack-infra | 20:33 | |
| *** salv-orlando has joined #openstack-infra | 20:33 | |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Require stevedore>=1.9.0 https://review.openstack.org/307475 | 20:33 |
| *** baoli has quit IRC | 20:33 | |
| *** baoli has joined #openstack-infra | 20:34 | |
| *** nwkarsten has quit IRC | 20:34 | |
| *** tonytan4ever has joined #openstack-infra | 20:34 | |
| *** BigWillie has quit IRC | 20:34 | |
| *** kzaitsev_mb has quit IRC | 20:35 | |
| *** _ody has quit IRC | 20:35 | |
| *** nadya has quit IRC | 20:36 | |
| *** xarses has quit IRC | 20:37 | |
| mordred | clarkb: STABSTABSTABSTABSTAB | 20:37 |
| *** _ody has joined #openstack-infra | 20:37 | |
| *** asettle has joined #openstack-infra | 20:37 | |
| *** tonytan_brb has quit IRC | 20:37 | |
| mordred | clarkb: I have a new line of analysis to go down | 20:38 |
| mordred | clarkb: today is "fun" | 20:38 |
| *** e0ne has joined #openstack-infra | 20:38 | |
| jeblair | mordred: i'm back from eating chili. 307439 lgtm. should i approve? | 20:40 |
| jeblair | mordred: or is your current stabbing related? | 20:40 |
| openstackgerrit | Merged openstack-dev/pbr: Fix wsgiref script use with oslo.config https://review.openstack.org/296061 | 20:43 |
| *** lucas-dinner has quit IRC | 20:44 | |
| *** xarses has joined #openstack-infra | 20:44 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Replace deprecated assertEquals with assertEqual https://review.openstack.org/307176 | 20:47 |
| *** eharney has quit IRC | 20:48 | |
| *** newjersey has joined #openstack-infra | 20:50 | |
| *** e0ne has quit IRC | 20:50 | |
| *** zeih has joined #openstack-infra | 20:51 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:51 | |
| *** lucasagomes has joined #openstack-infra | 20:53 | |
| *** sridhar_ram has joined #openstack-infra | 20:54 | |
| *** cloudtrainme has joined #openstack-infra | 20:54 | |
| *** Sukhdev has joined #openstack-infra | 20:55 | |
| *** hichihara has joined #openstack-infra | 20:57 | |
| *** fawadkhaliq has joined #openstack-infra | 20:57 | |
| *** baoli has quit IRC | 20:57 | |
| *** weshay has joined #openstack-infra | 20:58 | |
| anteaya | did we ever hear back from ovh? | 20:59 |
| *** kzaitsev_mb has quit IRC | 20:59 | |
| anteaya | on the ticket fungi submitted or any email (if sent)? | 20:59 |
| *** sdake has joined #openstack-infra | 21:00 | |
| *** bpokorny_ has joined #openstack-infra | 21:01 | |
| fungi | anteaya: still no update on the ticket and no reply from our contacts at ovh (i cc'd yolanda, clarkb and jeblair on the e-mail i sent to ovh since they were in on that conversation and so that i'm hopefully not the only one receiving a reply) | 21:02 |
| anteaya | fungi: thank you | 21:02 |
| *** kzaitsev_mb has joined #openstack-infra | 21:02 | |
| fungi | also this way if they don't reply until eu morning (which is entirely likely at this point) hopefully yolanda will be in a position to follow up as needed prior to my being conscious and caffeinated | 21:03 |
| *** rossella_s has quit IRC | 21:03 | |
| *** lucasagomes has quit IRC | 21:03 | |
| clarkb | pabelanger: so we need setfiles? | 21:03 |
| *** rhallisey has quit IRC | 21:04 | |
| clarkb | anteaya: ya like that | 21:04 |
| *** rossella_s has joined #openstack-infra | 21:04 | |
| anteaya | clarkb: thanks, I also need to remove it from system-config as well I have discovered | 21:04 |
| clarkb | anteaya: in system-config we may be setting a real value | 21:04 |
| clarkb | anteaya: we don't want to remove it from there if so | 21:05 |
| *** bpokorny has quit IRC | 21:05 | |
| anteaya | hmmmm, I got an error last round | 21:05 |
| anteaya | I'm trying again and will paste any error if I get it this time | 21:05 |
| anteaya | last time I also forgot I was setting a value for that param in my local.pp | 21:05 |
| anteaya | going to remove that this round | 21:05 |
| ianw | SpamapS: re http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf ; i did some work in 2004 around https://lkml.org/lkml/2004/10/30/75 and pluggable schedulers on IA64. NUMA was a weird supercomputing thing then :) Nobody wanted to hear about it; nice to see the idea coming back | 21:06 |
| anteaya | clarkb: thank you for helping me with this | 21:06 |
| fungi | ianw: as long as itanium isn't coming back too, i'm cool with that ;) | 21:07 |
| clarkb | fungi: coming back? it never left ;) | 21:07 |
| fungi | heh | 21:07 |
| pabelanger | clarkb: I guess so, going to get it a try | 21:08 |
| pabelanger | clarkb: but running out of time for today | 21:08 |
| ianw | fungi: we had pluggable page-table implementations too. that's probably more an 2018 idea, though :) | 21:08 |
| SpamapS | ianw: indeed, it's basically everywhere now | 21:08 |
| ianw | also, i noticed that x86 is getting VM protection keys -- that seems to have come directly from Itanium. so the good bits are leaking out | 21:10 |
| *** asettle has quit IRC | 21:11 | |
| *** shardy has quit IRC | 21:11 | |
| SpamapS | ianw: just like the good bits of Opteron leaked into the Pentium 4 ;) | 21:11 |
| clarkb | pabelanger: oh its the selinux is disabled on ubuntu problem, that will do it | 21:11 |
| *** ddieterly is now known as ddieterly[away] | 21:12 | |
| clarkb | so ya that is a noop in our image builds | 21:12 |
| clarkb | (it would be nice if dib could figure out how to deal with this generically for cross distro builds) | 21:12 |
| *** lucasagomes has joined #openstack-infra | 21:12 | |
| ianw | SpamapS : while around, I guess dib has got caught up in this yaml->release stuff because it's part of tripleo? | 21:12 |
| SpamapS | ianw: reno? | 21:13 |
| pabelanger | clarkb: Yup, I have to run now, but will catch ianw up on the finding and maybe he has some better solutions | 21:13 |
| pabelanger | BBIAB | 21:13 |
| ianw | SpamapS: https://review.openstack.org/#/c/306925/ | 21:13 |
| anteaya | clarkb: http://paste.openstack.org/show/494517/ | 21:14 |
| ianw | SpamapS : i have to say, my enthusiasm for adding more steps to getting something fixed in nodepool is extremely low. it already takes days to get a simple fix out there. this will probably add at least another day | 21:14 |
| anteaya | clarkb: Invalid parameter ssl_chain_file on Class[Mediawiki] | 21:14 |
| *** aysyd has quit IRC | 21:14 | |
| anteaya | clarkb: but you say I shouldn't remove the paramater from system-config | 21:14 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Improved Summit Entity Event Processing https://review.openstack.org/307484 | 21:14 |
| SpamapS | ianw: err, yeah, I have no idea what that is. | 21:15 |
| anteaya | clarkb: so I'm not sure how to best address that error | 21:15 |
| *** inc0 has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| clarkb | anteaya: it is saying that in system-config/modules/openstack_project/wiki.pp the call to mediawiki is passing an invalid parameter called ssl_chain_file | 21:15 |
| clarkb | anteaya: you will need to look in the mediawiki class to see what it expects | 21:16 |
| ianw | SpamapS: oh, well it's how dib gets released now. i might send a mail | 21:16 |
| SpamapS | ah | 21:16 |
| SpamapS | so death to the signed tags method? | 21:16 |
| anteaya | clarkb: okay thanks | 21:16 |
| clarkb | ianw: SpamapS aiui that is opt in for projects? | 21:16 |
| clarkb | ianw: SpamapS I don't think it is required for dib | 21:16 |
| anteaya | clarkb: I'll look after neutron meeting | 21:16 |
| *** gyee has quit IRC | 21:17 | |
| *** trown is now known as trown|outtypewww | 21:18 | |
| *** jheroux has quit IRC | 21:19 | |
| ianw | clarkb: i think it has come under tripleo. yeah i think i will send a mail about opting-out. there was pretty much zero interest in running dib from git, so adding even more overheads is just maddening to anyone trying to contribute | 21:19 |
| *** julim has quit IRC | 21:20 | |
| *** amitgandhinz has quit IRC | 21:20 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Improved Summit Entity Event Processing https://review.openstack.org/307484 | 21:20 |
| *** mylu has joined #openstack-infra | 21:22 | |
| *** vhosakot has quit IRC | 21:22 | |
| *** asettle has joined #openstack-infra | 21:23 | |
| *** cloudtrainme has quit IRC | 21:24 | |
| *** mylu has quit IRC | 21:24 | |
| *** flwang1 has joined #openstack-infra | 21:25 | |
| *** mylu has joined #openstack-infra | 21:25 | |
| *** infra-red has quit IRC | 21:26 | |
| *** asettle has quit IRC | 21:27 | |
| anteaya | clarkb: it expects '/etc/ssl/certs/intermediate.pem' | 21:27 |
| *** lakshmiS_ has quit IRC | 21:28 | |
| prometheanfire | well, next! | 21:31 |
| openstackgerrit | Matthew Thode proposed openstack-infra/bindep: Add Gentoo support to bindep https://review.openstack.org/307493 | 21:31 |
| *** maratoid has joined #openstack-infra | 21:31 | |
| craige | o/ | 21:32 |
| fungi | prometheanfire: color me excited! | 21:33 |
| prometheanfire | fungi: ya, working on the final things needed to get testing for openstack-ansible on gentoo working | 21:33 |
| prometheanfire | after this is gentoo support for the bindep element in DIB, then openstack-infra caching a gentoo image for openstack-ansible's use | 21:34 |
| clarkb | mordred: what is nodepool + shade + neutron fips situation? check queue is an entire work day behind right now :/ unfortunate to lose osic and bluebox | 21:34 |
| *** nwkarsten has joined #openstack-infra | 21:34 | |
| clarkb | we almost need to talk to our providers and convince them to not all break at the same time | 21:34 |
| clarkb | (we are currently running on 2 providers and 3 regions) | 21:35 |
| fungi | prometheanfire: once you have the fallback package list in project-config gentooized, we'll probably want to add a functional testing job for gentoo to bindep as well (at least advisory/non-voting) | 21:35 |
| Goneri | https://review.openstack.org/#/c/306196/ Can I get review for this little change. (ability for ipxe to retry on failure) | 21:36 |
| *** zeih_ has joined #openstack-infra | 21:36 | |
| Goneri | -EWRONGCHAN | 21:36 |
| *** ddieterly[away] is now known as ddieterly | 21:38 | |
| *** elmiko has left #openstack-infra | 21:39 | |
| *** zeih has quit IRC | 21:40 | |
| *** |-paul-| has joined #openstack-infra | 21:40 | |
| *** zeih_ has quit IRC | 21:40 | |
| prometheanfire | fungi: what do I need to do in project-config? | 21:40 |
| prometheanfire | does the packge list from other-requirements go through pkg-map? | 21:41 |
| *** nwkarsten has quit IRC | 21:41 | |
| *** dimtruck is now known as zz_dimtruck | 21:42 | |
| fungi | prometheanfire: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/data/bindep-fallback.txt gets fed to bindep and the result passed to the distro package manager for our other current supported platforms | 21:43 |
| *** gyee has joined #openstack-infra | 21:43 | |
| *** matrohon has quit IRC | 21:43 | |
| prometheanfire | ah, k | 21:43 |
| fungi | prometheanfire: and our functional testing is basically just running the http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/install-distro-packages.sh wrapper | 21:43 |
| mordred | clarkb: sorry,I have a new issue I just found to try to track down ... it's WEIRD | 21:44 |
| prometheanfire | fungi: doesn't seem to bad so far | 21:44 |
| anteaya | clarkb: I haven't figured out how to pass that information in in a way it likes it: http://paste.openstack.org/show/494521/ | 21:44 |
| prometheanfire | fungi: question about the un-platformed lines in the fallback | 21:44 |
| clarkb | mordred: another issue with floating IPs? | 21:44 |
| prometheanfire | gentoo specifies pacakges as category/package | 21:44 |
| clarkb | anteaya: ya that isn't a valid parameter on the wiki class | 21:44 |
| anteaya | because I removed it, yeah I know | 21:45 |
| clarkb | anteaya: the wiki class is passing it to the mediawiki class and that is what breaks | 21:45 |
| prometheanfire | so ant would become dev-java/ant | 21:45 |
| anteaya | :( | 21:45 |
| mordred | clarkb: might be the same one - just looking at it another way | 21:45 |
| clarkb | I do not know why it breaks you will have to look at the mediawiki code | 21:45 |
| fungi | prometheanfire: yeah, those lines are assumed to apply to all platforms in use for bindep, so for example if bindep grew some way to look up the category that might be a simpler and more friendly solution | 21:45 |
| anteaya | clarkb: okay thanks for your help | 21:45 |
| mordred | clarkb: they're leaking basically immediately | 21:45 |
| prometheanfire | fungi: it's specifically a problem for curl | 21:46 |
| mordred | clarkb: I did a cleanup, and then had 3 unattached ips 3 seconds later | 21:46 |
| prometheanfire | dev-haskell/curl net-misc/curl | 21:46 |
| fungi | ooh, fun | 21:46 |
| prometheanfire | you can't just emerge curl, you need to specify | 21:46 |
| clarkb | mordred: woo, should we revert? | 21:46 |
| mordred | clarkb: no - not yet | 21:46 |
| clarkb | mordred: zuul is basically toast today | 21:46 |
| mordred | yah. gimme a few more | 21:46 |
| prometheanfire | a negitive mask would work for us though | 21:46 |
| *** pcrews_ has joined #openstack-infra | 21:47 | |
| mordred | I'm still trying to gather the info I need to work on a fix - but I think I'm close to having enough info | 21:47 |
| *** matrohon has joined #openstack-infra | 21:47 | |
| clarkb | prometheanfire: fungi maybe just have bindep do most specific match wins? | 21:47 |
| prometheanfire | changing curl to be 'curl [platform:!emerge]' and then having 'net-misc/curl [platform:emerge]' on another line | 21:47 |
| fungi | prometheanfire: yeah, that could take some noodling. easy way out is to make the package list even more messy and duplicative, though it would be nice if maybe it could just use the full package path in cases where disambiguation is required? | 21:47 |
| clarkb | prometheanfire: fungi then you can leave curl in there but add net-misc/curl[platform:emerge] in addition | 21:47 |
| prometheanfire | how would it know that curl and net-misc/curl are the same | 21:48 |
| clarkb | the other way is to do things the dib way and use a logical name in the list with a lookup mapping for names that is more specific | 21:48 |
| prometheanfire | it sounds more like we need something like pkg-map | 21:48 |
| jeblair | clarkb: perhaps openstack releases encourage our providers to break at the same time | 21:48 |
| clarkb | prometheanfire: oh right | 21:48 |
| prometheanfire | do negative matches work? | 21:48 |
| prometheanfire | if so then we could do that instead without having to do more code | 21:49 |
| mordred | clarkb: ok. so - I cleared the fip leaks from osic and they don't seem to be leaking this time | 21:49 |
| clarkb | mordred: did the code change? | 21:49 |
| mordred | clarkb: I'm not sure why the previous leaks happened so quickly | 21:49 |
| mordred | clarkb: I mean, the one patch from earlier, but I already did somethign with those nodes | 21:49 |
| mordred | so I'm still sorting through it | 21:49 |
| clarkb | mordred: the wait patch? | 21:50 |
| mordred | yah | 21:50 |
| *** pcrews has quit IRC | 21:50 | |
| fungi | prometheanfire: if someone needs curl then the list specifies net-misc/curl explicitly but if they need wget we just use a plain wget entry in the list? | 21:50 |
| prometheanfire | fungi: at the moment we only have one package 'wget' | 21:50 |
| prometheanfire | but someone could add wget to another category just the same | 21:51 |
| fungi | prometheanfire: right. i guess the tricky part is that can bitrot if a second wget is added in a new section | 21:51 |
| prometheanfire | :D | 21:51 |
| *** sdake has quit IRC | 21:51 | |
| prometheanfire | Debian package names are used as a single source of truth - bindep can be taught the mapping onto specific packaging systems. | 21:52 |
| prometheanfire | would be useful to know what that means | 21:52 |
| *** matt-borland has quit IRC | 21:52 | |
| fungi | prometheanfire: i think lifeless was interested in adding something similar to pkg-map as a future feature of bindep (e.g. a built-in list of common resolutions) to keep the file lists people supply simpler, but that's not really been fleshed out afaik | 21:53 |
| *** nwkarsten has joined #openstack-infra | 21:53 | |
| fungi | so that comment in the docs might have been premature/predictive | 21:53 |
| jeblair | somehow osic thinks we have exceeded quota | 21:54 |
| fungi | jeblair: floating ips again? | 21:54 |
| *** IlyaG has joined #openstack-infra | 21:54 | |
| jeblair | instances | 21:54 |
| fungi | or ram quota? | 21:54 |
| * mordred is poking | 21:54 | |
| fungi | ahh | 21:54 |
| *** ddieterly is now known as ddieterly[away] | 21:54 | |
| jeblair | nodepool knows about 83 instances, osic says we are over 100 | 21:54 |
| *** mylu has quit IRC | 21:54 | |
| prometheanfire | fungi: boo | 21:54 |
| mordred | jeblair: I think it's related to the fip issue | 21:54 |
| prometheanfire | fungi: it does look like netative mapping works | 21:55 |
| fungi | prometheanfire: nifty | 21:56 |
| mordred | jeblair: yah. alien-list has 17 instances | 21:56 |
| fungi | prometheanfire: i'm in favor of whatever we need to do to balance platform support and workingness | 21:56 |
| *** stewie925 has quit IRC | 21:56 | |
| *** amrith is now known as _amrith_ | 21:56 | |
| jeblair | mordred, fungi: the multi-nodes bring us up to 99 instances | 21:57 |
| prometheanfire | ya | 21:57 |
| *** bpokorny_ has quit IRC | 21:57 | |
| prometheanfire | it looks like bindep will need to support 'foo/bar' package names though | 21:57 |
| prometheanfire | and if it is outputting a version it needs to prepend = to that | 21:57 |
| *** tonytan4ever has quit IRC | 21:58 | |
| fungi | prometheanfire: when i began hacking on it, deb support had been implemented, and rpm was an exercise still left to the reader... so this is a bit of a finding our path through the darkness exercise | 21:58 |
| jeblair | mordred: grep 259241 debug.log is interesting | 21:58 |
| prometheanfire | ya | 21:58 |
| stevemar | anteaya: ALL THE SUBJECT TAGS! | 21:58 |
| mordred | jeblair: I agree, that is interesting | 21:59 |
| fungi | prometheanfire: also the explicit version support may still be incomplete. the mechanisms it needs to do version lookups, at least for rpm-based platforms, may yet be missing | 21:59 |
| anteaya | stevemar: you missed a few, but that is a very impressive subject line | 21:59 |
| jeblair | mordred: it created the server, deleted it, but the subnode continued to build, and now is leaked, but for some reason seems unable to be cleaned up | 21:59 |
| jeblair | mordred: probably the most relevant part of that is why its unable to delete the leaked subnode | 21:59 |
| jeblair | (the rest could be normal behavior) | 22:00 |
| stevemar | anteaya: it's mostly "here's a random script that is unmaintained and untested, but is used to create keystone `data` " | 22:00 |
| prometheanfire | fungi: it looked like it was there when I was editing | 22:00 |
| mordred | TypeError: delete_floating_ip() got an unexpected keyword argument 'wait' | 22:00 |
| *** fawadkhaliq has quit IRC | 22:00 | |
| fungi | prometheanfire: oh, that's a good sign then | 22:00 |
| mordred | jeblair: that would be the problem | 22:00 |
| jeblair | mordred: ayup that'll do it | 22:00 |
| mordred | jeblair: THANK YOU | 22:00 |
| mordred | that was the thing I was missing | 22:00 |
| jeblair | yay! | 22:00 |
| anteaya | stevemar: yeah I got some of that | 22:01 |
| fungi | prometheanfire: the main reason i say version matching may not be fully implemented is that up to now at least we've been focused on platforms that rarely give you a choice of multiple versions of a package unless you're doing complex pinning between different suites/releases/distros | 22:01 |
| prometheanfire | and I think I have gentoo package parsing working | 22:02 |
| fungi | excellent | 22:02 |
| *** ociuhandu has quit IRC | 22:02 | |
| lifeless | fungi: pkg-map, I don't recall. Uhm, it was the abstraction layer, and it is very similar to bindep. | 22:03 |
| mordred | jeblair: I'm going to restart nodepool to remove that change | 22:04 |
| fungi | lifeless: right, you had expressed to me at one point that you were keen to bake in some pre-resolved platform-specific alternatives for common packages within bindep to simplify cross-platforms package lists people might feed to it | 22:04 |
| mordred | I think the previous leak rate was better than this at least | 22:04 |
| prometheanfire | yep, dev-lang/pythonasd shows up :D | 22:04 |
| fungi | lifeless: though i'm still not convinced it's needed or would be easy to maintain | 22:05 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Rework migration 1679b5bc102c to not use the db api https://review.openstack.org/301369 | 22:05 |
| *** bpokorny has joined #openstack-infra | 22:05 | |
| *** jrmyjohn has quit IRC | 22:05 | |
| mordred | jeblair: ok. restarted. leaked fip list for bb and osic are now 0 | 22:06 |
| mordred | and I'm watching the lists | 22:06 |
| clarkb | mordred: what code are we running now? | 22:07 |
| mordred | master | 22:07 |
| mordred | we were running with the wait patch added. now we're back to master | 22:07 |
| *** nikhil_k has joined #openstack-infra | 22:07 | |
| mordred | the wait patch, it turns out, was anti-helpful | 22:07 |
| openstackgerrit | Matthew Thode proposed openstack-infra/bindep: Add Gentoo support to bindep https://review.openstack.org/307493 | 22:07 |
| prometheanfire | fungi: there's the category fix | 22:08 |
| fungi | prometheanfire: awesome, thanks | 22:08 |
| lifeless | fungi: yeah. Shrug, what we have seems nice enough. | 22:08 |
| clarkb | mordred: but master is also broken | 22:08 |
| lifeless | right, ESTUFFTODO. | 22:08 |
| prometheanfire | well, it's both, not sure it should be split since it's really needed for gentoo support | 22:08 |
| prometheanfire | huh, platform:gentoo and platform:emerge both work, neat I guess | 22:09 |
| mordred | clarkb: right. but it's not _completely_ broken which the wait patch is. I can keep up with the leaking rate of master while poking | 22:09 |
| fungi | prometheanfire: platform:emerge would in theory be used for all gentoo derivatives as well, while platform:gentoo _should_ just apply to actual gentoo | 22:10 |
| fungi | prometheanfire: in most cases i'd recommend sticking to the broadest applicable classification, but for the gentoo ecosystem i have no idea what that actually means | 22:10 |
| clarkb | huh does that mean we will support coreos? | 22:10 |
| fungi | if it too uses emerge, then maybe? | 22:11 |
| prometheanfire | fungi: probably 'gentoo' | 22:11 |
| *** _amrith_ is now known as amrith | 22:11 | |
| prometheanfire | ya, dunno | 22:11 |
| prometheanfire | gentoo ONLY uses emerge | 22:11 |
| *** nikhil_k has quit IRC | 22:11 | |
| *** gordc has quit IRC | 22:11 | |
| prometheanfire | some others use other package managers but still use ebuilds and our category system | 22:11 |
| prometheanfire | so, it's complicated, like all things gentoo lol | 22:12 |
| fungi | well, i think it's fair to say that bindep only uses the _preferred_ package manager for a given platform (after all, you can run rpm on debian, dpkg on fedora, pip and npm most places, et cetera) | 22:12 |
| prometheanfire | good point | 22:13 |
| prometheanfire | still think gentoo is correct but it likely won't matter and we can change it if needed | 22:14 |
| *** hichihara has quit IRC | 22:14 | |
| * prometheanfire needs to look up how to do an external dependant review for the project config gentoo bindep support stuff | 22:15 | |
| clarkb | fungi: I once had to use a suse distro with packages primarily provided via apt | 22:15 |
| clarkb | fungi: do not recommend | 22:15 |
| *** imcsk8 has quit IRC | 22:15 | |
| prometheanfire | gentoo can use deb/apt/rpm as well, but we don't recommend it | 22:15 |
| *** imcsk8 has joined #openstack-infra | 22:15 | |
| prometheanfire | the sources our ebuilds pull down as the 'tarball' are sometimes deb/rpm | 22:16 |
| bkero | prometheanfire: paludis? | 22:16 |
| prometheanfire | bkero: ya, as an example | 22:16 |
| fungi | prometheanfire: i think depends-on isn't going to be of much help for the relevant jobs, but also the existing jobs aren't exercising the fallback list's gentoo-specific entries anyway so should probably be fine to just work on in parallel | 22:16 |
| *** ddieterly[away] is now known as ddieterly | 22:16 | |
| *** tpsilva has quit IRC | 22:17 | |
| prometheanfire | true | 22:17 |
| prometheanfire | I'll work on that when I get home then | 22:17 |
| mordred | jeblair, clarkb: SOOOOOOO | 22:17 |
| mordred | I'm watching the API calls | 22:17 |
| mordred | and nodepool/shade is totally making the right calls as we expect - and we just leaked a FIP on osic | 22:18 |
| mordred | my best theory now is that sometimes neutron just doesn't delete the fip when you ask it to | 22:18 |
| clarkb | mordred: yes, we alread yknow that, it is why we leak fips :) | 22:18 |
| clarkb | I tried explaining this way back when | 22:18 |
| mordred | well - before there was one more piece in the puzzle, which is that we were asking nova to ask neutron to delete the fip | 22:19 |
| clarkb | the issue is we lose track of that fip once the server is gone | 22:19 |
| mordred | now we're just talking to neutron | 22:19 |
| mordred | so I think the wait patch is the right choice - it just needs more code added to it | 22:19 |
| mordred | you konw - to actually wait | 22:19 |
| *** Sukhdev has quit IRC | 22:20 | |
| *** ddieterly is now known as ddieterly[away] | 22:20 | |
| mordred | so - revised patch coming ... and I'll keep the leaked fip count down in the mean time | 22:20 |
| mordred | (I was more worried that we were sometimes not calling the right thing for $reasons) | 22:20 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:20 | |
| *** MarkAtwood has quit IRC | 22:21 | |
| mtreinish | mordred: isn't a neutron floating ip delete sync? | 22:22 |
| mordred | mtreinish: one would think - except sometimes it doesn't work | 22:24 |
| clarkb | mordred: can we blame NAT? | 22:24 |
| *** nwkarsten has quit IRC | 22:24 | |
| fungi | returns 200 ok when it failed (or when it hasn't finished)? | 22:24 |
| nibalizer | clarkb: always | 22:24 |
| mordred | mtreinish: so in this case, "wait" is actually going to be "while not timeout: make the delete call, check to see if it deleted" | 22:24 |
| *** nwkarsten has joined #openstack-infra | 22:25 | |
| mordred | fungi: unknown - but I'm not seeing exceptions currently at all | 22:25 |
| mordred | fungi: so I believe it's silently just not deleting | 22:25 |
| clarkb | mordred: yes that is the behavior I am amiliar with | 22:25 |
| clarkb | basically delete, get 200 back, nothing deletes | 22:25 |
| mordred | yup | 22:25 |
| mtreinish | mordred: right that's a pretty standard pattern. (tempest does it all the time) but for floating ip deletes neutron returns a 204 not a 202 | 22:25 |
| mtreinish | that seems like it would be a neutron bug | 22:25 |
| *** ddieterly[away] is now known as ddieterly | 22:26 | |
| mordred | mtreinish: yes. it does doesn't it? | 22:26 |
| *** hichihara has joined #openstack-infra | 22:26 | |
| *** ayoung has quit IRC | 22:27 | |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/reviewday: Use pip to install gerrit-dash-creator https://review.openstack.org/307510 | 22:27 |
| *** matrohon has quit IRC | 22:27 | |
| *** yamahata has quit IRC | 22:27 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: grafana: added periodic oslo master jobs to unit test 'gate' dashboard https://review.openstack.org/307511 | 22:28 |
| *** markvoelker has quit IRC | 22:28 | |
| *** Adri2000 has quit IRC | 22:29 | |
| *** vgridnev_ has quit IRC | 22:30 | |
| *** ihrachys has quit IRC | 22:31 | |
| *** ddieterly is now known as ddieterly[away] | 22:31 | |
| *** stewie925 has joined #openstack-infra | 22:33 | |
| *** yamahata has joined #openstack-infra | 22:34 | |
| *** nwkarsten has quit IRC | 22:34 | |
| *** nwkarsten has joined #openstack-infra | 22:35 | |
| *** ddieterly[away] is now known as ddieterly | 22:35 | |
| *** camunoz has joined #openstack-infra | 22:36 | |
| *** nwkarsten has quit IRC | 22:39 | |
| *** ddieterly is now known as ddieterly[away] | 22:42 | |
| JayF | fungi: re: earlier conversation about IPA builds; https://review.openstack.org/#/c/307474/ changes it to use http.us. I'll work on something that allows something external to set a proxy so we'll be ready to drop it in when the infra proxy exists. | 22:42 |
| *** ddieterly[away] is now known as ddieterly | 22:44 | |
| fungi | JayF: well, we don't have proxies. we have actual mirrors. so while that may be what you meant, i just want to be clear that what you'd really want is to at least be able to set it to use the package repository urls local to the workers where those jobs are running, and ideally also copy in packages we've pre-cached on the worker image | 22:44 |
| *** eliqiao_ has joined #openstack-infra | 22:44 | |
| JayF | I meant mirror, absolutely | 22:44 |
| fungi | okay, cool | 22:45 |
| JayF | but we can't use it that way unless we change the IPA container back to ubuntu | 22:45 |
| *** eliqiao has quit IRC | 22:45 | |
| JayF | it was changed to debian after $canonical_copyright_shenanigans | 22:45 |
| *** sdague has quit IRC | 22:45 | |
| JayF | so unless debian is used in the gate, we can't steal mirror info verbatim. | 22:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Wait for floating ip deletion as part of server deletion https://review.openstack.org/307439 | 22:45 |
| fungi | right, so once we have a debian mirror covering the releases/suites you need | 22:45 |
| JayF | How will you all communicate that? Env variable? Sources.lists files in devstack builds I can pull in? | 22:46 |
| JayF | If you lmk what the interface is now, I can start building to it ;) | 22:46 |
| JayF | it's been on my list for a while to make IPA builds use more of this infra. | 22:46 |
| JayF | I guess I should go down the path of making it respect upper-constraints and use the pypi mirrors as well | 22:46 |
| mordred | clarkb, jeblair, Shrews: ^^ that patch above I think should do it | 22:46 |
| fungi | JayF: we keep metadata in /etc/nodepool like so http://paste.openstack.org/show/494526/ | 22:48 |
| *** |-paul-| has quit IRC | 22:49 | |
| fungi | JayF: fwiw, we don't consider that a guaranteed stable interface, but if you're trying to infer a debian mirror from an ubuntu worker that might be the only simple solution in the near term | 22:49 |
| JayF | fungi: Well, my hope was more like, MIRROR_HOST= available somewhere | 22:49 |
| JayF | fungi: would you all take a patch to put it in /etc/nodepool/provider? | 22:50 |
| fungi | JayF: actually, yeah that seems like it would probably be simpleish for us to abstract out | 22:51 |
| clarkb | JayF: fungi I think dib already has something for that, eg the images don't need anything new | 22:51 |
| clarkb | my preference would be not to bake more into the image if so | 22:51 |
| JayF | This build isn't using dib | 22:51 |
| fungi | clarkb: can you clarify? | 22:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: register jobs from JJB https://review.openstack.org/307515 | 22:51 |
| *** ddieterly is now known as ddieterly[away] | 22:51 | |
| jeblair | mordred, clarkb, fungi: zuulv2.5 part 2: parse jjb configs from yaml and register jobs | 22:52 |
| jeblair | mordred, clarkb, fungi: unanticipated benefit: jjb updates will take approx 5 seconds in the future | 22:52 |
| fungi | bwahahahaha | 22:52 |
| clarkb | fungi: JayF the dib tests run on ubuntu and centos, they then build ubuntu, centos, fedora, debian on those two host distros | 22:52 |
| jeblair | (i just imported jjb and told it to parse the yaml and then work from the resulting data structure) | 22:52 |
| clarkb | fungi: JayF when doing this the tests configure dib to use the region local mirrors, eg we alreayd provide sufficient data to do this | 22:53 |
| fungi | that's more or less what i expected libified jjb would be used for | 22:53 |
| clarkb | jeblair: hahaha nice | 22:53 |
| JayF | hmm. I guess I'll look at what that pulls from. | 22:53 |
| fungi | clarkb: so we've hard-coded mirror.%s.%s.openstack.org in or something? | 22:54 |
| fungi | just curious how that's getting passed in if not from the current NODEPOOL_CLOUD and NODEPOOL_REGION values | 22:54 |
| clarkb | fungi: no, it is all region specific | 22:54 |
| jeblair | oops, left a debug thing in; update incoming | 22:54 |
| clarkb | fungi: I think it may be from that | 22:54 |
| clarkb | fungi: I am saying do not make a MIRROR_HOST var | 22:55 |
| clarkb | because it wil lend up being a lock around any changes to our mirroring | 22:55 |
| JayF | So they have region-specific configs? | 22:55 |
| clarkb | JayF: yes they generate them when the job runs iirc | 22:55 |
| clarkb | (and it already can take us several days to update a single image) | 22:55 |
| fungi | clarkb: just pointing out that if so then we've hard-coded an assumption there that the mirror name will be some mirror.%s.%s.openstack.org pattern, but we could in theory provide the actual mirror hostname in /etc/nodepool/provider via the ready script | 22:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: register jobs from JJB https://review.openstack.org/307515 | 22:56 |
| clarkb | fungi: yes, I think hard coding the assumption is better than hard coding on the image | 22:56 |
| JayF | ++ fungi that's sorta what I was wondering, it sounds like if I do what DIB is doing, there's now a bonus place to update if mirrors change | 22:56 |
| fungi | clarkb: but /etc/nodepool/provider isn't hard-coded in the image | 22:56 |
| clarkb | even better is inheriting from the system | 22:56 |
| clarkb | because we already configure all of this stuff | 22:56 |
| fungi | oh, i get you. you mean avoid baking that url into the ipa image | 22:57 |
| fungi | i'm completely onboard with that | 22:57 |
| JayF | I can't inherit from the system unless I make broad assumptions that whatever host is used in /etc/apt/sources.list locally is going to serve debian too | 22:57 |
| clarkb | JayF: if you build on debian you can | 22:57 |
| JayF | fungi: clarkb: oh, it'd 100% be my intention to ship an image with a "clean" sources.list, pointing back to upstream mirrors | 22:57 |
| mordred | jeblair: awesome! | 22:58 |
| *** sdague has joined #openstack-infra | 22:58 | |
| clarkb | dib is a bit special because they are specifically testing the cross build setup | 22:58 |
| JayF | clarkb: that's a dependency chain of pain. We build IPA from source for some devstack jobs, and so it's built when devstack comes up... basically the deps on that resolve to 'run all Ironic tests on debian" | 22:58 |
| *** dingyichen has joined #openstack-infra | 22:58 | |
| clarkb | oh you build on demand for every job? | 22:58 |
| JayF | Not for every job, but for a class of jobs we do, yes | 22:58 |
| JayF | when we run tempest against new IPA code | 22:58 |
| JayF | These are the jobs that are failing without a mirror setup today. | 22:59 |
| clarkb | I see | 22:59 |
| JayF | I mean, I can make our automation try to pull a mirror from sources.list. I don't love it but I could make that whole set of assumptions. | 23:00 |
| clarkb | in that case I can see writing out a mirror_host type var which is the root url | 23:00 |
| clarkb | then expect consumers to know what the subpaths are (this is still an assumption but less of one I guess) | 23:01 |
| JayF | That's sorta what I wanted/was asking for | 23:01 |
| JayF | The nice thing about a var is it's a friendly interface for others as well | 23:01 |
| JayF | for instance, if I build an IPA ramdisk on my own on a Rackspace cloud server, I can MIRROR_HOST=mirror.rackspace.com make | 23:01 |
| JayF | and we haev a lot of devs behind restrictive firewalls or international, so that'd probably be appreciated. | 23:02 |
| *** ddieterly[away] is now known as ddieterly | 23:04 | |
| *** ddieterly is now known as ddieterly[away] | 23:04 | |
| *** ddieterly[away] has quit IRC | 23:04 | |
| *** doug-fish has quit IRC | 23:06 | |
| *** vhosakot has joined #openstack-infra | 23:06 | |
| *** xyang1 has quit IRC | 23:07 | |
| anteaya | if a channel op feels like refreshing the topic in -meeting, that would be great | 23:08 |
| fungi | done, thanks anteaya | 23:09 |
| jeblair | mordred: you still have a -2 on 307439 | 23:10 |
| anteaya | fungi: thank you | 23:10 |
| mordred | jeblair: thanks | 23:10 |
| mordred | jeblair: does that approach look sane-ish? | 23:11 |
| *** rbrndt_ has quit IRC | 23:11 | |
| jeblair | mordred: looking now | 23:11 |
| *** MarkAtwood has joined #openstack-infra | 23:12 | |
| *** asettle has joined #openstack-infra | 23:13 | |
| jeblair | clarkb, fungi, JayF: instead of/in addition to MIRROR_HOST, we could do DEBIAN_MIRROR_URL or something, and do that in the ready script for every kind of mirror we have (so that our paths are explicit in the vars rather than assumed) | 23:13 |
| *** EricGonc_ has joined #openstack-infra | 23:13 | |
| *** EricGonczer_ has quit IRC | 23:13 | |
| *** Sukhdev has joined #openstack-infra | 23:15 | |
| jeblair | mordred: yes generally. -1 with some comments | 23:16 |
| mordred | jeblair: cool | 23:16 |
| mordred | jeblair: I think we'd also need to set a list of pockets | 23:17 |
| jeblair | mordred: i'm trying to parse that sentence and it's all coming up funny | 23:17 |
| *** IlyaG has quit IRC | 23:17 | |
| openstackgerrit | Merged openstack/diskimage-builder: Prioritize venv python on host https://review.openstack.org/285885 | 23:17 |
| mordred | jeblair: like DEBIAN_MIRROR_POCKETS='jessie:main,contrib,updates' | 23:17 |
| *** stevelle has joined #openstack-infra | 23:17 | |
| stevelle | is there another place I should go to ask a Q about the mobile app for this summit? | 23:18 |
| jeblair | mordred: ah! | 23:18 |
| anteaya | stevelle: I don't know as there is a channel for that | 23:19 |
| anteaya | stevelle: are you making an inquiry? perhaps you should email the foundation | 23:19 |
| anteaya | stevelle: are you reporting a bug? | 23:19 |
| clarkb | mordred: I too have commented on that change | 23:19 |
| mordred | or, better example: UBUNTU_MIRROR_POCKETS='trusty:main,universe trusty-backports:main,universe xenial:main,universe" | 23:19 |
| jeblair | mordred: but mostly i was thinking about how to know to set it to /debian or /ceph-deb-hammer or whatnot | 23:19 |
| mordred | jeblair: totally | 23:19 |
| mordred | but with trusty, we actually have 4 base urls and then 3 pockets in each one | 23:20 |
| anteaya | stevelle: if you need to file a bug, here is the best place: https://bugs.launchpad.net/openstack-org | 23:20 |
| stevelle | anteaya: I am trying to login to the app. I don't know what credentials the app is asking for, "forgot password" is not delivering me an email I can find. I'm trying to figure out what to search for | 23:20 |
| stevelle | doubt this is a bug | 23:20 |
| stevelle | think it's just user error | 23:20 |
| anteaya | nope sounds like a bug to me | 23:20 |
| anteaya | if it is a user error and you don't know what to do next, that is a bug | 23:21 |
| jeblair | anteaya: ++ | 23:21 |
| clarkb | stevelle: it is your openstackid credentials. the same credentials you use to login to your openstack foundation profile | 23:21 |
| anteaya | stevelle: https://bugs.launchpad.net/openstack-org is my best suggestion | 23:21 |
| clarkb | stevelle: https://www.openstack.org/profile/ that thing | 23:21 |
| *** claudiub has quit IRC | 23:22 | |
| *** zz_dimtruck is now known as dimtruck | 23:22 | |
| stevelle | clarkb: apparently I'm logged in on the browser... thx that gives me something to work from | 23:22 |
| JayF | mordred: jeblair: I'd personally love that, but it seemed a little ... stronger in scope than folks wanted at first | 23:25 |
| anteaya | I've replied to the email telling everyone they have to get the app requesting the tell folks where to go for app support | 23:25 |
| dstanek | is the code for the summit calendar accepting pull requests? i'd love to fix up the real calendar instead of using my hack | 23:25 |
| anteaya | dstanek: https://bugs.launchpad.net/openstack-org | 23:25 |
| jhesketh | Morning | 23:26 |
| anteaya | dstanek: that is the only method I know of communicating with them | 23:26 |
| anteaya | jhesketh: morning | 23:26 |
| dstanek | anteaya: who is them? | 23:26 |
| clarkb | anteaya: dstanek they do have a github repo and I think they do accept pull requests | 23:26 |
| clarkb | let me get a link | 23:26 |
| anteaya | dstanek: the contractors the foundation hired to create the schedule | 23:26 |
| dstanek | clarkb: that would be cool | 23:26 |
| dstanek | anteaya: ah | 23:27 |
| dstanek | we should have just used the software pycon uses | 23:27 |
| *** EricGonc_ has quit IRC | 23:27 | |
| clarkb | https://github.com/OpenStackweb there | 23:27 |
| dstanek | clarkb: thanks | 23:27 |
| jeblair | dstanek: that would be nice. but the foundation makes these decisions without consulting the community. :( | 23:28 |
| *** markvoelker has joined #openstack-infra | 23:28 | |
| *** Qiming has joined #openstack-infra | 23:29 | |
| fungi | lifeless: mordred: btw, the same downstream from last week reported another pbr 1.9 regression https://review.openstack.org/306082 (additional effort for supporting older git still heavily in use in the wild) | 23:31 |
| clarkb | fungi: how old? | 23:31 |
| dstanek | ah, bummer. it looks like it's all PHP. not sure i have enough time to hack up a calendar view. | 23:32 |
| fungi | clarkb: current debian stable, and ubuntu precise | 23:32 |
| clarkb | so 1.7 ish | 23:32 |
| anteaya | dstanek: :( | 23:32 |
| clarkb | which is known to be really buggy :( | 23:32 |
| *** amrith is now known as _amrith_ | 23:33 | |
| *** markvoelker has quit IRC | 23:33 | |
| *** Adri2000 has joined #openstack-infra | 23:33 | |
| *** Adri2000 has quit IRC | 23:33 | |
| *** Adri2000 has joined #openstack-infra | 23:33 | |
| fungi | yeah, buggy but present in releases distros are still supporting | 23:33 |
| *** ddieterly has joined #openstack-infra | 23:35 | |
| *** ddieterly is now known as ddieterly[away] | 23:35 | |
| *** salv-orlando has quit IRC | 23:35 | |
| dstanek | anteaya: yeah. i'll just stick with mine for now. | 23:35 |
| fungi | oh, wait, debian wheezy (which is oldstable but still ostensibly "lts" supported) | 23:36 |
| *** MarkAtwood has quit IRC | 23:36 | |
| *** nwkarsten has joined #openstack-infra | 23:36 | |
| *** ddieterly[away] is now known as ddieterly | 23:37 | |
| lifeless | fungi: thanks for the headsup | 23:37 |
| mordred | dstanek: you have a calendar hack? | 23:37 |
| dstanek | mordred: i hacked up http://bit.ly/23SzXec to allow me to see overlapping sessions | 23:38 |
| *** eil397 has left #openstack-infra | 23:38 | |
| anteaya | dstanek: you speak wisdom, you do | 23:39 |
| clarkb | someone else hacked together ical feed too | 23:39 |
| clarkb | dstanek: how did you get the source data? the current data is behind some cookie wall | 23:39 |
| clarkb | so Igave up after about 5 minutes a wekeend or two ago | 23:39 |
| dstanek | clarkb: i took my cookie from chrome and used these two scripts http://bit.ly/1S5AVQi | 23:40 |
| dstanek | i just pulled the session json and put together a crude calendar | 23:41 |
| *** nwkarsten has quit IRC | 23:41 | |
| anteaya | dstanek: thank you, your crude calendar is awesome! | 23:41 |
| dstanek | clarkb: do you remember where you saw the ical feed? i was going to do that too, but if it's already done | 23:41 |
| clarkb | dstanek: yes give me a few I can find it in history somewhere | 23:41 |
| dstanek | clarkb: neat, thx | 23:41 |
| dstanek | anteaya: thx. i couldn't figure out my calendar without seeing the parallel tracks | 23:42 |
| anteaya | I hear ya | 23:42 |
| anteaya | I was putting it off myself because I couldn't visualize parallel tracks | 23:42 |
| anteaya | dstanek: you are a wonderful person, you are! | 23:42 |
| clarkb | dstanek: https://twitter.com/boot_vmlinuz/status/719149519972073473 | 23:43 |
| *** nwkarsten has joined #openstack-infra | 23:44 | |
| dstanek | clarkb: thx | 23:44 |
| dstanek | i'm going to see if i can do something like that for just the sessions i'm attending | 23:47 |
| fungi | it did take me a while when building the infra summit session schedule to work out who was conflicting with what so i could optimize session placement | 23:47 |
| clarkb | I will likely just print off a pdf friday night and make notes on paper | 23:48 |
| fungi | that's my plan. fold up, stick in bag, get on flight | 23:48 |
| *** nwkarsten has quit IRC | 23:49 | |
| fungi | i'm pleased it has a decent print view | 23:50 |
| clarkb | mordred: looks like we are up to 35 leaked fips in osic | 23:50 |
| anteaya | ha ha ha | 23:50 |
| anteaya | dstanek: if you figure that out, anne gentle wants the same thing | 23:50 |
| anteaya | dstanek: not sure if she did one for herself yet | 23:51 |
| *** mriedem is now known as mriedem_away | 23:51 | |
| fungi | i get the impression customizable ical feeds is something that dev team plans to have implemented by barcelona | 23:52 |
| *** ddieterly has quit IRC | 23:53 | |
| *** johnny1 has quit IRC | 23:55 | |
| *** erlon_ has quit IRC | 23:55 | |
| mordred | clarkb: fixed | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!