Tuesday, 2016-06-14

*** ffrank has quit IRC00:01
*** ihrachys has joined #openstack-infra00:01
*** xarses has quit IRC00:05
*** vhosakot has quit IRC00:05
*** EricGonczer_ has quit IRC00:08
*** anteaya has quit IRC00:09
pabelangerrcarrillocruz: looks like we have a DNS issue with baremetal00.vanilla, it is failing to resolve google DNS. Is there something local?00:09
*** anteaya has joined #openstack-infra00:09
*** Julien-zte has joined #openstack-infra00:10
clarkbpabelanger: we p robably dont want local as its been problematic in the past. we should make sure there are no firewall drop rules for our hosts00:11
*** ddieterly has joined #openstack-infra00:14
openstackgerritC.J. Collier proposed openstack-infra/gerritbot: Improve gerritbot error handling  https://review.openstack.org/28705600:14
*** claudiub has quit IRC00:15
zaroare you able post to SB using rest it’s rest api?00:16
zaroanteaya: i mean without gerri plugin?00:16
*** ihrachys has quit IRC00:17
*** sarob has quit IRC00:17
anteayazaro: good point, I should try that00:17
openstackgerritSachi King proposed openstack-infra/zuul: Support post jobs by supporting rev checkout  https://review.openstack.org/29319400:17
pabelangerclarkb: Agreed, will check firewalls. Was hoping something was local to testing networking00:18
Nakatojeblair: Newrev is up https://review.openstack.org/#/c/293194/1400:18
*** thorst has joined #openstack-infra00:19
pabelangerclarkb: current firewall settings: http://paste.openstack.org/show/515864/00:20
*** pvaneck has quit IRC00:20
*** tqtran has quit IRC00:21
*** sarob has joined #openstack-infra00:21
clarkbpabelanger: ya I am worried about hp firewalls not on host firewall00:21
*** gyee has quit IRC00:21
pabelangerclarkb: same00:21
clarkbnot being able to talk ti seems like hp firewall00:21
pabelangerI suspect that is the issue too00:22
anteayapleia2: have you any sense of why I would get a 302 returned when I get the systeminfo on the api when the same url resolves using my browser? http://paste.openstack.org/show/515865/00:22
*** Jeffrey4l has joined #openstack-infra00:23
openstackgerritPaul Belanger proposed openstack-infra/system-config: Remove vexxhost from nodepool.o.o clouds.yaml  https://review.openstack.org/32923400:25
*** denisra has quit IRC00:25
pabelangervexxhost properly cleaned up in nodepool.o.o now00:25
pleia2anteaya: hm, that's a temporary redirect notice00:26
*** piet_ has joined #openstack-infra00:26
*** denisra has joined #openstack-infra00:26
anteayapleia2: fair enough, why are we doing that?00:27
*** yuanying has quit IRC00:27
*** lvdongbing has joined #openstack-infra00:27
pleia2anteaya: oh, I think it's because http > https00:27
pleia2we're forcing use of https00:27
*** yuanying has joined #openstack-infra00:27
pleia2so as far as I can tell, you're going to http and it's sending you to https00:27
*** baoli has quit IRC00:28
pleia2anteaya: try: curl --insecure https://storyboard-dev.openstack.org/api/v1/systeminfo00:28
*** baoli has joined #openstack-infra00:28
anteayapleia2: I'll try that00:28
*** betherly has quit IRC00:28
*** betherly has joined #openstack-infra00:29
*** mtanin___ has joined #openstack-infra00:30
*** yamahata has quit IRC00:30
*** SotK has quit IRC00:31
*** mtanino has quit IRC00:31
*** SotK has joined #openstack-infra00:31
*** yamahata has joined #openstack-infra00:32
*** sflanigan has quit IRC00:34
*** sarob has quit IRC00:34
*** amitgandhinz has quit IRC00:35
*** kzaitsev_mb has quit IRC00:37
*** oanson has joined #openstack-infra00:39
*** jamesmcarthur has joined #openstack-infra00:40
*** yamada-h has joined #openstack-infra00:40
*** sarob has joined #openstack-infra00:40
*** thorst has quit IRC00:40
*** sarob has quit IRC00:40
*** bhavik has quit IRC00:40
*** zz_dimtruck is now known as dimtruck00:41
*** thorst has joined #openstack-infra00:41
*** jamesmcarthur has quit IRC00:44
*** yamada-h has quit IRC00:44
*** bhavik has joined #openstack-infra00:45
*** sputnik13_ has quit IRC00:46
*** sputnik13 has joined #openstack-infra00:47
*** Sukhdev has quit IRC00:49
*** thorst has quit IRC00:49
*** sflanigan has joined #openstack-infra00:51
anteayapleia2: sorry yes that was the issue when I tried this last week, thanks, I have no memory for anything anymore00:51
*** baoli has quit IRC00:53
*** dimtruck is now known as zz_dimtruck00:57
*** bpokorny_ has joined #openstack-infra01:01
*** bpokorny has quit IRC01:04
*** zul has joined #openstack-infra01:04
*** Apoorva_ has quit IRC01:04
*** bpokorny_ has quit IRC01:05
*** amotoki has joined #openstack-infra01:06
*** bhavik has quit IRC01:07
*** bhavik has joined #openstack-infra01:11
*** Daisy has joined #openstack-infra01:12
*** baoli has joined #openstack-infra01:16
*** rhallisey has quit IRC01:16
*** amotoki has quit IRC01:21
*** hongbin has joined #openstack-infra01:24
*** baoli has quit IRC01:26
*** grassass has joined #openstack-infra01:26
*** ddieterly is now known as ddieterly[away]01:28
*** bhavik has quit IRC01:29
*** rwsu has quit IRC01:30
*** yanyanhu has joined #openstack-infra01:32
*** ddieterly[away] has quit IRC01:33
*** Goneri has joined #openstack-infra01:34
*** thorst has joined #openstack-infra01:34
*** Daisy has quit IRC01:34
*** Daisy has joined #openstack-infra01:35
*** Daisy_ has joined #openstack-infra01:35
*** amitgandhinz has joined #openstack-infra01:36
*** Daisy_ has quit IRC01:36
*** Daisy_ has joined #openstack-infra01:36
openstackgerritNuman Siddique proposed openstack-infra/project-config: Add networking-ovn in the 'PROJECTS' for dsvm-functional job  https://review.openstack.org/32924101:37
*** Goneri has quit IRC01:38
*** piet_ has quit IRC01:39
*** Daisy has quit IRC01:39
*** amotoki has joined #openstack-infra01:39
*** tkelsey has joined #openstack-infra01:40
*** zz_dimtruck is now known as dimtruck01:41
*** zul has quit IRC01:41
*** amitgandhinz has quit IRC01:41
*** yamada-h has joined #openstack-infra01:43
*** tkelsey has quit IRC01:44
anteayapleia2: if you are still around I just tried `curl --insecure -i https://storyboard-dev.openstack.org/api/v1/stories/11/tasks/12 -X DELETE` which gave me a 500 but the way I read the documenation it is a valid command: http://docs.openstack.org/infra/storyboard/webapi/v1.html#delete--v1-stories--story_id--tasks01:45
anteayait is possible I am not doing something I should know to do01:45
anteayapleia2: I am curious what the storyboard-dev logs say in response to my DELETE attempt01:46
*** yamada-h has quit IRC01:47
*** sflanigan has quit IRC01:48
*** bhavik has joined #openstack-infra01:48
pleia2anteaya: I am working on other things this evening and haven't looked at the storyboard API at all, but presumably you need some kind of authentication token in order to do any edit-type tasks01:48
anteayaokay thanks, sorry to bother you01:49
anteayaand yeah that would make sense01:49
*** ddieterly has joined #openstack-infra01:49
pleia2anteaya: the straight curl you're doing doesn't pass auth info, so I think it's giving a server error since it doesn't know what to do if you're not "logged in"01:49
anteayaI'll try folks tomorrow to see what it is01:49
anteayaokay fair enough, thank you01:50
*** zul has joined #openstack-infra01:54
*** bhavik has quit IRC01:54
*** xarses has joined #openstack-infra01:57
anteayaI am able to query my user info: http://docs.openstack.org/infra/storyboard/webapi/v1.html#get--v1-users-%28user_id%2901:57
anteayaI figured out my user_id01:57
*** Daisy has joined #openstack-infra01:58
anteayaI'm not sure what bits it wants me to offer it in a edit command since anyone has access to that info01:58
*** ddieterly has quit IRC01:59
*** PollyZ has joined #openstack-infra01:59
*** sflanigan has joined #openstack-infra02:00
anteayathis did give me a 401 unauthorized: curl --insecure -i https://storyboard-dev.openstack.org/api/v1/users/13/preferences -X GET02:00
*** Daisy_ has quit IRC02:00
anteayaso I wonder why I got a 500 earlier instead of a 40102:01
anteayaanyway enough talking to myself, time to shut down for the night02:02
anteayathanks for your help02:02
Qiminginfra team, pls help invite openstackgerrit back to #senlin channel, thank you!!!02:02
*** PollyZ has quit IRC02:04
*** dimtruck is now known as zz_dimtruck02:06
*** erlon has quit IRC02:06
*** zz_dimtruck is now known as dimtruck02:06
*** amotoki has quit IRC02:07
*** piet_ has joined #openstack-infra02:07
*** amotoki has joined #openstack-infra02:07
*** Daisy has quit IRC02:11
*** Daisy has joined #openstack-infra02:11
*** yamahata has quit IRC02:12
clarkbQiming: http://docs.openstack.org/infra/system-config/irc.html you can propose the change but we also ask that the channel.be configured properly first02:13
Qimingthanks for the help clarkb, looking02:14
Qimingit was there, but we lost it for a few days now02:15
*** mtanino has joined #openstack-infra02:16
*** sarob has joined #openstack-infra02:16
*** r-mibu has quit IRC02:17
*** ramishra has quit IRC02:17
*** mtanin___ has quit IRC02:18
*** ramishra has joined #openstack-infra02:18
*** ramishra has quit IRC02:18
*** ramishra has joined #openstack-infra02:19
*** ramishra has quit IRC02:19
*** ramishra has joined #openstack-infra02:19
*** ramishra has quit IRC02:19
*** bhavik has joined #openstack-infra02:19
*** Daisy_ has joined #openstack-infra02:20
*** sarob has quit IRC02:21
*** ramishra has joined #openstack-infra02:21
*** ramishra has quit IRC02:21
*** Daisy has quit IRC02:23
Qimingclarkb, double checked system configuration, nothing has been changed, just that ... gerritbot is not in the #senlin channel, so we are missing all gerrit notifications recently02:24
*** zhurong has joined #openstack-infra02:27
*** ramishra has joined #openstack-infra02:27
*** ramishra has quit IRC02:27
*** ramishra has joined #openstack-infra02:28
*** ramishra has quit IRC02:29
*** julim has joined #openstack-infra02:29
*** tphummel has quit IRC02:31
*** mtanino has quit IRC02:37
*** r-mibu has joined #openstack-infra02:37
*** amitgandhinz has joined #openstack-infra02:38
openstackgerritPaul Belanger proposed openstack-infra/system-config: Remove vexxhost from nodepool.o.o clouds.yaml  https://review.openstack.org/32923402:38
*** gouthamr_ has quit IRC02:38
*** amotoki has quit IRC02:40
*** amitgandhinz has quit IRC02:42
*** zhurong has quit IRC02:44
*** ddieterly has joined #openstack-infra02:44
*** zhurong has joined #openstack-infra02:44
*** timothyb89 has quit IRC02:45
*** ddieterly has quit IRC02:46
*** timothyb89 has joined #openstack-infra02:47
*** woodster_ has quit IRC02:48
*** yuanying has quit IRC02:50
*** yamamoto_ has joined #openstack-infra02:50
*** yuanying has joined #openstack-infra02:52
openstackgerritEli Qiao proposed openstack-infra/project-config: Zun: rename higgins to zun (1)  https://review.openstack.org/32630602:52
openstackgerritEli Qiao proposed openstack-infra/project-config: Zun: rename higgins to zun (2)  https://review.openstack.org/32924702:52
*** amotoki has joined #openstack-infra02:54
*** amotoki has quit IRC02:57
*** thorst has quit IRC03:00
*** thorst has joined #openstack-infra03:00
*** yamamoto_ has quit IRC03:02
*** yamamoto_ has joined #openstack-infra03:05
*** links has joined #openstack-infra03:08
*** kushal has joined #openstack-infra03:09
*** thorst has quit IRC03:09
*** openstackgerrit has quit IRC03:11
*** openstackgerrit has joined #openstack-infra03:11
*** grassass has quit IRC03:12
pleia2Qiming: I restarted the bot to try to get it to return, but it seems like it's still not joining for some reason03:15
Qimingyes, noticed that, pleia203:15
pleia2and my dinner has just arrived, so I'll have to pick this up in the AM03:15
pleia2(unless someone else has a look before me)03:15
Qimingenjoy your dinner, pleia2, :)03:16
*** piet_ has quit IRC03:16
*** dimtruck is now known as zz_dimtruck03:18
*** bhavik has quit IRC03:20
*** zhurong has quit IRC03:22
*** csomerville has quit IRC03:24
*** PollyZ has joined #openstack-infra03:24
openstackgerritMerged openstack-infra/irc-meetings: Added meeting time for the JavaScript SDK & Tooling group.  https://review.openstack.org/32912003:26
*** binbincong has joined #openstack-infra03:26
*** afred312 has quit IRC03:26
openstackgerritMerged openstack-infra/irc-meetings: kuryr: Make wiki and ics match  https://review.openstack.org/32902303:27
*** yamahata has joined #openstack-infra03:29
openstackgerritMerged openstack-infra/irc-meetings: Change meeting time to 1500 UTC  https://review.openstack.org/32905703:29
*** yamamoto_ has quit IRC03:30
*** yuanying has quit IRC03:33
*** yuanying has joined #openstack-infra03:34
*** garyk has joined #openstack-infra03:34
*** yamamoto_ has joined #openstack-infra03:36
*** amitgandhinz has joined #openstack-infra03:39
*** asselin_ has quit IRC03:40
*** darvon has quit IRC03:40
*** darvon has joined #openstack-infra03:40
*** amitgandhinz has quit IRC03:43
*** sputnik13 has quit IRC03:44
*** ociuhandu has quit IRC03:45
*** PollyZ has quit IRC03:47
*** ociuhandu has joined #openstack-infra03:49
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Allow JJB to spoof plugin_info via yaml  https://review.openstack.org/32672203:50
*** cody-somerville has joined #openstack-infra03:53
*** ayoung has quit IRC03:56
*** amotoki has joined #openstack-infra03:58
*** hongbin has quit IRC04:00
*** PollyZ has joined #openstack-infra04:01
*** yamada-h has joined #openstack-infra04:03
*** yuanying has quit IRC04:05
*** ociuhandu has quit IRC04:05
*** alekhya has joined #openstack-infra04:06
tonybCan someone with access to the logs work out why this https://review.openstack.org/#/c/329097/ isn't progressing to the gate pipeline?04:07
*** thorst has joined #openstack-infra04:07
*** yuanying has joined #openstack-infra04:07
openstackgerritEli Qiao proposed openstack-infra/project-config: Zun: rename higgins to zun (1)  https://review.openstack.org/32630604:11
openstackgerritEli Qiao proposed openstack-infra/project-config: Zun: rename higgins to zun (2)  https://review.openstack.org/32924704:11
*** thorst has quit IRC04:14
*** xarses_ has joined #openstack-infra04:14
*** markvoelker has quit IRC04:16
*** andreww has joined #openstack-infra04:17
*** xarses has quit IRC04:17
*** andreww has quit IRC04:18
*** PollyZ has quit IRC04:18
*** andreww has joined #openstack-infra04:18
*** xarses_ has quit IRC04:20
*** cody-somerville_ has joined #openstack-infra04:23
openstackgerritSachi King proposed openstack-dev/pbr: Restore warnerrors behavior  https://review.openstack.org/22995104:24
*** alekhya has quit IRC04:24
*** grassass has joined #openstack-infra04:24
*** Daisy_ has quit IRC04:26
*** sree_ has joined #openstack-infra04:27
*** sree_ is now known as Guest9463104:27
*** afred312 has joined #openstack-infra04:27
*** psachin has joined #openstack-infra04:32
*** hdaniel has joined #openstack-infra04:32
*** afred312 has quit IRC04:33
*** cody-somerville has quit IRC04:37
*** zhurong has joined #openstack-infra04:38
*** pcaruana has quit IRC04:39
*** amitgandhinz has joined #openstack-infra04:39
*** tkelsey has joined #openstack-infra04:42
*** amitgandhinz has quit IRC04:44
*** tkelsey has quit IRC04:47
*** tlian has joined #openstack-infra04:48
*** nadya has joined #openstack-infra04:51
*** abregman has joined #openstack-infra04:52
*** yamada-h has quit IRC04:53
*** yamada-h has joined #openstack-infra04:54
*** zhenguo_ has joined #openstack-infra04:54
*** mrmartin has joined #openstack-infra04:58
*** yamada-h has quit IRC04:59
*** yamada-h has joined #openstack-infra05:05
*** pgadiya has joined #openstack-infra05:06
clarkbtonyb dont need logs it is because the verified +1 vote isnt changing so the gerrit event doesnt include that data due to no delta05:07
clarkbyou can just reapprove it to get it in the gate05:07
*** Daisy has joined #openstack-infra05:07
*** tphummel has joined #openstack-infra05:09
*** Daisy_ has joined #openstack-infra05:11
*** Daisy has quit IRC05:11
*** oanson has quit IRC05:11
*** jaosorior has joined #openstack-infra05:11
*** cody-somerville has joined #openstack-infra05:12
*** cody-somerville has joined #openstack-infra05:12
*** thorst has joined #openstack-infra05:12
*** cody-somerville has quit IRC05:16
*** cody-somerville_ has quit IRC05:16
*** markvoelker has joined #openstack-infra05:17
*** salv-orl_ has quit IRC05:17
*** salv-orlando has joined #openstack-infra05:18
*** thorst has quit IRC05:19
tonybclarkb: Thanks.  Why didn't it automatically progress when the first Verified came in?  (which looks to me after it had +W)05:20
clarkbtonyb: I think jeblair restarted zuul at some point05:21
tonybclarkb: Ah okay05:21
*** markvoelker has quit IRC05:22
tonybclarkb: Looks like it was betwn the +W and the +V on that review.05:23
* tonyb will add that to the list of things to check05:23
*** mrmartin has quit IRC05:24
*** yamamoto_ has quit IRC05:28
*** salv-orl_ has joined #openstack-infra05:29
*** nadya has quit IRC05:29
*** salv-orlando has quit IRC05:31
*** salv-orl_ has quit IRC05:34
*** salv-orlando has joined #openstack-infra05:34
*** amitgandhinz has joined #openstack-infra05:40
*** nadya has joined #openstack-infra05:41
*** sarob has joined #openstack-infra05:41
*** amitgandhinz has quit IRC05:44
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE TESTING  https://review.openstack.org/31643605:45
*** sarob has quit IRC05:46
*** Guest94631 has quit IRC05:46
*** sree has joined #openstack-infra05:47
*** savihou has joined #openstack-infra05:49
*** sree has quit IRC05:49
*** sree has joined #openstack-infra05:50
openstackgerritYAMADA Hideki proposed openstack-infra/project-config: Add Sheepdog test job to glance_store  https://review.openstack.org/32814805:52
*** nadya has quit IRC05:55
*** javeriak has joined #openstack-infra05:59
*** Sukhdev has joined #openstack-infra06:00
*** roxanaghe has joined #openstack-infra06:00
*** roxanaghe has quit IRC06:00
*** tlian has quit IRC06:00
*** savihou has quit IRC06:06
*** Daisy_ has quit IRC06:11
*** rcernin has joined #openstack-infra06:11
*** Daisy has joined #openstack-infra06:12
*** pfallenop has quit IRC06:12
*** salv-orlando has quit IRC06:13
*** salv-orlando has joined #openstack-infra06:14
*** tphummel has quit IRC06:14
*** pfallenop has joined #openstack-infra06:15
*** Daisy has quit IRC06:16
*** pcaruana has joined #openstack-infra06:16
*** thorst has joined #openstack-infra06:16
*** jbernard has quit IRC06:17
*** yamamoto_ has joined #openstack-infra06:17
*** jbernard has joined #openstack-infra06:19
openstackgerritMerged openstack-infra/zuul: Python 3 Fix: Divide operator returns float  https://review.openstack.org/32730306:19
openstackgerritMerged openstack-infra/project-config: Add networking-ovn in the 'PROJECTS' for dsvm-functional job  https://review.openstack.org/32924106:20
openstackgerritlvdongbing proposed openstack-infra/irc-meetings: Add the Bilean team meeting  https://review.openstack.org/32627106:21
yolandagood morning06:23
*** yamahata has quit IRC06:23
openstackgerritMerged openstack-infra/zuul: Fix timeout debug print in tests  https://review.openstack.org/32776806:24
*** thorst has quit IRC06:24
*** yamahata has joined #openstack-infra06:26
*** tphummel has joined #openstack-infra06:30
*** Daisy has joined #openstack-infra06:33
*** Sukhdev has quit IRC06:39
*** amitgandhinz has joined #openstack-infra06:41
*** ihrachys has joined #openstack-infra06:41
*** mrmartin has joined #openstack-infra06:42
*** andreas_s has joined #openstack-infra06:42
*** tkelsey has joined #openstack-infra06:44
openstackgerritMerged openstack-infra/project-config: Create new project karma-subunit-reporter  https://review.openstack.org/32794506:45
*** amitgandhinz has quit IRC06:45
*** nmagnezi has joined #openstack-infra06:46
*** openstackgerrit has quit IRC06:48
*** openstackgerrit has joined #openstack-infra06:48
openstackgerritMerged openstack-infra/project-config: Add non-voting coverage check job for neutron-lib  https://review.openstack.org/32844506:48
*** tkelsey has quit IRC06:49
AJaegermorning, yolanda !06:49
*** aviau has quit IRC06:49
*** aviau has joined #openstack-infra06:50
*** ociuhandu has joined #openstack-infra06:50
openstackgerritMerged openstack-infra/project-config: Add trove period jobs with oslo master  https://review.openstack.org/32908606:55
openstackgerritMerged openstack-infra/project-config: Add python version to wheel builders  https://review.openstack.org/32903606:55
*** tesseract has joined #openstack-infra06:56
*** samP has joined #openstack-infra06:56
*** ccamacho has quit IRC06:56
openstackgerritMerged openstack-infra/project-config: Add gates for deploying kolla-kubernetes  https://review.openstack.org/32881406:57
*** amoralej|off is now known as amoralej06:58
openstackgerritMerged openstack-infra/project-config: Add translation jobs to watcher  https://review.openstack.org/32901206:58
openstackgerritMerged openstack-infra/project-config: Add release notes jobs for watcher  https://review.openstack.org/32900706:59
*** mrmartin has quit IRC06:59
*** rossella_s has joined #openstack-infra07:00
openstackgerritMerged openstack-infra/project-config: [Manila] Add new experimental job  https://review.openstack.org/32836407:00
openstackgerritMerged openstack-infra/project-config: Disable convergence for jobs with legacy engine  https://review.openstack.org/32756007:01
*** grassass has quit IRC07:02
*** ccamacho has joined #openstack-infra07:06
*** zeih has joined #openstack-infra07:11
*** mrmartin has joined #openstack-infra07:12
*** xarses_ has joined #openstack-infra07:14
*** slaweq has joined #openstack-infra07:16
*** andreww has quit IRC07:17
*** markvoelker has joined #openstack-infra07:18
AJaegerflaper87: could you review 328148, please?07:19
openstackgerritMerged openstack-infra/tripleo-ci: Add new reference to puppet-certmonger to mirror list  https://review.openstack.org/32902507:19
*** mikelk has joined #openstack-infra07:22
*** thorst has joined #openstack-infra07:22
*** markvoelker has quit IRC07:22
*** vincentll has joined #openstack-infra07:23
openstackgerritMerged openstack-infra/project-config: Add tempest-dsvm-ironic-pxe_ipa-full-nv to Ironic  https://review.openstack.org/32824507:26
*** jlanoux has joined #openstack-infra07:26
*** arxcruz has joined #openstack-infra07:28
*** thorst has quit IRC07:29
*** abregman_ has joined #openstack-infra07:29
*** dizquierdo has joined #openstack-infra07:30
*** grassass has joined #openstack-infra07:30
*** dmk0202 has joined #openstack-infra07:32
*** ifarkas has joined #openstack-infra07:32
*** abregman has quit IRC07:33
*** sree has quit IRC07:33
*** sree has joined #openstack-infra07:34
*** jaosorior is now known as jaosorior_brb07:35
*** jlanoux has quit IRC07:37
*** tkelsey has joined #openstack-infra07:37
*** jlanoux has joined #openstack-infra07:38
*** sree has quit IRC07:38
*** jlanoux has quit IRC07:39
*** yamahata has quit IRC07:41
*** amitgandhinz has joined #openstack-infra07:41
*** nighttime has joined #openstack-infra07:41
*** sarob has joined #openstack-infra07:42
*** salv-orlando has quit IRC07:44
*** salv-orlando has joined #openstack-infra07:44
*** amitgandhinz has quit IRC07:46
*** sarob has quit IRC07:47
*** zeih has quit IRC07:47
*** Na3iL has joined #openstack-infra07:47
*** sree has joined #openstack-infra07:47
*** grazzazz has joined #openstack-infra07:49
*** grassass has quit IRC07:49
*** nadya has joined #openstack-infra07:50
*** nadya is now known as _nadya_07:50
*** grazzazz has quit IRC07:50
openstackgerritYuval Brik proposed openstack-infra/project-config: Use Smaug configuration from Smaug's repository  https://review.openstack.org/32930507:50
*** grassass has joined #openstack-infra07:51
*** Na3iL has quit IRC07:54
*** samP has quit IRC07:56
*** sorantis has joined #openstack-infra07:56
*** abregman_ has quit IRC07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** Na3iL has joined #openstack-infra08:00
*** kushal has quit IRC08:01
*** jlanoux has joined #openstack-infra08:02
*** dtantsur|afk is now known as dtantsur08:04
*** eezhova has joined #openstack-infra08:04
*** shardy has joined #openstack-infra08:04
*** zeih has joined #openstack-infra08:05
*** e0ne has joined #openstack-infra08:05
*** zeih has quit IRC08:06
openstackgerritDi XiaoLi proposed openstack-infra/project-config: Add a heat job to use devstack plugin  https://review.openstack.org/31781108:06
*** faizy has joined #openstack-infra08:08
*** zeih has joined #openstack-infra08:09
*** abregman_ has joined #openstack-infra08:11
openstackgerritElena Ezhova proposed openstack-infra/project-config: Pass extra parameter to hooks in neutron-lbaas tempest jobs  https://review.openstack.org/32281108:17
openstackgerritDi XiaoLi proposed openstack-infra/project-config: [WIP]Switch all heat jobs to use devstack plugin  https://review.openstack.org/31781708:18
openstackgerritElena Ezhova proposed openstack-infra/project-config: Add a functional job for neutron-lbaas  https://review.openstack.org/32281208:18
*** Daisy has quit IRC08:19
*** Daisy has joined #openstack-infra08:19
*** mhickey has joined #openstack-infra08:20
*** faizy_ has joined #openstack-infra08:21
*** faizy has quit IRC08:22
*** sorantis_ has joined #openstack-infra08:22
*** sorantis has quit IRC08:25
*** sorantis_ is now known as sorantis08:25
*** pgadiya has quit IRC08:25
*** pgadiya has joined #openstack-infra08:26
*** thorst has joined #openstack-infra08:26
*** javeriak has quit IRC08:27
*** afred312 has joined #openstack-infra08:29
*** chem`` has joined #openstack-infra08:31
*** chem` has quit IRC08:32
*** javeriak has joined #openstack-infra08:33
*** afred312 has quit IRC08:34
*** thorst has quit IRC08:34
openstackgerritDi XiaoLi proposed openstack-infra/project-config: Switch all heat jobs to use devstack plugin  https://review.openstack.org/31781708:35
*** sshnaidm has quit IRC08:39
*** jaosorior_brb has quit IRC08:39
*** tphummel has quit IRC08:39
*** jaosorior_brb has joined #openstack-infra08:40
*** derekh has joined #openstack-infra08:40
*** dingyichen has quit IRC08:41
*** amitgandhinz has joined #openstack-infra08:42
*** Qiming is now known as Qiming_afk08:43
*** kzaitsev_mb has joined #openstack-infra08:43
*** Julien-zte has quit IRC08:44
*** amitgandhinz has quit IRC08:47
*** kzaitsev_mb has quit IRC08:50
*** matrohon has joined #openstack-infra08:50
*** jaosorior_brb is now known as jaosorior08:53
*** dtardivel has joined #openstack-infra08:54
*** markusry has joined #openstack-infra08:54
*** javeriak has quit IRC08:56
*** acoles_ is now known as acoles08:58
*** hashar has joined #openstack-infra09:02
*** Daisy_ has joined #openstack-infra09:02
*** vgridnev has quit IRC09:03
*** Qiming_afk is now known as Qiming09:06
*** Daisy has quit IRC09:06
*** binbincong has quit IRC09:07
*** Daisy_ has quit IRC09:10
*** Daisy has joined #openstack-infra09:11
*** sambetts|afk is now known as sambetts09:14
*** jordanP has joined #openstack-infra09:14
*** zeih has quit IRC09:15
*** ams__ has joined #openstack-infra09:15
*** Daisy has quit IRC09:15
*** amotoki has quit IRC09:15
openstackgerritChuck Short proposed openstack-infra/project-config: Add check-requirements check job for nova-lxd  https://review.openstack.org/32933109:18
*** Na3iL has quit IRC09:19
openstackgerritMerged openstack-infra/irc-meetings: Add the Bilean team meeting  https://review.openstack.org/32627109:22
*** electrofelix has joined #openstack-infra09:26
*** zeih has joined #openstack-infra09:27
*** binbincong has joined #openstack-infra09:30
openstackgerritSam Betts proposed openstack-infra/elastic-recheck: Add query for nova resource tracker/scheduler race  https://review.openstack.org/31636409:30
*** javeriak has joined #openstack-infra09:32
*** thorst has joined #openstack-infra09:32
*** Na3iL has joined #openstack-infra09:34
*** snarwade has quit IRC09:35
*** yamada-h has quit IRC09:35
*** pgadiya has quit IRC09:36
*** thorst has quit IRC09:39
*** _degorenko|afk is now known as degorenko09:39
*** eezhova has quit IRC09:40
wznoinskhi all09:41
*** sshnaidm has joined #openstack-infra09:41
*** amitgandhinz has joined #openstack-infra09:43
*** sarob has joined #openstack-infra09:43
*** pgadiya has joined #openstack-infra09:43
openstackgerritJulien Danjou proposed openstack-infra/git-review: Set a default EXIT_CODE for GitReviewException  https://review.openstack.org/32395909:46
openstackgerritAles Komarek proposed openstack-infra/irc-meetings: Change opesntack-salt meeting chair  https://review.openstack.org/32934409:47
*** amitgandhinz has quit IRC09:47
*** sarob has quit IRC09:48
openstackgerritAles Komarek proposed openstack-infra/irc-meetings: Change openstack-salt meeting chair  https://review.openstack.org/32934409:48
*** zeih has quit IRC09:48
*** eezhova has joined #openstack-infra09:50
*** ociuhandu has quit IRC09:51
*** snarwade has joined #openstack-infra09:51
fricklerI'm seeing random failures where it looks like the task has been run by zuul instead of jenkins, like e.g. http://logs.openstack.org/61/327761/2/check/gate-nova-docs/6899468/ although the exit code is 0 (= success I assume), this is logged as failure09:53
*** javeriak has quit IRC09:56
openstackgerritLukas Bednar proposed openstack-infra/git-review: Fix AttributeError when can not connect to Gerrit  https://review.openstack.org/31818109:56
AJaegerfrickler: that looks to me like it should have been a success. Let's tell jeblair and pabelanger about your log file and have them look at it.09:58
AJaegerfrickler: Yes, we're replacing jenkins by zuul-launcher and hit a few problems - and jeblair and pabelanger look into these. Thanks for telling us!09:58
*** javeriak has joined #openstack-infra09:59
*** claudiub has joined #openstack-infra10:02
*** kzaitsev_mb has joined #openstack-infra10:02
*** lvdongbing has quit IRC10:03
*** amotoki has joined #openstack-infra10:04
*** tosky has joined #openstack-infra10:04
*** sree has quit IRC10:05
*** links has quit IRC10:07
*** dmk0202 has quit IRC10:08
openstackgerritMadhuri Kumari proposed openstack-infra/project-config: Add new project Zunclient to openstack repo  https://review.openstack.org/31769910:09
*** kushal has joined #openstack-infra10:11
*** oanson has joined #openstack-infra10:13
*** ociuhandu has joined #openstack-infra10:13
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repo for fuel-plugin-datera-cinder  https://review.openstack.org/31565110:13
*** javeriak has quit IRC10:14
*** snarwade has quit IRC10:20
*** links has joined #openstack-infra10:20
*** javeriak has joined #openstack-infra10:22
*** zeih has joined #openstack-infra10:23
*** yanyanhu has quit IRC10:24
*** zhurong has quit IRC10:24
*** jlanoux_ has joined #openstack-infra10:25
*** pgadiya has quit IRC10:26
*** pgadiya has joined #openstack-infra10:26
*** zeih has quit IRC10:27
*** jlanoux has quit IRC10:28
*** zeih has joined #openstack-infra10:28
*** afred312 has joined #openstack-infra10:31
*** sdague has joined #openstack-infra10:31
*** dizquierdo is now known as dizquierdo_afk10:32
*** amotoki_ has joined #openstack-infra10:34
*** afred312 has quit IRC10:35
*** thorst has joined #openstack-infra10:35
*** yamada-h has joined #openstack-infra10:36
*** snarwade has joined #openstack-infra10:36
*** amotoki_ has quit IRC10:37
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Fix tempest configuration  https://review.openstack.org/32605510:38
*** amotoki_ has joined #openstack-infra10:38
*** eezhova has quit IRC10:38
*** salv-orlando has quit IRC10:38
*** salv-orlando has joined #openstack-infra10:39
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE TESTING  https://review.openstack.org/31643610:39
*** Srinivas__ has joined #openstack-infra10:39
Srinivas__Hai. Fc passthrough will work only on  ubuntu 14.04.1 only or it can be used in any OS. Please help me. i am using FC passthrough on ubuntu 14.04.04 but it is failing while running this command check this link http://paste.openstack.org/show/515937/10:39
*** yamada-h has quit IRC10:41
*** amotoki has quit IRC10:42
*** amitgandhinz has joined #openstack-infra10:44
*** thorst has quit IRC10:44
*** dtantsur is now known as dtantsur|brb10:45
*** jlanoux has joined #openstack-infra10:46
*** jlanoux_ has quit IRC10:48
*** amitgandhinz has quit IRC10:48
*** amotoki_ has quit IRC10:52
*** amotoki has joined #openstack-infra10:52
*** esikachev has joined #openstack-infra10:54
Srinivas__<esikachev>Hai. Fc passthrough will work only on  ubuntu 14.04.1 only or it can be used in any OS. Please help me. i am using FC passthrough on ubuntu 14.04.04 but it is failing while running this command check this link http://paste.openstack.org/show/515937/10:54
*** salv-orlando has quit IRC10:58
*** salv-orlando has joined #openstack-infra10:58
*** zeih_ has joined #openstack-infra11:02
*** zeih has quit IRC11:02
openstackgerritHisashi Osanai proposed openstack-infra/project-config: Add jobs for Monasca-Analytics  https://review.openstack.org/32937611:07
*** permalac has quit IRC11:09
AJaegerSrinivas__: did you see http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-13.log.html#t2016-06-13T16:17:10 ?11:09
*** oanson has quit IRC11:10
AJaegerSrinivas__: and http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-13.log.html#t2016-06-13T16:41:22 ?11:10
*** Srinivas__ has left #openstack-infra11:14
*** ccamacho is now known as ccamacho|lunch11:19
*** eezhova has joined #openstack-infra11:20
*** ociuhandu has quit IRC11:23
*** ociuhandu has joined #openstack-infra11:23
*** ociuhandu has quit IRC11:26
*** ociuhandu has joined #openstack-infra11:26
*** ldnunes has joined #openstack-infra11:28
*** rhallisey has joined #openstack-infra11:28
*** salv-orl_ has joined #openstack-infra11:28
*** sree has joined #openstack-infra11:29
*** gordc has joined #openstack-infra11:30
*** weshay has joined #openstack-infra11:30
*** salv-orlando has quit IRC11:32
*** zeih_ has quit IRC11:32
*** sree has quit IRC11:34
*** rodrigods has quit IRC11:35
*** rodrigods has joined #openstack-infra11:35
*** kushal has quit IRC11:36
*** kushal has joined #openstack-infra11:37
*** dizquierdo_afk is now known as dizquierdo11:37
*** thorst has joined #openstack-infra11:41
*** thorst_ has joined #openstack-infra11:43
*** sarob has joined #openstack-infra11:44
*** piet has joined #openstack-infra11:44
*** exploreshaifali has joined #openstack-infra11:44
*** amitgandhinz has joined #openstack-infra11:44
*** jed56 has quit IRC11:45
*** thorst has quit IRC11:47
*** amotoki has quit IRC11:48
*** sarob has quit IRC11:48
*** thorst_ is now known as thorst11:49
*** ddieterly has joined #openstack-infra11:49
*** amitgandhinz has quit IRC11:49
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Rename expand-groups to ansible-expand-groups  https://review.openstack.org/32646511:50
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Tidy up ansible directories  https://review.openstack.org/32646411:50
*** baoli has joined #openstack-infra11:51
*** baoli_ has joined #openstack-infra11:52
*** raildo-afk is now known as raildo11:54
*** baoli has quit IRC11:56
*** thiagop has joined #openstack-infra11:57
*** aysyd has joined #openstack-infra11:57
*** tpsilva has joined #openstack-infra11:57
*** dtantsur|brb is now known as dtantsur11:58
*** piet has quit IRC11:58
*** exploreshaifali has quit IRC11:58
mordredjhesketh: do we care about deleting the old expand-groups script?12:00
mordredjhesketh: (I don't care a ton)12:00
jheskethmordred: I was just rerolling some old patches...12:01
jheskethmordred: is the plan to remove it?12:01
*** jed56 has joined #openstack-infra12:02
*** sree has joined #openstack-infra12:04
*** zeih has joined #openstack-infra12:06
*** esikachev has quit IRC12:06
*** yamada-h has joined #openstack-infra12:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Start fleshing out ansible-launcher tests  https://review.openstack.org/31748212:07
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Clean up results returned from ansible-launcher  https://review.openstack.org/31748112:07
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add jjb and zmq to requirements  https://review.openstack.org/31675212:07
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Prepare files for testing zuulv25  https://review.openstack.org/31675312:07
*** faizy_ has quit IRC12:07
*** markusry has quit IRC12:09
mordredjhesketh: no - I just mean, the patch renames it to ansible-expand-groups - but nothing cleans up after it12:09
jheskethoh right, I see what you mean... yeah I'll tidy that up12:10
*** yamada-h has quit IRC12:10
*** esikachev has joined #openstack-infra12:11
*** yamamoto_ has quit IRC12:11
*** daemontool has joined #openstack-infra12:11
*** pradk has joined #openstack-infra12:12
*** markvoelker has joined #openstack-infra12:13
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Rename expand-groups to ansible-expand-groups  https://review.openstack.org/32646512:14
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Remove cleanup directive  https://review.openstack.org/32941012:14
jheskethmordred: ^12:14
*** kgiusti has joined #openstack-infra12:15
*** esikachev has quit IRC12:15
*** matrohon has quit IRC12:15
*** ddieterly has quit IRC12:16
*** psilvad has joined #openstack-infra12:16
*** esikachev has joined #openstack-infra12:16
mordredjhesketh: +212:19
jheskethta :-)12:20
*** psilvad has quit IRC12:20
*** zeih has quit IRC12:21
*** esikachev has quit IRC12:21
*** dizquierdo has quit IRC12:21
*** eezhova has quit IRC12:23
*** ccamacho|lunch is now known as ccamacho12:23
*** matrohon has joined #openstack-infra12:25
*** Srinu has joined #openstack-infra12:26
SrinuHai. i am using this https://github.com/openstack/third-party-ci-tools/blob/master/provisioning_scripts/fibre_channel/invoke_fc_passthrough.sh12:26
*** psilvad has joined #openstack-infra12:26
*** kushal has quit IRC12:26
Srinuwhen running the virsh attach-device command in the script it is failing with the error http://paste.openstack.org/show/515937/12:27
Srinuwe are using Ubuntu 14.04.4 with QLOgic FC12:27
Srinuwe are using devstack master12:28
*** eezhova has joined #openstack-infra12:28
SrinuDid anyone face the issue like this12:28
*** kaisers1 has quit IRC12:29
Srinuwe are setting up CI for our FC driver and we need this script to be run to enable passthrough12:29
pabelangerfrickler: AJaeger: I suspect that is a know issue we fixed yesterday.  A recheck would be in order12:31
*** tlian has joined #openstack-infra12:31
*** psilvad has quit IRC12:31
*** afred312 has joined #openstack-infra12:32
mordredpabelanger: how's the 2.5 going?12:33
*** salv-orl_ has quit IRC12:33
*** salv-orlando has joined #openstack-infra12:34
*** rlandy has joined #openstack-infra12:34
pabelangermordred: Excellent I believe. jeblair has 5 zuul-launchers online (currently using 4 of them).  Finding some issues, but nothing major at this point12:34
mordredpabelanger: \o/12:34
*** psilvad has joined #openstack-infra12:35
pabelangermordred: jeblair: for example, http://paste.openstack.org/show/515966/ we need to see about disabling error handling in rsync.  We are currently failing to synchronize logs if we cannot ready them,12:35
*** esikachev has joined #openstack-infra12:35
pabelangerPOST_FAILURE is a new error we added, to replace UNSTABLE :)12:35
*** zeih has joined #openstack-infra12:36
*** afred312 has quit IRC12:37
*** ddieterly has joined #openstack-infra12:37
*** trown|outtypewww is now known as trown12:37
mordredooh. I like that as a better error12:38
*** nikeshm has joined #openstack-infra12:38
nikeshmwe are setting up CI for our kaminario FC driver12:39
*** psilvad has quit IRC12:39
nikeshmwe are following the steps required for FC passthrough12:40
AJaegernikeshm, Srinu did you see http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-13.log.html#t2016-06-13T16:41:22 ?12:41
*** sarob has joined #openstack-infra12:42
AJaegerand also http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-13.log.html#t2016-06-13T16:17:10 ?12:42
AJaegerpabelanger: thanks. I see a recheck was done earlier today and worked.12:45
*** amitgandhinz has joined #openstack-infra12:45
*** javeriak has quit IRC12:46
*** ddieterly is now known as ddieterly[away]12:46
*** exploreshaifali has joined #openstack-infra12:46
*** sarob has quit IRC12:47
*** psilvad has joined #openstack-infra12:47
*** pgadiya has quit IRC12:48
*** amoralej is now known as amoralej|lunch12:48
nikeshmAJaeger: thanks, since  infra channel helps us in configuring and understanding openstack CI for third party vendors, so people might be have faced such issue so they could help better12:48
yolandahi pabelanger , mordred, care reviewing https://review.openstack.org/326605 and https://review.openstack.org/327707 ?12:48
*** piet has joined #openstack-infra12:49
pabelangerAJaeger: We also are bootstrapping our logs correctly now: http://logs.openstack.org/91/328391/7/check/gate-manila-tempest-dsvm-postgres-generic-singlebackend/4985dfa/console.html  See first 2 lines12:49
*** mrmartin has quit IRC12:49
pabelangerjeblair: when we abort a job, should we also abort the post_playbook tasks? http://paste.openstack.org/show/515971/ I noticed this is passing this morning12:49
AJaegernikeshm: but that question is wrong here.12:49
AJaegerpabelanger: great!12:49
pabelangeryolanda: looking12:49
*** amitgandhinz has quit IRC12:50
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Add jobs based on ipmitool drivers to ironic-python-agent  https://review.openstack.org/32942812:51
*** piet has quit IRC12:51
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Make ironic-{pxe,agent}_ipmitool jobs voting  https://review.openstack.org/31259612:52
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Make ironic-{pxe,agent}_ipmitool jobs voting  https://review.openstack.org/31259612:52
rcarrillocruzpabelanger: sorry, was off yesterday and today i'll be around on and off12:53
*** piet has joined #openstack-infra12:53
rcarrillocruzi will check it out with the lab, cos not able to reach or whatever is just wrong12:53
*** yamamoto has joined #openstack-infra12:53
*** matt-borland has joined #openstack-infra12:53
pabelangerrcarrillocruz: thanks, I haven't tested this morning12:53
*** zeih has quit IRC12:54
*** rossella_s has quit IRC12:54
rcarrillocruzi also need to continue consolidating new ips and metadata for other servers, some are unreachable or had wrong creds, need to check it out with the ticket owner12:54
rcarrillocruzpabelanger: in other news, are we now on ansible 2.1 on puppetmaster or still pinned  on 2.0.x ?12:55
rcarrillocruzi'd like to move forward the cloud launcher replacement for launch-node changes12:55
yolandathx for reviews12:56
*** markusry has joined #openstack-infra12:57
openstackgerritMerged openstack-infra/puppet-nodepool: Fix duplicated package for build-essential  https://review.openstack.org/32660512:57
*** Hal has joined #openstack-infra12:58
*** yujunz has joined #openstack-infra12:58
*** Hal is now known as Guest1608712:58
pabelangerrcarrillocruz: still 2.0.x IIRC12:59
*** zeih has joined #openstack-infra12:59
pabelangerrcarrillocruz: I think the issue is related to the change of the default type for AnsibleModules12:59
rcarrillocruzcos i've tested ansible latest from pypi (2.1.0) on one of my vms for other ansible-puppet stuff and haven't hit issues, so unsure if it's just i've been lucky or the fixes have already merged12:59
*** salv-orlando has quit IRC13:00
*** salv-orlando has joined #openstack-infra13:00
*** zeih has quit IRC13:01
*** jcoufal has joined #openstack-infra13:02
*** xyang1 has joined #openstack-infra13:04
*** Goneri has joined #openstack-infra13:06
*** yamamoto has quit IRC13:07
*** yamamoto has joined #openstack-infra13:07
*** rossella_s has joined #openstack-infra13:08
openstackgerritPaul Belanger proposed openstack-infra/ansible-role-puppet: Make hostvars[inventory_hostname].group_names is a list  https://review.openstack.org/32943113:09
pabelangermordred: Shrews: rcarrillocruz: ^ I think that will help with ansible
*** mdrabe has joined #openstack-infra13:10
openstackgerritPaul Belanger proposed openstack-infra/ansible-role-puppet: Make puppet_get_hiera_file_list.groups a list  https://review.openstack.org/32943113:10
rcarrillocruzah nice, typing is good... Shrews did a great job cleaning up a lot of params on the openstack modules that lacked typing13:10
*** yamamoto has quit IRC13:12
pabelangerYa, I cannot remember our exact error, but know it was related to ansible-role-puppet13:13
*** afred312 has joined #openstack-infra13:14
*** zhurong has joined #openstack-infra13:15
*** mrmartin has joined #openstack-infra13:16
*** asselin_ has joined #openstack-infra13:16
*** zhurong has quit IRC13:18
Srinuasselin_: hi.i am using fc pass through in ubuntu 14.04.4 for our FC driver in in CI using this link https://github.com/openstack/third-party-ci-tools/blob/master/provisioning_scripts/fibre_channel/invoke_fc_passthrough.sh but it is failing at the commnad virsh attach-device giving error as  http://paste.openstack.org/show/515937/13:18
*** zhurong has joined #openstack-infra13:19
*** pilgrimstack has joined #openstack-infra13:20
Shrewspabelanger: yeah, you really need to be explicit with types13:20
Shrewspabelanger: you should make sure that the code that uses those params doesn't treat them as anything other than a list13:21
*** gouthamr has joined #openstack-infra13:21
*** piet has quit IRC13:22
pabelangerShrews: ya, we are pulling in the variable directly from ansible, group_names13:23
*** _ari_ has joined #openstack-infra13:23
*** pilgrimstack has quit IRC13:24
*** amotoki has joined #openstack-infra13:24
openstackgerritOmar Shykhkerimov proposed openstack-infra/project-config: Add murano periodic jobs with oslo master  https://review.openstack.org/32944013:25
Srinuamotoki: i am using fc pass through in ubuntu 14.04.4 for our FC driver in in CI using this link https://github.com/openstack/third-party-ci-tools/blob/master/provisioning_scripts/fibre_channel/invoke_fc_passthrough.sh but it is failing at the commnad virsh attach-device giving error as  http://paste.openstack.org/show/515937/13:27
asselin_Srinu, hi. Best to ask this in #openstack-cinder13:27
Srinuasselin_: Ok. Thank you13:28
amotokiSrinu: i think so. I wonder you find me..13:28
*** ddieterly has joined #openstack-infra13:28
asselin_patrickeast and hemna might be able to help13:29
*** ericksonsantos has joined #openstack-infra13:29
Srinuasselin_: Did you faced this issue anytime13:31
asselin_Srinu, I probably did, but don't remember. check that the drivers are loaded?13:32
fungiSrinu: i plugged the error you provided into the most advanced diagnostic system available... a web search engine... and it indicated that you might need to make allowances in apparmor for that13:33
*** rfolco has joined #openstack-infra13:33
*** woodster_ has joined #openstack-infra13:33
Srinuasselin_: which drivers i have to check13:33
*** mriedem_away is now known as mriedem13:33
*** annegentle has joined #openstack-infra13:34
*** erlon has joined #openstack-infra13:35
Srinuasselin_: i installed drivers correctly13:35
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Heat functional tests in Identity v3-only  https://review.openstack.org/31464313:35
fungiSrinu: check dmesg output for any apparmor="DENIED" lines13:36
asselin_check dmesg and syslog for more errors. I don't know where the virsh logs are. double check the fcoe and make sure it's correct for your fc environment13:36
*** amoralej|lunch is now known as amoralej13:37
*** piet has joined #openstack-infra13:39
fungiSrinu: nikeshm: also, as i and others have indicated, troubleshooting fc passthrough is pretty off-topic in this channel. you're better off seeking help from the developers of the software which is providing that error (virsh, qemu, nova, et cetera)13:40
*** rbrndt has joined #openstack-infra13:40
*** abregman_ has quit IRC13:40
*** edmondsw has joined #openstack-infra13:41
rookpabelanger another noob question re-infra... so, I have jenkins +1 but I am still unable to push ? https://review.openstack.org/#/c/328991/1413:41
*** PollyZ has joined #openstack-infra13:41
fungirook: you added your workflow +1 before there were any code-review +2 votes13:42
Srinuasselin_: the error in logs is http://paste.openstack.org/show/515992/13:42
fungirook: set workflow back to 0 and then to +1 again and it should trigger enqueuing into the gate13:42
*** mtanino has joined #openstack-infra13:43
*** faizy_ has joined #openstack-infra13:44
rookfungi ok - just did that.13:44
*** xarses_ has quit IRC13:45
*** eharney has joined #openstack-infra13:45
asselin_Srinu, i don't see you in #openstack-cinder. I suggest you ask there because all the users of that script are cinder folks.13:45
*** xarses_ has joined #openstack-infra13:45
*** amitgandhinz has joined #openstack-infra13:46
fungirook: note the automated comment which just appeared on that change. now search for 328991 on http://status.openstack.org/zuul/ to see its progress13:46
*** fifieldt has joined #openstack-infra13:47
*** ddieterly is now known as ddieterly[away]13:47
fungijeblair: mordred: olaph: oh, look, that change (328991) came from someone with an ncsu.edu address!13:47
*** dizquierdo has joined #openstack-infra13:47
openstackgerritMerged openstack-infra/storyboard-webclient: Allow the Subscribe directive to work without a list of subscriptions  https://review.openstack.org/32454913:47
olaphncsu ftw!13:49
rookfungi: yup! looks good!!13:49
rookfungi so... is there a way to remove jenkins from pushing the change?13:49
rookand allow the *-core group to do it13:50
rookolaph you talking about Sai?13:50
rookolaph he is a NCSU Grad!13:50
fungirook: jenkins doesn't "push" the change, it merely creates a merge commit for it (if needed)13:50
olaphrook: as am I!13:50
rookfungi - "Change has been successfully merged into the git repository by Jenkins"13:50
*** rbrndt has quit IRC13:51
rookolaph: my wife was a UNC grad... watch out :P13:51
rookolaph: and she is always right!13:51
*** amitgandhinz has quit IRC13:51
fungihalf my friends are unc grads and the other half ncsu grads. it's fun when they all come over for parties13:51
*** PollyZ has quit IRC13:52
rookfungi nice!13:52
*** mrmartin has quit IRC13:52
fungirook: look at the commit log at http://git.openstack.org/cgit/openstack/browbeat/log/ and you'll see it's the original commit13:52
* olaph assumes 'fun' is the polite term...13:52
fungiolaph: actually, they all get along fine13:53
Shrewsfungi: since i fall into neither of those groups, i now know where i stand13:53
fungiShrews: you should come over more (or at all)13:53
Shrewsfungi: last time i came over, i had to move furniture13:53
*** mrmartin has joined #openstack-infra13:54
Shrewsso...  fool me once, etc etc13:54
fungioh, oops!13:54
fungirook: the automation we have is basically taking your workflow +1 as an indication the change should merge, but defers merging it until it passes a final battery of testst (and until its prerequisites are merged, if there are any)13:56
*** PollyZ has joined #openstack-infra13:56
*** yamamoto has joined #openstack-infra13:56
fungirook: however, it preserves the original commit, and merely adds a merge commit if required to stitch it into the history13:57
*** ddieterly[away] is now known as ddieterly13:58
*** rbrndt has joined #openstack-infra13:59
*** xarses_ has quit IRC14:00
*** hongbin has joined #openstack-infra14:01
rooki see fungi - so dont +1 until we want to jenkins to push14:01
phschwartzclarkb: another review is comming for the launch_node.py that changes how we are using paramiko. Under certain situations the transport will throw a timeout socker.error exception that fails to be caught and is the cause of the issues I have been having. Cleaning up the catch all exception for the SSHException finally exposed the issue.14:01
rookfungi thanks... /me is used to the gerrithub.io workflow where I manually pushed14:01
fungirook: workflow +1, right14:01
Qiminghi, infra team, really appreciate it if you can help diagnose a problem at #senlin channel, gerritbot has disappeared for some days now ...14:01
thiagopWhatever you folks did to make devstack runs that fast, thank you so much!14:01
*** mhickey has quit IRC14:02
*** Julien-zte has joined #openstack-infra14:02
*** burgerk has joined #openstack-infra14:02
fungirook: there's a brief slide presentation at http://docs.openstack.org/infra/publications/zuul/ which covers the highlights/benefits of this model14:03
*** Julien-z_ has joined #openstack-infra14:03
*** zz_dimtruck is now known as dimtruck14:03
*** kushal has joined #openstack-infra14:04
*** pgadiya has joined #openstack-infra14:04
fungiQiming: i saw in scrollback where pleia2 was looking at it with you. skimming the logs for the bot, strangely the last two channels it _claims_ to join are #senlin and #openstack-searchlight but it never actually joins either of them that i can find (though it also doesn't mention any error)14:04
fungiQiming: pleia2: as far as i can tell the first 242 channels it joins, it shows up in, but those last 2 it doesn't14:05
*** jaosorior has quit IRC14:05
*** pgadiya has quit IRC14:05
*** yujunz has quit IRC14:05
Qimingmaybe it was getting tired knocking at so many doors ...14:05
fungii wonder if we're hitting the chanlimit14:06
*** mrmartin has quit IRC14:06
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run cinderclient functional tests in Identity v3-only  https://review.openstack.org/31356214:06
fungimaybe not. `/quote version` claims "CHANLIMIT=#:120"14:06
*** Julien-zte has quit IRC14:07
Qimingcould it be some channel flags changed?14:07
*** xarses_ has joined #openstack-infra14:08
*** xarses_ has quit IRC14:08
*** xarses_ has joined #openstack-infra14:09
*** yujunz has joined #openstack-infra14:10
*** amitgandhinz has joined #openstack-infra14:11
*** amitgandhinz has quit IRC14:11
*** cody-somerville has joined #openstack-infra14:11
*** cody-somerville has joined #openstack-infra14:11
*** amitgandhinz has joined #openstack-infra14:12
fungialso the debug log isn't showing a ERRTOOMANYCHANNELS or anything along those lines that i can find14:12
*** PollyZ has quit IRC14:13
*** PollyZ_ has joined #openstack-infra14:13
*** links has quit IRC14:14
*** cody-somerville has quit IRC14:16
*** mhickey has joined #openstack-infra14:17
*** faizy_ has quit IRC14:17
*** exploreshaifali has quit IRC14:18
fungion the other hand, gerritbot doesn't seem to log anything it receives from irc, so it may be that it's gettign an error back and we just don't know what14:19
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job  https://review.openstack.org/22978914:20
fungiprobably needs to be patched to log server messages at a minimum14:20
fungias a short-term fix, we could try pruning some unused channels from its configuration and see if it starts joining those last two14:21
*** xarses_ has quit IRC14:21
fungithose are both pretty low-hanging-fruit options if anyone has time to work on them14:22
fungi(patching gerritbot to record server messages, checking channels in our gerritbot config to identify some unused ones)14:23
yamamotopabelanger: ping14:24
openstackgerritDarragh Bailey proposed openstack-infra/project-config: Enable python3 jobs for git-upstream  https://review.openstack.org/32947514:25
*** matt-borland has quit IRC14:27
*** nmagnezi has quit IRC14:29
pabelangeryamamoto: hello14:30
yamamotopabelanger: hi14:30
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Remove pagination for timeline events  https://review.openstack.org/32768214:30
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Rework timeline events into a timeline  https://review.openstack.org/32723114:30
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a way to make time-moment not use am-time-ago  https://review.openstack.org/32947814:30
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run glanceclient functional tests in Identity v3-only  https://review.openstack.org/31357214:30
yamamotopabelanger: can you explain you comment on https://review.openstack.org/#/c/315033/ ?14:30
*** vhosakot has joined #openstack-infra14:30
*** mriedem1 has joined #openstack-infra14:31
yamamotodoes it imply some changes in grafyaml?14:31
pabelangeryamamoto: Right, lets create a new sub directory called openstack.  For now, we can add midonet into that, and do a follow up patch for tempest and neutron14:31
pabelangerI don't believe we need to update grafyaml14:32
*** _nadya_ has quit IRC14:32
*** _nadya_ has joined #openstack-infra14:33
sdaguemordred / jeblair - could one you upgrade mriedem & I to admin here - https://launchpad.net/~hacking-drivers/+members14:33
*** mriedem has quit IRC14:33
*** kaisers has joined #openstack-infra14:33
*** sree has quit IRC14:35
yamamotopabelanger: at least tools/grafyaml-validate.sh needs a change to visit the subdirectory, right?14:35
pabelangeryamamoto: let me check14:35
*** _nadya_ has quit IRC14:35
*** jamesmcarthur has joined #openstack-infra14:35
*** mhickey has quit IRC14:35
pabelangeryamamoto: let me move tempest first14:35
pabelangeryamamoto: then you can rebase onto it14:36
*** ayoung has joined #openstack-infra14:36
yamamotopabelanger: thank you!14:36
*** _nadya_ has joined #openstack-infra14:37
*** rfolco has quit IRC14:37
pabelangeryolanda: okay, that doesn't work, I'll add it to my list to fix.  In the mean time, lets see if AJaeger can help +A 31503314:41
*** _nadya_ has quit IRC14:41
* AJaeger checks14:41
*** mdrabe has quit IRC14:42
pabelangeryamamoto: ^ sorry yolanda14:42
yolandapabelanger, ah, i was reading backlog trying to figure what was the message about :)14:42
*** nmagnezi has joined #openstack-infra14:42
AJaegerpabelanger: is span 4 really right - or should it be 3?14:43
*** EricGonczer_ has joined #openstack-infra14:43
fungisdague: mriedem1: done. use your new-found superpowers only for good14:43
pabelangerAJaeger: should be, means panels are wider by will span 2 row.  There are 5 of them14:44
sdaguefungi: thanks14:44
mriedem1fungi: what do i have now and why?14:44
AJaegerpabelanger: ok14:44
*** _ari_ has quit IRC14:44
*** xarses_ has joined #openstack-infra14:44
*** _ari_ has joined #openstack-infra14:44
*** garyk has quit IRC14:44
fungimriedem1: sdague volunteered you to administer the hacking-drivers team on lp14:44
*** ajmiller has quit IRC14:45
AJaegersdague: could you review https://review.openstack.org/313588 , please?14:45
mriedem1oh i see14:45
fungimriedem1: perhaps because he knows you don't have nearly enough to do already? ;)14:45
* mriedem1 needs to update his resume then14:45
*** ajmiller has joined #openstack-infra14:45
*** eharney has quit IRC14:45
fungimriedem1: i hope your resume has a section called "thankless tasks"14:46
*** rfolco has joined #openstack-infra14:46
clarkbpabelanger: rcarrillocruz did we get anywhere with infra cloud firewall issues?14:46
*** inc0 has joined #openstack-infra14:46
pabelangerclarkb: I have not, but I also don't know who to contact other then rcarrillocruz14:46
*** julim has quit IRC14:47
*** mhickey has joined #openstack-infra14:47
*** reduxio has joined #openstack-infra14:48
Qimingfungi, following your suggestion, just digged through the channel list, cross verified the git repo status14:50
fungiQiming: excellent. find a few we should clean up?14:50
Qimingit seems to me, the following two channels can be removed: openstack-nova-solver-scheduler, openstack-magnetodb14:50
pabelangerclarkb: who is our contact at OSIC? I wanted to follow up with them about our SSH timeout issues on a set of public IPs14:50
Qimingboth channels are empty14:50
*** xarses_ is now known as xarses14:50
clarkbpabelanger: cloudnull hangs out here in channel14:50
Qimingboth projects seemingly dead14:50
pabelangerclarkb: cloudnull helped in the past, but he is currently away for a while longer14:51
Qimingfungi, as for "openstack-vahana", not so sure, it is a new thing, not commit for 6 months, channel empty14:51
*** zhurong has quit IRC14:52
jeblairpabelanger: i'm not entirely sure i want to ignore rsync errors.  i think it would be good to make the ansible logs available to users so they can diagnose that output themselves.  but really, i think this is exposing an actual malfunction in the job.14:52
clarkbpabelanger: JayF is also at OSIC I think and James Thorne is the person that set up the cluster for us14:52
reduxioHello. My company is developing a cinder driver, and I am currently in the process of setting up the third party CI. I understand that part of the process is to register as a third party company in order to be allowed to comment to gerrit. can anyone help with how i do this?14:52
JayFclarkb: pabelanger: I've got nothing to do with that OSIC cluster, but I can point people in your direction if needed.14:53
jeblairpabelanger: like -- someone said "copy a file from /var/log" but didn't correct the permissions to do that.  that's a bug in the job.14:53
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Support subscription to worklists in the API  https://review.openstack.org/32449314:53
JayFreduxio: I believe that's all fairly well documented here: http://docs.openstack.org/infra/system-config/third_party.html14:53
jeblairclarkb, pabelanger: what if we told zuul-launcher to put the ansible output the same place as the console log?  basically just overloaded the 'copy console log' publisher to also mean 'this is where you should copy the ansible log output'14:54
jeblairclarkb, pabelanger: we could either concatenate it to the console log, or make a separate file....14:55
* jeblair leans toward separate file14:55
clarkb+1 for separte file14:55
*** amotoki has quit IRC14:55
*** eharney has joined #openstack-infra14:56
fungiQiming: yeah, openstack/nova-solver-scheduler does look sort of dead (no commits this year, core reviewer team of 2, empty channel except for chanserv and openstackgerrit)14:56
fungiQiming: and i believe magnetodb was completely dissolved? double-checking14:57
reduxioJayf: In the link you sent, it says i need to join the weekly meeting and points me to the wiki. In the wiki, it states that if you want to participate in the meeting i need to add the issue to the meeting agenda. In order to add to the meeting agenda, i need to the login to the wiki. This fails with http error 500.14:57
AJaegerfungi: megnetodb is not in gerritbot setup AFAIK14:57
fungiAJaeger: Qiming: yeah, i don't see #openstack-magnetodb in the current config14:58
openstackgerritAlina Nesterova proposed openstack-infra/project-config: Migrated sahara-client-functional-job  https://review.openstack.org/32949714:58
JayFreduxio: Wiki logins were disabled due to spam. You're 100% right. If you ask in here (I don't have access, I just idle here :P), someone may be willing to enable wiki logins for you to get a user "created" with your openid. Alternatively, I bet anteaya would be willing to add you to the agenda as iirc she runs those meetings.14:58
pabelangerjeblair: I was actually thinking about add log_path to our ansible.cfg (per job), that would give us an output log per job run. The figure out the upload part14:58
*** mdrabe has joined #openstack-infra14:58
*** _ody has joined #openstack-infra14:58
fungireduxio: try logging in again. i've temporarily reenabled wiki account creation14:58
pabelangerjeblair: then upload zuul-launcher logs first, then job logs14:59
pabelangershould make debugging easier for the user14:59
clarkbfungi: how do you temporarily enable it? edit the config.php and restart apache?14:59
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Magnum functional tests in Identity v3-only  https://review.openstack.org/31511814:59
reduxiofungi: thanks! logged in now.14:59
fungiclarkb: simpler still. comment out line 149 (currently) of /srv/mediawiki/Settings.php14:59
fungiclarkb: #$wgGroupPermissions['*']['createaccount'] = false;14:59
clarkbfungi: no service restarts required?14:59
fungiclarkb: no restart needed, it's automatically reread on each connection apparently15:00
jeblairpabelanger: we still need the logs in the zuul-launcher logs.  but as long as we can do that.15:00
openstackgerritMerged openstack-infra/project-config: Add networking-midonet grafana dashboard  https://review.openstack.org/31503315:00
fungireduxio: awesome, let us know if you need anything else15:00
*** jamesmcarthur has quit IRC15:01
*** bhavik has joined #openstack-infra15:01
pabelangerjeblair: agreed. 326006 was my first attempt to do our uploads from zuul-launcher, but not sure I like it as it creates an empty publisher in jenkins15:01
pabelangerI believe 326006,1 over 326006,215:02
jeblairpabelanger: yeah, i'm thinking maybe we just interpret the definition of a console log liberally to include also ansible logs and maybe ansible config if we want15:02
pabelangerworks getter15:02
pabelangerjeblair: okay15:03
jeblairpabelanger: can we have ansible run every post task regardless of whether previous ones have succeeded or failed, but then still exit with a failure code if any have failed?15:03
jeblairShrews: ^15:03
openstackgerritAlina Nesterova proposed openstack-infra/project-config: Migrated sahara-client-functional-job  https://review.openstack.org/32949715:03
*** jamesmcarthur has joined #openstack-infra15:03
*** Swanson has joined #openstack-infra15:03
pabelangerjeblair: I would need to check15:03
clarkbpabelanger: are ssh timeouts the source of the fairly consistent node boot errors in osic?15:04
Shrewsjeblair: couldn't zuul handle that? you run each separately15:05
clarkbpabelanger: its possible we just need to bump up that timeout because cloud can't reliably do it in the allocated time15:05
pabelangerjeblair: I think we can use the always flag for block tasks15:05
pabelangerclarkb: Ya, it was a list of specific IPs. Let me see if I can find it again15:05
Shrewsjeblair: oh, i misunderstood. you mean isolated to the post playbook only15:05
*** yamahata has joined #openstack-infra15:05
jeblairShrews: because post will have several upload steps, and if one breaks, we still may want others to run, especially if the others tell us why the one broke15:06
Shrewsjeblair: i think there is an ignore_errors thing15:06
*** wc2912 has joined #openstack-infra15:06
jeblairShrews: (but i still want to know, at the end, if any failed with the exit code)15:06
*** abregman has joined #openstack-infra15:07
*** mhickey has quit IRC15:07
*** Jeffrey4l has quit IRC15:07
jeblairyeah, i think ignore_errors may be what we want as long as we still get the exit code (the docs are unspecific on that point)15:08
rcarrillocruzyup, passing ignore_errors true to a task makes it run regardless15:08
Shrewsjeblair: maybe with some sortof handler. i'd have to investigate15:08
rcarrillocruzbut that's a task only thing15:08
*** fifieldt has quit IRC15:08
clarkbrcarrillocruz: did you see there are firewall issues with the infracloud host(s)15:08
rcarrillocruzclarkb: i'll check15:08
Shrewsjeblair: i don't know what ignore_errors does for the playbook exit code.15:09
*** pradk has quit IRC15:11
*** rcernin has quit IRC15:11
rcarrillocruzi don't think it's a firewall issue on google dns nameservers, but rather an outbound gw issue, i can't ping anything15:11
rcarrillocruzi'll check with the net folks15:11
clarkbrcarrillocruz: its likely the same firewall also blocks icmp :)15:12
pabelangerclarkb: http://paste.openstack.org/show/507754/ that was the list of IPs we cannot SSH into15:13
clarkbpabelanger: so its a consistent list of IPs that fail? fun15:13
pabelangerclarkb: Yup, I am hoping some ACL on osic side15:13
*** julim has joined #openstack-infra15:14
*** lxsli has left #openstack-infra15:14
openstackgerritMerged openstack-infra/project-config: Pass extra parameter to hooks in neutron-lbaas tempest jobs  https://review.openstack.org/32281115:15
openstackgerritMerged openstack-infra/project-config: Add Sheepdog test job to glance_store  https://review.openstack.org/32814815:15
*** jlanoux has quit IRC15:18
*** croelandt has joined #openstack-infra15:18
*** psachin has quit IRC15:18
*** pilgrimstack has joined #openstack-infra15:18
rcarrillocruzclarkb: not just icmp, but everything then, cos http does not work either15:19
rcarrillocruzwhich is i why i beleive there's an issue with the gw15:19
rcarrillocruzchecking now with some folk15:19
zulmind someone +1ing this please? https://review.openstack.org/#/c/329331/15:20
openstackgerritPaul Belanger proposed openstack-infra/zuul: Refactor local_action SCP task into function  https://review.openstack.org/32560115:22
*** pilgrimstack has quit IRC15:24
Shrewsjeblair: a bit cumbersome with the registered vars, but this works: http://paste.openstack.org/show/516014/15:24
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add infrstucture scripts to prepare rh2  https://review.openstack.org/29524315:24
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/32952115:24
Shrewsjeblair: sort of a brute-force way which is kind of meh15:25
*** xarses_ has joined #openstack-infra15:25
*** salv-orlando has quit IRC15:25
jeblairShrews: yeah, our autogenerated playbook will definitely look like an autogenerated playbook.  :)15:25
jeblairShrews: i just confirmed ignore_errors exits 0 :(15:25
*** xarses has quit IRC15:26
Shrewsjeblair: yeah, i did too, which led me to that paste15:26
*** salv-orlando has joined #openstack-infra15:26
Shrewsjeblair: gotta run out for a bit. bbl15:26
jeblairShrews: thanks!15:27
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Allow JJB to spoof plugin_info via yaml  https://review.openstack.org/32672215:27
*** Julien-z_ has quit IRC15:28
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Allow update to provide plugin_info via yaml  https://review.openstack.org/32672215:28
openstackgerritPaul Belanger proposed openstack-infra/zuul: Add log_path to ansible.cfg  https://review.openstack.org/32952515:29
*** javeriak has joined #openstack-infra15:29
*** salv-orlando has quit IRC15:30
*** salv-orlando has joined #openstack-infra15:30
*** tongli has joined #openstack-infra15:31
dhellmannwe're seeing lots of failures for wheel-build-ubuntu-trusty-amd64-python2 and wheel-build-centos-7-amd64-python2 from releases this morning; those just update our cache, right? so we don't need to worry that they failed?15:31
*** matt-borland has joined #openstack-infra15:31
dhellmannthe error is "The executable {python} (from --python={python}) does not exist" -- http://logs.openstack.org/f5/f59009760a073b2125abdeeea703b2cc09cf17c9/release/wheel-build-centos-7-amd64-python2/363c6ea/console.html15:31
dhellmannlooks like some sort of templating issue15:32
pabelangerdhellmann: checking15:32
*** zeih has joined #openstack-infra15:32
*** zeih has quit IRC15:32
*** zeih has joined #openstack-infra15:32
pabelangerdhellmann: yes, I see the issue. Let me propose a fix.15:33
dhellmannpabelanger : great, thanks! do we need to requeue those jobs?15:34
*** jtomasek_ has joined #openstack-infra15:34
pabelangerdhellmann: I'd have to see, but we can if needed15:34
croelandtHey! I've been looking at bug https://bugs.launchpad.net/python-glanceclient/+bug/1546258 , and it seems that the tarball for glanceclient 1.2.0 is missing from PyPI (see https://pypi.python.org/pypi/python-glanceclient/json)15:35
openstackLaunchpad bug 1546258 in python-glanceclient "python-glanceclient==1.2.0 not found by setuptools" [Undecided,New]15:35
croelandtIs there anything that can be done about this?15:35
dhellmannpabelanger : ok, http://lists.openstack.org/pipermail/release-job-failures/2016-June/thread.html#0 is a good source for a list if we need to15:35
*** abregman is now known as abregman|afk15:35
jeblairclarkb, pabelanger: how about we modify 325602 so that instead of looking for specific jjb configuration, we add an extra task to the console-logs publisher that copies the logs and ansible config into the staging area so that it gets uploaded with the console log upload.15:35
*** Na3iL has quit IRC15:36
jamesmcarthurclarkb: fungi: mordred: Per our conversation in Prague, can we move forward with patch# 285641 and then we will work with jpmaxman on next steps?15:36
*** sorantis has quit IRC15:37
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: WIP: Remove 'partial' support for Grenade  https://review.openstack.org/32953215:37
*** zeih has quit IRC15:38
*** permalac has joined #openstack-infra15:38
thiagopGuys, do zuul developers chat on this channel too?15:38
pabelangerjeblair: clarkb: We can start with that, as it gets our logs uploaded for now. But ideally, I would like to have publishing task specific to zuul-launcher, before we attempt to fetch logs from the remote node.  That way, if that console task fails, we still have our logs published.15:39
*** eezhova has quit IRC15:39
jeblairpabelanger: i think we will... here, let me etherpad it15:39
clarkbjamesmcarthur: we should definitely keep moving on the things. I think fungi had some previously unknown info that maybe he wanted to share? (I think he and pleia2 were talking about doing the wiki upgrade real soon now like right now maybe?)15:39
clarkbthiagop: they do15:40
scottdaHi Infra. I'm looking for the place where I set a devstack variable. for example, L#240 in devstack-vm-gate.sh:15:40
*** nmagnezi has quit IRC15:40
fungijamesmcarthur: jpmaxman: sorry, been meaning to reply to the points in the e-mail thread you started with me, fifieldt, mordred, clarkb and ttx. apparently the discussion in prague was based on some misconceptions (pleia2 and i have picked up the wiki cleanup and upgrade work since it turns out pabelanger and nibalizer can't work on it after all), the patrolling feature fifieldt was asking to have15:40
thiagopclarkb: I'm suspecting canceled jobs aren't releasing mutexes, but my knowledge of the codebase is a little shallow15:40
fungiturned on is and was already enabled, we've been working around the account lockdown by assisting new users in the interim, et cetera15:40
scottdaI assume it is in some jenkins job?15:40
*** yujunz has quit IRC15:40
scottdaI cannot find that variable in jenkins-job-builder repo.15:41
scottdaIs there another repo for jenkins jobs?15:41
*** matt-borland has quit IRC15:41
scottdaThis is for Tempest testing in the merge queue15:41
fungijamesmcarthur: jpmaxman: also most of the work we need help with on the wiki doesn't require ssh access at all. for example the thousands of pages/uploads i deleted and hundreds of users i blocked were done entirely through the web interface and mediawiki api15:41
*** roxanaghe has joined #openstack-infra15:41
clarkbscottda: you set it in the environment calling devstack gate15:42
jamesmcarthurfungi: but the wiki is still locked down to new users and not usable by the community15:42
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Allow update to provide plugin_info via yaml  https://review.openstack.org/32672215:42
jpmaxmanfungi I think the key question is how the spam was getting into the wiki in the first place15:42
jpmaxmandid you figure that out?15:42
fungijpmaxman: that has never been a question?15:42
scottdaclarkb: And isn't that a jenkins job running tempest tests?15:42
clarkbscottda: I think that may be old code for older branches and current branches use a devstack plugin though15:42
clarkbscottda: which is why you don't see the var set in any of the jenkins jobs. Instead the ceph devstack plugin is enabled15:42
fungijpmaxman: i last week i even found an actual openstack community member whose lp sso account was compromised and used to add spam pages15:42
fungii've temporarily blocked them in the wiki until we can get up with them to reset their account on launchpad15:43
*** yamamoto has quit IRC15:43
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Add python version to wheel builders"  https://review.openstack.org/32953615:43
jpmaxmanin so far as I recall we had put captcha requests on edits and new page creations15:44
jpmaxmanyet spam was still getting in15:44
scottdaclarkb: OK, thanks. That makes sense. If I'm adding a new variable for DEVSTACK_CINDER_ENABLE_MULTI_BACKEND it won't be in a plugin. Where would I set that?15:44
*** pradk has joined #openstack-infra15:44
jpmaxmanso the question was is the captcha not good enough15:44
fungijpmaxman: oh, you're asking whether they're solving the captchas vs bypassing them by using page edit and create api calls?15:44
jpmaxmanor was there a way they were getting around15:44
pabelangerfungi: dhellmann: I've proposed a revert for now, so not to block things15:44
*** zeih has joined #openstack-infra15:44
clarkbscottda: you would set that in the calling context of devstack-gate, likely a jenkins job15:44
jamesmcarthurjpmaxman: fungi: the issue is that the wiki is still locked down. we proposed to attempt best practices and try different plugins to ensure that even with access turned on again, we could prevent spam.15:44
*** yamamoto has joined #openstack-infra15:44
jpmaxmanwhat I was wanting to figure out is exactly what the attack vectors were so we could then more knowledgeably move forward15:45
jpmaxmanbut if you want to start by upgrading everything and seeing where we're at15:45
clarkbjpmaxman: the attack vector is valid accounts were making edits and the rudimentary captcha on old mediawiki doesn't prevent those valid accounts from making edits15:45
jpmaxmanthat's reasonable too15:45
jamesmcarthurfungi: once we, in conjunction with infra, agreeed on a set of plugins that could be tested and work, we would commit back to the repo so that they would go in gerrit15:45
*** roxanaghe has quit IRC15:46
clarkbjpmaxman: at least that was my understanding, and part of the goal behind mediawiki upgrade was to use better captcha system available in newer mediawiki?15:46
jpmaxmancalrkb is that true?  If so I think it is a setting - you can set ALL accounts to be presented with captcha when making edits15:46
*** eezhova has joined #openstack-infra15:46
clarkbjpmaxman: we did set that and it didn't stop the spam15:46
pabelangerI would rather we spend the effort on upgrade our puppet manifest, and operating system, then reenable users on wiki.o.o.15:47
jpmaxmanso there's the problem15:47
scottdaclarkb: OK, thanks for the help.15:47
pabelangerthen iterate forward on taht15:47
jpmaxmanwhy didn't it stop the spam?15:47
clarkbjpmaxman: its too rudimentary iwth a tiny problem set15:47
jpmaxmanbecause they were able to bypass captcha?15:47
jamesmcarthurfungi: clarkb: mordred: I'm a little flumoxed here. I feel like we came up with a pretty solid plan.15:47
jpmaxmanor because the captcha was too easy?15:47
clarkbjpmaxman: yes that15:47
*** yamamoto has quit IRC15:47
jpmaxmanif I recall one thing I suggested was to set the captch as an impossible to answer question15:47
fungiwell, more that the current state of the spam mitigation extensions are driven by the spam handled by the wikipedia community, and since wikipedia always runs latest mediawiki those extensions are not well-supported on older releases15:47
*** bpokorny has joined #openstack-infra15:48
jpmaxmanto see if it stopped it15:48
jeblairclarkb, pabelanger, Shrews: https://etherpad.openstack.org/p/p25pzG6OSr15:48
jpmaxmanthen you'd know15:48
clarkbjpmaxman: yes that stopped it15:48
*** yamamoto has joined #openstack-infra15:48
jpmaxmanoh was that done?  I thought it never was...15:48
jamesmcarthurfungi: clarkb: mordred: Can we confirm that the plan is no longer to approve jpmaxman's patch?15:48
clarkbjpmaxman: it was, we set it to a random many byte string and it killed the spam but using the small set of easier to answer things that humans could know didn't15:48
clarkbjamesmcarthur: I am happy to continue with the old plan but didn't realize that othe rpeople had volunteered separately to do the work already. So just wanted that info to be added in15:49
fungialso we originally had the captcha set only for password-based account creation and all our accounts were being created through openid until we made the captcha answerable, then we started getting all sorts of accounts showing up created without openid15:49
jpmaxmanclarkb ok - in that case we could try: https://www.google.com/recaptcha/intro/index.html which may be the quickest fix15:49
*** yamamoto has quit IRC15:50
jamesmcarthurfungi: clarkb: again, the issue here is attempting to figure out which set of plugins are going to work best15:50
jpmaxmanfungi: there was also the mobile view vector - both of those should be disabled15:50
fungijpmaxman: right, the mobile view account creation was password-based, so it was unusable for creating new accounts up until we made the captcha something other than secret gibberish15:51
jamesmcarthurfungi: there was also a discussion of trying to get other people involved and contributing in positive ways to infra. Basically, the state of the wiki is "so bad we can't possibly do any worse" as ttx said.15:51
pabelangerjamesmcarthur: my objection is not to finding out which plugins work, it is the fact we are testing them on our production server. When in fact we should be setting up wiki-dev.o.o15:51
jpmaxmanfungi: right also the mobile view login wasn't redirecting to open id15:52
*** jlanoux has joined #openstack-infra15:52
jamesmcarthurpabelanger: traffic doesn't go to wiki-dev.o.o. Right now you have a completely unusable wiki.15:52
pabelangerwhich, to get wiki-dev.o.o, requires us to migrate to ubuntu-trusty and update our puppet manifests15:52
jamesmcarthurpabelenager: no, we do not need to migrate to ubuntu-trusty15:52
pabelangeryes we do15:52
jamesmcarthurwe are talking about utilizing existing plugins15:52
jamesmcarthurto help stop the spam as a first priority15:52
jamesmcarthurpabelanger: this circular reasoning is the reason we have been without a working wiki for 8 months15:53
jpmaxmanpabelanger: the production server is what is under attack - so to quickly know what stops the spam it is necessary to run some quick tests there as opposed to dev15:53
*** jamesmcarthur has quit IRC15:53
*** andreas_s has quit IRC15:53
jpmaxmananyway when I last looked at it15:53
fungithe confirmedit extension seems to support recaptcha and works with mw 1.25, which we could enable in the near term https://www.mediawiki.org/wiki/Extension:ConfirmEdit15:53
jpmaxmanthere was a unified anit-spam plugin (which is installed already)15:53
pabelangerregardless of the spam being stopped or not, we need to migrate wiki.o.o to ubuntu-trusty15:53
jpmaxmanand what we actually need to do is tweak that to use the right config of that plugin to be just hard enough to stop the spam but easy enough to let actual users publish15:53
clarkbfungi: right thoes types of steps are what I thin kwe were interested in doing in the very near term15:54
fungijamesmcarthur: i take exception to the claim that we've been without a working wiki at all, much less for 8 months. people are using it constantly15:54
clarkband jpmaxman/jamesmcarthur were willing to do the work when no one else was stepping up15:54
clarkbalso potentially using the patrol feature to better curate edits to automate spam cleanup as well15:54
*** jtomasek_ has quit IRC15:54
clarkbpabelanger: we do but thats independent of stopping spam15:55
clarkbpabelanger: and unfortunately tying them together has killed any progress in the space15:55
*** yamamoto has joined #openstack-infra15:55
fungiclarkb: patrol is already enabled, as i mentioned earlier. i really think that what we need _first_ are people willing to monitor edits and contributions from new accounts. that's basically all patrolling is anyway15:55
*** jamesmcarthur has joined #openstack-infra15:56
clarkbfungi: yup, we were mistaken that it wasn't enabled. So next step there would be recruiting wiki admins to do the patroling15:56
fungieveryone seems to think that this is something we can solve with a few patches and then turn it loose again, but if this is going to work at all it needs a bunch of people keeping an eye on the wiki. that doesn't involve ssh access at all, just wiki curators15:56
fungiit's time spent staring at a web browser15:57
*** brucet has joined #openstack-infra15:57
fungiso far very few people seem to want to do _that_ part15:57
jpmaxmanfungi: i prefer tailing logs ;)15:57
fungii spent two days last week doing it, and removed thousands of pages of spam and blocked hundreds of users15:57
bkerofungi: did you get through all of it? I'm willing to monitor the change list for incoming spam15:58
*** sshnaidm has quit IRC15:58
bkerobut that back log is scary15:58
jpmaxmananyway all I was going to do is refine the anti spam settings and test config changes wile watching the logs for the spam attacks15:58
jpmaxmanas I said earlier if you want to upgrade everything first (mediawiki version, plugins, trusty, etc)15:58
jpmaxmanand then see where you're at - I think that's valid too15:58
jamesmcarthurfungi: jpmaxman: right, but htat's been "the plan" for months now15:59
fungibkero: as far as i know it is all cleaned up. so far a few people have gone through and done random searches for the sorts of keywords we saw in spam previously and have found no new hits so i think it's 100% but if you spot anything i missed please tell me15:59
jpmaxmanIf you need some help on the devops side, I'm happy to jump in.  If you got it, that's cool too.16:00
*** e0ne has quit IRC16:00
bkerofungi: I'll see if I can spot anything new that pops up16:00
fungijpmaxman: jamesmcarthur: anyway, "best practices" is not a plan. "install and configure the confirmedit plugin to use recaptcha for page edits, reenable new account creation and then monitor activity to see what slips through" is a plan i can get behind16:00
jpmaxmanconfirmedit is already installed16:00
*** _nadya_ has joined #openstack-infra16:01
clarkbI think we need to decouple upgrade from spam handling. We know there are plugins and options available to us on current versions of things that we can use. We also know that we need people to curate wiki edits more directly (sounds like bkero is willing to help there and hopeflly the existing wiki admins too)16:01
jpmaxmanit just needs to be tweaked to use the right settings that block the spam assuming what clarkb said is correct that a complete hardening stopped it16:01
openstackgerritMikhail S Medvedev proposed openstack-infra/puppet-openstackci: Move subunit_worker.pp from system-config  https://review.openstack.org/32843816:01
clarkbthen separately some group of people can work on upgrading to newer ubuntu as the platform to run on16:01
*** thorst is now known as thorst_afk16:01
fungijpmaxman: configure confirmedit to use recaptcha, right. i stand corrected16:01
openstackgerritHisashi Osanai proposed openstack-infra/project-config: Add jobs for Monasca-Analytics  https://review.openstack.org/32937616:01
jamesmcarthurfungi: that's a bit frustrating. You know what the proposal was because we outlined it in an email after a discussion with mordred, clarkb, fifeldt, and ttx.16:01
*** yamamoto has quit IRC16:01
*** _nadya_ has quit IRC16:02
clarkband the third thing that needs to happen is to properly get the wiki into config mgmt16:02
*** markusry has quit IRC16:02
fungijamesmcarthur: no, you didn't. you wrote "Figure out best plugins for spam prevention/policing" in your plan16:02
jamesmcarthurfungi: right. There are a ton of options.16:02
clarkbthe second and third items may be related depending on how difficult an upgrade is without config mgmt16:02
jamesmcarthurWe have to see which set of plugins work together to solve the problem.16:02
bkerogetting it into config management sounds like a lot of reverse engineering since we don't exactly know how it is set up16:03
*** jcoufal_ has joined #openstack-infra16:03
fungijamesmcarthur: that's fine. and i like the initial suggestion of "let's see if the recaptcha implementation in confirmedit deters spammers better than the simple question/answer mechanism it has"16:03
jamesmcarthurfungi: if you want to know why people are having a hard time dealing with the infra team, this is a premier example. We came up with a very valid plan, working specifically with members of the infra team. Literally the phrase was "how can it get any worse"16:03
jamesmcarthurfungi: only to have it walked back two days later.16:03
jpmaxmanthere's also some cloud based solutions16:03
*** matrohon has quit IRC16:04
jpmaxman$8 / yr16:04
*** arxcruz has quit IRC16:04
jpmaxmanbut they ostensibly manage keeping up with the attack tactics16:04
*** hashar is now known as hasharOut16:04
ttxjamesmcarthur: The situation is not exactly what I thought it was when we had that discussion. But I would still support direct experimentation under the "can't really be worse" argument. That said I'm not infra :)16:04
fungijamesmcarthur: jpmaxman: i don't get what's being walked back. i'm in favor of adding ssh access for you to test out recaptcha support in confirmedit and see if it helps. that's likely to get done sooner if you have time to focus on it than if i'm stuck having to work it into my week16:05
*** gouthamr has quit IRC16:05
*** mikelk has quit IRC16:06
*** jcoufal has quit IRC16:06
fungii just wanted to follow up to your proposed plan to say that it should get a little more specific about which exact solutions were worth trying16:06
ttxIt feels like we are still following the spirit of that plan, from where I'm standing16:07
jamesmcarthurfungi: the point is that you have to try out different plugins while the wiki is active and determine which are working to stop spam and which arent'16:07
*** bpokorny has quit IRC16:07
jamesmcarthurwell step 1 in the plan was"Approve JP’s patch for root access (patch# 285641), addressing jeblair's comments.16:07
jpmaxmanfungi: I think the plan was to rapidly try several and watch the logs to see what stopped the spam then to identify those and present them as an official patch16:07
fungijamesmcarthur: yep, i get that. and the plugins/settings you wanted to try are what i was asking16:07
jpmaxmanfungi: but the entire idea is that in the time it has taken to have this conversation we could have already tried several variations of that plugin setting16:08
fungia couple of good suggestions (one free and one paid) have been mentioned in here now, and that's cool with me16:08
jamesmcarthurfungi: TBD once we were able to do it. There are a whole set of them.16:08
jpmaxmanthe last I checked the spam was coming in at such a rapid pace you'd know within minutes if what you did was effective16:08
jamesmcarthurAgain, and I quote from clarkb: "Communicate to Infra that you're physically working on box. Communuication is more important than the technical steps."16:08
jamesmcarthurThe point isn't outlining exactly which ones, just htat we were working on it.16:08
fungijpmaxman: jamesmcarthur: i get that communicating with a community is tiresome for you, but please let's have conversations about what you want to do rather than "just do it"16:08
jpmaxmanin the months we've been talking about it we could have written our own wiki :P16:09
jamesmcarthurfungi: that's a load of crap16:09
anteayajamesmcarthur: please watch your language16:09
jpmaxmanit's not tiresome, it's just talking in circles16:09
*** ddieterly is now known as ddieterly[away]16:09
jpmaxmanbecause you're not actually trying anything16:09
jamesmcarthurfungi: we have been talking for 8 months16:09
jpmaxmanso it's all conjecture and assumptions16:10
jamesmcarthurmeanwhile we have no working wiki16:10
*** yamamoto has joined #openstack-infra16:10
*** xyang1 has quit IRC16:10
jamesmcarthurfungi: i sat down for 2 hours in Prague and we all agreed that the wiki is a mess. It's broken. Nobody can use it.16:10
jamesmcarthurfungi: if you want to tell me that us sitting here having a conversation with you is us not participating in the community, I think you should take a long look at your definition of community16:11
fungii find the repeated assertions about "no working wiki for 8 months," "talking in circles" and "not actually trying anything" tiresome, but i'm willing to continue discussing16:11
electrofelixzxiiro: you happy with https://review.openstack.org/#/c/314900/ mutliple SCM fix?16:11
*** zeih has quit IRC16:11
*** yamamoto_ has joined #openstack-infra16:11
*** yamamoto has quit IRC16:12
*** ddieterly[away] is now known as ddieterly16:12
mtreinishjeblair, fungi: This failed the gate with a POST_FAILURE: https://review.openstack.org/#/c/328467/ I've not seen that before16:12
mtreinishI assume it's something related to all the recent zuul changes16:13
jpmaxmanfungi: you're right - so as I said moments ago and months ago, if you'd like my help I'm happy to get into the server and figure out the various spam attack vectors and what does the best job of stopping them then propose a solution to the community.16:13
*** xyang1 has joined #openstack-infra16:13
jpmaxmanif you don't, that's fine too, I got plenty to do :)16:13
jpmaxmanjamesmcarthur had come back from Prague hopeful that we could all work together16:13
ttxjamesmcarthur: I think communicating about what we are trying is still in the spirit of the plan. I remember clarkb suggesting just dropping a line on IRC16:13
jpmaxmanand had asked me to jump back in16:13
jeblairmtreinish: yeah, we're replacing UNSTABLE (cause 'what?') with that somewhat more specific error -- it means a publisher (like, log copying) failed.  we're working today on making those logs more visible so you can self-diagnose the problem.  let me look them up for you for that failure.16:14
ttxAlso "the plan" was just a suggested way forward, still had to be discussed with the rest of the infra team16:14
ttxI personally think it's acceptable, but I'm not an active infra team member16:14
*** yamahata has quit IRC16:14
fungijamesmcarthur: jpmaxman: yes, to get back to the root of the discussion, i'm in favor of turning new account creation back on if you are going to be tracking new accounts/edits for signs of spam and working on putting a better captcha in confirmedit, which will need ssh access i agree16:15
*** abregman|afk is now known as abregman16:15
jpmaxmancheers guys - I've got to turn my attention elsewhere. Let me know if you need anything.16:15
*** mriedem has joined #openstack-infra16:15
ttxI don't feel like requiring to drop a line on irc explaining what you're trying is an unreasonable requirement16:15
*** yamamoto_ has quit IRC16:15
fungijamesmcarthur: jpmaxman: it's a good next step as far as i'm concerned16:16
EmilienM50% packet loss on review.o.o16:16
*** sergmelikyan has joined #openstack-infra16:16
EmilienMjust fyi16:16
*** permalac has quit IRC16:16
jeblairEmilienM: v4 or v6?16:16
*** sergmelikyan has quit IRC16:16
fungijamesmcarthur: jpmaxman: since jeblair was the most outspoken about the original ssh access patch, i'd like to get him to weigh in so we can hopefully get on the same page there16:16
*** Apoorva has joined #openstack-infra16:16
clarkbEmilienM: I am not seeing loss from my location16:16
*** kevinshort has quit IRC16:17
jeblairclarkb, EmilienM: nor i16:17
EmilienMyou guys in west coas iirc16:17
*** zeih has joined #openstack-infra16:17
EmilienMmy remote is
*** Apoorva has quit IRC16:17
EmilienMfungi: you on east coast, can you try please?16:17
*** bhavik has quit IRC16:17
*** yamahata has joined #openstack-infra16:18
fungijamesmcarthur: jpmaxman: i simply wanted to have the discussion in a venue where the team can participate. those sorts of decisions aren't made in a private discussion with two infra sysadmins at a table in prague16:18
mtreinishEmilienM: fwiw, I did a ping and got 46% loss with 15 packets sent16:18
*** Apoorva has joined #openstack-infra16:18
*** mriedem1 has quit IRC16:18
EmilienMpaste.o.o -> same16:18
clarkbit sounds like people are mostly in violent agreement16:18
fungiEmilienM: ipv4 or ipv6?16:18
*** vincentll has quit IRC16:18
mtreinishheh, yeah I can't even get paste to load16:19
fungiEmilienM: so far no packet loss for me via ipv4 (broadband cable provider "charter communications" at my house)16:19
clarkbthere are a set of things we can do to the wiki that will help the situation. Doing them likely requires root, there is a small number of people volunteering to do the work16:19
EmilienMpings: http://paste.openstack.org/show/pXLQkNkGqeXUS1MjgIzC/16:19
*** kevinshort has joined #openstack-infra16:19
*** ifarkas has quit IRC16:19
fungi50 packets transmitted, 50 received, 0% packet loss, time 49063ms, rtt min/avg/max/mdev = 55.669/58.229/62.906/1.432 ms16:19
EmilienMmhh ok16:19
clarkbEmilienM: try mtr16:19
*** javeriak_ has joined #openstack-infra16:20
clarkbEmilienM: you might find the hole more easily that way16:20
fungiyeah, this is probably a problem peering somewhere between providers i'm not traversing16:20
*** javeriak has quit IRC16:20
jeblairfungi: thank you.  as someone who wasn't at the table or on the email thread, i appreciate that.  however, it was pretty clear that in this conversation, suggestions deviating from what was decided at the table are not welcome, so i judged that my participation would not be helpful.16:20
*** bpokorny has joined #openstack-infra16:21
AJaegerscottda: did you use http://codesearch.openstack.org/ ?16:21
EmilienMclarkb: indeed16:21
jeblairfungi, jamesmcarthur, jpmaxman: i have no objections to adding jpmaxman's key to wiki.  jpmaxman, could you propose the change to add your own key?16:22
*** zeih has quit IRC16:22
fungijeblair: i also wasn't at that table, got briefed mostly via a private e-mail from jamesmcarthur, so was asking for more details on the plan16:22
scottdaAJaeger: No, but thanks for the reminder!16:22
mtreinishEmilienM: I'm seeing a ~65% loss rate on: dls-b22-link.telia.net looking at mtr16:23
EmilienMclarkb: rackspace-ic-302090-dls-bb1.c.telia.net is loosing a lot of packets16:23
fungiEmilienM: mtreinish: do you get consistent loss for hops after that one as well?16:23
scottdaAJaeger: Unfortunately, it doesn't show  up there: http://codesearch.openstack.org/?q=DEVSTACK_GATE_CEPH&i=nope&files=&repos=16:23
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Remove gate-grenade-dsvm-partial-ironic  https://review.openstack.org/32955316:24
mtreinishfungi: yeah16:24
jpmaxman@jeblair it is proposed here patch# 28564116:24
mtreinishit's a bit less in the percentages but still >50%16:24
fungiEmilienM: mtreinish: the name looks like the peering between telia and rackspace16:24
jeblairjpmaxman: that is mrmarton adding your key, which is somewhat unusual -- is it possible for you to propose adding your own key?16:24
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Migrated sahara-client-functional-job  https://review.openstack.org/32949716:25
*** pradk has quit IRC16:26
fungiEmilienM: mtreinish: my traceroute seems to go into rackspace via a peering between my internet service provider and rackspace in the dfw equinix facility, so i guess i'm lucky ;)16:26
pleia2btw, I have the wiki upgrade spec drafted here: https://review.openstack.org/#/c/328455/16:26
jpmaxmanjeblair, he was just being helpful. If it makes a difference to you sure16:26
pleia2only anteaya has looked at it so far, it needs more input16:26
pabelangerjpmaxman: looking at etherpad now16:26
*** rwsu has joined #openstack-infra16:27
anteayamorning zaro16:27
*** gouthamr has joined #openstack-infra16:27
jeblairjpmaxman, pabelanger: i think that might have been meant fro me16:27
*** matt-borland has joined #openstack-infra16:27
fungijpmaxman: i too prefer the ssh key being added by the person claiming ownership of that key, from a standpoint of making sure we're adding the right key and tat the person proposing the change agrees about our access requirements16:27
jpmaxman"We are definitely okay with granting ssh access to production systems to folks with specialization in that area.  There are several precedents.16:27
jpmaxmanHowever, one of the things we consider most important is that operation of our systems is principally via manipulation of git repos with direct access and modification to be exceptions to the rules.  It would be good to see some history of collaboration in git first."16:27
pabelangerjpmaxman: jeblair: yes, tabbing fail16:27
EmilienMfungi: we should all move to your island16:27
EmilienMfungi: do you have place?16:28
jeblairjpmaxman: yes, it seems we have decided to waive that on the promise of future collaboration. :)16:28
jpmaxmanto address your concern - I don't have a lot of commits but have helped the foundation with their website and infrastructure for a number of years :)   Anyway, I'm happy to provide my own patch16:28
fungiEmilienM: i can probably fit 8 guests comfortably. everyone else will have to sleep on the dock16:28
*** daemontool has quit IRC16:28
jeblairjpmaxman: thanks! :)16:28
anteayajpmaxman: it is one small mechanism we use to ensure someone is acting with their own awareness and consent16:29
*** daemontool has joined #openstack-infra16:29
anteayajpmaxman: with thousands of developers we don't always know who has given permission for what actions16:29
AJaegerscottda: if it does not show up, then I expect nobody sets it ...16:29
*** njohnston|afk is now known as njohnston16:30
zxiiroelectrofelix: taking a look16:30
*** Sukhdev has joined #openstack-infra16:31
jpmaxmanI'm also happy to not do anything - I only really jumped i here because tom fifield seemed frustrated and having enjoyed some frosty cold adult beverages with him thought I'd help16:31
EmilienMfungi: coming16:31
jpmaxmananteaya it's understandable, I hadn't met Jim until very briefly in Austin.  Many of you I've enjoyed drinks with at many an openstack event :)16:32
pleia2fungi: re: gerritbot, there is a patch up here to add some error handling, I'm not sure if it adds enough logging for us, but it may be good place to start WRT our channel joining mystery https://review.openstack.org/#/c/287056/16:33
anteayajpmaxman: oh sure, and it is always a pleasure to meet in person :)16:33
anteayajpmaxman: one thing we have to be aware of in every action we take is that our decisons are available for evaulation by all our peers, including our peers who weren't there for drinks16:33
*** johnny_ has joined #openstack-infra16:34
pleia2fungi: I think the standard chanlimit is 120 as you mention, but stopping at 240 seems strangely suspicious as a number that may be a "bigger chanlimit that we have an exception to get"16:34
fungijpmaxman: i personally very much appreciate that you're willing to help us get this under control and more manageable, thanks! i mostly just want to make sure we don't sacrifice longer-term manageability in the process, which is part of why protracted discussions and more detailed planning become necessary16:34
anteayajpmaxman: which is why things like clear descriptive commit messages may seem pedantic until you have had to find a commit from 6 months prior and the only commit message is remove foo, without any motivation for the change16:34
*** gyee has joined #openstack-infra16:35
*** jlanoux has quit IRC16:36
*** yamamoto has joined #openstack-infra16:36
jpmaxman@fungi of course.  it is a unique situation because the production server is under active attack.  So fixes have to be tested in production to know if they work16:36
*** matt-borland_ has joined #openstack-infra16:36
pabelangerfungi: pleia2: do you mind helping revert 329536. Our wheel-builds are currently failing16:37
jeblairmtreinish: rsync: send_files failed to open \"/home/jenkins/workspace/gate-puppet-openstack_health-puppet-beaker-rspec-ubuntu-trusty/logs/kern_log.txt\": Permission denied (13)16:37
pabelangerworking on a patch, but talking longer then expected16:37
jpmaxmanso sometimes you have to be able to suspend normal operating procedures in order to re-enforce your processes16:37
anteayajpmaxman: as a note, you don't need to use @ in front of someone's name in irc, that is just for twitter (and perhaps hipchat?)16:37
fungipleia2: if we can get some clearer debugging from gerribot to confirm that hypothesis, there are ways we can work around it (for example, since it's an informational bot we could shard by channel)16:37
pleia2fungi: nods16:37
pleia2fungi: care to look at the debugging patch?16:37
anteayajpmaxman: well actually suspending normal operating procedures causes far more problems than it solves16:38
mtreinishjeblair: heh, I guess the kernel log is owner read only16:38
pabelangerjeblair: Yup, that is the rsync issue with misconfigured permissions for logs16:38
*** matt-borland has quit IRC16:38
*** sergmelikyan has joined #openstack-infra16:38
sergmelikyanjeblair, any chance you know if it's possible to use GitHub Pages with our repos given that they are mirrored to the GitHub?16:38
anteayajpmaxman: with as many different services running as we have and as many different people in different timezones that we do, follow our process is often the only thing we can depend on16:38
anteayajpmaxman: we always have some service in crisis of one form or another16:39
mordredsergmelikyan: I do not believe it is a thing we desire to support16:39
*** sergmelikyan has quit IRC16:39
*** tongli has quit IRC16:39
mordredsergmelikyan: we have the ability to publish web content to locations, what sort of content are you looking at?16:40
fungipleia2: taking a look, thanks for the pointer16:40
*** eezhova has quit IRC16:40
jeblairpabelanger: i didn't really intend to have the conversation in the etherpad...16:41
jeblairpabelanger: so -- console-log last:  i think it should be last so that when we use it to upload the ansible logs, we get as much as possible16:41
jpmaxman@anteaya sure - but you also have to be able to assign priority and set aside process to fix something that is urgent.  It's kind of the age old adage that while we were following the process the village was pillaged.16:41
pabelangerjeblair: sure, removed16:42
anteayajpmaxman: umm, it is been a while since I've worked on a pillage scenario so I might be out of my element on this one16:42
pabelangerjeblair: yes to last16:42
jeblairpabelanger: okay, what else we got?16:42
anteayajpmaxman: in any case, it is nice to have your help here, and I'm willing to help you learn about our workflow if that is useful to you16:42
*** daemontool has quit IRC16:43
*** tesseract has quit IRC16:43
jpmaxmananteaya: :D16:43
*** e0ne has joined #openstack-infra16:43
*** markusry has joined #openstack-infra16:43
jpmaxmanthank you! :)16:43
anteayayou're welcome16:44
anteayajpmaxman: as we get to learn how to work with each other, please ask any question even if you think it is silly, it probably isn't16:44
*** e0ne has quit IRC16:44
*** brucet has quit IRC16:44
anteayajpmaxman: and I'll do my best to keep an eye out for supportive suggestions, which you can accept or discard at your discresion16:45
anteayaalso I'm a horrible speller16:45
pabelangerjeblair: to upload or in general?16:45
*** sergmelikyan has joined #openstack-infra16:45
jeblairpabelanger: was that the only issue with the plan in the etherpad?16:45
fungipleia2: that's a great patch. i'm noticing we also need to update it for newer pbr and get rid of teh ancient <1.0 cap, but hopefully that can be done trivially in a separate change16:46
sergmelikyanmordred: sorry got disconnected, I was looking for hosting murano.io16:46
pleia2fungi: yeah, glad I fished it out of oblivion while I was catching up on my queue the other week16:47
pabelangerjeblair: I think so.  It should be a good start to get zuul-launcher logs into logs.o.o16:47
anteayasergmelikyan: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-14.log.html#t2016-06-14T16:40:1816:47
*** tkelsey has quit IRC16:47
sergmelikyanmurano.io hostname is used in murano in the same way org.java.* is used in Java16:47
jeblairclarkb: do you have any feedback?16:47
pleia2sergmelikyan: I think mordred was asking what kind of content (published docs, html pages, etc) you were looking to publish, not the domain :)16:48
clarkbjeblair: I don't know what is being discussed or what ehterpad is16:48
jeblairclarkb: https://etherpad.openstack.org/p/p25pzG6OSr16:48
sergmelikyanah it's just 1 html page16:48
mordredpleia2: ++16:48
mordredsergmelikyan: yah - so, publishing an html is definitely a thing that should be done via normal publishing mechanisms here16:49
mordredhowever, we've resisted per-project websites thusfar16:49
fungipleia2: it would be more reviewable as about half a dozen separate changes, but i think we can probably merge it. i'd like to actually give it a try on review.o.o first and see if it gets us any better debugging info. can take it for a spin right now16:49
*** salv-orlando has quit IRC16:49
mordredso it is also important to determine if we'd like to have such a thing - I think it's a worthy question for the TC16:49
*** salv-orlando has joined #openstack-infra16:50
mordredwhich I know may sound like it's a lot of effort for one html page - but nothing in openstack is ever just one html page once you multiply by the number of projects ;)16:50
sergmelikyanUnderstand that :)16:50
fungisergmelikyan: in fact, we've condensed a lot of old per-project vanity domains in recent years, turning them into deprecated redirects to official documentation sites16:50
*** sputnik13 has joined #openstack-infra16:50
openstackgerritPaul Belanger proposed openstack-infra/project-config: Fix wheel-build macro issue  https://review.openstack.org/32953616:51
fungie.g., http://nova.openstack.org/ and http://devstack.org/16:51
pabelangerfungi: dhellmann: AJaeger: ^ updated the revert with the actual fix16:51
clarkbjeblair: pabelanger the set of steps on the etherpad seems reasonable to me16:51
sergmelikyanwe are not looking to host documentation or anything significant there, we need some simple landing page16:51
clarkbbasically copy all the things and fail gracefully but still record and note failure16:51
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix Multiple SCMs Plugin child XML format  https://review.openstack.org/31490016:51
sergmelikyanidea is that we use this domain in the app definition language, same way as in Java org.java.* used16:52
*** Daisy has joined #openstack-infra16:52
sergmelikyanand we don't want to loose domain16:52
jeblairpabelanger: do you want to update your changes to do that?16:52
sergmelikyanand it make sense to have some simple landing page there instead of 40416:52
pabelangerjeblair: sure16:52
dhellmannpabelanger : thanks16:52
fungisergmelikyan: oh, you're needing to register an application namespace? that's a somewhat different use case than typical vanity domain splash pages16:53
*** hdaniel has quit IRC16:53
*** jordanP has quit IRC16:53
openstackgerritKhai Do proposed openstack-infra/system-config: run git gc instead of repack for git repos  https://review.openstack.org/32956616:53
*** sshnaidm has joined #openstack-infra16:54
notmorganclarkb: i'll have the log capture thing respun shortly. thanks for the comments yesterday16:54
sergmelikyanfungi: exactly, right now (and previous 3 years) it was hosted on my private hosting and domain is controlled by Mirantis, we want to move that to the community16:54
fungisergmelikyan: so why not appname.murano.openstack.org or something?16:54
fungisergmelikyan: legacy pre-openstack namespace i guess?16:54
sergmelikyanfungi, too long, https://github.com/openstack/murano/blob/master/meta/io.murano/Classes/resources/Instance.yaml#L1316:55
sergmelikyanexample ^16:55
sergmelikyanfungi: I would not say that it's exactly legacy, io.murano seems right and short name to use. Previously it was hosted this way because of exactly pre-openstack namespace rules16:56
fungithat doesn't seem to have stopped java and go communities from using insanely long package namespaces16:56
*** Daisy has quit IRC16:56
sergmelikyanfungi: that's why we need to have some short start like io.murano.* because people will end up with something like: io.murano.networking.neutron.loadbalancing16:57
sergmelikyanfungi: I am not looking for namespace for specific app16:57
sergmelikyanI am talking about base namespace for internal murano classes16:58
sergmelikyanused by other apps16:58
fungisergmelikyan: so anyway, the foundation does continue to maintain domains used by projects that later officially joined openstack, so this seems like a pretty similar request16:59
*** acabot has quit IRC16:59
fungie.g. devstack.org, refstack.org, trystack.org, et cetera16:59
sergmelikyanfungi: so should I add agenda item for TC meeting to approve moving murano.io to openstack-infra?17:00
openstackgerritMorgan Fainberg proposed openstack-infra/zuul: Capture Logs when Timeout is hit  https://review.openstack.org/32851617:00
notmorganmordred, jeblair, pabelanger, clarkb: ^17:01
*** trown is now known as trown|lunch17:01
fungisergmelikyan: no, if you simply want the foundation to take over ongoing registration of the murano.io domain i _think_ we can just do that. if jbryce is around hopefully he can confirm that's reasonable given past precedent for similar domains17:02
openstackgerritNate Johnston proposed openstack-infra/system-config: Add #openstack-fwaas to accessbot channels  https://review.openstack.org/32957017:02
openstackgerritKhai Do proposed openstack-infra/system-config: run git gc instead of repack for git repos  https://review.openstack.org/32956617:02
fungisergmelikyan: that's more just an openstack foundation decision and not really infra team/tc17:03
rcarrillocruzcan you try now17:03
rcarrillocruzi think the issue is solved17:03
sergmelikyanfungi: thank you for the help!17:03
*** acabot has joined #openstack-infra17:03
*** kushal has quit IRC17:04
fungisergmelikyan: as to whether there is web content which should be published for that url and whether it should be a redirect to a more standard openstack site, that's something worth discussing with us and the tc17:04
sergmelikyanfungi: I believe there is no specific requirements from the Murano team side regarding content. I think generic openstack site would be much better17:04
fungibut taking over the domain registration should be pretty straightforward17:05
pabelangerrcarrillocruz: sure17:05
pabelangerrcarrillocruz: ++17:05
sergmelikyanSummary: I need confirmation from TC about taking over murano.io, and than discuss with TC what we should show on that domain.17:06
*** PollyZ_ has quit IRC17:06
sergmelikyanI will write e-mail to jbryce, is any better way to contact him? or should I use foundation@openstack.org17:07
rcarrillocruzk, let me know if there are further issues17:07
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add ignore-projects option  https://review.openstack.org/29347217:07
fungisergmelikyan: i'm trying to raise him in some other places now17:07
*** thorst_afk is now known as thorst17:07
rcarrillocruzthe gw was blocking outbound access as I expected17:07
rcarrillocruzbtw, i asked the guy about the shutdown of the current datacenter and tells me it is expected to happen end of next month17:08
rcarrillocruzbut anyway, i'll pass along info when i hear something official17:08
clarkbrcarrillocruz: did all hosts get fixed or just the one?17:08
jbrycesergmelikyan: jonathan@openstack.org17:09
*** sergmelikyan has quit IRC17:10
*** sergmelikyan has joined #openstack-infra17:10
*** rossella_s has quit IRC17:11
rcarrillocruzthe fix has been put on the vlan17:11
*** PollyZ has joined #openstack-infra17:11
*** e0ne has joined #openstack-infra17:12
*** ramishra has joined #openstack-infra17:12
*** ramishra has quit IRC17:12
*** ramishra has joined #openstack-infra17:13
*** ramishra has quit IRC17:13
*** markus__ has joined #openstack-infra17:15
*** ddieterly is now known as ddieterly[away]17:15
jeblairpabelanger: aside from logs, what things from ansible are we interested in uploading?  just the playbooks?  or also the inventory and config files?17:15
jeblairpabelanger: and what about the scripts?  (the shell snippets that we upload and run with zuul_runner)17:16
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Remove unused plugins dir  https://review.openstack.org/32958117:16
*** PollyZ has quit IRC17:16
rcarrillocruzbut i can't check on other hosts, unless i reboot them via ilo and get into recovery mode. The other hosts have old IP configs on their NICs, thus are not reachable17:16
*** syjulian has joined #openstack-infra17:16
*** markusry has quit IRC17:16
clarkbrcarrillocruz: at least we know what the problem is if we run into it again17:17
* rcarrillocruz goes back to grunt work on checking statuses of different Jira issues17:17
*** pradk has joined #openstack-infra17:17
*** ramishra has joined #openstack-infra17:17
rcarrillocruzclarkb: yup, cos the NET guy i've been working to sort this out was like 'omg, you are really really really opening these machines to the outside bad Internet'17:17
rcarrillocruzso if it happens again , i can point to a ticket17:18
clarkbjlk: you around? I remember yo umentioned we would need to rebuild out the bluebox cloud config, do you know if the cloud credentials changed too? wondering how some of us can get started on that work17:18
*** qb_ has quit IRC17:18
rcarrillocruzi hope there's no security guy capping that at some point in future17:18
openstackgerritPaul Belanger proposed openstack-infra/zuul: Upload ansible playbook directory to logs server  https://review.openstack.org/32560217:19
*** ramishra has quit IRC17:19
pabelangerjeblair: clarkb: ^ Want to confirm that is what you expected?17:19
openstackgerritPaul Belanger proposed openstack-infra/zuul: Upload ansible playbook directory to logs server  https://review.openstack.org/32560217:20
*** dtantsur is now known as dtantsur|afk17:21
pabelangerjeblair: we still need to add the ignore_failure logic17:21
*** qb_ has joined #openstack-infra17:21
jeblairpabelanger: let's do that in its own change17:21
jeblairpabelanger: you'r chatting in the etherpad again17:22
*** alex_xu has quit IRC17:22
pabelangerjeblair: I think we need to change line 15 in etherpad, since it contains plain text passords17:23
jlkclarkb: the credentials might have changed, true. I think mordred has access to the Blue Box support system where credentials are stored.17:23
jeblairpabelanger: i had some questions in here about what you want to upload17:23
jeblairpabelanger: aside from logs, what things from ansible are we interested in uploading?  just the playbooks?  or also the inventory and config files?17:23
jeblairpabelanger: and what about the scripts?  (the shell snippets that we upload and run with zuul_runner)17:23
jlkIt'd be reduced down to just cloud_admin user account to start with.17:23
clarkbmordred: ^ maybe you want to bootstrap things for us?17:23
pabelangerjeblair: at first glance, if we do the whole ansible folder, however I have some concerns about the script sub directory17:24
jeblairpabelanger: so that's *yes* to inventory and ansible.cfg files?17:25
*** alex_xu has joined #openstack-infra17:25
pabelangerjeblair: yes, inverentory, both playbooks17:25
pabelangerI am not sure what is in plugins currently17:25
jeblairpabelanger: it's empty, i just pushed a change to delete it17:26
*** salv-orlando has quit IRC17:26
jeblairpabelanger: okay, what credential information could end up it scripts?17:26
*** yamamoto has quit IRC17:26
*** salv-orlando has joined #openstack-infra17:26
jeblairpabelanger: scripts just holds the shell snippets from jjb17:26
pabelangerjeblair: don't we store our FTP credentials there?17:26
jeblairpabelanger: no, we write those into the ftpdir that we create when we create an ftp task17:26
jeblairpabelanger: that *is* under ansible_root currently17:26
jeblairpabelanger: so based on your earlier answers, i think we can do the following:17:27
pabelangerYa, my patch above uploads the whole directory17:27
jeblairpabelanger: move logs under ansible/, and move ftp/scp staging directories *out* of ansible/17:27
jeblairpabelanger: then we can upload the whole ansible dir17:27
jeblairpabelanger: so it will be like the 'future layout' section at the bottom of the etherpad17:28
*** tphummel has joined #openstack-infra17:28
*** hongbin has quit IRC17:29
pabelangerjeblair: yes, I like that17:29
*** salv-orl_ has joined #openstack-infra17:29
jeblairpabelanger: 602 wasn't really what i had in mind -- that will make the log upload task be the first task on the first scp publisher, which may or may not be the correct target... i think i see how this should all fit together now; do you mind if i push up another revision of that patch?17:30
pabelangerjeblair: please do17:30
jeblairpabelanger: cool, i'll have something in a few minutes i think17:30
openstackgerritLuigi Toscano proposed openstack-infra/project-config: Enable Tempest test job also for sahara-tests  https://review.openstack.org/32959017:30
notmorganjlk: yay cloud_admin pattern ;)17:31
jlknot the way you'd like to see it17:31
jlksadly I think we were using the "cloud_admin" thing without looking at what Keystone was doing :(17:31
jlkfor us "cloud_admin" means "the customer account who can do (some) admin things on this cloud". Not all admin things though, that's left to just the "admin" account which we don't hand out.17:32
*** salv-orlando has quit IRC17:32
*** tqtran has joined #openstack-infra17:32
pabelangerjeblair: when we abort a job, exit code -9, should be continue onto post_playbook to upload logs files?17:36
*** mriedem1 has joined #openstack-infra17:38
*** PollyZ has joined #openstack-infra17:39
*** zz_ja is now known as zz_zz_ja17:39
jeblairpabelanger: a zuul abort should probably not do that.  a timeout should.17:40
*** mriedem has quit IRC17:40
*** eezhova has joined #openstack-infra17:40
pabelangerjeblair: I agree, let me get a patch17:40
*** e0ne has quit IRC17:41
*** tonytan4ever has joined #openstack-infra17:41
*** electrofelix has quit IRC17:41
pabelangerrcarrillocruz: incoming PM17:41
*** dizquierdo has quit IRC17:41
*** mriedem1 is now known as mriedem17:42
*** sambetts is now known as sambetts|afk17:42
*** _nadya_ has joined #openstack-infra17:42
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Ceilometer functional tests in Identity v3-only  https://review.openstack.org/31507717:42
*** roxanagh_ has joined #openstack-infra17:43
*** PollyZ has quit IRC17:43
*** tkelsey has joined #openstack-infra17:43
*** asselin_ has quit IRC17:44
*** ociuhandu has quit IRC17:45
clarkbjeblair: pabelanger fwiw jenkins aborts did result in logs but that was probable less desireable.17:45
clarkbsince an abort indicates an invalid state we shouldn't care about storing and indexing that info17:45
*** ihrachys has quit IRC17:46
AJaegerpabelanger fixed the wheel build with https://review.openstack.org/#/c/329536/2 - I'd like to have another project-config review on this one to merge quickly since wheel build is currently broken.17:47
*** roxanagh_ has quit IRC17:47
*** tkelsey has quit IRC17:47
*** jesusaur has quit IRC17:48
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Remove 'partial' support for Grenade jobs  https://review.openstack.org/32953217:48
*** jesusaur has joined #openstack-infra17:49
*** pvaneck has joined #openstack-infra17:49
jeblairclarkb: ++17:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute004.uswest metadata for better tracking  https://review.openstack.org/32960217:50
*** mtanino has quit IRC17:51
*** bpokorny has quit IRC17:51
*** tosky has quit IRC17:54
*** mhickey has joined #openstack-infra17:54
pabelangerclarkb: jeblair: Ya, not sure it is a bad thing. If we want to make the improvement, I can submit the patch17:57
*** Rockyg has joined #openstack-infra17:58
*** ajmiller_ has joined #openstack-infra17:58
*** ajmiller has quit IRC18:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Remove unused plugins dir  https://review.openstack.org/32958118:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Upload ansible playbook directory to logs server  https://review.openstack.org/32560218:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add log_path to ansible.cfg  https://review.openstack.org/32952518:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: move ftp/scp staging out of ansible dir  https://review.openstack.org/32960718:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: ensure that console/ansible publishing is last  https://review.openstack.org/32960818:01
jeblairpabelanger: yeah, i think we should improve it18:02
*** sergmelikyan has quit IRC18:02
jeblairpabelanger, clarkb, Shrews: ^ that's everything except the ignore_errors change.  it's all untested, but wanted to push that up for early feedback.  i'm going to write the ignore errors change, then test the whole stack.18:02
pabelangerjeblair: looking18:03
*** nighttime has quit IRC18:04
*** kushal has joined #openstack-infra18:05
*** robot has joined #openstack-infra18:05
*** bpokorny has joined #openstack-infra18:05
*** bhavik has joined #openstack-infra18:06
*** jamesmcarthur has quit IRC18:07
*** _nadya_ has quit IRC18:07
*** ddieterly[away] is now known as ddieterly18:08
*** mtanino has joined #openstack-infra18:08
*** dmk0202 has joined #openstack-infra18:09
Shrewsjeblair: if EVERY task in the post-playbook will have ignore_errors=true, then that block:always: wouldn't be necessary18:09
Shrewsjeblair: just the final task to check rc18:09
*** acoles is now known as acoles_18:10
*** salv-orl_ has quit IRC18:10
jeblairShrews: so skip the block construct, just have a list of tasks, all with ignore_errors, then at the end, one final task to check the registered rcs?18:11
*** salv-orlando has joined #openstack-infra18:11
clarkbjeblair: I notice that zl03 has a much larger number of used nodes than the other two zls, is that going o largely be driven by gearman response times?18:11
Shrewsjeblair: aye18:11
*** trown|lunch is now known as trown18:11
*** salv-orlando has quit IRC18:11
*** salv-orlando has joined #openstack-infra18:12
clarkbzl03 has roghly 2x the nodes as 01 and 02 combined18:12
jeblairclarkb: yes, it's the usual imperfect method of gearman load distribution  (if the server is busy, it's slower, right?)18:12
mordredclarkb, jlk: I hope to get the bluebox cloud re-bootstrapped tomorrow or thursday18:12
pabelangerrcarrillocruz: crinkle: clarkb:  ansible is now connected to baremetal00.vanilla.ic.openstack.org.  puppet currently failing with exit code 618:13
jeblairclarkb: maybe we should do "sleep(number_of_nodes)" before grabbing a job :)18:13
pabelangermordred: excellent18:13
pabelangerclarkb: I've noticed that too18:13
jeblairclarkb, pabelanger: before we jump to that though, we should see whether it's a problem18:14
jeblairlike, are we hitting resource limits18:14
clarkbya I don't actualy know if it is a problem18:14
*** eezhova has quit IRC18:14
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: ensure every publisher runs  https://review.openstack.org/32961218:14
crinklepabelanger: WARNING: The following packages cannot be authenticated!  libevent-2.0-5 libunbound2 unbound-anchor unbound18:14
pabelangerwhat about shutting down jenkins03 to see how zuul-launcher handle more nodes in the pool?18:15
rcarrillocruzpabelanger: sweet18:15
pabelangercrinkle: Hmm18:15
*** Sukhdev has quit IRC18:15
rcarrillocruzopening jiras for different issues i'm encointering in rack12 servers18:15
jeblairpabelanger: i think we should wait until we publish ansible logs so that people can diagnose POST_FAILURE conditions before we do that18:15
jeblairclarkb, Shrews, pabelanger: ignore_errors change: https://review.openstack.org/32961218:15
pabelangerjeblair: agreed.18:16
jeblairoops, one sec on 61218:16
pabelangerreally looking forward to an ansible-playbook test fixture in zuul!  Will make parsing dynamic playbooks way easier!18:17
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: ensure every publisher runs  https://review.openstack.org/32961218:17
jeblairpabelanger: i don't understand the last thing you just said18:17
Rockygmordred, you around?  What's the status of phabricator?  I'm in nova-bugs meeting and enquiring minds want to know18:19
*** sarob has joined #openstack-infra18:19
Rockygmordred, sorry, meeting-418:20
*** sarob has quit IRC18:20
Shrewsjeblair: lgtm. i'm excited to see if it works! :)18:20
mordredRockyg: https://review.openstack.org/#/c/314185/18:20
clarkbRockyg: I believe there was a recent ml update. Also fungi would like you to know that your lp account was possibly compromised and you should address that18:20
clarkbRockyg: it was part of the wiki spam attack18:20
clarkbaha mordred has the link18:20
*** PollyZ has joined #openstack-infra18:21
*** jamesmcarthur has joined #openstack-infra18:22
pabelangerjeblair: nothing major, hard to parse our generated playbooks from python code.  Was hoping one day we'd do something like JJB / grafyaml to compare the dynamic playbook to a know fixture.18:22
*** eezhova has joined #openstack-infra18:23
*** mhickey has quit IRC18:23
*** mhickey has joined #openstack-infra18:24
openstackgerritPaul Carver proposed openstack-infra/irc-meetings: Correction to meeting ID for Common Flow Classifier meeting  https://review.openstack.org/32961618:24
pabelangerjeblair: +2 for stack18:26
Rockygclarkb:  dang!  or maybe it was that I had found more spam pages and couldn't delete them so I moved them somewhere less obtrusive/irritating?18:27
Rockygclarkb, and thanks.  I guess I go and change my pw18:28
RockygIs tha all I need to do?18:28
anteayaRockyg: did you move spam pages?18:28
clarkbRockyg: fungi will have more info, I think he found spam pages iwth your name on them18:28
anteayaspam pages showed up in your account on June 8th18:28
fungiRockyg: there were 5 pages in total which claimed to have been created by the Rocky account on the wiki18:28
fungii'll link one in a sec18:28
RockygYeah.  They were in the middle of stuff some user groups would search so I moved them.  Sounds like the ones I moved.  I would have loved to have blown them away, but couldn18:29
anteayaRockyg: why couldn't you?18:29
RockygFor the future, what is the best wway to address?  Didn't have permission to delete.18:29
*** baoli_ has quit IRC18:29
*** dmk0202 has quit IRC18:30
fungiRockyg: https://wiki.openstack.org/w/index.php?title=JJJJ@_1866-532-0850_Turbotax_Customer_care_Technical_Support_Help_Phone_Number&action=edit&redlink=1 was one18:30
*** hasharOut has quit IRC18:30
*** harlowja has quit IRC18:30
RockygWhich is actually a good thing for most peopl.e18:30
*** esikachev has quit IRC18:30
fungiit does say something like "Rockyg moved page [spammy title] to Obsolete:[spammy title] (spam)" so that matches what you're saying18:31
fungii think that means your account is not compromised ;)18:31
fungii'll unblock it immediately18:31
anteayawhich is a relief18:31
RockygYup.  I could move the page to obsolete, so I did the top page of the ones that turned up in the search.18:31
fungiRockyg: i've unblocked it now. if you want to help patrol the wiki for spam and delete old material, i'm happy to add you to the wiki admins group there18:32
*** harlowja has joined #openstack-infra18:32
RockygThanks you!  And sorry for the scare.  I hate those f*&% spammers for grafittiing all over our wiki18:32
mtreinishjeblair: hmm, it failed on the same thing running on jenkins: http://logs.openstack.org/67/328467/1/check/gate-puppet-openstack_health-puppet-beaker-rspec-ubuntu-trusty/6da3f67/console.html#_2016-06-14_17_23_00_36018:33
Rockygfungi, sure.  Just need a delete page option, and a process so I don't scare the infra team when I do it.  Like a tag, or whatever.18:34
mordredRockyg: you will never scare us18:35
*** e0ne has joined #openstack-infra18:35
fungiRockyg: deletion is reversible18:35
mordredRockyg: well, I mean, except when you do18:35
pabelangermtreinish: that is going to be a bug with EmilienM and copy_logs.sh. I think we need to add logic to chown / chmod the logs folder to jenkins:jenkins18:35
RockygBwa ha ha ha ha!18:35
anteayaRockyg: my take on it is that there are a whole lot of people with not enough money to live on, so this is what they can do for pay18:35
mtreinishpabelanger: ah, ok18:35
anteayaRockyg: I don't fault the people, trying to earn a living, I fault a system that rewards them for doing this rather than useful work18:36
*** xenogear has joined #openstack-infra18:36
Rockyganteaya, yup.  Sad.  Especially since they seem to have at least rudimentory (or if they are writing the scripts, good) computer skills.18:37
pabelangercrinkle: clarkb: rcarrillocruz: puppet looks to have run correctly this time on baremetal00.vanilla18:37
fungiRockyg: your wiki account is unblocked, and you're now in the administrator and bureaucrat groups in the wiki like our other caretaker volunteers. thanks for helping!18:38
anteayaRockyg: agreed, they could be doing so much more, if only they could feed themselves when doing it18:42
*** hongbin has joined #openstack-infra18:45
*** salv-orlando has quit IRC18:45
RockygCool.  Thanks!18:45
openstackgerritgreghaynes proposed openstack-infra/jeepyb: Allow setting a default org in create_cgitrepos  https://review.openstack.org/32962618:46
*** sdague has quit IRC18:46
*** salv-orlando has joined #openstack-infra18:46
*** lifeless has quit IRC18:46
RockygI will wield my new power conservatively in the pursit of community.  As opposed to like a bureaucrat.18:46
*** lifeless has joined #openstack-infra18:47
*** qb_ has quit IRC18:48
*** qb_ has joined #openstack-infra18:49
Shrewsjeblair: processed your stack18:49
*** sarob has joined #openstack-infra18:50
*** e0ne has quit IRC18:53
*** mrmartin has joined #openstack-infra18:53
*** syjulian has quit IRC18:54
*** syjulian has joined #openstack-infra18:54
openstackgerritPaul Belanger proposed openstack-infra/zuul: Don't run post_playbook if ansible-playbook is aborted  https://review.openstack.org/32963118:55
*** ddieterly is now known as ddieterly[away]18:55
pabelangerjeblair: clarkb: I believe that will stop our post_playbook from running on abort.'18:55
*** eezhova has quit IRC18:56
syjulianhi guys, I'm trying to update a wiki page to grab some tasks, but for some reason I'm getting 500 errors when I log in with my Ubuntu One account. Anybody know what to do?18:56
*** eezhova has joined #openstack-infra18:56
anteayaI'm not a guy18:56
pabelangersyjulian: this is likely because you haven't logged into the wiki before18:56
anteayaand we had to disable new account creation on the wiki due to spamming18:57
pabelangersyjulian: if you give me a minute I can allow you to register for a 15mins window18:57
*** amoralej is now known as amoralej|off18:57
syjulianyes that would be great18:57
pabelangersyjulian: please try again18:57
*** sarob has quit IRC18:58
syjulianpabelanger: it works. thank you!18:59
pabelangersyjulian: np18:59
*** eezhova has quit IRC18:59
pleia2meeting time19:00
*** eezhova has joined #openstack-infra19:01
openstackgerritMerged openstack-infra/infra-specs: Mark 'Use Diskimage Builder in Nodepool' finished  https://review.openstack.org/32908019:01
pabelanger^ \o/19:01
*** PollyZ_ has joined #openstack-infra19:02
*** PollyZ has quit IRC19:05
*** dmk0202 has joined #openstack-infra19:05
*** bpokorny has quit IRC19:05
*** sdake has quit IRC19:06
*** sdake has joined #openstack-infra19:07
*** PollyZ has joined #openstack-infra19:07
*** PollyZ_ has quit IRC19:07
*** eezhova has quit IRC19:08
*** eezhova has joined #openstack-infra19:08
*** bpokorny has joined #openstack-infra19:09
*** jcoufal_ has quit IRC19:10
*** PollyZ has quit IRC19:14
openstackgerritJp Maxwell proposed openstack-infra/system-config: Adding ssh key for JP Maxwell to wiki.o.o  https://review.openstack.org/32963819:15
*** xarses_ has quit IRC19:16
rcarrillocruzclarkb: re: syslog on puppetmaster being interleaved, yup, i'll log to /var/log/ansible-puppet/<hostname> , being /var/log/ansible-puppet a role default, so it can tweaked19:16
*** markus__ has quit IRC19:17
*** ddieterly[away] is now known as ddieterly19:17
nibalizerpabelanger: awesome19:19
nibalizerre: puppet running correctly on baremetal0019:19
nibalizerexciting to have a cloud again19:19
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: translations: Fix error abort  https://review.openstack.org/32964219:19
*** PollyZ has joined #openstack-infra19:20
*** mrmartin has quit IRC19:20
*** kzaitsev_mb has quit IRC19:22
*** tonytan4ever has quit IRC19:25
*** dtardivel has quit IRC19:27
*** eezhova has quit IRC19:27
*** ociuhandu has joined #openstack-infra19:29
*** baoli has joined #openstack-infra19:31
*** mriedem has quit IRC19:31
*** sarob has joined #openstack-infra19:32
*** mriedem has joined #openstack-infra19:34
*** ayoung_ has joined #openstack-infra19:34
*** tkelsey has joined #openstack-infra19:35
*** gyee has quit IRC19:36
*** mrmartin has joined #openstack-infra19:37
*** eezhova has joined #openstack-infra19:39
*** tkelsey has quit IRC19:40
*** weshay is now known as weshay_mtg19:42
*** Sukhdev has joined #openstack-infra19:42
*** rfolco has quit IRC19:43
*** salv-orlando has quit IRC19:43
*** ajmiller_ has quit IRC19:44
*** salv-orlando has joined #openstack-infra19:44
*** eezhova has quit IRC19:44
*** dimtruck is now known as zz_dimtruck19:44
*** baoli has quit IRC19:44
*** roxanagh_ has joined #openstack-infra19:44
*** sarob has quit IRC19:45
*** baoli has joined #openstack-infra19:45
*** harlowja_ has joined #openstack-infra19:45
*** sarob has joined #openstack-infra19:45
*** derekh has quit IRC19:46
*** burgerk has quit IRC19:46
*** roxanagh_ has quit IRC19:49
*** harlowja has quit IRC19:49
*** jsavak has joined #openstack-infra19:50
*** cody-somerville has joined #openstack-infra19:50
*** cody-somerville has quit IRC19:50
*** cody-somerville has joined #openstack-infra19:50
*** rfolco has joined #openstack-infra19:50
jpmaxmanjeblair I put the patch here  - copied what Marton did.  https://review.openstack.org/32963819:51
jeblairjpmaxman: thanks!19:52
jpmaxmanit looks like Jenkins may not have liked it though in a couple of instances :(19:52
*** ffrank has joined #openstack-infra19:52
ttxjpmaxman: because in those tense situations we just get lucky :)19:52
ttx"Could not find puppet-4.5.2 in any of the sources"19:53
jeblairjpmaxman: yeah, a quick look at that suggests it's unrelated to your patch... we can probably track it down in a few minutes when the infra meeting wraps up19:53
jeblairttx: yeah, that19:53
pabelangerRight, we use the puppet modules beaker job, which doesn't use the OS version of puppet19:54
pabelangerit uninstalls it and them gem installs it19:54
jeblair4.5.2 was released today, so i wonder if this is rubygems.org being inconsistent19:55
jpmaxmanok cool - it's weird because it seemed pretty vanilla.  anyway let me know - happy to adjust19:55
*** sdague has joined #openstack-infra19:56
*** pilgrimstack has joined #openstack-infra19:56
jeblairjpmaxman: probably just bad timing; we can re-run the tests when we think rubygems.org is fixed19:57
nibalizeri didn't realize we'd moved the beaker jobs to 4.x19:57
jeblairTotal downloads 6,423,37319:57
jeblairFor this version 2419:57
nibalizerand yea... that is super strange19:57
*** ffrank has quit IRC19:58
jeblairthat's for version 4.5.219:58
jeblairso we might have tried to pull it literally the moment it went up19:58
*** e0ne has joined #openstack-infra19:58
*** rwsu_ has joined #openstack-infra19:58
jeblairi'm just going to recheck now and see if it's already cleared19:58
*** burgerk has joined #openstack-infra19:58
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Fix grenade exclusion for ironic  https://review.openstack.org/32965819:58
jrolleasy regex fix to save a few nodes ^19:58
nibalizerjroll: ++19:59
anteayajpmaxman: if you feel like attending infra weekly meetings, you may find them useful20:00
anteayajpmaxman: this week's just ended20:00
anteayajpmaxman: they take place in #openstack-meeting on Tuesdays at 19:00 utc20:00
*** rwsu has quit IRC20:00
*** jsavak has quit IRC20:03
jpmaxmancool I put it in my calendar20:03
*** eezhova has joined #openstack-infra20:03
jpmaxmanno wonder it was so quiet around here20:03
*** sarob has quit IRC20:04
anteayajpmaxman: great, it helps to get a feel for our flow20:04
fungiyeah, we were all focused in #openstack-meeting20:04
*** rwsu_ is now known as rwsu20:04
anteayajpmaxman: you never know when you learn something useful in a meeting20:04
fungiwe also tend to get pretty quiet for the hour after because we're either focused on the tc meeting or getting drunk. could be both20:04
jpmaxmanfungi +120:04
*** jsavak has joined #openstack-infra20:05
*** yamahata has quit IRC20:05
*** zz_dimtruck is now known as dimtruck20:06
*** ajmiller has joined #openstack-infra20:06
*** Goneri has quit IRC20:06
*** mrmartin has quit IRC20:09
*** flwang has quit IRC20:10
*** eezhova has quit IRC20:10
pabelangerclarkb: going back to the boot-timeout for osic-cloud1, did we want to try upping it to 240 (from 120)?20:10
clarkbpabelanger: might be worthwhile to at least confirm it isn't something silly like slow boots occasionally happening20:11
*** syjulian has left #openstack-infra20:12
*** pilgrimstack has quit IRC20:12
openstackgerritPaul Belanger proposed openstack-infra/project-config: Double our boot-timeout for osic-cloud1  https://review.openstack.org/32966220:13
pabelangerclarkb: ^20:13
pleia2off to lunch for a bit20:13
*** PollyZ has quit IRC20:14
*** kgiusti has quit IRC20:15
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: toci_gate_test: disable ipv6 on upgrade job for stable/liberty  https://review.openstack.org/32966320:16
anteayapleia2: enjoy lunch20:16
*** kzaitsev_mb has joined #openstack-infra20:17
*** xarses_ has joined #openstack-infra20:19
*** xarses_ has quit IRC20:24
*** xarses_ has joined #openstack-infra20:24
*** sarob has joined #openstack-infra20:24
*** flwang has joined #openstack-infra20:26
*** dmk0202 has quit IRC20:26
fungijeblair: 329638 lgtm, if you want to be the second +2 looks like it's all set for approval20:26
*** dmk0202 has joined #openstack-infra20:26
*** xarses_ has quit IRC20:27
*** xarses_ has joined #openstack-infra20:27
*** mriedem1 has joined #openstack-infra20:27
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: toci_gate_test: disable ipv6 on upgrade job for stable/liberty  https://review.openstack.org/32966320:27
pabelangerfungi: clarkb: do you mind reviewing: 329536 Our wheel-builds are currently broken20:28
*** sarob has quit IRC20:28
*** sarob has joined #openstack-infra20:28
*** e0ne has quit IRC20:29
*** PollyZ has joined #openstack-infra20:29
*** mriedem has quit IRC20:30
*** pradk has quit IRC20:31
*** rbrndt has quit IRC20:31
*** mriedem1 is now known as mriedem20:32
*** dmk0202 has quit IRC20:33
*** jsavak has quit IRC20:33
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: toci_gate_test: disable ipv6 on upgrade job for stable/liberty  https://review.openstack.org/32966320:33
*** jsavak has joined #openstack-infra20:33
*** cody-somerville has quit IRC20:35
*** gouthamr has quit IRC20:36
*** ddieterly is now known as ddieterly[away]20:37
openstackgerritMerged openstack-infra/system-config: Adding ssh key for JP Maxwell to wiki.o.o  https://review.openstack.org/32963820:38
prometheanfireianw: well, building now, thanks for the reminder, I didn't make a taks for that so it slipped my mind20:38
*** PollyZ has quit IRC20:40
*** ccamacho has quit IRC20:40
openstackgerritMerged openstack-infra/project-config: Fix wheel-build macro issue  https://review.openstack.org/32953620:42
*** rfolco has quit IRC20:42
*** mrmartin has joined #openstack-infra20:43
Shrewspabelanger: left you question on https://review.openstack.org/32963120:44
anteayajpmaxman: so now we wait for a puppet run to ensure your ssh key is deployed to the wiki server20:44
anteayawhat are we currently using as the puppet dashboard to track runs?20:44
*** roxanagh_ has joined #openstack-infra20:45
fungianteaya: hopefully still puppetboard.openstack.org but it's severely limited with the switch to ansible20:46
*** raildo is now known as raildo-afk20:47
*** PollyZ has joined #openstack-infra20:48
anteayaany objections to getting rid of the puppet-dashboard.openstack.org entry in cacti?20:48
pabelangerShrews: replied. we reason for leaving it20:48
anteayafungi: and thank you20:48
anteayajpmaxman: http://puppetboard.openstack.org/node/wiki.openstack.org20:48
fungianteaya: none from me. if it's still in the config, propose removal20:48
anteayaI will, thanks20:49
pabelangerShrews: basically, I've seen the abort request happen as we are uploading our publisher logs. Which is too late for the -9 to happen20:49
*** roxanagh_ has quit IRC20:50
*** _nadya_ has joined #openstack-infra20:51
anteayafungi: it isn't in system-config, I wonder how it is still displaying in cacti20:51
*** mrmartin has quit IRC20:52
fungianteaya: it may be that we don't have automated cleanup for things removed from cacti configs20:52
*** edmondsw has quit IRC20:52
*** yolanda has quit IRC20:52
Shrewspabelanger: ah, i see20:52
anteayaperhaps that is the thing20:53
pabelangerShrews: I think we can revisit that logic and optimize it better.  Sure jeblair has some ideas20:53
*** shardy has quit IRC20:53
*** thorst has quit IRC20:54
*** thorst has joined #openstack-infra20:55
Shrewspabelanger: these magic numbers concern me a bit. perhaps we should log the return code20:55
Shrewsoh, we do20:55
*** abregman is now known as abregman|ZzZzZz20:56
*** ayoung_ has quit IRC20:57
*** ayoung has quit IRC20:57
*** eezhova has joined #openstack-infra20:58
*** trown is now known as trown|outtypewww20:59
*** matt-borland has joined #openstack-infra20:59
*** thorst has quit IRC20:59
*** akshai has joined #openstack-infra20:59
*** _nadya_ has quit IRC20:59
jeblairThe conditional check 'task0.rc != 0 or task1.rc != 0 or task2.rc != 0 or task3.rc != 0 or task4.rc != 0' failed. The error was: error while evaluating conditional (task0.rc != 0 or task1.rc != 0 or task2.rc != 0 or task3.rc != 0 or task4.rc != 0): 'dict object' has no attribute 'rc'21:00
jeblairShrews, pabelanger: ^21:00
timothyb89are there any gerrit wizards around who could help setup initial permissions for karma-subunit-reporter?21:00
*** cody-somerville has joined #openstack-infra21:01
*** cody-somerville has joined #openstack-infra21:01
*** jd__ has quit IRC21:01
*** jd__ has joined #openstack-infra21:01
*** matt-borland_ has quit IRC21:01
*** eharney has quit IRC21:01
pabelangerjeblair: any more to the log?21:01
Shrewsjeblair: oh, hrm. rc might only be valid on shell commands21:01
Shrewswhich would stink21:01
pabelangerOh, hmm21:02
jeblairpabelanger, Shrews: http://paste.openstack.org/show/516063/21:02
Shrewsjeblair: ugh. yeah: http://docs.ansible.com/ansible/shell_module.html#return-values21:03
jeblairso probably synchronize or copy are the culprits21:03
*** csomerville has joined #openstack-infra21:03
*** darrenc is now known as darrenc_afk21:03
*** _ari_ has quit IRC21:04
pabelangermaybe checking stderr ?21:05
pabelangerbut, that might only be shell again21:05
fungitimothyb89: what was the change that added that repo?21:05
Shrewspabelanger: yep, only shell21:05
jeblairpabelanger, Shrews: for checking stderr -- we could look for the 'ignoring' message from ansible21:05
timothyb89fungi: https://review.openstack.org/#/c/327945/21:05
jeblair(i do not like log scraping as a solution, but if that's the best we have, that might work)21:06
*** e0ne has joined #openstack-infra21:06
jeblairi wonder if we should head over to #ansible for this one21:06
*** priteau has joined #openstack-infra21:06
*** gyee has joined #openstack-infra21:06
*** inc0 has quit IRC21:06
*** ddieterly[away] is now known as ddieterly21:06
Shrewsjeblair: probably21:06
*** cody-somerville has quit IRC21:07
fungiif you dangle "we have to parse your logs to solve this" in front of them, they might be compelled to point out far more palatable solutions21:07
*** psilvad has quit IRC21:07
*** aysyd has quit IRC21:08
*** gouthamr has joined #openstack-infra21:08
fungitimothyb89: done21:08
timothyb89fungi: great, thanks!21:09
*** abregman|ZzZzZz has quit IRC21:09
*** inc0 has joined #openstack-infra21:10
Shrewsjeblair: i'm stumped for ideas21:11
Shrewswe should as bcoca21:11
*** yamahata has joined #openstack-infra21:12
*** julim has quit IRC21:12
*** javeriak has joined #openstack-infra21:13
*** javeriak has quit IRC21:13
*** piet has quit IRC21:14
pabelangerjeblair: Shrews: same, unless we switch to using shell over synchronize. But not sure I like that21:15
jeblairi like log scraping better than that :(21:16
*** javeriak_ has quit IRC21:16
jeblair(and to reiterate, i do not like log scraping)21:16
jeblairi asked in #ansible; we'll see if that merits a response :)21:17
*** csomerville has quit IRC21:17
*** ldnunes has quit IRC21:18
*** csomerville has joined #openstack-infra21:18
*** woodster_ has quit IRC21:18
*** EricGonczer_ has quit IRC21:19
Shrewsjeblair: pabelanger: i have a feeling what we want is outside the norm and cannot be done w/o going all shell or scraping21:20
jeblairShrews: perhaps so, but i don't feel it's *that* crazy21:20
*** annegentle has quit IRC21:21
jeblair(it follows fairly naturally from the idea of idempotency)21:21
Shrewsjeblair: not *crazy*, just not often needed21:21
*** kzaitsev_mb has quit IRC21:24
*** sarob has quit IRC21:25
*** burgerk has quit IRC21:26
*** rhallisey has quit IRC21:26
*** kzaitsev_mb has joined #openstack-infra21:27
*** amitgandhinz has quit IRC21:27
*** amitgandhinz has joined #openstack-infra21:27
*** matt-borland has quit IRC21:28
*** eezhova has quit IRC21:30
jeblairShrews, pabelanger: perhaps the best thing to scan for would be "fatal:" at the start of a log line21:32
*** bpokorny has quit IRC21:32
openstackgerritMerged openstack-infra/project-config: Double our boot-timeout for osic-cloud1  https://review.openstack.org/32966221:33
*** eezhova has joined #openstack-infra21:34
*** inc0 has quit IRC21:34
*** elarson has joined #openstack-infra21:34
elarsonhey all, I've been getting disconnected from the list due to bounces and the folks managing my email could use some help figuring out why it is happening. is there a good contact I can provide them to help see what is happening?21:35
pabelangerjeblair: Shrews: quick test local: http://paste.openstack.org/show/516066/ I see failed = true for my register variable21:35
pabelangerjeblair: both for copy and synchronize21:35
*** dizquierdo has joined #openstack-infra21:35
pabelangerI'd have to test more to see if that world work21:36
jeblairpabelanger: i don't understand what you did21:36
Shrewspabelanger: with ignore_errors=true?21:36
jeblairpabelanger: can you post the playbook?21:37
*** haaj has joined #openstack-infra21:38
*** pradk has joined #openstack-infra21:39
fungielarson: i can look into it. you're not at rackspace by any chance? was just looking into a bunch of their mass-unsubscribe events with ttx21:39
*** bpokorny has joined #openstack-infra21:39
elarsonfungi: I am!21:39
elarsonfungi: there are a bunch of folks that are seeing the same thing. let me know how I can help!21:40
pabelangerShrews: jeblair: http://paste.openstack.org/show/516067/ was my example21:40
fungielarson: <fireeye@raxempsscanner.corp>: conversation with[] timed out while sending end of data -- message may be sent more than once21:40
pabelangerlooks like .failed is set on failure21:40
fungielarson: mailman is getting lots of bounces with that back from rackspace e-mail subscribers21:40
jeblairpabelanger, Shrews, clarkb: an alternative might be to use the regular behavior for most of the publishers inside of a block and then put the console log publisher (if present) at the end inside of an always21:40
*** thiagop has quit IRC21:41
*** eezhova has quit IRC21:41
fungielarson: i interpret that as a fireeye scanner at rackspace getting overloaded and the previous hop mta bouncing the message back rather than deferring/queuing21:41
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul: Turn off Ansible host fact gathering  https://review.openstack.org/32968821:41
dmsimardpleia2: you can merge https://review.openstack.org/#/c/321229/ whenever you'd like21:41
fungielarson: hopefully that's enough for your mailserver admins to go on?21:42
*** e0ne has quit IRC21:42
elarsonfungi: I'll update my internal ticket and see!21:42
fungielarson: ttx and i were just pondering who on that end to reach out to. much better if it comes from within21:42
jeblairfungi: oh, sorry, i somehow missed you and ttx working on this.  anything i can do to help?21:43
*** amitgandhinz has quit IRC21:43
Shrewspabelanger: if that exists on shell commands (don't see why it wouldn't), then we could use that in place of 'rc'21:43
*** bpokorny_ has joined #openstack-infra21:43
pabelanger  - debug: msg="{{ runner.failed|default(true) }}" could be the check21:44
*** bpokorny_ has quit IRC21:44
pabelangerif .failed is not setup, then default true21:44
fungijeblair: came up yesterday, some rackspace subscribers (adrian otto for one) had reached out to him and he was wondering if there were options to troubleshoot. i suggested he turn on the setting that sends unsub notices to the listadmin since it will attach the bounce21:45
*** eezhova has joined #openstack-infra21:45
pabelangererr default(false(21:45
*** bpokorny_ has joined #openstack-infra21:45
pabelangershell also setups .failed on failure21:45
fungijeblair: i don't think it was in channel, but we can move the conversation here when he's back at the computer (perhaps tomorrow)21:46
elarsonfungi: are there any logs of the bounces I can send along?21:46
*** Sukhdev has quit IRC21:46
pabelangerShrews: jeblair: updated test http://paste.openstack.org/show/516068/21:46
*** bpokorny has quit IRC21:47
fungielarson: ttx forwarded one to me that i can forward to you. what's the address?21:48
ianwprometheanfire: i'm not 100% sure what the status is, but yolanda had changes out to use ansible instead of puppet for the infra bits21:49
ianwif there's puppet issues, it might be worth just focusing on that21:49
*** stajkowski has joined #openstack-infra21:49
fungielarson: i just redirected a copy of what ttx sent me to you (Subject: Fwd: Bounce action notification)21:50
stajkowskilooking for some reviews for a new project https://review.openstack.org/#/c/31936821:50
prometheanfireianw: possibly, I think I'm about as good in either21:50
*** sileht has quit IRC21:51
ianwprometheanfire: https://review.openstack.org/#/c/283788/ was the review i was thinking of21:51
prometheanfirejust finished the cache-bindep portion I think on that image test21:51
Shrewspabelanger: jeblair: so i think 'failed' is only there on actual failure (https://github.com/ansible/ansible/blob/devel/lib/ansible/executor/task_executor.py#L115-L119) so what pabelanger proposed *should* work, i think21:51
elarsonfungi: got it, thanks!21:52
*** sileht has joined #openstack-infra21:52
jeblairShrews, pabelanger, clarkb: okay... should we try that, or should we consider the "exit on first failure except always run log upload" approach?21:52
*** sarob has joined #openstack-infra21:53
jeblairi think that will be sufficient for what we need -- it's just a matter of whether we think we also might benefit from partial uploads of other things (devstack logs... tarballs... etc)21:54
*** darrenc_afk is now known as darrenc21:54
Shrewsjeblair: honestly, if the only task we want to always execute is the log upload, that seems simpler21:54
clarkbpartial uploads of other things could be useful if the host has really ogtten into an unhappy place21:54
clarkbdstat logs are really important for example21:54
Shrewsjeblair: but isn't ansible-playbook exit code what we're really after here?21:55
jeblairclarkb: what we get in the partial upload could be very random21:55
pabelangeragreed, our zuul_launcher logs are key to detecting failues, so uploading them always is a +1 for me.  But, think we can also give .failed a go too21:55
jeblairShrews: yes21:55
jeblairto try to rephrase the situation:21:55
*** csomerville has quit IRC21:56
jeblairwe *require* that we always upload the console+ansible logs, and we *require* that upon any publisher failure, we know about that.  *optionally* we could try to continue past each publisher individually failing and try the next one.21:56
*** csomerville has joined #openstack-infra21:57
jeblairi think the last thing is the question -- whether we want to do that or not21:57
clarkbI like the idea of doing our best on error because when in an error situation you odn't know what will be useful info21:57
*** dimtruck is now known as zz_dimtruck21:58
*** gouthamr_ has joined #openstack-infra21:58
*** ddieterly is now known as ddieterly[away]21:58
*** sarob has quit IRC21:58
jeblairclarkb: ok.  i just want to make sure that we understand that if, say, the devstack logs fail because they try to copy kern.log, that doesn't mean we will get dstat.  it means we will copy whatever rsync happened to have copied so far (possibly nothing!) and still move on to the next publisher.21:58
jeblairso it's not "ignore individual file failures" it's "ignore publisher failures"21:59
*** Sukhdev has joined #openstack-infra21:59
jeblairstill worth it?21:59
clarkbmaybe we can ask rsync to be more greedy?21:59
pabelangerI have to step away, but agree that jeblair said on priority. I'll defer to which we do for now21:59
jeblairclarkb: a quick google says no21:59
jeblairclarkb: (solution is basically find+rsync)22:00
*** kzaitsev_mb has quit IRC22:00
*** ddieterly[away] is now known as ddieterly22:01
*** gouthamr has quit IRC22:01
anteayapriteau: hello, are you available for a chat?22:02
jeblairclarkb: still worth it?22:02
priteauYes for a quick chat22:02
fungigonna go find some dinner, back in a while22:02
anteayapriteau: thank you, I understand you have done a lot of work on Blazar22:02
priteauanteaya: Sorry I meant to reply to your email today22:02
clarkbjeblair: probably not22:02
anteayafungi: enjoy dinner22:02
pleia2dmsimard: thanks, approved22:02
anteayapriteau: no worries, irc is better anyway, larger audience more help22:03
anteayapriteau: first of all thank you for using Blazar22:03
anteayapriteau: that is great news22:03
anteayapriteau: what can I do to help you upstream your patches?22:03
jeblairclarkb, Shrews: so does "block: [publishers]; always: [log upload]" sound like a close-enough approximation of what we want?22:03
clarkbjeblair: yes I think so22:04
Shrewsjeblair: i think either is valid. but sounds like functionally, using the block seems to be more what you want22:04
priteauanteaya: I think the most helpful would be help to develop the tests, which are mostly missing at the moment22:05
jeblairit's at least simpler; i'll write that up and we can see what it looks like22:05
anteayapriteau: well I'm mostly interested in starting to get your work using gerrit22:05
anteayapriteau: was there a reason you didn't use our gerrit when you started added code to blazar in your fork?22:06
anteayapriteau: I should be clear, I'm a member of the infrastructure team, we help the developers use our workflow22:06
anteayapriteau: so having your work use gerrit would be helpful to having other openstack folks contribute to blazar, including getting tests working22:07
anteayapriteau: you wanted to upstream your work and I'd like to help you do that22:07
*** njohnston is now known as njohnston|afk22:07
priteauanteaya: when we started using Blazar in Chameleon, it was inactive with nothing happening in Gerrit, and we needed to get our project released so little time to iterate over patches22:08
openstackgerritMerged openstack-infra/system-config: Add #ara to the list of channels  https://review.openstack.org/32122922:08
anteayaokay fair enough22:08
anteayapriteau: and yes there has been little activity in blazar for a while now22:09
priteauanteaya: I should be able to submit most of our patches in July22:09
anteayaokay great, may I ask why July?22:09
priteauanteaya: I am out of the office starting this Friday22:10
priteaufor two weeks22:10
anteayapriteau: oh how lovely22:10
priteausorry, bad timing22:10
anteayaI hope you have a marvelous time off22:10
anteayanot at all, glad to meet you22:10
anteayalong enough to wish you good holiday22:10
priteau(honeymoon ;-) )22:10
*** jamesmcarthur has quit IRC22:10
anteayaoh even better22:10
priteaubut I will come back refreshed and eager to push these patches22:10
anteayamay you and your spouse share much love and happiness together22:10
Rockygcongratulations, priteau !22:11
anteayaha ha ha22:11
anteayayou have a lovely time22:11
*** mdrabe has quit IRC22:11
anteayaand you know where to find us now22:11
priteauyep, thanks for getting in touch22:11
anteayaand I look forward to picking this up when you are back online22:11
anteayasure thanks for the interest in Blazar22:11
*** kzaitsev_mb has joined #openstack-infra22:11
*** mriedem is now known as mriedem_away22:12
*** Rockyg has quit IRC22:12
*** fungi_lounge has joined #openstack-infra22:13
anteaya<-- dinner22:13
priteauhave a good dinner anteaya! talk to you soon22:14
*** Sukhdev has quit IRC22:14
*** psilvad has joined #openstack-infra22:15
*** Sukhdev has joined #openstack-infra22:15
anteayapriteau: thank you22:15
*** baoli has quit IRC22:15
*** Sukhdev has quit IRC22:16
*** dizquierdo has quit IRC22:16
*** akshai has quit IRC22:16
*** Sukhdev has joined #openstack-infra22:17
*** akshai has joined #openstack-infra22:17
*** priteau has quit IRC22:17
*** kzaitsev_mb has quit IRC22:18
*** eezhova has quit IRC22:19
*** mhickey has quit IRC22:20
*** netsin has quit IRC22:20
*** kzaitsev_mb has joined #openstack-infra22:21
*** sigmavirus24 is now known as sigmavirus24_awa22:23
*** openstackstatus has quit IRC22:25
*** openstack has joined #openstack-infra22:26
*** openstackstatus has joined #openstack-infra22:26
*** ChanServ sets mode: +v openstackstatus22:26
*** rcernin has joined #openstack-infra22:28
*** xarses has joined #openstack-infra22:30
*** xarses_ has quit IRC22:30
*** jsavak has quit IRC22:33
*** bpokorny_ has quit IRC22:35
*** ddieterly has quit IRC22:35
*** bpokorny has joined #openstack-infra22:35
*** bpokorny has quit IRC22:35
*** bpokorny has joined #openstack-infra22:36
*** psilvad has quit IRC22:37
*** furlongm_ has joined #openstack-infra22:37
*** rbrndt has joined #openstack-infra22:37
*** furlongm has quit IRC22:37
*** mtanino has quit IRC22:38
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: toci_gate_test: disable ipv6 on upgrade job for stable/liberty  https://review.openstack.org/32966322:39
*** tkelsey has joined #openstack-infra22:39
*** rbrndt has quit IRC22:39
asselinanyone know how to use ansible to find attached volumes and format partition them?22:40
*** mtanino has joined #openstack-infra22:43
*** tkelsey has quit IRC22:43
*** hongbin has quit IRC22:43
*** amitgandhinz has joined #openstack-infra22:44
clarkbasselin: at the very least you should be able to format and partition using shell directives. I do know not how to get the list of attached volumes in ansible though.22:44
*** ajmiller has quit IRC22:44
asselinyeah, I was hoping there was an 'ansible' way to do it. infra-ansbile has openstack.py which has the info, but I don't understand how it integrates...22:45
*** bhavik1 has joined #openstack-infra22:46
jeblairpabelanger: it looks like ansible logging is buffered, so when we upload the in-progress logs, we lose the last little bit22:46
*** roxanagh_ has joined #openstack-infra22:47
jeblairpabelanger: i mean, i was expecting not to have the results of actually uploading the log itself (for obvious reasons).  but we lose the last few tasks before that too it seems22:47
*** PollyZ_ has joined #openstack-infra22:47
clarkbasselin: typically I thin kyo uwould assert the attachment in ansible then operate off of that22:47
jeblairShrews, mordred: ^22:48
*** amitgandhinz has quit IRC22:48
clarkbasselin: but http://docs.ansible.com/ansible/os_server_facts_module.html will likely include that info22:48
*** bhavik has quit IRC22:48
*** bhavik1 is now known as bhavik22:48
jeblairShrews, mordred, pabelanger: doh! i'm wrong22:49
mordredjeblair: whee22:49
jeblairShrews, mordred, pabelanger: i saw the log end before i expected because we *copy* it, so it stopped at the copy task, which is before the upload task.22:49
*** PollyZ has quit IRC22:50
asselinclarkb, looks interesting. will try it out22:50
*** gongysh has joined #openstack-infra22:50
*** roxanagh_ has quit IRC22:51
*** ajmiller has joined #openstack-infra22:51
clarkbmordred: btw I notice in http://docs.ansible.com/ansible/os_server_volume_module.html that you allow setting the device. It should be noted that that is not reliable and is going away soon aiui22:51
*** Sukhdev has quit IRC22:51
*** PollyZ_ has quit IRC22:52
mordredclarkb: that's fantastic22:52
mordredclarkb: I love it when OpenStack removes interfaces to things22:52
*** vhosakot has quit IRC22:52
asselinlol...glad i didn't use that option22:53
*** gouthamr_ has quit IRC22:54
asselingot an unexpected error: http://paste.openstack.org/show/516069/ "shade is required for this module"}22:54
clarkbinstall shade?22:55
asselinit is installed....22:55
*** gouthamr has joined #openstack-infra22:55
asselinoh...maybe it needs to be installed on the target host?22:56
clarkbasselin: no virtualenvs are in use? anything like that22:56
clarkbasselin: it needs to be installed where ansible is running that lookup22:56
clarkbusually you would delegate to localhost I think since cloud22:57
*** tpsilva has quit IRC22:57
*** ajmiller_ has joined #openstack-infra23:01
notmorgananteaya: (ignore this checking something)23:02
notmorgananteaya: (your nick colour again:P)23:02
notmorgananteaya: awww... it's not as cool a color now. boo.23:03
*** ams__ has quit IRC23:03
*** netsin has joined #openstack-infra23:04
*** ajmiller has quit IRC23:05
anteayawhat colour am I now?23:06
anteayamy cat was torquoise and orange last night23:07
anteayashe looked so beautiful I couldn't stop petting her23:07
anteayanotmorgan: am I green still? green-blue?23:08
openstackgerritJames E. Blair proposed openstack-infra/zuul: Upload ansible playbook directory to logs server  https://review.openstack.org/32560223:10
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: ensure that console/ansible publishing is last  https://review.openstack.org/32960823:10
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: ensure log publishing always runs  https://review.openstack.org/32961223:10
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: set keep-hierarchy on log copying  https://review.openstack.org/32971223:10
jeblairclarkb, Shrews, pabelanger, mordred: ^23:10
jeblairi tested that, and it seems to work23:11
nibalizerasselin: clarkb yea when you have ansible in a venv it gets derpy23:12
nibalizerbecause the local connection loses the venv23:12
asselinnibalizer, go figure23:13
nibalizersometimes ill set python_interpreter on the host23:13
nibalizerto get aroubd it23:13
*** salv-orlando has quit IRC23:14
notmorganlifeless: let me know when you're about. :)23:14
anteayanotmorgan: well keep me posted if I become a better colour23:14
*** salv-orlando has joined #openstack-infra23:14
* anteaya thinks pink23:14
notmorganlifeless: want to start talking testrepository and such :) since ... I'm looking at things to dive into and fixup to make it all better for all of us23:15
notmorgananteaya: oh you don't want pink :P you want magenta ;) [i say this cause pink is kinda dull on this terminal]23:15
anteayaI want magenta23:15
anteayahow right you are23:16
pabelangerjeblair: +2 for the stack. Much simpler23:17
*** ayoung has joined #openstack-infra23:17
jeblairbtw, i did test that you can have a 'block' without an always (obvious to me); and an 'always' without 'block' (less obvious to me)23:17
jeblair(but still sensible i think)23:18
notmorganwwwwwwwww doh, pabelanger and anteaya now have the same coloured nicks now. I.. dunno how to fix this.23:18
mordredjeblair: you can have an always without a block?23:18
anteayanotmorgan: we're both magenta? well we are both canadian, so that fits23:19
notmorganno, trying to figure out if i can set a specific color for a specific nick23:19
anteayaplus our nicks are different lengths of characters23:19
anteayanotmorgan: what client?23:19
notmorgananteaya: weechat23:19
anteayayeah I just go with what it gives me23:20
mordredanteaya: in my terminal you are the same color of blue as jeblair - and notmorgan and pabelanger are the same magenta23:20
anteayayou're purple23:20
*** Sukhdev has joined #openstack-infra23:20
notmorganmordred: you have... dark cyan i think23:20
anteayamordred: you are the same blue as jeblair in my terminal23:20
anteayajeblair just goes with everything23:20
mordredhahaha. like a nice credenza23:20
*** sarob has joined #openstack-infra23:20
anteayaha ha ha23:21
notmorgananteaya: ++23:21
jeblaireveryone is the same color of grey in my terminal.23:21
* notmorgan is looking at the colorize nick code (yay python)23:21
anteayaha ha ha23:21
notmorganjeblair: so your terminal is color blind?23:21
mordredjeblair: how do you know who is talking?23:21
notmorganmordred: i know right?23:21
jeblairit makes you all easier to ignore ;)23:21
mordredjeblair: great plan23:22
anteayayeah, like how can I tell who to reply to unless I evaluate them with the credenza23:22
* jeblair tries out black23:22
anteayaha ha ha23:22
jeblairSO MUCH BETTER23:22
notmorganjeblair: i see what you did there...23:22
anteayajeblair's productivity goes up, way up23:22
*** akshai has quit IRC23:23
jeblairnotmorgan: i didn't!  ;)23:23
notmorganjeblair: ^523:23
jeblairnotmorgan and i are here all week folks!23:24
jeblairtip your irc bot!23:24
*** sarob has quit IRC23:24
*** sarob has joined #openstack-infra23:24
*** Guest16087 has quit IRC23:28
pabelangerTIL: people want to see what I say23:29
*** Jeffrey4l has joined #openstack-infra23:29
*** salv-orl_ has joined #openstack-infra23:29
clarkbpabelanger: error launches in osic seem to have fallen off but that may also be because overall demand has fallen23:29
*** yamada-h has joined #openstack-infra23:30
pabelangerclarkb: could be. I was working with somebody at rackspace today.  They are looking into the issue and expect to hear back from them tomorrow23:31
*** salv-orlando has quit IRC23:32
*** sdake has quit IRC23:32
*** sdake has joined #openstack-infra23:33
anteayapabelanger: yes, we do23:34
*** yamada-h has quit IRC23:34
*** xarses has quit IRC23:37
*** piet_ has joined #openstack-infra23:39
*** rcernin has quit IRC23:40
*** asselin_ has joined #openstack-infra23:40
lifelessnotmorgan: hai23:40
*** amitgandhinz has joined #openstack-infra23:44
*** asselin_ has quit IRC23:45
*** Sukhdev has quit IRC23:45
notmorganlifeless: ohai23:48
notmorganlifeless: so, in this channel or -dev better to talk testrepository and such?23:49
*** amitgandhinz has quit IRC23:49
lifelessnotmorgan: #subunit probably, its where most of the contributors idle23:49
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: Enable the Ubuntu binary build and deploy for all branch in kolla  https://review.openstack.org/32972123:52
*** sdague has quit IRC23:53
*** johnny_ has quit IRC23:54
*** xyang1 has quit IRC23:56
openstackgerritHisashi Osanai proposed openstack-infra/project-config: Add jobs for Monasca-Analytics  https://review.openstack.org/32937623:57
openstackgerritAndrea Frittoli proposed openstack-infra/project-config: Start using pre-prov creds in dsvm-neutron-full  https://review.openstack.org/32972323:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!