| dtrainor | excellent, i can't connect to http://myconnection.cox.com/ to get contact information for them. i'll give them a call. thanks for your time. | 00:00 |
|---|---|---|
| *** rhallisey has quit IRC | 00:00 | |
| *** gyee has joined #openstack-infra | 00:02 | |
| *** wolverineav has quit IRC | 00:02 | |
| *** sflanigan has quit IRC | 00:03 | |
| *** wolverin_ has quit IRC | 00:03 | |
| *** camunoz has quit IRC | 00:03 | |
| *** funzo has joined #openstack-infra | 00:03 | |
| ianw | speaking of networks, has anyone ever managed to get to a vnc console on a ci host? | 00:04 |
| clarkb | ianw: I think the rackspace one works if you do the things to run the java or whatever it is | 00:04 |
| *** pahuang has quit IRC | 00:04 | |
| clarkb | and hpcloud's worked once upon a time | 00:04 |
| *** marst has joined #openstack-infra | 00:04 | |
| ianw | clarkb: oh, sorry i'm talking about a console on an actual slave node for a devstack vm | 00:05 |
| clarkb | oh that | 00:05 |
| clarkb | by default devstack doesn't install vnc iirc | 00:05 |
| *** dingyichen has joined #openstack-infra | 00:05 | |
| *** smarcet has quit IRC | 00:06 | |
| ianw | i think it's all there ... i can telnet to localhost 6080 but either it's not listening on external interface or it's blocked ... | 00:06 |
| clarkb | oh we do block most ports if going from external | 00:07 |
| clarkb | on dib built images. try an ssh tunnel and connect to localhost? | 00:07 |
| *** funzo has quit IRC | 00:08 | |
| ianw | yeah i'm presuming it's security groups ... trying a tunnel | 00:08 |
| *** mdrabe has quit IRC | 00:08 | |
| *** wolverineav has joined #openstack-infra | 00:09 | |
| *** armax has quit IRC | 00:10 | |
| *** esikachev has joined #openstack-infra | 00:11 | |
| *** Sukhdev has quit IRC | 00:12 | |
| *** wolverineav has quit IRC | 00:13 | |
| ianw | cool, tunnel to 6800 works | 00:15 |
| *** esikachev has quit IRC | 00:16 | |
| *** camunoz has joined #openstack-infra | 00:16 | |
| *** zz_dimtruck is now known as dimtruck | 00:20 | |
| *** sflanigan has joined #openstack-infra | 00:20 | |
| *** pahuang has joined #openstack-infra | 00:22 | |
| *** MasterOfBugs has quit IRC | 00:26 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 00:30 |
| *** r-daneel has quit IRC | 00:30 | |
| *** rwsu has quit IRC | 00:31 | |
| *** vhosakot_ has joined #openstack-infra | 00:31 | |
| *** s-shiono has joined #openstack-infra | 00:33 | |
| *** vhosakot has quit IRC | 00:33 | |
| *** beslemon has quit IRC | 00:34 | |
| *** erikmwilson has quit IRC | 00:37 | |
| *** erikmwilson has joined #openstack-infra | 00:39 | |
| *** lseki has quit IRC | 00:40 | |
| *** beslemon has joined #openstack-infra | 00:40 | |
| openstackgerrit | Leehom Li proposed openstack/diskimage-builder master: Fix problem unable disable iptables service https://review.openstack.org/419703 | 00:40 |
| *** tphummel has quit IRC | 00:42 | |
| *** pvaneck has quit IRC | 00:43 | |
| *** ihrachys has joined #openstack-infra | 00:48 | |
| *** esikachev has joined #openstack-infra | 00:49 | |
| *** ZZelle has quit IRC | 00:49 | |
| *** lin_yang has joined #openstack-infra | 00:49 | |
| *** _Cyclone_ has quit IRC | 00:50 | |
| *** ihrachys_ has joined #openstack-infra | 00:50 | |
| *** armax has joined #openstack-infra | 00:51 | |
| *** Julien-zte has joined #openstack-infra | 00:52 | |
| *** kzaitsev_mb has quit IRC | 00:52 | |
| *** zhurong has joined #openstack-infra | 00:53 | |
| *** amotoki has joined #openstack-infra | 00:53 | |
| *** esikachev has quit IRC | 00:53 | |
| *** ssbarnea has quit IRC | 00:53 | |
| *** ihrachys has quit IRC | 00:54 | |
| *** esikachev has joined #openstack-infra | 00:56 | |
| *** esikachev has quit IRC | 01:00 | |
| *** sree has joined #openstack-infra | 01:02 | |
| *** sree has quit IRC | 01:02 | |
| *** sree has joined #openstack-infra | 01:02 | |
| *** stakeda has joined #openstack-infra | 01:03 | |
| *** beslemon has quit IRC | 01:04 | |
| *** funzo has joined #openstack-infra | 01:04 | |
| *** vhosakot_ has quit IRC | 01:07 | |
| *** erikmwilson has quit IRC | 01:07 | |
| *** erikmwilson has joined #openstack-infra | 01:08 | |
| *** funzo has quit IRC | 01:09 | |
| *** esikachev has joined #openstack-infra | 01:16 | |
| *** xarses_ has quit IRC | 01:17 | |
| *** esikachev has quit IRC | 01:20 | |
| *** dimtruck is now known as zz_dimtruck | 01:22 | |
| *** liujiong_lj has joined #openstack-infra | 01:23 | |
| *** yanyanhu has joined #openstack-infra | 01:27 | |
| *** esikachev has joined #openstack-infra | 01:29 | |
| *** esikachev has quit IRC | 01:33 | |
| *** liujiong_lj is now known as liujiong | 01:35 | |
| *** dingyichen has quit IRC | 01:35 | |
| *** eroux has joined #openstack-infra | 01:36 | |
| *** smarcet has joined #openstack-infra | 01:39 | |
| *** esikachev has joined #openstack-infra | 01:41 | |
| *** yonglihe has joined #openstack-infra | 01:43 | |
| *** mriedem has quit IRC | 01:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:48 | |
| *** esikachev has quit IRC | 01:49 | |
| *** aviau has quit IRC | 01:50 | |
| *** aviau has joined #openstack-infra | 01:50 | |
| *** xarses_ has joined #openstack-infra | 01:52 | |
| *** xarses_ has quit IRC | 01:53 | |
| *** xarses_ has joined #openstack-infra | 01:53 | |
| *** esikachev has joined #openstack-infra | 01:54 | |
| *** xarses_ has quit IRC | 01:58 | |
| *** esikachev has quit IRC | 02:04 | |
| *** funzo has joined #openstack-infra | 02:05 | |
| *** amotoki has quit IRC | 02:05 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Support swift logserver without Send-Temp-Url-Key https://review.openstack.org/270338 | 02:05 |
| *** amotoki has joined #openstack-infra | 02:05 | |
| *** EricGonczer_ has quit IRC | 02:07 | |
| *** erikmwilson has quit IRC | 02:07 | |
| *** yamahata has quit IRC | 02:08 | |
| *** erikmwilson has joined #openstack-infra | 02:09 | |
| *** sflanigan has quit IRC | 02:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/gerritbot master: Add _get_username method to discover username https://review.openstack.org/318096 | 02:09 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/gerritbot master: Add change-created event type https://review.openstack.org/286366 | 02:09 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add javascript license information https://review.openstack.org/335660 | 02:10 |
| *** funzo has quit IRC | 02:10 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Remove remaining apscheduler variables https://review.openstack.org/445719 | 02:13 |
| *** dhinesh has quit IRC | 02:13 | |
| *** gongysh has joined #openstack-infra | 02:14 | |
| *** EricGonczer_ has joined #openstack-infra | 02:16 | |
| *** sree has quit IRC | 02:17 | |
| openstackgerrit | Merged openstack-infra/release-tools master: Add --stop option for bugs-fixed-since https://review.openstack.org/445315 | 02:19 |
| *** sflanigan has joined #openstack-infra | 02:22 | |
| *** cody-somerville has joined #openstack-infra | 02:28 | |
| *** cody-somerville has joined #openstack-infra | 02:28 | |
| *** csomerville has quit IRC | 02:28 | |
| *** kzaitsev_mb has quit IRC | 02:34 | |
| *** gildub has quit IRC | 02:35 | |
| *** ijw has quit IRC | 02:36 | |
| *** erikmwilson has quit IRC | 02:37 | |
| *** zz_dimtruck is now known as dimtruck | 02:37 | |
| *** erikmwilson has joined #openstack-infra | 02:37 | |
| *** ijw has joined #openstack-infra | 02:38 | |
| *** ihrachys_ has quit IRC | 02:38 | |
| *** gouthamr has quit IRC | 02:39 | |
| *** EricGonczer_ has quit IRC | 02:40 | |
| *** rlandy has quit IRC | 02:40 | |
| *** ijw has quit IRC | 02:42 | |
| *** sree has joined #openstack-infra | 02:42 | |
| *** csomerville has joined #openstack-infra | 02:44 | |
| *** cody-somerville has quit IRC | 02:47 | |
| *** Apoorva has quit IRC | 02:52 | |
| *** esikachev has joined #openstack-infra | 02:54 | |
| *** hongbin has joined #openstack-infra | 02:56 | |
| *** esikachev has quit IRC | 02:58 | |
| *** sree has quit IRC | 03:00 | |
| *** gyee has quit IRC | 03:01 | |
| *** esikachev has joined #openstack-infra | 03:01 | |
| *** flepied has quit IRC | 03:04 | |
| *** gildub has joined #openstack-infra | 03:04 | |
| *** esikachev has quit IRC | 03:06 | |
| *** funzo has joined #openstack-infra | 03:06 | |
| *** markvoelker has quit IRC | 03:06 | |
| *** erikmwil_ has joined #openstack-infra | 03:09 | |
| *** erikmwilson has quit IRC | 03:09 | |
| *** funzo has quit IRC | 03:10 | |
| *** eroux has quit IRC | 03:14 | |
| *** gyee has joined #openstack-infra | 03:17 | |
| *** erikmwil_ has quit IRC | 03:17 | |
| *** knangia has quit IRC | 03:21 | |
| *** esikachev has joined #openstack-infra | 03:24 | |
| *** Jeffrey4l has quit IRC | 03:28 | |
| *** esikachev has quit IRC | 03:29 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:30 | |
| *** amotoki has quit IRC | 03:32 | |
| *** sflanigan has quit IRC | 03:33 | |
| *** aeng has quit IRC | 03:37 | |
| *** ihrachys has joined #openstack-infra | 03:37 | |
| *** esikachev has joined #openstack-infra | 03:38 | |
| *** jiaohaolin has joined #openstack-infra | 03:38 | |
| *** Jeffrey4l has joined #openstack-infra | 03:41 | |
| *** hongbin has quit IRC | 03:42 | |
| *** esikachev has quit IRC | 03:42 | |
| *** nicolasbock has quit IRC | 03:42 | |
| jamielennox | pabelanger: are you here? where did the -infra zuulv3 config go? | 03:45 |
| *** sflanigan has joined #openstack-infra | 03:46 | |
| *** ihrachys has quit IRC | 03:46 | |
| *** amotoki has joined #openstack-infra | 03:49 | |
| *** amotoki has quit IRC | 03:50 | |
| *** hashar has joined #openstack-infra | 03:52 | |
| *** amotoki has joined #openstack-infra | 03:52 | |
| *** amotoki has quit IRC | 03:54 | |
| *** knangia has joined #openstack-infra | 03:55 | |
| *** links has joined #openstack-infra | 03:57 | |
| *** Julien-zte has quit IRC | 04:03 | |
| *** Julien-zte has joined #openstack-infra | 04:04 | |
| *** Julien-zte has joined #openstack-infra | 04:05 | |
| *** Julien-zte has quit IRC | 04:05 | |
| *** Julien-zte has joined #openstack-infra | 04:05 | |
| *** Julien-zte has quit IRC | 04:06 | |
| *** Julien-zte has joined #openstack-infra | 04:06 | |
| *** funzo has joined #openstack-infra | 04:06 | |
| *** Julien-zte has joined #openstack-infra | 04:07 | |
| *** markvoelker has joined #openstack-infra | 04:07 | |
| *** Julien-zte has quit IRC | 04:07 | |
| *** rfolco has quit IRC | 04:10 | |
| *** funzo has quit IRC | 04:11 | |
| *** markvoelker has quit IRC | 04:13 | |
| *** ijw has joined #openstack-infra | 04:13 | |
| *** aeng has joined #openstack-infra | 04:14 | |
| *** psachin has joined #openstack-infra | 04:19 | |
| *** aeng has quit IRC | 04:19 | |
| *** amotoki has joined #openstack-infra | 04:22 | |
| *** ihrachys has joined #openstack-infra | 04:24 | |
| *** kzaitsev_mb has quit IRC | 04:25 | |
| *** smarcet has quit IRC | 04:25 | |
| *** Apoorva has joined #openstack-infra | 04:35 | |
| *** gongysh has quit IRC | 04:39 | |
| *** amotoki has quit IRC | 04:39 | |
| *** Apoorva has quit IRC | 04:45 | |
| *** beslemon has joined #openstack-infra | 04:46 | |
| *** anilvenkata has joined #openstack-infra | 04:47 | |
| *** beslemon_ has joined #openstack-infra | 04:51 | |
| *** beslemon has quit IRC | 04:51 | |
| *** beslemon_ has quit IRC | 04:55 | |
| jeblair | jamielennox: it's in the project-config and zuul repos | 04:57 |
| *** psachin has quit IRC | 05:00 | |
| *** nadya has joined #openstack-infra | 05:00 | |
| *** zhurong has quit IRC | 05:00 | |
| *** sree has joined #openstack-infra | 05:01 | |
| jamielennox | jeblair: thanks, i found http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/main.yaml but i was expecting to see more config changes | 05:03 |
| jamielennox | i think maybe i just don't follow the puppet well enough | 05:03 |
| *** david-lyle has quit IRC | 05:04 | |
| *** sree has quit IRC | 05:06 | |
| *** esikachev has joined #openstack-infra | 05:06 | |
| *** ihrachys has quit IRC | 05:06 | |
| jeblair | jamielennox: there's a /zuul.yaml in that repo as well, and one in zuul's repo, and then pabelanger has a stack of changes in review for it | 05:07 |
| jeblair | jamielennox: correction, in zuul's repo, it's /.zuul.yaml | 05:07 |
| *** funzo has joined #openstack-infra | 05:07 | |
| jamielennox | jeblair: so the zuul project is being specified twice there, once as project: in zuul.yaml and once as project-repo in main.yaml | 05:09 |
| jamielennox | what's the difference? | 05:09 |
| jeblair | jamielennox: main.yaml is the bootstrap configuration -- it tells zuul where to load the actual config | 05:09 |
| jamielennox | (i'm sure i can figure this out for myself if you're tying to go to bed) | 05:09 |
| *** nadya has quit IRC | 05:10 | |
| jeblair | jamielennox: main.yaml basically won't have much more in it than what's already there, except the list of repos will be longer. all the interesting stuff is in the (.)zuul.yaml in the repos. | 05:10 |
| jamielennox | jeblair: right, but given that openstack-infra is in main.yaml, what does | 05:11 |
| jamielennox | - project: | 05:11 |
| jamielennox | name: openstack-infra/zuul | 05:11 |
| jamielennox | check: | 05:11 |
| jamielennox | jobs: [] | 05:11 |
| *** funzo has quit IRC | 05:11 | |
| jamielennox | in project-config/zuul.yaml tell it that openstack-infra/zuul/.zuul.yaml doesn't? | 05:12 |
| jeblair | jamielennox: main.yaml is just for listing repos where zuul should read zuul.yaml files. so let's set that aside. the remainder of your question is "why is there a zuul project definition in both project-config/zuul.yaml and zuul/.zuul.yaml?" | 05:16 |
| *** psachin has joined #openstack-infra | 05:17 | |
| *** gongysh has joined #openstack-infra | 05:17 | |
| *** esikachev has quit IRC | 05:18 | |
| jamielennox | jeblair: yea, and i'm guessing the answer is they are merged | 05:18 |
| jeblair | jamielennox: short answer, yes | 05:18 |
| jeblair | jamielennox: longer answer: at the moment, we mostly just want to work with zuul/.zuul.yaml because it's self testing (unlike project-config/zuul.yaml). but there's a catch-22 where you can't dynamically add a project to a pipeline if that project isn't already in at least one pipeline. so at the PTG we added the project-config zuul.yaml so that we could bootstrap the zuul/.zuul.yaml. | 05:19 |
| jeblair | jamielennox: there's a story in storyboard about seeing if we can make that better :) | 05:19 |
| jeblair | jamielennox: but aside from strange bootstrap edge-cases, it's still sensible to have that project defn appear it 2 places so that, say, openstack can specify that projects should all run certain jobs centrally in project config, while still allowing projects to add their own jobs in their own repos. | 05:21 |
| jamielennox | because unless that project is already in one pipeline nothing knows that the .zuul.yaml file has changed to update the layout? | 05:21 |
| jeblair | jamielennox: exactly; the configuration change detection is attached to the event processing for pipelines. | 05:21 |
| *** kzaitsev_mb has joined #openstack-infra | 05:21 | |
| jamielennox | interesting, ok | 05:22 |
| jeblair | jamielennox: bedtime now; good night :) | 05:22 |
| jamielennox | jeblair: thanks for the explanation | 05:22 |
| jamielennox | good night | 05:23 |
| *** faizy has joined #openstack-infra | 05:28 | |
| *** hashar has quit IRC | 05:29 | |
| *** amotoki has joined #openstack-infra | 05:30 | |
| *** prateek has joined #openstack-infra | 05:32 | |
| *** prateek has quit IRC | 05:32 | |
| *** prateek has joined #openstack-infra | 05:33 | |
| *** vikrant has joined #openstack-infra | 05:35 | |
| *** eliqiao has quit IRC | 05:35 | |
| *** Julien-zte has joined #openstack-infra | 05:36 | |
| *** sandanar has joined #openstack-infra | 05:42 | |
| *** claudiub has joined #openstack-infra | 05:44 | |
| *** Julien-zte has quit IRC | 05:48 | |
| *** Julien-zte has joined #openstack-infra | 05:48 | |
| *** psachin has quit IRC | 05:49 | |
| openstackgerrit | venkata anil proposed openstack-infra/devstack-gate master: Set agent modes for DVR+HA multinode job https://review.openstack.org/383827 | 05:49 |
| *** Julien-zte has quit IRC | 05:51 | |
| *** gyee has quit IRC | 05:51 | |
| *** Julien-zte has joined #openstack-infra | 05:52 | |
| *** psachin has joined #openstack-infra | 05:57 | |
| *** MasterOfBugs has joined #openstack-infra | 05:58 | |
| *** adriant has quit IRC | 05:59 | |
| *** yamahata has joined #openstack-infra | 05:59 | |
| *** amotoki has quit IRC | 06:00 | |
| *** dhajare has joined #openstack-infra | 06:02 | |
| *** cshastri has joined #openstack-infra | 06:05 | |
| *** camunoz has quit IRC | 06:06 | |
| *** nadya has joined #openstack-infra | 06:07 | |
| *** Julien-zte has quit IRC | 06:07 | |
| *** sflanigan has quit IRC | 06:07 | |
| *** Julien-zte has joined #openstack-infra | 06:07 | |
| *** funzo has joined #openstack-infra | 06:08 | |
| *** markvoelker has joined #openstack-infra | 06:09 | |
| *** nadya has quit IRC | 06:11 | |
| *** amotoki has joined #openstack-infra | 06:11 | |
| *** funzo has quit IRC | 06:13 | |
| *** aeng has joined #openstack-infra | 06:13 | |
| *** markvoelker has quit IRC | 06:13 | |
| *** sree has joined #openstack-infra | 06:18 | |
| *** Julien-zte has quit IRC | 06:19 | |
| *** sflanigan has joined #openstack-infra | 06:19 | |
| *** sflanigan has joined #openstack-infra | 06:19 | |
| *** camunoz has joined #openstack-infra | 06:19 | |
| *** armax has quit IRC | 06:20 | |
| *** pahuang has quit IRC | 06:21 | |
| *** isaacb has joined #openstack-infra | 06:26 | |
| *** pahuang has joined #openstack-infra | 06:33 | |
| *** flepied has joined #openstack-infra | 06:35 | |
| *** e0ne has joined #openstack-infra | 06:37 | |
| *** e0ne has quit IRC | 06:38 | |
| *** sandanar has quit IRC | 06:45 | |
| *** Julien-zte has joined #openstack-infra | 06:49 | |
| *** jaosorior has joined #openstack-infra | 06:52 | |
| *** tuanluong has joined #openstack-infra | 06:54 | |
| *** john-davidge has joined #openstack-infra | 06:56 | |
| *** zhurong has joined #openstack-infra | 06:57 | |
| *** john-davidge has quit IRC | 07:01 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Removed some entities from L2 cache https://review.openstack.org/445777 | 07:03 |
| vsaienk0 | ianw: yolanda please add yo your review queue https://review.openstack.org/#/c/445555/ move ironic multinode multitenant to check pipeline as -nv. Thanks in advance! | 07:03 |
| *** tuanluong_ has joined #openstack-infra | 07:04 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Removed some entities from L2 cache https://review.openstack.org/445777 | 07:04 |
| *** andreas_s has joined #openstack-infra | 07:06 | |
| *** tuanluong has quit IRC | 07:07 | |
| *** jtomasek has quit IRC | 07:07 | |
| *** funzo has joined #openstack-infra | 07:09 | |
| *** flepied has quit IRC | 07:12 | |
| *** funzo has quit IRC | 07:13 | |
| *** isaacb has quit IRC | 07:18 | |
| *** Hal has joined #openstack-infra | 07:20 | |
| openstackgerrit | toan.t.pham proposed openstack-infra/jenkins-job-builder master: Add folder create for cloudbee folder jobs https://review.openstack.org/438795 | 07:23 |
| *** isaacb has joined #openstack-infra | 07:25 | |
| *** xarses_ has joined #openstack-infra | 07:26 | |
| *** alexchadin has joined #openstack-infra | 07:26 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: fix on missing dependencies on L2 cache https://review.openstack.org/445793 | 07:26 |
| *** andreww has joined #openstack-infra | 07:27 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: fix on missing dependencies on L2 cache https://review.openstack.org/445793 | 07:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Make python35 voting for monasca-common https://review.openstack.org/442278 | 07:28 |
| *** xarses has quit IRC | 07:28 | |
| *** andreww has quit IRC | 07:29 | |
| *** andreww has joined #openstack-infra | 07:29 | |
| *** xarses_ has quit IRC | 07:30 | |
| *** knangia has quit IRC | 07:31 | |
| *** eroux has joined #openstack-infra | 07:32 | |
| openstackgerrit | Merged openstack-infra/project-config master: ironic: Make grenade multi-node job non-voting https://review.openstack.org/445555 | 07:32 |
| *** xarses has joined #openstack-infra | 07:33 | |
| *** gongysh has quit IRC | 07:34 | |
| *** Hal has quit IRC | 07:34 | |
| *** andreww has quit IRC | 07:34 | |
| *** pgadiya has joined #openstack-infra | 07:34 | |
| *** gildub has quit IRC | 07:35 | |
| *** flepied has joined #openstack-infra | 07:42 | |
| *** tesseract has joined #openstack-infra | 07:44 | |
| *** flepied has left #openstack-infra | 07:45 | |
| *** jiaohaolin2 has joined #openstack-infra | 07:45 | |
| *** jiaohaolin has quit IRC | 07:45 | |
| *** flepied has joined #openstack-infra | 07:45 | |
| *** gongysh has joined #openstack-infra | 07:50 | |
| *** tuanluong_ has quit IRC | 07:52 | |
| *** tuanluong has joined #openstack-infra | 07:55 | |
| *** matrohon has joined #openstack-infra | 07:55 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** hashar has joined #openstack-infra | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** shardy_afk is now known as shardy | 08:00 | |
| *** matrohon has quit IRC | 08:02 | |
| *** Dinesh_Bhor has joined #openstack-infra | 08:03 | |
| *** yujunz has joined #openstack-infra | 08:03 | |
| *** pcaruana has joined #openstack-infra | 08:05 | |
| *** ssbarnea has joined #openstack-infra | 08:07 | |
| *** Hal has joined #openstack-infra | 08:09 | |
| *** funzo has joined #openstack-infra | 08:10 | |
| *** markvoelker has joined #openstack-infra | 08:10 | |
| *** bhavik1 has joined #openstack-infra | 08:12 | |
| *** markvoelker has quit IRC | 08:14 | |
| *** funzo has quit IRC | 08:14 | |
| *** tmorin has joined #openstack-infra | 08:16 | |
| *** tmorin has quit IRC | 08:16 | |
| zigo | ianw: Around? | 08:16 |
| *** florianf has joined #openstack-infra | 08:17 | |
| *** lin_yang has quit IRC | 08:17 | |
| *** ssbarnea has quit IRC | 08:21 | |
| *** ssbarnea has joined #openstack-infra | 08:23 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/system-config master: Do not install from puppetlab for deb https://review.openstack.org/445813 | 08:23 |
| *** bhavik1 has quit IRC | 08:25 | |
| *** martinkopec has joined #openstack-infra | 08:26 | |
| *** zhurong has quit IRC | 08:28 | |
| *** yamahata has quit IRC | 08:31 | |
| *** d0ugal has joined #openstack-infra | 08:31 | |
| *** esikachev has joined #openstack-infra | 08:32 | |
| *** amotoki has quit IRC | 08:35 | |
| *** amoralej|off is now known as amoralej | 08:37 | |
| *** alexchadin has quit IRC | 08:37 | |
| *** jpich has joined #openstack-infra | 08:39 | |
| *** tmorin has joined #openstack-infra | 08:44 | |
| *** markmcd has joined #openstack-infra | 08:45 | |
| tmorin | hi yolanda ! I was wondering if you could have a look at https://review.openstack.org/#/c/441725/ (simple, blessed by infra and liaison already, just missing a workflow+1...) | 08:46 |
| *** ccamacho has joined #openstack-infra | 08:48 | |
| *** markmcd has left #openstack-infra | 08:48 | |
| *** markmcd has joined #openstack-infra | 08:49 | |
| *** markmcd has left #openstack-infra | 08:50 | |
| *** matrohon has joined #openstack-infra | 08:50 | |
| *** markmcd has joined #openstack-infra | 08:50 | |
| *** alexchadin has joined #openstack-infra | 08:54 | |
| *** ralonsoh has joined #openstack-infra | 08:55 | |
| *** priteau has joined #openstack-infra | 08:55 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Clean up oooq jobs https://review.openstack.org/445628 | 08:55 |
| *** Julien-zte has quit IRC | 08:56 | |
| *** nadya has joined #openstack-infra | 08:57 | |
| *** john-davidge has joined #openstack-infra | 08:57 | |
| *** nadya has quit IRC | 08:58 | |
| *** nadya has joined #openstack-infra | 08:58 | |
| *** pgadiya has quit IRC | 08:59 | |
| *** s-shiono has quit IRC | 08:59 | |
| *** rossella_s has joined #openstack-infra | 09:00 | |
| *** flepied has quit IRC | 09:00 | |
| *** Julien-zte has joined #openstack-infra | 09:01 | |
| ianw | zigo: briefly | 09:01 |
| *** john-davidge has quit IRC | 09:02 | |
| *** prateek_ has joined #openstack-infra | 09:02 | |
| *** jiaohaolin has joined #openstack-infra | 09:04 | |
| *** prateek has quit IRC | 09:04 | |
| *** jiaohaolin2 has quit IRC | 09:05 | |
| *** nijaba has quit IRC | 09:06 | |
| ianychoi | ianw, if you have time now, could you check the speed of docs.o.o? | 09:07 |
| ianychoi | ianw, it was slow the simliar time period yesterday and it is slow now from my experience and previously after some filesystem checking the speed was recovered | 09:08 |
| tmorin | ianw: (hi) could you workflow+1 on https://review.openstack.org/#/c/441725/ ? (simple, blessed by infra and liaison already, really just missing a workflow+1...) | 09:08 |
| *** yujunz has quit IRC | 09:09 | |
| *** yujunz has joined #openstack-infra | 09:09 | |
| ianw | ianychoi: i think it's the same issue | 09:10 |
| *** dimtruck is now known as zz_dimtruck | 09:10 | |
| *** funzo has joined #openstack-infra | 09:10 | |
| *** nijaba has joined #openstack-infra | 09:11 | |
| ianw | ianychoi: i restarted it. clearly the worker mpm switch isn't helping. we'll have to think on plan b | 09:11 |
| tmorin | ianw: thanks! | 09:12 |
| ianychoi | ianw, thanks for resolving it! | 09:14 |
| *** pgadiya has joined #openstack-infra | 09:15 | |
| *** funzo has quit IRC | 09:15 | |
| *** csomerville has quit IRC | 09:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Don't run check-requirements on old branches fro networking-bgppvn https://review.openstack.org/441725 | 09:16 |
| *** lucas-afk is now known as lucasagomes | 09:17 | |
| *** liusheng has quit IRC | 09:18 | |
| *** jtomasek has joined #openstack-infra | 09:18 | |
| *** liusheng has joined #openstack-infra | 09:19 | |
| *** flepied has joined #openstack-infra | 09:19 | |
| *** e0ne has joined #openstack-infra | 09:19 | |
| *** yujunz has quit IRC | 09:20 | |
| *** furlongm has joined #openstack-infra | 09:20 | |
| *** furlongm has quit IRC | 09:20 | |
| *** priteau has quit IRC | 09:25 | |
| *** liujiong_lj has joined #openstack-infra | 09:27 | |
| *** liujiong has quit IRC | 09:27 | |
| *** zhurong has joined #openstack-infra | 09:28 | |
| *** jaosorior is now known as jaosorior_lunch | 09:28 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Disable tempest in container job https://review.openstack.org/445840 | 09:30 |
| *** Julien-zte has quit IRC | 09:33 | |
| *** alexchadin has quit IRC | 09:36 | |
| *** liujiong_lj is now known as liujiong | 09:36 | |
| *** neiljerram has joined #openstack-infra | 09:37 | |
| *** zhurong has quit IRC | 09:38 | |
| therve | Hello! I have a question about centos builds | 09:38 |
| therve | It seems they are significantly slower than ubuntu ones | 09:38 |
| therve | Is it missing some mirroring, should I use ubuntu wherever I can? | 09:38 |
| *** pgadiya has quit IRC | 09:40 | |
| *** derekh has joined #openstack-infra | 09:43 | |
| *** zhurong has joined #openstack-infra | 09:44 | |
| *** Hal has quit IRC | 09:44 | |
| *** ijw has quit IRC | 09:45 | |
| *** john-davidge has joined #openstack-infra | 09:45 | |
| *** mhickey has joined #openstack-infra | 09:46 | |
| *** vsaienko has joined #openstack-infra | 09:47 | |
| *** electrofelix has joined #openstack-infra | 09:49 | |
| *** panda|zZ is now known as panda | 09:50 | |
| andreas_s | hi, can anybody tell me where I can find the code for the OpenStack proposal bot? I want to understand which files are being considered for pushing updates (only requirements and test-requirements.txt, or also foo-requirements.txt?) | 09:50 |
| *** electrofelix has quit IRC | 09:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Re-enable Image v1 for OSC functional jobs https://review.openstack.org/445553 | 09:50 |
| *** ijw has joined #openstack-infra | 09:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove experimental ironic grenade multitenant single node job https://review.openstack.org/437936 | 09:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Cinder: Move NFS job from experimental to check https://review.openstack.org/443685 | 09:51 |
| openstackgerrit | Merged openstack-infra/project-config master: [magnum] Replace old jobs with the experimental https://review.openstack.org/442142 | 09:52 |
| *** zhurong has quit IRC | 09:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Adding new repositories for Octavia https://review.openstack.org/444057 | 09:53 |
| *** electrofelix has joined #openstack-infra | 09:53 | |
| *** dmellado has quit IRC | 09:54 | |
| *** amoralej has quit IRC | 09:54 | |
| *** pgadiya has joined #openstack-infra | 09:54 | |
| asettle | Hey guys - docs.o.o is slower than it has been for ages. Anything we can do? | 09:54 |
| openstackgerrit | Adam Spiers proposed openstack-infra/jenkins-job-builder master: add two new options to Log Parser Plugin https://review.openstack.org/445509 | 09:55 |
| *** martinkopec has quit IRC | 09:55 | |
| john-davidge | To add to asettle's comment, I'm getting actual page timeouts this morning | 09:57 |
| *** martinkopec has joined #openstack-infra | 09:57 | |
| *** ijw has quit IRC | 09:57 | |
| *** amoralej has joined #openstack-infra | 10:00 | |
| *** ijw has joined #openstack-infra | 10:01 | |
| *** dizquierdo has joined #openstack-infra | 10:01 | |
| *** dmellado has joined #openstack-infra | 10:02 | |
| *** nadya has quit IRC | 10:04 | |
| *** martinkopec has quit IRC | 10:06 | |
| *** martinkopec has joined #openstack-infra | 10:06 | |
| ianw | asettle / john-davidge : any better? | 10:09 |
| asettle | testing one twoo... | 10:10 |
| asettle | yes :) | 10:10 |
| asettle | For me, anyway | 10:10 |
| asettle | john-davidge ? | 10:10 |
| *** isaacb has quit IRC | 10:10 | |
| john-davidge | ianw: I'm still seeing the occasional slow load, but no timeouts at the moment | 10:10 |
| john-davidge | ianw: Thanks! What did you do? | 10:10 |
| *** markvoelker has joined #openstack-infra | 10:10 | |
| ianw | well i restarted it, but it's not much of a solution | 10:11 |
| asettle | That's a start ;) | 10:11 |
| *** funzo has joined #openstack-infra | 10:11 | |
| ianw | https://access.redhat.com/solutions/2078163 is interesting take on it ... | 10:13 |
| ianw | i wonder if something is trying a graceful restart of apache around this time | 10:14 |
| *** prateek__ has joined #openstack-infra | 10:15 | |
| *** markvoelker has quit IRC | 10:15 | |
| cmurphy | ianw: fungi had suggested it might be a noisy neighbor issue | 10:16 |
| *** funzo has quit IRC | 10:16 | |
| *** tmorin has left #openstack-infra | 10:16 | |
| cmurphy | ianw: it's also interesting that the load average isn't as spikey as it has been in the past http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=10229&rra_id=all | 10:16 |
| *** shu-mutou-AWAY is now known as shu-mutou | 10:17 | |
| frickler | ianw: can't read the rh link, but might log rotation trigger this? | 10:17 |
| *** saibarspeis has joined #openstack-infra | 10:18 | |
| *** prateek_ has quit IRC | 10:18 | |
| *** openstackgerrit has quit IRC | 10:18 | |
| *** tosky has joined #openstack-infra | 10:18 | |
| *** tuanluong has quit IRC | 10:19 | |
| *** rfolco has joined #openstack-infra | 10:19 | |
| frickler | response times are back to being pretty bad, too | 10:20 |
| *** yanyanhu has quit IRC | 10:20 | |
| *** adarazs has left #openstack-infra | 10:21 | |
| *** liujiong has quit IRC | 10:23 | |
| *** vsaienko has quit IRC | 10:23 | |
| ianw | frrickler: hmm, maybe | 10:26 |
| *** vsaienko has joined #openstack-infra | 10:26 | |
| *** sdague has joined #openstack-infra | 10:27 | |
| ianw | [Wed Mar 15 10:15:19.524417 2017] [mpm_worker:error] [pid 15827:tid 139979372996480] AH00287: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting | 10:29 |
| ianw | that was about 15 minutes ago | 10:29 |
| ianw | if someone wants to fiddle the maxrequestworkers settings in puppet, i guess it's worth turning it up? | 10:30 |
| *** saibarspeis has quit IRC | 10:30 | |
| ianw | that rh one says | 10:30 |
| ianw | MaxRequestWorkers is set equal to ThreadsPerChild. This means one httpd process takes up the full score board. When you call for a graceful, existing processes are told to finish current requests and then exit. So with MaxRequestWorkers = ThreadsPerChild, the one process and its threads go into this G state and cannot accept new requests. New processes cannot be created to serve new requests at this time as the G state threads consume the | 10:30 |
| ianw | full score board, generating the "scoreboard is full" errors. | 10:30 |
| *** flepied has quit IRC | 10:31 | |
| *** openstackgerrit has joined #openstack-infra | 10:34 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: quickstart transition: propagate deploy.env with existing functions https://review.openstack.org/445866 | 10:34 |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:34 | |
| *** ym200 has quit IRC | 10:36 | |
| *** jtomasek has quit IRC | 10:38 | |
| *** jtomasek has joined #openstack-infra | 10:38 | |
| *** florianf has quit IRC | 10:39 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] quickstart transition: propagate deploy.env with existing functions https://review.openstack.org/445866 | 10:42 |
| *** rhallisey has joined #openstack-infra | 10:42 | |
| *** scottda has quit IRC | 10:44 | |
| *** jtomasek has quit IRC | 10:45 | |
| *** jtomasek has joined #openstack-infra | 10:45 | |
| *** flepied has joined #openstack-infra | 10:47 | |
| *** florianf has joined #openstack-infra | 10:50 | |
| *** nicolasbock has joined #openstack-infra | 10:50 | |
| *** jkilpatr has quit IRC | 10:50 | |
| *** furlongm has joined #openstack-infra | 10:52 | |
| *** chem has joined #openstack-infra | 10:53 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 10:59 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Enable multinode oooq jobs for all branches https://review.openstack.org/445875 | 10:59 |
| *** Julien-zte has joined #openstack-infra | 11:01 | |
| *** gongysh has quit IRC | 11:01 | |
| *** saibarspeis has joined #openstack-infra | 11:10 | |
| *** funzo has joined #openstack-infra | 11:12 | |
| *** amotoki has joined #openstack-infra | 11:14 | |
| openstackgerrit | Merged openstack-infra/release-tools master: Add support for mail servers that require a login https://review.openstack.org/439968 | 11:15 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 11:15 |
| *** lucasagomes is now known as lucas-pto | 11:15 | |
| *** jkilpatr has joined #openstack-infra | 11:16 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT MERGE: testing hack for httpd https://review.openstack.org/445880 | 11:16 |
| *** funzo has quit IRC | 11:16 | |
| *** ociuhandu has quit IRC | 11:21 | |
| *** saibarspeis has quit IRC | 11:21 | |
| *** vsaienko has quit IRC | 11:21 | |
| *** amotoki has quit IRC | 11:21 | |
| *** vsaienko has joined #openstack-infra | 11:23 | |
| *** jkilpatr has quit IRC | 11:24 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 11:27 |
| andreykurilin_ | Hi folks! I wonder why we do not have https for lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ? | 11:28 |
| *** dtantsur|afk is now known as dtantsur | 11:29 | |
| openstackgerrit | Michele Baldessari proposed openstack-infra/tripleo-ci master: DONT MERGE: testing hack for httpd https://review.openstack.org/445880 | 11:29 |
| *** yolanda has quit IRC | 11:29 | |
| *** isaacb has joined #openstack-infra | 11:29 | |
| *** psachin has quit IRC | 11:30 | |
| *** psachin has joined #openstack-infra | 11:30 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Enable multinode oooq jobs for all branches https://review.openstack.org/445875 | 11:30 |
| *** gouthamr has joined #openstack-infra | 11:30 | |
| *** ralonsoh has quit IRC | 11:32 | |
| *** yolanda has joined #openstack-infra | 11:34 | |
| *** amotoki has joined #openstack-infra | 11:38 | |
| *** jkilpatr has joined #openstack-infra | 11:38 | |
| *** jaypipes has joined #openstack-infra | 11:40 | |
| *** _vsaienko has joined #openstack-infra | 11:40 | |
| *** vsaienko has quit IRC | 11:40 | |
| *** yolanda has quit IRC | 11:41 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 11:44 |
| *** ccamacho1 has joined #openstack-infra | 11:45 | |
| *** claudiub has quit IRC | 11:47 | |
| *** ccamacho has quit IRC | 11:47 | |
| *** _Cyclone_ has joined #openstack-infra | 11:48 | |
| *** amotoki has quit IRC | 11:48 | |
| *** yolanda has joined #openstack-infra | 11:48 | |
| *** jordanP has joined #openstack-infra | 11:52 | |
| *** _vsaienko has quit IRC | 11:54 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 11:55 |
| *** saibarspeis has joined #openstack-infra | 11:57 | |
| *** alexchadin has joined #openstack-infra | 11:57 | |
| *** faizy has quit IRC | 11:59 | |
| *** vsaienko has joined #openstack-infra | 12:00 | |
| *** vsaienko has quit IRC | 12:01 | |
| *** jcoufal has joined #openstack-infra | 12:01 | |
| *** ociuhandu has joined #openstack-infra | 12:04 | |
| *** gordc has joined #openstack-infra | 12:04 | |
| *** yolanda has quit IRC | 12:05 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/project-config master: Fix enabling glance v1 for OSC https://review.openstack.org/445900 | 12:05 |
| *** ccamacho1 is now known as ccamacho|lunch | 12:06 | |
| frickler | AJaeger: ds-gate-cores: the earlier patch to fix OSC gates doesn't seem to have worked, please see ^^ for a fix (hopefully ;) | 12:06 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: remove logs from subnode https://review.openstack.org/445902 | 12:08 |
| *** kornicameister has quit IRC | 12:09 | |
| *** markvoelker has joined #openstack-infra | 12:11 | |
| *** MasterOfBugs has quit IRC | 12:12 | |
| *** gongysh has joined #openstack-infra | 12:12 | |
| *** funzo has joined #openstack-infra | 12:13 | |
| *** yolanda has joined #openstack-infra | 12:14 | |
| *** yolanda has quit IRC | 12:14 | |
| *** yolanda has joined #openstack-infra | 12:14 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 12:16 |
| *** markvoelker has quit IRC | 12:16 | |
| *** funzo has quit IRC | 12:17 | |
| *** dprince has joined #openstack-infra | 12:18 | |
| *** jaosorior_lunch is now known as jaosorior | 12:21 | |
| *** saibarspeis has quit IRC | 12:22 | |
| *** nadya has joined #openstack-infra | 12:23 | |
| *** eroux has quit IRC | 12:24 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] oooq multinode: propagate STABLE_BRANCH to subnodes https://review.openstack.org/445866 | 12:24 |
| *** shu-mutou-AWAY has quit IRC | 12:24 | |
| *** hashar is now known as hasharLunch | 12:29 | |
| *** Julien-zte has quit IRC | 12:32 | |
| *** askb has quit IRC | 12:34 | |
| *** baoli has joined #openstack-infra | 12:36 | |
| *** markvoelker has joined #openstack-infra | 12:38 | |
| *** Wei_Liu has joined #openstack-infra | 12:38 | |
| *** baoli has quit IRC | 12:39 | |
| *** hasharLunch is now known as hashar | 12:39 | |
| *** baoli has joined #openstack-infra | 12:40 | |
| *** kzaitsev_mb has quit IRC | 12:41 | |
| *** Julien-zte has joined #openstack-infra | 12:41 | |
| *** wei_liu_ has joined #openstack-infra | 12:41 | |
| Wei_Liu | hi | 12:42 |
| *** wei_liu_ has left #openstack-infra | 12:42 | |
| *** eroux has joined #openstack-infra | 12:42 | |
| *** kgiusti has joined #openstack-infra | 12:43 | |
| *** shardy is now known as shardy_lunch | 12:43 | |
| *** ym200 has joined #openstack-infra | 12:44 | |
| *** baoli has quit IRC | 12:44 | |
| Wei_Liu | I am build local infra using zuul/gerrit/jenkins, I want to send the email to committer if job failed, how can I configure the "to" address in zuul pipeline? | 12:46 |
| *** jpich has quit IRC | 12:46 | |
| *** jaosorior has quit IRC | 12:46 | |
| *** jpich_ has joined #openstack-infra | 12:46 | |
| *** dalvarez has quit IRC | 12:47 | |
| *** jaosorior has joined #openstack-infra | 12:47 | |
| mordred | Wei_Liu: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n89 has an example | 12:48 |
| *** rlandy has joined #openstack-infra | 12:49 | |
| mordred | infra-root: I need to cut a shade release to help people out with the pbr 2.0 transition - I expect no impact to nodepool | 12:50 |
| *** trown|outtypewww is now known as trown | 12:50 | |
| *** jpich_ is now known as jpich | 12:51 | |
| *** erhudy has joined #openstack-infra | 12:52 | |
| *** eroux has quit IRC | 12:53 | |
| *** dalvarez has joined #openstack-infra | 12:53 | |
| *** gouthamr has quit IRC | 12:54 | |
| fungi | mordred: somehow i thought you already had. please go ahead | 12:56 |
| mordred | fungi: thanks - done! | 12:56 |
| *** faizy has joined #openstack-infra | 12:56 | |
| *** mriedem has joined #openstack-infra | 12:57 | |
| *** ym200 has quit IRC | 12:57 | |
| fungi | Wei_Liu: example from our running configuration: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n92 | 12:57 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Add oooq multinode jobs to ci status page https://review.openstack.org/445937 | 12:59 |
| cmurphy | fungi: we were having issues again with docs.o.o this morning | 12:59 |
| fungi | cmurphy: i saw it in scrollback | 12:59 |
| cmurphy | fungi: the load average didn't look spikey the way it usually does, could that be evidence of your noisy neighbors theory? | 13:00 |
| fungi | have my doubts the apache restart actually does much, since we generally see it recover on its own after a few hours | 13:00 |
| cmurphy | yeah, with or without restarts it comes and goes | 13:00 |
| *** dtardivel has joined #openstack-infra | 13:00 | |
| *** eroux has joined #openstack-infra | 13:00 | |
| *** ianychoi has quit IRC | 13:01 | |
| fungi | cmurphy: yeah, we've seen similar behavior with other instances where either network or disk contention causes apache to take a long time getting responses back to requestors | 13:01 |
| Wei_Liu | fungi, thanks | 13:01 |
| Wei_Liu | mordred: thanks | 13:02 |
| fungi | cmurphy: sometimes if we open a trouble-ticket with the provider they're able to rebalance things or shut down a problem customer, but usually we just end up having to redeploy the server and hope it gets scheduled somewhere quieter | 13:02 |
| *** links has quit IRC | 13:02 | |
| fungi | up-side is that replacing files01 with a files02 should be dead simple since there's nothing to migrate. all configuration is in puppet, all data in afs | 13:03 |
| Wei_Liu | fungi: I found "to: release-job-failures@lists.openstack.org", this is a mail list? | 13:04 |
| EmilienM | good morning infra, I have some patches in project config waiting for review to make changes in tripleo zuul layout: https://review.openstack.org/#/q/owner:%22Emilien+Macchi+%253Cemilien%2540redhat.com%253E%22+project:openstack-infra/project-config+status:open - thanks! | 13:04 |
| *** camunoz_ has joined #openstack-infra | 13:04 | |
| fungi | Wei_Liu: yes, in that example, jobs run in the release-post pipeline send results to that mailing list when they fail | 13:05 |
| *** ym200 has joined #openstack-infra | 13:05 | |
| cmurphy | fungi: I can spin up a quick change in manifests/site.pp to get the puppet part rolling if that would be helpful | 13:06 |
| *** jcoufal has quit IRC | 13:06 | |
| Wei_Liu | fungi: so, everyone in the list will receive the failure report? | 13:06 |
| fungi | Wei_Liu: in that particular case, yes | 13:07 |
| *** felipemonteiro has joined #openstack-infra | 13:07 | |
| fungi | cmurphy: sure, that would be great but maybe let's get input from a couple other infra-root admins before we head down that path since it's a bit of a heavy hammer solution | 13:07 |
| cmurphy | fungi: sure thing | 13:08 |
| mordred | fungi: you know - files01 is one of the few thigns we run that could actually fall into the "cattle" concept people always like talking about | 13:08 |
| cmurphy | it is remarkably well suited for that | 13:09 |
| fungi | i agree | 13:09 |
| *** eharney has joined #openstack-infra | 13:09 | |
| fungi | we can even keep files01 up for however long we want and flip dns back and forth to test the theory | 13:09 |
| *** cshastri has quit IRC | 13:09 | |
| *** ym200 has quit IRC | 13:10 | |
| *** pgadiya has quit IRC | 13:11 | |
| Wei_Liu | fungi: can I set "to: submitter.email", let the report just send failure report the submitter? | 13:11 |
| *** mdrabe has joined #openstack-infra | 13:11 | |
| fungi | Wei_Liu: what do you mean by "the submitter"? | 13:11 |
| Wei_Liu | fungi: I mean the gerrit patch submitter, | 13:12 |
| *** annegentle has joined #openstack-infra | 13:12 | |
| *** erikmwilson has joined #openstack-infra | 13:13 | |
| *** baoli has joined #openstack-infra | 13:13 | |
| *** claudiub has joined #openstack-infra | 13:14 | |
| Wei_Liu | fungi: the submitter submit gerrit review which will trigger jobs in the pipeline, if job failed, send the report to submitter | 13:14 |
| mordred | Wei_Liu: no, we do not have that feature - although now that you mention it it's not a terrible feature idea | 13:15 |
| *** matt-borland has joined #openstack-infra | 13:15 | |
| mordred | jeblair: ^^ for v3, maybe we should expose some info about the patch author in the zuul variables we pass to playbooks | 13:15 |
| *** annegentle has quit IRC | 13:17 | |
| *** ccamacho|lunch is now known as ccamacho | 13:17 | |
| *** jcoufal has joined #openstack-infra | 13:18 | |
| Wei_Liu | mordred: yes, | 13:18 |
| *** ym200 has joined #openstack-infra | 13:20 | |
| *** saibarspeis has joined #openstack-infra | 13:22 | |
| *** gouthamr has joined #openstack-infra | 13:22 | |
| *** prateek__ has quit IRC | 13:22 | |
| *** tongli has joined #openstack-infra | 13:23 | |
| *** xyang1 has joined #openstack-infra | 13:23 | |
| *** Hal has joined #openstack-infra | 13:24 | |
| *** saibarspeis has quit IRC | 13:24 | |
| *** psachin has quit IRC | 13:24 | |
| *** ym200_ has joined #openstack-infra | 13:25 | |
| *** Goneri has joined #openstack-infra | 13:25 | |
| *** ym200 has quit IRC | 13:25 | |
| *** jtomasek has quit IRC | 13:25 | |
| *** ym200 has joined #openstack-infra | 13:27 | |
| *** jtomasek has joined #openstack-infra | 13:28 | |
| *** rhallisey has quit IRC | 13:29 | |
| *** winggundamth has joined #openstack-infra | 13:29 | |
| *** ym200_ has quit IRC | 13:29 | |
| *** yujunz has joined #openstack-infra | 13:29 | |
| *** aysyd has joined #openstack-infra | 13:29 | |
| *** yujunz-zte has joined #openstack-infra | 13:31 | |
| sc` | how to get jjb to divulge its secrets? i'm attempting to test a job i wrote in preparation for spamming people with reviews, and all that i get, no matter which test, is a friendly "INFO:jenkins_jobs.builder:Number of jobs generated: 0" | 13:31 |
| *** ym200 has quit IRC | 13:31 | |
| sc` | -l debug is equally opaque | 13:31 |
| *** ym200 has joined #openstack-infra | 13:31 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove remaining apscheduler variables https://review.openstack.org/445719 | 13:31 |
| *** yujunz has quit IRC | 13:32 | |
| *** yujunz has joined #openstack-infra | 13:33 | |
| *** yujunz-zte has quit IRC | 13:34 | |
| *** amoralej is now known as amoralej|lunch | 13:34 | |
| *** ym200 has quit IRC | 13:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:36 | |
| *** fandi has joined #openstack-infra | 13:37 | |
| *** camunoz_ has quit IRC | 13:37 | |
| *** trown is now known as trown|brb | 13:38 | |
| pabelanger | mordred: ++ | 13:38 |
| *** ym200 has joined #openstack-infra | 13:38 | |
| *** ym200 has quit IRC | 13:40 | |
| *** kzaitsev_mb has quit IRC | 13:40 | |
| *** ym200 has joined #openstack-infra | 13:41 | |
| pabelanger | fungi: cmurphy: I think the next step clarkb mentioned was to maybe tune files.o.o via apache2.conf. But, I'm not sure which settings to use actually. | 13:41 |
| *** ym200 has quit IRC | 13:41 | |
| fungi | pabelanger: well, we still have no explanation for the strange times its behavior drags down | 13:42 |
| *** ym200 has joined #openstack-infra | 13:42 | |
| fungi | to suggest that tuning apache would in any way be a sensible "solution" to the problem | 13:42 |
| *** nadya has quit IRC | 13:43 | |
| pabelanger | fungi: understood, wasn't sure if that would help. Or if we seen large amount of traffic hitting apache during that time | 13:44 |
| pabelanger | I haven't looked into the issue at all | 13:44 |
| cmurphy | pabelanger: ianw had some thoughts about tuning MaxRequestWorkers | 13:46 |
| pabelanger | cmurphy: right, ianw. Thank you | 13:46 |
| *** jtomasek has quit IRC | 13:46 | |
| cmurphy | pabelanger: which would actually mean tuning ServerLimit according to https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers | 13:46 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove noisy log line https://review.openstack.org/445964 | 13:49 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/storyboard master: Increase project and project group name length limitation https://review.openstack.org/445965 | 13:51 |
| *** trown|brb is now known as trown | 13:53 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools master: fix mail sending https://review.openstack.org/445967 | 13:53 |
| *** markvoelker_ has joined #openstack-infra | 13:54 | |
| *** winggundamth has quit IRC | 13:55 | |
| *** markvoelker has quit IRC | 13:55 | |
| fungi | problem is, without knowing what's triggering the issue, the most we can do is try something (replace the server, adjust apache settings) and then wait for the issue to not recur and assume we probably took care of it | 13:56 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Populate requestor for min-ready requests https://review.openstack.org/445970 | 13:56 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: oooq multinode: fix for stable branches https://review.openstack.org/445866 | 13:57 |
| fungi | we don't have a reproducer other than waiting to see what happens, which granted it's become a daily occurrence recently but was less frequent prior to a few weeks ago | 13:57 |
| *** wolverineav has joined #openstack-infra | 13:58 | |
| cmurphy | it's still a problem basically every day, I don't think it's really less frequent | 14:00 |
| cmurphy | oh i think i misread the end of your sentence, nm | 14:00 |
| *** wolverin_ has joined #openstack-infra | 14:01 | |
| *** zz_dimtruck is now known as dimtruck | 14:01 | |
| mordred | yah - it's definitely a frequent problem, but we still have no clue what the cause or problem is :( | 14:01 |
| mordred | fungi: it might be interesting to spin up a files02 and then see if files02 has the same problem with files01 does | 14:02 |
| *** yujunz has quit IRC | 14:02 | |
| mordred | fungi: which could _maybe_ be a way to characterize if we're seeing an AFS system level problem or a problem on the files server | 14:02 |
| *** yujunz has joined #openstack-infra | 14:03 | |
| *** zhurong_ has joined #openstack-infra | 14:03 | |
| *** wolverineav has quit IRC | 14:04 | |
| *** alexchadin has quit IRC | 14:05 | |
| *** links has joined #openstack-infra | 14:06 | |
| fungi | which brings up a good point... i still think it would make sense to analyze the logs and see if maybe someone's got a daily download of all the docs site contents going or something... this is pretty suspicious network traffic: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2362&rra_id=all | 14:06 |
| *** zhurong_ has quit IRC | 14:06 | |
| fungi | actually, maybe not the docs site, it could just be an innocent bystander | 14:06 |
| *** knangia has joined #openstack-infra | 14:07 | |
| fungi | could we be backing up afs nightly? | 14:07 |
| *** baoli has quit IRC | 14:07 | |
| *** ihrachys has joined #openstack-infra | 14:08 | |
| fungi | yeah, the traffic graphs for files01 don't show anywhere near that volume | 14:08 |
| cmurphy | fungi: the times don't quite line up, the problem usually happens around 7-9:00 UTC | 14:09 |
| fungi | cmurphy: you're right, the issues seem to start after those spikes subside, so likely unrelated | 14:09 |
| *** shardy_lunch is now known as shardy | 14:10 | |
| fungi | though it does also look like we might be maxing out our bandwifth for afs01.dfw (independent of this) | 14:10 |
| *** baoli has joined #openstack-infra | 14:10 | |
| fungi | it's an 8gb flavor, don't recall what they're setting the bw cap to for those | 14:10 |
| jeblair | fungi, pabelanger: i suggested that we tune the apache variables because, simply, even under normal circumstances that server is near it's apache limits, but *nowhere* near its actual system limits. | 14:12 |
| fungi | i agree that's reasonable, regardless of whether the performance issues we're seeing are related to apache being a bottleneck | 14:12 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: oooq multinode: fix for stable branches https://review.openstack.org/445866 | 14:13 |
| *** dprince has quit IRC | 14:14 | |
| *** funzo_ has joined #openstack-infra | 14:14 | |
| *** john-davidge has quit IRC | 14:14 | |
| *** erikmwilson has quit IRC | 14:15 | |
| *** erikmwilson has joined #openstack-infra | 14:15 | |
| mordred | ++ | 14:17 |
| *** funzo_ has quit IRC | 14:19 | |
| openstackgerrit | Merged openstack-infra/release-tools master: fix mail sending https://review.openstack.org/445967 | 14:19 |
| fungi | found the culprit for the traffic spikes on afs01.dfw... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3209&rra_id=all | 14:19 |
| fungi | apparently some part of our mirror updating needs to read very large volumes of data out of afs | 14:19 |
| *** pradk has joined #openstack-infra | 14:20 | |
| mordred | oh. I wonder if one of the mirror processes is stating the files or something | 14:20 |
| *** skelso has joined #openstack-infra | 14:20 | |
| mordred | rsync. rsync is going to check to see what files are already there and if they're up to date | 14:20 |
| fungi | the volume there looks like more than a stat. maybe reprepro is rereading all files for the benefit of checksumming? | 14:21 |
| mordred | it shouldn't be - reprepro should be the best of them - it keeps a database and checks that against the remote packages list | 14:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create run-cover role https://review.openstack.org/441332 | 14:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable https://review.openstack.org/441441 | 14:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder https://review.openstack.org/441547 | 14:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Rename prepare-workspace role to bootstrap https://review.openstack.org/441440 | 14:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add run-docs role and tox-docs job https://review.openstack.org/441345 | 14:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role https://review.openstack.org/441617 | 14:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs https://review.openstack.org/441467 | 14:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job https://review.openstack.org/441609 | 14:22 |
| *** dprince has joined #openstack-infra | 14:23 | |
| mordred | although yah - the way we're using rsync also shouldn't be doing checksums | 14:23 |
| *** john-davidge has joined #openstack-infra | 14:24 | |
| mordred | fungi: maybe it's worth running each mirror type by hand and watching the traffic to see which one it is | 14:24 |
| *** cody-somerville has joined #openstack-infra | 14:24 | |
| fungi | well, first we can try to match spikes up to start/stop times i guess | 14:25 |
| *** EricGonc_ has joined #openstack-infra | 14:26 | |
| fungi | hrm... i don't think it's necessarily any of them | 14:26 |
| fungi | they're all either triggered every 5 minutes or every 2 hours | 14:26 |
| *** EricGonc_ has quit IRC | 14:26 | |
| fungi | not daily | 14:27 |
| mordred | oh good. so _sometimes_ there is a lot of data | 14:27 |
| *** EricGonc_ has joined #openstack-infra | 14:27 | |
| fungi | that makes it a lot harder to track down | 14:27 |
| *** esikachev has quit IRC | 14:27 | |
| *** rbrndt__ has joined #openstack-infra | 14:28 | |
| *** amotoki has joined #openstack-infra | 14:29 | |
| fungi | though whatever it is, is going on again right now from the look of the graphs | 14:31 |
| fungi | bandersnatch has been running for almost 40 minutes | 14:32 |
| fungi | according to ps | 14:32 |
| fungi | that start time coincides almost perfectly with the start of the current bandwidth spike | 14:33 |
| *** Julien-zte has quit IRC | 14:35 | |
| *** coreyob has quit IRC | 14:35 | |
| fungi | though npm-mirror-update has also been running for about 30 minutes | 14:35 |
| fungi | could be they're dogpiling on one another | 14:35 |
| *** marst has quit IRC | 14:36 | |
| *** Julien-zte has joined #openstack-infra | 14:36 | |
| *** Julien-zte has quit IRC | 14:36 | |
| fungi | npm-mirror-update just finished | 14:38 |
| *** dave-mccowan has joined #openstack-infra | 14:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 14:38 | |
| *** vikrant has quit IRC | 14:38 | |
| *** jordanP has quit IRC | 14:39 | |
| *** jordanP has joined #openstack-infra | 14:39 | |
| *** rook is now known as rook-mtg | 14:39 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove noisy log line https://review.openstack.org/445964 | 14:39 |
| fungi | waiting to see what bandersnatch logs having done when this completes. its previous run took just over one minute | 14:40 |
| *** dasanind has quit IRC | 14:40 | |
| *** dasanind has joined #openstack-infra | 14:41 | |
| *** yujunz has quit IRC | 14:41 | |
| *** saibarspeis has joined #openstack-infra | 14:41 | |
| *** d0ugal has quit IRC | 14:42 | |
| *** links has quit IRC | 14:42 | |
| *** erikmwilson has quit IRC | 14:42 | |
| *** eroux has quit IRC | 14:42 | |
| *** marst has joined #openstack-infra | 14:42 | |
| *** d0ugal has joined #openstack-infra | 14:42 | |
| *** cody-somerville has quit IRC | 14:43 | |
| *** csomerville has joined #openstack-infra | 14:43 | |
| *** hongbin has joined #openstack-infra | 14:43 | |
| *** erikmwilson has joined #openstack-infra | 14:44 | |
| *** dhajare has quit IRC | 14:45 | |
| *** tphummel has joined #openstack-infra | 14:45 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Tweak mpm_worker settings https://review.openstack.org/445991 | 14:45 |
| *** yujunz has joined #openstack-infra | 14:46 | |
| *** burgerk has joined #openstack-infra | 14:46 | |
| fungi | okay, bandersnatch just finished | 14:48 |
| *** saibarspeis has quit IRC | 14:49 | |
| *** eharney_ has joined #openstack-infra | 14:50 | |
| *** eharney has quit IRC | 14:50 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config master: experimental multinode job for sahara https://review.openstack.org/442382 | 14:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Update wait_for_threads https://review.openstack.org/445994 | 14:51 |
| *** gongysh has quit IRC | 14:51 | |
| *** saibarspeis has joined #openstack-infra | 14:52 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove ready-script support https://review.openstack.org/445567 | 14:52 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Stop writing nodepool bash variable on nodes https://review.openstack.org/445572 | 14:52 |
| fungi | total time was about 52 minutes, but the bandersnatch log pauses for over 30 minutes at one point and 10 minutes at another... seemingly for no reason (the packages being downloaded were relatively tiny) | 14:53 |
| *** Sukhdev_ has joined #openstack-infra | 14:55 | |
| *** lin_yang has joined #openstack-infra | 14:55 | |
| *** dhinesh has joined #openstack-infra | 14:55 | |
| *** tonytan4ever has joined #openstack-infra | 14:55 | |
| *** yamahata has joined #openstack-infra | 14:55 | |
| mordred | fungi: why do I have a feeling that we're not going to enjoy whatever the answer happens to be? | 14:56 |
| fungi | i guess it's possible it was blocked on the npm mirroring activity, unfortunately the npm mirror update logs are extremely sparse | 14:56 |
| *** esikachev has joined #openstack-infra | 14:56 | |
| *** eharney__ has joined #openstack-infra | 14:57 | |
| *** eharney_ has quit IRC | 14:57 | |
| *** eharney__ is now known as eharney | 14:57 | |
| fungi | npm mirroring ran from 14:05 to 14:30 | 14:57 |
| *** yujunz has quit IRC | 14:58 | |
| mordred | fungi: if only we had some sort of job-queueing system where we could run one and only one instance of an action on a given node at a time | 14:59 |
| mordred | jeblair: do you think we could write one of those? | 14:59 |
| fungi | gaps in the bandersnatch log are 13:55:34-14:27:53 and 14:27:58-14:46:00 (suggesting that it was probably working very slowly and got a word in edgewise for a few seconds over that half hour) | 14:59 |
| *** esikachev has quit IRC | 15:00 | |
| fungi | so the bandersnatch slowdown seems to have started too early to be blamed on npm, and ended much later (relatively speaking) | 15:01 |
| *** isaacb has quit IRC | 15:01 | |
| *** jtomasek has joined #openstack-infra | 15:01 | |
| *** john-davidge has quit IRC | 15:01 | |
| *** amoralej|lunch is now known as amoralej | 15:02 | |
| *** mhickey has quit IRC | 15:02 | |
| *** esikachev has joined #openstack-infra | 15:02 | |
| jeblair | catching up | 15:03 |
| jeblair | that was a lot of backscroll to read for a joke, but i'm glad i read it anyway. :) | 15:05 |
| *** czeq has joined #openstack-infra | 15:07 | |
| *** john-davidge has joined #openstack-infra | 15:08 | |
| jeblair | fungi: it looks like the bw spike starts between 13:55 and 14:00 ? | 15:08 |
| fungi | yeah | 15:09 |
| fungi | the spike also dipped when bandersnatch finished, and has now started back up again at the same time bandersnatch did (and it's been running again now for almost 20 minutes) | 15:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Disable functional tests for nodepool gearman-zk-shim https://review.openstack.org/446003 | 15:09 |
| mordred | jeblair: maybe I should tag jokes better | 15:09 |
| jeblair | fungi: oh did bandersnatch start before 13:55? thinking that may be it? | 15:09 |
| mordred | jeblair: ^^ does that patch look right to you for what we talked about for the shim? | 15:09 |
| fungi | jeblair: bandersnatch started at 13:55 the first time and ended at 14:47, then started up again at 14:50 ans it still running | 15:10 |
| *** Sukhdev_ has quit IRC | 15:10 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Make decision wheter to build images in CI or not https://review.openstack.org/442139 | 15:10 |
| *** Sukhdev_ has joined #openstack-infra | 15:11 | |
| jeblair | fungi: so more or less exactly corresponding to the spike? | 15:11 |
| fungi | yes | 15:11 |
| fungi | plus or minus the 5-minute sample granularity anyway | 15:12 |
| jeblair | apparently the rubygems mirror is also running? but doesn't log anything? | 15:12 |
| *** erikmwilson has quit IRC | 15:12 | |
| fungi | the rubygems mirror script seems to have been started under a screen session some time last year | 15:12 |
| jeblair | pstree 14347 | 15:12 |
| jeblair | flock───bash───cat | 15:12 |
| jeblair | fungi: ah, that explains that so much better than what i was thinking. | 15:13 |
| jeblair | so we can disregard | 15:13 |
| fungi | and, er, yeah isn't the actual script just some cat holding the lock blocking it | 15:13 |
| *** tongli has quit IRC | 15:13 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: New ovb playbook and settings https://review.openstack.org/431567 | 15:13 |
| *** erikmwilson has joined #openstack-infra | 15:14 | |
| jeblair | oh it's frida | 15:14 |
| *** rhallisey has joined #openstack-infra | 15:14 | |
| jeblair | bandersnatch is reading the contents of every frida release artifact | 15:14 |
| jeblair | they are each about 20M | 15:14 |
| *** EricGonczer_ has joined #openstack-infra | 15:15 | |
| fungi | does it release frequently? maybe they just pushed two releases of it in a row? | 15:15 |
| *** funzo_ has joined #openstack-infra | 15:15 | |
| jeblair | they do indeed | 15:15 |
| *** isaacb has joined #openstack-infra | 15:15 | |
| *** EricGonc_ has quit IRC | 15:15 | |
| *** Sukhdev_ has quit IRC | 15:15 | |
| jeblair | so i think every traffic spike is a frida release, as each one causes bandersnatch to read all of the released files for the package | 15:15 |
| fungi | aha, so this is making more sense. i looked at the pypi page for frida trying to suss it out, but the download size alone didn't seem relevant since 1) i would expect that to be write related, and 2) the downloads went very quickly according to the log | 15:16 |
| mordred | so - there is a potential improvement to be made in bandersnatch to maybe not read all of the files for a package when it's getting a new release | 15:16 |
| pabelanger | rubygems we can remove, we cannot mirror them right now | 15:16 |
| fungi | mordred: sounds like it, yes | 15:16 |
| pabelanger | it dumps all the gems into a single directory | 15:17 |
| fungi | okay, that mystery is solved, and while it's annoying it doesn't seem to have actually caused any noticeable user-facing impact | 15:17 |
| *** saibarspeis has quit IRC | 15:17 | |
| *** mhickey has joined #openstack-infra | 15:17 | |
| fungi | back to the anomalous behavior for files01, the most pronounced symptom showing in cacti seems to be the established tcp connection count, and the graphs for that show a definite weekday-only trend (with a lighter friday) starting a little over a month ago | 15:17 |
| jeblair | it's hard to look now, with the sharding by sha1 hash, but i think the last time i looked, a *substantial* portion of the pypi archive was frida. | 15:18 |
| *** funzo_ has quit IRC | 15:20 | |
| jeblair | i mean, we transferred 43GB from our fileserver for that. | 15:20 |
| mordred | wow | 15:20 |
| fungi | crazy | 15:21 |
| fungi | "Inject JavaScript to explore native apps on Windows, macOS, Linux, iOS, Android, and QNX" | 15:21 |
| * fungi has no idea how those words fit together | 15:21 | |
| mordred | maybe they're using pypi as a blob store | 15:22 |
| jeblair | fungi: yeah, i'm leaning toward apache tuning because we're "busy" (which may include some amount of bot activity, but hey, them's the breaks). it could be noisy neighbor, though when i logged in a few nights ago, it didn't seem very slow, but then, i wasn't doing many disk writes. | 15:22 |
| *** coreyob has joined #openstack-infra | 15:23 | |
| mnaser | i'm working a bit to improve the stability of the kolla ci -- it seems like it depends on published master tarballs for testing, going over project-config, i couldnt see if it is published on every commit or if its more of a periodic task | 15:23 |
| jeblair | fungi: (but reading log files and running links, etc, was responsive) | 15:23 |
| fungi | we moved the docs site to files01 on january 10 according to our #status log | 15:23 |
| jeblair | mnaser: should be every commit | 15:23 |
| mnaser | the reason i ask is because it seems like some rootwrap filters which were added 7 days ago are missing resulting in build failures | 15:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/gearman-zk-shim: Stub out methods needed for zk client https://review.openstack.org/435964 | 15:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/gearman-zk-shim: Hardcode min-ready to zero https://review.openstack.org/435963 | 15:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/gearman-zk-shim: Pass zk object in to provider manager https://review.openstack.org/446012 | 15:23 |
| fungi | so it was being served exclusively from there for at least a few weeks before this started up | 15:24 |
| mnaser | thanks jeblair .. i'll try to do a bit more invesigation surrounding this then | 15:24 |
| pabelanger | mnaser: jeblair: yes, there is a publisher for kolla that uploads some images to tarballs.o.o | 15:24 |
| mordred | mnaser: cool - we definitely expect the master tarballs to have content that matches the master branch of the repo | 15:24 |
| jeblair | mnaser: we can conjure up the logs for a specific build/upload if needed. just need the repo and git sha of the commit in question. | 15:25 |
| mnaser | thank you, we're trying to adopt kolla so we want to (hopefully) we can make jobs voting instead | 15:25 |
| fungi | well, these aren't master branch tarballs of kolla, but rather tarballs of docker image pieces or something from what i understand | 15:25 |
| jeblair | oh that may be different :) | 15:25 |
| mnaser | ex: http://logs.openstack.org/03/445803/1/check/gate-kolla-dsvm-deploy-centos-source-centos-7-nv/fcbce65/logs/kolla/neutron/neutron-l3-agent.txt.gz (as well as ubuntu source and oraclelinux source failed here) | 15:25 |
| mnaser | https://github.com/openstack/neutron/commit/3b22541a2aa9a5b06e2bff256701dbe24554c17c | 15:25 |
| mnaser | this should have addressed it but i assume thats not there, so im working on getting some local images and try to replicate this | 15:26 |
| jeblair | mnaser: you're saying the neutron commit is missing from something? | 15:26 |
| mnaser | what im *guessing* is for some reason, the rootwrap config is incomplete and therefore the job is failing (because the jobs installing from packages work fine) | 15:27 |
| mordred | mnaser: we also keep talking about getting a registry set up that things can publish to / read from - but that's a bit of work and hasn't made it to the top of anybody's stack yet | 15:27 |
| fungi | i think it's the result of the kolla-dsvm-build-.* jobs they have defined in their post pipeline which are being consumed http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n11287 | 15:27 |
| mnaser | http://logs.openstack.org/03/445803/1/check/gate-kolla-dsvm-deploy-centos-source-centos-7-nv/fcbce65/logs/ -- looks like system_log and system_logs is copied twice too | 15:28 |
| fungi | publishing to http://tarballs.openstack.org/kolla/images/ | 15:28 |
| mnaser | ooo thats interesting | 15:28 |
| mnaser | per build would be nice but i can imagine that would start using up a lot of space :( that way we can pull down a build and debug locally | 15:28 |
| *** dizquierdo has quit IRC | 15:29 | |
| *** xyang1 has quit IRC | 15:29 | |
| pabelanger | right, in an effort to reduce build times, they store images on tarballs.o.o, then download them each new job, stand up local registry, and install images from there | 15:29 |
| fungi | fun that every change which merges to kolla generates and replaces 31.5gb of data on tarballs.o.o | 15:29 |
| *** nadya has joined #openstack-infra | 15:29 | |
| *** xyang1 has joined #openstack-infra | 15:29 | |
| mnaser | thats no good.. | 15:29 |
| *** yujunz has joined #openstack-infra | 15:29 | |
| mnaser | pabelanger i think that could be the cause of the issue | 15:29 |
| fungi | (rough count based on skimming apache's reported file sizes) | 15:29 |
| mnaser | if the images are stored, they might be using a cached/older base of neutron | 15:30 |
| pabelanger | mnaser: correct | 15:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Ectomy some Jenkins out of the docs https://review.openstack.org/436452 | 15:30 |
| fungi | oh, i take that back, i was combining ocata and master sizes | 15:30 |
| fungi | so more like half of that | 15:30 |
| *** nadya has quit IRC | 15:31 | |
| mordred | pabelanger: feel like looking at https://review.openstack.org/#/c/446003/ ? | 15:31 |
| mnaser | okay well i just tried building it locally and its still failing | 15:32 |
| mnaser | so i guess caching isnt really the issue | 15:32 |
| pabelanger | mordred: +2 | 15:32 |
| mnaser | kolla meeting in 30 minutes so i hope that i can get more people onboard effective testing instead of just wasting compute cycles on non voting jobs :( | 15:32 |
| pabelanger | mnaser: it should only upload good images, meaning the job was able to build them | 15:33 |
| mnaser | cause i just rebuilt master and neutron-l3-agent from source just doesn't work (and seems like it hasn't been for a looong time) | 15:33 |
| pabelanger | mnaser: but, it is possible for the images to go stale somewhat quickly. | 15:33 |
| mnaser | so i mean.. either drop the source jobs or we fix them | 15:33 |
| *** amotoki has quit IRC | 15:33 | |
| *** oomichi has quit IRC | 15:36 | |
| *** armax has joined #openstack-infra | 15:37 | |
| sshnaidm | cores, please review patch for tripleo to enable newton jobs: https://review.openstack.org/#/c/445875/ | 15:38 |
| *** Jeffrey4l has quit IRC | 15:38 | |
| *** aviau has quit IRC | 15:39 | |
| *** Jeffrey4l has joined #openstack-infra | 15:39 | |
| *** aviau has joined #openstack-infra | 15:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable functional tests for nodepool gearman-zk-shim https://review.openstack.org/446003 | 15:39 |
| *** akscram has quit IRC | 15:40 | |
| *** dhinesh has quit IRC | 15:40 | |
| *** sandanar has joined #openstack-infra | 15:41 | |
| *** oomichi has joined #openstack-infra | 15:41 | |
| *** patriciadomin has quit IRC | 15:42 | |
| *** persia has quit IRC | 15:42 | |
| *** patriciadomin has joined #openstack-infra | 15:43 | |
| dtantsur | hi, quick question: will CI work correctly, if I use 2 Depends-On on different (and unrelated) patches from the same project? | 15:44 |
| *** esikachev has quit IRC | 15:44 | |
| *** persia has joined #openstack-infra | 15:45 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test ovb-oooq https://review.openstack.org/431103 | 15:45 |
| *** YorikSar has quit IRC | 15:47 | |
| pabelanger | dtantsur: might be better to rebase patch A onto B | 15:47 |
| *** YorikSar has joined #openstack-infra | 15:47 | |
| *** patriciadomin has quit IRC | 15:47 | |
| *** patriciadomin has joined #openstack-infra | 15:47 | |
| dtantsur | yeah, this is what I'd prefer to avoid. ok, I'll wait for either of them to merge. thanks! | 15:47 |
| *** tphummel has quit IRC | 15:48 | |
| fungi | dtantsur: using depends-on between changes in the same repo is usually discouraged over relying on sequencing the changes using git. it will generally work, but ends up just sequencing them in an arbitrary order (and it's also way easier to introduce dependency loops that way) | 15:49 |
| fungi | so if you have change C which depends on A and B, sequencing them using git parentage allows you to determine the order in which A and B get applied while having two depends-on headers in C referring to both A and B will not | 15:50 |
| *** akscram has joined #openstack-infra | 15:50 | |
| *** kzaitsev_mb has quit IRC | 15:51 | |
| fungi | but also since zuul considers both git parentage and depends-on headers, it's much easier to accidentally make a change a git parent and a depends-on resulting in an unresolvable loop (at which point zuul won't test or comment on any of the changes involved) | 15:51 |
| *** tosky has quit IRC | 15:51 | |
| dtantsur | ah, good. fortunately enough, in this case I don't care about ordering. | 15:52 |
| * fungi heads out to grab some quick lunch. bbiaw | 15:52 | |
| *** tosky has joined #openstack-infra | 15:53 | |
| clarkb | so much scrollback | 15:54 |
| *** skelso has quit IRC | 15:55 | |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable DPDK testing on gate https://review.openstack.org/445640 | 15:55 |
| *** skelso has joined #openstack-infra | 15:55 | |
| *** david-lyle has joined #openstack-infra | 15:56 | |
| *** fandi has quit IRC | 15:57 | |
| *** Sukhdev_ has joined #openstack-infra | 15:58 | |
| *** vhosakot has joined #openstack-infra | 15:59 | |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable DPDK testing on gate https://review.openstack.org/445640 | 15:59 |
| *** hashar is now known as hasharMeeting | 16:01 | |
| *** hasharMeeting has quit IRC | 16:02 | |
| *** vsaienk0 has quit IRC | 16:02 | |
| *** jaosorior has quit IRC | 16:02 | |
| *** mhickey has quit IRC | 16:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test new ovb with images building https://review.openstack.org/445228 | 16:02 |
| *** jkilpatr_ has joined #openstack-infra | 16:03 | |
| *** hongbin has quit IRC | 16:05 | |
| *** trown is now known as trown|lunch | 16:05 | |
| * frickler needs a second infra core to fix OSC gate https://review.openstack.org/445900 | 16:05 | |
| *** dhajare has joined #openstack-infra | 16:06 | |
| *** jkilpatr has quit IRC | 16:06 | |
| *** EricGonczer_ has quit IRC | 16:07 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] quickstart: test for missing failed deployment log https://review.openstack.org/446043 | 16:07 |
| *** EricGonczer_ has joined #openstack-infra | 16:07 | |
| *** isaacb has quit IRC | 16:08 | |
| *** tphummel has joined #openstack-infra | 16:10 | |
| *** rossella_s has quit IRC | 16:11 | |
| *** esikachev has joined #openstack-infra | 16:11 | |
| *** rossella_s has joined #openstack-infra | 16:11 | |
| *** yujunz has quit IRC | 16:12 | |
| EmilienM | could we have a review on https://review.openstack.org/#/c/445875/ please? it's blocking TripleO jobs in stable branches, thanks | 16:15 |
| *** mhickey has joined #openstack-infra | 16:15 | |
| *** funzo_ has joined #openstack-infra | 16:16 | |
| *** skelso has quit IRC | 16:17 | |
| *** rlandy is now known as rlandy|afk | 16:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix enabling glance v1 for OSC https://review.openstack.org/445900 | 16:19 |
| *** vsaienk0 has joined #openstack-infra | 16:20 | |
| *** funzo_ has quit IRC | 16:20 | |
| *** andreas_s has quit IRC | 16:20 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Containerized undercloud - install python-heat-agents-ansible https://review.openstack.org/446052 | 16:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Rename zuul-launcher to zuul-executor https://review.openstack.org/445594 | 16:21 |
| *** rossella_s has quit IRC | 16:21 | |
| *** sree has quit IRC | 16:22 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add per-repo public and private keys https://review.openstack.org/406382 | 16:23 |
| *** sree has joined #openstack-infra | 16:23 | |
| *** rbrndt__ has quit IRC | 16:24 | |
| *** _Cyclone_ has quit IRC | 16:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Port SQLAlchemy reporter to v3 driver structure https://review.openstack.org/442114 | 16:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move alembic into sql driver https://review.openstack.org/442124 | 16:25 |
| *** rbrndt__ has joined #openstack-infra | 16:25 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: remove logs from subnode https://review.openstack.org/445902 | 16:27 |
| *** sshnaidm is now known as sshnaidm|nnl | 16:27 | |
| *** sshnaidm|nnl is now known as sshnaidm|bbl | 16:27 | |
| *** sree has quit IRC | 16:27 | |
| *** GregHouse has quit IRC | 16:27 | |
| *** robputt_ has quit IRC | 16:28 | |
| *** clayton has quit IRC | 16:28 | |
| *** GregHouse has joined #openstack-infra | 16:30 | |
| *** robputt has joined #openstack-infra | 16:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove more swift configurations https://review.openstack.org/446056 | 16:31 |
| *** panda is now known as panda|bbl | 16:31 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/infra-specs master: Zuulv3 Launcher Security Enhancement https://review.openstack.org/444495 | 16:31 |
| *** clayton has joined #openstack-infra | 16:33 | |
| *** mdrabe has quit IRC | 16:33 | |
| *** david-lyle_ has joined #openstack-infra | 16:34 | |
| *** david-lyle has quit IRC | 16:34 | |
| *** faizy has quit IRC | 16:35 | |
| *** ggnel_t has quit IRC | 16:36 | |
| *** sandanar has quit IRC | 16:36 | |
| *** annegentle has joined #openstack-infra | 16:37 | |
| *** EricGonczer_ has quit IRC | 16:37 | |
| *** eharney has quit IRC | 16:37 | |
| *** sandanar has joined #openstack-infra | 16:37 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 16:38 |
| *** strigazi is now known as strigazi_AFK | 16:39 | |
| *** EricGonczer_ has joined #openstack-infra | 16:39 | |
| *** eroux has joined #openstack-infra | 16:40 | |
| openstackgerrit | Igor Duarte Cardoso proposed openstack-infra/irc-meetings master: Update Common Classification Framework meeting https://review.openstack.org/441068 | 16:40 |
| *** dhajare has quit IRC | 16:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Port SQLAlchemy reporter to v3 driver structure https://review.openstack.org/442114 | 16:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move alembic into sql driver https://review.openstack.org/442124 | 16:41 |
| *** Apoorva has joined #openstack-infra | 16:41 | |
| *** mdrabe has joined #openstack-infra | 16:43 | |
| armax | AJaeger: https://review.openstack.org/#/c/445037/ kindly needs your +W | 16:44 |
| *** yamahata has quit IRC | 16:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add per-repo public and private keys https://review.openstack.org/406382 | 16:47 |
| johnsom | Hello Infra folks, can I get the following groups seeded with 'octavia-core'? | 16:47 |
| johnsom | https://review.openstack.org/#/admin/groups/1730,members | 16:48 |
| johnsom | https://review.openstack.org/#/admin/groups/1728,members | 16:48 |
| johnsom | https://review.openstack.org/#/admin/groups/1729,members | 16:48 |
| johnsom | Thank you | 16:48 |
| *** Sukhdev_ has quit IRC | 16:48 | |
| *** gyee has joined #openstack-infra | 16:50 | |
| *** tesseract has quit IRC | 16:53 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename zuul-launcher to zuul-executor https://review.openstack.org/445594 | 16:55 |
| *** mhickey_ has joined #openstack-infra | 16:56 | |
| *** baoli has quit IRC | 16:56 | |
| *** mhickey has quit IRC | 16:56 | |
| *** baoli has joined #openstack-infra | 16:56 | |
| *** nadya has joined #openstack-infra | 16:57 | |
| *** baoli has quit IRC | 16:58 | |
| *** rook-mtg is now known as rook | 16:59 | |
| *** dhajare has joined #openstack-infra | 16:59 | |
| *** e0ne has quit IRC | 17:00 | |
| *** bh526r has joined #openstack-infra | 17:03 | |
| *** ldnunes has joined #openstack-infra | 17:03 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Update wait_for_threads https://review.openstack.org/445994 | 17:03 |
| *** trown|lunch is now known as trown | 17:04 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Clean up additional references to launcher https://review.openstack.org/446073 | 17:04 |
| *** rossella_s has joined #openstack-infra | 17:04 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Add note about multiple builders support https://review.openstack.org/439545 | 17:05 |
| *** baoli has joined #openstack-infra | 17:06 | |
| *** anilvenkata has quit IRC | 17:07 | |
| clarkb | johnsom: is there a governance change to add octavia-client to the octavia project list? | 17:08 |
| *** vsaienko has joined #openstack-infra | 17:08 | |
| clarkb | oh wait there it | 17:09 |
| johnsom | clarkb https://review.openstack.org/439798 | 17:09 |
| *** dhajare has quit IRC | 17:09 | |
| clarkb | johnsom: all done | 17:10 |
| johnsom | clarkb Thank you! | 17:10 |
| *** martinkopec has quit IRC | 17:10 | |
| *** erikmwilson has quit IRC | 17:12 | |
| *** erikmwilson has joined #openstack-infra | 17:13 | |
| *** rtheis has joined #openstack-infra | 17:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Clean up additional references to launcher https://review.openstack.org/446073 | 17:15 |
| *** rtheis has quit IRC | 17:16 | |
| *** funzo_ has joined #openstack-infra | 17:16 | |
| *** rossella_s has quit IRC | 17:18 | |
| mnaser | so working on making the kolla jobs more stable, it looks like eth1 just came up with no ip on a job resulting in a failure - ssh: connect to host 10.210.192.64 port 22: Connection timed out | 17:18 |
| *** erlon has joined #openstack-infra | 17:19 | |
| clarkb | mnaser: have a link to the job log? | 17:19 |
| mnaser | http://logs.openstack.org/32/446032/1/gate/gate-kolla-dsvm-build-centos-source-centos-7/2313b0a/console.html#_2017-03-15_17_06_45_788420 | 17:19 |
| mnaser | is it possible the centos image is not running dhcp on eth1 | 17:19 |
| *** pbourke has joined #openstack-infra | 17:19 | |
| clarkb | mnaser: in fact we know its not dhcping because rax doesn't dhcp :/ | 17:19 |
| mnaser | oh yeah | 17:20 |
| mnaser | magical agent | 17:20 |
| clarkb | mnaser: it is glean's job to read the net interface config from the config drive and then apply that to the interface | 17:20 |
| clarkb | thinking we may want to dump the config drive network data at that point in the jobs too | 17:20 |
| mnaser | i was going to suggest that | 17:21 |
| clarkb | because I have no idea if the config drive had the data and glean failed or if config drive/nova failed | 17:21 |
| mnaser | well id assume that it did have data because eth0 went up | 17:21 |
| *** funzo_ has quit IRC | 17:21 | |
| clarkb | mnaser: well the data is per interface so its possible eth0 had info and eth1 did not | 17:21 |
| clarkb | but ya it definitely had something in order to make eth0 function | 17:22 |
| mnaser | do you boot with configdrive on all clouds | 17:22 |
| mnaser | aka can we safely print out whats in there | 17:22 |
| clarkb | ya we should | 17:22 |
| clarkb | since its the same image across all clouds | 17:22 |
| mnaser | ok /mnt/config seems to contain the configdrive info | 17:23 |
| mnaser | ill have to try to remember how the formatting works again | 17:23 |
| clarkb | I think you want to do something like find /mnt/config -name '*network*' -exec cat {} ; | 17:24 |
| *** florianf has quit IRC | 17:24 | |
| clarkb | because different cloud use(d) different file names | 17:24 |
| *** yamahata has joined #openstack-infra | 17:25 | |
| mnaser | is there a way to test this out somehow before submitting a change on a node | 17:25 |
| mnaser | test this out (on a node) before submitting a change | 17:25 |
| mnaser | words are hard and we're drowning in snow | 17:25 |
| clarkb | mnaser: you could run the command on a nova VM booted with config drive. Or just push the chagne to devstack-gate since its self testing | 17:26 |
| clarkb | mnaser: and with the power of telnet you can see what is printed really early in the job so quick to iterate | 17:26 |
| mnaser | i think i'll just unlazy a bit and boot a vm with configdrive | 17:26 |
| mnaser | :-P | 17:26 |
| *** dtantsur is now known as dtantsur|afk | 17:27 | |
| pabelanger | clarkb: speaking of config-drive: https://review.openstack.org/#/c/349697/ defaults nodepool to yes | 17:28 |
| *** matrohon has quit IRC | 17:28 | |
| *** pcaruana has quit IRC | 17:28 | |
| *** rbrndt__ has quit IRC | 17:28 | |
| *** jkilpatr has joined #openstack-infra | 17:29 | |
| clarkb | mnaser: oh its actually a jjb macro dumping that info so not self testing | 17:29 |
| clarkb | so ++ to testing with a VM that way :) | 17:29 |
| mnaser | im thinking | 17:29 |
| mnaser | cat /mnt/config/openstack/latest/network_data.json | python -mjson.tool | 17:29 |
| *** szaher has quit IRC | 17:29 | |
| clarkb | mnaser: that sounds right, but I think some clouds call(ed) it network_info.jsob | 17:30 |
| clarkb | hence my find, but I think maybe that is all fixed in the various novas now so probably data is fine | 17:30 |
| mordred | I believe it's in vendor.json or vendor_data.json on rax | 17:30 |
| *** jkilpatr_ has quit IRC | 17:30 | |
| mordred | because their inclusion of that data pre-dates the network_info.json | 17:30 |
| mnaser | im going through glean's code to see if it uses network_info or vendor_data ? | 17:31 |
| clarkb | if you give me a few minutes to load an assh key I can check | 17:31 |
| mordred | (the list of files we have to look for is in glean | 17:31 |
| *** esikachev has quit IRC | 17:31 | |
| *** jordanP has quit IRC | 17:31 | |
| *** jpich has quit IRC | 17:31 | |
| *** Goneri has quit IRC | 17:31 | |
| mnaser | yeah found it | 17:31 |
| mordred | openstack/latest/vendor_data.json openstack/latest/network_info.json and openstack/latest/network_data.json | 17:31 |
| mordred | cool | 17:31 |
| mnaser | yup | 17:31 |
| mnaser | so print all 3? or we could just print everything inside the configdrive openstack api? | 17:32 |
| clarkb | mnaser: I think you want a if -f for each one and for each one that matches print | 17:33 |
| mnaser | yeah i think thats better because | 17:33 |
| clarkb | just in case the data is different between them if multiple files exist causing glean to act different | 17:33 |
| mnaser | meta_data has admin_pass | 17:33 |
| clarkb | mnaser: maybe only do the network ones? or do the vendor_data files also include meta data? | 17:34 |
| clarkb | I don't think we need the actual metadata fwiw | 17:34 |
| *** Sukhdev_ has joined #openstack-infra | 17:34 | |
| mnaser | yeah ill just print what glean looks for | 17:34 |
| clarkb | mnaser: the reason for that is we don't actually rely on the normal metadata for functiaonlity. We bake ssh keys in and don't use passwords etc | 17:35 |
| openstackgerrit | Merged openstack-infra/nodepool master: Default config-drive to true https://review.openstack.org/349697 | 17:36 |
| mnaser | clarkb mordred http://paste.openstack.org/show/602906/ thoughts? | 17:37 |
| mnaser | if that seems ok, i can do my effort to get it into the jjb macro (the same one that does the traceroutes etc) | 17:37 |
| mordred | mnaser: ++ | 17:38 |
| clarkb | mnaser: you don't need the cat just use python -m json.tool $1 but otherwise ya | 17:38 |
| *** flepied has quit IRC | 17:38 | |
| mnaser | TIL | 17:38 |
| *** szaher has joined #openstack-infra | 17:38 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Clean up additional references to launcher https://review.openstack.org/446073 | 17:39 |
| mnaser | are we okay with adding it here? https://github.com/openstack-infra/project-config/blob/5accc0fb4c21e4c59afc10c29b76ddc320fa9fb8/jenkins/jobs/macros.yaml#L747-L775 | 17:39 |
| *** szaher has quit IRC | 17:39 | |
| clarkb | mnaser: yup | 17:39 |
| clarkb | I think so | 17:39 |
| *** szaher has joined #openstack-infra | 17:39 | |
| clarkb | mordred: ^ there is no sensitive info that would be leaked on say the static nodes if we did that is there? | 17:39 |
| * clarkb checks if static jobs use net-info, I doubt they do | 17:40 | |
| clarkb | oh hey some do | 17:40 |
| *** ym200 has quit IRC | 17:40 | |
| *** ym200 has joined #openstack-infra | 17:41 | |
| clarkb | I'm going to login to proposal.slave.o.o and just double check the data on it | 17:41 |
| clarkb | if it looks fine I'm happy | 17:41 |
| mnaser | im not a bash expert | 17:41 |
| mnaser | but | 17:41 |
| mnaser | https://github.com/openstack-infra/project-config/blob/5accc0fb4c21e4c59afc10c29b76ddc320fa9fb8/jenkins/jobs/macros.yaml#L97-L100 | 17:41 |
| mnaser | any idea why there is double braces there | 17:41 |
| mnaser | aka should i have double braces too? | 17:42 |
| *** mhickey_ has quit IRC | 17:42 | |
| *** trown is now known as trown|mtg | 17:43 | |
| clarkb | mnaser: its beacuse that macro accepts a paramter and {} is parameter substitution in jjb {{}} escapes that | 17:43 |
| mnaser | aaaah i see | 17:43 |
| clarkb | mnaser: net-info doesn't take any parameters but I don't know if jjb distinguishes on that or if all macros are in param substitution mode | 17:44 |
| mnaser | better safe than sorry | 17:44 |
| mnaser | if this breaks all hell goes loose :p | 17:44 |
| clarkb | mnaser: you could do [ -f foo ] && python -m json.tool and not worry about it | 17:44 |
| clarkb | and just repeat rather than use function | 17:44 |
| mnaser | thats a better idea | 17:44 |
| mnaser | and takes less place too | 17:44 |
| mnaser | lemme change and try it | 17:44 |
| clarkb | either way I think jjb will complain if the "compile" is bad | 17:45 |
| *** Goneri has joined #openstack-infra | 17:45 | |
| *** MasterOfBugs has joined #openstack-infra | 17:45 | |
| clarkb | mordred: mnaser you want to know something "fun"? rackspace has both vendor_data and network_data. But the info is different | 17:46 |
| *** derekh has quit IRC | 17:47 | |
| clarkb | in particular vendor_data has static route info and network data does not | 17:47 |
| *** esikachev has joined #openstack-infra | 17:47 | |
| *** czeq has quit IRC | 17:47 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: ironic: Make stand-alone job non-voting https://review.openstack.org/446087 | 17:47 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Print out network configuration data in net-info https://review.openstack.org/446088 | 17:47 |
| mordred | clarkb: yah - I believe that's being worked in to being in nova | 17:48 |
| mnaser | network_data.json is nova generated | 17:48 |
| mordred | both are nova generated at rax - they have local patches | 17:48 |
| *** dhinesh has joined #openstack-infra | 17:48 | |
| clarkb | oh wait the routes are both there, what is this giant list of things | 17:48 |
| mnaser | on our vanilla cloud we have network_data.json | 17:48 |
| mordred | yah - that's hwere it should be | 17:48 |
| clarkb | ip_whitelist | 17:48 |
| mnaser | anyways i proposed that patch above, hopefully going to help clear things | 17:48 |
| mnaser | does glean have any logs which could be useful as well? | 17:49 |
| mnaser | maybe glean is pooping out on boot | 17:49 |
| mnaser | so maybe a useful traceback can help shed some light too | 17:49 |
| clarkb | mnaser: it should log to syslog | 17:50 |
| *** esikachev has quit IRC | 17:52 | |
| *** ianychoi has joined #openstack-infra | 17:52 | |
| mnaser | oh this is interesting | 17:52 |
| mnaser | http://logs.openstack.org/32/446032/1/gate/gate-kolla-dsvm-build-centos-source-centos-7/2313b0a/logs/syslog.txt.gz#_Mar_15_17_05_35 | 17:52 |
| mnaser | ifup[719]: /sbin/ifup: configuration for eth1 not found. | 17:52 |
| mnaser | im going to try downloading the journal and see if i can get more info | 17:54 |
| clarkb | mnaser: Mar 15 17:05:34 localhost systemd[1]: Starting Glean for interface eth1... is where glean should ingest config drive data and output configuration for eth1 | 17:54 |
| clarkb | unfortunately I don't really see any logging for what it attempted to do there | 17:55 |
| mnaser | but it looks like it didnt create the config for eth1 so ifup eth1 just fails | 17:55 |
| mnaser | it doesnt log much either, no traceback :( | 17:55 |
| mordred | oh goodie | 17:55 |
| *** armax has quit IRC | 17:55 | |
| mordred | is this one of our nodes? why does it have postfix installed? | 17:56 |
| mnaser | centos-7-rax-ord so i guess so? | 17:56 |
| clarkb | guessing postfix is default on centos? | 17:57 |
| mnaser | yeah it is | 17:57 |
| mordred | (I mean, I don't care about that - but we don't tend to use postfix, we tend to use exim- so it makes me wonder if something is weird with that config) | 17:57 |
| clarkb | mordred: except that when you centos-minimal you get a default minimal install | 17:57 |
| clarkb | if postfix is incldued in that then there you go | 17:57 |
| mordred | right. but then we puppet to add our user accounts and keys and I thought that installed our basic exim config | 17:57 |
| clarkb | likely just a difference between debuntu not installing mta by default but centos does | 17:58 |
| clarkb | mordred: ya I'm sure exim is there too | 17:58 |
| clarkb | its just insufficiently excising postfix | 17:58 |
| mordred | anyway - it's not important - it just jumped out at me | 17:58 |
| clarkb | actually yum search says no exim | 17:59 |
| clarkb | so maybe I am wrong about that | 17:59 |
| mnaser | i think there's not much more information that we have | 18:00 |
| mnaser | the patch once merged should give us more information if it fails again | 18:00 |
| mnaser | if someone could be kind enough to give it another poke :> | 18:01 |
| *** kzaitsev_mb has joined #openstack-infra | 18:01 | |
| clarkb | mordred: we don't install exim on our single use slaves | 18:02 |
| clarkb | mordred: so that explains it | 18:02 |
| *** rlandy|afk is now known as rlandy | 18:02 | |
| mordred | clarkb: ah. ok. cool. so that's not a sneaky symptom of some larger breakage | 18:02 |
| mordred | that's good | 18:02 |
| *** esikachev has joined #openstack-infra | 18:02 | |
| mordred | clarkb: we shoudl maybe get around to uninstalling postfix in our infra element - it's totally not needed in there | 18:03 |
| mordred | and it' snot like we're not tight on ram sometimes | 18:03 |
| clarkb | ya | 18:03 |
| mnaser | if i remember correctly | 18:03 |
| mnaser | uninstalling postfix in centos 7 removes something somwhat necessary | 18:03 |
| mnaser | nevermind, i lied | 18:04 |
| clarkb | hahaha of course | 18:04 |
| mnaser | (just tested) | 18:04 |
| clarkb | ah thats good | 18:04 |
| * clarkb remembers the pain of fixing setuptools | 18:04 | |
| amrith | hi infra folks, could I get some +2's on https://review.openstack.org/#/c/438668/ (putting mysql on a diet) | 18:04 |
| clarkb | amrith: you need qa folks | 18:04 |
| clarkb | we generally aren't devstack cores | 18:04 |
| amrith | yes, also devstack | 18:04 |
| *** nadya has quit IRC | 18:04 | |
| amrith | but without someone from infra saying yes ... | 18:04 |
| *** marst has quit IRC | 18:05 | |
| clarkb | amrith: I'm not sure I follow why someone from infra ahs to say yes? | 18:05 |
| *** marst has joined #openstack-infra | 18:05 | |
| amrith | well, that seems to be the same sentiment from devstack from the last time I broached the subject with them. | 18:05 |
| amrith | but, ok ... | 18:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Uninstall postfix on nodes if it's there https://review.openstack.org/446094 | 18:05 |
| mordred | clarkb: while I was thinking of it ^^ | 18:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Reset lost requests https://review.openstack.org/446095 | 18:06 |
| clarkb | amrith: if I were a devstack core I would get mysql people to chime in (mordred SpamapS devananda jaypipes etc) and if that are good with it then yay | 18:06 |
| clarkb | I just don't know what input I could provide | 18:06 |
| mordred | amrith: I have added a +1 (which is all I can do) | 18:07 |
| amrith | well, mordred SpamapS devananda jaypipes would you please chime in on https://review.openstack.org/#/c/438668/ | 18:07 |
| *** esikachev has quit IRC | 18:07 | |
| amrith | thanks mordred | 18:07 |
| *** Sukhdev_ has quit IRC | 18:07 | |
| openstackgerrit | Merged openstack-infra/project-config master: Print out network configuration data in net-info https://review.openstack.org/446088 | 18:09 |
| mnaser | thanks clarkb and mordred for that ^ | 18:09 |
| *** baoli has quit IRC | 18:09 | |
| * fungi is also not a core reviewer on devstack, but the reasoning seems sound enough so +1 | 18:09 | |
| *** baoli has joined #openstack-infra | 18:10 | |
| *** rbrndt__ has joined #openstack-infra | 18:12 | |
| *** wklm has quit IRC | 18:13 | |
| *** baoli has quit IRC | 18:14 | |
| *** esikachev has joined #openstack-infra | 18:14 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: Fix on openstackid production node https://review.openstack.org/446100 | 18:15 |
| *** sandanar has quit IRC | 18:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create run-cover role https://review.openstack.org/441332 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable https://review.openstack.org/441441 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder https://review.openstack.org/441547 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Rename prepare-workspace role to bootstrap https://review.openstack.org/441440 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add run-docs role and tox-docs job https://review.openstack.org/441345 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role https://review.openstack.org/441617 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs https://review.openstack.org/441467 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job https://review.openstack.org/441609 | 18:16 |
| *** funzo_ has joined #openstack-infra | 18:17 | |
| *** gyee has quit IRC | 18:18 | |
| *** esikachev has quit IRC | 18:19 | |
| *** tosky has quit IRC | 18:20 | |
| jaypipes | clarkb, amrith: done. | 18:20 |
| amrith | thx jaypipes | 18:21 |
| *** baoli has joined #openstack-infra | 18:22 | |
| *** funzo_ has quit IRC | 18:22 | |
| *** skelso has joined #openstack-infra | 18:23 | |
| *** pramodrj07 has joined #openstack-infra | 18:23 | |
| amrith | jaypipes, confirmed that the values are in fact the lowest allowable (at least per documentation). I haven't tried to set it to lower and verify | 18:23 |
| *** MasterOfBugs has quit IRC | 18:23 | |
| *** pramodrj07 has quit IRC | 18:25 | |
| *** MasterOfBugs has joined #openstack-infra | 18:25 | |
| *** john-davidge has quit IRC | 18:25 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/project-config master: Really do enable glance v1 for osc jobs https://review.openstack.org/446105 | 18:26 |
| *** krtaylor has quit IRC | 18:26 | |
| *** john-davidge has joined #openstack-infra | 18:26 | |
| *** _vsaienko has joined #openstack-infra | 18:26 | |
| * frickler tries to hide in a corner as far as he can ^^ | 18:26 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix on openstackid production node https://review.openstack.org/446100 | 18:26 |
| *** MasterOfBugs has quit IRC | 18:27 | |
| *** esikachev has joined #openstack-infra | 18:27 | |
| *** MasterOfBugs has joined #openstack-infra | 18:27 | |
| jaypipes | amrith: ya, cool with me :) | 18:27 |
| amrith | thx jaypipes | 18:27 |
| mnaser | can i recheck something that was +W'd (if im not core on the project) | 18:28 |
| *** vsaienko has quit IRC | 18:28 | |
| *** marst has quit IRC | 18:28 | |
| *** hashar has joined #openstack-infra | 18:29 | |
| clarkb | mnaser: ya you should be able to recheck as normal. +W just prevents merging | 18:29 |
| fungi | mnaser: if it simply has a negative (-1 or -2) verify from jenkins then yes, anybody can leave a recheck comment to trigger another pass | 18:29 |
| *** armax has joined #openstack-infra | 18:30 | |
| mnaser | i thought there was a listener so when 8 minutes later nothing happened i was questioning it :p | 18:30 |
| mnaser | i'll wait a bit | 18:30 |
| fungi | mnaser: what's the change? | 18:30 |
| mnaser | fungi https://review.openstack.org/#/c/446032/ | 18:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Deallocate ready nodes with no requests https://review.openstack.org/446110 | 18:31 |
| mnaser | added some stuff to net-info earlier to help figure out why nodes come up with no ip on eth1 | 18:31 |
| *** esikachev has quit IRC | 18:31 | |
| *** skelso has quit IRC | 18:31 | |
| fungi | mnaser: yeah, the status page shows that change queued in zuul for 9 minutes now | 18:31 |
| fungi | http://status.openstack.org/zuul/ | 18:32 |
| mnaser | i guess jenkins removes the -2 once it starts running | 18:32 |
| mnaser | (which is what confused me, thanks) | 18:32 |
| fungi | i think it only removes its verify vote once it reenqueues into the gate pipeline (and our current "clean check" policy requires that it goes back through the check pipeline first) | 18:33 |
| *** marst has joined #openstack-infra | 18:33 | |
| mnaser | gotcha | 18:33 |
| *** shardy has quit IRC | 18:34 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: GitHub file matching support https://review.openstack.org/446113 | 18:34 |
| clarkb | mnaser: I think the net info update should go in place soon | 18:34 |
| clarkb | mnaser: and I think we can use http://status.openstack.org/elastic-recheck/#1531187 to find recent occurences to track down | 18:35 |
| *** ym200 has quit IRC | 18:35 | |
| clarkb | ihrachys: any luck tracking down the dhcp issue? | 18:36 |
| mnaser | clarkb awesome, look forward for it. also that looks good | 18:36 |
| *** ym200 has joined #openstack-infra | 18:36 | |
| ihrachys | clarkb: I think it was odl issue, does it hit any other jobs? | 18:36 |
| *** sshnaidm|bbl is now known as sshnaidm | 18:36 | |
| ihrachys | clarkb: when I checked logstash, it was all odl | 18:36 |
| ihrachys | and they were experiencing some gate issue where they ended up disabling some tests | 18:37 |
| clarkb | ihrachys: oh right you said that. I need more caffeine. I asked because it looks like it may have tapered off | 18:37 |
| *** esikachev has joined #openstack-infra | 18:37 | |
| clarkb | that explains it | 18:37 |
| *** _vsaienko has quit IRC | 18:37 | |
| ihrachys | cool | 18:37 |
| johnsom | Hi, is there a known DNS problem? "Could not resolve host: git.openstack.org" in our gate jobs | 18:37 |
| johnsom | http://logs.openstack.org/59/446059/1/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial-nv/b9c73cc/logs/devstacklog.txt.gz#_2017-03-15_17_11_29_167 | 18:37 |
| ianychoi | Hello, anybody acknowleding that openstackid does not work now? For me, openstackid does not work for www.o.o, translate.o.o, and groups.o.o | 18:37 |
| *** vsaienko has joined #openstack-infra | 18:38 | |
| *** vhosakot has quit IRC | 18:38 | |
| *** vhosakot has joined #openstack-infra | 18:39 | |
| clarkb | ianychoi: yes there was a slight hickup in the recent upgrade I think fungi and smarcet have it almost sorted out | 18:39 |
| *** jtomasek has quit IRC | 18:39 | |
| ianychoi | Hmm might be related with https://review.openstack.org/#/c/446100/ - let's wait .. clarkb yep I have just seen it thanks! | 18:39 |
| clarkb | johnsom: not that I know of | 18:40 |
| *** eharney has joined #openstack-infra | 18:40 | |
| *** bnemec has quit IRC | 18:41 | |
| clarkb | johnsom: that ran in rackspace and I think glean broke our usage of local resolver ( working to check that now ) and rackspace has known issues with its resolvers | 18:41 |
| clarkb | mordred: ^ fyi | 18:41 |
| openstackgerrit | Ken Dreyer proposed openstack-infra/jenkins-job-builder master: builder: use newer get_plugins() API if available https://review.openstack.org/446116 | 18:41 |
| *** esikachev has quit IRC | 18:41 | |
| johnsom | Ok, thanks | 18:41 |
| clarkb | actually we are using the local resolver on random rax host I jumped on so I guess that got fixed. That kills that theory | 18:42 |
| pabelanger | Ya, I think we fixed it a while back | 18:44 |
| *** bnemec has joined #openstack-infra | 18:44 | |
| *** e0ne has joined #openstack-infra | 18:45 | |
| *** rossella_s has joined #openstack-infra | 18:45 | |
| fungi | ianychoi: yeah, it should be back online now | 18:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Really do enable glance v1 for osc jobs https://review.openstack.org/446105 | 18:46 |
| clarkb | I'm not seeing anything obvious in glean's git commit history | 18:46 |
| *** trown|mtg is now known as trown | 18:46 | |
| *** vsaienko has quit IRC | 18:46 | |
| clarkb | maybe we fixed it outside of glean somehow? | 18:46 |
| *** dprince has quit IRC | 18:47 | |
| fungi | ianychoi: there was a little bit missing in the production manifest for the openstackid.org server when the upgrade started a few minutes ago, but it seems to all be correct now | 18:47 |
| openstackgerrit | Michele Baldessari proposed openstack-infra/tripleo-ci master: WIP Make sure we upgrade to the right repos during an upgrade https://review.openstack.org/446121 | 18:47 |
| ianychoi | fungi, hmm there is one problem. openstackid authticates my id as a new identifier with translate.o.o | 18:48 |
| *** electrofelix has quit IRC | 18:48 | |
| ianychoi | fungi, it means I cannot login with my existing Zanata user id now.. | 18:48 |
| *** ym200 has quit IRC | 18:48 | |
| clarkb | pabelanger: ya I am totally not seeing how this was fixed in glean if it is fixed. Weseem to unconditionally write dns info if using config drive network data | 18:48 |
| ianychoi | www.openstack.org and groups.openstack.org seem fine | 18:48 |
| fungi | ianychoi: in what way did it change? we were only expecting spaces to be removed (the old version incorrectly kept spaces in the identifiers if present in names)? | 18:48 |
| *** afazekas_ is now known as afazekas | 18:50 | |
| ianychoi | fungi, maybe the spaces to be removed have the integration information between Zanata and openstackid? I just login to translate.openstack.org by clicking "Log In" button in Zanata but it redirects to "New User" registration and I cannot go further - Zanata tells that my 'ianychoi' ID is already registered | 18:50 |
| sshnaidm | pabelanger, can you plese +W this? https://review.openstack.org/#/c/445875/ | 18:51 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Removed some deprecation warnings and enabled py35 on tox. https://review.openstack.org/399994 | 18:51 |
| fungi | ianychoi: i bet it's related to spaces disappearing from the id. we may need to massage openid urls in the database | 18:52 |
| *** vsaienko has joined #openstack-infra | 18:52 | |
| clarkb | should be a straightforward update assuming the openid sanitization was done deterministically | 18:52 |
| ianychoi | Hmm, the spaces could be recovered? | 18:52 |
| *** rossella_s has quit IRC | 18:53 | |
| fungi | ianychoi: when you log in at https://www.openstackid.org/ what url does it end at (that should be the "new" openid it has for you) | 18:53 |
| fungi | er, https://openstackid.org/ i mean | 18:53 |
| ianychoi | fungi, in trusted sites? or in user actions? | 18:54 |
| *** ym200 has joined #openstack-infra | 18:54 | |
| *** marst has quit IRC | 18:54 | |
| mnaser | in an effort to speed up kolla oraclelinux builds, would it be possible (i dont mind doing the work) to create a mirror for it? | 18:55 |
| *** ym200 has quit IRC | 18:55 | |
| ianychoi | fungi, http://paste.openstack.org/show/602918/ | 18:55 |
| *** marst has joined #openstack-infra | 18:55 | |
| mnaser | they're significantly behind at the moment in terms of how long they take versus ubunt/centos | 18:55 |
| fungi | ianychoi: oh, if you're already logged in, then just whatever the "your openid" line says | 18:55 |
| *** ym200 has joined #openstack-infra | 18:56 | |
| *** ym200 has quit IRC | 18:57 | |
| clarkb | mnaser: is that somethign we can legally mirror? (honest question I have no idea what their licensing is like and if it requires a clone distro like centos to be used) | 18:57 |
| ianychoi | fungi, "Your OPENID: https://openstackid.org/ian.y..choi" | 18:57 |
| mnaser | clarkb that was actually my question but here is what i concluded https://community.oracle.com/docs/DOC-917963 | 18:57 |
| mnaser | they seem to list some mirrors there | 18:57 |
| mnaser | mirrors.kernel.org hosts a mirror so IANAL but that seems like it might be okay | 18:58 |
| *** annegentle has quit IRC | 18:58 | |
| mnaser | "Note that these mirrors are not run or managed by Oracle and this list is provided as a convenience only" | 18:58 |
| clarkb | cool | 18:58 |
| mnaser | looks like this is the root http://public-yum.oracle.com/ -- unsure if they run rsync | 18:59 |
| clarkb | pabelanger: I am really really failing at understanding how we are using a local resolver on rax now (I mean its great that we are but trying to udnerstand it in order to further debug dns resolution errors) | 19:00 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Disable tempest in container job https://review.openstack.org/445840 | 19:04 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Clean up oooq jobs https://review.openstack.org/445628 | 19:05 |
| *** skelso has joined #openstack-infra | 19:05 | |
| *** sekelso has joined #openstack-infra | 19:06 | |
| pabelanger | clarkb: where are you looking now? A held node? | 19:07 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/infra-specs master: Zuulv3 Launcher Security Enhancement https://review.openstack.org/444495 | 19:07 |
| clarkb | pabelanger: not a held node but rax xenial and trusty instances | 19:09 |
| clarkb | pabelanger: and comparing to glean source. Glean seems to always write dns info if writing networking interface config | 19:09 |
| clarkb | and these rax nodes have dns resolves local to rax that are not our unbound resolver that I would expect to be used | 19:09 |
| *** skelso has quit IRC | 19:10 | |
| fungi | ianychoi: okay, now i'm freed up to poke at the zanata db and see what needs fixing for the transformed openids | 19:11 |
| pabelanger | clarkb: ya, I know we write some DNS info with diskimage-builder (nodepool-base element) then glean also does it. | 19:11 |
| pabelanger | that's all I know for dns | 19:11 |
| *** erikmwilson has quit IRC | 19:12 | |
| clarkb | pabelanger: aha! it is nodepool-base that does it | 19:12 |
| *** dprince has joined #openstack-infra | 19:12 | |
| clarkb | pabelanger: so glean writes in the rax stuff then we come along and override it with our network setup scripts | 19:12 |
| clarkb | thats amazingly hacky | 19:12 |
| ianychoi | fungi, thanks for your investigation - maybe next time adding this as a unit test between openstackid-dev and translate-dev instance would be a good idea :) | 19:12 |
| fungi | clarkb: no /root/.my.cnf on translate01 huh? | 19:12 |
| clarkb | fungi: uh there should be | 19:13 |
| clarkb | oh right its not the default .my.cnf iirc | 19:13 |
| fungi | looks like backups refer to a /root/.translate_db.cnf | 19:13 |
| clarkb | ya we do that in case you talk to multiple dbs rather than merge the files iirc | 19:13 |
| clarkb | beacuse puppet defines | 19:13 |
| *** vsaienko has quit IRC | 19:13 | |
| *** erikmwilson has joined #openstack-infra | 19:13 | |
| fungi | that's fine | 19:13 |
| fungi | i'm in | 19:13 |
| pabelanger | clarkb: ya, we have a few places where DNS is setup. Not the best IMO | 19:13 |
| clarkb | johnsom: ok, now I understand the setup a bit better. We should be using the local resolver there reliably which then should resolve against google (8.8.8.8 and 8.8.4.4) | 19:14 |
| *** vsaienko has joined #openstack-infra | 19:15 | |
| fungi | ianychoi: found it. your old id was https://openstackid.org/ian%20y..choi | 19:15 |
| clarkb | looks like 8.8.8.8 and the ipv6 address for that "host" too | 19:15 |
| fungi | ianychoi: so the %20 got replaced by . | 19:15 |
| ianychoi | fungi, haha %20 and . ! | 19:16 |
| fungi | we have 42 rows with a %20 in their openids | 19:16 |
| *** krtaylor has joined #openstack-infra | 19:16 | |
| ianychoi | fungi, then manual Zanata data change would be better rather then changing in openstackid? | 19:16 |
| clarkb | ianychoi: yes, beacuse it turns out that some openid clients? barf on content like spaces | 19:17 |
| clarkb | so openstackid is trying to accomdoate them | 19:17 |
| fungi | ianychoi: yeah, because some openid implementations incorrectly handle %20 un/re-escaping | 19:17 |
| ianychoi | clarkb, fungi I am fine for this change assuming that current Zanata production data with openstackid information has been backed up | 19:17 |
| fungi | ianychoi: i updated your row in the zanata.HCredentials table to match the new id. can you try logging in again? | 19:18 |
| *** vsaienko has quit IRC | 19:18 | |
| clarkb | johnsom: so I think that means that either the packets went into the ether, google is blocking some of our cloud provider IPs, or maybe google had to go ask rax's dns servers and they failed? | 19:18 |
| *** funzo_ has joined #openstack-infra | 19:18 | |
| *** vsaienko has joined #openstack-infra | 19:18 | |
| fungi | ianychoi: yeah, we have database backups, but also i'll record exactly what i change when i do the mass migration in a moment | 19:18 |
| ianychoi | fungi, it works pretty well! | 19:18 |
| clarkb | ianychoi: yes we have daily db backups for zanata in production | 19:18 |
| *** bh526r has quit IRC | 19:18 | |
| fungi | ianychoi: cool, i'll work up a mitration table for the other 41 affected accounts now | 19:18 |
| ianychoi | clarkb, fungi: Thanks a lot for quick & great support on Zanata! | 19:19 |
| fungi | kudos to the zanata devs for having a db schema i could quickly grok | 19:19 |
| fungi | unlike, say, lodgeit or etherpad | 19:19 |
| johnsom | clarkb oh joy. Well, we can just recheck on that I guess | 19:20 |
| ianychoi | :) | 19:20 |
| *** amoralej is now known as amoralej|off | 19:20 | |
| clarkb | johnsom: according to logstash this has happend 5 times in the last 2 days. affecting rax-ord and internap-mtl01 and xenial and centos hosts | 19:22 |
| *** marst has quit IRC | 19:22 | |
| clarkb | which unfortuantely doesnt' really help narrow the problem down at all | 19:22 |
| *** dtrainor has quit IRC | 19:22 | |
| *** funzo_ has quit IRC | 19:23 | |
| *** Sukhdev has joined #openstack-infra | 19:23 | |
| *** ym200 has joined #openstack-infra | 19:24 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Add a UUID to the temporary email address https://review.openstack.org/444593 | 19:25 |
| *** trown is now known as trown|brb | 19:25 | |
| *** vsaienko has quit IRC | 19:27 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Fix on user identifer generator (openid) https://review.openstack.org/446137 | 19:28 |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable diskimage-builder / octavia gate https://review.openstack.org/446139 | 19:31 |
| *** john-davidge has quit IRC | 19:32 | |
| ihrachys | AJaeger: yolanda: what blocks https://review.openstack.org/#/c/445037/ ? | 19:33 |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable diskimage-builder / octavia gate https://review.openstack.org/446139 | 19:33 |
| clarkb | johnsom: and looks like we have seen it in ovh too | 19:35 |
| *** trown|brb is now known as trown | 19:37 | |
| *** eroux has quit IRC | 19:38 | |
| openstackgerrit | Merged openstack-infra/openstackid master: Fix on user identifer generator (openid) https://review.openstack.org/446137 | 19:39 |
| *** ociuhandu has quit IRC | 19:39 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config master: Add gate-hook to tempest job for networking-sfc https://review.openstack.org/445037 | 19:40 |
| ihrachys | armax: why does it affect stable only? | 19:40 |
| clarkb | johnsom: short of managing to hold a node with that failure I'm not sure how much more debugging we can do. It definitely seems infrequent and I'm guessing is related to hosting networking and/or google blacklisting | 19:41 |
| armax | ihrachys: ? | 19:41 |
| johnsom | clarkb yep. Thanks for having a look. | 19:42 |
| armax | ihrachys: don’t understand the question | 19:42 |
| ihrachys | armax: the sfc patch making api extensions explicit, why hasn't it failed in master without it but only in stable branch? | 19:42 |
| armax | ihrachys: because it’s a no-op in master | 19:42 |
| armax | ihrachys: the stable branch is busted regardless | 19:43 |
| clarkb | johnsom: might be worth proposing a change to elastic-recheck to track it if thee isn't one already though | 19:43 |
| *** fandi has joined #openstack-infra | 19:43 | |
| clarkb | johnsom: I've got to pop out for lunch now but happy to help with that if you are interseted | 19:43 |
| johnsom | Sure, send me the search string you used when you get a chance | 19:44 |
| clarkb | johnsom: message:"Could not resolve host" | 19:44 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Populate requestor for min-ready requests https://review.openstack.org/445970 | 19:44 |
| clarkb | we can probably reduce it down based on file but I so many things can fail to resolve I left it open | 19:44 |
| *** vgadiraj has quit IRC | 19:45 | |
| *** aunnam has quit IRC | 19:45 | |
| *** dharinic has quit IRC | 19:45 | |
| *** jascott1 has quit IRC | 19:45 | |
| *** soliosg has quit IRC | 19:45 | |
| *** sneti has quit IRC | 19:45 | |
| *** manjeets has quit IRC | 19:45 | |
| *** gordc has quit IRC | 19:48 | |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Voting functional py3x octavia test https://review.openstack.org/446148 | 19:49 |
| *** esikachev has joined #openstack-infra | 19:49 | |
| *** e0ne has quit IRC | 19:51 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Log GitHub API rate limit https://review.openstack.org/446150 | 19:52 |
| EmilienM | greghaynes: hey, when you have time can you look https://review.openstack.org/#/c/445617/ please, | 19:55 |
| *** e0ne has joined #openstack-infra | 19:55 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/gearman-zk-shim: Hardcode min-ready to zero https://review.openstack.org/435963 | 19:56 |
| *** esikachev has quit IRC | 19:57 | |
| greghaynes | EmilienM: yep, I need to send a ML, my plan is to add an item to the infra meeting agenda in two weeks (im out next week) so we chat about it briefly | 19:57 |
| *** david-lyle_ is now known as david-lyle | 19:57 | |
| *** esikachev has joined #openstack-infra | 19:57 | |
| EmilienM | greghaynes: can't we make progress before? do we really need to wait for a meeting to happen? | 19:58 |
| greghaynes | EmilienM: no, we dont, I was under the impression there wasnt any kind of rush | 19:58 |
| greghaynes | but we can definitely do it before... | 19:59 |
| EmilienM | greghaynes: no rush, but it's in the pipe for months | 19:59 |
| greghaynes | yea | 19:59 |
| EmilienM | I'm ok to wait if there are good reasons | 19:59 |
| greghaynes | sure, so the biggest reason I wanted to mention it in the infra meeting is jsut so more than tripleo / dib folks are aware and able to comment (specifically the infra folks). | 20:00 |
| greghaynes | So the alternative would be to use the ML for that I think. | 20:00 |
| EmilienM | my hope is we can solve this thing in async | 20:00 |
| *** matrohon has joined #openstack-infra | 20:00 | |
| greghaynes | EmilienM: ok, ill get together an ML I'll post to the -infra list | 20:01 |
| EmilienM | greghaynes: thanks and if there is a good reason to wait, then no rush. Enjoy the time off | 20:02 |
| *** Sukhdev has quit IRC | 20:02 | |
| greghaynes | EmilienM: btw I also made https://etherpad.openstack.org/p/dib-infra-move-deets | 20:02 |
| greghaynes | I wish it was time off, work travel :( | 20:02 |
| EmilienM | greghaynes: good luck then | 20:03 |
| *** kgiusti has left #openstack-infra | 20:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add secret top-level config object https://review.openstack.org/446156 | 20:06 |
| *** gus has quit IRC | 20:07 | |
| *** gus has joined #openstack-infra | 20:09 | |
| clarkb | ianychoi: if you are still around when is the meeting you wanted to wait for before deleting old zanata host? | 20:11 |
| *** docaedo has quit IRC | 20:12 | |
| ianychoi | clarkb, there will be I18n IRC meeting within less than 24 hours - Thursday 13:00 UTC. Can I tell after the meeting? | 20:12 |
| clarkb | yup, just wanted to amke sure I planned for removing the old host afterwads | 20:13 |
| *** hongbin has joined #openstack-infra | 20:13 | |
| sc` | what am i doing wrong if 'jenkins-jobs test' doesn't generate any jobs? even using a fresh clone of project-config without my changes results in 0 jobs generated | 20:13 |
| ianychoi | clarkb, I woke up today too early so I think I will sleep right after the meeting is finished. I will send an e-mail or let you know on Thursday afternoon (your timezone) :) | 20:13 |
| clarkb | sc`: are you pointing jenkins jobs at the config dir? | 20:15 |
| clarkb | sc`: I think by default it looks in /etc/jenkins-jobs or something like that | 20:15 |
| clarkb | sc`: project-config has the jjb tox target to make this easy | 20:16 |
| sc` | clarkb: i have a config in ~/.config that it picks up, but even -l debug does nothing to illuminate where things fall down | 20:16 |
| clarkb | sc`: `jenkins-jobs -l debug test -o {envdir}/tmp/jobs jenkins/jobs` is the command that tox runs | 20:17 |
| sc` | the last thing i want to do is spam the channel with a bunch of little stupid updates because i couldn't test locally | 20:17 |
| clarkb | sc`: right tox -e jjb is specifically for making that easier on you | 20:17 |
| fungi | i'm rerunning that locally right now to make sure it actually does the thing | 20:17 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Split webapp into its own nodepool application https://review.openstack.org/445675 | 20:18 |
| fungi | it certainly exercises my cpu fan ;) | 20:18 |
| *** funzo_ has joined #openstack-infra | 20:19 | |
| *** marst has joined #openstack-infra | 20:19 | |
| *** rwsu has joined #openstack-infra | 20:21 | |
| sc` | that's definitely more illuminating | 20:21 |
| sc` | way more than "lol, nothing generated" | 20:21 |
| fungi | cpu fan tests complete, tons of job configs written | 20:23 |
| *** funzo_ has quit IRC | 20:24 | |
| sc` | for reasons unknown, nor do i particularly care that much since i have new data to go on, running jenkins-jobs manually got me nowhere. tox -e jjb is getting somewhere | 20:25 |
| *** marst_ has joined #openstack-infra | 20:25 | |
| *** docaedo has joined #openstack-infra | 20:25 | |
| openstackgerrit | Kaitlin Farr proposed openstack-infra/project-config master: Update barbican kmip gate to non-voting check from experimental https://review.openstack.org/446161 | 20:25 |
| *** trown is now known as trown|outtypewww | 20:26 | |
| *** kzaitsev_mb has quit IRC | 20:27 | |
| *** marst has quit IRC | 20:28 | |
| *** dave-mccowan has quit IRC | 20:29 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Allow loading logging config from yaml https://review.openstack.org/445656 | 20:29 |
| jhesketh | Morning | 20:31 |
| *** matrohon has quit IRC | 20:31 | |
| *** ociuhandu has joined #openstack-infra | 20:33 | |
| *** marst_ has quit IRC | 20:36 | |
| *** bexelbie has quit IRC | 20:38 | |
| *** dprince has quit IRC | 20:38 | |
| *** bexelbie has joined #openstack-infra | 20:38 | |
| *** docaedo has quit IRC | 20:39 | |
| *** dtardivel has quit IRC | 20:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add secret top-level config object https://review.openstack.org/446156 | 20:40 |
| *** gus has quit IRC | 20:40 | |
| *** docaedo has joined #openstack-infra | 20:40 | |
| *** matt-borland has quit IRC | 20:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add reconfigure_handler for executor logging https://review.openstack.org/446172 | 20:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add secret top-level config object https://review.openstack.org/446156 | 20:40 |
| *** gus has joined #openstack-infra | 20:42 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: oooq multinode: fix for stable branches https://review.openstack.org/445866 | 20:43 |
| *** cody-somerville has joined #openstack-infra | 20:44 | |
| *** vsaienko has joined #openstack-infra | 20:45 | |
| *** jascott1 has joined #openstack-infra | 20:46 | |
| *** csomerville has quit IRC | 20:46 | |
| fungi | ianychoi: clarkb: this is what the transformation ended up being for the zanata db: http://paste.openstack.org/show/602930/ | 20:50 |
| ianychoi | fungi, thanks for sharing! by the way, can I also see zanata db schema? | 20:51 |
| ianychoi | (Or I can ask to Zanata dev team) | 20:51 |
| fungi | the transformation rules weren't exactly straightforward since there was some stripping of whitespace from fields before concatenation, some replacing of spaces with . and some simplifying of 8+ bit characters with 7-bit placeholders | 20:51 |
| fungi | ianychoi: i don't know a special mysql command to output the entire db schema. i ended up guessing a table based on its name and then looking at the fields | 20:52 |
| fungi | zanata has... lots of tables | 20:52 |
| clarkb | you can describe on a select of the table names? something liek that | 20:52 |
| clarkb | I'm sure its quite a bit of data | 20:52 |
| fungi | 59 tables | 20:52 |
| ianychoi | fungi, i see i do not want to execute additional commands - I can ask to zanata dev team don't worry about this | 20:53 |
| fungi | yeah, odds are they have a good document for it. i could probably script up something to dump a copy of all the table and field names and datatypes but it would be very long | 20:53 |
| sc` | now that tox -e jjb completes successfully, beyond proposing the changes, what guarantees do i have that things will actually work? | 20:53 |
| fungi | sc`: you can look in the xml file named for the job you expect to have created and see what shell blocks it contains | 20:54 |
| fungi | that's basically the end result of all macro/parameter expansion and object insertion | 20:55 |
| *** gouthamr has quit IRC | 20:55 | |
| ianychoi | Wow.. I have not recognized that there were too many "%20"s.. interesting. End trailings should not be replaced with period (.) | 20:55 |
| *** dimtruck is now known as zz_dimtruck | 20:55 | |
| *** panda|bbl is now known as panda | 20:55 | |
| sc` | fungi: okay, thanks. i can at least get to the point where i can say i need to work on it further | 20:56 |
| fungi | ianychoi: yeah, the openstackid backend expects a first name and a second name, and had a bug where it would preserve leading or trailing whitespace in each of those fields, so those got manually cleaned up in the db after that bug was fixed. but also it now reencodes any remaining spaces (not leading or trailing) with . and also (as before) puts a . between the two fields when it concatenates them | 20:56 |
| ianychoi | fungi, aha I got your point, thanks! | 20:58 |
| ianychoi | (with details) | 20:58 |
| fungi | so as some people would put a name and then a space followed by a middle initial and a . in the first name field like "FOO B.", that becomes FOO.B. and then gets concatenated with their second name with yet another . as a separator like "FOO.B..QUUX" | 20:58 |
| *** rwsu has quit IRC | 20:59 | |
| fungi | but there were some who had their names entered like " FOO" or "FOO " where those extra spaces had to be manually cleaned out of the db but had previously been also getting translated to %20 | 20:59 |
| *** vsaienko has quit IRC | 21:01 | |
| *** vhosakot has quit IRC | 21:01 | |
| fungi | and then to top that off, there's a hard-coded translation table for some extended/accented latin and cyrillic characters to one or more 7-bit latin replacements, but due to an oversight that only got applied _if_ the name already had a space in it | 21:01 |
| ianychoi | Haha.. yep I see. Just one more question for my curiousity: Then previously "ABCD " was not trimmed to "ABCD"? | 21:03 |
| fungi | ianychoi: right, not until the upgrade today | 21:03 |
| *** jascott1 has quit IRC | 21:03 | |
| fungi | as for the characterset "simplifaction" table, that's here for the curious http://git.openstack.org/cgit/openstack-infra/openstackid/tree/app/libs/Auth/UserNameGeneratorService.php#n22 | 21:04 |
| fungi | er, "simplification" | 21:04 |
| ianychoi | Yep then I think today fixing on openstackid is important and it is so fortunate that my name has a space and I was online | 21:04 |
| ianychoi | Wow looks too complex.. | 21:04 |
| fungi | ianychoi: i agree. i don't think the authors foresaw this disruption, but glad you spotted it quickly so we were able to work with them to make the zanata backend match | 21:05 |
| *** vsaienko has joined #openstack-infra | 21:05 | |
| fungi | i have a feeling groups and maybe refstack will run into similar challenges | 21:05 |
| *** hongbin has quit IRC | 21:06 | |
| ianychoi | fungi, aha yep please see it if refstack and others might have the similar problem. Again, thanks a lot for quick & great help on that issue! | 21:06 |
| *** zz_dimtruck is now known as dimtruck | 21:08 | |
| *** johndperkins has quit IRC | 21:09 | |
| *** sdague has quit IRC | 21:09 | |
| *** johndperkins has joined #openstack-infra | 21:11 | |
| *** dave-mccowan has joined #openstack-infra | 21:14 | |
| *** gouthamr has joined #openstack-infra | 21:14 | |
| *** askb has joined #openstack-infra | 21:16 | |
| *** jcoufal has quit IRC | 21:17 | |
| *** esikachev has quit IRC | 21:17 | |
| *** dave-mcc_ has joined #openstack-infra | 21:17 | |
| *** dave-mccowan has quit IRC | 21:19 | |
| clarkb | tonyb: hey are you around? I am working on a new email to the dev list about persistent gate issues and libvirt crashes are still showing up. Any progress made on that I can point to? | 21:26 |
| *** krtaylor has quit IRC | 21:27 | |
| *** ihrachys has quit IRC | 21:27 | |
| *** jtomasek has joined #openstack-infra | 21:28 | |
| *** burgerk has quit IRC | 21:28 | |
| *** flepied has joined #openstack-infra | 21:30 | |
| *** ldnunes has quit IRC | 21:32 | |
| *** tonytan4ever has quit IRC | 21:34 | |
| fungi | mrmartin: where does groups manage its mapping of openids to accounts? or is it relying on drupal to take care of that (in which case, where does drupal...) | 21:34 |
| *** Goneri has quit IRC | 21:35 | |
| fungi | i took a casual stroll through a mysqldump of its db and can't find anything which looks like an openid | 21:35 |
| clarkb | where does nova stash the metadata for ssh keys? is that in user data or in meta data? | 21:35 |
| clarkb | mordred: mnaser ^ do you know off the top of your head before I go digging? | 21:35 |
| *** Jeffrey4l_ has joined #openstack-infra | 21:35 | |
| *** EricGonczer_ has quit IRC | 21:35 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| mnaser | clarkb it goes in metadata | 21:35 |
| clarkb | mnaser: thanks | 21:36 |
| mnaser | clarkb np, i see this works for me - python -mjson.tool /mnt/config/openstack/latest/meta_data.json | 21:36 |
| mnaser | under public_keys | 21:36 |
| *** Sukhdev has joined #openstack-infra | 21:36 | |
| mgagne | Just a quick reminder that we (internap) will be performing a maintenance in ~24m and the Identity API will be unavailable for a period of up to 4h. Hopefully, we will keep it short and restore service in much less than 4h. Any running instances won't be affected. | 21:37 |
| *** jkilpatr has quit IRC | 21:37 | |
| clarkb | mgagne: thanks for the heads up (if it causes problems we can tell nodepool to temporarily stop booting instances there | 21:37 |
| clarkb | fungi: any idea why http://logs.openstack.org/50/439950/9/check/gate-tempest-dsvm-neutron-full-ssh/dd449bd/console.html#_2017-03-09_02_34_57_077403 would happen? | 21:38 |
| clarkb | fungi: I think that is breaking networking in the guest which causes ssh to the guest to not work and tempest fails | 21:38 |
| fungi | route: SIOCADDRT: File exists | 21:38 |
| fungi | translating: there is already a default route configured | 21:39 |
| *** ijw has quit IRC | 21:39 | |
| clarkb | so now Iguess the question is, is that default route valid or not | 21:39 |
| fungi | default via 10.0.0.1 dev ens3 onlink | 21:40 |
| fungi | http://logs.openstack.org/50/439950/9/check/gate-tempest-dsvm-neutron-full-ssh/dd449bd/console.html#_2017-03-09_01_26_12_156561 | 21:40 |
| clarkb | fungi: thats for the hypervisor, the error is for the guest | 21:40 |
| fungi | aha, i missed the change of context | 21:41 |
| *** Sukhdev has quit IRC | 21:41 | |
| fungi | i see now that's in the console output from a cirros boot | 21:41 |
| clarkb | ya sorry, its the cirros guest that appears to be unreachable from hypervisor. CUrrent theory is that the route is wrong since it didn't get set at the log location I linked | 21:41 |
| clarkb | so packets come into cirros then have no where to return (since we use the floating IPs and not addrs on a shared l2) | 21:42 |
| clarkb | or I guess they have somewhere to go since something is defined already, it just happens to be wrong maybe | 21:42 |
| fungi | i don't guess we can coerce it to klog the routing table before that | 21:42 |
| clarkb | we might be able to add a cloud init user script to run | 21:42 |
| clarkb | actually thats probably a good idea, I just have to figure out how to get that into tempest | 21:43 |
| fungi | or just spit it on stdout of the init process i guess | 21:43 |
| fungi | don't need to actually klog | 21:43 |
| fungi | since we probably already have the ability to run something under that init | 21:43 |
| clarkb | ya via cloud init user scripts | 21:43 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: support github pull reqeust labels https://review.openstack.org/444511 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow github trigger to match on branches/refs https://review.openstack.org/445625 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for dependent pipelines with github https://review.openstack.org/445292 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add basic Github Zuul Reporter. https://review.openstack.org/443323 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Configurable SSH access to GitHub https://review.openstack.org/444034 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'push' and 'tag' github webhook events. https://review.openstack.org/443947 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'pr-comment' github webhook event https://review.openstack.org/443959 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for github commit status https://review.openstack.org/444060 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Better merge message for GitHub pull reqeusts https://review.openstack.org/445644 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Encapsulate determining the event purpose https://review.openstack.org/445242 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow using webapp from connections https://review.openstack.org/439831 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Merge pull requests from github reporter https://review.openstack.org/444463 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support GitHub PR webhooks https://review.openstack.org/439834 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: GitHub file matching support https://review.openstack.org/446113 | 21:44 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Log GitHub API rate limit https://review.openstack.org/446150 | 21:44 |
| fungi | jlk is a madman | 21:44 |
| jlk | SORRY! had to rebase a patch set because of a internal attribute rename | 21:45 |
| clarkb | I think just having `ip route` should be sufficient and the init system should handle grabbing of that stdout | 21:45 |
| clarkb | anyways I will fiddle with getting that into a tempest patch | 21:45 |
| frickler | so I think mysql on diet looks rather broken: http://logs.openstack.org/75/443075/2/gate/gate-osc-dsvm-functional-ubuntu-xenial/ec15ddd/logs/screen-q-svc.txt.gz?level=WARNING#_2017-03-15_20_55_35_993 | 21:45 |
| fungi | jlk: no apologies needed, just expressing amazement! ;) | 21:45 |
| fungi | frickler: we should probably loop in amrith (but maybe in #openstack-qa?) | 21:46 |
| frickler | fungi: ack | 21:51 |
| *** esikachev has joined #openstack-infra | 21:54 | |
| *** baoli has quit IRC | 21:54 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Fixed HTML on openid consent form https://review.openstack.org/446193 | 21:54 |
| *** erlon has quit IRC | 21:55 | |
| *** jamielennox has quit IRC | 21:55 | |
| *** Sukhdev has joined #openstack-infra | 21:57 | |
| *** jkilpatr has joined #openstack-infra | 21:57 | |
| *** john-davidge has joined #openstack-infra | 21:59 | |
| *** Julien-zte has joined #openstack-infra | 21:59 | |
| *** esikachev has quit IRC | 21:59 | |
| frickler | I proposed a revert, at least neutron is failing, too https://review.openstack.org/#/c/446196/ | 21:59 |
| *** felipemonteiro has quit IRC | 22:00 | |
| *** esikachev has joined #openstack-infra | 22:03 | |
| openstackgerrit | Merged openstack-infra/openstackid master: Fixed HTML on openid consent form https://review.openstack.org/446193 | 22:03 |
| *** john-davidge has quit IRC | 22:03 | |
| *** ccamacho has left #openstack-infra | 22:05 | |
| *** mdrabe has quit IRC | 22:06 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/elastic-recheck master: Adding signature for "Could not resolve host" https://review.openstack.org/446200 | 22:06 |
| johnsom | clarkb https://bugs.launchpad.net/octavia/+bug/1673269 and ^^^ | 22:07 |
| openstack | Launchpad bug 1673269 in octavia "DNS resolver failure "Could not resolve host: git.openstack.org"" [Low,Triaged] | 22:07 |
| johnsom | Not sure which project to tag it against, so I just did octavia for now | 22:07 |
| *** esikachev has quit IRC | 22:07 | |
| *** dimtruck is now known as zz_dimtruck | 22:09 | |
| *** esikachev has joined #openstack-infra | 22:09 | |
| *** hashar has quit IRC | 22:09 | |
| *** aysyd has quit IRC | 22:09 | |
| clarkb | mriedem: dansmith so for nova's user-data parameter on server create that takes a base 64 encoded string. Does it take url safe or url unsafe input? I am assuming safe because otherwise how do you param it? | 22:10 |
| clarkb | the docs aren't clear | 22:10 |
| dansmith | clarkb: idk | 22:11 |
| clarkb | looks like novaclient does unsafe which is potentially problematic | 22:12 |
| * clarkb greps the server | 22:12 | |
| *** Julien-zte has quit IRC | 22:13 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] quickstart: test for missing failed deployment log https://review.openstack.org/446043 | 22:14 |
| *** e0ne has quit IRC | 22:14 | |
| *** adriant has joined #openstack-infra | 22:14 | |
| EmilienM | any chance to get approval on https://review.openstack.org/#/c/445875/ please? it would help us to have tripleo jobs green again on stable branches - thanks | 22:15 |
| *** sheeprine has quit IRC | 22:15 | |
| *** jamielennox has joined #openstack-infra | 22:15 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added missing ToRecord field to presentation json https://review.openstack.org/446203 | 22:15 |
| clarkb | oh looks like its in the body of the post so is fine either way | 22:15 |
| clarkb | and nova uses normal base64 | 22:16 |
| *** jascott1 has joined #openstack-infra | 22:16 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added missing ToRecord field to presentation json https://review.openstack.org/446203 | 22:17 |
| *** krtaylor has joined #openstack-infra | 22:17 | |
| *** esikachev has quit IRC | 22:17 | |
| *** soliosg has joined #openstack-infra | 22:18 | |
| *** e0ne has joined #openstack-infra | 22:18 | |
| *** sheeprine has joined #openstack-infra | 22:19 | |
| *** armax has quit IRC | 22:19 | |
| *** funzo has joined #openstack-infra | 22:20 | |
| *** jascott1 has quit IRC | 22:20 | |
| EmilienM | clarkb: sorry for direct ping, I just hope we can have it tonight https://review.openstack.org/#/c/445875/ | 22:23 |
| andreaf | etherpad is killing me, it would be so nice to get this in: https://github.com/ether/etherpad-lite/issues/2625 | 22:23 |
| andreaf | but I guess I can always use firefox :) | 22:23 |
| *** rfolco has quit IRC | 22:23 | |
| clarkb | andreaf: we have been wary to upgrade since the current version works and performs well and has no known security bugs | 22:24 |
| clarkb | andreaf: basically every time we do an upgrade we have to do a bunch of bug fixes that only actually break during summit/midcycle | 22:24 |
| clarkb | thats a fun bug though :/ | 22:24 |
| *** funzo has quit IRC | 22:25 | |
| andreaf | clarkb: heh ok, no worries I agree stable is better - and switching to firefox is easy enough | 22:25 |
| clarkb | andreaf: can you look at https://review.openstack.org/446205 and let me know if that is on the right track? | 22:26 |
| clarkb | andreaf: fwiw you could propose the upgrade to etherpad-dev and we could go from there | 22:26 |
| clarkb | andreaf: let me find you place to edit | 22:26 |
| clarkb | oh right we already do that on dev | 22:26 |
| clarkb | well then, andreaf can you test https://etherpad-dev.openstack.org fixes the issue for you? | 22:27 |
| clarkb | EmilienM: do you have an example run from today? I want to make sure that we got the logging sorted out before we apply it to more locations | 22:27 |
| mriedem | clarkb: not sure how the json schema handles this https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/user_data.py#L26 | 22:27 |
| mriedem | i.e. what is json schema for base64? | 22:28 |
| clarkb | mriedem: I don't know but I dug around in code and its vanilla non url safe base64 | 22:28 |
| andreaf | clarkb: no, same issue, at least with safari | 22:28 |
| clarkb | whcih is fine because its posted in the body not a url param | 22:28 |
| clarkb | andreaf: ok, that should be latest etherpad I think | 22:28 |
| mriedem | clarkb: ok | 22:28 |
| EmilienM | panda, sshnaidm: if you still around, i'm trying to get your patch merged to bring our stable CI jobs | 22:28 |
| mriedem | clarkb: do you think we need doc updates? | 22:28 |
| mriedem | to clarify? | 22:28 |
| clarkb | mriedem: probably not, I should just trust the docs when they say base64 | 22:29 |
| mriedem | fwiw i generally double check the code | 22:29 |
| mriedem | but that's just me | 22:29 |
| clarkb | EmilienM: sshnaidm specifically there was that etherpad we had about cleaning up oooq logging and want to make sure we addressed those problems | 22:29 |
| *** e0ne has quit IRC | 22:29 | |
| mriedem | as a user you shouldn't have to though | 22:29 |
| EmilienM | clarkb: the size of logs was fixed AFIK | 22:29 |
| *** esikachev has joined #openstack-infra | 22:30 | |
| *** rwsu has joined #openstack-infra | 22:30 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: readded missing query method https://review.openstack.org/446207 | 22:30 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: readded missing query method https://review.openstack.org/446207 | 22:30 |
| *** gildub has joined #openstack-infra | 22:34 | |
| *** esikachev has quit IRC | 22:34 | |
| *** baoli has joined #openstack-infra | 22:34 | |
| *** Goneri has joined #openstack-infra | 22:35 | |
| panda | clarkb: oooq, we didn't receive any other complaints about log size, we removed the most problematic inclusion in log collections, should be ok | 22:37 |
| *** gongysh has joined #openstack-infra | 22:37 | |
| clarkb | panda: I wrote a large ish etehrpad list of things to change | 22:38 |
| clarkb | with sshnaidm | 22:38 |
| * clarkb tries to dig it up | 22:38 | |
| *** baoli has quit IRC | 22:38 | |
| *** claudiub has quit IRC | 22:40 | |
| *** Goneri has quit IRC | 22:40 | |
| *** rbrndt__ has quit IRC | 22:41 | |
| *** rbrndt__ has joined #openstack-infra | 22:41 | |
| *** rbrndt__ has quit IRC | 22:41 | |
| *** esikachev has joined #openstack-infra | 22:41 | |
| clarkb | panda: EmilienM sshnaidm https://etherpad.openstack.org/p/tripleo-logs-actions found it | 22:41 |
| *** zz_dimtruck is now known as dimtruck | 22:41 | |
| *** hongbin has joined #openstack-infra | 22:43 | |
| *** mriedem1 has joined #openstack-infra | 22:43 | |
| *** mriedem has quit IRC | 22:44 | |
| EmilienM | clarkb: nice list of things todo, we need to add that in our todo | 22:44 |
| EmilienM | panda: any way we can track it in our list of things? | 22:44 |
| clarkb | its possible those things were done (which is what I was aksing for a link to an example run from today so I could look) | 22:45 |
| *** esikachev has quit IRC | 22:46 | |
| panda | clarkb: http://logs.openstack.org/43/446043/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/463d5a1/ | 22:46 |
| *** Gravious has quit IRC | 22:46 | |
| *** Gravious has joined #openstack-infra | 22:47 | |
| panda | clarkb: I'm checking and commenting on the etherpad | 22:47 |
| EmilienM | panda: http://logs.openstack.org/43/446043/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/463d5a1/logs/undercloud/etc/ | 22:47 |
| clarkb | http://logs.openstack.org/43/446043/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/463d5a1/logs/subnode-1/etc/ looks like there is still a lot of extra etc stuff at least | 22:47 |
| EmilienM | yeah, same for undercloud | 22:47 |
| EmilienM | clarkb: but progress was made | 22:47 |
| EmilienM | clarkb: we're still working on the next steps and we're moving fast I think | 22:47 |
| *** xyang1 has quit IRC | 22:49 | |
| clarkb | I'd really like to see the suggested cleanups done (where applicable) since this already caused problems once | 22:49 |
| clarkb | there is literally no reason to have eg http://logs.openstack.org/43/446043/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/463d5a1/logs/subnode-1/etc/GREP_COLORS.txt.gz and friends | 22:49 |
| panda | clarkb: http://logs.openstack.org/66/445866/11/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a64e43c/ this is from a successful run | 22:49 |
| *** smarcet_ has joined #openstack-infra | 22:49 | |
| *** jascott1 has joined #openstack-infra | 22:50 | |
| panda | clarkb: so it's not only a size problem ? | 22:51 |
| *** gyee has joined #openstack-infra | 22:52 | |
| clarkb | panda: it is largely a size problem, but we got into it by grabbing all the things | 22:52 |
| clarkb | I'm asking we don't grab all the things then we avoid the size problem | 22:53 |
| *** Gravious has quit IRC | 22:53 | |
| EmilienM | I have to go, I'll read the backlog later | 22:54 |
| panda | EmilienM: where are the rendered tripleo templates stored ? | 22:54 |
| EmilienM | panda: what do you mean? sorry | 22:54 |
| panda | EmilienM: onevermind, see you tomorrow. | 22:54 |
| EmilienM | panda: no, tell me | 22:55 |
| EmilienM | panda: the THT templates are in /usr/share AFIK | 22:55 |
| EmilienM | panda: was it your question? | 22:55 |
| *** Wei_Liu has quit IRC | 22:55 | |
| panda | EmilienM: I remember a request to show the rendered templates on the logs | 22:56 |
| *** ihrachys has joined #openstack-infra | 22:56 | |
| *** Wei_Liu has joined #openstack-infra | 22:56 | |
| *** jtomasek has quit IRC | 22:56 | |
| EmilienM | oh that | 22:56 |
| panda | EmilienM: but maybe I'm confusing things | 22:56 |
| EmilienM | panda: I don't know if we did it or not (though I don't think so) | 22:56 |
| *** jascott1 has quit IRC | 22:56 | |
| *** jascott1 has joined #openstack-infra | 22:56 | |
| * EmilienM afk now | 22:56 | |
| *** smarcet_ has quit IRC | 22:57 | |
| clarkb | re heat templates http://logs.openstack.org/43/446043/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/463d5a1/logs/undercloud/usr/share/openstack-tripleo-heat-templates/ is what I am talking about | 22:57 |
| clarkb | seems like those are artifacts independent of oooq and can just be referred to? | 22:57 |
| *** lin_yang has quit IRC | 22:57 | |
| clarkb | (I think they are in their own git repo?) | 22:58 |
| panda | clarkb: yes, I remember a request to show rendered templates, but maybe it was not for the logs, and we don't have rendered templates anywhere | 22:58 |
| panda | clarkb: yes, but since they are templates, they are rendered at a certain point, and people wanted to see those, but maybe I'm just confusing tasks, and we don't have rendered templates anywhere right now | 22:59 |
| clarkb | gotcha | 22:59 |
| *** ijw has joined #openstack-infra | 23:02 | |
| *** pradk has quit IRC | 23:02 | |
| *** timello has quit IRC | 23:02 | |
| *** rfolco has joined #openstack-infra | 23:04 | |
| panda | clarkb: do you remember what item 4 was about ? | 23:05 |
| *** ihrachys has quit IRC | 23:07 | |
| *** ijw has quit IRC | 23:07 | |
| panda | clarkb: and for item 7, we were talking about creating docs per build, not per commit, but I realize this may be a bit too much | 23:07 |
| *** ihrachys has joined #openstack-infra | 23:08 | |
| clarkb | panda: yes I don't think you should do that | 23:08 |
| *** jianghuaw-m has joined #openstack-infra | 23:08 | |
| clarkb | panda: what is the problem with building the docs in its own job once per commit? | 23:08 |
| *** MasterOfBugs has quit IRC | 23:08 | |
| clarkb | (this is how most other projects do it) | 23:08 |
| clarkb | re item4 I think that was done. Before we were getting an entire archive of a virtualenv? something like that hwich isn't actually portable or reusable so suggestion was to collect info about what was installed to the virtualenv instead | 23:09 |
| panda | clarkb: I think the general idea was to have docs included in the logs so people could understand what was going on in their failed builds | 23:09 |
| *** timello has joined #openstack-infra | 23:09 | |
| panda | but was only a preliminary consideration | 23:09 |
| clarkb | panda: maybe link to the docs build instead? | 23:10 |
| clarkb | panda: all those urls should be deterministic/relative to the other build job logs | 23:10 |
| clarkb | (I think even a symlink would work) | 23:10 |
| *** ihrachys has quit IRC | 23:10 | |
| *** sekelso has quit IRC | 23:11 | |
| *** esikachev has joined #openstack-infra | 23:11 | |
| panda | clarkb: we'll check if that's enough, docs are built dinamically from bash scripts, and those may change per build | 23:12 |
| panda | clarkb: ack about item 4 | 23:12 |
| *** mriedem1 is now known as mriedem | 23:12 | |
| *** ihrachys has joined #openstack-infra | 23:12 | |
| *** ijw has joined #openstack-infra | 23:13 | |
| *** aeng has quit IRC | 23:13 | |
| *** rfolco has quit IRC | 23:14 | |
| *** esikachev has quit IRC | 23:16 | |
| panda | clarkb: created bug https://bugs.launchpad.net/tripleo/+bug/1673282, now these shouldn't be overlooked anymore | 23:17 |
| openstack | Launchpad bug 1673282 in tripleo "quickstart logging: implement compliance with infra requirements" [Critical,Triaged] | 23:17 |
| panda | clarkb: thanks | 23:17 |
| *** ihrachys has quit IRC | 23:17 | |
| *** asilenkov has quit IRC | 23:18 | |
| *** vhosakot has joined #openstack-infra | 23:20 | |
| *** asilenkov has joined #openstack-infra | 23:20 | |
| *** rwsu has quit IRC | 23:21 | |
| *** funzo has joined #openstack-infra | 23:21 | |
| *** smarcet has joined #openstack-infra | 23:22 | |
| *** sree has joined #openstack-infra | 23:24 | |
| *** funzo has quit IRC | 23:26 | |
| *** dimtruck is now known as zz_dimtruck | 23:26 | |
| *** zz_dimtruck is now known as dimtruck | 23:26 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: removed evict cache policy from data updates https://review.openstack.org/446224 | 23:28 |
| *** sree has quit IRC | 23:28 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: removed evict cache policy from data updates https://review.openstack.org/446224 | 23:29 |
| *** panda is now known as panda|Zz | 23:30 | |
| openstackgerrit | Markos Chandras proposed openstack-infra/jenkins-job-builder master: modules: triggers: Do not ignore non-dict triggers https://review.openstack.org/446225 | 23:30 |
| *** aeng has joined #openstack-infra | 23:30 | |
| *** ssbarnea has quit IRC | 23:32 | |
| *** dimtruck is now known as zz_dimtruck | 23:36 | |
| *** smarcet has quit IRC | 23:36 | |
| *** jianghuaw-m has quit IRC | 23:37 | |
| *** s-shiono_ has joined #openstack-infra | 23:37 | |
| *** jianghuaw-m has joined #openstack-infra | 23:38 | |
| *** ihrachys has joined #openstack-infra | 23:40 | |
| *** ihrachys has quit IRC | 23:44 | |
| openstackgerrit | Markos Chandras proposed openstack-infra/jenkins-job-builder master: modules: triggers: Do not ignore non-dict triggers https://review.openstack.org/446225 | 23:46 |
| ianw | fungi / frickler : is 446196 fast merge urgent? | 23:48 |
| *** dingyichen has joined #openstack-infra | 23:51 | |
| *** zz_dimtruck is now known as dimtruck | 23:55 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use sphinx warning-is-error https://review.openstack.org/445254 | 23:56 |
| *** jianghuaw-m has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!