| *** M4g1c5t0rM has joined #openstack-infra | 00:02 | |
| *** edgewing2 has joined #openstack-infra | 00:02 | |
| *** aeng has quit IRC | 00:03 | |
| *** edgewing_ has quit IRC | 00:03 | |
| *** claudiub|2 has quit IRC | 00:03 | |
| *** edgewing2 has quit IRC | 00:05 | |
| *** xarses has quit IRC | 00:05 | |
| *** edgewing has joined #openstack-infra | 00:06 | |
| *** salv-orlando has quit IRC | 00:06 | |
| corvus | this probably goes without saying, but i stopped uploads of zuul-project changes quite a while ago due to the backlog | 00:08 |
|---|---|---|
| *** dhill_ has quit IRC | 00:12 | |
| *** salv-orlando has joined #openstack-infra | 00:12 | |
| *** aspiers[m] has quit IRC | 00:13 | |
| *** it-praktyk[m] has quit IRC | 00:13 | |
| *** ilpianista_ has quit IRC | 00:13 | |
| fungi | thanks for the heads up | 00:13 |
| *** masayukig[m] has quit IRC | 00:13 | |
| fungi | and for thinking to do that! | 00:13 |
| *** salv-orlando has quit IRC | 00:15 | |
| *** aeng has joined #openstack-infra | 00:15 | |
| *** bandini has quit IRC | 00:16 | |
| *** salv-orlando has joined #openstack-infra | 00:16 | |
| pabelanger | we seem to be making good process on node requests now. Another few hours and we'll be caught back up. | 00:17 |
| *** salv-orlando has quit IRC | 00:17 | |
| *** salv-orlando has joined #openstack-infra | 00:17 | |
| *** bandini has joined #openstack-infra | 00:18 | |
| *** tbachman has quit IRC | 00:18 | |
| *** tbachman has joined #openstack-infra | 00:18 | |
| *** tbachman has left #openstack-infra | 00:18 | |
| *** hemna_ has joined #openstack-infra | 00:19 | |
| *** M4g1c5t0rM has quit IRC | 00:22 | |
| *** r-daneel has quit IRC | 00:23 | |
| *** ijw has joined #openstack-infra | 00:24 | |
| *** ijw has quit IRC | 00:29 | |
| fungi | rax hasn't picked up our ticket about the v6 issues yet | 00:30 |
| *** hongbin has quit IRC | 00:35 | |
| *** edgewing has quit IRC | 00:36 | |
| *** edgewing_ has joined #openstack-infra | 00:37 | |
| *** esberglu has quit IRC | 00:38 | |
| *** sshnaidm has joined #openstack-infra | 00:40 | |
| *** sshnaidm is now known as sshnaidm|off | 00:40 | |
| *** zhurong has joined #openstack-infra | 00:42 | |
| *** masayukig[m] has joined #openstack-infra | 00:44 | |
| *** aeng has quit IRC | 00:46 | |
| *** agopi has joined #openstack-infra | 00:49 | |
| *** hamzy_ has quit IRC | 00:50 | |
| *** tosky has quit IRC | 00:52 | |
| clarkb | cloudnull: ^ not sure if that interests you? | 00:56 |
| *** Goneri has joined #openstack-infra | 00:56 | |
| clarkb | cloudnull: tldr is ipv6 stopped working on a good chunk of our zuul executors | 00:56 |
| cloudnull | :( | 00:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unused grafana/nodepool files https://review.openstack.org/541421 | 00:56 |
| cloudnull | I'll send out some feelers. | 00:56 |
| *** iyamahat_ has joined #openstack-infra | 00:56 | |
| *** caphrim007 has joined #openstack-infra | 00:57 | |
| cloudnull | clarkb: which region or is it all of the above? | 00:57 |
| clarkb | cloudnull: dfw | 00:57 |
| clarkb | cloudnull: 180206-iad-0005440 is the ticket that fungi opened | 00:58 |
| clarkb | not sure what it says iad when servers were in dfw | 00:58 |
| cloudnull | ok. I'll go bother some folks. | 00:58 |
| fungi | it always says iad when i open a ticket under that tenant, even when i select an instance in dfw | 00:58 |
| clarkb | huh | 00:59 |
| fungi | cloudnull: that said, i haven't done extensive searching to see if i can find similar behavior in other regions. we noticed it in dfw because almost all of our control plane is in that region | 00:59 |
| *** iyamahat has quit IRC | 00:59 | |
| *** cuongnv has joined #openstack-infra | 01:00 | |
| fungi | it could be more widespread, and it could presumably also impact some ipv4 connections/flows but i've only seen direct evidence so far of it being limited to ipv6 | 01:00 |
| *** caphrim007 has quit IRC | 01:01 | |
| *** gmann_ has joined #openstack-infra | 01:03 | |
| cloudnull | messages sent. | 01:04 |
| cloudnull | I'll let you all know if there's something I find out. | 01:04 |
| clarkb | cloudnull: thankyou! | 01:04 |
| fungi | thanks cloudnull! we'll try to keep an eye on the ticket updates too | 01:05 |
| cloudnull | ++ | 01:05 |
| *** iyamahat has joined #openstack-infra | 01:07 | |
| *** iyamahat_ has quit IRC | 01:07 | |
| fungi | it's annoying for us but we've managed to get some temporary relief workarounds in place. i can imagine this likely isn't limited to just our instances though and could potentially be similarly impacting lots of customers there | 01:07 |
| *** wxy has joined #openstack-infra | 01:08 | |
| *** hemna_ has quit IRC | 01:08 | |
| *** gongysh has joined #openstack-infra | 01:10 | |
| clarkb | fungi: http://lists.openstack.org/pipermail/openstack-internships/2018-February/000129.html that is probably something we need to debug | 01:11 |
| *** ijw has joined #openstack-infra | 01:11 | |
| *** ilpianista_ has joined #openstack-infra | 01:11 | |
| *** aspiers[m] has joined #openstack-infra | 01:11 | |
| *** it-praktyk[m] has joined #openstack-infra | 01:11 | |
| clarkb | fungi: is that mm just saying it doesn't know what the mail addr is? maybe they used a .foo or +foo or whatever the cool kid method is? | 01:11 |
| fungi | huh, that's a list i didn't know about | 01:11 |
| clarkb | ya I didn't know about it either but jesusaur was kind enough to point it out | 01:11 |
| *** salv-orlando has quit IRC | 01:12 | |
| *** aviau has quit IRC | 01:12 | |
| *** Goneri has quit IRC | 01:12 | |
| *** aviau has joined #openstack-infra | 01:12 | |
| fungi | that ml isn't one we're rejecting -owner e-mails for, so wouldn't possibly be that | 01:13 |
| fungi | i wonder whether they sent the unsubscribe request to the wrong address... they don't say what/where they sent it | 01:13 |
| *** hamzy_ has joined #openstack-infra | 01:13 | |
| fungi | we can unsubscribe them via the cli easily enough | 01:14 |
| clarkb | ya, worst case a list admin can unsub or we can from the cli | 01:14 |
| clarkb | jinx | 01:14 |
| fungi | an e-mail to openstack-internships-request@lists.openstack.org with a subject line of just "unsibscribe" (no quote marks) should normally do the trick | 01:15 |
| fungi | as long as they send it from the same address they have subscribed | 01:15 |
| corvus | also spelled right | 01:15 |
| clarkb | need to spell unsubscribe correctly | 01:15 |
| fungi | d'oh | 01:15 |
| fungi | yeah, that | 01:15 |
| fungi | or they can use the unsubscribe button at http://lists.openstack.org/cgi-bin/mailman/options/openstack-internships | 01:16 |
| fungi | thingee is a listadmin for that one too, though he's not in here at the moment | 01:17 |
| *** ijw has quit IRC | 01:17 | |
| clarkb | lsell too | 01:17 |
| fungi | well, she's not in here right now either | 01:18 |
| clarkb | I can go ask them to respond since I am guessing they are subscribed | 01:18 |
| fungi | sounds good | 01:18 |
| *** gongysh has quit IRC | 01:20 | |
| *** jamesmcarthur has quit IRC | 01:21 | |
| *** links has joined #openstack-infra | 01:28 | |
| *** links has quit IRC | 01:29 | |
| *** slaweq has joined #openstack-infra | 01:32 | |
| *** daidv has joined #openstack-infra | 01:33 | |
| *** liujiong has joined #openstack-infra | 01:33 | |
| *** slaweq has quit IRC | 01:37 | |
| *** ijw has joined #openstack-infra | 01:39 | |
| *** liusheng has quit IRC | 01:40 | |
| *** ijw has quit IRC | 01:44 | |
| *** liujiong has quit IRC | 01:45 | |
| *** liujiong has joined #openstack-infra | 01:46 | |
| *** gmann_ has quit IRC | 01:46 | |
| *** gmann_ has joined #openstack-infra | 01:46 | |
| *** ijw has joined #openstack-infra | 01:47 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 01:47 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 01:52 |
| *** ijw has quit IRC | 01:52 | |
| dmsimard | I guess I was panicking for nothing looking at the Zuul ram graphs | 01:53 |
| dmsimard | ¯\_(ツ)_/¯ | 01:53 |
| * dmsimard knocks on wood | 01:53 | |
| fungi | node requests count is finally almost down to our aggregate quota, so should be starting most of the remaining backlogged jobs within whatever our average job runtime is | 01:55 |
| dmsimard | #status log (dmsimard) OVH BHS1 and GRA1 both recovered on their own and are back at full capacity. | 01:55 |
| openstackstatus | dmsimard: finished logging | 01:55 |
| dmsimard | fungi: the cool stuff is that we have the data to calculate this average :) | 01:55 |
| dmsimard | fungi: I've done it in *shivers* spreadsheets derived from Zuul sql reporter data | 01:56 |
| fungi | yeah, though eyeballing the burndown, it's ~1hr | 01:56 |
| fungi | maybe slightly more | 01:56 |
| fungi | or slightly less since that's also taking into account new uploads/rechecks/approvals | 01:56 |
| clarkb | graphite will generate you a graph where you can figure that out based on derivatives and stuff too | 01:56 |
| fungi | yup | 01:56 |
| clarkb | (it will genreate the derivative even) | 01:57 |
| fungi | dmsimard: can you reply to your earlier e-mail to jean-daniel and damien letting them know things are all clear there? | 01:57 |
| dmsimard | fungi: yup I was writing something already | 01:57 |
| fungi | thanks!!! | 01:57 |
| *** masber has quit IRC | 01:59 | |
| *** gyee has quit IRC | 02:01 | |
| mriedem | https://review.openstack.org/#/c/540490/ and https://review.openstack.org/#/c/537951/ have the same change id but https://review.openstack.org/#/q/Ib9f8a52136e25d1cb609d465ca5d859523d9acc6 only shows the one on master? | 02:03 |
| mriedem | is there something obvious that i'm missing? | 02:03 |
| clarkb | mriedem: I want ot say that data is cached | 02:04 |
| mriedem | oh yeah... | 02:04 |
| clarkb | mriedem: if they two changes were submitted some "distance" apart (and change numbers indicated they were) its possible that only the first was cached for that query? | 02:04 |
| *** jlabarre has quit IRC | 02:05 | |
| *** salv-orlando has joined #openstack-infra | 02:10 | |
| *** gongysh has joined #openstack-infra | 02:10 | |
| *** links has joined #openstack-infra | 02:12 | |
| cloudnull | clarkb: fungi: folks are looking into the ipv6 issue as we speak. hopefully more soon. | 02:13 |
| cloudnull | is there a list of nodes that are kknown broken ? | 02:13 |
| fungi | cloudnull: oh, good. thanks again! | 02:13 |
| fungi | cloudnull: depends on which is the "broken end" i guess | 02:13 |
| fungi | if a can talk to b and c but b and c can't talk to each other, does that mean b is broken? c? both? | 02:14 |
| *** salv-orlando has quit IRC | 02:14 | |
| *** jamesmcarthur has joined #openstack-infra | 02:15 | |
| fungi | cloudnull: the other executors exhibiting similar behavior to ze09.openstack.org were ze01, ze03, ze06, ze07 and ze08 but we've temporarily deleted their global v6 addresses as a workaround until this blows over | 02:15 |
| cloudnull | so it's east west traffic that is not working right? | 02:15 |
| cloudnull | I can't see the ticket | 02:15 |
| cloudnull | sorry if this is asked and answered. | 02:16 |
| fungi | cloudnull: at least east-west, i haven't observed anything to indicate it's also north-south | 02:16 |
| fungi | here, lemme put the report up on paste.o.o real fast | 02:16 |
| fungi | easier than reiterating | 02:16 |
| cloudnull | ok. thanks! | 02:16 |
| *** hongbin has joined #openstack-infra | 02:18 | |
| fungi | cloudnull: http://paste.openstack.org/show/664362/ | 02:19 |
| fungi | so i guess listing "broken" instances would depend on whether you mean those not able to send ipv6 to some destinations, those not able to receive ipv6 from some destinations, or both | 02:20 |
| *** mriedem has quit IRC | 02:24 | |
| cloudnull | ok. i've relayed that info to the folks looking into this. | 02:26 |
| fungi | clarkb suggested maybe cross-communication between some specific nova cells is broken but only in one direction. it also looks to me a lot like issues with hash-based flow balancing, but since we don't really know your architecture under there speculation is pretty pointless from our end ;) | 02:27 |
| cloudnull | I'd list "broken" as anything that's not working as it should :) | 02:27 |
| fungi | in this case broken combinatorial involves pairings | 02:28 |
| fungi | er, broken involves combinatorial pairings | 02:28 |
| fungi | so at least set{ze01,ze03,ze06,ze07,ze08,ze09}->set{review,zuul01} is broken | 02:29 |
| fungi | though we've neutered v6 on all in that first set but ze09 to relieve the pain | 02:30 |
| *** nmathew has joined #openstack-infra | 02:32 | |
| *** nmathew has quit IRC | 02:33 | |
| fungi | status on the ticket has changed from "new" to "transferred" in the last few minutes, so i suppose that means it's come to the attention of the noc or something | 02:33 |
| *** nmathew has joined #openstack-infra | 02:33 | |
| *** cuongnv has quit IRC | 02:33 | |
| *** cuongnv has joined #openstack-infra | 02:34 | |
| fungi | anyway, i need to disappear for some rest, but i expect infra-root folk in other timezones will be around should questions arise | 02:34 |
| *** kiennt26 has joined #openstack-infra | 02:34 | |
| cloudnull | take care fungi. | 02:35 |
| fungi | thanks for your help, as always cloudnull! | 02:35 |
| * cloudnull hopes it gets fixed soon | 02:35 | |
| cloudnull | anytime! | 02:35 |
| fungi | we're not in dire straits, it would just be nice to reenable v6 on those servers | 02:35 |
| ianw | fungi: i'll just log in now to watch the ticket | 02:36 |
| fungi | thanks to you too ianw! | 02:36 |
| ianw | in case of questions | 02:36 |
| ianw | that i probably can't answer, but will make something up anyway :) | 02:36 |
| fungi | meh, you have all the information i do, summarized there in the ticket | 02:36 |
| fungi | most complicated thing i did was to run tcpdump, ping6/ping and nc -6/nc -4 | 02:37 |
| gongysh | dims, hi | 02:38 |
| *** gk__ has joined #openstack-infra | 02:40 | |
| gk__ | Hi everyone, someone can tell me the nickname of the gerrit admin? Thank you. | 02:41 |
| openstackgerrit | Merged openstack-infra/project-config master: grafana/networking-odl: add graphs for ODL oxygen https://review.openstack.org/521697 | 02:41 |
| fungi | gk__: what do you need? | 02:41 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/pipelines.json route https://review.openstack.org/541521 | 02:41 |
| ianw | gk__: just ask, as anyone who can fix it will be in here :) | 02:41 |
| fungi | gk__: there are about a dozen of us in here who are gerrit admins, yes | 02:41 |
| gk__ | fungi: our third party CI keeps opening many connections to gerrit | 02:42 |
| fungi | disco.ci right? | 02:42 |
| gk__ | fungi: I would like to clean it and let our IT to check our network and the switch | 02:43 |
| gk__ | fungi: yes | 02:43 |
| fungi | account 19852 corresponds to username:disco.ci | 02:43 |
| fungi | i see no established connections from that account to the gerrit api currently | 02:44 |
| gk__ | fungi: ok, thanks | 02:44 |
| gk__ | fungi: I will contact our IT to start the monitoring | 02:45 |
| fungi | i'll check the logs to see what ip address you were coming from and whether there are preauthenticated sessions hung open from that address | 02:45 |
| gk__ | fungi: ok | 02:46 |
| fungi | looks like it last connected from 113.196.154.248 and i also see no open connections from that ip address | 02:46 |
| fungi | only one connection since utc midnight today | 02:47 |
| fungi | no connections yesterday or the day before | 02:47 |
| fungi | gk__: when was the last time it worked (ran a job, reported on something)? | 02:47 |
| gk__ | fungi: the last time it was reported was on 01/20 | 02:48 |
| gk__ | fungi: Is the only connection still opened? | 02:49 |
| fungi | no, it logged in at 01:25:45 (a little over an hour ago) and out at 01:27:20 a couple minutes later | 02:51 |
| *** nmathew has quit IRC | 02:51 | |
| *** sshnaidm_ has joined #openstack-infra | 02:51 | |
| fungi | but from a different ip address than it had been using up until january 24 | 02:51 |
| fungi | previously it seems to have been using 60.251.195.198 | 02:51 |
| gk__ | fungi: Can you tell me which is the ip address? | 02:52 |
| fungi | but i don't see any current connections at all from that ip address either | 02:52 |
| gk__ | fungi: ok | 02:52 |
| *** yamamoto has joined #openstack-infra | 02:52 | |
| *** sshnaidm|off has quit IRC | 02:52 | |
| fungi | last login from that address was 2018-01-23 02:09:13 | 02:54 |
| *** it-praktyk[m] has quit IRC | 02:54 | |
| *** aspiers[m] has quit IRC | 02:54 | |
| *** masayukig[m] has quit IRC | 02:54 | |
| *** ilpianista_ has quit IRC | 02:55 | |
| gk__ | fungi: ok, I will discuss it with our IT | 02:55 |
| *** yamamoto has quit IRC | 02:55 | |
| gk__ | fungi: thanks for the checking | 02:55 |
| fungi | gk__: i'm still checking a few things | 02:56 |
| *** yamamoto has joined #openstack-infra | 02:56 | |
| gk__ | fungi: ok | 02:56 |
| *** gaurangt has quit IRC | 02:57 | |
| *** bandini has quit IRC | 02:57 | |
| *** bandini has joined #openstack-infra | 02:59 | |
| *** gaurangt has joined #openstack-infra | 03:00 | |
| *** zhenguo has joined #openstack-infra | 03:01 | |
| *** jamesmcarthur has quit IRC | 03:01 | |
| *** liusheng has joined #openstack-infra | 03:03 | |
| fungi | gk__: yeah, i don't see any sign of incoming connections or established connection from there, either within gerrit or at the operating system level | 03:06 |
| fungi | so we don't seem to be blocking anything | 03:06 |
| gk__ | fungi: ok thank you | 03:06 |
| fungi | you're welcome. good luck and don't hesitate to let us know if you need anything else | 03:07 |
| *** sshnaidm_ has quit IRC | 03:08 | |
| *** ijw has joined #openstack-infra | 03:09 | |
| *** ijw has quit IRC | 03:14 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-graphite master: Fix up log rotation https://review.openstack.org/541488 | 03:16 |
| *** namnh has joined #openstack-infra | 03:18 | |
| *** tinwood has quit IRC | 03:20 | |
| fungi | cloudnull: ianw: i was apparently unable to let it go, and i found something intruiging. i added this to the ticket just now: | 03:20 |
| fungi | In looking for commonalities I've found more instances unable to receive IPv6 packets from ze09.openstack.org, and all are within the 2001:4800:7819:103::/64 prefix. The instances we observed unable to send packets to those all fall within 2001:4800:7818:104::/64. | 03:21 |
| *** tinwood has joined #openstack-infra | 03:21 | |
| fungi | so, er, looks like it may be that 2001:4800:7819:103::/64 is unable to get packets to 2001:4800:7818:104::/64, plain and simple | 03:21 |
| *** sshnaidm_ has joined #openstack-infra | 03:23 | |
| fungi | after noticing that review and zuul01 were in the same prefix, i started going through our ansible inventory on a hunch looking for more instances in the same /64 | 03:24 |
| *** abelur has quit IRC | 03:25 | |
| fungi | and then similarly, all the affected ze* instances shared a different common prefix so i looked for more instances in the inventory there | 03:26 |
| fungi | and it seems to match up | 03:26 |
| *** olaph has joined #openstack-infra | 03:27 | |
| ianw | fungi: excellent sleuthing :) as i've always said, there's no better monitoring tool than just giving infra some cloud resources :) | 03:27 |
| fungi | for example, eavesdrop01.openstack.org is in 2001:4800:7818:104 and elasticsearch03.openstack.org in 2001:4800:7819:103... black hole between them | 03:28 |
| *** olaph1 has quit IRC | 03:28 | |
| *** sree has joined #openstack-infra | 03:30 | |
| fungi | so anyway, i have made that conjecture in the ticket... now i'm _really_ disappearing | 03:32 |
| *** gk__ has quit IRC | 03:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Third stage of removing trove-integration https://review.openstack.org/540997 | 03:35 |
| *** coolsvap has joined #openstack-infra | 03:36 | |
| *** ramishra has joined #openstack-infra | 03:37 | |
| *** ianychoi_ has quit IRC | 03:49 | |
| *** ianychoi_ has joined #openstack-infra | 03:50 | |
| *** hbrown has joined #openstack-infra | 03:50 | |
| *** VW_ has joined #openstack-infra | 03:51 | |
| VW_ | fungi, clarkb - have some folks looking at the IPV6 shennanigans - can you test to see if the ipv6 traffice between review.openstack and ze09 is working? | 03:53 |
| *** VW_ is now known as VW | 03:53 | |
| VW | if you are still about | 03:53 |
| clarkb | VW: I can check in a sec | 03:54 |
| VW | sure - no rush. Just relaying the request from some networking folks over here | 03:55 |
| clarkb | VW: `ping6 review.openstack.org` works from ze09 now and I was able to clone a git repo from review.openstack.org on ze09 as well | 03:57 |
| clarkb | VW: let me check that pings work from review.openstack.org to ze09 next | 03:58 |
| *** abelur_ has quit IRC | 03:58 | |
| clarkb | VW: yes ping6 seems to work in both directions | 03:58 |
| *** abelur has joined #openstack-infra | 03:58 | |
| VW | awesome - thanks for checking clarkb | 03:59 |
| clarkb | VW: and thank you for looking into it :) | 03:59 |
| VW | I'm just the messanger who poked the right folks, but glad we could sort it | 04:01 |
| *** openstackgerrit has quit IRC | 04:04 | |
| *** liusheng has quit IRC | 04:06 | |
| *** liusheng has joined #openstack-infra | 04:07 | |
| *** jamesmcarthur has joined #openstack-infra | 04:11 | |
| *** rosmaita has quit IRC | 04:14 | |
| *** slaweq has joined #openstack-infra | 04:15 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** VW has quit IRC | 04:16 | |
| ianw | thanks clarkb | 04:16 |
| ianw | i guess there's no rush to put interfaces back | 04:16 |
| clarkb | I don't think so | 04:17 |
| *** VW has joined #openstack-infra | 04:17 | |
| *** andreas_s has joined #openstack-infra | 04:19 | |
| *** slaweq has quit IRC | 04:20 | |
| *** andreas_s has quit IRC | 04:24 | |
| *** cshastri has joined #openstack-infra | 04:25 | |
| *** psachin has joined #openstack-infra | 04:25 | |
| gongysh | dhellmann | 04:26 |
| gongysh | hi | 04:26 |
| gongysh | ttx hi | 04:28 |
| clarkb | gongysh: is there something we can help with? | 04:29 |
| gongysh | how is tacker listed on https://releases.openstack.org/queens/index.html? | 04:29 |
| *** ykarel|away has joined #openstack-infra | 04:30 | |
| clarkb | https://git.openstack.org/cgit/openstack/releases/tree/deliverables/queens is what determines that I think | 04:30 |
| *** cuongnv has quit IRC | 04:31 | |
| gongysh | ok, I will submit a patch for it. | 04:31 |
| *** harlowja has quit IRC | 04:32 | |
| *** nicolasbock has quit IRC | 04:32 | |
| *** hbrown has quit IRC | 04:34 | |
| *** slaweq has joined #openstack-infra | 04:36 | |
| *** iyamahat has quit IRC | 04:38 | |
| clarkb | check is under 100 changes now | 04:39 |
| *** calebb has quit IRC | 04:41 | |
| *** slaweq has quit IRC | 04:42 | |
| *** namnh has quit IRC | 04:42 | |
| *** thorre has quit IRC | 04:43 | |
| *** thorre has joined #openstack-infra | 04:43 | |
| *** daidv has quit IRC | 04:44 | |
| *** janki has joined #openstack-infra | 04:46 | |
| *** kiennt26 has quit IRC | 04:46 | |
| *** calebb has joined #openstack-infra | 04:49 | |
| *** caphrim007 has joined #openstack-infra | 04:49 | |
| *** slaweq has joined #openstack-infra | 04:52 | |
| *** dsariel has quit IRC | 04:52 | |
| *** pgadiya has joined #openstack-infra | 04:53 | |
| *** caphrim007 has quit IRC | 04:54 | |
| *** slaweq has quit IRC | 04:57 | |
| *** eumel8 has joined #openstack-infra | 04:59 | |
| ianw | fungi / corvus : re the bad cacti data on ze02.openstack.org ... been digging through logs | 04:59 |
| ianw | try "snmpwalk -v1 -c public udp6:ze02.openstack.org .1.3.6.1.2.1.1.3.0" on cacti.o.o | 04:59 |
| *** edgewing2 has joined #openstack-infra | 05:00 | |
| *** edgewing2 has quit IRC | 05:00 | |
| ianw | as often as not, it fails : http://paste.openstack.org/show/664542/ | 05:01 |
| *** hongbin has quit IRC | 05:02 | |
| *** thorre has quit IRC | 05:03 | |
| *** thorre has joined #openstack-infra | 05:03 | |
| *** janki has quit IRC | 05:05 | |
| *** openstackgerrit has joined #openstack-infra | 05:06 | |
| openstackgerrit | Merged openstack-infra/zuul master: Sync when doing disk accountant testing https://review.openstack.org/541486 | 05:06 |
| *** janki has joined #openstack-infra | 05:06 | |
| ianw | that's between 2001:4800:7821:105:be76:4eff:fe04:b9a5/64 cacti -> 2001:4800:7819:104:be76:4eff:fe05:302/64 ze02 | 05:06 |
| *** slaweq has joined #openstack-infra | 05:08 | |
| *** zhenguo has quit IRC | 05:10 | |
| *** slaweq has quit IRC | 05:12 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use nested tempfile fixture for cleanups https://review.openstack.org/541487 | 05:13 |
| dmsimard | ianw: I remember something like that. I think if you're starting to query too many things you need to use getbulk instead of get ? | 05:19 |
| dmsimard | I've had to fix that on Nagios in a past life | 05:19 |
| *** claudiub|2 has joined #openstack-infra | 05:22 | |
| *** slaweq has joined #openstack-infra | 05:23 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix stuck node requests across ZK reconnection https://review.openstack.org/541454 | 05:24 |
| ianw | dmsimard: yeah ... but this is just one parameter, or whatever snmp calls it ... it seems like the packets always get cacti -> ze02 ... but not the other way around | 05:28 |
| *** slaweq has quit IRC | 05:28 | |
| *** kazsh_ has joined #openstack-infra | 05:28 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Move Magnum irc meeting to #openstack-containers https://review.openstack.org/541210 | 05:29 |
| dmsimard | ianw: it's the size of the answer, packet is too large for the window or something like that | 05:29 |
| *** kazsh_ has quit IRC | 05:29 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Revert "Test Use item.checkout from zuul.projects when mirroring" https://review.openstack.org/540950 | 05:30 |
| dmsimard | SNMP reflection is one of the most common DDoS techniques, along with NTP reflection :D | 05:30 |
| * dmsimard was DDoS'd by IP faxes and printers | 05:30 | |
| ianw | dmsimard: but it's only the uptime. and also, no other host appears to suffer from this | 05:30 |
| ianw | if you checkout the errors in the cacti logs, ze02 is wayyyy over represented | 05:31 |
| *** kjackal has joined #openstack-infra | 05:31 | |
| ianw | 16000 Host:'ze02.openstack.org', | 05:32 |
| ianw | 330 Host:'ze07.openstack.org', | 05:32 |
| ianw | (error rates grepped from the logs, top two results) | 05:32 |
| dmsimard | Hum, looping or stale process on ze02 ? On my phone :( | 05:33 |
| ianw | nup, i've restarted it several times. it really seems to be the network inbetween | 05:34 |
| *** VW_ has joined #openstack-infra | 05:34 | |
| *** VW has quit IRC | 05:34 | |
| ianw | need to double confirm by tracing packet flows, but that's where i'm at with it anyway. | 05:35 |
| *** harlowja has joined #openstack-infra | 05:37 | |
| *** caphrim007 has joined #openstack-infra | 05:38 | |
| *** zhurong has quit IRC | 05:39 | |
| *** VW_ has quit IRC | 05:40 | |
| *** jbadiapa has joined #openstack-infra | 05:42 | |
| *** olaph1 has joined #openstack-infra | 05:44 | |
| *** olaph has quit IRC | 05:45 | |
| *** jaosorior has joined #openstack-infra | 05:50 | |
| *** zhurong has joined #openstack-infra | 05:53 | |
| *** slaweq has joined #openstack-infra | 05:55 | |
| *** thorre has quit IRC | 05:58 | |
| *** thorre has joined #openstack-infra | 05:58 | |
| *** dhajare has joined #openstack-infra | 05:58 | |
| *** slaweq has quit IRC | 06:00 | |
| *** lathiat has quit IRC | 06:06 | |
| *** VW has joined #openstack-infra | 06:11 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** e0ne has joined #openstack-infra | 06:11 | |
| *** annp has joined #openstack-infra | 06:13 | |
| *** VW has quit IRC | 06:15 | |
| prometheanfire | you guys rate limit gerrit ssh connections? | 06:15 |
| prometheanfire | ssh://prometheanfire@review.openstack.org:29418/openstack-infra/project-config.git did not work. Description: Received disconnect from UNKNOWN port 65535:12: Too many concurrent connections (64) - max. allowed: 64 | 06:15 |
| prometheanfire | I was syncing all the repos and git reviewing them | 06:16 |
| *** slaweq has quit IRC | 06:16 | |
| prometheanfire | for prj in $(ssh review.openstack.org -qt "gerrit ls-projects" | grep -v All-Projects | grep -v All-Users | grep -v API-Projects); do d=$(dirname $prj); mkdir -p $d; (cd $d; git clone git://git.openstack.org/$prj); (cd $prj; git review -s </dev/null || true); done | 06:16 |
| prometheanfire | tonyb: might as well talk in here about it | 06:16 |
| *** e0ne has quit IRC | 06:17 | |
| *** jchhatbar has joined #openstack-infra | 06:19 | |
| *** jchhatbar has quit IRC | 06:20 | |
| *** lathiat has joined #openstack-infra | 06:20 | |
| *** jchhatbar has joined #openstack-infra | 06:21 | |
| *** VW has joined #openstack-infra | 06:21 | |
| *** janki has quit IRC | 06:21 | |
| *** ilpianista_ has joined #openstack-infra | 06:21 | |
| *** aspiers[m] has joined #openstack-infra | 06:23 | |
| *** masayukig[m] has joined #openstack-infra | 06:23 | |
| ianw | prometheanfire: you do only get so many connections, yep. as long as you're not leaking them, which has happened | 06:24 |
| prometheanfire | I don't see anything on my end | 06:25 |
| ianw | fungi: pretty convinced after looking at tcpdumps (http://paste.openstack.org/show/664605/) that there's also asymmetrical ipv6 packet loss between cacti & ze02. i've updated the ticket ... a hard reboot might fix this one | 06:25 |
| * prometheanfire is wondering how they were leaked | 06:25 | |
| *** it-praktyk[m] has joined #openstack-infra | 06:25 | |
| prometheanfire | I wonder if the git review -s crashing out was the cause | 06:25 |
| *** VW has quit IRC | 06:26 | |
| *** slaweq has joined #openstack-infra | 06:26 | |
| *** slaweq has quit IRC | 06:31 | |
| *** cuongnv has joined #openstack-infra | 06:36 | |
| *** sshnaidm_ has quit IRC | 06:37 | |
| *** sshnaidm_ has joined #openstack-infra | 06:41 | |
| *** slaweq has joined #openstack-infra | 06:42 | |
| *** slaweq has quit IRC | 06:47 | |
| *** e0ne has joined #openstack-infra | 06:48 | |
| *** hjensas has quit IRC | 06:52 | |
| *** harlowja has quit IRC | 06:52 | |
| *** e0ne has quit IRC | 06:54 | |
| *** sshnaidm_ has quit IRC | 06:55 | |
| *** VW has joined #openstack-infra | 06:56 | |
| *** slaweq has joined #openstack-infra | 06:58 | |
| *** dsariel has joined #openstack-infra | 06:58 | |
| prometheanfire | fungi: here's how you dos yourself out of gerrit, you'll hit your 64 connection limit. | 06:59 |
| prometheanfire | fungi: for prj in $(ssh review.openstack.org -qt "gerrit ls-projects" | grep -v All-Projects | grep -v All-Users | grep -v API-Projects); do d=$(dirname $prj); mkdir -p $d; (cd $d; git clone git://git.openstack.org/$prj); (cd $prj; git review -s </dev/null || true); done | 06:59 |
| prometheanfire | fungi: works without -t on the first ssh, also, controlmaster is enabled by default in my config :D | 07:00 |
| * ianw might have suggested that fungi loves weird bash/pty/fd duping issues :) | 07:01 | |
| *** VW has quit IRC | 07:01 | |
| prometheanfire | :D | 07:03 |
| prometheanfire | I think that's enough details on it for now | 07:03 |
| *** slaweq has quit IRC | 07:03 | |
| prometheanfire | ianw: let him know how you had to clear it too if you don't mind | 07:03 |
| ianw | review.o.o had the connection still ESTABLISHED ... my first guess is a zombie pty or something. i killed them with gerrit's close-connection | 07:04 |
| *** ykarel|away is now known as ykarel | 07:07 | |
| prometheanfire | fungi: 01:06 < ianw > review.o.o had the connection still ESTABLISHED ... my first guess is a zombie pty or something. i killed them with gerrit's close-connection | 07:08 |
| prometheanfire | thanks | 07:08 |
| AJaeger | ianw: could you review an older new repo request, please? they pinged me - https://review.openstack.org/511335 | 07:09 |
| AJaeger | ianw: and I wonder what you think about https://review.openstack.org/#/c/541000/ and https://review.openstack.org/#/c/540994/ - some nodepool jenkins cleanups. | 07:10 |
| *** alexchadin has joined #openstack-infra | 07:10 | |
| *** sshnaidm_ has joined #openstack-infra | 07:10 | |
| *** slaweq has joined #openstack-infra | 07:14 | |
| *** jbadiapa has quit IRC | 07:15 | |
| *** slaweq has quit IRC | 07:18 | |
| *** andreas_s has joined #openstack-infra | 07:26 | |
| *** VW has joined #openstack-infra | 07:27 | |
| *** sree has quit IRC | 07:28 | |
| *** slaweq has joined #openstack-infra | 07:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove jenkins/scripts/run-tox.sh https://review.openstack.org/540609 | 07:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove jenkins/scripts/run-tox.sh https://review.openstack.org/540609 | 07:31 |
| *** VW has quit IRC | 07:32 | |
| *** slaweq has quit IRC | 07:35 | |
| *** armaan has joined #openstack-infra | 07:36 | |
| kota_ | hello | 07:36 |
| kota_ | can i have a way to re-push a tag for release management? | 07:36 |
| prometheanfire | kota_: I'd ask smcginnis in #openstack-requirements but he's likely asleep | 07:37 |
| kota_ | I'v been trying to do a new release and then, push a tag to gerrit according to the docs at https://docs.openstack.org/infra/manual/drivers.html | 07:37 |
| kota_ | prometheanfire: hmm | 07:38 |
| prometheanfire | it's a managed project right? | 07:38 |
| kota_ | i found my typo in the tag message but no way for now I can find to delete or force to overwrite the tag. | 07:38 |
| kota_ | yeah | 07:38 |
| prometheanfire | ya, I'd ask in that channel then | 07:39 |
| prometheanfire | tags are supposed to be immutable for a reason though, but if it's just a comment change who knows | 07:39 |
| *** jtomasek has joined #openstack-infra | 07:40 | |
| kota_ | yeah, just a comment change | 07:40 |
| *** agopi has quit IRC | 07:43 | |
| AJaeger | kota_: what exactly is wrong with the tag? | 07:43 |
| *** slaweq has joined #openstack-infra | 07:43 | |
| *** slaweq has quit IRC | 07:44 | |
| AJaeger | kota_: I see - just leave it as is | 07:44 |
| *** slaweq has joined #openstack-infra | 07:44 | |
| kota_ | AJaeger: I had misspelled as "storelts", correctly "storlets" in the 1.0.0 release | 07:45 |
| kota_ | https://github.com/openstack/storlets/releases/tag/1.0.0 | 07:45 |
| kota_ | the project name :/ | 07:45 |
| kota_ | my god... | 07:45 |
| *** slaweq_ has joined #openstack-infra | 07:45 | |
| * kota_ sigh | 07:46 | |
| AJaeger | kota_: very few will notice. You can retag on the command line but I don't know whether gerrit allows repuplish. Release team might know - or some of the admins here but most are out now - best ask again at US time. | 07:47 |
| kota_ | i know that is not significant issue, just feels as sigh on the first 1.0.0 release. | 07:47 |
| kota_ | AJaeger: ok thx | 07:47 |
| AJaeger | kota_: yes, understand that | 07:47 |
| kota_ | I'll try to ask again here in US time. | 07:48 |
| *** rcernin has quit IRC | 07:48 | |
| AJaeger | kota_: reading further about git: IT probably won't work since deletion of tags is not possible. As soon as you push it to gerrit, it's distributed everywhere and then you cannot delete. Let's see whether somebody has an idea. | 07:49 |
| *** edgewing2 has joined #openstack-infra | 07:49 | |
| *** weshay has quit IRC | 07:50 | |
| *** slaweq_ has quit IRC | 07:50 | |
| *** pabelanger has quit IRC | 07:50 | |
| *** _ari__ has quit IRC | 07:51 | |
| *** rnoriega has quit IRC | 07:51 | |
| *** otherwiseguy_ has quit IRC | 07:51 | |
| *** myoung|off has quit IRC | 07:51 | |
| *** hjensas has joined #openstack-infra | 07:51 | |
| *** hjensas has quit IRC | 07:51 | |
| *** hjensas has joined #openstack-infra | 07:51 | |
| *** haleyb has quit IRC | 07:51 | |
| *** migi has quit IRC | 07:52 | |
| liujiong | hi infra team, our zuul job cannot recognize git revisions, any thought on this? | 07:53 |
| liujiong | http://logs.openstack.org/58/473658/19/check/openstack-tox-cover/0f6849c/job-output.txt.gz#_2018-02-07_05_43_43_163295 | 07:53 |
| *** jpena|off is now known as jpena | 07:59 | |
| AJaeger | liujiong: check http://logs.openstack.org/58/473658/19/check/openstack-tox-cover/0f6849c/job-output.txt.gz#_2018-02-07_05_40_59_201311 - you have git revisions but no *origin* | 08:00 |
| *** fultonj has quit IRC | 08:01 | |
| *** slaweq_ has joined #openstack-infra | 08:01 | |
| *** sree has joined #openstack-infra | 08:01 | |
| *** myoung has joined #openstack-infra | 08:02 | |
| *** haleyb has joined #openstack-infra | 08:02 | |
| *** _ari_ has joined #openstack-infra | 08:02 | |
| *** otherwiseguy has joined #openstack-infra | 08:02 | |
| *** pabelanger has joined #openstack-infra | 08:02 | |
| *** rnoriega has joined #openstack-infra | 08:02 | |
| *** migi has joined #openstack-infra | 08:03 | |
| *** weshay has joined #openstack-infra | 08:04 | |
| *** alexchadin has quit IRC | 08:04 | |
| *** VW has joined #openstack-infra | 08:05 | |
| *** caphrim007 has quit IRC | 08:06 | |
| *** slaweq_ has quit IRC | 08:06 | |
| *** slaweq has quit IRC | 08:06 | |
| *** slaweq has joined #openstack-infra | 08:07 | |
| *** VW has quit IRC | 08:09 | |
| *** zoli is now known as zoli|trng | 08:10 | |
| *** zoli|trng is now known as zoli | 08:10 | |
| *** slaweq has quit IRC | 08:11 | |
| *** amoralej|off is now known as amoralej | 08:13 | |
| *** annp has quit IRC | 08:14 | |
| *** cuongnv has quit IRC | 08:14 | |
| *** ianychoi_ has quit IRC | 08:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 08:14 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 08:14 |
| *** annp has joined #openstack-infra | 08:14 | |
| *** cuongnv has joined #openstack-infra | 08:15 | |
| *** alexchadin has joined #openstack-infra | 08:16 | |
| *** slaweq has joined #openstack-infra | 08:17 | |
| *** ianychoi_ has joined #openstack-infra | 08:19 | |
| *** sshnaidm_ is now known as sshnaidm | 08:19 | |
| *** dbecker has joined #openstack-infra | 08:20 | |
| *** slaweq has quit IRC | 08:22 | |
| *** threestrands has quit IRC | 08:22 | |
| liujiong | AJaeger: that's pretty weird, what does "mirror-workspace-git-repos" job do in our ci system? | 08:23 |
| *** tesseract has joined #openstack-infra | 08:24 | |
| *** zhurong has quit IRC | 08:25 | |
| *** zhurong has joined #openstack-infra | 08:25 | |
| AJaeger | liujiong: see http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/mirror-workspace-git-repos | 08:28 |
| *** ianychoi_ has quit IRC | 08:28 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 08:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 08:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 08:28 |
| *** neiljerram has quit IRC | 08:29 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 08:29 |
| AJaeger | frickler, jhesketh, could you please review https://review.openstack.org/538344 - final step before we can merge https://review.openstack.org/540609 | 08:29 |
| *** ianychoi_ has joined #openstack-infra | 08:30 | |
| liujiong | AJaeger: thx! | 08:30 |
| *** slaweq has joined #openstack-infra | 08:33 | |
| *** hashar has joined #openstack-infra | 08:33 | |
| *** pcaruana has joined #openstack-infra | 08:38 | |
| *** slaweq has quit IRC | 08:38 | |
| *** VW has joined #openstack-infra | 08:38 | |
| *** liujiong has quit IRC | 08:39 | |
| *** alexchadin has quit IRC | 08:39 | |
| *** VW has quit IRC | 08:43 | |
| *** ianychoi_ has quit IRC | 08:45 | |
| *** ianychoi_ has joined #openstack-infra | 08:46 | |
| *** slaweq has joined #openstack-infra | 08:49 | |
| *** dingyichen has quit IRC | 08:52 | |
| *** slaweq has quit IRC | 08:53 | |
| *** ralonsoh has joined #openstack-infra | 08:55 | |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** jamesmcarthur has joined #openstack-infra | 08:59 | |
| *** rossella_s has quit IRC | 08:59 | |
| *** rpittau has quit IRC | 09:01 | |
| *** jamesmcarthur has quit IRC | 09:04 | |
| *** slaweq has joined #openstack-infra | 09:04 | |
| jhesketh | AJaeger: looking | 09:04 |
| *** florianf has joined #openstack-infra | 09:08 | |
| AJaeger | thanks, jhesketh | 09:08 |
| *** dbecker has quit IRC | 09:08 | |
| *** slaweq has quit IRC | 09:09 | |
| *** stakeda has quit IRC | 09:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 09:09 |
| *** VW has joined #openstack-infra | 09:11 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove now obsolete jenkins-sudo-grep lines https://review.openstack.org/538344 | 09:11 |
| *** VW has quit IRC | 09:15 | |
| *** slaweq has joined #openstack-infra | 09:20 | |
| openstackgerrit | Lenny Verkhovsky proposed openstack/diskimage-builder master: Checking link status according to DIB_DHCP_TIMEOUT https://review.openstack.org/529976 | 09:23 |
| *** slaweq has quit IRC | 09:25 | |
| *** e0ne has joined #openstack-infra | 09:26 | |
| *** slaweq has joined #openstack-infra | 09:31 | |
| *** slaweq_ has joined #openstack-infra | 09:36 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Refactor status functions, add web endpoints, allow params https://review.openstack.org/536301 | 09:38 |
| *** slaweq has quit IRC | 09:38 | |
| *** slaweq has joined #openstack-infra | 09:38 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 09:39 |
| *** slaweq_ has quit IRC | 09:41 | |
| *** panda|off is now known as panda | 09:41 | |
| *** slaweq has quit IRC | 09:43 | |
| *** derekh has joined #openstack-infra | 09:44 | |
| *** VW has joined #openstack-infra | 09:46 | |
| *** bhavik1 has joined #openstack-infra | 09:48 | |
| *** askb_ has quit IRC | 09:48 | |
| *** abelur_ has joined #openstack-infra | 09:49 | |
| *** VW has quit IRC | 09:51 | |
| *** slaweq has joined #openstack-infra | 09:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove jenkins-sudo https://review.openstack.org/540994 | 09:53 |
| *** ykarel is now known as ykarel|lunch | 09:54 | |
| *** kopecmartin has joined #openstack-infra | 09:54 | |
| *** bhavik1 has quit IRC | 09:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: New project Tatu (SSH as a Service). https://review.openstack.org/511335 | 09:55 |
| *** olaph has joined #openstack-infra | 09:55 | |
| *** slaweq has quit IRC | 09:56 | |
| *** pcaruana has quit IRC | 09:57 | |
| *** olaph1 has quit IRC | 09:57 | |
| *** namnh has joined #openstack-infra | 09:57 | |
| *** alexchadin has joined #openstack-infra | 10:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove zuul-sudo-grep lines from nodepool https://review.openstack.org/541000 | 10:01 |
| *** olaph1 has joined #openstack-infra | 10:05 | |
| *** olaph has quit IRC | 10:06 | |
| *** slaweq has joined #openstack-infra | 10:08 | |
| *** sambetts|afk is now known as sambetts | 10:10 | |
| *** dougwig has quit IRC | 10:11 | |
| *** pgadiya_ has joined #openstack-infra | 10:12 | |
| *** slaweq has quit IRC | 10:13 | |
| *** pcaruana has joined #openstack-infra | 10:13 | |
| *** pgadiya has quit IRC | 10:13 | |
| *** strigazi_ has joined #openstack-infra | 10:16 | |
| *** VW has joined #openstack-infra | 10:17 | |
| *** strigazi has quit IRC | 10:19 | |
| *** strigazi_ is now known as strigazi | 10:19 | |
| *** gfidente has joined #openstack-infra | 10:20 | |
| *** ykarel|lunch is now known as ykarel | 10:20 | |
| *** VW has quit IRC | 10:21 | |
| *** ianychoi_ has quit IRC | 10:21 | |
| *** annp has quit IRC | 10:22 | |
| *** dbecker has joined #openstack-infra | 10:23 | |
| *** slaweq has joined #openstack-infra | 10:23 | |
| *** ianychoi_ has joined #openstack-infra | 10:24 | |
| *** dtantsur|afk is now known as dtantsur | 10:26 | |
| *** cuongnv has quit IRC | 10:27 | |
| *** slaweq has quit IRC | 10:28 | |
| *** gcb has quit IRC | 10:29 | |
| *** dizquierdo has joined #openstack-infra | 10:30 | |
| *** gongysh has quit IRC | 10:33 | |
| *** yamahata has quit IRC | 10:34 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant project group definition example and definition in the doc https://review.openstack.org/535730 | 10:34 |
| *** zhurong has quit IRC | 10:35 | |
| *** slaweq has joined #openstack-infra | 10:39 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 10:41 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: [WIP] webapp: add optional admin endpoint https://review.openstack.org/536319 | 10:42 |
| *** M4g1c5t0rM has joined #openstack-infra | 10:42 | |
| *** neiljerram has joined #openstack-infra | 10:43 | |
| *** ldnunes has joined #openstack-infra | 10:43 | |
| *** slaweq has quit IRC | 10:44 | |
| neiljerram | Morning all. | 10:44 |
| neiljerram | In gertty, how do I respond to pending comments on a previous patchset? | 10:44 |
| neiljerram | (Where by 'pending' I mean those that I haven't previously handled.) | 10:45 |
| *** lucas-afk is now known as lucasagomes | 10:45 | |
| *** nuggers has joined #openstack-infra | 10:46 | |
| *** sree has quit IRC | 10:52 | |
| *** sree has joined #openstack-infra | 10:53 | |
| sshnaidm | pabelanger, ping me please wrt docker proxy when you're online | 10:53 |
| *** pgadiya_ has quit IRC | 10:53 | |
| *** VW has joined #openstack-infra | 10:53 | |
| *** slaweq has joined #openstack-infra | 10:55 | |
| *** slaweq_ has joined #openstack-infra | 10:55 | |
| *** slaweq has quit IRC | 10:56 | |
| *** stevebaker has quit IRC | 10:56 | |
| *** slaweq has joined #openstack-infra | 10:57 | |
| *** sree has quit IRC | 10:57 | |
| *** VW has quit IRC | 10:57 | |
| *** slaweq has quit IRC | 10:59 | |
| *** andreas_s has quit IRC | 10:59 | |
| *** slaweq__ has joined #openstack-infra | 10:59 | |
| *** tpsilva has joined #openstack-infra | 10:59 | |
| *** andreas_s has joined #openstack-infra | 10:59 | |
| *** neiljerram has quit IRC | 11:00 | |
| *** slaweq_ has quit IRC | 11:00 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 11:01 |
| *** tosky has joined #openstack-infra | 11:01 | |
| *** slaweq__ has quit IRC | 11:04 | |
| *** stevebaker has joined #openstack-infra | 11:04 | |
| *** ianychoi_ has quit IRC | 11:05 | |
| *** pgadiya_ has joined #openstack-infra | 11:05 | |
| sambetts | Hey Infra is there gerrit -> storyboard intergration? e.g. can I do Closes-Bug: <storyboard id> etc to related a change to a story?? | 11:07 |
| *** ianychoi_ has joined #openstack-infra | 11:07 | |
| *** zoli is now known as zoli|lunch | 11:08 | |
| *** olaph has joined #openstack-infra | 11:08 | |
| *** andreas_s has quit IRC | 11:09 | |
| *** pcaruana has quit IRC | 11:09 | |
| frickler | sambetts: almost, you can use "Story: <story id>" + "Task: <task id>". But there also is https://storyboard.openstack.org/#!/story/2001411 making this only partially useful currently | 11:09 |
| *** olaph1 has quit IRC | 11:09 | |
| sambetts | frickler: ah ok, so we're getting there :D | 11:10 |
| *** slaweq has joined #openstack-infra | 11:11 | |
| openstackgerrit | Sam Betts proposed openstack-infra/zuul master: Ensure shlex.split takes comments into account https://review.openstack.org/541669 | 11:11 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Do not call merger:cat when all config items are excluded https://review.openstack.org/535509 | 11:12 |
| *** andreas_s has joined #openstack-infra | 11:14 | |
| *** slaweq has quit IRC | 11:15 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Zuul: Remove project name https://review.openstack.org/540381 | 11:19 |
| AJaeger | sambetts: should be documented in infra-manual | 11:21 |
| *** dmellado has quit IRC | 11:21 | |
| AJaeger | sambetts: https://docs.openstack.org/infra/manual/developers.html#working-on-bugs if you need a doc to share | 11:21 |
| sambetts | AJaeger: yeah, it was faster to ask than dig through the guides :) | 11:21 |
| AJaeger | sambetts: ;) | 11:21 |
| sambetts | AJaeger: thanks I pass the info on :) | 11:22 |
| AJaeger | ianw: thanks for reviews | 11:22 |
| *** slaweq has joined #openstack-infra | 11:27 | |
| *** dmellado has joined #openstack-infra | 11:27 | |
| *** namnh has quit IRC | 11:28 | |
| *** andreas_s has quit IRC | 11:28 | |
| *** andreas_s has joined #openstack-infra | 11:29 | |
| *** VW has joined #openstack-infra | 11:31 | |
| *** slaweq has quit IRC | 11:32 | |
| *** andreas_s has quit IRC | 11:34 | |
| *** VW has quit IRC | 11:36 | |
| *** andreas_s has joined #openstack-infra | 11:38 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: [WIP] webapp: add optional admin endpoint https://review.openstack.org/536319 | 11:41 |
| *** nicolasbock has joined #openstack-infra | 11:41 | |
| *** bcafarel has left #openstack-infra | 11:42 | |
| *** nuggers has quit IRC | 11:42 | |
| *** slaweq has joined #openstack-infra | 11:42 | |
| openstackgerrit | Lenny Verkhovsky proposed openstack/diskimage-builder master: Checking link status according to DIB_DHCP_TIMEOUT https://review.openstack.org/529976 | 11:43 |
| *** dizquierdo has quit IRC | 11:44 | |
| *** slaweq has quit IRC | 11:47 | |
| *** dmellado has quit IRC | 11:48 | |
| *** nuggers has joined #openstack-infra | 11:50 | |
| *** slaweq has joined #openstack-infra | 11:51 | |
| *** slaweq has quit IRC | 11:56 | |
| *** M4g1c5t0rM has quit IRC | 11:56 | |
| *** dbecker has quit IRC | 11:58 | |
| *** slaweq has joined #openstack-infra | 11:58 | |
| *** pgadiya_ has quit IRC | 11:59 | |
| *** dmellado has joined #openstack-infra | 11:59 | |
| *** pgadiya has joined #openstack-infra | 11:59 | |
| *** ykarel has quit IRC | 11:59 | |
| *** ykarel has joined #openstack-infra | 12:00 | |
| *** ianychoi_ has quit IRC | 12:00 | |
| *** edmondsw has joined #openstack-infra | 12:02 | |
| *** slaweq has quit IRC | 12:03 | |
| *** oidgar has joined #openstack-infra | 12:03 | |
| *** M4g1c5t0rM has joined #openstack-infra | 12:03 | |
| *** VW has joined #openstack-infra | 12:05 | |
| *** rfolco|off is now known as rfolco|ruck | 12:09 | |
| *** M4g1c5t0rM has quit IRC | 12:09 | |
| *** wolverineav has joined #openstack-infra | 12:10 | |
| *** VW has quit IRC | 12:10 | |
| *** dave-mccowan has joined #openstack-infra | 12:12 | |
| *** pcaruana has joined #openstack-infra | 12:13 | |
| *** slaweq has joined #openstack-infra | 12:14 | |
| *** alexchadin has quit IRC | 12:16 | |
| *** dave-mcc_ has joined #openstack-infra | 12:16 | |
| *** alexchadin has joined #openstack-infra | 12:16 | |
| *** dave-mccowan has quit IRC | 12:16 | |
| *** yamamoto has quit IRC | 12:17 | |
| *** yamamoto has joined #openstack-infra | 12:17 | |
| *** panda is now known as panda|lunch | 12:18 | |
| *** slaweq has quit IRC | 12:18 | |
| *** zoli|lunch is now known as zoli | 12:20 | |
| *** andreas_s_ has joined #openstack-infra | 12:20 | |
| *** hemna_ has joined #openstack-infra | 12:20 | |
| *** alexchadin has quit IRC | 12:21 | |
| *** alexchadin has joined #openstack-infra | 12:21 | |
| *** andreas_s has quit IRC | 12:23 | |
| *** nuggers has quit IRC | 12:26 | |
| Shrews | what, no fires to fight this morning? how dull | 12:29 |
| cmurphy | don't jinx it :P | 12:30 |
| *** slaweq has joined #openstack-infra | 12:30 | |
| Shrews | :) | 12:30 |
| *** dizquierdo has joined #openstack-infra | 12:30 | |
| *** oidgar has quit IRC | 12:34 | |
| *** slaweq has quit IRC | 12:35 | |
| *** jpena is now known as jpena|lunch | 12:39 | |
| *** VW has joined #openstack-infra | 12:40 | |
| *** VW has quit IRC | 12:44 | |
| *** slaweq has joined #openstack-infra | 12:45 | |
| AJaeger | Shrews: let's restart everything with IPv6 enabled ;) | 12:46 |
| *** nuggers has joined #openstack-infra | 12:46 | |
| *** olaph1 has joined #openstack-infra | 12:47 | |
| smcginnis | Looks like we had another release job timeout on a git command. | 12:48 |
| smcginnis | ui | 12:48 |
| smcginnis | http://logs.openstack.org/75/751450b7b55719fd0b37c02c651097ba75917b38/release-post/tag-releases/a2678e5/job-output.txt.gz#_2018-02-07_11_14_42_697495 | 12:49 |
| smcginnis | Would someone be able to rerun that one? | 12:49 |
| *** oidgar has joined #openstack-infra | 12:49 | |
| *** slaweq_ has joined #openstack-infra | 12:50 | |
| *** slaweq has quit IRC | 12:50 | |
| *** olaph has quit IRC | 12:50 | |
| *** slaweq_ has quit IRC | 12:50 | |
| *** slaweq has joined #openstack-infra | 12:51 | |
| *** alexchadin has quit IRC | 12:53 | |
| *** alexchadin has joined #openstack-infra | 12:54 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/puppet-zuul master: Remove 'restart' for v3 executor https://review.openstack.org/541707 | 12:55 |
| Shrews | pabelanger: ^^^ | 12:55 |
| *** sree has joined #openstack-infra | 12:58 | |
| *** alexchadin has quit IRC | 12:58 | |
| *** cshastri has quit IRC | 12:58 | |
| *** jamesmcarthur has joined #openstack-infra | 13:00 | |
| *** panda|lunch is now known as panda | 13:01 | |
| *** slaweq_ has joined #openstack-infra | 13:01 | |
| *** slaweq has quit IRC | 13:02 | |
| *** slaweq has joined #openstack-infra | 13:03 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** rosmaita has joined #openstack-infra | 13:06 | |
| *** slaweq_ has quit IRC | 13:06 | |
| *** slaweq has quit IRC | 13:08 | |
| *** pgadiya has quit IRC | 13:09 | |
| smcginnis | Nevermind my previous rerun request. It was for a patch where that will not be necessary. | 13:10 |
| *** sree has quit IRC | 13:14 | |
| *** sree has joined #openstack-infra | 13:14 | |
| *** sree has quit IRC | 13:14 | |
| *** sree has joined #openstack-infra | 13:15 | |
| *** gongysh has joined #openstack-infra | 13:15 | |
| *** sree has quit IRC | 13:16 | |
| *** sree has joined #openstack-infra | 13:17 | |
| *** slaweq has joined #openstack-infra | 13:17 | |
| *** VW has joined #openstack-infra | 13:17 | |
| *** dhajare has quit IRC | 13:21 | |
| *** sree has quit IRC | 13:21 | |
| *** slaweq has quit IRC | 13:22 | |
| *** VW has quit IRC | 13:23 | |
| *** dbecker has joined #openstack-infra | 13:24 | |
| *** rossella_s has joined #openstack-infra | 13:25 | |
| *** fultonj has joined #openstack-infra | 13:25 | |
| *** fultonj has quit IRC | 13:26 | |
| *** jamesmcarthur has joined #openstack-infra | 13:27 | |
| *** VW has joined #openstack-infra | 13:27 | |
| *** VW has quit IRC | 13:29 | |
| *** jlabarre has joined #openstack-infra | 13:31 | |
| *** slaweq has joined #openstack-infra | 13:33 | |
| *** jamesmcarthur has quit IRC | 13:33 | |
| *** dhajare has joined #openstack-infra | 13:34 | |
| *** amoralej is now known as amoralej|lunch | 13:35 | |
| *** jamesmcarthur has joined #openstack-infra | 13:36 | |
| *** efried has quit IRC | 13:36 | |
| *** fultonj has joined #openstack-infra | 13:37 | |
| *** hjensas has quit IRC | 13:37 | |
| *** eharney has quit IRC | 13:37 | |
| *** slaweq has quit IRC | 13:37 | |
| *** sree has joined #openstack-infra | 13:38 | |
| *** jpena|lunch is now known as jpena | 13:39 | |
| *** jamesmcarthur has quit IRC | 13:39 | |
| *** jamesmcarthur has joined #openstack-infra | 13:39 | |
| *** jcoufal has joined #openstack-infra | 13:39 | |
| *** coolsvap has quit IRC | 13:41 | |
| *** dhajare has quit IRC | 13:42 | |
| *** jchhatba_ has joined #openstack-infra | 13:45 | |
| *** rossella_s has quit IRC | 13:45 | |
| *** sree has quit IRC | 13:47 | |
| *** rossella_s has joined #openstack-infra | 13:47 | |
| *** jchhatbar has quit IRC | 13:48 | |
| *** slaweq has joined #openstack-infra | 13:49 | |
| fungi | Shrews: we _likely_ don't need to restart to get the v6 addresses back and can just readd them live from the command line. i only rebooted ze09 to do that so i was absolutely sure there was nothing weird on our end before using it to report the bug | 13:49 |
| fungi | smcginnis: did dhellmann's changes to make remote git operations in those jobs more robust merge yet? | 13:50 |
| *** dhajare has joined #openstack-infra | 13:50 | |
| smcginnis | fungi: Yep. This most recent failure was from somewhere else though. | 13:51 |
| smcginnis | fungi: Turns out it wasn't an actual release, so the failure is fine on that one. | 13:51 |
| fungi | k | 13:52 |
| *** slaweq has quit IRC | 13:53 | |
| *** kiennt26 has joined #openstack-infra | 13:53 | |
| *** alexchadin has joined #openstack-infra | 13:54 | |
| *** alexchadin has quit IRC | 13:55 | |
| *** alexchadin has joined #openstack-infra | 13:56 | |
| *** alexchadin has quit IRC | 13:56 | |
| *** alexchadin has joined #openstack-infra | 13:56 | |
| *** alexchadin has quit IRC | 13:57 | |
| *** markvoelker has joined #openstack-infra | 13:57 | |
| *** alexchadin has joined #openstack-infra | 13:57 | |
| *** alexchadin has quit IRC | 13:57 | |
| *** alexchadin has joined #openstack-infra | 13:58 | |
| *** alexchadin has quit IRC | 13:58 | |
| *** yamahata has joined #openstack-infra | 13:59 | |
| *** ijw has joined #openstack-infra | 13:59 | |
| *** alexchadin has joined #openstack-infra | 13:59 | |
| *** alexchadin has quit IRC | 13:59 | |
| *** pcichy has quit IRC | 14:00 | |
| *** alexchadin has joined #openstack-infra | 14:02 | |
| *** ijw has quit IRC | 14:03 | |
| *** r-daneel has joined #openstack-infra | 14:04 | |
| *** nuggers has quit IRC | 14:04 | |
| *** slaweq has joined #openstack-infra | 14:04 | |
| *** nuggers has joined #openstack-infra | 14:04 | |
| *** Goneri has joined #openstack-infra | 14:04 | |
| *** dhajare has quit IRC | 14:05 | |
| *** iyamahat has joined #openstack-infra | 14:06 | |
| fungi | infra-root: we'll want to be on the lookout for this in a week and probably make sure we get it patched quickly on systems which accept e-mail from the wild: http://www.openwall.com/lists/oss-security/2018/02/07/2 | 14:08 |
| *** jamesmcarthur has quit IRC | 14:09 | |
| fungi | also worth considering whether now is a good time to upgrade lists.o.o to xenial, or whether we expect ubuntu to still patch it in trusty just as quickly | 14:09 |
| *** slaweq has quit IRC | 14:09 | |
| *** links has quit IRC | 14:09 | |
| *** jamesmcarthur has joined #openstack-infra | 14:10 | |
| *** r-daneel has quit IRC | 14:12 | |
| *** daidv_ has joined #openstack-infra | 14:13 | |
| *** esberglu has joined #openstack-infra | 14:15 | |
| *** rossella_s has quit IRC | 14:16 | |
| openstackgerrit | Liam Young proposed openstack-infra/project-config master: Add vault charm to gerrit https://review.openstack.org/541287 | 14:17 |
| *** e0ne has quit IRC | 14:17 | |
| *** ianychoi has joined #openstack-infra | 14:17 | |
| *** jamesmcarthur has quit IRC | 14:18 | |
| *** rossella_s has joined #openstack-infra | 14:18 | |
| openstackgerrit | Liam Young proposed openstack-infra/project-config master: Add vault charm to gerrit https://review.openstack.org/541287 | 14:18 |
| *** slaweq has joined #openstack-infra | 14:20 | |
| *** dougwig has joined #openstack-infra | 14:20 | |
| *** esberglu has quit IRC | 14:21 | |
| *** hjensas has joined #openstack-infra | 14:23 | |
| *** hjensas has quit IRC | 14:23 | |
| *** hjensas has joined #openstack-infra | 14:23 | |
| *** eharney has joined #openstack-infra | 14:24 | |
| *** slaweq has quit IRC | 14:25 | |
| *** r-daneel has joined #openstack-infra | 14:31 | |
| *** slaweq has joined #openstack-infra | 14:32 | |
| *** esberglu has joined #openstack-infra | 14:32 | |
| *** slaweq has quit IRC | 14:32 | |
| *** slaweq has joined #openstack-infra | 14:33 | |
| *** oidgar has quit IRC | 14:35 | |
| *** lucasagomes is now known as lucas-hungry | 14:35 | |
| *** VW has joined #openstack-infra | 14:35 | |
| *** psachin has quit IRC | 14:36 | |
| *** slaweq_ has joined #openstack-infra | 14:36 | |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add support to change the default time filter range https://review.openstack.org/530798 | 14:37 |
| *** slaweq has quit IRC | 14:37 | |
| *** dhill_ has joined #openstack-infra | 14:38 | |
| *** slaweq_ has quit IRC | 14:40 | |
| *** hashar is now known as hasharAway | 14:41 | |
| frickler | bindep checks seem to be failing on debian-jessie due to broken packages, if someone wants to dig into that. otherwise I'll propose to make that check non-voting for now. http://logs.openstack.org/77/539777/1/check/bindep-fallback-debian-jessie/f1110a8/job-output.txt.gz#_2018-02-07_11_24_45_868795 | 14:41 |
| *** kopecmartin has quit IRC | 14:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove jenkins/scripts/run-tox.sh https://review.openstack.org/540609 | 14:43 |
| *** alexchadin has quit IRC | 14:44 | |
| AJaeger | infra-root, run-tox.sh is gone, we're now using only Zuul v3 native tox jobs! | 14:44 |
| AJaeger | config-core, could you review https://review.openstack.org/540647 to add missing required-project, please? | 14:45 |
| AJaeger | fungi, could you review two small infra-manual changes, please? https://review.openstack.org/#/c/540668/ and https://review.openstack.org/#/c/535725/ | 14:46 |
| *** amoralej|lunch is now known as amoralej | 14:46 | |
| dmsimard | AJaeger: worthy of a #success IMO :) | 14:47 |
| AJaeger | #success All tox jobs have been converted to Zuul v3 native syntax, run-tox.sh is gone. | 14:48 |
| openstackstatus | AJaeger: Added success to Success page (https://wiki.openstack.org/wiki/Successes) | 14:48 |
| AJaeger | dmsimard: good idea! | 14:48 |
| *** olaph has joined #openstack-infra | 14:49 | |
| *** jamesmcarthur has joined #openstack-infra | 14:50 | |
| *** olaph1 has quit IRC | 14:50 | |
| *** slaweq has joined #openstack-infra | 14:52 | |
| *** rossella_s has quit IRC | 14:52 | |
| *** rossella_s has joined #openstack-infra | 14:54 | |
| *** e0ne has joined #openstack-infra | 14:56 | |
| *** slaweq has quit IRC | 14:56 | |
| *** olaph1 has joined #openstack-infra | 14:56 | |
| *** zoli is now known as zoli|afk | 14:57 | |
| *** zoli|afk is now known as zoli | 14:57 | |
| *** olaph has quit IRC | 14:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove jenkins from DIB images https://review.openstack.org/514485 | 14:59 |
| *** felipemonteiro has joined #openstack-infra | 15:00 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:01 | |
| *** VW_ has joined #openstack-infra | 15:01 | |
| AJaeger | pabelanger: updated your change after I split out two changes ;) ^ | 15:01 |
| AJaeger | team, are we ready to move forward with removing jenkins? ^ | 15:02 |
| *** VW_ has quit IRC | 15:02 | |
| *** caphrim007 has joined #openstack-infra | 15:02 | |
| *** VW_ has joined #openstack-infra | 15:02 | |
| *** felipemonteiro has quit IRC | 15:05 | |
| *** VW has quit IRC | 15:05 | |
| dmsimard | Where do we submit bugs for the openstack.org website again ? | 15:08 |
| *** slaweq has joined #openstack-infra | 15:08 | |
| *** shardy has joined #openstack-infra | 15:08 | |
| pabelanger | AJaeger: thanks, I'll review shortly but we should do that after queens release to be safe | 15:09 |
| AJaeger | dmsimard: https://launchpad.net/openstack-org | 15:10 |
| *** yamamoto has quit IRC | 15:10 | |
| AJaeger | pabelanger: do you want to WIP again? | 15:10 |
| pabelanger | Shrews: looks good, but left a -1 with comments | 15:11 |
| pabelanger | AJaeger: sure | 15:11 |
| dmsimard | AJaeger: you are an ever ending source of knowledge <3 | 15:11 |
| AJaeger | pabelanger: I made one small change - as requested by ianw: Moving chmod around, so please double check that | 15:11 |
| Shrews | pabelanger: ah, what an excellect point | 15:11 |
| *** dhill__ has joined #openstack-infra | 15:12 | |
| AJaeger | pabelanger: we might want to remove nodepool/elements/nodepool-base/post-install.d/89-jenkins-scripts now in that case - to avoid jobs depending on it without installating them | 15:12 |
| *** dhill_ has quit IRC | 15:12 | |
| *** slaweq has quit IRC | 15:12 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/puppet-zuul master: Remove 'restart' for v3 executor https://review.openstack.org/541707 | 15:13 |
| *** kiennt26 has quit IRC | 15:14 | |
| *** sree has joined #openstack-infra | 15:14 | |
| *** david-lyle has quit IRC | 15:15 | |
| AJaeger | pabelanger: not sure whether we're ready for that - commenting | 15:15 |
| *** rossella_s has quit IRC | 15:16 | |
| *** coolsvap has joined #openstack-infra | 15:17 | |
| *** oidgar has joined #openstack-infra | 15:19 | |
| *** VW_ has quit IRC | 15:19 | |
| *** rossella_s has joined #openstack-infra | 15:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove zuul-sudo-grep.sh https://review.openstack.org/541746 | 15:21 |
| AJaeger | pabelanger, dmsimard, small cleanup, could you review, please? ^ | 15:21 |
| *** camunoz has joined #openstack-infra | 15:22 | |
| *** gongysh has quit IRC | 15:23 | |
| *** felipemonteiro__ has joined #openstack-infra | 15:23 | |
| *** slaweq has joined #openstack-infra | 15:23 | |
| pabelanger | AJaeger: where are we copying zuul-sudo-grep.sh from? | 15:25 |
| pabelanger | having trouble seeing that | 15:25 |
| *** mriedem has joined #openstack-infra | 15:26 | |
| *** yamamoto has joined #openstack-infra | 15:26 | |
| mordred | pabelanger: we copy the whole jenkins/scripts directory to /usr/local/jenkins/slave_scripts | 15:26 |
| *** felipemonteiro_ has quit IRC | 15:26 | |
| AJaeger | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/zuul-sudo-grep.sh | 15:27 |
| AJaeger | mordred: where??? | 15:27 |
| mordred | pabelanger: oh - wait - nevermind. I'm stupid | 15:27 |
| *** sree_ has joined #openstack-infra | 15:27 | |
| mordred | stil waking up | 15:27 |
| mordred | pabelanger: listen to AJaeger , not to me | 15:27 |
| *** sree_ is now known as Guest53871 | 15:27 | |
| pabelanger | oh, wow. I would never have looked there for that file | 15:27 |
| AJaeger | mordred: that's what I just commented on 514485 - we do not copy over the directory and thus would break lots of jobs if we remoe it from nodepool | 15:27 |
| *** slaweq has quit IRC | 15:28 | |
| AJaeger | so, for that single file we're fine - but for all the others we need more investigation. | 15:28 |
| pabelanger | slow and steady, we'll get there :) | 15:28 |
| pabelanger | +2 | 15:28 |
| AJaeger | pabelanger: yep, one file at a time... | 15:29 |
| AJaeger | (with my tox cleanup I removed 5 or 6 from that directory ;) | 15:29 |
| *** jbadiapa has joined #openstack-infra | 15:29 | |
| *** yamamoto has quit IRC | 15:30 | |
| mordred | \o/ | 15:30 |
| mordred | +A | 15:30 |
| *** sree has quit IRC | 15:30 | |
| AJaeger | thanks, pabelanger and mordred | 15:31 |
| *** lucas-hungry is now known as lucasagomes | 15:32 | |
| pabelanger | clarkb: fungi: looks like rackspace networking is fixed? At least according to backscroll, do we want to start zuul-executor on ze09? | 15:32 |
| AJaeger | could I get another +2 on https://review.openstack.org/540647 ? adds a missing required-project | 15:32 |
| AJaeger | pabelanger: +1 | 15:32 |
| *** salv-orlando has joined #openstack-infra | 15:32 | |
| *** panda is now known as panda|nappychang | 15:38 | |
| *** VW has joined #openstack-infra | 15:39 | |
| *** slaweq has joined #openstack-infra | 15:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove zuul-sudo-grep.sh https://review.openstack.org/541746 | 15:41 |
| *** Guest53871 has quit IRC | 15:41 | |
| pabelanger | sshnaidm: best to ask your question in channel for another infra-root to pick up too, we all have access logs for apache mirrors | 15:41 |
| *** sree has joined #openstack-infra | 15:42 | |
| *** slaweq has quit IRC | 15:43 | |
| mordred | AJaeger: do you have a sec to help me figure out the right way to fix a proposal job? | 15:44 |
| mordred | AJaeger: http://logs.openstack.org/50/50f29bc8288b20c4b470cd626a7859110b0c310c/post/propose-updates/fc39ff6/job-output.txt.gz#_2018-02-07_05_21_51_685518 is the error | 15:44 |
| mordred | AJaeger: it's for proposing updates to os-service-types when patches are landed to service-types-authority | 15:44 |
| mordred | AJaeger: the script is running from inside of src/git.openstack.org/openstack/service-types-authority since that's what triggered the job | 15:45 |
| *** dhill__ has quit IRC | 15:45 | |
| *** jbadiapa has quit IRC | 15:45 | |
| mriedem | zuul question, this ansible task didn't seem to have done anything, https://review.openstack.org/#/c/541474/1/playbooks/legacy/nova-next/run.yaml@26 http://logs.openstack.org/74/541474/1/check/nova-next/39df0c0/job-output.txt.gz#_2018-02-07_02_25_21_574704 even though we have other jobs that do the exact same thing with injecting a local.conf: https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-multiattach/run | 15:45 |
| mriedem | ml#L26 | 15:45 |
| mriedem | are my eyes missing something obvious? | 15:45 |
| AJaeger | mordred: checking quickly... | 15:46 |
| *** xarses has joined #openstack-infra | 15:46 | |
| mordred | AJaeger: should we add a propose-service-types-update that adds os-service-types to required-projects? or should we just clone it in the script? | 15:47 |
| *** xarses_ has joined #openstack-infra | 15:47 | |
| *** dhill_ has joined #openstack-infra | 15:48 | |
| *** sree has quit IRC | 15:48 | |
| *** panda|nappychang is now known as panda | 15:48 | |
| *** ramishra has quit IRC | 15:49 | |
| AJaeger | mordred: I'm confused right now ;/ | 15:50 |
| *** xarses has quit IRC | 15:51 | |
| mordred | AJaeger: I htink I see it - the update script is already doing the cloning, so the repo is there - but before the update function is called the script has popd'd back out of the directory | 15:51 |
| AJaeger | mordred: this is run on a checked out copy of os-service-types | 15:51 |
| mordred | AJaeger: yah. it's making more sense to me now | 15:51 |
| AJaeger | mordred: yep, indeed | 15:51 |
| *** bhavik1 has joined #openstack-infra | 15:52 | |
| *** dizquierdo has quit IRC | 15:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Change directories to os-service-types during update https://review.openstack.org/541759 | 15:54 |
| mordred | AJaeger: I think that should do it ^^ | 15:54 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move translation scripts out of jenkins/scripts https://review.openstack.org/541760 | 15:54 |
| AJaeger | mordred: will review - could you review mine, please? ;) ^ | 15:55 |
| *** slaweq has joined #openstack-infra | 15:55 | |
| AJaeger | mordred: LGTM | 15:56 |
| corvus | mriedem: if you look at http://logs.openstack.org/74/541474/1/check/nova-next/39df0c0/job-output.json and filter for 'dg-local.conf' you can confirm that the cat command ran as expected | 15:58 |
| *** slaweq_ has joined #openstack-infra | 15:58 | |
| mriedem | hmm, wtf, the contents are for a totally different job | 15:59 |
| *** felipemonteiro__ has quit IRC | 15:59 | |
| mriedem | it's cat'ing out the nova-multiattach localrc | 16:00 |
| mriedem | not the nova-next job localrc | 16:00 |
| *** jchhatba_ has quit IRC | 16:00 | |
| *** efried has joined #openstack-infra | 16:00 | |
| clarkb | pabelanger: ya I tested pings from review to ze09 and ze09 to review and it worked. I also did a git clone on ze09 against review.o.o. I think we likely can turn on the executor there and if it works reenable ipv6 on the others | 16:00 |
| *** slaweq has quit IRC | 16:00 | |
| mriedem | "cmd": "set -e\nset -x\ncat << 'EOF' >>\"/tmp/dg-local.conf\"\n[[local|localrc]]\nENABLE_VOLUME_MULTIATTACH=True\nENABLE_UBUNTU_CLOUD_ARCHIVE=False\n\nEOF\n", | 16:00 |
| mriedem | ^ is not at all what's in https://review.openstack.org/#/c/541474/1/playbooks/legacy/nova-next/run.yaml@26 | 16:00 |
| corvus | mriedem: oh yes, i agree those are different words. that's something we should be able to dig into | 16:02 |
| mriedem | OH i see the problem | 16:02 |
| mriedem | https://review.openstack.org/#/c/541474/1/.zuul.yaml@101 | 16:02 |
| *** vhosakot has joined #openstack-infra | 16:02 | |
| mriedem | corvus: thanks for the tip | 16:02 |
| mriedem | copy/paste error on my part | 16:02 |
| pabelanger | clarkb: sounds good | 16:02 |
| pabelanger | will do that now | 16:02 |
| corvus | mriedem: yay i helped! :) | 16:03 |
| *** slaweq_ has quit IRC | 16:03 | |
| *** pcaruana has quit IRC | 16:03 | |
| pabelanger | okay, ze09 started and looks to be cloning properly | 16:06 |
| *** hongbin has joined #openstack-infra | 16:07 | |
| *** andreas_s_ has quit IRC | 16:07 | |
| AJaeger | clarkb: do you want to remove the remaining two grafana boards from http://paste.openstack.org/show/663889/ today? | 16:07 |
| *** andreas_s has joined #openstack-infra | 16:08 | |
| EmilienM | pabelanger, mordred: who owns https://galaxy.ansible.com/openstack/ ? | 16:08 |
| clarkb | AJaeger: if the changes merged yes! I didn't get to them yesterday as they change hadn't merged yet | 16:08 |
| EmilienM | sdoran: ^ fyi | 16:08 |
| AJaeger | clarkb: merged now, thanks | 16:08 |
| pabelanger | EmilienM: I think mordred registered it | 16:08 |
| pabelanger | EmilienM: however, anybody can import projects into the namesspace I think | 16:09 |
| fungi | back to computer and catching up | 16:09 |
| mordred | pabelanger, EmilienM: it's connected to the github openstack org | 16:10 |
| mordred | because you cannot have things in galaxy any way other than github at the moment (and I haven't had time to fix galaxy) | 16:10 |
| *** slaweq has joined #openstack-infra | 16:11 | |
| mordred | pabelanger, EmilienM: however, last time I tried to run the galaxy 'look at the projects in the github org and register them' process it broke galaxy because we have a LOT of repos in our org | 16:11 |
| pabelanger | yah, when I last talked to ansible folks about galaxy, it really was a thin wrapper to github (APIs) | 16:11 |
| mordred | yes. it's one of my least favorite things about galaxy and I have complained MANY times about it | 16:11 |
| dmsimard | pabelanger: it's still that last I know | 16:11 |
| mordred | but then they open sourced galaxy and said "patches welcome" | 16:12 |
| dmsimard | pabelanger, mordred: puppet forge isn't that much different though, is it ? | 16:12 |
| pabelanger | mordred: yah | 16:12 |
| mordred | dmsimard: it is - you can totally upload things to puppet forge | 16:12 |
| *** sree has joined #openstack-infra | 16:12 | |
| dmsimard | however for some reasons I feel the puppet module "ecosystem" is better than what Ansible tried to do with Galaxy and I can't quite put my finger on it | 16:12 |
| pabelanger | galaxy doesn't cache or store anything, it is all on github | 16:12 |
| *** andreas_s has quit IRC | 16:12 | |
| dmsimard | mordred, pabelanger: right but ultimately you click on "project url" on the forge and it's on github like 90% of the time | 16:13 |
| dmsimard | at least it was last I know | 16:13 |
| mordred | dmsimard: yah - but it doesn't have to be | 16:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move translation scripts out of jenkins/scripts https://review.openstack.org/541760 | 16:13 |
| AJaeger | mordred: could you review again, please? Whitespace fixes;( ^ | 16:13 |
| mordred | dmsimard: that's just where people happen to have their stuff | 16:13 |
| mordred | dmsimard: but with galaxy there is literally no mechanism to import anything other than github | 16:13 |
| *** ykarel has quit IRC | 16:14 | |
| dmsimard | mordred: yeah but even if it wasn't for that, I generally feel like role sharing doesn't work as good as puppet module sharing | 16:14 |
| mordred | and as much as I find that displeasurable, even giving in and trying to use our github replicas it still doesn't work | 16:14 |
| dmsimard | "real" role re-usability is hard | 16:14 |
| mordred | dmsimard: I agree - I think a good amount of that may be age of ecosystem too - when we first started puppeting in infra the puppet module ecosystem was MUCH worse | 16:14 |
| fungi | frickler: thanks for the heads up on the bindep-fallback-debian-jessie job. i'll see if i can coax a more detailed error out of it and track that down shortly | 16:15 |
| dmsimard | I guess to some extent it's explained by the fact that Ansible has *modules* and *roles* while for Puppet, they're mostly one and the same.. like puppetlabs-apache, puppetlabs-mysql, etc. | 16:15 |
| fungi | AJaeger: congrats and thanks on legacy job detoxification! | 16:15 |
| AJaeger | ;) | 16:15 |
| *** slaweq has quit IRC | 16:15 | |
| dmsimard | For Ansible, modules and maintained in-tree upstream and roles are in this bit of an awkward spot | 16:16 |
| dmsimard | s/modules and/modules are/ | 16:16 |
| *** sree has quit IRC | 16:17 | |
| clarkb | AJaeger: the two extra dashboards have now been deleted | 16:17 |
| *** sree has joined #openstack-infra | 16:18 | |
| *** david-lyle has joined #openstack-infra | 16:18 | |
| AJaeger | thanks, clarkb ! | 16:18 |
| AJaeger | now we have a cleaner grafana board - great | 16:19 |
| *** katkapilatova has joined #openstack-infra | 16:19 | |
| kklimonda | is there a way to get a number of "buildsets" (sets of jobs) per project, per branch, per day? I'm trying to estimate how much load on our CI would it be to run tests for neutron | 16:20 |
| fungi | pabelanger: yes, reviewing the bug clarkb was on point that it was related to nova cells... connectivity from the cell ze09 and others were in couldn't reach a couple of specific v6 prefixes. reported as fixed in the ticket now and ianw confirmed it for ze09 at least. we should be able to readd v6 addresses to the other 5 executors where we did that temporary workaround (and i believe we should be | 16:20 |
| fungi | able to do it live without restarting anything) | 16:20 |
| EmilienM | mordred: yeah, sdoran and I are interested to register https://github.com/openstack/ansible-role-redhat-subscription when possible | 16:21 |
| *** andreas_s has joined #openstack-infra | 16:21 | |
| rcarrillocruz | oh look, sdoran is over here! | 16:22 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Correct django template pattern in babel-django.cfg https://review.openstack.org/540668 | 16:22 |
| EmilienM | :) | 16:22 |
| sdoran | I get around these days. 😉 | 16:22 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update retirement section for project description https://review.openstack.org/535725 | 16:22 |
| pabelanger | fungi: okay, I need to get more coffee, but happy to help re-enable ipv6 on our executors. Do we have any syntax to do so? | 16:23 |
| mordred | EmilienM: k. trying to do it by hand again | 16:24 |
| EmilienM | thanks | 16:25 |
| *** sree has quit IRC | 16:25 | |
| clarkb | pabelanger: I think rax writes that stuff to the /etc/network/interfaces config so a systemctl restart network may work? | 16:25 |
| clarkb | that shouldn't break any existing connections | 16:25 |
| clarkb | fungi: ^ | 16:25 |
| fungi | clarkb: ifup would probably work, though i was just going to use ip to readd the address | 16:26 |
| clarkb | that works too. So many options :) | 16:26 |
| *** ijw has joined #openstack-infra | 16:26 | |
| *** slaweq has joined #openstack-infra | 16:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Install install-distro-packages.sh only for legacy jobs https://review.openstack.org/541780 | 16:27 |
| fungi | mainly because those interfaces already have v4 addresses and the network config files may have trouble getting reapplied to already up interfaces since they may attempt to readd the addresses which are already active on them | 16:27 |
| *** ijw has quit IRC | 16:28 | |
| fungi | though we could test on an unimportant machine and find out i suppose | 16:28 |
| *** katkapilatova has quit IRC | 16:28 | |
| *** ijw has joined #openstack-infra | 16:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove install-distro-packages.sh from images https://review.openstack.org/541781 | 16:28 |
| clarkb | I'm good with just ip addr add. I just tend to be lazy and thought systemd might be able to do it for us :) | 16:28 |
| clarkb | but ya not sure if ipv4 already being there will make it have a sad | 16:29 |
| *** bhavik1 has quit IRC | 16:29 | |
| pabelanger | okay, I'll hold off until fungi has tested :) | 16:30 |
| pabelanger | also look like we are getting quota issues again in inap | 16:30 |
| pabelanger | likely out of sync again | 16:30 |
| AJaeger | team, https://review.openstack.org/541760 ; https://review.openstack.org/541780 and https://review.openstack.org/541781 rework the /usr/local/jenkins/slave_scripts so that we can remove it from nodes - what do you think? I can continue this... | 16:30 |
| *** slaweq has quit IRC | 16:31 | |
| mordred | AJaeger: I *love* it | 16:32 |
| mgagne | pabelanger: "No fixed IP addresses available for network" | 16:33 |
| mgagne | so it seems there are leaked neutron ports this time | 16:33 |
| AJaeger | mordred: ;) Ok, will continue it. | 16:33 |
| mgagne | because we "fixed" the nova quota issue, it gets refreshed after ~100 requests | 16:34 |
| pabelanger | mgagne: yah, just seen that now. Back from getting coffee | 16:34 |
| mgagne | pabelanger: will check on Neutron side | 16:34 |
| pabelanger | mgagne: great, thanks | 16:34 |
| *** dhajare has joined #openstack-infra | 16:38 | |
| *** salv-orlando has quit IRC | 16:38 | |
| dmsimard | sdoran: I'm surprised my IRC client even rendered that non-ascii emoji properly :p | 16:38 |
| *** VW_ has joined #openstack-infra | 16:38 | |
| *** andreas_s has quit IRC | 16:40 | |
| fungi | clarkb: what sysctl command were you suggesting to reapply network configuration? ifup as i predicted complains "ifup: interface eth0 already configured" | 16:40 |
| clarkb | fungi: systemctl restart network | 16:40 |
| mgagne | pabelanger: so around ~67 zombie ports (ports attached to deleted devices), deleting them | 16:40 |
| clarkb | which likely just runs an ifup | 16:40 |
| fungi | readding the deleted address via ip -6 addr add works fine in rax dfw, at least. confirmed that | 16:40 |
| mordred | dmsimard, sdoran mine did not :) | 16:41 |
| clarkb | kklimonda: you can query graphite for total jobs we've run (http://graphite.openstack.org) you can also query gerrit's api to count the total number of patchsets sent to neutron per day | 16:41 |
| fungi | clarkb: that seems redhattish... i don't think ubuntu has it? "Failed to restart network.service: Unit network.service not found." | 16:41 |
| *** VW has quit IRC | 16:41 | |
| *** salv-orlando has joined #openstack-infra | 16:41 | |
| clarkb | fungi: I thought it was a systemd required target | 16:41 |
| fungi | maybe it's not a "service" | 16:42 |
| clarkb | fungi: systemd defines a few networking things so that units can have a consistent set of deps | 16:42 |
| clarkb | ya it could be a meta target | 16:42 |
| *** VW has joined #openstack-infra | 16:42 | |
| fungi | systemctl restart seems to assume (and append on its own) a .service if you don't specify | 16:42 |
| *** slaweq has joined #openstack-infra | 16:42 | |
| clarkb | kklimonda: if you are wondering what the load would be for something like third party ci you may be best off asking gerrit for patchsets over some time span and then mathing that out into jobs you'd be running | 16:42 |
| pabelanger | mgagne: thanks for checking | 16:43 |
| *** dprince has joined #openstack-infra | 16:43 | |
| *** VW_ has quit IRC | 16:43 | |
| fungi | clarkb: aha, it's networkING | 16:43 |
| clarkb | ah | 16:43 |
| sdoran | Kids these days and their fancy non-ascii characters.... | 16:44 |
| fungi | clarkb: and it does appear to reapply the missing v6 address. i don't know whether it's in any way disruptive to the v4 addresses (like unapplying and reapplying or something) | 16:44 |
| mordred | dmsimard, pabelanger: could I get a quick +A on https://review.openstack.org/#/c/541759 ? | 16:44 |
| *** andreas_s has joined #openstack-infra | 16:44 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul system-required template for new project Tatu. https://review.openstack.org/511339 | 16:45 |
| fungi | sdoran: in my day, we didn't need 8 bits. that extra bit was for fancy folk | 16:45 |
| dulek | prometheanfire: Hi, so the projects you've listed on the ML need to prepare releases for Queens RC1? The deadline is tomorrow, right? | 16:45 |
| *** agopi has joined #openstack-infra | 16:45 | |
| openstackgerrit | Thomas Hisch proposed openstack-infra/git-review master: Fix git review -d M,N https://review.openstack.org/541791 | 16:45 |
| *** gyee has joined #openstack-infra | 16:45 | |
| clarkb | fungi: my understanding is it should not be disruptive but I haven't done extensive testing of that | 16:45 |
| *** salv-orlando has quit IRC | 16:46 | |
| kklimonda | @clarkb thanks, I didn't know gerrit API provides that - I'll have to take a look at the docs | 16:46 |
| fungi | clarkb: i'm willing to assume it's nondisruptive in this case and try it on ze01 | 16:46 |
| *** eumel8 has quit IRC | 16:46 | |
| prometheanfire | dulek: yes, and I believe so | 16:46 |
| clarkb | fungi: wfm | 16:46 |
| prometheanfire | I'm not the releases guy, just the reqs guy | 16:46 |
| *** salv-orlando has joined #openstack-infra | 16:47 | |
| *** slaweq has quit IRC | 16:47 | |
| *** markvoelker_ has joined #openstack-infra | 16:47 | |
| dulek | prometheanfire: Awesome, thank you, I think Kuryr will try to cut RC1 early tomorrow. :) | 16:47 |
| *** oidgar has quit IRC | 16:47 | |
| *** jpena is now known as jpena|brb | 16:48 | |
| dmsimard | mordred: mostly trusting AJaeger and you on this one | 16:49 |
| *** e0ne has quit IRC | 16:49 | |
| *** markvoelker has quit IRC | 16:49 | |
| *** andreas_s has quit IRC | 16:49 | |
| prometheanfire | dulek: k, I'd run it past smcginnis just to be sure | 16:49 |
| *** agopi has quit IRC | 16:49 | |
| *** yamamoto has joined #openstack-infra | 16:50 | |
| mordred | dmsimard: well - in this case that function can't get any *more* broken | 16:50 |
| *** hrw has quit IRC | 16:50 | |
| dmsimard | mordred: we generally need to find a strategy around all the bash that we have.. I don't have any great ideas right now and not everything is a good candidate to be ansibilized | 16:50 |
| *** katkapilatova has joined #openstack-infra | 16:50 | |
| *** katkapilatova has left #openstack-infra | 16:50 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 16:50 |
| dulek | smcginnis: As kuryr-kubernetes, do we need to do a release as well, or just cut the stable/queens branch? | 16:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Install proposal scripts only when needed https://review.openstack.org/541795 | 16:51 |
| AJaeger | mordred: one more ^ | 16:51 |
| *** hrw has joined #openstack-infra | 16:51 | |
| AJaeger | mordred: ah, need to wait for your change to merge first... | 16:52 |
| AJaeger | we touch the same file... | 16:52 |
| smcginnis | dulek: The branch needs to be cut from a tag/release, so as long as you have one that's reasonable for a stable/queens starting point, that's fine not to do a new one. | 16:52 |
| mordred | AJaeger: yah - or just rebase one on the other | 16:52 |
| AJaeger | mordred: will do | 16:52 |
| fungi | clarkb: pabelanger: `sudo systemctl restart networking` seems to have done the trick on ze01 but we probably want to keep an eye on it for a few minutes before we do the remaining four | 16:52 |
| pabelanger | ++ | 16:53 |
| smcginnis | dulek: I see 0.3.0 was just on Jan 3. If there is nothing significant since then, you can just request the branch from there. | 16:53 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Install proposal scripts only when needed https://review.openstack.org/541795 | 16:54 |
| AJaeger | mordred: done ^ | 16:54 |
| dulek | smcginnis: Awesome, I'll discuss with the team if we want to get anything else in before cutting. Thanks! | 16:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Change directories to os-service-types during update https://review.openstack.org/541759 | 16:55 |
| smcginnis | dulek: No problem. Feel free to ping us in #openstack-release if any questions come up. | 16:55 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul templates for new project tatu-dashboard. https://review.openstack.org/538307 | 16:55 |
| pabelanger | clarkb: corvus: ianw: I am also noticing the ram usage on ze02 seems to be down compared to other executors: http://grafana.openstack.org/dashboard/db/zuul-status I wonder if that is a sign the new kernel is working well | 16:56 |
| clarkb | pabelanger: possibly? I think we were more expecting that it would just stop OOMing when it had memory not necessarily that it would use less memory. But that could be a side effect too | 16:57 |
| *** felipemonteiro has joined #openstack-infra | 16:57 | |
| corvus | pabelanger: well, actually ram usage is 'up', since that's available ram. | 16:57 |
| corvus | maybe we should invert that graph :) | 16:57 |
| fungi | frickler: the apt-get install error details ended up in the job-output.json and look to me like we've got a mirror issue, maybe caught it mid-update: "The following packages have unmet dependencies: libcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.38.0-4+deb8u8) but 7.38.0-4+deb8u9 is to be installed" | 16:57 |
| pabelanger | Oh, right | 16:57 |
| *** yamamoto has quit IRC | 16:57 | |
| corvus | pabelanger, clarkb: however, ram usage being 'up' could still be good for us -- the main thing is whether the kernel is able to reclaim the memory it needs correctly. | 16:58 |
| clarkb | corvus: ya | 16:58 |
| *** slaweq has joined #openstack-infra | 16:58 | |
| pabelanger | fungi: I was thinking that could also be due to DIB not using AFS mirrors for debian. We could also make that change like we did for ubuntu | 16:58 |
| corvus | i think the experiment is: see which oom's first, ze02 or others. | 16:59 |
| pabelanger | yah, I haven't checked OOM yet on ze02 | 16:59 |
| clarkb | its like a snail race | 16:59 |
| *** dprince has quit IRC | 16:59 | |
| fungi | pabelanger: that's entirely possible, if we're trying to install libcurl4-gnutls-dev and it wants an older libcurl3-gnutls, but the error suggests we're trying to install a newer libcurl3-gnutls so i'm not convinced it's the new images with stale mirror case | 17:00 |
| *** shardy has quit IRC | 17:00 | |
| corvus | oh, interesting -- the ansible puppet wheel may not be running on some executors because it may be using ipv6 addrs. | 17:00 |
| corvus | (i just ran a manual ansible command on puppetmaster to check, and it failed to connecte to 3,6,7,8) | 17:00 |
| fungi | corvus: correct, though that will solve itself momentarily | 17:00 |
| fungi | or, rather, we will solve it shortly | 17:00 |
| fungi | ze01 would have also been in that list until a few minutes ago | 17:01 |
| fungi | just wanted to make sure it's sane before plowing forward with the remainder | 17:01 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul templates for new project python-tatuclient. https://review.openstack.org/538304 | 17:01 |
| *** HeOS has quit IRC | 17:01 | |
| *** efried has quit IRC | 17:01 | |
| *** vhosakot has quit IRC | 17:01 | |
| *** vhosakot has joined #openstack-infra | 17:02 | |
| fungi | corvus: but you are right in that those are exactly the 4 remaining executors which have no v6 address configured on them at the moment | 17:02 |
| *** dhajare has quit IRC | 17:02 | |
| *** slaweq has quit IRC | 17:03 | |
| *** pbourke has quit IRC | 17:04 | |
| *** pbourke has joined #openstack-infra | 17:05 | |
| *** dbecker has quit IRC | 17:09 | |
| fungi | pabelanger: yeah, as best as i can piece together, the current version of libcurl4-gnutls-dev in debian jessie/security is supposed to be 7.38.0-4+deb8u9 so looks like our mirrors are probably missing the updated version of libcurl4-gnutls-dev to match libcurl3-gnutls (both are supposed to be built from the same source package with versions in lockstep) | 17:10 |
| fungi | i'm hunting that down now | 17:10 |
| *** harlowja has joined #openstack-infra | 17:11 | |
| sshnaidm | pabelanger, just wanted to know - do we have docker cache proxy in infra? | 17:12 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul system-required template for new project Tatu. https://review.openstack.org/511339 | 17:12 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Complete retirement of horizon-cisco-ui https://review.openstack.org/541803 | 17:12 |
| clarkb | sshnaidm: yes | 17:12 |
| sshnaidm | clarkb, is it transparent? | 17:13 |
| fungi | pabelanger: yep, suspicions confirmed: http://mirror.dfw.rax.openstack.org/debian/pool/main/c/curl/ so this may fix itself on the next mirror pulse? i need to compare it to where we're mirroring from next | 17:13 |
| clarkb | sshnaidm: no, you have to point docker at our mirror nodes and it proxies | 17:13 |
| *** sree has joined #openstack-infra | 17:13 | |
| clarkb | sshnaidm: we don't do transparent because we can't easily lock down source IPsfor valid test nodes and don't want wide open proxies on the internet | 17:13 |
| sshnaidm | clarkb, so I need to pull like proxy.io/name/container instead of docker.io/name/container? | 17:13 |
| *** markvoelker_ has quit IRC | 17:14 | |
| *** slaweq has joined #openstack-infra | 17:14 | |
| clarkb | sshnaidm: http://$NODEPOOL_MIRROR_HOST:8081/registry-1.docker/ | 17:14 |
| *** markvoelker has joined #openstack-infra | 17:14 | |
| fungi | pabelanger: no, wait, i'm misreading there... so it looks like maybe we're missing the security suite in the pol | 17:14 |
| fungi | pool | 17:14 |
| *** mriedem has quit IRC | 17:14 | |
| sshnaidm | clarkb, and does it work only from CI? | 17:15 |
| smcginnis | If someone could take a look at https://review.openstack.org/541803 | 17:15 |
| clarkb | sshnaidm: you should only use it from our test nodes, please don't use it from outside our zuul jobs | 17:15 |
| smcginnis | That's blocking a requirements patch since validation fails due to not having a release job specified. | 17:15 |
| *** mriedem has joined #openstack-infra | 17:15 | |
| sshnaidm | clarkb, wanted just to test if it works.. | 17:15 |
| fungi | pabelanger: this is starting to look more like we build our images including packages from security.debian.org but then don't have them available in our mirror | 17:15 |
| clarkb | sshnaidm: for testing thats fine. I just want to be clear on that beause the rdo tests ddos'd one of our mirrors by using it instead of the lcoal resources in the rdo cloud | 17:16 |
| sshnaidm | clarkb, yeah | 17:16 |
| *** jpena|brb is now known as jpena | 17:17 | |
| corvus | i'm resuming the zuul-projects change uploads | 17:17 |
| *** harlowja has quit IRC | 17:17 | |
| pabelanger | fungi: odd | 17:19 |
| *** slaweq has quit IRC | 17:19 | |
| fungi | corvus: sounds good | 17:19 |
| *** dizquierdo has joined #openstack-infra | 17:19 | |
| pabelanger | sshnaidm: yes, the URL clarkb linked its the proxy | 17:19 |
| pabelanger | clarkb: Yah, RDO should have the same setup, but I haven't confirmed | 17:20 |
| pabelanger | fungi: I can help in a moment, just booking some travel first | 17:20 |
| *** sree has quit IRC | 17:20 | |
| clarkb | fungi: guessing debian doesn't just use that as a quicker way to get security patches otu and is a distinct repo? | 17:22 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Complete retirement of horizon-cisco-ui https://review.openstack.org/541803 | 17:22 |
| fungi | clarkb: yeah, security is a separate suite. packages from security eventually end up in main (along with stable proposed updates of a non-security kind) at stable point release time | 17:24 |
| fungi | though the pools are supposed to be merged these days so i need to dig deeper into this before proposing a solution | 17:24 |
| fungi | also need to go run a quick errand, so will pick it back up shortly | 17:24 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul web: add admin endpoint, enqueue commands https://review.openstack.org/539004 | 17:25 |
| pabelanger | fungi: Oh, right. We might not mirror security and expect to just get that directly from upstream | 17:26 |
| pabelanger | I seem to remember something like that | 17:27 |
| *** tesseract has quit IRC | 17:27 | |
| *** slaweq has joined #openstack-infra | 17:30 | |
| *** dtantsur is now known as dtantsur|afk | 17:33 | |
| *** rfolco|ruck is now known as rfolco|ruck|brb | 17:33 | |
| *** slaweq has quit IRC | 17:34 | |
| *** liusheng has quit IRC | 17:37 | |
| *** liusheng has joined #openstack-infra | 17:37 | |
| *** jpich has quit IRC | 17:37 | |
| sshnaidm | pabelanger, seems like docker proxy doesn't work for rdocloud | 17:41 |
| pabelanger | sshnaidm: you'll need to have jpena or dmsimard confirm. There is an apache rule we have setup in openstack-infra/system-config to handle the reverse proxy cache. | 17:43 |
| *** olaph1 is now known as olaph | 17:45 | |
| *** slaweq has joined #openstack-infra | 17:46 | |
| jpena | pabelanger: can you point me to the rule so I can check it? Maybe I missed some configuration while setting it up | 17:46 |
| *** yamahata has quit IRC | 17:47 | |
| *** iyamahat has quit IRC | 17:47 | |
| pabelanger | jpena: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n204 | 17:49 |
| *** rfolco|ruck|brb is now known as rfolco|ruck | 17:50 | |
| *** slaweq has quit IRC | 17:50 | |
| *** sshnaidm_ has joined #openstack-infra | 17:54 | |
| Roamer` | OK, so the StorPool third-party CI address appears in https://review.openstack.org/#/admin/groups/270,members so according to https://docs.openstack.org/infra/system-config/third_party.html this ought to mean that Gerrit will not send us e-mail on changes that our CI has commented on. However, we're still receiving those e-mails, e.g. Message-Id: <E1ejThx-000HmL-S5@review.openstack.org> from 15 m | 17:55 |
| Roamer` | inutes ago | 17:56 |
| jpena | pabelanger, sshnaidm: ok, that is set up | 17:56 |
| jpena | sshnaidm: have you been trying by hand, or using a job? | 17:56 |
| *** sshnaidm has quit IRC | 17:57 | |
| pabelanger | jpena: okay, might need to enable debug in apache to see what is happening. It is also possible sshnaidm doesn't have it configured properly, IIRC something about trailing slash, either docker needs it or doesn't | 17:58 |
| *** salv-orlando has quit IRC | 17:58 | |
| *** derekh has quit IRC | 17:59 | |
| jpena | sshnaidm_: I see some messages in the error log, it looks like one of the tests tried to access xxx:8081/v1 when it should be xxx:8081/registry-1.docker//v1 | 17:59 |
| *** dhajare has joined #openstack-infra | 17:59 | |
| jpena | not sure if that is your case | 17:59 |
| pabelanger | yah, that sill be malformed URL in docker client | 18:00 |
| *** jamesmcarthur has quit IRC | 18:00 | |
| pabelanger | trailing slash | 18:00 |
| sshnaidm_ | jpena, yeah, seems like.. | 18:01 |
| *** iyamahat has joined #openstack-infra | 18:01 | |
| sshnaidm_ | jpena, also needed to add it to insecure registries, because it's http | 18:01 |
| *** slaweq has joined #openstack-infra | 18:01 | |
| *** gfidente is now known as gfidente|afk | 18:02 | |
| *** salv-orlando has joined #openstack-infra | 18:02 | |
| *** jamesmcarthur has joined #openstack-infra | 18:03 | |
| *** e0ne has joined #openstack-infra | 18:04 | |
| *** slaweq has quit IRC | 18:06 | |
| *** dizquierdo has quit IRC | 18:06 | |
| sshnaidm_ | jpena, I have 403 error | 18:10 |
| sshnaidm_ | jpena, http://paste.openstack.org/show/664972/ | 18:11 |
| *** harlowja has joined #openstack-infra | 18:11 | |
| tobiash | sambetts: I've responded on https://review.openstack.org/#/c/541669 | 18:11 |
| sshnaidm_ | pabelanger, clarkb btw, why is docke rproxy http://$NODEPOOL_MIRROR_HOST:8081/registry-1.docker/ ? it should be without "http://" so it can be used with docker: docker pull $NODEPOOL_DOCKER_REGISTRY_PROXY | 18:13 |
| sshnaidm_ | or I miss something? | 18:14 |
| *** mriedem has quit IRC | 18:14 | |
| pabelanger | sshnaidm_: IIRC, docker.json file needs URL (with http) | 18:14 |
| pabelanger | which is how jobs have been setup to use it | 18:15 |
| sshnaidm_ | pabelanger, do you mean daemon.json? it does not require http iirc | 18:15 |
| sambetts | tobiash: it seem this is the "fix" in upstream ansible https://github.com/ansible/ansible/commit/85f727ad4bfbd21091beeeb69e3eb89bb47b6b49#diff-90085fdcec6ed8b273ba885eaee60328 | 18:16 |
| sambetts | tobiash: however it doesn't really fix it, it just hides the problem behind an if _debug statement | 18:16 |
| pabelanger | sshnaidm_: right. I'd have to see what the syntax is these days. | 18:16 |
| *** slaweq has joined #openstack-infra | 18:17 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/540647/ and https://review.openstack.org/541760 , please? | 18:17 |
| *** harlowja has quit IRC | 18:18 | |
| tobiash | sambetts: so this should be 'fixed' by the ansible upgrade? | 18:19 |
| sambetts | tobiash: if we pull in that change as part of it | 18:19 |
| sambetts | tobiash: the ansible basics module has changed alot since it was forked by zuul | 18:19 |
| *** panda is now known as panda|off | 18:20 | |
| *** sshnaidm_ is now known as sshnaidm|afk | 18:21 | |
| *** slaweq has quit IRC | 18:22 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:22 | |
| *** sree has joined #openstack-infra | 18:23 | |
| tobiash | sambetts: looks like the ansible basics module is not forked by zuul so this would be used directly from upstream | 18:23 |
| *** caphrim007_ has joined #openstack-infra | 18:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Complete retirement of horizon-cisco-ui https://review.openstack.org/541803 | 18:24 |
| sambetts | tobiash: all the code in the zuul_run_command function in command.py module, comes from ansible's module_lib/basics.py run_command function | 18:24 |
| sambetts | module_utils rather | 18:25 |
| sambetts | https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/basic.py#L2632 | 18:25 |
| sambetts | the upstream command module calls module.run_command which calls ^ | 18:26 |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: WIP - Fix yaxes format for newer versions of Grafana https://review.openstack.org/541825 | 18:26 |
| mordred | fungi: ifyou have a sec, https://review.openstack.org/#/c/517105 is needed for loci to cut a 1.0 - it looks good to me | 18:26 |
| sambetts | tobiash: the zuul command module, replaces the basic run_command, with zuul_run_command | 18:26 |
| *** felipemonteiro has quit IRC | 18:26 | |
| *** caphrim007 has quit IRC | 18:27 | |
| *** eumel8 has joined #openstack-infra | 18:27 | |
| *** jamesmcarthur has quit IRC | 18:27 | |
| jpena | sshnaidm|afk: I've been doing some tests locally, it looks like at one point the docker client tries to use /v1 or /v2 paths directly, instead of /registry-1.docker/v1 | 18:27 |
| *** sree has quit IRC | 18:27 | |
| jpena | sshnaidm|afk: gotta go now, i'll keep on checking tomorrow | 18:27 |
| *** r-daneel_ has joined #openstack-infra | 18:28 | |
| *** jpena is now known as jpena|off | 18:28 | |
| *** VW has quit IRC | 18:28 | |
| *** VW has joined #openstack-infra | 18:29 | |
| *** r-daneel has quit IRC | 18:29 | |
| *** r-daneel_ is now known as r-daneel | 18:29 | |
| fungi | mordred: looking but will take a bit... i completely missed the "groups" feature add so am quickly familiarizing myself with it | 18:30 |
| mordred | fungi: awesome | 18:30 |
| tobiash | sambetts: ah, right | 18:31 |
| fungi | looks like it got a mention in the readme and we added reno-style release notes to bindep at the same time, but nobody pushed for actual documentation? | 18:31 |
| * tobiash is not yet that familiar with that ansible code | 18:31 | |
| clarkb | fungi: https://review.openstack.org/#/c/532359/ is a git review chagne that is also worth looking over to fix a bug | 18:32 |
| fungi | mordred: ahh, i guess the majority of the documentation is in the readme anyway so close enough | 18:32 |
| *** slaweq has joined #openstack-infra | 18:33 | |
| *** VW has quit IRC | 18:33 | |
| mordred | fungi: yah | 18:33 |
| *** pcichy has joined #openstack-infra | 18:34 | |
| dmsimard | Looks like the newest version of grafana will support configuring data sources and dashboards from a configuration file http://docs.grafana.org/guides/whats-new-in-v5/ | 18:36 |
| jlvillal | clarkb, FYI: https://review.openstack.org/#/c/541125/ This is a follow-up to the unit test patch to gerritbot that added the .zuul.yaml file. The patch removes the pep8 job from project-config. | 18:36 |
| *** slaweq has quit IRC | 18:37 | |
| *** VW has joined #openstack-infra | 18:40 | |
| prometheanfire | AJaeger: mnaser: https://review.openstack.org/#/c/539927/ broke requirements | 18:40 |
| prometheanfire | AJaeger: we need to remove stuff from projects.txt BEFORE removing the project config stuff | 18:41 |
| prometheanfire | AJaeger: mnaser see http://logs.openstack.org/52/541752/1/check/requirements-tox-validate-projects/8200c0c/job-output.txt.gz#_2018-02-07_15_57_06_828118 | 18:41 |
| *** sambetts is now known as sambetts|afk | 18:41 | |
| *** tosky has quit IRC | 18:42 | |
| fungi | or else the projects.txt removal needs to be the very next change to go into requirements | 18:42 |
| fungi | (but yes, before would be preferable) | 18:43 |
| *** jamesmcarthur has joined #openstack-infra | 18:43 | |
| mordred | mnaser: thank you for running many openstack services in vexxhost - it just made tracking down an api discovery issue very easy :) | 18:45 |
| AJaeger | prometheanfire: could you update the retirement instructions, please? | 18:45 |
| AJaeger | prometheanfire: https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project - and sorry about that. | 18:45 |
| mnaser | mordred: cool! Thank you! :) | 18:45 |
| prometheanfire | AJaeger: k :D | 18:46 |
| *** jamesmca_ has joined #openstack-infra | 18:46 | |
| prometheanfire | AJaeger: what repo is that? | 18:47 |
| AJaeger | prometheanfire: openstack-infra/infra-manual | 18:47 |
| prometheanfire | k | 18:48 |
| *** slaweq has joined #openstack-infra | 18:49 | |
| *** zoli is now known as zoli|gone | 18:49 | |
| *** zoli|gone is now known as zoli | 18:49 | |
| *** jamesmcarthur has quit IRC | 18:51 | |
| corvus | clarkb, fungi, mordred: do you remember what pip was trying to do with the origin remote which caused those warnings when running under zuul? | 18:53 |
| *** slaweq has quit IRC | 18:53 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 18:54 |
| clarkb | corvus: yes it is trying to make the freeze output be an http link to the egg at the upstream location | 18:55 |
| *** jamesmca_ has quit IRC | 18:55 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/infra-manual master: Update retirement to remove from projects.txt https://review.openstack.org/541838 | 18:55 |
| corvus | clarkb: thanks | 18:55 |
| prometheanfire | AJaeger: there you go ^ | 18:56 |
| clarkb | corvus: https://github.com/pypa/pip/pull/4760 is my proposed fix | 18:56 |
| prometheanfire | AJaeger: let me know if that's clear enough, it should be removed from projects.txt for all branches it's in | 18:56 |
| *** e0ne has quit IRC | 18:59 | |
| AJaeger | prometheanfire: thanks, I commented | 18:59 |
| AJaeger | prometheanfire: just nits ;) | 18:59 |
| openstackgerrit | Matthew Thode proposed openstack-infra/infra-manual master: Update retirement to remove from projects.txt https://review.openstack.org/541838 | 19:02 |
| prometheanfire | AJaeger: updated | 19:02 |
| AJaeger | prometheanfire: thanks, LGTM. fungi, what do you think? ^ | 19:02 |
| *** amoralej is now known as amoralej|off | 19:03 | |
| *** slaweq has joined #openstack-infra | 19:04 | |
| *** iyamahat_ has joined #openstack-infra | 19:05 | |
| *** iyamahat has quit IRC | 19:05 | |
| *** salv-orlando has quit IRC | 19:06 | |
| *** xarses_ has quit IRC | 19:07 | |
| *** xarses has joined #openstack-infra | 19:07 | |
| *** sree has joined #openstack-infra | 19:08 | |
| *** tosky has joined #openstack-infra | 19:09 | |
| *** Goneri has quit IRC | 19:09 | |
| *** slaweq has quit IRC | 19:09 | |
| *** e0ne has joined #openstack-infra | 19:10 | |
| prometheanfire | AJaeger: https://review.openstack.org/541841 https://review.openstack.org/541752 thanks | 19:10 |
| *** Goneri has joined #openstack-infra | 19:10 | |
| hjensas | Changes in the gate look like they run to completion, and then restart. 8-9 hours now, 3-4 times I have seen the jobs go from all success -> bland -> queue. There is a depends on in another repo gating at the same time, same thing happens there. Do we have to land the Depends-On first, then the other ones? | 19:11 |
| *** sree has quit IRC | 19:12 | |
| *** jamesmcarthur has joined #openstack-infra | 19:12 | |
| pabelanger | hjensas: which patch are you looking at? | 19:13 |
| *** jamesmcarthur has quit IRC | 19:13 | |
| *** r-daneel_ has joined #openstack-infra | 19:13 | |
| hjensas | pabelanger: openstack/instack-undercloud 437544 and openstack/tripleo-heat-templates 473817 | 19:14 |
| *** r-daneel has quit IRC | 19:14 | |
| *** r-daneel_ is now known as r-daneel | 19:14 | |
| pabelanger | hjensas: right, all the previous patches before those will need to first pass and merge. What you are likely seeing, is when on job above those fails it will remove that patchset from the queue and restart testing | 19:16 |
| pabelanger | hjensas: https://docs.openstack.org/infra/zuul/user/gating.html#testing-in-parallel explains more about the process | 19:17 |
| fungi | those diagrams never get old | 19:18 |
| *** slaweq has joined #openstack-infra | 19:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: zuul.d: gerritbot: Remove check and gate jobs as now in repo https://review.openstack.org/541125 | 19:21 |
| *** jamesmcarthur has joined #openstack-infra | 19:22 | |
| hjensas | pabelanger: 533369 and 533370 you mean? I've seen all three with all jobs 'success', and then restart. I'll leave it for one more iteration and see how goes. | 19:22 |
| clarkb | hjensas: the gate is a queue, first in first out. We test in parallel to speed things up with the hope that things pass and we can merge things in order all at once. If however something at the front of the queue fails we remove it from the queue and reenqueue everyone else without the failing change included. This requires that the tests be restarted as the previous results may no longer be valid | 19:24 |
| *** slaweq has quit IRC | 19:25 | |
| pabelanger | hjensas: you might want to follow up with #tripleo to see if there is any know issues of their queue. It is possible there is ongoing issues which is causing the queue to reset | 19:27 |
| *** vhosakot has quit IRC | 19:28 | |
| *** vhosakot has joined #openstack-infra | 19:28 | |
| hjensas | clarkb: pabelanger: I see, I will monitor the triple heat templates queue then, not just my change. Looks like there are some other changes infront that had issues. Thanks for the explanations. | 19:28 |
| *** iyamahat_ has quit IRC | 19:29 | |
| *** iyamahat has joined #openstack-infra | 19:30 | |
| *** yamahata has joined #openstack-infra | 19:32 | |
| *** markvoelker has quit IRC | 19:33 | |
| *** lucasagomes is now known as lucas-afk | 19:33 | |
| *** markvoelker_ has joined #openstack-infra | 19:33 | |
| *** jamesmcarthur has quit IRC | 19:34 | |
| *** bobh has joined #openstack-infra | 19:35 | |
| *** mriedem has joined #openstack-infra | 19:35 | |
| *** slaweq has joined #openstack-infra | 19:36 | |
| *** ralonsoh has quit IRC | 19:37 | |
| *** anticw has quit IRC | 19:38 | |
| *** sree has joined #openstack-infra | 19:39 | |
| *** anticw has joined #openstack-infra | 19:39 | |
| *** slaweq has quit IRC | 19:41 | |
| fungi | okay, so back to digging into the debian-security mirroring, it _is_ included in our reprepro config as a suite within the mirror we're creating, so i don't _think_ it's a mirror configuration issue. that leaves us with either stale mirroring or a failure to configure our images to use the security suite in their sources.list | 19:41 |
| *** coolsvap has quit IRC | 19:41 | |
| fungi | checking mirror logs next | 19:41 |
| clarkb | fungi: mordred corvus and other infra-roots interested in the project renaming. smcginnis indicated that march 16 would work as long as they finish up the trailing cycle releases the day before. Otherwise we can look at the week after that | 19:41 |
| fungi | clarkb: sounds great | 19:42 |
| openstackgerrit | Merged openstack-infra/subunit2sql master: remove oslosphinx from test-requirements https://review.openstack.org/477161 | 19:42 |
| corvus | clarkb: i expect to be available around that time. | 19:43 |
| mordred | clarkb: WFM | 19:43 |
| ianw | wfm modulo timezones | 19:44 |
| ianw | fungi: did you see my update to that ticket re ze02 ipv6 connections to cacti? were you seeing 100% packet loss? | 19:44 |
| *** sree has quit IRC | 19:44 | |
| fungi | ianw: i saw your update on the ticket, i have not done any testing with regard to ze02 packet loss though | 19:45 |
| ianw | infra-root: could i request a couple of reviews of https://review.openstack.org/#/c/541488/ which fixes up logrotation on graphite. i think i'll need to manually clear out the old logrotate | 19:45 |
| fungi | last update to /var/log/reprepro/debian-jessie-security.log is from 2018-01-23 but no erroring, so next likely possibility is that we're building images with security.debian.org (which gets updated quickly) in our sources.list but then mirroring from a site which lags in security updates. if _that's_ the case, then yes we can likely apply the same fix we did for ubuntu to use our mirrors when | 19:46 |
| fungi | building images | 19:46 |
| ianw | fungi: but in your testing, it was 100% loss right? in ze02's case, it's intermittent | 19:46 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update retirement to remove from projects.txt https://review.openstack.org/541838 | 19:46 |
| fungi | ianw: correct | 19:46 |
| *** gfidente|afk has quit IRC | 19:46 | |
| prometheanfire | woot | 19:46 |
| corvus | fungi: i think it's okay to re-enable ipv6 on the other hosts -- ze01's squiggly lines look like the others | 19:48 |
| corvus | er, i mean "metric regression analysis reports negative impact" | 19:49 |
| fungi | hah, excellent | 19:49 |
| fungi | doing now | 19:49 |
| clarkb | ianw: logrotate change is approved | 19:50 |
| fungi | #status log ipv6 addresses have been readded to all zuul executors | 19:51 |
| openstackstatus | fungi: finished logging | 19:51 |
| *** pcaruana has joined #openstack-infra | 19:52 | |
| *** slaweq has joined #openstack-infra | 19:52 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add /node-list to the webapp https://review.openstack.org/535562 | 19:53 |
| clarkb | mordred: AJaeger is https://review.openstack.org/#/c/541760/2 correct in putting the scripts in the tasks/ dir of the role? | 19:56 |
| *** slaweq has quit IRC | 19:56 | |
| AJaeger | clarkb: thanks for double checking . Let me investigate some more... | 19:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix openstack-tox-py35-with-ovsdbapp-master for networking-ovn https://review.openstack.org/540647 | 20:00 |
| mnaser | prometheanfire: sorry for that breakage | 20:00 |
| mnaser | TIL | 20:00 |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Fix yaxes format for newer versions of Grafana https://review.openstack.org/541825 | 20:00 |
| prometheanfire | mnaser: wasn't doc'd now it is | 20:01 |
| VW | I should know this, but can you guys point me to where you define the number of OpenStack Jenkins boxes per Rackspace region? | 20:03 |
| prometheanfire | fungi: let me know if you want to figure out the tcp leaking thing | 20:03 |
| prometheanfire | I would like to finish cloning | 20:03 |
| mordred | AJaeger, clarkb: oh- actually, good call - those should go into a directory called "files" since they're in a role | 20:03 |
| VW | just refreshing my memory of how many in each location | 20:03 |
| AJaeger | mordred, clarkb will update - thanks | 20:03 |
| mordred | AJaeger, clarkb: the other patch was putting them in the playbooks dir in ozj which *is* the right place | 20:03 |
| *** dhajare has quit IRC | 20:04 | |
| AJaeger | mordred: ah, so for playbooks in the directory, for roles in files directory - will do | 20:04 |
| *** dhill_ has quit IRC | 20:04 | |
| *** jpich has joined #openstack-infra | 20:04 | |
| clarkb | VW: https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nl01.openstack.org.yaml max-servers is the key to look for for that number | 20:05 |
| VW | gracias clarkb | 20:05 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move translation scripts out of jenkins/scripts https://review.openstack.org/541760 | 20:05 |
| AJaeger | mordred: do I need to give any special location? please double check ^ | 20:05 |
| AJaeger | mordred: I mean in referencing the files | 20:05 |
| mordred | AJaeger: nope - in a role references to copying files should find them in the files dir just fine | 20:06 |
| fungi | prometheanfire: yeah, are you able to controllably leak just one? we've seen it for example with newer paramiko when programs forget to explicitly close a socket | 20:06 |
| fungi | prometheanfire: but your shell example looked interesting | 20:06 |
| *** pcaruana has quit IRC | 20:07 | |
| *** slaweq has joined #openstack-infra | 20:08 | |
| fungi | and yes gerrit as of at least 2.13 does by default impose a maximum of 64 concurrent connections to the ssh api per account | 20:08 |
| fungi | so it's possible you were leaking when running against earlier gerrit versions and just didn't notice | 20:08 |
| AJaeger | mordred: thanks, then it should be fine - see https://review.openstack.org/541760 and also https://review.openstack.org/541795 | 20:09 |
| *** kjackal has quit IRC | 20:10 | |
| *** slaweq has quit IRC | 20:10 | |
| *** slaweq has joined #openstack-infra | 20:10 | |
| *** dhill_ has joined #openstack-infra | 20:11 | |
| openstackgerrit | Merged openstack-infra/puppet-graphite master: Fix up log rotation https://review.openstack.org/541488 | 20:12 |
| fungi | okay, found our debian mirroring issue... "The lock file '/afs/.openstack.org/mirror/debian/db/lockfile' already exists." | 20:12 |
| fungi | i'll work on clearing that now | 20:12 |
| *** rfolco|ruck is now known as rfolco|off | 20:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 20:14 |
| *** dhill_ has quit IRC | 20:14 | |
| *** ldnunes has quit IRC | 20:15 | |
| *** kjackal has joined #openstack-infra | 20:15 | |
| *** hasharAway has quit IRC | 20:15 | |
| *** dhill_ has joined #openstack-infra | 20:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: rm unused jenkins/scripts/* https://review.openstack.org/541861 | 20:18 |
| AJaeger | mordred, clarkb, found some completely unused scripts ^ | 20:18 |
| * AJaeger puts jenkins/scripts on a diet | 20:18 | |
| mordred | AJaeger: I like your diet plans | 20:19 |
| prometheanfire | fungi: I think I can | 20:19 |
| prometheanfire | fungi: if you have some time we can test | 20:19 |
| mordred | corvus: can we trigger post jobs that had a dependency? we can't do that right? | 20:20 |
| mordred | corvus: cause they would need to re-run the first job,yeah? | 20:20 |
| *** dhill_ has quit IRC | 20:21 | |
| *** dhill_ has joined #openstack-infra | 20:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Install proposal scripts only when needed https://review.openstack.org/541795 | 20:24 |
| *** eharney has quit IRC | 20:24 | |
| fungi | prometheanfire: i'm caught up as of just now, waiting a few minutes for a manual update of our debian mirrors to complete | 20:25 |
| fungi | were you able to distill the poc down at all? | 20:25 |
| prometheanfire | fungi: no, ian had to leave/sleep | 20:26 |
| prometheanfire | fungi: I think that it is the initial clone though, not the gerrit setup | 20:26 |
| fungi | yeah, i have your original. digging into it now | 20:26 |
| prometheanfire | or maybe it's the gerrit setup, but we can test that | 20:26 |
| AJaeger | config-core, tatu is a new project that got imported earlier today - care to review the system-required addition for it? https://review.openstack.org/511339 | 20:26 |
| prometheanfire | just remove the remote and re-run git review -s | 20:27 |
| fungi | prometheanfire: yeah, i think it's the git review -s | 20:32 |
| *** gridinv has quit IRC | 20:32 | |
| fungi | just looking at the command, that's the only thing inside the loop which will hit the gerrit ssh api | 20:33 |
| prometheanfire | yep | 20:33 |
| prometheanfire | otherwise we are hitting git.o.o | 20:33 |
| fungi | because it wants to retrieve the post-commit hook | 20:33 |
| fungi | happen to have your numeric gerrit account id handy? | 20:33 |
| prometheanfire | sec | 20:34 |
| prometheanfire | 14288 | 20:34 |
| fungi | thanks, checking current connection count | 20:35 |
| *** pcichy has quit IRC | 20:36 | |
| fungi | i have 6 established connections from you right now to the ssh api | 20:36 |
| prometheanfire | ya, I'm doing nothing atm | 20:36 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move wheel-build scripts out of jenkins/scripts https://review.openstack.org/541868 | 20:36 |
| AJaeger | and next diet step ^ | 20:36 |
| prometheanfire | let me try re-running -s | 20:37 |
| fungi | prometheanfire: okay, i'll kill those before we do anything | 20:37 |
| fungi | give it a sec | 20:37 |
| prometheanfire | ok | 20:37 |
| *** hemna_ has quit IRC | 20:37 | |
| corvus | mordred: yeah, we can enqueue items, which will run the entire job graph, but not individual jobs | 20:37 |
| *** liusheng has quit IRC | 20:38 | |
| *** liusheng has joined #openstack-infra | 20:38 | |
| fungi | prometheanfire: okay, i've closed those and primed my command history with a loop i can use to clean up hung connections by account id | 20:38 |
| fungi | currently no established connections for your account | 20:39 |
| *** salv-orlando has joined #openstack-infra | 20:39 | |
| prometheanfire | k | 20:39 |
| prometheanfire | ok, just ran git review -s | 20:39 |
| fungi | and it returned control to you? | 20:39 |
| prometheanfire | connection should be closed | 20:39 |
| fungi | confirmed, no established connection | 20:39 |
| fungi | i wonder if the backgrounding is disassociating in such a way that the connection doesn't close | 20:40 |
| fungi | i've had that with ssh in other situations | 20:40 |
| prometheanfire | let me try something more manual | 20:41 |
| prometheanfire | check now? | 20:41 |
| fungi | nope, still no connections from you | 20:41 |
| fungi | my gut says that calling git-review in the background is interfering with the way it's spawning the ssh client subprocess | 20:42 |
| prometheanfire | ok, going to run the following one line at a time asking for counts between each line | 20:42 |
| prometheanfire | mkdir -p openstack-infra | 20:42 |
| prometheanfire | (cd openstack-infra; git clone git://git.openstack.org/openstack-infra/meetbot) | 20:42 |
| prometheanfire | (cd openstack-infra/meetbot; git review -s </dev/null || true) | 20:42 |
| prometheanfire | now? (clone) | 20:43 |
| fungi | nada | 20:43 |
| fungi | betting git-review with no stdin for openssh to inherit is the issue though | 20:43 |
| prometheanfire | now? | 20:43 |
| fungi | yep, now there's one | 20:43 |
| prometheanfire | (cd openstack-infra/meetbot; git review -s </dev/null || true) | 20:43 |
| prometheanfire | ok, let's try the next one without the stdin | 20:44 |
| prometheanfire | now (before) | 20:44 |
| fungi | right now i only have the one hung connection from a minute ago (i haven't closed that) | 20:44 |
| prometheanfire | how about now? just ran (cd openstack-infra/mqtt_statsd; git review -s || true) | 20:45 |
| fungi | there are three new connections for you now | 20:45 |
| prometheanfire | lol | 20:45 |
| fungi | _strange_ | 20:45 |
| openstackgerrit | Ian Wienand proposed openstack-infra/git-review master: Testing getting specific patchset https://review.openstack.org/541892 | 20:45 |
| fungi | why would that be THREE? | 20:45 |
| prometheanfire | I did do a clone as well (cd openstack-infra; git clone git://git.openstack.org/openstack-infra/mqtt_statsd) | 20:45 |
| prometheanfire | but that's not to gerrit | 20:45 |
| fungi | right | 20:46 |
| prometheanfire | ok, let's try removing one thing at a time | 20:46 |
| prometheanfire | removing || true next | 20:46 |
| fungi | how about i close these first | 20:46 |
| fungi | clean slate before the next test | 20:46 |
| *** gridinv has joined #openstack-infra | 20:46 | |
| prometheanfire | ok, go ahead | 20:46 |
| fungi | okay, squeaky-clean, have at it | 20:46 |
| prometheanfire | just ran (cd openstack-infra/nodepool; git review -s) | 20:47 |
| prometheanfire | and have my console back | 20:47 |
| fungi | there was just one new connection, then i saw a second one appear | 20:47 |
| prometheanfire | ok, let me run with --verbose on the git review | 20:48 |
| fungi | keep in mind, `git review -s` may do several things over the ssh api so it may depend on the state of the repo too | 20:48 |
| prometheanfire | want to clean it up first? | 20:48 |
| fungi | like, i think it also does a test push which it aborts once it confirms whether or not there will be an error | 20:48 |
| fungi | so that may explain the multiple connections if they all hang | 20:49 |
| fungi | okay, removed those two connections | 20:49 |
| *** dave-mcc_ has quit IRC | 20:49 | |
| prometheanfire | just ran (cd openstack-infra/nose-html-output; git review -s --verbose) | 20:49 |
| prometheanfire | next one I'll run the commands listed via debug manually one at a time so we can narrow it down | 20:50 |
| fungi | one connection | 20:50 |
| prometheanfire | close it :D | 20:50 |
| fungi | wiped | 20:50 |
| *** dave-mccowan has joined #openstack-infra | 20:50 | |
| AJaeger | mordred: could i trouble you to review https://review.openstack.org/#/c/541760/ again today, please? | 20:51 |
| fungi | curious to see whether it's openssh in a subshell or openssh via python popen in a subshell at fault | 20:51 |
| * AJaeger waves good bye | 20:51 | |
| fungi | have a good evening, AJaeger1 | 20:51 |
| fungi | er, AJaeger! | 20:51 |
| prometheanfire | fungi: now? (cd openstack-infra/odsreg; git push --dry-run ssh://prometheanfire@review.openstack.org:29418/openstack-infra/nose-html-output.git --all) | 20:51 |
| fungi | prometheanfire: nope, no new connections hanging | 20:52 |
| prometheanfire | just ran (cd openstack-infra/odsreg; git remote add -f gerrit ssh://prometheanfire@review.openstack.org:29418/openstack-infra/nose-html-output.git) | 20:52 |
| fungi | prometheanfire: bingo, one connection open | 20:53 |
| prometheanfire | ok, go ahead and close it, I'll remove the remote and do it without the subshell | 20:54 |
| fungi | closed | 20:54 |
| prometheanfire | just ran (cd openstack-infra/odsreg; git push --dry-run ssh://prometheanfire@review.openstack.org:29418/openstack-infra/odsreg.git --all) | 20:55 |
| prometheanfire | should show 0 | 20:55 |
| kota_ | hello | 20:56 |
| fungi | prometheanfire: yep, no connections | 20:56 |
| prometheanfire | fungi: now? | 20:56 |
| prometheanfire | ran git remote add -f gerrit ssh://prometheanfire@review.openstack.org:29418/openstack-infra/odsreg.git without subshell | 20:56 |
| kota_ | does anyone know how to rewrite tag message after pushing the tag to gerrit? | 20:56 |
| fungi | prometheanfire: one connection open | 20:56 |
| prometheanfire | so even without the subshell it does it | 20:56 |
| kota_ | I'd like to reword it but it seems immutable. | 20:57 |
| prometheanfire | can you confirm? just in case | 20:57 |
| prometheanfire | kota_: join the -release channel and ask there :D | 20:57 |
| fungi | kota_: the tag message is part of the data signed in the tag. once it's pushed you're stuck with it. you'd need to delete and recreate and push a new tag, but tag deletion/replacement doesn't propagate via pulls, fetches or remote updates | 20:57 |
| prometheanfire | ah, didn't know it was part of the tag | 20:57 |
| openstackgerrit | Merged openstack-infra/project-config master: Zuul system-required template for new project Tatu. https://review.openstack.org/511339 | 20:58 |
| fungi | prometheanfire: yep, much in the same way that a commit message is covered by the commit object hash | 20:58 |
| prometheanfire | With -f option, git fetch <name> is run immediately after the remote information is set up. | 20:58 |
| fungi | prometheanfire: yeah, so makes sense that would cause it to connect | 20:59 |
| prometheanfire | fungi: I'll try running that manually next | 20:59 |
| kota_ | fungi: can i delete the tag if i asuume it's ok that it doesn't propagate? | 20:59 |
| fungi | prometheanfire: want me to delete the previous connection? | 20:59 |
| prometheanfire | ya | 20:59 |
| kota_ | on the gerrit, we use in the openstack. | 20:59 |
| prometheanfire | tell me when to run git fetch gerrit | 20:59 |
| fungi | kota_: not really effectively. we have lots of automation which will fail to notice a new tag if you attempt to push a replacement. but also you'd need a special acl permission to remotely delete a tag from gerrit | 21:00 |
| prometheanfire | kota_: if it's published it's in git.openstack.org and mirrored to github iirc | 21:00 |
| kota_ | at least, `git push -f` didn't work to me tho. | 21:01 |
| fungi | from my perspective, the bigger concern is not git.o.o and gh mirrors, but for example the clones maintained by zuul executors which will continue to have and use teh old tag, or people who have cloned the repo already | 21:02 |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Allow dashboards to be tagged https://review.openstack.org/541898 | 21:02 |
| prometheanfire | fungi: say when | 21:02 |
| fungi | prometheanfire: cleaned up, try again now | 21:03 |
| prometheanfire | fungi: git fetch gerrit | 21:03 |
| fungi | no new open connections from that | 21:03 |
| prometheanfire | so, there's a problem, not sure on which end, on how it handles git remote add -f | 21:04 |
| prometheanfire | if run as a separate connection it's fine | 21:04 |
| *** jpich has quit IRC | 21:04 | |
| fungi | prometheanfire: bizarre | 21:04 |
| mordred | prometheanfire, fungi: this is with git review? | 21:04 |
| fungi | mordred: yep | 21:05 |
| * mordred suggests everyone adopts 'git config --global gitreview.usepushurl 1' | 21:05 | |
| prometheanfire | mordred: and what does that do | 21:06 |
| mordred | prometheanfire: it sets the gerrit info as the pushurl on the origin remote | 21:06 |
| mordred | prometheanfire: I got tired of 'git update' pulling refs once from git.o.o and once from gerrit | 21:06 |
| prometheanfire | mordred: not sure that'd fix this | 21:06 |
| prometheanfire | but ya, that's still a nice change | 21:06 |
| prometheanfire | set | 21:07 |
| mordred | might not - an din any case, figuring out why this is behaving weird is also a thing we shoudl do | 21:07 |
| fungi | #status log cleared stale reprepro update lockfile for debian, manually ran mirror update | 21:07 |
| openstackstatus | fungi: finished logging | 21:07 |
| fungi | SamYaple: ^ | 21:07 |
| fungi | should be safe to recheck the bindep change if it failed on the debian job | 21:07 |
| *** timothyb89 has quit IRC | 21:09 | |
| *** timothyb89 has joined #openstack-infra | 21:10 | |
| prometheanfire | fungi: anything else for me to do? | 21:10 |
| prometheanfire | fungi: just ran scp -P29418 prometheanfire@review.openstack.org:hooks/commit-msg .git/hooks/commit-msg | 21:11 |
| prometheanfire | fungi: was the last thing for git review -s | 21:11 |
| fungi | prometheanfire: nope, thanks! all good info. next is for me or someone else to try reproducing so we can rule out client-side differences as a cause | 21:15 |
| prometheanfire | sure | 21:15 |
| fungi | and no lingering connections | 21:15 |
| prometheanfire | thanks | 21:15 |
| *** salv-orlando has quit IRC | 21:18 | |
| *** tpsilva has quit IRC | 21:19 | |
| ianw | i've cleaned up the old logrotate files on graphite.o.o and made a note to check back on it in a few days to make sure it's rotating as we expect | 21:19 |
| *** eharney has joined #openstack-infra | 21:20 | |
| ianw | also cleared out the ~20gb of old logs i saved in /var/lib/graphite/storage/carbon-cache-a.backup.2018-02-06/ as we don't need them | 21:20 |
| *** dhill__ has joined #openstack-infra | 21:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: rm unused jenkins/scripts/* https://review.openstack.org/541861 | 21:21 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove storelets-release permissions to create tags https://review.openstack.org/541908 | 21:23 |
| *** dhill_ has quit IRC | 21:23 | |
| *** e0ne has quit IRC | 21:23 | |
| ianw | fungi / corvus : also, do you think it's worth me stopping ze02 and trying a) a hard reboot or b) detach interface and re-attach to see if it helps with the ipv6 packet loss? | 21:25 |
| corvus | ianw: i do, but maybe after the oom test? | 21:25 |
| fungi | ianw: i take it you've confirmed there's no observable v4 packet loss, only v6? | 21:25 |
| openstackgerrit | Merged openstack-infra/nodepool master: Convert from legacy to native devstack job https://review.openstack.org/535899 | 21:25 |
| ianw | fungi: doing my same test with the snmpwalk, i never saw a failure with ipv4 | 21:26 |
| corvus | incidentally, it looks like our executor governors are doing a very good job right now -- so much that i kind of want to dial them back a bit so we can overload the executors and cause more oom errors :| | 21:26 |
| *** e0ne has joined #openstack-infra | 21:27 | |
| SamYaple | fungi: thanks buddy | 21:28 |
| fungi | ianw: this was between cacti (not cacti01?) and ze02 right? | 21:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/git-review master: Fix git review -d M,N https://review.openstack.org/541791 | 21:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/git-review master: Testing getting specific patchset https://review.openstack.org/541892 | 21:28 |
| fungi | ianw: oh, looks like the cacti alias goes to cacti02 now? i wonder if we can delete the other instance | 21:29 |
| fungi | looks like we have cacti, cacti01 and cacti02 in our ansible inventory | 21:29 |
| corvus | (the only dead log streamer right now is ze04) | 21:29 |
| ianw | fungi: yep (cacti02) ... try "snmpwalk -v1 -c public udp6:ze02.openstack.org .1.3.6.1.2.1.1.3.0" and confirm that i'm not nuts :) | 21:30 |
| fungi | cacti and cacti01 seem to be cruft in our ansible inventory | 21:30 |
| fungi | they're not in the server list in dfw | 21:30 |
| fungi | clearing the inventory cache manually to see if they disappear on a refresh now | 21:31 |
| fungi | ianw: more bizarre is that i can't `ping6 ze02.openstack.org` from cacti02 | 21:32 |
| fungi | ze01 is fine though | 21:33 |
| fungi | ianw: but i agree, intermittent (25-50%?) timeouts for udp6 snmpgets to ze02, no problem with straight ipv4 udp | 21:35 |
| ianw | true! i feel a bit silly, i tried that but just assumed icmp was blocked, didn't think to ping another host. certainly seems part of it | 21:35 |
| fungi | aha! found the cacti and cacti01 cruft entries in the hostlist ;) | 21:36 |
| fungi | they're still in the emergency file. i'll remove them | 21:36 |
| fungi | done | 21:37 |
| *** armaan has quit IRC | 21:37 | |
| *** armaan has joined #openstack-infra | 21:38 | |
| *** eumel8 has quit IRC | 21:39 | |
| *** threestrands has joined #openstack-infra | 21:39 | |
| *** threestrands has quit IRC | 21:39 | |
| *** threestrands has joined #openstack-infra | 21:39 | |
| openstackgerrit | Merged openstack-infra/storyboard master: import blog post "Things that StoryBoard does differently" https://review.openstack.org/534412 | 21:39 |
| fungi | ianw: i'll check the inventory to see if i can spot a similar prefix-to-prefix blackhole like we had with the others | 21:40 |
| fungi | ianw: no dice this time. logstash-worker01 is also in 2001:4800:7821:105::/64 and ze02 can ping6 that just fine | 21:42 |
| fungi | was worth checking anyway | 21:44 |
| *** slaweq has quit IRC | 21:44 | |
| *** ijw has quit IRC | 21:51 | |
| openstackgerrit | Merged openstack-infra/storyboard master: separate worklist instructions from theory https://review.openstack.org/534413 | 21:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: fix document cross-link syntax https://review.openstack.org/534414 | 21:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: fix syntax for toctree https://review.openstack.org/534415 | 21:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: fix literal block markup for blocks within bullet lists https://review.openstack.org/534416 | 21:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: escape an example URL as literal text to avoid build warning https://review.openstack.org/534417 | 21:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: fix field list markup syntax https://review.openstack.org/534418 | 21:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: highlight notes about permissions on API endpoints https://review.openstack.org/534419 | 21:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: suppress warnings from wsmeext.sphinxext https://review.openstack.org/534872 | 21:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: treat doc build warnings as errors https://review.openstack.org/534873 | 21:52 |
| *** mriedem has quit IRC | 21:53 | |
| *** ijw has joined #openstack-infra | 21:54 | |
| *** Sukhdev has joined #openstack-infra | 21:59 | |
| *** slaweq has joined #openstack-infra | 22:00 | |
| *** slaweq has quit IRC | 22:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Add default logging configuration https://review.openstack.org/541921 | 22:06 |
| *** r-daneel has quit IRC | 22:08 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Remove vagrant setup https://review.openstack.org/532367 | 22:12 |
| openstackgerrit | Merged openstack-infra/bindep master: Fix logic for groups https://review.openstack.org/517105 | 22:13 |
| ianw | fungi: yeah, i feel like it's probably a random thing that would be fixed with, at worst, a rebuild | 22:14 |
| fungi | sure | 22:15 |
| fungi | i concur | 22:15 |
| *** florianf has quit IRC | 22:15 | |
| *** VW_ has joined #openstack-infra | 22:16 | |
| *** VW has quit IRC | 22:19 | |
| *** VW_ has quit IRC | 22:20 | |
| *** claudiub|2 has quit IRC | 22:22 | |
| *** claudiub|2 has joined #openstack-infra | 22:22 | |
| *** rcernin has joined #openstack-infra | 22:25 | |
| dhellmann | I'm trying to figure out how to access group settings through gerrit's REST API and I keep getting authentication errors. I'm using my http password from the settings page in the UI and digest authentication. Can anyone tell me if there's something obviously wrong with that setup? Should I use basic auth? something else? | 22:25 |
| *** felipemonteiro_ has quit IRC | 22:25 | |
| *** agopi has joined #openstack-infra | 22:27 | |
| *** slaweq has joined #openstack-infra | 22:28 | |
| clarkb | dhellmann: I think you have to use the /a or similar urls for things that are auth'd? the gerrit api docs should cover that | 22:29 |
| dhellmann | hmm, ok, let me go look at those docs again | 22:29 |
| dhellmann | ooh, that was it, thanks clarkb | 22:30 |
| clarkb | ya /a/ prefix and digest auth | 22:30 |
| *** slaweq has quit IRC | 22:32 | |
| *** jlabarre has quit IRC | 22:32 | |
| mgagne | any issue with inap-mtl01 since compute node updates? (like network hang in jobs) | 22:35 |
| clarkb | mgagne: I havent noticed any. Have been on the lookout for old in use nodes in nodepool to catch it | 22:37 |
| mgagne | clarkb: thanks | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Launcher: exit immediately https://review.openstack.org/541930 | 22:39 |
| *** bobh has quit IRC | 22:40 | |
| *** fultonj has quit IRC | 22:42 | |
| corvus | dhellmann: system-config/tools/update-gerrit-group.py and maintain-infra-groups.py may be helpful | 22:46 |
| dhellmann | corvus : thanks, I have it now. I wasn't using the /a urls | 22:47 |
| *** e0ne has quit IRC | 22:47 | |
| corvus | dhellmann: oh yeah, i saw -- just thought they might have some reusable code for ya | 22:47 |
| dhellmann | oh, thanks | 22:48 |
| *** fultonj has joined #openstack-infra | 22:49 | |
| *** Goneri has quit IRC | 22:50 | |
| *** slaweq has joined #openstack-infra | 22:52 | |
| dhellmann | yeah, that gerrit client class looks a lot like the one I just wrote :-) | 22:52 |
| *** iyamahat_ has joined #openstack-infra | 22:56 | |
| *** slaweq has quit IRC | 22:56 | |
| *** sree has joined #openstack-infra | 22:57 | |
| *** iyamahat has quit IRC | 22:57 | |
| *** sree has quit IRC | 23:01 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 23:02 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 23:02 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 23:05 |
| *** sree has joined #openstack-infra | 23:07 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/git-review master: Testing getting specific patchset https://review.openstack.org/541892 | 23:07 |
| *** sree has quit IRC | 23:11 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/git-review master: Testing getting specific patchset https://review.openstack.org/541892 | 23:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Builder: exit immediately https://review.openstack.org/541935 | 23:16 |
| *** dhill__ has quit IRC | 23:16 | |
| *** dhill_ has joined #openstack-infra | 23:21 | |
| *** Sukhdev has quit IRC | 23:23 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make timeout value apply to entire job https://review.openstack.org/541485 | 23:25 |
| *** masber has joined #openstack-infra | 23:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Add default logging configuration https://review.openstack.org/541921 | 23:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Launcher: exit immediately https://review.openstack.org/541930 | 23:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Builder: exit immediately https://review.openstack.org/541935 | 23:30 |
| openstackgerrit | Ian Wienand proposed openstack-infra/git-review master: Testing getting specific patchset https://review.openstack.org/541892 | 23:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Noop https://review.openstack.org/541937 | 23:32 |
| *** stakeda has joined #openstack-infra | 23:34 | |
| *** Sukhdev has joined #openstack-infra | 23:36 | |
| *** edmondsw has quit IRC | 23:37 | |
| *** iyamahat_ has quit IRC | 23:37 | |
| *** iyamahat has joined #openstack-infra | 23:38 | |
| *** s-shiono has joined #openstack-infra | 23:39 | |
| fungi | corvus: dhellmann: also the tools/who-approves.py does a lot with group querying (though likely needs updates to deal with the loss of gitweb) | 23:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Update tox docs environment to match build-sphinx-docs https://review.openstack.org/541939 | 23:40 |
| fungi | yeah, still wants to retrieve the projects list and acls via gitweb | 23:41 |
| *** dave-mccowan has quit IRC | 23:45 | |
| *** markvoelker_ has quit IRC | 23:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Add default logging configuration https://review.openstack.org/541921 | 23:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Launcher: exit immediately https://review.openstack.org/541930 | 23:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Builder: exit immediately https://review.openstack.org/541935 | 23:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use python3 version of docs jobs https://review.openstack.org/541940 | 23:49 |
| mordred | corvus, clarkb: ^^ | 23:49 |
| mordred | corvus, clarkb: I'm betting that's the issue - but I haven't reproduced the issue locally yet | 23:50 |
| corvus | mordred: that sounds plausible. zuul is unhappy with that change | 23:50 |
| *** dave-mccowan has joined #openstack-infra | 23:51 | |
| corvus | mordred: though... if that were the case, wouldn't my noop change have failed? | 23:51 |
| *** hongbin has quit IRC | 23:52 | |
| clarkb | corvus: potentially not if the existing code was python2 safe? | 23:52 |
| mordred | yah - perhaps somehting in the logging patch is not python2 safe | 23:52 |
| clarkb | I'm testing it really quick on python2 to see if I can get better error message | 23:52 |
| mordred | clarkb, corvus: yes | 23:52 |
| corvus | ooooh i see | 23:52 |
| mordred | python2 fails on that patch locally | 23:52 |
| mordred | with the same error | 23:53 |
| clarkb | def _read_config_file(filename: str): yup thats not python2 | 23:53 |
| clarkb | isn't even python3.4 safe | 23:53 |
| *** dingyichen has joined #openstack-infra | 23:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove nodepool docs jobs from project-config https://review.openstack.org/541940 | 23:54 |
| mordred | corvus, clarkb: ^^ that matches what goes on in zuul | 23:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Update tox docs environment to match build-sphinx-docs https://review.openstack.org/541939 | 23:56 |
| mordred | corvus, clarkb: ^^ and that should update the nodepool jobs to be correct | 23:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove tox-cover from nodepool https://review.openstack.org/541941 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!