| cloudnull | so nothing super important. | 00:00 |
|---|---|---|
| * cloudnull just tooling about | 00:00 | |
| clarkb | fungi: gerrit continues to look happy | 00:02 |
| clarkb | even with all of today's extra change churn | 00:03 |
| clarkb | and now time to start working on dinner | 00:03 |
| *** xarses has joined #openstack-infra | 00:03 | |
| jeblair | well, except for *right now* of course, because of mysqldump :) | 00:03 |
| *** aeng has quit IRC | 00:03 | |
| *** camunoz_ has joined #openstack-infra | 00:04 | |
| *** baoli has joined #openstack-infra | 00:04 | |
| *** sflanigan has quit IRC | 00:05 | |
| *** baoli_ has joined #openstack-infra | 00:05 | |
| *** zhurong has quit IRC | 00:06 | |
| *** camunoz has quit IRC | 00:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Update osic-cloud1 credential format https://review.openstack.org/356702 | 00:06 |
| oomichi | hi, now we don't see testr_results.html.gz on the gate logs/ | 00:07 |
| openstackgerrit | Craige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio https://review.openstack.org/342481 | 00:07 |
| clarkb | oomichi: I filed a bug against devstack about it. its not writing the subunit file | 00:07 |
| fungi | oomichi: if you happen to know someone on the qa team, maybe they could fix it ;) | 00:08 |
| clarkb | I didnt manage to root cause the issue in drvstack thoigh | 00:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 00:08 |
| oomichi | clarkb: cool, thanks. can we know the link of LP | 00:08 |
| oomichi | fungi: heh, yeah we should help up:) | 00:08 |
| *** baoli has quit IRC | 00:09 | |
| clarkb | oomichi: https://bugs.launchpad.net/devstack/+bug/1617476 | 00:10 |
| openstack | Launchpad bug 1617476 in devstack "devstack.subunit is not generated" [Undecided,Confirmed] | 00:10 |
| oomichi | clarkb: thanks :) | 00:10 |
| pabelanger | clarkb: fungi: osic-cloud8 looks to be ready now. I've update the patches ^ | 00:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 00:11 |
| *** dingyichen has joined #openstack-infra | 00:13 | |
| *** yamahata has quit IRC | 00:14 | |
| *** chlong has quit IRC | 00:16 | |
| *** sdake has joined #openstack-infra | 00:16 | |
| *** sputnik13 has quit IRC | 00:18 | |
| *** pvinci has joined #openstack-infra | 00:19 | |
| pvinci | Hi. | 00:20 |
| *** gouthamr has joined #openstack-infra | 00:20 | |
| pvinci | I have a change that is stuck in "Needs Verified" for the last 6 hours. https://review.openstack.org/358359 | 00:21 |
| ianw | clarkb / oomichi : hmm, having it in that exit trap is rather annoying, since we've closed off all the logging before we run it. no other trap calls either ... i wonder if there's other weird ways you can affect signal masks | 00:21 |
| *** aeng has joined #openstack-infra | 00:21 | |
| *** sflanigan has joined #openstack-infra | 00:21 | |
| pvinci | Can someone help me troubleshoot why? | 00:21 |
| *** thorst has joined #openstack-infra | 00:21 | |
| *** sdake has quit IRC | 00:21 | |
| fungi | pvinci: it depends on an abandoned change for stable/mitaka | 00:22 |
| clarkb | pvinci: your depends on is abandned and cannot merge | 00:22 |
| pvinci | It needed the Depends-On intitialy to get through check. | 00:23 |
| oomichi | ianw: oh, cannot we see any log about that now? that seems difficult to dig | 00:24 |
| *** Swami__ has joined #openstack-infra | 00:24 | |
| *** gildub has joined #openstack-infra | 00:24 | |
| *** ddieterly has joined #openstack-infra | 00:25 | |
| *** Julien-zte has joined #openstack-infra | 00:25 | |
| clarkb | ianw: ya I think the exit trap does it so it happens on fails too but it us annoying | 00:25 |
| *** sdake has joined #openstack-infra | 00:25 | |
| pvinci | Do I just need to create a new patch-set remove the Depends-On:? | 00:25 |
| clarkb | maybe do it in a controlled manner on success but also have trap do it for fails? | 00:25 |
| pabelanger | fungi: clarkb: confirmed osic-cloud8 works with SSL, puppetmaster credentials updated. | 00:25 |
| pabelanger | We can start work in the morning to bring it online | 00:26 |
| clarkb | pvinci: you either need to get the depends on to merge or remove it | 00:26 |
| clarkb | pabelanger: yay | 00:26 |
| *** dfflanders has joined #openstack-infra | 00:26 | |
| pabelanger | I mean, we could start uploading nodepool images this evening | 00:26 |
| pvinci | clarkb: ok. Thank you. | 00:26 |
| clarkb | requires service restarts | 00:26 |
| fungi | pvinci: yeah, taking the depends-on line out of the commit message now should work, as your master branch dependency already merged in neutron | 00:26 |
| fungi | pvinci: but you change will need to get approved again | 00:27 |
| fungi | since the review and workflow votes will be cleared on a commit message update | 00:27 |
| pvinci | fungi: OK. Thanks! | 00:27 |
| *** AnarchyAo has quit IRC | 00:27 | |
| clarkb | pabelanger: I am pretty checked out now with dinner and kids though | 00:27 |
| ianw | oomichi / clarkb : ahh, i guess at least we "set +o xtrace" for that little status message, then never turn it back on, so that might hide what's actually going on | 00:28 |
| ianw | we could start there | 00:28 |
| *** Swami has quit IRC | 00:28 | |
| fungi | mordred: i'm going to give launch-node.py a try with your working ansible virtualenv once 363312 makes it onto puppetmaster.o.o | 00:29 |
| fungi | will let you know how it goes | 00:29 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade: Prevent Negative Caching https://review.openstack.org/363321 | 00:30 |
| *** itisha has quit IRC | 00:30 | |
| pabelanger | clarkb: enjoy, we can wait until tomorrow | 00:31 |
| clarkb | pabelanger: do we have a mirror there yet? | 00:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Upload nodepool images to osic-cloud8 https://review.openstack.org/357364 | 00:32 |
| *** sdague has joined #openstack-infra | 00:32 | |
| pabelanger | clarkb: no, that will be launched in the morning. | 00:32 |
| *** tonytan4ever has quit IRC | 00:33 | |
| *** gildub has quit IRC | 00:33 | |
| *** gildub has joined #openstack-infra | 00:33 | |
| *** rfolco has joined #openstack-infra | 00:33 | |
| *** mtanino has quit IRC | 00:34 | |
| fungi | argh. i should not have a system-config change bounce out of the gate twice on nondeterministic failures. first gate-openstackci-beaker-centos-7 ended up with no rspec installed, now gate-infra-puppet-apply-fedora-23 spontaneously failed to resolve git.openstack.org in dns | 00:34 |
| *** thcipriani is now known as thcipriani|afk | 00:34 | |
| fungi | yeah, the first was actually a connection timeout in osic: http://logs.openstack.org/12/363312/1/gate/gate-openstackci-beaker-centos-7/5b0764a/console.html#_2016-08-31_00_12_45_960072 | 00:36 |
| fungi | the eventual failure was a cascade error stemming from a failure to connect to rubygems.org | 00:36 |
| clarkb | did that go throught nat? | 00:37 |
| *** thorst_ has joined #openstack-infra | 00:37 | |
| clarkb | we might he seeing the limits of the nat there? | 00:37 |
| fungi | great question... i think yes nat. there is no aaaa for that name | 00:37 |
| *** spzala has joined #openstack-infra | 00:39 | |
| ianw | pabelanger: urgh ... you know how i said fedora24 was ready .... 3 hour timeout on this one :( http://logs.openstack.org/12/363212/1/check/gate-tempest-dsvm-platform-fedora24-nv/fc07025/ | 00:39 |
| fungi | the other failure was in bluebox though... http://logs.openstack.org/12/363312/1/gate/gate-infra-puppet-apply-fedora-23/c183f88/console.html#_2016-08-31_00_31_52_724775 | 00:39 |
| clarkb | fungi: bluebox is also nat but 1:1 | 00:39 |
| clarkb | I lioe blaming nat | 00:39 |
| *** thorst has quit IRC | 00:39 | |
| fungi | me too, but i think any further natshaming on my part will require beer | 00:41 |
| fungi | it's just too late in the evening for it not to | 00:41 |
| fungi | anyway, here's hoping third time gating is a charm | 00:41 |
| *** pvaneck has quit IRC | 00:41 | |
| oomichi | clarkb: ianw: We can see testr_results.html on stable branch tests still, that seems happening on master only | 00:41 |
| ianw | oomichi: always? or sometimes? | 00:43 |
| *** spzala has quit IRC | 00:43 | |
| oomichi | ianw: always on my checking(10+ tests), but I'd like to check more | 00:43 |
| fungi | oomichi: that suggests it's a recent regression in devstack's master branch which hasn't been backported to any stable branches? | 00:44 |
| ianw | oomichi: that sounds pretty deterministic | 00:44 |
| pabelanger | ianw: ouch | 00:44 |
| *** yamamoto_ has joined #openstack-infra | 00:44 | |
| oomichi | fungi: yeah, I guess so. and I checked the history of devstack and devstack-gate, but I cannot catch it yet | 00:44 |
| ianw | pabelanger: i hate to say it, but i think ansible might be having some of the blame -> http://paste.openstack.org/show/564955/ | 00:45 |
| ianw | oomichi: i think to start we can put a "trap -p" at the end to make sure the trap is still registered, and turn on tracing. that will give us a clue | 00:46 |
| ianw | pabelanger: a 10734s timeout seems like an odd number | 00:47 |
| *** yamahata has joined #openstack-infra | 00:47 | |
| pabelanger | ianw: 179 mins | 00:49 |
| pabelanger | something is setting it to 3 hours | 00:49 |
| *** M-docaedo_vector has quit IRC | 00:50 | |
| oomichi | ianw: that seems useful, could you help that? | 00:50 |
| ianw | maybe it's already gone for a minute and that's where it comes from? | 00:50 |
| ianw | oomichi: https://review.openstack.org/#/c/363326/ | 00:51 |
| *** caowei has joined #openstack-infra | 00:52 | |
| oomichi | ianw: awesome, thanks :) I'd like to check the result soon | 00:52 |
| *** asettle has joined #openstack-infra | 00:54 | |
| ianw | pabelanger: so the weird thing is, something oddly asynchronous is going on | 00:55 |
| ianw | pabelanger: console log ends at 21:27 http://logs.openstack.org/12/363212/1/check/gate-tempest-dsvm-platform-fedora24-nv/fc07025/console.html | 00:55 |
| *** Goneri has quit IRC | 00:56 | |
| ianw | pabelanger: but the ansible log ends at 00:29 http://logs.openstack.org/12/363212/1/check/gate-tempest-dsvm-platform-fedora24-nv/fc07025/_zuul_ansible/ansible_log.txt | 00:56 |
| *** asettle has quit IRC | 00:59 | |
| *** gouthamr_ has joined #openstack-infra | 01:01 | |
| fungi | pabelanger: oomichi: ianw: clarkb: could it be an ansible-on-xenial behavior (stable branches are still using trusty)? | 01:01 |
| ianw | pabelanger: yeah, we got a problem growing disk on !raxspace hosts? http://paste.openstack.org/show/564957/ | 01:02 |
| ianw | no ... same thing on rax hosts, i guess they just have enough disk to limp through | 01:02 |
| *** Julien-z_ has joined #openstack-infra | 01:02 | |
| *** Julien-zte has quit IRC | 01:03 | |
| ianw | arrgh! why is nothing ever simple! :) | 01:03 |
| fungi | wow, my system-config change just hit another nondeterministic failure ("Unable to look up git.openstack.org" on gate-infra-puppet-apply-fedora-23 in bluebox again) | 01:03 |
| *** salv-orlando has joined #openstack-infra | 01:03 | |
| ianw | fungi: are there ipv6 addresses in there? that was what i saw the other day | 01:03 |
| fungi | ianw: looks like the only ipv6 we have in bluebox is linklocal | 01:04 |
| ianw | fungi: i mean did that message bail out and only put in an ipv6 address as the uncontactable address? if that makes sense, probably not | 01:05 |
| *** gouthamr has quit IRC | 01:05 | |
| ianw | http://logs.openstack.org/12/363312/1/check/gate-infra-puppet-apply-fedora-23/57c387c/console.html <- nup, different error to what i saw yesterday | 01:05 |
| *** Apoorva has quit IRC | 01:06 | |
| *** gongysh has joined #openstack-infra | 01:06 | |
| *** shashank_hegde has quit IRC | 01:06 | |
| oomichi | ianw: clarkb: we lost testr_result between 2016/08/22 - 2016/08/23 | 01:07 |
| oomichi | 08/22: Exist: http://logs.openstack.org/15/352715/1/check/gate-tempest-dsvm-neutron-dvr/2beb15a/logs/ | 01:07 |
| oomichi | 08/23: Lost: http://logs.openstack.org/37/356237/4/check/gate-tempest-dsvm-full-ubuntu-xenial/e476e95/logs/ | 01:07 |
| oomichi | will check git history | 01:08 |
| *** Apoorva has joined #openstack-infra | 01:09 | |
| *** gouthamr_ is now known as gouthamr | 01:09 | |
| oomichi | ianw: clarkb: https://review.openstack.org/#/c/355234/ seems much related.. | 01:11 |
| *** salv-orlando has quit IRC | 01:11 | |
| *** esp has quit IRC | 01:12 | |
| ianw | mmm, could be ... | 01:12 |
| *** salv-orlando has joined #openstack-infra | 01:13 | |
| sdake | hey folks - noticed one of my cohorts commits is not listed in zuul | 01:13 |
| sdake | are only certain projects listed in zuul? | 01:13 |
| fungi | sdake: which change? | 01:13 |
| *** zhurong has joined #openstack-infra | 01:13 | |
| sdake | fungi https://review.openstack.org/#/c/363319/ | 01:13 |
| sdake | networking-vpp repo - whatever that is :) | 01:13 |
| fungi | i saw some networking-vpp changes showing up in the zuul status page earlier | 01:14 |
| sdake | ok then | 01:14 |
| sdake | maybe its just gate overload | 01:14 |
| fungi | i remember because there was a change chain several dozen long | 01:14 |
| sdake | lol | 01:15 |
| fungi | yeah, looks like that change is in a stack of 70+ changes | 01:15 |
| openstackgerrit | Merged openstack-infra/system-config: Use regular expressions for wiki hostgroups https://review.openstack.org/363312 | 01:15 |
| sdake | fungi wow | 01:16 |
| fungi | that's crazy long for a dependent patch series | 01:16 |
| *** psilvad has joined #openstack-infra | 01:17 | |
| *** ijw has joined #openstack-infra | 01:17 | |
| sdake | fungi apparenty its initial git history | 01:17 |
| fungi | the zuul layout.yaml claims it should be running the python-jobs template, so presumably those changes would be tested normally in the check pipeline | 01:17 |
| ijw | fungi: sorry I'm a bad person. | 01:17 |
| fungi | ijw: not at all, just a surprisingly long series of changes | 01:17 |
| *** salv-orlando has quit IRC | 01:17 | |
| clarkb | initial historry can be imported when we create the project fwiw | 01:18 |
| fungi | it's possible it confused something--i'm trying to track down what that might have been | 01:18 |
| clarkb | too late now though | 01:18 |
| sdake | clarkb that is precisely what i told ian :) | 01:18 |
| ijw | clarkb: Yeah, but unfortunately I didn't create the project... | 01:18 |
| oomichi | https://review.openstack.org/#/c/363336/ is a reverting patch | 01:18 |
| *** M-docaedo_vector has joined #openstack-infra | 01:18 | |
| sdake | fungi so the 70 commits - should punish the gate for several days I thnk :( | 01:19 |
| fungi | zuul's debug log is taking a while to open | 01:19 |
| ijw | Anyway, if this patch chain turns out to be an eldritch horror, just make recommendations and I'll do what I can | 01:19 |
| fungi | ijw: i recommend a gate seal ;) | 01:19 |
| sdake | ijw considered squashing commit? | 01:19 |
| * fungi makes terrible hpl joke | 01:19 | |
| sdake | fungi wtb gate seal for kolla :) | 01:19 |
| ijw | sdake: I was asked to preserve the history (and believe me it's already squashed from what it was) | 01:20 |
| fungi | it's too bad the great old ones weren't viable mascot choices | 01:20 |
| ijw | And teh staypuft man is probably copyright | 01:20 |
| sdake | by red hat no doubt ;-) | 01:20 |
| ijw | But conveniently sticky if you do need to seal the gate | 01:20 |
| sdake | why oh why would they name a product staypuft | 01:21 |
| sdake | ijw I htink the problem yu will have is your first commit will fail the gate | 01:21 |
| sdake | along with the other 70 | 01:21 |
| ijw | Yeah, understood | 01:22 |
| fungi | sdake: ijw: looking back at https://review.openstack.org/363319 it seems it's not enqueued into the gate because "Change <Change 0x7facc939f690 363289,2> is needed but can not be merged" | 01:22 |
| sdake | fungi thanks i didn't catch it was 70 patches long :( | 01:22 |
| ijw | OK, that's probably a good thing | 01:22 |
| *** david-lyle_ has joined #openstack-infra | 01:23 | |
| *** esikachev has quit IRC | 01:23 | |
| fungi | yeah, so basically the changes have to merge in their expressed dependency order. start with the ones closest to be branch tip and see why they're not merging, then work your way up fixing whatever the ci complains about | 01:23 |
| sdake | fungi i think he will need da nonvoting gate for that to work | 01:23 |
| sdake | fungi but yup sounds vible ;) | 01:24 |
| clarkb | or just make it work... | 01:24 |
| clarkb | its trivial to bootstrap that | 01:24 |
| clarkb | for python jobs at least | 01:24 |
| *** baoli_ has quit IRC | 01:24 | |
| *** M-docaedo_vector has quit IRC | 01:24 | |
| ijw | clarkb: ? | 01:24 |
| sdake | clarkb agree - i'll get em going | 01:24 |
| sdake | ijw he means to make your gate job nonvoting | 01:24 |
| ijw | Ah | 01:24 |
| clarkb | no | 01:24 |
| clarkb | I mean make them pass :) | 01:24 |
| sdake | oh wrong sorry :) | 01:24 |
| fungi | it's shim in enough boilerplate ahead of those changes to get all jobs passing, and then make sure they all pass at every stage in the series | 01:24 |
| clarkb | ya that | 01:25 |
| ijw | fungi: you ask much | 01:25 |
| ijw | Let me see what I can do about it | 01:25 |
| sdake | fungi - gog:) | 01:25 |
| fungi | however, doing that for 71 changes may be too daunting | 01:25 |
| ijw | I can squash it more | 01:25 |
| ijw | Till it squeaks | 01:25 |
| sdake | ijw i'd jut turn the gate nonvoting | 01:25 |
| sdake | get the history in | 01:25 |
| sdake | turn the gate voting | 01:25 |
| sdake | victory | 01:25 |
| *** akshai has quit IRC | 01:26 | |
| fungi | you might replace the python-jobs template in zuul's layout.yaml with the noop-jobs template if you're just going to set all those jobs nonvoting | 01:26 |
| pabelanger | ianw: I can dig into it tomorrow morning | 01:27 |
| fungi | then put python-jobs back once you have your "import" completed and properly test subsequent proposed changes | 01:27 |
| ijw | That's fine with me (not least because I don't want to be eating gate time) | 01:27 |
| ianw | pabelanger: yeah ... so no idea why growroot isn't working :( i'll try and figure that out | 01:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 01:27 |
| *** fguillot has quit IRC | 01:27 | |
| fungi | ijw: sdake: but clarkb is right that making your changes pass the already defined jobs is less work on us, as we don't need to review project-config changes tu turn your jobs off and on again | 01:28 |
| pabelanger | ianw: do you need to revert? | 01:28 |
| ijw | Given nothing has gone in, not yet | 01:28 |
| *** rfolco has quit IRC | 01:28 | |
| *** Swami has joined #openstack-infra | 01:28 | |
| sdake | fungi - understood - not sure if the cats woring on this can get networking-vpp going and preserve the history | 01:28 |
| ianw | pabelanger: i think might have to ... it's a crap shoot if it has enough space or not depending on which host it runs on | 01:28 |
| clarkb | and not only that it helps you see that your code works | 01:28 |
| pabelanger | ianw: also, do you have a minute to review 362908, 362900 | 01:28 |
| ianw | pabelanger: it would be ok if the job died in a timely fashion ... but sitting there for hours is bad news | 01:29 |
| ijw | And fungi: let me see what I can do. Again, this is history. The historical versions didn't pass the tests, so I can condense them down to a handful of versions that do, or I can just erase all history and add the authors in (but I would prefer not to if I could avoid it) | 01:29 |
| fungi | ijw: sdake: basically, it's probably possible to just configure the tox.ini to return true for the pep8 and py27 envs, but docs is going to be harder since that relies on `python setup.py build_sphinx` not bombing out with a nonzero exit code | 01:29 |
| sdake | fungi thats a good idea | 01:30 |
| ijw | Argh, two ianw's in the same channel, that's just confusing | 01:30 |
| *** fguillot has joined #openstack-infra | 01:30 | |
| *** Apoorva has quit IRC | 01:30 | |
| ijw | fungi: that one is comparatively easy - I had to put in a template doc, and I can just move that patch up to the biginning of the stack rather than the end trivially enough | 01:30 |
| fungi | ijw: yeah, i also used to get the two of you confused occasionally | 01:30 |
| * ijw is easily confused | 01:31 | |
| *** yamamoto_ has quit IRC | 01:31 | |
| sdake | i feel used all the time ;) | 01:31 |
| ijw | Let me rewrite history | 01:31 |
| * ijw feels like the next Dr Who | 01:31 | |
| *** yanyanhu has joined #openstack-infra | 01:32 | |
| *** Swami__ has quit IRC | 01:32 | |
| fungi | have sonic screwdriver, will time-travel | 01:32 |
| *** sdague has quit IRC | 01:32 | |
| fungi | yay! my wiki ansible groups fix merged finally | 01:33 |
| *** tonytan4ever has joined #openstack-infra | 01:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Revert "Switch in Fedora 24 devstack job" https://review.openstack.org/363340 | 01:33 |
| ianw | pabelanger: ^ :( | 01:34 |
| ijw | OK, if I rewrite history I'm guessing I want shot of the change-ids in the commit messages, just to confirm | 01:34 |
| sdake | ,ozzxjkzxxxhhhhhnope | 01:34 |
| sdake | nope | 01:34 |
| sdake | ijw no change ids will cahnge just orderin gerrit | 01:34 |
| fungi | if rewriting history means just using rebase -i to reorder some commits, then you should be fine | 01:35 |
| fungi | what you _don't_ want is altering the change-ids in the commit messages, since that will cause the old changes associated with them to become orphaned cruft which will need to be separately abandoned | 01:35 |
| fungi | and with 70 some changes, that would be pretty annoying to clean up | 01:36 |
| openstackgerrit | Merged openstack-infra/irc-meetings: kuryr: update meeting settings https://review.openstack.org/362852 | 01:36 |
| sdake | fungi the onl ytime that makes sensse is during a sqush | 01:36 |
| fungi | sdake: yep, in a squash case you'll need to manually abandon the old changes associated with the no longer needed change-ids | 01:36 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add a UTC clock and integrate local meeting time https://review.openstack.org/353008 | 01:37 |
| ijw | Let me see what I can do with this. I should be able ot script up a nasty little walk through with tox locally to see if this is working | 01:37 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Check chairs are in the 'correct' format as part of pep8 https://review.openstack.org/347511 | 01:38 |
| *** thorst_ has quit IRC | 01:38 | |
| *** yamamoto_ has joined #openstack-infra | 01:38 | |
| *** thorst has joined #openstack-infra | 01:38 | |
| *** tonytan4ever has quit IRC | 01:38 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Run placement-api job in devstack experimental https://review.openstack.org/363342 | 01:39 |
| *** chlong has joined #openstack-infra | 01:41 | |
| *** changzhi has joined #openstack-infra | 01:42 | |
| *** kzaitsev_mb has quit IRC | 01:44 | |
| *** yuanying has quit IRC | 01:45 | |
| *** yuanying has joined #openstack-infra | 01:46 | |
| *** ddieterly has quit IRC | 01:46 | |
| *** Goneri has joined #openstack-infra | 01:46 | |
| *** thorst has quit IRC | 01:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Run windmill-jobs-trusty(-nv) on master https://review.openstack.org/362900 | 01:48 |
| openstackgerrit | Merged openstack-infra/project-config: Switch windmill-jobs-centos7 for voting https://review.openstack.org/362908 | 01:49 |
| *** M-docaedo_vector has joined #openstack-infra | 01:52 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove the check target https://review.openstack.org/347512 | 01:52 |
| *** netsin has joined #openstack-infra | 01:52 | |
| fungi | mordred: that (newer ansible) seems to have worked. now i'm just down to mundane vcsrepo troubleshooting | 01:53 |
| *** zshuo has joined #openstack-infra | 01:55 | |
| *** spzala has joined #openstack-infra | 01:55 | |
| fungi | ianw: I'm +2 on that entire chain ending in 359500. worked great once i got past the group expansion related issues with the older ansible we're pinned to | 01:55 |
| *** zhurong_ has joined #openstack-infra | 01:59 | |
| *** kuntelin has joined #openstack-infra | 02:00 | |
| *** pvinci has quit IRC | 02:01 | |
| *** pvinci has joined #openstack-infra | 02:02 | |
| *** zhurong has quit IRC | 02:02 | |
| amrith | clarkb, thanks for confirming earlier today that you were able to install tempest in a venv. I'm re-running now ... | 02:03 |
| amrith | and it appears to be working better. I have enough to debug the failure (and it is local to my machines). | 02:04 |
| amrith | clarkb ^^ | 02:04 |
| *** kuntelin_ has joined #openstack-infra | 02:06 | |
| *** kuntelin has quit IRC | 02:06 | |
| *** ddieterly has joined #openstack-infra | 02:10 | |
| *** aeng has quit IRC | 02:10 | |
| *** dchen has joined #openstack-infra | 02:11 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: NPM DSVM jobs are now voting. https://review.openstack.org/363200 | 02:11 |
| *** sflanigan has quit IRC | 02:14 | |
| *** pblaho has quit IRC | 02:14 | |
| *** Goneri has quit IRC | 02:15 | |
| *** sdake has quit IRC | 02:16 | |
| *** shashank_hegde has joined #openstack-infra | 02:16 | |
| *** sdake has joined #openstack-infra | 02:17 | |
| *** kuntelin_ has quit IRC | 02:17 | |
| *** vinaypotluri has quit IRC | 02:22 | |
| *** tqtran has quit IRC | 02:22 | |
| *** Sukhdev has joined #openstack-infra | 02:23 | |
| *** aeng has joined #openstack-infra | 02:23 | |
| *** Swami__ has joined #openstack-infra | 02:25 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: cover more combinations of version https://review.openstack.org/347045 | 02:26 |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: add job for OpenDaylight carbon https://review.openstack.org/363356 | 02:26 |
| *** spzala has quit IRC | 02:26 | |
| *** Swami has quit IRC | 02:28 | |
| *** Swami_ has quit IRC | 02:29 | |
| *** Swami has joined #openstack-infra | 02:29 | |
| *** tphummel has joined #openstack-infra | 02:29 | |
| *** fguillot has quit IRC | 02:30 | |
| *** mriedem has quit IRC | 02:31 | |
| *** david-lyle_ has quit IRC | 02:31 | |
| *** gyee has quit IRC | 02:31 | |
| *** psilvad has quit IRC | 02:32 | |
| *** ddieterly has quit IRC | 02:33 | |
| *** baoli has joined #openstack-infra | 02:34 | |
| *** tonytan4ever has joined #openstack-infra | 02:34 | |
| *** yamahata has quit IRC | 02:35 | |
| *** dimtruck is now known as zz_dimtruck | 02:37 | |
| *** Jeffrey4l_ has joined #openstack-infra | 02:38 | |
| *** tonytan4ever has quit IRC | 02:38 | |
| *** tonytan4ever has joined #openstack-infra | 02:38 | |
| *** baoli has quit IRC | 02:40 | |
| *** gildub has quit IRC | 02:42 | |
| *** thorst has joined #openstack-infra | 02:45 | |
| *** reed_ has joined #openstack-infra | 02:46 | |
| *** reed_ has quit IRC | 02:47 | |
| *** yamamoto_ has quit IRC | 02:48 | |
| *** thorst has quit IRC | 02:52 | |
| *** pblaho has joined #openstack-infra | 02:54 | |
| *** pblaho has quit IRC | 02:59 | |
| *** gnuoy has quit IRC | 03:00 | |
| *** gnuoy` has joined #openstack-infra | 03:00 | |
| *** mriedem has joined #openstack-infra | 03:00 | |
| *** jamespage has quit IRC | 03:01 | |
| *** mriedem has quit IRC | 03:01 | |
| *** jamespag` has joined #openstack-infra | 03:01 | |
| *** mriedem has joined #openstack-infra | 03:01 | |
| *** mriedem has quit IRC | 03:01 | |
| *** yamamoto_ has joined #openstack-infra | 03:02 | |
| *** Genek has joined #openstack-infra | 03:16 | |
| *** salv-orlando has joined #openstack-infra | 03:17 | |
| *** tqtran has joined #openstack-infra | 03:21 | |
| *** yamamoto_ has quit IRC | 03:22 | |
| *** yamamoto_ has joined #openstack-infra | 03:22 | |
| *** yamamoto_ has quit IRC | 03:22 | |
| *** changzhi has quit IRC | 03:24 | |
| *** salv-orlando has quit IRC | 03:24 | |
| *** pblaho has joined #openstack-infra | 03:24 | |
| *** yamamoto_ has joined #openstack-infra | 03:25 | |
| *** aeng has quit IRC | 03:26 | |
| *** Ravikiran_K has joined #openstack-infra | 03:27 | |
| *** winggundamth has quit IRC | 03:27 | |
| *** winggundamth has joined #openstack-infra | 03:29 | |
| *** yamamoto_ has quit IRC | 03:29 | |
| *** flepied has joined #openstack-infra | 03:30 | |
| *** AnarchyAo has joined #openstack-infra | 03:32 | |
| *** esp has joined #openstack-infra | 03:34 | |
| *** shashank_hegde has quit IRC | 03:36 | |
| *** woodster_ has quit IRC | 03:39 | |
| *** vikrant has joined #openstack-infra | 03:40 | |
| *** adriant has quit IRC | 03:41 | |
| *** aeng has joined #openstack-infra | 03:43 | |
| openstackgerrit | Craige McWhirter proposed openstack-infra/puppet-phabricator: Configure HTTPD and HTTPS certificates https://review.openstack.org/350370 | 03:44 |
| *** changzhi has joined #openstack-infra | 03:45 | |
| *** shashank_hegde has joined #openstack-infra | 03:45 | |
| ianw | oomichi: so the trap is set ... that's something ... http://logs.openstack.org/26/363326/1/check/gate-tempest-dsvm-full-ubuntu-xenial/aec60b5/logs/devstacklog.txt.gz#_2016-08-31_01_12_51_020 | 03:49 |
| *** esp has quit IRC | 03:49 | |
| *** thorst has joined #openstack-infra | 03:50 | |
| *** gouthamr has quit IRC | 03:51 | |
| *** esp has joined #openstack-infra | 03:51 | |
| *** cody-somerville has quit IRC | 03:51 | |
| *** cody-somerville has joined #openstack-infra | 03:52 | |
| *** thorst has quit IRC | 03:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Switch in Fedora 24 devstack job" https://review.openstack.org/363340 | 03:57 |
| *** ijw has quit IRC | 04:04 | |
| *** dingyichen has quit IRC | 04:05 | |
| *** ijw has joined #openstack-infra | 04:08 | |
| *** cody-somerville has quit IRC | 04:08 | |
| *** cody-somerville has joined #openstack-infra | 04:08 | |
| *** cody-somerville has quit IRC | 04:08 | |
| *** cody-somerville has joined #openstack-infra | 04:08 | |
| ianw | pabelanger: ^ see https://bugzilla.redhat.com/show_bug.cgi?id=1371761 . i'll see what response i get in the bug then decide what to do | 04:11 |
| openstack | bugzilla.redhat.com bug 1371761 in util-linux "sfdisk return code breaks growpart" [Unspecified,New] - Assigned to kzak | 04:11 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework https://review.openstack.org/322307 | 04:14 |
| *** crinkle_ has joined #openstack-infra | 04:16 | |
| *** cschwede has quit IRC | 04:17 | |
| *** crinkle has quit IRC | 04:17 | |
| *** cschwede has joined #openstack-infra | 04:17 | |
| *** kaisers has quit IRC | 04:17 | |
| *** kaisers has joined #openstack-infra | 04:19 | |
| *** Genek has quit IRC | 04:21 | |
| *** jdg has joined #openstack-infra | 04:24 | |
| jdg | #join #openstack-cinder | 04:25 |
| ijw | OK, I have rewritten history. In this version of history my changes should pass the tests. Also, gerrit is not purple, so I consider that a bonus. | 04:26 |
| *** spzala has joined #openstack-infra | 04:26 | |
| *** salv-orlando has joined #openstack-infra | 04:27 | |
| *** ijw has quit IRC | 04:27 | |
| *** pgadiya has joined #openstack-infra | 04:28 | |
| *** ijw has joined #openstack-infra | 04:28 | |
| *** yamamoto_ has joined #openstack-infra | 04:29 | |
| *** spzala has quit IRC | 04:31 | |
| *** jdg has quit IRC | 04:32 | |
| *** salv-orlando has quit IRC | 04:32 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Make neutron-fwaas functional job not experimental https://review.openstack.org/359320 | 04:32 |
| *** ijw has quit IRC | 04:33 | |
| *** yuanying has quit IRC | 04:34 | |
| *** yuanying has joined #openstack-infra | 04:34 | |
| *** dtantsur|afk has quit IRC | 04:36 | |
| *** coolsvap_ has joined #openstack-infra | 04:36 | |
| *** links has joined #openstack-infra | 04:36 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Make neutron-fwaas functional job not experimental https://review.openstack.org/359320 | 04:37 |
| *** senk has joined #openstack-infra | 04:38 | |
| *** chlong has quit IRC | 04:38 | |
| *** chlong has joined #openstack-infra | 04:39 | |
| *** vinaypotluri has joined #openstack-infra | 04:41 | |
| *** sarob has joined #openstack-infra | 04:41 | |
| ianw | oomichi: ok, so exit trap is getting called -> http://logs.openstack.org/26/363326/2/check/gate-tempest-dsvm-full-ubuntu-xenial/ecaf107/logs/devstacklog.txt.gz | 04:43 |
| *** esp has quit IRC | 04:45 | |
| *** chlong has quit IRC | 04:45 | |
| *** sarob has quit IRC | 04:46 | |
| *** tphummel has quit IRC | 04:47 | |
| *** esp has joined #openstack-infra | 04:54 | |
| *** thorst has joined #openstack-infra | 04:54 | |
| *** dingyichen has joined #openstack-infra | 04:55 | |
| *** ijw has joined #openstack-infra | 04:55 | |
| *** Julien-z_ has quit IRC | 04:56 | |
| *** Jeffrey4l_ has quit IRC | 04:56 | |
| *** asettle has joined #openstack-infra | 04:57 | |
| *** chlong has joined #openstack-infra | 04:58 | |
| *** ijw has quit IRC | 05:00 | |
| *** claudiub has joined #openstack-infra | 05:01 | |
| *** thorst has quit IRC | 05:01 | |
| *** asettle has quit IRC | 05:02 | |
| *** yamahata has joined #openstack-infra | 05:04 | |
| *** psachin has joined #openstack-infra | 05:05 | |
| *** yamahata has quit IRC | 05:06 | |
| *** sdake has quit IRC | 05:10 | |
| *** sdake_ has joined #openstack-infra | 05:10 | |
| *** salv-orlando has joined #openstack-infra | 05:13 | |
| *** ilyashakhat has joined #openstack-infra | 05:16 | |
| *** changzhi has quit IRC | 05:18 | |
| *** senk has quit IRC | 05:20 | |
| *** jaosorior has joined #openstack-infra | 05:21 | |
| *** yonglihe has joined #openstack-infra | 05:22 | |
| *** hichihara has joined #openstack-infra | 05:24 | |
| *** hichihara has quit IRC | 05:24 | |
| *** roxanaghe has joined #openstack-infra | 05:25 | |
| *** sdake_ has quit IRC | 05:26 | |
| *** javeriak has joined #openstack-infra | 05:26 | |
| *** ilyashakhat has quit IRC | 05:30 | |
| *** gildub has joined #openstack-infra | 05:30 | |
| *** roxanaghe has quit IRC | 05:32 | |
| *** sdake has joined #openstack-infra | 05:33 | |
| *** Na3iL has joined #openstack-infra | 05:34 | |
| craige | Is there problem with the checks or am I missing something? http://logs.openstack.org/70/350370/3/check/gate-puppet-phabricator-puppet-lint/ec4c9d1/console.html | 05:38 |
| AJaeger | morning ianw, could you review https://review.openstack.org/#/c/362839/2 , please? | 05:38 |
| AJaeger | craige: http://logs.openstack.org/70/350370/3/check/gate-puppet-phabricator-puppet-lint/ec4c9d1/console.html#_2016-08-31_03_48_50_273894 - timed out | 05:39 |
| *** Julien-zte has joined #openstack-infra | 05:39 | |
| craige | I thought that was it but was unsure. thanks for confirming AJaeger | 05:40 |
| *** caowei has quit IRC | 05:41 | |
| openstackgerrit | Merged openstack-infra/project-config: Introduce functional/fullstack Neutron Xenial jobs https://review.openstack.org/359843 | 05:42 |
| *** nwkarsten has quit IRC | 05:42 | |
| AJaeger | craige: I expect that's it ;) So, recheck... | 05:42 |
| *** nwkarsten has joined #openstack-infra | 05:42 | |
| *** caowei has joined #openstack-infra | 05:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove unused gate-tempest-dsvm-full-ceph-plugin-src filter https://review.openstack.org/363205 | 05:44 |
| *** nwkarsten has quit IRC | 05:47 | |
| *** mtanino has joined #openstack-infra | 05:48 | |
| *** ifarkas_afk has quit IRC | 05:50 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Change tripleo ha2 job types back too ha https://review.openstack.org/363409 | 05:51 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove the ha2 JOBTYPE https://review.openstack.org/363411 | 05:52 |
| *** senk has joined #openstack-infra | 05:54 | |
| *** _nadya_ has joined #openstack-infra | 05:54 | |
| *** armax has quit IRC | 05:56 | |
| *** sandanar has joined #openstack-infra | 05:57 | |
| *** ilyashakhat has joined #openstack-infra | 05:57 | |
| *** aeng has quit IRC | 05:57 | |
| *** camunoz_ has quit IRC | 05:57 | |
| *** dingyichen has quit IRC | 05:58 | |
| *** markvoelker has joined #openstack-infra | 05:58 | |
| *** _nadya_ has quit IRC | 05:59 | |
| *** thorst has joined #openstack-infra | 06:00 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Higgins: Add post script for tempest api testing https://review.openstack.org/363416 | 06:00 |
| vrovachev1 | Hello, dear colleagues. I was created patch for fuel-qa project with fix in misprint for gates. Please take a look https://review.openstack.org/#/c/362871 | 06:01 |
| *** martinkopec has joined #openstack-infra | 06:01 | |
| *** oanson has joined #openstack-infra | 06:02 | |
| *** Swami has quit IRC | 06:04 | |
| *** Swami__ has quit IRC | 06:04 | |
| odyssey4me | hmm, it looks like there are a bunch of nodes in a waiting state which aren't transitioning to any other state | 06:06 |
| *** thorst has quit IRC | 06:07 | |
| *** salv-orl_ has joined #openstack-infra | 06:07 | |
| *** salv-orlando has quit IRC | 06:10 | |
| *** aeng has joined #openstack-infra | 06:10 | |
| *** camunoz_ has joined #openstack-infra | 06:11 | |
| *** dingyichen has joined #openstack-infra | 06:11 | |
| *** Genek has joined #openstack-infra | 06:11 | |
| *** ianychoi has quit IRC | 06:12 | |
| *** ianychoi has joined #openstack-infra | 06:12 | |
| *** esikachev has joined #openstack-infra | 06:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Adds Magnum API Reference jobs to gate and check builds https://review.openstack.org/362839 | 06:14 |
| *** pcaruana has joined #openstack-infra | 06:15 | |
| *** claudiub has quit IRC | 06:17 | |
| *** javeriak has quit IRC | 06:18 | |
| *** ilyashakhat has quit IRC | 06:19 | |
| *** andreas_s has joined #openstack-infra | 06:20 | |
| *** rcernin has joined #openstack-infra | 06:21 | |
| *** sdake has quit IRC | 06:22 | |
| odyssey4me | jhesketh AJaeger any idea what's up with the hanging nodes in nodepool? | 06:24 |
| *** salv-orl_ has quit IRC | 06:24 | |
| jhesketh | odyssey4me: something does look odd there, yes | 06:27 |
| odyssey4me | jhesketh it looks like there are a small number running, but there are tons of merge check jobs waiting | 06:27 |
| odyssey4me | and the check queue is slowly climbing - even though there should be plenty of nodes to consume | 06:27 |
| jhesketh | jobs aren't launching, so it's probably something to do with zuul or zuul-launcher | 06:28 |
| *** ijw has joined #openstack-infra | 06:28 | |
| *** amotoki has joined #openstack-infra | 06:29 | |
| *** Genek has quit IRC | 06:29 | |
| *** Illumitardi has joined #openstack-infra | 06:30 | |
| *** abregman has quit IRC | 06:31 | |
| *** mtanino has quit IRC | 06:32 | |
| *** watanabe_isao has joined #openstack-infra | 06:32 | |
| *** markvoelker has quit IRC | 06:34 | |
| *** dingyichen has quit IRC | 06:35 | |
| HeOS | Hello, infra-team! I'd like to ask to review the following request: https://review.openstack.org/#/c/362002/. I'll really appreciate your help. | 06:36 |
| *** gildub has quit IRC | 06:37 | |
| *** nwkarsten has joined #openstack-infra | 06:44 | |
| ianw | jhesketh: hmm, something's going on but it's not immediately obvious to me | 06:44 |
| jhesketh | ianw: yep, agreed... I'm poking around at the logs atm | 06:45 |
| *** mikelk has joined #openstack-infra | 06:45 | |
| ianw | yeah, nodepool seems to be assigning and deleting nodes | 06:45 |
| jhesketh | ianw: looks like something may have just gotten unstuck... | 06:47 |
| ianw | there was something earlier with like 70 dependent changes, an initial import put through as separate changes | 06:48 |
| ianw | i wonder if that's being digested | 06:48 |
| *** nwkarsten has quit IRC | 06:49 | |
| *** javeriak has joined #openstack-infra | 06:49 | |
| *** ilyashakhat has joined #openstack-infra | 06:49 | |
| jhesketh | I was wrong about it being unstuck.. | 06:50 |
| ianw | 2016-08-30 07:07:33,795 ERROR gear.Client.unknown: Exception in poll loop: | 06:53 |
| ianw | this seems to be the most recent error in zuul | 06:53 |
| *** ijw has quit IRC | 06:53 | |
| *** camunoz_ has quit IRC | 06:53 | |
| ianw | 2016-08-31 06:06:10,989 ERROR zuul.Scheduler: Exception in run handler: | 06:54 |
| ianw | zuul@zuul:/var/log/zuul$ date | 06:54 |
| ianw | Wed Aug 31 06:54:08 UTC 2016 | 06:54 |
| *** ilyashakhat has quit IRC | 06:54 | |
| ianw | jhesketh: ^ maybe the scheduler loop has stopped? 50 minutes ago might be about right | 06:54 |
| *** amotoki has quit IRC | 06:56 | |
| ianw | http://paste.openstack.org/show/565070/ | 06:56 |
| jhesketh | ianw: maybe, but it looks like it should continue okay : http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/scheduler.py#n993 | 06:58 |
| *** javeriak has quit IRC | 06:59 | |
| jhesketh | ianw: gearman is showing no jobs requested or running... it shows a high number of workers | 07:00 |
| jhesketh | that supports your scheduler theory | 07:00 |
| ianw | jhesketh: i guess the logs don't really, right after that is "2016-08-31 06:06:10,991 INFO zuul.Scheduler:" so zuul.Scheduler is still logging | 07:01 |
| *** javeriak has joined #openstack-infra | 07:02 | |
| dfflanders | rcarrillocruz, will you be in Barcelona? Hoping to request your skills to do team teaching with us on shade for AppDev again? | 07:03 |
| *** thorst has joined #openstack-infra | 07:05 | |
| *** esikachev has quit IRC | 07:05 | |
| * AJaeger just one message where a change finished testing, so something is going... | 07:05 | |
| AJaeger | morning ianw and jhesketh , thanks for looking into this! | 07:05 |
| jhesketh | ianw: something may have unstuck now however | 07:06 |
| jhesketh | 267 nodes building | 07:06 |
| AJaeger | yeah! | 07:07 |
| ianw | hmm, yes ... | 07:07 |
| *** dtardivel has joined #openstack-infra | 07:07 | |
| *** tesseract- has joined #openstack-infra | 07:08 | |
| *** bethwhite_ has joined #openstack-infra | 07:08 | |
| ianw | hmm, glitch in the matrix? | 07:09 |
| *** florianf has joined #openstack-infra | 07:11 | |
| jhesketh | yeah, I don't like not knowing why it unstuck... | 07:11 |
| jhesketh | the merge-check pipeline is still huge... | 07:11 |
| *** yaume has joined #openstack-infra | 07:11 | |
| *** thorst has quit IRC | 07:12 | |
| *** shardy has joined #openstack-infra | 07:12 | |
| *** kaisers_ has joined #openstack-infra | 07:13 | |
| AJaeger | 1100 entries... | 07:13 |
| *** ihrachys has joined #openstack-infra | 07:14 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:15 | |
| jhesketh | and growing :-s | 07:16 |
| AJaeger | jhesketh, ianw: Since yolanda is on vacation, I would appreciate if you help a bit more than usual with reviewing of changes. Could you go this week some time over open project-config changes, please? | 07:16 |
| AJaeger | jhesketh: so, back to normal ;) | 07:16 |
| jhesketh | heh | 07:16 |
| *** amotoki has joined #openstack-infra | 07:16 | |
| jhesketh | AJaeger: I'm always happy to help with reviews, just point me somewhere :-) | 07:17 |
| jhesketh | I have done a poor job the last month or so on picking up fresh reviews sadly :-( | 07:17 |
| ianw | AJaeger: sure | 07:17 |
| AJaeger | A dashboard with jobs that have seen one +2 already: | 07:17 |
| AJaeger | https://review.openstack.org/#/dashboard/?foreach=%2528project%253Aopenstack%252Dinfra%252Fproject%252Dconfig%2529+status%253Aopen+NOT+label%253AWorkflow%253C%253D%252D1+label%253AVerified%253E%253D1%252Cjenkins+is%253Amergeable&title=AJ+Review+Inbox&Needs+final+%252B2=NOT+label%253ACode%252DReview%253C%253D%252D1%252Cproject%252Dconfig%252Dcore+label%253ACode%252DReview%253E%253D2 | 07:17 |
| AJaeger | Note that some might wait on dependent changes, some might wait on Neutron Infra liason. | 07:18 |
| *** nmagnezi has joined #openstack-infra | 07:18 | |
| *** ifarkas has joined #openstack-infra | 07:18 | |
| AJaeger | And there're also some changes that are unreviewed... | 07:19 |
| *** jerryz has joined #openstack-infra | 07:19 | |
| *** dizquierdo has joined #openstack-infra | 07:20 | |
| * AJaeger is offline for a bit now... | 07:21 | |
| *** hichihara has joined #openstack-infra | 07:22 | |
| *** esikachev has joined #openstack-infra | 07:22 | |
| *** drifterza has joined #openstack-infra | 07:23 | |
| *** hichihara has quit IRC | 07:24 | |
| *** yanyanhu has quit IRC | 07:25 | |
| *** Hal1 has quit IRC | 07:26 | |
| *** yanyanhu has joined #openstack-infra | 07:26 | |
| *** spzala has joined #openstack-infra | 07:26 | |
| *** jerryz has quit IRC | 07:27 | |
| *** jlanoux has joined #openstack-infra | 07:29 | |
| *** markvoelker has joined #openstack-infra | 07:30 | |
| *** spzala has quit IRC | 07:31 | |
| *** markvoelker has quit IRC | 07:35 | |
| *** kaisers_ has quit IRC | 07:35 | |
| *** jamespag` is now known as jamespage | 07:35 | |
| *** abregman has joined #openstack-infra | 07:36 | |
| *** jpich has joined #openstack-infra | 07:37 | |
| *** abregman has quit IRC | 07:37 | |
| *** abregman has joined #openstack-infra | 07:38 | |
| *** shardy has quit IRC | 07:39 | |
| *** salv-orlando has joined #openstack-infra | 07:39 | |
| *** shardy has joined #openstack-infra | 07:40 | |
| *** gnuoy` is now known as gnuoy | 07:40 | |
| *** Hal1 has joined #openstack-infra | 07:41 | |
| *** vincentll has joined #openstack-infra | 07:42 | |
| *** spzala has joined #openstack-infra | 07:42 | |
| *** _nadya_ has joined #openstack-infra | 07:43 | |
| *** hashar has joined #openstack-infra | 07:44 | |
| *** vincentll has quit IRC | 07:44 | |
| *** zhurong_ has quit IRC | 07:45 | |
| openstackgerrit | Vadim Rovachev proposed openstack-infra/project-config: Fix branches for fuel-qa gates https://review.openstack.org/362871 | 07:46 |
| *** jlanoux has quit IRC | 07:47 | |
| *** spzala has quit IRC | 07:47 | |
| *** zhurong has joined #openstack-infra | 07:48 | |
| openstackgerrit | Vadim Rovachev proposed openstack-infra/project-config: Fix branches for fuel-qa gates https://review.openstack.org/362871 | 07:50 |
| *** Sukhdev has quit IRC | 07:52 | |
| *** cdent has joined #openstack-infra | 07:56 | |
| *** priteau has joined #openstack-infra | 07:57 | |
| *** shashank_hegde has quit IRC | 07:57 | |
| *** spzala has joined #openstack-infra | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** mugsie|alt has quit IRC | 08:00 | |
| *** hichihara has joined #openstack-infra | 08:00 | |
| *** mugsie|alt has joined #openstack-infra | 08:01 | |
| *** markvoelker has joined #openstack-infra | 08:01 | |
| *** matthewbodkin has joined #openstack-infra | 08:01 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| AJaeger | ianw, jhesketh : Still something is odd with zuul - having entries for 3+ hours in merge-check is odd. Also, post and periodic queues are not served. | 08:02 |
| *** r-mibu has quit IRC | 08:02 | |
| *** spzala has quit IRC | 08:03 | |
| *** matrohon has joined #openstack-infra | 08:05 | |
| *** chlong has quit IRC | 08:06 | |
| *** markvoelker has quit IRC | 08:07 | |
| hashar | the job queue is quite fuel, maybe it is a spam of merge:merge jobs? | 08:09 |
| *** thorst has joined #openstack-infra | 08:09 | |
| hashar | fuel .. | 08:10 |
| hashar | full | 08:10 |
| *** claudiub has joined #openstack-infra | 08:11 | |
| hashar | AJaeger: looks like nodepool is spinning instances again as of 7:00utc with lot of "in use" nodes. So I guess the stack is going to catch up all fine | 08:11 |
| hashar | AJaeger: and 'post' and 'periodic' have a low precedence, so if something is starved those queues are probably not processed at all | 08:13 |
| *** rossella_s has joined #openstack-infra | 08:16 | |
| *** thorst has quit IRC | 08:17 | |
| acabot | hi guys, sorry to interupt, I dont really understand why jenkins fail on https://review.openstack.org/#/c/362984/. Any guess ? Thx | 08:17 |
| acabot | logs are http://pastebin.com/0T3xTLcB | 08:18 |
| acabot | and I dont see where I need to define those jobs... | 08:18 |
| acabot | thx | 08:18 |
| jhesketh | AJaeger: Yeah, I'm going to let it clear out some of the other queues first... as hashar points out the post/periodic ones are low priority | 08:19 |
| jhesketh | the merge though isn't looking healthy | 08:19 |
| jhesketh | acabot: | 08:19 |
| hashar | maybe some zuul-merger have troubles catching up | 08:19 |
| *** lucas-dinner is now known as lucasagomes | 08:20 | |
| *** javeriak has quit IRC | 08:20 | |
| jhesketh | acabot: you need to define the jobs in jenkins/jobs/ if you're familiar with that? | 08:20 |
| AJaeger | jhesketh: I would expect the proposal slave - translation jobs - run. That slave is only used in periodic and post, so should be served. But I don't see it in use. | 08:20 |
| AJaeger | jhesketh: nc proposal.slave.openstack.org 19885 - last action hours ago ;( | 08:20 |
| AJaeger | is it offline? | 08:20 |
| jhesketh | hmm | 08:21 |
| acabot | jhesketh : I dont think I need to add a job as I'm using a standard job 'publish-to-pypi" in layout.yaml | 08:22 |
| *** kong has quit IRC | 08:22 | |
| jhesketh | acabot: that only exists if you have that template initiated in jjb | 08:22 |
| *** javeriak has joined #openstack-infra | 08:22 | |
| *** yanyanhu has quit IRC | 08:22 | |
| acabot | jhesketh : jjb ? | 08:23 |
| AJaeger | acabot: it'S not a standard job. It's a job specific to your repository | 08:23 |
| jhesketh | acabot: jenkins job builder | 08:23 |
| AJaeger | acabot: Read http://jaegerandi.blogspot.de/2016/02/creating-new-test-jobs-in-openstack-ci.html | 08:23 |
| *** kong has joined #openstack-infra | 08:23 | |
| *** yanyanhu has joined #openstack-infra | 08:23 | |
| acabot | AJaeger : thx let me look at this :-) | 08:24 |
| *** e0ne has joined #openstack-infra | 08:24 | |
| AJaeger | acabot: also documented in the Infra Manual docs.openstack.org/infra/manual | 08:24 |
| jhesketh | AJaeger: zlstatic and proposal.slave look okay... (still poking) | 08:24 |
| *** martinkopec has quit IRC | 08:25 | |
| *** martinkopec has joined #openstack-infra | 08:25 | |
| openstackgerrit | Antoine Cabot proposed openstack-infra/project-config: Add publish-to-pypi for watcher-dashboard https://review.openstack.org/362984 | 08:25 |
| *** gongysh has quit IRC | 08:26 | |
| *** dchen has quit IRC | 08:26 | |
| *** oomichi has quit IRC | 08:27 | |
| *** zubchick has quit IRC | 08:27 | |
| *** javeriak has quit IRC | 08:28 | |
| *** javeriak has joined #openstack-infra | 08:29 | |
| *** oomichi has joined #openstack-infra | 08:29 | |
| *** markvoelker has joined #openstack-infra | 08:29 | |
| *** gildub has joined #openstack-infra | 08:29 | |
| openstackgerrit | Vadim Rovachev proposed openstack-infra/project-config: Change ACLs for fuel-qa https://review.openstack.org/359704 | 08:30 |
| *** auggy has quit IRC | 08:30 | |
| *** zubchick has joined #openstack-infra | 08:30 | |
| openstackgerrit | Vadim Rovachev proposed openstack-infra/project-config: Change ACLs for fuel-qa project https://review.openstack.org/359704 | 08:30 |
| *** derekh has joined #openstack-infra | 08:31 | |
| *** rossella_s has quit IRC | 08:31 | |
| *** auggy has joined #openstack-infra | 08:31 | |
| AJaeger | acabot: you need to *Define* a job in jjb and then you can schedule the job in zuul. | 08:32 |
| AJaeger | Your change to jjb is just a variable in an existing job, no definition of a new one | 08:32 |
| *** r-mibu has joined #openstack-infra | 08:33 | |
| *** Julien-zte has quit IRC | 08:33 | |
| *** markvoelker has quit IRC | 08:34 | |
| openstackgerrit | Arie Bregman proposed openstack-infra/zuul: Add 'reset_branch' option to Merger https://review.openstack.org/362049 | 08:36 |
| AJaeger | jhesketh: is zlstatic processing merge-check as well? | 08:36 |
| *** Julien-zte has joined #openstack-infra | 08:36 | |
| *** binbincong has joined #openstack-infra | 08:37 | |
| openstackgerrit | Antoine Cabot proposed openstack-infra/project-config: Add publish-to-pypi for watcher-dashboard https://review.openstack.org/362984 | 08:37 |
| *** ihrachys has quit IRC | 08:37 | |
| *** samueldmq has quit IRC | 08:38 | |
| *** JerryOpenix has quit IRC | 08:39 | |
| *** samueldmq has joined #openstack-infra | 08:39 | |
| jhesketh | AJaeger: it shouldn't... merge-check should just go to the zuul-mergers | 08:39 |
| *** mhickey has joined #openstack-infra | 08:39 | |
| *** javeriak has quit IRC | 08:40 | |
| AJaeger | ah, ok. Just wild guessing to see whether there was a direct connection | 08:41 |
| ianw | oomichi: http://logs.openstack.org/26/363326/3/check/gate-tempest-dsvm-full-ubuntu-xenial/3ccdbee/logs/devstacklog.txt.gz#_2016-08-31_07_17_14_339 | 08:43 |
| *** sarob has joined #openstack-infra | 08:43 | |
| ianw | oomichi: in short, it seems to be running, and it seems it would output to the right location. i think that maybe leaves something changing on the copying side. if you don't get a chance, i'll look into it my time tomorrow | 08:44 |
| *** JerryOpenix has joined #openstack-infra | 08:44 | |
| *** yolanda has joined #openstack-infra | 08:45 | |
| *** sarob has quit IRC | 08:48 | |
| *** watanabe_isao has quit IRC | 08:51 | |
| acabot | AJaeger : thx for your help and sorry for not looking again at the doc ;-) | 08:54 |
| *** pilgrimstack has quit IRC | 08:56 | |
| *** asettle has joined #openstack-infra | 08:58 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Higgins: Add post script for tempest api testing https://review.openstack.org/363416 | 08:58 |
| *** AnarchyAo has quit IRC | 08:59 | |
| *** hichihara has quit IRC | 08:59 | |
| *** markvoelker has joined #openstack-infra | 08:59 | |
| *** pilgrimstack has joined #openstack-infra | 08:59 | |
| *** andreykurilin_ has joined #openstack-infra | 09:00 | |
| *** markvoelker has quit IRC | 09:04 | |
| jhesketh | AJaeger: merge-check is going down... I think we just need to give it time :-) | 09:05 |
| *** nijaba has quit IRC | 09:05 | |
| hashar | is that a function to attempt to do a merge:merge via zuul-merger ? | 09:06 |
| jhesketh | hashar: is that a question to me, I'm not sure I follow sorry | 09:07 |
| *** nijaba has joined #openstack-infra | 09:07 | |
| *** nijaba has joined #openstack-infra | 09:07 | |
| *** nwkarsten has joined #openstack-infra | 09:07 | |
| *** electrofelix has joined #openstack-infra | 09:07 | |
| *** nwkarste_ has joined #openstack-infra | 09:09 | |
| *** nwkarst__ has joined #openstack-infra | 09:10 | |
| AJaeger | jhesketh: great | 09:10 |
| AJaeger | jhesketh: and proposal slave is handling translation jobs... | 09:10 |
| AJaeger | That took a long time to recover ;( Sorry for beeing impatient | 09:11 |
| jhesketh | yeah I think it was just the priority settings | 09:11 |
| jhesketh | all good.. zuul does a lot so I've learned to be patient | 09:11 |
| *** nwkarsten has quit IRC | 09:11 | |
| AJaeger | jhesketh: Ah, requirements updates first... | 09:12 |
| *** nwkarste_ has quit IRC | 09:13 | |
| *** dtantsur has joined #openstack-infra | 09:13 | |
| *** cdent has left #openstack-infra | 09:14 | |
| *** vgridnev has quit IRC | 09:14 | |
| *** ihrachys has joined #openstack-infra | 09:14 | |
| *** thorst has joined #openstack-infra | 09:14 | |
| *** nwkarst__ has quit IRC | 09:14 | |
| *** abregman has quit IRC | 09:15 | |
| *** jlibosva has joined #openstack-infra | 09:16 | |
| jlibosva | AJaeger: hi | 09:16 |
| AJaeger | hi jlibosva | 09:16 |
| jlibosva | AJaeger: thanks for help with the xenial jobs. I think we/I have a problem | 09:16 |
| jlibosva | AJaeger: it seems the functional-ubuntu-trusty is now not part of check queue | 09:17 |
| AJaeger | jlibosva: that sounds right | 09:17 |
| *** yanyanhu has quit IRC | 09:18 | |
| AJaeger | jlibosva: you replaced gate-neutron-dsvm-functional-nv by gate-neutron-dsvm-functional-ubuntu-trusty-nv in the check queue | 09:18 |
| AJaeger | Or which job are you talking about? | 09:19 |
| *** chem has joined #openstack-infra | 09:20 | |
| *** jlanoux has joined #openstack-infra | 09:20 | |
| jlibosva | AJaeger: I wanted to rename gate-neutron-dsvm-functional to gate-neutron-dsvm-functional-ubuntu-trusty | 09:21 |
| *** vgridnev has joined #openstack-infra | 09:21 | |
| AJaeger | and that happened, didn't it? | 09:22 |
| *** thorst has quit IRC | 09:22 | |
| AJaeger | jlibosva: you have me confused, please explain what went wrong and show some proof of it | 09:22 |
| jlibosva | AJaeger: It seems it did not. The test disappeared | 09:22 |
| jlibosva | AJaeger: ok :) | 09:22 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci: Add quotes around systemctl command in test https://review.openstack.org/363518 | 09:22 |
| jlibosva | AJaeger: gimme a minute I'll find examples | 09:22 |
| jlibosva | AJaeger: previously, we ran functional tests on Neutron - it ran on trusty: https://review.openstack.org/#/c/351287/ - name of the job is gate-neutron-dsvm-functional | 09:23 |
| *** pzhurba has joined #openstack-infra | 09:24 | |
| *** andreykurilin_ has quit IRC | 09:25 | |
| jlibosva | AJaeger: What I intended was to keep the functional tests running for Neutron on Trusty, until we have a confidence to switch to Xenial. So as I modified template, I renamed this job to gate-neutron-dsvm-functional-ubuntu-trusty. | 09:25 |
| jlibosva | AJaeger: to keep it running as it was before - but it's not what happened - https://review.openstack.org/#/c/333804/ | 09:25 |
| pzhurba | Hello | 09:25 |
| jlibosva | AJaeger: I'd expect to have gate-neutron-dsvm-functional-ubuntu-trusty there | 09:25 |
| pzhurba | Review please https://review.openstack.org/#/c/362950 | 09:26 |
| *** lock_ has joined #openstack-infra | 09:26 | |
| *** andreykurilin has joined #openstack-infra | 09:26 | |
| AJaeger | jlibosva: checking... | 09:26 |
| *** andreykurilin has left #openstack-infra | 09:27 | |
| AJaeger | jlibosva: http://logs.openstack.org/43/359843/8/gate/gate-project-config-layout/e1563ca/console.html#_2016-08-31_05_41_42_847966 | 09:27 |
| *** abregman has joined #openstack-infra | 09:28 | |
| AJaeger | the change you did for gate-neutron-dsvm-fullstack-ubuntu-trusty with the branch needs to be done for this job as well | 09:28 |
| *** markvoelker has joined #openstack-infra | 09:29 | |
| jlibosva | AJaeger: it's shouldn't have -nv suffix | 09:29 |
| jlibosva | s/it's/it/ | 09:29 |
| *** ianychoi has quit IRC | 09:29 | |
| AJaeger | jlibosva: http://logs.openstack.org/43/359843/8/gate/gate-project-config-layout/e1563ca/console.html#_2016-08-31_05_41_43_098043 | 09:29 |
| AJaeger | Look elsewhere in the file - both are wrong ;) | 09:30 |
| *** yolanda has quit IRC | 09:30 | |
| *** andreykurilin has joined #openstack-infra | 09:30 | |
| AJaeger | jlibosva: do you know what you need to do? | 09:30 |
| jlibosva | AJaeger: I need to define regex for our job to be running on all branches | 09:31 |
| jlibosva | our job = funcitonal-ubuntu-trusty | 09:31 |
| *** andreykurilin has left #openstack-infra | 09:31 | |
| jlibosva | AJaeger: thanks for your help! And sorry for screwing up, I'm quite n00b in project-config :) | 09:31 |
| AJaeger | jlibosva: you're welcome - we need more help here ;) | 09:32 |
| *** andreykurilin has joined #openstack-infra | 09:33 | |
| *** andreykurilin has left #openstack-infra | 09:33 | |
| *** markvoelker has quit IRC | 09:33 | |
| *** andreykurilin has joined #openstack-infra | 09:34 | |
| rcarrillocruz | dfflanders: it is a bit soon to know if i'll get funding for the summit | 09:34 |
| *** andreykurilin__ has quit IRC | 09:36 | |
| BobBall | mordred: https://review.openstack.org/gitweb?p=openstack%2Fnova.git;a=commitdiff;h=1bb5a0d1017dd634444932dc87dc8d6c4460934b - I guess it may not have made it into Rackspace's cloud yet, but XenAPI should indeed support disk labels | 09:36 |
| BobBall | mordred: Or perhaps I mis-understood the conversation... hmmmz | 09:37 |
| BobBall | :) | 09:37 |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config: Run functional-ubuntu-trusty jobs on all branches https://review.openstack.org/363531 | 09:37 |
| jlibosva | AJaeger: ^^ I hope this is it :) | 09:37 |
| openstackgerrit | Merged openstack-infra/project-config: Add publish-to-pypi for watcher-dashboard https://review.openstack.org/362984 | 09:37 |
| AJaeger | jlibosva: let'S wait for build results and then double check the gate-project-config-layout lines | 09:38 |
| jlibosva | ack | 09:38 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/puppet-os_client_config: Fill module by manifest https://review.openstack.org/363533 | 09:39 |
| openstackgerrit | Merged openstack-infra/project-config: Fix branches for fuel-qa gates https://review.openstack.org/362871 | 09:40 |
| *** chem has quit IRC | 09:41 | |
| *** chem has joined #openstack-infra | 09:42 | |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config: Run functional-ubuntu-trusty jobs on all branches https://review.openstack.org/363531 | 09:43 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Re-add note that markdown formatting is supported https://review.openstack.org/363162 | 09:43 |
| *** salv-orlando has quit IRC | 09:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Add tripleo-centos-7-ovb-ha-ipv6 experimental job https://review.openstack.org/362966 | 09:47 |
| openstackgerrit | Merged openstack-infra/project-config: Check for jobs without attributes in Zuul layout.yaml https://review.openstack.org/363214 | 09:48 |
| *** pgadiya has quit IRC | 09:49 | |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/jenkins-job-builder: Add support for Parameterized Scheduler Plugin https://review.openstack.org/353165 | 09:49 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses https://review.openstack.org/363545 | 09:50 |
| *** vinaypotluri has quit IRC | 09:52 | |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses https://review.openstack.org/363545 | 09:52 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/puppet-os_client_config: Add manifest https://review.openstack.org/363533 | 09:56 |
| *** markvoelker has joined #openstack-infra | 09:57 | |
| dlahn | morning | 09:59 |
| *** yamamoto_ has quit IRC | 10:01 | |
| *** zhurong has quit IRC | 10:01 | |
| *** markvoelker has quit IRC | 10:02 | |
| *** salv-orlando has joined #openstack-infra | 10:02 | |
| *** eranrom has quit IRC | 10:02 | |
| *** ifarkas_ has joined #openstack-infra | 10:05 | |
| *** pgadiya has joined #openstack-infra | 10:07 | |
| openstackgerrit | Merged openstack-infra/project-config: networking-midonet: Fix gate_hook check https://review.openstack.org/360312 | 10:08 |
| *** javeriak has joined #openstack-infra | 10:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Add k8s-docker-suite-app-murano project to openstack https://review.openstack.org/357745 | 10:09 |
| *** yaume_ has joined #openstack-infra | 10:09 | |
| *** yaume has quit IRC | 10:12 | |
| jlibosva | AJaeger: do we have any other magician who could help us merging it in emea region? :) | 10:13 |
| AJaeger | jlibosva: jhesketh is reviewing right now (thanks!) | 10:16 |
| jlibosva | thank you! | 10:16 |
| AJaeger | jlibosva: I mean reviewing in general... | 10:17 |
| *** javeriak has quit IRC | 10:19 | |
| AJaeger | jhesketh: https://review.openstack.org/#/c/363531 - and I would approve that without waiting for armax to fix a regression | 10:19 |
| jhesketh | AJaeger, jlibosva: looking already ;-) | 10:19 |
| *** Julien-zte has quit IRC | 10:19 | |
| *** thorst has joined #openstack-infra | 10:20 | |
| AJaeger | thanks! | 10:20 |
| jhesketh | AJaeger, jlibosva: I'm confused though.. if the job/branch were omitted wouldn't it already run on every branch? | 10:20 |
| jlibosva | jhesketh: IIUC there is some magic that ubuntu-trusty runs only on some branches | 10:21 |
| AJaeger | jhesketh: we have the global regexes for ubuntu-trusty and ubuntu-xenial | 10:21 |
| jlibosva | by default | 10:21 |
| AJaeger | jhesketh: line 1171 | 10:21 |
| jhesketh | ah that's right, I see | 10:22 |
| jhesketh | thanks | 10:22 |
| * jhesketh missed the ubuntu part of the name | 10:22 | |
| *** _degorenko|afk is now known as degorenko | 10:22 | |
| jhesketh | +w | 10:22 |
| jlibosva | jhesketh: AJaeger thanks! I owe you a beer or something :) | 10:23 |
| jhesketh | anytime :-) | 10:23 |
| AJaeger | jlibosva: you're welcome | 10:23 |
| *** markvoelker has joined #openstack-infra | 10:26 | |
| *** thorst has quit IRC | 10:26 | |
| openstackgerrit | Merged openstack-infra/project-config: add in missing gnocchi 2.2 job references https://review.openstack.org/361268 | 10:29 |
| *** HeOS has quit IRC | 10:30 | |
| *** markvoelker has quit IRC | 10:30 | |
| *** HeOS has joined #openstack-infra | 10:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Run functional-ubuntu-trusty jobs on all branches https://review.openstack.org/363531 | 10:31 |
| *** bethwhite_ has quit IRC | 10:31 | |
| wznoinsk | hi infra, I get the below when run manually or when building image with openstack-repos element: | 10:33 |
| wznoinsk | git clone git://git.openstack.org/openstack/k8s-docker-suite-app-murano.git | 10:33 |
| wznoinsk | Cloning into 'k8s-docker-suite-app-murano'... | 10:33 |
| wznoinsk | fatal: remote error: access denied or repository not exported: /openstack/k8s-docker-suite-app-murano.git | 10:33 |
| AJaeger | wznoinsk: when was your change merged? IT takes an hour or two for new repos to be setup properly. So, please try again later and if after two hours, it's not merged, tell us again. | 10:35 |
| wznoinsk | AJaeger: yeah, I see it's a fresh project-config change, not mine, caught it only while rebuilding my image using dib | 10:37 |
| AJaeger | wznoinsk: So, either use an old config file that does not reference it or wait until it's cloned (and there are few more repos that were just approved) | 10:39 |
| *** ansiwen has quit IRC | 10:42 | |
| *** sarob has joined #openstack-infra | 10:44 | |
| AJaeger | jlibosva: note that we run job changes via cron every 15 mins - it might take an hour until everything is updated. So, your change is not live yet. | 10:45 |
| wznoinsk | AJaeger: not really doable, element sources the project list of https://git.openstack.org/cgit/openstack-infra/project-config/plain/gerrit/projects.yaml, will have to hack by hand | 10:45 |
| *** mestery has quit IRC | 10:46 | |
| AJaeger | jlibosva: http://status.openstack.org/zuul/ has at bottom "Last reconfigured: " - check when your change merged and whether the reconfigure is afater that - and then your change is life. BUT any change currently tested will get the new job after reconfiguring... | 10:46 |
| openstackgerrit | Yuval Brik proposed openstack-infra/project-config: Rename Smaug to Karbor https://review.openstack.org/353304 | 10:46 |
| *** sdague has joined #openstack-infra | 10:46 | |
| *** sarob has quit IRC | 10:49 | |
| AJaeger | wznoinsk: have a lunch break ;) | 10:49 |
| *** sdake has joined #openstack-infra | 10:49 | |
| electrofelix | zaro: could you check out my response on https://review.openstack.org/#/c/312885/2/git_review/cmd.py | 10:55 |
| *** markvoelker has joined #openstack-infra | 10:56 | |
| *** markvoelker has quit IRC | 11:00 | |
| *** dizquierdo has quit IRC | 11:01 | |
| *** rhallisey has joined #openstack-infra | 11:02 | |
| *** esikachev has quit IRC | 11:03 | |
| *** salv-orlando has quit IRC | 11:04 | |
| *** vstoiko has quit IRC | 11:05 | |
| *** amotoki has quit IRC | 11:09 | |
| *** bethwhite_ has joined #openstack-infra | 11:09 | |
| *** bethwhite__ has joined #openstack-infra | 11:10 | |
| *** bethwhite_ has quit IRC | 11:10 | |
| *** bethwhite__ has quit IRC | 11:10 | |
| *** bethwhite_ has joined #openstack-infra | 11:10 | |
| *** Na3iL has quit IRC | 11:10 | |
| *** amotoki has joined #openstack-infra | 11:14 | |
| *** javeriak has joined #openstack-infra | 11:15 | |
| *** Ahharu has joined #openstack-infra | 11:16 | |
| Ahharu | Hello | 11:16 |
| Ahharu | Is there issues with openstack git? | 11:16 |
| AJaeger | Ahharu: not a known one - what is your problem? | 11:16 |
| Ahharu | SSL problems when trying to download the packages by r10k | 11:17 |
| *** wznoinsk has quit IRC | 11:18 | |
| *** sarob has joined #openstack-infra | 11:19 | |
| *** ramishra has quit IRC | 11:21 | |
| AJaeger | Ahharu: sorry, can't help | 11:21 |
| AJaeger | Ahharu: hope somebody else will be around to help soon - how can we reproduce your problem? Did it work before? | 11:22 |
| *** ramishra has joined #openstack-infra | 11:23 | |
| *** shardy is now known as shardy_lunch | 11:24 | |
| *** sarob has quit IRC | 11:24 | |
| *** markvoelker has joined #openstack-infra | 11:25 | |
| Ahharu | yes it was working before, the thing is that SSL connection to git.openstack.org sometimes gets stuck | 11:26 |
| Ahharu | plain http works fine | 11:26 |
| *** jkilpatr has joined #openstack-infra | 11:26 | |
| *** salv-orlando has joined #openstack-infra | 11:26 | |
| sdague | AJaeger: yeh, my devstack is hung on a git fetch right now | 11:26 |
| sdague | any infra-root up? | 11:27 |
| AJaeger | sdague: not that I'm aware off | 11:27 |
| jhesketh | sdague: I'm around | 11:28 |
| sdague | something is up with the git servers | 11:28 |
| *** rtheis has joined #openstack-infra | 11:28 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Only ask for the overcloud-full image https://review.openstack.org/363592 | 11:29 |
| sdague | I had a dead hang locally when trying to run devstack | 11:29 |
| sdague | and someone else above just reported similar issue | 11:29 |
| *** markvoelker has quit IRC | 11:29 | |
| jhesketh | sdague: do you have any more to go off? eg can you reproduce with -vvv? | 11:31 |
| sdague | nope, I restarted this devstack run. It was buried inside enough layers there all I had was the git fetch hang | 11:32 |
| AJaeger | jhesketh: I thought it was too late for you... | 11:32 |
| *** wznoinsk has joined #openstack-infra | 11:33 | |
| jhesketh | AJaeger: it's evening, but always happy to help if I can | 11:33 |
| openstackgerrit | Liam Young proposed openstack-infra/project-config: Add service-control charm interface https://review.openstack.org/363595 | 11:33 |
| AJaeger | thanks, jhesketh ! | 11:33 |
| *** esikachev has joined #openstack-infra | 11:34 | |
| *** nwkarsten has joined #openstack-infra | 11:34 | |
| jhesketh | sdague: which repo was it fetching? | 11:36 |
| sdague | requirements | 11:36 |
| sdague | maybe it was a blip, and it's back now, I seem to be getting further now | 11:36 |
| AJaeger | looks also like puppet hasn't been run the last 90 minutes - looking at http://puppetboard.openstack.org/ | 11:37 |
| *** kzaitsev_mb has joined #openstack-infra | 11:37 | |
| *** YorikSar has quit IRC | 11:37 | |
| *** baoli has joined #openstack-infra | 11:37 | |
| jhesketh | sdague: hmm okay... let me know if you see it again.. fwiw there is nothing obvious that I can see in the server stats etc | 11:38 |
| *** xyang1 has quit IRC | 11:39 | |
| jlibosva | AJaeger: ok, thanks for info | 11:39 |
| *** nwkarsten has quit IRC | 11:39 | |
| *** ldnunes has joined #openstack-infra | 11:40 | |
| *** dfflanders has quit IRC | 11:41 | |
| openstackgerrit | Liam Young proposed openstack-infra/project-config: Add service-control charm interface https://review.openstack.org/363595 | 11:42 |
| *** lucasagomes is now known as lucas-hungry | 11:43 | |
| rcarrillocruz | mordred: when you get around, gimme a ping, i'm seeing some oddness in glance/nova interactions in the infracloud | 11:43 |
| *** thorst has joined #openstack-infra | 11:43 | |
| *** YorikSar has joined #openstack-infra | 11:43 | |
| *** baoli_ has joined #openstack-infra | 11:46 | |
| *** rfolco has joined #openstack-infra | 11:47 | |
| *** coolsvap_ is now known as coolsvap | 11:47 | |
| *** baoli has quit IRC | 11:49 | |
| *** nwkarsten has joined #openstack-infra | 11:49 | |
| *** sshnaidm is now known as sshnaidm|afk | 11:50 | |
| fungi | sdague: i'm around now too for a bit | 11:51 |
| *** asettle has quit IRC | 11:53 | |
| *** psilvad has joined #openstack-infra | 11:54 | |
| AJaeger | jhesketh, fungi : It looks like none of the merges from over an hour ago is active - is puppet running that long? Looking at puppetboard, I see last results are from 1h:47. Is that ok? | 11:54 |
| AJaeger | morning, fungi ! | 11:54 |
| *** rhallisey has quit IRC | 11:54 | |
| *** nwkarsten has quit IRC | 11:54 | |
| rcarrillocruz | mordred: the issue is http://paste.openstack.org/show/565138/, i get failures creating instances (not sure when this happened, it was working last week) due to nova asking to glance for the image id on the v1 endpoint, despite being the image published on the v2. I've tried putting IMAGE_API_VERSION: '2' on the OSCC clouds.yaml but it doesn't make any difference | 11:55 |
| fungi | sdague: according to http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n8 our image updates start at 10:34 utc, but i don't see any significant traffic spikes on the git server from that (at least not in excess of what we see from our ci later in the day) http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all | 11:56 |
| *** markvoelker has joined #openstack-infra | 11:56 | |
| fungi | AJaeger: i have to step away for a few minutes, but i can check our ansible log when i get back and see if it's spending lots of excess time somewhere in the loop | 11:57 |
| *** ddieterly has joined #openstack-infra | 11:57 | |
| fungi | should be around again in ~30 minutes | 11:57 |
| AJaeger | fungi: sure - perhaps it'S back fine then ;) | 11:57 |
| *** asettle has joined #openstack-infra | 11:58 | |
| *** asettle has quit IRC | 11:58 | |
| *** larainema has quit IRC | 11:58 | |
| *** asettle has joined #openstack-infra | 11:59 | |
| *** yamamoto has joined #openstack-infra | 11:59 | |
| *** spzala has joined #openstack-infra | 11:59 | |
| *** psilvad has quit IRC | 11:59 | |
| *** rhallisey has joined #openstack-infra | 11:59 | |
| *** yamamoto_ has joined #openstack-infra | 11:59 | |
| *** markvoelker has quit IRC | 12:01 | |
| *** larainema has joined #openstack-infra | 12:02 | |
| *** ddieterly has quit IRC | 12:02 | |
| *** Na3iL has joined #openstack-infra | 12:03 | |
| *** tongli has quit IRC | 12:03 | |
| *** spzala has quit IRC | 12:03 | |
| *** yamamoto has quit IRC | 12:03 | |
| *** salv-orlando has quit IRC | 12:04 | |
| *** asselin_ has quit IRC | 12:04 | |
| jhesketh | AJaeger, fungi: ansible/puppet haven't ran on non infra-cloud nodes for a while... I can't tell if that's just because of locks though | 12:04 |
| *** jaosorior has quit IRC | 12:04 | |
| *** jlibosva has quit IRC | 12:05 | |
| rcarrillocruz | jhesketh , fungi : i run manual ansible puppet runs against infracloud to bring up computes, that's why there are recent runs against them | 12:05 |
| rcarrillocruz | fyi | 12:05 |
| jhesketh | ah okay, thanks rcarrillocruz | 12:06 |
| *** jaosorior has joined #openstack-infra | 12:06 | |
| AJaeger | rcarrillocruz: I'm talking about runs on zuul etc | 12:09 |
| *** rodrigods has quit IRC | 12:09 | |
| *** rodrigods has joined #openstack-infra | 12:09 | |
| *** gouthamr has joined #openstack-infra | 12:09 | |
| *** Jeffrey4l_ has joined #openstack-infra | 12:10 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: to accelerate placement api work, put it in nv check https://review.openstack.org/363613 | 12:11 |
| openstackgerrit | Bartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029 | 12:11 |
| *** salv-orlando has joined #openstack-infra | 12:12 | |
| sdague | fungi / AJaeger / jhesketh / rcarrillocruz can I get a fast review on https://review.openstack.org/363613 | 12:12 |
| sdague | it's moving a job from experimental into nv check for nova | 12:12 |
| *** dtantsur is now known as dtantsur|bbl | 12:12 | |
| AJaeger | looking... | 12:12 |
| *** ddieterly has joined #openstack-infra | 12:12 | |
| *** dprince has joined #openstack-infra | 12:13 | |
| *** shardy_lunch is now known as shardy | 12:15 | |
| sdague | hmm... same issue | 12:16 |
| sdague | same git issue | 12:16 |
| jhesketh | fungi: looks like ansible is running puppet across our nodes okay now (probably was before too) | 12:16 |
| dhellmann | good morning | 12:16 |
| jhesketh | sdague: same repo? | 12:16 |
| sdague | yes | 12:16 |
| jhesketh | hmm | 12:17 |
| sdague | if I jump in from another session, and try again, it works | 12:17 |
| sdague | but this is going to be painful if 50% of devstack builds just hang locally | 12:17 |
| jhesketh | sdague: where in devstack is it failing (do you have a line) | 12:19 |
| jhesketh | gerrit appears to be syncing requirements without any trouble | 12:19 |
| mordred | sdague: also, any idea if we're seeing this in a specific regoin? | 12:19 |
| AJaeger | jhesketh: indeed, zuul was just reconfigured | 12:19 |
| sdague | mordred: the region is my house | 12:20 |
| *** pradk has joined #openstack-infra | 12:20 | |
| *** berendt has joined #openstack-infra | 12:20 | |
| mordred | sdague: oh. right | 12:20 |
| mordred | sdague: sorry, still coffeeing - I remember you said that now | 12:20 |
| sdague | yep, no worries | 12:20 |
| jhesketh | AJaeger: and looks like puppetboard is updating now too | 12:20 |
| AJaeger | jhesketh, fungi: confirming ^ | 12:20 |
| sdague | jhesketh: it's running the git fetch | 12:20 |
| AJaeger | so, we're green again from my side ;) | 12:20 |
| sdague | I blasted and restarted again, because I actually need to test patches | 12:21 |
| sdague | yep, hanging again, this time on nova repo | 12:22 |
| sdague | I at least changed the git timeout here, so maybe it will blast and retry | 12:22 |
| sdague | yeh, so setting GIT_TIMEOUT=30 locally is making it retry these network hanges | 12:22 |
| sdague | it defaults to 0 | 12:23 |
| jhesketh | fungi: hmm, this could be a problem: http://paste.openstack.org/show/565142/ | 12:23 |
| sdague | but it's blipping a lot | 12:23 |
| mordred | sdague: quick non-related question ... with https://review.openstack.org/#/c/346282 I get a shade gate failure: http://logs.openstack.org/01/362901/1/check/gate-shade-dsvm-functional-neutron/9698d83/console.html#_2016-08-30_18_56_58_838512 | 12:23 |
| jhesketh | fungi: if we're only running in a 2GB vm we should look to upgrade that | 12:23 |
| mordred | sdague: which is blockign landing the patch we need to get nodepool updated ... I tried this: https://review.openstack.org/#/c/363157/ | 12:24 |
| pabelanger | jhesketh: no, that is a known issue | 12:24 |
| dhellmann | mordred : when you have a sec, I'd like to go ahead with enabling the tagging automation by landing https://review.openstack.org/#/c/363156 | 12:24 |
| *** markvoelker has joined #openstack-infra | 12:24 | |
| pabelanger | jhesketh: we run out of memory when posting logs to puppetboard | 12:24 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses https://review.openstack.org/363545 | 12:24 |
| pabelanger | also, morning | 12:25 |
| mordred | because that's what we originally had a month ago or so ... but it's feeling like there is something I should be setting explicitly in the job config that I'm not | 12:25 |
| jhesketh | pabelanger: oh, okay... we should fix that | 12:25 |
| sdague | mordred: so, honestly, I don't know. I'd see what sc68cal says when he gets up. I'm kind of heads down on placement-api this week so can't really get this into my stack | 12:25 |
| pabelanger | jhesketh: yes, sadly we need to rebuild the server | 12:25 |
| mordred | sdague: k, no problem. mostly asked in case you happened to know the answer off the top of your head | 12:25 |
| *** hashar has quit IRC | 12:25 | |
| fungi | okay, back | 12:26 |
| mordred | dhellmann: okie. I have hit the +A | 12:26 |
| *** hashar has joined #openstack-infra | 12:26 | |
| dhellmann | mordred : thanks! | 12:26 |
| sdague | on that one I was mostly trusting the other folks. But devstack config for neutron is a bit of a whack-a-mole at times, at least off the default path | 12:26 |
| *** gordc has joined #openstack-infra | 12:27 | |
| *** Genek has joined #openstack-infra | 12:27 | |
| *** psilvad has joined #openstack-infra | 12:27 | |
| *** kgiusti has joined #openstack-infra | 12:28 | |
| mordred | sdague: woot | 12:28 |
| fungi | jhesketh: yeah, i seem to recall there were memory leaks/oom issues with ansible someone was trying to work out | 12:28 |
| openstackgerrit | Merged openstack-infra/project-config: to accelerate placement api work, put it in nv check https://review.openstack.org/363613 | 12:28 |
| sdague | mordred: at least we have a working default path now :) | 12:28 |
| sdague | baby steps | 12:28 |
| fungi | oh, nevermind, pabelanger repluied | 12:28 |
| *** markvoelker has quit IRC | 12:29 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:29 | |
| *** nwkarsten has joined #openstack-infra | 12:29 | |
| *** trown|outtypewww is now known as trown | 12:29 | |
| *** Ravikiran_K has quit IRC | 12:29 | |
| *** ddieterly has quit IRC | 12:29 | |
| jhesketh | :-) | 12:30 |
| *** javeriak has quit IRC | 12:31 | |
| fungi | fwiw, when i hacked launch-node to add -vvv on ansible calls, i noticed it transfers an _insane_ amount of data during what looks like fact collection | 12:31 |
| *** rossella_s has joined #openstack-infra | 12:32 | |
| mordred | sdague: that's fantastic! now I just need tofigure out how to test creating a provider network and I'm set :) | 12:32 |
| fungi | i assume it does that every time it runs against a server but normally hides that from its stdout | 12:32 |
| mordred | fungi: it's possible to turn fact collection off | 12:32 |
| sdague | yeh, sane provider network setup wasn't quite in the mix yet. Maybe next cycle. | 12:33 |
| fungi | while having the fact reporting in puppetdb is useful, we collect many orders of magnitude more detail than we really need to | 12:33 |
| openstackgerrit | Merged openstack-infra/project-config: enable release tagging for all repos https://review.openstack.org/363156 | 12:33 |
| mordred | fungi: hrm. we have gather_facts set to true explicitly. I feel like we did that for a reason ... one sec | 12:33 |
| *** openstackgerrit has quit IRC | 12:34 | |
| *** openstackgerrit has joined #openstack-infra | 12:34 | |
| fungi | i bet we could speed up launch-node.py by turning that off in its custom play at least | 12:34 |
| fungi | ooh, i managed to catch gerrit with 36 httpd threads running just now | 12:36 |
| fungi | that's the highest i've spotted yet | 12:36 |
| AJaeger | fungi, team: I did some short analysis on docs.o.o content and put it on https://etherpad.openstack.org/p/CGMHA4ANGZ | 12:36 |
| AJaeger | If you have any questions or anything your want me to look at, please tell me. | 12:37 |
| AJaeger | I'll then talk with Docs team... | 12:37 |
| fungi | AJaeger: one comment on the _sources directories... the "show source" links in the rendered documentation relies on those so it can show you the restructuredtext version of a page | 12:39 |
| *** markusry has joined #openstack-infra | 12:40 | |
| fungi | at least in normal sphinx-type documents | 12:40 |
| paulobanon | anyone has any idea what might be causing this: https://storyboard.openstack.org/#!/story/2000618 | 12:40 |
| dhellmann | fungi, mordred : now that https://review.openstack.org/#/c/363156/ has landed, what's the lag between that version of the script being applied to the signing node? that's a puppet update, right? | 12:41 |
| mordred | dhellmann: yah. so 15-30 minutes ish | 12:42 |
| fungi | paulobanon: yes, jenkins job builder needs permissions to modify jobs | 12:42 |
| dhellmann | mordred : cool, thanks | 12:42 |
| paulobanon | fungi, only works when the jjb apiuser im using has administrator permissions. Even with all Job permissions gives the same error | 12:43 |
| AJaeger | fungi, I know we cannot remove it. But we could not generate it going forward - and then the "show source" will not be there | 12:43 |
| fungi | paulobanon: my guess is that jenkins changed their permission model for the api in 2.7 and 1.6.1 | 12:44 |
| *** rlandy has joined #openstack-infra | 12:44 | |
| *** zhurong has joined #openstack-infra | 12:44 | |
| fungi | paulobanon: we stopped using jenkins here so i haven't been following its development closely, but electrofelix or zxiiro might know | 12:44 |
| paulobanon | fungi, i see, ill try to look on jenkins side of things. Thanks! | 12:45 |
| *** lucas-hungry is now known as lucasagomes | 12:45 | |
| *** ifarkas has quit IRC | 12:46 | |
| rcarrillocruz | mordred: do you have a sec to check http://paste.openstack.org/show/565138/ ? not sure why nova hits v1 on glance.... | 12:46 |
| mordred | rcarrillocruz: I think that's going to be in nova.ini | 12:46 |
| mordred | rcarrillocruz: there will be a glance endpoint configured? | 12:46 |
| *** ifarkas_ is now known as ifarkas | 12:47 | |
| rcarrillocruz | what we have in /etc/nova/nova.conf in the controller is this: | 12:47 |
| rcarrillocruz | [glance] | 12:47 |
| rcarrillocruz | api_servers=https://controller00.vanilla.ic.openstack.org:9292 | 12:47 |
| *** tongli has joined #openstack-infra | 12:47 | |
| rcarrillocruz | should it read /v2 ? | 12:47 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Disable rax-iad due to launch failure rate" https://review.openstack.org/362885 | 12:47 |
| *** tongli has quit IRC | 12:48 | |
| mordred | rcarrillocruz: what version are we useing? | 12:48 |
| *** woodster_ has joined #openstack-infra | 12:48 | |
| *** tongli has joined #openstack-infra | 12:49 | |
| rcarrillocruz | as in the cloud version? mitaka | 12:49 |
| mordred | rcarrillocruz: I think there is not support for v2 in nova in mitaka ... sdague, you added that in newton, right? | 12:50 |
| mordred | rcarrillocruz: so I think we need to tell glance to also run a v1 endpoint | 12:50 |
| sdague | api_servers don't have a version url in them | 12:50 |
| sdague | so https://controller00.vanilla.ic.openstack.org:9292 is right | 12:50 |
| rcarrillocruz | sdague: mind checking http://paste.openstack.org/show/565138/ | 12:51 |
| rcarrillocruz | that's where i show the issue | 12:51 |
| sdague | all the version selection is done behind the scenes | 12:51 |
| rcarrillocruz | long story short: i upload an image, it gets published as v2/uuid | 12:51 |
| rcarrillocruz | but then i boot the server with that image, nova fails cos it tries to find it on v1/uuid on glance | 12:51 |
| mordred | rcarrillocruz: it looks like we are running the v1 endpoint though | 12:51 |
| *** ijw has joined #openstack-infra | 12:52 | |
| rcarrillocruz | mordred: so, then we should instruct nodepool oscc and our manual glance commands to upload images with v1 | 12:52 |
| rcarrillocruz | so nova doesn't complain | 12:52 |
| electrofelix | paulobanon: there were changes in Jenkins 2.0 and newer to require more privileges to get the plugins_info | 12:52 |
| sdague | rcarrillocruz: look at the nova logs | 12:52 |
| rcarrillocruz | let me test that thing | 12:52 |
| andreas_s | Hi fungi, do you know from where the email address is taken that is used for the Summit ATC emails? | 12:52 |
| mordred | rcarrillocruz: it shouldn't matter | 12:52 |
| mordred | rcarrillocruz: it's the same backend set of images ... | 12:52 |
| sdague | rcarrillocruz: Error finding address for | 12:52 |
| sdague | really sounds like DNS resolution fail | 12:52 |
| *** mestery has joined #openstack-infra | 12:52 | |
| sdague | a 404 would probably be a different issue | 12:53 |
| mordred | sdague: that would be running on the nova computes, right? | 12:53 |
| sdague | yes | 12:53 |
| paulobanon | electrofelix, got it thank you | 12:53 |
| fungi | andreas_s: yes, i send them to every e-mail address you have configured in your gerrit account at https://review.openstack.org/#/settings/contact | 12:53 |
| mordred | sdague: tahnks! | 12:53 |
| electrofelix | paulobanon: you can either a) disable getting it at all, or b) provide a file containing what would have been if you had access, that allows for creating a job to be run by a user with more privs than your default one | 12:53 |
| *** markvoelker has joined #openstack-infra | 12:54 | |
| sdague | mordred: it might be run from other nodes as well, but definitely from computes | 12:54 |
| mordred | rcarrillocruz: so we need to make sure that dns resolution is working on the compute nodes | 12:54 |
| electrofelix | paulobanon: and then passing the plugin info to be used by a user with less privileges to only update jobs | 12:54 |
| sdague | I can't remember if api hits it as a precheck | 12:54 |
| rcarrillocruz | root@compute001:/var/log/nova# host controller00.vanilla.ic.openstack.org | 12:54 |
| rcarrillocruz | controller00.vanilla.ic.openstack.org has address 15.184.64.5 | 12:54 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: move tripleo scenario jobs to check pipeline, non-voting https://review.openstack.org/363629 | 12:55 |
| rcarrillocruz | i can run an ansible run with that against all computes, maybe i'm hitting a compute that has resolution hosed | 12:55 |
| andreas_s | fungi, ah ok. So if I change my mail there today, I'll get the ATC pass sent to that new email, also if my commits happened on the old email? | 12:55 |
| *** javeriak has joined #openstack-infra | 12:55 | |
| sdague | rcarrillocruz: yeh, or flip it to the ip addr to see if that makes it go away | 12:55 |
| *** javeriak has quit IRC | 12:55 | |
| sdague | rcarrillocruz: you know which compute failed? | 12:55 |
| sdague | can you get on it's logs and backtrack from there | 12:55 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Install tripleo-admin ssh keys on CI nodes https://review.openstack.org/363630 | 12:56 |
| AJaeger | andreas_s: didn't you get already one? fungi is only sending it out now to those that didn't get one yet. | 12:56 |
| *** jcoufal has joined #openstack-infra | 12:56 | |
| rcarrillocruz | sdague: how do I know? cos from the nova-scheduler i just see messages like: "Successfully synced instances from host 'compute032.vanilla.ic.openstack.org'" | 12:56 |
| rcarrillocruz | i would assume i'd get which compute gets selected on the scheduler | 12:56 |
| rcarrillocruz | so probably a log level tweak is needed? | 12:56 |
| *** ijw has quit IRC | 12:57 | |
| fungi | andreas_s: only if you didn't have any changes merged to official repos between april 7 (mitaka release day) and august 15 (when i sent the most recent batch). otherwise i've already sent one to whatever address(es) you have configured previously. though i can resend it to your updated address now if you tell me your account id number from https://review.openstack.org/#/settings | 12:57 |
| mordred | rcarrillocruz: perhaps ansible all of the hosts with "grep c6bd9eba-bf07-4320-99d8-e407d0d76331 /var/log/nova/*" :) | 12:57 |
| rcarrillocruz | that works too | 12:58 |
| *** YorikSar has quit IRC | 12:58 | |
| *** markvoelker has quit IRC | 12:58 | |
| *** ilyashakhat has joined #openstack-infra | 12:59 | |
| sdague | yeh, if you have distributed logs, what mordred said | 12:59 |
| sdague | ELK would be a ++ as soon as you can get it up | 12:59 |
| *** YorikSar has joined #openstack-infra | 13:00 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Remove unused builder.Builder.update_job method https://review.openstack.org/319752 | 13:00 |
| *** mriedem has joined #openstack-infra | 13:00 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Rename Builder.delete_job to Builder.delete_jobs. https://review.openstack.org/319753 | 13:00 |
| *** pradk has quit IRC | 13:01 | |
| electrofelix | waynr zxiiro: https://review.openstack.org/#/c/319754/15 on merging objects to a JenkinsManager one just a few minor nits, we can tidy up these up subsequently if preferred, main item is a question around what we're testing that can follow up subsequently but not in that patch. let me know whether you want to handle the nits in this patch or follow up | 13:01 |
| *** dizquierdo has joined #openstack-infra | 13:03 | |
| *** gildub has quit IRC | 13:03 | |
| *** markvoelker has joined #openstack-infra | 13:04 | |
| *** spzala has joined #openstack-infra | 13:04 | |
| *** vikrant has quit IRC | 13:05 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/gertty: Use urlparse from six for python 3 compat https://review.openstack.org/363637 | 13:05 |
| *** ddieterly has joined #openstack-infra | 13:05 | |
| *** Julien-zte has joined #openstack-infra | 13:06 | |
| *** amitgandhinz has quit IRC | 13:06 | |
| *** amitgandhinz has joined #openstack-infra | 13:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Run ansible-role-ubuntu-trusty jobs on master https://review.openstack.org/363639 | 13:08 |
| *** ddieterly has quit IRC | 13:09 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add OSA keystone uwsgi functional tests https://review.openstack.org/363640 | 13:09 |
| *** psachin has quit IRC | 13:09 | |
| rcarrillocruz | bang | 13:10 |
| rcarrillocruz | 2016-08-31 11:49:09.220 16435 ERROR nova.image.glance CommunicationError: Error finding address for https://controller00.vanilla.ic.openstack.org:9292/v1/images/b37fd797-f863-434d-ab9e-4d27557432f5: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed | 13:10 |
| rcarrillocruz | sdague , mordred , thx for the help, something with ssl | 13:10 |
| *** markvoelker has quit IRC | 13:10 | |
| AJaeger | fungi, mail send out to docs list to suggest removal of 887 MB from docs.openstack.org. | 13:10 |
| sdague | rcarrillocruz: nice | 13:11 |
| mordred | rcarrillocruz: woot! and we learn things | 13:11 |
| fungi | AJaeger: awesome--thanks for going through all that | 13:11 |
| *** fguillot has joined #openstack-infra | 13:11 | |
| rcarrillocruz | so yeah | 13:11 |
| rcarrillocruz | there are couple things | 13:11 |
| rcarrillocruz | that we have no puppetized | 13:11 |
| rcarrillocruz | like | 13:11 |
| rcarrillocruz | you know | 13:11 |
| mordred | certs? | 13:11 |
| rcarrillocruz | trustying our self-signed certificates | 13:11 |
| *** raildo has joined #openstack-infra | 13:11 | |
| rcarrillocruz | :D | 13:11 |
| mordred | :) | 13:11 |
| rcarrillocruz | when i first crreated an instance last week | 13:12 |
| mordred | rcarrillocruz: I have this hunch that the next thing to puppet is trusting our self-signed certs | 13:12 |
| rcarrillocruz | it worked | 13:12 |
| sdague | heh | 13:12 |
| andreas_s | AJaeger, fungi, ok, got it thanks. I have mine already - just was wondering if that works out without larger impacts - and ATC was one of them I had in mind thanks! | 13:12 |
| rcarrillocruz | cos i trusted the cert on the couple machines i provisioned | 13:12 |
| rcarrillocruz | but i have not with the forty something i provisioned afterwards | 13:12 |
| pabelanger | sdague: fungi: Did we figure out why git clones were hanging this morning? Or is that still an ongoing issue | 13:12 |
| AJaeger | wznoinsk: I can clone the repo now - hope everything works for you as well | 13:13 |
| wznoinsk | looks good, thanks | 13:14 |
| fungi | andreas_s: yeah, handling address changes is tricky. basically when i send updated batches of invites i diff against the previous list by gerrit account id number. that should _normally_ be enough but there are cases where people accidentally create and begin using new (duplicate) gerrit accounts or we combine duplicate accounts... the list generator tries to find and deduplicate them where possible | 13:14 |
| fungi | but there's not always enough overlap in data to find them all | 13:14 |
| *** mtanino has joined #openstack-infra | 13:15 | |
| fungi | andreas_s: it's further complicated by additions/removals from the extra-atcs lists in governance, since someone can be both an extra-atc and a contributor (perhaps to a different team than they're an extra-atc on), so identifying duplication and changes in that duplication over the course of a cycle between batches is especially complicated | 13:15 |
| *** sdake_ has joined #openstack-infra | 13:16 | |
| andreas_s | fungi, that seems to be a really complex process :D | 13:16 |
| andreas_s | fungi, thanks for the insights! | 13:17 |
| *** hewbrocca has joined #openstack-infra | 13:17 | |
| fungi | andreas_s: i'm hoping with the coming project-team gatherings we'll be able to just make admission free (or trivially inexpensive) for anyone who wants to attend, and then use attendance to one of more ptgs over a reasonably lengthy period of time to get discounted/free access to the summits. so hopefully this becomes much simpler in the future | 13:18 |
| *** rhallisey has quit IRC | 13:18 | |
| *** mdrabe has joined #openstack-infra | 13:18 | |
| *** zz_dimtruck is now known as dimtruck | 13:18 | |
| *** sdake has quit IRC | 13:19 | |
| *** esberglu has joined #openstack-infra | 13:20 | |
| *** yamamoto_ has quit IRC | 13:20 | |
| *** ilyashakhat has quit IRC | 13:20 | |
| mat128 | Hi group, I'm getting IPv6 links for running jobs in the zuul status page | 13:21 |
| mat128 | if, for example you search for "363294" | 13:21 |
| mat128 | "telnet://2001:4800:1ae1:18:f816:3eff:fe6e:1042:19885" | 13:21 |
| pabelanger | fungi: sdague: it looks like tripleo-test-cloud-rh1 can reproduce the git clone hanging failures this morning. All of their devstack jobs are timing out after 3 hours | 13:21 |
| AJaeger | mat128: YEs, that's correct. Welcome to the 21st century ;) | 13:22 |
| pabelanger | mat128: yes, we have an IPv6 only cloud, osic-cloud1 | 13:22 |
| mat128 | oh wow :) | 13:22 |
| mat128 | now I have a real reason to ask my provider for ipv6.. | 13:22 |
| *** sdake_ is now known as sdake | 13:22 | |
| mat128 | lol | 13:22 |
| sdague | would be really great to get that web console proxy.... :) | 13:23 |
| *** coolsvap is now known as _coolsvap_ | 13:23 | |
| fungi | mat128: i've been having lots of luck with a free v6 tunnel from hurricane electric (tunnelbroker.net) for many years now | 13:24 |
| fungi | mat128: but yes, just in case there are still isps out there who claim there's no rush to implement ipv6 because nobody actually has v6-only content... well be happy to use us as an example! ;) | 13:25 |
| *** Goneri has joined #openstack-infra | 13:27 | |
| rtheis | not sure if this is the correct channel for grafana dashboard questions ... | 13:28 |
| *** flepied has quit IRC | 13:28 | |
| rtheis | networking-ovn dashboard has several issues that I'm not sure how to resolve: http://grafana.openstack.org/dashboard/db/networking-ovn-failure-rate | 13:28 |
| mat128 | fungi: thanks, I'll try that | 13:29 |
| *** zul has quit IRC | 13:30 | |
| rtheis | The "Unit Test Failure Rate" graph has no data. And while the job names look good in project config, I'm not finding success or fail data posted to graphite | 13:30 |
| *** rossella_s has quit IRC | 13:31 | |
| *** jamesdenton has joined #openstack-infra | 13:31 | |
| zxiiro | electrofelix: cool, I'll leave that to waynr since it's his patch. Let me fix the merge conflict in the last patch listed though since it doesn't depend on anything | 13:32 |
| *** cardeois has joined #openstack-infra | 13:33 | |
| *** zul has joined #openstack-infra | 13:33 | |
| *** ansiwen has joined #openstack-infra | 13:34 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Simplify delete by removing unnecessary loop https://review.openstack.org/357990 | 13:35 |
| EmilienM | hello infra! can I have a review on https://review.openstack.org/#/c/363629/ please? | 13:35 |
| *** hongbin has joined #openstack-infra | 13:36 | |
| *** matt-borland has joined #openstack-infra | 13:36 | |
| fungi | rtheis: when was that graph added, and how often do changes get approved for that repo? do you also have rules to skip unit tests for certain kinds of patches (for example those only modifying documentation files)? | 13:37 |
| *** mtanino has quit IRC | 13:37 | |
| rtheis | fungi: the graph has been around for at least a few weeks | 13:38 |
| zxiiro | electrofelix: fyi I added the my patch to fix the disabled always returning true patch to the v2 list since I think it's important to merge that | 13:38 |
| rtheis | we have changes getting approved usually on a daily basis | 13:38 |
| fungi | rtheis: i wonder if none of them have failed a unit test job in the gate pipeline since the graph was added | 13:39 |
| rtheis | I believe there are unit test skips for doc-only changes | 13:39 |
| *** ddieterly has joined #openstack-infra | 13:39 | |
| zxiiro | electrofelix: can you review the patches after the one you left a comment on just in case if it's a trivial rebase we can merge it when you're gone? | 13:39 |
| rtheis | fungi: that may be possible | 13:39 |
| waynr | electrofelix zxiiro: i'll probably try to address the nits in this patch, not sure until i have time to take a closer look though...did you see my comments the other day about renaming jenkins_jobs.builder to jenkins_jobs.manager? any thoughts on that? | 13:39 |
| rtheis | but the check queue certainly had failures and the py34 and py35 data isn't shown | 13:40 |
| AJaeger | rtheis: create a change that fails and let's see whether it shows up;) | 13:40 |
| zxiiro | electrofelix: waynr with that said I think we're very close to having this all done. Just need to figure out if we still need the 3 patches that are in merge conflict at the bottom https://review.openstack.org/#/q/status:open+project:openstack-infra/jenkins-job-builder+branch:master+topic:jjb-2.0.0-api | 13:40 |
| *** Guest81 has joined #openstack-infra | 13:40 | |
| rtheis | AJaeger: I'll see if we have one already | 13:41 |
| AJaeger | rtheis: but indeed, this looks odd... | 13:41 |
| rtheis | AJaeger: here is one https://review.openstack.org/#/c/362494/ | 13:41 |
| sambetts | Hi infra, where can I find the rules/regs for whether a project is allowed to publish docs to docs.openstack.org/developer ?? | 13:41 |
| dhellmann | I have an unexpected job failure for pypi-both-upload on the monasca-events-api repo. The ACLs look like they use the default, and I don't think the release team tagged anything. Could the job have been triggered by a repo import? http://logs.openstack.org/00/004f1a23226101c6d4349a74462899f59c08dd93/release/monasca-events-api-pypi-both-upload/69f2304/console.html | 13:41 |
| *** isaacb has joined #openstack-infra | 13:42 | |
| AJaeger | sambetts: any project in the big tent can. See also infra manual Creator'S guide | 13:42 |
| fungi | AJaeger: rtheis: getting a failing change to show up on the gate pipeline graph would be hard since it would need to succeed in check to get enqueued | 13:42 |
| AJaeger | fungi, oh, yes. Silly me... | 13:42 |
| AJaeger | sambetts: which repo do you care about? | 13:43 |
| sambetts | AJaeger: thanks, I'm trying to work out if networking-cisco is still allowed to publish docs, I vaguly remember something about not being allowed too | 13:43 |
| fungi | usually the way you get hits in the gate pipeline failures is by having nondeterministic failures | 13:43 |
| rtheis | Another oddity is that is the "Integrated Failure Rates" still shows an old job name | 13:43 |
| AJaeger | sambetts: it'S not in governance/reference/projects.yaml | 13:43 |
| rtheis | gate-tempest-dsvm-networking-ovn-native-l3 instead of gate-tempest-dsvm-networking-ovn-native-services | 13:44 |
| AJaeger | sambetts: so, should not publish on docs.o.o | 13:44 |
| *** sai has joined #openstack-infra | 13:44 | |
| *** xyang1 has joined #openstack-infra | 13:44 | |
| *** Guest81 has quit IRC | 13:44 | |
| pabelanger | fungi: sdague: Wonder if adding http://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=d53ad0b07d3e7bdd2668c2d3f1815d95d4b8f532 to devstack-gate could help with network issue for git clones | 13:45 |
| fungi | rtheis: if you rename a job in the ci system, you'd also need to adjust the graphite counter names in your grafana dashboards to match | 13:45 |
| rtheis | fungi: we did | 13:45 |
| sambetts | AJaeger: so I'm still a little confused where we stand now, because we were kicked out of the neutron umbrella thing, but I thougth that meant we were still a big tent project | 13:45 |
| jkilpatr | pabelanger, is there anyway to separate the pep8 and ansible linting jobs for openstack/browbeat? | 13:46 |
| rtheis | yet the old name still shows in the graphs | 13:46 |
| *** amotoki has quit IRC | 13:46 | |
| rtheis | *graph | 13:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/363669 | 13:46 |
| sdague | pabelanger: I'm not running devstack-gate | 13:46 |
| sdague | pabelanger: this is running devstack, locally | 13:46 |
| fungi | rtheis: oh, so the grafana dashboard isn't updated to what the configuration in networking-ovn.yaml says it should be... interesting | 13:46 |
| sdague | and we're hanging today, a lot | 13:47 |
| pabelanger | jkilpatr: yes it is possible, you'd create a new job breaking them apart. However, we're trying to use the linters / pep8 target to group them together | 13:47 |
| AJaeger | sambetts: https://review.openstack.org/303026 removed networking-cisco from the Big Tent. Removal from the Stadium included removal from Big Tent | 13:47 |
| rtheis | fungi: yes, that appears to be the case | 13:47 |
| pabelanger | sdague: right, it is 100% reproducible with tripleo jobs right now | 13:47 |
| sdague | pabelanger: which indicates that it's going to be an issue for lots of folks | 13:47 |
| pabelanger | yes | 13:48 |
| *** njohnston has left #openstack-infra | 13:48 | |
| jkilpatr | pabelanger, The issue is we're likely to fail ansible linters for a while since we want to actually refactor whats failing instead of bandaid. But this means python errors don't get -1'd | 13:48 |
| rtheis | fungi: although I did notice that the latest "Periodic jobs" graph addition did show up | 13:48 |
| AJaeger | infra-root, setup of https://github.com/openstack/k8s-docker-suite-app-murano did not work. The repo was approved earlier today but is not mirrored. Could somebody fix this, please? | 13:48 |
| sambetts | AJaeger: Can a project like networking-cisco still become part of the big tent? Do I need to reapply? | 13:49 |
| rtheis | fungi: it is almost like the existing graphs which were update in the configuration aren't being used in dashboard ... only new graphs | 13:49 |
| fungi | dhellmann: https://pypi.org/project/monasca-events-api/#history shows a 0.0.5 version uploaded more than a year ago, and a 0.0.6 uploaded 10 months ago | 13:49 |
| AJaeger | sambetts: I'm not in the TC. But according to my understanding and the messaging I remember: Yes. | 13:49 |
| fungi | rtheis: pabelanger may have some more insights once he | 13:49 |
| fungi | 's freed up | 13:49 |
| dhellmann | fungi : yeah | 13:49 |
| pabelanger | jkilpatr: Yup, understood | 13:49 |
| rtheis | thanks | 13:49 |
| fungi | he's our grafana specialist and the grafyaml author | 13:50 |
| AJaeger | sambetts: best talk with a TC member for a more educated answer | 13:50 |
| sambetts | AJaeger: Ok thanks! I'll go and do some more digging | 13:50 |
| AJaeger | sambetts: there are conditions for a project to be part of big tent, check whether you meet them. | 13:50 |
| *** dtantsur|bbl is now known as dtantsur | 13:51 | |
| sambetts | AJaeger: We've been working to maintain our OpenStackness because we assumed we were still a part of it, just not in the neutron thingy anymore, so I hope we still meet them | 13:51 |
| fungi | dhellmann: they very well may have manually uploaded early releases to pypi | 13:51 |
| dhellmann | fungi : yeah. I'm trying to understand what triggered a job to run today or yesterday, though | 13:52 |
| fungi | dhellmann: oh! looking | 13:52 |
| *** rossella_s has joined #openstack-infra | 13:52 | |
| AJaeger | fungi, the repo was imported today | 13:52 |
| dhellmann | ok, I thought that might be the case because it seemed new | 13:52 |
| fungi | pabelanger: sdague: zuul-cloner added timeout and retry options, so maybe devstack-gate just isn't using them (yet)? | 13:52 |
| dhellmann | fungi, AJaeger: it's interesting that importing a repo triggers the jobs like that | 13:52 |
| *** ekhugen has quit IRC | 13:53 | |
| dhellmann | hmm | 13:53 |
| AJaeger | fungi, dhellmann https://review.openstack.org/362462 merged 5 hours ago | 13:53 |
| *** amotoki has joined #openstack-infra | 13:53 | |
| dhellmann | fungi, mordred : it doesn't look like the signing node has been updated with the new version of the release script (it still has the skip in) | 13:53 |
| jkilpatr | pabelanger, need me to do anything to make the job? | 13:53 |
| pabelanger | fungi: Okay, digging into that | 13:53 |
| *** esp has quit IRC | 13:54 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 13:54 |
| sdague | fungi: ... it's not devstack-gate | 13:54 |
| *** eharney has joined #openstack-infra | 13:54 | |
| fungi | sdague: oh, i was responding to pabelanger's suggestion of adding git timeout support to devstack-gate | 13:54 |
| pabelanger | jkilpatr: sorry, distracted with another issue. What is the issue you are trying to solve? | 13:54 |
| fungi | too many conversations at once for me today | 13:55 |
| jkilpatr | pabelanger, different jobs in CI for pep8 and ansible-linters, so that one can be voting and the other won't be for now | 13:55 |
| *** jamielennox|away is now known as jamielennox | 13:55 | |
| pabelanger | jkilpatr: I'd just comment out the code in tox.ini for now, make the existing job voting, then once you are ready, uncomment the tox change | 13:56 |
| jkilpatr | pabelanger, ok then, will do. Thanks for the help | 13:56 |
| *** psachin has joined #openstack-infra | 13:57 | |
| *** ekhugen has joined #openstack-infra | 13:57 | |
| *** pgadiya has quit IRC | 13:57 | |
| waynr | zxiiro electrofelix: I would like to get https://review.openstack.org/351743 and https://review.openstack.org/333076 into 2.0.0 API | 13:58 |
| *** tongli_ has joined #openstack-infra | 13:58 | |
| fungi | dhellmann: so my guess is the reasons you don't normally see that happen: 1. projects are often imported into gerrit before they add release jobs; 2. most repos that get imported don't already have tags in them | 13:58 |
| fungi | dhellmann: but yeah, we "import" projects with a scripted push of the content from an existing repository somewhere, and gerrit emits tag-related events when those tags are pushed in | 13:59 |
| zxiiro | waynr: ok cool, we just need to fix the merge conflicts, I might be able to help with that | 13:59 |
| dhellmann | fungi : those reasons make sense. I'm not too worried, since nothing is actually broken, but I wanted to make sure we understood what was going on. | 13:59 |
| *** zshuo has quit IRC | 14:00 | |
| *** links has quit IRC | 14:00 | |
| dhellmann | fungi : I have to step away. When you have a few minutes, could you look at puppet on the signing node to make sure it's running ok? we merged the change to enable release automation everywhere, but that version of the script doesn't seem to have been pushed out yet. The patch landed ~90 minutes ago. https://review.openstack.org/#/c/363156/ | 14:01 |
| *** tongli has quit IRC | 14:01 | |
| fungi | dhellmann: earlier today puppet went several hours without updating servers, so we may have some issues with our configuration management getting applied in a timely manner | 14:01 |
| *** jamielennox is now known as jamielennox|away | 14:01 | |
| pabelanger | fungi: do you have a link handy for the zuul-cloner timeout / retry you mentioned? I don't see anything in the documentation. | 14:02 |
| dhellmann | fungi : ah! ok. | 14:02 |
| *** wgd3 has joined #openstack-infra | 14:02 | |
| *** jamielennox|away is now known as jamielennox | 14:02 | |
| *** pradk has joined #openstack-infra | 14:02 | |
| *** rfolco has quit IRC | 14:03 | |
| *** zhurong has quit IRC | 14:03 | |
| *** zhurong has joined #openstack-infra | 14:03 | |
| *** rfolco has joined #openstack-infra | 14:04 | |
| AJaeger | pabelanger, fungi: and jobs don't use it either if my grep in jenkins/jobs is correct | 14:04 |
| *** dimtruck is now known as zz_dimtruck | 14:04 | |
| fungi | i'm reading teh zuul.lib.cloner module now to see where that is | 14:05 |
| fungi | maybe the change i'm remembering never merged? | 14:05 |
| *** cdent_ has joined #openstack-infra | 14:05 | |
| mordred | dhellmann: sorry - was writing a long email to someone ... | 14:05 |
| mordred | thanks fungi for responding | 14:06 |
| *** sdake_ has joined #openstack-infra | 14:06 | |
| fungi | pabelanger: AJaeger: aha, 282099 and 282102 are what i was thinking of | 14:06 |
| fungi | not yet merged | 14:06 |
| *** akshai has joined #openstack-infra | 14:07 | |
| *** cardeois_ has joined #openstack-infra | 14:07 | |
| *** cardeois has quit IRC | 14:07 | |
| *** yamamoto has joined #openstack-infra | 14:07 | |
| *** cdent_ has quit IRC | 14:07 | |
| *** cdent has joined #openstack-infra | 14:07 | |
| *** sdake has quit IRC | 14:07 | |
| AJaeger | ;( | 14:08 |
| *** tonytan4ever has quit IRC | 14:09 | |
| *** matt-borland has quit IRC | 14:09 | |
| *** tonytan4ever has joined #openstack-infra | 14:09 | |
| *** egarbade_ has joined #openstack-infra | 14:09 | |
| * sc68cal connects | 14:10 | |
| sc68cal | mordred: looking | 14:10 |
| mordred | sc68cal: yay! thanks | 14:11 |
| *** rfolco has quit IRC | 14:11 | |
| *** cardeois_ is now known as cardeois | 14:11 | |
| sc68cal | mordred: I have to do a little research, but basically we create a physical network named public, and I don't think more than one network can use the same physical net | 14:11 |
| *** eranrom has joined #openstack-infra | 14:11 | |
| *** spzala has quit IRC | 14:12 | |
| *** rfolco has joined #openstack-infra | 14:12 | |
| *** spzala has joined #openstack-infra | 14:12 | |
| *** rbrndt has joined #openstack-infra | 14:12 | |
| sc68cal | mordred: http://logs.openstack.org/01/362901/1/check/gate-shade-dsvm-functional-neutron/9698d83/logs/etc/neutron/plugins/ml2/ml2_conf.ini.txt.gz | 14:13 |
| sc68cal | scroll to the bottom | 14:13 |
| mordred | sc68cal: ah - so, that's the thing that creates the public physical network | 14:13 |
| *** matbu is now known as matbu|mtg | 14:14 | |
| *** rhallisey has joined #openstack-infra | 14:14 | |
| sc68cal | mordred: DevStack now creates a public network and uses the "public" physical network | 14:14 |
| sc68cal | .... we may need to rename it. | 14:14 |
| sc68cal | but basically we create a neutron network, that then maps it to the br-ex device | 14:14 |
| *** eventingmonkey has joined #openstack-infra | 14:14 | |
| * sc68cal digs for it | 14:14 | |
| *** sandanar has quit IRC | 14:15 | |
| sc68cal | mordred: http://logs.openstack.org/01/362901/1/check/gate-shade-dsvm-functional-neutron/9698d83/logs/devstacklog.txt.gz#_2016-08-30_18_46_38_671 | 14:15 |
| mordred | ah. | 14:15 |
| *** abregman has quit IRC | 14:16 | |
| mordred | so - our test is trying to do that, but it's failing because devstack has already done that | 14:16 |
| sc68cal | I believe so | 14:16 |
| *** amotoki has quit IRC | 14:16 | |
| fungi | mordred: did /var/log/ansible.log replace /var/log/run_all.log on puppetmaster.o.o? | 14:16 |
| sc68cal | it's a recent development, thanks to kevinbenton 's https://review.openstack.org/346282 | 14:16 |
| *** tpsilva has joined #openstack-infra | 14:16 | |
| *** spzala has quit IRC | 14:17 | |
| sc68cal | mordred: The actual first commit - https://review.openstack.org/#/c/343072/ - details | 14:17 |
| pabelanger | fungi: Ah, thanks. | 14:17 |
| sc68cal | mordred: we had to revert the revert a couple times :) | 14:17 |
| mordred | sc68cal: gotcha. so, it's a great thing generally, but may mean that this particular test won't work on devstack anymore, since the base devstack setup has already done that thing | 14:17 |
| *** Guest81 has joined #openstack-infra | 14:17 | |
| mordred | fungi: maybe? I dont remember thathappening | 14:17 |
| jroll | sc68cal: the latest revert of the revert broke ironic CI, fwiw. we're working on fixing but would you prefer to revert in the meantime? | 14:17 |
| sc68cal | mordred: yeah just put some logic in to check for a public network that has the physnet set | 14:17 |
| mordred | sc68cal: cool. and skip-if that is true I guess | 14:18 |
| sc68cal | jroll: The issue is we've deprecated the way we were doing it before. Like, setting external_network_bridge=br-ex for the l3 agent will soon not work | 14:18 |
| *** tonytan4ever has quit IRC | 14:18 | |
| sc68cal | it's going bye-bye | 14:18 |
| sc68cal | per https://review.openstack.org/#/c/343072/ | 14:19 |
| sc68cal | I should write something on the ML | 14:19 |
| fungi | mordred: nevermind, it was puppet_run_all.log (i was spacing on correct the name) | 14:19 |
| mordred | fungi: phew | 14:19 |
| *** esp has joined #openstack-infra | 14:19 | |
| *** tonytan4ever has joined #openstack-infra | 14:20 | |
| jroll | sc68cal: I don't know how all of this works, but I do hope we can find something workable for ironic before we make that not work | 14:20 |
| fungi | it looks like maybe puppeting all the infracloud nodes has significantly increased the time to complete our update loops | 14:20 |
| fungi | to the point where it's taking a couple hours to complete now | 14:20 |
| *** mtanino has joined #openstack-infra | 14:20 | |
| anteaya | fungi: :( | 14:20 |
| mordred | fungi: oh. perhaps we shold put infracloud puppet on a different loop | 14:20 |
| mordred | fungi: so that infracloud puppet does not block infra puppet? | 14:21 |
| fungi | some of this could be reachability issues with some of the infra-cloud nodes | 14:21 |
| mordred | yah | 14:21 |
| fungi | so it may be waiting for ssh timeouts to kick in | 14:21 |
| mordred | also - perhaps increasing our parallelism is warranted | 14:21 |
| fungi | i'm working out the phase timing now | 14:22 |
| *** psachin has quit IRC | 14:22 | |
| fungi | looks like we do the git servers first and that currently takes ~3 minutes | 14:22 |
| AJaeger | bbl | 14:22 |
| jroll | sc68cal: for now, we're setting Q_USE_PROVIDERNET_FOR_PUBLIC=False to get us back up and running, still waiting for CI on that though | 14:23 |
| fungi | oh, looking at the tun that started at 12:30 utc, the review.o.o phase took almost half an hour on its own | 14:23 |
| fungi | s/tun/run | 14:23 |
| sc68cal | jroll: OK. hope that unblocks you. Again sorry for the surprise.... | 14:24 |
| jroll | sc68cal: yeah, I'd just like some help figuring out the future here (probably after freeze) | 14:24 |
| mordred | sc68cal: oh - if I do Q_USE_PROVIDERNET_FOR_PUBLIC=False ... that would not use the physical network for the public network? | 14:24 |
| fungi | after review.o.o we take another ~3 minutes to do other nodes, then we start on the infracloud nodes | 14:24 |
| *** zz_dimtruck is now known as dimtruck | 14:25 | |
| sc68cal | mordred: yes that should have that effect | 14:25 |
| fungi | then ansible times out for some 15+ minutes on "fatal: [controller00.vanilla.ic.openstack.org]: UNREACHABLE!" | 14:25 |
| mordred | sc68cal: ok. cool. I might add a job then that has that set ... so that I'll skip trying to create a provider net in one of the jobs, but create one in the other job | 14:25 |
| sc68cal | mordred: k | 14:26 |
| *** Guest81 has quit IRC | 14:26 | |
| *** amotoki has joined #openstack-infra | 14:26 | |
| fungi | and it got a network disconnect after some 10+ minutes writing to compute037.vanilla.ic.openstack.org | 14:27 |
| fungi | another disconnect on compute040.vanilla.ic.openstack.org | 14:27 |
| *** rajinir has joined #openstack-infra | 14:28 | |
| *** spzala has joined #openstack-infra | 14:28 | |
| fungi | so basically on the curent pulse it started working through infracloud nodes at ~1300 utc and is still going 1.5 hours later | 14:28 |
| fungi | current | 14:28 |
| fungi | rcarrillocruz: ^ ideas? this is pretty crippling at the moment | 14:29 |
| *** yamamoto has quit IRC | 14:29 | |
| mordred | sc68cal: https://review.openstack.org/363715 I think this should do it ... thnak you for your help! | 14:29 |
| *** yamamoto has joined #openstack-infra | 14:29 | |
| fungi | rcarrillocruz: can we temporarily disable ansible for infra-cloud until it can be switched to a separate loop | 14:29 |
| fungi | ? | 14:29 |
| mordred | fungi: ++ | 14:29 |
| rcarrillocruz | unreachable controller00 ? | 14:30 |
| rcarrillocruz | i'm in now | 14:30 |
| rcarrillocruz | let me check the others | 14:30 |
| pabelanger | fungi: So, I was thinking the other day, we should consider moving infracloud ansible wheel to a new server. As we don't affect our control plane for the project. This also gives starts our migration away from puppetmaster.o.o | 14:30 |
| *** tonytan4ever has quit IRC | 14:30 | |
| rcarrillocruz | ok, so my ansible play run against infracloud just finished | 14:31 |
| rcarrillocruz | i get as unrechable: | 14:31 |
| rcarrillocruz | compute016 | 14:31 |
| *** abregman has joined #openstack-infra | 14:31 | |
| rcarrillocruz | compute005 | 14:31 |
| *** tonytan4ever has joined #openstack-infra | 14:31 | |
| fungi | rcarrillocruz: well, ansible timed out trying to reach it from puppetmaster. no idea if that means it's a network problem or an ansible problem. regardless the sheer number of nodes we're updating in infra-cloud coupled with the complexity of what's being done on them plus possible network problems there causes it to dwarf the time needed to run against our typical servers | 14:31 |
| rcarrillocruz | and compute043 | 14:31 |
| fungi | rcarrillocruz: trace through /var/log/puppet_run_all.log and look at timestamps | 14:32 |
| *** akshai has quit IRC | 14:32 | |
| *** ZZelle has joined #openstack-infra | 14:32 | |
| rcarrillocruz | so, how's the usual pattern, put servers with problems in emergency group, yeah? | 14:33 |
| sc68cal | mordred: no problem..... sorry it kind of is after the fact rather than warning people in advance | 14:33 |
| sc68cal | jroll: ^ | 14:33 |
| *** nmagnezi has quit IRC | 14:33 | |
| *** akshai has joined #openstack-infra | 14:33 | |
| rcarrillocruz | so that doesn't run by the run_all.sh playbooks | 14:33 |
| fungi | rcarrillocruz: yeah, though if they're all already in a group maybe the group itself can be disabled? | 14:33 |
| rcarrillocruz | well, i believe putting the whole group is a bit too much, | 14:34 |
| fungi | i am admittedly fuzzy on how ansible groups other than the disabled group play into disabling | 14:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Run puppet on infracloud in a different cron https://review.openstack.org/363719 | 14:34 |
| mordred | rcarrillocruz, fungi: ^^ | 14:34 |
| rcarrillocruz | some failures are legit, cos they are not completely configured , provisioned | 14:34 |
| amrith | is there a way to get to CI gate pipeline logs before the jobs are done? I was too late to grab the output with netcat from the job but it failed and I'd like to see the output if possible. commit 5b3f953b0d858f62a469a115fb5aa345333a9fd1, job output telnet://104.130.73.19:19885 | 14:34 |
| fungi | rcarrillocruz: sure, but even the successful runs against those servers seem to be... slow | 14:34 |
| fungi | compared to our typical virtual machines anyway | 14:35 |
| rcarrillocruz | mordred: good, that'll give me breathing time till i pinpoint the individual offending ones | 14:35 |
| mordred | rcarrillocruz: ++ | 14:35 |
| *** amotoki has quit IRC | 14:35 | |
| fungi | and it seems fine if we have a different update frequency for infra-cloud nodes vs our virtual servers | 14:36 |
| *** oanson has quit IRC | 14:36 | |
| *** ansiwen has quit IRC | 14:36 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Run puppet on infracloud in a different cron https://review.openstack.org/363719 | 14:37 |
| *** david-lyle_ has joined #openstack-infra | 14:37 | |
| fungi | okay, so i've confirmed the delay this run on review.o.o is that Exec[manage_projects] took nearly half an hour to return | 14:37 |
| *** ansiwen has joined #openstack-infra | 14:37 | |
| *** salv-orlando has quit IRC | 14:38 | |
| rcarrillocruz | mordred: how about logging the ansible puppet runs to a different log file? | 14:38 |
| pabelanger | mordred: fungi: rcarrillocruz: Any chance for reviews on https://review.openstack.org/#/c/356702/ https://review.openstack.org/#/c/356703 ? osic-cloud8 is ready and want to get the credentials onto puppetmaster.o.o | 14:38 |
| *** ddieterly is now known as ddieterly[away] | 14:38 | |
| *** senk has quit IRC | 14:39 | |
| mordred | rcarrillocruz: sure? | 14:39 |
| *** ansiwen has quit IRC | 14:39 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Run puppet on infracloud in a different cron https://review.openstack.org/363719 | 14:39 |
| *** ansiwen has joined #openstack-infra | 14:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 14:40 |
| *** ddieterly[away] is now known as ddieterly | 14:40 | |
| openstackgerrit | Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950 | 14:40 |
| rcarrillocruz | mordred: ^ | 14:40 |
| rcarrillocruz | i'm cool landing as is | 14:40 |
| rcarrillocruz | fungi, pabelanger ^ | 14:40 |
| *** kushal has joined #openstack-infra | 14:41 | |
| rcarrillocruz | pabelanger: i just created the mirror on the infracloud and put the DNS | 14:41 |
| pabelanger | rcarrillocruz: isn't there going to be issues which update_puppet.yaml playbook? | 14:41 |
| rcarrillocruz | will add the node on site.pp shortly | 14:41 |
| rcarrillocruz | let me review your changes | 14:41 |
| pabelanger | that new crontab will not update puppet bit locally | 14:41 |
| pabelanger | so there could be some sync issues | 14:41 |
| mordred | yah. I didn't have a good answer for that | 14:42 |
| mordred | but figured for now infracloud having eventual consistency was likely fine | 14:42 |
| pabelanger | okay | 14:42 |
| *** eventingmonkey has quit IRC | 14:42 | |
| rcarrillocruz | we can always put an additional update_puppet on the run_all_infracloud afterwards, let's see howw it goes for now | 14:42 |
| fungi | lgtm, +3 | 14:43 |
| *** tonytan4ever has quit IRC | 14:43 | |
| fungi | thanks! | 14:43 |
| *** wgd3 is now known as wgd3[away] | 14:43 | |
| *** tonytan4ever has joined #openstack-infra | 14:43 | |
| fungi | so digging further in the review.o.o logs, manage_projects was correctly triggered due to a change for k8s-docker-suite-app-murano | 14:44 |
| *** ansiwen has quit IRC | 14:44 | |
| *** eventingmonkey has joined #openstack-infra | 14:44 | |
| *** ansiwen has joined #openstack-infra | 14:45 | |
| fungi | i wonder if the script simply needs some refactoring to handle the scale of repos we've grown to | 14:45 |
| rcarrillocruz | question: the emergency hosts group, is that backed in repo or just a file on the puppetmaster | 14:45 |
| rcarrillocruz | ? | 14:45 |
| rcarrillocruz | version controlled i mean | 14:45 |
| *** mdrabe has quit IRC | 14:46 | |
| fungi | rcarrillocruz: there are two. the emergency one is not in any git repo, but there is also a disabled group in the groups file in system-config | 14:46 |
| fungi | for longer-term disablement, we should use the file in system-config | 14:46 |
| openstackgerrit | Bartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029 | 14:46 |
| rcarrillocruz | ah sweet | 14:46 |
| *** sputnik13 has joined #openstack-infra | 14:46 | |
| rcarrillocruz | i'll put the 3 servers i cannot provision them on the long-term file then, thx | 14:46 |
| *** amotoki has joined #openstack-infra | 14:48 | |
| *** amitgandhinz has quit IRC | 14:48 | |
| *** amitgandhinz has joined #openstack-infra | 14:49 | |
| *** tonytan_brb has joined #openstack-infra | 14:49 | |
| skraynev | AJaeger: fungi: Sorry for the interruption. I just worry about patch https://review.openstack.org/#/c/357745/, I see ,that it was finally merged. And I can see a new repository, but it looks like provided info of existing repo was not used. May be I did something wrong or need to wait a bit more time? | 14:49 |
| *** eventingmonkey has quit IRC | 14:50 | |
| *** tonytan4ever has quit IRC | 14:50 | |
| jroll | sc68cal: no worries, it happens | 14:52 |
| *** Swami has joined #openstack-infra | 14:52 | |
| fungi | so looking at the manage-projects debug log, for every single repo we have in projects.yaml it's doing 11 git calls and a file copy (at least as far as what it logs that it's doing) seemingly in an attempt to see whether re-pushing the acl into gerrit will result in an update? that definitely seems excessive when the update to project-config only touched one repo | 14:52 |
| *** markvoelker has joined #openstack-infra | 14:52 | |
| skraynev | AJaeger: fungi: oh Jesus. I missed ".git" | 14:52 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add non-voting governance tag validation job https://review.openstack.org/363165 | 14:52 |
| vrovachev1 | andreas_s: I got +1 from fuel PTL for patch https://review.openstack.org/#/c/359704/ | 14:53 |
| andreas_s | vrovachev1, you probably meant AJaeger :) | 14:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Skip test creating provider network if one exists https://review.openstack.org/363715 | 14:54 |
| fungi | skraynev: looks like it imported fine to me http://git.openstack.org/cgit/openstack/k8s-docker-suite-app-murano/ | 14:54 |
| *** mdrabe has joined #openstack-infra | 14:54 | |
| vrovachev1 | andreas_s: Oh, yes, I'm so sorry :) | 14:54 |
| vrovachev1 | AJaeger: I got +1 from fuel PTL for patch https://review.openstack.org/#/c/359704/ | 14:55 |
| *** afred312 has joined #openstack-infra | 14:55 | |
| anteaya | fungi: wow, that is a lot of work manage-projects is doing for little effect | 14:56 |
| *** nt has joined #openstack-infra | 14:57 | |
| fungi | anteaya: yeah, i have a feeling it could be made _waaaay_ more efficient | 14:57 |
| fungi | though i'm not sure i have teh available bandwidth to hack on it yet | 14:57 |
| nt | hey folks, I get the following warning after updating from JJB 1.3.0 to JJB 1.6.1: WARNING:jenkins_jobs.modules.publishers:trigger-parameterized-builds:Using deprecated order for parameter sets in triggered-parameterized-builds. This will be changed in a future release to inherit the order from the user defined yaml. To enable this behaviour immediately, set the config option '__future__.param_order_from_yaml' to 'true' and change the input job | 14:57 |
| nt | configuration to use the desired order | 14:57 |
| skraynev | fungi: Great. Then it's just a some delay for updating github copy of the repo. I checked https://github.com/openstack/k8s-docker-suite-app-murano | 14:57 |
| nt | is there something I should adjust in my JJB templates to resolve that warning? | 14:57 |
| skraynev | fungi: thank you. you saved my nerves :) | 14:58 |
| anteaya | fungi: agreed on both points | 14:58 |
| fungi | skraynev: oh, github's api is terrible. we don't really support that github mirror more than on a best-effort basis. this is what it did according to our logs: http://paste.openstack.org/show/565208/ | 14:58 |
| anteaya | nt: did you consider following the instructions that accompany the warning? | 14:59 |
| *** david-lyle_ has quit IRC | 14:59 | |
| fungi | skraynev: i'll poke at it and see what happened, but odds are it created the repo in github but returned a 404 response because github is terribly broken for real automation, so we subsequently assumed the repo wasn't there and never tried to grant gerrit permission to replicate into it | 14:59 |
| skraynev | fungi: O_o. I have not known about it. | 14:59 |
| *** asselin_ has joined #openstack-infra | 14:59 | |
| *** ijw has joined #openstack-infra | 15:00 | |
| nt | anteaya, yes, but i just want to be sure that i don't need to adjust my templates in any way. | 15:00 |
| fungi | skraynev: yeah, git.openstack.org is where our official git repos are served from. github.com is not a service we control, and most of us wish we could just drop it entirely but then someone else would run an even worse mirror of our repos on github because there are too many people who think that github is the official place for everything open-source | 15:01 |
| skraynev | fungi: ok. thank you for the attention. It's really not big deal to have a copy on github IMO. However it confused me, because I was not familiar with such issue | 15:01 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add lpmqtt project https://review.openstack.org/363296 | 15:01 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add puppet-lpmqtt project https://review.openstack.org/363297 | 15:01 |
| *** inc0 has joined #openstack-infra | 15:01 | |
| skraynev | fungi: I WAS one of such people ;) | 15:01 |
| *** tongli_ has quit IRC | 15:01 | |
| *** isaacb has quit IRC | 15:02 | |
| nt | anteaya, i have my parameterized builds ordered already as i would prefer. | 15:02 |
| *** tongli has joined #openstack-infra | 15:02 | |
| fungi | skraynev: i've manually corrected the permissions for that repo in github and gerrit has successfully replicated into it now | 15:02 |
| mordred | electrofelix: ^^ nt has some questions | 15:02 |
| inc0 | hey guys, did you manage to check out our little cluster? | 15:03 |
| inc0 | pabelanger ^ | 15:03 |
| openstackgerrit | afazekas proposed openstack/os-testr: Error on invalid list parameter combination https://review.openstack.org/363739 | 15:04 |
| pabelanger | inc0: Yes, have patches up now to bring it online. | 15:04 |
| inc0 | sorry, hey and gals, ladies and gentleman, hello to everyone;) | 15:04 |
| *** dmellado is now known as dmellado|mtg | 15:04 | |
| pabelanger | https://review.openstack.org/#/q/topic:osic-cloud8 | 15:05 |
| pabelanger | rcarrillocruz: how did using cloud-launcher for infracloud mirror go? | 15:05 |
| *** jamielennox is now known as jamielennox|away | 15:05 | |
| inc0 | thank you sir | 15:05 |
| rcarrillocruz | worked just fine | 15:06 |
| rcarrillocruz | i had issues, unrelated to launcher | 15:06 |
| rcarrillocruz | computes did not have the self-signed certificate for the controller trusted | 15:06 |
| pabelanger | rcarrillocruz: cool. I'll use it for osic-cloud8 then | 15:06 |
| rcarrillocruz | but i did an ansible -a 'update-ca-certificates' against all and then i could create it fine | 15:06 |
| rcarrillocruz | pabelanger: ++ | 15:06 |
| skraynev | fungi: thank you :) | 15:06 |
| inc0 | btw pabelanger, me and mrhillsman have talk in Barcelona about benchmarking you cloud with openstack infra | 15:06 |
| inc0 | fyi:) | 15:07 |
| *** tongli has quit IRC | 15:07 | |
| fungi | ooh, i'll need to sit in on that one | 15:07 |
| fungi | i promise i won't heckle | 15:07 |
| fungi | (much) | 15:07 |
| pabelanger | sounds fun | 15:07 |
| inc0 | we'll try to convince people out there to do what we're doing | 15:07 |
| pabelanger | indeed | 15:07 |
| fungi | sound logic. how could i disagree? | 15:07 |
| inc0 | fungi, there always is a way | 15:08 |
| inc0 | somebody is wrong somewhere, always | 15:08 |
| inc0 | internet taught me this | 15:08 |
| *** bethwhite_ has quit IRC | 15:08 | |
| *** jaosorior has quit IRC | 15:08 | |
| fungi | granted, our application is a bit atypical for gloud apps in general, as we tend to do a lot of expensive (boot and delete) operations, repeated huge image uploads, et cetera | 15:09 |
| *** ijw has quit IRC | 15:09 | |
| *** armax has joined #openstack-infra | 15:09 | |
| *** _coolsvap_ is now known as coolsvap | 15:09 | |
| *** thcipriani|afk is now known as thcipriani | 15:09 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Split additional features across the periodic jobs https://review.openstack.org/362904 | 15:09 |
| inc0 | fungi, well, benchmarking is about stressing your env right? | 15:09 |
| fungi | yeah, we'll stress it in some ways your typical customers probably won't | 15:09 |
| inc0 | we'll try to characterize this workload and see if we need to add any artificial on top of it | 15:09 |
| *** yamahata has joined #openstack-infra | 15:10 | |
| fungi | i have a feeling we put a lot of load on the api and storage backend | 15:10 |
| inc0 | we'll figure it out, still it will test out basic operations heavily - and that is valuable too | 15:10 |
| mordred | fungi: otoh - I think nodepool is what people tell me "cloud native" workloads are supposed to lok like | 15:10 |
| *** matbu|mtg is now known as matbu | 15:10 | |
| mordred | fungi: you know, "cattle" that you delete and replace rather than fix - tons of elastic things | 15:11 |
| *** zhurong has quit IRC | 15:11 | |
| *** ddieterly is now known as ddieterly[away] | 15:11 | |
| inc0 | mordred, another thing is that we intend to drop live migration and move stuff around too | 15:11 |
| inc0 | at some point | 15:11 |
| *** ddieterly[away] is now known as ddieterly | 15:11 | |
| fungi | mordred: until you say "oh, we delete these servers on average every 30 minutes" | 15:11 |
| mordred | fungi: funny enough - the cloudfoundry and kubernetes folks seem to think that's _slow_ | 15:11 |
| fungi | wow | 15:11 |
| *** ddieterly is now known as ddieterly[away] | 15:11 | |
| electrofelix | nt: you don't need to update your templates, instead this is a ini file change, see http://docs.openstack.org/infra/jenkins-job-builder/execution.html#future-section | 15:11 |
| mordred | fungi: also, our control plane is apparently "not cloud enough" | 15:11 |
| mordred | fungi: so somehow we're both "too cloud" and "not cloud enough" simultaneously | 15:12 |
| fungi | mordred: our control plane is not cloud at all. we hardly ever delete it, and we get upset when it breaks | 15:12 |
| mordred | it makes me wonder if any of the people making up these terms have ever actually used a cloud, or if they've ever even run a service | 15:12 |
| inc0 | well, tbh k8s suck ass in terms of database, which is not a cattle by definition | 15:12 |
| *** sputnik13 has quit IRC | 15:12 | |
| mordred | inc0: funny that, right? | 15:12 |
| inc0 | mordred, that being said I am core in kolla-k8s too;) | 15:13 |
| *** sputnik13 has joined #openstack-infra | 15:13 | |
| *** sputnik13 has quit IRC | 15:13 | |
| fungi | yeah, i'm curious how you treat your cloudfoundry or kubernetes control plane as cattle | 15:13 |
| mordred | inc0: db is one of the things I always bring up when people tell me I shoul dhave no persistent or special computers | 15:13 |
| *** dprince has quit IRC | 15:13 | |
| inc0 | we're actually trying to make control plane work and there are nice things about it | 15:13 |
| *** ggnel_t has quit IRC | 15:13 | |
| inc0 | yeah, it's a religion which has holes, like most of religions | 15:13 |
| electrofelix | nt: this will change the order the sets of parameters are combined when using 'trigger-parameterized-builds' module from using the order defined in code to using the order you define in the template definition. In certain cases depending on the order of various parameters sets if you have a same parameter defined multiple times, the last definition of a parameter wins | 15:13 |
| fungi | mordred: databases aren't computers! they're just a haze of magic electrons permeating all your cattle | 15:14 |
| fungi | you can safely ignore databases | 15:14 |
| fungi | because cloud | 15:14 |
| nt | electrofelix, thank you, changed the ini file as suggested and the warnings went away. i was just a bit confused on what the ordering stuff meant. thanks for clarifying! | 15:14 |
| electrofelix | nt: which can affect the values of the parameters in the triggered build | 15:14 |
| *** tongli has joined #openstack-infra | 15:14 | |
| inc0 | when I get into this discussion I've even heard answers like "proper enterprises uses oracle cluster aside"...which is well... | 15:14 |
| pleia2 | good morning | 15:14 |
| inc0 | fungi, or even better, use mongo db | 15:14 |
| pabelanger | mordred: so, speaking of not cloud enough. Thoughts on standing up a 2nd nodepool.o.o server to build DIBs for our control plane servers? I'd like to see us do that | 15:14 |
| inc0 | it's webscale | 15:14 |
| jroll | if you use /dev/null for your db you can totally do it cloudy | 15:14 |
| electrofelix | nt: it should be an edge case if it bites people, hence the warning being present for a while | 15:14 |
| fungi | inc0: i happen to have seen what happens when your proper enterprise oracle cluster members get deleted or taken up and down rapidly | 15:14 |
| *** vhosakot has joined #openstack-infra | 15:15 | |
| fungi | it's... not pretty | 15:15 |
| inc0 | jroll, https://github.com/dcramer/mangodb | 15:15 |
| jroll | inc0: ++ | 15:15 |
| *** vinaypotluri has joined #openstack-infra | 15:15 | |
| inc0 | so I need to teach openstack to run on top of riak | 15:15 |
| *** tonytan4ever has joined #openstack-infra | 15:15 | |
| inc0 | riak works. | 15:15 |
| openstackgerrit | Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950 | 15:15 |
| *** Guest81 has joined #openstack-infra | 15:16 | |
| mordred | fungi: I honestly thought you said "premeating your cattle" | 15:18 |
| *** tongli has quit IRC | 15:18 | |
| fungi | mordred: pre-meat is what cattle are, after all | 15:18 |
| *** tonytan_brb has quit IRC | 15:18 | |
| mordred | fungi: yah. but maybe if you meat your meat something wonderful happens | 15:18 |
| fungi | pre-meat your cattle for extra meatiness | 15:19 |
| inc0 | kill a server before you even start it? that's so cloud! | 15:19 |
| *** jamesdenton has quit IRC | 15:20 | |
| *** tphummel has joined #openstack-infra | 15:20 | |
| *** ddieterly[away] is now known as ddieterly | 15:20 | |
| *** jamesdenton has joined #openstack-infra | 15:20 | |
| inc0 | everything is cloud native if you kill it early enough | 15:20 |
| inc0 | I think I'll make tshirt with this phrase. | 15:21 |
| *** hockeynut has joined #openstack-infra | 15:21 | |
| *** pcaruana has quit IRC | 15:22 | |
| *** rcernin has quit IRC | 15:22 | |
| openstackgerrit | Merged openstack-infra/system-config: Run puppet on infracloud in a different cron https://review.openstack.org/363719 | 15:22 |
| *** mdrabe has quit IRC | 15:24 | |
| *** mdrabe has joined #openstack-infra | 15:24 | |
| anteaya | good morning pleia2 | 15:25 |
| clarkb | rcarrillocruz: fwiw I thought we had solved the cert trustying issue in puppet for the cloud because we ran into that previously | 15:25 |
| mordred | inc0: ++ | 15:25 |
| clarkb | also has anyone investigated devstack git timeouts further? does thsi affect our cloud instances or just sdague ? | 15:25 |
| rcarrillocruz | clarkb: we have a cacert.pp manifest to do that, but i think the logic is off | 15:25 |
| anteaya | clarkb: I have not investigated nor have I witnessed that anyone else has either | 15:26 |
| rcarrillocruz | https://github.com/openstack-infra/puppet-infracloud/blob/master/manifests/cacert.pp#L24 | 15:26 |
| anteaya | though I did miss a good bit of yesterday due to weather in my area | 15:26 |
| rcarrillocruz | that looks to me is only going to exec if the file changes | 15:26 |
| clarkb | rcarrillocruz: ok I know there was some trouble with it in the past but I thought yolanda and pabelanger and crinkle_ sorted it out | 15:26 |
| rcarrillocruz | but on first deploy, the update-ca-certificates never gets run | 15:26 |
| *** yamamoto has quit IRC | 15:26 | |
| *** andreas_s has quit IRC | 15:27 | |
| pabelanger | clarkb: I've poked around on tripleo-test-cloud-rh1, basically confirming there is an issue. Sounds like we have a patch to zuul-cloner to expose timeouts and retries but that hasn't landed yet | 15:27 |
| fungi | clarkb: i made a cursory look at the cacti graphs for git.o.o, but didn't really dig deeper | 15:27 |
| rcarrillocruz | i mean, i ran an ansible -a 'update-ca-certificates' and pretty much all servers added the cert on the command output | 15:27 |
| rcarrillocruz | so it never ran | 15:27 |
| rcarrillocruz | despite the file being on /usr/local/share/ca-certificates | 15:27 |
| clarkb | anyone have a source IP address for one of the timeouts? we can check against the haproxy log | 15:28 |
| fungi | as far as i know it's not impacting our ci, but we also have a warm cache so maybe this is more pronounced for people without that to reduce their git remote operations dramatically | 15:28 |
| anteaya | pabelanger: sorry I missed most of the conversation yesterday, what was the outcome of the rename all the things to windmill discussion? | 15:28 |
| clarkb | rcarrillocruz: or the command variable is wrong? | 15:28 |
| *** Guest81 has quit IRC | 15:28 | |
| *** esp has quit IRC | 15:29 | |
| rcarrillocruz | oh wait | 15:29 |
| rcarrillocruz | i believe fatih from opnfv had a change for that | 15:29 |
| rcarrillocruz | you may be right | 15:29 |
| rcarrillocruz | sec | 15:29 |
| pabelanger | anteaya: no problem, I've abandoned the rename process for now. We can have some more discussions about it in the future | 15:29 |
| *** sandanar has joined #openstack-infra | 15:29 | |
| rcarrillocruz | yup | 15:29 |
| rcarrillocruz | https://review.openstack.org/#/c/361652/1/manifests/params.pp | 15:29 |
| rcarrillocruz | clarkb: ^ | 15:29 |
| clarkb | `sudo tail -f /var/log/haproxy.log | grep -v -- --` is the command to see less normal connections on the haproxy instance | 15:30 |
| anteaya | pabelanger: oh okay, thanks, I appreciate that, happy to participate in the future discussion if my participation is helpful | 15:30 |
| pabelanger | clarkb: 66.187.229.153 is from tripleo-test-cloud-rh1 | 15:30 |
| anteaya | yay 3 things to look at for friday | 15:30 |
| anteaya | fungi: did anyone create an etherpad for friday yet, do you know? | 15:30 |
| clarkb | pabelanger: grepping for that IP in the haproxy log doesn't show me anything that looks broken. There are connections that were ended normally according to the log | 15:31 |
| clarkb | from just over 2 hours ago | 15:31 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute016.vanilla to disabled group https://review.openstack.org/363751 | 15:32 |
| *** esikachev has quit IRC | 15:32 | |
| rcarrillocruz | fungi: is that ^ right to disabling a long-term server | 15:32 |
| rcarrillocruz | ? | 15:32 |
| fungi | anteaya: not yet, no. i'm still on the fence about using the documented playbook since it really only takes care of the easy parts (directory moves, database update queries, gerrit group renames) and punts on the hard things (restarting/requeuing zuul, moving/transferring in github) | 15:32 |
| clarkb | pabelanger: is there a job log I can compare timestamps with for that IP? | 15:32 |
| rcarrillocruz | pabelanger: i just completed puppet ansible run on infracloud mirror | 15:33 |
| anteaya | fungi: I feel we need an etherpad regardless of the chosen workflow | 15:33 |
| anteaya | fungi: would you agree? | 15:33 |
| rcarrillocruz | it's nice there's a wildcard on the site.pp, i didn't have to add anything | 15:33 |
| fungi | anteaya: sure, but having an etherpad is as easy as making up a title for it | 15:33 |
| rcarrillocruz | SO | 15:33 |
| *** dmellado|mtg is now known as dmellado | 15:33 | |
| rcarrillocruz | given that we have 42 computes | 15:33 |
| rcarrillocruz | and the mirror is up | 15:33 |
| rcarrillocruz | and dns is up | 15:33 |
| anteaya | fungi: yes, i just didn't want to do that if one existed already | 15:33 |
| fungi | anteaya: it's what to put in that etherpad that still needs to be decided | 15:33 |
| rcarrillocruz | i think we are good to add some servers on infracloud to nodepool? | 15:33 |
| mordred | rcarrillocruz: wow, awesome | 15:33 |
| rcarrillocruz | clarkb, fungi , pabelanger ^ | 15:34 |
| anteaya | fungi: agreed, thought I can add the patches and their status and leave the remainder blank for the moment | 15:34 |
| fungi | anteaya: thanks | 15:34 |
| anteaya | okay thank you | 15:34 |
| *** jlanoux has quit IRC | 15:34 | |
| fungi | rcarrillocruz: neat! did you do any benchmarking yet to figure out what the flavor needs to have (particularly from a cpu performance perspective)? | 15:35 |
| clarkb | confirmed it is /usr/sbin/ on ubuntu not just debian | 15:35 |
| rcarrillocruz | fungi: i ran a nova dsvm full tempest last week | 15:35 |
| rcarrillocruz | it took a bit less than an hour | 15:35 |
| rcarrillocruz | which is in line to an osic run | 15:35 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Set default for pingtest template https://review.openstack.org/363753 | 15:35 |
| rcarrillocruz | obv. , there were no neighbours on that compute :-) | 15:35 |
| anteaya | fungi: https://etherpad.openstack.org/p/project-renames-Septemeber-2016 | 15:35 |
| rcarrillocruz | i suggest we bump little by little and see how it goes? | 15:36 |
| mordred | rcarrillocruz: ++ | 15:36 |
| *** tonytan_brb has joined #openstack-infra | 15:36 | |
| mordred | rcarrillocruz: it'll be fun to watch load on the cloud as we increase nodepool load | 15:36 |
| rcarrillocruz | indeed | 15:36 |
| rcarrillocruz | i believe we've never seen so much capacity if we are able to bring up the whole infracloud PLUS the new osic | 15:36 |
| rcarrillocruz | \o/ | 15:36 |
| *** tesseract- has quit IRC | 15:36 | |
| *** JerryOpenix has quit IRC | 15:37 | |
| * clarkb has started a local clone nova loop to see if the issue can be reproduced from here | 15:38 | |
| rcarrillocruz | i'll propose the patch | 15:38 |
| *** tonytan4ever has quit IRC | 15:38 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add lpmqtt project https://review.openstack.org/363296 | 15:39 |
| clarkb | rcarrillocruz: and you have trusted the cert everywhere? | 15:39 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add puppet-lpmqtt project https://review.openstack.org/363297 | 15:39 |
| rcarrillocruz | clarkb: yeah, i did an ansible -a 'update-ca-certificates' ~compute | 15:39 |
| fungi | clarkb: rcarrillocruz: it looks like the ca-certificates package provides /usr/sbin/update-ca-certificates on my debian systems | 15:39 |
| fungi | same as on ubuntu 14.04 | 15:39 |
| rcarrillocruz | y | 15:40 |
| *** salv-orlando has joined #openstack-infra | 15:40 | |
| clarkb | rcarrillocruz: its not just compute that needs it fwiw | 15:40 |
| clarkb | rcarrillocruz: the controller(s) should also get that as things talk to keystone for example | 15:40 |
| *** sdake_ is now known as sdake | 15:40 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Enable infracloud servers in Nodepool https://review.openstack.org/363756 | 15:41 |
| rcarrillocruz | the controller i did myself manually | 15:41 |
| clarkb | (its a good thing my ISP doesn't have a quota on my usage as I clone nova over and over and over again) | 15:42 |
| * rcarrillocruz goes grab some coffee | 15:42 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenari001, 002 and 003 https://review.openstack.org/362504 | 15:43 |
| clarkb | given that there hasn't been tons of screaming on the git thing is it possible that sdague and tripleo cloud share a common networking issue to git.o.o? I haven't been able to reproduce this yet locally from the other coast and haproxy logs from a quick grep look happy | 15:43 |
| *** mikelk has quit IRC | 15:44 | |
| *** cody-somerville has quit IRC | 15:45 | |
| *** nstolyarenko has joined #openstack-infra | 15:45 | |
| *** armax has quit IRC | 15:45 | |
| mordred | clarkb: maybe sdague and the tripleo cloud are actually the same person ... have you ever seen them in the same place at the same time?? | 15:45 |
| anteaya | the higgins to zun rename patch isn't a rename patch I just discovered, it merged in june and changes the names of their irc channels | 15:46 |
| *** salv-orlando has quit IRC | 15:46 | |
| *** _nadya_ has quit IRC | 15:46 | |
| anteaya | trying to flag down one of their developers to get them to offer a rename patch | 15:46 |
| *** esikachev has joined #openstack-infra | 15:47 | |
| *** matt-borland has joined #openstack-infra | 15:47 | |
| *** armax has joined #openstack-infra | 15:47 | |
| *** abregman has quit IRC | 15:47 | |
| *** mkarpin has joined #openstack-infra | 15:47 | |
| *** matrohon has quit IRC | 15:48 | |
| *** markvoelker has quit IRC | 15:48 | |
| *** kaisers_ has joined #openstack-infra | 15:48 | |
| mkarpin | Hello all! are there some issues with git.openstack.org or review.openstack.org? | 15:48 |
| nstolyarenko | Hello Folks! Could you please review my patch https://review.fuel-infra.org/#/c/25430/. It is very important for us. Thank you | 15:49 |
| clarkb | mkarpin: that is something we are investigating. Are you noticing timeouts to git.openstack.org? have any more info? | 15:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: sort output of latest-deliverable-versions by team https://review.openstack.org/363763 | 15:49 |
| mordred | sc68cal: wanna see a funny typo? | 15:50 |
| mordred | sc68cal: filters={'pysical_network': 'public'}) | 15:50 |
| rcarrillocruz | fwiw i had random lags pulling from git.openstack.org about an hour a half ago | 15:50 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Fix path to update-ca-certificates for Debian https://review.openstack.org/361652 | 15:50 |
| mordred | sc68cal: not surprisingly, that did not find the network | 15:50 |
| sc68cal | mordred: haha so that's why it failed | 15:50 |
| mat128 | mordred: too much python :) | 15:50 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: sort output of latest-deliverable-versions by team https://review.openstack.org/363763 | 15:51 |
| clarkb | pabelanger: I am looking at https://review.openstack.org/#/c/356703/7/modules/openstack_project/templates/nodepool/clouds.yaml.erb will cloud8 not have the same network setup as cloud1? eg we need to explicitly list that the v6 network has public v6 and private v4? | 15:51 |
| sc68cal | nstolyarenko: wrong channel. #fuel ? | 15:51 |
| mkarpin | clarkb yes on my third party ci i have devstack stucked on | 15:51 |
| mkarpin | 2016-08-31 14:26:37.704 | ++ /opt/stack/murano/devstack/plugin.sh:install_murano:357 : git_clone git://git.openstack.org/openstack/murano.git /opt/stack/murano master | 15:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Skip test creating provider network if one exists https://review.openstack.org/363715 | 15:52 |
| jeblair | clarkb, rcarrillocruz, mordred: i know everyone working on nodepool stuff has access to the server now, but it still might be nice to get this in: https://review.openstack.org/346943 | 15:52 |
| mkarpin | clarkb https://murano-ci.mirantis.com/jenkins/job/gate-murano-dashboard-ubuntu/975/consoleFull | 15:52 |
| rcarrillocruz | looking | 15:52 |
| mordred | jeblair: lgtm | 15:52 |
| rcarrillocruz | jeblair: lgtm | 15:53 |
| clarkb | mkarpin: does that host have ipv6? | 15:54 |
| clarkb | mkarpin: or would this be ipv4 only? | 15:54 |
| *** rcernin has joined #openstack-infra | 15:54 | |
| mordred | clarkb: btw - a devstack change made the shade gate not work, so we didn't land the revert so I didn't restart nodepool | 15:54 |
| mkarpin | I think ipv4 only | 15:54 |
| clarkb | mordred: kk | 15:55 |
| mordred | clarkb: fix to the test case is in flight, at which point we can land the revert | 15:55 |
| nstolyarenko | Sorry, wrong patch URL. Please review this one https://review.openstack.org/#/c/361965/ | 15:55 |
| mkarpin | clarkb I think ipv4 only, need to check with it guys | 15:55 |
| clarkb | mkarpin: looking at the log origin is git.openstack.org not review.openstack.org. Do you have some other information that would indicate review.openstack.org is also affected? | 15:55 |
| jeblair | rcarrillocruz: are we about to turn on infracloud for realz? | 15:55 |
| mkarpin | clarkb i think i have something one moment | 15:56 |
| *** bethwhite_ has joined #openstack-infra | 15:56 | |
| clarkb | (if review.o.o is also affected I am more likely to blame the network as they sit in the same DC but are otherwise hosted on completely different systems) | 15:57 |
| rcarrillocruz | jeblair: yes, i think we are good to test it | 15:57 |
| rcarrillocruz | to recap: | 15:57 |
| rcarrillocruz | vanilla has 48 nodes | 15:57 |
| rcarrillocruz | mitaka | 15:58 |
| jeblair | rcarrillocruz: 48 compute hosts? | 15:58 |
| rcarrillocruz | 2 nodes are taken for bifrost/baremetal and another one for controller | 15:58 |
| bnemec | clarkb: FWIW, this is what I get when I clone from the tripleo cloud: http://paste.openstack.org/show/565231/ | 15:58 |
| rcarrillocruz | out of 46 compute hosts, 3 are not working, due to NIC / vlan issues | 15:58 |
| clarkb | why does bifrost need 2 instances? | 15:58 |
| bookwar | clarkb: we have following bug reported https://bugs.launchpad.net/mos/+bug/1618936 | 15:58 |
| jeblair | gotcha | 15:58 |
| openstack | Launchpad bug 1618936 in Mirantis OpenStack "[pkgs-ci-pub] Unusual upstream gerrit behavior" [Critical,New] - Assigned to Fuel CI (fuel-ci) | 15:58 |
| bnemec | Oh wait, it finally got off 36%. | 15:58 |
| rcarrillocruz | clarkb: just one, bifrost/baremetal is the same thing | 15:59 |
| mgagne | is there a way to rebase a whole topic in Gerrit? suppose you don't have the commit locally yet | 15:59 |
| rcarrillocruz | it's just the bifrost machine is called baremetal00 | 15:59 |
| bnemec | Now it's stuck at Receiving objects: 1% (134/13306), 44.00 KiB | 1024 bytes/s | 15:59 |
| *** shamail has joined #openstack-infra | 15:59 | |
| *** ansiwen has quit IRC | 15:59 | |
| fungi | dhellmann: looks like puppet finally applied 363156 to signing01.ci.o.o | 15:59 |
| rcarrillocruz | i'll sort out the offending servers with the DC folks | 15:59 |
| rcarrillocruz | and as for chocolate | 15:59 |
| mgagne | git review -d <change-id> will download the chain in its current state but won't update the chain to latest patchset | 15:59 |
| rcarrillocruz | we have more machines | 15:59 |
| jeblair | clarkb: at one point i think we decided to do that for safety/redundancy | 15:59 |
| rcarrillocruz | but HW is not as good | 15:59 |
| dhellmann | fungi : thanks for the heads-up | 15:59 |
| clarkb | bookwar: to be clear the https and ssh hosts are completely different htere | 16:00 |
| *** matthewbodkin has quit IRC | 16:00 | |
| mkarpin | clarkb i have zuul often stucked with thing like that for example http://paste.openstack.org/show/565230/, it just stucked, and do not merge in ps it looks like http://paste.openstack.org/show/565233/ | 16:00 |
| rcarrillocruz | jeblair, clarkb : we had baremetal00 and baremetal01 cos we had east/west in different locations | 16:00 |
| pabelanger | clarkb: yes, I should fix that | 16:00 |
| clarkb | bookwar: you may want to clarify that in the bug as otherwise its really really confusing | 16:00 |
| rcarrillocruz | but now we have just one bifrost serving pxe boot to both | 16:00 |
| jeblair | mgagne: not a whole topic at once, i don't think. (in gerrit, the 'cherry-pick to branch' command will probably work for one change) | 16:00 |
| jeblair | mgagne: but! | 16:00 |
| *** abregman has joined #openstack-infra | 16:00 | |
| mkarpin | clarkb its only today | 16:00 |
| mgagne | gretty again? =) | 16:00 |
| openstackgerrit | Peter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare. https://review.openstack.org/362950 | 16:00 |
| *** gyee has joined #openstack-infra | 16:00 | |
| fungi | mgagne: you might want to `git review -d somechange` followed by `git restack` https://pypi.org/project/git-restack/ | 16:01 |
| *** Ahharu has quit IRC | 16:01 | |
| fungi | mgagne: though as jeblair points out, a patch series and a topic are distinct concepts in gerrit. probably would help to know for sure which you mean | 16:01 |
| mgagne | fungi: how will restack know about latest version of a change from Gerrit? it looks like git-review will cherry-pick and won't know about updated changes | 16:01 |
| jeblair | mgagne: you can download the last change, then run 'git restack'. though that's usually best for *not* rebasing a series. if you *do* want to rebase the series, then plain old 'git rebase' may be what you want. | 16:01 |
| mgagne | fungi: anything form of chain of commits | 16:02 |
| *** weshay is now known as weshay_food | 16:02 | |
| jeblair | mgagne: 'git review -d <tip>' will download the whole patch series | 16:02 |
| Zara | question from the StoryBoard meeting-- we have a pythonclient! it has docs (and further docs-in-progress)! the docs aren't rendered anywhere handy! we think it's better to keep things modular and keep them living in the pythonclient repo, so now we're wondering: how we can ensure they're tracked on docs.openstack.org? | 16:02 |
| fungi | mgagne: oh, is the problem you're seeing that you have a series of dependent changes A,B,C and someone has uploaded a new version of B without rebasing C? | 16:02 |
| mkarpin | clarkb bookwar i am experiencing exactly the same as https://bugs.launchpad.net/mos/+bug/1618936 | 16:02 |
| openstack | Launchpad bug 1618936 in Mirantis OpenStack "[pkgs-ci-pub] Unusual upstream gerrit behavior" [Critical,New] - Assigned to Fuel CI (fuel-ci) | 16:02 |
| mgagne | the case is: the one performing the rebase doesn't have the latest version locally and can only access Gerrit | 16:02 |
| anteaya | I've informed hongbin in the -zun channel that we need a rename patch for their project to be renamed: http://eavesdrop.openstack.org/irclogs/%23openstack-zun/%23openstack-zun.2016-08-31.log.html#t2016-08-31T15:43:05 | 16:02 |
| clarkb | rcarrillocruz: gotcha | 16:02 |
| anteaya | in case someone shows up asking about it and I'm not around | 16:03 |
| mgagne | fungi: that's one example. I know you can rebase from UI but it's say for the sack of this example, that there is a million change. can't click forever | 16:03 |
| zaro | morning | 16:03 |
| *** apuimedo is now known as apuimedo|away | 16:03 | |
| *** trown is now known as trown|brb | 16:03 | |
| anteaya | morning zaro | 16:03 |
| jeblair | Zara: add jobs similar to the storyboard jobs in project-config | 16:03 |
| clarkb | error: RPC failed; curl 56 SSL read: error:00000000:lib(0):func(0):reason(0), errno 104 success! now to see what haproxy/apache say about my ip | 16:03 |
| Zara | jeblair: aha, thank you | 16:03 |
| *** edtubill has joined #openstack-infra | 16:04 | |
| jeblair | Zara: 'infra-publish-jobs' looks like the name in zuul | 16:04 |
| *** devananda|MOVING is now known as devananda | 16:05 | |
| *** ansiwen has joined #openstack-infra | 16:05 | |
| jeblair | Zara: looks like they are already setup in jjb, so just the zuul layout.yaml change is needed | 16:05 |
| jeblair | oh wait | 16:05 |
| jeblair | what's python-storyboardclient-infra-docs-tags-only | 16:06 |
| Zara | I have no idea but it sounds exciting. | 16:06 |
| *** ildikov has quit IRC | 16:06 | |
| jeblair | description: Publish infra documents, use when only publish on tag | 16:06 |
| clarkb | [31/Aug/2016:15:55:05.089] balance_git_https balance_git_https/git08.openstack.org 1/0/253881 108025 cD 73/58/58/7/0 0/0 | 16:06 |
| jeblair | Zara: so it looks like storyboardclient docs are already set to be published, but only when the repo is tagged | 16:07 |
| *** rockyg has joined #openstack-infra | 16:07 | |
| clarkb | and from the haproxy manual: this is oftne caused by network failures on the client side | 16:07 |
| *** hewbrocca is now known as hewbrocca-afk | 16:07 | |
| jeblair | Zara: so when it is tagged, it should show up here: http://docs.openstack.org/infra/python-storyboardclient/ | 16:08 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730 | 16:08 |
| clarkb | the tripleo IP that pabelanger provided did not close any connections with the cD state though | 16:08 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729 | 16:08 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Don't allow users to subscribe to private worklists they can't see https://review.openstack.org/363776 | 16:08 |
| *** trown|brb is now known as trown | 16:08 | |
| jeblair | Zara: i don't know why that choice was made for that repo | 16:08 |
| SotK | I suspect it would be sensible for us to undo that choice | 16:09 |
| *** drifterza has quit IRC | 16:09 | |
| *** hashar has quit IRC | 16:09 | |
| *** Julien-zte has quit IRC | 16:09 | |
| jeblair | clarkb: http://grafana.openstack.org/dashboard/db/git-load-balancer?panelId=11&fullscreen | 16:10 |
| jeblair | clarkb: zoom out a bit there | 16:10 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/363669 | 16:10 |
| Zara | (I suppose the choice has meant I've just learned more about how project-config works, but I agree) | 16:10 |
| clarkb | tahts neat I go from 7ms to 72ms rtt in seattle | 16:10 |
| clarkb | its like they have thousands of miles of cable looped up to spin my packets around in | 16:10 |
| jeblair | clarkb: oh, here's how you link zoom: http://grafana.openstack.org/dashboard/db/git-load-balancer?from=1471968641519&to=1472659841519 | 16:10 |
| anteaya | clarkb: just for you | 16:10 |
| jeblair | grr.. that's the whole dashboard | 16:11 |
| *** links has joined #openstack-infra | 16:11 | |
| jeblair | clarkb: so nevermind -- do that first link then zoom out :) | 16:11 |
| anteaya | like flash boys I think is the name of the book | 16:11 |
| openstackgerrit | Merged openstack-infra/system-config: Update osic-cloud1 credential format https://review.openstack.org/356702 | 16:11 |
| fungi | mgagne: yeah, i agree the git-review -d behavior seems to be to download the latest patchset of the change you specify (assuming you don't include a patchset number for it) along with the specific patchsets (not latest patchsets) it depends on in other changes | 16:11 |
| *** Sukhdev has joined #openstack-infra | 16:11 | |
| clarkb | jeblair: ok | 16:11 |
| clarkb | fungi: mgagne yes because its just fetching the patchset you told it to and git is pulling in the parents automagically | 16:12 |
| mgagne | fungi: yea, will check with the one with the problem, it's not me. Deps aren't showing well in Gerrit so it's hard to visualize the state of things. maybe it's just a matter of git review -d the chain and rebase against his own change. | 16:12 |
| mgagne | clarkb: true. I think it's just a matter of education around what git-review is really doing | 16:12 |
| *** asettle has quit IRC | 16:12 | |
| clarkb | jeblair: interesting. What does that map to in the haproxy logs? is it the health checks on the backend | 16:12 |
| *** javeriak has joined #openstack-infra | 16:13 | |
| *** tongli has joined #openstack-infra | 16:13 | |
| *** ianw has quit IRC | 16:13 | |
| jeblair | clarkb: i will refresh my memory :) | 16:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730 | 16:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729 | 16:13 |
| *** asettle has joined #openstack-infra | 16:13 | |
| *** abregman has quit IRC | 16:13 | |
| mordred | odyssey4me, rbergeron: re: galaxy api ... it appears that there IS a REST API - it's just not documented. sniffing the network traffic of the web ui shows the API interactions to do things like "hey, plesae import repo X" | 16:13 |
| fungi | mgagne: right, i'm thinking through ways we could enhance git-review to make that easier. it's a complex problem because you ultimately need to initiate a rebase for any changes after one which has a newer patchset in gerrit, and that cascades the rest of the way up the series, potentially resulting in merge conflicts on multiple commits along teh way | 16:14 |
| jeblair | clarkb: it's eresp: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/git/haproxy-statsd.py#n76 | 16:14 |
| clarkb | AH01215: fatal: The remote end hung up unexpectedly I see those in the git error log for httpd on git03 | 16:15 |
| clarkb | (picked git03 as it seemed to be errory according to the graph jeblair linked) | 16:15 |
| Zara | jeblair: hm, looking at `release:` there, it seems storyboard has a similar setting (storyboard-infra-docs-tags-only) | 16:16 |
| Zara | but docs for storyboard are rendered | 16:16 |
| Zara | so I'm wondering if there's anything else affecting it | 16:16 |
| ihrachys | so what's about those network glitches when working with openstack infra resources (git, gerrit)? is it a known thing? | 16:16 |
| clarkb | jeblair: that includes write errors on the client sockets but haproxy claims that won't be counted against the server stats. Not sure how else it would show them in that case | 16:16 |
| Zara | (or hm, maybe the repo only needs one tag and doesn't have one) | 16:17 |
| *** Sukhdev has quit IRC | 16:17 | |
| *** yamamoto has joined #openstack-infra | 16:17 | |
| Zara | (I parsed it as 'release on new tag') | 16:17 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Move unsuccessful non-voting OSA jobs to experimental https://review.openstack.org/363783 | 16:17 |
| *** javeriak has quit IRC | 16:17 | |
| *** asettle has quit IRC | 16:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 16:17 |
| *** javeriak has joined #openstack-infra | 16:18 | |
| pabelanger | raddaoui: ^ updates for osic-cloud8 | 16:18 |
| *** kaisers_ has quit IRC | 16:18 | |
| *** akshai has quit IRC | 16:18 | |
| clarkb | for my specific connection git said: fatal: The remote end hung up unexpectedly on my desktop. Haproxy said client disconnected | 16:18 |
| clarkb | this tells me neither end wanted to close the tcp connection but something did | 16:18 |
| mgagne | fungi: yes. I think what that person asked me is a bit far fetched where he wanted to rebase someone's else series of changes which you usually don't do yourself. | 16:18 |
| mgagne | fungi: and as you said, you could end up in merge conflicts hell | 16:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 16:19 |
| raddaoui | nice pabelanger . mrhillsman: ^ | 16:19 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Proxy nodepool webapp status commands https://review.openstack.org/346943 | 16:19 |
| clarkb | none of the git hosts appear anywhere near their bw limits. Grafana shows that haproxy statistics include an increase in https server eresp errors | 16:20 |
| jeblair | mgagne: sometimes i use gertty for this -- i walk down the tree and hit 'x' to cherry pick the latest version of each patchset in whatever order i want to do it in. | 16:20 |
| *** ijw has joined #openstack-infra | 16:20 | |
| clarkb | but review.openstack.org is also apparently affected making me further think its less of a git.openstack.org issue as those two stacks are so vastly different. Different gits apaches ssls kernels etc | 16:20 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 16:21 |
| mgagne | jeblair: yes. It happens the person isn't used to gerrit workflow but to github PR's one instead which hides a lot of git concepts =) | 16:21 |
| pabelanger | raddaoui: mrhillsman: if you want to check our configuration for cloud8 ^ | 16:21 |
| Zara | (and hm, `git tag` doesn't show any tags for either repo; am I thinking of the right sort of tag?) | 16:21 |
| AJaeger | Zara, just finished reading backscroll... | 16:21 |
| AJaeger | Zara, let me help: We publish in general - the client projects only when there are tags, since most users will install a release. | 16:22 |
| AJaeger | And for server projects we publish with each version - so that developers get the info. | 16:22 |
| *** martinkopec has quit IRC | 16:22 | |
| *** Benj_ has joined #openstack-infra | 16:22 | |
| mordred | rbergeron, odyssey4me: I have updated the galaxy-issues bug with information | 16:22 |
| AJaeger | But we can change that. If you want to change the job for your client, just send a proposal. | 16:22 |
| AJaeger | Zara: Or do a release ;) | 16:22 |
| jeblair | clarkb: the only other thing i note from the graphs is there seems to be a bit more http (not https) current sessions tonday than normal. it's a stretch. you kind of have to squint to see it. i'm not giving it a lot of weight. | 16:23 |
| anteaya | AJaeger: any idea what yuval's irc nick is? https://review.openstack.org/#/c/353304 | 16:23 |
| *** Swami has quit IRC | 16:23 | |
| odyssey4me | mordred reverse engineers yet another api :) | 16:24 |
| anteaya | AJaeger: and i don't know if you caught it in backscroll but zun doesn't have a rename patch in gerrit | 16:24 |
| AJaeger | anteaya: http://stackalytics.com/?user_id=jhamhader -> https://launchpad.net/~jhamhader -> it's youval | 16:24 |
| AJaeger | anteaya: didn't catch that one ;( | 16:24 |
| anteaya | AJaeger: their link in the wiki was to a patch to rename their channels that you merged in june | 16:24 |
| fungi | mgagne: what i've usually done is isolate the first change depending on an outdated patchset, then `git review -d` the latest patchset for its parent change id and `git review -x` the change in question followed by all child changes in the series one by one, fixing merge conflicts as i go | 16:24 |
| AJaeger | yuval I mean | 16:24 |
| rbergeron | mordred: thank you! :) | 16:24 |
| AJaeger | anteaya: ;( | 16:24 |
| *** ijw has quit IRC | 16:24 | |
| anteaya | AJaeger: neither did I, so I posted in their channel | 16:24 |
| Zara | AJaeger: Thanks, that makes things clearer. Though I'm still confused because the jobs suggest that docs for both projects are released on a 'only when tagged' basis, but we have docs up for one and not the other, and neither has any tags. I'm wondering if the client also needs to be listed explicitly here: https://git.openstack.org/cgit/openstack-infra/project-config/tree/docs-site/infra-documents. | 16:24 |
| *** yaume_ has quit IRC | 16:25 | |
| Zara | yaml ? | 16:25 |
| anteaya | AJaeger: so far no response | 16:25 |
| Zara | what a wonderful place to split a url | 16:25 |
| Zara | https://git.openstack.org/cgit/openstack-infra/project-config/tree/docs-site/infra-documents.yaml | 16:25 |
| mordred | rbergeron: honestly, just docs on auth handshake would be Good Enough | 16:25 |
| AJaeger | anteaya: https://review.openstack.org/#/c/329247 | 16:25 |
| mgagne | fungi: yea but I strongly suggest to that person: yea, don't do that, let that person deals with his own changes. | 16:25 |
| fungi | Zara: it's possible the one which has docs and no tags originally had a different docs publication job in place | 16:25 |
| anteaya | AJaeger: and thank you, yuval or someone else from smaug/karbor, this patche needs a rebase please: https://review.openstack.org/#/c/353304 | 16:25 |
| *** shashank_hegde has joined #openstack-infra | 16:25 | |
| anteaya | AJaeger: ah thank you | 16:26 |
| AJaeger | anteaya: will you update the wiki? And update the topic on all changes for consistency? | 16:26 |
| *** tongli_ has joined #openstack-infra | 16:26 | |
| clarkb | jeblair: the connection retries for the backends seem to be sitting consistently at 0 which makes me think that the backends are pretty happy | 16:26 |
| anteaya | topic is updated as long as they keep the same topic during rebases | 16:26 |
| anteaya | will update the wiki, thank you | 16:27 |
| jeblair | Zara: yes it does need to be listed there, but only after being published; that's just the index page | 16:27 |
| clarkb | total conenctions I don't see going over 100 so should be well below any system fd limits | 16:27 |
| AJaeger | fungi, that's the same approach I use | 16:27 |
| AJaeger | thanks, anteaya | 16:27 |
| clarkb | the cD state does seem to occur after roughly 120000 milliseconds between connection accept and close | 16:28 |
| clarkb | (there are some outliers to that but not many) | 16:28 |
| Zara | fungi: ah, okay. I don't have history so I don't know if that happened or not, am looking at storyboard over here: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n4705 | 16:28 |
| *** roxanaghe has joined #openstack-infra | 16:28 | |
| anteaya | AJaeger: thank you, also fixed in etherpad which is https://etherpad.openstack.org/p/project-renames-Septemeber-2016 | 16:28 |
| *** ddieterly is now known as ddieterly[away] | 16:28 | |
| AJaeger | Zara, yes, you need to list it there. But let's check the contents on docs.o.o... | 16:29 |
| *** jpich has quit IRC | 16:29 | |
| clarkb | its like something is just nuking any connection taking longer than 2 minutes | 16:29 |
| clarkb | (except there are plenty of happy connectiosn that take longer too) | 16:29 |
| AJaeger | Zara, last push of storyboard docs was on 26th of August | 16:29 |
| *** tongli has quit IRC | 16:29 | |
| *** dprince has joined #openstack-infra | 16:30 | |
| *** tongli_ has quit IRC | 16:30 | |
| AJaeger | Zara, for storyboard there's the template infra-publish-jobs used - which publishes after each commit. | 16:30 |
| *** yamamoto has quit IRC | 16:30 | |
| AJaeger | So, we do both right now - publish after tag and publish after each commit. | 16:30 |
| clarkb | the server and backend queues are also sitting at 0 telling me that haproxy isn't having to park things waiting on backends | 16:30 |
| *** weshay_food is now known as weshay | 16:30 | |
| AJaeger | Zara, I can clean this up. Do you want to change python-storyboard publishing as well? | 16:31 |
| clarkb | which is expected when well below limits | 16:31 |
| Zara | AJaeger: ahhhh, right. yeah, I'd like python-storyboardclient to publish on each commit, so I'm guessing we want it to have use same template | 16:32 |
| *** tongli has joined #openstack-infra | 16:32 | |
| fungi | clarkb: this is a pretty common behavior when another instance sharing the same hypervisor is eating up most of the nic buffers | 16:32 |
| mgagne | update on mtl01 at internap: replaced network hardware, tests are in progress and so far, it's doing well. will update once completed. | 16:33 |
| fungi | the host's tcp/ip stack begins to exhibit pathological behaviors that result in all manner of odd disconnects and timeouts for guests | 16:33 |
| *** nstolyar_ has joined #openstack-infra | 16:33 | |
| *** nstolyarenko has quit IRC | 16:33 | |
| Zara | AJaeger: if you're able to do it, that would be wonderful (and thank you!) otherwise I'm happy to :) | 16:33 |
| clarkb | fungi: ya everything I can see from what we control and have access to looks happy | 16:34 |
| mordred | mgagne: cool. did we find a real problem for you then? | 16:34 |
| *** wgd3[away] has quit IRC | 16:34 | |
| clarkb | fungi: I think its something in the network stack between $client and git.openstack.org | 16:34 |
| zaro | Online reindex testing on review-dev.openstack.org will commence in about 30 mins. Would appreciate volunteers to bang at gerrit. Do as much banging as you like, a few mins to about 1 hr. | 16:34 |
| *** ddieterly[away] is now known as ddieterly | 16:34 | |
| mgagne | mordred: so far, all points to a faulty network hardware | 16:34 |
| mordred | excellent | 16:34 |
| jeblair | clarkb, fungi: this is affecting all of the backend servers though, right? | 16:34 |
| clarkb | fungi: apparently review.openstack.org is also exhibiting some of this maybe it shares a hypervisor or switch or router or something | 16:34 |
| anteaya | zaro: I'm not sure my timing will line up, I'm about to go offline for the rest of the day | 16:35 |
| fungi | clarkb: it could certainly be a switch with a nearly-full bridge table or something | 16:35 |
| anteaya | zaro: will participate if I am online | 16:35 |
| clarkb | jeblair: yes my grep for cDs shows a pretty good spread | 16:35 |
| mgagne | my own tests didn't show the problem since I landed on the non-faulty hardware. in fact, half of the hardware was faulty, not the other. | 16:35 |
| clarkb | jeblair: but the errors are happenign in front of haproxy I think not behind. We are not queuing or needing to retry any connections to the backends | 16:35 |
| mordred | mgagne: oh lovely | 16:35 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Update storyboard publishing https://review.openstack.org/363795 | 16:35 |
| AJaeger | Zara: ^ | 16:35 |
| nt | Does Jenkins Job Builder still support Python 2.6? A lot of projects are dropping 2.6 support and I'm just checking on JJB. I looked through the docs and didn't see anything about verson compatibility. | 16:35 |
| jeblair | clarkb: oh hrm, i thought that would be the error that's not counted in eresp... | 16:36 |
| *** esp has joined #openstack-infra | 16:36 | |
| AJaeger | Zara: once that's in and you have documents published for the client, please send yourself a change for the yaml file so that the index file gets updated. | 16:36 |
| clarkb | nt: we don't have a test platform for python2.6 so its not tested there at least | 16:36 |
| *** tongli has quit IRC | 16:36 | |
| zaro | nt: it's no longer tested against py26 only py27 and py34 | 16:36 |
| clarkb | jeblair: eresp can include client errors according to the haproxy manual. I think we are seeing these cDs show up in the eresp stats | 16:36 |
| *** shashank_hegde has quit IRC | 16:37 | |
| clarkb | and its pretty consistently 2 minutes and bam connection is cD | 16:37 |
| jeblair | clarkb: write error on the client socket (won't be counted for the # server stat) | 16:37 |
| nt | clarkb, zaro, thanks for the info. I get deprecation warnings about 2.6 from some of the dependencies, so this is good info. | 16:37 |
| Zara | AJaeger: yay, thanks! Will do. | 16:37 |
| zaro | anteaya: sounds good. | 16:37 |
| jeblair | clarkb: that makes me think that client errors would show up in stats.haproxy.balance_git_https.BACKEND.eresp but not stats.haproxy.balance_git_https.git01.eresp | 16:37 |
| clarkb | jeblair: ya thats how I would interpret it too but the logs themselves don't seem to line up with that. eg tehre are no retries | 16:38 |
| *** tongli has joined #openstack-infra | 16:38 | |
| clarkb | (I would expect a retry when a backend eresps) | 16:38 |
| fungi | clarkb: thinking back, i've seen similar behavior on overloaded switchrouters, where the routing is tightly coupled to bridge flows and so actively disconnects open sockets its tracking when the flow table begins to fill up | 16:38 |
| Zara | zaro: what's the best way for us to hammer gerrit? | 16:39 |
| fungi | by spoofing tcp/rst or similar | 16:39 |
| clarkb | granted I am looking at the subset of the logs that matches grep -v -- -- | 16:40 |
| fungi | clarkb: are we seeing this for git protocol too? or just http(s)? | 16:40 |
| *** tphummel has quit IRC | 16:41 | |
| zaro | Zara: the test will be around changes. so any type of operation that involves a change. like create patchset/review it/update it/merge it/download it/etc.. | 16:41 |
| clarkb | fungi: ya the cDs appear to be all protocols. Interesting I just saw some SDs | 16:41 |
| clarkb | those would be server disconnects | 16:41 |
| *** ddieterly is now known as ddieterly[away] | 16:42 | |
| anteaya | zaro: my company has arrived, I'm offline now, sorry for the poor timing hope you get some volunteers | 16:42 |
| anteaya | zaro: thanks for testing | 16:42 |
| *** tongli has quit IRC | 16:43 | |
| *** ianw has joined #openstack-infra | 16:43 | |
| *** _nadya_ has joined #openstack-infra | 16:43 | |
| clarkb | SD "The connection to the server died with an error during the data transfer. This usually means that haproxy has received an RST from the server or an ICMP message from an intermediate equipment while exchanging data with the server. This can be caused by a server crash or by a network issue on an intermediate equipment." | 16:43 |
| clarkb | so still doesn't rule out network issues | 16:43 |
| Zara | zaro: okay, sounds straightforward! | 16:43 |
| *** tongli has joined #openstack-infra | 16:44 | |
| *** fguillot_ has joined #openstack-infra | 16:44 | |
| *** _nadya_ has quit IRC | 16:44 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add OSA keystone uwsgi functional tests https://review.openstack.org/363640 | 16:44 |
| clarkb | jeblair: the SDs are much less frequent and we got two in close proximity to each other for different backend hosts | 16:44 |
| pleia2 | zaro: I got asked to join a call, is it ok to push out testing a few? | 16:44 |
| clarkb | but that could possibly explain the server specific eresps | 16:44 |
| clarkb | in addition to the client disconnect cDs | 16:44 |
| *** lucasagomes has quit IRC | 16:44 | |
| zaro | pleia2: well reindex will take 1.5 hrs. so as long as it's not a super long call you should have plenty of time to bang. | 16:45 |
| *** tphummel has joined #openstack-infra | 16:45 | |
| *** lucasagomes has joined #openstack-infra | 16:46 | |
| jeblair | clarkb: here's the doc; you can find the stats in sec 9.1: http://www.haproxy.org/download/1.5/doc/configuration.txt | 16:46 |
| pleia2 | zaro: ok :) | 16:46 |
| jeblair | clarkb: i agree with your observation about eresp. i can't explain the apparent discrepancy. i don't feel like reading the haproxy source right now though. :) | 16:46 |
| *** yamamoto has joined #openstack-infra | 16:47 | |
| *** tongli has quit IRC | 16:48 | |
| *** Apoorva has joined #openstack-infra | 16:48 | |
| clarkb | I am going to restart my cloen nova test on a backend directly and have it talk to lcoalhost | 16:48 |
| clarkb | to at least bolster the argument that the service itself is fine | 16:48 |
| *** ddieterly[away] is now known as ddieterly | 16:49 | |
| *** daemontool has joined #openstack-infra | 16:50 | |
| *** akshai has joined #openstack-infra | 16:50 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute038.vanilla to disabled group https://review.openstack.org/363805 | 16:51 |
| *** ddieterly is now known as ddieterly[away] | 16:52 | |
| *** _nadya_ has joined #openstack-infra | 16:52 | |
| *** senk has joined #openstack-infra | 16:52 | |
| clarkb | ya I am seeing the SDs happen to all of the instances | 16:52 |
| clarkb | er all of the backend instances. So this doesn't appear to be any single backend being unruly | 16:52 |
| *** piet has joined #openstack-infra | 16:52 | |
| *** roxanaghe has quit IRC | 16:52 | |
| clarkb | that coupled with the cDs makes it hard for me to think our service is broken, it just can't tcp | 16:53 |
| clarkb | fungi: anything else you can think that would be worth checking? | 16:54 |
| *** roxanaghe has joined #openstack-infra | 16:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Skip test creating provider network if one exists https://review.openstack.org/363715 | 16:54 |
| * mordred hits head on wall | 16:54 | |
| clarkb | wait | 16:54 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Disable compute025.vanilla https://review.openstack.org/363806 | 16:54 |
| clarkb | mordred: don't you want to test that regardless? | 16:54 |
| mordred | clarkb: it's not possible to test if the devstack has already created a network using that physical network | 16:55 |
| fungi | clarkb: my usual go-to would be analyzing the kernel's interface buffer/queue utilization... it's been a while so i'm rereading how one does it these days | 16:55 |
| mordred | clarkb: in order to execute that test now, I need to create a new job that configures devstack differently | 16:55 |
| clarkb | mordred: you can't have two physical networks? | 16:55 |
| clarkb | that seems broken | 16:55 |
| mordred | clarkb: you cannot have two neutron networks that map to the same underlying physical network | 16:55 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Disable compute008.vanilla https://review.openstack.org/363809 | 16:56 |
| clarkb | mordred: ya make another disjoint one | 16:56 |
| clarkb | or maybe even overlapping? | 16:56 |
| mordred | it's not possible | 16:56 |
| clarkb | eg /25 instead of /24? | 16:56 |
| clarkb | mordred: why not? | 16:56 |
| mordred | that's not the conflict | 16:56 |
| mordred | you define a pre-existing physical network in the ml2.conf file by name | 16:56 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Move placement job to nova's check queue https://review.openstack.org/363810 | 16:57 |
| mordred | then, when you create the neutron network object, you say "provider:physical_network = name_defined_in_file" | 16:57 |
| mriedem | sdague: ^ | 16:57 |
| mordred | and that defines the mapping between the neutron entwork and the underlying physical netowrk it represents | 16:57 |
| *** salv-orlando has joined #openstack-infra | 16:57 | |
| mordred | so what I need to do for the shade test is to make a new job that runs devstack without the provider network defined | 16:57 |
| *** tongli has joined #openstack-infra | 16:57 | |
| clarkb | wow | 16:57 |
| mordred | so that we can define it in the test | 16:57 |
| rcarrillocruz | folks, gotta run a few errands, i may be later online should the infracloud being put online today | 16:58 |
| rcarrillocruz | laterz | 16:58 |
| sdague | mordred / clarkb - quick hit on that - https://review.openstack.org/363810 ? | 16:58 |
| *** trown is now known as trown|lunch | 16:58 | |
| *** shamail has left #openstack-infra | 16:59 | |
| sdague | I was apparently slightly uncaffinated this morning when I made the first change | 16:59 |
| fungi | clarkb: jeblair: at least the system interrupts for eth0 only seem to be consuming ~10% of one cpu on the proxy server (and ~1% on each of the backends) | 16:59 |
| zaro | Ok. online reindex testing on review.o.o will commence now: https://etherpad.openstack.org/p/gerrit-online-index-testing | 17:00 |
| zaro | i'm going to hijack #openstack-meeting channel for this so join me over there if interested. | 17:01 |
| zaro | opps meant openstack-sprint channel | 17:01 |
| *** tongli has quit IRC | 17:02 | |
| mordred | sdague: done | 17:03 |
| clarkb | fungi: let me know if I can help somehow with the linux tcp investigating | 17:04 |
| clarkb | the cloen nova over and over again on git03 is going fine | 17:04 |
| mordred | sc68cal: just for your amusement - it's "provider:physical_network" ... not "physical_network" ... also, I decided to change the test to be for provider:network_type instead, in case devstack decides it wants to change the network name | 17:05 |
| *** yamahata has quit IRC | 17:05 | |
| clarkb | fungi: also I think there is some weight to your idea that our caches improve things because the client disconnects at least all seem to happen after 2 minutes | 17:06 |
| clarkb | fungi: in theory our caches will reduce the total connection time to something much shorter | 17:06 |
| *** akshai has quit IRC | 17:07 | |
| Shrews | mordred: we need a new job for the one shade test?? | 17:07 |
| Shrews | did i read that sb right? | 17:07 |
| clarkb | Shrews: I think you can likely replace your current job with the new job but yes | 17:07 |
| *** jerryz has joined #openstack-infra | 17:08 | |
| *** derekh has quit IRC | 17:09 | |
| *** spzala has quit IRC | 17:09 | |
| pabelanger | Shrews: jeblair: any thoughts about exposing a CLI command to individually launch a node? Basically to be used for the purpose of debugging cloud failures. Today, I have a shade script to do it, but not exactly how nodepool will launch a server | 17:10 |
| *** sputnik13 has joined #openstack-infra | 17:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Re-enable test_failed_change_at_head https://review.openstack.org/363821 | 17:12 |
| *** akshai has joined #openstack-infra | 17:12 | |
| pabelanger | Shrews: jeblair: or expanding out auto hold feature to include launch failures (scp nodepool-script / ready-script) | 17:12 |
| mordred | Shrews: if we want to execute that test, yes | 17:12 |
| *** tongli has joined #openstack-infra | 17:12 | |
| jeblair | pabelanger: can we do that as a v3 TODO? i think it will be easier then | 17:12 |
| *** amitgandhinz has quit IRC | 17:12 | |
| mordred | Shrews, clarkb: honestly, I think testing that shade works in clouds wiht provider networks is more important than testing that shade can create a provider network | 17:12 |
| pabelanger | jeblair: Of course | 17:13 |
| mordred | so I would not recommend replacing the job with a non-provider network config | 17:13 |
| *** ddieterly[away] is now known as ddieterly | 17:13 | |
| *** amitgandhinz has joined #openstack-infra | 17:13 | |
| clarkb | pabelanger: are there specific instances where nova boot hasn't been sufficient? might help shape the way the command works (fwiw I have had really good luck just noav booting things) | 17:13 |
| mordred | BUT - I do see value in testing that shade works witha provider-network config and also with a config that doesn't have one and requires floating ips ... so I could see making both of those | 17:13 |
| *** annegentle has joined #openstack-infra | 17:13 | |
| mordred | and don't think it would be a waste of energy | 17:13 |
| *** akshai has quit IRC | 17:14 | |
| clayton | it appears dmsimard is on pto, does anyone know if his ARA tool requires Ansible 2.x? | 17:14 |
| pabelanger | clarkb: no, thats what I do today, but with shade. I just figure it would be easier to debug failures, using the same code path we use to launch server. | 17:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add OSA keystone uwsgi functional tests https://review.openstack.org/363640 | 17:15 |
| clarkb | pabelanger: maybe? I haven't experienced any cases where I need nodepool to boot an instance to debug why it was failing so I don't know | 17:15 |
| *** markvoelker has joined #openstack-infra | 17:15 | |
| *** e0ne has quit IRC | 17:16 | |
| pabelanger | clarkb: Ya, it is rare. But that's what I am trying to figure out with rax-iad right now. Server boots, we cannot connect to it via SSH | 17:16 |
| clarkb | was wondering if you had run into that with something like writing /etc/nodepool contents | 17:16 |
| fungi | ooh, sorry, just got distracted by a large delivery. back now and looking at the git servers some more | 17:16 |
| mordred | clayton: I do not know - but I would be surprised if it didn't | 17:16 |
| *** tongli has quit IRC | 17:17 | |
| clarkb | pabelanger: thats likely not related to nodepool at all but instead glean? | 17:17 |
| clarkb | pabelanger: did you attach a config drive? | 17:17 |
| mordred | clayton: it's new enough that I'd be very surprised if he wrote it for 1.9 | 17:17 |
| clayton | mordred: that was my thought also. the callback api changed between 1.9 and 2.x | 17:17 |
| pabelanger | clarkb: possible, yes using config-drive | 17:17 |
| mordred | clayton: yah | 17:17 |
| clayton | unfortunately we're still on 1.9 | 17:17 |
| *** tphummel has quit IRC | 17:17 | |
| * mordred recommends upgrading ;) | 17:17 | |
| *** akshai has joined #openstack-infra | 17:17 | |
| clarkb | fungi: fwiw I am basically ready to write a support ticket and see what rax says, but will wait on doing that | 17:18 |
| pabelanger | clarkb: agreed, I don't think it is nodepool either, but would make things easy if we could tell nodepool to keep that failure online then toss the UUID over the wall to rackspace. | 17:18 |
| pabelanger | right now, I'm manually trying to reproduce, if this server will boot | 17:18 |
| *** rossella_s has quit IRC | 17:18 | |
| *** fguillot_ has quit IRC | 17:18 | |
| clarkb | pabelanger: oh is it not consistent? gotcha that would be one case where it would be helpful (though you'd need more of an auto hold than a boot command) | 17:18 |
| *** lucasagomes is now known as lucas-dinner | 17:19 | |
| *** mhickey has quit IRC | 17:19 | |
| *** rossella_s has joined #openstack-infra | 17:19 | |
| fungi | `ethtool -S eth0` is decidedly unhelpful on xen guests, it seems | 17:19 |
| jeblair | clarkb, fungi: are we still interested in having nodepool image builds attempt to happen at a given time of day? or would logic like "rebuild if the image is older than X hours, regardless of time of day" work? (obvs that would probably start out being every 24 hours at the same time, but would probably fairly quickly start to move around). cc: Shrews | 17:19 |
| openstackgerrit | Merged openstack-infra/project-config: Move placement job to nova's check queue https://review.openstack.org/363810 | 17:20 |
| clarkb | jeblair: the benefit to having it happen at a certain time when snapshot/upload was consistent was that if an image had problems we could ensure they wouldn't start until roughly when the humans that could address those problems were present | 17:20 |
| fungi | jeblair: i think either would work out for us, except that we might go into image rebuild loops when there are issues, unless we also added a throttle | 17:20 |
| mordred | jeblair: I like rebuild if image is older ... also, a while back I was thinking that nodepool considering images older than X hours as not being viable to boot content on would be another neat knob | 17:20 |
| sc68cal | mordred: ok. :) | 17:20 |
| clarkb | jeblair: but now glance image upload is so unreliable that we can't depend on that and also we have more people around the globe making that less important | 17:20 |
| pabelanger | clarkb: Yup | 17:20 |
| sc68cal | mordred: isn't openstack f.u.n.? | 17:21 |
| mordred | sc68cal: so f.u.n. | 17:21 |
| clarkb | mordred: if you put that in place most of our providers would stop working :) | 17:21 |
| jeblair | fungi: yeah, though i'm considering auto image rebuild loops a feature -- since right now, clarkb frequently manually executes a human-powered image rebuild loop :) | 17:21 |
| jeblair | well, usually upload, not build, but sometimes build too. | 17:22 |
| pabelanger | \o/ | 17:22 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic?from=1472577734067&to=1472664134067&var-provider=All | 17:22 |
| AJaeger | project-config cores, could you review the storyboard publishing change, please? https://review.openstack.org/363795 | 17:22 |
| clarkb | the build problems tend to be more consistent until addressed where as upload tends to work eventually if you try hard enough | 17:22 |
| pabelanger | officially 24 hours and osic-cloud1 has reported 0 launch node failures | 17:22 |
| *** itisha has joined #openstack-infra | 17:22 | |
| pabelanger | cloudnull: ^ | 17:22 |
| jeblair | clarkb: right, with the notable exception of 'git clone/jeepyb' :) | 17:22 |
| *** ddieterly is now known as ddieterly[away] | 17:22 | |
| clarkb | jeblair: so maybe builds should have a hard limit of retries but uploads could continue with a backoff | 17:22 |
| *** tphummel has joined #openstack-infra | 17:22 | |
| *** dizquierdo has quit IRC | 17:22 | |
| cloudnull | pabelanger: ++ | 17:22 |
| fungi | clarkb: depends. we have plenty of transient image build problems too (especially around caching git repos when new projects are being added) | 17:23 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic?from=1472577734067&to=1472664134067 is the actually URL | 17:23 |
| cloudnull | thats even after a massive spinup last evening | 17:23 |
| clarkb | fungi: thats true | 17:23 |
| fungi | probably what jeblair also meant | 17:23 |
| jeblair | words | 17:23 |
| * fungi gets back to reading very dry rhel 7 performance tuning guides | 17:23 | |
| clarkb | I think if we did a thing that retried a failed build once (or some small number of times) then waited again for some period of image is X time old retry that would work well | 17:23 |
| *** akshai has quit IRC | 17:23 | |
| pabelanger | cloudnull: Ya, seems to be holding its own well | 17:24 |
| mtreinish | AJaeger: any idea what I'm missing on: https://review.openstack.org/#/c/363297/3 it's probably something so obvious I'm blind to it :) | 17:24 |
| clarkb | then for uploads have them retry over and over with a backoff between uploads that is reset whenever a new image is built | 17:24 |
| cloudnull | time to find the next break point :) | 17:24 |
| *** tongli has joined #openstack-infra | 17:24 | |
| clarkb | is that complicated enough? :) | 17:24 |
| mtreinish | AJaeger: it's failing on the merge template check: http://logs.openstack.org/97/363297/3/check/gate-project-config-layout/0b2b23a/console.html#_2016-08-31_17_04_51_494298 | 17:24 |
| *** kzaitsev_mb has quit IRC | 17:24 | |
| fungi | for those who need some good bedtime reading material... https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Performance_Tuning_Guide/chap-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Networking.html | 17:24 |
| *** pt_15 has joined #openstack-infra | 17:24 | |
| jeblair | clarkb: we could add limits, but honestly, i'd be okay with our dedicated image builder continually running trying to build an image, as long as it didn't starve other image builds (ie, it looped through all of them before retrying) | 17:24 |
| pabelanger | hopefully we can get osic-cloud8 up later today, once the fires are out | 17:24 |
| clarkb | jeblair: ya that would probably work too we would just churn the disk of that machine if/when things break | 17:25 |
| jeblair | pabelanger: also https://review.openstack.org/363756 | 17:25 |
| clarkb | but hey cloud :P | 17:25 |
| pabelanger | jeblair: woah | 17:25 |
| *** akshai has joined #openstack-infra | 17:25 | |
| jeblair | i think that's ready to go if anyone wants to babysit :) | 17:25 |
| pabelanger | +3 | 17:25 |
| fungi | clarkb: your idea holds merit: "Network performance problems are most often the result of hardware malfunction or faulty infrastructure. Red Hat highly recommends verifying that your hardware and infrastructure are working as expected before beginning to tune the network stack." | 17:25 |
| pabelanger | Yup, have some cycles to watch | 17:25 |
| fungi | clarkb: red hat says we should check out our network hardware first | 17:26 |
| clarkb | fungi: wfm :) | 17:26 |
| *** akshai has quit IRC | 17:26 | |
| clarkb | pabelanger: I am trying to rereview cloud8 stack now that network info is updated | 17:26 |
| clarkb | also I really want it to be cloud9 | 17:26 |
| *** _nadya_ has quit IRC | 17:26 | |
| pabelanger | clarkb: cool, I added both networks like we did with cloud1 | 17:27 |
| clarkb | yup looks goood | 17:27 |
| mordred | clarkb: link? | 17:27 |
| *** ddieterly[away] is now known as ddieterly | 17:27 | |
| clarkb | mordred: topic:osic-cloud8 | 17:28 |
| mordred | thanks | 17:28 |
| *** akshai has joined #openstack-infra | 17:28 | |
| *** tongli has quit IRC | 17:28 | |
| mordred | pabelanger: +A | 17:29 |
| AJaeger | mtreinish: looking... | 17:29 |
| mordred | pabelanger: I left the nodepool change cause it has 2 +2s | 17:29 |
| mordred | and likely wants to be watched | 17:29 |
| pabelanger | ya | 17:29 |
| pabelanger | we need to restart nodepool-builder for that too | 17:29 |
| clarkb | jeblair: the new scaling pain point definitely seems to be image upload reliability as we add more and more regions fwiw. So I think changing how we do those uploads is a good thing | 17:29 |
| clarkb | pabelanger: are you going to +A that one or should I? | 17:30 |
| clarkb | then I need to submit a rax ticket | 17:31 |
| pabelanger | clarkb: sure if you want to | 17:31 |
| *** tongli_ has joined #openstack-infra | 17:31 | |
| clarkb | done | 17:31 |
| fungi | the featureset for the nic driver in these xen guests is pretty limited in what we can adjust. it does at least have tso and gso support (and both are enabled) | 17:32 |
| fungi | gro as well | 17:32 |
| * AJaeger sends two "-" signs to mtreinish to fix his change | 17:32 | |
| clarkb | fungi: so tl;dr of ticket would be noticing tcp disconnects that both sides of connection does not expect for git.openstack.org (uuid here) between hosts that are external to rax and to those within rax using their public IPs. Provide list of example IP addrs and timestemps for disconnects then cross fingers? | 17:32 |
| clarkb | fungi: anything else you think we should add to that | 17:32 |
| *** manjeets_ has quit IRC | 17:34 | |
| fungi | clarkb: it's a shot in the dark, but sure. we should just expect a slow and fairly unsatisfying response | 17:34 |
| *** tongli_ has quit IRC | 17:34 | |
| *** watanabe_isao has joined #openstack-infra | 17:34 | |
| *** yamamoto has quit IRC | 17:34 | |
| *** tongli_ has joined #openstack-infra | 17:35 | |
| fungi | i've not had much luck reporting nuanced network issues to rackspace in the past, except when they've been able to spot a noisy neighbor on the same hypervisor host | 17:35 |
| * fungi remembers the ages we went round and round on their ipv6 dscp issues | 17:36 | |
| *** dtantsur is now known as dtantsur|afk | 17:36 | |
| *** watanabe_isao has quit IRC | 17:36 | |
| clarkb | my git03 clone against locahost https is continuing to be happy | 17:37 |
| fungi | where openssh changed their qos defaults and rackspace's gear was just outright dropping packets for ssh sessions after the qos shift in the session | 17:37 |
| *** abregman has joined #openstack-infra | 17:37 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 17:37 |
| *** bradjones has quit IRC | 17:38 | |
| openstackgerrit | Merged openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 17:38 |
| *** bradjones has joined #openstack-infra | 17:38 | |
| *** tonytan_brb has quit IRC | 17:38 | |
| *** pcaruana has joined #openstack-infra | 17:39 | |
| *** tongli_ has quit IRC | 17:39 | |
| *** bradjones is now known as Guest30374 | 17:39 | |
| fungi | i suppose we could up our rmem_default (it's currently 212992 while rmem_max is 33554432). it'll potentially make latency a little worse but may smooth out some bumps if we're briefly overrunning the buffer at times | 17:39 |
| openstackgerrit | Merged openstack-infra/project-config: Enable infracloud servers in Nodepool https://review.openstack.org/363756 | 17:39 |
| *** tongli has joined #openstack-infra | 17:40 | |
| fungi | clarkb: jeblair: looked into the mysterious xen_netfront/xennet errors in dmesg yet? | 17:41 |
| mtreinish | AJaeger: haha, yeah that is pretty obvious now that you've pointed it out. I knew it was gonna be a dumb mistake like that | 17:42 |
| fungi | i have to say i find their wording amusing if nothing else | 17:42 |
| fungi | [Wed Aug 31 17:39:12 2016] xen_netfront: xennet: skb rides the rocket: 20 slots | 17:42 |
| *** degorenko is now known as _degorenko|afk | 17:42 | |
| * fungi can't imagine a more cryptic error) | 17:42 | |
| clarkb | fungi: no, I checked dmesg and it was mostly just auth stuff | 17:42 |
| clarkb | didn't see that | 17:42 |
| *** yamahata has joined #openstack-infra | 17:42 | |
| *** electrofelix has quit IRC | 17:42 | |
| fungi | http://www.brendangregg.com/blog/2014-09-11/perf-kernel-line-tracing.html | 17:42 |
| *** sambetts is now known as sambetts|afk | 17:43 | |
| *** ddieterly is now known as ddieterly[away] | 17:43 | |
| *** tongli has quit IRC | 17:43 | |
| *** markvoelker has quit IRC | 17:43 | |
| fungi | "It's a driver bug with TSO. A very large skb can span too many pages (more than 16) to be put in the driver ring buffer. One workaround is "sudo ethtool -K eth0 tso off", for your interface. There's plenty of articles about this on the Internet, and they are easy to find thanks to our mysterious message." | 17:44 |
| *** tongli has joined #openstack-infra | 17:44 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Remove image-update cron https://review.openstack.org/363837 | 17:44 |
| clarkb | huh do we want to try that before submitting the rax ticket? | 17:44 |
| *** rwsu has quit IRC | 17:44 | |
| fungi | bug 1317811 | 17:45 |
| openstack | bug 1317811 in linux (Ubuntu Utopic) "Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"" [Medium,Fix released] https://launchpad.net/bugs/1317811 | 17:45 |
| clarkb | fungi: we run centos on these machines fwiw | 17:45 |
| pabelanger | clarkb: okay, reproduced my network issue in rax-iad. Server online but no SSH access | 17:45 |
| fungi | clarkb: yep, just a datapoint. different distros, same kernel driver though | 17:46 |
| clarkb | ya | 17:46 |
| *** tongli has quit IRC | 17:46 | |
| *** tongli has joined #openstack-infra | 17:46 | |
| clarkb | fungi: seems like it would be simple to disable tso, keep an eye on cpu utilization and see if haproxy is happier | 17:47 |
| *** shashank_hegde has joined #openstack-infra | 17:48 | |
| fungi | clarkb: yeah, i'm leaning that way but still reading | 17:48 |
| fungi | i mean, tso exists for a reason. without it, our cpu utilization may go up a lot. on the other hand, we have way more available processing power on this machine than we use even at peak | 17:49 |
| oomichi | pleia2: hello, thanks for reviewing. | 17:50 |
| *** tongli has quit IRC | 17:50 | |
| clarkb | devstack-gate cores can I get reviews on https://review.openstack.org/#/c/312647/ thats another thing that will help increase the speed of our test jobs | 17:50 |
| oomichi | pleia2: can you take a look at another https://review.openstack.org/#/c/358149/ ? | 17:50 |
| clarkb | fungi: yup, I think we will definitely want to watch cpu utilization closely | 17:50 |
| *** tongli has joined #openstack-infra | 17:50 | |
| pleia2 | oomichi: I'll add it to my list | 17:50 |
| pabelanger | clarkb: we don't manage security groups in rackspace, do we? | 17:51 |
| clarkb | pabelanger: there are no security groups in rackspace so no | 17:51 |
| *** dimtruck is now known as zz_dimtruck | 17:51 | |
| pabelanger | clarkb: Ya, I thought that was the case. | 17:51 |
| pabelanger | so, networking issue or iptables for rax-iad | 17:52 |
| jeblair | clarkb, fungi: if you have a sec to give a quick "+1 in principle" to https://review.openstack.org/363837 that would be nice | 17:52 |
| jeblair | mordred: ^ | 17:52 |
| clarkb | jeblair: can trade you review for 312647? just another small tweak to speed up our jobs a little bit | 17:52 |
| jeblair | clarkb: we have xfs? | 17:53 |
| *** Na3iL has quit IRC | 17:53 | |
| clarkb | jeblair: we don't anymore beacuse we switched to dib build images everywhere, but centos defaults to xfs and I thin fedora does too | 17:54 |
| *** rvasilets__ has left #openstack-infra | 17:54 | |
| clarkb | jeblair: so people using devstack-gate on not our images may run into it | 17:54 |
| *** ihrachys has quit IRC | 17:54 | |
| pabelanger | rcarrillocruz: I manually kill some ansible-playbook processes on puppetmaster.o.o, I think there were 2 different playbooks running on infracloud between the 2 crontab processes | 17:55 |
| clarkb | actually that statement isn't entirely correct | 17:55 |
| clarkb | we switch to dib build "minimal" images which don't rely on the prebuilt distro iamges which ship with an opinion on fs | 17:55 |
| clarkb | if you use the non minimal centos dib buulds you will get xfs | 17:55 |
| *** _nadya_ has joined #openstack-infra | 17:55 | |
| jeblair | clarkb: +2 but i rechecked since i didn't see current logs | 17:56 |
| jeblair | (so couldn't double check that it worked) | 17:56 |
| phschwartz | fungi: who is best to talk to, I just realized today is the final day to get tickets with the ATC code only to be shocked that I never got my atc code | 17:56 |
| clarkb | jeblair: sounds good | 17:57 |
| *** akshai has quit IRC | 17:57 | |
| mordred | jeblair: ++ | 17:57 |
| fungi | phschwartz: i'll check my logs | 17:57 |
| phschwartz | fungi: ty | 17:58 |
| *** rbrndt has quit IRC | 17:58 | |
| *** tongli has quit IRC | 17:58 | |
| pabelanger | jeblair: clarkb: I think 363837 is great! Thanks for doing that | 17:58 |
| *** tongli has joined #openstack-infra | 17:58 | |
| pleia2 | oomichi: just a request to add some documentation about this change to the README | 17:58 |
| *** e0ne has joined #openstack-infra | 17:59 | |
| fungi | phschwartz: i sent it to both your linux.vnet.ibm.com and progmad.com e-mail addresses on june 20 | 17:59 |
| phschwartz | fungi: ah, found it. It was flagged as spam by ibm's mail server :( | 17:59 |
| phschwartz | fungi: ty for looking for me | 18:00 |
| *** dkehn has quit IRC | 18:00 | |
| *** tongli_ has joined #openstack-infra | 18:00 | |
| fungi | phschwartz: use it now, before it no longer covers 100% of teh registration cost | 18:00 |
| *** dkehn_ has quit IRC | 18:00 | |
| fungi | which is, like, tomorrow | 18:00 |
| pabelanger | jeblair: clarkb: does this mean moving forward we submit individual image builds to nodepool-builder, rather then all at one? | 18:00 |
| oomichi | pleia2: so fast review, thanks :) OK, I will update README ASAP | 18:00 |
| clarkb | pabelanger: its more subtle then that. The builders will used a shared db (zk) to store info on when they last built and uploaded images. When they see that they need to be updated they will start doing the work | 18:01 |
| clarkb | pabelanger: so its less about instructing a builder from some central brain and more about decentralized coordination based on synchronized data | 18:01 |
| *** zz_dimtruck is now known as dimtruck | 18:01 | |
| pabelanger | clarkb: Right, thats much better | 18:02 |
| *** trown|lunch is now known as trown | 18:02 | |
| *** tongli has quit IRC | 18:03 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add puppet-lpmqtt project https://review.openstack.org/363297 | 18:03 |
| *** sarob has joined #openstack-infra | 18:03 | |
| *** tphummel has quit IRC | 18:04 | |
| AJaeger | mtreinish: do you have a governance change for that? | 18:04 |
| *** spzala has joined #openstack-infra | 18:05 | |
| *** akshai has joined #openstack-infra | 18:06 | |
| *** tphummel has joined #openstack-infra | 18:07 | |
| *** dkehn has joined #openstack-infra | 18:07 | |
| *** pt_15 has quit IRC | 18:08 | |
| *** salv-orl_ has joined #openstack-infra | 18:08 | |
| *** larainema has quit IRC | 18:08 | |
| mtreinish | AJaeger: not yet, I can push one up for it right now | 18:08 |
| AJaeger | mtreinish: please do - and then amend your two changes and add Needed-By | 18:09 |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/jenkins-job-builder: Add support for Parameterized Scheduler Plugin https://review.openstack.org/353165 | 18:09 |
| *** jkilpatr has quit IRC | 18:10 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Allow str for ip_version param in create_subnet https://review.openstack.org/363846 | 18:10 |
| *** srobert has joined #openstack-infra | 18:10 | |
| *** salv-orlando has quit IRC | 18:10 | |
| *** pvinci has quit IRC | 18:11 | |
| *** tongli has joined #openstack-infra | 18:11 | |
| *** larainema has joined #openstack-infra | 18:11 | |
| *** pt_15 has joined #openstack-infra | 18:12 | |
| *** dkehn_ has joined #openstack-infra | 18:12 | |
| *** eeiden has quit IRC | 18:13 | |
| *** stewie925 has quit IRC | 18:13 | |
| *** wcriswell has quit IRC | 18:13 | |
| clarkb | fungi: find anything else? want to go ahead and disable tso? | 18:14 |
| *** skipp has quit IRC | 18:14 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: require CLA for release-tools https://review.openstack.org/363851 | 18:14 |
| *** sarob has quit IRC | 18:14 | |
| *** tongli_ has quit IRC | 18:14 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add puppet-lpmqtt project https://review.openstack.org/363297 | 18:14 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add lpmqtt project https://review.openstack.org/363296 | 18:14 |
| fungi | clarkb: sorry, got pulled into troubleshooting two other things in the middle of this. getting back to it now | 18:15 |
| AJaeger | team, FYI, I have serious problems to reach even docs.openstack.org - there might be other networking problems with Rackspace... | 18:15 |
| AJaeger | a reload helps normally | 18:15 |
| mtreinish | AJaeger: ^^^ | 18:15 |
| AJaeger | fungi, 363297 and 363296 need your review | 18:15 |
| AJaeger | mtreinish: once fungi is happy, I'll review ;) | 18:16 |
| AJaeger | thanks, mtreinish | 18:16 |
| *** florianf has quit IRC | 18:16 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/bugdaystats: Add "daily" argument to update_stats() https://review.openstack.org/358149 | 18:16 |
| oomichi | pleia2: ^^^: thanks for your reviewing. updated | 18:17 |
| pleia2 | oomichi: thanks for adding that :) | 18:17 |
| *** wcriswell has joined #openstack-infra | 18:17 | |
| AJaeger | mtreinish: +4 | 18:18 |
| pleia2 | oomichi: hm, wouldn't this be optional? | 18:18 |
| *** skipp has joined #openstack-infra | 18:18 | |
| *** tonytan4ever has joined #openstack-infra | 18:18 | |
| oomichi | pleia2: +1 for making it optional :) | 18:18 |
| oomichi | pleia2: please give me -1 on the patch | 18:19 |
| pleia2 | oomichi: ok | 18:19 |
| mtreinish | AJaeger: heh, is fungi ever unhappy :) | 18:19 |
| mtreinish | AJaeger: cool, thanks | 18:19 |
| fungi | mtreinish: these hawaiian shirts reflect my state of mind | 18:20 |
| AJaeger | ;) | 18:20 |
| AJaeger | project-config cores, could you review the storyboard publishing change so that Zara has documents, please? https://review.openstack.org/363795 | 18:21 |
| * AJaeger waves good bye | 18:21 | |
| *** e0ne has quit IRC | 18:21 | |
| pleia2 | oh yay, storyboard docs | 18:21 |
| pabelanger | infracloud-vanilla lives: http://logs.openstack.org/05/293305/49/check/gate-tempest-dsvm-neutron-linuxbridge/20aacae/console.html | 18:22 |
| pabelanger | job failed however | 18:22 |
| pabelanger | rcarrillocruz: looks like quota issues in infracloud-vanilla | 18:22 |
| *** e0ne has joined #openstack-infra | 18:22 | |
| *** hashar has joined #openstack-infra | 18:23 | |
| *** jkilpatr has joined #openstack-infra | 18:23 | |
| pabelanger | http://mirror.regionone.infracloud-vanilla.openstack.org/ | 18:23 |
| pabelanger | that's the issue | 18:23 |
| pabelanger | we have no AFS data | 18:23 |
| pabelanger | rcarrillocruz: ^ | 18:23 |
| *** hashar is now known as hasharAway | 18:23 | |
| *** ddieterly[away] is now known as ddieterly | 18:24 | |
| clarkb | pabelanger: possible firewall issues? | 18:24 |
| pabelanger | clarkb: checking | 18:24 |
| clarkb | blocking our afs udp packets/ | 18:24 |
| *** shardy is now known as shardy_afk | 18:24 | |
| pabelanger | Oh | 18:25 |
| *** shardy_afk has quit IRC | 18:25 | |
| pabelanger | I don't think the server was reboot after coming online | 18:25 |
| pabelanger | AFS module /lib/modules/3.13.0-93-generic/fs/openafs.ko does not exist. | 18:25 |
| clarkb | pabelanger: so the modules are not loaded? launch node should always restart the instances... | 18:25 |
| pabelanger | clarkb: rcarrillocruz used cloud-launcher | 18:25 |
| pabelanger | so this is the likely issue | 18:26 |
| pabelanger | let me reboot and see if that fixes it | 18:26 |
| fungi | clarkb: at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/comments/22 it's suggested that disabling scatter-gather similarly solved it | 18:26 |
| openstack | Launchpad bug 1317811 in linux (Ubuntu Utopic) "Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"" [Medium,Fix released] | 18:26 |
| clarkb | kk | 18:26 |
| fungi | clarkb: which `ethtool -k eth0`says is also enabled | 18:26 |
| *** zaro has quit IRC | 18:26 | |
| *** sandanar_ has joined #openstack-infra | 18:27 | |
| clarkb | fungi: I like that comments also confirms mtu is not at fault here since we are also 1500 mtu | 18:27 |
| *** Sukhdev has joined #openstack-infra | 18:27 | |
| *** zaro has joined #openstack-infra | 18:27 | |
| fungi | right, i checked ip link show eth0 there to be sure | 18:27 |
| *** sarob has joined #openstack-infra | 18:28 | |
| *** _nadya_ has quit IRC | 18:28 | |
| *** sandanar_ has quit IRC | 18:28 | |
| fungi | supposedly linux 3.17 has a workaround, but no idea if that's backported to rhel 7's 3.10 | 18:28 |
| *** rbrndt has joined #openstack-infra | 18:28 | |
| clarkb | turning off sg will also likely lead to more cpu utilization or at least more blocking for the reads and writes ya? | 18:28 |
| fungi | yep | 18:28 |
| pabelanger | clarkb: still no kernel module. Trying to find out why | 18:30 |
| *** sandanar has quit IRC | 18:30 | |
| pabelanger | likely want to revert until we confirm mirror is working | 18:30 |
| clarkb | pabelanger: did the iamge we based that on use the hardware support for ubuntu which doesn't have working afs module? | 18:30 |
| pabelanger | clarkb: I am not sure, rcarrillocruz launched the mirror this time | 18:31 |
| clarkb | pabelanger: what is the kernel version? | 18:31 |
| *** amotoki has quit IRC | 18:31 | |
| pabelanger | Linux mirror 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | 18:31 |
| clarkb | I think tahts the right one | 18:31 |
| clarkb | fungi: ethtool -k eth0 to see the features enabled? | 18:33 |
| rcarrillocruz | I launched it with the trusty dib image | 18:33 |
| *** vhosakot has quit IRC | 18:34 | |
| openstackgerrit | Merged openstack-infra/shade: Skip test creating provider network if one exists https://review.openstack.org/363715 | 18:34 |
| pabelanger | Oh | 18:34 |
| pabelanger | rcarrillocruz: ya, we've never tested that | 18:34 |
| pabelanger | rcarrillocruz: for now, I've just used ubuntu cloud image | 18:34 |
| clarkb | fungi: I say lets try that and watch the cpu utilization and grep -v -- -- /var/log/haproxy.log | 18:34 |
| *** _nadya_ has joined #openstack-infra | 18:35 | |
| *** rockyg has quit IRC | 18:35 | |
| fungi | clarkb: yeah | 18:35 |
| *** yamamoto has joined #openstack-infra | 18:35 | |
| *** vhosakot has joined #openstack-infra | 18:35 | |
| clarkb | fungi: you going to run the command or should I? | 18:35 |
| pabelanger | so, I see some puppet error around openafs | 18:35 |
| pabelanger | openafs-modules-dkms was also missing | 18:36 |
| pabelanger | so, I don't think we launched the server properly | 18:36 |
| fungi | clarkb: i can... did you have a reliable way to reproduce the issue other than tailing haproxy logs? | 18:36 |
| mgagne | clarkb: alright, mtl01 is ready to go. We tested all compute nodes and they didn't show any problem. | 18:36 |
| *** crst has joined #openstack-infra | 18:36 | |
| pabelanger | and we should delete and reprovision again | 18:36 |
| *** dprince has quit IRC | 18:36 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:36 | |
| *** crst is now known as Guest5541 | 18:36 | |
| clarkb | fungi: no just tailing logs. I also have a local clone loop I can run against it which was good for confirming the fail here was unexpected and unexpected in haproxy | 18:36 |
| clarkb | fungi: I can restart my clone loop as soon as you tell me sg is off | 18:36 |
| fungi | clarkb: should we try disabling tso first, or sg, or both? | 18:37 |
| clarkb | fungi: I think sg | 18:37 |
| clarkb | as the tso thing seems more related to funny MTUs which we don't have | 18:37 |
| *** dtardivel has quit IRC | 18:37 | |
| fungi | agreed, that seems like it would be slightly lower-impact too | 18:37 |
| fungi | clarkb: oh, sg is needed for tso | 18:38 |
| clarkb | ah | 18:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Enable infracloud servers in Nodepool" https://review.openstack.org/363881 | 18:38 |
| pabelanger | rcarrillocruz: clarkb^ | 18:38 |
| clarkb | so both go off if we turn off sg? | 18:38 |
| rcarrillocruz | pabelanger: i was not sure which image i should based it on | 18:38 |
| fungi | so it's possible they saw disabling sg fix it because disabling sg disables tso too | 18:38 |
| rcarrillocruz | asked it and got to use dib | 18:38 |
| rcarrillocruz | which seems | 18:38 |
| clarkb | fungi: ya | 18:38 |
| rcarrillocruz | to lack features to have afs working :/ | 18:38 |
| rcarrillocruz | so, kernel not having afs, no? | 18:38 |
| fungi | clarkb: anyway, http://paste.openstack.org/show/565280/ | 18:38 |
| pabelanger | rcarrillocruz: what I've used for the last 3 mirrors: https://cloud-images.ubuntu.com/trusty/current/ | 18:39 |
| rcarrillocruz | k | 18:39 |
| fungi | disabling sg also disabled tso and gso | 18:39 |
| pabelanger | rcarrillocruz: ya, openafs is missing kernel modules | 18:39 |
| rcarrillocruz | then leave that with me | 18:39 |
| rcarrillocruz | i'll nuke the image on the openstackci tenant | 18:39 |
| rcarrillocruz | upload from cloud images | 18:39 |
| rcarrillocruz | and launch it again | 18:39 |
| clarkb | fungi: my cloen loop is running as is my tail piped through grep -v -- -- | 18:39 |
| rcarrillocruz | i'll do when back at home | 18:39 |
| fungi | clarkb: so far cpu utilization seems unchanged | 18:39 |
| pabelanger | rcarrillocruz: we also need to update cloud-launcher to reboot the server after puppet runs | 18:39 |
| rcarrillocruz | i've +2 the change you linked | 18:39 |
| zaro | fungi: are you monitoring http threads on javamelody? | 18:40 |
| zaro | for gerrit | 18:40 |
| pabelanger | rcarrillocruz: we also have a quota issue, going to dig into that now | 18:40 |
| clarkb | fungi: still seeing a few cD's maybe those settings won't take effect on old connections? I don't know how the kernel deals with that | 18:40 |
| pabelanger | rcarrillocruz: but for bluebox, I manually did it since we don't have ansible modules yet | 18:40 |
| fungi | zaro: no, virtual network interface driver tuning on the git load balancer | 18:40 |
| rcarrillocruz | pabelanger: well, the pre-post provisioning should be really out of the launcher, launcher just provisions | 18:40 |
| rcarrillocruz | i'll do the swap | 18:40 |
| rcarrillocruz | set hostname | 18:40 |
| rcarrillocruz | and reboot | 18:40 |
| fungi | clarkb: yeah, i'm expecting a potentially delayed reaction | 18:40 |
| rcarrillocruz | manually | 18:40 |
| rcarrillocruz | i really need to get back to fix os_server to make the pre/post actions on the launcher | 18:41 |
| rcarrillocruz | i def. want to chat with folks about what i have in mind at mid cycle | 18:41 |
| *** ddieterly is now known as ddieterly[away] | 18:41 | |
| zaro | ok, i wasn't able to find it on javamelody so was wondeering | 18:41 |
| rcarrillocruz | potentially, it can even help for replacing / upgrading servers | 18:41 |
| clarkb | pabelanger: rcarrillocruz it is odd that the package manager wouldn't pull in what we need for dkmsing afs | 18:41 |
| fungi | clarkb: si on Cpu7 seems a little elevated now over what it was hovering at before the change | 18:41 |
| rcarrillocruz | pabelanger: back in a bit, leave the thing with me, i'll do in an hour when back | 18:42 |
| fungi | clarkb: also sy is up a bit | 18:42 |
| *** yamamoto has quit IRC | 18:42 | |
| clarkb | just saw another cD in haproxy logs | 18:42 |
| *** tongli has quit IRC | 18:43 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config: Zun: rename higgins to zun (2) https://review.openstack.org/329247 | 18:43 |
| *** Sukhdev has quit IRC | 18:43 | |
| fungi | i assume the clarkb no new "rides the rocket" in dmesg since 18:37:20 | 18:43 |
| *** chem has quit IRC | 18:43 | |
| clarkb | I haven't checked that but can | 18:43 |
| pabelanger | clarkb: looks like it tried, but failed for some reason | 18:43 |
| fungi | that was a weird started-typing-one-thing-then-decided-to-type-something-else | 18:44 |
| *** _nadya_ has quit IRC | 18:44 | |
| *** tongli has joined #openstack-infra | 18:44 | |
| fungi | clarkb: no new "rides the rocket" in dmesg since 18:37:20 | 18:44 |
| fungi | is what i meant to say | 18:44 |
| *** chem has joined #openstack-infra | 18:44 | |
| clarkb | ah yup I confirm | 18:44 |
| clarkb | unitl I can get my client to disconnect locally with a cD in haproxy we can't really be sure that those clients didn't just disappear on their own | 18:45 |
| fungi | also entirely possible our disconnects are not at all related to packet loss from skb reassembly issues | 18:45 |
| clarkb | yup | 18:45 |
| *** ddieterly[away] is now known as ddieterly | 18:45 | |
| *** eeiden has joined #openstack-infra | 18:45 | |
| fungi | but sy and si on Cpu7 have climbed quite a bit more, which lends credence to your theory that it doesn't take effect for established sockets | 18:46 |
| *** ddieterly is now known as ddieterly[away] | 18:46 | |
| *** esp has quit IRC | 18:46 | |
| oomichi | pleia2: I did re-think more on https://review.openstack.org/#/c/358149/3/README.rst , I feel it is better to avoid it optional | 18:46 |
| pabelanger | clarkb: for some reason, puppet didn't properly install opensafs-client: http://paste.openstack.org/show/565281/ | 18:46 |
| oomichi | pleia2: because the HTML will have to contain both data pages for current one and daily | 18:46 |
| *** tongli_ has joined #openstack-infra | 18:46 | |
| zaro | fungi: we finished the oline index testing and the result shows that reindex worked without error while our small band of testers were banging on it during the index. | 18:47 |
| *** stewie925 has joined #openstack-infra | 18:47 | |
| pleia2 | oomichi: I see, I'll make some time to play around with it then | 18:47 |
| clarkb | fungi: we can also watch that grafana graph that jeblair pointed out /me finds that tab | 18:47 |
| pabelanger | clarkb: and rather then shave the yak with a ubuntu-trusty DIB, I'd say switch back to ubuntu-cloud image for now. Then work on our DIB images for control plan servers strategy | 18:48 |
| pleia2 | oomichi: I'll comment in the review once I finish | 18:48 |
| fungi | zaro: that's great news! i'm looking forward to the maintenance on friday | 18:48 |
| *** tongli has quit IRC | 18:48 | |
| * pleia2 still hopes most folks will have wandered off by rename time on Friday ;) | 18:48 | |
| clarkb | pabelanger: ya that seems fine | 18:48 |
| oomichi | pleia2: thanks again :) | 18:48 |
| pleia2 | it was a bit slow here and there | 18:48 |
| zaro | fungi: only thing to note is that the online reindex pegged the CPU on review-dev instance. so it was slow during the index | 18:48 |
| pleia2 | yeah, we'll want to keep an alert going throughout the reindex | 18:49 |
| *** hasharAway is now known as hashar | 18:49 | |
| clarkb | pabelanger: huh it couldn't get an apt lock | 18:49 |
| clarkb | pabelanger: possibly a dirty image build? | 18:49 |
| pleia2 | noting about potential degraded performance or somesuch | 18:49 |
| zaro | you can view the javamelody logs https://review-dev.openstack.org/monitoring | 18:49 |
| fungi | zaro: yep, i expect performance may be slow | 18:49 |
| *** senk has quit IRC | 18:49 | |
| fungi | but it'll also be a very low-activity time for us | 18:49 |
| pabelanger | clarkb: possible | 18:50 |
| clarkb | pabelanger: rcarrillocruz so why didn't the ansible cloud launch fail ? | 18:50 |
| openstackgerrit | Arie Bregman proposed openstack-infra/zuul: Handle non-valid HEAD https://review.openstack.org/362049 | 18:50 |
| clarkb | I think we need to make sure that cloud launch reboots servers after booting and configing them and it should fail when puppet fails | 18:50 |
| rcarrillocruz | cos it just provisions | 18:50 |
| rcarrillocruz | it doesn't run puppet | 18:50 |
| clarkb | what runs puppet? | 18:50 |
| pabelanger | Oh | 18:51 |
| rcarrillocruz | i ran it the launcher from clouds_layouts | 18:51 |
| zaro | fungi: i assume zuul will not be running? but users will continue to be able to access it? | 18:51 |
| rcarrillocruz | not the launch-node ansible thingy | 18:51 |
| clarkb | rcarrillocruz: yes I know | 18:51 |
| clarkb | rcarrillocruz: I am saying that what you ran has some things we need to address including failing when it should fail and rebooting thei nstace | 18:51 |
| rcarrillocruz | i kicked puppet manually myself afterwards | 18:51 |
| clarkb | uh | 18:51 |
| fungi | zaro: zuul will be running, but we've not had trouble keeping up with volume this week so i don't expect it will be particularly hampered by gerrit slowness | 18:51 |
| clarkb | ok I don't think we should be using that tool for now if it isn't going to do these things for us (we have automated them in launch node because they are important) | 18:51 |
| pabelanger | Agreed, don't want to loose that step right now | 18:52 |
| pabelanger | lose* | 18:52 |
| *** Guest81 has joined #openstack-infra | 18:52 | |
| *** david-lyle_ has joined #openstack-infra | 18:52 | |
| *** david-lyle_ has quit IRC | 18:52 | |
| zaro | fungi: ok, we should at least notify users that they may see "Working" on Gerrit UI during downtime. that's what was happening for us | 18:53 |
| clarkb | fungi: I am not really seeing the SDs and cDs go away. I would've expected htat after 10 minutes or so they should be down to a trickle | 18:53 |
| clarkb | fungi: probably time to submit that rax ticket after all | 18:53 |
| clarkb | though my local cloen is still happy my throughput has fallen a little bit | 18:54 |
| *** Guest81 has quit IRC | 18:54 | |
| clarkb | we were doing 3-7MBps but now its 2-5MBps might be in the margin of internet error though | 18:54 |
| clarkb | oh just had one under 1MBps | 18:55 |
| *** akshai has quit IRC | 18:55 | |
| *** senk has joined #openstack-infra | 18:55 | |
| *** senk has quit IRC | 18:55 | |
| *** akshai has joined #openstack-infra | 18:55 | |
| clarkb | jeblair: also I have roughly been able to correlate SDs in haproxy log to the eresps in grafana | 18:56 |
| *** ijw has joined #openstack-infra | 18:57 | |
| clarkb | fungi: any opposition to me filing that ticket now? I don't want to file it if you think its something we need to fix on our end | 18:57 |
| *** Guest5541 has quit IRC | 18:58 | |
| clarkb | but I need to pop out and find lunch so want to file it before leaving for a bit | 18:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add osic-cloud8 to cloud-launcher https://review.openstack.org/363890 | 18:58 |
| pabelanger | clarkb: fungi: have a moment to review^, adds osic-cloud8 to cloud-launcher so we can setup security groups | 18:58 |
| fungi | clarkb: nah, i'm in favor of the ticket | 18:59 |
| mgagne | clarkb, pabelanger: Let me know when you are ready to enable mtl01. it's ready now. | 18:59 |
| pabelanger | mgagne: sure, give me a few minutes | 18:59 |
| *** ddieterly[away] is now known as ddieterly | 18:59 | |
| *** alexey_weyl has joined #openstack-infra | 18:59 | |
| clarkb | pabelanger: mgagne did we still want to rerun a test job on it first? | 18:59 |
| *** dprince has joined #openstack-infra | 18:59 | |
| clarkb | I can do that after lunch its pretty simple. Or if we are confident the issue is resolved we can just reenable | 19:00 |
| *** Guest81 has joined #openstack-infra | 19:00 | |
| mgagne | clarkb: up to you at this point, better be safe I gues | 19:00 |
| *** mwhahaha has joined #openstack-infra | 19:00 | |
| pabelanger | clarkb: mgagne: Ya, lets do that | 19:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add osic-cloud8 to cloud-launcher https://review.openstack.org/363890 | 19:00 |
| *** kushal has quit IRC | 19:01 | |
| clarkb | pabelanger: mgagne especially since its just before feature freeze we should avoid making tests unhappy as much as possible | 19:01 |
| *** ildikov has joined #openstack-infra | 19:01 | |
| mgagne | hehe | 19:01 |
| *** esp has joined #openstack-infra | 19:01 | |
| clarkb | pabelanger: I can run that test after lunch if you like or if you want to do it just boot the nodepool image then run the reproduce.sh script as jenkins | 19:01 |
| clarkb | I think taht should work | 19:01 |
| pabelanger | Yup, I can do that now | 19:01 |
| clarkb | fungi: jroll JayF 160831-dfw-0002533 has been submitted re the haproxy unhappyness | 19:02 |
| *** hrybacki is now known as hrybacki|afk | 19:02 | |
| JayF | *blink* | 19:03 |
| JayF | I have no context for that? | 19:03 |
| jroll | ditto | 19:03 |
| fungi | clarkb: i guess in a few minutes i'll reenable sg/tso/gso | 19:03 |
| clarkb | JayF: jroll basically tcp connectivity in rax dfw seems to be flaky. Its really noticeable on our git mirror but there are some reports it may be affecting review.o.o and possibly docs hosting as well | 19:03 |
| clarkb | JayF: jroll client and server both notice that a tcpconnection has gone away unexpectedly for both sides | 19:04 |
| jroll | clarkb: fun stuff | 19:04 |
| fungi | it definitely seems to have shifted more work into the cpu as expected, and got rid of the xen_netfront/xennet errors the kernel was spewing, but if it doesn't fix the disconnects i'd rather we switch back to kernel tuning defaults anyway | 19:04 |
| clarkb | JayF: jroll we see it from internet connections and between hsots in the same region so likely in rax and not on the internets | 19:04 |
| clarkb | fungi: yup | 19:04 |
| jroll | clarkb: not sure I can escalate that too much (mostly don't know how/where right off hand) but I'll take a look and see what I can do | 19:04 |
| *** ilyashakhat has joined #openstack-infra | 19:05 | |
| clarkb | ok lunch now | 19:05 |
| fungi | jroll: suspicion is some device in the network (probably in relatively close topological proximity to our git.openstack.org haproxy instance) is closing active flows >120 seconds in age | 19:05 |
| clarkb | jroll: JayF fungi has all the details too if you need more datas | 19:05 |
| fungi | clarkb: i'm curious though what it is about the sessions going through the haproxy hitting this, that our ssh sessions aren't impacted | 19:06 |
| jroll | cool | 19:06 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Enable infracloud servers in Nodepool" https://review.openstack.org/363881 | 19:07 |
| fungi | though there are a number of potential factors, not the least of which are ipv4 vs ipv6 and different qos levels/dscp precedence | 19:07 |
| fungi | interactive openssh sessions use dscp 0x04 while bulk protocols tend to set 0x02 | 19:09 |
| *** egarbade_ has quit IRC | 19:10 | |
| *** ifarkas is now known as ifarkas_afk | 19:11 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: be more careful using setuptools commands in release script https://review.openstack.org/363895 | 19:12 |
| *** Guest81 has quit IRC | 19:12 | |
| fungi | or 0x0 it looks like | 19:13 |
| *** links has quit IRC | 19:13 | |
| fungi | clarkb: are v4 and v6 sessions equally affected? | 19:16 |
| *** ddieterly is now known as ddieterly[away] | 19:16 | |
| pabelanger | clarkb: mgagne: running on 198.72.124.70 | 19:16 |
| clarkb | fungi: its mostly v4 but thr occasional v6 is seenin haproxy log | 19:16 |
| fungi | k | 19:17 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tripleo-ui: add missing jobs for release management https://review.openstack.org/363897 | 19:17 |
| *** piet has quit IRC | 19:18 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tripleo-ui: add missing jobs for release management https://review.openstack.org/363897 | 19:18 |
| *** salv-orl_ has quit IRC | 19:19 | |
| *** e0ne has quit IRC | 19:20 | |
| *** sarob has quit IRC | 19:20 | |
| *** ddieterly[away] is now known as ddieterly | 19:20 | |
| *** coolsvap has quit IRC | 19:22 | |
| *** rfolco has quit IRC | 19:22 | |
| fungi | clarkb: worth noting, if i `time nc git.openstack.org ssh` it closes at real 2m0.167s (coincidence?) | 19:23 |
| *** daemontool has quit IRC | 19:23 | |
| *** rfolco has joined #openstack-infra | 19:23 | |
| fungi | http closed for me much earlier though, at real 0m51.389s | 19:24 |
| *** Guest81 has joined #openstack-infra | 19:25 | |
| *** akshai has quit IRC | 19:25 | |
| fungi | consistently at 51s | 19:25 |
| *** Guest81 has quit IRC | 19:27 | |
| fungi | that may be apache on a backend closing it for lack of inbound request though | 19:27 |
| *** kzaitsev_mb has joined #openstack-infra | 19:28 | |
| *** akshai has joined #openstack-infra | 19:28 | |
| fungi | clarkb: `time nc git.openstack.org git` also ends at real 2m0.164s | 19:28 |
| *** claudiub has quit IRC | 19:29 | |
| *** gyee has quit IRC | 19:29 | |
| rcarrillocruz | can i please get reviews for https://review.openstack.org/#/c/363751/ https://review.openstack.org/#/c/363805/ https://review.openstack.org/#/c/363806/ | 19:30 |
| *** Swami has joined #openstack-infra | 19:30 | |
| rcarrillocruz | and https://review.openstack.org/#/c/363809/ | 19:30 |
| rcarrillocruz | to pull them out infracloud | 19:30 |
| *** jkilpatr has quit IRC | 19:30 | |
| pabelanger | rcarrillocruz: wow, is that how we disable hosts? | 19:31 |
| pabelanger | 1 massive line with hostnames? | 19:31 |
| *** akshai has quit IRC | 19:31 | |
| pabelanger | much sadness | 19:31 |
| rcarrillocruz | not sure, that's why i wanted to ask for reviews | 19:31 |
| rcarrillocruz | i understand from fungi that for long-term disabling, that's the way | 19:31 |
| rcarrillocruz | but i may be wrong | 19:32 |
| *** akshai has joined #openstack-infra | 19:32 | |
| *** sdake_ has joined #openstack-infra | 19:32 | |
| rcarrillocruz | we can put them on emergency, not back in git repo, but not sure when those servers will be fixed | 19:32 |
| openstackgerrit | Alexey Weyl proposed openstack-infra/project-config: Vitrage tempests https://review.openstack.org/363905 | 19:32 |
| rcarrillocruz | i agree the mechanism seems ugly enough :/ | 19:32 |
| fungi | rcarrillocruz: well, that's the way to disable hosts from the dynamic inventory. if infra-cloud uses a static inventory you could just remove the inventory entries? | 19:32 |
| rcarrillocruz | ok | 19:33 |
| rcarrillocruz | that too | 19:33 |
| rcarrillocruz | i'll abandon | 19:33 |
| *** sdake has quit IRC | 19:33 | |
| rcarrillocruz | better not be consistent here instead of putting a long line of hosts | 19:33 |
| rcarrillocruz | pabelanger: trusty image for mirror, yeah? | 19:34 |
| *** _nadya_ has joined #openstack-infra | 19:34 | |
| alexey_weyl | Hi, | 19:34 |
| pabelanger | rcarrillocruz: I just downloaded one into /tmp | 19:34 |
| pabelanger | for osic-cloud8 | 19:34 |
| alexey_weyl | please approve the following change: | 19:34 |
| alexey_weyl | https://review.openstack.org/#/c/363905/ | 19:34 |
| rcarrillocruz | on puppetmaster? | 19:34 |
| pabelanger | rcarrillocruz: also, I think I just recreated the mirror in infra-cloud | 19:34 |
| pabelanger | rcarrillocruz: yes | 19:34 |
| *** sdake has joined #openstack-infra | 19:35 | |
| rcarrillocruz | let me use it then | 19:35 |
| pabelanger | rcarrillocruz: I ran cloud-launcher to confirm osic-cloud8 settings | 19:35 |
| rcarrillocruz | i'll push tomorrow the change for being able to run the launcher against a single cloud | 19:35 |
| *** larsks has joined #openstack-infra | 19:35 | |
| rcarrillocruz | i.e. ansible-playbook run_launcher blah -e "cloud=osic-cloud1" | 19:35 |
| openstackgerrit | Richard Theis proposed openstack-infra/irc-meetings: Add networking-ovn meeting https://review.openstack.org/363906 | 19:36 |
| rcarrillocruz | pabelanger: also, not sure if you know | 19:36 |
| rcarrillocruz | but you can pass tags to the launcher | 19:36 |
| rcarrillocruz | e.g. | 19:36 |
| *** sdake_ has quit IRC | 19:36 | |
| rcarrillocruz | you want to create just projects | 19:37 |
| rcarrillocruz | you run | 19:37 |
| rcarrillocruz | ansible-playbook blah --tags projects | 19:37 |
| pabelanger | ya | 19:37 |
| rcarrillocruz | and will just process projects from the clouds_layouts.yml | 19:37 |
| rcarrillocruz | k | 19:37 |
| *** flepied has joined #openstack-infra | 19:37 | |
| pabelanger | I think tomorrow we should see how to loop it into our ansible wheel, its been stable for a while | 19:37 |
| *** sarob has joined #openstack-infra | 19:38 | |
| *** tphummel has quit IRC | 19:38 | |
| rcarrillocruz | mordred: hah, https://github.com/ansible/ansible-modules-core/issues/1658#issuecomment-236480459 , we were chatting about that yesterday, just got it updated on my mailbox :D | 19:39 |
| mordred | rcarrillocruz: haha | 19:39 |
| rcarrillocruz | pabelanger: if we had ^, we could just drive images with the launcher by pointing to the cloud-images url, howerver mordred says the 'feature' is not really a thing in v2 or smth :/ | 19:40 |
| fungi | clarkb: on a lark, i'm doing an isolated packet capture of both ends for an idle netcat socket to git.o.o:git so i can compare what they see at the 2-minute mark socket termination | 19:41 |
| mordred | flaper87: ^^ | 19:41 |
| *** sarob has quit IRC | 19:42 | |
| clarkb | fungi: ok curious what you find | 19:42 |
| fungi | clarkb: i'll paste.o.o the result | 19:42 |
| pabelanger | rcarrillocruz: Ya, would rather just work on 2nd nodepool instances to manage control plane DIBs | 19:42 |
| *** pvaneck has joined #openstack-infra | 19:43 | |
| *** berendt has quit IRC | 19:43 | |
| *** jkilpatr has joined #openstack-infra | 19:43 | |
| *** rfolco has quit IRC | 19:43 | |
| pabelanger | mrhillsman: raddaoui: Around to help with an network issue for osic-cloud8? having issues sshing into 172.22.132.39 | 19:44 |
| *** tphummel has joined #openstack-infra | 19:44 | |
| mordred | rcarrillocruz: yah - I see no mention of it in the glance v2 api docs | 19:44 |
| pabelanger | mrhillsman: raddaoui: 544c3700-c31f-4270-ac3b-a7bea98fd742 in question | 19:44 |
| *** rfolco has joined #openstack-infra | 19:44 | |
| mordred | rcarrillocruz: it's possible there might be a Task that does it - but I do not think we should support that | 19:45 |
| pabelanger | mtreinish: raddaoui: Did I need to attach a FIP? or will external-v4 provide me the address? | 19:45 |
| rcarrillocruz | pabelanger: i really think we should have maybe a dib for the mirror with our keys baked in... that or figure out why the currenty ubuntu-trusty dib image doesn't work for afs | 19:45 |
| fungi | clarkb: hrm... server initiates a fin at the 2-minute mark, so maybe this is a coincidence and it's also the default time to close a git socket with no request? http://paste.openstack.org/show/565286/ | 19:45 |
| rcarrillocruz | cos with it we don't have to generate a tmp keypair, we just launch the mirror and we have our keys baked in | 19:45 |
| *** ddieterly has quit IRC | 19:45 | |
| *** Guest81 has joined #openstack-infra | 19:46 | |
| pabelanger | rcarrillocruz: yes, but I want us to have the build process automated first. With out that, we're basically in the same boat | 19:46 |
| mtreinish | pabelanger: ? | 19:46 |
| pabelanger | mtreinish: mistab sorry | 19:46 |
| mtreinish | heh, no worries | 19:46 |
| clarkb | fungi: oh thats goign through haproxy right? I think haproxy will time things out which lines up with the cD explanation | 19:46 |
| clarkb | fungi: maybe 2 minutes is that timeout | 19:46 |
| openstackgerrit | Merged openstack-infra/project-config: require CLA for release-tools https://review.openstack.org/363851 | 19:46 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Allow str for ip_version param in create_subnet https://review.openstack.org/363846 | 19:46 |
| clarkb | fungi: that doesn't explain why ssh would do similar though | 19:46 |
| fungi | clarkb: yeah, i'm capturing one of those next | 19:47 |
| rcarrillocruz | yup, i def. want to work on that when infracloud is rolling | 19:47 |
| fungi | having trouble figuring out how to tail sshd logs under systemd though | 19:47 |
| clarkb | fungi: journalctl -f something something somethign | 19:47 |
| pabelanger | rcarrillocruz: if we can get quorum, we'd launch nodepool02.o.o, have it use all-clouds.yaml and be the delivery system for images to our control plane. Starting with -minimal images, and iterating on that | 19:47 |
| clarkb | fungi: journalctl -f -u sshd | 19:48 |
| fungi | how appropriate | 19:48 |
| clarkb | -f for follow and -u to specify the unit | 19:48 |
| fungi | yeah, i read it another way, but that works too | 19:48 |
| clarkb | -u will accept a pattern too if you need to be fancier | 19:48 |
| *** alexey_weyl has quit IRC | 19:49 | |
| *** akshai has quit IRC | 19:49 | |
| pabelanger | mrhillsman: raddaoui: Just tried external-v6, but looks like we also got an ipv4 address | 19:50 |
| pabelanger | | addresses | external-v6=2001:4800:1ae1:17:f816:3eff:fe1d:6ef0, 172.22.180.54 | 19:50 |
| mordred | pabelanger: I do not think that's a terrible idea | 19:50 |
| pabelanger | mordred: right, we just need to make sure we have enough of them | 19:51 |
| clarkb | mordred: pabelanger fwiw I really don't want to rely on glance image upload for servers that matter right now | 19:51 |
| clarkb | IMO we really need much more reliable glance before thats doable | 19:51 |
| mrhillsman | should be good pabelanger | 19:51 |
| mrhillsman | i want to say that those ipv4 addresses are just tag alongs | 19:52 |
| *** javeriak_ has joined #openstack-infra | 19:52 | |
| pabelanger | clarkb: Yes, we need make things more stable for sure | 19:52 |
| mrhillsman | as single stack ipv6 is no bueno | 19:52 |
| clarkb | its possible the fail rate is realted to iamge size which will be better for our other servers. Just don't want us to decide to switch before we have a reliable service backing the new images | 19:52 |
| fungi | clarkb: same story... server initiated a fin at the 2-minute mark http://paste.openstack.org/show/565287/ | 19:53 |
| mrhillsman | raddaoui am i correct assuming that? | 19:53 |
| pabelanger | clarkb: right, we'd need to do some testing for sure | 19:54 |
| pabelanger | mrhillsman: okay, will relaunch here in a minute | 19:54 |
| clarkb | fungi: huh | 19:54 |
| fungi | clarkb: also sshd doesn't log the connection/disconnect when there were never any bytes transmitted over the socket | 19:55 |
| *** annegentle has quit IRC | 19:55 | |
| fungi | other than the sshd banner | 19:55 |
| *** javeriak has quit IRC | 19:55 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure https://review.openstack.org/363918 | 19:56 |
| fungi | clarkb: strangely, if i sigint my nc before the timeout, sshd logs "Did not receive identification string" | 19:56 |
| clarkb | huh | 19:57 |
| clarkb | so much huh | 19:57 |
| fungi | but if i let it go the full 2 minutes, it never logs anything about the termination | 19:57 |
| *** tphummel has quit IRC | 19:58 | |
| *** tphummel has joined #openstack-infra | 19:59 | |
| fungi | i'll check some other systems to see if i can identify consistent behaviors | 20:00 |
| *** aimeeu has quit IRC | 20:00 | |
| *** aimeeu has joined #openstack-infra | 20:00 | |
| clarkb | fungi: and this is happening with nc right not with an interactive ssh proper connection | 20:00 |
| *** ldnunes has quit IRC | 20:00 | |
| *** annegentle has joined #openstack-infra | 20:00 | |
| fungi | clarkb: correct, i wanted to rule out ssh keepalives | 20:00 |
| openstackgerrit | Richard Theis proposed openstack-infra/irc-meetings: Add networking-ovn meeting https://review.openstack.org/363906 | 20:00 |
| fungi | and key renegotiation and all that | 20:01 |
| clarkb | ya | 20:01 |
| *** scottynomad has quit IRC | 20:01 | |
| fungi | confirmed 2 minutes to a personal debian server in rackspace iad | 20:02 |
| fungi | trying a local debian server on the same lan as my client next | 20:02 |
| *** ldnunes has joined #openstack-infra | 20:02 | |
| *** javeriak has joined #openstack-infra | 20:03 | |
| *** gyee has joined #openstack-infra | 20:03 | |
| *** javeriak_ has quit IRC | 20:03 | |
| *** _nadya_ has quit IRC | 20:03 | |
| *** tonytan4ever has quit IRC | 20:03 | |
| *** azvyagintsev has quit IRC | 20:03 | |
| *** jamesdenton has quit IRC | 20:04 | |
| *** rfolco has quit IRC | 20:04 | |
| *** sarob has joined #openstack-infra | 20:05 | |
| *** rfolco has joined #openstack-infra | 20:05 | |
| *** tphummel has quit IRC | 20:05 | |
| pabelanger | mrhillsman: still getting SSH timeouts, mind taking a look? e6f9fe21-24ea-4d5b-9682-779a69ce06f7 | 20:06 |
| pabelanger | mrhillsman: using external-v6 network | 20:06 |
| openstackgerrit | Merged openstack/os-client-config: Go ahead and handle YAML list in region_name https://review.openstack.org/362483 | 20:07 |
| *** sigmavirus is now known as sigmavirus|awa | 20:07 | |
| fungi | clarkb: any idea roughly how many lines of logs per day we index in logstash? | 20:08 |
| clarkb | fungi: I can give you exact numbers if you want :) | 20:08 |
| raddaoui | pabelanger: was in meeting, reading | 20:08 |
| fungi | clarkb: it's for something anecdotal, so don't spend time looking it up | 20:08 |
| clarkb | well I need to look it up to know anyways | 20:09 |
| *** esp has quit IRC | 20:09 | |
| fungi | right, i mean it's unimportant | 20:09 |
| clarkb | yseterday was 728 million documents | 20:09 |
| clarkb | one doc is roughly one line | 20:09 |
| fungi | but thanks! | 20:09 |
| fungi | clarkb: confirmed, a debian server on my local lan also disconnects an initially idle netcat socket to its sshd at the 2-minute mark as well | 20:10 |
| raddaoui | pabelanger: yeah single stack ipv6 dosent work, that why external-v6 has two subnets, but you can only connect to VM with the ipv6 one | 20:10 |
| *** eharney has quit IRC | 20:11 | |
| pabelanger | raddaoui: sure, the issue right now is, using external-v6, we're not able to SSH into the server on both ipv4 or ipv6 | 20:12 |
| pabelanger | raddaoui: our security groups look to be correct | 20:12 |
| clarkb | pabelanger: this is cloud8? | 20:12 |
| pabelanger | clarkb: yes | 20:12 |
| clarkb | I can take a quick look oh except osc is still broken with neutron /me shakes fist | 20:12 |
| pabelanger | 2001:4800:1ae1:17:f816:3eff:fe04:b485 | 20:12 |
| *** ilyashakhat_mobi has joined #openstack-infra | 20:13 | |
| pabelanger | is the IP is question | 20:13 |
| *** bethwhite_ has quit IRC | 20:13 | |
| clarkb | pabelanger: whats the isntance uuid? | 20:13 |
| pabelanger | clarkb: e6f9fe21-24ea-4d5b-9682-779a69ce06f7 | 20:13 |
| haleyb | clarkb: did you break ipv6 again? :) | 20:13 |
| pabelanger | clarkb: it might die shortly, launch-node about to time out | 20:13 |
| pabelanger | Ya, just got deleted | 20:14 |
| clarkb | pabelanger: server list shows ya no instances | 20:14 |
| pabelanger | let me launch another with --keep | 20:14 |
| * clarkb will try booting one manually | 20:14 | |
| raddaoui | so for you openstackci mirror project, you should attach your VM to an internal network attached to external-v4 and assign it the fip | 20:14 |
| clarkb | oh I can wait | 20:14 |
| raddaoui | like the test VM I had before | 20:14 |
| *** ilyashakhat has quit IRC | 20:14 | |
| pabelanger | Oh | 20:14 |
| pabelanger | okay, I am not doing that | 20:14 |
| clarkb | hrm? | 20:14 |
| raddaoui | and then you can use the public ip mapped to it as specified on the email | 20:15 |
| pabelanger | let me quickly try that | 20:15 |
| *** piet has joined #openstack-infra | 20:15 | |
| raddaoui | yeah I tested that and works fine | 20:15 |
| clarkb | oh its like the old style cloud1 | 20:15 |
| clarkb | so you have to create a network, router, subnet, dhcp range, dns servers, etc etc | 20:15 |
| clarkb | then wire that all up to be able to get floating IPs on external-v4 | 20:16 |
| fungi | clarkb: `sudo ethtool -K eth0 sg on` conveniently reenabled tx-scatter-gather, tcp-segmentation-offload, tx-tcp-segmentation and generic-segmentation-offload again | 20:16 |
| clarkb | fungi: ncie | 20:16 |
| raddaoui | there is one fip allocated to that project 172.22.132.35 | 20:16 |
| *** tongli_ has quit IRC | 20:16 | |
| raddaoui | which is mapped internally to the public ip | 20:17 |
| fungi | clarkb: and the "xen_netfront: xennet: skb rides the rocket" lines in dmesg instantly came back | 20:17 |
| pabelanger | 1 sec | 20:17 |
| clarkb | fungi: our low docs on the weekend is 126million per day and our high is 750million per day | 20:17 |
| clarkb | raddaoui: uhm | 20:17 |
| fungi | clarkb: thanks! | 20:17 |
| clarkb | isn't 172.22.132.35 not routable? | 20:17 |
| clarkb | er rather not globally routable? | 20:17 |
| clarkb | fungi: we consistently float right around 700million during work days | 20:17 |
| *** kgiusti has left #openstack-infra | 20:17 | |
| raddaoui | yeah but it is mapped to 72.3.183.45 | 20:18 |
| clarkb | raddaoui: so we are behind two NATs? | 20:18 |
| *** apuimedo|away is now known as apuimedo | 20:18 | |
| *** waht has joined #openstack-infra | 20:18 | |
| clarkb | 72.3.183.45 to 172.22.132.35 to whatever range we choose for our neutron subnet? | 20:18 |
| raddaoui | yeah I think those ipas are mapped in the firewall | 20:19 |
| raddaoui | ips* | 20:19 |
| raddaoui | yes | 20:19 |
| clarkb | :( | 20:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure https://review.openstack.org/363918 | 20:19 |
| *** esikachev has quit IRC | 20:20 | |
| clarkb | pabelanger: fwiw its almost impossible to get the router + subnet + network + wiring + dhcp + dns resolvers etc stuff correctfrom the command line. Since it only ever needs to be done once I have chaeted and its the one thing I use horizon for. That said I think this cloud may go away under us ya? so this is a good candidate to be added to cloud launch if we can figure out the incantation | 20:20 |
| openstackgerrit | Oleksandr Berezovskyi proposed openstack-infra/project-config: Clone sources of ironic-inspector and ironic-inspector-client https://review.openstack.org/363922 | 20:21 |
| raddaoui | you can attach it directly to the external-v4 but you should make sure vm has 172.22.132.35 | 20:21 |
| *** esp has joined #openstack-infra | 20:21 | |
| clarkb | raddaoui: you mean by statically configuring the network on the host? | 20:22 |
| clarkb | our tooling assumes dhcp or config drive network info. Not sure we can hack that to get that addr | 20:22 |
| *** dimtruck is now known as zz_dimtruck | 20:23 | |
| pabelanger | clarkb: sure, I can work on cloud_launcher | 20:23 |
| *** sarob has quit IRC | 20:23 | |
| *** sarob has joined #openstack-infra | 20:24 | |
| *** sarob has quit IRC | 20:25 | |
| *** akshai has joined #openstack-infra | 20:25 | |
| raddaoui | well that's why I allocated that fip from the beggining to the project so you can allocate it whenever you attach your VM to an internal network | 20:26 |
| *** hockeynut has quit IRC | 20:26 | |
| *** cdent has quit IRC | 20:27 | |
| fungi | i get that neutron probably can't handle a /32 host route with an rfc-1918 serial, but i'm curious why that global address isn't itself the fip pool | 20:28 |
| *** sarob has joined #openstack-infra | 20:28 | |
| pabelanger | raddaoui: Ya, we'd like to automate that process. We've actually rebuilt our mirrors a fair bit recently | 20:29 |
| openstackgerrit | afazekas proposed openstack/os-testr: Construct a list of test cases instead of passing a regexp https://review.openstack.org/348878 | 20:29 |
| *** vhosakot has quit IRC | 20:29 | |
| *** jpeeler has quit IRC | 20:30 | |
| *** Goneri has quit IRC | 20:30 | |
| *** vhosakot has joined #openstack-infra | 20:30 | |
| *** ilyashakhat_mobi has quit IRC | 20:31 | |
| fungi | though i guess the way it's worked around in the cloud1 redesign is that there's just a pat in front of the rfc-1918 "public" neutron net? | 20:31 |
| fungi | so no fip | 20:31 |
| *** sarob has quit IRC | 20:31 | |
| fungi | no, wait, that's for the openstackjenkins project, but not the mirror server in the openstackci project | 20:32 |
| clarkb | fungi: in cloud1 they gave us a provider network with directly assigned "public" IPs | 20:32 |
| clarkb | no rfc-1918 involved | 20:32 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Implement scenari001, 002 and 003 https://review.openstack.org/362504 | 20:32 |
| raddaoui | I don't see the right fip allocated to the project anymore. let me fix this and then will assign it to one of your running VM | 20:32 |
| clarkb | before they did that the floating Ips were properly routable and we just had a single NAT | 20:32 |
| fungi | clarkb: that's not what i see bount to eth0 | 20:32 |
| fungi | er, bound | 20:32 |
| fungi | inet 172.99.106.183/24 brd 172.99.106.255 scope global eth0 | 20:32 |
| clarkb | fungi: its possible the mirror is still using the old setup | 20:32 |
| fungi | that's currently mirror.regionone.osic-cloud1.o.o | 20:32 |
| clarkb | fungi: oh wait the mirror is funny because its got both networks on it iirc | 20:32 |
| *** claudiub has joined #openstack-infra | 20:32 | |
| * clarkb logs in | 20:32 | |
| fungi | it's also still not correctly dual-stack | 20:33 |
| clarkb | ya ok eth1 is ipv6 global network. eth0 is on the private network with rfc1918 addr and that is NATed by neutron with a floating IP | 20:33 |
| clarkb | fungi: it isn't? | 20:33 |
| fungi | oh, eth1! | 20:33 |
| fungi | no wonder i was confused | 20:34 |
| *** Jeffrey4l__ has joined #openstack-infra | 20:34 | |
| clarkb | ya eth1 is sepoarate | 20:34 |
| *** markusry has quit IRC | 20:34 | |
| *** markusry has joined #openstack-infra | 20:34 | |
| fungi | so 10.0.13.115 for the "public" v4 i guess | 20:34 |
| *** markusry has quit IRC | 20:34 | |
| * mordred reads scrollback | 20:34 | |
| fungi | amusing to see eth1 as the global/egress interface | 20:34 |
| clarkb | 10.0.13.115 is the ipv4 addr from the ipv4 subnet that shares the network with the ipv6 subnet for the eth1 interface | 20:34 |
| *** Jeffrey4l_ has quit IRC | 20:35 | |
| clarkb | eth0 is the ipv4 addr on the ipv4 subnet that is on the network that we can attach floating IPs to | 20:35 |
| clarkb | so your ipv4 global traffic goes over eth0 and ivp6 over eth1. Then you just use local routes for eth1 ipv4 | 20:35 |
| pabelanger | okay, I have the mirror attaching to both networks | 20:35 |
| mordred | this is the thing that's fixed by the new shade patch | 20:35 |
| pabelanger | just working on assigning FIP now | 20:35 |
| pabelanger | clarkb: http://paste.openstack.org/show/565316/ | 20:36 |
| mordred | which will get rolled out as soon as the caching revert patch makes its way through CI | 20:36 |
| pabelanger | okay, neat | 20:36 |
| clarkb | well shade won't change how we have things configured | 20:36 |
| clarkb | it will just change how shade reports on them | 20:36 |
| mordred | that is correct | 20:36 |
| mordred | just saying - this is the networking setup that is at th eroot of why we had to make that change | 20:37 |
| fungi | clarkb: indeed, looks like routing table has "default via 172.99.106.1 dev eth0" and "default via fe80::def dev eth1" | 20:37 |
| clarkb | pabelanger: did you configure int_netw or was it preexisting? | 20:37 |
| clarkb | mordred: ya | 20:37 |
| pabelanger | clarkb: I used --nics to assign it. It was existing | 20:37 |
| clarkb | mordred: its also going to get more fun where the public addr shade sees is not the actual public addr because of another layer of nat we can't see | 20:37 |
| pabelanger | I have not created any networks | 20:37 |
| clarkb | pabelanger: ah ok then we will not need to add anything to cloud laucnher | 20:38 |
| fungi | clarkb: so i guess in cloud1 there's a "provider" network for 172.99.106.0/24 and our eth0 is assigned into that? no nat involved? | 20:38 |
| clarkb | fungi: yup | 20:38 |
| fungi | nifty | 20:39 |
| *** sarob has joined #openstack-infra | 20:39 | |
| clarkb | fungi: before pabelanger rebuilt that mirror to have the v6 interface we had nat with a floating IP and only v4 connectivity | 20:39 |
| mordred | clarkb: wat? | 20:39 |
| clarkb | mordred: yes 72.3.183.45 to 172.22.132.35 to whatever range we choose for our neutron subnet? | 20:39 |
| clarkb | mordred: in this case looks like neutron subnet add ris 192.168.2.9 | 20:39 |
| fungi | clarkb: i wonder if they couldn't extend that same 172.99.106.0/24 provider net to the openstackci tenant in cloud8 | 20:39 |
| fungi | then we could do something similar | 20:40 |
| mordred | holy crap that's terrible | 20:40 |
| mordred | I can't even .. | 20:40 |
| clarkb | its definitely a bunch of magicallness | 20:40 |
| clarkb | raddaoui: are there no alternatives to that? eg we can't expose the 72.3.183.45 addr to neutron as a single ip floating ip pool or something? | 20:40 |
| clarkb | raddaoui: then that way the cloud reflects reality for us? | 20:41 |
| mordred | like, honestly - it would be better to just not have the v4 'provider net' at all | 20:41 |
| clarkb | you have to have it for afs | 20:41 |
| mordred | if it's not actually routing things | 20:41 |
| mordred | clarkb: afs works with floating ips | 20:41 |
| *** ansiwen has quit IRC | 20:41 | |
| clarkb | mordred: yes this is still floating IPs | 20:41 |
| mordred | wow | 20:41 |
| mordred | that's | 20:41 |
| clarkb | mordred: its real public addr on firwall outside of openstack to neutron floating ip that is rfc1918 addr to private subnet rfc1918 addr | 20:42 |
| raddaoui | well I dont think that is possible because 72.3.183.45 is not routed inside our private network | 20:42 |
| raddaoui | mrhillsman: ^ | 20:42 |
| clarkb | raddaoui: you just have to route it to the neutron router I think | 20:42 |
| clarkb | I don't know how that relates to your private network | 20:42 |
| fungi | zaro: i wonder if you have any idea what the deal is with the gerrit behavior observed in this thread: http://lists.openstack.org/pipermail/openstack-dev/2016-August/102639.html | 20:43 |
| mrhillsman | reading | 20:44 |
| raddaoui | yeah I can do that but after how packets will be routed internally from that VM to the firewall | 20:44 |
| mordred | oh - is this because the kolla team are themselves "clients" of the osic stuff, so the kolla control plane is not 'trusted' in the same way as the things run by the osic humans? | 20:45 |
| *** sarob has quit IRC | 20:45 | |
| mrhillsman | kolla is not relevant to cloud8 | 20:46 |
| mrhillsman | i'm still a bit lost on the issue | 20:46 |
| rcarrillocruz | pabelanger, clarkb : i have problems to create the mirror with launch-node.py | 20:47 |
| rcarrillocruz | http://paste.openstack.org/show/565318/ | 20:47 |
| clarkb | mrhillsman: the issue is we don't want to be behind two nats preventing our cloud apis from tellin us what the reality is for our floating IP situation | 20:47 |
| rcarrillocruz | i believe is because we don't have neutron ns metadata proxy to inject key | 20:47 |
| rcarrillocruz | thus | 20:47 |
| mrhillsman | as i understood there was on a need for the one floating ip for mirror VM | 20:47 |
| clarkb | mrhillsman: instead it would be nice if neutron could attach the actual publci IP as a floating IP so that cloud queries return complete info | 20:47 |
| rcarrillocruz | we need to pass a config drive ? | 20:47 |
| clarkb | mrhillsman: there is a need for one globally routable ipv4 addresss | 20:47 |
| rcarrillocruz | with the key to bake it in | 20:47 |
| *** sarob has joined #openstack-infra | 20:47 | |
| clarkb | mrhillsman: aiui the way this is being presented to us is via a magical NAT on a firewall somewhere that our cloud api queries will not be privy to | 20:48 |
| pabelanger | rcarrillocruz: launch node will try to use its own key | 20:48 |
| clarkb | so it will just happen to wkrk which is less than ideal | 20:48 |
| rcarrillocruz | yeah , it creates one of the fly but isn't that injected on clouds with neutron metadata server ? | 20:48 |
| pabelanger | rcarrillocruz: I needed --config-drive with tripleo-test-cloud-rh1 I think | 20:48 |
| rcarrillocruz | yep, what i thought | 20:49 |
| rcarrillocruz | do we have a config drive on the puppetmaster to use ? | 20:49 |
| mgagne | pabelanger: any success with mtl01? | 20:49 |
| fungi | mrhillsman: yeah, the need isn't specifically for a floating ip. if for example the 172.99.106.0/24 provider network we've got an address in for cloud1 could also be extended to a provider network for cloud8 we could skip dealing with fips entirely | 20:49 |
| mrhillsman | unfortunately cloud1 and cloud8 are segregated so this would not be possible | 20:50 |
| *** pvinci has joined #openstack-infra | 20:50 | |
| pabelanger | rcarrillocruz: we use cloud.create_keypair() then add the key to create_server | 20:50 |
| fungi | a global ipv4 address pool of /32 size would suit us, even without a fip at all | 20:50 |
| pabelanger | mgagne: test just finished | 20:51 |
| pabelanger | mgagne: and passed | 20:51 |
| mgagne | awesome! | 20:51 |
| pabelanger | clarkb: mgagne: ^ so I think we can bring mtl01 back online | 20:51 |
| clarkb | pabelanger: mgagne yay | 20:51 |
| mrhillsman | so are you wanting to just attach that public IP directly to the mirror VM? | 20:51 |
| mordred | that would be perfect | 20:51 |
| fungi | mrhillsman: but certainly if we do have to have a fip, then having the fip be a global ipv4 address rather than another rfc-1918 address would help. with the double-nat, the openstack api doesn't tell us at all what our routable ipv4 address for that instance is | 20:51 |
| mordred | yah. what fungi said | 20:52 |
| zaro | fungi: the doc says that it will stop email only to reviewers and watchers. | 20:52 |
| rcarrillocruz | ah ok, i thought i had to pass a path to the config-drive param | 20:52 |
| mordred | the only way we konw how to connect to the machien is to ask nova "what's your ip address" - and then we ssh to that | 20:52 |
| rcarrillocruz | going thru now | 20:52 |
| pabelanger | clarkb: mgagne: lets start with 10 nodes first? | 20:52 |
| fungi | mrhillsman: if we could attach the global address to the mirror instance directly (e.g. neutron address pool just large enough to provide us 1 address) that would be even better than a fip, yes | 20:52 |
| mrhillsman | there is only one IP though am i right? | 20:52 |
| clarkb | pabelanger: mgagne 10 is fine by me | 20:52 |
| mrhillsman | it should not change? | 20:52 |
| mgagne | I think it is a good thing to start slow | 20:52 |
| zaro | fungi: i guess owners and subscribers still get emails | 20:53 |
| zaro | ohh i guess subscriber and watcher are the same. | 20:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Slowly bring internap-mtl01 back online https://review.openstack.org/363931 | 20:53 |
| pabelanger | clarkb: mgagne:^ | 20:54 |
| fungi | mrhillsman: yeah, we only need one globally-routable ipv4 address. if we need to replace the server, we'll tear the old server down first or hot-attach the interface later when we move traffic | 20:54 |
| mrhillsman | there is only one floating ip address available | 20:54 |
| zaro | so it continues to email authors and people who starred the change. | 20:54 |
| pabelanger | rcarrillocruz: in fact, we should just make config-drive true by default | 20:54 |
| mordred | mrhillsman: our ansible inventory uses the openstack api to construct itself dynamically | 20:54 |
| fungi | mrhillsman: or if it has to be a fip, we'll reassign the fip when we replace the server | 20:54 |
| mordred | mrhillsman: if nova lies to us about the actual public address, none of our automation will be able to talk to the server | 20:54 |
| clarkb | and aiui in the current situation the ip we will see is the one for the neutron floating ip which is not the actual address we should talk to | 20:55 |
| mordred | mrhillsman: so it's not like we can just learn the address once and write it down | 20:55 |
| mrhillsman | but it should never lie because there is only one | 20:55 |
| zaro | fungi: i don't think there's a way to completely stop emails and it's difficult to seperate bot and human reviews since both use the same comments channel to post info about the change. | 20:55 |
| mrhillsman | only one floating ip available, only one public address available | 20:55 |
| mrhillsman | everything else is ipv6 except for that mirror VM | 20:55 |
| clarkb | mrhillsman: there is the publicly routable IP taht is NAT'd to an rfc 1918 address which is then NAT'd to another rfc 1918 address | 20:55 |
| *** eharney has joined #openstack-infra | 20:55 | |
| zaro | although i believe newer version of Gerrit does have the feature to stop emails. | 20:55 |
| mordred | mrhillsman: the one that nova knows about is not the actual address | 20:55 |
| clarkb | mrhillsman: the cloud apis will show us the two rfc 1918 addresses not the actual publicly routable IP that is sitting on a firweall NATed to our floating IP | 20:56 |
| mrhillsman | ok, i got you | 20:56 |
| pabelanger | rcarrillocruz: if you don't mind: https://review.openstack.org/#/c/363931/ | 20:56 |
| *** e0ne has joined #openstack-infra | 20:56 | |
| mordred | woot | 20:56 |
| *** raildo has quit IRC | 20:56 | |
| clarkb | 72.3.183.45 to 172.22.132.35 to 192.168.2.9 | 20:56 |
| mrhillsman | right | 20:56 |
| *** rtheis has quit IRC | 20:56 | |
| clarkb | we have no visibility into what 72.3.183.45 is | 20:56 |
| rcarrillocruz | sure | 20:56 |
| clarkb | we only see 172.22.132.35 and 192.168.2.9 | 20:56 |
| mrhillsman | let me see if there is a way to change without having to make significant adjustment | 20:56 |
| *** jamesdenton has joined #openstack-infra | 20:56 | |
| fungi | so either we'd like to be able to bind 72.3.183.45 directly to a virtual ethernet interface in the server instance, or at worst have 72.3.183.45 be the fip | 20:56 |
| rcarrillocruz | jebus, MOAR CLOUDS pls | 20:57 |
| pabelanger | clarkb: stepping away here for family time, I'll let you decided on bringing mtl01 online tonight. | 20:58 |
| clarkb | pabelanger: I just approved it | 20:58 |
| clarkb | pabelanger: I will keep an eye on it | 20:58 |
| pabelanger | clarkb: Also, do you mind restarting nodepool-builder so we can pickup osic-cloud8? | 20:58 |
| pabelanger | clarkb: great, thanks | 20:58 |
| clarkb | pabelanger: well we have to fgire out this networking thing before that matters | 20:59 |
| clarkb | but yes I can do that if we sort out something that will work | 20:59 |
| * dhellmann wonders what networking-vpp is and why they've chosen to import their repo via one git review at a time *this week* | 20:59 | |
| clarkb | dhellmann: ijw and sdague can probably tell you about it | 20:59 |
| *** rwsu has joined #openstack-infra | 21:00 | |
| pabelanger | clarkb: Yup, figured we'd get the images in place in case it was a simple fix | 21:00 |
| *** rfolco has quit IRC | 21:00 | |
| dhellmann | clarkb : thanks | 21:00 |
| *** ekhugen has left #openstack-infra | 21:00 | |
| fungi | clarkb: dhellmann: ijw and sdake (i doubt sdague cares at all about it) | 21:00 |
| dhellmann | ah | 21:01 |
| clarkb | oh sorry my bad on tab completing | 21:01 |
| *** rhallisey has quit IRC | 21:01 | |
| clarkb | ooh its completely stopped zuul from doing anything useful | 21:01 |
| *** matt-borland has quit IRC | 21:01 | |
| dhellmann | fungi : we may want to have a soft policy about not importing new repos during deadline weeks, esp. next cycle since it's so much shorter | 21:01 |
| clarkb | jeblair: ^ you may be interested in this | 21:01 |
| dhellmann | yeah, there's a huge check queue now | 21:01 |
| *** jkilpatr has quit IRC | 21:01 | |
| clarkb | dhellmann: well I think this repo was put into gerrit forever ago they just didn't import their code at that time (I dunno why) | 21:01 |
| sdake | dhellmann - no idea what it is, it was imported one commit at a time to preerve commit history - no idea why they didn't wait until after milestone3 | 21:01 |
| fungi | dhellmann: it sounded something like someone at cisco created that as a new repo but continued to commit to an internal copy instead, and then later wanted their work imported into gerrit after the project already existed in it | 21:02 |
| pabelanger | dhellmann: fungi: Ah, yes. The old bomb zuul with 9k events patch-set | 21:02 |
| sdake | dhellmann they didn't use the upstream flag on their new repo submission | 21:02 |
| sdague | ..... :( | 21:02 |
| sdague | let me say, I have words, which are not suitable for irc for that | 21:02 |
| sdake | don't blame me, I wasn't consulted prior to the work ;) | 21:02 |
| *** baoli_ has quit IRC | 21:02 | |
| dhellmann | is there any way to do anything about it now? | 21:02 |
| pabelanger | zuul will eventually recover | 21:03 |
| sdague | pabelanger: eventually | 21:03 |
| clarkb | dhellmann: we could probably dump zuul's queues. remove that projects changes, restart zuul and requeue everything else and make that project read only | 21:03 |
| pabelanger | ya | 21:03 |
| sdague | the point is we have this freeze | 21:03 |
| *** baoli has joined #openstack-infra | 21:03 | |
| dhellmann | I mean, zuul's doing it's thing so we're not blocked, but the queue is pretty long | 21:03 |
| sdague | and we have important content to land | 21:03 |
| *** jamesdenton has quit IRC | 21:03 | |
| sdague | and this is effectively a DOS attack | 21:03 |
| *** jamesdenton has joined #openstack-infra | 21:03 | |
| *** salv-orlando has joined #openstack-infra | 21:03 | |
| *** psilvad has quit IRC | 21:03 | |
| sdake | i hear you, again I can't undo what they have done - they clearly should have read about the upstream: tag | 21:03 |
| *** ansiwen has joined #openstack-infra | 21:03 | |
| sdake | if you would lilke me tog et them to stop rebasing or a week | 21:04 |
| *** tonytan4ever has joined #openstack-infra | 21:04 | |
| sdake | i can do so | 21:04 |
| sdake | or/for | 21:04 |
| sdake | i think there are continual rebases to get the gate jobs working | 21:04 |
| dhellmann | sdake : yes, please ask them to stop doing anything with those patches for now | 21:04 |
| sdague | sdake: also, why are they doing this this way at all | 21:04 |
| sdake | dhellmann have timeframe when they can open it up again | 21:04 |
| sdague | there is a whole git import infrastructure | 21:04 |
| pabelanger | sadly, I have to run. So I won't be able to support the effort. But I think we need to come up with a proper fix, this is about the 4th time in 6 months this has happend | 21:04 |
| dhellmann | sdake : next week at the earliest | 21:04 |
| sdake | sdague what i was told was the documentation stated how to import a repo two different ways in the documentation | 21:04 |
| sdake | and they did the wwrong way | 21:05 |
| fungi | yep. there's a delicate balance with new project creation. we purposely didn't create a provision for post-creation bulk import because it could be used to bypass code review entirely, but by the same token we don't want to set a precedent that the infra team will manually import your repo for you if you forget to specify it at creation time | 21:05 |
| clarkb | sdague: yes they failed to use that infrastructure so now its either push to gerrit like this or get a gerrit admin to force push (which we really don't like doing) | 21:05 |
| dhellmann | fungi : maybe we should remove the version of the import that doesn't expect an upstream repo | 21:05 |
| jeblair | dhellmann: i've never imported a repo | 21:05 |
| sdague | clarkb: well force push is a lot less evil that destroying critical merge time | 21:05 |
| dhellmann | jeblair : you're special | 21:05 |
| jeblair | dhellmann: i always create new projects from scratch in our infra | 21:05 |
| jeblair | dhellmann: i hope not | 21:05 |
| jeblair | i believe in our community process | 21:06 |
| clarkb | sdague: yes but there is another alternative | 21:06 |
| fungi | i see creating a project outside the openstack community and importing it later as a bit of an anti-pattern | 21:06 |
| sdake | jeblair i usse cookiecutter and start from there | 21:06 |
| dhellmann | jeblair: In general I do, too, but I'm having a bit of trouble believing in a process that leads to this result. | 21:06 |
| sdague | anyway, regardless of that | 21:06 |
| clarkb | they could push a reasonable set of chagnes at a time in order to move things along without DOSing | 21:06 |
| clarkb | eg 5 instead of 70 | 21:06 |
| sdague | remember zuul is crazy slow on 170 deep patch series | 21:06 |
| sdague | there is an n^2 problem | 21:06 |
| sdake | fungi I think its ok in some circumstances - heat was created in this way, kolla and magnum all used the upstream patch | 21:07 |
| sdague | we hit this before | 21:07 |
| clarkb | sdague: yup | 21:07 |
| sdague | so this either has to be dumped out of zuul | 21:07 |
| sdague | otherwise we just destroyed freeze | 21:07 |
| sdake | clarkb ya i hear ya - Iguess their orders were to preserve history | 21:07 |
| fungi | sdake: i agree there are times when projects start outside the community and join us later, but by the same token i don't want to make it seem like that's the preferred default behavior pattern | 21:07 |
| sdake | fungi I typically create a cookiecutter on github and use upstream | 21:07 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Run with cells v2 in placement and neutron grenade jobs https://review.openstack.org/363937 | 21:07 |
| sdake | but again, not consulted | 21:07 |
| jeblair | dhellmann: i don't even know what to say to that. i'm clearly not defending this. someone made an error. | 21:08 |
| sdague | ok, problem at hand. | 21:08 |
| sdake | was pinged lat night about why networking-vpp wasn't showingup in zuul | 21:08 |
| sdague | lets take the philosophy to later | 21:08 |
| *** Guest81 has quit IRC | 21:08 | |
| sdague | can this get dumped? | 21:08 |
| sdake | so reachedout to openstack-infra | 21:08 |
| dhellmann | jeblair : yeah, I think my more common pattern is what sdake just said: create something with cookiecutter then import it. maybe I'm the special one. | 21:08 |
| sdake | sdague if thats possible | 21:08 |
| sdake | sdague do so, and these guys cn do this disruptive work in a week | 21:08 |
| jeblair | dhellmann: i use cookiecutter to create the initial commit | 21:08 |
| clarkb | does zuul have an unenqueue to go with enqueue? that might be the other alternative btu I think not without dumping, restarting, and enqueuing only what we want? | 21:08 |
| jeblair | i'm just incensed by the idea that we would force people to not use our system because of tihs | 21:09 |
| sdake | cookiecutter a bit out of date so needs some manual fixups | 21:09 |
| pabelanger | what is the downside of zuul continuing to merge the patches? Missed deadlines right? | 21:09 |
| *** tonytan4ever has quit IRC | 21:09 | |
| jeblair | instead of just saying, hey, someone messed up | 21:09 |
| jeblair | people do that | 21:09 |
| jeblair | let's fix it | 21:09 |
| sdague | jeblair: right, I agree | 21:09 |
| dhellmann | jeblair : I also like to try to set up the jobs when I import the repo, so the tests run from the start with any "real" content. like I said, maybe I'm doing it wrong. | 21:09 |
| sdague | lets fix it | 21:09 |
| fungi | clarkb: the dwqueue feature was never completed. i think that patch is still partially implemented and under review | 21:09 |
| clarkb | pabelanger: potentially yes since in ~12 hours is freeze crunch time I think | 21:09 |
| sdague | pabelanger: right | 21:09 |
| clarkb | pabelanger: ttx's working day basically | 21:09 |
| sdague | basic it's a DOC on our release | 21:09 |
| sdague | DOS | 21:09 |
| jeblair | i want to help, but i need a minute to cool off | 21:09 |
| sdague | jeblair: ok, cool, np | 21:09 |
| pabelanger | IIRC, this happend this morning too. Has anybody see what the downtime was? | 21:10 |
| dhellmann | jeblair : sorry, didn't mean to tick you off :-( | 21:10 |
| clarkb | pabelanger: there is no downtime | 21:10 |
| clarkb | pabelanger: it just slows things down due to the n^2 merge problem | 21:10 |
| sdake | sorry guys -if i had been consulted - different outcome | 21:10 |
| clarkb | (at least that is my understanding of it) | 21:10 |
| dhellmann | jeblair : I guess I'm just coming from a different perspective | 21:10 |
| clarkb | pabelanger: so new changes are not queued to run their jobs as the zuul mergers are all working overtime to enqueue these changes | 21:10 |
| fungi | pabelanger: no downtime, just a prioritization concern. people want to make sure that release-critical (to openstack) work isn't slowed by non-release-critical/unofficial project testing that isn't tied to the release | 21:10 |
| clarkb | eventually it will get through it | 21:10 |
| pabelanger | clarkb: well, nodepool is currently not launching nodes. That's the downtime I was referring too | 21:10 |
| *** priteau has quit IRC | 21:11 | |
| clarkb | pabelanger: yes its not doing that because zuul is only very slowly queueing new jobs due to the merge backlog (I think) | 21:11 |
| pabelanger | Right | 21:11 |
| sdake | so AI or me is to get them to stop all work on networking-vpp until dhellmann gives me a green light | 21:11 |
| sdake | anything else? | 21:11 |
| fungi | we've generally treated all projects equally, in some part because implementing a project prioritization solution would be complicated | 21:11 |
| jeblair | yeah, the queue processors are running | 21:11 |
| pabelanger | clarkb: fungi: So a quick look at grafana shows it took about 4 hours for zuul to clear out | 21:11 |
| jeblair | merge backlog seems plausible | 21:11 |
| rcarrillocruz | pabelanger: ok, the mirror is up | 21:11 |
| rcarrillocruz | how can i make sure the afs is sane and all | 21:12 |
| rcarrillocruz | ? | 21:12 |
| jeblair | do we have a number on that? | 21:12 |
| *** jcoufal has quit IRC | 21:12 | |
| jeblair | rcarrillocruz: just access the mirror over http | 21:12 |
| clarkb | jeblair: I haven't checked gearman but can pretty quickly | 21:12 |
| pabelanger | rcarrillocruz: if you can access http://mirror.regionone.infracloud-vanilla.openstack.org/ and see repos, that is usually all I do | 21:12 |
| clarkb | merger:merge 7668 8 8 | 21:12 |
| clarkb | so ya I think thats it | 21:12 |
| rcarrillocruz | o-k, that's fast, i thought it neeeded to transfer stuff or something | 21:13 |
| pabelanger | now I run, will catch up on backscroll | 21:13 |
| mordred | rcarrillocruz: it does the transfer in the background as needed | 21:13 |
| fungi | and probably ~1.3k of that is the backlog in the merge-check pipeline | 21:13 |
| jeblair | yeah, looking at a merger, it's chugging through vpp work | 21:13 |
| fungi | since it's low priority | 21:13 |
| jeblair | fungi: that's fine and shouldn't affect other things | 21:13 |
| rcarrillocruz | clarkb, clarkb : do I revert the revert for nodepool infracloud now | 21:13 |
| rcarrillocruz | ? | 21:13 |
| clarkb | rcarrillocruz: if the mirror is working then sure | 21:14 |
| sdake | clarkb fungi dhellmann need any other actions out o me? | 21:14 |
| sdake | o/of | 21:14 |
| rcarrillocruz | yup, i see folders, and they have content | 21:14 |
| fungi | right, just saying we'll get to merge requests for all other changes before the merge requests for the merge-check pipeline are processed | 21:14 |
| rcarrillocruz | i'll push | 21:14 |
| jeblair | probably we're looking at check changes being behind a backlog of merges for the vpp changes in check | 21:14 |
| jeblair | fungi: yeah | 21:14 |
| raddaoui | clarkb: just FYI I tried to create an instance with the image uploaded (trusty) and I clouldn't ssh or ping its ipv6 address unlike the one we have | 21:14 |
| fungi | so ultimately it's more like a 6.6k backlog on merges | 21:14 |
| jeblair | and the mergers have to stack 177 changes up for each | 21:14 |
| jeblair | the *good* news is this gradually gets faster... | 21:15 |
| clarkb | raddaoui: the image we uploaded? | 21:15 |
| jeblair | as the mergers start to remember changes they've seen and don't need to fetch them | 21:15 |
| sdague | jeblair: umm... yeh, but iirc the last time it was like days | 21:15 |
| clarkb | sdague: no this happened last night it took a few hours | 21:15 |
| sdague | and after 12 hours we just killed and restarted | 21:15 |
| raddaoui | yes the Ubuntu 14.04.5 LTS (Trusty Tahr) Daily | 21:15 |
| sdague | clarkb: at 177? | 21:15 |
| sdake | clarkb dhellmann crafting email now - please let me know ifi should provide further instructions rather then "STOP ALL ACTIVITY" | 21:15 |
| sdague | oh, I see the node graph now | 21:16 |
| rcarrillocruz | in other news, i got a quick reply from HPE DC folks, they claim they fixed the cabling issues of at least a couple servers | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| raddaoui | the Ubuntu 14.04 LTS works fine with ipv6 | 21:16 |
| rcarrillocruz | i'll provision them tomorrow then | 21:16 |
| sdague | right, so that's like a 6 hour halt? | 21:16 |
| clarkb | sdague: I think it would be ok for them to push a few patches at a time and iterate through them that way | 21:16 |
| clarkb | er | 21:16 |
| clarkb | sdake: ^ | 21:16 |
| mordred | rcarrillocruz: you're killing it with the new servers. I'm also impressed with the response you're getting from the DC ops folks | 21:16 |
| *** tphummel has joined #openstack-infra | 21:16 | |
| sdague | clarkb: probably, but given that we've had directions challenges in the past, it's probably just safer to ask them to wait 2 weeks | 21:16 |
| sdake | rcarrillocruz ++ :) | 21:16 |
| clarkb | raddaoui: we uploaded a daily? I am confused | 21:17 |
| rcarrillocruz | ;-) | 21:17 |
| *** shashank_hegde has quit IRC | 21:17 | |
| sdague | and regroup with a mentor after that | 21:17 |
| sdake | i should be their mentor | 21:17 |
| sdake | not sure why they didn't ask | 21:17 |
| *** shashank_hegde has joined #openstack-infra | 21:17 | |
| sdake | probably because i am so overloaded i can brely come up forair.. :( | 21:17 |
| clarkb | raddaoui: pabelanger likely knows what the image story is but if you did two nics I want to say there may be issues auto configing the second | 21:17 |
| *** matbu is now known as matbu|afk | 21:17 | |
| raddaoui | yeah I didn't upload it and you guys were using it with your VMs | 21:17 |
| fungi | jeblair: if we rebased just the change closest to the branch tip so that all the other changes are invalidated for an unmergable parent, would that clear them out quickly or do the mergers still have to try to merge each of those? | 21:18 |
| clarkb | raddaoui: I want to say we ran into that on cloud1 when we did the two nics there. We basically just had to enable eth1 and then it picked up the RAs | 21:18 |
| *** rossella_s has quit IRC | 21:18 | |
| *** annegentle has quit IRC | 21:18 | |
| *** e0ne has quit IRC | 21:18 | |
| *** rossella_s has joined #openstack-infra | 21:18 | |
| jeblair | fungi: checking | 21:18 |
| *** gouthamr has quit IRC | 21:18 | |
| raddaoui | no actually I am just using one interface eth0 | 21:19 |
| fungi | i guess the mergers would still have to pick the work requests up out of gearman, but still might short-circuit after that | 21:19 |
| *** dprince has quit IRC | 21:19 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/infra-manual: emphasize the prefered way for importing repository history https://review.openstack.org/363941 | 21:19 |
| dhellmann | jeblair : ^^ | 21:19 |
| *** e0ne has joined #openstack-infra | 21:19 | |
| *** e0ne has quit IRC | 21:20 | |
| jeblair | fungi: i don't think we cancel merger requests, and merger requests have the whole data set with them, so i don't think that would work | 21:21 |
| *** ansiwen has quit IRC | 21:21 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Revert "Revert "Enable infracloud servers in Nodepool"" https://review.openstack.org/363942 | 21:21 |
| jeblair | fungi: it's a good idea, and if this weren't changing in v3 anyway, i'd say it should behave like that :) | 21:21 |
| *** spzala has quit IRC | 21:21 | |
| clarkb | I am wathcing the merge job count fall so we are working through it just not super quickly | 21:21 |
| *** spzala has joined #openstack-infra | 21:22 | |
| clarkb | 7589 queued now | 21:22 |
| rcarrillocruz | clarkb, pabelanger : ^ if you don't mind... | 21:22 |
| jeblair | clarkb: have you worked out a slope yet to get an eta? (i think you can subtract 1.6k as fungi said) | 21:22 |
| clarkb | raddaoui: unfortunately I don't know where that image came from so don't know much about how it is configured to do networking | 21:22 |
| clarkb | jeblair: pabelanger said it took 4 hours last night | 21:22 |
| * clarkb looks at irc timestamps to produce rough jobs per minute number | 21:23 | |
| fungi | got it. so really if we're not going to wait it out, we need to dump a copy of the pipelines we can salvage (check and gate), restart zuul, edit the check export to remove all changes for networking-vpp, then start zuul and then reenqueue the other old check and gate changes | 21:23 |
| jeblair | fungi: oh, one other option: | 21:23 |
| rcarrillocruz | thx pleia2 | 21:23 |
| pleia2 | sure | 21:23 |
| pleia2 | rcarrillocruz: exciting times :) | 21:24 |
| rcarrillocruz | indeed ! :D | 21:24 |
| fungi | it looks like the most recent tag for the release pipeline ran the important jobs, so we won't lose anything critical there | 21:24 |
| fungi | though we'll lose a couple hours of jobs queued up for the post pipeline | 21:24 |
| *** mdrabe has quit IRC | 21:24 | |
| clarkb | at this rate less the 1.6k we will be done in about 12.5 hours | 21:24 |
| *** mdrabe has joined #openstack-infra | 21:25 | |
| clarkb | rate is ~8 per minute | 21:25 |
| jeblair | fungi: we could write a zuul-merger that pops jobs off the stack and fails them quickly, and exits as soon as it sees something that isn't networking-vpp. | 21:25 |
| fungi | clarkb: taking acceleration into account, or was that just a linear burn-down estimate? | 21:25 |
| clarkb | fungi: linear | 21:25 |
| jeblair | clarkb: the rate will increase, but not dramatically. i think that helps us get an order of magnitude. | 21:25 |
| fungi | fair enough | 21:25 |
| jeblair | i don't think the custom zuul-merger is a good idea. i think we should dump/edit/reload. | 21:25 |
| *** spzala has quit IRC | 21:26 | |
| *** aviau has quit IRC | 21:26 | |
| *** yolanda has joined #openstack-infra | 21:26 | |
| fungi | i think getting dequeue working (i guess for v3 at this point) is also a good longer-term idea | 21:26 |
| *** aviau has joined #openstack-infra | 21:26 | |
| clarkb | jeblair: ya customer zuul-merger seems like a way to potentially mangle things with bad merge info | 21:26 |
| *** fguillot_ has joined #openstack-infra | 21:27 | |
| clarkb | (if we get it wrong0 | 21:27 |
| fungi | but yeah, i feel like there's a lot more risk if we rush a hacked-up merger into place to shear away the networking-vpp changes | 21:27 |
| fungi | or what clarkb said | 21:27 |
| jeblair | oh, heh, that's actually a better idea than mine. but they're still both harder than just restarting. | 21:27 |
| fungi | at least dump/reenqueue is a devil we know | 21:27 |
| jeblair | where are we on nodepool restart? | 21:27 |
| jeblair | should we roll one into this? | 21:27 |
| jeblair | mordred: ^? | 21:27 |
| clarkb | the builder needs a restart to pick up new cloud infos but unsure of the main daemon and shade situation | 21:28 |
| *** sdague has quit IRC | 21:28 | |
| fungi | the gearman server clearing is going to tank all our in progress image uploads anyway, right/. | 21:28 |
| fungi | ? | 21:28 |
| clarkb | fungi: yup | 21:28 |
| clarkb | so I will restart the builder | 21:28 |
| jeblair | yeah, so we can do the builder restart with no additional impact | 21:28 |
| fungi | so might as well roll a nodepool and builder restart into the mix | 21:28 |
| *** waht has quit IRC | 21:29 | |
| mordred | jeblair: the shade patch still hasnt' landed | 21:29 |
| fungi | i'm happy to do either the zuul or the nodepool part | 21:29 |
| fungi | mordred: did it pass tests? | 21:29 |
| mordred | fungi: yes. well, it passed the important ones | 21:29 |
| mordred | fungi: the current issue was battling the devstack config change | 21:30 |
| fungi | we could just apply it while we're restarting | 21:30 |
| jeblair | mordred: is it a thing we should manually apply and restart, or does it still need more time? | 21:30 |
| *** Illumitardi has quit IRC | 21:30 | |
| mordred | we could do that - I don't think it's terribly dangerous - shall I get prepped for that? | 21:30 |
| jeblair | mordred: yeah, let's | 21:30 |
| mordred | k. one sec | 21:30 |
| *** zz_dimtruck is now known as dimtruck | 21:31 | |
| fungi | hopefully at least not dangerous the way asps are (very!) dangerous | 21:31 |
| clarkb | I can get the nodepool builder since I told pabelanger I would do that earlier | 21:31 |
| jeblair | dhellmann: ++ | 21:31 |
| *** nstolyar_ has quit IRC | 21:32 | |
| mordred | fungi: ASPs? | 21:32 |
| fungi | mordred: those are, in fact, especially dangerous | 21:32 |
| fungi | glad to no longer have to care about any of the webservers that ran them | 21:33 |
| jeblair | fungi: bad dates. | 21:33 |
| mordred | I'm installing new shade before restart | 21:33 |
| *** fguillot_ has quit IRC | 21:33 | |
| fungi | heh. nice cross-reference | 21:33 |
| * clarkb hears a whoosh go over his head | 21:33 | |
| fungi | clarkb: indiana jones and the lost ark quotes | 21:34 |
| mordred | new shade installed - http://paste.openstack.org/show/565320/ are the dependencies that changed in case we need to revert | 21:34 |
| fungi | thanks mordred | 21:34 |
| jeblair | there are only 4 vpp changes in check | 21:34 |
| *** annegentle has joined #openstack-infra | 21:35 | |
| jeblair | so when we're ready to start -- | 21:35 |
| fungi | the rest haven't been queued yet i guess? | 21:35 |
| jeblair | i'll stop/edit/start/re-enqueue zuul | 21:35 |
| jeblair | fungi: there's no backlog | 21:36 |
| fungi | ahh | 21:36 |
| jeblair | fungi: i think it's just 4 changes with hundreds of dependencies | 21:36 |
| jeblair | clarkb: will restart builder | 21:36 |
| fungi | oh, yep. that 'splains it | 21:36 |
| jeblair | mordred: will restart nodepool | 21:36 |
| mordred | yup | 21:36 |
| jeblair | fungi: will quote movies | 21:36 |
| clarkb | and I can get nodepool-builder if mordred isn't including that in nodepoold restarting | 21:36 |
| mordred | I'm tailing the nodepool debug log to look for bad tracebacks | 21:36 |
| jeblair | clarkb: let's put you on builder duty | 21:37 |
| clarkb | wfm | 21:37 |
| jeblair | mordred: will just do main daemon | 21:37 |
| jeblair | everybody set? | 21:37 |
| mordred | main daemon, standing by | 21:37 |
| *** sarob has quit IRC | 21:37 | |
| clarkb | I am going to go ahead and stop builder now, then start again when mordred is happy with nodepoold | 21:37 |
| mordred | ++ | 21:37 |
| jeblair | oh wait | 21:38 |
| clarkb | thats done so ready when you are | 21:38 |
| *** sarob has joined #openstack-infra | 21:38 | |
| jeblair | fungi: i have a job for you | 21:38 |
| *** thorst has quit IRC | 21:38 | |
| mordred | 2016-08-31 21:38:06,393 DEBUG nodepool.NodePool: Instance ubuntu-trusty-rax-dfw-9172205 (3aad4b32-694a-4964-9bce-9b67e1f20c2a) in rax-dfw has no nodepool metadata | 21:38 |
| jeblair | fungi: can you stand by to restart all the zuul mergers? | 21:38 |
| *** rwsu has quit IRC | 21:38 | |
| mordred | I don't believe I've seen that before | 21:38 |
| fungi | jeblair: standing by now | 21:38 |
| jeblair | mordred: still ready? | 21:38 |
| *** rwsu has joined #openstack-infra | 21:38 | |
| mordred | standing by | 21:39 |
| jeblair | mordred, clarkb, fungi: and go :) | 21:39 |
| jeblair | zuul is stopped | 21:39 |
| mordred | nodepool stopped | 21:39 |
| mordred | nodepool started | 21:40 |
| mordred | jeblair: do you care about this: http://paste.openstack.org/show/565321/ | 21:40 |
| jeblair | zuul is restarted | 21:41 |
| clarkb | mordred: did that happen before or after shutdown? | 21:41 |
| fungi | mergers have all restarted now | 21:41 |
| mordred | clarkb: after the start | 21:41 |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Add craton-dashboard repository (Horizon Plugin) https://review.openstack.org/354274 | 21:41 |
| mordred | clarkb: but I'm not seeing it again | 21:41 |
| clarkb | mordred: ya I think thats a race between main thread initing db and setting global config and event handler for job finished | 21:42 |
| jeblair | mordred: huh, maybe there's a race... right that. :) | 21:42 |
| clarkb | mordred: it may mean that we will leak that node though | 21:42 |
| clarkb | mordred: so maybe check that instance in 15 minutes to see if the cleanup routines don't somehow handle it | 21:42 |
| fungi | just confirmed, zuul-merger processes all have a recent start time now | 21:43 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 21:43 |
| mordred | ok. I'm not seeing anything particularly unhappy in the logs | 21:43 |
| jeblair | the 8 hour timeout for used nodes would catch it eventually | 21:43 |
| clarkb | mordred: ya from what I see of the service it looks to be happy so far | 21:44 |
| clarkb | mordred: we want to see osic instances get the right ip info though right? | 21:44 |
| ijw | Hey, sorry about the patch push earlier | 21:45 |
| sdake | dhellmann clarkb - all work on networing-vpp is halted until clarkb gives me a green light | 21:45 |
| ijw | As we talked this over yesterday afternoon I thought I was in the clear | 21:45 |
| mordred | clarkb: yah | 21:45 |
| *** thiagop has quit IRC | 21:46 | |
| clarkb | wait I am the stoplight? | 21:46 |
| sdake | iwj sierra happens :) | 21:46 |
| clarkb | ijw: we didn't realize at the time that it was gumming up the works | 21:46 |
| sdake | ijw that is | 21:46 |
| fungi | ijw: yesterday it was at a time of day where there was very little going on | 21:46 |
| clarkb | ijw: and basically we want to get feature freeze out the door | 21:46 |
| mrhillsman | mordred fungi clarkb can you all just use the mirror in cloud1 for cloud8? | 21:46 |
| *** dimtruck is now known as zz_dimtruck | 21:46 | |
| mrhillsman | they are both in the same DC | 21:46 |
| dhellmann | sdake, clarkb : I can act as stop light | 21:46 |
| mordred | mrhillsman: not really | 21:46 |
| clarkb | dhellmann: yes I think you would be ebtter than me :) | 21:47 |
| sdake | dhellmann sounds good ill sync up with you next week adn week after | 21:47 |
| ijw | And per previous comments, I've come to this somewhat late, so the problem we have is one group created the repo empty - hence ther otherwise silly way to import history | 21:47 |
| dhellmann | sdake : ok | 21:47 |
| mordred | mrhillsman: the mirrors are inferred from cloudname+regionname in the setup scripts on the build nodes | 21:47 |
| dhellmann | ijw : I think we're just going to want you to go a few patches at a time. zuul doesn't cope well with extremely deep series like that. | 21:47 |
| ijw | But yes, sdake got to me and I can certainly stop. | 21:47 |
| jeblair | dhellmann, sdake, ijw: particularly -- merging the patches is important | 21:48 |
| ijw | dhellmann: can do if you like. | 21:48 |
| fungi | mordred: while it does feel suboptimal, and might pose scaling problems, what are the other complications besides just having to put the cloud8 mirror name in dns for the same ip addresses as the cloud1 mirror? | 21:48 |
| mrhillsman | ok, i'll work on it | 21:48 |
| mordred | fungi: oh - that's an idea | 21:48 |
| clarkb | mordred: http://paste.openstack.org/show/565322/ is that logging what you expect out of osic cloud1? | 21:48 |
| dhellmann | ijw, sdake : let's sync up tuesday (monday being a holiday) and see how things are looking | 21:48 |
| ijw | For what it's worth, I think it is a shortcoming that we can't import history post-creation, though I understand your viewpoint | 21:48 |
| jeblair | dhellmann, sdake, ijw: so not only just pushing a small number of patches, but since they are dependent on each other, making sure that only a small number are open. that's what hurt us this time, that zuul was preparing hundreds of patches together for a single change | 21:48 |
| mordred | clarkb: nope. one sec ... | 21:48 |
| sdake | jeblair ya makes sensse | 21:49 |
| dhellmann | jeblair : I was going to suggest that they start cherry-picking from the bottom of their stack and go ~5 at a time. Would that work? | 21:49 |
| ijw | Yeah, fine - I wasn't aware it was setting up an O(n^2) task (it's not obvious from the outside). | 21:49 |
| clarkb | fungi: I think that may be the best workaround if we can't get the IP into the cloud in a way that tools want | 21:49 |
| ijw | Let me go patch weeding, and I'll submit a couple this evening and see how that works. | 21:50 |
| mrhillsman | i'm am quite sure this will not get looked at by networking until tomorrow | 21:50 |
| clarkb | mrhillsman: we aren't going to have to deal with weird bw bottlenecks if we do that? | 21:50 |
| dhellmann | ijw : no, please do not submit any more patches until next week | 21:50 |
| mrhillsman | but should be possible | 21:50 |
| clarkb | mrhillsman: like only 100mbps between regions or similar? | 21:50 |
| mrhillsman | i do not believe so but would hate for that not to be the truth | 21:50 |
| jeblair | dhellmann: if you literally mean cherry-picking -- i think the same patch history can still be preserved -- ie, it's okay for them to be dependent on each other (still have the same git parents). we just want to keep the number of outstanding unmerged patches small. | 21:51 |
| mordred | clarkb: oh! yes. | 21:51 |
| mordred | clarkb: that is correct | 21:51 |
| mordred | clarkb: nodepool is reporting public v4 and v6 | 21:51 |
| mrhillsman | they should all be next to each other but i'd imagine it would be a concern of the routing | 21:51 |
| mordred | and we expect this node to not have public v4 | 21:51 |
| clarkb | mordred: yup that sounds right to me | 21:51 |
| mrhillsman | i'll work on getting the proper setup in place regarding that address | 21:51 |
| dhellmann | jeblair : ok, I'm not sure how to take N patches in a series and only submit 5 of them without picking them into a new branch that doesn't include the N-5 patches. | 21:51 |
| clarkb | mordred: you think this is happy then? I can start the builder up? | 21:51 |
| mordred | clarkb: yah | 21:52 |
| mrhillsman | just hoping delay is not much | 21:52 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/project-config: Add integration tests between system-config and logstash-filters https://review.openstack.org/320729 | 21:52 |
| clarkb | mordred: jeblair fungi I am starting the builder now | 21:52 |
| *** zz_dimtruck is now known as dimtruck | 21:52 | |
| dhellmann | jeblair : I mean, I guess just "git checkout $sha" at the 5th item? | 21:52 |
| clarkb | and thats done. says it is listening for jobs | 21:52 |
| jeblair | dhellmann: ah yeah, i see what you're saying. yes i think that will work. | 21:53 |
| fungi | dhellmann: basically what we need to avoid is a zuul-triggering event for any change with lots of open parent _or_ child changes in gerrit | 21:54 |
| *** tphummel has quit IRC | 21:54 | |
| dhellmann | ok. so there's also a long list of existing open patches in that repo https://review.openstack.org/#/q/project:openstack%2Fnetworking-vpp+is:open,n,z | 21:55 |
| dhellmann | it sounds like those should be merged before we do anything else? | 21:55 |
| jeblair | fungi, dhellmann, sdake: yeah, we'll want to be careful approving the existing changes in gerrit. should only approve those one at a time, for starters, starting with the change closest to the branch tip | 21:55 |
| mordred | clarkb: uhm ... | 21:55 |
| clarkb | mordred: yes? | 21:56 |
| jeblair | zuul will need to walk the whole tree to see if it needs to enqueue the children. that will take it a little while. | 21:56 |
| jeblair | (the first time, at least, when nothing is cached) | 21:56 |
| dhellmann | right | 21:56 |
| raddaoui | clarkb, pabelanger can you look at the VMs in openstackci project they are both directly connected to provider network they are sshable | 21:56 |
| mordred | clarkb: we're at very low building-nodes count | 21:56 |
| mordred | clarkb: http://grafana.openstack.org/dashboard/db/nodepool | 21:56 |
| jeblair | mordred: i agree. looking. | 21:56 |
| clarkb | mordred: we have a bunch ready | 21:56 |
| *** yolanda has quit IRC | 21:56 | |
| clarkb | well relative to building | 21:56 |
| *** nwkarsten has quit IRC | 21:57 | |
| jeblair | oh, we probably leaked a bunch during the restart | 21:57 |
| raddaoui | yeah clarkb from reading history I think rcarrillocruz uploaded the trusty image | 21:57 |
| clarkb | yup there is a ton used | 21:57 |
| *** jamesdenton has quit IRC | 21:57 | |
| clarkb | raddaoui: ah in that case it may be a dib built image which we will want to replace with the canonical/ubuntu published ones for now | 21:57 |
| jeblair | clarkb, mordred: i'm going to clean some of those up | 21:58 |
| clarkb | mordred: jeblair I think we can just delete all used > 20 minutes | 21:58 |
| raddaoui | it does not work with both v4 and v6 I guess | 21:58 |
| clarkb | jeblair: kk | 21:58 |
| clarkb | raddaoui: it depends on config drive, is one attached? | 21:58 |
| clarkb | actually thats not true hrm | 21:58 |
| ijw | dhellmann, jeblair: I mean literally a couple of patches. If I have to patch / approve / patch / approve, so be it, but I understand what I shouldn't be doing at this point, I think | 21:58 |
| clarkb | it should dhcp by default for v4 but maybe it doesn't v6 in that case | 21:58 |
| *** manjeets- has joined #openstack-infra | 21:58 | |
| dhellmann | ijw : really, seriously, and truly please do not do anything with that repo this week. | 21:58 |
| ijw | dhellmann: would you like me to kill the patches up for review | 21:59 |
| *** hrybacki|afk is now known as hrybacki | 21:59 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Add craton-dashboard repository (Horizon Plugin) https://review.openstack.org/354274 | 21:59 |
| manjeets- | hello infra folks I want to enable a extension_driver in conf file by default for some tempest tests | 21:59 |
| *** javeriak has quit IRC | 21:59 | |
| dhellmann | ijw : I would like nothing to be touched at all, for now. When I'm done with the milestone I will have time to help get things merged carefully. I don't have that time this week. | 21:59 |
| ijw | dhellmann: ok, all good | 21:59 |
| dhellmann | ijw : thanks | 21:59 |
| raddaoui | no just eth0 | 22:00 |
| *** tphummel has joined #openstack-infra | 22:00 | |
| mordred | clarkb, raddaoui: I believe we've had issues with second interfaces not being configured to pick up stuff by default | 22:00 |
| mordred | pabelanger has puppet to fix it for the osic-cloud1 mirror | 22:01 |
| manjeets- | for devstack gate deployment i want test to enable a extension driver by default, | 22:01 |
| mordred | so - it's a known thing with the base images aiui | 22:01 |
| jeblair | 2016-08-31 22:01:07,294 INFO nodepool.NodePool: Need to launch 401 ubuntu-xenial nodes for zuul on osic-cloud1 | 22:01 |
| mordred | jeblair: that'll be fun | 22:01 |
| mordred | cloudnull: ^^ buckle up | 22:01 |
| jeblair | it's doing it right now :) | 22:01 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 22:01 |
| jeblair | well, actually it's issuing glance image list and flavor list over and over | 22:02 |
| jeblair | cause i guess we're not caching those right yet | 22:02 |
| mordred | jeblair: to the same cloud? | 22:02 |
| jeblair | yep | 22:02 |
| mordred | sigh | 22:02 |
| mordred | I thought that was long since sorted | 22:02 |
| *** shashank_hegde has quit IRC | 22:02 | |
| clarkb | manjeets-: if you want something enabled by default the best place to do that is devstack | 22:03 |
| clarkb | manjeets-: devsatck-gate should really only be used to do things like configure non defaults for specific tests or make testing work in a non interactive manner | 22:03 |
| mordred | jeblair: I'll look in to that tomorrow - unless you think it's choking us too badly right now | 22:04 |
| manjeets- | clarkb, https://review.openstack.org/#/c/354447/ I have a api test which covers a scenario | 22:04 |
| clarkb | mordred: we have 955 in building so I think its probably working ok | 22:04 |
| jeblair | mordred: yeah, it's just slowing us, not killing us | 22:05 |
| manjeets- | all I want to set a parameter in ml2.conf.ini extension_drivers = dns | 22:05 |
| *** ansiwen has joined #openstack-infra | 22:05 | |
| mordred | jeblair: ok. good. it sounds like a good "first thing when I wake up" thing to fix - rather than a "last thing before I start drinking" | 22:05 |
| dhellmann | thanks for resetting things, everyone, it looks like the jobs at the front of the queues have started up again | 22:06 |
| jeblair | mordred: ++ | 22:06 |
| jeblair | dhellmann: np | 22:06 |
| *** fguillot has quit IRC | 22:07 | |
| cloudnull | mordred: rutro-shaggy | 22:07 |
| * cloudnull goes for a beer leaving pager on desk | 22:08 | |
| clarkb | manjeets-: as I said if you want to set a default configuration for one of the projects typically the best place for that is devstack | 22:08 |
| mordred | cloudnull: I lost my pager ... golly, 15 years ago? | 22:08 |
| *** hashar has quit IRC | 22:08 | |
| cloudnull | sadly i have a "smart" phone now. | 22:09 |
| *** signed8bit has joined #openstack-infra | 22:09 | |
| *** fguillot has joined #openstack-infra | 22:09 | |
| mordred | cloudnull: my phone has been set in "do not disturb or even ring" mode for quite some time :) | 22:09 |
| clarkb | I remember when I had two because I refused to be on call on my personal phone | 22:09 |
| cloudnull | so far so good, life has note exploded yet. | 22:09 |
| rbergeron | i actually flushed my pager down a toilet once | 22:09 |
| clarkb | mordred: android has a handy feature that is only make noise if its from someone in this list | 22:09 |
| rbergeron | during my on-call week no less | 22:10 |
| clarkb | rbergeron: were you testing its water resistance? | 22:10 |
| *** spzala has joined #openstack-infra | 22:10 | |
| *** esp has quit IRC | 22:10 | |
| *** adriant has joined #openstack-infra | 22:11 | |
| clarkb | manjeets-: though devstack may just copy whatever is in neutron's example config for that | 22:11 |
| *** esp has joined #openstack-infra | 22:11 | |
| clarkb | manjeets-: so you may have to update neutrons example config | 22:11 |
| cloudnull | 271939 info and 355 error messages processed in the last 10 min. with the spike going down on every refresh. | 22:11 |
| cloudnull | i think we're through the build storm | 22:12 |
| manjeets- | clarkb, thats autogenerated i guess | 22:12 |
| pabelanger | catching up on backscoll | 22:12 |
| *** mriedem has quit IRC | 22:12 | |
| manjeets- | need to figure what it reads before autogenerating | 22:12 |
| cloudnull | rbergeron: thats how you win the pager game. | 22:12 |
| cloudnull | :) | 22:12 |
| *** akshai has quit IRC | 22:12 | |
| rbergeron | clarkb: not really, it just fell out of its holster as i flushed... it's as though it knew its true destiny | 22:12 |
| cloudnull | hahahahaha ^ | 22:12 |
| clarkb | manjeets-: ya I think it uses the oslo config objects inside neutron to generate the file so if you edit that it may do the right thing | 22:13 |
| rbergeron | cloudnull: would have been better if it hadn't been my joke threat for years | 22:13 |
| clarkb | when I was oncall we did once get a nice smartphone photo of another smartphone in a toilet | 22:14 |
| cloudnull | it was a freudian slip, into the toilet | 22:14 |
| rbergeron | "if we dont get more than 3 ppl in this rotation, one of these days i swear i'm gonna flush this thing down the toilet" | 22:14 |
| *** asettle has joined #openstack-infra | 22:14 | |
| *** gordc has quit IRC | 22:14 | |
| pabelanger | raddaoui: clarkb: checking the server in osic-clou8 now | 22:14 |
| *** spzala has quit IRC | 22:15 | |
| *** piet has quit IRC | 22:15 | |
| clarkb | pabelanger: did you see the suggestion of maybe just put dns records inplace that point cloud8 at cloud1 for now | 22:15 |
| *** spzala has joined #openstack-infra | 22:15 | |
| *** xyang1 has quit IRC | 22:15 | |
| pabelanger | clarkb: Ya, we can do that if people are fine with that | 22:16 |
| mrhillsman | on a side note i submitted change request regarding the IP; not sure again of time to resolution but ball is rolling | 22:16 |
| mrhillsman | but if we can do something temporarily to help out like the dns change, that would be great | 22:16 |
| mrhillsman | at least the resources can be used in some manner | 22:16 |
| pabelanger | Yup, if other infra-root are good with that, I can update DNS records now | 22:16 |
| *** inc0 has quit IRC | 22:17 | |
| *** eharney has quit IRC | 22:17 | |
| *** esberglu has quit IRC | 22:17 | |
| clarkb | ya I think that would work fine for now particularly if we start with a small number of instances (which we have been) | 22:17 |
| *** esberglu has joined #openstack-infra | 22:18 | |
| mordred | yah | 22:19 |
| mordred | I thnk it's a fine thing to do | 22:19 |
| pabelanger | http://mirror.regionone.osic-cloud8.openstack.org | 22:19 |
| mordred | and then also, if there are problems, we can konw about it | 22:19 |
| *** asettle has quit IRC | 22:19 | |
| mordred | woot | 22:19 |
| clarkb | pabelanger: also builder was restarted | 22:19 |
| pabelanger | clarkb: nice | 22:20 |
| *** srobert has quit IRC | 22:20 | |
| *** cardeois has quit IRC | 22:21 | |
| pabelanger | clarkb: ha, we need to restart it again. osic-cloud8 patch hasn't landed on nodepool.o.o yet | 22:21 |
| *** piet has joined #openstack-infra | 22:21 | |
| jhesketh | Morning | 22:23 |
| *** esberglu has quit IRC | 22:24 | |
| jeblair | oh huh | 22:25 |
| jeblair | i don't really understand why we're doing weird things with dns for cloud9 | 22:25 |
| jeblair | cloud8 | 22:25 |
| jeblair | can someone status log that? | 22:25 |
| jeblair | and maybe ping infra-root with an explanation | 22:26 |
| jeblair | cause that's super weird | 22:26 |
| mordred | jeblair: lemme tl;dr you first - and lets see if we can turn it into a useful status | 22:26 |
| * clarkb can backup explain | 22:26 | |
| mordred | jeblair: the networking in cloud8 is such that our mirror is behind the double nat - so our automation has no idea what the actual ip of the server is ... the cloud8 people are looking in to fixing this, but there are things outside of their immediate control | 22:27 |
| mordred | jeblair: in the mean time, it was suggested as a workaround to just use the cloud1 mirror since they're in the same data center by pointing the dns record there | 22:27 |
| mordred | that way the cloud8 people can work on getting the ips sorted in parallel | 22:27 |
| jeblair | gotcha | 22:27 |
| jeblair | mordred: i think i'd just status log that :) | 22:27 |
| mordred | that's a lot of status log :) | 22:28 |
| * mordred tries | 22:28 | |
| *** Sukhdev has joined #openstack-infra | 22:28 | |
| mordred | #status the networking in cloud8 is such that our mirror is behind the double nat - so our automation has no idea what the actual ip of the server is ... the cloud8 people are looking in to fixing this, but there are things outside of their immediate control | 22:28 |
| openstackstatus | mordred: unknown command | 22:28 |
| *** chem is now known as chem|off | 22:28 | |
| mordred | gah | 22:28 |
| mordred | it's #status log isn't it? | 22:28 |
| jeblair | yep | 22:28 |
| clarkb | ya | 22:28 |
| mordred | #status log the networking in cloud8 is such that our mirror is behind the double nat - so our automation has no idea what the actual ip of the server is ... the cloud8 people are looking in to fixing this, but there are things outside of their immediate control | 22:28 |
| openstackstatus | mordred: finished logging | 22:28 |
| mordred | #status log in the mean time, it was suggested as a workaround to just use the cloud1 mirror since they're in the same data center by pointing the dns record there | 22:29 |
| openstackstatus | mordred: finished logging | 22:29 |
| mordred | #status log that way the cloud8 people can work on getting the ips sorted in parallel | 22:29 |
| openstackstatus | mordred: finished logging | 22:29 |
| *** annegent_ has joined #openstack-infra | 22:30 | |
| jeblair | mordred: why doesn't this affect nodepool's use? | 22:31 |
| jeblair | (only affects v4?) | 22:32 |
| *** akshai has joined #openstack-infra | 22:32 | |
| mordred | jeblair: yah - this is only for the floating ip | 22:32 |
| mordred | jeblair: the single floating ip we have there - because the cloud doens't need any other ipv4 networks | 22:32 |
| jeblair | gotcha | 22:33 |
| *** annegentle has quit IRC | 22:33 | |
| *** mdrabe has quit IRC | 22:33 | |
| *** Sukhdev has quit IRC | 22:34 | |
| *** akshai has quit IRC | 22:34 | |
| *** signed8bit is now known as signed8bit_Zzz | 22:35 | |
| *** sbezverk_ has quit IRC | 22:36 | |
| *** piet has quit IRC | 22:37 | |
| pabelanger | the other issue is, eth1 is currently down. So, if that is our ipv6 interface, we still cannot SSH | 22:37 |
| *** shashank_hegde has joined #openstack-infra | 22:37 | |
| pabelanger | when I did the mirror is osic-cloud1, ipv4 was eth0 | 22:37 |
| mordred | pabelanger: yah | 22:37 |
| mordred | I mean - one way to make that easier ... | 22:37 |
| mordred | would be to attach the neutron router to GATEWAY_NET_V6 | 22:38 |
| mordred | so that the fip would attach to the ipv4 address on the same interface as the v6 | 22:38 |
| cloudnull | pabelanger: our error node launch attempt record is now ruined... trieste mi vida... :) | 22:38 |
| mordred | and the boot command would just be a single network - so a single nic | 22:38 |
| mordred | cloudnull: yah? | 22:39 |
| cloudnull | http://grafana.openstack.org/dashboard/db/nodepool-osic?panelId=11&fullscreen | 22:39 |
| cloudnull | ^ sad sad days. | 22:39 |
| pabelanger | pretty sure that is a nodepool failure, not osic-cloud1 | 22:39 |
| pabelanger | let me confirm | 22:39 |
| jeblair | what an utter failure! | 22:39 |
| jeblair | let's pack it all up and go home | 22:39 |
| cloudnull | jeblair: IRK... | 22:40 |
| cloudnull | it was good while it lasted | 22:40 |
| cloudnull | :) | 22:40 |
| pabelanger | Yup, IndexError | 22:40 |
| jeblair | pabelanger: what happened with the logging change? | 22:40 |
| pabelanger | jeblair: I'm just checking that actually | 22:40 |
| *** vhosakot has quit IRC | 22:41 | |
| pabelanger | 362455, needs a +3 | 22:41 |
| *** akshai has joined #openstack-infra | 22:42 | |
| jeblair | well, that would have been nice to have in before this restart | 22:42 |
| *** tphummel has quit IRC | 22:42 | |
| dhellmann | does anything special need to be done on the signing node to have it start picking up tag-releases jobs again because of the restart? | 22:44 |
| *** Sukhdev has joined #openstack-infra | 22:45 | |
| clarkb | dhellmann: we might have to requeue jobs | 22:46 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/project-config: Ensure that gerrit projects have zuul pipelines https://review.openstack.org/363969 | 22:47 |
| dhellmann | clarkb : I just approved a patch and there's a job queued up now | 22:47 |
| clarkb | jeblair: did you capture the release queue by chance when you restarted zuul? | 22:47 |
| *** zhurong has joined #openstack-infra | 22:47 | |
| *** mriedem has joined #openstack-infra | 22:47 | |
| dhellmann | clarkb : usually those are picked up in seconds | 22:47 |
| dhellmann | oh, this job wasn't even enqueued until 5 minutes ago | 22:47 |
| clarkb | dhellmann: hrm maybe I misunderstand what you are asking | 22:47 |
| dhellmann | there's a job queued up in release-post. it has been waiting for 6 minutes for the only node that can run it, which shouldn't be doing anything else afaik. is there a way to see what's in the queue for the special signing node? | 22:48 |
| dhellmann | I'm wondering if a worker lost contact with a server or something | 22:48 |
| jeblair | clarkb: no, someone said it was okay | 22:49 |
| clarkb | dhellmann: oh gotcha | 22:49 |
| dhellmann | jeblair : yeah, there weren't any queued up anyway | 22:50 |
| clarkb | the last log line for the zlstatic launcher is from 2211UTC | 22:50 |
| clarkb | I wonder if it didn't reregister with the gearman server after it restarted | 22:50 |
| clarkb | that log line was an onfinalized message so presumably it finished that job then didn't start any others | 22:51 |
| jeblair | clarkb: | 22:51 |
| jeblair | https://review.openstack.org/350807 | 22:51 |
| *** zhurong has quit IRC | 22:52 | |
| clarkb | does that mean I should restart hte zuul launcer on static now? | 22:52 |
| jeblair | clarkb: yep. also +3ing the change would be a nice touch. | 22:52 |
| clarkb | ya will review that as soon as I restart the launcher process | 22:53 |
| clarkb | hrm service restart didn't work | 22:53 |
| clarkb | it stopped but didn't start /me tries explicit start | 22:53 |
| jeblair | fungi: how would you feel about increasing the max line width to 100 in zuul? | 22:55 |
| *** Sukhdev has quit IRC | 22:55 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate: DNM: testing cellsv2 grenade/devstack run https://review.openstack.org/363971 | 22:55 |
| dhellmann | clarkb : there it goes | 22:55 |
| jeblair | tests/test_scheduler.py:386:80: E501 line too long (80 > 79 characters) dict(name='project-merge', result='SUCCESS', changes='1,1 2,1 3,1'), | 22:55 |
| clarkb | dhellmann: cool means restart worked | 22:56 |
| clarkb | now to review the fix | 22:56 |
| jeblair | fungi: i'm dealing with a bunch of those sorts of things -- it's a case where i think we're generally hurting legibility | 22:56 |
| dhellmann | clarkb , jeblair : thanks again! | 22:56 |
| jeblair | dhellmann: np, hopefully that's the last time we hit that error :) | 22:56 |
| dhellmann | jeblair : I have to wrap at 65 cols in my book, so I feel your pain | 22:56 |
| dhellmann | hmm, the lp comment script looks like it's hung again, though. telnet://signing01.ci.openstack.org:19885 | 22:57 |
| clarkb | jeblair: I can do 3 columns at 80 wide on current monitor which is kind of nice | 22:58 |
| jeblair | dhellmann, clarkb: this diff makes me sad: http://paste.openstack.org/show/565447/ | 22:58 |
| dhellmann | jeblair: yeah | 22:59 |
| dhellmann | and there goes the lp script, too | 22:59 |
| dhellmann | woot, we just tagged a release in ci | 22:59 |
| *** rbrndt has quit IRC | 22:59 | |
| jeblair | all hail ci releasing overlords | 22:59 |
| jeblair | dhellmann: that means you're free to write books now, right? | 23:00 |
| dhellmann | jeblair : it at least means I can go on vacation on the final release date again, since I can do that from my phone | 23:00 |
| pabelanger | Yay, that is nice | 23:01 |
| *** annegent_ has quit IRC | 23:01 | |
| *** gouthamr has joined #openstack-infra | 23:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Re-enable test_failed_change_at_head https://review.openstack.org/363821 | 23:02 |
| dhellmann | I'm going to try another one, skipping the "clarkb restarts the service" step this time. ;-) | 23:03 |
| fungi | sorry, had to disappear to make/eat food. catching back up | 23:03 |
| dhellmann | fungi : release automation is working! | 23:03 |
| *** pradk has quit IRC | 23:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Slowly bring internap-mtl01 back online https://review.openstack.org/363931 | 23:04 |
| fungi | mordred: jeblair: clarkb: one thing i didn't even bring up, but i have a suspicion that afs through a double-nat would be... troublesome | 23:06 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Revert "Enable infracloud servers in Nodepool"" https://review.openstack.org/363942 | 23:07 |
| fungi | though maybe it would just be fine | 23:07 |
| jeblair | fungi: well, any nat is 'trouble' so a double nat is 'double trouble'.... | 23:07 |
| jeblair | fungi: but theoretically, it maybe might still possibly work. | 23:07 |
| rcarrillocruz | weeeeee | 23:08 |
| jeblair | fungi: as long as you don't have more than one client | 23:08 |
| *** esp has quit IRC | 23:08 | |
| *** Benj_ has quit IRC | 23:09 | |
| fungi | jeblair: you mean increase the max width of lines in code? i could survive it | 23:09 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Add stable-maint-core to os-vif gerrit ACL https://review.openstack.org/363978 | 23:10 |
| pabelanger | here we go | 23:10 |
| pabelanger | internap-mtl01 and infracloud-vanilla just came online in nodepool | 23:10 |
| mgagne | :D | 23:10 |
| *** edtubill has quit IRC | 23:10 | |
| fungi | jeblair: i mean, conventional wisdom is that if you have overly long lines because of lots of levels of indentation, you need to break it up into more modular functions/methods, but sometimes that's just bs | 23:10 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: wait to publish releases.o.o until after tagging https://review.openstack.org/363979 | 23:11 |
| mgagne | yea, saw the merge and update in grafana | 23:11 |
| *** coreyob has quit IRC | 23:11 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 23:11 |
| mgagne | fungi: when you know that line length limit takes its origin from punch cards :D | 23:11 |
| rcarrillocruz | 2016-08-31 23:12:37,481 INFO nodepool.NodeLauncher: Creating server with hostname ubuntu-xenial-infracloud-vanilla-4022814 in infracloud-vanilla from image ubuntu-xenial for node id: 4022814 | 23:12 |
| pabelanger | rcarrillocruz: did you say ipv6 is now working in infracloud network? | 23:13 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Add nova-stable-maint to os-vif gerrit ACL https://review.openstack.org/363978 | 23:14 |
| rcarrillocruz | haven't asked to the NET folks yet | 23:14 |
| rcarrillocruz | will follow up with my EMEA contact tomorrow | 23:14 |
| *** tpsilva has quit IRC | 23:14 | |
| pabelanger | rcarrillocruz: cool | 23:14 |
| *** jamielennox|away is now known as jamielennox | 23:14 | |
| fungi | mgagne: i have a fondness for punch cards | 23:14 |
| fungi | way better than programming a computer by reordering circuit boards in the frame | 23:15 |
| *** akshai_ has joined #openstack-infra | 23:15 | |
| mgagne | https://en.wikipedia.org/wiki/Characters_per_line | 23:15 |
| mgagne | with some model this number was either reduced by half to 40 CPL | 23:15 |
| mgagne | 40 max line length, awesome :D | 23:16 |
| jeblair | fungi: http://paste.openstack.org/show/565447/ is specifically what i'm looking at | 23:17 |
| pabelanger | rcarrillocruz: clarkb: I'm going to work on a buildimage job tomorrow to create ubuntu-minimal images for our control plan, to at least get the ball rolling | 23:17 |
| pabelanger | then see why the mirror failed with afs | 23:18 |
| rcarrillocruz | cool | 23:18 |
| *** Julien-zte has joined #openstack-infra | 23:19 | |
| *** akshai has quit IRC | 23:19 | |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: re-register functions after disconnect https://review.openstack.org/350807 | 23:21 |
| *** krtaylor has quit IRC | 23:22 | |
| *** jamielennox is now known as jamielennox|away | 23:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Upload nodepool images to osic-cloud8 https://review.openstack.org/357364 | 23:23 |
| *** gyee has quit IRC | 23:24 | |
| pabelanger | mgagne: nc 198.72.124.71 19885 | 23:24 |
| mgagne | so you found the one job running in mtl01 =) | 23:24 |
| clarkb | wait did the cahnge we needed for the builder not even merge yet :P ok we can restart it again now that there are no jobs | 23:24 |
| pabelanger | clarkb: ya | 23:25 |
| pabelanger | infracloud failed to schedule the node | 23:25 |
| *** salv-orlando has quit IRC | 23:25 | |
| *** Julien-zte has quit IRC | 23:25 | |
| *** hockeynut has joined #openstack-infra | 23:26 | |
| fungi | jeblair: yeah, the only way to wrap and keep it semi-readable is to switch to one parameter per line in those functions, but that just means a lot fewer function calls in your screen | 23:26 |
| *** jamielennox|away is now known as jamielennox | 23:28 | |
| *** hongbin has quit IRC | 23:28 | |
| pabelanger | mgagne: success | 23:28 |
| mgagne | :D | 23:28 |
| mgagne | pabelanger: up to you to increase quota. | 23:29 |
| pabelanger | mgagne: lets see what tomorrow holds | 23:30 |
| pabelanger | but I don't see a reason not too | 23:30 |
| *** nwkarsten has joined #openstack-infra | 23:30 | |
| mgagne | just less humans to respond in case of problem tonight =) | 23:30 |
| mgagne | pabelanger: 150 would be a reasonable value, (up from previous suggested 120) | 23:31 |
| *** fguillot has quit IRC | 23:31 | |
| mgagne | maybe prepare the change and merge tomorrow | 23:31 |
| pabelanger | mgagne: Yup, I'm about to walk away for the night, but feel free to propose it | 23:31 |
| mgagne | ok, will do | 23:31 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Add cinder-stable-maint to os-brick https://review.openstack.org/363982 | 23:32 |
| rcarrillocruz | i'm also walking away | 23:32 |
| *** esp has joined #openstack-infra | 23:32 | |
| rcarrillocruz | talk to you tomorrow folks | 23:32 |
| rcarrillocruz | g'night | 23:32 |
| pleia2 | rcarrillocruz: nice job today! night :) | 23:33 |
| *** kzaitsev_mb has quit IRC | 23:33 | |
| *** krtaylor has joined #openstack-infra | 23:34 | |
| *** coreyob has joined #openstack-infra | 23:35 | |
| *** jamielennox is now known as jamielennox|away | 23:36 | |
| *** gongysh has joined #openstack-infra | 23:37 | |
| *** Swami has quit IRC | 23:37 | |
| *** oomichi has quit IRC | 23:39 | |
| *** gildub has joined #openstack-infra | 23:39 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config: Set max-servers value to 150 for internap-mtl01 https://review.openstack.org/363984 | 23:39 |
| *** oomichi has joined #openstack-infra | 23:39 | |
| *** oomichi has quit IRC | 23:40 | |
| *** gyee has joined #openstack-infra | 23:41 | |
| *** Sukhdev has joined #openstack-infra | 23:41 | |
| *** claudiub has quit IRC | 23:42 | |
| *** Sukhdev has quit IRC | 23:43 | |
| *** yuanying has quit IRC | 23:43 | |
| *** yuanying has joined #openstack-infra | 23:44 | |
| *** oomichi has joined #openstack-infra | 23:44 | |
| *** fguillot has joined #openstack-infra | 23:48 | |
| *** jerryz has quit IRC | 23:49 | |
| *** dingyichen has joined #openstack-infra | 23:54 | |
| clarkb | mgagne: you happy for ^ to happen whenever? | 23:58 |
| mgagne | pabelanger: suggested we merge this change tomorrow | 23:58 |
| mgagne | clarkb: pabelanger suggested ^ | 23:58 |
| mgagne | clarkb: since there will be more humans available to react | 23:59 |
| clarkb | kk I +2'd it | 23:59 |
| mgagne | clarkb: I'm leaving the office now, but it's up to you or any infra-root | 23:59 |
| *** rwsu has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!