| fungi | doing 10 in parallel over the ~300 which had been in a delete state for at least 0.3 hours | 00:00 |
|---|---|---|
| jeblair | StevenK: no, a change to nodepool ended up slowing our node turnover rate; we've reverted it and are cleaning up | 00:00 |
| StevenK | jeblair: Ah, excellent. | 00:01 |
| *** oubiwann_ has quit IRC | 00:01 | |
| *** oanufriev has quit IRC | 00:02 | |
| *** oubiwann_ has joined #openstack-infra | 00:02 | |
| *** sarob has quit IRC | 00:03 | |
| *** sarob has joined #openstack-infra | 00:03 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 00:04 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add oslo fixture module and update everything else https://review.openstack.org/69830 | 00:04 |
| *** mrodden has joined #openstack-infra | 00:05 | |
| notmyname | how is a new meeting time scheduled? | 00:06 |
| notmyname | doesn't seem like I was able to catch ttx while he was online today | 00:07 |
| *** mestery has joined #openstack-infra | 00:07 | |
| jeblair | notmyname: add it to the wiki page | 00:07 |
| jeblair | notmyname: ttx watches that and will update the ical feed | 00:07 |
| notmyname | jeblair: I did | 00:07 |
| notmyname | ah ok | 00:07 |
| notmyname | I'll send him an email too. thanks | 00:07 |
| jeblair | notmyname: cool | 00:07 |
| *** sarob has quit IRC | 00:07 | |
| *** mestery has quit IRC | 00:08 | |
| jeblair | fungi: yay noticible upticks on the graphs | 00:09 |
| fungi | seems so anyway | 00:09 |
| clarkb | jog0: elasticsearch/logstash is just over an hour behind | 00:09 |
| fungi | at peak today, the jobs launched graph says we were pushing ~1000/hr | 00:10 |
| clarkb | wow | 00:10 |
| fungi | with gerrit events up around 400/hr | 00:10 |
| clarkb | fungi: is now still a bad time to start an image build? | 00:10 |
| *** krotscheck has joined #openstack-infra | 00:11 | |
| *** wchrisj__ has quit IRC | 00:11 | |
| *** wchrisj_ has joined #openstack-infra | 00:11 | |
| fungi | clarkb: if you pick one of the check images or a provider with low representation, it's probably not too impactful | 00:11 |
| * fungi assumes we don't currently have a good way to trigger new image builds besides deleting an existing image | 00:12 | |
| clarkb | fungi: was thinking devstack-precise-check and a bare-precise to start | 00:12 |
| jog0 | clarkb: cool thanks | 00:12 |
| clarkb | fungi: there is an image built command | 00:12 |
| fungi | clarkb: sure, why not. sounds fine | 00:12 |
| fungi | oh, nodepool image-build? | 00:12 |
| fungi | i should --help more often | 00:12 |
| clarkb | image-update iirc | 00:12 |
| *** ruhe has joined #openstack-infra | 00:13 | |
| fungi | so there is. lookie there | 00:13 |
| fungi | i think i must have known about that | 00:13 |
| clarkb | building bare-precise in rax-ord first | 00:13 |
| jeblair | fungi: how are your deletes coming along? | 00:14 |
| fungi | oh, i know why i had sometimes resorted to deletes. when a provider is failing to build a fresh image 90% of the time, it gets the weeks-old stale image out of circulation and causes nodepool to loop creation until it sticks. ugly, but it works | 00:14 |
| fungi | jeblair: still running | 00:14 |
| *** sarob has joined #openstack-infra | 00:15 | |
| fungi | we seem to be turning over building->used about as quickly as deletes are getting cleared, so the building count isn't jumping much but the used count sure is | 00:16 |
| jeblair | fungi: i wonder if we should have a max-age for images (where it does that for us)? | 00:16 |
| *** wchrisj_ has quit IRC | 00:16 | |
| fungi | jeblair: that, and also i need to spend some time coming up with a better answer for weeks-old images than "that provider az is terrible" | 00:16 |
| *** chris_johnson has joined #openstack-infra | 00:17 | |
| fungi | clearly there's something we're not accomodating with regard to the situation. first skim of the logs suggested ruby timeouts in the puppet apply | 00:17 |
| jeblair | gah | 00:17 |
| fungi | but only on hpcloud-az2 | 00:17 |
| fungi | and the -bare images had it worse than -devstack | 00:18 |
| fungi | maybe because moar puppet or something | 00:18 |
| jeblair | fungi: clarkb's change gives us less puppet | 00:19 |
| jeblair | so yay | 00:19 |
| fungi | yes | 00:19 |
| clarkb | but it doesn't seem to be working | 00:19 |
| jeblair | the spike is rounding off | 00:19 |
| clarkb | the value passed to the sudo flag was an empty string | 00:19 |
| jeblair | clarkb: did nodepool detect the failure correctly? | 00:20 |
| *** mgagne has joined #openstack-infra | 00:20 | |
| jog0 | random question: any way to get a list of blueprints via launchpadlib? | 00:20 |
| *** matsuhashi has joined #openstack-infra | 00:20 | |
| jeblair | clarkb: (so we don't have to worry about it using nodes from borked images) | 00:20 |
| clarkb | jeblair: yes, it bailed out, threw an exception and died | 00:20 |
| jeblair | jog0: probably, and probably by querying the 'openstack' project group | 00:20 |
| clarkb | jeblair: and said INFO:nodepool.NodePool:Deleted image id: 154829 | 00:21 |
| jeblair | clarkb: cool | 00:21 |
| jeblair | oh yay, the gate got nodes | 00:22 |
| *** jergerber has quit IRC | 00:22 | |
| *** mgagne1 has joined #openstack-infra | 00:23 | |
| jog0 | jeblair: just found the asnwer https://github.com/markmc/openstack-lp-scripts/blob/master/mistargeted-blueprints.py | 00:24 |
| *** david-lyle has quit IRC | 00:24 | |
| *** mgagne has quit IRC | 00:25 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: make command parsing compatible with python 2.6 https://review.openstack.org/71424 | 00:25 |
| *** sarob has quit IRC | 00:25 | |
| jeblair | jog0: cool | 00:25 |
| *** sarob has joined #openstack-infra | 00:26 | |
| *** esker has joined #openstack-infra | 00:26 | |
| clarkb | sudo puppet apply --modulepath=/root/config/modules:/etc/puppet/modules -e 'class {'\''openstack_project::single_use_slave'\'': sudo => , }' | 00:28 |
| jog0 | jeblair: trying to generate a list of patchs that don't have approved BPs | 00:28 |
| clarkb | not sure why $SUDO became '' there | 00:28 |
| StevenK | clarkb: Undefined? | 00:28 |
| clarkb | StevenK: earlier in the script it is set to $2 which should be set | 00:29 |
| jeblair | jog0: btw, this is the kind of thing we want to make a lot easier with storyboard | 00:29 |
| jog0 | jeblair: cool | 00:29 |
| clarkb | StevenK: I suppose it could be a problem a few levels higher | 00:29 |
| jog0 | actually this should be pretty easy now | 00:29 |
| *** markwash has quit IRC | 00:29 | |
| *** ryanpetrello has joined #openstack-infra | 00:29 | |
| jog0 | lp.projects("nova").valid_blueprints is all valid blueprints | 00:29 |
| jog0 | then just need to search through gerrit commit messages | 00:30 |
| *** sarob has quit IRC | 00:30 | |
| jeblair | jog0: well, i mean making this kind of workflow easier with closer integration (with storyboard tasks directly linked to gerrit reviews) | 00:30 |
| jog0 | jeblair: right | 00:31 |
| jog0 | this is a ugly hack until then | 00:31 |
| *** reed has quit IRC | 00:31 | |
| jeblair | jog0: yeah, in the future, it can be in the ui and not just a script | 00:31 |
| jeblair | jog0: just an fyi, not attempting to alter your current plans | 00:32 |
| fungi | clarkb: is there any chance that it's using the old slave scripts for some reason? | 00:32 |
| *** krotscheck has quit IRC | 00:32 | |
| fungi | clarkb: nevermind. then it would have old prepare_node.sh too | 00:33 |
| jeblair | we have 492 nodes in use and 12 being deleted | 00:33 |
| clarkb | fungi: ya the only way would be new prepare_node.sh and old prepare_node_bare.sh | 00:33 |
| jeblair | and 66 being built | 00:33 |
| jog0 | jeblair: I look forward to that day | 00:33 |
| clarkb | fungi: which maybe that is the case? /me reads up on how file copies happen | 00:33 |
| *** sarob has joined #openstack-infra | 00:33 | |
| fungi | jeblair: yeah, my delete loops all completed | 00:34 |
| clarkb | the file in /etc/nodepool/scripts are up to date | 00:34 |
| *** CaptTofu has quit IRC | 00:34 | |
| *** CaptTofu has joined #openstack-infra | 00:34 | |
| * clarkb tries again, maybe those files were not up to date previously | 00:34 | |
| fungi | jeblair: it's possible it also lost track of some used nodes which should have transitioned to delete, but i'll give it a couple hours and then do a follow-on pass over slaves which have been used for far too long | 00:35 |
| *** krotscheck has joined #openstack-infra | 00:35 | |
| jeblair | clarkb: i believe i saw them get updated when i ran puppet agent --test | 00:35 |
| clarkb | jeblair: was that a few hours ago? | 00:35 |
| jeblair | fungi: yeah, hopefully not too many if the downtime was short | 00:35 |
| *** oubiwann_ has quit IRC | 00:35 | |
| zaro | clarkb: could you kindly unblock me from testing zuul-gerrit? https://review.openstack.org/#/c/71410 | 00:35 |
| jeblair | clarkb: right around 22:30 based on my irc log | 00:35 |
| clarkb | zaro: done | 00:36 |
| clarkb | jeblair: thanks | 00:36 |
| zaro | awesome, thanks! | 00:36 |
| jeblair | i wish the yaml parser could catch that | 00:36 |
| *** ryanpetrello has quit IRC | 00:37 | |
| * fungi needs to take a break and have some semblance of an evening/dinner/whatev... bbiab | 00:37 | |
| jeblair | fungi: good evening, thanks! | 00:37 |
| *** yamahata has joined #openstack-infra | 00:37 | |
| jeblair | i'm going to wind down too | 00:37 |
| clarkb | is false being interpretted as a shell built in? | 00:37 |
| jeblair | maybe tomorrow i'll get to work on zuul | 00:37 |
| clarkb | it shouldn't be, it is in an argument list | 00:37 |
| *** chris_johnson has quit IRC | 00:37 | |
| *** jcoufal has joined #openstack-infra | 00:39 | |
| clarkb | still doing it so yeah files were probably up to date a long time ago | 00:39 |
| *** markwash has joined #openstack-infra | 00:40 | |
| *** shivh has quit IRC | 00:43 | |
| *** sarob has quit IRC | 00:44 | |
| *** ken1ohmichi has joined #openstack-infra | 00:44 | |
| *** sarob has joined #openstack-infra | 00:44 | |
| openstackgerrit | A change was merged to openstack-infra/config: remove extra projects section https://review.openstack.org/71410 | 00:46 |
| *** shivh has joined #openstack-infra | 00:47 | |
| *** markwash has quit IRC | 00:48 | |
| *** jnoller has joined #openstack-infra | 00:48 | |
| *** sarob has quit IRC | 00:49 | |
| *** atiwari has quit IRC | 00:49 | |
| *** thuc has quit IRC | 00:50 | |
| *** sarob has joined #openstack-infra | 00:50 | |
| *** dcramer_ has joined #openstack-infra | 00:51 | |
| *** thuc has joined #openstack-infra | 00:51 | |
| *** Sukhdev has quit IRC | 00:54 | |
| *** thuc has quit IRC | 00:55 | |
| *** mgagne has joined #openstack-infra | 00:55 | |
| *** nosnos has joined #openstack-infra | 00:56 | |
| *** mgagne1 has quit IRC | 00:57 | |
| *** shivh has quit IRC | 00:58 | |
| *** markmcclain has joined #openstack-infra | 01:00 | |
| *** sarob has quit IRC | 01:01 | |
| *** sarob has joined #openstack-infra | 01:01 | |
| *** thuc has joined #openstack-infra | 01:01 | |
| *** jnoller has quit IRC | 01:04 | |
| *** thuc has quit IRC | 01:04 | |
| *** ryanpetrello has joined #openstack-infra | 01:05 | |
| *** sarob has quit IRC | 01:06 | |
| *** thuc has joined #openstack-infra | 01:06 | |
| *** ryanpetrello has quit IRC | 01:07 | |
| *** sarob has joined #openstack-infra | 01:07 | |
| *** mgagne has quit IRC | 01:07 | |
| jog0 | jeblair: the whole script to get the patches with unapproved BPs is just 50 lines | 01:09 |
| *** mrodden has quit IRC | 01:10 | |
| *** hogepodge has quit IRC | 01:11 | |
| *** stack555 has quit IRC | 01:12 | |
| *** bhuvan has joined #openstack-infra | 01:12 | |
| *** jdurgin has quit IRC | 01:15 | |
| *** igor has joined #openstack-infra | 01:15 | |
| *** igor____ has quit IRC | 01:16 | |
| *** jdurgin has joined #openstack-infra | 01:17 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Be explicit about node_prepare.sh args. https://review.openstack.org/71432 | 01:18 |
| *** ruhe has quit IRC | 01:18 | |
| clarkb | jeblair: fungi ^ that is the fix for the scripts. By the way, bare-precise does not include mysql and postgres and a bunch of other stuff for testing | 01:18 |
| clarkb | but that isn't a result of my change, it has been that way for a while | 01:19 |
| fungi | oh | 01:19 |
| *** markmcclain has quit IRC | 01:21 | |
| mattoliverau | jeblair: re: nodepool image logs, is this what you had in mind: http://is.gd/AjNsIK | 01:21 |
| *** prad has joined #openstack-infra | 01:21 | |
| *** esmute has quit IRC | 01:22 | |
| fungi | clarkb: how have nova py27 jobs been working with no mysql? | 01:22 |
| clarkb | actually hmm, maybe my change changes that, looking at it more | 01:22 |
| clarkb | fungi: those tests don't run if mysql isn't present, but yeah I think this may have worked before. I had a small misunderstanding | 01:23 |
| *** markmcclain has joined #openstack-infra | 01:23 | |
| clarkb | so while the old bare_slave application could not fix sudo, it did install the additional packages and things | 01:23 |
| clarkb | fungi: fixing now | 01:24 |
| fungi | i'm still around to approve stuff if you want to keep prodding it. i'll postpone any particularly heavy drinking | 01:25 |
| *** cody-somerville has quit IRC | 01:25 | |
| clarkb | fungi: thank you, should have a second patch up in must a few minutes | 01:25 |
| *** thuc_ has joined #openstack-infra | 01:25 | |
| fungi | the first one lgtm too... i had started pondering down those lines, but it didn't dawn on me that hostname isn't set sometimes (doesn't the default shell environment have it even?) | 01:26 |
| *** mrda is now known as mrda_away | 01:27 | |
| clarkb | its confusing because we call it a bare node but it isn't a bare node | 01:27 |
| *** gothicmindfood has quit IRC | 01:27 | |
| fungi | it's less bare than the devstack nodes in some ways | 01:27 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] MVP Storyboard Client https://review.openstack.org/70897 | 01:28 |
| *** thuc has quit IRC | 01:29 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Handle the slave bare flag properly in nodepool. https://review.openstack.org/71433 | 01:29 |
| clarkb | fungi: ^ there we go | 01:29 |
| clarkb | fungi: I am going to apply those by hand on nodepool.o.o and rebuild rax-ord's iamge | 01:29 |
| *** thuc_ has quit IRC | 01:30 | |
| *** bhuvan has quit IRC | 01:30 | |
| *** wenlock has quit IRC | 01:31 | |
| fungi | l | 01:31 |
| fungi | er, okay | 01:31 |
| *** thuc has joined #openstack-infra | 01:32 | |
| clarkb | image is rebuilding | 01:32 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: [WIP] MVP API for Storyboard https://review.openstack.org/71435 | 01:33 |
| *** mgagne has joined #openstack-infra | 01:33 | |
| *** freyes has joined #openstack-infra | 01:33 | |
| openstackgerrit | A change was merged to openstack-infra/config: Be explicit about node_prepare.sh args. https://review.openstack.org/71432 | 01:34 |
| *** krotscheck has quit IRC | 01:34 | |
| clarkb | fungi: so while applying bare_slave.pp was intended to remove sudo rights it actually added a bunch of needed packages | 01:35 |
| clarkb | and i missed it until just now :/ | 01:35 |
| fungi | clarkb: 71433 is a bit too quick and dirty for my tastes. the true and false parameters need some context. how about we just gratuitously set those as local vars and then pass the varnames on script invocation? | 01:35 |
| clarkb | fungi: sure | 01:36 |
| *** markmcclain has left #openstack-infra | 01:36 | |
| fungi | it would make it far more readable without needing to flip back and forth through the spaghetti of scripts calling scripts to figure out what the parameters are | 01:36 |
| *** sc68cal has quit IRC | 01:37 | |
| * fungi is having sco flashbacks. everything was a tangle of ksh scripts invoking one another | 01:37 | |
| fungi | if you never admin'ed sco unix, count yourself lucky. it was even worse than hp/ux, and i did rather hate hp/ux too | 01:38 |
| clarkb | oh gah, locally applying that to nodepool.o.o doesn't help because the git repo is cloned | 01:38 |
| * clarkb hurries up to address comments so we can merge this | 01:38 | |
| *** jcoufal has quit IRC | 01:39 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Handle the slave bare flag properly in nodepool. https://review.openstack.org/71433 | 01:40 |
| clarkb | fungi: ^ there you go | 01:40 |
| fungi | lookin' | 01:40 |
| *** jcoufal has joined #openstack-infra | 01:40 | |
| *** paul-- has quit IRC | 01:41 | |
| *** jnoller has joined #openstack-infra | 01:42 | |
| *** amcrn has quit IRC | 01:43 | |
| fungi | yep, lgtm. i had to spend a moment convincing myself that quoting the var which was assigned a quoted string wouldn't add a layer of quotes | 01:44 |
| fungi | ' vs " brain fart | 01:44 |
| fungi | wow, my initial approve on that caused gerrit to return an internal server error page | 01:45 |
| clarkb | o_O | 01:45 |
| fungi | retried and it worked | 01:45 |
| clarkb | I have deleted the bad rax-ord image manually | 01:46 |
| clarkb | there may be some straggler nodes out there though :( | 01:46 |
| fungi | yeah, that's one thing nodepool misses tracking is the image id associated with a given node | 01:47 |
| *** balar has quit IRC | 01:47 | |
| *** paul-- has joined #openstack-infra | 01:47 | |
| fungi | so you can't say "let's delete all nodes built from image x" | 01:47 |
| fungi | built from image id x, not image name x (it knows the latter, of course) | 01:48 |
| *** balar has joined #openstack-infra | 01:51 | |
| fungi | "No local packages or download links found for pbr>=0.5.16,<0.6" | 01:51 |
| clarkb | what failed on that? | 01:52 |
| fungi | that requirements change dying at the top of the gate right now | 01:52 |
| fungi | hit it in the requirements-integration job | 01:52 |
| clarkb | iirc that job talks to pypi.python.org | 01:52 |
| fungi | https://pypi.python.org/simple/pbr/ shows plenty of versions in the acceptable range for that | 01:53 |
| *** weshay has joined #openstack-infra | 01:54 | |
| openstackgerrit | A change was merged to openstack-infra/config: Handle the slave bare flag properly in nodepool. https://review.openstack.org/71433 | 01:55 |
| clarkb | ok trying a new build of rax-ord bare-precise in jus a moment | 01:56 |
| *** sc68cal has joined #openstack-infra | 01:56 | |
| *** mestery has joined #openstack-infra | 01:57 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Map failed jobs to bugs in gerrit comment https://review.openstack.org/71148 | 01:57 |
| clarkb | gah looks like the postgres stuff may have failed | 02:02 |
| *** gokrokve has quit IRC | 02:02 | |
| clarkb | this should cause the build to fail though and I am about ready to deal with day old images and fix it tomorrow | 02:02 |
| *** jnoller has quit IRC | 02:04 | |
| *** ianw has quit IRC | 02:06 | |
| *** ianw has joined #openstack-infra | 02:06 | |
| *** ryanpetrello has joined #openstack-infra | 02:07 | |
| *** melwitt has quit IRC | 02:08 | |
| clarkb | https://github.com/puppetlabs/puppetlabs-postgresql/commit/acabbbf53066a9b760def6136d8f2d7b8ac65738 need to bump the version of puppet-postgresql to 3.1.0 | 02:10 |
| clarkb | if this image build properly fails and is deleted I will push that off until tomorrow | 02:10 |
| clarkb | hrm doesn't look like it will fail nicely | 02:12 |
| *** prad has quit IRC | 02:13 | |
| *** sarob_ has joined #openstack-infra | 02:14 | |
| *** mestery has quit IRC | 02:15 | |
| *** sarob has quit IRC | 02:16 | |
| *** dcramer_ has quit IRC | 02:16 | |
| ArxCruz | jhesketh__: clarkb do you have any documentation about how to configure jcloud + swift ? | 02:16 |
| clarkb | ArxCruz: no, its an upstream plugin | 02:16 |
| ArxCruz | clarkb: do you know the plugin name? Just jclouds? I have one installed on my jenkins, however, it seems to be pre-configured with some cloud services | 02:17 |
| ArxCruz | I have my own swift server :/ | 02:17 |
| clarkb | jclouds jenkins plugin | 02:17 |
| clarkb | it should support raw swift as well | 02:18 |
| *** jnoller has joined #openstack-infra | 02:18 | |
| *** jcoufal has quit IRC | 02:18 | |
| *** sarob_ has quit IRC | 02:18 | |
| *** nicedice has quit IRC | 02:19 | |
| *** nicedice has joined #openstack-infra | 02:19 | |
| *** ryanpetrello has quit IRC | 02:20 | |
| ArxCruz | clarkb: oh, mordred is one of the maintainers | 02:23 |
| ArxCruz | mordred: help! :D | 02:23 |
| ArxCruz | mordred: well, I can bug you tomorrow, good night :) | 02:23 |
| *** sarob has joined #openstack-infra | 02:25 | |
| *** morganfainberg is now known as morganfainberg_Z | 02:28 | |
| *** thuc has quit IRC | 02:28 | |
| *** mrda_away is now known as mrda | 02:29 | |
| *** thuc has joined #openstack-infra | 02:29 | |
| clarkb | https://github.com/puppetlabs/puppetlabs-postgresql/issues/336 so I think this must have been happening for a while | 02:31 |
| *** thuc has quit IRC | 02:33 | |
| *** gothicmindfood has joined #openstack-infra | 02:34 | |
| *** mgagne has quit IRC | 02:34 | |
| *** banix has joined #openstack-infra | 02:37 | |
| *** sc68cal has quit IRC | 02:38 | |
| *** jnoller has quit IRC | 02:39 | |
| banix | have a question about using git review; for a patch I have been making changes and doing git commit --amend and then git review. Now, I am getting an error when I do git review: | 02:41 |
| banix | error: could not apply e12dee1... Deals with fails in update-*-postcommit ops | 02:43 |
| banix | When you have resolved this problem run "git rebase --continue". | 02:43 |
| banix | If you would prefer to skip this patch, instead run "git rebase --skip". | 02:43 |
| banix | To check out the original branch and stop rebasing run "git rebase --abort". | 02:43 |
| banix | I did a git rebase --continue which I am think I shouldn't have | 02:43 |
| *** dims has joined #openstack-infra | 02:44 | |
| banix | now with review I get the following error: | 02:44 |
| banix | (you are not allowed to upload merges) | 02:44 |
| banix | error: failed to push some refs .... | 02:44 |
| *** harlowja has quit IRC | 02:44 | |
| *** dstufft|laptop has joined #openstack-infra | 02:45 | |
| banix | any suggestions as what I need to do to fix this? | 02:45 |
| clarkb | banix: you want to go back to the old state before the rebase (you can use git reflog to find the old sha), then do the rebase and fix the merge conficts before doing git rebase --continue | 02:47 |
| banix | clarkb: Thank you | 02:48 |
| *** jasondotstar has joined #openstack-infra | 02:50 | |
| *** gokrokve has joined #openstack-infra | 02:50 | |
| *** dkranz has quit IRC | 02:52 | |
| *** harlowja has joined #openstack-infra | 02:55 | |
| *** gokrokve has quit IRC | 02:55 | |
| *** mgagne has joined #openstack-infra | 02:55 | |
| *** gokrokve has joined #openstack-infra | 02:56 | |
| *** freyes has quit IRC | 02:57 | |
| *** mgagne has quit IRC | 02:59 | |
| *** gokrokve has quit IRC | 03:00 | |
| *** gokrokve has joined #openstack-infra | 03:00 | |
| *** mgagne has joined #openstack-infra | 03:02 | |
| *** sarob has quit IRC | 03:06 | |
| *** gokrokve has quit IRC | 03:06 | |
| clarkb | banix: I can be more verbose if it helps too :) sorry I just realized that my directions may be a bit vague | 03:06 |
| *** sarob has joined #openstack-infra | 03:06 | |
| clarkb | let me know if you want me to expand on anything | 03:06 |
| *** dstufft|laptop has quit IRC | 03:06 | |
| *** dstufft|laptop has joined #openstack-infra | 03:07 | |
| banix | clarkb: Thanks. Trying to figure out where to rebase to. so I have done a git reflag and | 03:07 |
| clarkb | banix: the rebase was started before because git-review detected a merge conflict. Were you proposing your change against master? if so you want to rebase onto an up to date master | 03:08 |
| *** zehicle_at_dell has quit IRC | 03:09 | |
| *** zehicle_at_dell has joined #openstack-infra | 03:09 | |
| *** briancline has quit IRC | 03:10 | |
| banix | so would I do that by making my master up to date and then rebasing my branch? | 03:10 |
| clarkb | yup | 03:11 |
| banix | I may not make sense so let me explain a bit more. I have a master and say branch b1 where I do my work. So I should go to master and do a git pull … there, come back to b1 and rebase? | 03:11 |
| clarkb | yes, I like to do `git remote update ; git checkout master ; #check that I can fast forward ; git pull origin master ; git checkout b1 ; git rebase -i master` | 03:12 |
| *** coolsvap_away has quit IRC | 03:13 | |
| hub_cap | random Q. does gerrit support git notes? like if i wanted to put a list of Co-Authors or a list of dependent reviews (for instance in another git tree, referring to their change-id) | 03:13 |
| StevenK | clarkb: If you never make changes on master, you should always be able to fast-forward | 03:13 |
| clarkb | hub_cap: gerrit does, but that is where it writes review data | 03:13 |
| hub_cap | clarkb: for sure i sees that on commits | 03:13 |
| clarkb | hub_cap: so I am not sure how it will handle other people using them for stuff | 03:13 |
| banix | I start in b1 or master? | 03:13 |
| clarkb | hub_cap: eg I don't know if you can read them via gerrit | 03:14 |
| clarkb | banix: either one | 03:14 |
| *** sc68cal_ has joined #openstack-infra | 03:14 | |
| *** jhesketh__ has quit IRC | 03:14 | |
| hub_cap | clarkb: sure, or if itll clobber it when it writes its own notes :) | 03:14 |
| hub_cap | so the answer is, i should do some digging if im interested :D | 03:14 |
| clarkb | hub_cap: yup, feel free to give it a spin on review-dev | 03:15 |
| *** sc68cal_ has quit IRC | 03:15 | |
| banix | clarkb: Will this have any effect on my other branches? | 03:16 |
| clarkb | StevenK: right, don't assume that is the case :) | 03:16 |
| clarkb | StevenK: quite a few of us have bad habits of hacking on master and using git reset --hard later | 03:16 |
| clarkb | banix: it shouldn't | 03:16 |
| clarkb | banix: the git remote update will only fetch upstream changes not pull them | 03:16 |
| StevenK | clarkb: I've explicitly tried to avoid that particular habit | 03:16 |
| clarkb | StevenK: it is a noble thing, but once you have a gerrit... meh | 03:17 |
| banix | clarkb: Ok thanks; let me try it. | 03:17 |
| StevenK | clarkb: But branching is *so cheap* | 03:17 |
| hub_cap | clarkb: what is this review-dev in which u speak??? | 03:18 |
| *** sc68cal_ has joined #openstack-infra | 03:18 | |
| clarkb | StevenK: `git branch | wc -l` says I have 288 branches in openstack-infra/config | 03:18 |
| *** sc68cal_ has quit IRC | 03:18 | |
| clarkb | hub_cap: https://review-dev.openstack.org | 03:18 |
| StevenK | clarkb: Haha | 03:19 |
| hub_cap | what the deuce | 03:19 |
| *** jhesketh__ has joined #openstack-infra | 03:19 | |
| hub_cap | thx clarkb i will totally give it a whirl. i hate seeing all the Ugly-Extra-Junk: in_my_commits | 03:19 |
| StevenK | clarkb: I tend to git branch -d <foo> after a pull when it's merged | 03:19 |
| clarkb | StevenK: so for a long time we had openstack-infra/config set to cherry pick not merge. This meand git branch -d never worked | 03:20 |
| *** wenlock has joined #openstack-infra | 03:20 | |
| clarkb | so you had to cross your fingers and git branch -D, which leads to git reset --hard origin master | 03:20 |
| dims | Anyone recognize this? "Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class openstack_project::slave_template at line 2" - trying to followi README.rst from devstack-gate to build a vm locally | 03:20 |
| StevenK | clarkb: Oh, cherry-pick doesn't preserve the commits from the other branch? | 03:22 |
| clarkb | dims: oh fun, I need to update README.rst in d-g | 03:22 |
| clarkb | dims: sorry we just renamed some stuff to make it clearer in puppet. Try using openstack_project::single_use_slave instead | 03:22 |
| banix | clarkb: At the last step above 'git rebase -i master' I get an editor opened where I see noon at the top of the file and everything else commented (list of commands). Just close the editor or do I need to add anything to the file? | 03:22 |
| clarkb | StevenK: nope and Gerrit makes sure of it by changing the commit messages | 03:22 |
| banix | I see noop | 03:22 |
| dims | clarkb, thx will try | 03:22 |
| clarkb | banix: so your commit isn't listed? | 03:23 |
| clarkb | banix: your commit(s) should be listed with a prefix of 'pick' | 03:23 |
| dims | clarkb, much better :) | 03:23 |
| banix | I see noon then the lines are all starting with # and the first line i: # Rebase db6dafc..76a0486 onto db6dafc and then list of commands | 03:25 |
| banix | noon and noon | 03:25 |
| banix | noop not noon | 03:25 |
| banix | sorry cannot type! | 03:26 |
| *** lcheng_ has joined #openstack-infra | 03:26 | |
| *** prad has joined #openstack-infra | 03:26 | |
| *** prad has quit IRC | 03:28 | |
| *** sc68cal_ has joined #openstack-infra | 03:28 | |
| *** weshay has quit IRC | 03:29 | |
| *** thuc has joined #openstack-infra | 03:31 | |
| *** thuc_ has joined #openstack-infra | 03:32 | |
| banix | clarkb: any comment as to what I need to do next? | 03:33 |
| *** amotoki has joined #openstack-infra | 03:34 | |
| clarkb | banix: if local git thinks you are good I would try git review again on the b1 branch | 03:34 |
| *** odyssey4me has joined #openstack-infra | 03:35 | |
| banix | clarkb: None of my changes are in my branch any more | 03:35 |
| clarkb | that would explain why the rebase was clean, you need to restore your changes by referring to the reflog and checking out the sha with your changes on it | 03:36 |
| *** thuc has quit IRC | 03:36 | |
| banix | clarkb: ok thanks | 03:36 |
| *** zehicle_at_dell has quit IRC | 03:37 | |
| *** zehicle_at_dell has joined #openstack-infra | 03:38 | |
| *** sc68cal_ has quit IRC | 03:39 | |
| *** odyssey4me2 has joined #openstack-infra | 03:39 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add a testr specific builder https://review.openstack.org/70511 | 03:39 |
| *** odyssey4me has quit IRC | 03:40 | |
| clarkb | harlowja: we already have one of those... | 03:40 |
| harlowja | clarkb ? | 03:40 |
| harlowja | clarkb should i call it unittest (not sure of the best name) | 03:41 |
| clarkb | the commit message confuses me | 03:41 |
| clarkb | Extend unittest jenkins builder <- or similar might be slightly more indicatve of what the cahnge is doing | 03:41 |
| harlowja | clarkb k | 03:41 |
| harlowja | clarkb agreed | 03:41 |
| harlowja | fixing | 03:41 |
| clarkb | we do still run nose too :( | 03:41 |
| *** ken1ohmichi has quit IRC | 03:42 | |
| harlowja | ya, let me adjust the names, wasn't quite sure what was a good name :) | 03:42 |
| *** zehicle_at_dell has quit IRC | 03:42 | |
| *** zehicle_at_dell has joined #openstack-infra | 03:43 | |
| *** sc68cal has joined #openstack-infra | 03:45 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Extend unittest jenkins builder https://review.openstack.org/70511 | 03:46 |
| clarkb | thanks | 03:47 |
| harlowja | np | 03:47 |
| openstackgerrit | A change was merged to openstack/requirements: Update python-openstacksdk from the global-requirements file https://review.openstack.org/69701 | 03:50 |
| openstackgerrit | A change was merged to openstack/requirements: pbr: update to 0.6 https://review.openstack.org/70955 | 03:50 |
| openstackgerrit | A change was merged to openstack/requirements: Add support for Python 3 requirements https://review.openstack.org/58770 | 03:50 |
| *** david-lyle has joined #openstack-infra | 03:51 | |
| openstackgerrit | Jenkins proposed a change to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/71412 | 03:54 |
| jog0 | mordred: http://logs.openstack.org/17/70717/1/gate/gate-tempest-dsvm-neutron/5b46b42/logs/screen-key.txt.gz | 03:56 |
| jog0 | ImportError: No module named pbr.version | 03:56 |
| jog0 | where did the beer go | 03:56 |
| *** mriedem has quit IRC | 03:56 | |
| fungi | went flat | 03:57 |
| clarkb | fungi: I think I have the postgres trouble debugged and will submit a fix tomorrow, but everything should be working as is tonight (at least as compared to yesterday | 03:57 |
| *** emagana has quit IRC | 03:57 | |
| *** thuc_ has quit IRC | 03:58 | |
| *** thuc has joined #openstack-infra | 03:58 | |
| *** Hefeweiz1n has quit IRC | 03:59 | |
| fungi | clarkb: looking forward to seeing the fix--thanks | 03:59 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Rename slave_template to single_use_slave https://review.openstack.org/71450 | 04:02 |
| *** thuc has quit IRC | 04:03 | |
| *** lcheng_ has quit IRC | 04:06 | |
| *** dcramer_ has joined #openstack-infra | 04:08 | |
| *** HenryG has joined #openstack-infra | 04:09 | |
| *** pcrews has quit IRC | 04:10 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make gerrit/ES timeout configurable https://review.openstack.org/68799 | 04:15 |
| *** gokrokve has joined #openstack-infra | 04:24 | |
| lifeless | clarkb: what do you mean 'lifeless style of review' ? | 04:31 |
| lifeless | jog0: that was a mistake :) | 04:31 |
| jog0 | lifeless: heh I know, I was told to never let you live that down | 04:31 |
| clarkb | lifeless: you push commits for every little refactor | 04:32 |
| StevenK | fungi, clarkb: If you're still around, does it look like the running job on 71054,1 has hung? | 04:32 |
| clarkb | lifeless: when really one or two commits would cover the bases of distinct changes and still be very reviewable | 04:32 |
| clarkb | StevenK: it probably is, timeout will kick in soon I think | 04:33 |
| StevenK | clarkb: And then everything gets retried :-( | 04:33 |
| clarkb | yup | 04:33 |
| lifeless | clarkb: thats not a review style, its a commit style | 04:33 |
| lifeless | clarkb: </pedant> | 04:33 |
| lifeless | clarkb: I see jeblair is reverting the nodepool work - did we capture the data needed to postmortem? | 04:33 |
| StevenK | lifeless: Like you'll stop being a pedant. | 04:33 |
| lifeless | StevenK says, being a pedant | 04:34 |
| clarkb | lifeless: yes I believe we have the extra log data on the server | 04:34 |
| StevenK | :-D | 04:34 |
| lifeless | clarkb: ok cool; I'll need that if I'm to track down what went wrong | 04:34 |
| clarkb | lifeless: roger, will try to remember to dig it out of the logs tomorrow | 04:35 |
| *** lcheng_ has joined #openstack-infra | 04:38 | |
| *** CaptTofu has quit IRC | 04:42 | |
| *** jcooley_ has quit IRC | 04:43 | |
| *** coolsvap has joined #openstack-infra | 04:44 | |
| *** ryanpetrello has joined #openstack-infra | 04:46 | |
| *** sc68cal has quit IRC | 04:50 | |
| *** sc68cal has joined #openstack-infra | 04:50 | |
| *** gokrokve has quit IRC | 04:50 | |
| *** dstufft|laptop has quit IRC | 04:51 | |
| *** gokrokve has joined #openstack-infra | 04:52 | |
| *** gokrokve has quit IRC | 04:53 | |
| *** kraman1 has quit IRC | 04:56 | |
| *** jasondotstar has quit IRC | 04:57 | |
| *** nosnos_ has joined #openstack-infra | 05:02 | |
| *** wenlock has quit IRC | 05:02 | |
| *** dims has quit IRC | 05:03 | |
| *** Ryan_Lane has joined #openstack-infra | 05:03 | |
| *** dims has joined #openstack-infra | 05:04 | |
| *** sarob has quit IRC | 05:05 | |
| *** harlowja has quit IRC | 05:05 | |
| *** nosnos has quit IRC | 05:06 | |
| *** emagana has joined #openstack-infra | 05:08 | |
| *** sarob has joined #openstack-infra | 05:10 | |
| *** emagana has quit IRC | 05:13 | |
| *** ryanpetrello has quit IRC | 05:13 | |
| *** harlowja has joined #openstack-infra | 05:14 | |
| *** sarob has quit IRC | 05:15 | |
| *** kraman1 has joined #openstack-infra | 05:16 | |
| *** talluri has joined #openstack-infra | 05:17 | |
| *** dstufft|laptop has joined #openstack-infra | 05:17 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update gat trigger to match new workflow label on review-dev https://review.openstack.org/71458 | 05:18 |
| *** blamar has quit IRC | 05:21 | |
| *** malini_afk has quit IRC | 05:24 | |
| zaro | clarkb: almost verified every pipeline on zuul-dev. just need to verify one more, the gate. got time for a quick review? | 05:24 |
| *** dstufft|laptop has quit IRC | 05:25 | |
| zaro | clarkb: i can't believe how many times the radio stations are rerunning the superbowl game, it's getting rediculous. | 05:25 |
| *** malini_afk has joined #openstack-infra | 05:26 | |
| *** malini_afk is now known as malini | 05:27 | |
| *** mgagne1 has joined #openstack-infra | 05:35 | |
| *** dstufft-gone has quit IRC | 05:36 | |
| *** dstufft|laptop has joined #openstack-infra | 05:36 | |
| *** mgagne has quit IRC | 05:38 | |
| *** e0ne has joined #openstack-infra | 05:40 | |
| *** nicedice has quit IRC | 05:40 | |
| *** DinaBelova_ is now known as DinaBelova | 05:41 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update git trigger to match new workflow label on review-dev https://review.openstack.org/71458 | 05:42 |
| *** sandywalsh has quit IRC | 05:46 | |
| *** banix has quit IRC | 05:47 | |
| *** jcooley_ has joined #openstack-infra | 05:49 | |
| *** dstufft has joined #openstack-infra | 05:50 | |
| *** paul--_ has joined #openstack-infra | 05:56 | |
| *** emagana has joined #openstack-infra | 05:57 | |
| *** paul-- has quit IRC | 05:58 | |
| *** julim has joined #openstack-infra | 06:00 | |
| *** denis_makogon has joined #openstack-infra | 06:00 | |
| *** DinaBelova is now known as DinaBelova_ | 06:06 | |
| *** matsuhashi has quit IRC | 06:07 | |
| *** matsuhashi has joined #openstack-infra | 06:07 | |
| *** matsuhas_ has joined #openstack-infra | 06:09 | |
| *** matsuhashi has quit IRC | 06:09 | |
| *** jcooley_ has quit IRC | 06:11 | |
| *** sarob has joined #openstack-infra | 06:17 | |
| *** mgagne1 has quit IRC | 06:17 | |
| *** igor has quit IRC | 06:18 | |
| *** igor has joined #openstack-infra | 06:18 | |
| *** sarob has quit IRC | 06:21 | |
| *** jcooley_ has joined #openstack-infra | 06:24 | |
| *** denis_makogon_ has joined #openstack-infra | 06:24 | |
| *** dstufft|laptop has quit IRC | 06:25 | |
| *** denis_makogon has quit IRC | 06:26 | |
| *** mrda is now known as mrda_away | 06:27 | |
| *** e0ne has quit IRC | 06:27 | |
| *** paul--_ has quit IRC | 06:28 | |
| *** slong has quit IRC | 06:33 | |
| *** paul-- has joined #openstack-infra | 06:35 | |
| *** paul-- has quit IRC | 06:38 | |
| *** nosnos has joined #openstack-infra | 06:40 | |
| *** nosnos_ has quit IRC | 06:42 | |
| *** julim has quit IRC | 06:42 | |
| *** CaptTofu has joined #openstack-infra | 06:42 | |
| *** hdd_ has quit IRC | 06:46 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 06:48 | |
| *** coolsvap_away has joined #openstack-infra | 06:51 | |
| *** coolsvap has quit IRC | 06:52 | |
| *** markwash has joined #openstack-infra | 06:53 | |
| *** jpich has joined #openstack-infra | 06:54 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 06:54 | |
| *** dstanek has quit IRC | 06:55 | |
| *** HenryG_ has joined #openstack-infra | 06:55 | |
| *** hawi has quit IRC | 06:57 | |
| *** hawi has joined #openstack-infra | 06:57 | |
| *** yamahata__ has quit IRC | 06:57 | |
| *** yamahata__ has joined #openstack-infra | 06:57 | |
| *** bhuvan has joined #openstack-infra | 06:57 | |
| *** HenryG has quit IRC | 06:58 | |
| *** masayukig has quit IRC | 06:58 | |
| *** mugsie has quit IRC | 07:00 | |
| *** mugsie has joined #openstack-infra | 07:01 | |
| *** ok_delta has joined #openstack-infra | 07:01 | |
| *** coolsvap_away has quit IRC | 07:02 | |
| *** esmute has joined #openstack-infra | 07:04 | |
| *** nati_ueno has quit IRC | 07:06 | |
| *** coolsvap has joined #openstack-infra | 07:07 | |
| *** hawi has quit IRC | 07:07 | |
| *** hawi has joined #openstack-infra | 07:07 | |
| *** CaptTofu has quit IRC | 07:08 | |
| *** coolsvap_away has joined #openstack-infra | 07:11 | |
| *** coolsvap has quit IRC | 07:12 | |
| *** lcheng_ has quit IRC | 07:14 | |
| *** kraman1 has quit IRC | 07:15 | |
| *** lcheng_ has joined #openstack-infra | 07:15 | |
| *** harlowja is now known as harlowja_away | 07:23 | |
| *** markwash has quit IRC | 07:24 | |
| *** rlandy has joined #openstack-infra | 07:27 | |
| *** bhuvan has quit IRC | 07:28 | |
| *** yolanda has joined #openstack-infra | 07:30 | |
| *** markwash has joined #openstack-infra | 07:33 | |
| *** coolsvap_away is now known as coolsvap | 07:37 | |
| *** flaper87|afk is now known as flaper87 | 07:39 | |
| *** emagana has quit IRC | 07:43 | |
| *** e0ne has joined #openstack-infra | 07:47 | |
| *** sc68cal is now known as sc68cal_ | 07:50 | |
| *** nibalizer has joined #openstack-infra | 07:51 | |
| *** sarob has joined #openstack-infra | 07:52 | |
| *** sc68cal_ is now known as sc68cal | 07:52 | |
| *** julim has joined #openstack-infra | 07:54 | |
| *** denis_makogon_ has quit IRC | 07:55 | |
| *** kevinbenton has joined #openstack-infra | 08:00 | |
| *** markwash has quit IRC | 08:02 | |
| *** jpich has quit IRC | 08:06 | |
| *** pblaho has joined #openstack-infra | 08:07 | |
| *** lcheng_ has quit IRC | 08:12 | |
| *** jcoufal has joined #openstack-infra | 08:17 | |
| *** markwash has joined #openstack-infra | 08:18 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 08:20 | |
| *** AlexF has joined #openstack-infra | 08:22 | |
| *** dizquierdo has joined #openstack-infra | 08:26 | |
| *** bogdando has joined #openstack-infra | 08:30 | |
| *** nati_ueno has joined #openstack-infra | 08:40 | |
| *** afazekas has joined #openstack-infra | 08:42 | |
| *** markwash has quit IRC | 08:43 | |
| *** jcoufal has quit IRC | 08:50 | |
| *** hashar has joined #openstack-infra | 08:53 | |
| *** bookwar has joined #openstack-infra | 08:59 | |
| *** salv-orlando has quit IRC | 08:59 | |
| *** sarob has quit IRC | 09:00 | |
| *** sarob has joined #openstack-infra | 09:01 | |
| *** fbo is now known as fbo_away | 09:01 | |
| *** odyssey4me2 is now known as odyssey4me | 09:01 | |
| *** sarob has quit IRC | 09:06 | |
| *** jcoufal has joined #openstack-infra | 09:06 | |
| *** fbo_away is now known as fbo | 09:06 | |
| *** bradm has quit IRC | 09:07 | |
| *** bradm has joined #openstack-infra | 09:08 | |
| *** CaptTofu has joined #openstack-infra | 09:09 | |
| *** e0ne_ has joined #openstack-infra | 09:12 | |
| *** rossella_s has joined #openstack-infra | 09:12 | |
| *** CaptTofu has quit IRC | 09:14 | |
| *** e0ne has quit IRC | 09:14 | |
| *** yassine has joined #openstack-infra | 09:16 | |
| *** AlexF has quit IRC | 09:16 | |
| *** jpich has joined #openstack-infra | 09:20 | |
| *** nati_ueno has quit IRC | 09:23 | |
| *** cody-somerville has joined #openstack-infra | 09:28 | |
| *** cody-somerville has quit IRC | 09:28 | |
| *** cody-somerville has joined #openstack-infra | 09:28 | |
| *** nati_ueno has joined #openstack-infra | 09:29 | |
| *** igor_ has joined #openstack-infra | 09:30 | |
| *** igor has quit IRC | 09:32 | |
| *** jcooley_ has quit IRC | 09:32 | |
| *** markmc has joined #openstack-infra | 09:34 | |
| *** nati_ueno has quit IRC | 09:36 | |
| *** DinaBelova_ is now known as DinaBelova | 09:37 | |
| *** jooools has joined #openstack-infra | 09:37 | |
| *** reed has joined #openstack-infra | 09:38 | |
| *** derekh has joined #openstack-infra | 09:38 | |
| *** johnthetubaguy has joined #openstack-infra | 09:39 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:40 | |
| *** johnthetubaguy has quit IRC | 09:40 | |
| *** jhesketh__ has quit IRC | 09:40 | |
| *** johnthetubaguy has joined #openstack-infra | 09:42 | |
| *** hashar_ has joined #openstack-infra | 09:44 | |
| *** johnthetubaguy1 has quit IRC | 09:44 | |
| *** hashar has quit IRC | 09:48 | |
| *** hashar has joined #openstack-infra | 09:51 | |
| *** bhuvan has joined #openstack-infra | 09:52 | |
| *** hashar_ has quit IRC | 09:55 | |
| *** hashar_ has joined #openstack-infra | 09:57 | |
| *** jasondotstar has joined #openstack-infra | 10:00 | |
| *** hashar has quit IRC | 10:01 | |
| *** johnthetubaguy has quit IRC | 10:03 | |
| *** johnthetubaguy has joined #openstack-infra | 10:05 | |
| *** sarob has joined #openstack-infra | 10:11 | |
| *** salv-orlando has joined #openstack-infra | 10:14 | |
| *** sarob has quit IRC | 10:16 | |
| *** ruhe has joined #openstack-infra | 10:21 | |
| *** max_lobur_afk is now known as max_lobur | 10:26 | |
| *** e0ne_ has quit IRC | 10:26 | |
| *** mrda_away is now known as mrda | 10:31 | |
| *** salv-orlando has quit IRC | 10:31 | |
| *** ken1ohmichi has joined #openstack-infra | 10:32 | |
| *** ken1ohmichi has quit IRC | 10:32 | |
| *** hashar_ is now known as hashar | 10:32 | |
| *** odyssey4me2 has joined #openstack-infra | 10:33 | |
| *** odyssey4me has quit IRC | 10:34 | |
| *** odyssey4me2 has quit IRC | 10:38 | |
| *** apevec has joined #openstack-infra | 10:49 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 10:54 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 10:57 | |
| *** jasondotstar has quit IRC | 11:00 | |
| *** yassine has quit IRC | 11:01 | |
| *** mrda is now known as mrda_away | 11:02 | |
| ttx | notmyname: updated on the ical | 11:04 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Database fixture added https://review.openstack.org/69384 | 11:08 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 11:09 |
| *** CaptTofu has joined #openstack-infra | 11:10 | |
| *** salv-orlando has joined #openstack-infra | 11:14 | |
| *** CaptTofu has quit IRC | 11:15 | |
| *** ociuhandu has joined #openstack-infra | 11:15 | |
| *** david-lyle has quit IRC | 11:17 | |
| *** boris-42_ has quit IRC | 11:18 | |
| *** andreaf has joined #openstack-infra | 11:22 | |
| *** coolsvap_away has joined #openstack-infra | 11:29 | |
| *** coolsvap has quit IRC | 11:30 | |
| *** salv-orlando has quit IRC | 11:30 | |
| *** ociuhandu has quit IRC | 11:31 | |
| *** ociuhandu has joined #openstack-infra | 11:31 | |
| *** ken1ohmichi has joined #openstack-infra | 11:34 | |
| *** fbo is now known as fbo_away | 11:35 | |
| *** apevec is now known as apc | 11:41 | |
| *** apc is now known as apevec | 11:41 | |
| *** ociuhandu has quit IRC | 11:41 | |
| *** salv-orlando has joined #openstack-infra | 11:43 | |
| *** ociuhandu has joined #openstack-infra | 11:44 | |
| *** CaptTofu has joined #openstack-infra | 11:46 | |
| *** coolsvap_away is now known as coolsvap | 11:46 | |
| *** jp_at_hp has joined #openstack-infra | 11:46 | |
| *** salv-orlando has quit IRC | 11:49 | |
| *** jasondotstar has joined #openstack-infra | 11:50 | |
| *** DinaBelova is now known as DinaBelova_ | 11:52 | |
| *** CaptTofu has quit IRC | 11:57 | |
| *** CaptTofu has joined #openstack-infra | 11:58 | |
| *** jnoller has joined #openstack-infra | 11:58 | |
| *** ociuhandu has quit IRC | 11:59 | |
| *** jooools has quit IRC | 12:00 | |
| *** matrohon has quit IRC | 12:02 | |
| *** ociuhandu has joined #openstack-infra | 12:02 | |
| *** dizquierdo has quit IRC | 12:02 | |
| *** DinaBelova_ is now known as DinaBelova | 12:05 | |
| *** hashar has quit IRC | 12:12 | |
| *** yassine has joined #openstack-infra | 12:13 | |
| *** yassine_ has joined #openstack-infra | 12:13 | |
| *** yamahata has quit IRC | 12:14 | |
| *** ociuhandu has quit IRC | 12:16 | |
| ArxCruz | mordred: around ? | 12:16 |
| *** miqui has joined #openstack-infra | 12:18 | |
| *** alexpilotti has joined #openstack-infra | 12:18 | |
| *** jnoller has quit IRC | 12:26 | |
| *** jcooley_ has joined #openstack-infra | 12:26 | |
| *** dizquierdo has joined #openstack-infra | 12:26 | |
| *** CaptTofu has quit IRC | 12:28 | |
| *** e0ne has joined #openstack-infra | 12:29 | |
| *** boris-42_ has joined #openstack-infra | 12:30 | |
| *** jcooley_ has quit IRC | 12:31 | |
| *** malini is now known as malini_afk | 12:33 | |
| *** rfolco has joined #openstack-infra | 12:33 | |
| *** coolsvap has quit IRC | 12:34 | |
| *** coolsvap has joined #openstack-infra | 12:34 | |
| *** lcostantino has joined #openstack-infra | 12:35 | |
| *** ryanpetrello has joined #openstack-infra | 12:35 | |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added MagentoDB (https://wiki.openstack.org/wiki/MagnetoDB) project to Stackforge https://review.openstack.org/71308 | 12:35 |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add script trigger. https://review.openstack.org/70971 | 12:36 |
| lcostantino | Hi, i am reworking some patches for a bp that were abandoned by another author. Is it possible to restore them to keep the same review? | 12:38 |
| *** jooools has joined #openstack-infra | 12:39 | |
| *** igor has joined #openstack-infra | 12:45 | |
| *** igor_ has quit IRC | 12:46 | |
| *** ken1ohmichi has quit IRC | 12:49 | |
| *** isviridov has joined #openstack-infra | 12:49 | |
| *** weshay has joined #openstack-infra | 12:53 | |
| *** ken1ohmichi has joined #openstack-infra | 12:53 | |
| *** coolsvap has quit IRC | 12:55 | |
| *** fbo_away is now known as fbo | 12:55 | |
| *** yassine has quit IRC | 12:55 | |
| *** CaptTofu has joined #openstack-infra | 12:57 | |
| *** jasondotstar has quit IRC | 13:00 | |
| *** smarcet has joined #openstack-infra | 13:01 | |
| *** CaptTofu has quit IRC | 13:01 | |
| SergeyLukjanov | lcostantino, you can ask infra folks in openstack-infra@lists.openstack.org mailing lists to do it | 13:03 |
| *** Shrews has joined #openstack-infra | 13:06 | |
| *** freyes has joined #openstack-infra | 13:06 | |
| *** mriedem has joined #openstack-infra | 13:07 | |
| lcostantino | SergeyLukjanov, thanks, will do that. | 13:08 |
| *** prad_ has joined #openstack-infra | 13:08 | |
| openstackgerrit | A change was merged to openstack/requirements: Unpin kazoo https://review.openstack.org/69815 | 13:09 |
| *** freyes has quit IRC | 13:16 | |
| *** max_lobur is now known as max_lobur_afk | 13:17 | |
| *** jasondotstar has joined #openstack-infra | 13:19 | |
| *** julim has quit IRC | 13:23 | |
| *** max_lobur_afk is now known as max_lobur | 13:29 | |
| *** hashar has joined #openstack-infra | 13:29 | |
| *** dizquierdo has quit IRC | 13:31 | |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added new MagentoDB project to Stackforge https://review.openstack.org/71308 | 13:33 |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added new MagnetoDB project to Stackforge https://review.openstack.org/71308 | 13:34 |
| ArxCruz | SergeyLukjanov: hey, around ? | 13:38 |
| SergeyLukjanov | ArxCruz, yup | 13:38 |
| ArxCruz | do you know about jclouds + swift configuration in jenkins ? | 13:38 |
| *** prad_ has quit IRC | 13:38 | |
| SergeyLukjanov | ArxCruz, nope :( probably I could have some ideas, do you have a specific question? | 13:39 |
| ArxCruz | I want to upload my jenkins job logs to my swift server | 13:39 |
| ArxCruz | the jclouds seems to have a few providers already configured | 13:39 |
| ArxCruz | but I have my own swift | 13:39 |
| *** hashar has quit IRC | 13:40 | |
| SergeyLukjanov | ArxCruz, it's your own jenkins? | 13:42 |
| ArxCruz | SergeyLukjanov: yup | 13:42 |
| SergeyLukjanov | ArxCruz, AFAIK jclouds plugin in jenkins is only for slaves creation | 13:43 |
| ArxCruz | SergeyLukjanov: according clarkb it has support to swift :/ | 13:43 |
| SergeyLukjanov | ArxCruz, oh, correct, I've found the README section about it - https://github.com/jenkinsci/jclouds-plugin#adding-a-blobstore-profile-for-storing-build-artifacts | 13:44 |
| ArxCruz | SergeyLukjanov: checking, thanks | 13:44 |
| isviridov | SergeyLukjanov, thx for support with patch | 13:44 |
| SergeyLukjanov | isviridov, np | 13:45 |
| * ArxCruz should search more before ask | 13:45 | |
| SergeyLukjanov | ArxCruz, are you using only jenkins w/o zuul? | 13:45 |
| *** ken1ohmichi has quit IRC | 13:45 | |
| ArxCruz | SergeyLukjanov: I'm using the same infrastructure that o-infra uses | 13:45 |
| ArxCruz | SergeyLukjanov: i'm aware about the zuul patch | 13:46 |
| SergeyLukjanov | ArxCruz, than probably it'll easier to use swift-related patch | 13:46 |
| SergeyLukjanov | oh :) | 13:46 |
| ArxCruz | SergeyLukjanov: I'm following closer this, but I don't know when the core will decide to merge that | 13:46 |
| *** jooools has quit IRC | 13:47 | |
| *** milki has quit IRC | 13:48 | |
| *** milki has joined #openstack-infra | 13:48 | |
| *** hashar has joined #openstack-infra | 13:49 | |
| *** thomasem has joined #openstack-infra | 13:49 | |
| *** jooools has joined #openstack-infra | 13:49 | |
| *** e0ne_ has joined #openstack-infra | 13:51 | |
| *** ktbenton has quit IRC | 13:52 | |
| *** e0ne has quit IRC | 13:54 | |
| miqui | hello.... | 13:55 |
| *** talluri has quit IRC | 13:56 | |
| miqui | so in vm i am seeing hundreds of /usr/local/bin/recheckwatch processes (and the vm is not really busy with many jobs) | 13:56 |
| miqui | sound familiar? | 13:56 |
| *** talluri has joined #openstack-infra | 13:56 | |
| *** dprince has joined #openstack-infra | 13:56 | |
| ArxCruz | SergeyLukjanov: The plugin also provides a way to store your build artifacts on JClouds supported cloud storage providers | 13:57 |
| *** jcooley_ has joined #openstack-infra | 13:57 | |
| ArxCruz | bleh, only for supported cloud storage providers :@ | 13:58 |
| SergeyLukjanov | ArxCruz, I think swift is supported | 13:59 |
| SergeyLukjanov | ArxCruz, btw afaik swift provides s3 compatibleapi | 13:59 |
| *** dkranz has joined #openstack-infra | 14:00 | |
| ArxCruz | SergeyLukjanov: yeah, it's supported, but only for the supported providers | 14:00 |
| ArxCruz | like s3, azure, hpcloud etc | 14:00 |
| *** talluri has quit IRC | 14:00 | |
| *** jcooley_ has quit IRC | 14:03 | |
| *** bknudson has quit IRC | 14:03 | |
| *** apevec has left #openstack-infra | 14:05 | |
| *** dcramer_ has quit IRC | 14:08 | |
| fungi | miqui: the recheckwatch service is for building a page like http://status.openstack.org/recheck/ so you probably don't need to run it on your system | 14:08 |
| *** dstanek has joined #openstack-infra | 14:09 | |
| miqui | thanks fungi....its running many times and eating up resources and brings everything else down with it.. | 14:09 |
| miqui | , but thing is fungi...is that perhaps we do...i am mean if you want to see the status of the zuul gates.... | 14:10 |
| fungi | miqui: the recheck list page is completely separate from the zuul status page | 14:11 |
| miqui | fungi: btw, i get a 404 on your posted URL... | 14:11 |
| miqui | ah ok... | 14:11 |
| miqui | thanks fungi: so how do i disable recheck? | 14:11 |
| fungi | sorry, my browser has decided to eat all my ram so i had to guess at the url... i think it might be rechecks (plural) | 14:11 |
| fungi | miqui: it's got an initscript. i would just configure that not to run or if you're using puppet then configure that service not to start | 14:12 |
| miqui | yeah its plural.. | 14:12 |
| miqui | cool...thanks fungi | 14:12 |
| fungi | it's basically just a little daemon which connects to gerrit, watches the event stream and any time it sees a recheck or reverify comment with a bug number it adds the statistic to a table in a pickle file, then occasionally writes out that bug list html from it | 14:13 |
| miqui | k..thanks | 14:13 |
| *** crank has quit IRC | 14:14 | |
| fungi | we use it purely for keeping track of how many rechecks/reverifies there are for which bug numbers, but that page and service are being obsoleted by http://status.openstack.org/elastic-recheck/ soon anyway | 14:14 |
| *** yamahata has joined #openstack-infra | 14:15 | |
| fungi | it was a fairly quick and dirty service to start with, so we'd rather just get rid of it than spend time trying to track down and fix problems in it | 14:15 |
| fungi | ugh... firefox. y u want 18gb ram? | 14:16 |
| miqui | but fungi: you guys are running so, my next question is you are probably seeing many rechecks running...so what ram capacity do you have now | 14:16 |
| miqui | to deal with this currently.? | 14:16 |
| fungi | miqui: there should only be one instance of the recheckwatch service running, so if you have multiples of them then something has gone terribly wrong with it | 14:17 |
| *** dstufft|laptop has joined #openstack-infra | 14:18 | |
| fungi | yeah, just checked on zuul.openstack.org and there's only one running recheckwatch process, started about a month ago (the last time we rebuilt zuul) | 14:18 |
| miqui | k, thanks fungi... | 14:19 |
| fungi | you're welcome! | 14:19 |
| *** yamahata has quit IRC | 14:22 | |
| *** bknudson has joined #openstack-infra | 14:22 | |
| *** yamahata has joined #openstack-infra | 14:23 | |
| *** eharney has joined #openstack-infra | 14:24 | |
| *** yamahata has quit IRC | 14:25 | |
| openstackgerrit | A change was merged to openstack/requirements: Add XenAPI to OpenStack dependencies https://review.openstack.org/68181 | 14:25 |
| *** yamahata has joined #openstack-infra | 14:25 | |
| *** hdd_ has joined #openstack-infra | 14:27 | |
| SergeyLukjanov | fungi, is it difficult to restart zuul? it'll expose last_reconfigured time in API that'll be available on status pages too | 14:28 |
| *** IvanBerezovskiy has joined #openstack-infra | 14:28 | |
| SergeyLukjanov | fungi, morning | 14:29 |
| *** malini_afk is now known as malini | 14:31 | |
| *** prad_ has joined #openstack-infra | 14:31 | |
| *** prad_ has quit IRC | 14:32 | |
| dstanek | i have a patch that fails installing dependencies because GCC can't find libxml/xmlversion.h. where should i file that bug? | 14:32 |
| fungi | dstanek: file it against openstack-ci... but it's probably from one of clarkb's failed experiments building new images last night--i know he ended up with a few stray vms launched from one which was lacking some necessary system packages | 14:34 |
| dstanek | fungi: thanks | 14:34 |
| *** thomasem_ has joined #openstack-infra | 14:35 | |
| fungi | SergeyLukjanov: good morning! restarting zuul is not complicated, but it does require waiting for it to quiesce all running jobs so that it can safely restart without losing results | 14:35 |
| dstanek | fungi: should i bother filing it then? or do you want to keep track of these kinds of failures? | 14:35 |
| fungi | dstanek: we don't really keep track of them per se, but your job probably wasn't the only one hit so someone will likely wind up opening a bug to recheck against anyway | 14:36 |
| *** thomasem has quit IRC | 14:38 | |
| *** dims has quit IRC | 14:38 | |
| SergeyLukjanov | fungi, oh, got it, SIGUSR1 + restart when queue is empty | 14:39 |
| fungi | SergeyLukjanov: yes. and it was last restarted ~2 weeks ago according to the process list | 14:40 |
| *** julim has joined #openstack-infra | 14:41 | |
| *** julim has quit IRC | 14:42 | |
| *** burt1 has joined #openstack-infra | 14:43 | |
| *** julim has joined #openstack-infra | 14:43 | |
| *** sandywalsh has joined #openstack-infra | 14:43 | |
| *** dims has joined #openstack-infra | 14:44 | |
| *** dhellmann_ is now known as dhellmann | 14:44 | |
| *** changbl has quit IRC | 14:45 | |
| *** jooools has quit IRC | 14:50 | |
| *** zehicle_at_dell has quit IRC | 14:50 | |
| *** blamar has joined #openstack-infra | 14:55 | |
| *** cody-somerville has quit IRC | 14:57 | |
| *** dcramer_ has joined #openstack-infra | 14:58 | |
| *** crank has joined #openstack-infra | 15:00 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 15:01 |
| *** jooools has joined #openstack-infra | 15:01 | |
| *** boris-42_ has quit IRC | 15:02 | |
| *** amotoki has quit IRC | 15:02 | |
| *** esker has quit IRC | 15:07 | |
| *** jswarren has joined #openstack-infra | 15:07 | |
| *** esker has joined #openstack-infra | 15:08 | |
| *** prad_ has joined #openstack-infra | 15:08 | |
| *** esker has quit IRC | 15:09 | |
| *** mestery has joined #openstack-infra | 15:09 | |
| *** virmitio has joined #openstack-infra | 15:10 | |
| *** jpich has quit IRC | 15:12 | |
| *** jpich has joined #openstack-infra | 15:15 | |
| *** krotscheck has joined #openstack-infra | 15:16 | |
| *** matsuhas_ has quit IRC | 15:17 | |
| *** rcleere has joined #openstack-infra | 15:18 | |
| *** mfer has joined #openstack-infra | 15:19 | |
| *** matrohon has joined #openstack-infra | 15:20 | |
| *** mestery has quit IRC | 15:20 | |
| *** markmc has quit IRC | 15:20 | |
| *** nosnos has quit IRC | 15:21 | |
| *** mestery has joined #openstack-infra | 15:21 | |
| *** mrodden has joined #openstack-infra | 15:23 | |
| *** ok_delta has quit IRC | 15:24 | |
| *** jnoller has joined #openstack-infra | 15:27 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 15:29 |
| *** jnoller_ has joined #openstack-infra | 15:30 | |
| *** jnoller has quit IRC | 15:33 | |
| *** banix has joined #openstack-infra | 15:34 | |
| *** david-lyle has joined #openstack-infra | 15:36 | |
| *** jgrimm has joined #openstack-infra | 15:36 | |
| openstackgerrit | A change was merged to openstack/requirements: Update python-keystoneclient to 0.5.0 https://review.openstack.org/70929 | 15:36 |
| *** thomasbiege has joined #openstack-infra | 15:36 | |
| *** atiwari has joined #openstack-infra | 15:36 | |
| *** kraman has joined #openstack-infra | 15:37 | |
| miqui | fungi: diggin a little on recheckwatch it seems pid file is not being created during /etc.init.d/recheckwatch start | 15:37 |
| miqui | checked perms,etc...etc...but no dice.. | 15:38 |
| fungi | miqui: maybe the directory it wants to put that file in doesn't exist? usually the initscript should create and chown that directory (in my opinion) but we might have missed doing that | 15:38 |
| *** david_lyle_ has joined #openstack-infra | 15:39 | |
| fungi | if you're running it on a system which keeps /var/run in tmpfs for example (which is somewhat common practice) | 15:39 |
| *** jgrimm has quit IRC | 15:41 | |
| *** david-lyle has quit IRC | 15:43 | |
| *** wenlock has joined #openstack-infra | 15:44 | |
| *** jcooley_ has joined #openstack-infra | 15:46 | |
| *** dcramer_ has quit IRC | 15:46 | |
| *** mestery has quit IRC | 15:47 | |
| *** mrodden has quit IRC | 15:47 | |
| fungi | miqui: the permissions/ownership on that directory for us looks like drwxr-xr-x 2 recheckwatch root 40 Jan 18 03:14 /var/run/recheckwatch | 15:49 |
| miqui | thanks fungi: i'll grab this info and comp... | 15:51 |
| miqui | the /var/run/ dir does exist and the final path of recheckwatch/ does seem to be created..but no pid file after that.. | 15:52 |
| *** jcooley_ has quit IRC | 15:53 | |
| miqui | ..thanks again... | 15:53 |
| banix | Hi everybody. A question: Are the Infra people also deal with/review/maintain devstack? | 15:54 |
| fungi | banix: not as much, though we do sometimes contribute patches to fix problems in it | 15:55 |
| banix | fungi: Thanks. Do you know who are the main people there? Can't seem to get an answer to my questions regarding adding a file there to support a new plugin. | 15:56 |
| fungi | banix: https://review.openstack.org/#/admin/groups/50,members | 15:56 |
| banix | fungi: Thanks! | 15:56 |
| fungi | banix: primarily dtroyer, sdague and chmouel i think. so maybe try to bring it up in #openstack-qa | 15:57 |
| *** AlexF has joined #openstack-infra | 15:57 | |
| banix | fungi: Thanks, will do. | 15:58 |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added clone-workspace publisher https://review.openstack.org/69532 | 15:58 |
| *** esker has joined #openstack-infra | 15:58 | |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added clone-workspace publisher https://review.openstack.org/69532 | 15:59 |
| *** mrodden has joined #openstack-infra | 16:01 | |
| *** che-arne has joined #openstack-infra | 16:01 | |
| *** markmc has joined #openstack-infra | 16:02 | |
| *** max_lobur is now known as max_lobur_afk | 16:02 | |
| *** bhuvan has quit IRC | 16:04 | |
| *** smarcet has left #openstack-infra | 16:04 | |
| *** dstufft|laptop has quit IRC | 16:04 | |
| *** IvanBerezovskiy has left #openstack-infra | 16:05 | |
| *** jpich has quit IRC | 16:07 | |
| *** hdd_ has quit IRC | 16:09 | |
| *** andreaf2 has joined #openstack-infra | 16:12 | |
| *** jaypipes has quit IRC | 16:12 | |
| *** andreaf has quit IRC | 16:12 | |
| *** dizquierdo has joined #openstack-infra | 16:14 | |
| *** boris-42_ has joined #openstack-infra | 16:15 | |
| *** thuc has joined #openstack-infra | 16:15 | |
| *** jpich has joined #openstack-infra | 16:15 | |
| *** thuc_ has joined #openstack-infra | 16:16 | |
| *** ociuhandu has joined #openstack-infra | 16:17 | |
| *** salv-orlando has joined #openstack-infra | 16:18 | |
| *** mestery has joined #openstack-infra | 16:18 | |
| *** dklyle has joined #openstack-infra | 16:19 | |
| *** jp_at_hp has quit IRC | 16:19 | |
| *** thuc has quit IRC | 16:20 | |
| *** david_lyle_ has quit IRC | 16:22 | |
| *** thomasbiege has left #openstack-infra | 16:24 | |
| *** salv-orlando has quit IRC | 16:26 | |
| dhellmann | fungi: ping? | 16:27 |
| *** chris_johnson has joined #openstack-infra | 16:28 | |
| *** nicedice has joined #openstack-infra | 16:28 | |
| *** jcooley_ has joined #openstack-infra | 16:30 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 16:32 |
| flaper87 | clarkb: hey, could you take a look at this patch again? https://review.openstack.org/#/c/69344/ | 16:32 |
| flaper87 | clarkb: sdague and I commented there | 16:33 |
| flaper87 | clarkb: and there's a -1 you left :D | 16:33 |
| fungi | dhellmann: hi there | 16:33 |
| *** salv-orlando has joined #openstack-infra | 16:34 | |
| dhellmann | fungi: I'm looking into renaming oslo.sphinx to get it out of the oslo namespace package to fix a devstack problem | 16:34 |
| dhellmann | it *looks* like I can just do all of that in the current repo, without renmaing it | 16:34 |
| dhellmann | we had an issue with a project where the repo name didn't match the package name in the past (WSME?) but I think you fixed that, and I wanted to confirm | 16:34 |
| dhellmann | s/renmaing/renaming/ | 16:35 |
| fungi | dhellmann: yes, we can do that (the script now looks at the metadata in the repo for the package name), though it should be avoided for the sake of clarity if at all possible | 16:36 |
| *** bhuvan has joined #openstack-infra | 16:36 | |
| dhellmann | fungi: ok, I'll go ahead and submit another pull request to rename the repo. maybe include it in the list tomorrow? | 16:36 |
| fungi | what are you looking at renaming it to? and how do we think the requirement transition will go down? | 16:37 |
| dhellmann | oslo.sphinx -> oslosphinx | 16:37 |
| *** dklyle is now known as david-lyle | 16:37 | |
| *** sparkycollier has joined #openstack-infra | 16:37 | |
| fungi | should presumably be able to do a new release of the new name mostly the same as the old release and then edit the requirements on depending projects accordingly i guess | 16:37 |
| dhellmann | we can change the name, tag a release, publish, wait for the mirror to sync, add it to requirements, update the projects, then remove the old name from requirements | 16:37 |
| mtreinish | dhellmann: how do you pronounce that? | 16:37 |
| fungi | oz-loss-pinks | 16:38 |
| dhellmann | mtreinish: "the sphinx theme for openstack" :-) | 16:38 |
| mtreinish | heh | 16:38 |
| *** max_lobur_afk is now known as max_lobur | 16:38 | |
| dhellmann | would have called it openstacksphinx but bnemec pointed out possible issues with using the trademark name there and we didn't want to wait for an answer to the question | 16:38 |
| sdague | fungi: any word on when tripleo is going to have working nodepool again? The 100+ jobs that are filling up the status page blocked on that kind of bothers me | 16:39 |
| *** pblaho has quit IRC | 16:39 | |
| fungi | sdague: i haven't heard any new timeline, and when i ask in #tripleo nobody seems to speak up. i'll check again today but i suspect we won't know anything until lifeless is up and about | 16:40 |
| sdague | fungi: alternatively, can we make tripleo be a separate check queue? I feel it's kind of a UI denial of service right now | 16:40 |
| sdague | because it makes it much harder to understand the upstream state | 16:41 |
| *** sdake has quit IRC | 16:41 | |
| *** thomasbiege1 has joined #openstack-infra | 16:42 | |
| fungi | sdague: agreed. we might look at clearing their special-node jobs from zuul's config and flushing them out of the pipeline if it goes on much longer. they're definitely piling up | 16:42 |
| sdague | yeh | 16:42 |
| *** hashar has quit IRC | 16:42 | |
| sdague | and honestly, I feel like if someone wants upstream zuul running against their backend nodes, we should have an expectation that nodepool outages will be kept to < 1 day | 16:43 |
| fungi | zuul itself seems to be handling the situation just fine, but i agree the status page is becoming harder to review | 16:43 |
| sdague | it's been 4 days now (almost) since top of check came in | 16:44 |
| *** mestery has quit IRC | 16:44 | |
| sdague | which I find really irresponsible that the tripleo team hasn't dropped everything else to fix this | 16:44 |
| fungi | i think the tripleo plan is to have redundant regions, so ought to be less of an issue for them once that happens | 16:44 |
| sdague | fungi: redundant regions is only good if the regions are inherently more reliable than this | 16:45 |
| fungi | well, this time was more of a freak occurrence i think. someone accidentally deleted the entire cloud so it's being rebuilt from scratch | 16:46 |
| sdague | 1) how did that happen? | 16:46 |
| sdague | 2) that takes 4 days? | 16:46 |
| jeblair | sdague: yes, we definitely have some expectations around service level; part of this experiment is burning in the tripleo cloud to see if it meets them | 16:46 |
| jeblair | it doesn't yet | 16:46 |
| jeblair | sdague: but it is _well_ understood that this is not the sort of thing we expect to happen when it's in production | 16:47 |
| sdague | I always expect the unexpected :) | 16:47 |
| sdague | jeblair: so how about a 2nd check queue for them, so it doesn't push all the working parts off the page? | 16:48 |
| jeblair | sdague: if we're going to reconfigure, i say we yank the jobs. | 16:49 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Rename oslo.sphinx to oslosphinx https://review.openstack.org/71593 | 16:50 |
| *** krotscheck has quit IRC | 16:50 | |
| sdague | ok, they have an ETA for when this is going to be back? Because given their throughput, I think it will take a week to clear that backlog | 16:50 |
| *** gyee has joined #openstack-infra | 16:50 | |
| *** jcoufal has quit IRC | 16:52 | |
| jeblair | sdague: let's remove the jobs today, and i'll look into adding a (configurable) timeout to zuul, where, say, after 24 hours it declares a job LOST... | 16:54 |
| fungi | they very well may want those pending job runs cleared anyway, since many are probably no longer relevant to the changes in question | 16:54 |
| *** dcramer_ has joined #openstack-infra | 16:54 | |
| sdague | jeblair: sounds good, that needs to be per/queue though | 16:55 |
| sdague | because gate has legitimate reasons to back up that far | 16:55 |
| *** bhuvan has quit IRC | 16:55 | |
| sdague | check does not | 16:55 |
| jeblair | sdague: i was thinking that it would be if a job hasn't started 24 hours after zuul requested it be built... so per-build not per-change (queue item) | 16:56 |
| jeblair | sdague: jobs should _generally_ start within moments of being requested, and if we are exhausting slaves, maybe a few hours. | 16:56 |
| *** mestery has joined #openstack-infra | 16:56 | |
| *** jcooley_ has quit IRC | 16:56 | |
| *** jcooley_ has joined #openstack-infra | 16:57 | |
| jeblair | sdague: so 24h is probably generous enough for us in current circumstances. but definitely configurable. | 16:57 |
| notmyname | I'm seeing some swift-docs jobs failing (on jenkins05 only). https://jenkins05.openstack.org/job/gate-swift-docs/32/console https://jenkins05.openstack.org/job/gate-swift-docs/29/console and https://jenkins05.openstack.org/job/gate-swift-docs/27/console | 16:57 |
| fungi | right, even in the worst of the churn we saw last month, it never got to the point of taking 24 hours to start a job | 16:57 |
| sdague | fungi: fair | 16:58 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:58 | |
| fungi | notmyname: that looks like the job got cancelled. i'll see if it was for one which was actually reported to the change or behind a failing change in a gate reset | 16:58 |
| jeblair | notmyname: what change is that for? | 16:58 |
| notmyname | fungi: thanks | 16:59 |
| notmyname | jeblair: not sure yet. I started looking when I saw my graph of gate success rates take a dip | 16:59 |
| fungi | huh, weird. i second jeblair's question... it says "No changes." | 16:59 |
| *** sparkycollier has quit IRC | 16:59 | |
| jeblair | notmyname: that job was not reported to gerrit | 16:59 |
| *** shivh has joined #openstack-infra | 16:59 | |
| pleia2 | morning | 17:00 |
| fungi | notmyname: java.lang.InterruptedException is what we often see when zul cancels a running job. jenkins incorrectly reports that as a failure | 17:00 |
| jeblair | notmyname: it's one of the builds that zuul canceled because its speculative execution failed | 17:00 |
| *** markmc has quit IRC | 17:00 | |
| jeblair | notmyname: here's the change: https://review.openstack.org/#/c/71405/ | 17:00 |
| notmyname | jeblair: ie a gate reset? | 17:00 |
| jeblair | notmyname: yep | 17:00 |
| fungi | notmyname: this is why the qa team stopped tracking failure stats for the gate directly... jenkins often reports failures for cancelled jobs | 17:00 |
| notmyname | ok, thanks. | 17:00 |
| notmyname | fungi: so what's the better way to track failures? | 17:01 |
| fungi | which makes failure stats look far, far worse the longer the gate gets | 17:01 |
| jeblair | notmyname: zuul cancels _so many_ jobs. | 17:01 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Rename oslo.sphinx to oslosphinx https://review.openstack.org/71593 | 17:01 |
| jeblair | notmyname: which metric were you watching? | 17:01 |
| notmyname | and cancel is reported as failure? that would inflate numbers | 17:01 |
| fungi | notmyname: i believe jog0 was working on something to do log analysis to confirm whether the failure was for a job which completed | 17:01 |
| *** emagana has joined #openstack-infra | 17:01 | |
| jeblair | notmyname: it depends on where jenkins is; sometimes it's reported as ABORTED, sometimes FAILURE. | 17:02 |
| notmyname | jeblair: on http://not.mn/swift_gate_status.html the graphite measurement of gate-swift-docs (SUCCESS / (SUCCESS+FAIURE)) | 17:02 |
| *** sarob has joined #openstack-infra | 17:02 | |
| shivh | fungi: will i be able to get the service account today to allow me to i start posting votes? | 17:02 |
| fungi | shivh: hopefully. i want to catch up on the past couple days of third-party account requests after lunch | 17:03 |
| *** freyes has joined #openstack-infra | 17:03 | |
| shivh | fungi: thanks | 17:03 |
| *** portante has left #openstack-infra | 17:04 | |
| jeblair | notmyname: zuul should not report the results of canceled builds to statsd, so that should not affect that metric | 17:04 |
| *** DinaBelova is now known as DinaBelova_ | 17:04 | |
| jeblair | notmyname: there could be a bug in that though; it's a somewhat indirect process | 17:05 |
| *** e0ne_ has quit IRC | 17:05 | |
| jeblair | notmyname: if you can't find another explanation, we should entertain that possibility | 17:05 |
| *** e0ne has joined #openstack-infra | 17:05 | |
| *** lcheng_ has joined #openstack-infra | 17:06 | |
| notmyname | jeblair: ok. jenkins05 was the only box to show any failed runs for that job. and the 3 console links I sent are all I found | 17:06 |
| *** yassine_ has quit IRC | 17:06 | |
| *** ashepelev1 has joined #openstack-infra | 17:07 | |
| *** sarob has quit IRC | 17:07 | |
| *** e0ne has quit IRC | 17:08 | |
| *** ashepelev1 has left #openstack-infra | 17:08 | |
| *** ashepelev1 has joined #openstack-infra | 17:08 | |
| *** kiall has quit IRC | 17:08 | |
| *** GheRivero has quit IRC | 17:09 | |
| *** sdake has joined #openstack-infra | 17:09 | |
| *** sdake has quit IRC | 17:09 | |
| *** sdake has joined #openstack-infra | 17:09 | |
| *** Shrews has quit IRC | 17:10 | |
| jeblair | notmyname: http://graphite.openstack.org/render/?yMaxRight=100&areaAlpha=.15&height=308&yStepRight=20&yStepLeft=20&showTarget=color%28alias%28secondYAxis%28smartSummarize%28asPercent%28stats.zuul.pipeline.gate.job.gate-swift-dsvm-functional.SUCCESS%2Csum%28stats.zuul.pipeline.gate.job.gate-swift-dsvm-functional.{SUCCESS%2CFAILURE}%29%29%2C%2712hours%27%2C%27avg%27%29%29%2C%27gate-swift-dsvm-functional%27%29%2C%27brown%27%29&vtitleRight=Succ | 17:11 |
| jeblair | oops | 17:11 |
| jeblair | 1 sec | 17:11 |
| notmyname | :-) | 17:11 |
| ArxCruz | clarkb: jeblair do you know a way to get the console log from jenkins? I see that the scp-plugin is the one who generate it | 17:11 |
| clarkb | flaper87: responded | 17:12 |
| jeblair | notmyname: http://graphite.openstack.org/render/?width=586&height=308&_salt=1391706819.229&target=stats.zuul.pipeline.gate.job.gate-swift-docs.SUCCESS&target=stats.zuul.pipeline.gate.job.gate-swift-docs.FAILURE | 17:12 |
| clarkb | ArxCruz: jenkins generates it, scp plugin has a hack to grab it and put it somewhere else | 17:13 |
| jeblair | notmyname: i don't see any gate failures for the past 24 hours for that job | 17:13 |
| ArxCruz | clarkb: scp get it through build.getLogText() | 17:13 |
| ArxCruz | then pass to an outputstream transforming in html | 17:13 |
| clarkb | ArxCruz: yes, and it does that in an independent thread so that the job thread can complete and write the entire log | 17:14 |
| *** shivh has quit IRC | 17:14 | |
| ArxCruz | clarkb: exactly, so, this console log doesn't exist physically in the VM correct ? | 17:14 |
| clarkb | ArxCruz: not on the VM, but it does "physically" exist on the master as something weird like a base64 encoded file | 17:15 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 17:15 |
| *** markmcclain has joined #openstack-infra | 17:15 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Rename oslo.sphinx to oslosphinx https://review.openstack.org/71593 | 17:17 |
| notmyname | jeblair: thanks for looking | 17:18 |
| clarkb | I think whatever mordred had must've crossed oceans, /me is fending of $disease today | 17:19 |
| zaro | clarkb: quick review to test zuul-gerrit? https://review.openstack.org/#/c/71458 | 17:20 |
| *** dripton is now known as dripton_lunch | 17:20 | |
| *** markwash has joined #openstack-infra | 17:21 | |
| clarkb | zaro: is that in support of WIP? I think you need to keep the approval vote | 17:21 |
| clarkb | zaro: or is approval being replaced by workflow in gerrit too? | 17:21 |
| zaro | clarkb: yes, it was jeblair's suggestion to combine them. | 17:21 |
| clarkb | ah ok I get it | 17:21 |
| fungi | yeah, that's the -1=wip/0=ready/+1=approved | 17:23 |
| *** chris_johnson is now known as wchrisj|away | 17:24 | |
| *** wchrisj|away is now known as chris_johnson | 17:24 | |
| *** jamielennox is now known as jamielennox|away | 17:24 | |
| *** pafuent has joined #openstack-infra | 17:25 | |
| *** markwash has quit IRC | 17:26 | |
| pafuent | Hi. I'm working on devstack support for Climate project. The thing is that when Jenkins run the job can't clone one of the repositories of the project (climate-nova). Here are the logs: https://jenkins01.openstack.org/job/gate-climate-devstack-dsvm/62/console | 17:29 |
| *** ashepelev1 has quit IRC | 17:29 | |
| clarkb | pafuent: if you look at the localrc you will see ERROR_ON_CLONE is true | 17:29 |
| clarkb | pafuent: this is because we require that devstack-gate set the state of all repos under test before devstack runs | 17:30 |
| clarkb | I think you may need to update the PROJECTS list to have it do the right thing | 17:30 |
| pafuent | clarkb: This file modules/openstack_project/files/jenkins_job_builder/config/projects.yaml? | 17:31 |
| clarkb | pafuent: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/solum.yaml#n18 like that | 17:31 |
| clarkb | pafuent: no devstack-gate has a PROJECTS variable | 17:31 |
| clarkb | pafuent: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n27 is the default value | 17:32 |
| clarkb | but stackforge projects can extend it as in the solum tet | 17:32 |
| *** dstanek_afk has joined #openstack-infra | 17:33 | |
| openstackgerrit | A change was merged to openstack-infra/config: update git trigger to match new workflow label on review-dev https://review.openstack.org/71458 | 17:34 |
| pafuent | clarkb: So I need to add all the projects that I need to clone on that variable, right? | 17:35 |
| *** dstanek has quit IRC | 17:36 | |
| clarkb | pafuent: yes I believe so | 17:36 |
| *** pcrews has joined #openstack-infra | 17:36 | |
| pafuent | clarkb: Thanks | 17:36 |
| *** DinaBelova_ is now known as DinaBelova | 17:37 | |
| *** dstanek_afk is now known as dstanek | 17:37 | |
| *** salv-orlando has quit IRC | 17:38 | |
| *** julim has quit IRC | 17:38 | |
| *** dkliban has quit IRC | 17:39 | |
| *** jnoller_ is now known as jnoller | 17:41 | |
| *** jcooley_ has quit IRC | 17:41 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Start renaming oslo.sphinx to oslosphinx https://review.openstack.org/71612 | 17:41 |
| *** jcooley_ has joined #openstack-infra | 17:42 | |
| *** AlexF has quit IRC | 17:42 | |
| clarkb | dhellmann: what is the difference between oslosphinx and oslo.sphinx? | 17:42 |
| fungi | clarkb: namespaces | 17:43 |
| dhellmann | clarkb: what fungi said | 17:43 |
| clarkb | why the change? | 17:43 |
| clarkb | oh I see in sb now | 17:43 |
| dhellmann | having oslo.sphinx installed in non-editable mode with oslo.config and oslo.messaging installed in editable mode breaks devstack sometimes | 17:43 |
| dhellmann | since oslo.sphinx is not production code, we're just moving it out of the namespace package | 17:43 |
| openstackgerrit | Pablo Andres Fuente proposed a change to openstack-infra/config: Add the climate-nova project https://review.openstack.org/71614 | 17:44 |
| dhellmann | clarkb: http://lists.openstack.org/pipermail/openstack-dev/2013-December/022586.html | 17:44 |
| fungi | dhellmann: it's technically only a documentation building/rendering requirement, right? | 17:44 |
| fungi | not run-time or test-time or install-time? | 17:44 |
| dhellmann | fungi: yes | 17:44 |
| dhellmann | although because of the way the doc build works, it is listed in test-requirements.txt | 17:44 |
| * dhellmann would like to change that, but it would require even more work than this | 17:45 | |
| fungi | sure. we overload test-requirements.txt to mean "anything which isn't a run-time requirement" | 17:45 |
| dhellmann | right | 17:45 |
| fungi | so sort of poorly named | 17:45 |
| clarkb | are namespaces just broken? (my impression of them a year ago was that they were and caused all sorts of fun problems with setuptools) | 17:46 |
| dhellmann | I'd like to just have a doc-requirements.txt too | 17:46 |
| dhellmann | clarkb: they are broken in some specific cases, such as this | 17:46 |
| dhellmann | but they do work quite a lot of the time | 17:46 |
| *** rlandy is now known as rlandy|bbl | 17:46 | |
| dhellmann | although dstufft doesn't like them :-) | 17:46 |
| clarkb | dhellmann: my gripe with that is (and I don't want to sound like a politician) but where does it stop? | 17:46 |
| dhellmann | clarkb: where does what stop? | 17:47 |
| clarkb | I don't think we need that level of distinction between non runtime requirements | 17:47 |
| dhellmann | oh | 17:47 |
| dhellmann | a new developer coming into a project has no idea how to even build the docs | 17:47 |
| dhellmann | the answer is something like: | 17:47 |
| fungi | i would be fine with just not calling the file test-requirements.txt i think | 17:47 |
| dhellmann | tox -e venv python setup.py build_sphinx | 17:47 |
| dhellmann | which is sort of nuts | 17:47 |
| dhellmann | I'd like the answer to be "tox -e doc" | 17:47 |
| clarkb | dhellmann: why is that nuts? | 17:47 |
| clarkb | oh I see | 17:47 |
| dhellmann | because someone who doesn't know how tox, sphinx, and setuptools works doesn't have any idea what that command line even means :-) | 17:48 |
| clarkb | dhellmann: right so maybe building docs is an opportunity to teach? | 17:48 |
| dhellmann | and if we have "tox -e doc" then we can have doc-requirements.txt used only for that env (with the other 2) | 17:48 |
| * fungi doesn't know what maven build command lines mean | 17:48 | |
| dhellmann | clarkb: there are other times and easier examples | 17:48 |
| clarkb | I think our devs need to have a greater understanding of what is going on under the hood | 17:48 |
| dhellmann | I would like that, too | 17:48 |
| clarkb | but tox -edoc is nice and convenient regardless of understanding | 17:49 |
| clarkb | fungi: maven build lines are basically make but with xml | 17:49 |
| dhellmann | yeah, I don't know how many other places we use that "-e venv" trick, but the doc case is one where I think we could benefit from a specific tox target | 17:49 |
| fungi | sudo maven me an xml sandwich | 17:49 |
| * dhellmann doesn't think that sounds like a very good sandwich | 17:50 | |
| fungi | i tried it. it's not | 17:50 |
| pleia2 | hah | 17:50 |
| dhellmann | xml is sort of stringy | 17:50 |
| *** kraman has left #openstack-infra | 17:50 | |
| clarkb | protip, if you stand outside in the could for an hour or two with 700k people you will be sick the next day | 17:50 |
| clarkb | ugh, should've watched from my window | 17:51 |
| fungi | dhellmann: i use the "-e whatev" trick constantly. it's very useful for choosing specific tests/actions | 17:51 |
| dhellmann | clarkb: what were you doing? | 17:51 |
| clarkb | dhellmann: superbowl parade | 17:51 |
| dhellmann | fungi: sure, I'm not saying remove it, I'm saying for the things we do a lot (and want other devs to do) we should make it harder to type the wrong command so we're always consistent | 17:51 |
| dhellmann | clarkb: ah | 17:51 |
| *** cadenzajon has joined #openstack-infra | 17:51 | |
| fungi | dhellmann: oh, i see. you mean "venv" literally as the tox env name. right it's a bit of a hack | 17:52 |
| *** mrodden has quit IRC | 17:52 | |
| dhellmann | fungi: yeah | 17:52 |
| dhellmann | it's a clever hack, but clever tends to be bad for repeatability and teaching | 17:52 |
| fungi | in jobs we use it for sdist builds too | 17:52 |
| fungi | and i agree. anywhere we use it is fair game for a more specific env name | 17:53 |
| dhellmann | is that so pbr is installed in the tox env? | 17:53 |
| dhellmann | maybe next cycle I'll be able to carve out some time to work on that | 17:53 |
| clarkb | dhellmann: I think so | 17:53 |
| fungi | i'm not even sure. i should try writing one called "tarball" to encapsulate the same concept and see if that works as expected | 17:53 |
| clarkb | fungi: dhellmann: I actually dislike having too many targets | 17:53 |
| clarkb | the nice thing about the venv target is I can change one file and update how we do doc builds | 17:54 |
| *** nati_ueno has joined #openstack-infra | 17:54 | |
| clarkb | if we go to tox -edoc I have to change 20 soemthing files | 17:54 |
| fungi | clarkb: you're such a make-hater ;) | 17:54 |
| clarkb | fungi: :) | 17:54 |
| fungi | in makeland, every file is a target | 17:54 |
| clarkb | the implicit rules are great | 17:54 |
| fungi | in former soviet make, files target you! | 17:55 |
| clarkb | seriously though, if tox fixes themselves we have to udpate a ton of files | 17:55 |
| *** mrodden has joined #openstack-infra | 17:55 | |
| dhellmann | clarkb: yeah, holger needs stern talking to about that INI change | 17:55 |
| fungi | he's been basically silent on that bug report | 17:55 |
| clarkb | fungi: but it has been so much fun | 17:56 |
| fungi | i think he's letting the dev who implemented the change defend it or take the flak | 17:56 |
| *** smarcet has joined #openstack-infra | 17:56 | |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added clone-workspace publisher https://review.openstack.org/69532 | 17:56 |
| clarkb | dhellmann: I mean it would be fine if they said "whoops" and worked to fix it in a 1.7.1 | 17:56 |
| *** markwash has joined #openstack-infra | 17:56 | |
| clarkb | or even stated they would accept a fix for it | 17:56 |
| clarkb | but the response of "no, that is intended behavior we need to change our docs" makes me cranky | 17:56 |
| *** AlexF has joined #openstack-infra | 17:56 | |
| fungi | the last word in the bug sounds to me like "old behavior changed and is now the new behavior--if you still want old behavior that should become a new feature in a later release" | 17:57 |
| *** rossella_s has quit IRC | 17:58 | |
| *** harlowja_away is now known as harlowja | 17:59 | |
| *** david_lyle_ has joined #openstack-infra | 18:00 | |
| *** sc68cal_ has joined #openstack-infra | 18:01 | |
| *** davidhadas_ has joined #openstack-infra | 18:01 | |
| dhellmann | fungi, clarkb : do you have a link to that bug handy? | 18:01 |
| dhellmann | if not I can find it | 18:01 |
| clarkb | dhellmann: https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror | 18:01 |
| *** derekh has quit IRC | 18:02 | |
| dhellmann | clarkb: thanks | 18:02 |
| *** david-lyle has quit IRC | 18:02 | |
| *** jpich has quit IRC | 18:02 | |
| *** davidhadas has quit IRC | 18:02 | |
| *** fbo is now known as fbo_away | 18:03 | |
| *** sarob has joined #openstack-infra | 18:03 | |
| *** dstufft|laptop has joined #openstack-infra | 18:04 | |
| *** amcrn has joined #openstack-infra | 18:06 | |
| *** sarob has quit IRC | 18:07 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:08 | |
| *** dripton_lunch is now known as dripton | 18:08 | |
| dhellmann | clarkb: that answer seems exceptionally unhelpful | 18:09 |
| dhellmann | (on the bug) | 18:09 |
| dhellmann | is there something we can do to our tox.ini files to make them work with 1.7? | 18:09 |
| *** DinaBelova is now known as DinaBelova_ | 18:09 | |
| clarkb | dhellmann: not if we want to continue to pass posargs as arguments to a specific argument in a subcommand | 18:10 |
| clarkb | dhellmann: the new behavior basically ensure posargs will be posargs to the subcommand | 18:10 |
| *** talluri has joined #openstack-infra | 18:10 | |
| * dhellmann doesn't like how quickly this is making him think about forking tox | 18:10 | |
| dkranz | clarkb: Are we supposed to do "reverify bug xxx" when elastic recheck posts in a comment to do that, or not? | 18:10 |
| clarkb | dkranz: its up to you | 18:11 |
| dkranz | clarkb: ok, thanks | 18:11 |
| *** DinaBelova_ is now known as DinaBelova | 18:11 | |
| clarkb | dkranz: ideally folks would review the bug, see if it applies then make a decision on whether or not to run the gauntlet again | 18:11 |
| *** johnthetubaguy has quit IRC | 18:11 | |
| clarkb | dkranz: its possible the failure rate is 100% and should not be reverified | 18:12 |
| dkranz | clarkb: Got it. | 18:12 |
| *** shivh has joined #openstack-infra | 18:12 | |
| jeblair | clarkb, dhellmann: hashar just added a 'doc' env for zuul: https://review.openstack.org/#/c/65987/ | 18:13 |
| clarkb | dhellmann: the trouble here is we want to do `python setup.py test -t ''` in the case where no posargs are passed and `python setup.py test -t 'my args'` when posargs are passed. The current implementation of tox breaks that because it eats the ''s | 18:13 |
| jeblair | dhellmann: basically the same arguments | 18:13 |
| clarkb | dhellmann: that leaves us with `python setup.py -t` when no posargs are passed which is an argument parser error because -t take arguments | 18:14 |
| annegentle | I'm behind on what's up with storyboard... I see meetings, who all's involved? | 18:14 |
| jeblair | clarkb: i think whether we use -evenv or -edoc for doc builds is a separate question from whether -edoc exists; there's ups/downs to both | 18:14 |
| clarkb | dhellmann: we could tell everyone to `tox -epy27 -- ''` to get around that but ugh | 18:14 |
| jeblair | annegentle: infra, hp and mirantis folks so far | 18:14 |
| jeblair | annegentle: also jcoufal | 18:14 |
| jeblair | annegentle: ttx | 18:15 |
| clarkb | dhellmann: or we can ditch python setup.py test and go back to the clunky multistep inline bash script | 18:15 |
| SergeyLukjanov | jeblair, highlighter triggered :) | 18:15 |
| *** dkliban has joined #openstack-infra | 18:15 | |
| *** sarob has joined #openstack-infra | 18:15 | |
| openstackgerrit | Steve Baker proposed a change to openstack-infra/config: tempest-dsvm-neutron-heat-slow graduates from experimental https://review.openstack.org/70037 | 18:15 |
| *** dizquierdo has quit IRC | 18:15 | |
| dstufft|laptop | clarkb: Why can't you just let tox handle all the run stuff? What does the bash script do that tox can't? | 18:15 |
| dstufft|laptop | or what would it do I mean | 18:15 |
| jeblair | dstufft|laptop: lots of file moves based on zuul branch/tag names for one | 18:16 |
| fungi | dstufft|laptop: have the ability to correctly pass arguments to subcommands | 18:16 |
| jeblair | fungi: zing | 18:16 |
| jeblair | annegentle: we're trying to get a really basic system up asap so it's self-hosting | 18:17 |
| annegentle | jeblair: cool thanks. storyboard on openstack | 18:17 |
| clarkb | dstufft|laptop: is this re targets or re tox 1.7? | 18:17 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 18:17 | |
| fungi | annegentle: were you breaking the ice in here because you're about to ask for an ops guide edits branch, or are you not ready for that just yet? | 18:18 |
| clarkb | dstufft|laptop: re tox 1.7 is tox fails fast, sometimes you want the collective result of several commands, need a bash wrapper for that | 18:18 |
| clarkb | dstufft|laptop: re targets, it requires updating files all over the place, it is just a lot of busy work | 18:18 |
| dstufft|laptop | clarkb: ahh, gotcha | 18:18 |
| dstufft|laptop | also jeblair and fungi too :) | 18:18 |
| *** lcheng_ has quit IRC | 18:19 | |
| dstufft|laptop | mostly wondering because I hadn't seen many people actually use setup.py test outisde of openstack :) | 18:19 |
| clarkb | dstufft|laptop: because they all use nose and nose wraps this stuff for you too | 18:19 |
| dstufft|laptop | nose is crazy pants | 18:19 |
| dstufft|laptop | py.test ftw :D | 18:19 |
| fungi | also nose cuts off you to spite your face | 18:19 |
| dstufft|laptop | but yea | 18:19 |
| clarkb | nose is equivalent to setup.py test in this case. The difference is the name of the command, but you pass different flags based on the additional actions you want | 18:19 |
| *** sarob has quit IRC | 18:20 | |
| dstufft|laptop | clarkb: I actually want folks to use a standard entry point to testing eventually, so I can implement something that'll record test runs ala cpan testers | 18:20 |
| dstufft|laptop | ~someday~ | 18:20 |
| *** mestery has quit IRC | 18:20 | |
| clarkb | so we could probably tackle this by exposing testr args directly though setup.py test rather than doing a pass through | 18:21 |
| *** jcooley_ has quit IRC | 18:21 | |
| clarkb | eg python setup.py test --slowest becomes testr run --slowest | 18:21 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:21 | |
| fungi | or we prefix and mangle args like python setup.py test --testr-slowest (to give us less chance for namespace conflicts) | 18:22 |
| *** morganfainberg_Z is now known as morganfainberg | 18:22 | |
| *** DinaBelova is now known as DinaBelova_ | 18:23 | |
| *** david_lyle_ is now known as david_lyle | 18:23 | |
| *** AlexF has quit IRC | 18:23 | |
| mtreinish | clarkb: that'll get tricky --slowest isn't a testr arg. It's a pbr flag that calls 'testr slowest' (maybe just a bad example) | 18:23 |
| *** GheRivero has joined #openstack-infra | 18:23 | |
| *** CaptTofu has joined #openstack-infra | 18:24 | |
| clarkb | mtreinish: yup I would rather avoid it | 18:24 |
| fungi | i also feel like we've got an unusually high number of layers there... tox->setup->pbr->testr | 18:25 |
| fungi | though i suppose setup.py and pbr are sort of one in the same when you pbr | 18:25 |
| *** changbl has joined #openstack-infra | 18:27 | |
| markwash | maybe wrong room, but. . anybody here know how to run testr (via tox) with an option to suppress log output for failed tests? | 18:28 |
| clarkb | markwash: OS_LOG_CAPTURE=0 may work | 18:30 |
| clarkb | that might also dump your logs to stderr though... | 18:30 |
| *** dstufft|laptop has quit IRC | 18:30 | |
| mtreinish | clarkb: won't that just disable the fixture that injects the logs into the subunit stream | 18:32 |
| mtreinish | the regular logging will still happen | 18:32 |
| mtreinish | oh that's what markwash wants nm | 18:32 |
| clarkb | mtreinish: ya so now it wont show in the failed test results instead it will go to someplace else | 18:33 |
| mtreinish | clarkb: it depends on how logging is configured it will do whatever is set there | 18:34 |
| *** smurugesan has joined #openstack-infra | 18:34 | |
| *** igor has quit IRC | 18:40 | |
| *** smurugesan has quit IRC | 18:41 | |
| *** mestery has joined #openstack-infra | 18:41 | |
| *** e0ne has joined #openstack-infra | 18:43 | |
| *** chris_johnson has quit IRC | 18:44 | |
| *** igor has joined #openstack-infra | 18:44 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 18:45 | |
| *** e0ne has quit IRC | 18:45 | |
| *** gokrokve has joined #openstack-infra | 18:46 | |
| *** smurugesan has joined #openstack-infra | 18:47 | |
| *** dangers_away is now known as dangers | 18:47 | |
| *** ArxCruz has quit IRC | 18:47 | |
| *** ArxCruz has joined #openstack-infra | 18:47 | |
| *** dstufft|laptop has joined #openstack-infra | 18:50 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 18:51 | |
| *** talluri has quit IRC | 18:52 | |
| *** talluri has joined #openstack-infra | 18:53 | |
| *** lcheng_ has joined #openstack-infra | 18:55 | |
| *** talluri has quit IRC | 18:57 | |
| *** jcooley_ has joined #openstack-infra | 18:58 | |
| *** emagana has quit IRC | 18:58 | |
| *** ruhe_ has joined #openstack-infra | 18:59 | |
| *** dstufft|laptop has quit IRC | 19:00 | |
| *** krtaylor has quit IRC | 19:01 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Fix management queue test errors https://review.openstack.org/71626 | 19:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Make queue processing more efficient https://review.openstack.org/71627 | 19:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Split the merger into a separate process https://review.openstack.org/71628 | 19:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Move the merger to a JSON-compatible API https://review.openstack.org/70310 | 19:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Remove Zuul ref replication. https://review.openstack.org/70309 | 19:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Remove push refs to gerrit feature https://review.openstack.org/68723 | 19:01 |
| jeblair | yaaaay! | 19:01 |
| SergeyLukjanov | jeblair, you're a batch processor in terms of Big Data frameworks xD | 19:02 |
| clarkb | fun things to review! but first I need to push my postgres test slave fixe | 19:05 |
| *** dstanek has quit IRC | 19:05 | |
| openstackgerrit | A change was merged to openstack-infra/gitdm: Adding Matt Farina. https://review.openstack.org/70422 | 19:06 |
| *** dstanek has joined #openstack-infra | 19:06 | |
| *** godp1301 has joined #openstack-infra | 19:08 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:09 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make postgres work on Jenkins slaves again. https://review.openstack.org/71629 | 19:09 |
| markwash | mtreinish: any theories about how I can simulate nose's --nologoutput option with testr? | 19:10 |
| clarkb | fungi: jeblair ^ that should fix postgres for any new slaves built (including single use slaves) | 19:10 |
| clarkb | markwash: did OS_LOG_CAPTURE=0 not help? | 19:10 |
| markwash | ah sorry thought that was shot down, trying now. . . | 19:11 |
| pafuent | NikitaKonovalov_: Hi. I noticed that you have assigned https://bugs.launchpad.net/climate/+bug/1252704. Are you ok if I continue with it? | 19:11 |
| uvirtbot | Launchpad bug 1252704 in climate "Value error raises if 'lease_create' get incorrect date format" [High,Confirmed] | 19:11 |
| clarkb | markwash: I think it may be the first piece of a two piece puzzle. Second piece is probably setting log config such that logging doesn't happen (or redirecting stderr to elsewhere) | 19:11 |
| markwash | gotcha | 19:12 |
| *** DinaBelova_ is now known as DinaBelova | 19:14 | |
| *** e0ne has joined #openstack-infra | 19:14 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:15 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 19:15 |
| openstackgerrit | Aaron Greengrass proposed a change to openstack-infra/config: Module to handle package management via hiera https://review.openstack.org/71630 | 19:16 |
| lifeless | fungi: derek was working on bringing it up last night, I'm back on it today (yesterday was a pub holiday...) | 19:19 |
| *** mestery has quit IRC | 19:19 | |
| zaro | alright, i believe i've finally finished testing zuul gerrit 2.8 integration | 19:20 |
| zaro | clarkb: ^ thanks for the help with zuul. | 19:20 |
| markwash | clarkb: no extra step needed, thanks! | 19:21 |
| fungi | lifeless: my country needs more pub holidays. not enough days on the calendar dedicated to spending time in pubs | 19:21 |
| lifeless | fungi: you know it | 19:21 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:23 | |
| *** melwitt has joined #openstack-infra | 19:24 | |
| *** kiall has joined #openstack-infra | 19:24 | |
| zaro | is it normal for zuul to trigger a check job on -1 vote? https://review.openstack.org/#/c/69608 | 19:26 |
| *** dstanek has quit IRC | 19:27 | |
| *** ruhe_ has quit IRC | 19:27 | |
| *** dstanek has joined #openstack-infra | 19:27 | |
| fungi | zaro: as of yesterday, if the change has not been tested for at least 48 hours, then yes | 19:27 |
| fungi | any comment left on a change will trigger that recheck | 19:28 |
| fungi | this helps us ensure fairly recent check results for changes under active development | 19:28 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:28 | |
| *** max_lobur is now known as max_lobur_afk | 19:30 | |
| zul | fungi: hey for the ": Cannot find 'pm-is-supported' in path: No such file or directory" in the libvirt error logs pm-utils needs to be installed (# 1274772) I just fixed it in trusty | 19:30 |
| *** sarob has joined #openstack-infra | 19:30 | |
| fungi | zul: oh, interesting | 19:31 |
| zul | fungi: its listed as a suggests | 19:31 |
| fungi | zul: will the pm-utils in precise{,-backports} do or is it going to be in uca? | 19:31 |
| zul | it should do | 19:31 |
| *** mrodden has quit IRC | 19:32 | |
| fungi | zul: do we think that's the only known blocker now for switching our ci to current libvirt in uca? | 19:32 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:32 | |
| zul | fungi: except for the nwfilter race as well should be fixed in trusty (then via uca) | 19:33 |
| fungi | and in a related vein, can the libvirt package in uca get a backport of the suggests->recommends/depends patch? | 19:33 |
| fungi | (so apt-get will pull in pm-utils when libvirt is upgraded?) | 19:34 |
| zul | fungi: if you are using the icehouse pocket (libvirt 1.2.x it will alrready be there) | 19:34 |
| fungi | oh, excellent | 19:34 |
| fungi | dims: ^ | 19:34 |
| zul | when i upload it | 19:34 |
| fungi | ;) | 19:34 |
| *** rlandy|bbl is now known as rlandy | 19:35 | |
| fungi | zul: you're uploading that to precise uca, not trusty uca though, right? | 19:35 |
| *** jcooley_ has quit IRC | 19:35 | |
| * fungi can't remember whether icehouse uca is trusty-only | 19:35 | |
| zul | precise-icehouse uca | 19:35 |
| fungi | perfect | 19:35 |
| *** dstanek has quit IRC | 19:36 | |
| *** jcooley_ has joined #openstack-infra | 19:36 | |
| *** gokrokve has quit IRC | 19:36 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:38 | |
| zul | does someone have the bug number for the nwfilter bug? | 19:39 |
| *** dstanek has joined #openstack-infra | 19:39 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:39 | |
| *** chris_johnson has joined #openstack-infra | 19:39 | |
| mriedem | i'm probably behind on the latest gossip, but why did jenkins just 'no score' this? https://review.openstack.org/#/c/70817/ | 19:40 |
| mriedem | zul: nwfilter in use libvirt error? | 19:40 |
| *** jcooley_ has quit IRC | 19:40 | |
| mriedem | it's fixed | 19:40 |
| *** jcooley_ has joined #openstack-infra | 19:41 | |
| mriedem | zul: see https://bugs.launchpad.net/nova/+bug/1276772 | 19:41 |
| uvirtbot | Launchpad bug 1276772 in nova "Instance deletion fails due to race condition on Hyper-V" [High,In progress] | 19:41 |
| mriedem | zul: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRXJyb3IgZnJvbSBsaWJ2aXJ0IGR1cmluZyB1bmZpbHRlci4gQ29kZT01NSBFcnJvcj1SZXF1ZXN0ZWQgb3BlcmF0aW9uIGlzIG5vdCB2YWxpZDogbndmaWx0ZXIgaXMgaW4gdXNlXCIgQU5EIGZpbGVuYW1lOlwibG9ncy9zY3JlZW4tbi1jcHUudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImFsbCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTE2MjM5MzE5MDN9 | 19:41 |
| mriedem | the revert yesterday made that drop off logstash | 19:41 |
| sdague | mriedem: that's the auto recheck feature | 19:41 |
| *** pcrews has quit IRC | 19:41 | |
| mriedem | sdague: auto-what-now?! | 19:41 |
| mriedem | clearly i'm behind on mail | 19:41 |
| sdague | so reviews without current results are getting auto rechecked when people comment on them | 19:42 |
| zul | https://bugs.launchpad.net/nova/+bug/1228977 is the one i was referring to | 19:42 |
| uvirtbot | Launchpad bug 1228977 in nova "n-cpu seems to crash when running with libvirt 1.1.1 from ubuntu cloud archive" [High,Confirmed] | 19:42 |
| sdague | so we don't have the issue with month old invalid test results pushing things into gate queue | 19:42 |
| dims | zul, fungi , i am trying to build/run the 1.1.1-maint libvirt git tree since this morning https://review.openstack.org/#/c/67564/ it has the patches from Daniel | 19:42 |
| mriedem | zul: ah | 19:42 |
| mriedem | sdague: cool - and that's not bashing the check queue? | 19:43 |
| sdague | doesn't seem to be | 19:43 |
| sdague | we're set to 48hr freshness right now | 19:44 |
| *** mrodden has joined #openstack-infra | 19:44 | |
| mriedem | like a spring rain... | 19:44 |
| *** sarob has quit IRC | 19:44 | |
| zul | dims: if you gimme a couple of minutes i can throw what im working on into a ppa for you until it hits the cloud archive | 19:45 |
| *** jcooley_ has quit IRC | 19:45 | |
| *** markmcclain has quit IRC | 19:45 | |
| *** sarob has joined #openstack-infra | 19:45 | |
| *** markmcclain has joined #openstack-infra | 19:45 | |
| *** thomasbiege1 has quit IRC | 19:45 | |
| zul | dims: so you can test it out, etc, etc, etc, etc | 19:45 |
| *** thomasbiege has joined #openstack-infra | 19:46 | |
| *** che-arne has quit IRC | 19:49 | |
| *** jcooley_ has joined #openstack-infra | 19:50 | |
| dims | zul, the environment is precise with uca - so it has libvirt 1.1.1. will the ppa be 1.1.1 based? | 19:50 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 19:52 |
| *** torgomatic has left #openstack-infra | 19:52 | |
| *** sarob has quit IRC | 19:54 | |
| *** sarob has joined #openstack-infra | 19:54 | |
| *** emagana has joined #openstack-infra | 19:54 | |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added clone-workspace publisher https://review.openstack.org/69532 | 19:56 |
| *** jcooley_ has quit IRC | 19:56 | |
| *** ladquin has joined #openstack-infra | 19:57 | |
| zul | dims: trusty has 1.2.1 | 19:57 |
| *** jcooley_ has joined #openstack-infra | 19:57 | |
| dims | zul, understood, all our gate/check jobs are on precise. | 19:59 |
| *** sarob has quit IRC | 19:59 | |
| zul | dims: so with the ca enabled it will be with 1.2.1 when its put through the paces | 19:59 |
| dims | you mean 'deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/havana main' will end up providing us libvirt 1.2.1? | 20:01 |
| zul | dims: | 20:03 |
| zul | oh no i mean precise-updates/icehouse | 20:03 |
| *** lcheng_ has quit IRC | 20:03 | |
| *** godp1301 has quit IRC | 20:03 | |
| zul | ill see what i can do to get it backported | 20:03 |
| dims | zul, ah ok. thanks! | 20:04 |
| *** HenryG_ has quit IRC | 20:04 | |
| *** HenryG_ has joined #openstack-infra | 20:05 | |
| *** HenryG_ has quit IRC | 20:05 | |
| *** rfolco has quit IRC | 20:05 | |
| *** HenryG has joined #openstack-infra | 20:06 | |
| *** lcheng_ has joined #openstack-infra | 20:06 | |
| fungi | clarkb: not sure if you caught it in scrollback, but do you expect bug 1277109 was from one of the handful of servers launched off that failed image build experiment? | 20:10 |
| uvirtbot | Launchpad bug 1277109 in openstack-ci "Job failed to install dependencies - libxml dev package doesn't appear to be installed" [Undecided,New] https://launchpad.net/bugs/1277109 | 20:10 |
| clarkb | fungi: yes | 20:10 |
| clarkb | the timestamps seem to line up too | 20:10 |
| *** jcooley_ has quit IRC | 20:12 | |
| *** jcooley_ has joined #openstack-infra | 20:12 | |
| fungi | cool, updated and closed out. thanks clarkb | 20:14 |
| *** sarob has joined #openstack-infra | 20:18 | |
| mriedem | is there a way to filter logstash queries based on stable branches only? | 20:19 |
| *** thomasbiege has quit IRC | 20:20 | |
| mriedem | i want to see if we backported a race bug from master | 20:20 |
| clarkb | mriedem: yes there is a new build_branch field | 20:21 |
| fungi | mriedem: AND build_branch:"stable/havana" | 20:21 |
| mriedem | nice, thanks | 20:21 |
| *** lcestari has quit IRC | 20:22 | |
| *** mrda_away is now known as mrda | 20:22 | |
| mriedem | jog0: does e-r report on stable branch patches? not seeing this https://review.openstack.org/#/c/70556/ reporting on https://review.openstack.org/#/c/70474/ | 20:23 |
| mriedem | hmm, logstash isn't finding it either | 20:23 |
| mriedem | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU1NIRXhjZXB0aW9uOiBFcnJvciByZWFkaW5nIFNTSCBwcm90b2NvbCBiYW5uZXJcIiBBTkQgZmlsZW5hbWU6XCJjb25zb2xlLmh0bWxcIiBBTkQgYnVpbGRfYnJhbmNoOlwic3RhYmxlL2hhdmFuYVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiJhbGwiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkxNzE4MTU0MjQ4LCJtb2RlIjoidGVybXMiLCJhbmFseXplX2ZpZWxkIjoiYnVpbGRfbmFtZSJ9 | 20:23 |
| mriedem | on stable/havana at least? | 20:23 |
| clarkb | mriedem: please don't search all time... | 20:24 |
| clarkb | I am going to rip all time out of the options list | 20:24 |
| mriedem | ok, 7 days doesn't hit either | 20:24 |
| clarkb | right it won't affect the content of results, but all time is basically go nuts mode for elasticsearch | 20:25 |
| mriedem | yeah | 20:25 |
| clarkb | when you give it boundaries queries are much more responsive | 20:25 |
| clarkb | mriedem: can you link to a log file that should match that? | 20:25 |
| mriedem | ok, so back to my main issue - i'm seeing an error we have an e-r query for in a stable/havana patch but it's not hitting in logstash | 20:25 |
| mriedem | http://logs.openstack.org/74/70474/1/check/check-tempest-dsvm-neutron/df2d911/console.html | 20:25 |
| mriedem | well damn | 20:26 |
| mriedem | i think our e-r query must be busted.... | 20:26 |
| *** markmcclain has quit IRC | 20:26 | |
| mriedem | i think jog0 and i must have been drunk one night when we changed it... | 20:26 |
| *** jcoufal has joined #openstack-infra | 20:28 | |
| *** malini is now known as malini_afk | 20:29 | |
| *** sarob has quit IRC | 20:29 | |
| *** smurugesan has quit IRC | 20:30 | |
| *** jooools has quit IRC | 20:30 | |
| *** sarob has joined #openstack-infra | 20:30 | |
| mriedem | doesn't matter anyway, stable/havana isn't showing up in logstash for this: | 20:30 |
| mriedem | filename:"console.html" AND message:"SSHException: Error reading SSH protocol banner" | 20:30 |
| mriedem | but it's in that log above | 20:30 |
| clarkb | and the log is indexed | 20:31 |
| *** smurugesan has joined #openstack-infra | 20:31 | |
| *** freyes has quit IRC | 20:31 | |
| openstackgerrit | A change was merged to openstack/requirements: Start renaming oslo.sphinx to oslosphinx https://review.openstack.org/71612 | 20:31 |
| *** oubiwann_ has joined #openstack-infra | 20:31 | |
| clarkb | mriedem: filename:"console.html" AND build_branch:"stable/havana" AND message:"SSHException: Error reading SSH protocol banner" AND build_uuid:"df2d911e4e674e0088ce3340a208baa4" | 20:31 |
| *** smurugesan has quit IRC | 20:31 | |
| *** smurugesan has joined #openstack-infra | 20:31 | |
| mriedem | hmm, why is build_uuid needed to make it work? | 20:32 |
| clarkb | mriedem: I was just using it to filter more, It shouldn't be necessary | 20:32 |
| mriedem | ah weird, this works filename:"console.html" AND build_branch:"stable/havana" AND message:"SSHException: Error reading SSH protocol banner" | 20:33 |
| mriedem | it seems that ordering of the filters matters? | 20:33 |
| clarkb | maybe? that would be weird | 20:33 |
| *** sarob has quit IRC | 20:34 | |
| mriedem | clarkb: well now this is working for me: message:"raise SSHException(\'Error reading SSH protocol banner\' + str(x))" AND filename:"console.html" AND build_branch:"stable/havana" | 20:35 |
| russellb | oops, looks like we have gantt in the integrated gate ... probably should take it out for now | 20:35 |
| mriedem | clarkb: did you hit the "pay clarkb today" button? | 20:36 |
| clarkb | no that happens in ~9 days | 20:36 |
| russellb | so how do i get gantt to run run in the main gate queue | 20:38 |
| russellb | it's not worth letting it get in the way of other things right now | 20:38 |
| *** krtaylor has joined #openstack-infra | 20:39 | |
| russellb | not* run | 20:39 |
| clarkb | russellb: we need to edit the zuul layout.yaml fiel so that gantt doesn't share tests with any of the integrated projects | 20:39 |
| annegentle | fungi: or clarkb: or jeblair: requesting a branch of openstack/operations-guide cut from commit 3a86619f1b41b3d41a7ecf05495c1079cab01fc3 what should it be named? It's the one for O'Reilly | 20:39 |
| clarkb | the tempest tests are probably shared which is why it ends up there | 20:39 |
| *** cadenzajon has quit IRC | 20:40 | |
| russellb | clarkb: ACK, will do | 20:40 |
| fungi | annegentle: feature/edits maybe? | 20:40 |
| *** jamielennox|away is now known as jamielennox | 20:40 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 20:41 |
| fungi | russellb: if gantt wants equivalents of the integrated gate jobs, it can instantiate some from similarly-named but not identical templates, as placeholders until it's graduated | 20:41 |
| clarkb | fungi: russellb: ironic does this with -nv named jobs | 20:41 |
| fungi | russellb: and marconi now | 20:42 |
| SergeyLukjanov | clarkb, fungi, russellb, and savanna already have -nv named jobs | 20:42 |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Enhance project specific parameters example. https://review.openstack.org/71647 | 20:42 |
| fungi | right. there's plenty of precedent then! | 20:42 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Trim down gantt check/gate jobs https://review.openstack.org/71648 | 20:42 |
| SergeyLukjanov | fungi, yup, we can now name in standard approach :) | 20:42 |
| russellb | clarkb: fungi thanks! No need right now, until we actually have variants of those jobs that can use it | 20:43 |
| russellb | but i've asked the folks working on it to look at doing that next | 20:44 |
| fungi | annegentle: i've created a feature/edits branch of openstack/operations-guide from existing commit 3a86619f1b41b3d41a7ecf05495c1079cab01fc3 | 20:46 |
| annegentle | fungi: sure | 20:46 |
| annegentle | fungi: and thanks! | 20:46 |
| fungi | annegentle: we'll presumably want to set up an acl to allow merging between that branch and master | 20:47 |
| annegentle | fungi: yes! | 20:47 |
| annegentle | fungi: please! | 20:47 |
| fungi | annegentle: but we can address that later when you're ready to try to sync things up if you want | 20:47 |
| annegentle | fungi: it can just be me for now | 20:47 |
| fungi | annegentle: sure thing. i'll put together the acl change for that | 20:47 |
| annegentle | fungi: I'll email O'Reilly now | 20:47 |
| fungi | annegentle: and i'll get you the link to the feature branch workflow documentation | 20:47 |
| annegentle | fungi: also do we need any special ACL for that branch specifically? I don't think that was a requiremen but my memory is fuzzy | 20:48 |
| annegentle | requirement | 20:48 |
| fungi | annegentle: just for being able to push merge commits to review. for normal change commits it should work similarly to master | 20:49 |
| annegentle | fungi: oh ok, pushing merge commits to review is usually which entity? | 20:49 |
| fungi | merge commits will be needed when the time comes to either import updates from master into the feature branch or fold the feature branch back into master | 20:49 |
| annegentle | fungi: ok, I know we need that bidirectional movement for commits | 20:49 |
| fungi | annegentle: the feature branch merge commit concept is detailed in https://wiki.openstack.org/wiki/GerritJenkinsGit#Merge_Commits | 20:51 |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin https://review.openstack.org/71650 | 20:51 |
| annegentle | fungi: who else uses feature branches? nova? | 20:51 |
| *** julim has joined #openstack-infra | 20:52 | |
| fungi | annegentle: keystone was using them for api v3 i believe, swift for erasure coding, zuul did for gearman support. there have probably been more | 20:52 |
| *** banix has quit IRC | 20:52 | |
| *** yolanda has quit IRC | 20:52 | |
| fungi | so the workflow has at least been tried out a few times with successful results | 20:52 |
| annegentle | fungi: nice, thanks, hadn't made that connection | 20:52 |
| annegentle | fungi: So glad you guys make these connections for me :) | 20:53 |
| *** julim has quit IRC | 20:53 | |
| openstackgerrit | Terry Howe proposed a change to openstack-infra/gitdm: Terry Howe added to other files https://review.openstack.org/71651 | 20:53 |
| * fungi connects things all day | 20:53 | |
| clarkb | dstufft: greghaynes has a problem downloading ipaddr 2.1.11 from google via pypi. On pypi that release seems to have disappeared. Is this a case of a removed release? | 20:54 |
| clarkb | jeblair: these zuul changes are pretty meaty | 20:54 |
| clarkb | https://review.openstack.org/#/c/71627/1/zuul/scheduler.py in particular is spinning some gears in my head | 20:55 |
| *** jcoufal has quit IRC | 20:56 | |
| *** dkliban is now known as dkliban_afk | 20:56 | |
| clarkb | and the one after that is going to take some time | 20:57 |
| *** julim has joined #openstack-infra | 20:58 | |
| *** denis_makogon_ has joined #openstack-infra | 21:00 | |
| *** andreaf2 has quit IRC | 21:00 | |
| *** denis_makogon_ is now known as denis_makogon | 21:01 | |
| fungi | i haven't had time to start looking at it yet, but figure i'll need to be in the mindframe for meaty-meat-pie reviewing | 21:01 |
| *** slong has joined #openstack-infra | 21:01 | |
| clarkb | fungi: it ramps you up | 21:01 |
| clarkb | there are a few simple changes, then you get into the slightly more complicated change then bam rip out mergers | 21:02 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 21:02 | |
| devananda | is jenkins not voting for checks now? | 21:03 |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin https://review.openstack.org/71655 | 21:04 |
| clarkb | devananda: depends on the tests run | 21:04 |
| devananda | clarkb: https://review.openstack.org/#/c/71297/2 | 21:04 |
| *** smarcet has left #openstack-infra | 21:04 | |
| devananda | clarkb: check jobs finished ~2hr ago, i don't see another in the queue, but there's no +1/-1 | 21:05 |
| clarkb | devananda: kill all drafts with fire | 21:05 |
| devananda | draft? | 21:05 |
| clarkb | yes I believe thta is a draft change | 21:05 |
| *** thomasbiege has joined #openstack-infra | 21:06 | |
| clarkb | which is why no one can vote on it | 21:06 |
| devananda | oh jeez | 21:06 |
| devananda | yea | 21:06 |
| devananda | Status: review in progress.... | 21:06 |
| devananda | it would be great if the status said "draft" or something | 21:06 |
| devananda | sorry for the noise | 21:06 |
| *** pafuent has left #openstack-infra | 21:07 | |
| clarkb | devananda: they are supposed to be secure ish | 21:07 |
| clarkb | I think the idea is you don't want to leak state | 21:07 |
| clarkb | but it does so anyways | 21:08 |
| devananda | right | 21:08 |
| devananda | yea, i get the draft thing, just forget to look for that litte "(draft)" tag on each rev, and given that you guys changed jenkins yesterday, my mind jumped to maybe-its-related | 21:09 |
| devananda | rather than thinking "look for taht wierd draft thing" | 21:09 |
| *** slong_ has joined #openstack-infra | 21:10 | |
| *** thuc_ has quit IRC | 21:10 | |
| *** thuc has joined #openstack-infra | 21:11 | |
| *** ken1ohmichi has joined #openstack-infra | 21:11 | |
| fungi | always be on the lookout for that wierd draft thing | 21:12 |
| *** markwash has quit IRC | 21:12 | |
| *** pcrews has joined #openstack-infra | 21:12 | |
| *** gokrokve has joined #openstack-infra | 21:14 | |
| *** slong has quit IRC | 21:14 | |
| *** thuc has quit IRC | 21:15 | |
| *** sarob has joined #openstack-infra | 21:17 | |
| *** lcostantino has quit IRC | 21:18 | |
| *** markmcclain has joined #openstack-infra | 21:18 | |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: added missing blank line https://review.openstack.org/71656 | 21:20 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/git-review: Fix the regex for setting topic https://review.openstack.org/71657 | 21:20 |
| *** ryanpetrello has quit IRC | 21:20 | |
| *** rlandy has quit IRC | 21:20 | |
| *** ryanpetrello has joined #openstack-infra | 21:22 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 21:23 |
| *** krtaylor has quit IRC | 21:24 | |
| *** DinaBelova is now known as DinaBelova_ | 21:25 | |
| *** jcooley_ has quit IRC | 21:28 | |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin https://review.openstack.org/71658 | 21:29 |
| jhesketh_ | Morning | 21:29 |
| *** jcooley_ has joined #openstack-infra | 21:29 | |
| *** cadenzajon has joined #openstack-infra | 21:29 | |
| *** klrmn has joined #openstack-infra | 21:30 | |
| *** jcoufal-mobile has joined #openstack-infra | 21:31 | |
| *** sarob has quit IRC | 21:33 | |
| *** sarob has joined #openstack-infra | 21:33 | |
| clarkb | jhesketh_: morning | 21:34 |
| jeblair | clarkb: yeah, to be fair, including docs with the change makes it look a little bit bigger; also there's a git mv of merger.py which pretty much doesn't change anything... | 21:35 |
| jeblair | clarkb: but the last one isn't small | 21:35 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make postgres work on Jenkins slaves again. https://review.openstack.org/71629 | 21:35 |
| clarkb | fungi: jeblair ^ should be ready now | 21:35 |
| jeblair | clarkb: let me know if there's anything i can elucidate | 21:35 |
| *** dprince has quit IRC | 21:35 | |
| clarkb | jeblair: I am writing notes as I go, one of the earlier changes has questions | 21:35 |
| *** e0ne has quit IRC | 21:36 | |
| *** sarob has quit IRC | 21:36 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to set_hooks_commit_msg https://review.openstack.org/70746 | 21:36 |
| jeblair | clarkb: ok cool, i'll do a pass now; looks like there's some verify-1s too | 21:36 |
| *** sarob has joined #openstack-infra | 21:37 | |
| jeblair | jhesketh_: good morning; i just want to let you know the swift stuff is on my radar; i've been looking at bits of it casually and hope to do proper reviews soon... | 21:37 |
| jeblair | jhesketh_: i kind of had to get the merger change out of my system | 21:37 |
| jhesketh_ | no worries jeblair :-) | 21:37 |
| *** masayukig has joined #openstack-infra | 21:38 | |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added clone-workspace publisher https://review.openstack.org/69532 | 21:39 |
| *** klrmn has left #openstack-infra | 21:40 | |
| *** sarob has quit IRC | 21:41 | |
| *** zehicle_at_dell has joined #openstack-infra | 21:42 | |
| *** mgagne has joined #openstack-infra | 21:42 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow merge commits for operations-guide https://review.openstack.org/71661 | 21:47 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 21:48 |
| *** UtahDave has joined #openstack-infra | 21:52 | |
| clarkb | jeblair: I have commented on the full stack at this point | 21:53 |
| lifeless | clarkb: reminder to get me the diagnostic logs | 21:54 |
| clarkb | lifeless: thanks | 21:54 |
| lifeless | my hunch is rate limiting making the batch delete slow | 21:55 |
| openstackgerrit | A change was merged to openstack-infra/config: Make postgres work on Jenkins slaves again. https://review.openstack.org/71629 | 21:55 |
| lifeless | so it's waiting a full batch | 21:55 |
| *** zehicle_at_dell has quit IRC | 21:55 | |
| *** zehicle_at_dell has joined #openstack-infra | 21:55 | |
| jeblair | clarkb: thx, i'm working through it now | 21:56 |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: added support for Environment Script Plugin https://review.openstack.org/71663 | 22:01 |
| *** dkliban_afk has quit IRC | 22:04 | |
| *** jcoufal-mobile has quit IRC | 22:06 | |
| *** chris_johnson has quit IRC | 22:06 | |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: added support for Environment Script Plugin https://review.openstack.org/71663 | 22:08 |
| *** david_lyle has quit IRC | 22:09 | |
| lifeless | clarkb: how long was the new logging running for ? | 22:09 |
| *** sc68cal_ has quit IRC | 22:10 | |
| lifeless | clarkb: I can only see rax-dfw finishing and starting *once* in that log | 22:10 |
| lifeless | clarkb: which is hours long | 22:10 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 22:11 | |
| jeblair | lifeless, clarkb: i believe two of the rax providers stopped working | 22:11 |
| clarkb | lifeless: it was just a handful of hours | 22:11 |
| clarkb | not very long | 22:11 |
| jeblair | lifeless, clarkb: that may be related to adding the rate limit value to the config | 22:11 |
| lifeless | jeblair: in nodepool (we stopped using it) or externally (it died and we couldn't use it) ? | 22:11 |
| jeblair | lifeless: in nodepool | 22:11 |
| fungi | lifeless: it was fairly brief. part of the problem was that the new changes seem to have made the 'nodepool delete' cli a no-op (as far as i was able to observe) and we were hemmoraging pending deletes all over our quotas so when we saw that it wasn't bringing down the delete volume appreciatively, we reverted it all asap | 22:12 |
| jeblair | lifeless: so that could be a related or separate bug... | 22:12 |
| fungi | because without nodepool delete as a fallback, we really had no way to easily bring the pending delete volume back down far enough as to no longer be starved for nodes | 22:13 |
| jeblair | lifeless: it's possible that changing a rate value in the config would even break nodepool without the delete changes; i think it's untested. | 22:13 |
| lifeless | fungi: ah yes, cli goes in through the path that became nonblocking | 22:13 |
| lifeless | fungi: rolling back was entirely sensible | 22:13 |
| lifeless | I just want to figure out what went wrong enough to either discard the approach or fix it | 22:13 |
| *** wenlock has quit IRC | 22:14 | |
| fungi | limiting the volume of testing so as to leave some buffer/breathing room for deletes to happen seems to have made improving delete performance less urgent, but i still agree it would be nice to improve it | 22:14 |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: added support for Environment Script Plugin https://review.openstack.org/71663 | 22:15 |
| *** A-G has joined #openstack-infra | 22:15 | |
| *** thomasem_ has quit IRC | 22:15 | |
| openstackgerrit | A change was merged to openstack-infra/config: Make requirement update proposals more robust. https://review.openstack.org/69969 | 22:15 |
| *** VijayT has joined #openstack-infra | 22:16 | |
| *** mfer has quit IRC | 22:21 | |
| *** banix has joined #openstack-infra | 22:23 | |
| *** jhesketh__ has joined #openstack-infra | 22:24 | |
| zaro | *** review-dev.o.o will be down for a few minutes for testing **** | 22:25 |
| lifeless | fungi: so its taking up to a minute to go from node to node in the loop cleaning them up | 22:26 |
| lifeless | fungi: I think its the rate limiting | 22:26 |
| lifeless | because our deletes are so inefficient | 22:27 |
| fungi | zaro: thanks for the heads-up! | 22:28 |
| fungi | lifeless: this sounds entirely possible given the behavior we were seeing | 22:28 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Make queue processing more efficient https://review.openstack.org/71627 | 22:29 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Fix management queue test errors https://review.openstack.org/71626 | 22:29 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Remove push refs to gerrit feature https://review.openstack.org/68723 | 22:29 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Split the merger into a separate process https://review.openstack.org/71628 | 22:29 |
| *** jnoller has quit IRC | 22:29 | |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Nodepool image logs accessible via HTTP https://review.openstack.org/71190 | 22:30 |
| jeblair | lifeless: i'm not following: rate limiting should account for no more than 1 second of time between calls; a delete should only take ~3 calls at most, so rate limiting should account for 3 seconds of the node-to-node time | 22:30 |
| *** dizquierdo has joined #openstack-infra | 22:31 | |
| jeblair | (at most) | 22:31 |
| jeblair | (0 seconds if the far end is slow enough) | 22:31 |
| lifeless | jeblair: if there are no other sources of API requests within nodepool | 22:31 |
| lifeless | clarkb: can you extract (for the same time period) /everything/ that mentions rax-dfw ? | 22:32 |
| clarkb | lifeless: sure | 22:32 |
| * fungi is being called to cook dinner, but will read scrollback in a bit | 22:32 | |
| jeblair | lifeless: are you suggesting the server builds are impacting it...? | 22:33 |
| jeblair | that could be the case... | 22:34 |
| lifeless | so, data we have: from before 2014-02-05 20:28:00 (start of log extract) until 2014-02-05 21:32:5 dfw was doing a cleanup | 22:34 |
| jeblair | lifeless: perhaps it's processing all of the build-related requests between each node delete. | 22:34 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Nodepool image logs accessible via HTTP https://review.openstack.org/71190 | 22:34 |
| lifeless | then it started another at 2014-02-05 21:33:00, and did not finish until after 2014-02-05 23:33:10, (end of log extract) | 22:34 |
| lifeless | thats > 1 hour for the first segment, and > 2 hours for the second | 22:35 |
| lifeless | that explains why node deletes were not happening fast, because they were picked up on the next cycle | 22:35 |
| lifeless | but the cycles were meant to be fast | 22:35 |
| jeblair | lifeless: the current approach tries to be fair to all tasks by having each one in a thread, and they all run as fast as possible while the task manager handles their requests in order | 22:36 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt https://review.openstack.org/67564 | 22:37 |
| jeblair | lifeless: it effectively serializes the requests across all of them because they all put in their requests at once, and when they get a response, if they are looping, they put in another request which goes to the end of the queue | 22:37 |
| lifeless | jeblair: thats definitely not fair - because python threading has no fairness guarantees on wakeups etc | 22:37 |
| jeblair | lifeless: but the task manager processes them in order | 22:37 |
| jeblair | lifeless: and that _is_ fair | 22:37 |
| lifeless | sure | 22:37 |
| lifeless | anyhow, if there were say 130 servers to cleanup | 22:38 |
| jeblair | lifeless: so you end up with something that looks very close to the task manager servicing one request for each thread, for all of the threads | 22:38 |
| lifeless | and we have 3 requests per thats 690 API calls, or 690 seconds at the 1/second default. | 22:38 |
| lifeless | thats mumble 10 minutes which is nowhere near hours. | 22:38 |
| jeblair | lifeless: or in other words, the behavior of the current approach converges on something close to servicing all of the nodes, then servicing them all again... | 22:39 |
| jeblair | lifeless: your approach explicitly puts all of the delete servers in a loop | 22:40 |
| jeblair | lifeless: to try to get roughly the same effect, but possibly with more strict sequencing, in a single thread | 22:40 |
| jeblair | lifeless: that's probably not compatible with the other part of the program still using the other method | 22:40 |
| lifeless | jeblair: and the [not implemented] opportunity to remove a bunch of the api calls | 22:40 |
| jeblair | lifeless: yeah, i'm very keen on that | 22:41 |
| lifeless | jeblair: IIRC there are two other parts of the rest of the program | 22:41 |
| lifeless | ignoring template builds | 22:41 |
| lifeless | there are probes for servers | 22:41 |
| lifeless | and there are spawns | 22:41 |
| jeblair | lifeless: yeah, it's the spawns that i'm thinking about -- that's where you could end up with GetServer calls for every spawning server between each of the node deletes | 22:42 |
| *** thuc has joined #openstack-infra | 22:42 | |
| jeblair | lifeless: (because there are 100 spawn threads all able to submit a task while there is only one delete thread able to submit a task) | 22:42 |
| lifeless | jeblair: we talked about using batch launches possibly | 22:42 |
| *** krtaylor has joined #openstack-infra | 22:43 | |
| lifeless | jeblair: e.g. ask the provider to create 50 servers at once rather than 50x1 ? | 22:43 |
| jeblair | lifeless: that sounds great | 22:43 |
| lifeless | jeblair: perhaps that would be a safe thing to bring in before the delete work ? | 22:43 |
| *** ociuhandu has quit IRC | 22:43 | |
| lifeless | since its strictly a reduced number of API calls (but perhaps the providers are fragile?) | 22:44 |
| *** thuc_ has joined #openstack-infra | 22:44 | |
| jeblair | lifeless: i think we'd want to be careful that we don't invert the problem | 22:44 |
| lifeless | jeblair: we'd still create N threads | 22:45 |
| lifeless | jeblair: just after doing a single create call | 22:45 |
| jeblair | lifeless: wasn't there a way to check on the status of multiple servers too? | 22:45 |
| lifeless | jeblair: just listServers should do it | 22:46 |
| *** thuc has quit IRC | 22:47 | |
| lifeless | clarkb: jeblair: visual inspection suggests 90% of rax-dfw traffic was GetServerTask | 22:47 |
| openstackgerrit | Michael Still proposed a change to openstack-infra/elastic-recheck: Don't encourage reverify https://review.openstack.org/70632 | 22:47 |
| lifeless | I'll poke more closely later. Right now, I have a cloud to build | 22:48 |
| mikal | Hey, is https://review.openstack.org/#/c/70632/ an example of gerrit noticing a comment on an old patch and rechecking? | 22:48 |
| jeblair | lifeless: ok. well, i have the feeling it would be safest to drop the hundreds-of-threads approach for both spawn and delete at once, but perhaps there's a path where that could be incremental | 22:48 |
| lifeless | and meetings and shit. | 22:48 |
| mikal | Cause if it is, that's cool | 22:48 |
| clarkb | mikal: yes | 22:48 |
| mikal | clarkb: awesome | 22:48 |
| jeblair | mikal: we hope to remove the 'starting check jobs' spam, but it's required atm | 22:49 |
| jeblair | clarkb: i thought of another in-gerrit way to address that... | 22:49 |
| jeblair | clarkb: if we could remove the vote without leaving a comment (cause really it's leaving the comment that bothers us about that) | 22:49 |
| *** dstanek is now known as dstanek_afk | 22:49 | |
| jeblair | clarkb: so an api equivalent of the "x" box near the vote in the ui | 22:50 |
| clarkb | I like that | 22:50 |
| clarkb | might be visually confusingto see things disappear but only for a moment | 22:50 |
| jeblair | clarkb: something to consider if the first idea isn't well received | 22:50 |
| jeblair | clarkb: yes. s/moment/50 minutes/ | 22:50 |
| jeblair | clarkb: also, i think i botched the git rebase on that zuul stack. it'll take me a few mins to fix. | 22:51 |
| *** markwash has joined #openstack-infra | 22:58 | |
| *** david-lyle has joined #openstack-infra | 22:59 | |
| *** markmcclain has quit IRC | 23:02 | |
| *** amcrn has quit IRC | 23:03 | |
| *** salv-orlando has joined #openstack-infra | 23:05 | |
| *** amcrn has joined #openstack-infra | 23:06 | |
| zaro | clarkb: i'm getting an error when attempting to clone from review-dev.o.o, error is "SSL certificate problem: Invalid certificate chain" | 23:06 |
| zaro | clarkb: only happens when clone with https, works ok when clone with ssh protocol | 23:07 |
| clarkb | zaro: yes, git checks SSL certs, but the one for review-dev is self signed. You need to tell git to ignore bad certs or add the review-dev cert to a trusted list | 23:07 |
| *** dkranz has quit IRC | 23:08 | |
| zaro | clarkb: how is review different in that i don't have to do that? | 23:08 |
| *** dizquierdo has quit IRC | 23:09 | |
| zaro | jeblair, clarkb, fungi: i converted tables on review-dev to utf8 and seems to be working ok, bug 979227 | 23:10 |
| uvirtbot | Launchpad bug 979227 in openstack-ci "convert gerrit's mysql tables to utf8" [Medium,Triaged] https://launchpad.net/bugs/979227 | 23:10 |
| zaro | was wondering whether that's something we should do before OR after gerrit upgrade? | 23:10 |
| *** enikanorov has joined #openstack-infra | 23:11 | |
| *** eharney has quit IRC | 23:11 | |
| clarkb | zaro: we did not pay money for the cert on review-dev | 23:12 |
| clarkb | it uses the self generated snakeoil certs | 23:12 |
| *** enikanorov__ has quit IRC | 23:12 | |
| fungi | zaro: there is an envvar you can pass with git to have it ignore cert verification issues | 23:13 |
| fungi | google knows it, i just keep losing it out one of the holes in my head | 23:13 |
| zaro | fungi: yeah, i think i saw that on google, but was wondering why review and review-dev was different. | 23:13 |
| fungi | zaro: right, we generally only get ca-signed certs for production systems. dev systems use self-signed snakeoil | 23:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Make queue processing more efficient https://review.openstack.org/71627 | 23:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Fix management queue test errors https://review.openstack.org/71626 | 23:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Move the merger to a JSON-compatible API https://review.openstack.org/70310 | 23:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Remove Zuul ref replication. https://review.openstack.org/70309 | 23:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Remove push refs to gerrit feature https://review.openstack.org/68723 | 23:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Split the merger into a separate process https://review.openstack.org/71628 | 23:14 |
| jeblair | okay. jeez. thanks git. | 23:15 |
| zaro | fungi: cool. any opinion on my question about that bug? | 23:15 |
| *** dcramer_ has quit IRC | 23:15 | |
| *** burt1 has quit IRC | 23:16 | |
| jeblair | zaro: let's make sure to ask mordred about that | 23:16 |
| fungi | zaro: oh, utf-8 upgrade | 23:16 |
| fungi | yeah, ask mysql people, but in general i would say before we upgrade (less to convert in case gerrit adds more data) but in general anything added as part of the upgrade is probably ascii anyway so it's still the same under iso-8601 or utf-8 | 23:17 |
| jeblair | lifeless: sorry if this is a dupe question; but would you say your eta for tripleo cloud is more like some time today or after today? | 23:18 |
| clarkb | jeblair: is the last reconfigured note at the bottom of the zuul status page empty because the html/js is up to date but the running zuul is not? | 23:20 |
| fungi | from what i gathered, apparently building an openstack (multi-node, non-devstack) cloud from scratch using master branches of everything is a great way to find new bugs ;) | 23:20 |
| clarkb | fungi: I think they are also running on some really flaky mellanox gear | 23:20 |
| lifeless | jeblair: have finally confirmed that trunk of everything works again | 23:20 |
| lifeless | jeblair: so am now doing a single meeting, then bringup | 23:20 |
| fungi | lifeless: if it's before i fall asleep, i can get a revert of the tripleo removal from nodepool.yaml pushed through and new images built/nodes launching | 23:22 |
| jeblair | fungi: has someone written that change yet? | 23:22 |
| fungi | jeblair: i'll just do that now | 23:23 |
| jeblair | clarkb: yes, that's the case, re zuul. | 23:23 |
| lifeless | fungi: thanks. | 23:24 |
| *** rcleere has quit IRC | 23:24 | |
| jeblair | fungi: oh, revert of nodepool. right. misread. carry on. | 23:24 |
| jeblair | what i had meant to ask was if someone had written the 'remove from zuul' change | 23:25 |
| fungi | jeblair: i'll write that too, in case we end up needing it | 23:25 |
| fungi | should be easy/quick | 23:25 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Revert "Temporarily remove tripleo-ci nodepool provider" https://review.openstack.org/71684 | 23:26 |
| jeblair | fungi: sounds good. let's go ahead and put that in; i'm sure tripleo doesn't want to run all those jobs anyway. | 23:26 |
| clarkb | jeblair: is it ok for me to approve changes like https://review.openstack.org/#/c/66173 now? or do you have a plan for avoiding merge conflicts in the zuul work? | 23:28 |
| *** wenlock has joined #openstack-infra | 23:29 | |
| *** julim has quit IRC | 23:29 | |
| jeblair | clarkb: i have no plans; aprv at will | 23:29 |
| *** dims has quit IRC | 23:30 | |
| mordred | jeblair: ola! | 23:31 |
| fungi | it's a mordred! | 23:31 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Temporarily stop running tripleo seed/undercloud https://review.openstack.org/71685 | 23:31 |
| mordred | zaro: what should you ask me about? | 23:31 |
| *** banix has quit IRC | 23:33 | |
| zaro | mordred: bug 979227, should we convert gerrit db tables to UTF8 before OR after upgrading gerrit? | 23:33 |
| uvirtbot | Launchpad bug 979227 in openstack-ci "convert gerrit's mysql tables to utf8" [Medium,Triaged] https://launchpad.net/bugs/979227 | 23:33 |
| mordred | hrm | 23:33 |
| openstackgerrit | A change was merged to openstack-infra/config: Temporarily stop running tripleo seed/undercloud https://review.openstack.org/71685 | 23:35 |
| mordred | the script works on 2.8 still? | 23:35 |
| *** prad_ has quit IRC | 23:35 | |
| mordred | I'd do it then | 23:35 |
| mordred | or, wait | 23:35 |
| * mordred sucks | 23:35 | |
| mordred | (was thinking about migrating data, not in place upgrade) | 23:35 |
| clarkb | jeblair: https://review.openstack.org/#/c/68788/ doc fix for zuul's rate limiting | 23:35 |
| mordred | still - if the script works on 2.8 and it looks good, I say do it post upgrade | 23:36 |
| *** A-G has quit IRC | 23:37 | |
| *** A-G has joined #openstack-infra | 23:37 | |
| zaro | i feel like i'm pretty much done with testing gerrit 2.8 upgrade on review-dev so i'm going to stop doing that. if anyone has concerns or questions go ahead and ask. | 23:37 |
| *** A-G has quit IRC | 23:38 | |
| clarkb | zaro: https://review.openstack.org/#/c/61321/7 were you going to redup the nodepool manifests? | 23:39 |
| clarkb | doesn't look like that happened | 23:39 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Don't encourage reverify https://review.openstack.org/70632 | 23:39 |
| clarkb | zaro: I think you want a nodepool.pp and a nodepool_dev.pp | 23:40 |
| clarkb | basically add a new manifest rather than having different ones | 23:40 |
| *** esker has quit IRC | 23:40 | |
| *** sarob has joined #openstack-infra | 23:40 | |
| *** denis_makogon has quit IRC | 23:41 | |
| jeblair | mikal: what's the expected outcome of https://review.openstack.org/70632 ? | 23:41 |
| jeblair | sdague: ^ ? | 23:41 |
| zaro | clarkb: that wasn't mentioned in the review, but ok. i'll check that out. | 23:42 |
| clarkb | zaro: it was on the previous patchset | 23:42 |
| *** lcheng_ has quit IRC | 23:42 | |
| sdague | jeblair: just to stop educating people to do blind reverifies | 23:42 |
| clarkb | jeblair: my interpretation. There were classified bugs that resulted in comments to reverify. Problem is that several of those bugs failed 100% of the time | 23:42 |
| sdague | which they do, way too much | 23:42 |
| jeblair | sdague: but an e-r prompt isn't blind, right? | 23:42 |
| clarkb | so people would just reverify the bug and the gate would explode | 23:43 |
| clarkb | jeblair: correct, but it may classify bugs that are known to just cause trouble | 23:43 |
| jeblair | clarkb: oh, i see | 23:43 |
| sdague | jeblair: so because a run failed and hit a race, doesn't mean it *only* failed because of that race | 23:43 |
| sdague | it may have also failed for other reasons | 23:43 |
| clarkb | jeblair: in particular the neutron nbd stuff was a near 100% fail | 23:43 |
| clarkb | we classified it but neutron devs needed to stop reverifying until the underlying issue was fixed | 23:44 |
| sdague | honestly, I still think we should completely remove reverify | 23:44 |
| jeblair | sdague, clarkb: okay; my main concern is that a possble outcome of this is that we provide worse service to developers who don't know they can reverify when it's really okay and create more work for core devs | 23:44 |
| jeblair | and that defititely will create more work for core devs :) | 23:44 |
| sdague | jeblair: it will | 23:45 |
| clarkb | jog0: https://review.openstack.org/#/c/64200/ is that still a thing? | 23:45 |
| sdague | however, my experience watching this for the last month | 23:45 |
| *** sarob has quit IRC | 23:45 | |
| sdague | we get tons of reverify on patches that can't pass | 23:45 |
| sdague | by devs | 23:45 |
| sdague | even by core devs | 23:45 |
| jog0 | clarkb: its a WIP in progress still | 23:45 |
| sdague | so clearly we've failed in educating people, don't do that | 23:45 |
| jog0 | no need to review it now | 23:46 |
| clarkb | jog0: can you WIP it? | 23:46 |
| jeblair | sdague: the require-check change will mitigate that, right? | 23:46 |
| jeblair | sdague: | 23:46 |
| jog0 | clarkb: done | 23:46 |
| *** dims has joined #openstack-infra | 23:46 | |
| zaro | mordred: fungi says we should do before upgrade. | 23:46 |
| sdague | jeblair: maybe | 23:46 |
| jeblair | sdague: (if it can't pass at all, it won't pass check) | 23:46 |
| mordred | zaro: I'm fine either way - I do not think it matters | 23:46 |
| jog0 | clarkb sdague: but https://review.openstack.org/#/q/status:open++branch:master+topic:graph-gate,n,z is still a think | 23:46 |
| mordred | fungi: why do you think it matters? | 23:46 |
| sdague | the hyperv guys just run reverify on any patch in the gate when it gets kicked out | 23:46 |
| jog0 | thing | 23:46 |
| sdague | they were doing it on merge conflict patches | 23:47 |
| fungi | zaro: i defer to mordred on database things. i hardly know how to spell db | 23:47 |
| *** changbl has quit IRC | 23:47 | |
| sdague | so honestly, I think there are too many bad actors at this point to police | 23:47 |
| sdague | and the net effect is worse experience for everyone | 23:47 |
| mordred | fungi: I'm deferring to you | 23:47 |
| fungi | mordred: i don't think it matters, which is why i said might as well do it first. might as well also do it last | 23:47 |
| mordred | gotcha | 23:47 |
| mordred | I COMPLETELY agree | 23:47 |
| mordred | might as well do it first. might as well also do it last | 23:47 |
| zaro | ok just need to get on the schedule then. | 23:48 |
| sdague | btw, it's nice that incheon airport has free wifi :) | 23:48 |
| fungi | i have doubts that any new transforms or data added by the gerrit upgrade will include any data which is non-ascii, so the utf-8 conversion should be basically the same regardless of whether it's before or after the upgrade | 23:48 |
| jeblair | mordred, zaro: has that been tested with a copy of our production db? | 23:49 |
| *** yamahata has quit IRC | 23:49 | |
| zaro | jeblair: no, are you suggesting we copy data from prod to review-dev and then convert tables? | 23:49 |
| *** reed has quit IRC | 23:50 | |
| zaro | jeblair: there is a revert option if there are errors. | 23:50 |
| jeblair | mordred: didn't darragh identify problems with the utf8 upgrade? did those get resolved? | 23:51 |
| mikal | jeblair: the outcome of that review is just that we don't actively encourage reverify any more | 23:51 |
| *** changbl has joined #openstack-infra | 23:51 | |
| jeblair | mikal: yeah, i chatted with sdague about it. i'm concerned about increasing core reviewer workload, but not enough to object to it. | 23:51 |
| fungi | mikal: the reason for encouraging (well, mentioning) reverify syntax in the comment was because otherwise contributors wander into irc channels or onto mailing lists asking how to get their change resubmitted to the gate. if there's something different we should be telling them, we could in theory do that in the comment too | 23:52 |
| sdague | jeblair: honestly, I expect it's offset by less unpassable code in the gate, which means less time core folks are babysitting patches | 23:52 |
| jeblair | zaro: i think the upgrade should at least be run on the real data. testing it with a gerrit is optional. | 23:53 |
| jeblair | sdague: sure, again though, the 'unpassable' code problem should be fixed by requiring check jobs to pass. that leaves actual passable code, which means _someone_ has to re-enqueue it. this path heads toward core reviewers doing that work. | 23:54 |
| mordred | jeblair: I believe darragh's concerns were that he wasn't 100% sure that all comment data went right | 23:54 |
| fungi | jeblair: zaro: i can copy a mysqldump from review.o.o somewhere (to review-dev even) so it could be tried out | 23:54 |
| fungi | we have nightlies anyway, so it's just a quick scp away | 23:54 |
| mordred | but yeah - I agree about running it on copy of prod | 23:54 |
| mikal | jeblair: I think you have a point, but core reviewers are also the ones best positioned to identify if a reverify is the right answer | 23:55 |
| zaro | fungi: wfm | 23:55 |
| mikal | I imagine that new devs are going to come and ask a core anyways | 23:55 |
| jeblair | fungi, zaro: we should be careful that we don't have a gerrit start spamming people or anything. | 23:55 |
| fungi | zaro: the bigger issue is knowing what to look for in the db as a sign that the conversion was bad | 23:55 |
| *** paul-- has joined #openstack-infra | 23:55 | |
| *** thuc_ has quit IRC | 23:55 | |
| fungi | jeblair: right, i didn't mean to load into a running gerrit | 23:55 |
| *** thuc has joined #openstack-infra | 23:56 | |
| fungi | could shut down gerrit service, import the prod dump, run the conversion, look it over, load the -dev backup and then start gerrit again | 23:56 |
| jeblair | mikal, sdague: k. i'm willing to go along with it. just consider me cautious and concerned. :) | 23:56 |
| jeblair | fungi: or just load it into an alternate db | 23:56 |
| *** thuc has quit IRC | 23:56 | |
| fungi | jeblair: zaro: or simply load the dump into a local mysql server | 23:56 |
| fungi | jeblair: or that | 23:56 |
| *** thuc has joined #openstack-infra | 23:57 | |
| zaro | i'll check with darragh to see if he knows what to look for. otherwise i'm clueless. | 23:57 |
| sdague | jeblair: if I hadn't witnessed so much abuse, I'd be with you. But I ended up pulling 100 patches out of the gate manually during the week of death. | 23:57 |
| sdague | :) | 23:57 |
| jeblair | sdague: look at the check sparkline | 23:57 |
| sdague | heh | 23:58 |
| fungi | zaro: mostly it'll be finding places in gerrit where there is arbitrary text (comments, display names, et cetera) which contain non-ascii characters and make sure they end up with the utf-8 equivalents. particularly for codepoints where iso-8601 and utf-8 don't overlap | 23:58 |
| sdague | it would be nice if the sparklines actually started at y=0 | 23:58 |
| clarkb | fungi: zaro: you should be able to identify bad strings with a simple python script | 23:59 |
| sdague | otherwise they are a little hard to tell what's going on, except in a case like this | 23:59 |
| *** che-arne has joined #openstack-infra | 23:59 | |
| *** bknudson has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!