| jeblair | fungi: that risk is not automatically eliminated with afs, but we at least have some options to deal with it | 00:00 |
|---|---|---|
| *** weshay_xchat has quit IRC | 00:00 | |
| *** yamamoto_ has joined #openstack-infra | 00:00 | |
| *** dingyichen has joined #openstack-infra | 00:01 | |
| clarkb | if we wanted resiliency ceph(fs) might be an option but then we are basically reimplementing an object store on top of cinder in acloud with an object store already whichseems dirty | 00:02 |
| *** ybathia has joined #openstack-infra | 00:02 | |
| clarkb | so I agree with jhesketh agreeing on some fundamental requirements/needs would probably help guide us | 00:02 |
| *** ddieterl_ has quit IRC | 00:03 | |
| *** pahuang has joined #openstack-infra | 00:03 | |
| *** dingyichen has quit IRC | 00:03 | |
| clarkb | 8TB of storage for ~6 months of log retention? have to be able to host indexes to the logs so that periodic jobs and friends are browseable, untrusted job workers need some safe way of writing to their specific location, expect that every file will be read at least one (logstash), | 00:03 |
| clarkb | that is rough stuff off the top of my head | 00:04 |
| jeblair | clarkb: i don't think the untrusted workers thing is necessarily true anymore... | 00:04 |
| *** kaisers has quit IRC | 00:05 | |
| jeblair | clarkb: it's what pushed us in the direction of swift for where we thought we were going to be with zuulv2 post jenkins... but i think the situation is different for zuulv3 | 00:05 |
| *** thorst has joined #openstack-infra | 00:05 | |
| jeblair | clarkb: i wrote a bit about that in the afs log spec | 00:05 |
| clarkb | jeblair: because anisble can do the same sort of file indirection as jenkins? | 00:05 |
| jeblair | clarkb: yeah basicall | 00:05 |
| jeblair | y | 00:05 |
| *** rbrndt has quit IRC | 00:06 | |
| jeblair | clarkb: (i mean, i have an answer for how to do that with afs, but it's somewhat complicated -- enough so that it wasn't clearly simpler than swift at the time) | 00:07 |
| jeblair | (so i'll be happy if we don't need it) | 00:07 |
| clarkb | gotcha | 00:07 |
| clarkb | logs should be relatively easily expired | 00:07 |
| jhesketh | so the browsable/index requirement, does that extend to "must look like apache dir listing"? because maybe we just need a better way to browse logs (in particular, periodic ones) | 00:08 |
| clarkb | jhesketh: I think they do not need to be just like apache but they do need to be sortable bsaed on mod time | 00:08 |
| *** sdake_ has joined #openstack-infra | 00:08 | |
| jeblair | (right, bonus points if we make it actually look better, but we can do that with apache too :) | 00:09 |
| *** doug-fish has joined #openstack-infra | 00:09 | |
| *** bryan_att has quit IRC | 00:09 | |
| clarkb | jhesketh: to allow for people going "I need to look at results from one week ago" or "I need to look at a particular day" | 00:09 |
| *** dingyichen has joined #openstack-infra | 00:10 | |
| *** sdake has quit IRC | 00:11 | |
| jhesketh | I guess the extension of my question is do we not change anything from the current swift storage and just make publishing log listings better... eg a database the zuul inserts into that we create a frontend for browsing | 00:11 |
| *** Swami has quit IRC | 00:11 | |
| fungi | which is also potentially solvable instead with a zuul mysql reporter? if the data reported into the db included the timestamps and base index urls | 00:11 |
| *** Swami has joined #openstack-infra | 00:11 | |
| fungi | heh, i think we were just typing the same thing | 00:11 |
| *** MarkAtwood1 has quit IRC | 00:14 | |
| *** doug-fish has quit IRC | 00:14 | |
| *** markvoelker has quit IRC | 00:14 | |
| *** rkukura has quit IRC | 00:14 | |
| *** dchen has joined #openstack-infra | 00:15 | |
| *** piet has quit IRC | 00:15 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: AFS servers use server class instead of template https://review.openstack.org/269930 | 00:15 |
| *** piet has joined #openstack-infra | 00:15 | |
| fungi | jeblair: mordred: clarkb: ^ should in theory fix up the sources.lists on those stragglers | 00:15 |
| krotscheck | jeblair: Not yet. I was planning on sitting on mordred's head, but if you're volunteering ;) | 00:15 |
| jeblair | krotscheck: it sounds like some combination of fungi and i are volunteering | 00:16 |
| krotscheck | Oh good | 00:16 |
| * krotscheck reads backscroll | 00:16 | |
| fungi | those entries were the only direct callers of openstack_project::template in the global site manifest | 00:16 |
| jeblair | possibly the top part of fungi and the bottom part of me | 00:16 |
| clarkb | jhesketh: fungi I do worry about tightly coupling the job runner to log storage, though I don't believe that is a necessary requirement | 00:16 |
| fungi | one of those mix-and-match outfit books | 00:16 |
| *** moravec has joined #openstack-infra | 00:16 | |
| krotscheck | jeblair, fungi: I was going to get working on converting-mirrors-to-AFS next, but if y'all want to do that.... | 00:16 |
| clarkb | it just makes working with it harder | 00:17 |
| clarkb | if zuul is down you can't get logs sort of thing | 00:17 |
| *** markvoelker has joined #openstack-infra | 00:17 | |
| jhesketh | clarkb: only if zuul's db is down which could be done with galera or something with a bit more stability | 00:18 |
| fungi | maybe zuul could buffer and flush durable index updates into afs | 00:18 |
| jhesketh | but that's a design point in my opinion | 00:18 |
| fungi | or, yeah, the database not running on zuul.o.o itself | 00:18 |
| fungi | which is how we'd likely deploy anyway | 00:18 |
| jhesketh | we're still talking about a decent amount of extra work | 00:18 |
| jhesketh | for what return | 00:18 |
| fungi | report into a trove backend and have read-only creds for the logserver | 00:19 |
| anteaya | fungi: I could read all the scrollback and figure out if I can do anything to help or I could just ask, can I do anything to help? | 00:19 |
| fungi | anteaya: probably not for the moment? | 00:19 |
| fungi | mostly hacking on getting kernel updates installed on our servers | 00:19 |
| anteaya | very good | 00:19 |
| * anteaya goes back to scrollback | 00:19 | |
| *** shashank_hegde has quit IRC | 00:19 | |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: add the ability to get an object back from swift https://review.openstack.org/269925 | 00:20 |
| jeblair | yeah, i keep coming back to the idea that we get everything we need feature-wise by dumping it into a filesystem (whether that's on disk, afs, ceph, whatever), and we can always build other stuff on top of that. | 00:20 |
| krotscheck | fungi, jeblair: When in the "Between 4AM and Noon PST" Timeframe this week would work for you? | 00:21 |
| *** kaisers has joined #openstack-infra | 00:21 | |
| *** shashank_hegde has joined #openstack-infra | 00:21 | |
| *** rkukura has joined #openstack-infra | 00:21 | |
| anteaya | internap nodes are unhappy, ovh nodes are unhappy | 00:21 |
| jhesketh | so it seems the best reason to not continue storing on disk is that it's fragile and likely to fail at some point | 00:21 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 00:21 |
| *** baoli has joined #openstack-infra | 00:23 | |
| jeblair | fail again even :) | 00:23 |
| fungi | jhesketh: well, and there is still the scalability concern. today we have no easy answer to what happens when our storage needs double. we're attaching the maximum number of cinder volumes of the maximum possible size to the server already | 00:23 |
| jhesketh | so are the alternative filesystems (afs, ceph etc) going to be stable enough, and is the stability worth the effort in setting it up? | 00:23 |
| jhesketh | fungi: hmm, didn't know that | 00:23 |
| jeblair | jhesketh: well, we've already set up afs | 00:23 |
| jeblair | so that's basically a sunk cost | 00:24 |
| jhesketh | sure, just thinking through the options. For example, if we have playbooks to help recover from various disk failures how bad is it if we lose a day of logs? Jobs can just be re-run for those who need them. | 00:24 |
| *** sdake_ has quit IRC | 00:24 | |
| fungi | jhesketh: maximum cinder device rackspace allows us to provision is 1tb and due to xen limitations we can't attach more than 14 (supports up to 16 vbd nodes, two of which are taken up by rootfs and ephemeral) | 00:24 |
| jhesketh | (not that I Think that's a good idea) | 00:24 |
| jeblair | if we want _more_ stability from afs than our current system, we may need to spend a little more effort into making it more robust, but that's also something we can fairly easily evolve over time | 00:25 |
| fungi | so basically with no redundancy we top out around 14tb capacity | 00:25 |
| jhesketh | jeblair: it sounds like there is still some effort in tuning the afs and testing, although compared to the other options that doesn't seem like too much work | 00:25 |
| *** rkukura has quit IRC | 00:25 | |
| jeblair | (volume migrations make that easy) | 00:25 |
| *** sdake has joined #openstack-infra | 00:25 | |
| *** moravec has quit IRC | 00:27 | |
| *** prometheanfire has quit IRC | 00:28 | |
| jeblair | krotscheck: i get online around 8am; i can try to reserve mornings for this -- i did want to update my bandersnatch patch first, so maybe we should plan for thurs/friday for the serious push. | 00:28 |
| *** sflanigan has joined #openstack-infra | 00:29 | |
| *** sflanigan has joined #openstack-infra | 00:29 | |
| *** sridhar_ram has joined #openstack-infra | 00:29 | |
| *** piet has quit IRC | 00:30 | |
| jeblair | so maybe i can spend tomorrow getting bandersnatch writing into afs | 00:30 |
| *** [1]Thelo has joined #openstack-infra | 00:31 | |
| *** baoli has quit IRC | 00:31 | |
| clarkb | fyi rebooting nodepool now to apply new nodepool changes and patch kernel | 00:32 |
| jeblair | jhesketh, clarkb, fungi, mordred: so maybe we should pause the swift logs stuff for now and take a bit of time to consider it, during which we can see if the pypi in afs effort falls flat or what else we might learn from it? | 00:32 |
| *** Thelo has quit IRC | 00:33 | |
| *** [1]Thelo is now known as Thelo | 00:33 | |
| openstackgerrit | Merged openstack-infra/system-config: AFS servers use server class instead of template https://review.openstack.org/269930 | 00:34 |
| krotscheck | jeblair: That works for me, that leaves me tomorrow open to contribute to bower and get them more distro-package-policy savvy | 00:35 |
| *** yamamoto_ has quit IRC | 00:35 | |
| anteaya | I'm thinking of writing a story where the narative begins at the punchline and the story is revealed in reverse order one page at a time | 00:37 |
| *** Swami has quit IRC | 00:37 | |
| nibalizer | anteaya: so its a puppet manifest? | 00:38 |
| anteaya | ha ha ha | 00:38 |
| anteaya | it could be written in ruby with lots of rockets, sure | 00:38 |
| jhesketh | jeblair: yeah I'm okay with it being on the back burner, but I'd love for the spec to not drag on. I thought this is why we revisited it during the summit. | 00:39 |
| openstackgerrit | Christopher Aedo proposed openstack-infra/project-config: Update App Catalog dead link checker https://review.openstack.org/269935 | 00:40 |
| ianw | clarkb: when it's settled, can you ensure a build is started so we can see if the logs are making it out? | 00:41 |
| clarkb | ianw: ya I will trigger a build shortly | 00:41 |
| clarkb | oh except I think it will write to my console | 00:42 |
| jeblair | jhesketh: yeah, i'm with you. there are two things that make me think we should pause, for at least a couple days: 1) changing direction probably shouldn't be a snap decision if we can take a moment to consider. 2) given a week, i think we can put something production into afs which will help us better evaluate that alternative. i don't think we need to drag it out. | 00:43 |
| *** baoli has joined #openstack-infra | 00:44 | |
| *** ZZelle_ has quit IRC | 00:44 | |
| *** Sukhdev has quit IRC | 00:44 | |
| anteaya | jhesketh: I do acknowledge and am grateful for all the hard work you have put into swift logs to get us this far | 00:44 |
| anteaya | jhesketh: thank you | 00:45 |
| anteaya | you are awesome | 00:45 |
| jeblair | everything is awesome, but especially jhesketh | 00:45 |
| anteaya | ++ | 00:45 |
| *** Qiming has quit IRC | 00:45 | |
| jhesketh | lol | 00:46 |
| jhesketh | thankyou :-) | 00:46 |
| jhesketh | jeblair: I'm fine with that way forward | 00:46 |
| openstackgerrit | Jay_Clark proposed openstack-infra/irc-meetings: adding the irc meeting slot for the openstack-salt team. https://review.openstack.org/269936 | 00:46 |
| anteaya | did we restart nodepool recently? | 00:48 |
| ianw | anteaya: clarkb just rebooted it | 00:49 |
| anteaya | ah thank you | 00:49 |
| anteaya | there it is in scrollback, sorry | 00:50 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Use devstack subunit stream in output https://review.openstack.org/234447 | 00:50 |
| *** piet has joined #openstack-infra | 00:51 | |
| anteaya | clarkb: I don't know if you saw but the patch to devstack master to remove the cpu flags failed in check: https://review.openstack.org/#/c/269739/ | 00:52 |
| *** eil397 has quit IRC | 00:52 | |
| anteaya | clarkb: for after the world has been patched | 00:52 |
| *** Thelo_ has quit IRC | 00:53 | |
| *** baoli has quit IRC | 00:53 | |
| *** bpokorny has joined #openstack-infra | 00:55 | |
| *** MarkAtwood1 has joined #openstack-infra | 00:55 | |
| clarkb | ok geaman is still not getting jobs registered for the image deletes | 00:55 |
| clarkb | greghaynes: tells me likely because the builder relies on what is on disk to know what to register jobs for | 00:56 |
| clarkb | so | 00:56 |
| greghaynes | ya | 00:56 |
| clarkb | I can do manual cleanup of the db | 00:56 |
| *** jcoufal has joined #openstack-infra | 00:57 | |
| greghaynes | builders only register delete/upload jobs for stuff that exists on disk, if a build fails and the file is not on disk then no -delete is registered | 00:57 |
| greghaynes | so when nodepool is out of sync and wants to delete an image that lacks a job it just hnnnng | 00:57 |
| *** kzaitsev_mb has quit IRC | 00:58 | |
| *** MarkAtwood1 has quit IRC | 00:58 | |
| clarkb | for now I htink the best option is to manually remove those rows from the db | 00:58 |
| clarkb | then work on making the deletes more resilient to this failure | 00:59 |
| clarkb | under typical operation this should't happen | 00:59 |
| *** MarkAtwood has quit IRC | 00:59 | |
| *** abitha has quit IRC | 00:59 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:00 | |
| anteaya | sandbox has the +1, 0, -1 that it used to have for registered users again | 01:00 |
| anteaya | https://review.openstack.org/#/c/269387/ | 01:00 |
| *** yonglihe has quit IRC | 01:01 | |
| *** ybathia has quit IRC | 01:02 | |
| *** sputnik13 has quit IRC | 01:04 | |
| *** unicell1 has joined #openstack-infra | 01:05 | |
| *** baoli has joined #openstack-infra | 01:05 | |
| *** kzaitsev_mb has quit IRC | 01:05 | |
| *** aeng has joined #openstack-infra | 01:07 | |
| *** unicell has quit IRC | 01:07 | |
| clarkb | so I think I need to do a 'DELETE FROM dib_image WHERE state = 1; | 01:07 |
| clarkb | jeblair: ^ that look right ot you? | 01:07 |
| *** vivekd has joined #openstack-infra | 01:09 | |
| *** salv-orlando has quit IRC | 01:12 | |
| clarkb | ok confirmed they are not on disk so I am going to go ahead and run that sql query | 01:13 |
| jeblair | clarkb: yep. | 01:14 |
| *** baoli has quit IRC | 01:15 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add optional run_metadata filter to get_test_runs_by_test_test_id https://review.openstack.org/269950 | 01:15 |
| teran | Hi folks | 01:16 |
| *** lewo has quit IRC | 01:17 | |
| clarkb | teran: hello | 01:17 |
| mtreinish | lifeless: have you seen this error before? http://logs.openstack.org/47/234447/5/check/gate-tempest-dsvm-full/f57080f/logs/devstacklog.txt.gz#_2016-01-20_01_10_44_137 | 01:17 |
| *** Sukhdev has joined #openstack-infra | 01:17 | |
| clarkb | jeblair: ianw ok I have run `nodepool image-build ubuntu-trusty` as a first attempt (but with the sudo things which may not be necesary any longer with builders | 01:17 |
| teran | clarkb: Do you probably know is there any chance to get more logs(like dmesg/kernel.log) from nodes running gate tests? | 01:17 |
| teran | like this one http://logs.openstack.org/64/269564/1/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/ | 01:17 |
| lifeless | mtreinish: either bad packaging, or something ran 'python setup.py install' the six project | 01:18 |
| clarkb | jeblair: ianw and the logs are being written to dib.ubuntu-trusty.log | 01:18 |
| clarkb | jeblair: ianw so progress | 01:18 |
| dstufft | FYI I just cut pip 8.0 | 01:18 |
| clarkb | teran: we should record those logs already | 01:18 |
| dstufft | so if things go boom, I'll be hiding over here | 01:18 |
| teran | clarkb: oh, where to look for them? | 01:18 |
| mtreinish | dstufft: oh, that could be what broke in that link I just pasted | 01:18 |
| mtreinish | because everything was working a few minutes ago | 01:18 |
| clarkb | teran: oh doesn't look like fuel puppet runs do | 01:19 |
| mtreinish | lifeless: ^^^ | 01:19 |
| lifeless | mtreinish: I don't *think* its a new thing in pip 8 | 01:19 |
| *** kaisers1 has joined #openstack-infra | 01:19 | |
| lifeless | mtreinish: but it could be | 01:19 |
| clarkb | teran: we do for mots of the openstack integration tests, you will have to add it to that job if you wan tthem | 01:19 |
| dstufft | I don't think that's new or an actual error | 01:19 |
| *** Sukhdev has quit IRC | 01:19 | |
| lifeless | mtreinish: certainly the limitation on uninstalling things without that metadata isn't new | 01:19 |
| *** Sukhdev has joined #openstack-infra | 01:19 | |
| lifeless | mtreinish: isn't | 01:19 |
| teran | clarkb: hm... ok, we'll try to get them. Thanks a lot! | 01:20 |
| lifeless | 2016-01-20 01:10:44.467 | ./stack.sh: line 486: generate-subunit: command not found | 01:20 |
| lifeless | the error ? | 01:20 |
| lifeless | hmm, no | 01:20 |
| lifeless | dstufft: it does look like 8 bailed on that | 01:20 |
| *** kaisers has quit IRC | 01:21 | |
| * dstufft finds commit | 01:21 | |
| lifeless | dstufft: elif distutils_egg_info: | 01:21 |
| lifeless | raise UninstallationError( | 01:21 |
| dstufft | oh right | 01:21 |
| mtreinish | lifeless: http://logs.openstack.org/32/234232/9/check/gate-tempest-dsvm-full/a54f412//logs/devstacklog.txt.gz#_2016-01-19_22_55_33_451 | 01:22 |
| lifeless | dstufft: thats from 2014 though | 01:22 |
| dstufft | it was in my general "we're doing a pip 8 so remove all the deprecated things" | 01:22 |
| mtreinish | it looks like it was just a deprecation warning before and now its fatal | 01:22 |
| dstufft | https://github.com/pypa/pip/commit/6afc718307fea36b9ffddd376c1395ee1061795c | 01:22 |
| *** Sukhdev_ has joined #openstack-infra | 01:22 | |
| ianw | clarkb: yay! | 01:22 |
| *** Sukhdev has quit IRC | 01:22 | |
| lifeless | dstufft: oh no, I misread blame | 01:23 |
| lifeless | dstufft: so yeah | 01:23 |
| lifeless | dstufft: this is going to break everyone using trusty that has six installed from the distro ;) | 01:23 |
| *** lewo has joined #openstack-infra | 01:23 | |
| dstufft | basically we stopped lying and pretending we were uninstalling something that we weren't actually uninstalling | 01:23 |
| lifeless | dstufft: sure, I get the logic | 01:23 |
| *** yamamoto_ has joined #openstack-infra | 01:24 | |
| mtreinish | yeah, it's gonna break every dsvm run: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22DEPRECATION%3A%20Uninstalling%20a%20distutils%20installed%20project%5C%22 | 01:24 |
| lifeless | dstufft: the concern I have is that the deprecation cycle is faster than the reaction-to-deprecations for distro packages | 01:24 |
| mtreinish | has 200177 hits in the last 7 days | 01:24 |
| *** ccrouch has quit IRC | 01:24 | |
| *** zz_dimtruck is now known as dimtruck | 01:24 | |
| lifeless | mtreinish: can you see if its always associated with six? | 01:24 |
| lifeless | mtreinish: or if there are other packages | 01:25 |
| mtreinish | it's more than just six | 01:25 |
| teran | clarkb: one more question, is there a chance to get contrack limits and yes/no about it's reaching from hardware nodes instances for those tests are running? | 01:25 |
| lifeless | mtreinish: if its just a few we might be able to uninstall them like we uninstall python-pip, python-pip3 etc | 01:25 |
| *** Sukhdev_ has quit IRC | 01:25 | |
| mtreinish | theres urllib3, jsonpointer, chardet, libvirt-python, pyOpenSSL, PyYAML, netaddr | 01:25 |
| mtreinish | and that's just where I got until laziness kicked in | 01:26 |
| mtreinish | although, that might be the complete list | 01:26 |
| lifeless | mtreinish: ok, lets cap the install of pip to 7.1 | 01:26 |
| *** mtanino has quit IRC | 01:26 | |
| lifeless | mtreinish: get-pip.py is a pip itself | 01:26 |
| lifeless | mtreinish: so doing get-pip.py install pip<8 should work IIRC | 01:26 |
| dstufft | no | 01:27 |
| dstufft | it won't, because double requirement | 01:27 |
| lifeless | dstufft: -c ? | 01:28 |
| dstufft | -c should work yea | 01:28 |
| mtreinish | lifeless, dstufft: so 'get-pip.py install -c pip<8' ? | 01:29 |
| lifeless | python ./get-pip.py -c <(echo "pip<8") | 01:29 |
| *** akshai has quit IRC | 01:29 | |
| lifeless | mtreinish: -c takes a file as parameter | 01:29 |
| dstufft | do we support stdin on that? | 01:29 |
| *** zhurong has joined #openstack-infra | 01:29 | |
| dstufft | that's neat | 01:29 |
| lifeless | mtreinish: so probably a little clearer to write a file with pip<8 in it, rather than my shell hackery | 01:29 |
| lifeless | mtreinish: since we need to tunnel through sudo | 01:29 |
| lifeless | dstufft: no, thats a shell thing, uses /proc/fd | 01:30 |
| *** harlowja has quit IRC | 01:30 | |
| dstufft | oh | 01:30 |
| lifeless | <(expression) -> run expression with its output a pipe, pass the pipe as an open fd via fork, pass /proc/fd/N in the cmdline | 01:30 |
| *** harlowja has joined #openstack-infra | 01:30 | |
| *** david-lyle has quit IRC | 01:31 | |
| *** Jeffrey4l has joined #openstack-infra | 01:31 | |
| mtreinish | lifeless, dstufft: https://review.openstack.org/#/c/269954/ | 01:31 |
| mtreinish | oh, I can change it to a file | 01:31 |
| lifeless | mtreinish: yah, will need to | 01:32 |
| lifeless | mtreinish: because sudo opens another subprocess and closes extraneous fd's | 01:32 |
| dstufft | lifeless: I didn't mean to ignore the thing earlier, feeling crummy and I got distracted looking to see how long ago we deprecated it-- We may roll that back. We deprecated it almost exactly a year ago (11 days shy), though I'm not sure that a longer period of time is going to actually cause any movement on it, the tl;dr is that some projects use distutils in their setup.py and if they do distros won't drop the right metadata, fix is | 01:33 |
| dstufft | either get distros to switch to setuptools, projects, or both. Just continuing to lie about uninstalling doesn't feel like a great solution, but we'll see how many people start yelling at me | 01:33 |
| lifeless | dstufft: so we may be unable to avoid the lying on distros | 01:33 |
| mtreinish | lifeless: ok: https://review.openstack.org/#/c/269954/ uses a seperate file | 01:35 |
| clarkb | teran: I don't understand the question | 01:35 |
| mtreinish | I'll fast approve it if it passes the gate | 01:35 |
| lifeless | dstufft: because the distro package manager will have a hernia if we uninstall things high up the dep list | 01:35 |
| mtreinish | I probably should have opened a bug for it | 01:35 |
| lifeless | dstufft: (and things with CLI's other programs need, for instance) | 01:35 |
| dstufft | I don't feel like logging into review.o.o atm but that looks legit to me | 01:36 |
| dstufft | lifeless: virtual environments! | 01:36 |
| lifeless | dstufft: so -> #pypa-dev perhaps ? | 01:37 |
| teran | clarkb: so, we have a connectivity issue with these tests and one of the most obvious guesses why could it happen that's because of conntrack limit in hardware node | 01:37 |
| dstufft | pip install --ignored-installed six will (I think) install six as pip<8 did too fwiw | 01:37 |
| dstufft | lifeless: sure | 01:37 |
| *** dimtruck is now known as zz_dimtruck | 01:37 | |
| teran | clarkb: so someone who has access to hardware node could just check it's dmesg for conntrack limit reaching messages that would be very useful | 01:38 |
| teran | clarkb: could you please do that or who should I ask about that? | 01:39 |
| clarkb | teran: we don't have access to that you would have to contact the individual clouds | 01:39 |
| teran | clarkb: ok, got it. Thank you! | 01:39 |
| fungi | at least until infra-cloud goes live | 01:40 |
| openstackgerrit | Jay_Clark proposed openstack-infra/irc-meetings: adding the irc meeting slot for the openstack-salt team. https://review.openstack.org/269936 | 01:42 |
| *** Qiming has joined #openstack-infra | 01:42 | |
| *** ddieterly has joined #openstack-infra | 01:44 | |
| dims | lifeless : mtreinish : https://review.openstack.org/#/c/269954/ is the plan of action? :) | 01:44 |
| mtreinish | dims: assuming it passes check yes | 01:45 |
| mtreinish | well in the short term at least | 01:45 |
| dims | mtreinish : guess we need request to promote it once it passes check? | 01:45 |
| mtreinish | fungi: btw, I set the subunit2sql db prune cron job to run on Sundays right? Was there anything in the log file, because it doesn't look like anything trimmed | 01:45 |
| mtreinish | dims: yep, that was my plan :) | 01:45 |
| dims | mtreinish : great thanks | 01:46 |
| *** otter768 has joined #openstack-infra | 01:46 | |
| *** hemna has quit IRC | 01:47 | |
| *** ksantoshk has quit IRC | 01:48 | |
| *** pvaneck has quit IRC | 01:48 | |
| *** Daisy has joined #openstack-infra | 01:49 | |
| *** Sukhdev has joined #openstack-infra | 01:49 | |
| *** Sukhdev has quit IRC | 01:50 | |
| lifeless | dims: yes | 01:50 |
| ianw | clarkb: cool, finally that git clone line looks right and the old branches deleted (01:21:00,296 in http://nodepool.openstack.org/dib.ubuntu-trusty.log) | 01:50 |
| *** armax has quit IRC | 01:51 | |
| jeblair | lol: the atlassian instructions for "how to resolve a merge conflict" describe git, not mercurial. | 01:52 |
| *** keedya has quit IRC | 01:52 | |
| lifeless | can someone confirm for me - pip in unittest images is done at image build time, not test run time ? | 01:53 |
| jeblair | lifeless: hrm, i think we may actually run get-pip at runtime... | 01:54 |
| *** armax has joined #openstack-infra | 01:54 | |
| jeblair | lifeless: i think we cache _get-pip.py itself_ at image-build, but run it at job-execution... | 01:55 |
| jeblair | i still want to double check myself on that though | 01:55 |
| lifeless | jeblair: ok; thinking through, I actually care about virtualenv, not pip | 01:55 |
| lifeless | jeblair: since the virtualenv package determines the pip we end up with in a virtualenv made by tox | 01:56 |
| lifeless | jeblair: and virtualenvs with system site packages enabled may run into this issue too | 01:56 |
| jeblair | lifeless: i think we have exised all site-packages-enabled venvs | 01:56 |
| *** bpokorny_ has joined #openstack-infra | 01:56 | |
| lifeless | oh that would be loverly | 01:56 |
| jeblair | lifeless: i think virtualenv is image build time | 01:57 |
| greghaynes | I dont see anything that runs get-pip.py in our elements | 01:58 |
| jeblair | lifeless: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/template.pp#n231 | 01:58 |
| greghaynes | so that jives with what jeblair thins | 01:58 |
| mtreinish | greghaynes: I think something is installing a newer pip in the images: http://logs.openstack.org/47/234447/5/check/gate-tempest-dsvm-full/f57080f/console.html#_2016-01-20_00_56_51_068 | 01:58 |
| greghaynes | thinks | 01:58 |
| lifeless | mtreinish: that dvsm | 01:58 |
| fungi | http://codesearch.openstack.org/?q=get-pip.py&i=nope&files=&repos= | 01:58 |
| lifeless | mtreinish: we're now talking unit tests | 01:59 |
| greghaynes | mtreinish: yep, at runtime though | 01:59 |
| lifeless | mtreinish: I know exactly where dvsm does its stuff :) | 01:59 |
| mtreinish | lifeless: heh, ok | 01:59 |
| jeblair | lifeless: you may be the only one ;) | 01:59 |
| fungi | there are a few dib elements in dib itself which seem to use it | 02:00 |
| lifeless | jeblair: we should rewrite it in Make | 02:00 |
| fungi | also ansible's pip role | 02:00 |
| jeblair | lifeless: don't tempt clarkb | 02:00 |
| *** bpokorny has quit IRC | 02:00 | |
| clarkb | I already did it | 02:00 |
| fungi | er, ansible's pip_install role | 02:00 |
| lifeless | jeblair: nah, clarkb doesn't have time to rewrite devstack in Make | 02:00 |
| clarkb | oh devstack | 02:00 |
| mtreinish | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/fix_pip.sh looks like it could be a problem | 02:00 |
| *** maishsk has joined #openstack-infra | 02:00 | |
| fungi | also our cache-devstack element | 02:01 |
| *** bpokorny_ has quit IRC | 02:01 | |
| jeblair | mtreinish: i can't find that called anywhere | 02:01 |
| lifeless | mtreinish: we only need to cap it where we see failures for now IMO | 02:01 |
| fungi | oh, i guess that's just downloading not calling | 02:01 |
| *** akshai has joined #openstack-infra | 02:01 | |
| mtreinish | ugh, stable: http://logs.openstack.org/54/269954/2/check/gate-grenade-dsvm/312754b//logs/grenade.sh.txt.gz#_2016-01-20_01_58_30_538 | 02:01 |
| lifeless | mtreinish: we're going to get back onto latest soon, once we figure an answer for https://github.com/pypa/pip/issues/3384 | 02:02 |
| *** aeng has quit IRC | 02:02 | |
| fungi | the cache-devstack hits i mean | 02:02 |
| mtreinish | I'm gonna have to push a backport before we can land master | 02:02 |
| *** sflanigan_ has joined #openstack-infra | 02:02 | |
| *** sflanigan_ has joined #openstack-infra | 02:02 | |
| lifeless | mtreinish: is it a backport if it lands first ? | 02:02 |
| lifeless | mtreinish: or is the master change a forward port ?:) | 02:02 |
| *** sflanigan has quit IRC | 02:03 | |
| *** nelsnelson has quit IRC | 02:03 | |
| mtreinish | lifeless: heh, fair enough :) | 02:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Remove unused fix_pip.sh https://review.openstack.org/269962 | 02:03 |
| greghaynes | jeblair: Also https://review.openstack.org/#/c/245297/ | 02:04 |
| greghaynes | jeblair: We can remove out logic which removes pip from system packages pretty soon | 02:04 |
| lifeless | greghaynes: oh ? | 02:05 |
| jeblair | greghaynes: that's more than i can deal with right before dinner :) | 02:05 |
| greghaynes | lifeless: ya, dib has the smarts to bulid an image using get-pip.py now (it was stuck in t-i-e but recently moved up) | 02:06 |
| greghaynes | jeblair: It depends on something still, but just FYI | 02:06 |
| mtreinish | fungi, jeblair, clarkb: so we're gonna need to promote: https://review.openstack.org/#/c/269964/ https://review.openstack.org/#/c/269963/ and https://review.openstack.org/#/c/269954/ in that order when they pass check | 02:06 |
| jeblair | jhesketh: considering time of day, we may need your help with the above ^ | 02:07 |
| fungi | we've also got http://git.openstack.org/cgit/openstack-infra/puppet-pip/tree/manifests/init.pp#n16 | 02:07 |
| jhesketh | jeblair: sure, what can I do | 02:07 |
| *** piet has quit IRC | 02:08 | |
| jeblair | jhesketh: see request from mtreinish; i'm about to get food | 02:08 |
| fungi | http://git.openstack.org/cgit/openstack-infra/system-config/tree/install_puppet.sh#n217 | 02:09 |
| fungi | maybe | 02:09 |
| jeblair | i'm rebooting codesearch.o.o | 02:09 |
| jhesketh | mtreinish: looking.. | 02:11 |
| dims | mtreinish : 269964 will pass check first and would have to be merged before 269963 will pass check. right? | 02:12 |
| *** armax has quit IRC | 02:12 | |
| dims | mtreinish : so we may as well request a ninja merge of those 3 rather than a promotion | 02:12 |
| mtreinish | dims: yep, we've got to work are way up from kilo | 02:12 |
| mtreinish | dims: that's probably a good idea | 02:13 |
| mtreinish | jhesketh: ^^^ | 02:13 |
| *** aeng has joined #openstack-infra | 02:14 | |
| jeblair | so two things about hound: | 02:14 |
| jeblair | one, it did not start at boot | 02:14 |
| *** yonglihe has joined #openstack-infra | 02:14 | |
| jhesketh | mtreinish: happy to ninja merge, but I havne't done that before... fungi are you still around to advise on policy here? | 02:14 |
| jeblair | two, it seems to be cycling through pids at an insane rate | 02:15 |
| jeblair | you almost would not notice it because they loop around so quickly | 02:15 |
| openstackgerrit | Drew Thorstensen proposed openstack-infra/nodepool: Bump APScheduler to >=3.0 https://review.openstack.org/262017 | 02:15 |
| fungi | dims: mtreinish: usually you can address that with a depends-on | 02:16 |
| fungi | then they can all go into the gate together rather than needing one merged before check succeeds on the next | 02:16 |
| mtreinish | fungi: they all have the same Change ID | 02:16 |
| fungi | yeah, that won't work then | 02:16 |
| dims | yep, unfortunately not an option here | 02:17 |
| fungi | probably a terrible idea to keep using the same change id on backports if we want to be able to do cross-branch dependencies | 02:17 |
| mtreinish | yeah, I didn't think about that when I pushed it | 02:17 |
| *** tiswanso has quit IRC | 02:17 | |
| *** tiswanso has joined #openstack-infra | 02:17 | |
| *** piet has joined #openstack-infra | 02:20 | |
| jeblair | jhesketh, fungi: as for policy, generally by the time we get to "nothing will work at all unless we merge something" i'm pretty okay with ninja merges. | 02:20 |
| jeblair | however, if things are testable, i do prefer to try to merge them naturally, because they might not work or even make things worse. | 02:21 |
| jhesketh | yep, that makes sense | 02:21 |
| jeblair | i'm not quite sure where we are on this... | 02:21 |
| mtreinish | jeblair: yep, that's the case here all dsvm jobs are stuck on this | 02:21 |
| jhesketh | mtreinish: so this is just affecting stable right? | 02:21 |
| mtreinish | jhesketh: it's affecting all dsvm jobs | 02:21 |
| mtreinish | stable and master | 02:21 |
| jhesketh | oh.. | 02:21 |
| jeblair | mtreinish: though it's apparently a testable change, yeah? | 02:21 |
| dims | jhesketh : nope every one | 02:21 |
| *** nelsnelson has joined #openstack-infra | 02:21 | |
| mtreinish | jeblair: it is testable, https://jenkins02.openstack.org/job/gate-tempest-dsvm-full/31437/console shows that the fix worked | 02:22 |
| mtreinish | jeblair: the issue is there is a stable branch cross-dependency | 02:22 |
| dims | jeblair : however the grenade jobs will fail | 02:22 |
| mtreinish | master can't land until liberty has it (because of grenade) | 02:22 |
| dims | yep | 02:22 |
| mtreinish | and liberty can't land until kilo | 02:22 |
| *** armax has joined #openstack-infra | 02:22 | |
| jhesketh | right, but that's not a cyclic dependency no? | 02:22 |
| fungi | and they can't be declared as deps to zuul because they reused the change id on the backports | 02:22 |
| jhesketh | as in, if we do them in sequence they should go in neatly | 02:23 |
| jeblair | yeah, so we could re-propose as 3 different changes, promote, and they should merge naturally (barring random errors) | 02:23 |
| jhesketh | would it be so bad to use a different changeid? | 02:23 |
| mtreinish | jhesketh: except the sequence is kilo passes check we promote it passes gate, then we have to recheck liberty and repeat that and then do it again for master | 02:23 |
| mtreinish | it's a ~5-6 hour process | 02:23 |
| jeblair | mtreinish: no with depends on we should be able to do all 3 at once | 02:24 |
| *** ashleighfarnham has quit IRC | 02:24 | |
| fungi | unless they're reproposed with new change ids, set with dependency headers, and get a free pass out of check straight to the gate and promoted | 02:24 |
| fungi | them ~1 hour barring failures | 02:24 |
| fungi | er, then | 02:24 |
| jeblair | fungi: right | 02:24 |
| mtreinish | jeblair: fungi: sure, that's fine, I'll do that now | 02:24 |
| jhesketh | fungi: how do you get a free pass out of check? wouldn't we have to +1 as jenkins (according to the gate rules) | 02:25 |
| fungi | zuul enqueue | 02:25 |
| jhesketh | ah, and dequeue (if we could) from check | 02:25 |
| fungi | you can just ignore the instance in check | 02:25 |
| fungi | it'll report, but won't influence what's already running in the gate | 02:25 |
| openstackgerrit | Drew Thorstensen proposed openstack-infra/nodepool: Bump APScheduler to >=3.0 https://review.openstack.org/262017 | 02:25 |
| jhesketh | yep | 02:25 |
| jhesketh | okay so are we comfortable with setting the depends-on and then promoting straight to top of gate | 02:26 |
| jeblair | jhesketh: definitely comfortable with that, and if we get into trouble, i'm not opposed to ninja merging these either. | 02:26 |
| fungi | yeah, so long as they have distinct change-ids should be sane | 02:26 |
| jhesketh | okay | 02:26 |
| jhesketh | mtreinish: once you shoot through the new changeid's I'll do that thing | 02:26 |
| *** zz_dimtruck is now known as dimtruck | 02:27 | |
| mtreinish | https://review.openstack.org/#/c/269954/ | 02:27 |
| mtreinish | https://review.openstack.org/#/c/269970/ and https://review.openstack.org/#/c/269969/ | 02:27 |
| mtreinish | jhesketh: ^^^ | 02:27 |
| *** jcoufal has quit IRC | 02:28 | |
| *** shashank_hegde has quit IRC | 02:28 | |
| fungi | i'm with jeblair on policy, basically ninja-merge as a workaround to not wanting to wait for thorough testing is not my preference, but skipping check and prioritizing the gate is a sufficient compromise most of the time so that we can shorten the delays significantly | 02:28 |
| *** sridhar_ram has quit IRC | 02:28 | |
| dims | fungi : cool, did not about that option | 02:29 |
| jhesketh | mtreinish: do yo uwant to chuck a +W on them, then I'll enqueue to gate | 02:29 |
| jhesketh | otherwise once jenksin finishes it won't be able to submit | 02:29 |
| fungi | they'll need code review +2 and workflow +1 before you an enqueue, right | 02:29 |
| mtreinish | jhesketh: ok, should have +3 on all 3 | 02:30 |
| jhesketh | fungi: I suspect it'll enqueue correctly, but not submit | 02:30 |
| jhesketh | mtreinish: thanks | 02:30 |
| fungi | jhesketh: actually it didn't in the past, since zuul does a can-merge check of what gerrit wants on enqueuing | 02:31 |
| *** rossella_s has quit IRC | 02:31 | |
| fungi | which has actually tripped me up in the past until i realized that's why enqueue wasn't working | 02:31 |
| *** rossella_s has joined #openstack-infra | 02:31 | |
| *** baoli has joined #openstack-infra | 02:32 | |
| jhesketh | ah okay | 02:33 |
| jhesketh | mtreinish: okay looks like they are at the top of the gate | 02:33 |
| jhesketh | jeblair, fungi: thanks for your extra help, sorry to keep you away from dinner etc | 02:34 |
| mtreinish | jhesketh: ok cool, thanks | 02:34 |
| * mtreinish crosses his fingers | 02:34 | |
| fungi | jhesketh: not keeping me from dinner--i'm already in bed for the night | 02:34 |
| jhesketh | well sorry for keeping you from sleep :-( | 02:34 |
| mtreinish | fungi: at 9:30? :) | 02:35 |
| fungi | mtreinish: well, testing out a new mattress, also watching some cartoons | 02:35 |
| thorst | Hi. Wondering if someone can take a look at https://review.openstack.org/#/c/262017 ? | 02:35 |
| thorst | It looks like the corresponding zuul change merged 2 weeks ago (https://review.openstack.org/#/c/260637/) and now anyone running zulu & nodepool on the same system have an incompatibility with APScheduler until that change goes through. | 02:35 |
| *** wei-wang has joined #openstack-infra | 02:36 | |
| *** edmondsw has quit IRC | 02:37 | |
| ianw | thorst: looking | 02:37 |
| *** doug-fish has joined #openstack-infra | 02:39 | |
| dims | fungi : sounds like fun :) | 02:39 |
| thorst | thanks...I'm not sure context of it. But I proposed an update that I think addresses the -1 that had been on it earlier. Tox seems to work now for me locally. | 02:40 |
| dmsimard | Is there a list of openstack projects available somewhere ? Ideally somewhere parseable | 02:40 |
| thorst | and we're going to try to redeploy our CI to validate further...but that may run into tomorrow | 02:40 |
| *** davideagnello has quit IRC | 02:40 | |
| *** davideagnello has joined #openstack-infra | 02:41 | |
| dims | dmsimard : http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml ? | 02:41 |
| dmsimard | dims: yeah just found it too | 02:42 |
| dmsimard | dims: I'll try that, thanks | 02:42 |
| *** dansmith has quit IRC | 02:42 | |
| *** markvoelker has quit IRC | 02:42 | |
| *** annegentle has quit IRC | 02:43 | |
| *** doug-fish has quit IRC | 02:44 | |
| dmsimard | ehhhhhhh, it's dirty but it works: curl http://git.openstack.org/cgit/openstack/governance/plain/reference/projects.yaml | egrep "^\S+:$" |cut -f1 -d : | 02:44 |
| *** dimtruck is now known as zz_dimtruck | 02:44 | |
| mtreinish | ugh, kilo failed: http://logs.openstack.org/69/269969/1/gate/gate-tempest-dsvm-neutron-large-ops/4bea847//logs/devstacklog.txt.gz#_2016-01-20_02_42_37_774 | 02:44 |
| mtreinish | pip 8 ended up in the tox venv? | 02:45 |
| mtreinish | but not on the system, how did that happen? | 02:45 |
| *** MarkAtwood has joined #openstack-infra | 02:45 | |
| fungi | mtreinish: installed a newer version of virtualenv? | 02:45 |
| fungi | usually they get released in tandem since virtualenv bundles its own pip | 02:46 |
| *** tjones has joined #openstack-infra | 02:46 | |
| mtreinish | fungi: right, that's likely what it is | 02:46 |
| mtreinish | oh right it's kilo so no constraints | 02:46 |
| fungi | or else maybe something did a pip install -U pip in the virtualenv? or something's depending on pip and we install -U whatever that thing is? | 02:46 |
| *** gongysh has joined #openstack-infra | 02:47 | |
| fungi | aha, yeah there's that too | 02:47 |
| mtreinish | I'll push a g-r cap on kilo for virtualenv | 02:47 |
| mtreinish | this chain just keeps getting bigger | 02:47 |
| fungi | testing is helping at least | 02:47 |
| *** PsionTheory has quit IRC | 02:47 | |
| mtreinish | well I just want to unblock master, let mriedem worry about stable. (its his job now :) ) | 02:48 |
| *** dansmith has joined #openstack-infra | 02:48 | |
| *** dansmith is now known as Guest38462 | 02:48 | |
| jhesketh | Anybody here know much about the afs nodes? | 02:48 |
| fungi | jhesketh: they're not (yet) really in production, so if you reboot them and things don't come back up we can troubleshoot at our leisure | 02:50 |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Cap virtualenv<14 https://review.openstack.org/269974 | 02:50 |
| dims | mtreinish : did you see the "Detected a distutils installed project ('argparse') which we cannot uninstall."? | 02:51 |
| dims | ah you did | 02:51 |
| jhesketh | fungi: so my method was to do 'ps aux | grep -i afs' before restarting and then after.. the first node came back up with the same asf processes, the second one had /sbin/asfd running (which the first one didn't) which didn't come back up | 02:51 |
| jhesketh | so if somebody wants to work from that, then that's cool | 02:51 |
| jhesketh | otherwise I'll keep moving | 02:51 |
| mtreinish | jhesketh, fungi: ok can we try this again with: https://review.openstack.org/#/c/269974/ in addition to the other 3 | 02:52 |
| jhesketh | mtreinish: hmm, do the other ones need a new dependson? | 02:52 |
| mtreinish | jhesketh: only the kilo one does | 02:53 |
| dims | mtreinish : that will just let the kilo change in | 02:53 |
| jhesketh | right, but that affects the whole series | 02:53 |
| stevemar | something going on with the CI? | 02:53 |
| mtreinish | constraints should protect us on liberty and master | 02:53 |
| dims | stevemar : pip8 broke dsvm jobs | 02:53 |
| openstackgerrit | Merged openstack-infra/nodepool: Uncap pyzmq requirements https://review.openstack.org/262019 | 02:53 |
| stevemar | dims: wonderful! | 02:53 |
| stevemar | :) | 02:53 |
| dims | mtreinish : wouldn't bet on it | 02:53 |
| stevemar | mtreinish: master seems to be messed up | 02:54 |
| mtreinish | stevemar: every dsvm job is broken | 02:54 |
| mtreinish | I was talking about the patch series to fix it | 02:54 |
| dims | mtreinish : but worth a try | 02:55 |
| mtreinish | dims: well my earlier test runs showed virtualenv wasn't getting pulled in | 02:55 |
| *** jaypipes has quit IRC | 02:55 | |
| dims | mtreinish : ack | 02:55 |
| mtreinish | the master patch got to tempest everywhere except for grenade :) | 02:55 |
| *** zz_dimtruck is now known as dimtruck | 02:55 | |
| dims | mtreinish : jhesketh : 269974 has to be the first one in the queue for it to work i think | 02:56 |
| jhesketh | dims: yeah, I'm reworking the queue now | 02:56 |
| dims | thanks jhesketh | 02:57 |
| *** changzhi has joined #openstack-infra | 02:57 | |
| *** baoli has quit IRC | 02:57 | |
| stevemar | dims: mtreinish well that's one way to clear out the check and gate pipelines | 02:58 |
| dims | LOL stevemar | 02:58 |
| *** binbincong has quit IRC | 02:59 | |
| mtreinish | stevemar: it is a milestone week, this just comes with the territory :) | 02:59 |
| *** rkukura has joined #openstack-infra | 02:59 | |
| stevemar | dims: mtreinish: hehe, i think some of the patches in gate may still survive | 02:59 |
| *** MarkAtwood has quit IRC | 03:00 | |
| stevemar | oh boy, i can see the gate drop off in numbers | 03:00 |
| stevemar | 75.... 69 | 03:00 |
| stevemar | 651 | 03:00 |
| stevemar | 61 | 03:00 |
| jhesketh | so got the requirements change to the top, but the devstack ones aren't queuing.. | 03:01 |
| jhesketh | oh this is the thing fungi had mentioned | 03:01 |
| jhesketh | mtreinish: your +W from 269969 was lost, could you please readd it | 03:01 |
| * jhesketh hasn't got +w | 03:01 | |
| mtreinish | jhesketh: oops, sorry. Added it again | 03:01 |
| mtreinish | forgot that pushing a new rev would lose that :) | 03:02 |
| jhesketh | ta | 03:02 |
| *** unicell1 has quit IRC | 03:03 | |
| jhesketh | mtreinish: okay they are the top 4 now | 03:03 |
| dims | mtreinish : https://review.openstack.org/#/c/269974/ and https://review.openstack.org/#/c/269969/ wont' merge without each other i think | 03:04 |
| mtreinish | jhesketh: oh crap, the requirements patch is gonna fail because it needs the pip cap from the devstack change | 03:04 |
| dims | yep | 03:04 |
| mtreinish | dims: jinx | 03:04 |
| mtreinish | we've got a circular dependency now I guess it's time to ninja merge :( | 03:04 |
| dims | right | 03:05 |
| jhesketh | hmm sounds like it.. probably best to merge the requirements change and then queue back in the devstack ones | 03:05 |
| fungi | in which case you pick the simpler of the two and we manually submit it for merge | 03:05 |
| fungi | yeah, i agree probably the stable reqs change | 03:05 |
| jhesketh | fungi: which group do I need to add myself to to submit changes? | 03:06 |
| jhesketh | or is there more magic than that | 03:06 |
| * jhesketh hasn't ninja'd before and pretty sure he isn't stealthy enough to not trip over or cut himself with his stars | 03:06 | |
| dims | jhesketh : +1 to merge requirements and queue the rest back in | 03:06 |
| dims | haha | 03:07 |
| fungi | jhesketh: project bootstrappers | 03:07 |
| jhesketh | ta | 03:07 |
| *** baoli has joined #openstack-infra | 03:08 | |
| jhesketh | fungi: is there anything else I need to do? I can't see the verified or submit options | 03:09 |
| fungi | jhesketh: force-reload | 03:09 |
| fungi | your browser | 03:09 |
| *** thorst has quit IRC | 03:09 | |
| jhesketh | yep, did that | 03:09 |
| * jhesketh is probably missing something obvious | 03:09 | |
| fungi | gerrit caches some mighty strange things in browser | 03:09 |
| *** moravec has joined #openstack-infra | 03:10 | |
| fungi | maybe also flush gerrit's caches via the ssh api | 03:10 |
| fungi | gerrit flush-caches | 03:10 |
| fungi | and then force-reload your browser once that completes | 03:10 |
| *** moravec has quit IRC | 03:10 | |
| *** pahuang has quit IRC | 03:11 | |
| *** dingyichen has quit IRC | 03:11 | |
| *** mudassirlatif has quit IRC | 03:12 | |
| jhesketh | fungi: hmm still nothing | 03:13 |
| *** apoorvad has quit IRC | 03:13 | |
| fungi | still no verified? | 03:13 |
| fungi | note that submit no longer shows up there in 2.11 | 03:13 |
| jhesketh | nope, no verified options | 03:14 |
| *** rlandy has quit IRC | 03:14 | |
| *** wei-wang has quit IRC | 03:14 | |
| *** ashishb has joined #openstack-infra | 03:15 | |
| *** dimtruck is now known as zz_dimtruck | 03:15 | |
| *** gongysh has quit IRC | 03:16 | |
| ianw | mtreinish: this pip thing is going to be a PITA | 03:16 |
| *** achanda has joined #openstack-infra | 03:17 | |
| fungi | jhesketh: i get a verified after adding myself to project bootstrappers and going to https://review.openstack.org/269974 and replying | 03:17 |
| fungi | jhesketh: when you're _on_ the reply pop-up dialogue, force refresh your browser there | 03:18 |
| jhesketh | wow, that worked | 03:18 |
| jhesketh | that's incredibly weird | 03:18 |
| *** yuanying_ has joined #openstack-infra | 03:18 | |
| mtreinish | ianw: heh, yep | 03:18 |
| *** aeng has quit IRC | 03:18 | |
| fungi | see my earlier comment about the weird ways gerrit caches in a browser | 03:18 |
| jhesketh | agreed | 03:19 |
| *** dchen has quit IRC | 03:19 | |
| *** sflanigan_ has quit IRC | 03:19 | |
| *** tangchen has joined #openstack-infra | 03:19 | |
| *** yuanying has quit IRC | 03:19 | |
| fungi | jhesketh: so anyway, +2 code review, +2 verify, +1 workflow with a comment that we're bypassing gating due to a circular fix and then once that's showing there should be a submit change button | 03:19 |
| fungi | and then we should see gerritbot report the merge event in here | 03:20 |
| *** coolsvap|away is now known as coolsvap | 03:20 | |
| *** yamahata has quit IRC | 03:20 | |
| jhesketh | fungi: done the +2 verify, but can't see a submit button | 03:20 |
| jhesketh | the change says "not verified". I wonder if it's because Jenkins snuck in a -2 first | 03:20 |
| *** bardia has quit IRC | 03:20 | |
| fungi | oh, and delete the -2, yeah | 03:21 |
| *** bardia has joined #openstack-infra | 03:21 | |
| openstackgerrit | Merged openstack/requirements: Cap virtualenv<14 https://review.openstack.org/269974 | 03:21 |
| jhesketh | ah there we go | 03:21 |
| mtreinish | \o/ 1 down 3 to go | 03:21 |
| jhesketh | okay putting the devstack changes back to the top | 03:23 |
| *** ashishb has quit IRC | 03:23 | |
| * mtreinish crosses his fingers again | 03:23 | |
| jhesketh | mtreinish: can you abandon the backports with the same id's please | 03:23 |
| jhesketh | nevermind, you have | 03:24 |
| jhesketh | okay, those are top of the queue | 03:24 |
| *** zz_dimtruck is now known as dimtruck | 03:24 | |
| jhesketh | I have to run to an appointment but will be able to jump back in if anything gets stuck or blows up | 03:24 |
| jhesketh | be back soon | 03:24 |
| *** pahuang has joined #openstack-infra | 03:24 | |
| *** piet has quit IRC | 03:25 | |
| *** dingyichen has joined #openstack-infra | 03:25 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements https://review.openstack.org/246114 | 03:28 |
| *** ccrouch has joined #openstack-infra | 03:30 | |
| *** dchen has joined #openstack-infra | 03:30 | |
| *** sflanigan_ has joined #openstack-infra | 03:31 | |
| *** sflanigan_ has joined #openstack-infra | 03:31 | |
| *** achanda has quit IRC | 03:32 | |
| *** pahuang has quit IRC | 03:35 | |
| *** sflanigan_ has quit IRC | 03:35 | |
| *** achanda has joined #openstack-infra | 03:38 | |
| mtreinish | ugh, it's still installing virtualenv 14 | 03:38 |
| *** aeng has joined #openstack-infra | 03:39 | |
| *** dimtruck is now known as zz_dimtruck | 03:39 | |
| dims | mtreinish : requirements/test-requirements.txt (line 12) ? | 03:41 |
| *** vivekd has quit IRC | 03:42 | |
| mtreinish | dims: no, it's kilo devstack | 03:42 |
| mtreinish | pip_install doesn't do g-r | 03:42 |
| mtreinish | https://github.com/openstack-dev/devstack/blob/stable/kilo/inc/python#L71 | 03:43 |
| mtreinish | there was a separate pip_install_gr function | 03:43 |
| *** dchen has quit IRC | 03:43 | |
| mtreinish | so I think just setting the pip_install -U virtualenv to pip_install -U virtualenv<14 in the kilo patch will be the right way | 03:44 |
| *** gyee has quit IRC | 03:44 | |
| *** vivekd has joined #openstack-infra | 03:44 | |
| dims | mtreinish : not https://github.com/openstack/requirements/blob/stable/kilo/test-requirements.txt#L12 ? | 03:44 |
| mtreinish | dims: that only matters for the requirements repo tests | 03:44 |
| *** zz_dimtruck is now known as dimtruck | 03:44 | |
| mtreinish | kilo global requirements might not be able to pass it's tests without the cap added there, but thats of lesser concern | 03:45 |
| dims | mtreinish : i see it kick in http://logs.openstack.org/69/269969/2/gate/gate-tempest-dsvm-postgres-full/a05abef//logs/devstacklog.txt.gz#_2016-01-20_03_38_25_321 | 03:45 |
| *** woodster_ has quit IRC | 03:46 | |
| mtreinish | dims: what kicks in? | 03:47 |
| *** pahuang has joined #openstack-infra | 03:47 | |
| *** akshai has quit IRC | 03:48 | |
| mtreinish | dims: I'm going to add: http://paste.openstack.org/show/484370/ to the kilo patch | 03:48 |
| dims | mtreinish : http://logs.openstack.org/69/269969/2/gate/gate-tempest-dsvm-postgres-full/a05abef//logs/devstacklog.txt.gz#_2016-01-20_03_33_58_721 | 03:48 |
| dims | mtreinish : ok | 03:49 |
| mtreinish | dims: there is also: http://logs.openstack.org/69/269969/2/gate/gate-tempest-dsvm-postgres-full/a05abef//logs/devstacklog.txt.gz#_2016-01-20_03_33_46_199 | 03:49 |
| *** andreaf has quit IRC | 03:50 | |
| mtreinish | which happens right before that | 03:50 |
| dims | y | 03:50 |
| mtreinish | that link you pasted is happening inside the wheelhouse venv, which doesn't look like it's causing an issue | 03:51 |
| *** ashishb has joined #openstack-infra | 03:51 | |
| dims | ok | 03:51 |
| mtreinish | dims: https://review.openstack.org/#/c/269969/ is updated now | 03:52 |
| *** rhallisey has quit IRC | 03:52 | |
| *** SotK has quit IRC | 03:52 | |
| mtreinish | jhesketh: we do need to promote everything again when you're back | 03:52 |
| *** andreaf has joined #openstack-infra | 03:52 | |
| *** dimtruck is now known as zz_dimtruck | 03:52 | |
| *** SotK has joined #openstack-infra | 03:52 | |
| dims | mtreinish : will check up when i wake up :) | 03:53 |
| mtreinish | dims: oh, right inside a venv won't be an issue because there aren't any ubuntu packages in there which are doing a distutils install | 03:53 |
| dims | good | 03:53 |
| mtreinish | which is one of the potential solutions lifeless mentioned in: https://github.com/pypa/pip/issues/3384 | 03:53 |
| dims | ok cross my fingers | 03:53 |
| dims | signing off now | 03:53 |
| mtreinish | err, well maybe it was on irc, because its not in the bug :) | 03:54 |
| mtreinish | dims: ok, o/ | 03:54 |
| mtreinish | thanks for helping | 03:54 |
| *** achanda_ has joined #openstack-infra | 03:54 | |
| dims | :) | 03:54 |
| *** dims has quit IRC | 03:54 | |
| *** dchen has joined #openstack-infra | 03:56 | |
| *** achanda_ has quit IRC | 03:56 | |
| *** aeng has quit IRC | 03:56 | |
| *** achanda has quit IRC | 03:56 | |
| *** Daisy has quit IRC | 03:56 | |
| madorn | there is currently an issue with openstack-dev/sandbox where registered users can no longer +2 code reviews...it appears to have happened after the gerrit upgrade..i know clarkb and anteaya were looking at it yesterday and confirmed the issue.. | 03:57 |
| *** akshai has joined #openstack-infra | 03:57 | |
| *** pahuang has quit IRC | 03:57 | |
| *** dingyichen has quit IRC | 03:57 | |
| jhesketh | mtreinish: heading home, will look asap | 03:58 |
| mtreinish | jhesketh: ok thanks, I'm about to call it a night myself | 03:58 |
| mtreinish | so hopefully someone will take over if the most recent kilo rev doesn't work | 03:59 |
| *** akshai_ has joined #openstack-infra | 03:59 | |
| *** dims has joined #openstack-infra | 03:59 | |
| ianw | mtreinish: i'll keep an eye ... but this is a mess. we've been overwriting installed packages for years | 04:00 |
| *** flwang1 has quit IRC | 04:01 | |
| *** treaki_ has joined #openstack-infra | 04:01 | |
| *** akshai has quit IRC | 04:02 | |
| *** ddieterly has quit IRC | 04:02 | |
| ianw | i'm rather worried if this will affect image builds too | 04:03 |
| mtreinish | ianw: cool thanks | 04:03 |
| mtreinish | ianw: yeah the image builds are a risk, if they pull in pip8 or virtualenv 14 it could cause issues | 04:03 |
| *** treaki has quit IRC | 04:04 | |
| *** yuanying_ has quit IRC | 04:05 | |
| *** links has joined #openstack-infra | 04:06 | |
| *** annegentle has joined #openstack-infra | 04:06 | |
| *** akshai_ has quit IRC | 04:06 | |
| *** akshai has joined #openstack-infra | 04:07 | |
| *** rguillebert has quit IRC | 04:07 | |
| *** yuanying_ has joined #openstack-infra | 04:07 | |
| *** Jeffrey4l has quit IRC | 04:08 | |
| *** pahuang has joined #openstack-infra | 04:09 | |
| *** dingyichen has joined #openstack-infra | 04:09 | |
| *** otter768 has quit IRC | 04:10 | |
| *** aeng has joined #openstack-infra | 04:10 | |
| *** annegentle has quit IRC | 04:10 | |
| *** akshai has quit IRC | 04:12 | |
| *** akshai has joined #openstack-infra | 04:13 | |
| *** dims has quit IRC | 04:13 | |
| *** flwang has quit IRC | 04:13 | |
| mtreinish | ianw: ugh, I need quotes: http://logs.openstack.org/69/269969/3/check/gate-tempest-dsvm-full/e71c689//logs/devstacklog.txt.gz#_2016-01-20_04_12_00_600 | 04:14 |
| *** markvoelker has joined #openstack-infra | 04:14 | |
| *** flwang has joined #openstack-infra | 04:14 | |
| * jhesketh returns | 04:15 | |
| jhesketh | mtreinish: what needed changing? | 04:15 |
| mtreinish | jhesketh: kilo devstack doesnt respect g-r when installing virtualenv | 04:16 |
| mtreinish | so I had to manually cap all the places kilo devstack was installing virtualenv | 04:16 |
| mtreinish | I just pushed a new rev on: https://review.openstack.org/#/c/269969/ | 04:17 |
| mtreinish | which should do the trick, hopefully | 04:17 |
| jhesketh | mtreinish: okay, and the other patches are the same? | 04:17 |
| jhesketh | ie unchanged | 04:17 |
| mtreinish | yep | 04:17 |
| *** fitoduarte has quit IRC | 04:17 | |
| *** akshai has quit IRC | 04:18 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Support old setuptools for LTS distro venvs https://review.openstack.org/269999 | 04:18 |
| jhesketh | mtreinish: needs as +W | 04:19 |
| jhesketh | *a | 04:19 |
| *** yuanying has joined #openstack-infra | 04:20 | |
| mtreinish | jhesketh: heh, I'll never remember that | 04:20 |
| mtreinish | it should have it now | 04:21 |
| *** aeng has quit IRC | 04:21 | |
| *** yuanying_ has quit IRC | 04:22 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Support old setuptools for LTS distro venvs https://review.openstack.org/269999 | 04:23 |
| *** tjones has quit IRC | 04:25 | |
| *** baoli has quit IRC | 04:25 | |
| *** pahuang has quit IRC | 04:26 | |
| *** watanabe_isao has quit IRC | 04:27 | |
| jhesketh | mtreinish: okay shuffled to the top | 04:28 |
| jhesketh | lets see how they go now | 04:28 |
| mtreinish | jhesketh: this is my last shot for tonight, if it doesn't work I'm passing the baton to you and ianw | 04:28 |
| jhesketh | yep happy to do what I can | 04:29 |
| jhesketh | thanks for your help :-) | 04:29 |
| *** MarkAtwood has joined #openstack-infra | 04:29 | |
| madorn | is there a place to submit bugs for openstack-dev..? | 04:29 |
| *** dingyichen has quit IRC | 04:30 | |
| *** bpokorny has joined #openstack-infra | 04:31 | |
| *** ashishb has quit IRC | 04:32 | |
| *** Hal has joined #openstack-infra | 04:34 | |
| *** Hal is now known as Guest35147 | 04:34 | |
| *** aeng has joined #openstack-infra | 04:35 | |
| *** sdake has quit IRC | 04:36 | |
| *** sdake has joined #openstack-infra | 04:37 | |
| *** yuanying has quit IRC | 04:37 | |
| *** pahuang has joined #openstack-infra | 04:38 | |
| *** yuanying has joined #openstack-infra | 04:39 | |
| zxiiro | anyone here know what project provides this page? http://zuul.openstack.org/ | 04:39 |
| zxiiro | is it a part of zuul-server or something else? | 04:39 |
| *** yuanying has quit IRC | 04:40 | |
| *** bpokorny has quit IRC | 04:40 | |
| *** bpokorny has joined #openstack-infra | 04:42 | |
| *** doug-fish has joined #openstack-infra | 04:42 | |
| *** dingyichen has joined #openstack-infra | 04:43 | |
| *** nelsnelson has quit IRC | 04:43 | |
| *** yuanying has joined #openstack-infra | 04:44 | |
| *** david-lyle has joined #openstack-infra | 04:44 | |
| *** bpokorny has quit IRC | 04:44 | |
| *** Daisy has joined #openstack-infra | 04:45 | |
| *** dingyichen has quit IRC | 04:45 | |
| *** fedexo has joined #openstack-infra | 04:46 | |
| *** Jeffrey4l has joined #openstack-infra | 04:46 | |
| *** doug-fish has quit IRC | 04:46 | |
| *** Jeffrey4l has quit IRC | 04:47 | |
| *** Jeffrey4l has joined #openstack-infra | 04:47 | |
| *** yuanying_ has joined #openstack-infra | 04:49 | |
| *** doug-fish has joined #openstack-infra | 04:49 | |
| *** yuanying has quit IRC | 04:50 | |
| *** tiswanso has quit IRC | 04:50 | |
| *** doug-fish has quit IRC | 04:53 | |
| stevemar | mtreinish: looks like the check/gate are still borked eh? | 04:54 |
| mtreinish | stevemar: the fix is still in progress, it needed several more revs | 04:54 |
| mtreinish | you can read the backscroll for all the gory details :) | 04:54 |
| stevemar | mtreinish: link? | 04:54 |
| stevemar | oh i see it | 04:54 |
| stevemar | 269969/ | 04:54 |
| mtreinish | but it's at the top of the gate now and it looks like it's working | 04:54 |
| mtreinish | yep, that's the start of it | 04:55 |
| mtreinish | there is an equiv change on liberty and master too | 04:55 |
| stevemar | man, that a storm | 04:55 |
| dstufft | lifeless: curl https://t.co/1cT8QG2j00 | python - 'pip<8' | 04:55 |
| dstufft | er | 04:55 |
| dstufft | stupid thing | 04:55 |
| dstufft | lifeless: curl https://bootstrap.pypa.io/get-pip.py | python - 'pip<8' | 04:55 |
| dstufft | that works now | 04:55 |
| *** Ramanjaneya has joined #openstack-infra | 04:56 | |
| stevemar | mtreinish: well i guess i'll recheck in a few hours | 04:56 |
| dstufft | or mtreinish or whatever, if you prefer that form anyways | 04:56 |
| *** Ramanjaneya_ has joined #openstack-infra | 04:56 | |
| *** bpokorny has joined #openstack-infra | 04:57 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 04:58 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add generic script to recover diagnostic or service config and logs https://review.openstack.org/270004 | 04:58 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Leverage default-logs for puppet-openstack gate jobs https://review.openstack.org/270005 | 04:58 |
| *** treaki__ has joined #openstack-infra | 04:59 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Leverage default-logs for puppet-openstack gate jobs https://review.openstack.org/270005 | 04:59 |
| *** jamespage has quit IRC | 05:00 | |
| *** gnuoy has quit IRC | 05:00 | |
| *** gnuoy has joined #openstack-infra | 05:00 | |
| *** jamespage has joined #openstack-infra | 05:00 | |
| *** Ramanjaneya has quit IRC | 05:01 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 05:02 |
| *** treaki_ has quit IRC | 05:02 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 05:03 |
| dmsimard | Sorry for the spam ^ | 05:03 |
| dmsimard | Going to sleep now :) | 05:03 |
| *** shashank_hegde has joined #openstack-infra | 05:03 | |
| *** ddieterly has joined #openstack-infra | 05:03 | |
| greghaynes | clarkb: jeblair fungi I am EOD so I cant debug now but it seems pip 8.0 has broken dib, so I suspect image building tomorrow will be broken as a result | 05:04 |
| greghaynes | just FYI | 05:04 |
| *** Daisy has quit IRC | 05:04 | |
| *** oomichi_away has quit IRC | 05:04 | |
| *** sdake_ has joined #openstack-infra | 05:05 | |
| *** jaosorior has joined #openstack-infra | 05:06 | |
| *** watanabe_isao has joined #openstack-infra | 05:06 | |
| *** sdake has quit IRC | 05:06 | |
| *** ddieterly has quit IRC | 05:08 | |
| *** armax has quit IRC | 05:09 | |
| *** oomichi has joined #openstack-infra | 05:10 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Properly fail/trap in eval_run_d https://review.openstack.org/259734 | 05:11 |
| *** Sukhdev has joined #openstack-infra | 05:11 | |
| *** maishsk has quit IRC | 05:11 | |
| *** nelsnelson has joined #openstack-infra | 05:13 | |
| ianw | lifeless / dstufft: http://lists.openstack.org/pipermail/openstack-dev/2016-January/084490.html <- on this over-install failure. i'm not sure where we will go from here :( | 05:15 |
| *** piet has joined #openstack-infra | 05:16 | |
| ianw | greghaynes: yeah, just as I feared | 05:16 |
| greghaynes | ianw: ah, there is some ML tl;dr I have missed | 05:16 |
| dstufft | ianw: to be clear, pip didn't remove any directories for those packages | 05:17 |
| *** armax has joined #openstack-infra | 05:17 | |
| dstufft | all it did was remove the .egg-info file that said "hey X is installed" | 05:17 |
| ianw | greghaynes: there's nothing to tl;dr at this point ... i'm not sure how we're going to fix it | 05:17 |
| ianw | dstufft: yeah, sorry, and then overwrote | 05:17 |
| ianw | dstufft: one thing i'm thinking is we just remove the egg-info in devstack | 05:17 |
| ianw | ugly though | 05:18 |
| greghaynes | I dont entirely understand why pip is trying to uninstall that package - is it because it is attempting to upgrade a system installed package? | 05:18 |
| dstufft | yes | 05:18 |
| ianw | greghaynes: yeah, generally anything left behind is part of some system dependency we can't remove (like yum on centos, etc) | 05:18 |
| greghaynes | Ya, so it is a combination of the distutils in setup.py + allowing system-site-packages | 05:18 |
| greghaynes | ok | 05:19 |
| greghaynes | fun | 05:19 |
| ianw | dstufft: if i proposed something for pip that did "if os.environ("ALLOW_UNSAFE_DISTUTILS_UNINTSALL") or something and did what it used to do, would that be rejected? | 05:19 |
| ianw | i can't see any precedent for that, so i'm guessing yes | 05:19 |
| *** Daisy has joined #openstack-infra | 05:20 | |
| *** sabeen has quit IRC | 05:20 | |
| dstufft | ianw: fwiw I'm not opposed to rolling it back to a warning, but I'd want at least some idea how we're going to actually move forward on the issue if we do that. As it is right now the warning has been there a year and it appears folks (distros, packagers, users) just ignored it until it became not a warning so it's likely if we reverted it back to a warning folks would just go back to ignoring it | 05:22 |
| *** vivekd has quit IRC | 05:23 | |
| *** vivekd_ has joined #openstack-infra | 05:23 | |
| ianw | dstufft: does pip really care? it's not ideal but maybe it's good enough. aren't distros moving towards setuptools based intsalls, where pip can actually properly remove distro installed packages? it might just fix itself | 05:23 |
| *** vivekd_ is now known as vivekd | 05:24 | |
| ianw | basically devstack has shot itself in the foot with it so many times we have no foot left to shoot off :) | 05:24 |
| dstufft | ianw: I think Fedora is planning (or is) on using pip to install things so they would, Debuntu side they do not seem to interested in it | 05:25 |
| dstufft | ianw: as far as does pip really care -- Yes-ish, it's the cause of weird bugs when you overlay two packages ontop of each other like that and it's in preperation for pip to start hard failing on overwriting all together without a flag to force it | 05:26 |
| ianw | dstufft: so if this warning was replaced with a flag, i think that would be fine... | 05:27 |
| ianw | what used to be a warning, i mean | 05:27 |
| ianw | also, the breakage around it seems like a good argument for debuntu to care | 05:27 |
| dstufft | (I suspect the Debuntu answer will be the standard, don't use pip outside of --user and virtual envs) | 05:32 |
| dstufft | I think they've modified pip in their repos to not be able to touch system files at all | 05:32 |
| *** f1ller has quit IRC | 05:32 | |
| *** vivekd_ has joined #openstack-infra | 05:33 | |
| *** f1ller has joined #openstack-infra | 05:35 | |
| *** vivekd has quit IRC | 05:35 | |
| *** vivekd_ is now known as vivekd | 05:35 | |
| *** achanda has joined #openstack-infra | 05:36 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add generic script to recover diagnostic or service config and logs https://review.openstack.org/270004 | 05:37 |
| *** shashank_hegde has quit IRC | 05:38 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic https://review.openstack.org/270010 | 05:39 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Leverage default-logs for puppet-openstack gate jobs https://review.openstack.org/270005 | 05:39 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 05:39 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic https://review.openstack.org/270010 | 05:39 |
| *** bpokorny has quit IRC | 05:43 | |
| *** vgridnev has joined #openstack-infra | 05:44 | |
| *** wei-wang has joined #openstack-infra | 05:46 | |
| *** Jeffrey4l has quit IRC | 05:47 | |
| *** salv-orlando has joined #openstack-infra | 05:49 | |
| *** MarkAtwood has quit IRC | 05:50 | |
| *** abregman has joined #openstack-infra | 05:51 | |
| *** yamahata has joined #openstack-infra | 05:53 | |
| *** gongysh has joined #openstack-infra | 05:54 | |
| lifeless | dstufft: oh ? | 05:54 |
| lifeless | dstufft: did you tweak the get-pip glue code? | 05:55 |
| dstufft | yea | 05:55 |
| lifeless | dstufft: cool | 05:55 |
| *** otherwiseguy has quit IRC | 05:58 | |
| *** otherwiseguy has joined #openstack-infra | 05:58 | |
| *** laurelm has quit IRC | 05:58 | |
| *** laurelm has joined #openstack-infra | 05:59 | |
| *** vgridnev has quit IRC | 06:00 | |
| *** fawadkhaliq has joined #openstack-infra | 06:02 | |
| *** MarkAtwood has joined #openstack-infra | 06:02 | |
| *** _et_ has quit IRC | 06:02 | |
| *** nelsnelson has quit IRC | 06:03 | |
| *** Sukhdev has quit IRC | 06:03 | |
| *** mrmartin has joined #openstack-infra | 06:04 | |
| *** ddieterly has joined #openstack-infra | 06:04 | |
| *** unicell has joined #openstack-infra | 06:05 | |
| *** annegentle has joined #openstack-infra | 06:06 | |
| *** Jeffrey4l has joined #openstack-infra | 06:07 | |
| ianw | dstufft: i've proposed the revert in https://github.com/pypa/pip/pull/3389 ... i'm sure there will be opinions | 06:09 |
| *** fedexo has quit IRC | 06:09 | |
| *** ddieterly has quit IRC | 06:09 | |
| *** vgridnev has joined #openstack-infra | 06:09 | |
| *** otter768 has joined #openstack-infra | 06:11 | |
| *** annegentle has quit IRC | 06:11 | |
| *** vivekd has quit IRC | 06:11 | |
| *** ashishb has joined #openstack-infra | 06:11 | |
| *** gongysh has quit IRC | 06:13 | |
| *** aopincar has joined #openstack-infra | 06:13 | |
| *** jtomasek has joined #openstack-infra | 06:14 | |
| *** piet has quit IRC | 06:15 | |
| *** otter768 has quit IRC | 06:15 | |
| jhesketh | yay, looks like the dsvm fixes are all in | 06:16 |
| *** yfried has quit IRC | 06:17 | |
| *** Jeffrey4l has quit IRC | 06:18 | |
| *** SamYaple is now known as SamYample | 06:20 | |
| *** SamYample is now known as SamYaple | 06:20 | |
| *** Jeffrey4l has joined #openstack-infra | 06:21 | |
| *** achanda has quit IRC | 06:25 | |
| *** Jeffrey4l has quit IRC | 06:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/267962 | 06:27 |
| *** achanda has joined #openstack-infra | 06:28 | |
| *** rossella_s has quit IRC | 06:31 | |
| *** rossella_s has joined #openstack-infra | 06:31 | |
| *** bardia has quit IRC | 06:31 | |
| *** achanda has quit IRC | 06:32 | |
| *** achanda has joined #openstack-infra | 06:32 | |
| *** MarkAtwood has quit IRC | 06:33 | |
| *** unicell has quit IRC | 06:34 | |
| *** unicell has joined #openstack-infra | 06:36 | |
| *** achanda has quit IRC | 06:38 | |
| *** vgridnev has quit IRC | 06:39 | |
| *** nmagnezi has joined #openstack-infra | 06:40 | |
| *** jamielennox is now known as jamielennox|away | 06:40 | |
| *** unicell1 has joined #openstack-infra | 06:42 | |
| *** fawadk has joined #openstack-infra | 06:43 | |
| *** unicell has quit IRC | 06:43 | |
| *** maishsk has joined #openstack-infra | 06:43 | |
| *** odyssey4me has quit IRC | 06:44 | |
| *** bradjones has quit IRC | 06:45 | |
| *** bradjones has joined #openstack-infra | 06:45 | |
| *** bradjones has quit IRC | 06:45 | |
| *** bradjones has joined #openstack-infra | 06:45 | |
| *** Jeffrey4l has joined #openstack-infra | 06:47 | |
| *** fawadkhaliq has quit IRC | 06:47 | |
| *** bardia has joined #openstack-infra | 06:48 | |
| *** mrmartin has quit IRC | 06:50 | |
| *** odyssey4me has joined #openstack-infra | 06:51 | |
| *** gildub has quit IRC | 06:52 | |
| *** maishsk has quit IRC | 06:53 | |
| *** jaosorior has quit IRC | 06:54 | |
| *** shardy has joined #openstack-infra | 06:58 | |
| *** vivekd has joined #openstack-infra | 07:01 | |
| *** rcernin has joined #openstack-infra | 07:02 | |
| *** aeng is now known as aeng_afk | 07:02 | |
| *** ddieterly has joined #openstack-infra | 07:05 | |
| *** _et_ has joined #openstack-infra | 07:08 | |
| *** vgridnev has joined #openstack-infra | 07:09 | |
| *** shashank_hegde has joined #openstack-infra | 07:10 | |
| *** ddieterly has quit IRC | 07:10 | |
| *** mrmartin has joined #openstack-infra | 07:11 | |
| *** markvoelker has quit IRC | 07:11 | |
| *** markvoelker has joined #openstack-infra | 07:12 | |
| *** bardia has quit IRC | 07:13 | |
| *** markvoelker_ has joined #openstack-infra | 07:18 | |
| *** harlowja_at_home has joined #openstack-infra | 07:19 | |
| *** scheuran has joined #openstack-infra | 07:19 | |
| *** markvoelker has quit IRC | 07:20 | |
| *** kaisers1 has quit IRC | 07:20 | |
| *** tdurakov has joined #openstack-infra | 07:25 | |
| *** kaisers has joined #openstack-infra | 07:29 | |
| *** HeOS has quit IRC | 07:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Pin pip to < 8 https://review.openstack.org/270038 | 07:32 |
| *** mudassirlatif has joined #openstack-infra | 07:32 | |
| *** yfried has joined #openstack-infra | 07:33 | |
| *** _nadya_ has joined #openstack-infra | 07:34 | |
| *** tdurakov has quit IRC | 07:34 | |
| AJaeger | dstufft, ian, lifeless : WE'Ve broken also pep8 and docs targets with pip 8 | 07:35 |
| AJaeger | what do you think of 270038 as quick measure? | 07:35 |
| *** tdurakov has joined #openstack-infra | 07:36 | |
| AJaeger | dstufft, ianw, lifeless, jhesketh : See https://review.openstack.org/#/c/268687/ for an example breakage | 07:36 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add networking-generic-switch project https://review.openstack.org/267664 | 07:37 |
| *** markvoel_ has joined #openstack-infra | 07:42 | |
| *** markvoelker_ has quit IRC | 07:43 | |
| *** Kennan has quit IRC | 07:45 | |
| *** tdurakov has quit IRC | 07:47 | |
| *** GheRivero is now known as Rivero | 07:48 | |
| *** GheRiver1 is now known as GheRivero | 07:49 | |
| *** salv-orlando has quit IRC | 07:49 | |
| *** Kennan has joined #openstack-infra | 07:50 | |
| *** sdake_ has quit IRC | 07:55 | |
| *** mrmartin has quit IRC | 07:57 | |
| *** harlowja_at_home has quit IRC | 07:59 | |
| *** otter768 has joined #openstack-infra | 08:00 | |
| *** otter768 has quit IRC | 08:02 | |
| *** mrmartin has joined #openstack-infra | 08:02 | |
| *** otter768_ has joined #openstack-infra | 08:02 | |
| *** otter768 has joined #openstack-infra | 08:04 | |
| *** otter768_ has quit IRC | 08:04 | |
| *** otter768 has quit IRC | 08:05 | |
| *** markus_z has joined #openstack-infra | 08:06 | |
| *** otter768 has joined #openstack-infra | 08:06 | |
| *** ddieterly has joined #openstack-infra | 08:06 | |
| *** mrmartin has quit IRC | 08:06 | |
| *** arxcruz has joined #openstack-infra | 08:07 | |
| *** hdd has quit IRC | 08:07 | |
| *** annegentle has joined #openstack-infra | 08:07 | |
| *** otter768_ has joined #openstack-infra | 08:08 | |
| *** dtardivel has joined #openstack-infra | 08:08 | |
| *** Jeffrey4l has quit IRC | 08:08 | |
| *** otter76__ has joined #openstack-infra | 08:09 | |
| *** otter768_ has quit IRC | 08:10 | |
| *** acruz has joined #openstack-infra | 08:10 | |
| *** otter768 has quit IRC | 08:10 | |
| *** ddieterly has quit IRC | 08:10 | |
| *** otter76__ has quit IRC | 08:11 | |
| *** annegentle has quit IRC | 08:12 | |
| *** arxcruz has quit IRC | 08:12 | |
| *** gongysh has joined #openstack-infra | 08:13 | |
| *** Jeffrey4l has joined #openstack-infra | 08:14 | |
| *** matrohon has joined #openstack-infra | 08:16 | |
| *** vivekd_ has joined #openstack-infra | 08:17 | |
| *** ihrachys has joined #openstack-infra | 08:18 | |
| *** vivekd has quit IRC | 08:19 | |
| *** vivekd_ is now known as vivekd | 08:19 | |
| *** gongysh has quit IRC | 08:21 | |
| *** HeOS has joined #openstack-infra | 08:23 | |
| therve | AJaeger, Don't we control the mirror from which pip is installed? | 08:24 |
| therve | Syncing requirements will mean a 12+h breakage | 08:25 |
| *** arxcruz has joined #openstack-infra | 08:26 | |
| *** dtantsur|afk is now known as dtantsur | 08:27 | |
| *** jcoufal has joined #openstack-infra | 08:27 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Rename sahara-scenario -> sahara-tests https://review.openstack.org/269815 | 08:27 |
| *** esikachev has joined #openstack-infra | 08:28 | |
| *** acruz has quit IRC | 08:28 | |
| AJaeger | therve: I'm not aware that we're manipulating our mirrors | 08:28 |
| *** fawadk has quit IRC | 08:29 | |
| *** zhurong has quit IRC | 08:29 | |
| therve | Hum okay, I thought things like pypi.region-b.geo-1.openstack.org were maintained by infra | 08:30 |
| AJaeger | therve: yes, they are but I'm not aware that we manipulate these... | 08:31 |
| *** mudassirlatif_ has joined #openstack-infra | 08:31 | |
| *** gongysh has joined #openstack-infra | 08:31 | |
| therve | AJaeger, Well I'm sure it's not usual practice, but there ought to be a way | 08:31 |
| *** zhurong has joined #openstack-infra | 08:32 | |
| *** mudassirlatif has quit IRC | 08:32 | |
| *** mudassirlatif_ is now known as mudassirlatif | 08:32 | |
| *** yfried_ has joined #openstack-infra | 08:35 | |
| *** yfried has quit IRC | 08:35 | |
| *** abregman is now known as abregman|mtg | 08:35 | |
| *** watanabe_isao has quit IRC | 08:36 | |
| *** gongysh has quit IRC | 08:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:40 | |
| *** abregman|mtg is now known as abregman | 08:40 | |
| *** shashank_hegde has quit IRC | 08:40 | |
| *** coolsvap is now known as coolsvap|away | 08:43 | |
| *** jlanoux has joined #openstack-infra | 08:44 | |
| jhesketh | AJaeger: it doesn't look like every pep8/docs job is broken | 08:46 |
| jhesketh | AJaeger: besides, if we pin it with global reqs we'd have to still update every individual project's reqs too no? | 08:46 |
| *** derekh has joined #openstack-infra | 08:47 | |
| *** samuelBartel has joined #openstack-infra | 08:48 | |
| AJaeger | jhesketh: yes, we requirements sync. | 08:49 |
| AJaeger | jhesketh: not every - but some. And for those some I checked and they still use pip 7.2. | 08:50 |
| AJaeger | and succeeded | 08:50 |
| *** dchen has quit IRC | 08:50 | |
| *** zeih has joined #openstack-infra | 08:50 | |
| AJaeger | https://review.openstack.org/265627 and https://review.openstack.org/268687 are two broken ones | 08:50 |
| *** arxcruz has quit IRC | 08:51 | |
| *** fawadkhaliq has joined #openstack-infra | 08:51 | |
| esikachev | AJaeger: hi! take a look on https://review.openstack.org/#/c/269815/ | 08:51 |
| AJaeger | esikachev: why? | 08:52 |
| *** arxcruz has joined #openstack-infra | 08:52 | |
| AJaeger | esikachev: let me -2 that ;) | 08:52 |
| AJaeger | esikachev: change is fine but needs gerrit downtime, just -2 for procedural reasons | 08:53 |
| AJaeger | jhesketh: I don't understand why some jobs get pip 8 and other 7.2... | 08:54 |
| jhesketh | AJaeger: are we better to just pin individual projects if they need it? | 08:54 |
| esikachev | AJaeger: This was discussed at a meeting on Tuesday | 08:54 |
| jhesketh | hmm, maybe 8 hasn't gotten to every mirror yet | 08:54 |
| AJaeger | jhesketh: if we pin individual projects, the check-requirements job will fail | 08:56 |
| *** fawadkhaliq has quit IRC | 08:56 | |
| AJaeger | esikachev: we didn't have time to discuss this yesterday - and it still cannot merge now, it needs a gerrit downtime with a database migration ;( | 08:57 |
| jhesketh | I thought it was only if it's not compatible with the global reqs | 08:57 |
| * jhesketh should look at the check requirements job again | 08:57 | |
| *** nikil89_ has joined #openstack-infra | 08:57 | |
| *** nikil89_ is now known as nandhu | 08:57 | |
| *** nandhu is now known as naggappan | 08:57 | |
| AJaeger | pip is not in the requirements at all of any of the projects, so even doing the sync will not update any project ;( | 08:58 |
| AJaeger | It would still need manual updates | 08:58 |
| *** kzaitsev_mb has quit IRC | 08:59 | |
| esikachev | AJaeger: when the patch can be merged? | 08:59 |
| jhesketh | AJaeger: is it pip and setuptools on the nodes maybe | 09:00 |
| jhesketh | would explain different versions | 09:00 |
| *** yfried has joined #openstack-infra | 09:00 | |
| AJaeger | esikachev: it needs to be scheduled with a global 30 mins downtime for gerrit - nobody will be able to access review.o.o at that time. And it's not scheduled yet. | 09:01 |
| *** yamahata has quit IRC | 09:01 | |
| *** fhubik has joined #openstack-infra | 09:01 | |
| esikachev | AJaeger: ok, thanks | 09:02 |
| *** mudassirlatif has quit IRC | 09:02 | |
| AJaeger | jhesketh: I'm lost here and won't have time to further investigate ;( | 09:03 |
| *** yfried_ has quit IRC | 09:04 | |
| *** ociuhandu has joined #openstack-infra | 09:05 | |
| *** ddieterly has joined #openstack-infra | 09:07 | |
| *** flwang1 has joined #openstack-infra | 09:07 | |
| *** zeih has quit IRC | 09:11 | |
| *** ddieterly has quit IRC | 09:12 | |
| *** dtantsur is now known as dtantsur|brb | 09:14 | |
| *** acruz has joined #openstack-infra | 09:15 | |
| *** jistr has joined #openstack-infra | 09:16 | |
| AJaeger | jhesketh: see http://logs.openstack.org/63/270063/1/check/gate-openstack-doc-tools-releasenotes/0383c50/console.html and change 270063 - pinning might not help at all ;( | 09:17 |
| *** arxcruz has quit IRC | 09:18 | |
| jhesketh | AJaeger: yes I think it's the pip version installed on the node itself | 09:19 |
| jhesketh | should possibly try and figure out what tox doesn't like about pip8 | 09:19 |
| *** dizquierdo has joined #openstack-infra | 09:21 | |
| *** hashar has joined #openstack-infra | 09:23 | |
| *** fawadkhaliq has joined #openstack-infra | 09:26 | |
| *** BobBall_AWOL is now known as BobBall | 09:30 | |
| mrunge | mtreinish, jhesketh are you sure your fix for kilo works? https://review.openstack.org/#/c/269974/ | 09:31 |
| ianw | AJaeger: yeah, i think the conversation will have to be had probably at https://github.com/pypa/pip/issues/3384 | 09:31 |
| jhesketh | mrunge: yeah that failed because of the circular dependency so we force merged it | 09:31 |
| mrunge | it looks like all kilo projects are broken right now | 09:31 |
| mescanef | hello openstack infra. if any root around please review&approve https://review.openstack.org/#/c/263334/ :-) | 09:32 |
| ianw | mrunge: a fix to kilo merged, and it passed CI. do you have an example? | 09:32 |
| mrunge | ianw, yes, just a sec | 09:32 |
| mrunge | http://logs.openstack.org/periodic-stable/periodic-nova-python27-kilo/25109b8/ | 09:33 |
| mrunge | http://logs.openstack.org/periodic-stable/periodic-ceilometer-python27-kilo/a9b02a6/ | 09:33 |
| mrunge | http://logs.openstack.org/periodic-stable/periodic-horizon-python27-kilo/89e2b50/ | 09:33 |
| mrunge | ianw, and more | 09:33 |
| jhesketh | those aren't the devstack problem that was fixed | 09:33 |
| ianw | mrunge: oh ... non devstack jobs. yeah | 09:33 |
| jhesketh | mrunge: they looked related to the issue ianw linked to on github | 09:34 |
| ianw | yeah, they are | 09:34 |
| *** Jeffrey4l has quit IRC | 09:34 | |
| jhesketh | which is the same as AJaeger's issues | 09:34 |
| *** vgridnev has quit IRC | 09:34 | |
| *** vgridnev has joined #openstack-infra | 09:37 | |
| ianw | hmmm, that's in a virtualenv?! is pip detecting installed packages *outside*? | 09:38 |
| openstackgerrit | Dina Belova proposed openstack-infra/project-config: Add py27 job for performance-docs https://review.openstack.org/270088 | 09:41 |
| *** melwitt_ has joined #openstack-infra | 09:42 | |
| *** melwitt_ has quit IRC | 09:43 | |
| odyssey4me | hmm, morning all | 09:46 |
| odyssey4me | can tox be configured to use an isolated venv? | 09:46 |
| odyssey4me | if that works, that should make the conflicts with distro packages in the gate a moot point, mostly | 09:47 |
| *** flwang1 has quit IRC | 09:50 | |
| *** vivekd_ has joined #openstack-infra | 09:52 | |
| *** flwang1 has joined #openstack-infra | 09:53 | |
| *** vivekd has quit IRC | 09:54 | |
| *** vivekd_ is now known as vivekd | 09:54 | |
| *** rguillebert has joined #openstack-infra | 10:01 | |
| *** jaosorior has joined #openstack-infra | 10:02 | |
| *** abregman is now known as abregman|foody | 10:04 | |
| *** jaosorior has quit IRC | 10:04 | |
| *** jaosorior has joined #openstack-infra | 10:04 | |
| *** dizquierdo has quit IRC | 10:05 | |
| *** annegentle has joined #openstack-infra | 10:08 | |
| *** ddieterly has joined #openstack-infra | 10:09 | |
| *** jaosorior_ has joined #openstack-infra | 10:09 | |
| *** jaosorior has quit IRC | 10:11 | |
| *** Qiming has quit IRC | 10:12 | |
| *** annegentle has quit IRC | 10:13 | |
| *** ddieterly has quit IRC | 10:13 | |
| *** zeih has joined #openstack-infra | 10:14 | |
| *** dtantsur|brb is now known as dtantsur | 10:15 | |
| *** electrofelix has joined #openstack-infra | 10:16 | |
| ianw | AJaeger: somehow, those jobs are breaking out of their virtualenv | 10:21 |
| *** zeih has quit IRC | 10:24 | |
| odyssey4me | ianw yeah, I see that tox 2 automatically isolates the venv... so how is it that the OS package installed python bits are bing picked up at all | 10:24 |
| *** e0ne has joined #openstack-infra | 10:25 | |
| odyssey4me | perhaps the issue is that pip's installed on the host and it's able to see all things? | 10:25 |
| odyssey4me | ie it's not properly isolated to the venv | 10:25 |
| *** fawadkhaliq has quit IRC | 10:26 | |
| *** fawadkhaliq has joined #openstack-infra | 10:26 | |
| markus_z | craige: You are working on the maniphest migration, aren't you? I try to main the bug list for Nova and I use scripts to interact with Launchpad. | 10:26 |
| markus_z | craige: My hope was that Maniphest provides the same possibilities. | 10:27 |
| markus_z | craige: I've seen that you asked for input in yesterdays infra IRC meeting. That's why I'm reaching out to you. | 10:27 |
| *** links has quit IRC | 10:28 | |
| craige | heya markus_z | 10:28 |
| craige | I've ot a dev site you can hit up here: http://os03.mcwhirter.io/ | 10:30 |
| craige | (uses Launchpad creds). | 10:30 |
| craige | So you can have a poke around and see if it's doing what you need. | 10:31 |
| *** rossella_s has quit IRC | 10:31 | |
| *** [1]Thelo has joined #openstack-infra | 10:31 | |
| markus_z | craige: yesss, exactly what I want :) thanks! | 10:31 |
| craige | It's got an import of all Storyboard tickets from mid December IIRC. | 10:31 |
| *** rossella_s has joined #openstack-infra | 10:31 | |
| craige | That site is stable, as doing dev elsewhere, so it will show you where migration was up to mid-December markus_z | 10:32 |
| markus_z | yep, I see the tickets, that's enough data to see if/how it works for me | 10:32 |
| craige | Excellent :-) | 10:33 |
| markus_z | craige: If I have feedback, should I ping you here or post on the ML or otherwise? | 10:33 |
| *** Thelo has quit IRC | 10:33 | |
| *** [1]Thelo is now known as Thelo | 10:33 | |
| craige | Either works markus_z. I'm on AEST so I'm about gone for the night. | 10:34 |
| * jhesketh is restarting etherpad to upgrade kernel | 10:34 | |
| markus_z | craige: ok, cool, takes me some days. thanks for the help! | 10:35 |
| *** kzaitsev_mb has joined #openstack-infra | 10:35 | |
| craige | No rush markus_z. I look forward to hearing from you. | 10:36 |
| openstackgerrit | Dmitry Tantsur proposed openstack/requirements: Bump upper-constraints for python-ironic-inspector-client to 1.4.0 https://review.openstack.org/270113 | 10:38 |
| *** Daisy has quit IRC | 10:39 | |
| *** jaosorior_ is now known as jaosorior | 10:39 | |
| *** ldnunes has joined #openstack-infra | 10:39 | |
| ianw | AJaeger / odyssey4me : http://lists.openstack.org/pipermail/openstack-dev/2016-January/084518.html | 10:41 |
| *** abregman|foody is now known as abregman | 10:41 | |
| ianw | on the argparse leaking issue | 10:41 |
| lennyb | Infra, is git OK? I am getting timeout error lately [ERROR] /jenkins/workspace/Neutron-ML2-MLNX/devstack/functions-common:607 git call failed: [git clone git://git.openstack.org/openstack/neutron.git /opt/stack/neutron] | 10:44 |
| lennyb | 11:53:25 2016-01-20 09:54:51.292 | Error on exit | 10:44 |
| *** arxcruz has joined #openstack-infra | 10:44 | |
| *** acruz has quit IRC | 10:45 | |
| odyssey4me | hmm, ianw AJaeger I seem to have a docs job that's actually passing: https://review.openstack.org/269668 | 10:45 |
| Zara | markus_z: btw, the storyboard team are interested in your requirements, since even if you don't intend to use storyboard, you're probably not the only person who'll want that feature in a task-tracker. so any info you can post here about the kinds of things you need to be able to do would be useful for us. :) | 10:47 |
| *** yamamoto_ has quit IRC | 10:49 | |
| ianw | odyssey4me: sure that's using a virtualenv to build though? | 10:50 |
| *** yamamoto has joined #openstack-infra | 10:50 | |
| odyssey4me | ianw well, it's using tox and is configured much like most other docs jobs | 10:50 |
| *** rfolco has joined #openstack-infra | 10:51 | |
| odyssey4me | I'm trying to get our specs repo working by changing requirements to see if it works... as that appears to be the only real difference in the configs for the docs job | 10:51 |
| *** links has joined #openstack-infra | 10:51 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new repo for fuel-plugin-zaqar in /Openstack https://review.openstack.org/270121 | 10:52 |
| ianw | odyssey4me: yeah, i dunno sorry. definite problem with trusty virtualenvs and argparse, but maybe you have a different platform or something funky | 10:52 |
| odyssey4me | ianw heh, interesting - that works: https://review.openstack.org/270114 | 10:54 |
| *** fhubik is now known as fhubik_brb | 10:54 | |
| jhesketh | craige: noting that you're around.. are you working on phabricator.openstack.org or is it safe for me to restart it | 10:56 |
| * jhesketh restarts it as there doesn't seem to be anything running atm | 10:59 | |
| Zara | email notifications in storyboard work everywhere except storyboard.openstack.org. SotK tried deploying with puppet, pointing to a server running locally, and everything worked fine, but we're seeing nothing from s.o.o. anyone have any idea what could be going on? it's hard for us to debug without access to the box. | 10:59 |
| *** vgridnev has quit IRC | 11:01 | |
| *** _et_ has quit IRC | 11:03 | |
| *** gongysh has joined #openstack-infra | 11:04 | |
| *** yamamoto has quit IRC | 11:04 | |
| *** rcernin has quit IRC | 11:05 | |
| *** links has quit IRC | 11:05 | |
| *** rcernin has joined #openstack-infra | 11:07 | |
| *** sfinucan has joined #openstack-infra | 11:07 | |
| *** Qiming has joined #openstack-infra | 11:08 | |
| openstackgerrit | Alexander Maretskiy proposed openstack-infra/project-config: Increase timeout for job gate-rally-dsvm-cli https://review.openstack.org/270134 | 11:09 |
| *** ddieterly has joined #openstack-infra | 11:10 | |
| *** fhubik_brb is now known as fhubik | 11:13 | |
| *** rcernin is now known as rcernin|lunch | 11:13 | |
| *** ddieterly has quit IRC | 11:14 | |
| *** vgridnev has joined #openstack-infra | 11:15 | |
| *** kashyap has joined #openstack-infra | 11:16 | |
| dstufft | ianw: I am now awake again, (but taking Alyssa to the bus stop in 10 minutes or so) | 11:18 |
| *** zhurong has quit IRC | 11:18 | |
| *** aysyd has joined #openstack-infra | 11:20 | |
| *** daemontool has joined #openstack-infra | 11:21 | |
| odyssey4me | ianw AJaeger dstufft ok, so as far as I can see the tox docs/specs jobs that fail are due to the argparse issue which ianw has raised - in our case we removed testtools from requirements.txt in the specs repository and the repo builds properly again | 11:21 |
| odyssey4me | other jobs which have more complex requirements and don't use venvs are going to have a bad day :) | 11:22 |
| dstufft | I haven't investigated it yet, but the argparse thin, on the surface, just seems like a full out bug and shouldn't be hitting this error case | 11:22 |
| dstufft | argparse in a virtual env is being installed as a wheel | 11:22 |
| *** ashishb has quit IRC | 11:23 | |
| *** aysyd has quit IRC | 11:23 | |
| *** flwang1 has quit IRC | 11:24 | |
| odyssey4me | dstufft agreed, that does seem to be a proper bug | 11:24 |
| *** aysyd has joined #openstack-infra | 11:24 | |
| *** wei-wang has quit IRC | 11:25 | |
| ianw | dstufft: yeah ... i mean probably for openstack now 2.6 is gone, who needs argparse ... but i guess it gets dragged in from other things that still support 2.6 and have it in requirements? (testtools as odyssey4me described) | 11:30 |
| *** zhurong has joined #openstack-infra | 11:30 | |
| dstufft | presumably | 11:31 |
| dstufft | they could optionally depend on argparse for 2.6 only | 11:31 |
| *** placko_ has joined #openstack-infra | 11:31 | |
| markus_z | Zara: ok, will do. | 11:32 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add output of top to hostinfo https://review.openstack.org/270139 | 11:32 |
| *** links has joined #openstack-infra | 11:32 | |
| craige | I don;t have access to that host jhesketh and it's not being used for dev work. | 11:33 |
| *** ashishb has joined #openstack-infra | 11:33 | |
| *** changzhi has quit IRC | 11:33 | |
| jhesketh | craige: cool, thanks. Doesn't look like it was/is doing much to be honest | 11:33 |
| markus_z | Zara: As a bit of a background, I need a way to collect timeseries data in an automated way to fill a dashboard for bugs, e.g. my local PoC: http://www.tiikoni.com/tis/view/?id=ccdf40c | 11:34 |
| markus_z | Zara: But I will play around with maniphest and create a list of requirements and let you know | 11:35 |
| AJaeger | argparse is a requirement of oslo.config, so if we want to block that, we first need to release olso.config... | 11:35 |
| craige | Now that I know it exists, I will wna to use it for closer to production testing, jhesketh :-) | 11:35 |
| jhesketh | :-) | 11:35 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make large-ops jobs non voting https://review.openstack.org/270141 | 11:36 |
| openstackgerrit | Angus Lees proposed openstack-infra/project-config: Create new tempest-dsvm-full-head job-template https://review.openstack.org/270143 | 11:38 |
| Zara | markus_z: thank you, we really appreciate it! :) requirements are always helpful, and it's also very useful to know what goals you're trying to achieve. so thanks for the background, too! | 11:38 |
| *** links has quit IRC | 11:38 | |
| jhesketh | #status alert review.openstack.org is being restarted to apply patches | 11:39 |
| openstackstatus | jhesketh: sending alert | 11:39 |
| -openstackstatus- NOTICE: review.openstack.org is being restarted to apply patches | 11:42 | |
| *** ChanServ changes topic to "review.openstack.org is being restarted to apply patches" | 11:42 | |
| *** openstackgerrit has quit IRC | 11:43 | |
| *** openstackgerrit has joined #openstack-infra | 11:44 | |
| jhesketh | ... and back | 11:44 |
| openstackstatus | jhesketh: finished sending alert | 11:45 |
| jhesketh | heh, beat the alert | 11:45 |
| AJaeger | congrats! | 11:45 |
| Zara | :) | 11:46 |
| jhesketh | I did delay actually restarting until at least #openstack-dev had the message | 11:46 |
| *** zeih has joined #openstack-infra | 11:46 | |
| *** gongysh has quit IRC | 11:46 | |
| AJaeger | jhesketh: so time to send the finish #status? | 11:49 |
| AJaeger | Btw. A #notice would have been enough in this case, #status is for longer periods of time | 11:50 |
| jhesketh | AJaeger: woops, I meant to send a notice | 11:50 |
| jhesketh | hence forgetting to send ok | 11:50 |
| *** fhubik is now known as fhubik_brb | 11:50 | |
| jhesketh | #status ok Restart done, review.openstack.org is available | 11:51 |
| openstackstatus | jhesketh: sending ok | 11:51 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Add Create Story Button in Worklists https://review.openstack.org/269656 | 11:51 |
| *** fhubik_brb is now known as fhubik | 11:52 | |
| odyssey4me | AJaeger yeah, I see that oslo.config requiring argparse in turn affects installations of all the clients | 11:53 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 11:53 | |
| *** zeih has quit IRC | 11:53 | |
| -openstackstatus- NOTICE: Restart done, review.openstack.org is available | 11:53 | |
| *** vgridnev has quit IRC | 11:54 | |
| tonyb | So, trying to catch up where is the pip8 issue at? Anythign I can do to help ? | 11:54 |
| *** vgridnev has joined #openstack-infra | 11:55 | |
| *** gongysh has joined #openstack-infra | 11:55 | |
| jhesketh | tonyb: https://github.com/pypa/pip/issues/3384 | 11:56 |
| openstackstatus | jhesketh: finished sending ok | 11:56 |
| jhesketh | there are suggested solutions and work arounds | 11:56 |
| AJaeger | odyssey4me, dstufft: IS the a proper solution of oslo.config: https://review.openstack.org/270149 ? | 11:56 |
| jhesketh | tonyb: we could perhaps force nodes to install an older version | 11:56 |
| jhesketh | I'm not sure how many nodes have updated to 8 yet | 11:56 |
| tonyb | jhesketh: ok, I'll read the URL. It seems like forcing it would be good. | 11:57 |
| dstufft | AJaeger: If you don't support 2.6 there's no reason to keep argparse, if you dos upport 2.6 you can make that conditional | 11:57 |
| tonyb | I know all the periododic-stable jobs barfed because of it | 11:57 |
| AJaeger | dstufft: oslo.config HEAD does not support 2.6 it anymore - so reviews are welcome on 270149 | 11:58 |
| AJaeger | it still needs a release of oslo.config to help us, but it might be one piece of the puzzle | 11:58 |
| *** fawadkhaliq has quit IRC | 12:02 | |
| *** _amrith_ is now known as amrith | 12:03 | |
| AJaeger | but oslo.config imports netaddr - and probably will fail due to that as well ;( | 12:04 |
| *** amotoki has joined #openstack-infra | 12:04 | |
| *** gongysh has quit IRC | 12:07 | |
| *** hashar has quit IRC | 12:08 | |
| *** vgridnev has quit IRC | 12:08 | |
| *** annegentle has joined #openstack-infra | 12:09 | |
| *** vgridnev has joined #openstack-infra | 12:10 | |
| *** ddieterly has joined #openstack-infra | 12:11 | |
| openstackgerrit | Kirill Zaitsev proposed openstack-infra/project-config: Add translation jobs to murano-dashboard https://review.openstack.org/262073 | 12:12 |
| *** annegentle has quit IRC | 12:13 | |
| *** zeih has joined #openstack-infra | 12:13 | |
| *** boris-42 has quit IRC | 12:13 | |
| *** ddieterly has quit IRC | 12:15 | |
| *** exploreshaifali has joined #openstack-infra | 12:15 | |
| *** zeih has quit IRC | 12:18 | |
| samueldmq | morning all | 12:18 |
| samueldmq | stackalytics.openstack.org is giving me 500 | 12:19 |
| *** achanda has joined #openstack-infra | 12:19 | |
| jhesketh | samueldmq: yes, that's my fault... restarted for a kernel upgrade and it hasn't come up nicely | 12:20 |
| jhesketh | samueldmq: currently looking at it | 12:21 |
| jhesketh | http://stackalytics.com/ is working though | 12:21 |
| jhesketh | and if anybody here knows much about stackalytics, ping me | 12:21 |
| *** Jeffrey4l has joined #openstack-infra | 12:21 | |
| samueldmq | jhesketh: nice, I made stackalytics.o.o my choice, I will wait for it, thanks | 12:22 |
| odyssey4me | AJaeger I take it that netaddr is also a legacy requirement to have that's included in python 2.7? | 12:23 |
| *** jaypipes has joined #openstack-infra | 12:23 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/project-config: Make TripleO puppet check jobs voting https://review.openstack.org/270165 | 12:24 |
| *** gordc has joined #openstack-infra | 12:25 | |
| *** doug-fish has joined #openstack-infra | 12:25 | |
| *** yamamoto has joined #openstack-infra | 12:28 | |
| *** ccarmack has joined #openstack-infra | 12:28 | |
| *** ashishb has quit IRC | 12:28 | |
| *** yamamoto has quit IRC | 12:29 | |
| *** yamamoto_ has joined #openstack-infra | 12:29 | |
| *** kzaitsev_mb has quit IRC | 12:30 | |
| *** yamamoto has joined #openstack-infra | 12:31 | |
| *** otter768 has joined #openstack-infra | 12:31 | |
| *** hashar has joined #openstack-infra | 12:32 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add sahara scenario gate jobs in sahara-scenario https://review.openstack.org/270166 | 12:33 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make large-ops jobs non voting https://review.openstack.org/270141 | 12:33 |
| *** yamamoto_ has quit IRC | 12:34 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add sahara scenario gate jobs in sahara-scenario https://review.openstack.org/270166 | 12:34 |
| *** rhallisey has joined #openstack-infra | 12:35 | |
| *** otter768 has quit IRC | 12:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:37 | |
| *** zeih has joined #openstack-infra | 12:37 | |
| *** dizquierdo has joined #openstack-infra | 12:38 | |
| *** claudiub has joined #openstack-infra | 12:39 | |
| *** fhubik is now known as fhubik_brb | 12:41 | |
| *** ddieterly has joined #openstack-infra | 12:41 | |
| *** weshay_xchat has joined #openstack-infra | 12:42 | |
| *** _et_ has joined #openstack-infra | 12:43 | |
| *** SongmingYan has joined #openstack-infra | 12:44 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add sahara scenario gate jobs in sahara-scenario https://review.openstack.org/270166 | 12:45 |
| *** vgridnev has quit IRC | 12:46 | |
| *** thorst has joined #openstack-infra | 12:46 | |
| *** baoli has joined #openstack-infra | 12:47 | |
| sdague | now that the governance change landed, can we get - https://review.openstack.org/#/c/267133/ reviewed so that Nova can start using constraints in our primary jobs | 12:47 |
| *** dims has joined #openstack-infra | 12:49 | |
| mordred | sdague: lgtm | 12:49 |
| *** SongmingYan has quit IRC | 12:50 | |
| *** moravec has joined #openstack-infra | 12:51 | |
| *** baoli_ has joined #openstack-infra | 12:52 | |
| *** achanda has quit IRC | 12:54 | |
| *** rlandy has joined #openstack-infra | 12:55 | |
| *** baoli has quit IRC | 12:55 | |
| *** vgridnev has joined #openstack-infra | 12:55 | |
| pedroalvarez | Hi zaro, I know this happened long time ago, but I've just discovered the its-storyboard plugin for gerrit. | 12:56 |
| pedroalvarez | I note that in your last commit you said: "Updating of story status will be implemented in a follow on change." | 12:56 |
| pedroalvarez | did that ever happened? I'd be intereste on that | 12:56 |
| *** Jeffrey4l has quit IRC | 12:59 | |
| *** davidhadas has joined #openstack-infra | 12:59 | |
| *** wei-wang has joined #openstack-infra | 13:02 | |
| AJaeger | odyssey4me: don't know... | 13:03 |
| hashar | pedroalvarez: there is apparently no follow up change in Gerrit https://gerrit-review.googlesource.com/#/q/project:plugins/its-storyboard | 13:04 |
| *** matrohon has quit IRC | 13:04 | |
| pedroalvarez | yeah.. | 13:04 |
| *** matrohon has joined #openstack-infra | 13:05 | |
| *** rcernin|lunch is now known as rcernin | 13:07 | |
| *** dkranz has joined #openstack-infra | 13:09 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add sahara scenario gate jobs in sahara-scenario https://review.openstack.org/270166 | 13:09 |
| *** vgridnev has quit IRC | 13:11 | |
| *** kzaitsev_mb has quit IRC | 13:13 | |
| *** doug-fish has quit IRC | 13:14 | |
| *** ccarmack has left #openstack-infra | 13:15 | |
| *** zeih has quit IRC | 13:15 | |
| *** doug-fish has joined #openstack-infra | 13:16 | |
| *** Jeffrey4l has joined #openstack-infra | 13:17 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add sahara scenario gate jobs in sahara-scenario https://review.openstack.org/270166 | 13:17 |
| *** Guest35147 has quit IRC | 13:18 | |
| *** daemontool has quit IRC | 13:18 | |
| mescanef | mordred: hello. could I ask you for one quick review one one patch and wf+1? it already has +2CR | 13:19 |
| *** sulo has joined #openstack-infra | 13:19 | |
| SotK | Is there anyone around who could assist me in debugging a problem with storyboard.openstack.org that we can't seem to reproduce anywhere else? | 13:20 |
| *** vgridnev has joined #openstack-infra | 13:20 | |
| *** dguitarbite has quit IRC | 13:23 | |
| *** tlian has joined #openstack-infra | 13:24 | |
| *** trown|outttypeww is now known as trown | 13:24 | |
| *** ayoung has quit IRC | 13:24 | |
| *** rfolco has quit IRC | 13:25 | |
| Zara | for background: SotK has now tried deploying a storyboard instance with puppet and pointing to a remote database server, and email notifications still work. but not on s.o.o! :( | 13:25 |
| *** rfolco has joined #openstack-infra | 13:25 | |
| *** dprince has joined #openstack-infra | 13:26 | |
| *** dguitarbite has joined #openstack-infra | 13:28 | |
| *** exploreshaifali has quit IRC | 13:29 | |
| *** vgridnev has quit IRC | 13:31 | |
| SotK | indeed, I can't see anything wrong with the config (and the same config except private credentials is working on my test instance) | 13:32 |
| SotK | I don't think I can debug any further without having some help | 13:32 |
| *** dslev has joined #openstack-infra | 13:34 | |
| *** Ramanjaneya_ has quit IRC | 13:34 | |
| mordred | mescanef: you can ... but it might help if you provided the link :) | 13:35 |
| sdague | AJaeger / pleia2 someone asked in #openstack-nova | 13:36 |
| sdague | about i18n | 13:36 |
| sdague | python setup.py compile_catalog in nova throws errors | 13:36 |
| sdague | apparently due to bad translations coming in | 13:36 |
| AJaeger | sdague: let me check | 13:36 |
| sdague | error: nova/locale/es/LC_MESSAGES/nova.po:2127: placeholders are incompatible | 13:36 |
| sdague | error: nova/locale/es/LC_MESSAGES/nova.po:2223: placeholders are incompatible | 13:36 |
| sdague | error: nova/locale/es/LC_MESSAGES/nova.po:3807: incompatible format for placeholder u'ports': 's' and 'd' are not compatible | 13:36 |
| sdague | it seems like that should be validated before proposed | 13:37 |
| sdague | any example | 13:37 |
| sdague | #, python-format | 13:37 |
| sdague | msgid "The number of defined ports: %(ports)d is over the limit: %(quota)d" | 13:37 |
| sdague | msgstr "El número de puertos definidos: %(ports)s es más del lÃmite: %(quota)d" | 13:37 |
| sdague | msgid "Invalid range expression %r" | 13:38 |
| sdague | msgstr "Expresión de intérvalo inválida %" | 13:38 |
| sdague | yeh, someone is typoing replacements when translating | 13:38 |
| AJaeger | sdague: keystone tests in pep8: msgfmt --check-format -o /dev/null, apparently nova does not. Shall I propose a change? | 13:38 |
| *** eharney has quit IRC | 13:39 | |
| AJaeger | sdague: I can change upstream sources ;) | 13:39 |
| sdague | AJaeger: after someone fixes upstream :) | 13:39 |
| AJaeger | sdague: let me propose the change and then approve once upstream is fixed (we can "recheck") | 13:39 |
| AJaeger | first upstream... | 13:39 |
| sdague | AJaeger: sounds good | 13:40 |
| *** moravec has quit IRC | 13:41 | |
| sdague | AJaeger: for reference - http://paste.openstack.org/show/484403/ | 13:42 |
| sdague | 14 fatal errors in our translations right now | 13:42 |
| AJaeger | sdague: https://review.openstack.org/270190 | 13:42 |
| AJaeger | found the same list ;) | 13:43 |
| AJaeger | a bit more than expected ;( | 13:43 |
| *** dslev has quit IRC | 13:43 | |
| *** aysyd has quit IRC | 13:43 | |
| sdague | yeh, though is there a reason we don't use 'python setup.py compile_catalog' instead of that? | 13:44 |
| *** amrith is now known as _amrith_ | 13:44 | |
| *** annegentle has joined #openstack-infra | 13:44 | |
| *** MCoLo has quit IRC | 13:44 | |
| *** MCoLo has joined #openstack-infra | 13:45 | |
| *** aysyd has joined #openstack-infra | 13:45 | |
| mescanef | mordred: please look https://review.openstack.org/#/c/263334/ | 13:46 |
| *** esikache1 has joined #openstack-infra | 13:46 | |
| AJaeger | sdague: it's quicker ;) but feel free to propose an alternative patch. I just don't want to test that right now... | 13:46 |
| *** ihrachys_ has joined #openstack-infra | 13:47 | |
| *** hashar_ has joined #openstack-infra | 13:47 | |
| * AJaeger updated the validation options of nova to avoid these problems... | 13:47 | |
| *** jcoufal_ has joined #openstack-infra | 13:47 | |
| sdague | AJaeger: yeh, no problem, I'm just trying to understand why python setup.py compile_catalog misses issues | 13:48 |
| sdague | so your approach is better | 13:48 |
| *** esikache1 has quit IRC | 13:48 | |
| *** cbader_ has joined #openstack-infra | 13:48 | |
| *** ociuhandu_ has joined #openstack-infra | 13:48 | |
| *** phschwartz_ has joined #openstack-infra | 13:49 | |
| sdague | AJaeger: in an ideal world, those tests would get run in zanata before things could be saved there | 13:49 |
| sdague | so it doesn't keep proposing broken translations | 13:49 |
| *** crinkle_ has joined #openstack-infra | 13:49 | |
| *** taron1 has joined #openstack-infra | 13:49 | |
| AJaeger | sdague: that's waht I meant with "updated validation options" - these are now hard errors and not warnings | 13:49 |
| *** mahatic_ has joined #openstack-infra | 13:49 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pass timeout through to floating ip creation https://review.openstack.org/270193 | 13:50 |
| AJaeger | I fixed the strings with: Enable errros for this, search for errors ;) | 13:50 |
| *** beisner- has joined #openstack-infra | 13:51 | |
| mordred | mescanef: done | 13:51 |
| *** nihilife1 has joined #openstack-infra | 13:51 | |
| *** dosaboy_ has joined #openstack-infra | 13:51 | |
| mescanef | mordred: thanks! | 13:51 |
| *** lifeless_ has joined #openstack-infra | 13:51 | |
| *** tcammann1 has joined #openstack-infra | 13:51 | |
| *** kzaitsev1ws has joined #openstack-infra | 13:51 | |
| *** frickler_ has joined #openstack-infra | 13:51 | |
| *** tonyb_ has joined #openstack-infra | 13:51 | |
| *** Odd_Blok1 has joined #openstack-infra | 13:51 | |
| *** jokke__ has joined #openstack-infra | 13:51 | |
| *** j_king_ has joined #openstack-infra | 13:51 | |
| *** jkt_ has joined #openstack-infra | 13:51 | |
| AJaeger | sdague: all fixed - tomorrows import should be fine... | 13:51 |
| *** gothicmi1dfood has joined #openstack-infra | 13:51 | |
| sdague | AJaeger: ok, great | 13:51 |
| *** dhp_ has joined #openstack-infra | 13:52 | |
| *** vgridnev has joined #openstack-infra | 13:52 | |
| AJaeger | sdague: so, tomorrow I rebase my change on top of the translation import and ask for reviews;) | 13:53 |
| sdague | AJaeger: yep | 13:53 |
| *** MCoLo has left #openstack-infra | 13:53 | |
| *** coreycb` has joined #openstack-infra | 13:54 | |
| *** ianw_ has joined #openstack-infra | 13:54 | |
| *** hughhalf_ has joined #openstack-infra | 13:54 | |
| *** bookwar_ has joined #openstack-infra | 13:54 | |
| *** itsuugo_ has joined #openstack-infra | 13:54 | |
| *** GonZo2000 has joined #openstack-infra | 13:54 | |
| *** GonZo2000 has quit IRC | 13:54 | |
| *** GonZo2000 has joined #openstack-infra | 13:54 | |
| *** kambiz_ has joined #openstack-infra | 13:54 | |
| *** purp_too has joined #openstack-infra | 13:55 | |
| *** raginbaj- has joined #openstack-infra | 13:55 | |
| *** electrichead has joined #openstack-infra | 13:55 | |
| *** robcresswell_ has joined #openstack-infra | 13:55 | |
| *** schang_ has joined #openstack-infra | 13:55 | |
| *** toabctl_ has joined #openstack-infra | 13:55 | |
| *** bknudson_ has joined #openstack-infra | 13:55 | |
| *** krotscheck_ has joined #openstack-infra | 13:55 | |
| *** htruta` has joined #openstack-infra | 13:55 | |
| *** diltram_ has joined #openstack-infra | 13:55 | |
| *** hrubi_ has joined #openstack-infra | 13:55 | |
| *** mkoderer_ has joined #openstack-infra | 13:55 | |
| *** bradjones_ has joined #openstack-infra | 13:55 | |
| *** leifmadsen_ has joined #openstack-infra | 13:55 | |
| *** electrichead is now known as Guest10164 | 13:55 | |
| *** cinerama` has joined #openstack-infra | 13:55 | |
| AJaeger | mordred: could you review this jeepyb change, please? https://review.openstack.org/#/c/253229/ | 13:55 |
| *** cfarquhar_ has joined #openstack-infra | 13:55 | |
| *** bradjones_ is now known as Guest7791 | 13:55 | |
| *** kushal has quit IRC | 13:55 | |
| *** ddieterly has quit IRC | 13:56 | |
| *** hashar has quit IRC | 13:56 | |
| *** ociuhandu has quit IRC | 13:56 | |
| *** ihrachys has quit IRC | 13:56 | |
| *** notmorgan has quit IRC | 13:56 | |
| *** diltram has quit IRC | 13:56 | |
| *** bknudson has quit IRC | 13:56 | |
| *** itsuugo has quit IRC | 13:56 | |
| *** zzzeek has quit IRC | 13:56 | |
| *** dougshelley66 has quit IRC | 13:56 | |
| *** Sam-I-Am has quit IRC | 13:56 | |
| *** beisner has quit IRC | 13:56 | |
| *** ianw has quit IRC | 13:56 | |
| *** d0ugal has quit IRC | 13:56 | |
| *** jkt has quit IRC | 13:56 | |
| *** mnaser has quit IRC | 13:56 | |
| *** dosaboy has quit IRC | 13:56 | |
| *** akscram has quit IRC | 13:56 | |
| *** mahatic has quit IRC | 13:56 | |
| *** gothicmindfood has quit IRC | 13:56 | |
| *** raginbajin has quit IRC | 13:56 | |
| *** redrobot has quit IRC | 13:56 | |
| *** phschwartz has quit IRC | 13:56 | |
| *** schang has quit IRC | 13:56 | |
| *** hughhalf has quit IRC | 13:56 | |
| *** anteaya has quit IRC | 13:56 | |
| *** tcammann has quit IRC | 13:56 | |
| *** Odd_Bloke has quit IRC | 13:56 | |
| *** frickler has quit IRC | 13:56 | |
| *** plol has quit IRC | 13:56 | |
| *** vponomaryov has quit IRC | 13:56 | |
| *** clarkb has quit IRC | 13:56 | |
| *** taron has quit IRC | 13:56 | |
| *** coreycb has quit IRC | 13:56 | |
| *** htruta has quit IRC | 13:56 | |
| *** jcoufal has quit IRC | 13:56 | |
| *** esikachev has quit IRC | 13:56 | |
| *** bradjones has quit IRC | 13:56 | |
| *** nmagnezi has quit IRC | 13:56 | |
| *** nihilifer has quit IRC | 13:56 | |
| *** cfarquhar has quit IRC | 13:56 | |
| *** toabctl has quit IRC | 13:56 | |
| *** cbader has quit IRC | 13:56 | |
| *** bookwar has quit IRC | 13:56 | |
| *** mkoderer has quit IRC | 13:56 | |
| *** dulek has quit IRC | 13:56 | |
| *** crinkle has quit IRC | 13:56 | |
| *** hrubi has quit IRC | 13:56 | |
| *** skraynev has quit IRC | 13:56 | |
| *** BobBall has quit IRC | 13:56 | |
| *** clayton has quit IRC | 13:56 | |
| *** tonyb has quit IRC | 13:56 | |
| *** kambiz has quit IRC | 13:56 | |
| *** andreykurilin has quit IRC | 13:56 | |
| *** kzaitsev_ws has quit IRC | 13:56 | |
| *** purp has quit IRC | 13:56 | |
| *** cinerama has quit IRC | 13:56 | |
| *** mancdaz has quit IRC | 13:56 | |
| *** lifeless has quit IRC | 13:56 | |
| *** leifmadsen has quit IRC | 13:56 | |
| *** robcresswell has quit IRC | 13:56 | |
| *** jokke_ has quit IRC | 13:56 | |
| *** bkero has quit IRC | 13:56 | |
| *** krotscheck has quit IRC | 13:56 | |
| *** |GonZo2000| has quit IRC | 13:56 | |
| *** dhp has quit IRC | 13:56 | |
| *** j_king has quit IRC | 13:56 | |
| *** Zara has quit IRC | 13:56 | |
| *** jzerebecki has quit IRC | 13:56 | |
| *** ociuhandu_ is now known as ociuhandu | 13:56 | |
| *** schang_ is now known as schang | 13:56 | |
| *** kambiz_ is now known as kambiz | 13:56 | |
| *** robcresswell_ is now known as robcresswell | 13:56 | |
| *** ianw_ is now known as ianw | 13:56 | |
| *** beisner- is now known as beisner | 13:56 | |
| *** itsuugo_ is now known as itsuugo | 13:56 | |
| *** kushal has joined #openstack-infra | 13:56 | |
| *** krotscheck_ is now known as krotscheck | 13:56 | |
| *** leifmadsen_ is now known as leifmadsen | 13:56 | |
| *** zzzeek has joined #openstack-infra | 13:56 | |
| *** dulek has joined #openstack-infra | 13:56 | |
| *** Zara has joined #openstack-infra | 13:56 | |
| *** raginbaj- is now known as raginbajin | 13:56 | |
| *** clayton has joined #openstack-infra | 13:57 | |
| *** ccrouch has quit IRC | 13:57 | |
| *** dougshelley66 has joined #openstack-infra | 13:57 | |
| *** toabctl_ is now known as toabctl | 13:57 | |
| *** ccrouch has joined #openstack-infra | 13:57 | |
| *** ikalnitsky has joined #openstack-infra | 13:58 | |
| *** BobBall has joined #openstack-infra | 13:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Set DocImpact group for LMA projects https://review.openstack.org/263334 | 13:58 |
| openstackgerrit | Merged openstack-infra/project-config: Update fuel-plugins on 'group' and 'docimpact' flag https://review.openstack.org/268101 | 13:58 |
| *** BobBall is now known as Guest45232 | 13:59 | |
| *** migi has quit IRC | 13:59 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new repo for fuel-plugin-zaqar in /Openstack https://review.openstack.org/270121 | 13:59 |
| *** bcourt has quit IRC | 13:59 | |
| *** yamamoto has quit IRC | 13:59 | |
| *** toabctl is now known as toabctl_ | 14:00 | |
| *** migi has joined #openstack-infra | 14:00 | |
| *** xyang1 has joined #openstack-infra | 14:00 | |
| mordred | AJaeger: lgtm | 14:01 |
| *** eharney has joined #openstack-infra | 14:01 | |
| *** ansiwen has joined #openstack-infra | 14:01 | |
| *** Jeffrey4l has quit IRC | 14:01 | |
| *** PsionTheory has joined #openstack-infra | 14:02 | |
| *** eharney is now known as Guest983 | 14:02 | |
| *** skraynev has joined #openstack-infra | 14:02 | |
| ikalnitsky | Hey folks! We (fuel-plugins core team) had a short conversation, and have decided to remove some branches in our repo. How can we do this? Gerrit says me I have no force permissions. Could someone help me? | 14:02 |
| *** d0ugal has joined #openstack-infra | 14:02 | |
| *** nmagnezi has joined #openstack-infra | 14:02 | |
| *** akscram has joined #openstack-infra | 14:03 | |
| openstackgerrit | Merged openstack-infra/jeepyb: Add docimpact tag for bugs that target project queues https://review.openstack.org/253229 | 14:03 |
| *** anteaya has joined #openstack-infra | 14:03 | |
| AJaeger | thanks, mordred | 14:03 |
| *** yamamoto has joined #openstack-infra | 14:03 | |
| *** sambetts has joined #openstack-infra | 14:03 | |
| *** clarkb has joined #openstack-infra | 14:03 | |
| *** tonyb_ is now known as tonyb | 14:03 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 14:04 |
| *** bkero has joined #openstack-infra | 14:04 | |
| *** vponomaryov has joined #openstack-infra | 14:04 | |
| *** andreykurilin has joined #openstack-infra | 14:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pass timeout through to floating ip creation https://review.openstack.org/270193 | 14:04 |
| mordred | ikalnitsky: which branches and which repo? | 14:05 |
| *** kzaitsev_mb has quit IRC | 14:05 | |
| *** mnaser has joined #openstack-infra | 14:05 | |
| *** esikachev has joined #openstack-infra | 14:05 | |
| openstackgerrit | Angus Lees proposed openstack-infra/project-config: Create new tempest-dsvm-full-head job-template https://review.openstack.org/270143 | 14:05 |
| mescanef | ikalnitsky: https://wiki.openstack.org/wiki/Fuel/Plugins#Deleting_branches ;-) | 14:05 |
| *** Sam-I-Am has joined #openstack-infra | 14:06 | |
| *** notmorgan has joined #openstack-infra | 14:06 | |
| *** notmorgan has joined #openstack-infra | 14:06 | |
| *** dkranz has quit IRC | 14:06 | |
| mordred | wow. there is a document alread. I'm impressed | 14:06 |
| mescanef | ikalnitsky: id go with sending an email to openstack infra ML | 14:06 |
| *** kzaitsev_mb has joined #openstack-infra | 14:07 | |
| ansiwen | AJaeger: hi, sdague told me I should ask you about that: python setup.py compile_catalog compiles just some of the .po files (at least in nova). Is this a temporary issue, or is it just not taken care of at the moment? | 14:07 |
| AJaeger | ansiwen: interesting, let me check... | 14:08 |
| AJaeger | ansiwen: should I join #openstack-nova? | 14:08 |
| ansiwen | AJaeger: I don't mind | 14:09 |
| *** thorst has quit IRC | 14:09 | |
| AJaeger | let me join... | 14:09 |
| *** thorst has joined #openstack-infra | 14:09 | |
| *** Guest983 is now known as eharney | 14:10 | |
| *** eharney has quit IRC | 14:10 | |
| *** eharney has joined #openstack-infra | 14:10 | |
| *** abregman is now known as abregman|mtg | 14:10 | |
| thomasem | Hello! I'm pretty unfamiliar with a lot of the process regarding nodes for nodepool in devstack-gate. I'm working on getting libvirt-lxc experimental gate working reliably so we can promote it to non-voting, then voting and so on for Nova. | 14:12 |
| thomasem | The thing causing failures appears to be Libvirt/LXC not working properly on the very latest 3.13.0 kernel from Ubuntu's updates repo | 14:12 |
| *** nihilife1 has quit IRC | 14:13 | |
| *** peristeri has joined #openstack-infra | 14:13 | |
| *** mhickey has joined #openstack-infra | 14:13 | |
| *** kgiusti has joined #openstack-infra | 14:13 | |
| *** ccrouch has quit IRC | 14:13 | |
| thomasem | So, wondering about setting up a new node with 3.13.0-58-generic, which is available to trusty out of the box, rather than 3.13.0-74-generic where it breaks. I confirmed that an upstream 3.13.0 kernel works also, and other newer ones do as well. I'm concerned it's just somewhere between 3.13.0-58-generic and 3.13.0-74-generic that it's getting broken. | 14:14 |
| *** nihilifer has joined #openstack-infra | 14:14 | |
| mordred | thomasem: so actually, we just updated all the long-lived servers to 3.13.0-76-generic due to a CVE | 14:14 |
| mordred | the nodepool nodes should follow suit shortly | 14:14 |
| thomasem | No kidding? Let me give that a go! | 14:15 |
| *** bcourt has joined #openstack-infra | 14:15 | |
| *** ari has joined #openstack-infra | 14:15 | |
| mordred | but I would recommend digging in with that kernel - I imagine if libvirt-lxc doesn't work with the CVE-fixed kernel that many people will be unhappy | 14:15 |
| *** sabeen1 has joined #openstack-infra | 14:15 | |
| thomasem | mordred: Thanks!. I haven't tried that one yet... is that in the trusty-updates? | 14:15 |
| mordred | thomasem: might just be in security | 14:15 |
| *** dslevin has joined #openstack-infra | 14:15 | |
| thomasem | Cool, I'll pull it down and run the tests | 14:15 |
| *** dslevin has quit IRC | 14:16 | |
| thomasem | see what happens | 14:16 |
| mordred | ossum | 14:16 |
| mordred | however - as to your question | 14:16 |
| *** dslevin has joined #openstack-infra | 14:16 | |
| mordred | it's not really a thing we can do to have a custom kernel on nodes for tests - we're trying actually to decrease the number of divergent nodes so that we can get better throughput overall for the gate | 14:16 |
| mordred | (we pre-boot nodes and keep them in a ready pool before tests are run - so the node is already running by the time it gets assigned to a test) | 14:17 |
| thomasem | I see | 14:17 |
| thomasem | That makes sense, just not sure what to do when it simply won't pass on that kernel due to some funniness with libvirt + kernel, was my main concern. | 14:18 |
| thomasem | I'd prefer not skip tests over it, if I can help it | 14:18 |
| thomasem | The long-term solution would be troubleshooting what the problem is between the two and getting that fixed. Although, I know for sure this all works on much newer kernels. | 14:19 |
| *** ddieterly has joined #openstack-infra | 14:19 | |
| *** sabeen1 has quit IRC | 14:20 | |
| mordred | thomasem: yeah - it's a tricky area | 14:21 |
| openstackgerrit | Merged openstack-infra/irc-meetings: adding the irc meeting slot for the openstack-salt team. https://review.openstack.org/269936 | 14:21 |
| mordred | so far we've tried to keep to latest kernel on trusty and wanted ot see the breaks - because that would be what someone gets if they were to install openstack on trusty in production | 14:21 |
| mordred | (so we'd rather be made aware that we're currently broken with existing kernels) - but in times like this when we're a few months away from a new LTS it starts to get ... trying | 14:22 |
| *** sabeen has joined #openstack-infra | 14:23 | |
| mordred | infra-root: http://puppetboard.openstack.org/report/ask-staging.openstack.org/444f17f7be378f6c3318f5658f40168879102519 <-- that's a puppetboard report from having run puppet apply via ansible | 14:23 |
| *** acruz has joined #openstack-infra | 14:23 | |
| mordred | AJaeger: ^^ you too | 14:23 |
| thomasem | mordred: Ahhh, I see | 14:23 |
| *** tiswanso has joined #openstack-infra | 14:23 | |
| mordred | AJaeger: (oh, wait, that wasn't you asking the question about that) | 14:23 |
| *** arxcruz has quit IRC | 14:23 | |
| thomasem | mordred: that's fair. Well, let me see if we even have a problem anymore before I get all worried. I'm just trying to keep the lxc stuff from getting deprecated, and in fact promoted to a more supported state. | 14:23 |
| mordred | thomasem: totally - and a laudable goal | 14:24 |
| thomasem | Once we can get more backing, with getting that all working with Magnum and what not too. | 14:24 |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Use VPN devstack plugin in heat jobs https://review.openstack.org/259463 | 14:25 |
| *** bcourt has quit IRC | 14:25 | |
| mordred | infra-root: so - with that said - I need one more +2 on https://review.openstack.org/#/c/249492 and I can roll puppet apply live | 14:25 |
| *** sabeen3 has joined #openstack-infra | 14:26 | |
| *** moravec has joined #openstack-infra | 14:26 | |
| mordred | fungi: also, based on yesterday incident scrollback - I think maybe we should have a delete_node.py script to compliment launch_node.py that clears the cache for us? | 14:27 |
| *** acruz has quit IRC | 14:27 | |
| *** sabeen has quit IRC | 14:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pass timeout through to floating ip creation https://review.openstack.org/270193 | 14:29 |
| *** dkranz has joined #openstack-infra | 14:29 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins https://review.openstack.org/269007 | 14:29 |
| *** kzaitsev_mb has quit IRC | 14:30 | |
| *** tiswanso has quit IRC | 14:30 | |
| *** rossella_s has quit IRC | 14:31 | |
| *** rossella_s has joined #openstack-infra | 14:31 | |
| *** tiswanso has joined #openstack-infra | 14:31 | |
| *** mriedem has joined #openstack-infra | 14:31 | |
| *** otter768 has joined #openstack-infra | 14:32 | |
| *** thorst has quit IRC | 14:32 | |
| *** edmondsw has joined #openstack-infra | 14:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove a done todo list item https://review.openstack.org/269120 | 14:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pass timeout through to floating ip creation https://review.openstack.org/270193 | 14:33 |
| mordred | Shrews: ^^ updated the cache patch too - you were right about that question about setting it to 2 | 14:33 |
| *** yamamoto has quit IRC | 14:33 | |
| *** kzaitsev_mb has joined #openstack-infra | 14:33 | |
| Shrews | mordred: wunderbar | 14:36 |
| *** mtanino has joined #openstack-infra | 14:36 | |
| *** otter768 has quit IRC | 14:36 | |
| *** bcourt has joined #openstack-infra | 14:37 | |
| sambetts | Have there been any further updates about the pip > 8 issue? I'm seeing failures locally and in CI for the python27 job, I see there has been a fix put in place for devstack, but is there any way to fix the py27 job? | 14:38 |
| *** moravec has quit IRC | 14:38 | |
| *** yamamoto has joined #openstack-infra | 14:39 | |
| mordred | sambetts: I am not up to date on the current state of the pip breakage | 14:41 |
| *** sdake has joined #openstack-infra | 14:41 | |
| *** Swami has joined #openstack-infra | 14:41 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 14:42 |
| *** flepied1 has quit IRC | 14:42 | |
| fungi | mordred: well, deleting the cache was as simple as removing a file, but yeah if we use a script to wrap nova delete then it could call that too so we don't forget | 14:43 |
| *** aysyd has quit IRC | 14:43 | |
| *** aysyd has joined #openstack-infra | 14:43 | |
| *** acruz has joined #openstack-infra | 14:44 | |
| AJaeger | sambetts: there's no general solution for py27/pep8/docs if those fails. One might be to remove argparse from requirements but that comes in from oslo.config, so I made a change for that, but now we need to merge that and release it first... | 14:45 |
| mordred | fungi: yah. although I guess what I _actually_ want is to finish ansible launch-node now that we've got puppet apply | 14:45 |
| mordred | fungi: in which case flipping a node to "state:deleted" should do the right thing | 14:45 |
| AJaeger | sambetts: there'S discussion going on at github, see the link in backscroll, on a pip 8.0.1 that fixes this somehow | 14:45 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: WIP: Add new django extraction https://review.openstack.org/266565 | 14:45 |
| fungi | zxiiro: yes, that http://zuul.openstack.org/ page is the zuul server's built-in web rendering of its status.json blob. we maintain a custom rendering at http://status.openstack.org/zuul/ which is pulling the same json | 14:46 |
| zxiiro | fungi: thanks! | 14:46 |
| *** knikolla has joined #openstack-infra | 14:47 | |
| AJaeger | amotoki: just looked at your change, great! I'll review later... | 14:47 |
| zxiiro | fungi: how do i access that if i have a zuul instance? i don't see it documented | 14:47 |
| zxiiro | fungi: I was able to figure out http://server:8001/status for the json but I couldn't find the page rendering | 14:47 |
| amotoki | AJaeger: thanks. I try to check murano-dashboard but there is no version in zanata... can you create a version on Zanata? | 14:48 |
| AJaeger | amotoki: I'll do a change later today... | 14:50 |
| *** jkt_ is now known as jkt | 14:50 | |
| sambetts | AJaeger: here https://github.com/pypa/pip/issues/3384 | 14:50 |
| sambetts | ? | 14:50 |
| kzaitsev_mb | amotoki: it's not yet there, since my commit that adds translations initially sparked this whole situation ) | 14:51 |
| *** acruz has quit IRC | 14:52 | |
| fungi | Zara: SotK: i'll take a quick look at the mail queue and exim logs on storyboard.o.o. is it delivering to 127.0.0.1 via smtp or calling /usr/lib/sendmail and having that inject? | 14:52 |
| SotK | the former I believe | 14:53 |
| kzaitsev_mb | AJaeger: just wanted to note, that I rebased https://review.openstack.org/#/c/262073/3 on top of 266565, not attempting to add it as a separate patch =) | 14:53 |
| *** ashishb has joined #openstack-infra | 14:53 | |
| amotoki | AJaeger: no need to rush :) | 14:53 |
| Zara | fungi: thank you! :) | 14:54 |
| *** naggappan has quit IRC | 14:56 | |
| amotoki | kzaitsev_mb: ah... I got the situation. I thought murano-dashboard exists in Zanata... | 14:56 |
| kzaitsev_mb | amotoki: to certain extent you can use magnum-ui as a substitute. | 14:58 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 14:58 |
| AJaeger | sambetts: yes | 14:59 |
| *** electrofelix has quit IRC | 14:59 | |
| kzaitsev_mb | not sure if that's possible though, but the two projects are similar in regard of i18n stuff | 14:59 |
| fungi | Zara: SotK: simple answer! exim on storyboard is configured to perform sender verification, even on messages injected via loopback, and the messages are using a nonexistent sender | 14:59 |
| *** markmcd__ has quit IRC | 14:59 | |
| *** markmcd has quit IRC | 14:59 | |
| AJaeger | kzaitsev_mb: let me pull your change apart later ;) | 14:59 |
| sambetts | AJaeger: Ok thanks :) | 14:59 |
| fungi | Zara: SotK: 2016-01-20 13:46:58 H=localhost (storyboard.openstack.org) [127.0.0.1] sender verify fail for <no-reply@storyboard.openstack.org>: Unknown user | 14:59 |
| AJaeger | amotoki: I took murano-dashboard since it did not exist yet ;) That way we cannot break anything ;) | 15:00 |
| *** bcourt has quit IRC | 15:00 | |
| SotK | fungi: heh, thanks! | 15:01 |
| kzaitsev_mb | AJaeger: sure =) | 15:01 |
| *** piet has joined #openstack-infra | 15:01 | |
| *** ashishb has quit IRC | 15:01 | |
| SotK | fungi: I assume the solution is to create a user for storyboard to use? | 15:01 |
| fungi | SotK: likely we should look at how gerrit's e-mail messages are formed, for precedent | 15:02 |
| *** ashishb has joined #openstack-infra | 15:02 | |
| *** daemontool has joined #openstack-infra | 15:02 | |
| fungi | SotK: looks like we use a real address <review@openstack.org> in that situation | 15:03 |
| fungi | and have a mailbox set up to accept for that | 15:03 |
| fungi | checking now to see what happens if i reply to it | 15:04 |
| *** bhunter71 has joined #openstack-infra | 15:04 | |
| fungi | aha, gerrit also sets reply-to to the e-mail address of the account for the originating action | 15:05 |
| *** bcourt has joined #openstack-infra | 15:05 | |
| fungi | so if i upload a patchset or make a comment in gerrit, it sets reply-to: my e-mail address instead | 15:05 |
| *** jaosorior has quit IRC | 15:06 | |
| *** bryan_att has joined #openstack-infra | 15:06 | |
| fungi | i guess that cuts down on random replies to the service's from address | 15:06 |
| openstackgerrit | Merged openstack-infra/project-config: Add sahara scenario gate jobs in sahara-scenario https://review.openstack.org/270166 | 15:06 |
| *** zz_dimtruck is now known as dimtruck | 15:06 | |
| mordred | I believe there are violent opinions in the world on whether setting reply-to original sender is good or bad | 15:06 |
| SotK | we aim to have reply-to set to something which storyboard can watch and use to post comments on stories and such-like eventually | 15:06 |
| *** daemontool has quit IRC | 15:07 | |
| *** daemontool has joined #openstack-infra | 15:07 | |
| *** thorst has joined #openstack-infra | 15:08 | |
| *** jsavak has joined #openstack-infra | 15:08 | |
| *** mancdaz has joined #openstack-infra | 15:08 | |
| fungi | but yeah, in this case it's the sender (envelope mail-from) at issue. so far looks like if i e-mail review@openstack.org directly it's a black hole. we've probably got a mountain of spam in a mailbox assigned to that address | 15:08 |
| *** thorst has quit IRC | 15:08 | |
| *** thorst has joined #openstack-infra | 15:08 | |
| *** EricGonczer_ has joined #openstack-infra | 15:08 | |
| *** markmcd has joined #openstack-infra | 15:09 | |
| fungi | so anyway, there are a few options. we could do like gerrit is set up and have the sender be an actual black hole address somewhere, or we could configure exim to skip sender verification for messages from localhost... the latter though seems like it's just kicking the can down the road and the message will get rejected at lots of destinations instead because they'll attempt sender verification | 15:10 |
| *** EricGonczer_ has quit IRC | 15:10 | |
| *** acruz has joined #openstack-infra | 15:11 | |
| fungi | jeblair's been doing a lot of our role address mailbox administration, so i'd like to get his input too once he's awake | 15:11 |
| *** sambetts has left #openstack-infra | 15:11 | |
| *** EricGonczer_ has joined #openstack-infra | 15:12 | |
| SotK | yeah, doing the same as gerrit seems sensible | 15:12 |
| * SotK waits for jeblair | 15:12 | |
| *** acruz has quit IRC | 15:13 | |
| *** markmcd has quit IRC | 15:13 | |
| *** daemontool has quit IRC | 15:13 | |
| *** ashishb has quit IRC | 15:13 | |
| *** electrofelix has joined #openstack-infra | 15:13 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Add osprofiler events to #openstack-oslo channel https://review.openstack.org/270238 | 15:13 |
| *** bkero has quit IRC | 15:13 | |
| *** bkero has joined #openstack-infra | 15:13 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:13 | |
| *** rbrndt has joined #openstack-infra | 15:14 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 15:16 |
| *** boris-42 has joined #openstack-infra | 15:16 | |
| *** electrofelix has quit IRC | 15:16 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for pip 8 release explosion https://review.openstack.org/270239 | 15:16 |
| *** zeih has joined #openstack-infra | 15:17 | |
| *** electrofelix has joined #openstack-infra | 15:17 | |
| mordred | fungi: nudge nudge https://review.openstack.org/#/c/249492/ nudge nudge :) | 15:18 |
| mordred | fungi: I have verified that it works - ready to flip the switch | 15:19 |
| *** jokke__ is now known as jokke_ | 15:19 | |
| mordred | fungi: woot! | 15:22 |
| mordred | here goes nothing | 15:22 |
| *** andymaier has joined #openstack-infra | 15:22 | |
| *** hdd has joined #openstack-infra | 15:23 | |
| *** andymaier has quit IRC | 15:23 | |
| *** andymaier has joined #openstack-infra | 15:23 | |
| *** Keedya has joined #openstack-infra | 15:24 | |
| *** phschwartz_ is now known as phschwartz | 15:25 | |
| Keedya | Hi all , I m adding a new project to Openstack and need help setting up Jenkins jobs and Zulu. Anyone :)? | 15:25 |
| *** abregman|mtg is now known as abregman | 15:25 | |
| *** bauzas has quit IRC | 15:25 | |
| *** markvoel_ has quit IRC | 15:25 | |
| Keedya | Zuul* | 15:26 |
| *** zeih has quit IRC | 15:26 | |
| *** fhubik_brb is now known as fhubik | 15:27 | |
| *** EricGonczer_ has quit IRC | 15:28 | |
| openstackgerrit | Merged openstack-infra/system-config: Use puppet apply instead of puppet agent https://review.openstack.org/249492 | 15:29 |
| *** bauzas has joined #openstack-infra | 15:30 | |
| *** yamahata has joined #openstack-infra | 15:31 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Use devstack subunit stream in output https://review.openstack.org/234447 | 15:31 |
| *** tonytan4ever has joined #openstack-infra | 15:32 | |
| *** _et_ has quit IRC | 15:33 | |
| *** _nadya_ has quit IRC | 15:34 | |
| *** zhurong has quit IRC | 15:35 | |
| *** fhubik is now known as fhubik_brb | 15:35 | |
| *** vgridnev has quit IRC | 15:35 | |
| mordred | fungi: btw - we do not track puppetmaster in cacti | 15:36 |
| *** piet has quit IRC | 15:37 | |
| *** _nadya_ has joined #openstack-infra | 15:38 | |
| *** ajmiller has quit IRC | 15:38 | |
| tristanC | should I worry if some changes didn't get picked by jenkins/zuul 30 minutes after being proposed ? e.g.: https://review.openstack.org/#/c/270234/ ? | 15:38 |
| *** jpr1 has joined #openstack-infra | 15:38 | |
| *** tiswanso has quit IRC | 15:39 | |
| tristanC | oh nevermind, they are in zuul, the review is just missing a comment | 15:39 |
| openstackgerrit | Maksim Malchuk proposed openstack-infra/project-config: Make gate-fuel-library-puppet-syntax-4-dsvm-centos7 job voting in the check queue https://review.openstack.org/270256 | 15:39 |
| *** frickler_ is now known as frickler | 15:39 | |
| openstackgerrit | Maksim Malchuk proposed openstack-infra/project-config: Make gate-fuel-library-puppet-syntax-4-dsvm-centos7 job voting in the check queue https://review.openstack.org/270256 | 15:39 |
| *** esikachev has quit IRC | 15:40 | |
| *** tiswanso has joined #openstack-infra | 15:41 | |
| *** Hal has joined #openstack-infra | 15:41 | |
| *** fhubik_brb is now known as fhubik | 15:41 | |
| *** jordanP has joined #openstack-infra | 15:41 | |
| *** Hal is now known as Guest33708 | 15:42 | |
| *** yamamoto has quit IRC | 15:42 | |
| *** ajmiller has joined #openstack-infra | 15:42 | |
| *** electrofelix has left #openstack-infra | 15:42 | |
| *** yamamoto has joined #openstack-infra | 15:42 | |
| openstackgerrit | Shamail Tahir proposed openstack-infra/irc-meetings: Add new bi-weekly APAC meeting for user committee https://review.openstack.org/269427 | 15:43 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Change the date and chair of the HA guide team meetings https://review.openstack.org/252388 | 15:43 |
| *** devananda has quit IRC | 15:45 | |
| openstackgerrit | Maksim Malchuk proposed openstack-infra/project-config: Make gate-fuel-library-puppet-syntax-4-dsvm-centos7 job voting in the check queue https://review.openstack.org/270256 | 15:46 |
| *** markvoelker_ has joined #openstack-infra | 15:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Stop listing hosts as groups in the disabled file https://review.openstack.org/270262 | 15:47 |
| mordred | fungi: ^^ urgentish | 15:47 |
| *** _nadya_ has quit IRC | 15:47 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for pip 8 release explosion https://review.openstack.org/270239 | 15:48 |
| mordred | fungi: tl;dr - with the update we made to openstack inventory a little while ago to fix the logs having useless hostnames, we actually broke (but didn't notices) our disabled file - because puppet agent as _also_ disabled on those nodes | 15:48 |
| mordred | fungi: however, we're in luck, because most of those nodes also haven't had puppet run since we update the puppet environmnet file to list the in-tree hiera location and thus fail to compile the manifest/site.pp | 15:49 |
| *** Guest10164 is now known as redrobot | 15:49 | |
| dtantsur | hi folks! could you review https://review.openstack.org/#/c/264168/ please? it's been around for 2 weeks already. thanks! | 15:50 |
| *** Sukhdev has joined #openstack-infra | 15:50 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Delete old queries for fixed bugs https://review.openstack.org/269305 | 15:50 |
| mordred | fungi: actually, I'm just going to approve it - I've made the changes locally on the puppetmaster alread | 15:51 |
| *** devananda has joined #openstack-infra | 15:51 | |
| openstackgerrit | Maksim Malchuk proposed openstack-infra/project-config: Make gate-fuel-library-puppet-syntax-4-dsvm-centos7 job voting in the check queue https://review.openstack.org/270256 | 15:51 |
| *** Eva-i has quit IRC | 15:52 | |
| *** aopincar has quit IRC | 15:52 | |
| *** shashank_hegde has joined #openstack-infra | 15:52 | |
| *** Eva-i has joined #openstack-infra | 15:52 | |
| *** matrohon has quit IRC | 15:53 | |
| *** zeih has joined #openstack-infra | 15:53 | |
| *** vivekd has quit IRC | 15:53 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder: Mark ironic-discoverd-ramdisk as deprecated in favor of ironic-agent https://review.openstack.org/270267 | 15:54 |
| *** _nadya_ has joined #openstack-infra | 15:54 | |
| *** armax has left #openstack-infra | 15:55 | |
| *** ccrouch has joined #openstack-infra | 15:56 | |
| Keedya | Hi all, is there documentation to help me add Zuul jobs for nodejs ? | 15:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Raise SQLAlchemy upper cap to match global requirements https://review.openstack.org/267636 | 15:57 |
| dmsimard | Is there anyone that can help me understand the reason why the gate-project-config-layout job failed in https://review.openstack.org/#/c/269918/ ? | 15:58 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes https://review.openstack.org/270273 | 15:58 |
| fungi | Keedya: you may want to talk to krotscheck... he's possibly been doing some javascript toolchain work for horizon's jobs | 15:58 |
| *** piet has joined #openstack-infra | 15:58 | |
| pabelanger | clarkb: fungi: jeblair: simple rebase for sqlalchemy upper cap for nodepool^ | 15:58 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add some swap to the undercloud https://review.openstack.org/270274 | 15:58 |
| fungi | Keedya: i don't think we have any node.js-based projects yet (though i could be wrong) so a lot of it may need to be designed from scratch | 15:59 |
| krotscheck | fungi: Yes, he's been trying to get in contact with me, but I've been caught up in maintaining the CORS body of work as well as the pre-npm-mirror wheel-mirror things. | 15:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Update docs because hosts are no longer groups https://review.openstack.org/270276 | 15:59 |
| mordred | fungi: ^^ there is the doc update to go along with the patch I self-approved a few moments ago | 15:59 |
| pabelanger | clarkb: fungi: do you mind reviewing 262017 for nodepool too | 16:00 |
| AJaeger | dmsimard: It's template: -name X but gate: X | 16:00 |
| krotscheck | Also, helping Bower become more distro-savvy, figuring out the AFS things I've been told to do to allow npm mirrors, and general Ironic-UX things | 16:00 |
| AJaeger | dmsimard: remove "name" from check/gate | 16:00 |
| dmsimard | AJaeger: oooh, will try that. Thanks. | 16:01 |
| pabelanger | clarkb: greghaynes: So far nodepool-builder on a remote system is working fine | 16:01 |
| *** HeOS has quit IRC | 16:04 | |
| *** bigjools has quit IRC | 16:05 | |
| *** zeih has quit IRC | 16:05 | |
| *** mriedem has quit IRC | 16:05 | |
| *** nmagnezi has quit IRC | 16:05 | |
| *** rajinir has joined #openstack-infra | 16:05 | |
| *** jaypipes has quit IRC | 16:06 | |
| *** hughhalf_ has quit IRC | 16:06 | |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/irc-meetings: add nova-bugs-team meeting https://review.openstack.org/270281 | 16:06 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 16:07 |
| *** hughhalf has joined #openstack-infra | 16:07 | |
| *** purp_too is now known as purp | 16:07 | |
| fungi | mordred: that static list of disabled hosts is stale | 16:07 |
| *** bigjools has joined #openstack-infra | 16:07 | |
| mordred | yes | 16:07 |
| mordred | I'm about to fpush up another list | 16:07 |
| fungi | cool, just making sure you were aware | 16:07 |
| *** jaosorior has joined #openstack-infra | 16:08 | |
| *** mriedem has joined #openstack-infra | 16:08 | |
| *** jaypipes has joined #openstack-infra | 16:11 | |
| *** jaosorior has quit IRC | 16:11 | |
| *** arxcruz has joined #openstack-infra | 16:12 | |
| *** EricGonczer_ has joined #openstack-infra | 16:14 | |
| *** ashishb has joined #openstack-infra | 16:16 | |
| *** Keedya has quit IRC | 16:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use groups.txt for disabling hosts for puppet https://review.openstack.org/270296 | 16:19 |
| mordred | fungi: ^^ | 16:19 |
| dmsimard | pabelanger: thanks for your help. For the part about projects.yaml, I can add the jobgroup there indeed but it says node: trusty. Should I worry about that ? The jobs are not meant to run on Ubuntu. | 16:19 |
| dmsimard | ref: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml#L4530-L4536 | 16:19 |
| pabelanger | dmsimard: no, you setup ostype in your job-group | 16:20 |
| dmsimard | pabelanger: okay, it's already setup then. Thanks, will submit a patch soon. | 16:20 |
| *** shashank_hegde has quit IRC | 16:21 | |
| *** Keedya has joined #openstack-infra | 16:22 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 16:22 |
| *** zhurong has joined #openstack-infra | 16:23 | |
| *** mudassirlatif has joined #openstack-infra | 16:23 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: increase large-ops job timeout https://review.openstack.org/270304 | 16:24 |
| *** placko_ has quit IRC | 16:24 | |
| sdague | mordred / fungi / mtreinish ^^^ | 16:24 |
| *** Jeffrey4l has joined #openstack-infra | 16:24 | |
| *** rcernin has quit IRC | 16:25 | |
| *** Jeffrey4l has quit IRC | 16:25 | |
| *** Jeffrey4l has joined #openstack-infra | 16:25 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add generic script to recover diagnostic or service config and logs https://review.openstack.org/270004 | 16:26 |
| *** krtaylor has quit IRC | 16:29 | |
| fungi | mordred: so the group matches are only a regex if they start with ~ and are otherwise glob matches? | 16:29 |
| mordred | fungi: yes | 16:30 |
| *** vgridnev has joined #openstack-infra | 16:30 | |
| *** hdd has quit IRC | 16:30 | |
| *** salv-orlando has joined #openstack-infra | 16:30 | |
| *** jordanP has quit IRC | 16:30 | |
| *** hashar_ has quit IRC | 16:30 | |
| fungi | mordred: also just to confirm, the list you translated over into 270296 is still outdated | 16:32 |
| *** Guest33708 has quit IRC | 16:32 | |
| mordred | fungi: it is? | 16:32 |
| mordred | fungi: I removed two things from it | 16:32 |
| mordred | that are not real things | 16:32 |
| mordred | but that's what's on the box right now? | 16:32 |
| *** nelsnelson has joined #openstack-infra | 16:32 | |
| fungi | what i meant earlier was that puppet agent has since been enabled on a lot of those | 16:32 |
| mordred | fungi: unless you mean "let's also move the things in that are in emergency righ tnow" | 16:32 |
| mordred | ah | 16:32 |
| fungi | the pypi mirrors and review-dev at least | 16:33 |
| mordred | kk. I can update | 16:33 |
| *** regXboi has joined #openstack-infra | 16:33 | |
| thomasem | mordred: I'm seeing the same problems with 3.13.0-76-generic. :\ | 16:33 |
| *** ashishb has quit IRC | 16:33 | |
| *** otter768 has joined #openstack-infra | 16:33 | |
| *** yamamoto has quit IRC | 16:33 | |
| fungi | mordred: at least as of last night, all our ubuntu trusty servers were running puppet (because i relied on that to get the sources.list updates in place) | 16:34 |
| mordred | fungi: agent is not disabled on subunit-worker either | 16:34 |
| *** tiswanso has quit IRC | 16:34 | |
| fungi | thomasem: have you tried seeing if there are similar issues on latest centos 7? are these tests something that could be run on centos? | 16:34 |
| *** KimJ has joined #openstack-infra | 16:34 | |
| mordred | fungi: so maybe we leave it at backup and jenkins-dev? | 16:34 |
| fungi | mordred: that sounds good to me | 16:35 |
| *** davidhadas has quit IRC | 16:35 | |
| thomasem | fungi: I haven't tried on CentOS... these are the libvirt/LXC tests, fwiw. | 16:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use groups.txt for disabling hosts for puppet https://review.openstack.org/270296 | 16:35 |
| thomasem | I was using trusty since that was the prescribed method for simulating gate tests | 16:36 |
| thomasem | Do you have any docs on simulating gate on CentOS handy? | 16:36 |
| cbader_ | I am trying to install common-ci and I am running into an error on startup of pip install -q --upgrade diskimage-builder throwing a SNIMissingWaring anyone seen this today. | 16:36 |
| *** Keedya has quit IRC | 16:36 | |
| *** bpokorny has joined #openstack-infra | 16:37 | |
| fungi | thomasem: oh, are these tests running in tempest? not unit tests? | 16:37 |
| *** otter768 has quit IRC | 16:37 | |
| *** cinerama` is now known as cinerama | 16:37 | |
| thomasem | fungi: yeah | 16:38 |
| thomasem | one of the symptoms is that nova stop <instance> fails with: 2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1 | 16:38 |
| thomasem | and not much else to go on | 16:38 |
| *** ashleighfarnham has joined #openstack-infra | 16:38 | |
| thomasem | 3.13.0-58-generic doesn't have the problem | 16:38 |
| thomasem | and 3.13.0 upstream doesn't seem to either | 16:38 |
| *** mudassirlatif has quit IRC | 16:38 | |
| *** esikachev has joined #openstack-infra | 16:38 | |
| thomasem | seems like something up with the ubuntu kernel's latter builds specifically | 16:39 |
| fungi | thomasem: i _think_ devstack/tempest work on centos 7 but don't know to what extent, or if it's regularly tested (yet anyway) | 16:39 |
| mordred | fungi: maybe I should just squash the two patches - the doc update and the use-groups.txt | 16:39 |
| fungi | so, yeah, getting that to work on centos may be more work than running down the lernel issue | 16:39 |
| *** electrofelix has joined #openstack-infra | 16:39 | |
| fungi | mordred: i think i would prefer that | 16:39 |
| thomasem | haha yeah, probably | 16:39 |
| *** fawadkhaliq has joined #openstack-infra | 16:39 | |
| fungi | s/lernel/kernel/ | 16:39 |
| fungi | thomasem: is there an open bug for this? | 16:40 |
| *** ihrachys_ has quit IRC | 16:40 | |
| *** davidhadas_ has joined #openstack-infra | 16:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use groups.txt for disabling hosts for puppet https://review.openstack.org/270296 | 16:40 |
| mordred | fungi: thereyago | 16:40 |
| fungi | thomasem: like, have you checked the bug trackers for libvirt/lxc to see if it's a known incompatibility with certain kernels? | 16:40 |
| *** krtaylor has joined #openstack-infra | 16:40 | |
| thomasem | fungi: I've looked around the bug trackers but didn't find anything quite matching this issue. | 16:41 |
| *** unicell1 has quit IRC | 16:41 | |
| mordred | zul ^^ trusty might be broken with libvirt/lxc | 16:41 |
| mordred | jamespage: ^^ | 16:41 |
| thomasem | Appears to be when the virDomainShutdown call happens | 16:42 |
| *** ashishb has joined #openstack-infra | 16:42 | |
| zul | mordred: hmm? | 16:42 |
| mordred | zul: thomasem is working on getting libvirt/lxc driver voting, but is running in to some breaks that don't happen with 3.13.0-58-generic or with 3.13.0 upstream | 16:43 |
| mordred | zul: I figured that if there was an ubuntu kernel patch that had broken it between -58 and now you might be interested :) | 16:44 |
| zul | thomasem: can you open up a bug please and we'll start from there | 16:44 |
| thomasem | zul: yeah, against which project? | 16:44 |
| zul | libvirt-bin in launchpad for ubuntu | 16:44 |
| thomasem | okey doke | 16:44 |
| mordred | thomasem: zul works on ubuntu server so is basically a magical pony for fixing all of the ubuntu problems you ever encounter | 16:45 |
| thomasem | LOL | 16:45 |
| thomasem | ahhh I see | 16:45 |
| thomasem | okay, awesome, glad I asked | 16:45 |
| thomasem | Thanks a 'mil, zul! | 16:45 |
| * mordred hands zul a somewhat undrunk coffee | 16:45 | |
| *** unicell has joined #openstack-infra | 16:46 | |
| *** annegentle has quit IRC | 16:46 | |
| * fungi prefers his coffee completely drunken | 16:46 | |
| *** Sukhdev has quit IRC | 16:47 | |
| *** annegentle has joined #openstack-infra | 16:47 | |
| clarkb | is pip8 still affecting us or have we applied sufficient workaround? | 16:47 |
| *** fawadkhaliq has quit IRC | 16:48 | |
| *** fawadkhaliq has joined #openstack-infra | 16:48 | |
| dhellmann | I need some help debugging a release notes publishing job. Is this a good time to ask, or are you all still working bigger gate issues? | 16:49 |
| *** abregman has quit IRC | 16:50 | |
| *** fawadkhaliq has quit IRC | 16:50 | |
| *** dizquierdo has quit IRC | 16:51 | |
| dhellmann | fungi, clarkb : have time to lend a hand? ^^ | 16:53 |
| fungi | dhellmann: sure, i can take a peek--have details? | 16:53 |
| *** apoorvad has joined #openstack-infra | 16:53 | |
| *** andymaier has quit IRC | 16:53 | |
| dhellmann | fungi : yeah, I'm pulling them together in https://etherpad.openstack.org/p/ironic-inspector-client-reno-job-debugging | 16:53 |
| *** ashishb has quit IRC | 16:53 | |
| *** andymaier has joined #openstack-infra | 16:54 | |
| *** yfried has quit IRC | 16:54 | |
| *** melwitt_ has joined #openstack-infra | 16:54 | |
| nibalizer | mordred: thoughts on http://puppetboard.openstack.org/node/038e80f5-15aa-4f69-8c6c-0f43b3587778 ? | 16:55 |
| nibalizer | doesn't look right to me | 16:55 |
| nibalizer | however 'pbx' is reporting fine it looks like | 16:56 |
| *** _nadya_ has quit IRC | 16:56 | |
| clarkb | dhellmann: give mr a few still trying to catch up on the overnight fun | 16:56 |
| nibalizer | so maybe that is cruft | 16:56 |
| dhellmann | clarkb : np, thanks | 16:56 |
| fungi | nibalizer: maybe that's an extra pbx? like the centos 6 one never got deleted from rax? | 16:56 |
| clarkb | it looks like pip8 broke many thing bht they aren't currently brraking so we must've addressed it somehow | 16:56 |
| fungi | dhellmann: looking at the publisher in that log, i think it may not be dropping the files into the subtree you expect | 16:56 |
| mordred | nibalizer: checking | 16:57 |
| fungi | dhellmann: where's that defined in project-config? | 16:57 |
| nibalizer | mordred: also http://puppetboard.openstack.org/nodes has a very specific and incorrect 'catalog compiled at' | 16:57 |
| mordred | yup. therea re two pbx's | 16:57 |
| mordred | root@puppetmaster:~# ansible pbx.openstack.org --list-hosts | 16:57 |
| mordred | hosts (2): | 16:57 |
| mordred | 038e80f5-15aa-4f69-8c6c-0f43b3587778 | 16:57 |
| mordred | 102f69b8-3173-4565-86fa-e0f539e9e254 | 16:57 |
| mordred | nibalizer: I'm not sure we ever properly dealt with catalog compiled at | 16:58 |
| nibalizer | its fine if its an error | 16:58 |
| mordred | nibalizer: I think we should likely investigate and see if we can make it betterer | 16:58 |
| nibalizer | I'm guessing we have a string hardcoded in the python from an example payload | 16:58 |
| dhellmann | fungi : looking | 16:58 |
| *** bpokorny_ has joined #openstack-infra | 16:59 | |
| mordred | clarkb, nibalizer: also https://review.openstack.org/270296 | 16:59 |
| mordred | came out of looking at disabled problems this morning | 16:59 |
| dhellmann | fungi : jenkins/jobs/openstack-publish-jobs.yaml | 16:59 |
| *** melwitt_ has quit IRC | 16:59 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 16:59 |
| nibalizer | yup http://git.openstack.org/cgit/openstack-infra/ansible-puppet/tree/library/puppet_post_puppetdb#n84 | 17:00 |
| dhellmann | fungi : the fact that zuul-cloner is reporting an earlier commit isn't suspicious to you? | 17:00 |
| *** achanda has joined #openstack-infra | 17:01 | |
| *** melwitt_ has joined #openstack-infra | 17:01 | |
| *** armax has joined #openstack-infra | 17:01 | |
| phschwartz | Has anyone seen an error of "ImportError: No module named schedulers.background" while starting zuul recently? | 17:01 |
| *** hashar has joined #openstack-infra | 17:02 | |
| fungi | dhellmann: oh, perhaps i missed the problem statement | 17:02 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 17:02 |
| dhellmann | fungi : notes do get published, I think they're just the wrong notes, but I'm not confident I've identified the right log file, either | 17:02 |
| *** bpokorny has quit IRC | 17:02 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 17:03 |
| *** jsavak has quit IRC | 17:03 | |
| fungi | dhellmann: so, this is a publication job run in the post pipeline? could there have been two commits approved at about the same time? sometimes jobs race uploading in post and you'll get an earlier commit's content overwriting a later commit if the later commit's jobs run more quickly for any reason | 17:03 |
| nibalizer | mordred: looks good | 17:03 |
| *** trown is now known as trown|lunch | 17:03 | |
| *** fedexo has joined #openstack-infra | 17:03 | |
| mordred | nibalizer: also - dude. we're appluing | 17:03 |
| mordred | applying | 17:03 |
| *** jsavak has joined #openstack-infra | 17:03 | |
| dhellmann | fungi : that's possible, I suppose | 17:04 |
| nibalizer | see what we can accomplish when we apply ourselves? | 17:04 |
| *** jistr has quit IRC | 17:04 | |
| clarkb | no 8.0.1 release has been made | 17:04 |
| fungi | dhellmann: easiest way is to look at the same log for the prior commit and compare timestamps for the ftp commands | 17:04 |
| dhellmann | fungi : looking at http://git.openstack.org/cgit/openstack/python-ironic-inspector-client/log/ I don't see commits that close together | 17:04 |
| *** ddieterly has quit IRC | 17:05 | |
| *** markmcd has joined #openstack-infra | 17:05 | |
| *** tiswanso has joined #openstack-infra | 17:05 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds openstack-doc-tools as a group for fairy-slipper https://review.openstack.org/268313 | 17:05 |
| fungi | dhellmann: and this job was triggered by a commit on python-ironic-inspector-client not releases? | 17:05 |
| fungi | ahh, yep | 17:05 |
| *** gyee has joined #openstack-infra | 17:06 | |
| *** tiswanso has quit IRC | 17:06 | |
| dhellmann | fungi : yeah, dtantsur is trying to put together a release and the current notes looked wrong so we're trying to figure that out before releasing | 17:07 |
| *** tiswanso has joined #openstack-infra | 17:07 | |
| *** ociuhandu has quit IRC | 17:07 | |
| dhellmann | fungi : this job did run after the previous one by almost 2 hrs (2016-01-19 19:04:14.490 vs. 2016-01-19 21:49:44.254 for the current) | 17:07 |
| dhellmann | sorry, almost 3 hrs | 17:08 |
| *** piet has quit IRC | 17:08 | |
| fungi | dhellmann: yep, just got finished pulling and confirming it | 17:08 |
| *** piet has joined #openstack-infra | 17:08 | |
| fungi | http://logs.openstack.org/62/62ef4eb178728b13af862e61ba595df2c114eed7/post/python-ironic-inspector-client-releasenotes/3367ad0/console.html | 17:08 |
| thomasem | fungi: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1536280 | 17:08 |
| openstack | Launchpad bug 1536280 in libvirt (Ubuntu) "domain shutdown fails for libvirt/lxc" [Undecided,New] | 17:08 |
| thomasem | That's one of them | 17:08 |
| thomasem | There were several tests that failed in this kernel, but passed fine on 3.13.0-58-generic | 17:09 |
| *** jlanoux has quit IRC | 17:09 | |
| thomasem | so, going to manually run those and open any bugs that aren't related to this one | 17:09 |
| dhellmann | fungi : the commit mentioned by zuul-cloner comes before the commit with the missing note, so the fact that zuul seems to be looking at something old would explain everything, except why it's looking at something old (assuming I'm interpreting that correctly) | 17:09 |
| fungi | dhellmann: was just starting to look into that too | 17:10 |
| *** yamahata has quit IRC | 17:10 | |
| fungi | INFO:zuul.Cloner:Prepared openstack/python-ironic-inspector-client repo with commit f90cebe0c40d87513372eed447d572cf698a3a81 | 17:10 |
| fungi | http://git.openstack.org/cgit/openstack/python-ironic-inspector-client/commit/?id=f90cebe0c40d87513372eed447d572cf698a3a81 | 17:10 |
| *** samuelBartel has quit IRC | 17:10 | |
| fungi | definitely a different commit | 17:10 |
| *** dtantsur is now known as dtantsur|afk | 17:11 | |
| fungi | that was 3 commits before the one which should have triggered the job | 17:11 |
| dhellmann | it's a shame the logs don't show all of the zuul variables for the job, just how the job uses them | 17:11 |
| dhellmann | right, and the note was added as a result of the merge commit that triggered the job | 17:12 |
| fungi | we may still be able to pull them from jenkins | 17:12 |
| dhellmann | ok, good | 17:12 |
| *** melwitt__ has joined #openstack-infra | 17:12 | |
| fungi | dhellmann: https://jenkins06.openstack.org/job/python-ironic-inspector-client-releasenotes/2/parameters/ | 17:13 |
| *** e0ne has quit IRC | 17:13 | |
| *** scheuran has quit IRC | 17:14 | |
| dhellmann | fungi : odd. I don't see "f90cebe0c40d87513372eed447d572cf698a3a81" on that page at all | 17:14 |
| *** monester2 has joined #openstack-infra | 17:14 | |
| fungi | i don't see f90cebe0c40d87513372eed447d572cf698a3a81 | 17:14 |
| fungi | heh, indeed | 17:14 |
| dhellmann | nor a shorter prefix that could have been misinterpreted somehow | 17:14 |
| *** melwitt_ has quit IRC | 17:15 | |
| *** monester2 has quit IRC | 17:15 | |
| zaro | morning | 17:15 |
| *** monester2 has joined #openstack-infra | 17:16 | |
| *** dims has quit IRC | 17:16 | |
| dhellmann | fungi : do you know if that output from zuul-cloner is saying "I was told to check out commit X" or "now that I've done what I was told, I see commit X in the local dir"? | 17:16 |
| fungi | dhellmann: what's dragging in the -e git:/.. to the pip install there? | 17:16 |
| dhellmann | I wonder if there was some issue updating the repo, for example? | 17:16 |
| dhellmann | excellent question | 17:16 |
| dhellmann | ah! there's that hash | 17:17 |
| zaro | pedroalvarez: it was my intention to add a follow on change to its-storyboard but i have lost motivation for it, sorry. | 17:17 |
| fungi | tox -v -ereleasenotes | 17:17 |
| kzaitsev_mb | we're having problems with murano-dsvm job today — it's failing at some point inside tempest | 17:17 |
| kzaitsev_mb | ImportError: No module named fixtures | 17:17 |
| pedroalvarez | zaro: no worries, I understand :) | 17:17 |
| kzaitsev_mb | are we hitting a known issue? | 17:17 |
| fungi | dhellmann: oh, that's probably reporting that we'd previously installed that package from source | 17:17 |
| pedroalvarez | I might find some motivation to continue that work | 17:17 |
| dhellmann | fungi : yeah, I think that's what that is | 17:17 |
| *** _amrith_ is now known as amrith | 17:18 | |
| zaro | pedroalvarez: cool! | 17:18 |
| dhellmann | fungi : "git" does not appear in tox.ini, requirements.txt, or test-requirements.txt in that repo | 17:18 |
| *** root_ has joined #openstack-infra | 17:19 | |
| fungi | dhellmann: so my next guess is that f90cebe0c40d87513372eed447d572cf698a3a81 was the state of the on-disk repo cache and for some reason zuul-cloner didn't fetch a newer commit from the merger | 17:19 |
| fungi | jeblair: ^ interesting situation | 17:19 |
| clarkb | does zuul cloner work in post? | 17:19 |
| zaro | pedroalvarez: its-* plugins are a bit of a bugger to install/configure. let me know if you need help there. | 17:19 |
| *** root_ has quit IRC | 17:19 | |
| *** monester2 has quit IRC | 17:20 | |
| fungi | clarkb: oh, very good question. this might be it punting and just using the local copy | 17:20 |
| kzaitsev_mb | oh it seems to also be connected to the 'Detected a distutils installed project ('argparse')' thing =/ | 17:20 |
| fungi | kzaitsev_mb: sounds like the recently discovered argparse bug exposed by the pip 8 release yesterday | 17:20 |
| dhellmann | clarkb, fungi : what? the post jobs don't get the latest version of the thing they're processing? | 17:20 |
| pedroalvarez | zaro: thanks for the offer, I'll ask you whenever I get to it. :) | 17:21 |
| fungi | dhellmann: this may be a behavior difference between what gerrit-git-prep did and what zuul-cloner is doing (missing feature parity? looking for variables which aren't set in post?) | 17:21 |
| *** dizquierdo has joined #openstack-infra | 17:21 | |
| kzaitsev_mb | fungi: is there anything I can do to fix it or is there some work already happening around it? | 17:22 |
| *** sputnik13 has joined #openstack-infra | 17:22 | |
| dhellmann | fungi : oh, so is this a recent change between tools? | 17:22 |
| *** sputnik13 has quit IRC | 17:22 | |
| fungi | dhellmann: we have the parameter list for that job, so can easily compare to what zuul-cloner wants | 17:22 |
| *** FallenPegasus has joined #openstack-infra | 17:22 | |
| *** asselin has joined #openstack-infra | 17:22 | |
| fungi | dhellmann: zuul-cloner is a fairly recent tool, and it's just starting to see wider adoption but so far mostly in change-based pipelines. it's possible we're missing something still to make it work for ref-based pipelines | 17:23 |
| AJaeger | pleia2, clarkb : Could you review and vote on the spec for translation setup, please? https://review.openstack.org/262545 | 17:23 |
| dhellmann | fungi : ah, I see that some other job definitions use gerrit-git-prep instead. Are we migrating, or did I build this job using zuul-cloner without realizing there was an alternative? | 17:23 |
| fungi | kzaitsev_mb: i think there's at least a bug open against pip and some discussion of a pip 8.0.1 on the way to fix it. trying to pin pip everywhere is tough since it's a bootstrapping issue most places in our infrastructure | 17:24 |
| fungi | dhellmann: gerrit-git-prep is not multi-repo aware, whereas zuul-cloner is | 17:24 |
| *** melwitt__ has quit IRC | 17:24 | |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/irc-meetings: add nova-bugs-team meeting https://review.openstack.org/270281 | 17:24 |
| fungi | gerrit-git-prep was (still is most places) used only for self-contained jobs | 17:24 |
| dhellmann | fungi : ok, in this case we don't need multiple repos so maybe I should change the job | 17:24 |
| fungi | single-repo jobs i should say | 17:24 |
| *** vgridnev has quit IRC | 17:25 | |
| *** tonytan4ever has quit IRC | 17:25 | |
| dhellmann | I mean, I'm happy to help further debugging, tool, but as a more immediate fix... | 17:25 |
| fungi | oh, the clonemap only has a single repo in it? | 17:25 |
| dhellmann | the job is supposed to run the release notes build after something merges into a project repo, so there's only one repo involved | 17:25 |
| fungi | yeah, if you're not integrating files from another repo then gerrit-git-prep should be fine for now | 17:25 |
| dhellmann | I'm not sure if that's what the clonemap is | 17:25 |
| dhellmann | ok, I'll do that right now | 17:25 |
| fungi | INFO:zuul.Cloner:Preparing 1 repositories | 17:26 |
| fungi | so yeah, zuul-cloner isn't being used for multiple repos there anyway | 17:26 |
| AJaeger | clarkb: pip it's still failing, see https://review.openstack.org/#/c/268753/1 | 17:26 |
| *** dims has joined #openstack-infra | 17:26 | |
| diltram_ | hello everyone, is any of cores available right nowe? | 17:26 |
| dhellmann | fungi : oh, wait, maybe not -- we want the job to run against master always, no matter which branch received the commit | 17:26 |
| clarkb | AJaeger: ok so we havent done anything yet | 17:26 |
| dhellmann | fungi : that way if you add a release note to the liberty branch, it shows up without losing the master notes | 17:26 |
| openstackgerrit | Merged openstack-infra/release-tools: strip _ prefix from independent series https://review.openstack.org/269802 | 17:27 |
| fungi | dhellmann: aha, that's a little trickier then. that might be what we were trying to work around by using z-c | 17:27 |
| dhellmann | fungi: yeah, there's a branch-override setting there | 17:27 |
| fungi | dhellmann: so the job only uses files from the master branch? it doesn't also try to checkout other branches? | 17:27 |
| dhellmann | hmm, it is "branch-override: master" instead of "branch-override: origin/master" I wonder if that would do it | 17:27 |
| *** diltram_ is now known as diltram | 17:27 | |
| dhellmann | fungi : it scans the other branches, but does not check them out | 17:27 |
| *** hashar has quit IRC | 17:27 | |
| dhellmann | reno does all of that | 17:28 |
| *** markus_z has quit IRC | 17:28 | |
| dhellmann | I wonder if we need a custom builder to update the local repo for this case? | 17:28 |
| *** bardia has joined #openstack-infra | 17:28 | |
| dhellmann | it can be "git remote update; git checkout master; git pull" | 17:28 |
| fungi | and i guess if we only ever run this in post-merge pipelines then there's no risk of it needing to take dependent changes into account | 17:28 |
| *** unicell has quit IRC | 17:28 | |
| dhellmann | yeah | 17:28 |
| fungi | we do have gerrit-branch-prep (i think that's the name of the builder) which takes an explicit branch name | 17:29 |
| dhellmann | fungi : do you think the fact that it says "master" instead of "origin/master" is significant? I don't know whether zuul-cloner automatically forces a local master branch to be up to date | 17:29 |
| *** flepied has joined #openstack-infra | 17:29 | |
| fungi | dhellmann: no, i suspect clarkb is right and zuul isn't providing the complete set of envvars zuul-cloner needs to be able to checkout the correct ref | 17:30 |
| fungi | but i haven't dug into the source yet to confirm | 17:30 |
| dhellmann | ok | 17:30 |
| dhellmann | I don't see "gerrit-branch-prep" anywhere in project-config files | 17:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-exim: Accept configurable routers/transports https://review.openstack.org/270329 | 17:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add storyboard exim router/transports https://review.openstack.org/270331 | 17:30 |
| *** knikolla has quit IRC | 17:31 | |
| fungi | dhellmann: branch-git-prep looks like | 17:31 |
| jeblair | SotK, Zara, fungi, mordred: ^ this is how i think storyboard should work with exim | 17:31 |
| jeblair | fungi, mordred: ^ we should also re-do the mailman config to use that | 17:31 |
| jeblair | fungi, dhellmann: what's up? :) | 17:31 |
| fungi | and needs a branch parameter set. which gets exported as BRANCH before invoking gerrit-git-prep.sh | 17:31 |
| dhellmann | jeblair : it appears that a post job meant to build release notes after a change merges to a repo is not checking out the most current version of the repository | 17:32 |
| dhellmann | jeblair : see https://etherpad.openstack.org/p/ironic-inspector-client-reno-job-debugging for notes | 17:32 |
| fungi | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/gerrit-git-prep.sh | 17:32 |
| *** sridhar_ram has joined #openstack-infra | 17:33 | |
| fungi | interprets BRANCH as ZUUL_REF which gets passed directly to git checkout | 17:33 |
| jeblair | fungi: did you dig up the jenkins link for http://logs.openstack.org/f7/f77bd83f10835b3188c6687f8238dfc6d30183e6/post/python-ironic-inspector-client-releasenotes/64703cc/console.html ? | 17:33 |
| *** ashishb has joined #openstack-infra | 17:33 | |
| fungi | so just set the branch parameter to a name suitable for git checkout | 17:33 |
| jeblair | oh on the bottom | 17:33 |
| fungi | jeblair: yep https://jenkins06.openstack.org/job/python-ironic-inspector-client-releasenotes/2/parameters/ | 17:33 |
| Zara | jeblair: ooh, thank you, I'll take a look at that :) (SotK's afk but should be back in a bit) | 17:34 |
| *** daemontool has joined #openstack-infra | 17:34 | |
| fungi | diltram: better if you say what you need help with | 17:34 |
| *** EricGonczer_ has quit IRC | 17:34 | |
| *** zhurong has quit IRC | 17:34 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: switch releasenotes job to branch-git-prep https://review.openstack.org/270333 | 17:34 |
| dhellmann | fungi : like this? ^^ | 17:34 |
| fungi | we're all very busy already so taking time to ask you what you want is time we could spend answering your question instead | 17:34 |
| anteaya | jeblair: hound can find no entries for fix_pip.sh: http://codesearch.openstack.org/?q=fix_pip.sh&i=nope&files=&repos= | 17:34 |
| diltram | fungi: I need help with locked account because of merging my account, I cannot login to my old account on gerrit using SSO | 17:34 |
| fungi | dhellmann: yep, +2 | 17:35 |
| dhellmann | fungi : great, thanks for your help with this! | 17:35 |
| diltram | fungi: I wrote an email about that on mailing list | 17:35 |
| fungi | diltram: yes, saw the e-mail. we can try to merge the new openid into your old account and then deactivate the errant newer one | 17:35 |
| fungi | diltram: what's the new account id number displayed at https://review.openstack.org/#/settings/ when you're logged in right now? | 17:36 |
| diltram | fungi: it will be great, do you need something from me to proceed with that? | 17:36 |
| diltram | fungi: 19086 | 17:36 |
| *** bpokorny_ has quit IRC | 17:37 | |
| jeblair | fungi, dhellmann: why was it using zuul-git-branch-prep? | 17:37 |
| *** bpokorny has joined #openstack-infra | 17:38 | |
| fungi | diltram: looks like your earlier account was id 14591 (username diltram) | 17:38 |
| dhellmann | jeblair : either copy-pasta on my account, or because we wanted to force the job to always use the master branch instead of wherever the commit landed and that's how I was told to do that, I'm honestly not sure | 17:38 |
| *** EricGonczer_ has joined #openstack-infra | 17:38 | |
| diltram | fungi: exactly :) | 17:38 |
| *** hashar has joined #openstack-infra | 17:38 | |
| dhellmann | I mean, I know we wanted to force the branch, I'm not sure if someone told me to do it that way or if I guessed it myself | 17:39 |
| jeblair | dhellmann: i see the comment that says "always published from master branch". what do you want to happen when something lands on stable/liberty? | 17:39 |
| *** hashar is now known as hasharAway | 17:39 | |
| jeblair | dhellmann: the job not to run, or something runs on master anyway? | 17:39 |
| dhellmann | jeblair : reno looks for notes on branches itself, so I want it to run on master | 17:39 |
| AJaeger | jeblair: run on master | 17:39 |
| jeblair | (or something i haven't thought of) | 17:39 |
| dhellmann | that way a patch on liberty doesn't remove the published notes from other branches | 17:39 |
| *** Swami has quit IRC | 17:39 | |
| *** Jeffrey4l has quit IRC | 17:40 | |
| jeblair | dhellmann: this is an unusual situation. i need to think about it for a minute. | 17:40 |
| dhellmann | jeblair : what I need is "git remote update; git checkout master; git pull" and then run sphinx. Reno will scan the branches it is told to scan. | 17:40 |
| dhellmann | it is usually told to do things like scan origin/stable/liberty and the "current" branch, for example | 17:40 |
| dhellmann | as separate steps, preparing notes docs for different branches | 17:41 |
| fungi | diltram: okay, log out of gerrit and back in again and it should be fixed now | 17:41 |
| *** _et_ has joined #openstack-infra | 17:41 | |
| diltram | fungi: thank you very much, it works :) | 17:41 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Always go to Story Detail Page after adding Story https://review.openstack.org/270337 | 17:41 |
| pleia2 | AJaeger: done (also, airport bound for a conference, will be mostly gone for the rest of the week) | 17:42 |
| phschwartz | Has there been a change in the dependencies for zuul and nodepool around apscheduler? It seems that nodepool is looking for an older version then zuul which moves aspscheduler.scheduler. (zuul fails on the older version and works on the latest) | 17:42 |
| clarkb | pabelanger: ^ | 17:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Set logserver variables using temp url key only https://review.openstack.org/270338 | 17:42 |
| openstackgerrit | Merged openstack-infra/irc-meetings: add nova-bugs-team meeting https://review.openstack.org/270281 | 17:43 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 17:43 |
| dhellmann | jeblair : The gate job is more normal. There we need it to run against the current branch, so we can ensure that new notes are included in the check and don't break the doc build. But the output isn't published, so it doesn't matter if it misses some things on a different branch. | 17:43 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Always go to Story Detail Page after adding Story https://review.openstack.org/270337 | 17:43 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Set logserver variables using temp url key only https://review.openstack.org/270338 | 17:44 |
| *** esikachev has quit IRC | 17:44 | |
| pabelanger | clarkb: phschwartz: right, have you upgraded apscheduler? Right now, zuul and nodepool depend on different versions, trying to get 262017 for nodepool to get them back on the same version | 17:44 |
| *** daemontool has quit IRC | 17:44 | |
| pabelanger | phschwartz: clarkb: so, nodepool and zuul on same host will likely fail | 17:44 |
| fungi | phschwartz: pabelanger has a change proposed to fix that i think | 17:44 |
| fungi | oh, he just said that | 17:45 |
| AJaeger | pleia2: thanks! Safe travels! | 17:45 |
| fungi | reminder that i should read all of scrollback before replying | 17:45 |
| pabelanger | fungi: Yup, I think it is actually read to land now | 17:45 |
| fungi | i thought i approved it but maybe not | 17:45 |
| fungi | oh, nevermind, what i approved was the sqlalchemy cap bum[ | 17:46 |
| fungi | bump | 17:46 |
| phschwartz | pabelanger: ok, that will be a huge help, but not until it is pushed to pip. By anychance do you know what versions these were added in so I can pin them in my puppet manifests so I can get my single node running again | 17:46 |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config: Remove fuel-library puppet 3.6 and 3.7 unit jobs https://review.openstack.org/270340 | 17:46 |
| *** ayoung has joined #openstack-infra | 17:47 | |
| pabelanger | phschwartz: I'm hoping for a nodepool release shortly. Want to confirm clarkb and greghaynes are happy with nodepool-builder first | 17:47 |
| anteaya | pleia2: happy conferencing | 17:47 |
| greghaynes | pabelanger: hrm? | 17:47 |
| *** esikachev has joined #openstack-infra | 17:47 | |
| pabelanger | phschwartz: iirc, 2.1.4 was the old, and 3.0.5 is new apscheduler | 17:47 |
| Swanson | nodepool question for anyone. I have an image that never built and node pool has been trying to delete it for a couple hours now. How do I sort this? | 17:47 |
| greghaynes | Swanson: Are you running off master? | 17:47 |
| phschwartz | pabelanger: I meant, what version of zuul moved to the new apscheduler | 17:47 |
| greghaynes | Swanson: because there is a bug right now related to that | 17:48 |
| phschwartz | pabelanger: that would be the quickest fix. move back to the old zuul | 17:48 |
| pabelanger | phschwartz: oh, we have not released it to pip yet | 17:48 |
| clarkb | pabelanger: I think the answer is we dont know yet because evrything keeps breaking | 17:48 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds openstack-doc-tools as a group for doc tools https://review.openstack.org/268313 | 17:48 |
| fungi | phschwartz: commit 1a4ad06 is what changed it, but hasn't made it into any tag get according to git tag --contains | 17:48 |
| clarkb | dib pip nodepool etc | 17:48 |
| phschwartz | hmm. I am curious as to how I am getting it. | 17:48 |
| greghaynes | Yep :( | 17:48 |
| pabelanger | phschwartz: 2.1.0 zuul should still have the old apscheduler requirment | 17:48 |
| Swanson | greghaynes, I believe so. This is asselin's one stop CI. I just loaded it up yesterday. | 17:48 |
| pabelanger | requirement* | 17:48 |
| *** asalkeld has joined #openstack-infra | 17:48 | |
| fungi | phschwartz: installing zuul from master branch tip rather than release version? | 17:49 |
| fungi | phschwartz: yeah, i believe it's a continuous deployment model in that case | 17:49 |
| asselin | what's up? | 17:49 |
| greghaynes | Swanson: ah, so yes, there was a bug where if an image build failed nodepool fails to delete the image. That bug is now fixed on master but sounds like you hit that issue - to remedy you have to remove the image from the db | 17:49 |
| asselin | cbader_, ^^ | 17:50 |
| clarkb | Swanson: DELETE FROM dib_image WHERE state = 1; | 17:50 |
| openstackgerrit | Merged openstack-infra/project-config: Remove unused fix_pip.sh https://review.openstack.org/269962 | 17:50 |
| asalkeld | hi, sorry is this is already "a known thing", but kolla-mesos is hitting this pip 8.0 issue (conflicts with argparse) do you guys have any suggestions on what to do? http://logs.openstack.org/51/270151/3/check/gate-kolla-mesos-pep8/1832137/console.html#_2016-01-20_15_50_30_167 | 17:50 |
| Swanson | greghaynes, clarkb, I was afraid of that. Thanks for the sql line. | 17:50 |
| clarkb | asalkeld: I think we may be trying to figure that out. ianw has proposed a revert to pip as one option | 17:51 |
| *** _nadya_ has joined #openstack-infra | 17:51 | |
| asalkeld | ok, clarkb so just wait a bit? | 17:51 |
| *** FallenPegasus has quit IRC | 17:51 | |
| clarkb | asalkeld: I guess? its a tricky situation and not sure we have a good idea of the best fix yet so waiting cant hurt | 17:52 |
| asalkeld | thanks - I'll chill out and do some coding ... | 17:52 |
| *** FallenPegasus has joined #openstack-infra | 17:53 | |
| clarkb | pinning pip is sort of difficult | 17:53 |
| AJaeger | asalkeld: good idea ;) | 17:53 |
| clarkb | and not good long term | 17:53 |
| clarkb | distros could fix packaging but thats slow | 17:53 |
| clarkb | we could run everything out of a system venv | 17:54 |
| fungi | well, the venv "fix" doesn't help the argparse bug | 17:54 |
| AJaeger | clarkb: there seem to be some genuine bugs in pip 8, like the argparse one | 17:54 |
| fungi | we're hitting it even in virtualenvs | 17:54 |
| *** jcoufal_ has quit IRC | 17:54 | |
| *** vgridnev has joined #openstack-infra | 17:54 | |
| asalkeld | could pip accept a patch to change the error into a warning? | 17:54 |
| clarkb | fungi oh because its part of python in 2.7 | 17:54 |
| AJaeger | clarkb: yes | 17:54 |
| fungi | asalkeld: that's what the revert would do | 17:55 |
| fungi | asalkeld: it was a warning for about the past year | 17:55 |
| asalkeld | cool | 17:55 |
| clarkb | ya so that is proposed | 17:55 |
| AJaeger | clarkb: and oslo.config has it in requirements | 17:55 |
| greghaynes | clarkb: fungi if you have system-site-packages on youlll hit issues... | 17:55 |
| dstufft | https://github.com/pypa/pip/issues/3384#issuecomment-173303806 | 17:55 |
| fungi | asalkeld: part of the issue is that one year is, like, half a release cycle for debian/ubuntu lts | 17:55 |
| dstufft | also https://github.com/pypa/pip/issues/3384#issuecomment-173231050 | 17:55 |
| clarkb | AJaeger: we can remove it from requirements there since it was only necessary for python 2.6 | 17:55 |
| AJaeger | dhellmann, dims: What about approving https://review.openstack.org/#/c/270149/ and cuting a new olso.config release? | 17:55 |
| AJaeger | clarkb: See https://review.openstack.org/#/c/270149/ ;) | 17:56 |
| greghaynes | oh wow, argparse :( | 17:56 |
| openstackgerrit | Merged openstack-infra/project-config: increase large-ops job timeout https://review.openstack.org/270304 | 17:56 |
| greghaynes | this is a good day to go hide | 17:56 |
| dims | AJaeger : has a +2 from me | 17:56 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/irc-meetings: Move the Neutron drivers meeting https://review.openstack.org/269780 | 17:56 |
| dhellmann | AJaeger : +2a on the patch, I'll let dims decide on the release | 17:56 |
| *** Swami has joined #openstack-infra | 17:57 | |
| fungi | greghaynes: even without system site packages, installing argparse in a virtualenv will run into this | 17:57 |
| *** knikolla has joined #openstack-infra | 17:57 | |
| greghaynes | fungi: wow | 17:58 |
| asalkeld | dhellmann: just a note, kolla-mesos runs into this issue even though we don't have it in requirements | 17:58 |
| asalkeld | (argparse) | 17:58 |
| fungi | because the egg-info file is provided by stdlib and so always in the path | 17:58 |
| dougwig | topic is bugged in #openstack-meeting-4, fyi | 17:58 |
| asalkeld | so i don't think that patch is going to do anything | 17:58 |
| dhellmann | asalkeld : are you using oslo.config or another project that does include it? | 17:58 |
| *** weshay_xchat has quit IRC | 17:58 | |
| fungi | dougwig: thanks, i'll fix | 17:58 |
| asalkeld | dhellmann: aaa, that might be the case... we do use oslo.config | 17:58 |
| dhellmann | asalkeld : yeah, that's probably it | 17:59 |
| *** shardy has quit IRC | 17:59 | |
| AJaeger | but there are more repos with argparse in requirements ;( | 17:59 |
| dougwig | fungi: ty | 17:59 |
| fungi | any of your transitive dependencies could potentially trigger it, not just ones we maintain | 17:59 |
| AJaeger | exactly | 18:00 |
| *** jsavak has quit IRC | 18:00 | |
| dims | dhellmann : https://review.openstack.org/270346 | 18:00 |
| fungi | say you have a declared dependency on pyfoo and pyfoo in turn wants to support python 2.6 so includes argparse in its requirements | 18:00 |
| *** jsavak has joined #openstack-infra | 18:01 | |
| *** sfinucan has quit IRC | 18:01 | |
| dhellmann | dims : how is that differennt from AJaeger's patch? | 18:01 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 18:01 |
| AJaeger | all the python-*clients require argparse ;( | 18:01 |
| *** tonytan4ever has joined #openstack-infra | 18:01 | |
| *** bardia has quit IRC | 18:01 | |
| asalkeld | doh! | 18:01 |
| * AJaeger works on patches now... | 18:02 | |
| fungi | maybe it's time to remove those, since we no longer support python 2.6 on any of our branches | 18:02 |
| dims | dhellmann : oops i thought that was a requirements/ repo patch from him :) | 18:02 |
| *** weshay_xchat has joined #openstack-infra | 18:02 | |
| dims | dhellmann : y, that's enough | 18:02 |
| dhellmann | dims : :-) | 18:02 |
| dhellmann | fungi : ++ | 18:02 |
| clarkb | fungi ya just would be nice to not need it under such circumstances | 18:02 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic https://review.openstack.org/270010 | 18:02 |
| clarkb | but good cleanup regardless | 18:02 |
| fungi | but it's not guaranteed to solve this if we still depend on something else which transitively declares a dependency on argparse | 18:02 |
| clarkb | nodepool for example | 18:03 |
| fungi | due to actually still having upstream support for 2.6 | 18:03 |
| fungi | (not an openstack project) | 18:03 |
| dims | dhellmann : spotted argparse in osprofiler as well https://review.openstack.org/270347 | 18:03 |
| *** weshay_xchat is now known as weshay | 18:03 | |
| jeblair | http://codesearch.openstack.org/?q=argparse&i=nope&files=requirements.txt&repos= | 18:04 |
| *** crinkle_ is now known as crinkle | 18:04 | |
| * greghaynes wonders what that i=nope arg is for | 18:04 | |
| fungi | greghaynes: case sensitivity | 18:04 |
| jeblair | greghaynes: ignore case? | 18:04 |
| greghaynes | haha, and nope is a boolean? | 18:04 |
| greghaynes | someone had fun | 18:04 |
| jeblair | 68 file it says | 18:04 |
| fungi | though i have to say, i approve of "nope" as a boolean value | 18:05 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 18:05 |
| fungi | with "yep" being the inverse | 18:05 |
| dhellmann | dims : +2a | 18:05 |
| fungi | i assume it's unlikely we'll have argparse in test-requirements.txt | 18:05 |
| fungi | or setup.py | 18:05 |
| *** wei-wang has quit IRC | 18:07 | |
| dhellmann | dims, fungi : http://paste.openstack.org/show/484442/ | 18:07 |
| jeblair | dhellmann, fungi: i agree that zuul-cloner is missing some things needed for post jobs. i also worry a little bit about whether any of our methods will result in the correct values for branches other than the branch the commit landed on -- at least, i don't think we've ever covered that in testing. | 18:07 |
| dhellmann | jeblair : I'm not aware of any jobs with this need, so maybe a custom script to supplement something that is close to correct is in order | 18:08 |
| dhellmann | ? | 18:08 |
| *** esikachev has quit IRC | 18:08 | |
| *** unicell has joined #openstack-infra | 18:08 | |
| jeblair | dhellmann: maybe if we find that it is somehow incompatible with what we want zuul-cloner to do in post | 18:09 |
| stevemar | so, what exactly is the logic behind when a job that is gating, and seems to have passed everything, resets? | 18:09 |
| jeblair | dhellmann: i don't know that's the case -- i mostly point it out as something we've never considered worth testing. when we add post support to cloner, we can add tests for that if we think it's compatible | 18:09 |
| asselin | &ngelica0116121 | 18:09 |
| *** esikachev has joined #openstack-infra | 18:10 | |
| AJaeger | dhellmann: I'm currently going through the list you pasted... | 18:10 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: remove argparse from requirements list https://review.openstack.org/270354 | 18:10 |
| *** Keedya has joined #openstack-infra | 18:10 | |
| *** derekh has quit IRC | 18:10 | |
| dhellmann | dims , fungi : ^^ | 18:10 |
| phschwartz | What would cause zuul to return "uat-tempest-run: NOT_REGISTERED" even if a status shows "build:uat-tempest-run:devstack-trusty 0 0 1" | 18:10 |
| Swanson | greghaynes, clarkb, That worked a treat. Thanks! | 18:10 |
| AJaeger | dhellmann: had it already generated locally ;) | 18:10 |
| lifeless_ | AJaeger: pin to pip <8 | 18:11 |
| dims | dhellmann : that's a lot :( | 18:11 |
| dhellmann | jeblair : ok, I didn't want to add requirements to zuul-cloner unnecessarily | 18:11 |
| fungi | lifeless_: easier said that done | 18:11 |
| anteaya | stevemar: if a patch in the gate is behind any number of patches and any of the preceding patches fails the first failed patch is removed and all subsequent patches are retested without the failing patch | 18:11 |
| lifeless_ | fungi: yeah, I know | 18:11 |
| anteaya | stevemar: even if the tests on the subsequent patches were passing | 18:11 |
| AJaeger | dhellmann: https://review.openstack.org/#/q/topic:rm-argparse | 18:11 |
| dhellmann | dims : yeah, as lifeless_ says we may get farther by pinning pip | 18:11 |
| fungi | er, than | 18:11 |
| AJaeger | lifeless_: we cannot easily pin pip | 18:11 |
| anteaya | stevemar: as they may have been passing in error due to the failing patch | 18:11 |
| lifeless_ | argparse can turn up anywhere in the transitive dep chain | 18:12 |
| *** lifeless_ is now known as lifeless | 18:12 | |
| jeblair | dhellmann: well, 'works in post' is definitely a requirement -- i think once we dig into it we'll know if there's any additional complexity around other branches | 18:12 |
| lifeless | and cause the issue | 18:12 |
| openstackgerrit | Andreas Jaeger proposed openstack/diskimage-builder: Remove argparse from requirements https://review.openstack.org/270356 | 18:12 |
| lifeless | e.g. if testrepository depends on argparse | 18:12 |
| *** trown|lunch is now known as trown | 18:12 | |
| fungi | worse, the numerous patches needed to pin pip everywhere need to get tracked and then undone again once we're able to support newer pip, and also identifying that we _have_ gotten everything working with newer pip everywhere is hard without trying to undo those, so may take multiple iterations | 18:12 |
| *** ociuhandu has joined #openstack-infra | 18:12 | |
| lifeless | we've got nearly 400 deps in our transitive chain | 18:12 |
| *** sputnik13 has joined #openstack-infra | 18:13 | |
| *** Keedya_ has joined #openstack-infra | 18:13 | |
| stevemar | anteaya: seems like a bit hit to restart everything just cause one thing failed, especially if the projects are unrelated | 18:13 |
| lifeless | so if just one has an argparse reference we'll still hit the issue | 18:13 |
| *** jsavak has quit IRC | 18:13 | |
| jeblair | lifeless: i take it we still need post-trusty pip in order to have constraints support? | 18:13 |
| AJaeger | removing argparse does not hurt - but indeed is not the silver bullet | 18:13 |
| dstufft | 7.0 | 18:13 |
| lifeless | jeblair: 7+ | 18:13 |
| dstufft | or 7.1? | 18:13 |
| anteaya | stevemar: fire fighting is not the best time to try to redesign the gate algorithm | 18:13 |
| dstufft | for constraints | 18:14 |
| fungi | yeah, didn't mean to imply that removing argparse from our direct requirements would solve much of this, just that it's cleanup we should eventually be doing anyway | 18:14 |
| *** jsavak has joined #openstack-infra | 18:14 | |
| anteaya | stevemar: after the fire is out if folks would like to discuss the gate algorithm design that is fine | 18:14 |
| lifeless | jeblair: maybe 7.1; I'd need to check | 18:14 |
| *** kgiusti1 has joined #openstack-infra | 18:14 | |
| *** NikitaKonovalov2 has joined #openstack-infra | 18:14 | |
| *** aysyd_ has joined #openstack-infra | 18:14 | |
| *** maximov_ has joined #openstack-infra | 18:14 | |
| *** xek_ has joined #openstack-infra | 18:14 | |
| anteaya | stevemar: I hear your frustration | 18:14 |
| stevemar | anteaya: of course, just talking aloud | 18:14 |
| *** othermordred_ has joined #openstack-infra | 18:14 | |
| *** dtardivel_ has joined #openstack-infra | 18:14 | |
| anteaya | stevemar: thanks for sharing :) | 18:14 |
| lifeless | fungi: so, I'd actually argue that we need to solve this systematically in pip, because any package published before argparse was included in the stdlib will trigger this forever, and we can't require thousands of packages to update | 18:15 |
| *** jtomasek has quit IRC | 18:15 | |
| lifeless | fungi: and we don't have systematic widespread use of markers yet to do clean conditional dependencies on argparse | 18:15 |
| *** rajinir_ has joined #openstack-infra | 18:15 | |
| dstufft | pip 8.0.1 will fix argparse specifically for sure. | 18:15 |
| stevemar | anteaya: data points would be helpful | 18:15 |
| fungi | lifeless: i never disagreed with that | 18:15 |
| *** yfried has joined #openstack-infra | 18:15 | |
| lifeless | fungi: cool | 18:15 |
| *** lindsayk- has joined #openstack-infra | 18:15 | |
| anteaya | stevemar: data is always helpful, agreed | 18:15 |
| *** _nadya__ has joined #openstack-infra | 18:15 | |
| fungi | lifeless: all i did was say we could stand to clean up some python 2.6-only cruft while we're at it | 18:15 |
| dstufft | as well as fix virtualenv's with system-site-pacakges for sure | 18:15 |
| lifeless | fungi: I have not read the 8 hours of backscroll; I commented here in reply to AJaeger's ping | 18:15 |
| stevemar | anteaya: anywho, time and place :) | 18:15 |
| anteaya | stevemar: yup, happy to listen further at another time and place :) | 18:16 |
| clarkb | stevemar: anteaya the zuul docs cover this in depth fwiw | 18:16 |
| stevemar | anteaya: it just came up now, cause like you said, wee bit of frustration | 18:16 |
| lifeless | anyhow, I need to go feed the tribe | 18:16 |
| *** taron2 has joined #openstack-infra | 18:16 | |
| clarkb | if they arent related then dont share a queue | 18:16 |
| anteaya | stevemar: yup, I hear that, you are in good company | 18:16 |
| *** migi_ has joined #openstack-infra | 18:16 | |
| clarkb | if they are related its basically tequired | 18:16 |
| fungi | lifeless: what i think we're weighing is whether we can effectively pin pip <8.0.0 everywhere before there exists a version of pip which makes doing that unnecessary | 18:16 |
| stevemar | clarkb: and all the 'integrated' projects are required by each other | 18:17 |
| *** smcginni1 has joined #openstack-infra | 18:17 | |
| *** kragniz_ has joined #openstack-infra | 18:17 | |
| *** nihilife1 has joined #openstack-infra | 18:18 | |
| openstackgerrit | Merged openstack-infra/nodepool: Raise SQLAlchemy upper cap to match global requirements https://review.openstack.org/267636 | 18:18 |
| *** e0ne has joined #openstack-infra | 18:18 | |
| *** alazarev_ has joined #openstack-infra | 18:18 | |
| *** Keedya_ has quit IRC | 18:18 | |
| *** aviau_ has joined #openstack-infra | 18:19 | |
| *** edwarnicke_ has joined #openstack-infra | 18:19 | |
| *** mriedem has quit IRC | 18:19 | |
| *** dhellmann_ has joined #openstack-infra | 18:20 | |
| openstackgerrit | Merged openstack-infra/project-config: switch releasenotes job to branch-git-prep https://review.openstack.org/270333 | 18:20 |
| *** dteselkin_ has joined #openstack-infra | 18:20 | |
| *** peterlisak_ has joined #openstack-infra | 18:20 | |
| *** isq_ has joined #openstack-infra | 18:20 | |
| *** a2hill has joined #openstack-infra | 18:21 | |
| clarkb | stevemar: anteaya http://docs.openstack.org/infra/zuul/gating.html | 18:21 |
| *** clayton_ has joined #openstack-infra | 18:21 | |
| anteaya | clarkb: thank you :) | 18:21 |
| *** ajo_ has joined #openstack-infra | 18:21 | |
| *** kzaitsev_mb has quit IRC | 18:21 | |
| *** Keedya has quit IRC | 18:22 | |
| *** _nadya_ has quit IRC | 18:22 | |
| *** dizquierdo has quit IRC | 18:22 | |
| *** annegentle has quit IRC | 18:22 | |
| *** rajinir has quit IRC | 18:22 | |
| *** Eva-i has quit IRC | 18:22 | |
| *** thorst has quit IRC | 18:22 | |
| *** aysyd has quit IRC | 18:22 | |
| *** nihilifer has quit IRC | 18:22 | |
| *** kgiusti has quit IRC | 18:22 | |
| *** migi has quit IRC | 18:22 | |
| *** clayton has quit IRC | 18:22 | |
| *** taron1 has quit IRC | 18:22 | |
| *** claudiub has quit IRC | 18:22 | |
| *** dtardivel has quit IRC | 18:22 | |
| *** treaki__ has quit IRC | 18:22 | |
| *** edwarnicke has quit IRC | 18:22 | |
| *** othermordred has quit IRC | 18:22 | |
| *** maximov has quit IRC | 18:22 | |
| *** dteselkin has quit IRC | 18:22 | |
| *** med_ has quit IRC | 18:22 | |
| *** lindsayk has quit IRC | 18:22 | |
| *** xek has quit IRC | 18:22 | |
| *** peterlisak has quit IRC | 18:22 | |
| *** ajo has quit IRC | 18:22 | |
| *** tchaypo has quit IRC | 18:22 | |
| *** cody-somerville- has quit IRC | 18:22 | |
| *** aviau has quit IRC | 18:22 | |
| *** mrunge has quit IRC | 18:22 | |
| *** wfoster has quit IRC | 18:22 | |
| *** dhellmann has quit IRC | 18:22 | |
| *** isq has quit IRC | 18:22 | |
| *** hamzy has quit IRC | 18:22 | |
| *** NikitaKonovalov has quit IRC | 18:22 | |
| *** ptoohill has quit IRC | 18:22 | |
| *** akuznetsova has quit IRC | 18:22 | |
| *** sbadia has quit IRC | 18:22 | |
| *** alazarev has quit IRC | 18:22 | |
| *** kragniz has quit IRC | 18:22 | |
| *** smcginnis has quit IRC | 18:22 | |
| *** aysyd_ is now known as aysyd | 18:22 | |
| *** NikitaKonovalov2 is now known as NikitaKonovalov | 18:22 | |
| *** alazarev_ is now known as alazarev | 18:22 | |
| *** asalkeld has left #openstack-infra | 18:22 | |
| *** othermordred_ is now known as othermordred | 18:22 | |
| *** peterlisak_ is now known as peterlisak | 18:22 | |
| *** akuznetsova has joined #openstack-infra | 18:22 | |
| *** dtardivel_ is now known as dtardivel | 18:22 | |
| *** rajinir_ is now known as rajinir | 18:22 | |
| clarkb | so should we start working on pins? | 18:22 |
| *** clayton_ is now known as clayton | 18:22 | |
| *** dhellmann_ is now known as dhellmann | 18:22 | |
| *** gyee has quit IRC | 18:22 | |
| *** clayton has quit IRC | 18:22 | |
| *** santoshk has joined #openstack-infra | 18:22 | |
| *** med_ has joined #openstack-infra | 18:22 | |
| *** cody-somerville- has joined #openstack-infra | 18:23 | |
| *** maximov_ is now known as maximov | 18:23 | |
| *** gyee has joined #openstack-infra | 18:23 | |
| *** davidhadas_ has quit IRC | 18:23 | |
| clarkb | devstack all branches, grenade?, image builds (puppet install script) | 18:23 |
| *** med_ is now known as Guest65103 | 18:23 | |
| clarkb | then rebuild assuming we dont need something else in dib | 18:23 |
| *** clayton has joined #openstack-infra | 18:23 | |
| *** sbadia has joined #openstack-infra | 18:23 | |
| clarkb | the get pip cached file will bewrong | 18:23 |
| *** mhickey has quit IRC | 18:23 | |
| dims | clarkb : should we ask dstufft ? | 18:23 |
| dims | clarkb : about a revert? | 18:23 |
| clarkb | dims ianw proposed it but not sure if that will happen or how far oht it is | 18:24 |
| *** mriedem has joined #openstack-infra | 18:24 | |
| *** wfoster has joined #openstack-infra | 18:24 | |
| *** annegentle has joined #openstack-infra | 18:24 | |
| dims | clarkb : y reading https://github.com/pypa/pip/issues/3384 | 18:25 |
| *** fedexo has quit IRC | 18:25 | |
| greghaynes | clarkb: I think dib itself works, theres some obvious element breakage depending on what you try to pip install in an element | 18:25 |
| clarkb | greghaynes: gotcha | 18:25 |
| *** mrunge has joined #openstack-infra | 18:26 | |
| greghaynes | dib gates on being able to build with ironic which isn't able to install ATM so trying to pin around that | 18:26 |
| *** edwarnicke_ is now known as edwarnicke | 18:26 | |
| cbader_ | so I am caught in the pip 8.0.0 is there a way to downgrade my pip to something else? I have new install of Common-CI I can't complete because of the pip issue. | 18:27 |
| *** _nadya__ has quit IRC | 18:27 | |
| dstufft | the distutils thing is 3 things, 2/3 are going to be flat out fixed | 18:28 |
| *** treaki__ has joined #openstack-infra | 18:28 | |
| dstufft | 1/3 things I don't know yet, it won't be a revert without a plan that isn't just "wait another year and hope this time people actually read the words on their screen" | 18:28 |
| *** dizquierdo has joined #openstack-infra | 18:28 | |
| clarkb | cbader_: we are currently discussing how to handle it | 18:28 |
| cbader_ | clarkb, ok thanks I have been watching and googling but haven't figured out how to downgrade pip. | 18:29 |
| *** thorst has joined #openstack-infra | 18:29 | |
| *** tchaypo has joined #openstack-infra | 18:29 | |
| dstufft | pip install 'pip<8' | 18:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/release-tools: Remove argparse from requirements https://review.openstack.org/270404 | 18:30 |
| *** asselin__ has joined #openstack-infra | 18:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/reviewstats: Remove argparse from requirements https://review.openstack.org/270405 | 18:30 |
| *** baoli_ has quit IRC | 18:30 | |
| *** Eva-i has joined #openstack-infra | 18:30 | |
| lifeless | clarkb: well mtreinish put up pins for devstack already; I guess they failed? | 18:30 |
| *** rossella_s has quit IRC | 18:31 | |
| AJaeger | lifeless: that does not work for pep8 or docs job | 18:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/storyboard: Remove argparse from requirements https://review.openstack.org/270406 | 18:31 |
| lifeless | AJaeger: I know | 18:31 |
| *** baoli has joined #openstack-infra | 18:31 | |
| AJaeger | lifeless: I think devstack worked | 18:31 |
| *** rossella_s has joined #openstack-infra | 18:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul: Remove argparse from requirements https://review.openstack.org/270407 | 18:31 |
| *** hamzy has joined #openstack-infra | 18:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/yaml2ical: Remove argparse from requirements https://review.openstack.org/270408 | 18:32 |
| *** asselin has quit IRC | 18:32 | |
| *** esikachev has quit IRC | 18:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/jeepyb: Remove argparse from requirements https://review.openstack.org/270409 | 18:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/git-review: Remove argparse from requirements https://review.openstack.org/270410 | 18:32 |
| *** otter768 has joined #openstack-infra | 18:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/git-restack: Remove argparse from requirements https://review.openstack.org/270411 | 18:34 |
| fungi | lifeless: part of the issue mtreinish encountered with the devstack pins, i think, was tempest still breaking on its virtualenv-based install | 18:34 |
| *** Qiming has quit IRC | 18:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/gear: Remove argparse from requirements https://review.openstack.org/270412 | 18:35 |
| *** kushal has quit IRC | 18:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/git-restack: Fix .gitreview https://review.openstack.org/270413 | 18:35 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/elastic-recheck: Remove argparse from requirements https://review.openstack.org/270414 | 18:35 |
| fungi | also complications of pinning in kilo pre-constraints, since grenade won't pass master without liberty working, and won't pass liberty without kilo working | 18:35 |
| *** wfoster has quit IRC | 18:35 | |
| *** annegentle has quit IRC | 18:35 | |
| * AJaeger is done with spamming ;) | 18:36 | |
| *** kushal has joined #openstack-infra | 18:36 | |
| *** Sukhdev has joined #openstack-infra | 18:36 | |
| *** baoli_ has joined #openstack-infra | 18:36 | |
| *** ayoung has quit IRC | 18:37 | |
| dims | AJaeger : nice work :) | 18:37 |
| *** ayoung has joined #openstack-infra | 18:37 | |
| clarkb | ya you have to it in order | 18:37 |
| *** wfoster has joined #openstack-infra | 18:37 | |
| *** annegentle has joined #openstack-infra | 18:37 | |
| fungi | yep, just pointing out why kilo needed fixing urgently, even though it's harder to skin since pre-constraints | 18:38 |
| lifeless | fungi: makes sense | 18:38 |
| *** otter768 has quit IRC | 18:38 | |
| lifeless | what can I do to help ? | 18:39 |
| fungi | oh, and the fact that for much of this what we need to pin is not pip directly, but virtualenv since it vendors pip | 18:39 |
| *** baoli has quit IRC | 18:39 | |
| lifeless | yeah, virtualenv<14 | 18:39 |
| *** yfried has quit IRC | 18:39 | |
| *** KimJ has quit IRC | 18:40 | |
| jpr1 | in the new http://docs.openstack.org/infra/openstackci/third_party_ci.html docs (nice work btw), the configuration of the common.yaml for the ci node (that runs jenkins etc) nicely guides the users to provide data for the non-commented out fields. oscc_file_contents is one such uncommented field, however the default provided contents says "Do Not Edit - - Generated & Managed by Puppet". Is this really the value we want there or should we put the ou | 18:40 |
| Sam-I-Am | if i'm looking to disable expensive jobs for docs-only patches to networking-ovn, is there any special ordering or other magic i should be aware of in layout.yaml? | 18:40 |
| Sam-I-Am | sort of unfamiliar territory for me | 18:40 |
| AJaeger | Sam-I-Am: see how other repos do it.. | 18:41 |
| clarkb | jpr1 you cut off but I think you want to update puppet to set the values | 18:41 |
| AJaeger | Sam-I-Am: add to the list starting at line 723 "Skip these long-running and expensive devstack jobs when the..." | 18:42 |
| Sam-I-Am | AJaeger: thats what i was looking at | 18:43 |
| Sam-I-Am | didnt know if there was some separation in there between 'core' and not-really-core projects | 18:43 |
| Sam-I-Am | things seem to be scattered around | 18:43 |
| fungi | lifeless: i'm not 100% sure where we are at this point. i think the pip 8 with python 2.7 installing argparse in even in isolated virtualenvs has us sufficiently deadlocked that we may be just waiting to see what pip upstream does | 18:43 |
| Sam-I-Am | like openstack-ansible is by itself | 18:43 |
| openstackgerrit | Armando Migliaccio proposed openstack/requirements: Cap keystonemiddleware version https://review.openstack.org/270417 | 18:43 |
| jpr1 | ok. will try that route. it's not clear if the later puppet apply gathers this info from somewhere or if it needs to be fed in | 18:43 |
| fungi | lifeless: mtreinish and/or dims may have more current status info than i do | 18:43 |
| jpr1 | I assume this is pointing at the cloud the ci functions will use as the build platform, is that right? | 18:44 |
| clarkb | jpr1 fed in via hiera I think | 18:44 |
| clarkb | jpr1 yes | 18:44 |
| fungi | lifeless: the up-side, i guess, is that this has also broken diskimage-builder so we may not be seeing a lot of workers end up with pip 8/virtualenv 14 preinstalled yet | 18:45 |
| fungi | except bare-.* workers and devstack-.* in rackspace | 18:45 |
| AJaeger | dims: do you want to abandon https://review.openstack.org/#/c/270346/ since it's a duplicate? | 18:45 |
| fungi | bare-.* workers in rackspace and hpcloud | 18:45 |
| dims | AJaeger : already done | 18:45 |
| AJaeger | dims: oh you did already, sorry, ignore me | 18:45 |
| openstackgerrit | Drew Thorstensen proposed openstack-infra/nodepool: Bump APScheduler to >=3.0 https://review.openstack.org/262017 | 18:46 |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental neutron jobs for the new pecan based api server https://review.openstack.org/268234 | 18:46 |
| *** electrofelix has quit IRC | 18:47 | |
| cbader_ | so I got past the pip issue but looks like the puppet script the created the common-ci didn't create zuul, nodepool and apache conf files so looks like wait for pip fix I guess | 18:47 |
| lifeless | fungi: so, I'm catching up on the pip issue and will be helping there as I can | 18:48 |
| *** fhubik has quit IRC | 18:48 | |
| jpr1 | clarkb: i thought environments/common.yaml is the heira source | 18:48 |
| fungi | lifeless: awesome--thanks | 18:48 |
| *** wfoster has quit IRC | 18:48 | |
| *** annegentle has quit IRC | 18:48 | |
| AJaeger | thanks, lifeless | 18:48 |
| *** wfoster has joined #openstack-infra | 18:48 | |
| *** annegentle has joined #openstack-infra | 18:48 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add storyboard exim router/transports https://review.openstack.org/270331 | 18:48 |
| clarkb | jpr1 asselin_ would know better than me | 18:48 |
| lifeless | fungi: I think I've suggested before we should have constraints of some sort on the things we install via pip - I think effort put into pins now would be mainly useful if it was done with an eye to having a central control point for future firedrills | 18:49 |
| fungi | jeblair: is 270331 based on the discussion i was having with SotK and Zara earlier? | 18:49 |
| *** doug-fish has quit IRC | 18:49 | |
| jpr1 | ok. btw, i'm referencing this line in common.yaml https://github.com/openstack-infra/puppet-openstackci/blob/master/contrib/single_node_ci_data.yaml#L30 | 18:50 |
| *** claudiub has joined #openstack-infra | 18:50 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 18:50 |
| jpr1 | i'll run it with the provided value | 18:51 |
| *** gothicmi1dfood has quit IRC | 18:53 | |
| *** bardia has joined #openstack-infra | 18:53 | |
| AJaeger | sdague: could you review https://review.openstack.org/#/c/270143 and the logic behind it some time, please? (not urgent) | 18:53 |
| *** doug-fish has joined #openstack-infra | 18:53 | |
| *** hdd has joined #openstack-infra | 18:54 | |
| *** tjones1 has joined #openstack-infra | 18:54 | |
| *** flaper87 has quit IRC | 18:54 | |
| *** flaper87 has joined #openstack-infra | 18:54 | |
| pabelanger | fungi: clarkb: do you mind approving 262017 before another rebase :) | 18:55 |
| *** gothicmindfood has joined #openstack-infra | 18:56 | |
| *** ansiwen has quit IRC | 18:56 | |
| *** ansiwen has joined #openstack-infra | 18:56 | |
| *** eharney has quit IRC | 18:57 | |
| *** eharney has joined #openstack-infra | 18:57 | |
| *** 77CAALV3Z has joined #openstack-infra | 18:57 | |
| *** ari has quit IRC | 18:57 | |
| *** ari has joined #openstack-infra | 18:57 | |
| *** dkranz has quit IRC | 18:57 | |
| *** dkranz has joined #openstack-infra | 18:57 | |
| *** edmondsw has quit IRC | 18:57 | |
| *** edmondsw has joined #openstack-infra | 18:57 | |
| *** mtanino has quit IRC | 18:57 | |
| *** mtanino has joined #openstack-infra | 18:57 | |
| *** bcourt has quit IRC | 18:57 | |
| *** bcourt has joined #openstack-infra | 18:57 | |
| *** bryan_att has quit IRC | 18:57 | |
| *** bryan_att has joined #openstack-infra | 18:57 | |
| *** rfolco has quit IRC | 18:57 | |
| *** boris-42 has quit IRC | 18:57 | |
| *** boris-42 has joined #openstack-infra | 18:57 | |
| *** ajmiller has quit IRC | 18:57 | |
| *** ajmiller has joined #openstack-infra | 18:57 | |
| *** andymaier has quit IRC | 18:58 | |
| *** andymaier has joined #openstack-infra | 18:58 | |
| *** piet has quit IRC | 18:58 | |
| *** piet has joined #openstack-infra | 18:58 | |
| *** doug-fish has quit IRC | 18:58 | |
| pabelanger | fungi: Regarding the 'upstream development' track for Austin summit. Do we want to have an -infra collaboration on picking a talk, or do some sort of free for all? | 18:58 |
| *** weshay has quit IRC | 18:58 | |
| *** weshay has joined #openstack-infra | 18:58 | |
| anteaya | fungi: based on this comment in backscroll I'm going with yes: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-20.log.html#t2016-01-20T17:31:16 | 18:58 |
| *** Sukhdev has quit IRC | 18:58 | |
| *** maximov has quit IRC | 18:58 | |
| *** maximov has joined #openstack-infra | 18:58 | |
| *** othermordred has quit IRC | 18:58 | |
| *** othermordred has joined #openstack-infra | 18:58 | |
| *** dtardivel has quit IRC | 18:58 | |
| *** dtardivel has joined #openstack-infra | 18:58 | |
| *** rajinir has quit IRC | 18:58 | |
| *** rajinir has joined #openstack-infra | 18:58 | |
| anteaya | fungi: I just happened to be at that place in backscroll | 18:58 |
| *** migi_ has quit IRC | 18:59 | |
| *** migi_ has joined #openstack-infra | 18:59 | |
| *** 77CAALV3Z has quit IRC | 18:59 | |
| *** wfoster has quit IRC | 18:59 | |
| *** annegentle has quit IRC | 18:59 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: [WIP] trying to add one block node https://review.openstack.org/270423 | 18:59 |
| *** edwarnicke has quit IRC | 18:59 | |
| *** edwarnicke has joined #openstack-infra | 18:59 | |
| *** ajo_ has quit IRC | 18:59 | |
| *** ajo_ has joined #openstack-infra | 18:59 | |
| *** mriedem has quit IRC | 18:59 | |
| *** mriedem has joined #openstack-infra | 18:59 | |
| dstufft | lifeless: oh | 18:59 |
| dstufft | by the way | 18:59 |
| *** baoli_ has quit IRC | 18:59 | |
| *** baoli_ has joined #openstack-infra | 18:59 | |
| *** ayoung has quit IRC | 18:59 | |
| *** ayoung has joined #openstack-infra | 18:59 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Minimal translation setup for murano-dashboard https://review.openstack.org/270424 | 18:59 |
| dstufft | y'all probably want to add --no-download to your virtualenv invocations, or for compatibility with < 14.0, --never-download | 18:59 |
| *** mriedem has quit IRC | 19:00 | |
| *** knikolla has quit IRC | 19:00 | |
| *** ybathia has joined #openstack-infra | 19:01 | |
| *** bpokorny_ has joined #openstack-infra | 19:01 | |
| fungi | dstufft: does virtualenv >=14 look for newer pip/setuptools and pull them in if available to replace the vendored versions? | 19:01 |
| *** bpokorny_ has quit IRC | 19:01 | |
| lifeless | dstufft: oh? | 19:01 |
| fungi | that might get tricky to mitigate since most of our virtualenv invocation is happening indirectly via tox | 19:02 |
| dstufft | fungi: it doesn't "replace" the vendored versions, but it'll pull the latest versions in via pip during install | 19:02 |
| dstufft | or set PIP_NO_INDEX=1 I suppose | 19:02 |
| *** bpokorny_ has joined #openstack-infra | 19:02 | |
| dstufft | I think it'll work | 19:02 |
| *** andymaier has quit IRC | 19:02 | |
| fungi | so we may want to be able to have tox tell virtualenv not to grab newer pip/setuptools but still pip install other things over the network | 19:03 |
| *** notnownikki has joined #openstack-infra | 19:03 | |
| dstufft | (virtualenv's pip isntall works by taking the vendored copy of pip, and using it to install pip/setuptools/wheel, previously it used --no-index when it did that so it would only isntall the bundled .whl's, now it only adds --no-index if you use --(no|never)-download | 19:03 |
| *** woodster_ has joined #openstack-infra | 19:03 | |
| *** cfarquhar_ has quit IRC | 19:03 | |
| SergeyLukjanov | fungi, hi, re project renames, do you have few minutes to discuss? | 19:04 |
| anteaya | phschwartz: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-20.log.html#t2016-01-20T02:35:51 | 19:04 |
| anteaya | that may be what you re looking for | 19:04 |
| fungi | SergeyLukjanov: yep, we ran out of time during yesterday's meeting so i didn't get to bring up scheduling | 19:04 |
| fungi | infra-root: ^ SergeyLukjanov is interested in volunteering to do the current batch of pending renames this weekend | 19:04 |
| *** bpokorny has quit IRC | 19:05 | |
| dstufft | lifeless: fungi obviously it doesn't matter much right now, since if you're using pip 14 you'll be using pip 8 anyways, but when pip 9 is released! | 19:05 |
| dstufft | s/pip 14/virutalenv 14/ | 19:05 |
| nibalizer | this is abad weekend for me because of scale | 19:05 |
| fungi | dstufft: right, was just thinking ahead to how we'd skin that inside tox | 19:05 |
| nibalizer | so my availablility is poo | 19:05 |
| *** mriedem has joined #openstack-infra | 19:05 | |
| fungi | i'm travelling this weekend for a foundation thing, as is clarkb | 19:05 |
| fungi | and i gather pleia2 is at scale as well | 19:05 |
| dstufft | fungi: presumably tox will expose the option somehow | 19:06 |
| fungi | dstufft: this is my hope, because using PIP_NO_INDEX to make that happen is presumably going to have side effects for its other invocations of pip after it builds the virtualenv | 19:06 |
| *** cfarquhar has joined #openstack-infra | 19:07 | |
| *** cfarquhar has quit IRC | 19:07 | |
| *** cfarquhar has joined #openstack-infra | 19:07 | |
| fungi | unless we maybe expressly unset that with an install command override | 19:07 |
| fungi | anyway, could get murky in the config | 19:07 |
| sdague | AJaeger: ok, done. I think there are a few things that can be cleaned up | 19:07 |
| lifeless | yeah, setting PIP_NO_INDEX is hard | 19:07 |
| lifeless | we need a tox feature to control this | 19:08 |
| *** pvaneck has joined #openstack-infra | 19:08 | |
| AJaeger | thanks, sdague | 19:08 |
| lifeless | fungi: virtualenv creation isn't currently configurable in tox; IIRC there are plans to let it be so | 19:08 |
| fungi | so hopefully before pip9/venv15 tox will do something about it | 19:08 |
| sdague | hmmm internap is really slow huh? | 19:08 |
| fungi | or else one of us will have a fire lit under them to submit a patch for it | 19:09 |
| AJaeger | dims, so, looking at the failing patches for the arg-parse: Mmost of them do get argparse in via oslo.config, so a new release would help with them | 19:09 |
| *** doug-fish has joined #openstack-infra | 19:09 | |
| fungi | AJaeger: but would they still be hitting it somewhere else later even if oslo.config were no longer responsible? | 19:09 |
| fungi | this can still happen via transitive dependencies over which we have basically no control | 19:10 |
| *** eharney has quit IRC | 19:10 | |
| *** doug-fis_ has joined #openstack-infra | 19:10 | |
| *** esikachev has joined #openstack-infra | 19:10 | |
| lifeless | dstufft: so the idea is that virtualenv no longer needs to release coordinated with pip ? | 19:10 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/system-config: Change logstash multiline stream identity https://review.openstack.org/190345 | 19:10 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/system-config: Logstash 2.0 compat ES output rule https://review.openstack.org/247714 | 19:10 |
| *** unicell has left #openstack-infra | 19:11 | |
| AJaeger | fungi: yes, it could. But I'm not seeing that ;( | 19:11 |
| AJaeger | fungi, looking at storyboard as an example: http://logs.openstack.org/06/270406/1/check/gate-storyboard-docs/1aa1626/console.html | 19:11 |
| *** ashishb has quit IRC | 19:11 | |
| AJaeger | "Collecting argparse (from oslo.config>=1.11.0->storyboard==0.0.1.dev661)" | 19:11 |
| dstufft | lifeless: yes, I mean we'll likely still do it for the offline case, but also wheels and setuptools too | 19:11 |
| sdague | I just had a patch die on neutron job timeouts, pip install time is up to 16 minutes ... - http://logs.openstack.org/15/270215/1/check/gate-tempest-dsvm-neutron-full/19bd306//logs/devstacklog.txt.gz | 19:11 |
| dstufft | virtualenv <14 is broken on 3.5 by default because the wheel version is too old for it | 19:11 |
| fungi | krotscheck: what were the mirror patches you wanted me to look over yesterday? | 19:12 |
| AJaeger | fungi: so, oslo.config is just a minor help I guess -but the one that shows up in most failures | 19:12 |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins https://review.openstack.org/269567 | 19:12 |
| *** knikolla has joined #openstack-infra | 19:13 | |
| *** doug-fish has quit IRC | 19:14 | |
| jeblair | dstufft: hey, we're going to try to put our new pypi mirrors into production this week; when you have a second, i updated my bandersnatch PR with what we need to do that: https://bitbucket.org/pypa/bandersnatch/pull-requests/16/add-option-to-dir-hash-index-files/diff | 19:14 |
| AJaeger | fungi: flame gets argparse via python-keystoneclient, fuel-octane is the only odd one: octane and stevedeore | 19:14 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-jenkins: Use more generic variable name for ruby packages https://review.openstack.org/270433 | 19:16 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-jenkins: Inherit from param class instead of including it https://review.openstack.org/270434 | 19:16 |
| dstufft | jeblair: probably not going to look at it until after pip 8 settles down, but jsut as a reminder, I don't have commit on that repo (other than implicitly from being a pypa org admin) | 19:16 |
| *** Sukhdev has joined #openstack-infra | 19:16 | |
| jeblair | dstufft: if you can poke the right people, that would be appreciated :) | 19:16 |
| fungi | krotscheck: nevermind, dug it out of scrollback. https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open+project:openstack-infra/system-config | 19:17 |
| *** bnemec has quit IRC | 19:17 | |
| *** EricGonczer_ has quit IRC | 19:17 | |
| *** dtardivel has quit IRC | 19:17 | |
| mtreinish | fungi, lifeless: as far as I know all my pins landed and unblocked most of the jobs | 19:18 |
| *** ZZelle_ has joined #openstack-infra | 19:19 | |
| fungi | mtreinish: excellent, so devstack/tempest/grenade jobs are a-okay now? | 19:19 |
| fungi | sounds like we're just down to solving for tox at the moment | 19:20 |
| *** knikolla has quit IRC | 19:20 | |
| *** knikolla has joined #openstack-infra | 19:20 | |
| sdague | does anyone have more generic internap run timings? | 19:20 |
| *** ociuhandu has quit IRC | 19:20 | |
| clarkb | sdague: grafana has some | 19:20 |
| mtreinish | fungi: I think so: http://status.openstack.org/elastic-recheck/gate.html#1536229 | 19:20 |
| clarkb | sdague: http://grafana.openstack.org/dashboard/db/nodepool-internap | 19:20 |
| sdague | clarkb: url? | 19:20 |
| clarkb | they are at the bottom of the page | 19:21 |
| sdague | clarkb: oh, but not tempest run times? | 19:21 |
| anteaya | SergeyLukjanov: I am not available this weekend | 19:21 |
| *** achanda has quit IRC | 19:21 | |
| sdague | ah, at the bottom, I see | 19:21 |
| sdague | right, so internap looks slower again than everything else | 19:21 |
| *** yfried has joined #openstack-infra | 19:21 | |
| mtreinish | sdague: I can get that with subunit2sql, it's on my todo to add those graphs to the approapriate pages | 19:21 |
| SergeyLukjanov | seems like it'll be better to try discuss it on the next week meeting | 19:21 |
| SergeyLukjanov | fungi ^^ | 19:21 |
| sdague | and is pushing fail rates up due to our timeouts | 19:22 |
| mtreinish | woah, yeah 1.8 hrs is kinda long | 19:22 |
| sdague | right, some neutron jobs are timing out a 2 hrs now | 19:22 |
| clarkb | fwiw rax is doing that too as is ovh | 19:22 |
| clarkb | I don't think it is unique to internap | 19:23 |
| dims | AJaeger : can cut a release when your argparse review lands | 19:23 |
| *** exploreshaifali has joined #openstack-infra | 19:24 | |
| fungi | SergeyLukjanov: yep, i need to clean up the agenda from this week and i'll add it as a discussion topic. i won't be around this tuesday but pleia2 has volunteered to chair the meeting in my absence | 19:24 |
| openstackgerrit | Merged openstack-infra/release-tools: Remove argparse from requirements https://review.openstack.org/270404 | 19:25 |
| *** EricGonczer_ has joined #openstack-infra | 19:25 | |
| *** flepied1 has joined #openstack-infra | 19:25 | |
| *** bnemec has joined #openstack-infra | 19:25 | |
| *** flepied has quit IRC | 19:26 | |
| *** eharney has joined #openstack-infra | 19:26 | |
| sdague | clarkb: ok, gotcha | 19:26 |
| *** hasharAway is now known as hashar | 19:26 | |
| sdague | just wanted to double check | 19:26 |
| *** rossella_s has quit IRC | 19:26 | |
| *** eharney is now known as Guest80152 | 19:26 | |
| clarkb | so probably need it identify where the time is being spent and find the regression | 19:27 |
| sdague | clarkb: the theory is keystone middleware, there is a pin posted | 19:27 |
| *** esikachev has quit IRC | 19:27 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet/spec_helper: run the script from project dir https://review.openstack.org/269914 | 19:28 |
| *** mgkwill is now known as mgkwill_training | 19:28 | |
| lifeless | sdague: psycopg2, numpy, libvirt, lxml, pycrypto and cryptography are a chunk of that; | 19:28 |
| lifeless | sdague: I can't see any single omg-bad call to pip | 19:28 |
| sdague | lifeless: no, there is no omg-bad call to pip | 19:28 |
| dstufft | spoiler: all calls to pip are bad | 19:29 |
| sdague | it's just the continuous upward creep + new nodes that are slower than rax | 19:29 |
| lifeless | sdague: whats the normal time you'd expect? | 19:29 |
| sdague | and the lack of a wheel mirror | 19:29 |
| sdague | I'm used to 10 minutes, though that's bad. Before upper-constraints it was more like 2 minutes. | 19:29 |
| sdague | because we used packages for the expensive ones | 19:29 |
| sdague | like numpy | 19:30 |
| *** tonytan4ever has quit IRC | 19:30 | |
| sdague | any update on the wheel mirror? | 19:30 |
| mordred | it's in active work | 19:30 |
| sdague | because we're going to just need to start mass bumping up timeouts all over the place | 19:30 |
| mordred | krotscheck has done work on it and jeblair is connecting with him to get it landed and stuff | 19:31 |
| fungi | i'm reviewing the patches krotscheck asked for oversight on in the next hour | 19:31 |
| fungi | wrt the wheel mirror implementation | 19:31 |
| sdague | fungi / mordred - great. | 19:31 |
| mordred | so - yeah - mirror stuff has gotten a bump up in priority | 19:31 |
| sdague | because it's definitely a big component in our current fail rates | 19:31 |
| fungi | and jeblair is trying to get one patch into bandersnatch to support our filename hashing needs | 19:31 |
| sdague | none of our config accounted for the idea that pip install overhead could be more than a couple of minutes | 19:32 |
| openstackgerrit | Matthew Kassawara proposed openstack-infra/project-config: Add logic for networking-ovn jobs https://review.openstack.org/270444 | 19:33 |
| *** Guest80152 is now known as eharney | 19:33 | |
| *** eharney has quit IRC | 19:33 | |
| *** eharney has joined #openstack-infra | 19:33 | |
| *** sridhar_ram has quit IRC | 19:34 | |
| *** piet has quit IRC | 19:34 | |
| *** bpokorny_ has quit IRC | 19:34 | |
| *** FallenPegasus has quit IRC | 19:34 | |
| *** bpokorny has joined #openstack-infra | 19:35 | |
| *** sridhar_ram has joined #openstack-infra | 19:36 | |
| sdague | also, if anyone else wants to review this change - https://review.openstack.org/#/c/267133/ - we could land some nova code that will drop 5 nodes per nova check change, which will put more resource back into the queue | 19:36 |
| sdague | it's the constraints related one | 19:36 |
| * mordred has already reviewed it | 19:37 | |
| *** ddieterly has joined #openstack-infra | 19:38 | |
| *** smcginni1 is now known as smcginnis | 19:38 | |
| sdague | mordred: yes, and thank you for that :) | 19:38 |
| * mordred _occasionally_ does thing | 19:38 | |
| AJaeger | speaking of less jobs per change, here's one for keystoneclient: https://review.openstack.org/#/c/269270/ | 19:39 |
| AJaeger | mordred: not only occassionally ;) | 19:39 |
| clarkb | sdague: do you know if the plan is to switch neutron back to the py27 regular job with constraints enabled there? I think it is based on TC meeting yesterday | 19:42 |
| mtreinish | fungi: or any infra-root I guess, when you get a chance can you check the logs for the subunit2sql pruning cron job? I think it was supposed to run already, but the db doesn't look like it's shrunk at all | 19:43 |
| clarkb | sdague: the two tarball jobs in that file are not updated to add constraints, but there are constraints specific tarball jobs, were you wanting to condensce those too? | 19:43 |
| *** mtanino has quit IRC | 19:45 | |
| openstackgerrit | Merged openstack-infra/shade: Remove a done todo list item https://review.openstack.org/269120 | 19:46 |
| sdague | clarkb: I do not know for neutron, my guess is yes, I was currently trying to get the nova situation sorted. I might have missed some jobs, if so point them out and I'll update. | 19:46 |
| *** trown is now known as trown|mtg | 19:51 | |
| clarkb | sdague: commented, there were 3 that I found | 19:51 |
| sdague | clarkb: ok, will fix right now | 19:52 |
| clarkb | but this way we can remove all the -constraints jobs in favor of projects opting in project side with the normal jobs | 19:52 |
| cbader_ | clarkb, was there a fix for the dib-image-build hang found? | 19:52 |
| clarkb | cbader_: I don't know what issue that is | 19:52 |
| sdague | clarkb: the pylint on didn't have the zuul cloner in it | 19:52 |
| *** vilobhmm111 has joined #openstack-infra | 19:53 | |
| clarkb | sdague: oh but there is a constraints pylint job? let me double check | 19:53 |
| sdague | there is, it was wrong. I'll just fix it anyway | 19:53 |
| clarkb | sdague: ha so I think that is just wrong ok | 19:53 |
| *** piet has joined #openstack-infra | 19:53 | |
| cbader_ | clarkb, on first creation of the dib-image-build it states receiving packets but just hangs and can't cntrl-c to break out of it have to kill -9 to stop it. | 19:53 |
| cbader_ | clarkb, this is on a clean machine first run with nodepool started per common-ci instructions. | 19:54 |
| clarkb | cbader_: we recently switched to using gearman to trigger image builds so we can run builders on external systems. I bet the ^C not working is related to the gear library | 19:55 |
| *** achanda has joined #openstack-infra | 19:55 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: always make upper-constraints available to python jobs https://review.openstack.org/267133 | 19:55 |
| clarkb | cbader_: but while it sits there it should be building an image, check your logs | 19:55 |
| sdague | clarkb / mordred ok, I believe it's fixed now | 19:55 |
| fungi | mtreinish: i don't see any output in the subunit2sql log from that cronjob, though the file was last modified at the date and time it was scheduled to run (suggesting it did fire and produced no stdout/stderr) | 19:56 |
| clarkb | sdague: +2 | 19:56 |
| cbader_ | clarkb, so looks like I can cntrl-c now. dib-image-build says building but nothing hitting cache | 19:56 |
| clarkb | cbader_: look at your image build logs | 19:56 |
| cbader_ | clarkb, ok looking thanks | 19:57 |
| mtreinish | fungi: hmm that's curious | 19:57 |
| clarkb | becuase the build itself has been pushed out of the process you are running to the gearman worker ou don't get logs on the console | 19:57 |
| fungi | mtreinish: also no cronspam sent to the sysadmins from it, so didn't exit nonzero i gues | 19:57 |
| fungi | s | 19:57 |
| clarkb | the logs should however end up in your image logs | 19:57 |
| mtreinish | it definitely looks like we've still got a years worth of data in there: http://paste.openstack.org/show/484448/ | 19:57 |
| clarkb | cbader_: if you still have issues make sure you ahve latest nodepool as we have pushed a few fixes in the last da | 19:57 |
| *** rkukura has quit IRC | 19:58 | |
| mtreinish | fungi: the mystery deepens | 19:58 |
| cbader_ | clarkb, I pulled latest common-ci last night. I don;t see anything in the /var/log/nodepool/image location is that the right place? | 19:58 |
| *** flwang1 has joined #openstack-infra | 19:58 | |
| clarkb | cbader_: yes that should be the location | 19:59 |
| cbader_ | clarkb, ok then nothing is being created. | 19:59 |
| clarkb | cbader_: check the version of nodepool you are running, update and restart if not latest | 20:00 |
| cbader_ | clarkb, I liked it better when I saw output of the build could at least see it working | 20:00 |
| clarkb | cbader_: unfortunately that doesn't scale well | 20:00 |
| clarkb | (and you can see the output in the logs) | 20:00 |
| clarkb | I have a hunch you are missing a fix for logging in your current install though | 20:01 |
| cbader_ | clarkb, I understand the reason but just wish could turn it on when needed. | 20:01 |
| cbader_ | clarkb, fix for logging in your current install? | 20:01 |
| *** tjones1 has quit IRC | 20:02 | |
| clarkb | yes master nodepool should have the fix for logging (I confirmed it was working at about 5pm PST yesterday) | 20:02 |
| mtreinish | fungi: I guess I should have made it output something when it works :p | 20:02 |
| fungi | mtreinish: with a timestamp ;) | 20:02 |
| *** FallenPegasus has joined #openstack-infra | 20:02 | |
| cbader_ | clarkb, I have nodepool 0.2.1 | 20:02 |
| clarkb | cbader_: look at pbr freeze to get the sha1 | 20:02 |
| clarkb | you can then compare that against current master of nodepool | 20:03 |
| cbader_ | clarkb, nodepool==0.2.1.dev31 | 20:03 |
| clarkb | cbader_: there should be a sha1 to the right of that | 20:04 |
| *** dizquierdo has quit IRC | 20:04 | |
| lifeless | clarkb: I am pretty sure we don't export that in the version anymore | 20:04 |
| lifeless | clarkb: since the PEP-440 thing landed in a pip release | 20:04 |
| cbader_ | clarkb, nope just that on the line from pip freeze | grep nodepool | 20:04 |
| clarkb | lifeless: yes that is why you have to use pbr freeze | 20:04 |
| clarkb | cbader_: correct you have to use pbr freeze | 20:05 |
| *** weshay has quit IRC | 20:06 | |
| cbader_ | clarkb, sorry haven't done that before so here is what I got, nodepool==0.2.1.dev31 # git sha de4a318 | 20:06 |
| *** alkhodos_ has joined #openstack-infra | 20:06 | |
| clarkb | ok that is the latest commit on master | 20:07 |
| sdague | mordred: you want to hit me up with final +2 on that - https://review.openstack.org/#/c/267133/ ? | 20:07 |
| clarkb | cbader_: how long has your nodepoold process been running? if it is > 16 hours or so you may need to restart it | 20:07 |
| *** knikolla has quit IRC | 20:07 | |
| alkhodos_ | Hi everyone. I have a patch that is waiting Jenkins verify, but it keeps restarting with such things in logs: 2016-01-20 19:48:47.466 | Build was aborted 2016-01-20 19:48:47.467 | Aborted by anonymous , any idea why? | 20:08 |
| cbader_ | clarkb, this is a new installation of common-ci I haven't started the nodepool service yet per instructions to manually install first dib-image. | 20:08 |
| clarkb | alkhodos_: that happens when a change ahead of your change fails and zuul has to reassemble the git trees to remove the failing change | 20:08 |
| openstackgerrit | Merged openstack-infra/system-config: Add more logstash timestamp formats https://review.openstack.org/251518 | 20:08 |
| clarkb | alkhodos_: when it does that it cancels job results for the discarded tree and starts over using the new tree | 20:08 |
| clarkb | alkhodos_: covered in detail at http://docs.openstack.org/infra/zuul/gating.html | 20:09 |
| alkhodos_ | clarkb: ah ok, thanks | 20:09 |
| *** dteselkin_ has quit IRC | 20:09 | |
| alkhodos_ | clarkb: so that's normal behavior, right? | 20:09 |
| clarkb | cbader_: ok running the nodepoold or a builder daemon is required | 20:09 |
| alkhodos_ | clarkb: pretty much all I wanted to know :) | 20:09 |
| clarkb | alkhodos_: yes it is normal | 20:10 |
| alkhodos_ | clarkb: thanks! | 20:10 |
| zxiiro | Does anyone know why Zuul wouldn't kick off a job in Jenkins? I got a pretty simple job configured with zuul and my Jenkins Gearman plugin appears to have connected to zuul based on log output and I can see that Zuul is scheduling the job when I run "zuul show running-jobs" but the job never gets assigned a worker and Jenkins never fires off the job. I'm | 20:10 |
| zxiiro | using this is my simple layout.yaml http://pastebin.com/kMwcwGpj | 20:10 |
| cbader_ | clarkb, so I can run the nodepoold or just start the nodepool service correct? | 20:11 |
| clarkb | cbader_: yup, I would start the service | 20:11 |
| cbader_ | clarkb, ok will try that do I have to restart or kill the current dib-image-build and if so how would I do that. | 20:12 |
| clarkb | cbader_: no I think once the nodepool service starts it should pick up the job to build the image meaning you don't need to kill the current build | 20:12 |
| cbader_ | clarkb, great thanks. | 20:12 |
| clarkb | cbader_: after ou start it look in the images log dir again and see if logs show up | 20:13 |
| cbader_ | clarkb, ok thanks will do. | 20:13 |
| cbader_ | clarkb, I have a log forming, thank you for the help. | 20:14 |
| *** dteselkin has joined #openstack-infra | 20:14 | |
| *** jzerebecki has joined #openstack-infra | 20:15 | |
| fungi | jeblair: were you also wanting to review the three topic:wheel_mirror changes pending for system-config or should i go ahead and approve them if they look okay (mordred's already reviewed the latest patchsets on them and yolanda has also positively reviewed the last change in the series) | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| cbader_ | clarkb, but it is a zero bite but will give it a few minutes. | 20:15 |
| jeblair | fungi: feel free to approve | 20:15 |
| fungi | jeblair: i guess it would help for me to read your specs change first and make sure these aren't contrary to that plan | 20:16 |
| *** salv-orlando has quit IRC | 20:18 | |
| *** annegentle has joined #openstack-infra | 20:18 | |
| *** matrohon has joined #openstack-infra | 20:18 | |
| *** wfoster has joined #openstack-infra | 20:18 | |
| *** wfoster has quit IRC | 20:18 | |
| *** wfoster has joined #openstack-infra | 20:18 | |
| fungi | oh, wait, that's for logs not really for mirrors | 20:18 |
| *** gongysh has joined #openstack-infra | 20:18 | |
| *** weshay has joined #openstack-infra | 20:18 | |
| *** davidlenwell has quit IRC | 20:18 | |
| mtreinish | mordred: if you get a sec I think you'll like: https://review.openstack.org/234447 | 20:18 |
| *** changbl has quit IRC | 20:19 | |
| *** davidlenwell has joined #openstack-infra | 20:20 | |
| mtreinish | it'll let us track devstack in the health dashboard | 20:20 |
| openstackgerrit | Merged openstack-infra/infra-specs: Add some assignees to zuulv3 https://review.openstack.org/266538 | 20:20 |
| openstackgerrit | Merged openstack-infra/nodepool: Bump APScheduler to >=3.0 https://review.openstack.org/262017 | 20:20 |
| * krotscheck heard his name? | 20:21 | |
| *** simonmcc has quit IRC | 20:21 | |
| *** matrohon has quit IRC | 20:21 | |
| openstackgerrit | Merged openstack-infra/infra-specs: Adjust assignees for translations check site https://review.openstack.org/267785 | 20:21 |
| krotscheck | Ah, gotcha. | 20:21 |
| openstackgerrit | Merged openstack-infra/project-config: Use linters job for keystoneclient starting with mitaka https://review.openstack.org/269270 | 20:21 |
| *** simonmcc has joined #openstack-infra | 20:21 | |
| krotscheck | So, wheel mirros, afs mirros, and then we'll start adding other mirros into the mix. | 20:21 |
| *** FallenPegasus has quit IRC | 20:22 | |
| *** matrohon has joined #openstack-infra | 20:22 | |
| openstackgerrit | Merged openstack-infra/infra-specs: Use git.openstack.org everywhere https://review.openstack.org/262311 | 20:23 |
| stevemar | notmorgan: sounds like we should be blocking 4.1.0 of ksm | 20:23 |
| notmorgan | stevemar: ? | 20:23 |
| *** gongysh has quit IRC | 20:23 | |
| stevemar | notmorgan: we can add memcache to devstack independent of blocking 4.1.0 ksm | 20:24 |
| notmorgan | stevemar: we can. this really is a minimal devstack patch. let me get on it | 20:24 |
| clarkb | are we expecting this to use a lot of extra memory? | 20:25 |
| jeblair | fungi, krotscheck: yeah, my plan (now that i've updated bandersnatch) is to write a change to start populating afs with it. i don't think it contradicts anything krotscheck has done, so we can just keep landing those and spin up the new servers and hopefully we can use afs on them instead of the local bandersnatch. | 20:25 |
| mtreinish | clarkb: heh, concerns about the swap thing? | 20:25 |
| notmorgan | clarkb: no | 20:25 |
| notmorgan | clarkb: this should actually reduce memory usage | 20:26 |
| notmorgan | clarkb: since all the services will share the cache for validation rather than each worker process having it's own version | 20:26 |
| *** rkukura has joined #openstack-infra | 20:26 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 20:26 |
| *** sridhar_ram has quit IRC | 20:26 | |
| *** mtanino has joined #openstack-infra | 20:27 | |
| notmorgan | clarkb: or at worst, you'll see a minor uptick since memcache relies on LRU rather than an expensive scrub of the in-process cache on each lookup | 20:27 |
| notmorgan | clarkb: but it should be at worst near wash compared to today, at best a reduction of used memory | 20:27 |
| clarkb | ok | 20:27 |
| *** rguillebert has quit IRC | 20:27 | |
| krotscheck | jeblair: Is your plan to reate sortof a "Mirror master" machine whose job it is to populate afs? | 20:27 |
| *** apoorvad has quit IRC | 20:27 | |
| krotscheck | We can run all the various sync jobs on it. | 20:27 |
| clarkb | mtreinish: ya any time more mmemory comes up I get worried since we apparently rely on swap | 20:27 |
| *** mtanino has quit IRC | 20:28 | |
| *** matrohon has quit IRC | 20:28 | |
| *** annegentle has quit IRC | 20:28 | |
| *** wfoster has quit IRC | 20:28 | |
| mtreinish | that still surprises me, because I definitely thought we fixed all of that > 1 yr ago | 20:28 |
| mtreinish | I actually wrote blog posts about debugging that | 20:29 |
| *** ccrouch has quit IRC | 20:29 | |
| mordred | krotscheck: yes, I believe that is the goal | 20:30 |
| mordred | krotscheck: s/goal/plan/ | 20:30 |
| clarkb | mtreinish: well I think it is newer jobs like magnum and kolla and openstack ansible | 20:30 |
| clarkb | mtreinish: the "core/integrated/smalltent" is fine | 20:30 |
| mtreinish | ah ok | 20:30 |
| mordred | clarkb: zomg. please let smalltent be a real new term | 20:30 |
| jeblair | krotscheck, mordred: yes -- so the equiv npm process can run there too | 20:30 |
| mtreinish | heh, that is a good term | 20:30 |
| mordred | and maybe even a reprepro process in the future! :) | 20:31 |
| sdague | mordred: puptent | 20:31 |
| mordred | sdague: bivvy sack? | 20:31 |
| *** ccrouch has joined #openstack-infra | 20:31 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 20:31 |
| clarkb | mordred: you put the small tent in the big tent | 20:31 |
| clarkb | mordred: that way when it rains you stay dry | 20:31 |
| *** bpokorny has quit IRC | 20:33 | |
| *** jsavak has quit IRC | 20:33 | |
| *** Thelo has quit IRC | 20:33 | |
| *** otter768 has joined #openstack-infra | 20:35 | |
| *** jsavak has joined #openstack-infra | 20:35 | |
| *** amitgandhinz has joined #openstack-infra | 20:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: openstack-doc-tools,openstackdocstheme: Use linters https://review.openstack.org/270469 | 20:36 |
| *** dprince has quit IRC | 20:36 | |
| clarkb | fungi: so sounds like pip + virtualenv + tox pinning is actually quite complicated and you are inclined to wait for pip to release 8.0.1 with fixes for at least 2/3 of the issues? | 20:38 |
| anteaya | I'm in with the smalltent | 20:38 |
| fungi | clarkb: that's where i'm leaning at this point. it sounds like also the various argparse cleanups landing all over the place may get a few more tox jobs working with pip 8.0.0 in the meantime | 20:39 |
| *** otter768 has quit IRC | 20:39 | |
| *** piet has quit IRC | 20:39 | |
| asselin_ | jpr1, didn't follow your question. The "Do Not Edit - Generated & Managed by Puppet" is just a comment header. You should populate the real data underneath | 20:39 |
| ianw | clarkb: when you've got some cycles, i'm going to need some help with poking at logs/nodepool for fedora23. build looks OK now. i can build locally, upload to rax and ssh in. but still coming up as NOT_REGISTERED for zuul. can you check what our nodepool thinks about the whole thing | 20:39 |
| clarkb | ianw can do | 20:40 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 20:40 |
| jpr1 | ah ok | 20:40 |
| dstufft | I can probably commit to an 8.0.1 with at least 2/3 of the things fixed in the next few hours. the other thing I'll try to include something for, or at least a decision on who will do what | 20:40 |
| ianw | dstufft: the 2/3's being the virtualenv stuff, and the other bit being removing the egg file? | 20:41 |
| jpr1 | so that "puppet configed" warning is for the reader of the eventually changed system's config file then? | 20:41 |
| *** ociuhandu has joined #openstack-infra | 20:42 | |
| dstufft | ianw: 2/3 == argparse (and wsgiref, but that' less common) and virtual environments with system site packages, 1/3 being -> if we legimately need to *actually* uninstall a distutils installed project (can't shadow or something) do we error? continue to warn? something else? | 20:42 |
| *** Thelo has joined #openstack-infra | 20:42 | |
| *** mtanino has joined #openstack-infra | 20:42 | |
| *** matrohon has joined #openstack-infra | 20:42 | |
| *** annegentle has joined #openstack-infra | 20:42 | |
| *** wfoster has joined #openstack-infra | 20:42 | |
| *** wfoster has quit IRC | 20:42 | |
| *** wfoster has joined #openstack-infra | 20:42 | |
| *** mtanino has quit IRC | 20:42 | |
| *** mtanino has joined #openstack-infra | 20:42 | |
| *** bpokorny has joined #openstack-infra | 20:43 | |
| jpr1 | and I can use the output of os-client-config to populate (as in `python -m os_client_config.config`)? | 20:43 |
| *** boris-42 has quit IRC | 20:43 | |
| jeblair | mordred: does this line do anything? http://git.openstack.org/cgit/openstack-infra/puppet-bandersnatch/tree/templates/bandersnatch.conf.erb#n34 | 20:44 |
| sdague | dhellmann / lifeless / ttx - could we get a +A on https://review.openstack.org/#/c/270417/ so that we could then promote it | 20:44 |
| asselin_ | jpr1, it's a notice for someone reading the file in the filesystem, so that if they update it there, then they've been warnined if their updates are overwritten | 20:44 |
| *** weshay has quit IRC | 20:44 | |
| sdague | and neutron failure rates will go back to baseline | 20:44 |
| asselin_ | jrp1, * overwritten by puppet | 20:45 |
| notmorgan | sdague, stevemar, clarkb: https://review.openstack.org/#/c/270474/ something like that. oops missed a file. | 20:45 |
| ianw | dstufft: got it, thanks | 20:45 |
| lifeless | sdague: 2A | 20:45 |
| Swanson | More CI fun. If the Zuul status page is showing up with a status.json: Proxy Error it is failing to connect to Zuul itself and not something else? | 20:45 |
| asselin_ | jpr1, but if you think it's confusing, submit a patch set that you think clarifies it. | 20:46 |
| *** bknudson_ has quit IRC | 20:46 | |
| notmorgan | ok something like that. | 20:46 |
| *** sridhar_ram has joined #openstack-infra | 20:46 | |
| *** bknudson has joined #openstack-infra | 20:47 | |
| krotscheck | Merge failure. Doh | 20:47 |
| krotscheck | Sec | 20:47 |
| krotscheck | Oh. | 20:47 |
| krotscheck | Not a merge cailure at all | 20:47 |
| krotscheck | jeblair: Before these things merge, can you take a close look at https://review.openstack.org/#/c/253521/10/modules/openstack_project/templates/pypi.vhost.erb to make sure that it's the folder structure you're expecting? | 20:47 |
| stevemar | notmorgan: noice | 20:47 |
| krotscheck | jeblair: Specifically at the rewrite ules | 20:47 |
| jpr1 | will do. just trying to get a little clarity on what the format is of the data that goes there. | 20:48 |
| jpr1 | read over os-client-config which appears to be how to generate it but not sure which output format is desired for this part of the config file | 20:49 |
| notmorgan | stevemar: it might need "adjusting" and i haven't run it locally just tossing it up to quickly iterate / get feedback while i conintue to work on it | 20:49 |
| mtreinish | notmorgan: reviewed | 20:49 |
| fungi | mordred: greghaynes: for 165240 i assume we need an ansible host group and associated group hiera file? | 20:50 |
| notmorgan | mtreinish: so, i was lifting previous logic [will cc this in the review] for the python-memcache vs python-memcached | 20:50 |
| fungi | mordred: greghaynes: for keys associated with node /.*wheel-mirror-.*\.openstack\.org/ | 20:50 |
| notmorgan | mtreinish: i can do a pip install instead if that is preferred. | 20:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: openstack-doc-tools,openstackdocstheme: Use linters https://review.openstack.org/270469 | 20:51 |
| asselin_ | jpr1, this link shoudl explian it: https://git.openstack.org/cgit/openstack/os-client-config/tree/README.rst | 20:51 |
| notmorgan | mtreinish: and uhm, if files guarantee install, sure, we can skip the install memcached thing | 20:51 |
| asselin_ | jpr1, https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp#n108 | 20:51 |
| mtreinish | notmorgan: I think pip install is prefered because we can tailor the versions with g-r and constraints | 20:51 |
| notmorgan | mtreinish: so, in short: files means that install is no longer needed ? (you can confirm that for me). and uhm.. the pip thing hmm. | 20:51 |
| mtreinish | using the ubuntu package has been problematic for other things in the past | 20:52 |
| *** piet has joined #openstack-infra | 20:52 | |
| notmorgan | mtreinish: i don't think we have a package that explicitly relies on python-memcache | 20:52 |
| mtreinish | I thought we stopped using ubuntu packages for python things everywhere else in the past | 20:52 |
| jpr1 | yes. that's the one I read. :) | 20:52 |
| notmorgan | and a raw pip install isn't good. | 20:52 |
| notmorgan | because that isn't g-r gated | 20:52 |
| mtreinish | I know that mordred was beating that drum for a while | 20:52 |
| mtreinish | notmorgan: pip_install will respect g-r | 20:52 |
| mtreinish | pip install won't :) | 20:53 |
| notmorgan | mtreinish: oh cool | 20:53 |
| notmorgan | mtreinish: then i'll switch to that. | 20:53 |
| mtreinish | it's all about that underline | 20:53 |
| notmorgan | give me a moment. | 20:53 |
| fungi | mtreinish: notmorgan: at least in liberty and later | 20:53 |
| *** Thelo has quit IRC | 20:53 | |
| *** mtanino has quit IRC | 20:53 | |
| *** matrohon has quit IRC | 20:53 | |
| *** annegentle has quit IRC | 20:53 | |
| *** wfoster has quit IRC | 20:53 | |
| sdague | infra-root - please promote - https://review.openstack.org/#/c/270417/ | 20:54 |
| fungi | mtreinish: notmorgan: for kilo devstack i think it needed to be in a requirements file which was subjected to the version syncing | 20:54 |
| sdague | it has passed all the relevant tests, the test fails that is has are related to unrelated race conditions | 20:54 |
| *** changbl has joined #openstack-infra | 20:54 | |
| mtreinish | fungi: right, it's only >= liberty but this is a change on master :) | 20:54 |
| notmorgan | fungi: this should only matter for mitaka+ | 20:55 |
| notmorgan | fungi: so we're good | 20:55 |
| sdague | notmorgan: if you address mtreinish's comments on the devstack patch, I'm +2 | 20:55 |
| notmorgan | mtreinish: and so files will take care of the server bits? | 20:55 |
| notmorgan | sdague: cool. | 20:56 |
| notmorgan | sdague: almost ready to push an update addressing those | 20:56 |
| notmorgan | mtreinish: so i can just drop the install_package bit | 20:56 |
| *** piet has quit IRC | 20:56 | |
| sdague | notmorgan: for memcached | 20:56 |
| notmorgan | sdague: cool | 20:56 |
| mtreinish | notmorgan: yes | 20:56 |
| mtreinish | I was just looking up the ref for you: https://github.com/openstack-dev/devstack/blob/c461db4a2f75333dceb85c9fb5c90660c0623b11/functions-common#L1159 | 20:56 |
| *** piet has joined #openstack-infra | 20:56 | |
| notmorgan | sdague: then 20 seconds and i'll have another patch. | 20:56 |
| sdague | you should convert the client side one to a pip_install | 20:56 |
| notmorgan | sdague: already done. :) | 20:56 |
| asselin_ | jpr1, this wasn't helpful? https://git.openstack.org/cgit/openstack/os-client-config/tree/README.rst#n86 | 20:57 |
| asselin_ | jpr1, which 'cloud' are you using? | 20:57 |
| notmorgan | sdague, mtreinish: new patch uploaded with fixes | 20:57 |
| fungi | sdague: i've enqueued 270417,1 into the gate and promoted it to the front | 20:57 |
| *** tonytan4ever has joined #openstack-infra | 20:57 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/ansible-puppet: Conditionally manage puppet config files https://review.openstack.org/243399 | 20:58 |
| jpr1 | asselin_: think i'm getting it. so the "multi-line contents of os-client-config" look like the yaml here https://git.openstack.org/cgit/openstack/os-client-config/tree/README.rst#n108 or is it intended to be an output format produced by running the os-client-config module? | 20:58 |
| jpr1 | asselin_: running against our local openstack | 20:58 |
| notmorgan | sdague, mtreinish: i am now doing the final 100% confirmation this is working as expected. | 20:58 |
| sdague | fungi: thanks | 20:58 |
| asselin_ | jpr1, yes, actually I never ran os-client-config module so I don't know what that looks like. | 20:59 |
| notmorgan | i don't see why it wouldn't but it really just is an eyeball check so give me a moment to spin up a VM running keystone and ... uhmm.... cinder yeah | 20:59 |
| mtreinish | notmorgan: I'm +2 on it | 20:59 |
| notmorgan | we can let the gate chew on it for now too so we can be closer to it landing it all good :) | 20:59 |
| *** kgiusti1 has quit IRC | 20:59 | |
| mtreinish | it looks like it would work | 20:59 |
| notmorgan | mtreinish: aye. i figure it will work but... let me be 100% on it while the check queue runs | 20:59 |
| *** weshay has joined #openstack-infra | 21:00 | |
| notmorgan | mtreinish: worst case, i'll hav eanother minor fix before it's ready. best case, we're confirmed ant closer to it landing :) | 21:00 |
| jeblair | krotscheck: that looks right to me | 21:01 |
| jpr1 | asselin_: ah ok. I think i just misinterpreted the format since the comment in the single_node_ci.pp says check out the docs for os-client-config and that page starts off by saying it "is a library for collecting client configuration" wasn't sure if I was feeding an input format or using it to generate a desired output | 21:01 |
| *** jsavak has quit IRC | 21:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-bandersnatch: Add hash_index support https://review.openstack.org/270480 | 21:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-bandersnatch: Add support to install from alternate locations https://review.openstack.org/270481 | 21:02 |
| krotscheck | jeblair: Thanks :) | 21:02 |
| jpr1 | asselin_: think i got it now though that os-client-config is the likely the lib use to parse whatever i put in this part of the common.yaml on the ci node, so it should be in the input format expected by os-client-config | 21:02 |
| *** ihrachys has joined #openstack-infra | 21:03 | |
| asselin_ | jpr1, that sounds right, and nodepool uses os-client-config to get the auth configurations for each cloud | 21:04 |
| *** sridhar_ram1 has joined #openstack-infra | 21:05 | |
| *** ccrouch has quit IRC | 21:05 | |
| *** sridhar_ram has quit IRC | 21:05 | |
| *** ccrouch has joined #openstack-infra | 21:05 | |
| *** jsavak has joined #openstack-infra | 21:05 | |
| *** erikwilson has joined #openstack-infra | 21:06 | |
| openstackgerrit | Merged openstack-infra/system-config: Add wheel mirror to mirrors https://review.openstack.org/253521 | 21:08 |
| *** erikwilson has quit IRC | 21:08 | |
| *** bpokorny has quit IRC | 21:09 | |
| *** amitgandhinz has quit IRC | 21:09 | |
| *** EricGonc_ has joined #openstack-infra | 21:09 | |
| *** bpokorny has joined #openstack-infra | 21:09 | |
| jeblair | gah | 21:10 |
| jeblair | we aren't using puppet-bandersnatch | 21:10 |
| *** EricGonczer_ has quit IRC | 21:10 | |
| jeblair | is anyone still working on the module split out? | 21:11 |
| krotscheck | Ok, first wheel patch merged. Let's see if it breaks everything. | 21:12 |
| *** alkhodos_ has quit IRC | 21:13 | |
| *** ari has quit IRC | 21:13 | |
| *** bcourt has quit IRC | 21:15 | |
| *** hdd has quit IRC | 21:15 | |
| *** Sukhdev has quit IRC | 21:16 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-bandersnatch: Make bandersnatch::httpd class https://review.openstack.org/270486 | 21:16 |
| AJaeger | dims: could you release stevedore as well, please? That one requires also argparse i nlatest release but it's removed since two weeks | 21:16 |
| *** hdd has joined #openstack-infra | 21:16 | |
| dims | AJaeger : ack | 21:17 |
| AJaeger | thanks | 21:17 |
| krotscheck | jeblair: Copy pasta in the copyright ;) | 21:17 |
| mordred | pasta is tasty | 21:18 |
| *** salv-orlando has joined #openstack-infra | 21:18 | |
| *** salv-orlando has quit IRC | 21:19 | |
| *** salv-orlando has joined #openstack-infra | 21:19 | |
| clarkb | ianw: http://paste.openstack.org/show/484457/ | 21:19 |
| clarkb | ianw: it looks like ssh is failing so we are in a build delete loop | 21:19 |
| jeblair | krotscheck: where? | 21:20 |
| jeblair | krotscheck: ah yes. tasty pasta is actually correct. | 21:21 |
| jeblair | krotscheck: i didn't actually write any code in that patch :) | 21:21 |
| krotscheck | jeblair: Oh! Well then. | 21:21 |
| krotscheck | Ironic, because that company doesn't really exist anymore | 21:22 |
| jeblair | omg if we updated the copyright lines everytime hp had an identity crisis :) | 21:23 |
| mordred | :) | 21:23 |
| *** rguillebert has joined #openstack-infra | 21:23 | |
| *** thelo_ has joined #openstack-infra | 21:23 | |
| *** ayoung has quit IRC | 21:24 | |
| jeblair | (someone from hpe should really do that for all files in all repos and pretty much win stackalytics for this cycle) | 21:24 |
| mordred | I kind of want to email Eileen and Nissa just to point out the amusing "Jim submitted a patch as an IBM employee with an HPDC copyright header from 2015 ... and that's all accurate" | 21:24 |
| mordred | jeblair: we could have someone from HP have the proposal bot do it | 21:25 |
| * mordred looks sideways at pleia2 | 21:25 | |
| lifeless | deity no | 21:25 |
| mordred | :) | 21:25 |
| jeblair | why can't i get codesearch to tell me how many files that is? | 21:25 |
| mordred | lifeless: just think of all the productive threads it would start though! | 21:26 |
| *** achanda has quit IRC | 21:26 | |
| mordred | jeblair: cloud. file counts are not cloud-native | 21:26 |
| jeblair | hewlett returns 0; packard returns 1 | 21:26 |
| jeblair | does it not search comments? | 21:26 |
| jeblair | ERROR:open /home/hound/data/idx-dfb868e849e9b9a9/raw/elements/testenv-worker/bin/ci_commands: too many open files | 21:27 |
| jeblair | also that happened | 21:27 |
| jeblair | i broke hound | 21:27 |
| mordred | jeblair: http://codesearch.openstack.org/?q=Hewlett-Packard.Development.Company&i=nope&files=&repos= | 21:27 |
| *** aeng_afk is now known as aeng | 21:27 | |
| jeblair | huh | 21:27 |
| mordred | it defaults to case sensitive | 21:27 |
| jeblair | mordred: i was doing that too | 21:28 |
| *** vgridnev has quit IRC | 21:28 | |
| mordred | 3889 files, according to codesearch, fwiw | 21:28 |
| *** apoorvad has joined #openstack-infra | 21:28 | |
| jeblair | and now it works for me | 21:28 |
| jeblair | even just searching for hewlett | 21:28 |
| mordred | jeblair: cloud | 21:28 |
| jeblair | mordred: i think so | 21:29 |
| jeblair | mordred: either i loosened the jar lid for you, or there was something wrong with my session | 21:29 |
| dims | AJaeger : hop onto #openstack-release please? | 21:29 |
| ianw | clarkb: well, yeah ... i dunno. i mean i can build the image, upload it and ssh in. i know it's almost impossible but are there any boot logs or console? maybe glean is still messing up, but i don't know how | 21:29 |
| mordred | jeblair: https://twitter.com/hergertme/status/689869072385703936 | 21:29 |
| jeblair | okay, back to whatever it was i was doing | 21:29 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone-only uwsgi job https://review.openstack.org/264991 | 21:30 |
| clarkb | ianw I can try getting logs from nova | 21:31 |
| clarkb | ianw my money is on systemd / glean interactions | 21:31 |
| clarkb | but if you can get yours to boot then maybe not | 21:32 |
| ianw | clarkb: well yeah, there were a lot of issues, but i feel like i've worked around them all ... | 21:32 |
| *** apoorvad has quit IRC | 21:33 | |
| clarkb | ianw also Ican boot off the image outside of nodepool as another option | 21:33 |
| ianw | clarkb: yeah, can you share that image with my rax tenant? i feel like there was a way to do that | 21:33 |
| clarkb | ianw I can look into it | 21:34 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone run py34 unit tests only on master https://review.openstack.org/269896 | 21:34 |
| *** apoorvad has joined #openstack-infra | 21:34 | |
| krotscheck | Hrm, that wheel patch of mine wasn't picked up in the last run. Is there a cache thing I'm waiting for? | 21:34 |
| openstackgerrit | Merged openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 21:36 |
| lifeless | sigmavirus24: https://review.openstack.org/#/c/270384/1 - I don't think that should land | 21:37 |
| *** htruta` is now known as htruta | 21:38 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Put wheel-mirror workers in a common group https://review.openstack.org/270495 | 21:38 |
| fungi | mordred: i've created a /etc/puppet/hieradata/production/group/wheel-mirror.yaml for the keys added in 165240 but i need to do ^ for that yeah? | 21:39 |
| *** hashar has quit IRC | 21:40 | |
| fungi | running down all those ssh public host keys was somewhat time consuming. i wonder if we shouldn't put those in public hiera in the future | 21:40 |
| *** amrith is now known as _amrith_ | 21:40 | |
| mordred | fungi: you need to do that- you need to add them to groups.txt | 21:40 |
| *** exploreshaifali has quit IRC | 21:40 | |
| fungi | mordred: oh, thanks. adding to the same change then | 21:40 |
| mordred | fungi: eventually you will not need the $group line | 21:40 |
| mordred | and we'll just depend on the ansible groups totally | 21:41 |
| mordred | but for now you need both | 21:41 |
| fungi | hrm, tricky to make that change not merge-conflict with your groups.txt reorg | 21:43 |
| jeblair | mordred, fungi, clarkb: should puppet-bandersnatch grow the ability to know about afs, or should i keep that in system-config/openstack_project ? | 21:43 |
| fungi | mordred: has that not landed yet? | 21:43 |
| mordred | fungi: zomg | 21:44 |
| mordred | fungi: let's land that change | 21:44 |
| jeblair | mordred, fungi, clarkb: (basically, wendering where i should put the vos release commands, etc) | 21:44 |
| clarkb | jeblair probably system config since thats animplementation detail I think | 21:44 |
| mordred | jeblair: that is a fascinating question | 21:44 |
| fungi | jeblair: i feel like that's outside the scope of puppet-bandersnatch, so either system-config or a new wrapper module | 21:44 |
| clarkb | you could use other backends or local disk etc | 21:44 |
| jeblair | ok, so system-config will have to manage the cron then | 21:44 |
| mordred | the check+vos-release stuff is an us thing | 21:44 |
| mordred | so I almost think that's a puppet-inframirror or something | 21:45 |
| mordred | and bandersnatch is one of the things it knows how to do | 21:45 |
| mordred | but - you know - I'm overthining | 21:45 |
| jeblair | so that we can wrap the cron with k5start and run vos release on success | 21:45 |
| jeblair | as long as that sounds fine to folks, i'll do that | 21:45 |
| jeblair | and we can make puppet-afsmirror later | 21:45 |
| *** ihrachys has quit IRC | 21:45 | |
| fungi | though, yeah, the cronjob to run bandersnatch maybe also doesn't belong in puppet-bandersnatch (or should be optional if it's not already) | 21:45 |
| jeblair | mordred: (actually, rather than a puppet module we may want a python program that understands "run a command with k5start and if it succeeds run vos release") | 21:46 |
| notmorgan | clarkb, mtreinish, stevemar, sdague: that proposed patch looks to work except it had some check errors, package ordering and ironic-something | 21:46 |
| jeblair | fungi: yeah, it's about to become optional :) | 21:46 |
| *** xwizard has joined #openstack-infra | 21:47 | |
| *** _amrith_ is now known as amrith | 21:47 | |
| xwizard | Hi there | 21:47 |
| fungi | heh | 21:47 |
| *** wfoster has joined #openstack-infra | 21:47 | |
| mordred | jeblair: ++ | 21:47 |
| *** dizquierdo has joined #openstack-infra | 21:47 | |
| *** Thelo has joined #openstack-infra | 21:48 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-bandersnatch: Make bandersnatch::cron class https://review.openstack.org/270498 | 21:48 |
| jhesketh | Morning | 21:48 |
| fungi | mordred: so... not merged because its parent is failing the trusty beaker job for openstackci | 21:48 |
| *** jecarey has joined #openstack-infra | 21:48 | |
| * mordred cries | 21:48 | |
| clarkb | xwizard: hello | 21:49 |
| mordred | fungi: hrm | 21:49 |
| xwizard | does aybody know about issues with ci jobs, for example http://logs.openstack.org/13/270213/2/check/gate-kolla-mesos-pep8/787d37c/console.html | 21:49 |
| xwizard | it looks like all kolla-mesas commits affected but it is not clear how to fix it | 21:50 |
| xwizard | it works fine on my desktop | 21:50 |
| clarkb | xwizard is it pip complaining? | 21:50 |
| notmorgan | mtreinish, sdague, clarkb, stevemar: pushing a change to cover the package ordering. | 21:50 |
| mtreinish | notmorgan: huh, I didn't realized we asserted alphabetical order on the package lists | 21:50 |
| notmorgan | mtreinish: yeah enither did i | 21:50 |
| mtreinish | and yeah the ironic job is broken iirc | 21:50 |
| xwizard | clarkb, probably yes | 21:50 |
| clarkb | mtreinish: we shouldnt for pip since order matters | 21:50 |
| mordred | fungi: I do not understand what broke | 21:50 |
| sdague | clarkb: it's for distro packages | 21:50 |
| mordred | nibalizer: ^^ can you help please? | 21:50 |
| notmorgan | ok cool pushed up a fix then, a couple of the dsvm jobs completed, | 21:51 |
| mtreinish | clarkb: real packages | 21:51 |
| mordred | nibalizer: http://logs.openstack.org/96/270296/3/check/gate-openstackci-beaker-trusty-dsvm/a6dcc31/console.html | 21:51 |
| notmorgan | hopefully it'll all be happy | 21:51 |
| sdague | order does not matter there | 21:51 |
| mordred | crinkle: ^^ ? | 21:51 |
| clarkb | xwizard: new release of pip waiting for upstream to release fix | 21:51 |
| notmorgan | and yeah local test looks good for validate | 21:51 |
| nibalizer | mordred: i | 21:51 |
| notmorgan | shoved the token as expected into memcache [shared] | 21:51 |
| xwizard | clarkb, ok, got it :) Thank you! | 21:51 |
| nibalizer | *hi | 21:51 |
| *** davidlenwell has quit IRC | 21:52 | |
| mordred | nibalizer: yay you fix problems yay!!! | 21:52 |
| fungi | nibalizer fix brokens | 21:53 |
| jhesketh | So how are things looking after yesterday's fun? | 21:53 |
| fungi | jhesketh: still plenty of fun to spread around | 21:53 |
| *** davidlenwell has joined #openstack-infra | 21:54 | |
| fungi | i've not had time to dig into your reboot notes yet, but thanks for taking care of those | 21:54 |
| jhesketh | That's okay | 21:54 |
| clarkb | ianw: ok waiting for building node to get far enough along to console log it. I did confirm that config drive is set to true on a nova show | 21:54 |
| nibalizer | oh we broke openstack ci? | 21:54 |
| mordred | yeah | 21:54 |
| mordred | well | 21:54 |
| mordred | not really yet- but that patch wanted to merge this morning (oops) | 21:54 |
| nibalizer | I think its just the same pip stuff bubbbling everywherer | 21:55 |
| nibalizer | http://logs.openstack.org/96/270296/3/check/gate-openstackci-beaker-trusty-dsvm/a6dcc31/console.html#_2016-01-20_18_17_53_787 | 21:55 |
| * krotscheck patiently waits for puppet to run again | 21:55 | |
| nibalizer | eh? | 21:55 |
| *** changbl has quit IRC | 21:55 | |
| jhesketh | Right. Has pip been pinned on the nodes or anything yet? | 21:56 |
| mordred | oh. piddle | 21:56 |
| anteaya | jhesketh: the devstack changes are in | 21:57 |
| anteaya | jhesketh: pip 8 is tripping on argparse, some patches to remove argparse are up | 21:57 |
| *** ybathia has quit IRC | 21:57 | |
| lifeless | I've gone and -1'd all the client ones | 21:58 |
| anteaya | I do believe that pinning pip is hard as it draws in latest of things (virtualenv?) | 21:58 |
| lifeless | testtools depends on unittest2 depends on argparse, and supports python2.6 | 21:58 |
| anteaya | jhesketh: I think fungi's current stance is wait for pip 8.0.1 | 21:58 |
| anteaya | jhesketh: what lifeless said | 21:58 |
| clarkb | lifeless: whats wrong with the clients removing explicit dep on argparse? | 21:58 |
| lifeless | we can't fix unittests by dropping argparse; we may want to - but thats a separate discussion | 21:58 |
| clarkb | lifeless: it is ok for testtools to do that but we don't need it in the clients | 21:58 |
| lifeless | clarkb: it shifts from best-effort for 2.6 support, to 2.6 simply won't work | 21:59 |
| clarkb | I mean we were never beest effort were we? | 21:59 |
| clarkb | we said "python 2.6 is not supported" | 21:59 |
| lifeless | I thought we were | 21:59 |
| jhesketh | anteaya: okay thanks :-) | 21:59 |
| lifeless | anyhow, I'm not core, and even if I was I would only have -1'd - to prompt the discussion | 21:59 |
| *** aeng has quit IRC | 22:00 | |
| *** davidlenwell has quit IRC | 22:00 | |
| *** erikwilson has joined #openstack-infra | 22:01 | |
| *** annegentle has joined #openstack-infra | 22:01 | |
| *** davidlenwell has joined #openstack-infra | 22:02 | |
| *** mgkwill_training is now known as mgkwill_away | 22:02 | |
| fungi | yeah, AJaeger seemed fairly confident that the clients and oslo.config were the primary culprits for pulling in argparse in our dependency chain most places. i was (and still am) skeptical, but noting that it's not a magic bullet it's still cleanup we can do post 2.6. i didn't really plan for people to see it as a general fix for the current issues, hopefully they don't | 22:02 |
| *** mgkwill_away is now known as mgkwill_training | 22:02 | |
| fungi | especially since, even if it were true, we'll probably have pip 8.0.1 and virtualenv 14.0.1 sooner than most of those patches can merge | 22:03 |
| *** annegent_ has joined #openstack-infra | 22:04 | |
| *** annegentle has quit IRC | 22:04 | |
| *** annegentle has joined #openstack-infra | 22:04 | |
| jeblair | fungi, mordred, nibalizer: i think even with 8.0.1 our troubles are just beginning | 22:05 |
| jeblair | mordred's patch failed because of netaddr | 22:05 |
| *** amrith is now known as _amrith_ | 22:05 | |
| anteaya | jhesketh: welcome | 22:05 |
| jeblair | i didn't see dstufft say that 8.0.1 is going to whitelist netaddr like it will argparse | 22:05 |
| anteaya | jhesketh: has taken me most of the day to get that far in understanding | 22:06 |
| krotscheck | I already have 2x +2 on the last wheel system-config patch, can someone come along and add the +A? https://review.openstack.org/#/c/267751/2 | 22:06 |
| *** aeng has joined #openstack-infra | 22:06 | |
| *** hdd has quit IRC | 22:06 | |
| jhesketh | anteaya: I appreciate it :-) | 22:07 |
| cbader_ | clarkb, so question about the nodepool/image/image.log it is still zero after a few hours so I guess it is not building the image right? | 22:07 |
| anteaya | jhesketh: :) | 22:07 |
| *** kzaitsev_mb has joined #openstack-infra | 22:07 | |
| clarkb | cbader_: ya I don't think it is in that case, you can try rerunning the build command to see if that forces the job through | 22:07 |
| *** erikwilson has quit IRC | 22:07 | |
| clarkb | cbader_: you will also need a gearman server running (which is part of zuul typically) | 22:07 |
| cbader_ | clarkb, ok thanks just wanted to ask before trying to run. | 22:08 |
| jeblair | fungi, mordred, nibalizer: i suspect that pretty much all of our projects are uninstallable on our servers now | 22:08 |
| nibalizer | jeblair: neat! | 22:08 |
| *** annegent_ has quit IRC | 22:08 | |
| jeblair | fungi, mordred, nibalizer: (in other words, i think the failure nibz pointed out on mordred's patch is probably actually happening on our servers) | 22:08 |
| *** notnownikki has quit IRC | 22:09 | |
| notmorgan | anteaya: you had a link for a tool that was like average runtim of a given ci job? or was that something else? | 22:09 |
| jeblair | oh, well, or it would be if pip were upgraded. we may not be automatically upgrading pip, so this might only affect _new_ servers. yay | 22:10 |
| anteaya | notmorgan: ahhh, do you remember how long ago I shared or mentioned this link? | 22:10 |
| nibalizer | jeblair: http://puppetboard.openstack.org/report/pypi.ord.openstack.org/cffac692046e62e9eb3db5e80627157cdc6bfb06 seems to say it is working | 22:11 |
| *** aeng has quit IRC | 22:11 | |
| *** annegentle has quit IRC | 22:11 | |
| anteaya | notmorgan: might we be talking about http://status.openstack.org/openstack-health/#/ | 22:11 |
| nibalizer | ie. we don't have an explosion | 22:11 |
| clarkb | nova console-log fail against rax, is that expected? | 22:11 |
| *** trown|mtg is now known as trown|outttypeww | 22:11 | |
| *** aeng has joined #openstack-infra | 22:11 | |
| *** annegentle has joined #openstack-infra | 22:11 | |
| mordred | jeblair: so, we could update install-puppet.sh -isn't that where we install pip for our servers? | 22:12 |
| *** kragniz_ is now known as kragniz | 22:12 | |
| mtreinish | fungi, nibalizer: if you get a sec can you check if there are any tracebacks on health.o.o the api server seems to be stuck again... :( | 22:12 |
| nibalizer | I think we would also only hit that if we were installing something, and the exec['install-loganalyze'] doesn't fire | 22:12 |
| nibalizer | mtreinish: i wil llook | 22:12 |
| anteaya | mtreinish: ah thanks, wondered what was going on | 22:12 |
| *** jsavak has quit IRC | 22:12 | |
| mtreinish | nibalizer: cool, thanks | 22:12 |
| *** ybathia has joined #openstack-infra | 22:12 | |
| clarkb | ianw: doesn't look like I can use nova console-log against rax VMs, I have confirmed that the IP address unpings and doesn't accept ssh connections | 22:13 |
| *** thelo_ has quit IRC | 22:13 | |
| clarkb | so I do not think this is part of any of the recent nodepool chnages | 22:13 |
| clarkb | ianw: now to look into sharing the image | 22:13 |
| *** achanda has joined #openstack-infra | 22:14 | |
| mordred | lifeless: what was your cantrip for installing with get-pip <8 ? | 22:14 |
| mtreinish | anteaya: I mean it also looks like the db is taking it's time: http://paste.openstack.org/show/484459/ but even non db requests aren't responding | 22:14 |
| *** thorst has quit IRC | 22:14 | |
| mordred | lifeless: python get-pip.py -c <$(echo 'pip<8') ? | 22:14 |
| anteaya | I'm for going back to bed | 22:14 |
| anteaya | we can just call off the rest of the day | 22:15 |
| *** thorst has joined #openstack-infra | 22:15 | |
| anteaya | milk and cookies for everybody | 22:15 |
| anteaya | and try again tomorrow | 22:15 |
| jeblair | nibalizer, mordred: right - | 22:15 |
| anteaya | today just doesn't seem to be working | 22:15 |
| jeblair | nibalizer, mordred: we aren't upgrading pip automatically which is protecting our existing servers | 22:15 |
| jeblair | nibalizer, mordred: but if we spin up a new server, and then install something, it is likely to hit this | 22:15 |
| cbader_ | clarkb, I think I know what my problem might be, I had to change the /etc/puppet/modules/diskimage_builder/manifests/init.pp ensure => latest to '1.7.1' to get and error to stop. what is the latest number for diskimage-builder? | 22:15 |
| anteaya | mtreinish: :( | 22:16 |
| clarkb | cbader_: I think 1.7.2 is latest | 22:16 |
| clarkb | cbader_: but you can check on pypi | 22:16 |
| nibalizer | mtreinish: pretty empty log | 22:16 |
| nibalizer | http://paste.openstack.org/show/484460/ | 22:17 |
| mtreinish | oh, and I just got a 500 from health.o.o so there's definitely a stack trace in the logs :) | 22:17 |
| nibalizer | it looks like very little activity | 22:17 |
| mtreinish | nibalizer: hmm I thought I fixed that NoneType exception already | 22:17 |
| clarkb | ianw: IAD is where these fails are occuring if that region works for you I can share the image with an account id looks like | 22:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Pin pip to less than version 8 in install_puppet.sh https://review.openstack.org/270503 | 22:17 |
| mordred | jeblair: ^^ | 22:17 |
| *** FallenPegasus has joined #openstack-infra | 22:18 | |
| *** dkranz has quit IRC | 22:18 | |
| nibalizer | mtreinish: this is the most recent stuff | 22:18 |
| nibalizer | http://paste.openstack.org/show/484461/ | 22:18 |
| cbader_ | clarkb, thanks | 22:18 |
| *** thorst has quit IRC | 22:19 | |
| mtreinish | nibalizer: hmm I would have expected more activity than that. I mean I only see 2 requests from me, and I've made at least 3 in the past few mins :) | 22:19 |
| mtreinish | oh, its the error log, nm | 22:19 |
| nibalizer | yes | 22:19 |
| nibalizer | but it does seem pretty empty | 22:20 |
| mtreinish | I guess that script timeout error was the 500 I hit | 22:20 |
| *** ayoung has joined #openstack-infra | 22:20 | |
| * greghaynes returns | 22:20 | |
| greghaynes | I hear there are glean issues afoot? | 22:20 |
| clarkb | greghaynes: fedora-23 images build and boot but nodepool can't ssh to them | 22:21 |
| mtreinish | which doesn't really help with finding what is the issue :) | 22:21 |
| greghaynes | clarkb: ah, fun | 22:21 |
| clarkb | I have confirmed that they do not ping and do not accept ssh connections | 22:21 |
| greghaynes | clarkb: is it just in rax? | 22:21 |
| clarkb | greghaynes: thats the only place attempting boots so far | 22:21 |
| *** edmondsw has quit IRC | 22:21 | |
| lifeless | mordred: dstufft fixed it, you can do get-pip.py "pip<8" | 22:22 |
| ianw | clarkb: pm'd you my account number, if you could share the image that would be great | 22:22 |
| *** aeng has quit IRC | 22:22 | |
| *** annegentle has quit IRC | 22:22 | |
| lifeless | fungi: unittest2 has argparse and won't be dropping it today | 22:22 |
| greghaynes | ah, if ianw is on it I can step back for a bit. I also have the ability to make/upload rax images if we need that... | 22:22 |
| nibalizer | mordred: 'puppet apply /opt/system-config/production/manifests/site.pp' is how I run puppet right | 22:22 |
| lifeless | fungi: thus all our unittests need argparse | 22:22 |
| ianw | greghaynes: yeah, just your regular day of debugging/yak shaving around here | 22:23 |
| greghaynes | ianw: new day, same yak | 22:23 |
| *** boris-42 has joined #openstack-infra | 22:23 | |
| *** dims has quit IRC | 22:24 | |
| *** EricGonc_ has quit IRC | 22:24 | |
| *** tiswanso has quit IRC | 22:25 | |
| greghaynes | ianw: speaking of - https://review.openstack.org/#/c/270010/ | 22:25 |
| greghaynes | ianw: to unbreak dib gate | 22:25 |
| mtreinish | nibalizer: so here's the fix for that traceback in the log https://review.openstack.org/#/c/270505/ | 22:26 |
| *** silos has joined #openstack-infra | 22:26 | |
| *** sridhar_ram1 has quit IRC | 22:26 | |
| mtreinish | I wonder if that was really causing the issue | 22:26 |
| cbader_ | clarkb, So I tried to do a nodepool dib-image-delete 1 and it hangs, here is a pastebin I think my system is not configure to talk to nodepool upstream is that what the pastebin shows,http://paste.openstack.org/show/484462/ | 22:26 |
| *** tiswanso has joined #openstack-infra | 22:26 | |
| *** tiswanso has quit IRC | 22:26 | |
| mtreinish | it could have, I guess if it fails I don't know what the recovery story is | 22:26 |
| cbader_ | clarkb, is there someone else I can ask these questions of? | 22:26 |
| openstackgerrit | Merged openstack/diskimage-builder: Move speedup section to image building guide https://review.openstack.org/268769 | 22:27 |
| *** sridhar_ram has joined #openstack-infra | 22:27 | |
| anteaya | cbader_: if you attend third party meetings you meet other third party operators there | 22:27 |
| anteaya | cbader_: and get to ask questions of each other, it is very helpful | 22:27 |
| clarkb | cbader_: your nodepool should talk to your local gearman | 22:27 |
| cbader_ | shows this 2016-01-20 14:23:59,505 INFO gear.Connection.nodepool: Connected to 127.0.0.1 port 4730 | 22:28 |
| *** rlrossit_ has joined #openstack-infra | 22:28 | |
| rlrossit_ | clarkb: mriedem sent me here, but I've been noticing that bluebox nodes always fail the grenade multinode jobs with the same error every time, and they're the only nodes doing so | 22:29 |
| mriedem | rlrossit_: link? | 22:29 |
| rlrossit_ | finding | 22:29 |
| clarkb | cbader_: if you telent to localhost 4730 and enter status you should get a job listing | 22:29 |
| cbader_ | anteaya, yes I have it in my calendar next one is 26th at 11:00PST correct | 22:29 |
| clarkb | cbader_: in there you can see if you have the image build jobs | 22:29 |
| rlrossit_ | clarkb: mriedem: http://logs.openstack.org/56/268856/1/check/gate-grenade-dsvm-multinode/01039c5/ | 22:30 |
| clarkb | rlrossit_: can you link to the error too? | 22:30 |
| anteaya | cbader_: 26th at 15:00 utc | 22:30 |
| *** dims has joined #openstack-infra | 22:30 | |
| rlrossit_ | clarkb: bad link, don't listen to that one | 22:30 |
| anteaya | whatever that is in pst | 22:30 |
| cbader_ | anteays, thank you i had it wrong so 15:00 utc is 8:00 PST will attend. | 22:32 |
| rlrossit_ | mriedem: here ya go https://jenkins06.openstack.org/job/gate-grenade-dsvm-multinode/2033/consoleFull | 22:32 |
| anteaya | cbader_: sorry it is the 25th the monday | 22:32 |
| anteaya | mmedvede will be chairing as I will be away, it is a good place to ask the kinds of questions you are asking | 22:33 |
| clarkb | greghaynes: new issue with nodepool image builders | 22:33 |
| ianw | clarkb: yeah, that imge is coming up with a selinux failure on boot -> i fixed that with https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=b0eff0df074b33f5f2d1ef40a31d1c280ea44407 | 22:33 |
| clarkb | greghaynes: dib is leaky and leaves a bunch of .d/ dirs in the images path | 22:33 |
| clarkb | greghaynes: these all get registered as possible images to upload and delete in gearman | 22:33 |
| *** xwizard has quit IRC | 22:33 | |
| greghaynes | clarkb: ah | 22:33 |
| ianw | so old images is looking suspect | 22:33 |
| greghaynes | clarkb: easy enough to fix | 22:33 |
| mriedem | silly rlrossit_: http://logs.openstack.org/17/270417/1/gate/gate-grenade-dsvm-multinode/0858eae//logs/grenade.sh.txt.gz#_2016-01-20_21_46_46_369 | 22:34 |
| mriedem | test_volume_boot_pattern ssh fail | 22:34 |
| greghaynes | clarkb: those .d's are intentional btw. You might be able to turn it off | 22:34 |
| mriedem | rlrossit_: the thing you saw in the jenkins log was generic console output | 22:34 |
| clarkb | ianw: greghaynes as for fedora failing to boot we seem to be waiting on uploads for all the images | 22:34 |
| clarkb | ianw: greghaynes in gearman I see that there is one worker registered for all these upload jobs but no workers in the running column | 22:34 |
| *** dizquierdo has quit IRC | 22:35 | |
| *** dizquierdo has joined #openstack-infra | 22:36 | |
| *** changbl has joined #openstack-infra | 22:36 | |
| Sam-I-Am | lifeless: ping | 22:36 |
| cbader_ | anteaya so every monday or every other monday | 22:36 |
| lifeless | pong | 22:36 |
| clarkb | I am reading logs to see if I can tell where it got stuck | 22:36 |
| Sam-I-Am | lifeless: should https://review.openstack.org/#/c/270417/ get a recheck or is something else up? | 22:37 |
| greghaynes | clarkb: hrmmm | 22:37 |
| greghaynes | clarkb: is it uploading something else? | 22:37 |
| greghaynes | clarkb: or building? | 22:37 |
| anteaya | cbader_: every monday at 15:00 utc and every tuesday at 08:00 utc for the ones I chair | 22:37 |
| *** jecarey has quit IRC | 22:38 | |
| Sam-I-Am | lifeless: err, maybe there's already a discussion in here about it | 22:38 |
| clarkb | greghaynes: ianw ok looks like it managed to upload fedora-23 to hpcloud-b1 and last log for upload is for devstack-precise to hpcloud-b1 | 22:38 |
| clarkb | greghaynes: ianw so either that has fialed and it didn't realize or it is still in progress | 22:38 |
| clarkb | that started at 18:39 UTC or about 4 hours ago | 22:39 |
| jeblair | mordred, nibalizer: yeah, i think a pin on install_puppet.sh will help our real servers, and then i also think we need a pin for our tests | 22:39 |
| greghaynes | clarkb: and gearman shows that job as not running? | 22:39 |
| clarkb | greghaynes: I am going to double check on that now that I hav ea speific imgae job name | 22:39 |
| nibalizer | jeblair: ok | 22:39 |
| clarkb | correct image-upload:176178001 | 22:40 |
| *** thorst has joined #openstack-infra | 22:40 | |
| clarkb | er image-upload:176178 0 0 1 | 22:40 |
| clarkb | greghaynes: there may be an issue uplodaing >1 image? we only do one in the dsvm job | 22:40 |
| cbader_ | anteaya, so that would be 12:00 PST so I will attend the Monday for now but thanks I already work 5:00am to 5:00pm | 22:41 |
| notmorgan | dhellmann: this is finally passin check https://review.openstack.org/#/c/267270/ | 22:41 |
| *** diazjf has joined #openstack-infra | 22:41 | |
| jeblair | nibalizer, mordred: maybe a 'pin-pip' macro in jjb that we can add to the 'gate-openstackci-beaker-{ostype}-dsvm' job ? | 22:41 |
| anteaya | cbader_: you asked if there was anyone one other than clarkb you can ask questions of, I gave you an answer | 22:41 |
| greghaynes | clarkb: that 1 on the end is number of workers who can perform that job? | 22:41 |
| *** rlrossit_ has left #openstack-infra | 22:41 | |
| anteaya | cbader_: use it or not as you see fit | 22:41 |
| dhellmann | notmorgan : +2 | 22:42 |
| nibalizer | jeblair: and then all the other jobs that grow that need, I like it | 22:42 |
| notmorgan | dhellmann: yay | 22:42 |
| lifeless | Sam-I-Am: I don't know; it may be impacted by the pip 8 thing | 22:42 |
| cbader_ | anteaya, thank you very much for the help, if I need to attend the midnight meeting to meet people I will. | 22:42 |
| clarkb | greghaynes: yes, queued running num_workers | 22:42 |
| *** burgerk_ has quit IRC | 22:42 | |
| greghaynes | clarkb: ok, so it isn't queued. Is one of the other ones queued in gearman? | 22:42 |
| *** regXboi has quit IRC | 22:42 | |
| notmorgan | dhellmann: hopefully we can land that so i can cleanup the code in keystoneclient/keystonemiddleware/keystoneauth and not worry about weird dependency graphs | 22:43 |
| greghaynes | clarkb: trying to figure out if nodepool is actually asking it to upload anything | 22:43 |
| dhellmann | notmorgan : is positional in our ci? | 22:43 |
| *** hdd has joined #openstack-infra | 22:43 | |
| notmorgan | dhellmann: why can't py2 just have that nice positional/kw-only syntax :P | 22:43 |
| notmorgan | dhellmann: no. it's not in openstacks. i can move it there | 22:43 |
| dhellmann | notmorgan : nah, I was just checking | 22:43 |
| notmorgan | dhellmann: it's in github and run through travis w/ tox | 22:43 |
| dhellmann | I mean, do if you want, but it's not required | 22:43 |
| fungi | mordred: nibalizer: the recheck on 270262 passed check and failed gate again | 22:43 |
| notmorgan | dhellmann: for all PRs. it just is so small it felt silly to put it in our ci | 22:43 |
| dhellmann | sure | 22:44 |
| notmorgan | dhellmann: i expect it to... well frankly never need to change. | 22:44 |
| clarkb | mriedem: rlrossit left :( the issue is http://logs.openstack.org/17/270417/1/gate/gate-grenade-dsvm-multinode/0858eae/logs/grenade.sh.txt.gz#_2016-01-20_21_46_46_405 which attempts to ssh over the private IP | 22:44 |
| notmorgan | unless we have a broken pbr sometime. | 22:44 |
| diazjf | Any chance you can help with the docs-gate failing because of the new pip version (http://logs.openstack.org/68/241068/16/check/gate-barbican-specs-docs/d45ba92/console.html), bug is related to https://github.com/openstack-dev/devstack/commit/1d27155f650e9b8e4053a201f1f08c494b820de2 | 22:44 |
| clarkb | mriedem: I do not know if sshing over the private ip is expected to work since that network is isolated from the host node where tempest runs | 22:44 |
| nibalizer | jeblair: we have this http://git.openstack.org/cgit/openstack-infra/puppet-pip/tree/manifests/init.pp | 22:44 |
| *** thorst has quit IRC | 22:44 | |
| clarkb | mriedem: the node where tempest runs will need to have an interface on that network or a route to that network in order for that to work | 22:44 |
| notmorgan | dhellmann: if it ever becomes an issue i'll just move it over to our ci, would be quick to do so | 22:45 |
| *** rlrossit_ has joined #openstack-infra | 22:45 | |
| fungi | mordred: nibalizer: the failures were in rackspace, successes were not. suggests our inability to run dib has spared non-rackspace devstack images a pip 8 fate? | 22:45 |
| dhellmann | notmorgan : yeah, we're not obsessing about importing everything any more, so it's not a big deal | 22:45 |
| *** tonytan4ever has quit IRC | 22:45 | |
| nibalizer | fungi: heh | 22:45 |
| fungi | should i roll back rackspace's devstack snapshot images from today? | 22:46 |
| mriedem | clarkb: mtreinish pointed out grenade multinode is 100% fail on bluebox http://status.openstack.org/openstack-health/#/g/node_provider/bluebox-sjc1 | 22:46 |
| *** doug-fis_ has quit IRC | 22:46 | |
| nibalizer | I'm going to lose connectivity here in a bit | 22:46 |
| clarkb | mriedem: I suppose its possible that if we get an IP for the hypervisor on 10.1.0.0/24 for example on bluebox then it will think it can route via the private IP | 22:47 |
| *** jamielennox|away is now known as jamielennox | 22:47 | |
| clarkb | mriedem: except it can't | 22:47 |
| clarkb | I can login to a bluebox machine and check the IPs | 22:47 |
| mgagne | mriedem cool, didn't know about this URL thanks! | 22:47 |
| clarkb | or thats recorded in the job itself | 22:47 |
| clarkb | inet 10.220.0.29/22 so thats not it | 22:47 |
| *** weshay has quit IRC | 22:47 | |
| cbader_ | clarkb, is there a place for the labels for the telnet status? | 22:47 |
| clarkb | mriedem: any idea why the flating IP is not used? | 22:48 |
| *** hdd has quit IRC | 22:48 | |
| clarkb | cbader_: no, its queued, running, workers that can run this job | 22:48 |
| mtreinish | clarkb: it could be a tempest configuration issue, how tempest ssh's into a guest is configurable | 22:48 |
| mtreinish | but that whole code path is a mess, so its difficult to track down | 22:48 |
| clarkb | mriedem: but I am fairly certain that that IP is not routable from tempest | 22:48 |
| mriedem | mtreinish: clarkb: yeah tempest.conf says to use fixed http://logs.openstack.org/17/270417/1/gate/gate-grenade-dsvm-multinode/0858eae/logs/old/tempest_conf.txt.gz | 22:49 |
| mriedem | neutron jobs using floating | 22:49 |
| mriedem | and this test fails far less often in neutron multinode jobs it looks like (at least from logstash) | 22:49 |
| cbader_ | clarkb, so have merger:merge 1 workers, image-delete:1 1 quesed and merger:update 1 worker so it isn' doing anything righe | 22:50 |
| mtreinish | mriedem: oh, right we "fixed" that not too long ago | 22:50 |
| mriedem | is that a pun? | 22:50 |
| mriedem | congratulations if so | 22:50 |
| nibalizer | mtreinish: I ran puppet on health.o.o | 22:50 |
| clarkb | cbader_: you should have an image-build:image-name I think | 22:50 |
| *** ajmiller_ has joined #openstack-infra | 22:50 | |
| clarkb | cbader_: if not then it is probably missing from your config? | 22:50 |
| mtreinish | mriedem: https://github.com/openstack-dev/devstack/commit/563a7e75b7d26275a7416eb4d6641fcfe867b45a | 22:50 |
| mtreinish | nibalizer: ok, cool. To pull in the 1 liner? | 22:51 |
| cbader_ | clarkb, http://paste.openstack.org/show/484466/ | 22:51 |
| mtreinish | mriedem: oh, it is a pun I guess | 22:51 |
| mtreinish | I didn't even make the connection | 22:51 |
| mriedem | mtreinish: hrm | 22:51 |
| clarkb | mriedem: mriedem if you want to make that routable on multinode we may need another bridge to carry the private traffic between hypervisors and VMs | 22:51 |
| *** jpr1 has quit IRC | 22:52 | |
| mriedem | mtreinish: would really need sdague to see if 563a7e75b7d26275a7416eb4d6641fcfe867b45a is causing issues now, or making something worse in unintended ways | 22:52 |
| notmorgan | mtreinish: is there anything i should need to do for the memcache thing for middleware for grenade? [backport to the previous version of devstack once it lands in master?] | 22:53 |
| clarkb | cbader_: ya s oI don't think you have an image build job | 22:53 |
| notmorgan | mriedem: or not something we care about | 22:53 |
| clarkb | cbader_: make sure yo uhave an image configured in your nodepool.yaml | 22:53 |
| * mriedem has to bail for awhiole | 22:53 | |
| mriedem | also, | 22:53 |
| * mriedem is not mtreinish | 22:53 | |
| *** ajmiller has quit IRC | 22:53 | |
| anteaya | no! | 22:54 |
| mtreinish | clarkb, mriedem: we can add: TEMPEST_SSH_CONNECT_METHOD to the localrc passthrough on the multinode jobs I think that'll set it | 22:54 |
| mtreinish | https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L243 | 22:54 |
| notmorgan | mriedem: so, about that tempest stuff. | 22:54 |
| notmorgan | mriedem: *shiftyeyes* | 22:54 |
| mtreinish | so if we need floating to work on multinode | 22:54 |
| *** devananda has quit IRC | 22:54 | |
| mtreinish | notmorgan: if you backport the change to liberty devstack it'll run during grenade jobs | 22:54 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:54 | |
| notmorgan | mtreinish: great once it lands in master i'll do that | 22:55 |
| clarkb | mtreinish: the alternative is bridge the private Ips onto the hypervisors | 22:55 |
| clarkb | and put an interface on that network on the hypervisors | 22:55 |
| notmorgan | mtreinish: that way we can ensure we're getting the most out of it in all tangible ways | 22:55 |
| Sam-I-Am | clarkb: eeewww | 22:55 |
| cbader_ | clarkb, http://paste.openstack.org/show/484467/ I think I have what I need am I missing something | 22:55 |
| *** achanda has quit IRC | 22:55 | |
| clarkb | Sam-I-Am: exactly | 22:55 |
| *** mriedem has quit IRC | 22:55 | |
| notmorgan | mtreinish: fwiw, it does look like it knocked a chunk of time off devstack runs based upon the check run that is currently going | 22:56 |
| clarkb | it turns out there are a lot of assumptions about network design that make testing terrible | 22:56 |
| mtreinish | clarkb: is that what the other clouds do? because this looks like it's only an issue on bluebox | 22:56 |
| clarkb | especially when you run inside of the network you are testing with a bunch of tunnels | 22:56 |
| Sam-I-Am | fips are the only way to go unless the vms are booted on the same network as tempest, or tempest runs in a vm attached to the priv net | 22:56 |
| clarkb | mtreinish: it should fail that way on every cloud if you use the private IPs | 22:56 |
| clarkb | mtreinish: we install our own overlay network to run openstack against | 22:57 |
| clarkb | which should be consistent across clouds | 22:57 |
| Sam-I-Am | is there anything i can help with here? i use multinode in pubcloud for my own labs and have to host-tunnel and bunch of crap to make it work. | 22:57 |
| Sam-I-Am | s/and/a | 22:57 |
| mtreinish | clarkb: the numbers show the test is only failing 10% of the time on multinode: http://status.openstack.org/openstack-health/#/job/gate-grenade-dsvm-multinode | 22:57 |
| *** dimtruck is now known as zz_dimtruck | 22:57 | |
| clarkb | mtreinish: are you using floating IPs on the other jobs? | 22:57 |
| clarkb | I don't know what determines private vs floating | 22:57 |
| clarkb | Sam-I-Am: we already have it in place | 22:57 |
| mtreinish | the config should be the same for all n-net jobs | 22:58 |
| mtreinish | unless there is a local override in project config (or d-g I guess) | 22:58 |
| cbader_ | anteaya, do you have any clue why my dib-image-build process is not running. | 22:58 |
| *** achanda has joined #openstack-infra | 22:58 | |
| *** devananda has joined #openstack-infra | 22:59 | |
| *** ccrouch has quit IRC | 22:59 | |
| craige | o/ | 22:59 |
| clarkb | mtreinish: so when I got all this working only floating IP was ever used | 22:59 |
| cbader_ | anteaya, should i restart nodepool on ps -ef for nodepool I see nodepoold nodepool.yaml and nodepool.conf | 22:59 |
| clarkb | so use of private is new | 22:59 |
| Sam-I-Am | clarkb: yeah, just didnt know if this was a weird case | 22:59 |
| clarkb | regardless of cloud and I would not expect it to work | 22:59 |
| mtreinish | clarkb: here is it working on rax: http://logs.openstack.org/52/267152/3/gate/gate-grenade-dsvm-multinode/15a6c81/ (with fixed) | 23:00 |
| mtreinish | clarkb: right, sdague changed it recently in devstack | 23:00 |
| mtreinish | for n-net | 23:00 |
| mtreinish | clarkb: https://github.com/openstack-dev/devstack/commit/563a7e75b7d26275a7416eb4d6641fcfe867b45a | 23:00 |
| clarkb | mtreinish: ok then it working in some places is the anomoly | 23:00 |
| boris-42 | clarkb: hi | 23:00 |
| clarkb | the reason is we use mulit host or whatever its called which makes each hypervisor its own router | 23:00 |
| clarkb | neutron backhauls everything back to the controller unless you dvr in the multinode jobs | 23:01 |
| clarkb | so you get things routable from there over the backhual but nova net does not do this | 23:01 |
| boris-42 | clarkb: rally installation script stopped working on ubuntu nodes, it says something strange | 23:01 |
| *** ccrouch has joined #openstack-infra | 23:01 | |
| *** rlrossit_ has quit IRC | 23:01 | |
| *** weshay has joined #openstack-infra | 23:01 | |
| boris-42 | clarkb: like 2016-01-20 22:58:10.691 | Detected a distutils installed project ('netaddr') which we cannot uninstall. The metadata provided by distutils does not contain a list of files which have been installed, so pip does not know which files to uninstall. | 23:01 |
| anteaya | cbader_: O dpm | 23:01 |
| clarkb | boris-42: yes pip is broken | 23:01 |
| *** nelsnels_ has joined #openstack-infra | 23:01 | |
| mtreinish | boris-42: that's the pip 8 bug | 23:02 |
| anteaya | cbader_: I don't know the current status of your system or what you have already done | 23:02 |
| boris-42 | mtreinish: clarkb ah ok thanks! | 23:02 |
| cbader_ | anteaya, sorry O dpm ? | 23:02 |
| clarkb | mtreinish: http://logs.openstack.org/52/267152/3/gate/gate-grenade-dsvm-multinode/15a6c81/logs/tempest.txt.gz#_2016-01-20_17_44_43_966 that says public | 23:02 |
| anteaya | cbader_: however the portion of your nodepool.yaml that you included in the paste does match what is in the project config example | 23:02 |
| mtreinish | clarkb: I just did a quick check, it only looks like it doesn't work on bluebox and internap | 23:02 |
| *** bardia_ has joined #openstack-infra | 23:02 | |
| mtreinish | all the other node providers pass the grenade multinode job | 23:02 |
| *** doug-fish has joined #openstack-infra | 23:03 | |
| anteaya | cbader_: that was "I don" one space displaced on my keyboard | 23:03 |
| *** markmcd1 has joined #openstack-infra | 23:03 | |
| *** andrearo1 has joined #openstack-infra | 23:03 | |
| anteaya | I made a mistake | 23:03 |
| clarkb | mtreinish: ok, liek I said we need to figure out why it works on any of them | 23:03 |
| clarkb | mtreinish: because it should not work | 23:03 |
| clarkb | that it works is a bug :) | 23:03 |
| *** silos has left #openstack-infra | 23:03 | |
| *** kmartin_ has joined #openstack-infra | 23:03 | |
| clarkb | at http://logs.openstack.org/52/267152/3/gate/gate-grenade-dsvm-multinode/15a6c81/logs/tempest.txt.gz#_2016-01-20_17_46_52_811 it says private | 23:03 |
| clarkb | http://logs.openstack.org/52/267152/3/gate/gate-grenade-dsvm-multinode/15a6c81/logs/tempest.txt.gz#_2016-01-20_17_46_56_034 is private again | 23:04 |
| *** ccrouch has quit IRC | 23:04 | |
| notmorgan | mtreinish: ok proposed it as a backport... and bah, grenade transient failure hit the master change :( | 23:04 |
| mtreinish | clarkb: right the first config output in the tempest log is tempest-verify-config which is run before devstack finishes configuring | 23:04 |
| clarkb | are these interleaved? | 23:04 |
| *** andrearosa has quit IRC | 23:04 | |
| *** asselin_ has quit IRC | 23:04 | |
| mtreinish | clarkb: http://logs.openstack.org/52/267152/3/gate/gate-grenade-dsvm-multinode/15a6c81/logs/old/tempest_conf.txt.gz | 23:04 |
| notmorgan | mtreinish: will recheck once it finishes. | 23:04 |
| *** asselin has joined #openstack-infra | 23:04 | |
| mtreinish | clarkb: oslo.log dumps the tempest config file 5 times during a run :( | 23:04 |
| mtreinish | once for verify-tempest-config | 23:04 |
| *** bardia has quit IRC | 23:04 | |
| *** markmcd has quit IRC | 23:04 | |
| *** nelsnelson has quit IRC | 23:04 | |
| *** kmartin has quit IRC | 23:04 | |
| *** mordred has quit IRC | 23:04 | |
| mtreinish | and then once per testr worker | 23:04 |
| mtreinish | the testr once will be interleaved because they start up around the same time | 23:05 |
| *** dingyichen has joined #openstack-infra | 23:05 | |
| mtreinish | it's easier to just look at the actually config file | 23:05 |
| clarkb | http://logs.openstack.org/52/267152/3/gate/gate-grenade-dsvm-multinode/15a6c81/logs/tempest.txt.gz#_2016-01-20_17_48_20_913 there we see it successfuly connect tothe private net | 23:05 |
| mtreinish | notmorgan: ok, I'll take a look | 23:05 |
| *** mordred has joined #openstack-infra | 23:05 | |
| clarkb | so I think ^ is not expected to work | 23:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-openafs: Add kstart to client https://review.openstack.org/270516 | 23:05 |
| *** ccrouch has joined #openstack-infra | 23:05 | |
| clarkb | if we didn't use multi host it would work | 23:06 |
| clarkb | so thats the next thing to check | 23:06 |
| *** ybathia_ has joined #openstack-infra | 23:06 | |
| mtreinish | clarkb: do we use multi host? | 23:06 |
| *** zhiyan has quit IRC | 23:06 | |
| clarkb | mtreinish: we should yes | 23:06 |
| clarkb | oh I know | 23:06 |
| clarkb | this will also work if those VMs boot on the controller and not the compute host | 23:06 |
| *** ybathia has quit IRC | 23:06 | |
| *** ybathia_ is now known as ybathia | 23:06 | |
| mtreinish | we do | 23:06 |
| mtreinish | http://logs.openstack.org/52/267152/3/gate/gate-grenade-dsvm-multinode/15a6c81/logs/etc/nova/nova.conf.txt.gz | 23:06 |
| clarkb | so need to see if compute host is doing anything | 23:07 |
| *** e0ne has quit IRC | 23:07 | |
| *** dims has quit IRC | 23:07 | |
| clarkb | depending on how the scheduler schedules this is a race | 23:07 |
| clarkb | anyways quick fix is go back to using floating ips | 23:08 |
| clarkb | and in meantime see if we can confirm it works when sshing to controller and fails when sshing to compute host | 23:08 |
| *** doug-fish has quit IRC | 23:08 | |
| *** zhiyan has joined #openstack-infra | 23:08 | |
| mtreinish | clarkb: oh, that could be. Although if that was the case I wouldn;t expect the failure pattern we're seeing | 23:08 |
| mtreinish | where it's 100% fail on 2 providers an almost always succesful everywhere else | 23:09 |
| mgagne | mtreinish which providers? =) | 23:09 |
| mtreinish | mgagne: bluebox and internap are 100% fail for this job | 23:09 |
| *** piet has quit IRC | 23:10 | |
| mtreinish | http://status.openstack.org/openstack-health/#/g/node_provider/internap-nyj01 and http://status.openstack.org/openstack-health/#/g/node_provider/bluebox-sjc1 | 23:10 |
| clarkb | mtreinish: it could be a timing thing | 23:10 |
| clarkb | regardless this is not supposed to be reliable | 23:10 |
| clarkb | so I don't feel like we need to keep chasing down a broken thing that is known to be broken | 23:10 |
| mgagne | mtreinish if you could find UUID for both nodes, I can check if they were on the same physical nodes or not | 23:10 |
| clarkb | mgagne: I don't think we need to bother | 23:10 |
| clarkb | mgagne: tempest is doing an unsupported operation | 23:11 |
| *** achanda has quit IRC | 23:11 | |
| mtreinish | clarkb: well I'd say just throw up the project-config change to tell devstack to configure tempest to use floating instead of fixed | 23:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add mirror_update.openstack.org https://review.openstack.org/270520 | 23:11 |
| jeblair | fungi, krotscheck, mordred: https://review.openstack.org/270520 is the root of the afs mirror patch tree; topic:afs-mirror | 23:12 |
| dstufft | jeblair: netaddr isn't special like argparse is, breaking the situation netaddr in was purposeful (unlike argparse).Part of what I'll be doing here is deciding if we want to roll that back or not. | 23:13 |
| *** johnthetubaguy has quit IRC | 23:13 | |
| mgagne | mtreinish list of jobs on the provider page looks to not be specific to the provider, while reading logs, I saw it ran at bluebox | 23:13 |
| mgagne | or in fact, there is no easy way to list specific failures, once you click on the job name, you end up on a non-provider specific page which list all job runs | 23:14 |
| clarkb | mtreinish: I think that should be a tempest default since tempest is supposed to run black box | 23:14 |
| clarkb | mtreinish: and VM private networks should only be accessbile from hosts on those networks | 23:14 |
| *** rajinir has quit IRC | 23:15 | |
| *** johnthetubaguy has joined #openstack-infra | 23:16 | |
| mtreinish | mgagne: yeah, that's expected behavior/bug right now. The per job page is not filtered by the previous pages | 23:16 |
| boris-42 | clarkb: btw could you help with getting this in project-config https://review.openstack.org/#/c/270134/ | 23:16 |
| clarkb | boris-42: timeouts are likely due toe keystone middleeware | 23:17 |
| mtreinish | clarkb: it's not so straightforward, it depends on the cloud and how the networks are configured | 23:17 |
| boris-42 | clarkb: work on rally is currently blocked because this job is facing timeouts 100% | 23:17 |
| clarkb | boris-42: fix is enroute you shouldn't incrase your timeouts | 23:17 |
| clarkb | mtreinish: well you are basically saying that tempest has to run on a vm in the cloud if you do that | 23:17 |
| clarkb | mtreinish: wheras public should be reachable from everywhere | 23:17 |
| boris-42 | clarkb: ya that job is doing a lot of keystone calls | 23:17 |
| clarkb | boris-42: so no I think we should -2 that change and/or abandon it | 23:17 |
| boris-42 | clarkb: ok | 23:18 |
| mtreinish | clarkb: which options are you talking about? | 23:18 |
| clarkb | mtreinish: the validate ssh network | 23:18 |
| clarkb | mtreinish: when set to private that implies that tempest is running on the private network too otherwise ssh will fail | 23:18 |
| *** gordc has quit IRC | 23:19 | |
| mtreinish | clarkb: so public is the default for that option in tempest: https://github.com/openstack/tempest/blob/master/tempest/config.py#L697 | 23:21 |
| mtreinish | devstack is setting it otherwise: https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L454 | 23:21 |
| clarkb | mtreinish: ok, then we should stop doing that in devstack | 23:21 |
| mtreinish | also in n-net I thought those were just labels and the names didn't have any special meaning | 23:21 |
| *** diazjf has quit IRC | 23:22 | |
| mtreinish | like public doesn't actually mean it's a public network (although that case would be terribad) | 23:22 |
| clarkb | mtreinish: the way we deploy it thats what it means | 23:22 |
| mordred | it also does not mean that in neutron | 23:22 |
| mordred | BUT | 23:22 |
| mordred | tools make hueristic choices bsaed on what they can figure out | 23:23 |
| clarkb | private isn't bridged across the hypervisor | 23:23 |
| clarkb | public is | 23:23 |
| clarkb | beause it is public | 23:23 |
| santoshk | Hello openstack-infra, I am test engineer working for tacker project in openstack (https://github.com/openstack/tacker) . We are looking to setup gate jobs which can setup 2 devstack setups with each having its own keystone servers.... | 23:23 |
| clarkb | every hypervisor gets an IP on the public net too which sets up routes | 23:23 |
| santoshk | Testcase we are trying to test is basically we will have 2 (or more) such devstack setup where tacker server runs only on one devstack and will be able to spawn vms on both devstacks (local and also remote). Are there any jobs in any project currently having a setup like this.. | 23:23 |
| mtreinish | clarkb: ok, then we should change that in devstack | 23:24 |
| clarkb | santoshk: there are not | 23:24 |
| santoshk | clarkb thankyou for reply, if we want to request a setup like that...can we open a story/enhancement bug | 23:25 |
| clarkb | santoshk: there was an attempt to do federated keystones across multiple installs but I don't think that finished implementation | 23:25 |
| clarkb | santoshk: it would probably be better if you just made the changes to run your jobs that way | 23:25 |
| clarkb | santoshk: you can run on multiple hosts and you could run distinct devstack installs, no one does it today | 23:25 |
| *** eil397 has joined #openstack-infra | 23:26 | |
| notmorgan | sdague: https://review.openstack.org/#/c/270474/ will show you the timeings on the jobs there, transient failure on grenade. | 23:26 |
| notmorgan | sdague: also https://review.openstack.org/#/c/270515/ so we can get the same benefit in grenade [backport to stable/liberty] | 23:26 |
| clarkb | mtreinish: I think our options are 1) revert use of private net 2) bridge private net across hypervisors 3) route private net over public net (wat) 4) stop using multihost which was broken when we switched to multi host and no one was interested in fixing not multi host | 23:27 |
| clarkb | specifically was metadata server didn't work properly on the compute node. The traffic wasn't backhauled ot api service iirc | 23:28 |
| mtreinish | clarkb: https://review.openstack.org/#/c/270528/ | 23:28 |
| *** mudassirlatif has joined #openstack-infra | 23:28 | |
| *** mudassirlatif has quit IRC | 23:29 | |
| *** gildub has joined #openstack-infra | 23:30 | |
| mtreinish | clarkb: well I went with #1, although revert might not be the right word because it's been set that way for as long as I can remember | 23:30 |
| *** sabeen3 has quit IRC | 23:31 | |
| *** dizquierdo has quit IRC | 23:31 | |
| mtreinish | clarkb: like: https://github.com/openstack-dev/devstack/blob/grizzly-eol/lib/tempest#L217 | 23:31 |
| *** dims has joined #openstack-infra | 23:32 | |
| *** jaypipes has quit IRC | 23:32 | |
| clarkb | mtreinish: when I got the tests working it wsa definitely sing the public net | 23:33 |
| clarkb | which was why that got bridged to the hypervisors | 23:33 |
| clarkb | anyways I left a +1 and explanation for why I think it is correct | 23:33 |
| *** sridhar_ram1 has joined #openstack-infra | 23:33 | |
| * mordred agrees | 23:33 | |
| mtreinish | clarkb: well I think it was the floating ip thing. It was creating a floating ip and using that to ssh into the guest until sdague switched it back to using the fixed network for n-net jobs | 23:34 |
| mordred | so... | 23:34 |
| mordred | I'm going to try to say this without it turning into a floating ip rant | 23:35 |
| mordred | but fixed_network_name $PUBLIC_NETWORK_NAME seems wrong given the way we're deploying devstack | 23:35 |
| notmorgan | lifeless: re https://review.openstack.org/#/c/270417 - i commented and would rather push to land the devstack change at this point than the cap. | 23:35 |
| mordred | fixed_network in this cloud is the private network | 23:35 |
| *** sridhar_ram has quit IRC | 23:35 | |
| mordred | because the cloud follows the model of fixed private / floating public | 23:36 |
| mordred | now, I'd LOVE it if the default devstack deploy became fixed public provider net + optional floating and optional private | 23:36 |
| mordred | but unless something has changed that's just not the devstack deployment choice | 23:36 |
| clarkb | so thats doable to, if we did that we would need to bridge both the provider net and the fip net | 23:37 |
| mordred | it's the same net | 23:37 |
| mordred | in that cloud setup | 23:37 |
| mtreinish | mordred: I just did a find replace for s/PRIVATE_NETWORK_NAME/PUBLIC_NETWORK_NAME in that change that was probably wrong | 23:37 |
| *** xyang1 has quit IRC | 23:37 | |
| clarkb | oh you can do that? even better | 23:37 |
| mordred | there is one 'public' network | 23:37 |
| mtreinish | I didn't factor in neutron at all | 23:37 |
| mordred | you attach a vm to it and it behaves like we like from rax | 23:37 |
| mordred | but you can also create a network and a neutron router | 23:37 |
| notmorgan | lifeless: and i only say that since the requirements change just bounced off the gate. | 23:37 |
| clarkb | so neutron gets lucky in that they don't do the equivalent of mutlihost in most of their testing | 23:37 |
| mordred | and then you can also get floating ips from the same network | 23:37 |
| clarkb | and neutron agents automagically setup the backhaul for private nets | 23:38 |
| mtreinish | mordred: so I'm confused the issue we're hitting is only n-net. I haven't looked at how we set that up in devstack | 23:38 |
| *** dizquierdo has joined #openstack-infra | 23:38 | |
| lifeless | notmorgan: the main thing we get from that is the cap in requirements; we don't need to actually sync it into other project releases | 23:39 |
| clarkb | mtreinish: because neutron hsa extra smarts to make this work | 23:39 |
| lifeless | notmorgan: but, I'm easy. Push your stuff through :) | 23:39 |
| clarkb | mtreinish: specifically they set up their own overlay network automagically | 23:39 |
| Sam-I-Am | clarkb: yay, one neutron win :) | 23:39 |
| mordred | I'm confused that we still test nova-net - but then I'm weird | 23:39 |
| notmorgan | lifeless: right, both are net the same effect. | 23:39 |
| mtreinish | clarkb: sure, we also don't run any multinode gate jobs with neutron :) | 23:39 |
| clarkb | mtreinish: so you have that 10 net across all of them | 23:39 |
| Sam-I-Am | one of these days devstack will default to neutron | 23:39 |
| mordred | Sam-I-Am: maybe when almost all of the public openstack clouds use it - oh, wait - that day has long since come and gone | 23:40 |
| mtreinish | Sam-I-Am: bug sc68cal to finish the lib/neutron patch | 23:40 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add helper script to delete Gerrit groups https://review.openstack.org/270537 | 23:40 |
| Sam-I-Am | mordred: now they all use frankennets | 23:40 |
| mordred | Sam-I-Am: maybe when it's the most widely deployed network config? oh, wait, nope, that's also already true | 23:40 |
| *** aeng has joined #openstack-infra | 23:40 | |
| notmorgan | lifeless: :) since we have a devstack fix, might as well avoid churn in requirements. but tbh, i wouldn't be upset either way, just trying to avoid busy work of cap/uncap/cap/uncap :P | 23:40 |
| Sam-I-Am | mtreinish: we're about to pick that back up | 23:40 |
| *** achanda has joined #openstack-infra | 23:40 | |
| clarkb | mordred: well in fairness to devstack the devstack neutron stuff is/was really bad | 23:40 |
| lifeless | notmorgan: sure. Actually its a !=, not a cap. | 23:40 |
| clarkb | mordred: and was switched for about a day | 23:40 |
| *** hdd has joined #openstack-infra | 23:41 | |
| lifeless | notmorgan: but shrug :) | 23:41 |
| mtreinish | clarkb: it's still really bad :) | 23:41 |
| lifeless | notmorgan: I won't push it through | 23:41 |
| notmorgan | lifeless: well we wouldn't be reverting the underlying change | 23:41 |
| clarkb | mordred: and no one could get working networking on their devstacks | 23:41 |
| Sam-I-Am | clarkb: it could use better docs | 23:41 |
| clarkb | Sam-I-Am: thats part of it | 23:41 |
| lifeless | notmorgan: but if your stuff gets wedged, we do need to do something soon | 23:41 |
| *** ybathia has quit IRC | 23:41 | |
| Sam-I-Am | clarkb: docs would solve lots of problems | 23:41 |
| clarkb | Sam-I-Am: but even simple things like make sure my VMs can talk off host don't work right | 23:41 |
| notmorgan | right. it is just a question of getting votes on it and pushing it through | 23:41 |
| clarkb | Sam-I-Am: I know this beacuse I had a hell of a time figuring it out getting multinode + dvr t owork | 23:42 |
| notmorgan | either should get us clear. | 23:42 |
| Sam-I-Am | clarkb: with devstack or using the network guide? | 23:42 |
| clarkb | Sam-I-Am: devstack | 23:42 |
| clarkb | Sam-I-Am: we don't test with the netowrk guide | 23:42 |
| Sam-I-Am | oh, i thought you were just setting this up on your own somewhere | 23:42 |
| Sam-I-Am | crazy thoughts i know | 23:42 |
| clarkb | (the issue is that br-ex isn't attached to anything for you, so you have a completely detached networking) | 23:42 |
| clarkb | so what you have to do is manually configure br-ex before running devstack | 23:43 |
| clarkb | then it works | 23:43 |
| Sam-I-Am | the devstack instructions for dvr last i looked were for single-node, which was sort of lol | 23:43 |
| mordred | clarkb: right. I get that - but as long as nova-net is still there it's a crutch and causes nobody to fix it | 23:43 |
| mtreinish | Sam-I-Am: wait how is that d? | 23:43 |
| anteaya | okay so on our test nodes graph, the number of nodes in use have decreased | 23:43 |
| anteaya | anyone else see that? | 23:43 |
| clarkb | Sam-I-Am: well thats because when I got dvr working on multinode it didn't work :) | 23:43 |
| clarkb | Sam-I-Am: I think it is working now though | 23:43 |
| Sam-I-Am | mtreinish: lets not go there :) | 23:43 |
| Sam-I-Am | mtreinish: turns out no one ever tested dvr on > 1 node before it was released in juno... and i was the first person to figure out it didn't work with > 1 node. | 23:44 |
| *** gildub has quit IRC | 23:44 | |
| mtreinish | :( | 23:44 |
| Sam-I-Am | which soured a lot of people who were waiting a long time for a dvr thing | 23:44 |
| clarkb | and the multinode job confirmed it | 23:44 |
| clarkb | but it also gave a way for people to iterate on making it work | 23:44 |
| clarkb | so hopeully made a difference | 23:44 |
| Sam-I-Am | there's some adoption of it now, but its rather complex and fragile | 23:44 |
| Sam-I-Am | yeah, something that depends on muti-something needs to be tested with multi-something | 23:45 |
| Sam-I-Am | i was looking through devstack code the other day and realized its full of hacks that break what its trying to test. then i went to go cry in a corner. | 23:45 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 23:46 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add helper script to delete Gerrit groups https://review.openstack.org/270537 | 23:46 |
| Sam-I-Am | specifically, we put an ip address on br-ex and create the 'public' neutron net as vxlan, which should not even work | 23:46 |
| mtreinish | Sam-I-Am: to be fair I've had the lib/neutron rewrite marked as a prio item for 3 cycles | 23:46 |
| clarkb | I think if munging br-ex onto your actual interface became a thing then switching devstack to it would be much easier for hte masses | 23:46 |
| Sam-I-Am | but it does, because we still use external_network_bridge = br-ex, so that stuffs qg-* into br-ex and effectively provides an alternate path for traffic | 23:47 |
| *** aysyd has quit IRC | 23:47 | |
| clarkb | Sam-I-Am: uh devstack doesn't touch br-ex by default | 23:47 |
| Sam-I-Am | clarkb: neutron-legacy does it, iirc | 23:47 |
| clarkb | no | 23:47 |
| sc68cal | it does | 23:47 |
| clarkb | it may create one but thats it | 23:47 |
| sc68cal | if OVS_PHYSICAL_BRIDGE=br-ex | 23:47 |
| *** mriedem has joined #openstack-infra | 23:47 | |
| *** dizquierdo_ has joined #openstack-infra | 23:48 | |
| clarkb | right but al lthat does is make a br-ex iirc | 23:48 |
| Sam-I-Am | if you dont set use_providernet (or whatever it is), an ip winds up on br-ex | 23:48 |
| clarkb | it won't attach your eth0 to it | 23:48 |
| *** Sukhdev has joined #openstack-infra | 23:48 | |
| sc68cal | but again this is back to the mismatch between what br-ex means in a deployment vs. devstack-gate | 23:48 |
| clarkb | it won't move the ips around to keep networking working | 23:48 |
| sc68cal | clarkb: oh yes it will | 23:48 |
| Sam-I-Am | oh, there's no interface attached to it | 23:48 |
| *** dizquierdo has quit IRC | 23:48 | |
| clarkb | sc68cal: huh thats new in the last 9 months or so then | 23:48 |
| sc68cal | clarkb: yep. | 23:48 |
| Sam-I-Am | but there's an ip, which makes sense if you want to test connectivity to vms that would have fips on that network | 23:48 |
| clarkb | Sam-I-Am: ah ok | 23:48 |
| Sam-I-Am | otherwise its just layer2, and tempest cant see into it | 23:48 |
| clarkb | because the interface is the important part | 23:48 |
| sc68cal | clarkb: you have to have OVS_PHYSICAL_BRIDGE set, and I think PUBLIC_INTERFACE and a couple other knobs | 23:49 |
| clarkb | without it everyone complains their devstack doesn't work | 23:49 |
| sc68cal | because I hate people and want to make neutron-lib impossible to understand and configure | 23:49 |
| sc68cal | err | 23:49 |
| sc68cal | neutron-legacy | 23:49 |
| Sam-I-Am | clarkb: for local devstack aios, you can put an ip on br-ex thats the same as your neutron public subnet, OR plug an interface into it. | 23:49 |
| clarkb | anyways 9 months ago when we built out the working mutlinode stuff we just made br-ex the cross node bridge | 23:50 |
| Sam-I-Am | for multinode you'd need an interface in br-ex | 23:50 |
| clarkb | and everything worked | 23:50 |
| clarkb | Sam-I-Am: riht the prolem when devstack defaulted t neutron was that none f that is done for you | 23:50 |
| clarkb | so all of a sudden none of your VMs can talk to the world | 23:50 |
| clarkb | which is a thing people had come to expect from nova net which isn't shy about munging your interfaces | 23:50 |
| Sam-I-Am | kind of curious whats going to happen when external_network_bridge goes away | 23:51 |
| Sam-I-Am | its been deprecated and should = '' now rather than br-ex | 23:51 |
| Sam-I-Am | i suspect things will break | 23:51 |
| clarkb | (and nova net does that not devstack which is probably part of the confusion | 23:52 |
| clarkb | Sam-I-Am: what do you use instead? | 23:52 |
| Sam-I-Am | for now, 'external_network_bridge = ' ... when thats gone, it should more or less emulate that setting. | 23:53 |
| Sam-I-Am | what we have no relies on it being set to br-ex | 23:54 |
| clarkb | gotcha | 23:54 |
| clarkb | it should be arbitrary | 23:54 |
| clarkb | but is not yet | 23:54 |
| *** kzaitsev_mb has quit IRC | 23:55 | |
| openstackgerrit | Ian Cordasco proposed openstack/requirements: Update flake8 constraint for the off-by-default feature https://review.openstack.org/270541 | 23:55 |
| Sam-I-Am | what we need to do is what 'use_providernet' does, more or less, but also make sure br-ex gets an IP address | 23:55 |
| fungi | what have we merged to zuul recentlyish that would cause what looks like a job queue stats reporting counting error? it looks like we get this somewhat steady climbing envelope which abruptly resets on config reload | 23:55 |
| Sam-I-Am | if you use use_providernet now, br-ex does not get an ip and breaks tempest | 23:55 |
| openstackgerrit | Ian Cordasco proposed openstack/requirements: Update flake8 constraint for the off-by-default feature https://review.openstack.org/270541 | 23:55 |
| clarkb | fungi: its nodepool | 23:55 |
| clarkb | or was I think | 23:56 |
| adam_g | greghaynes, is this news to you? seems to be showing up in other jobs that use DIB https://bugs.launchpad.net/diskimage-builder/+bug/1536410 | 23:56 |
| openstack | Launchpad bug 1536410 in diskimage-builder "get-pip.py failing, using an outdated locatino for get-pip.py.script" [Undecided,New] | 23:56 |
| *** ddieterly has quit IRC | 23:56 | |
| greghaynes | adam_g: fun | 23:56 |
| greghaynes | adam_g: new to me | 23:56 |
| adam_g | greghaynes, from looking at DIB it looks like it should be pulling it from the right place, but ive got no idea why its caching it from github instead? | 23:57 |
| greghaynes | adam_g: probably a cache on the build host | 23:57 |
| docaedo | For using Shade, are there any docs outside what can be found in the shade repository? | 23:57 |
| openstackgerrit | Ian Cordasco proposed openstack-dev/hacking: Update flake8 requirement for 0.11.0 https://review.openstack.org/270544 | 23:57 |
| greghaynes | adam_g: we might not invalidate the cache based on url | 23:57 |
| nibalizer | docaedo: what are you trying to do with it? | 23:57 |
| jeblair | nibalizer, crinkle, fungi: i have a secret i want to put in hiera that needs to end up as a binary file on disk. any suggestions? | 23:58 |
| docaedo | nibalizer: App Developer Working Group is trying to use it as part of a "writing your first app on OpenStack" | 23:58 |
| fungi | docaedo: i think the first app crowd have been writing their tutorials around shade... reed may have details? | 23:58 |
| adam_g | greghaynes, i forget how the dib caching works, but its actually going and downloading the thing | 23:58 |
| fungi | docaedo: oh, that's why you're asking. nevermind! | 23:59 |
| clarkb | dougwig: why are we abandoning https://review.openstack.org/#/c/269323/ ? | 23:59 |
| docaedo | fungi: haha, yeah, trying to help on that front :) | 23:59 |
| greghaynes | adam_g: oh, is that from a run on an old release? | 23:59 |
| nibalizer | jeblair: there is a hiera-file backend that lets you store files in hiera | 23:59 |
| adam_g | greghaynes, nope, 1.7.1 | 23:59 |
| clarkb | dougwig: the proposal bot will push new patchsets to an existing change, it is safe to ignore it | 23:59 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!