Wednesday, 2013-08-07

clarkbnati_ueno: markmcclain was working that one by fixing the problem in novaclient iirc00:00
nati_uenoclarkb: so mark was working on that to update remove cap for nova00:00
nati_uenoclarkb: yes that's one00:00
clarkbya, not sure if there was a bug for that00:00
lifelessjeblair: if you want to avoid a new C binary, we could use lmirror, which is python and has a wsgi server that streams quite efficiently.00:00
clarkbfungi may know as he was involved with the requests pinning00:00
lifelessjeblair: (just jumping back to one of your other comments)00:00
lifelessjeblair: that would run over https too and that might avoid concerns about folk mitming the content)00:00
nati_uenoclarkb: Thanks. fungi: do you know something?00:01
jeblairlifeless: disregarding my concerns (which are more around risk/reward, and i'm coming over to understanding the reward part), which do you think would be better?00:02
fungiclarkb: nati_ueno: the reason we had originally pinned requests in nova is that nova was tested with site-packages enabled, nova depended on another openstack project which depended on requests. requests 1.2.1 pulled in sphinx as a new dependency which in turn depended on something else which depended on a newer jinja2 than the globally-installed rpm on centos 6.400:02
*** CaptTofu has quit IRC00:02
*** CaptTofu has joined #openstack-infra00:03
fungithat's as "briefly" as i can summarize the original issue00:03
clarkbfungi: that sentence is full of ugh00:03
fungiswitching tests to pip install -U dependencies solved this because we started pulling in a sufficiently new jinja2 in the virtualenv and were able to stop capping requests in nova as a result00:04
*** CaptTofu has quit IRC00:04
funginova never directly depended on requests anyway00:05
fungior hadn't prior to sticking it in as a workaround for that specific situation00:05
*** CaptTofu has joined #openstack-infra00:05
nati_uenofungi: thanks. Do you have bug report for this issue? I wanna know it is solved or not00:05
jeblairlifeless: yes, we can certainly revisit that decision, and i think you make a compelling case that the tripleo program has a real general need for this.00:06
funginati_ueno: there's a chain from the requirements cap revert to the nova cap revert to the original patch which closed the bug. i'll hunt it down00:06
jeblairlifeless: i'm now on board with running rsync or lmirror as appropriate, and segregating the hosts from static.o.o as needed for improved security.00:07
nati_uenofungi: Thanks. so is the change finished? I see mark's patch is merged00:07
clarkbnati_ueno: which patch merged?00:08
jeblairlifeless: so i'd just want to make sure the other coremudgeons agree with the scope expansion, though it seems like they have been.  :)00:08
clarkbI think one thing rsync has going for it is the puppet module... if we start using https on static I can see lmirror being useful00:09
funginati_ueno: it00:09
funginati_ueno: it's bug 118227100:09
uvirtbotLaunchpad bug 1182271 in nova "Nova unit tests fail on CentOS 6 when python-jinja2 package is installed" [Medium,In progress]
jeblairthis does dovetail into the earlier https for pypi.o.o discussion00:09
nati_uenofungi: Thansk!00:09
*** dina_belova has joined #openstack-infra00:09
locke105pip forces SSL?00:11
nati_uenoclarkb: sorry my mistake. It looks it is not merged yet
locke105(sorry was reading HTTPS discussion)00:11
clarkblocke105: yes, after it became an internet meme that pip was insecure (I think everyone sort of knew about this before hand...) dstufft and I assume the other pypi/pip folks worked to fix it00:12
clarkblocke105: it did other bad things like not enfoce md5sum checks00:12
*** CaptTofu has quit IRC00:12
locke105my internal "mirror" doesn't use that...00:12
locke105HTTPS that is00:12
clarkblocke105: s/pip/pypi/00:13
*** CaptTofu has joined #openstack-infra00:13
locke105yeah that makes sense00:13
locke105my squid busted after they did that00:13
locke105which is why i am rebuilding a full internal mirror instead of some packages00:13
fungipip doesn't enforce https, pypi now redirects from http to https as of a month or so00:13
fungiahh, right00:13
clarkbjeblair: I think there is value in https everywhere. We could create our own CA so that nodes trust each other properly while still self signing00:13
* locke105 hates dealing with certs00:14
jeblairclarkb: i am in favor of https everywhere; i'm less in favor of filing expense reports for buying certs for transient servers with numbers in their names. :/00:14
fungiclarkb: mordred already suggested that once today. having managed certificate authorities before, i'm glad you two are volunteering to run this one instead ;)00:14
jeblairthus the current 'https most places'.00:14
*** dina_belova has quit IRC00:14
clarkbfungi: oh man I raelly don't want to manage a CA >_>00:15
clarkbbut it is a possibility00:15
locke105although the nova-cert service theorectically turns most of it into an API00:15
locke105its kind of cool00:15
jeblairif we expand the scope of pypi as lifeless is suggesting, that increases my desire for it to have a ca-signed cert00:15
fungithe mechanics of ca management aren't that bad actually. easy enough to script. doing proper due diligence to secure your ca key is another matter entirely00:16
locke105yeah thats true00:16
clarkbjeblair: I agree for pypi.o.o scope changing wants a proper cert00:16
locke105what exactly is the scope change?00:17
*** weshay has joined #openstack-infra00:17
*** emagana has joined #openstack-infra00:17
clarkblocke105: providing this as a service to downstreams outside of our infrastructure00:17
clarkblocke105: so that people doing openstack dev/deployments/whatever can use our mirror instead of pypi00:18
locke105i probably would still need to maintain an internal mirror for bandwidth reasons00:18
ianwharlowja: do you use the -p command on ever step?00:18
harlowjanah, its actually cached :)00:18
*** reed has quit IRC00:18
harlowjajust the first one00:18
jeblairlocke105: lifeless wants to make that easier for you by supporting rsync or something like it.00:18
locke105yeah that would kick ass00:18
harlowjaianw /etc/anvil/passwords.cfg  , yes its not secure, but oh well, ha00:18
clarkbfungi: jeblair: does /refs/heads/* not match /regs/heads/stable/* ?00:19
harlowjasorry i mean /etc/anvil/settings.yaml , the passwords should be somewhat secure, ha00:19
fungiclarkb: /regs/heads/stable/* is a more specific match and is present in all-projects so needs to be overridden there00:19
fungier, refs00:20
jeblairclarkb: the exclusive on global refs/heads/stable matches more closely, so it needs an even more closer match (project is closer than all-projects)00:20
ianwhmm, ok, install step is failing to find openstack-cinder,cinder==2013.2.a154.g8aac388, openstack-glance,glance==2013.2.a54.g463f322, ... and pretty much everything else00:20
jeblairwhat fungi said :)00:20
*** salv-orlando has quit IRC00:20
fungijeblair said it better00:20
clarkbI see, /me looks at all projects00:20
harlowjaianw u did, ./smithy -a prepare, ./smithy -a build, ./smithy -a install right00:21
harlowjano skipping :-p00:21
*** nijaba has quit IRC00:21
locke105i have noticed that with my mirror lagging behind on reqs sometimes, we catch a lot of bugs00:21
ianwno skipping :)00:21
fungiclarkb: it's our blanket rule to grant the stable release managers exclusive control to those branches00:21
jeblairclarkb: there ought to be similar existing samples00:21
harlowja$ ls /etc/yum.repos.d/anvil*00:21
jeblairharlowja: wow that's really close to "smitty" and i was having aix flashbacks00:21
clarkbjeblair: there are. It just occurred to me that it shouldn't be necessary but the blanket rule explains it00:21
harlowjaianw ^ ?00:21
*** nijaba has joined #openstack-infra00:22
harlowjasmithy is anvils blacksmith jeblair , ha00:22
*** derekh has quit IRC00:22
ianwharlowja:  not found ... so hmm00:22
fungijeblair: good flashbacks! filesystem/block device management with smitty was great00:22
lifelessjeblair: sorry, ELOCAL.00:22
harlowjahmmmm, ianw wanna sudo rm -rf ~/openstack then do the steps again00:23
harlowjathose 2 repo files should exist after the build step00:23
ianwharlowja: hang on, i think i *did* forget build00:23
harlowjathe install step actually verifies that all packages it wants to install are there, its the late catch anything missing thing00:23
ianwyeah, sorry, lost track00:24
openstackgerritA change was merged to openstack-infra/config: Allow puppet-manager-core to review stable branches
harlowjanp :)00:24
lifelessjeblair: uhm, so I think the set of functional requirements we want: - streaming; no content alteration by MITMers; multi-concurrent-writer-safe.00:24
locke105smitty can die in a fire along with AIX and its crappy network device model00:24
lifelessjeblair: I designed lmirror for single writer millions-of-files, it's not a great fit here - only mentioned it due to your C daemon comment.00:25
harlowjaianw there was a time ago that anvil would run the dependent operations, but it was just split into these steps, for better or worse :-p00:25
harlowjatime long ago, ha00:25
ianwsmithy --help giving an overview of the steps might be good00:26
ianwi think i forgot because i had the rst of the docs in less00:26
ianwand it all looks the same :)00:26
harlowjabut ya smithy --help would make sense00:27
jeblairhrm, just thought of something else...00:28
jeblairsupporting rsync makes it more difficult to store the mirror in object storage (and front with, eg a wsgi app), if we wanted to go that way.00:28
lifelessjeblair: so I think stunnel in inetd rather than rsyncd running; use rsync -e stunnel ... to connect to the rsync server.00:29
clarkbjeblair: it does00:29
lifelessjeblair: it does, thats true; how many thousand files is the mirror?00:29
jeblairbut if we did that, we could more easily provide the (accurate) index files lifeless originally wanted00:29
lifelessjeblair: so I only went to index files after talking to clarkb here.00:29
lifelessjeblair: rsync was my first thought, I think :)00:30
jeblairfind -type f |wc -l00:30
clarkbI didn't like rsync without more discussion for the reasons jeblair describes00:30
lifelessjeblair: teeny :)00:30
jeblairlifeless: expact that to quintuple due to added branches.00:30
clarkbI did however think hosting a few more files would be fine ...00:30
jeblairbut yes, still teeny.00:30
lifelessjeblair: so we could always stash that in object storage as well, just a cron job on the host to sync regularly; the size is small enough that that wouldn't have scaling/overhead issues.00:31
*** zul has quit IRC00:32
clarkbI am going to walk home now. Back later00:32
jeblairlifeless: i think if we used object storage, we wouldn't want to support rsync because we'd need the whole tree locally; we'd want to use object storage exclusive of local storage (except for caching)00:32
lifelessjeblair: I'm saying that the 'whole tree locally' only becomes an issue at scale; at a scale we aren't at.00:33
clarkbcrazy idea before I go: I believe swift allows you to replicate containers. Might be possible to do something with that?00:33
lifelessjeblair: [and have no reason to think we'll reach for a long time]00:33
jeblairlifeless: well, its absence will be an issue.00:33
lifelessjeblair: I don't follow00:33
clarkbreplicate to lifeless' container and have him do what he wants? it will be an auth nightmware though00:33
jeblairbecause i really don't want to have an authoritative copy on object storage and one locally.  this is supposed to make things simpler.00:34
lifelessjeblair: in that if the scale is small enough one can make a local tree in cron on a cloud server00:34
Alex_Gaynormordred: man, today is just bringing first class bummers, nonstop00:35
jeblairlifeless: yes, and rsync is inaccurate until the cron runs; it may be empty when the server spins up, and different servers may return different values.00:35
lifelessjeblair: I see; I presumed we didn't put things into rotation until they'd set themselves up.00:36
*** sandywalsh has quit IRC00:36
jeblairlifeless: having dumber application servers that rely entirely on cloud services is a direction we'd like to head.  it's going to take a while to get there, but some of the static content servers are serious candidates.  logs or talballs may be first.  pypi could be on the list.00:36
lifelessjeblair: ack on the skew between cron window thing, but the empty dir case is much more of an issue00:36
lifelessjeblair: there's no in-principle reason that swift can't support rsync. Heck, S3 supports bittorrent.00:37
jeblairlifeless: we haven't engineered the system yet, but i'd certainly want to keep it as simple as possible.00:38
locke105i think the problem is that we want to rsync a tree of objects00:38
locke105not just the objects00:38
locke105swift uses/used rsync for object sync already00:39
locke105the problem we have is the one git solves00:39
clarkbjeblair ++00:39
locke105only in git, the backing store is gzipped files...00:40
jeblairi have to run now00:40
fungithere's an idea... publish the pypi mirror via bt00:41
locke105i'm not sure how that would work with a backing object store and wsgi frontend still...00:43
locke105plus if the file data changes you need to build a new torrent00:44
locke105where did the object store requirement come from?00:46
lifelessfungi: I would be happy with that too :)00:46
fungilocke105: the object store benefit is something we've discussed in the past. being able to turn up a new server and just point it at the data in a provider rather than needing to restore copies from backup when replacing one00:48
dkehnclarkb, well the devstack with neutron looks worthless again00:48
locke105so use the list of objects in a container as your index00:48
locke105and the objects are the tarballs or zips etc00:48
locke105yeah i like that00:48
harlowjaianw let me know how it goes :)00:49
harlowjahopefully ok this time, ha00:49
fungibut that's a long-term possibility for our static and semi-static content00:49
locke105i believe that is how works00:49
locke105serves out of a cdn with wsgi00:49
funginear term we're still serving off filesystems on local block devices00:49
fungimainly because we haven't had time to reimplement something browseable like apache's autoindex00:50
locke105right you need something to do the translation from object store to file-like system00:50
locke105can't imagine it would be too hard to hack up00:51
fungiit's not a super-complex problem, but we have plenty of other places we need to spend our time improving higher priority things00:51
locke105yeah i have the same issue :(00:51
locke105hosting the current file set with git or rsync would be easiest00:53
fungihuh, git's a possibility i hadn't considered. it's not great for larger binary blobs though so it would need to be something more like bup00:54
locke105yeah you guys said mirror + rsync and my mind jumped to the Gentoo portage tree00:54
locke105which uses rsync to mirror around00:55
locke105they have been trying to move it to Git for some time...00:55
locke105thats a lot of tiny files00:55
locke105not binaries00:55
locke105git would work well, since it would only grab the latest set in the tree00:56
locke105if you did shallow clones00:56
locke105but at the same time we don't want to keep the large binaries in history00:56
fungithat's basically what bup does, but it's not plain git (so that it can optimize for those issues)00:57
locke105never heard of bup... looking into it now...00:58
fungiwe're in the process of rolling it out for server backups00:58
fungiit's a very neat implementation00:58
locke105It uses a rolling checksum algorithm (similar to rsync) to split large files into chunks. The most useful result of this is you can backup huge virtual machine (VM) disk images, databases, and XML files incrementally, even though they're typically all in one huge file, and not use tons of disk space for multiple versions.00:59
locke105first bullet point and i'm sold00:59
lifelesslocke105: the pypi CDN is backed by a regular server with a regular filesystem.00:59
lifelesslocke105: it's just an aggressive HTTP cache in front of everything.00:59
locke105the one?00:59
locke105not python.org00:59
lifelessoh, sorry, not sure.01:00
lifelessbut IIUC dstufft is moving the one onto the same codebase.01:00
lifelessso, same same.01:00
dstufftit's a whole new codebase01:00
dstufftthe codebase sucks01:00
dstufftfirst systems etc01:01
dstufftI've learned a ton since I wrote it01:01
dstufftlots of mistakes01:01
fungiproof of concept ;)01:01
lifelessfungi: jeblair: so, I'd be happy with a few-minutes-delayed rsyncable tree.01:01
dstufftthe PyPI CDN is basically just varnish01:01
fungilike when you're making pancakes, the first one never looks as good as the rest01:01
lifelessall the systems that need latest-and-greatest could happily run off an object storage container.01:01
lifelessBoth would be eventually consistent given the cache properties of object stores.01:02
lifelessso - my suggestion is stunnel in inetd wired up to invoke rsync - getting us end to end encryption.01:02
locke105tbh my mirror lags behind further than a few minutes (more like days) and its fine01:02
lifelessI don't know if the rsync module from puppet labs does that.01:02
dstufftBut yea Crate stores it's files in S3 (and PyPI will eventually too)01:03
dstufftor rather not S3 for PyPI, but an object store01:03
clarkbdstufft: are you going to host your own?01:03
dstufftclarkb: probably, our infra lead doesn't like depending on external things if we can easily host it ourselves01:03
fungilifeless: we could wire up stunnel to rsync fairly easily on a second port01:03
lifelessFWIW lmirror could happily back onto S3 - it has a simple file based journal that clients can interpret; you could run the server against it.01:03
lifelessfungi: or that, whatever you prefer.01:04
dstufftclarkb: We've talked about using Ceph or Swift, and we want to switch to OpenStack for our ifnra anyways01:04
dstufftright now using Ganeti because OSUOL recommended it when we setup there01:05
fungidstufft: simultaneously stick copies in several swift providers and that way you can weather outages of a provider01:05
clarkbdstufft: ya its what Lance prefers01:05
clarkbganeti is nice and simple01:05
Alex_Gaynorfungi: "why would a highly available object store ever go down" (snrk)01:05
* locke105 googles ganeite01:06
dstufftI don't have access to Ganeti so I don't know anything about it01:06
fungiAlex_Gaynor: i have seen what happens when a tornado cuts a top-notch data center in half01:06
* locke105 tries again with correct spelling...01:06
clarkbunlike openstack it is highly opinionated about what your setup should look like01:06
dstufftother than it doesn't support LXC and we want to use LXC for isolation01:06
clarkbiirc KVM only with only local storage for VM images01:06
Alex_Gaynorfungi: I'm curious how swifts new global replication would weather a tornado01:07
clarkbthat makes it simple to setup configure etc01:07
dtroyerganeti is great for smallish relatively static clusters…with live drbd-based failover01:07
*** sandywalsh has joined #openstack-infra01:07
fungiAlex_Gaynor: yes, globally distributed in a single provider is almost as good as distributing across multiple providers01:07
clarkbdstufft: I am guessing the fact that a majority of their employees are students helps influence those types of decisions01:07
harlowjaianw :)01:08
*** weshay has quit IRC01:09
dstufftclarkb: probably yea, There was some misconceptions about how much OSOUL was going to actively do so we'll likely be moving to something our members have more experience with and know people to complain at when things break ;)01:10
dstufftplus written in python!01:10
*** dina_belova has joined #openstack-infra01:10
dtroyerganeti is python01:10
locke105looks like a lot of .hs files...01:12
locke105and .pys01:12
dstufftdtroyer: welp I refer to the above I don't know anything about it other than it doesn't support LXC :V01:13
dtroyerdstufft: no, and the fact that it doesn't use libvirt makes that a more difficult problem to fix.  but it was rock solid for me for a couple of years running basic infra stuff.  it solves a basic virt manager problem, not clouds01:15
*** dina_belova has quit IRC01:15
clarkbdtroyer: exactly01:15
clarkbit is actually a pretty awesome tool, just different01:15
clarkbone of the OSUOSLers wrote the web front end for it during an internship at google iirc. which is how it first ended up in their infra iirc01:16
dstufftYea I'm saying it's bad I don't have any data to claim that :) I just know we're looking at moving to OpenStack in general01:16
locke105tbh... who isn't looking at moving to openstack...01:17
*** mestery_ has joined #openstack-infra01:17
locke105i have a meeting tomorrow with some internal labops guys that want to know how to set one up01:18
fungii always keep an extra openstack in my back pocket. you never know when you might need one01:19
*** mestery has quit IRC01:19
locke105fungi: ++ i have around 3 or 4 now...01:19
locke105they tend to be pretty popular and fill up fast :)01:20
locke105i wish i could get stats of ubuntu/rhel installs with openstack vs ESXi in our labs and how they have changed over the past year01:21
*** nijaba has quit IRC01:22
*** nijaba has joined #openstack-infra01:22
*** nijaba has quit IRC01:22
*** nijaba has joined #openstack-infra01:22
clarkblocke105: glance/nova doesn't keep track?01:25
locke105clarkb: idk. how?01:26
clarkblocke105: I am guessing logs01:27
clarkbwhich you could then process and spit numbers out to graphite with01:28
locke105the lab hardware is either xcat provisioned or manually provisioned (yes sad panda i know)01:28
locke105thats what i would be interested in getting stats off of01:28
locke105i could probably whip up some xcat magic01:29
fungibtw, reinstalling python-pip on all the centos servers seems not to have rebroken after the puppet update circulated01:30
fungithough pbx and centos6-dev1 have separate issues on top of that so they're still erroring01:30
jog0clarkb: where is the code for the gerrit irc bot?01:30
jog0I wantto make it less wordy01:31
fungijog0: openstack-infra/gerritbot01:31
jog0fungi: thanks01:31
clarkbjog0: less wordy?01:31
clarkbjog0: its already only one message per announcement01:31
clarkbwhcih is what matters in IRC01:31
jog0I have word wrap :(01:31
jog0as do others I bet01:31
clarkbI do... doesn't bother me01:32
jog0clarkb: I find it breaks up conversations sometimes01:33
clarkbmaybe a client thing?01:33
jog0clarkb:  I was thinking instead of %s propsed a change to %s ...01:34
jog0just listing everything01:34
*** gyee has quit IRC01:35
jog0mayvbe %s proposed %s ...01:35
*** yaguang has joined #openstack-infra01:35
clarkbjog0: I am not really picky. The current behavior hasn't bothered me even on my phone... So I am open to whatever you think will help01:36
*** markmcclain has quit IRC01:37
fungiit will be an invitation for bikeshed painters01:37
jog0lets paint it red01:38
clarkbmy gut is telling me it is the wrong kind of optimization01:38
clarkbbecause it doesn't matter from IRC's perspective01:38
locke105+1 for red01:38
clarkbbut because I feel that it doesn't matter I will withhold the -1s :)01:38
* fungi left his paintbrush in his other pants01:38
*** pcrews has quit IRC01:39
clarkbas long as you are under 512 bytes it is good :)01:39
openstackgerritJoe Gordon proposed a change to openstack-infra/gerritbot: Make proposed and merged messages more terse
jog0nice and wordy ^01:39
jog0the nova room has been empty lately and i think gerrit bot is to blame01:40
jog0thats just my hunch01:40
clarkbjog0: you can turn it off in the channel too if you think it might help01:40
clarkbwe are still spitting the same info to openstack-dev iirc01:40
jog0clarkb: this seemed like a less drastic first step01:40
jog0clarkb: I already ignore openstack-dev because of gerritbot though :)01:40
fungi-dev may only be getting merge comments01:40
fungiwe actually make use of the gerritbot comments in here, and to some degree they drive additional discussion01:41
fungii would miss them01:41
clarkbjog0: so the proper way to deal with this in IRC is to set up filters/ignores01:41
jog0fungi: sure but this is a much smaller team,01:41
clarkbdon't not sit in a channel because a bot is chatty, just ignore the bot01:42
jog0clarkb: perhaps, but right around when we started using gerritbot in openstack-nova I think it became less used01:42
fungibut a channel of 162 even if most lurk, and we probably get at least as many patches across our collection of projects as nova+novaclient01:42
jog0fungi: how many patches a day do you get?01:43
clarkbfungi: I think jeblair counted once and we are about half of nova01:43
clarkbfungi: which is chattier than any other project iirc01:43
fungiahh, okay, so not quite as high as i though01:43
clarkbso not nova bad but not quiet either01:43
*** changbl_ has joined #openstack-infra01:44
fungiit's within the same order of magnitude ;)01:44
jog0either way I don't see why the bot needs to be so verbose.  I do see value in the bot, figure this would be a good safe experiment01:44
*** zaro has quit IRC01:44
*** gyee has joined #openstack-infra01:44
*** yaguang has quit IRC01:45
jog0 before gerritbot I wired up growl (on OS X) to do the same thing01:45
*** yaguang has joined #openstack-infra01:45
*** zaro has joined #openstack-infra01:46
*** adalbas has quit IRC01:47
*** zaro0508 has joined #openstack-infra01:52
*** nati_uen_ has joined #openstack-infra01:54
*** nati_ueno has quit IRC01:56
*** gyee has quit IRC01:57
*** davidhadas has quit IRC01:57
*** davidhadas has joined #openstack-infra01:58
*** thomasm has joined #openstack-infra01:59
*** anteaya has quit IRC02:04
openstackgerritJeremy Stanley proposed a change to openstack-infra/gitdm: Add to foundation group file
*** enikanorov_ has quit IRC02:06
*** soren has quit IRC02:10
*** soren has joined #openstack-infra02:10
*** prad has joined #openstack-infra02:10
*** dina_belova has joined #openstack-infra02:10
*** dina_belova has quit IRC02:15
*** Ryan_Lane has joined #openstack-infra02:16
openstackgerritSean Dague proposed a change to openstack/requirements: Raise warlock requirement
openstackgerritSean Dague proposed a change to openstack/requirements: Raise eventlet to 0.13.0
openstackgerritSean Dague proposed a change to openstack/requirements: Add pysnmp needed for Ceilometer/HardwareAgent/SNMPInspector
openstackgerritSean Dague proposed a change to openstack/requirements: Bump pbr requirment to 0.5.21
*** Ryan_Lane has quit IRC02:19
*** emagana has quit IRC02:20
*** Ryan_Lane has joined #openstack-infra02:21
*** nijaba has quit IRC02:21
*** nijaba has joined #openstack-infra02:22
*** nijaba has quit IRC02:22
*** nijaba has joined #openstack-infra02:22
openstackgerritSean Dague proposed a change to openstack/requirements: Update keyring minimum version
*** Ryan_Lane has quit IRC02:30
openstackgerritSean Dague proposed a change to openstack/requirements: remove netifaces as a requirement
openstackgerritSean Dague proposed a change to openstack/requirements: Raise psutil requirement
Alex_GaynorIs there a script to add the apache header to files?02:33
clarkbAlex_Gaynor: I don't think so02:34
clarkbhacking does check that it exists in files longer than 10 lines though02:34
* Alex_Gaynor gets his copy-paste hat on02:34
* locke105 is not aware of any script that does that either02:34
clarkbfor file in `git status --porcelain | grep ^(A02:36
* clarkb tries again02:36
fungigrr... non-py3k-compatible global requirements are preventing us from building a pypi mirror on mirror33:
locke105i've tried to :wq my irc so many times its not funny02:39
clarkbfor file in `git status --porcelain | grep ^(A|\?\?) | sed -n 's/(A|\?\?)\(.*\)/\1/p'` do ; cp $HEADER_FILE $TMPFILE && cat $file >> $TMPFILE && mv $TMPFILE $file ; done02:39
clarkbso uh I am sure the regexes in that are wrong, but you get the idea02:40
clarkblocke105: I recently switch bash to vi mode. It has been fun02:41
locke105clarkb: wut?02:41
clarkblocke105: best of all dd works02:41
*** emagana has joined #openstack-infra02:41
clarkband hitting v in command mode opens vim02:41
locke105thats freakin' awesome... must try...02:42
clarkblocke105: `set -o vi` iirc02:42
clarkblocke105: hjkl work too02:44
clarkbI haven't figured out the shortcut for clear though02:44
locke105wow awesome02:44
locke105except it opens nano for some reason on my ubuntu machine....02:44
locke105must have not set EDITOR02:45
locke105wow this is great02:45
*** sarob has joined #openstack-infra02:46
fungiupdate-alternatives --set editor /usr/bin/vim.basic02:46
clarkbit has definitely made not learning emacs things easy02:46
clarkbas I try random vi things and they just work :)02:47
locke105clarkb: you rock, thanks for the tip02:47
clarkbfungi: do you vi mode too?02:47
clarkbat times it feels half backed but I don't use a lot of the emacs stuff because it isn't familiar to me02:47
fungiclarkb: not previously... already know a lot of classic readline shortcuts anyway02:47
fungiassuming you mean vi mode in bash (as opposed to vi mode in emacs)02:48
locke105i got so excited i forgot what i was doing now02:48
clarkbfungi: also if doing emacs in vi mode you should use elvis because the name is awesome02:49
clarkbI really want an ex mode when you type :02:50
mordredfungi: we could land my "install what you can" patch02:50
clarkbmordred: your global venv change failed in the gate :(02:50
fungimordred: it does suddenly become much more compelling02:50
mordredclarkb: ossume!02:50
mordredfungi: see... I had reasons for everything... ;)02:50
clarkbfungi: mordred doesn't it already install what it can?02:51
clarkbI mean things blow up and it still copies the packages02:51
fungiclarkb: it gives up on the first exception02:52
mordredwell, it blows up at first error02:52
*** melwitt has quit IRC02:52
clarkbi see02:52
mordredwhich means we get predictably broken things02:52
clarkband that is a pip thing right?02:52
fungiand then copies however far it made it, but doesn't try to install anything else02:52
mordredthis has repeatedly been a problem for me in pbr gate02:52
* dstufft thinks it should just blow up if pip install fails, not silently error 02:52
*** emagana has quit IRC02:53
clarkbmordred: have you seen ?02:53
clarkbthis is going to be seriously dangerous for my food intake02:53
pleia2I use that + delivery.com02:55
mordredclarkb: I use seamless - but I'm in NY :)02:55
mordreddstufft: it does that already02:55
mordreddstufft: BUT02:55
clarkbpleia2: there is another one I have bene told about to use in SEA but I forget what it is02:55
mordreddstufft: if a pip blows up in teh woods and there is nobody to see it...02:56
clarkbpleia2: I am trying to cook more and amazon fresh helps on that front02:56
dstufftmordred: no I mean the job should fail if pip blows up02:56
mordreddstufft: right... but who is going to notice02:56
mordreddstufft: these are periodic jo bs02:56
mordredwhen it breaks, it regularly stays broken for weks02:56
clarkbdstufft: I thought so too, but we had this discussion and we want to the breakage to percolate downstream so that the projects fix it themselves02:56
mordredwhat I want is as much of the mirror mirrored as possible02:57
clarkbbecause swift is in a better position to make a decision about exploding xattr than I am02:57
mordredand I don't want a breakage in prettytable02:57
pleia2clarkb: no amazon fresh but we have here, can order from a few local grocery stores, very nice :)02:57
mordredto affect whether or not we can mirror eventlet02:57
mordreddstufft: it's possible we're not using all of our words thogh02:57
clarkbpleia2: the other problem I have is five point is a block and a half away02:57
mordreddstufft: the current process does a pip install -d . -r requirements.txt02:57
clarkbpleia2: so whenever I am too lazy to make somethign I can just walk over there and eat horrible^HHHHHHHH awesome greasy pub food02:58
mordreddstufft: to grab _all_ of the thigns that opentsack needs in the mirror02:58
mordreddstufft: and as soon as it can't download any of them, it stops trying02:58
pleia2clarkb: haha, doh :)02:58
*** amotoki has joined #openstack-infra02:58
clarkbfungi: have you been to five point?02:58
clarkbif not maybe we should beer there02:58
dstufftmordred: yea, and then it just moves onto the next pip install -d right?02:58
clarkbbecause it is awesome02:58
mordredthere is no next install -d02:58
fungiclarkb: i have not, but certainly want to02:58
dstufftmordred: isn't there multiple mirrors?02:59
mordredthere is a single, global, requirements list with every requirement needed by all of openstack02:59
dstufftgrizzly, havana etc02:59
mordrednot yet, no02:59
dstufftMaybe my brain is remembering it wrong02:59
fungiworking on that this week02:59
dstufftor I didn't understand the script well enough02:59
mordredit _also_ tries putting the other branches into that mirror02:59
mordredso yes, it will tryu those too02:59
mordredthey are all in alpah order03:00
fungisoon it will be putting those branches in separate mirrors03:00
clarkbwe have worked around it so far because the old versions of thigns worked with the stable branches03:00
mordredso it's likely that grizzly's version will fail in the same place03:00
clarkband we don't remove old packages03:00
mordredbut in any case- we wound up with a case where a thing early in the list was bombing on cffi03:00
dstufftmordred: I think installing each item with pip seperately is going to bite you fwiw but I think I've mentioned that before. But it's your pain if it does :)03:00
fungiin this case, boto does not play well with (py3k) others03:01
mordreddstufft: why?03:01
fungiFile "./boto/pyami/", line 18503:01
fungiprint s.getvalue()03:01
mordreddstufft: if we do each individually, will we not wind up with the things we asked for in the list, plus a myriad of other crap?03:02
fungimordred: dstufft: the main difference i've seen it cause is pulling in newer packages than what our requirements list caps at (in addition to the versions we do want)03:03
mordredfungi: I think I'd rather have that, tbh03:03
mordredas long as we do get the ones we aske for too03:03
clarkbmordred: for the reason that it did bite us03:03
fungiwhich shouldn't itself cause issues with our requirements sync anywa03:03
mordredbtw- clarkb was right03:03
clarkbI was right about something? what was I right about?03:04
clarkbmordred: the biting happens because pip doesn't resolve dependencies together. It isn't great at that but individual installs are even worse03:04
mordredclarkb: well, at this point, we are enforcing versions strongly through other means03:04
mordredso, as logn as the versions that are in the per-project lists are in the mirror03:04
dstufftmordred: Yea becasue of higher versions, and that if project A depends on Foo<=1.0, and project B depends on Bar, which Depends on Foo w/o a version spec03:04
mordredour installation of _those_ all in one go should get us the right hting03:04
dstufftthenw hen you install project B alone from the mirror you get a different set of versions thenw hen you install project A and Project B together03:05
mordredI do not think that's solvable by the m irror03:05
mordredwhich is what clarkb was saying originally03:05
mordredwhat Iwant the mirror to be now03:05
mordredis a thing that makes us not dependent on the state of an external network resrouces for the gate03:05
mordredand let version mnanagement happen in the projects03:06
mordredwhere order of install _is_ curated03:06
mordredso a large corpus of versions in the mirror is great03:06
*** dkliban has quit IRC03:06
mordredI don't need the mirror to protect me unnaturally from a situation that would occur in the wild03:06
clarkbmordred: that makes me so happy :)03:07
mordredclarkb: see! you were right!03:07
*** prad has quit IRC03:07
*** emagana has joined #openstack-infra03:07
mordredI disagreed with you before the current batch of requireents wrk03:07
clarkband I understand why03:07
mordredbecause we did _not_ protect strongly enough03:07
mordredI mean - yeek03:07
mordredwe still barely do03:07
clarkbthere was definitely the large step that needed to be made to get from here to there03:07
*** emagana has quit IRC03:07
dstufftmordred: You might have less pain not using pip to mirror then once you no longer have deps hosted externally to PyPI03:07
mordredyeah - less pain ... but still - if we depend on external resources03:08
mordredthen we're at your personal mercy03:08
mordredshould something go down03:08
dstufftno I mean03:08
mordrednot that I don't like you :)03:08
dstufftusing something other than pip03:08
mordredlike what?03:08
mordredEVERY other piece of software I've seen and tried is TERRIBLE03:08
mordredI mean UNWORKABLE03:09
dstufftmordred: You've tried bandersnatch?03:09
mordredand non functional03:09
mordredand doesn't do anything03:09
mordredomg. there's another one? /me goes to see in what ways it does not work...03:09
dstufftmordred: it's the new recommended mirroring client over pep381client03:09
fungipip and friends will figure out our dependency tree. bandersnatch will try to mirror all of pypi right?03:09
mordreddstufft: what does it mirror?03:10
clarkbI am not opposed to mirroring all of pypi if external links go away03:10
dstufftmordred: It'll mirror PyPI (hence the once you no longer have deps hosted externally)03:10
dstufftthe benefit is it'll be WAY faster03:10
clarkbthat avoids the branch problem if we restrict versions through some other mechanism (which was my original argument :) )03:10
dstufftwell the intial sync will suck03:10
dstufftbut after that you can just run the mirror in a cronjob once a minute or once every 5 minutes or so, and not have to rebuild the entire mirror03:11
dstufftYou'll get your own minature version of PyPI, which will only help you once all your deps are PyPI hosted only03:11
*** dina_belova has joined #openstack-infra03:11
mordreddo we know where we are on that?03:11
dstufftI haven't looked recently. I know some of them moved over though03:12
dstufftI can take a look though03:12
fungiit was previously suggested that we wanted to be able to identify and curate our transitive dependencies, and mirroring the entire dependency tree from our direct requirements list accomplishes that03:12
mordredfungi: right. I believe that's an out of date desire03:12
dstufftfungi: so does pip instaling into a venv and using pip freeze whenever you need to see the entire dep tree :)03:12
mordredwhich was a thing we wanted before we had strong reuqirements gating03:12
mordredhowever - _I_ like run-mirror because I can run it on my laptop03:13
mordredand get a usable mirror for when I'm on a plane :)03:13
mordredand that's why lifeless is wanting to add rsync support ot our mirror03:13
* mordred might spend a lot of time in planes03:14
dstufftmordred: I run bandersnatch on my laptop :D, takes up above 53GB though03:15
lifelessdstufft: mordred: so what I want is a /mini/ pyPI with just the transitive deps in it.03:15
lifelessbecause 53GB to install tripleo is bad.03:15
mordreddstufft: 57M/home/mordred/cache/pypi/03:16
lifeless100MB would be about right.03:16
mordred57M is the size03:16
*** dina_belova has quit IRC03:16
mordredand that's with old things in it, because I jus update it03:16
fungiso 3 orders of magnitude reduction03:16
dstufftpossible the author of bandersnatch would support partial mirroring03:16
dstufftjust an idea so your mirroring is way faster :)03:16
lifelessdstufft: rsync is widely available03:16
dstufftlifeless: you're not gonna get rsync from pypi03:16
lifelessdstufft: I'd rather use widely available tools even if they aren't domain specific.03:17
mordreddstufft: he will from our mirror03:17
lifelessdstufft: I don't care about pypi, just pypi.o.o.03:17
dstufftsure, I don't care what you do from your mirror, I'm saying to get things onto your mirror03:17
mordredah - well, in general, mirror speed _in_ to our mirror is not a problem03:18
mordredthat's the size of our curent mirror03:18
mordredlifeless: ^^03:18
lifelessmordred: thats tolerable03:18
mordredlifeless: just letting you know03:18
lifelessmordred: base ubuntu image is within margin-of-error of that.03:18
mordredlifeless: we have literally never deleted anything from it03:18
lifelessmordred: 250MB or whatever.03:18
fungii intend to clean that up with the branch-specific mirrors anyway, so we'll only carry 6 months of cruft per mirror once that's working03:18
lifelessfungi: whats this per-mirror thing?03:19
fungilifeless: separate mirror for stable branches of openstack/requirements03:19
lifelessfungi: do you mean the havana/grizzly/ etc stuff?03:19
mordredlifeless: yah03:19
lifelessfungi: I thought it was folded intothe same tree?03:19
lifelessfungi: so there is no redundancy?03:19
dstufftmordred: okies, well just an idea, bandersnatch is also smarter about not accepting old versions from the CDN and such but if you're rebuilding your mirror periodically that's less important because you just pay the cost overtime by needing to redo the entire mirroring operation03:19
*** mestery_ is now known as mestery03:19
mordredyeah. I mean, bandersnatch seems to be a MUCH better choice fora full mirror03:20
fungilifeless: originally it was folded together out of necessity, because we needed to see it in action before adding the necessary complexity. also we haven't been diverging requirements enforcement for new releases yet03:20
lifelessI don't see how bandersnatch could do partial for us anyhow, not unless I convince mordred and jeblair to list the transitive deps03:20
mordredthe problem is - there are still external deps03:20
mordredand transitive deps03:21
lifelessfungi: so, I think folding it together is a good thing.03:21
mordredso, bandersnatch is unlikely to grab either for a while03:21
mordredduring which time, we'll have ot continue to engineer our current thing03:21
lifelessfungi: don't need to download the same file twice..03:21
mordredfungi: honestly, I'd like to revisit per-pbranch mirrors with jeblair and you in themorning03:22
*** nijaba has quit IRC03:22
fungilifeless: well, that's a new possibility we've just started discussing... basically relying on comparison of requirements lists alone and not expecting the mirror scope to limit access to packages03:22
mordredfungi: I think it's a project we started before the requirements stuff changed03:22
mordredand I want to make sure it's still a rabbit we need03:22
lifelessfungi: right.03:22
*** nijaba has joined #openstack-infra03:23
fungimordred: i agree that it's worth reevaluating, but i'm not strongly invested in either position at the moment03:23
mordredin fact, the more I think about it, th emore I think it might be counter productive, because it won't tell us how our requirmeents lists fare in teh cold light of public pypi03:23
mordredfungi: me either - I'm only just now rethining my position03:23
*** nati_uen_ has quit IRC03:23
mordredfungi: speaking of - should I look at the rsync patch?03:24
fungithe last time the topic surfaced there was fairly strong dissent. but the requirements enforcement has lurched forward since03:24
fungimordred: you can look at the rsync patch if you like. i think jeblair is still unconvinced it's a safe move (or is maybe still on the fence)03:24
fungithe change to implement it is extremely trivial03:25
mordredhe didn't -1 though - just asked the question, which I tihnk its fair03:25
mordredI'm going to +2 but ping him in the morning before aprv03:25
fungiit got discussed heavily in here after that, i just haven't recapped it there yet03:26
mordredoh - did it?03:26
mordredtl;dr ?03:26
lifelesswe should add stunnel03:26
lifelessso we have ssl - prevent hostile attackers modifying packages via MITM03:26
mordred++ I'm on board with that03:26
fungialso possibly move pypi off to a separate server again. i think that's where he was still waffling03:27
lifelesscan still have anonymous non-ssl rsync, but tripleo would want to be more secure, all things being equal.03:27
fungithen we moved on to discussing how having a filesystem rsync could grok might be counter to any future move of the mirror to an object storage backend03:28
fungiand looking into rsync alternatives which could be integrated into the web services layer instead03:29
fungiand then he had to bail for a prior engagement03:30
lifelessI would like to ignore that though, as a 500M mirror isn't a stateless-scaling-problem; it's what - < 60 seconds to mirror from an existing server, or put it on a block device.03:31
mordrednod. well, not to be my brazen self - but the object storage part reminds me of the gerrit folks and "we might have a generalized workflow engine some day"...03:31
mordredand what lifeless said03:31
mordredI think our build logs need to go in object storage badly03:32
mordredthe mirror - meh03:32
mordredbut the stunnel and the put-mirror-on-own-box seem quite legit03:33
*** sarob has quit IRC03:35
*** sarob has joined #openstack-infra03:35
*** arosen1 has joined #openstack-infra03:37
mordredwow. I just tweeted a link to gerrit docs about prolog03:38
fungiyou need to get out more ;)03:38
*** dkliban has joined #openstack-infra03:39
mordredfungi: I just had dinner and drinks in brazil! how much more out can I get?03:39
*** sarob has quit IRC03:40
fungiif sabdfl can do it...03:41
*** changbl_ has quit IRC03:42
fungithough i hear the dehydrated ice cream and tang gets tiresome03:43
mordredI love dehydrated ice cream03:45
clarkbmordred: I like the context of that tweet too03:47
mordredclarkb: trying one more devstack patch03:48
*** sdake has quit IRC03:48
*** enikanorov has joined #openstack-infra03:51
*** ladquin has quit IRC03:53
*** SergeyLukjanov has joined #openstack-infra03:59
*** kpepple has quit IRC04:00
mordredok. I have actually read the scrollback on the earlier discussion around rsync and object store and I have more context04:01
lifelessmordred: devstack - dstufft had a suggestion for the same thing when it bit tripleo.04:01
mordredand I do think a general look at the system design we'd like to be at is warranted04:01
lifelesspip install -U setuptools after installing distribute.04:01
mordredlifeless: what?04:01
mordredlifeless: right. we already do that04:01
mordredlifeless: it breaks on redhat04:01
lifelessactually, not dstufft, was rslage or something - for fixing redhat.04:02
lifelessmordred: interesting, since it was submitted to fix redhat :)04:02
mordredlifeless: it turns out that "redhat" are a conflicting set of needs04:02
mordredlifeless: I have at least three diffrent patches around devstack that fix devstack for one person's use on fedora and breaks it for anothers04:02
* mordred tries to remember why pip install -U setuptools was insufficient04:03
dstufftit breaks yum04:04
mordredlifeless: what would be great is if I could figure out why starting the services in devstack does not work with the software installed in a virtualenv04:04
mordreddstufft: is that what it is?04:04
dstufftyea I think so04:04
dstufftbecause it uninstalsl setuptools that yum installed04:04
dstufftand installs a new version04:05
mordredoh. right04:05
dstufftbut doesn't tell yum it did that04:05
mordredbecause pip uninstalls then reinstalls04:05
lifelessoh yay. Well, I'm sure the redhatters will followup ;)04:05
mordredlifeless: I'm surrounded by redhatters04:05
lifelessmordred: where art thou?04:05
mordredI'm more and more in favor of a single global virtualenv04:05
mordredlifeless: brazil04:05
lifelessmordred: I knew that...04:05
mordredI'm not literally surrounded by them04:05
mordredI'm not actually surrounded by them04:06
lifelessmordred: one venv per service :>04:06
mordredI'm figuratively surrounded by them04:06
lifelessmordred: because that way conflicting requirements between services isn't a problem.04:06
mordredno, I very specifically do not want that04:06
*** zul has joined #openstack-infra04:06
mordredwe've already solved that04:06
mordredconflicting requirements is fixed04:06
dstuffta virtualenv is pretty similar to a software collection really04:06
dstufftexcept a software collection is controlled by rpm and handles more than python04:06
lifelessmordred: I don't think you have in general. You have for the 'running trunk of everything' special case.04:06
mordredlifeless: we're not designing a general deployment solution - you are04:07
lifelessmordred: doh, how did I end up doing that :)04:07
mordredlifeless: in this case, _I_ need to care that we prevent conflicting dependencies04:07
lifelessmordred: you may need to revisit the 'single global venv' when we start gating on trunk vs release scenarios.04:07
*** sarob has joined #openstack-infra04:07
mordredas otherwise the distros will kill me04:08
mordredlifeless: why?04:08
lifelessmordred: because the last release of e.g. nova may have conflicting requirements after 5 months.04:08
mordredlifeless: we already do this in grenade04:08
lifelessmordred: grenade installs e.g. released nova, trunk nova client ?04:09
mordredlifeless: the only thing we don't try to do is install different verions on the same machine simultaneously04:09
mordredlifeless: yes04:09
mordredand that should always work :)04:09
lifelesscause I thought grenade tested upgrades.04:09
mordredit does04:09
mordredthe client libraries do not have stable releases04:09
mordredI'm picking a nit04:09
lifelessclearly I need to learn more about it04:09
* lifeless is shocked04:09
mordredI'm also speaking flippantly04:10
mordredright now, it does not entirely do that, because grizzly has some bad arbitrary version caps04:10
lifeless$jeepybvenv/bin/run-mirror -b remotes/origin/master --verbose -c $tmpdir/mirror.yaml --no-process04:10
lifelesswhat is the branch used for ?04:10
lifelessdo I need a checkout of each thing whose requirements I'm going to mirror ?04:10
mordredlifeless: there are some historical bits in there ...04:11
mordredit was originally designed to work on the list of projects in our yaml file04:11
*** pcrews has joined #openstack-infra04:11
mordrednow we only operate on the requirments repo04:11
*** dina_belova has joined #openstack-infra04:11
mordredbut still treat it with the wider logic04:12
lifelessok, so I clone openstack-infra/requirements?04:12
mordredrun-mirror should do that for you04:13
mordredif you list it in your mirror.yaml file04:13
lifelessoh, so in projects in the yaml I list the github repo?04:13
mordred(and it's openstack/requirements - not openstack-infra/requirements)04:13
mordredlifeless: there's a simple example of usage...04:13
mordredhrm. maybe it's not that simple04:14
lifeless    for project in mirror['projects']:04:14
lifelessKeyError: 'projects'04:14
lifelessoh, damn copy paste fail04:15
*** dina_belova has quit IRC04:16
*** changbl has joined #openstack-infra04:17
*** nayward has quit IRC04:20
lifelesswhat does ffi need to build ?04:20
lifelesssorry, xattr04:20
lifeless  Running egg_info for package xattr04:20
lifeless    c/_cffi_backend.c:14:17: fatal error: ffi.h: No such file or directory04:20
lifeless    compilation terminated.04:20
*** nijaba has quit IRC04:22
lifelessmordred: ^04:22
mordredlifeless: you need libffi-dev04:23
*** nijaba has joined #openstack-infra04:23
*** nijaba has quit IRC04:23
*** nijaba has joined #openstack-infra04:23
*** reed has joined #openstack-infra04:23
mordredlifeless: current list there, in case it prevents you from chasing further04:24
Alex_Gaynormordred: gerrit requires like a server to run :effort:. ZaaS please04:25
lifelessI'll note that this is one of my objections to run-mirror ;)04:25
mordredAlex_Gaynor: gimme a little bit, I'm working on it...04:25
Alex_Gaynormordred: A+ awesome :)04:26
mordredand with that ... I'm off to bed04:26
lifelessAlex_Gaynor: disk-image-builder + heat. doit.04:27
Alex_Gaynorlifeless: Sounds like a thing someone who actually enjoys ops should do04:27
lifeless  File "/tmp/tmpWHXU3n/venv/local/lib/python2.7/site-packages/pip/", line 887, in filename04:29
lifeless    assert name, ('URL %r produced no filename' % self.url)04:29
lifelesssomething wrong there...04:29
lifelessAssertionError: URL 'file://' produced no filename04:29
Alex_Gaynorlifeless: file a bug :)04:29
lifelessAlex_Gaynor: sure, but what on...04:29
lifelessAlex_Gaynor: dunno cause yet.04:29
Alex_Gaynorlifeless: pip I imagine, things should never die with an assertion error04:30
lifelesswell, dunno reproduction instructions yet04:30
lifelessAlex_Gaynor: this is a reasonable assertion, the issue for me is the input04:30
lifelessgotta run04:30
Alex_Gaynordstufft: Look a thing is broken!04:31
*** dina_belova has joined #openstack-infra04:33
*** nati_ueno has joined #openstack-infra04:33
*** afazekas has joined #openstack-infra04:35
dstufftAlex_Gaynor: :[04:36
*** sarob has quit IRC04:36
dstufftAlex_Gaynor: 1.4.1 is gonna come out tomorrow btw04:36
dstufftwith fixes for stuff04:36
Alex_Gaynordstufft: wassit do?04:36
*** sarob has joined #openstack-infra04:36
Alex_Gaynordstufft: stuff I filed I guess?04:36
*** dina_belova has quit IRC04:36
*** pcrews has quit IRC04:39
*** sarob has quit IRC04:41
*** sdake has joined #openstack-infra04:49
*** reed has quit IRC04:50
*** sdake_ has quit IRC04:50
*** sarob has joined #openstack-infra05:00
*** dina_belova has joined #openstack-infra05:05
*** emagana has joined #openstack-infra05:09
*** shhu has quit IRC05:09
*** boris-42 has joined #openstack-infra05:11
*** jhesketh has quit IRC05:14
*** jhesketh has joined #openstack-infra05:14
*** nijaba has quit IRC05:22
*** dina_belova has quit IRC05:22
*** nijaba has joined #openstack-infra05:23
*** nijaba has quit IRC05:23
*** nijaba has joined #openstack-infra05:23
*** dina_belova has joined #openstack-infra05:24
*** dina_belova has quit IRC05:24
*** sarob has quit IRC05:29
*** sarob has joined #openstack-infra05:30
jheskethHi guys, how do I get gerrit/jenkins to recheck a patchset?05:32
*** sarob has quit IRC05:34
pleia2jhesketh: this should help
jheskeththanks pleia2, my search foo failed me05:40
pleia2you're welcome05:41
jheskethso why do we keep running gates that seem to fail/not work that are non-voting?05:41
*** pcrews has joined #openstack-infra05:43
*** nicedice has quit IRC05:43
*** sarob has joined #openstack-infra05:44
*** SergeyLukjanov has quit IRC05:48
*** sarob has quit IRC05:49
*** sarob has joined #openstack-infra05:49
clarkbthe goal is they eventually pass and we make them vote05:51
clarkbit helps development of new tests. the tempest testr test for example05:51
Alex_Gaynorclarkb: I was suggesting to mordred the other week that if a team wants to add a non-gating test they should be required to provide a description, a timeline, and under what conditions it'll be upgraded to gating05:52
clarkbthats not a bad idea05:52
*** sarob has quit IRC05:53
clarkbthere are legit reasons for somethibg to always be non voting. our jjb diff job is that way. it provides a binary state with logs to help reviewers decide if the change is passing or failing05:54
Alex_Gaynorthat's definitely the exception, not the rule though05:54
jheskethso why not use the silent pipeline for non-voting jobs?05:55
clarkbjhesketh because we want them to have visibility05:56
clarkbthe failure rate shoulr motivate you to fix it05:56
*** nayward has joined #openstack-infra05:56
jheskethheh, okay05:56
clarkband yes the jjb diff job is an exception05:57
*** arosen1 has quit IRC05:57
*** vogxn has joined #openstack-infra05:57
clarkbAlex_Gaynor: the swift functional tests graduated to gating status this week \o/05:58
clarkbjhesketh: you will probably start seeing non voting py3k testa soon too05:59
clarkbagain we dont expect them to pass but should provide good info to careful reviewers05:59
*** arosen1 has joined #openstack-infra06:00
jheskethso I think I follow that zuul triggers jenkins through gearman and jenkins-job-builder builds configurations for jenkins, but I'm unsure where the job code itself is stored06:17
jheskethfor example, where is the code that gate-nova-python27 runs?06:18
lifelesshmm python-cinderclient not mirroring properly06:19
lifelessclarkb: so run-mirror, do you guys basically run it again and again and again until it works ?06:19
*** sarob has joined #openstack-infra06:20
*** nijaba has quit IRC06:22
*** nijaba has joined #openstack-infra06:23
*** nijaba has quit IRC06:23
*** nijaba has joined #openstack-infra06:23
*** sarob has quit IRC06:26
lifelessmordred: oh btw services in venvs not starting - we had a minor glitchon that in neutron with hardcoded paths not being linked into the global path, but I suspect we were doing something unique...06:27
lifelessmordred: what symptoms do you have?06:28
*** marun has quit IRC06:31
*** marun has joined #openstack-infra06:35
*** marun has quit IRC06:42
*** sarob has joined #openstack-infra06:52
*** sarob has quit IRC06:59
*** sdake_ has joined #openstack-infra07:01
*** sdake_ has joined #openstack-infra07:01
*** salv-orlando has joined #openstack-infra07:12
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added Robot Framework reposts publisher
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added Robot Framework reports publisher
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added pre-scm-buildstep wrapper
*** nijaba has quit IRC07:23
*** woodspa has joined #openstack-infra07:23
*** nijaba has joined #openstack-infra07:24
*** nijaba has quit IRC07:24
*** nijaba has joined #openstack-infra07:24
*** sarob has joined #openstack-infra07:25
*** sarob has quit IRC07:29
*** SergeyLukjanov has joined #openstack-infra07:30
*** jhesketh has quit IRC07:31
*** psedlak has joined #openstack-infra07:32
*** afazekas_ has joined #openstack-infra07:33
*** afazekas has quit IRC07:36
*** afazekas_ is now known as afazekas07:36
*** dina_belova has joined #openstack-infra07:40
*** derekh has joined #openstack-infra07:40
*** woodspa has quit IRC07:41
*** psedlak has quit IRC07:43
*** psedlak has joined #openstack-infra07:45
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added more options to the Build Timeout plugin
*** BobBall_Away is now known as BobBall07:52
*** vogxn has quit IRC07:53
*** odyssey4me has joined #openstack-infra08:01
*** Ryan_Lane has joined #openstack-infra08:02
*** fbo_away is now known as fbo08:02
*** jpich has joined #openstack-infra08:12
*** derekh has quit IRC08:13
*** odyssey4me has quit IRC08:16
*** derekh has joined #openstack-infra08:18
*** psedlak_ has joined #openstack-infra08:20
*** odyssey4me has joined #openstack-infra08:21
*** nijaba has quit IRC08:24
*** psedlak has quit IRC08:24
*** nijaba has joined #openstack-infra08:24
*** sarob has joined #openstack-infra08:25
*** sarob has quit IRC08:30
*** Ryan_Lane has quit IRC08:33
openstackgerritRoman Podolyaka proposed a change to openstack-infra/config: Modify running of sqlalchemy-migrate tests
*** ruhe has joined #openstack-infra08:37
*** vogxn has joined #openstack-infra08:43
uvirtbotLaunchpad bug 1209132 in openstack-ci "run-mirror fails to mirror cinderclient" [Undecided,New]08:45
*** odyssey4me has quit IRC08:47
*** odyssey4me has joined #openstack-infra08:55
*** dina_belova has quit IRC09:12
*** ruhe has quit IRC09:15
*** ruhe has joined #openstack-infra09:17
*** nijaba has quit IRC09:23
*** nijaba has joined #openstack-infra09:24
*** nijaba has joined #openstack-infra09:24
*** sarob has joined #openstack-infra09:26
*** sarob has quit IRC09:31
*** dina_belova has joined #openstack-infra09:42
openstackgerritA change was merged to openstack-infra/storyboard: Edit README to include amended initial data instructions.
openstackgerritA change was merged to openstack-infra/storyboard: projects view now accepts projects with spaces
*** dina_belova has quit IRC09:51
*** GlaciErr has joined #openstack-infra09:56
*** Ryan_Lane has joined #openstack-infra10:03
*** nijaba has quit IRC10:24
*** nijaba has joined #openstack-infra10:25
*** nijaba has quit IRC10:25
*** nijaba has joined #openstack-infra10:25
*** ruhe has quit IRC10:25
*** sarob has joined #openstack-infra10:26
*** dina_belova has joined #openstack-infra10:31
*** sarob has quit IRC10:31
*** vogxn has quit IRC10:31
*** vogxn has joined #openstack-infra10:32
*** yaguang has quit IRC10:39
*** emagana has quit IRC10:41
BobBallsdague / anyone: What/where is EXTRA_CONF? I'm probably be being blind... but I don't see it10:42
*** nati_ueno has quit IRC10:42
*** ruhe has joined #openstack-infra10:55
*** nayward has quit IRC10:55
*** Ryan_Lane has quit IRC10:55
lifelessmordred: btw just needs +2's AFAICT11:01
*** nayward has joined #openstack-infra11:01
*** ruhe has quit IRC11:02
*** weshay has joined #openstack-infra11:02
*** fbo is now known as fbo_away11:04
*** emagana has joined #openstack-infra11:11
*** nayward has quit IRC11:17
*** boris-42 has quit IRC11:19
*** emagana has quit IRC11:19
*** nijaba has quit IRC11:24
*** adalbas has joined #openstack-infra11:25
*** nijaba has joined #openstack-infra11:25
*** nijaba has quit IRC11:25
*** nijaba has joined #openstack-infra11:25
*** sarob has joined #openstack-infra11:27
*** chuckieb has quit IRC11:28
*** woodspa has joined #openstack-infra11:29
*** lcestari has joined #openstack-infra11:30
*** sarob has quit IRC11:32
*** CliMz has joined #openstack-infra11:33
*** GlaciErr has quit IRC11:36
*** CliMz has quit IRC11:37
*** vogxn has quit IRC11:38
*** CliMz has joined #openstack-infra11:41
*** prad_ has joined #openstack-infra11:43
sdagueBobBall: actually it's called EXTRA_OPTS11:44
*** ArxCruz has joined #openstack-infra11:44
BobBallI searched for CONF too *grin*11:44
sdaguewe should probably restructure those variables for clarity and get them into the README11:44
sdagueyeh, my bad11:44
sdagueI think it used to be more documented than it currently is, the libraryizing of devstack hid some things away that were in stack.sh11:45
*** nayward has joined #openstack-infra11:47
*** ruhe has joined #openstack-infra11:58
*** sandywalsh has quit IRC12:01
*** dkehn_ has joined #openstack-infra12:01
*** mdurnosvistov has joined #openstack-infra12:02
*** rfolco has joined #openstack-infra12:02
*** psedlak_ has quit IRC12:02
*** psedlak_ has joined #openstack-infra12:02
*** dkehn has quit IRC12:02
*** afazekas has quit IRC12:02
sdaguemordred: when you get in I've got a bunch of requirements rebases that need +A12:05
*** CaptTofu has quit IRC12:06
*** CaptTofu has joined #openstack-infra12:07
*** mriedem has joined #openstack-infra12:07
*** CaptTofu has quit IRC12:08
*** CaptTofu has joined #openstack-infra12:08
*** fbo_away is now known as fbo12:09
*** prad_ has quit IRC12:10
*** sandywalsh has joined #openstack-infra12:14
*** afazekas has joined #openstack-infra12:14
*** nijaba has quit IRC12:25
*** nijaba has joined #openstack-infra12:27
*** nijaba has quit IRC12:27
*** nijaba has joined #openstack-infra12:27
*** dprince has joined #openstack-infra12:29
*** boris-42 has joined #openstack-infra12:30
*** afazekas has quit IRC12:36
*** psedlak__ has joined #openstack-infra12:41
*** psedlak_ has quit IRC12:45
mordredsdague: on it12:50
*** adalbas has quit IRC12:50
*** adalbas has joined #openstack-infra12:55
*** sarob has joined #openstack-infra12:58
*** afazekas has joined #openstack-infra13:00
sdaguemordred: and this time I was smart, and made them a patch series :)13:01
sdaguethough I needed to do a lot of cherry-pick --strategy none because it wanted to merge into the test file instead13:01
mordredsdague: yes! well done13:02
mordredand all +A'd13:02
*** sarob has quit IRC13:02
*** anteaya has joined #openstack-infra13:03
openstackgerritMonty Taylor proposed a change to openstack/requirements: Remove unneeded tests dir reference
openstackgerritMonty Taylor proposed a change to openstack/requirements: Update ourselves to ourselves
openstackgerritMonty Taylor proposed a change to openstack/requirements: Add pep8 checks
mordredsdague: ^^ fixed/rebased those13:06
*** weshay has quit IRC13:06
sdaguemordred: ok, cool, pep8 was still waiting on landing it as a gate check though, right?13:07
sdagueand the rest sat on that13:07
mordredsdague: oh - right13:07
mordredI thought it was on clark's print_fuction quesiton13:07
mordredone sec13:07
sdagueno look at clarkb's comments on 3996713:07
mordredyeah - I remember now. stil pre-coffee13:08
sdagueyep, no worries13:08
sdaguethat's the reason I didn't bother to go through those last night though13:08
sdaguealso, tempest gate on requirements is fantastic13:09
sdagueit means the sqla version uncap gets tested that it blows up before we break anyone13:09
sdaguethat made me very happy last night13:09
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Added pep8 checks to requirements
*** krtaylor has quit IRC13:11
*** nayward has quit IRC13:13
mordredsdague: +10013:16
BobBallsdague: - devstack will fail to run if the directories permissions are 0700 (  I assumed that 777 was needed because there's no guarantee that any daemons we run will be in our group if they aren't our user?13:19
*** pentameter has joined #openstack-infra13:21
*** vogxn has joined #openstack-infra13:21
*** nijaba has quit IRC13:25
*** nijaba has joined #openstack-infra13:26
*** nijaba has quit IRC13:26
*** nijaba has joined #openstack-infra13:26
*** larsks has quit IRC13:26
*** dkehn_ is now known as dkehn13:27
*** afazekas has quit IRC13:35
zaromordred: would you be able to take a look?
*** vijendar has joined #openstack-infra13:36
openstackgerritA change was merged to openstack/requirements: Raise warlock requirement
openstackgerritwill soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin
*** weshay has joined #openstack-infra13:38
*** dina_belova has quit IRC13:38
*** prad_ has joined #openstack-infra13:41
*** ruhe has quit IRC13:42
sdagueBobBall: what about just turning on rx bits?13:43
*** changbl has quit IRC13:43
BobBallIf that's all that's needed I'm happy - I was being paranoid :)13:43
openstackgerritDirk Mueller proposed a change to openstack/requirements: Always specify a lower bound on packages
*** ruhe has joined #openstack-infra13:44
*** markmcclain has joined #openstack-infra13:47
openstackgerritA change was merged to openstack/requirements: Bump pbr requirment to 0.5.21
*** pabelanger_ has joined #openstack-infra13:49
*** pabelanger_ has quit IRC13:50
*** pabelanger_ has joined #openstack-infra13:50
BobBallactually, convinced myself that 755 is fine - that's what devstack dir uses and all of the dirs it pulls down13:50
*** pabelanger has quit IRC13:50
BobBallso root directory being 755 is fine13:50
*** pabelanger_ is now known as pabelanger13:50
*** pabelanger_ has joined #openstack-infra13:50
*** pabelanger has quit IRC13:50
openstackgerritwill soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin
*** pabelanger has joined #openstack-infra13:51
*** krtaylor has joined #openstack-infra13:51
*** cppcabrera has joined #openstack-infra13:52
*** cppcabrera has left #openstack-infra13:52
*** datsun180b has joined #openstack-infra13:53
openstackgerritwill soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin
*** wu_wenxiang has joined #openstack-infra13:54
*** woodspa has quit IRC13:56
*** woodspa has joined #openstack-infra13:57
*** sarob has joined #openstack-infra13:58
openstackgerritwill soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin
BobBallarghhhhh - sdague are you there? or someone else who knows what's going on with setuptools...14:02
BobBallTrying to run devstack on CentOS - it used to work but now the code in lib/infra for unfubar_setuptools fails because after we've yum removed it, pip has gone too14:02
*** sarob has quit IRC14:03
BobBallso pip_install of setuptools doesn't work :)14:03
*** dina_belova has joined #openstack-infra14:05
*** avtar has joined #openstack-infra14:06
openstackgerritThierry Carrez proposed a change to openstack-infra/storyboard: Introducing project groups
*** tiamar has quit IRC14:07
mordredBobBall: yes. we're having rh/centos issues14:09
BobBallknown problem then :)14:09
mordredtry commenting out the removal14:09
mordredand see if that breaks more or less for you14:10
*** _TheDodd_ has joined #openstack-infra14:10
*** ^d has joined #openstack-infra14:11
*** ^d has joined #openstack-infra14:11
*** ruhe has joined #openstack-infra14:15
BobBallless so far... ;)14:24
*** nijaba has quit IRC14:25
*** nijaba has joined #openstack-infra14:26
openstackgerritThierry Carrez proposed a change to openstack-infra/storyboard: Introducing project groups
sdagueBobBall: so there is a proposed fix for it that dtroyer wrote14:28
sdagueBobBall: try this on for size -
BobBallI've got confused in the last few days - too many similar issues going round in my head14:29
sdagueBobBall: dude, I'm with you14:29
sdagueI've been down this rabbit hole for 3 weeks now14:29
*** enikanorov-w has joined #openstack-infra14:29
*** burt has joined #openstack-infra14:29
sdaguebut I think we actually have a light at the end of the tunnel14:29
BobBallAh - not still going down then :D14:29
BobBallI like his final comment14:30 runs to completion on F18 and CentOS6 but in my environment14:30
BobBallinstances do not have working networks14:30
BobBallwho needs networking?! ;)14:30
chmoueldoes any of you guys have by any chances the latest standard license header?14:32
*** jbjohnso_ has quit IRC14:34
sdaguechmouel: it's in hacking/hacking/core.py14:34
clarkbchmouel: should be in every repo14:34
sdagueas we have a check for it14:34
clarkbthe header is at the end of the text14:34
*** emagana has joined #openstack-infra14:35
chmouelcool thanks14:35
*** Ryan_Lane has joined #openstack-infra14:36
chmouelif there is some vim: stuff in the files can i add my emacs stuff there as well ?14:36
chmouel(it's a troll)14:36
sdaguechmouel: I do :)14:37
*** _TheDodd_ has left #openstack-infra14:38
sdaguethough emacs normally doesn't need other stuff to make it not scramble the files :)14:38
*** _TheDodd_ has joined #openstack-infra14:38
fungia proper troll would have been to ask whether you can add ms-project metadata14:38
fungisorry, visual studio14:38
fungii get all their product names mixed up14:39
zaromorning fungi14:39
fungii also don't actually like the vim comments in files, even though i use vim. i have y vim configured for "the one true style" for whatever file type i happen to be editing14:40
fungimorning zaro!14:40
clarkbfungi ++ annotations in file bother me for some reason14:42
*** changbl has joined #openstack-infra14:45
*** rcleere has joined #openstack-infra14:46
*** afazekas has joined #openstack-infra14:48
openstackgerritA change was merged to openstack/requirements: Raise eventlet to 0.13.0
openstackgerritA change was merged to openstack/requirements: Add pysnmp needed for Ceilometer/HardwareAgent/SNMPInspector
openstackgerritA change was merged to openstack/requirements: Update keyring minimum version
openstackgerritA change was merged to openstack/requirements: Raise psutil requirement
openstackgerritA change was merged to openstack/requirements: remove netifaces as a requirement
*** mdurnosvistov has quit IRC14:52
*** mdurnosvistov has joined #openstack-infra14:52
*** dina_belova has quit IRC14:54
Mithrandirfungi: clearly you mean MS word?14:57
chmouelfungi: +1 i am not sure why people do that since it's not like people switch between editor configs all the time14:57
fungiMithrandir: the only ones i remember using were ms lookout! and m sexchange14:57
fungithough i've heard of m sword, it sounds rather dangerous14:58
*** sarob has joined #openstack-infra14:59
*** nayward has joined #openstack-infra14:59
dtroyerBobBall: :)  It was late when I wrote that… IOW I know it isn't done but I wanted to get it out for incremental testing.  And more verification that Ububntu is working.15:02
ArxCruzHello, does anyone know where can I find the hiera data in order to use puppet to install some openstack infra projects?15:02
ArxCruzI'm trying to run puppet and I'm getting this error:15:02
ArxCruzCould not find data item sysadmins in any Hiera data file and no default supplied at /opt/config/production/manifests/site.pp:7 on node
*** boris-42 has quit IRC15:03
*** boris-42_ has joined #openstack-infra15:03
*** dina_belova has joined #openstack-infra15:03
clarkbArxCruz: you have to provide it or swap out the hiera calls with your data15:04
*** sarob has quit IRC15:04
clarkbsysadmins is a list of email addresses to get root mail iirc15:04
ArxCruzclarkb: thanks, but where can I find an hiera sample ?15:04
fungiArxCruz: we have some suggestions at
fungithe local.pp example there is of swapping out variables we store in hiera with local definitions15:05
clarkbArxCruz: we don't currently have a sample file. we could add one or comment values in site.pp about what their contents should be15:06
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Improved the gerrit trigger
clarkbArxCruz: the only place we use hiera is in site.pp to avoid needing complicated samples requiring hiera15:06
*** HenryG has quit IRC15:06
*** krtaylor has quit IRC15:07
ArxCruzfungi: clarkb thanks15:07
ArxCruzhaving a sample file it's a good idea, mainly for people starting to use puppet and openstack like me :)15:07
*** UtahDave has joined #openstack-infra15:07
*** boris-42_ is now known as boris-4215:08
clarkbit doesn't hurt, but it is an implementation detail and not a requirement. I think it may be better to properly document out class parameters15:08
fungiArxCruz: well, bear in mind our puppet configuration repo at openstack-infra/config will not help you set up openstack in any way. it will help you set up servers like we use to provide resources to our developer community (code review, wiki, pastebin, et cetera)15:09
fungiArxCruz: if you're interested in using puppet to set up openstack, see the various puppet-* repositories at
*** dprince has quit IRC15:10
*** vogxn has quit IRC15:13
*** jjmb has joined #openstack-infra15:15
*** pabelanger has quit IRC15:15
*** krtaylor has joined #openstack-infra15:16
*** CliMz has quit IRC15:17
*** rnirmal has joined #openstack-infra15:18
*** dina_belova has quit IRC15:20
*** SergeyLukjanov has quit IRC15:20
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added on-cache option
*** mrodden1 has quit IRC15:25
*** nijaba has quit IRC15:26
*** vogxn has joined #openstack-infra15:26
*** nijaba has joined #openstack-infra15:27
*** nijaba has quit IRC15:27
*** nijaba has joined #openstack-infra15:27
*** vijendar has quit IRC15:31
*** sarob has joined #openstack-infra15:31
*** Ryan_Lane has quit IRC15:32
*** mrodden has joined #openstack-infra15:35
*** HenryG has joined #openstack-infra15:36
*** emagana has quit IRC15:36
*** sarob has quit IRC15:37
*** Ryan_Lane has joined #openstack-infra15:37
*** ruhe has quit IRC15:41
*** zaro0508 has quit IRC15:41
*** Ryan_Lane has quit IRC15:44
*** reed has joined #openstack-infra15:52
*** olaph has quit IRC15:54
chmouelsomebody can help with some weirdness with testr/jenkins?15:55
*** salv-orlando has quit IRC15:55
clarkbchmouel what are you seeing?15:57
chmouelclarkb: welll that's for this bug
chmouelI am getting an error in
chmouelwhich is cool and dandy15:58
chmouelbut launching the tox manually shows me no error15:58
chmouellaunching nosetests pass as well15:58
chmouelbut launching the test manually fail :15:59
clarkbchmouel: don't use nosetests :)15:59
chmouelyep was just invesgiating :)15:59
clarkbso, testr forks and execs testr runners and collects the results over a subunit stream15:59
*** ruhe has joined #openstack-infra15:59
clarkbthis means that your tests are partitioned and run in multiple processes concurrently16:00
*** krtaylor has quit IRC16:00
clarkbthe other thing testr does is it may not make those partitions deterministic especially across file systems16:00
clarkball of that to say when you run tox locally the tests may run in a different order and in a different set of processes than they do on jenkins16:01
*** olaph has joined #openstack-infra16:01
*** w_ has joined #openstack-infra16:01
*** w_ has quit IRC16:01
chmouelhumm so how do I fix that?16:02
clarkbchmouel: the fact that running the test individual with nose (you can do this with tox by doing `tox -epy27 $testname` where $testname is a regex that is used for matching the name) fails implies that there is some hidden dependency in the test16:02
chmouelrerun the tests until that succed?16:02
clarkbchmouel: no! :) you fix the test to remove the hidden dependency16:02
chmouelfun times16:03
*** CliMz has joined #openstack-infra16:03
clarkbthe test is broken if it cannot run on its own, you will need to figure out what that dependency is and correct it16:03
clarkbthe fact that it passes when run by nose with the rest of the suite implies there is some other test that is satisfying the dependency implicitly16:04
chmouelthe thing is that i can't reproduce the randomness16:04
fungior rattle the appropriate cages of people more incented to fix the test, at least16:04
clarkbchmouel: I thought you said you could reproduce by running the test individually?16:04
clarkbchmouel: that should be sufficient to debug16:04
chmouelclarkb: oh i guess, will try16:04
clarkbchmouel: cms.subprocess is None16:05
*** emagana has joined #openstack-infra16:05
clarkbchmouel: my guess is somewhere another test is either importing the proper bits or instantiating that object16:05
clarkband this test needs to do the same16:05
chmouelah ok yeah i see16:06
*** odyssey4me has quit IRC16:07
clarkbSome days i really want a jenkins job that runs each test in its own process16:08
zulcan we get a newer oslo.config  that builds with python3 on pypi?16:08
clarkbzul: markmc does those releases iirc16:08
zulclarkb:  k16:08
chmouelclarkb: I think I have seem someone doing that with docker16:10
clarkbchmouel: _ensure_subprocess needs to be called in keystoneclient.common.cms16:11
clarkbchmouel: that is a private method so you will need to call one of the things the sideffects it16:12
chmouelclarkb: cool you had it all debugged for me :)16:12
chmoueli was filling up the bug report
uvirtbotLaunchpad bug 1209300 in python-keystoneclient "concurrency error in auth_token error" [Undecided,New]16:12
chmouelprob need a better subject16:12
clarkbchmouel: I think it is a fast fix :)16:13
chmouelfeel free to send it over :)16:13
*** CliMz has quit IRC16:13
chmouelif you like I don't mind doing it...16:14
clarkbI am just trying to figure out the best way to do this16:15
clarkbIt may require calling the private function directly which seems a little dirty16:15
ArxCruzfungi: ty16:15
*** pabelanger_ has quit IRC16:16
*** pabelanger has joined #openstack-infra16:16
*** marun has joined #openstack-infra16:16
zaromorning pleia216:17
zarofungi: would you be able to take a look?
clarkbchmouel: there is a chicken and egg16:18
clarkbthe assertRaises is calling the method that will init subprocess16:18
clarkbbut it is referring to the exception prior to calling it :)16:18
zaroclarkb: not coming in today.16:18
clarkbzaro: ok16:18
clarkbchmouel: there are other tests that call the same method, if they run first the test passes16:18
chmoueli see16:19
fungiclarkb: calling private methods from tests doesn't seem that dirty to me. they're tests. sometimes you may even want to directly test a private method16:19
clarkbchmouel: I will push a naive patch and people can argue if it is good enough :)16:19
*** dprince has joined #openstack-infra16:20
clarkbrunning tox now16:21
chmouelclarkb: let me know when you have done so I can review16:23
*** krtaylor has joined #openstack-infra16:23
*** vijendar has joined #openstack-infra16:25
*** nijaba has quit IRC16:26
*** nijaba has joined #openstack-infra16:27
*** nijaba has quit IRC16:27
*** nijaba has joined #openstack-infra16:27
clarkbchmouel: I have confirmed tox -epy27 test_request_no_token_dummy fails without the patch and passes with it16:29
clarkbfungi: it is dirty because we want the side effect. we aren't actually directly testing it16:30
chmouelclarkb: cool that was quick16:30
fungiclarkb: in that case, calling any method purely for an unintended side effect seems specious16:30
clarkbchmouel: if you like we can move that into the setUp() call16:31
chmouelI would be honest I don't really understand very well the auth_token middleware structure and inheritance to be confident how it works16:32
clarkbyou probably understand it better than me :)16:32
chmouelthe test i mean16:32
chmouelI think  I wrote the first version of those16:33
chmouelbut things has much changed since then :)16:33
*** vogxn has quit IRC16:33
CaptTofudevananda: ping!16:33
clarkbchmouel: updated the commit message with the bug16:34
clarkbcompletely forgot to do that in my haste16:34
chmouelclarkb: but I think in seTup sounds like a better place16:34
clarkbgah ok :) I will push that16:34
chmouelsince there is only one test in this class16:34
chmouelbut probably would have to the same bug if there is a new one16:35
*** BobBall is now known as BobBall_AFK16:36
*** derekh has quit IRC16:37
chmouelclarkb: thanks16:38
clarkbchmouel: third patchset pushed, should be good now16:38
chmouelperfect will ping the keystone guys to get this merged quickly16:39
fungiclarkb: jeblair: mordred: i have temporarily stopped puppet and apache2 on pending a usn and precise security update16:41
fungiclarkb: jeblair: mordred:
fungiat your option, i can also temporarily comment out the contents of /etc/cron.d/cacti to stop polling, depending on how uncomfortable you are with the security of the systems we're monitoring16:44
fungibut it will of course leave gaps in all the graphs if i do16:44
*** boris-42 has quit IRC16:45
fungiand it doesn't look like any of the exploitation vectors are in the return values from the oids being polled, unless i'm misreading the patches, so probably not a risk16:46
fungi(my php is a bit rusty these days, thankfully)16:47
jeblairfungi: thanks; i think leaving the poller running is ok16:51
fungii'll keep banging on the usn list until i see an update16:52
clarkbrunning tests individually is really slow16:52
clarkbI could see this potentially being a periodic job16:53
fungiclarkb: run them individually in parallel on separate machines ;)16:53
fungithat should be really, really fast. you just need really, really many machines16:53
*** HenryG has quit IRC16:53
*** HenryG has joined #openstack-infra16:54
* clarkb AFKs for a bit17:00
*** ruhe has quit IRC17:01
*** nayward has quit IRC17:03
*** fbo is now known as fbo_away17:03
*** dkranz has quit IRC17:04
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Unset ZUUL_PROJECT on periodic devstack jobs
*** ^demon has joined #openstack-infra17:07
*** ^d is now known as Guest8096817:08
*** psedlak__ has quit IRC17:09
*** Guest80968 has quit IRC17:09
*** ^demon is now known as ^d17:09
jeblairfungi: did you want to review ?17:10
fungii'm sure i did want to. i'll do that now ;)17:11
jeblairfungi, clarkb: is a futher operational fix needed by the stable jobs17:11
*** ruhe has joined #openstack-infra17:11
fungiadded it next in line17:12
clarkbjeblair: in you are setting ZUUL_BRANCH which seems to conflict with teh commit message17:16
clarkbone of the two is set not both or none17:16
jeblairclarkb: wow, that was a poor commit message wasn't it?17:17
*** nati_ueno has joined #openstack-infra17:17
clarkbmy brain still feels like it is far too early in the morning, wanted to make sure I wasn't mis parsing something17:17
*** jpich has quit IRC17:17
fungiyeah, i'm confused by that one as well17:17
*** sarob has joined #openstack-infra17:18
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Unset ZUUL_PROJECT on periodic devstack jobs
openstackgerritA change was merged to openstack-infra/gearman-plugin: Add OFFLINE_NODE_WHEN_COMPLETE option
fungifrom the change i gather zuul is passing ZUUL_PROJECT in the environment and d-g is bailing on it17:18
jeblairclarkb, fungi: forget everything i said earlier and try that instead.  :)17:18
*** dkranz has joined #openstack-infra17:18
fungithat makes waaaay more sense17:19
*** hub_cap has left #openstack-infra17:19
*** pabelanger has quit IRC17:20
fungialso, our gerrit triggers seem not to be running (no trivial rebase comment there about only the commit message changing, several recent bugs which didn't get updated when patches merged). i'm going to start looking into whether we've broken jeepyb again17:21
fungioh, nevermind. it did comment, just took longer than usual17:21
fungii still suspect we may have issues though17:21
zarojeblair: does review 40041 fix bug 1082803?17:24
uvirtbotLaunchpad bug 1082803 in openstack-ci "Manage jenkins global config and plugin list" [Medium,Triaged]
_TheDodd_fungi, I hope not. Out of curiosity, what do you think is broken?17:24
sarobanyone up for helping with errors?17:24
jeblairzaro: oh, i guess it's a partial fix17:24
fungi_TheDodd_: no idea yet. i'm looking at tracebacks fromit in the gerrit logs ow and will paste a summary link here in a bit17:24
zarocool thing with that uvirtbot showing the bug info.  can we do same thing with review info?17:25
zarojeblair: would you mind updating bug with what else is needed for full fix?17:26
*** afazekas has quit IRC17:26
clarkbzaro: you could extend gerritbot to do that, right now it does not read from irc, it only writes, but reading and responding with review info shouldn't be too hard to add17:26
*** nijaba has quit IRC17:26
clarkbI think some folks feel that gerritbot is too chatty so it will need to be configurable on a per channel basis17:27
*** odyssey4me has joined #openstack-infra17:27
*** reed has quit IRC17:27
*** nijaba has joined #openstack-infra17:27
_TheDodd_fungi: great. I will try to stay in the loop as much as possible.17:29
zaroclarkb: i just like that it prints more info to irc channel when bug XXX is mentioned.  what does that have to do with gerrit being chatty?17:29
fungi_TheDodd_: maybe false alarm. this is the most recent exception i saw it throw, which is potentially legit (not sure which project it was for)
*** wu_wenxiang has quit IRC17:29
clarkbzaro: because having extra info about reviews would make that bot more chatty17:29
clarkbzaro: I like the extra info as well and think there are better ways for removing noise if individuals wish (IRC clients come with ignore/filter tools)17:30
*** ruhe has quit IRC17:30
fungii think we would probably want to run gerritbot not-on-the-review-server if it's acting on random input17:30
*** sarob_ has joined #openstack-infra17:30
zaro17:22:22            zaro | jeblair: does review 40041 fix bug 1082803?17:31
uvirtbotLaunchpad bug 1082803 in openstack-ci "Manage jenkins global config and plugin list" [Medium,Triaged]
zaro17:22:24        uvirtbot | Launchpad bug 1082803 in openstack-ci "Manage jenkins global config and plugin list"17:31
uvirtbotLaunchpad bug 1082803 in openstack-ci "Manage jenkins global config and plugin list" [Medium,Triaged]
zaro                         | [Medium,Triaged]
fungii'm uncomfortable enough with it running there now and acting on semi-dangerous input from the commit message texts, but at least that's more sanitized and leaves a much more substantial audit trail17:32
clarkbfungi: I would be completely open to moving it17:32
*** sarob_ has quit IRC17:33
*** sarob__ has joined #openstack-infra17:33
clarkbfungi: eavesdrop is probably a half decent place for it as the only stuff hosted there is irc and meeting logs (maybe we should treat that data as more important than I am implying though)17:33
fungiin my opinion that would be a prerequisite for turning it into a read+write ircbot17:33
fungi(moving it off review.o.o)17:34
*** sarob has quit IRC17:34
jeblairyes, we should put it on eavesdrop17:35
zarobug 1209335 entered.17:36
uvirtbotLaunchpad bug 1209335 in openstack-ci "gerritbot should print review info to irc channel" [Undecided,New]
*** ruhe has joined #openstack-infra17:39
jeblairjog0: if the actual issue is that gerritbot interrupts conversations in #openstack-nova, have you considered either removing it from openstack-nova, tweaking its config so it announces only interesting things (only merges?  only new patches?), or giving it the ability to be defer announcements while conversations are happening?17:40
_TheDodd_fungi, a quick look through the stack trace that you linked above seems to indicate that the problem is an authorization issue, where, perhaps, the dev who submitted the code is unauthorized to make changes to that particular bugs status.17:43
jog0jeblair: i do like having gerritbot and didn't want to remove it17:44
_TheDodd_fungi, you obviously know the code significantly better than I, but that is what I gathered upon initial evaluation with my current knowledge of our systems.17:44
jog0jeblair: but not sure what I would keep and remove if tweaking it17:44
jog0merges aren't that important, as everything is done17:44
jog0new patches are interesting, as are updates I think17:45
*** sarob__ has quit IRC17:45
jog0jeblair: and sometimes we refer to bot msgs, like you guys too17:45
*** sarob has joined #openstack-infra17:45
*** sarob has quit IRC17:45
*** harlowja has quit IRC17:45
*** nayward has joined #openstack-infra17:46
*** sarob has joined #openstack-infra17:46
*** ruhe has quit IRC17:46
jeblairjog0: turning of merge messages is easy17:46
jog0jeblair: in that lets turn of merges17:47
jog0as I don't think they really are relevant to discussion usually17:47
fungi_TheDodd_: yeah, that's why i said it's probably a legitimate exception and not indicative of a problem in the script17:47
jeblairjog0: openstack-infra/config/modules/gerritbot/files/gerritbot_channel_config.yaml17:48
*** harlowja has joined #openstack-infra17:48
mordredjeblair, jog0: I think that might be a good change to make for #openstack-dev too17:48
mordredfor the same reasons17:48
jog0mordred: will do17:49
jeblairmordred: maybe openstack-dev doesn't want announcements at all.17:49
jeblair[though we'll just miss zuul merge-bombs.  :( ]17:49
fungii find those amusing, if probably extremely disruptive to conversation on -dev17:50
*** boris-42 has joined #openstack-infra17:51
jeblairi actually like the "don't interrupt people" idea and think it could be quite workable.17:51
fungii concur17:52
fungiooh, new topic. look at the graphs on
fungii think we've plateaued at a maximum throughput for new jobs there ;)17:53
pleia2jeblair: replied to your comment on - if there is time a few of us should probably do some brainstorming to get this finally sorted17:54
*** odyssey4me has quit IRC17:54
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Don't post merges in openstack-dev and openstack-nova
jeblairpleia2: i don't quite understand.  are you saying puppet can't handle passing a string with "${name}" in it?17:55
jeblairpleia2: even one in single quotes?17:55
pleia2jeblair: so ${name} in the manifest means something different than name in a template for gerrit17:56
*** odyssey4me has joined #openstack-infra17:56
pleia2it's a specific variable17:56
*** sarob has quit IRC17:57
pleia2so I'm pretty sure puppet will expand it to it's default value in puppet in the manifest, and pass that wrong thing to the template rather than passing the string itself17:57
mgagnepleia2: ping17:57
pleia2mgagne: hey17:57
*** sarob has joined #openstack-infra17:57
jeblairdoesn't using single quotes cause it to not be substituted?  or isn't there a way to escape it?17:57
mgagnepleia2: can I be of any help?17:57
pleia2jeblair: that's where puppet-lint gets us, it won't let us17:58
*** mrodden1 has joined #openstack-infra17:58
mgagnejeblair: puppet-lint will cry17:58
jeblairbut it really works?17:58
jeblairwith puppet?17:58
*** avtar has quit IRC17:58
pleia2it should17:58
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Remove oslo gerritbot posts from openstack-nova
mgagnejeblair: you can use single-quote to avoid variable interpolation17:58
mgagnejeblair: but puppet-lint won't let you unless you globally disable the check17:58
jeblairso, ah, kill puppet lint? :)17:58
pleia2puppet-lint is usually lovely :)17:58
jeblairi mean, what we have here is an inability to actually do the work that we want to do17:59
pleia2I wouldn't want to disable it for gerrit.pp, that's a complicated file and link checks are nice17:59
pleia2lint too17:59
mgagnepleia2: you can disable this specific check until we (or someone else) find a way to fix it in a nicer way17:59
*** mrodden has quit IRC18:00
jeblairi wouldn't want to hinder our ability to provide a useful way to configure gerrit that doesn't assume specific git repo layouts18:00
pleia2anyway, in my patch I used fungi's fix of simply putting ${name}.git in the template, it works fine, but now we're stuck on true/false18:00
mgagnepleia2: maybe with the addition of a "#nolint" flag for puppet-lint18:00
jeblairmgagne: that would be lovely18:00
*** sarob_ has joined #openstack-infra18:00
jeblairpleia2: so what's the true/false issue?18:00
mgagnepleia2: what's up with the bool?18:00
pleia2same thing, we can't single quote them18:01
pleia2but I guess in this case it'll pass the right thing, so maybe it's ok18:01
jeblairpuppet lint doesn't let you pass the string 'true' ?18:01
mgagnejeblair: it won't allow you18:01
pleia2single quotes bad18:01
mgagnejeblair: true will be true anyway in the template18:02
*** sarob has quit IRC18:02
jeblairmgagne: and likewise false, i assume?18:02
mgagnejeblair: yes18:02
pleia2so that's the solution for ${name}.git we came up with18:02
jeblairok, that seems fine then; i think it makes sense for these args to feel more puppet-native anyway.18:03
*** odyssey4me has quit IRC18:03
mgagnejeblair: unless you *need* True, in which case you can use @var.to_s.capitalize if puppet-lint cries too much18:03
pleia2ok, and I'll drop the quotes around true/false, since there is no scary variable substitution, it'll just stay true/false18:03
pleia2mgagne: the word "true" needs to be assigned to the variable "mirror" to be passed into a template18:04
mgagnepleia2: how are bools managed by gerrit? does it have to be true or True?18:04
*** sarob_ has quit IRC18:04
pleia2mgagne: true is fine18:05
mgagnepleia2: The true puppet boolean will be substituted to true in a template. you should be fine.18:05
pleia2great, thanks18:05
* pleia2 gets to patching18:05
*** dina_belova has joined #openstack-infra18:06
jeblairpleia2, mgagne: thanks!18:06
*** sarob has joined #openstack-infra18:06
*** nayward has quit IRC18:07
jeblairfungi: i believe that graph should more correctly be labeled "Zuul Jobs Completed (per Hour)"18:08
jeblair(it does still include aborted jobs)18:09
fungialso a more interesting metric than started anyway18:13
jeblairit's about the same, but time shifted18:13
*** dina_belova has quit IRC18:14
fungiwould be really neat to make it multiline and also incorporate successful vs failed vs lost vs unstable vs... but that would maybe be better applied to specific pipeline stats (gate in particular)18:14
*** pabelanger has joined #openstack-infra18:17
*** svarnau has joined #openstack-infra18:18
fungitoo much noise in the check pipeline, whereas any bias away from success in the gate pipeline is an interesting statistic18:18
clarkbfungi: we should cleanup the non voting jobs in the gate if we do that18:19
clarkbthey will just make noise18:19
pleia2hmm, in replication.config, what happens if we set "authGroup = " no definition, for stanzas that don't need it? or can we do if statements in there?18:19
jeblairyep.  the data should all be there.  i think i'd like to try something like sparklines in the pipeline headers.18:19
clarkbfungi: but those numbers would be good to see18:19
jeblairpleia2: you can put if statements in there, there should be in-tree examples18:20
fungiclarkb: ahh, tracking voting vs. non-voting to filter those out maybe18:20
pleia2jeblair: ok thanks, I'll look around18:20
jeblairpleia2: (that compare to the empty string which should be exactly what you want)18:20
jeblairclarkb, fungi: ok we're not tracking voting/non-voting separately.18:20
jeblairi'm not entirely sure we'd want to...18:21
clarkbjeblair: fungi I was thinking of pulling them out of the layout file sections for gate18:21
jeblairclarkb: ?18:21
clarkbjeblair: just don't run them in the gate queue18:21
jeblairclarkb: ah yes18:22
jeblairclarkb: i think that's mostly the case right now?18:22
clarkbI think now that neutron is voting again the instance of that is low18:22
jeblairclarkb: anyway, yes, in that case, agreed, and should be simple.18:22
*** ladquin has joined #openstack-infra18:23
*** nijaba has quit IRC18:24
emaganafolks, is devstack broken?18:24
emaganakeep getting this error:18:25
emaganasqlalchemy.exc.OperationalError: (OperationalError) (1005, "Can't create table 'neutron_linux_bridge.routerroutes' (errno: 150)") '\nCREATE TABLE routerroutes18:25
clarkbemagana: it runs about a thousands times per day on the devstack gate slaves, but there have been issues around setuptools particularly on fedora/rhel/centos18:25
clarkbemagana: that particular error is not one I am familiar with. Perhaps your mysql/postgres permissions are not quite right?18:26
mgagneDo LP project groups have a bug tracker or is it just a container for other projects?18:26
emaganaclarkb: I just install a new ubuntu 12.04 and run devstack18:26
emaganaclarkb: I was expecting no errors at all  :-(  I am using neutron with linuxbridge plugin18:27
*** dkranz has quit IRC18:27
*** nijaba has joined #openstack-infra18:27
*** nijaba has quit IRC18:27
*** nijaba has joined #openstack-infra18:27
clarkbemagana: can you check which version of sqlalchemy is isntalled? `pip freeze` should show it18:27
jeblairmgagne: i _think_ it's just a container; possibly mordred or ttx could confirm with more authority18:28
clarkbI don't expect that to be the issue but there was recently talk of bumping the sqlalchemy version18:28
mordredmgagne: I belive it's just a contained - I think it might show you bugs associated with al lof its sub containers thought18:28
emaganaclarkb: sqlalchemy-migrate==0.7.218:28
*** dina_belova has joined #openstack-infra18:28
clarkbemagana: cool that is the version we want18:28
clarkbemagana: the neutron devstack testing is still fairly rudimentary aiui. Do you have anything special in your localrc related to neutron? It is possible devstack is trying to do something we don't test18:29
emaganaclarkb: will try another plugin, maybe this is neutron linuxbridge related18:29
HenryGWhen I do "git review -d 34642" I don't get the same code as I see here:
clarkbemagana: is the localrc we use in testing18:30
HenryGWhat am I doing wrong?18:30
clarkbemagana: note some of those options you will not want because they are related to testing like ERROR_ON_CLONE, but most of them should be fine if you want to copy that18:30
clarkbHenryG: the sha1's are different?18:31
*** SergeyLukjanov has joined #openstack-infra18:31
dansmithI'm seeing a new nova unit test failure related to neutron that seems to be affecting a lot of folks:
dansmithI don't understand it, and on top of the various other things going on right now, my patches are getting nailed 100% of the time18:31
HenryGclarkb: I did not check that. Where do I look?18:32
openstackgerritA change was merged to openstack-infra/config: Fix puppet-lint for bare puppet modules
clarkbHenryG: `git log -1 --oneline` will show you the local one18:32
clarkbHenryG: then you can compare to the on on gerrit18:32
emaganaclarkb: checking out the localrc18:32
clarkbdansmith: did neutronclient change their MockCLient version or something?18:32
jeblairHenryG: i tried locally and get the same sha1 in gerrit def56cfaeb9a093888396c5daae891431258cf2e18:33
*** nayward has joined #openstack-infra18:33
clarkbdansmith: it almost looks like neutron's code base expects one thing and nova's another18:33
dansmithclarkb: I dunno, I thought we were all unified now18:33
clarkbdansmith: mostly unified18:33
dansmithis that like "mostly pregnant" ?18:33
dansmith"officer, I'm mostly sober"18:33
clarkbwe allow ranges and don't currently require you update all projects when a requirement changes18:34
clarkbthat is changing though in part to avoid issues like the supposed one above18:34
dansmithclarkb: so the real question is.. can we blame this on sdague somehow?18:34
clarkbdansmith: no, he is fixing this :)18:34
dansmithcrap, okay.18:34
sdagueis this on unit tests ?18:35
HenryGclarkb: jeblair: thanks. I did "git review -u -d 34642" and that seems to have fixed it.18:35
jeblairsdague: yes18:35
sdagueright, I was hoping phase 2 was going to be able to wait longer, but I guess not18:35
sdaguedansmith: where's the review link?18:35
jeblairsdague: run every projects unit tests as requirements gate?18:35
clarkbdansmith: sdague actually, looking closer it appears that nova is mocking neutronclient?18:36
dansmithsdague: I checked a few of the ones in the check queue with failing unit tests and they're because of that18:36
clarkband the mock isn't complete enough to include EXTED_PLURALS18:36
clarkbdansmith: so possibly an cross test reaction where someone isn't cleaning up their mocks properly18:36
jeblairwhat's an exted?18:36
clarkbjeblair: no idea18:37
clarkboh even better18:38
*** vijendar1 has joined #openstack-infra18:38
clarkbdansmith: neutronclient changed the internals of the thing being mocked so the existing mock is no longer sufficient?18:38
*** vijendar1 has quit IRC18:38
*** vijendar1 has joined #openstack-infra18:38
*** vijendar has quit IRC18:38
emaganaclarkb: It seems that the problem is with linuxbridge plugin, basically something is broken during the alembic.migration18:38
jeblairclarkb: uploaded today18:39
clarkbjeblair: that'll do it18:39
*** dkranz has joined #openstack-infra18:39
jeblairdansmith, sdague: so possibly neutronclient released a new version that broke backwards compatability with nova unit tests18:39
sdaguejeblair: yeh, sounds like18:40
sdagueso this was one of the edge cases I was worried about that we hadn't hit yet, that python clients are out of git in the devstack nodes, but not in the unit tests18:40
sdagueso these kinds of breaks end up being allowed18:40
clarkbsdague: I think this would've been allowed anyways18:41
clarkbsdague: because the fail would show up after the thing causing the breakage18:41
clarkb(anyways = even if you pulled client from git)18:41
sdagueright, the issue is on the client landing side18:41
jeblairsdague: yeah, we are following the 'treat clients as external releases' model for unit tests18:41
sdaguejeblair right... so we've got another asymetric gate18:41
sdaguethe clients actually need to be gated on the unit tests of the projects that consume them as well18:42
clarkbits asymetric but making it symetric does not fix this particular edge case18:42
sdaguewhich has never been in the equation18:42
fungisdague: it would be asymmetric regardless, as clarkb points out18:42
clarkbit may fix others though18:42
clarkbsdague: oh, yes that would fix it18:42
sdaguefungi: not if python-neutronclient gates on nova unit tests18:42
jeblairsdague: well, i think it's more subtle than that; we've never treated unit tests as integration tests.18:42
fungisdague: oh, okay granted18:43
clarkbjeblair: and I think it is a good thing we don't18:43
sdaguejeblair: it is... however we're at 20 git trees that all need to work together18:43
clarkbbecause running `tox` locally shouldn't be complicated18:43
jeblairsdague: i think i'm in full agreement with you on the technical aspects; this can be solved in the way you describe.18:43
clarkbso one thing lifeless suggested in a mail list thread was that each clinet/server provide mocks to other people18:43
clarkbthen nova imports netronclient.mock18:44
*** CaptTofu has quit IRC18:44
clarkbthen we can gate the mocks with an integration test18:44
sdagueclarkb: I think it would just change the surface of the break18:44
clarkbsdague: it changes the surface of what you need to test18:44
*** CaptTofu has joined #openstack-infra18:44
clarkband gives us one mock per projects instead of nova's, keystone's etc that will each needfixing18:45
*** dkranz has quit IRC18:45
sdagueoh what a tangle web we weave....18:46
clarkbit doesn't outright solve the problem though18:46
jeblairdansmith: since we're about to get into the weeds as we do; do you have the actionable info you need?  i think the choices are (a) pin the neutron requirement (bad) (b) update nova to match (c) ask neutronclient to fix/revert and release an update18:46
*** sarob has quit IRC18:47
sdaguejeblair: I think he just ran to lunch18:47
jeblairsdague: probably a good choice18:47
*** sarob has joined #openstack-infra18:47
*** sarob has quit IRC18:48
clarkband if you want to live with a more limited set of corner cases run neutronclient unittests against that mock too18:48
clarkbwith the theory being that sufficient test coverage will uncover problems locally18:48
jeblairnati_ueno: it looks like the neutronclient release that included this commit has broken nova unit tests:
*** sarob has joined #openstack-infra18:49
jeblairnati_ueno: you can see the failure here:
nati_uenojeblair: gotcha do you have bug report?18:49
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add replication of git from gerrit to git.o.o
jeblairnati_ueno: i haven't seen one18:50
clarkbdansmith might18:50
nati_uenoOK I'll register it18:50
dansmithI didn't18:50
sdaguejeblair: heh, I just ran over the the neutron channel to ask about it18:50
sdaguebut now we have nati_ueno here, so it's all good18:50
nati_uenoThank you for your sharing issues18:51
nati_uenojeblair: how did you find is the cause?18:53
clarkbnati_ueno: it added new things to the client that nova needed to mock but hadn't18:54
nati_uenoI registerd
uvirtbotLaunchpad bug 1209364 in neutron "AttributeError: 'MockClient' object has no attribute 'EXTED_PLURALS' " [Critical,In progress]18:54
clarkbnotice that EXTED_PLURALS is missing at line 38, 33187 seemed to add that to neutronclient18:54
nati_uenogotcha let's me check18:56
nati_uenoOK I confirmed the patch is the cause18:57
*** ruhe has joined #openstack-infra18:58
*** ^d has quit IRC18:58
nati_uenomarkmcclain: I would like to revert 33187. because FWaaS is not fully merged on server side.18:59
nati_uenomarkmcclain: IMO, we should fix nova unit test before the patch is merged18:59
nati_uenomarkmcclain: How do you think?18:59
*** avtar has joined #openstack-infra18:59
*** dkranz has joined #openstack-infra19:00
lifelessclarkb: actually I suggested each project export *test doubles*, not mocks per se : actual mocks are per-test, not-per-codebase -> you have to fix every single mock on every single API change always19:01
lifelessmordred: so - run-mirror sadness.19:02
lifelessmordred: I'd like a hand with it, unless rsync is unblocked [I need something working]19:02
*** changbl has quit IRC19:05
*** vijendar has joined #openstack-infra19:05
markmcclainnati_ueno: looking19:05
nati_uenomarkmcclain: I updated bug report
uvirtbotLaunchpad bug 1209364 in neutron "AttributeError: 'MockClient' object has no attribute 'EXTED_PLURALS' " [Critical,In progress]19:05
*** vijendar has quit IRC19:06
lifelessclarkb: (but some times things that are not 'mocks' are labelled Mock* :)19:06
clarkblifeless: I think that is the case here :)19:06
*** vijendar has joined #openstack-infra19:06
lifelessmordred: some scrollback we should follow up on - and also and also you mentioned services in venvs not starting - do you need a hand? what symptoms do you have?19:07
uvirtbotLaunchpad bug 1209132 in openstack-ci "run-mirror fails to mirror cinderclient" [Undecided,New]19:07
*** vijendar1 has quit IRC19:08
*** ^d has joined #openstack-infra19:10
openstackgerritA change was merged to openstack-infra/config: Unset ZUUL_PROJECT on periodic devstack jobs
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added no-cache option
fungion the topic of run-mirror, this is probably a good time to rehash last night's discussion with mordred and lifeless about requirements enforcement/stable branch mirrors in the presence of clarkb, jeblair and sdague19:12
clarkbfungi: we are in a call, but happy to in about 45 minutes19:13
fungioh, no sweat. i can wait ;)19:13
jeblairlunch calls are evil19:14
pleia2less evil than 7AM :)19:14
lifelessIt *is* a 7AM call.19:14
pleia2sorry lifeless19:14
lifelesspleia2: :) At least it's not a 6AM call.19:15
nati_uenomarkmcclain: I pushed patch, if you don't want revert i'll abandon this patch
mgagnepuppet-lint is finally working on bare puppet modules :D19:15
jgriffithfungi: I do have strong feelings about the double negative logic statements :)19:15
fungijgriffith: you are entitled to them!19:16
fungisometimes code review is like group therapy, with red foam bats labeled "-1"19:17
jgriffithfungi: haha19:17
jgriffithfungi: stuff like that I just give my 2 cents but still +119:18
jgriffithfungi: not important enough or something that we have a set standard on.  Hopefully folks rethink, and often they do19:18
jgriffithgood enough for me19:18
sdagueman, chasing the project deps makes me understand the facebook model of every line of code in the company being in one git tree19:18
jgriffithsdague: :)19:19
*** vipul is now known as vipul-away19:19
*** vipul-away is now known as vipul19:19
fungibetter than every line of code being in it own separate git tree19:19
* jgriffith likes that idea 19:19
mgagneatm, is it normal/expected to have a ~45m delay between a "recheck" and the results?19:25
clarkbmgagne: we are seeing resource starvation because everyone else is doing the same thing :)19:26
clarkbmgagne: expected definitely when load ramps up like this. Maybe not normal though19:26
openstackgerritA change was merged to openstack-infra/config: Prepare to test git-review
*** nijaba has quit IRC19:27
fungiit's times like this when i wish i didn't have to keep cacti unviewable, because i'd love to see what the actual resource utilization is like on zuul and the jenkins masters19:27
fungistill no updated cacti packages though19:27
*** mrmartin has joined #openstack-infra19:28
* fungi uses ssh19:28
jeblairzuul load average: 0.31, 0.72, 0.7919:28
markmcclainsdague: we're going to make the code in neutronclient a little more forgiving of a bad mock for the time being… it is the least  bad of the solutions19:28
*** nijaba has joined #openstack-infra19:28
markmcclainsdague: long term we need to clean the mock in Nova but I'd like to fix a few warts in the client before doing so19:28
*** mrodden has joined #openstack-infra19:29
jeblairfungi: the jenkins webuis seem responsive, and there is no backlog on devstack-gate update jobs19:29
*** moted has joined #openstack-infra19:29
mgagneclarkb: according to the graph, it isn't "normal". I mean, "patchset created" isn't abnormally high atm. Unless slaves are clogged running lengthy tests. ^^'19:29
fungijeblair: yeah, the control servers seem to not be heavily loaded, so it's just availability of slave servers/pool size i guess19:30
*** ruhe has quit IRC19:30
*** mrodden1 has quit IRC19:31
jeblairwe could probably increase the number of slaves slightly, but i'd like to avoid doing it too much yet until we have a new devstack-gate system19:31
fungithe launch slaves also seem not at all loaded19:32
clarkbmgagne: I think that is a big part of it, there has been an extended period of running long running tests19:32
*** thomasbiege1 has joined #openstack-infra19:33
mgagneclarkb: would there be a way to buy a flash pass to avoid waiting in queue? :D19:33
clarkbthomasbiege1: hello19:34
fungimgagne: the approve button is the priority customer line19:34
*** avtar has quit IRC19:34
fungimgagne: puts you in the gate queue which gets first dibs on available test resources19:35
*** sandywalsh has quit IRC19:37
thomasbiege1I am investigating ways to add security tests (against a deployed openstack setup) to the openstack infrastructure, but for me it is not very clear where it would fit best. There is devstack, tempest and smokestack. Additionall deployment tests seem just to focus on integration validation. Maybe you have a hint for me?19:39
*** ^d has quit IRC19:41
jeblairthomasbiege1: perhaps they would be a good fit for tempest, which has a number of different test groupings (including long-running stress tests, for example)19:41
jeblairthomasbiege1: you may want to start by talking to sdague19:41
*** ^d has joined #openstack-infra19:41
thomasbiege1jeblair: thanks!19:42
fungithomasbiege1: also, a lot of the discussion around planning tests takes place in #openstack-qa so you might want to ask in there19:42
jeblairthomasbiege1: are you thinking of tests such as fuzz-testing, or something completely different?19:42
markmcclainsdague: fix working its way through the gate19:43
*** sarob has quit IRC19:43
thomasbiege1fungi: ah, thanks for the hint19:43
*** sarob has joined #openstack-infra19:43
thomasbiege1jeblair: yes, fuzz testing and web-security testing (based on the OWASP testing guide)19:44
*** changbl has joined #openstack-infra19:44
jeblairthomasbiege1: do you have a really rough idea of how long these tests might take (minutes, hours, days?)19:44
thomasbiege1jeblair: well the web-security test w/o fuzzing only need minutes19:45
* fungi imagines a devstack-gate based test which stands up horizon, installs a local openvas and scans it to generate audit reports19:45
thomasbiege1fungi: sounds good :)19:46
*** ^d has quit IRC19:46
jeblairthomasbiege1: so at least some of this could be considered for check and gate jobs; maybe the fuzz testing would need to run nightly due to length19:46
fungii guess actually something light like whisker would be sufficient against horizon alone, but openvas might be great to bombard all the devstack services19:47
jeblairthomasbiege1: at any rate, when you find the right home for them, we'll be happy to run them, and it sounds like it shouldn't be too much trouble.19:47
*** sarob has quit IRC19:47
thomasbiege1jeblair: what is the official/openstack way of introducing such test-suites?19:48
fungii'm personally extremely excited by the prospect of automated vulnerability scans against proposed changes, if it turns out to be feasable in check and gate jobs19:48
dtroyerthomasbiege1: be aware that DevStack's deployment of OpenStack is not considered secure out of the box.  I would love to change that19:48
jeblairthomasbiege1: we strongly favor consolidation into tempest if feasible (sdague and folks in #openstack-qa should be able to help determine that)19:49
*** sandywalsh has joined #openstack-infra19:49
sdaguemarkmcclain: cool, thanks much19:49
jeblairthomasbiege1: it will make it much easier to run that set of tests, as tempest is already well integrated into the infrastructure; and we run tempest on a deployment made by devstack -- so that's the kind of environment you can expect it to test against19:50
sdaguethomasbiege1: yes, tempest would be the right place. Probably either in the scenario subdir, or if it grows enough, have a dedicated security directory19:50
fungithomasbiege1: another good candidate which might be better placed as per-project testing would be static analysis for risky code (looking for potential format string vulrabilities for example)19:50
fungieither run the way we do coverage reporting, or maybe as a flake8 plugin if the false-positive rate is low enough19:51
thomasbiege1fungi: code scanning is also on my list19:51
thomasbiege1fungi: but I thnik this can go directly into Jenkins (at least this is the way we implemented it at our company)19:52
fungithomasbiege1: potentially, though depending on how quickly it runs, being able to lump it in with other static analysis tests might help avoid unnecessary additional resource consumption19:53
jeblairthomasbiege1: tempest has developer documentation here:
thomasbiege1ah, thanks for the pointer19:54
jeblairthomasbiege1: and if you want to read up on the project infrastructure, our docs are here:
thomasbiege1jeblair: read it already :)19:55
jeblairthomasbiege1: the sections on devstack-gate, jenkins job builder, and zuul may be interesting to help you ...19:55
jeblair :)19:55
jeblairthomasbiege1: that's never happened.  that's really cool.  :)19:55
thomasbiege1hahaha :-D19:55
fungiyou've just blown jeblair's mind by reading documentation before asking questions19:56
*** sandywalsh has quit IRC19:56
*** UtahDave has quit IRC19:56
thomasbiege1nevertheless knowing now the best place to start this little project helps me a lot. thank you guys19:56
jeblairthomasbiege1: you're welcome.  looking forward to it!19:56
fungiyou bet! i'm excited to see this come about19:57
*** dprince has quit IRC19:57
*** sarob has joined #openstack-infra20:00
lifelessmordred: ok so...20:01
mordredfungi: ok - clarkb and I are back -20:01
*** krtaylor has quit IRC20:01
*** isviridov has joined #openstack-infra20:01
mordredfungi: so we can talk about mirrors if you want20:01
fungimordred: cool20:01
mordredalso, apparently run-mirror doesn't work for lifeless20:01
fungiyes, that's the bug he opened i guess?20:02
fungii haven't tried to reproduce that yet20:02
fungijeblair: clarkb: sdague: it was suggested last night that maybe the newer strong requirements enforcement we have going into the gate obviates any need for separate stable-branch pypi mirrors, and that the gains from keeping that collapsed into one tree are worth not giving up20:03
clarkband if that is the case we can just run a full pypi mirror using something snatch20:04
clarkbor snap or snarf20:04
clarkbI forget the name20:04
jeblairfungi: i think i agree with that.20:04
jeblairfungi: the _current_ stable branches are worth considering...20:04
fungii know we've beaten the topic around previously, but the last time it came up was before the requirements enforcement work lurched forward so significantly20:04
mordredyeah. basically, I think I agree with what clarkb was originally saying20:05
mordredthat the mirror should not act as a part of the enforcement of the requirements20:05
sdaguefungi: ok, honestly I'd need to think through it. And I'm only a few days away from vacation, so my thinking hat is probably not in the state of usefulness20:05
jeblairin that they are not currently subject to brutal requirements enforcement20:05
clarkbjeblair: good point20:05
mordredjeblair: this is true - however, if we have a mirror which has less things than the outside world...20:05
mordredwe won't be testing what life will be like out of our world (which I think is what clarkb was originally trying to tell us)20:06
jeblairmordred: yes -- something needs to enforce requirements, it no longer needs to be the mirror.20:06
mordredsdague: I know your thinking hat isn't on ...20:06
jeblairmordred: i don't know what you're trying to say there talking about worlds20:06
mordredsdague: but what do you thinkabout backporting strong requirements enforcement20:06
fungisdague: keep your party hat on (i recommend using a lampshade, if available)20:06
jeblairmordred: perhaps i should finish what i was saying...20:06
mordredjeblair: ha. yes lease20:06
sdaguemordred: I'm +1 to unified requirements on stable20:07
sdagueif that's the question  you are asking20:07
mordredsdague: do you think we should just try backporting the patch to devstack stable/grizzly ?20:07
sdaguewe have requirements stable/grizzly/20:08
clarkbsdague: iirc yes20:08
* clarkb double checks20:08
fungisdague: i don't think the requirements list even needs to be unified on stable branches, more just that the mirror itself might not need to be separated out. but i want to wait for whatever jeblair's going to say which makes my assumptions silly after all20:08
sdagueso, here's the thing, I expect that requirements stable/grizzly is totally crap based on the state requirements was in, so my expectation is iteration would be required20:08
jeblairwe should probably _not_ clean up old packages in the mirror yet, which stable branches may be relying on20:08
mordredjeblair: +100020:09
clarkbsdague: probably so, the initial seed was very naive20:09
*** sandywalsh has joined #openstack-infra20:09
mordredthen additionally, I believe we were going to try to cap the folsom branch20:09
mordredso that one should actually be easyish20:09
jeblairand yeah, if we want to backport enforcement to devstack, and spruce up the requirements repo, that could be nice.20:09
sdagueclarkb: it turns out we're saved in the devstack-gate case by the fact that we never install the minimum, but always the maximum (unless we've got an apt dep)20:09
sdaguebut in real people's devstacks, it causes havoc20:10
sdaguethat was the reason to push through all those updates last night, after figuring out with mtreinish why g-r broke him20:10
mordredbtw - I'm going to re-push requirement update patches20:10
fungig-r... git-review? something else?20:10
fungiTHAT. okay20:11
clarkbso it sounds like we have agreement on not worrying about stable branch mirrors if we backport and cleanup20:12
clarkbwhat about possibly mirroring all the things if external links go away?20:12
mordredI think I'd lke to revisit that if/when external links go away20:13
jeblairclarkb: that would be fine, but possibly in contradiction to our tentatively accepted scope expansion of the mirror as a resource for devs and maybe deployers for things openstack needs.20:13
clarkb++ has the list gotten any shorter since we started advertising weshouldn't use those projects or get them to push to pypi properly?20:13
mordredclarkb: I believe some of them have hosted now20:13
clarkbjeblair: yes, I think we would want to make it a proper dedicated host in that case20:13
clarkbjeblair: it is definitely not as simple as running that mirror script20:14
mordredI mena- the question becomes, is a partial mirror more helpful than a full mirror20:14
mordredto our devs20:14
jeblairclarkb: i don't think that would help -- i don't believe lifeless and others want to rsync all of pypi.python.org20:14
clarkbmy personal interest is for a full mirror so that I don't have to change settings locally20:14
jeblairclarkb: change settings locally?20:14
fungiif they did want that, there are already scripts to do it (several at least)20:14
clarkbI am also not on the other side of the planet so I don't need a local cache20:14
clarkbjeblair: I use our mirror locally, but when touching python things that are not mirrored by our mirror I have to edit files20:15
lifelessclarkb: Full pypi is 53GB20:15
lifelessclarkb: I want the 400MB one that pypi.o.o is.20:15
*** adalbas has quit IRC20:15
mordredyeah - without partial mirror, the pbr integration test would be hard20:15
mordredI think that's my main usecase for the mirror being both small and reconsumable20:15
*** dkranz has quit IRC20:15
mordredif I can grab a copy of the mirror at the top of that20:15
clarkbwhat if we had two mirrors (I am hating this already) the full mirror which is consumed by the subest mirror20:15
jeblairclarkb: lifeless being on the other side of the planet is not a good reason for us to run a mirror, but he cited a need of devs who are on this side that could benefit from it, and others chimed in and said it would help with their site-local mirror needs too.20:15
fungiclarkb: you shouldn't need to use our mirror any longer since we're not mirroring unreleased tarballs on it any longer, which was the only reason i used to point to it for tests20:15
mordredand then just inject the local to-be-tested pbr20:16
mordredthat would be great20:16
clarkbfungi: I use our mirror because it is better than pypi20:16
clarkbyou should try it, much more reliable and fast20:16
fungiah, that's a different argument entirely ;)20:16
mordredif I have to try to inject the to-be-tested pbr intoa full mirror20:16
mordredto test that packages can find and install it automatically properly20:16
mordredthen things just got honestly quite difficult20:16
mordredso, since we have a partial mirror pretty much working, I would be quite sad if it went away20:17
mordredbecause it would make testing some things harder20:17
clarkbmordred: agreed20:17
jeblairmordred: when we can rsync the mirror, perhaps that should be a devstack-gate image prep step so most packages are local.20:17
clarkbI am beginning to think that a full mirror would need to support a partial mirror too which is extra work20:17
clarkbsince what we need is the partial mirror20:17
mordredjeblair: +10020:17
mordredjeblair: I think that would be great20:17
fungiin general i concede that we probably do want to continue having a partial mirror. the question at hand is whether we want more than one20:18
jeblairthomasbiege1: drop in anytime!20:18
jeblairfungi: one is enough for me!20:18
mordredfungi: I'm not sure I can state any currently compelling arguments for two20:18
mordredor more20:18
fungijeblair: by "more than one" i mean mirror per stable branch20:18
mordredI do think we need to backport reqs stuff to stable/* - but I'll work on htat20:18
clarkbmordred: my only compelling argument is that a full mirror makes local use of the mirror easy20:19
fungiso anyway, based on last night's rethink, i mostly need to know whether i continue down the path of separating out different mirrors for each branch after all, or whether that effort would be better spent on something else (improving the mirroring implementation in general for more robustness, requirements enforcement improvements, whatever)20:19
clarkbmordred: for general pythoness and that is low priority because it isn't strictly openstack related20:19
mordredclarkb: yup20:19
mordredfungi: I think the two things in ()s in there would be more useful overall20:19
jeblairfungi: yep, i say abort.20:20
fungiokay, so i'll update the existing bug on branch mirrors as being an obsolete direction, and then reroute my efforts to those other bits accordingly20:20
* clarkb can find lunch now20:21
funginow, as to lifeless's run-mirror bug report... any ideas on that issue? i can try to reproduce and dissect, but we likely want to narrow that down to a more specific failure20:22
*** pabelanger has quit IRC20:22
*** krtaylor has joined #openstack-infra20:22
*** thomasbiege1 has quit IRC20:22
*** thomasbiege1 has joined #openstack-infra20:23
*** sdake has quit IRC20:23
lifelessso my run-mirror thing is so that I can get a test mirror up and running and not be blocked. It's probably more interesting to push an official mirror forward faster than spend time on my issue. Buttt.... if the official mirror needs a few more days, I'd really appreciate a pointer/aid on that bug.20:23
fungibug 120913220:23
uvirtbotLaunchpad bug 1209132 in openstack-ci "run-mirror fails to mirror cinderclient" [Undecided,New]
*** boris-42 has quit IRC20:24
fungiand it's also worth repeating for the present crowd that run-mirror's current failure modes make assumptions which aren't actually good when it comes to py3k support20:25
fungiexample run:
lifelessohho this might be it20:27
lifelesscat .pydistutils.cfg20:27
lifelessindex_url = file://20:27
lifelessrogue configu files20:27
*** nijaba has quit IRC20:27
lifelesslets see20:27
fungithat looks like an easy_install optimization to try and use a local on-disk replica20:27
*** sdake has joined #openstack-infra20:28
fungiand does seem to correspond with the exception you quoted20:28
*** nijaba has joined #openstack-infra20:28
*** nijaba has quit IRC20:28
*** nijaba has joined #openstack-infra20:28
*** dkranz has joined #openstack-infra20:29
fungiha, the script output in your paste even says "Please make sure you remove any previous custom paths from your /home/ubuntu/.pydistutils.cfg file.20:29
lifelesswell there was so much crap in there how could I tell :)20:30
*** sdake has quit IRC20:30
fungiit is rather wordy20:30
*** sdake has joined #openstack-infra20:31
*** dina_belova has quit IRC20:34
fungithis is a beautiful sight, btw...
openstackgerritMathieu Gagné proposed a change to openstack-infra/jenkins-job-builder: Make parameter description optional
clarkbfungi: !!!!!20:38
fungiyes, !!!1!eleven20:38
*** sdake has quit IRC20:38
*** sdake has joined #openstack-infra20:38
*** sdake has quit IRC20:38
*** sdake has joined #openstack-infra20:38
fungiclarkb: though for some reason is still not generating a testr_results.html when run from .tox/py33/bin/python20:40
*** cppcabrera has joined #openstack-infra20:41
lifelessI think it would be interesting to stream the subunit straight out and process offline into html20:41
*** cppcabrera has left #openstack-infra20:41
*** ^d has joined #openstack-infra20:41
fungiyeah, and subunit_log.txt exists, so it's something non-obvious happening within to cause it to silently fail there20:41
lifelessnext thing...20:43
lifelessUnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 62: ordinal not in range(128)20:43
lifelesspip install did not indicate success20:43
lifelesslooks like it's compiling an LDAP module20:43
fungitoo fun20:44
fungiand yes, it will compile things. many nasty things20:44
*** mrmartin has quit IRC20:45
fungibecause it has to run setup for each thing it downloads to make sure it catches any other python packages which get pulled in by $random_maintainers_setup_madness in them20:45
*** ^d has quit IRC20:46
fungione of those things which will hopefully become much simpler in that utopian future where every python package uses consistent tooling20:46
s1rplifeless: just a heads up: in nova at least ldap as test-requirement is going away
fungii suspect it's not going away for keystone though20:49
*** thomasbiege1 has quit IRC20:53
openstackgerritMonty Taylor proposed a change to openstack/requirements: Revert "remove netifaces as a requirement"
mordredsdague: ^^20:54
*** ^d has joined #openstack-infra20:54
*** ^d has joined #openstack-infra20:54
lifelesss1rp: sure, but what binary package do I need to be able to build it /20:54
clarkbfungi: I can take a look at the subunit stuff20:56
*** pabelanger has joined #openstack-infra20:57
clarkbfungi: I am just going to go into that workspace and fiddle with running subunit2html. I assume this is ok as you won't be running that test a whole bunch20:57
sdaguemordred: ok, why?20:59
fungiclarkb: yeah, i'm actually running it a few times, but it runs quickly20:59
mordredsdague: swift still uses it20:59
mordredsdague: the patch to remove it has not, in fact, landed20:59
mordredand is being contested20:59
sdagueoh, ok20:59
mordredsdague: for the record20:59
*** melwitt has joined #openstack-infra20:59
sdagueso basically right now the gate is bust if any nova patches go to it21:00
sdagueany idea where that fix is sitting for this?21:00
jeblairsdague: i believe is the fix21:01
jeblairsdague: which is at the head21:01
jeblairsdague: though i believe a neutronclient release will need to be made after that21:01
sdagueok, but that requires a neutron client release as well21:01
sdaguethere was no way to fix it on the nova side?21:01
openstackgerritlifeless proposed a change to openstack-dev/pbr: Fix python-ldap mirroring.
jeblairsdague: i don't know if anyone looked into that21:02
sdagueoh, ok, that's what I thought was being looked at... :)21:02
*** ArxCruz has quit IRC21:02
sdagueoh well21:02
clarkbfungi: I am beginning to think it may be a subunit bug21:02
clarkbfungi: the script runs to completion. No exception thrown (that bubble out at least)21:03
clarkbunless maybe it has to do with some weird bytes vs string thing21:03
fungiclarkb: is the file opened with an encoding specified (that's usually the first thing to bite me)?21:04
*** thomasm has quit IRC21:04
*** vipul is now known as vipul-away21:05
*** vipul-away is now known as vipul21:05
dansmithsdague, clarkb: it's scrolled off my buffer now I think, but what was the outcome?21:05
fungithough i think that's one of the methods which are new in 2.7 so we probably need a conditional21:05
clarkbfungi: no, but the write encodes strings21:05
jeblairmordred: ping21:05
sdaguedansmith: neutron client revert, then new release21:05
sdaguethen things should be ok again21:05
clarkbfungi: and that should be compatible with 2.621:05
mordredjeblair: pong21:05
fungiyeah, hrm21:05
dansmithso, we're just sunk right now?21:05
sdaguethough I actually think it would be fixable with just nova patches as well21:05
sdaguedansmith: yep21:05
dansmithalrighty then21:06
lifelesswell, that patch won't be enough, but will iterate21:06
jeblairmordred: so i'm starting rough-in work on the new devstack-gate pooling code21:06
jeblairclarkb, fungi: you too21:06
mordredjeblair: ooh, neat21:06
jeblairwill you take a look at this proposed config file:
jeblairand tell me if that looks reasonable...21:07
jeblairmordred: i want to make sure i accomodate a possible move to dib21:07
*** isviridov has quit IRC21:07
jeblairmordred: so i'm expecting the "configuration:" entry to be a shell script that does what the image script currently does for now21:07
jeblairmordred: but possibly something else that specifies how dib could build the image in the future21:08
fungijeblair: yaml! looks like what we want--has the right info and organized sanely21:08
mordredyeah - we could even collapse _this_ file in to not having a configuration entry21:08
jeblairmordred: so in addition to anything else you think of, specific attention to forward-comptability there would be good21:08
mordredwe could keep this as a config file for both things21:08
mordredif we put the elements list into configuration for dib21:09
jeblairmordred: i think (without knowing the terms) that's what i was thinking21:09
*** lcestari has quit IRC21:10
fungijeblair: i think you inadvertently duplicated your providers key under targets images21:10
jeblairmordred: so maybe we add something like "elements: [puppet, git, pypi-mirror]" later?21:10
clarkbjeblair: can we abstract the provider such that maybe d-g can spin up and connect to gearman directly?21:10
clarkbjeblair: maybe that means having a provider type21:10
jeblairfungi: yes, copy/paste error there21:10
clarkbI might be thinking too far ahead here21:11
mordredthen base-image could be the base image to use - which might want us to have an image-name21:11
mordredoh - but there is one21:11
mordrednevermind - I think that's all we need21:11
clarkbbut something like that would make it useful for general worker pool management21:11
clarkbeg logstash21:11
*** SergeyLukjanov has quit IRC21:11
mordredjeblair: yes21:11
sdagueso... I had this other idea, last one for the day before I run off21:11
fungijeblair: but otherwise lgtm and should be extensible enough if we want to add more there later21:11
sdagueto make sure that global-requirements actually worked at the minimums, have a version of the job which runs tempest gate with min versions21:12
sdaguebecause in the base case we basically run at max21:12
sdagueso we only test one end of the range21:12
jeblairsdague: sounds good21:12
fungijeblair: okay, cool. that's what i assumed you meant there21:12
mordredsdague: neat21:12
* zaro is addicted to looking at activity on jenkins01 & jenkins0221:13
sdagueok, maybe I can actually get that nailed before I start road tripping next week :)21:13
fungisdague: that would need more work on the mirror script to make sure we actually provided those minimum package versions21:14
sdaguefungi: yes, probably21:14
jeblairclarkb: so maybe we do that in the way i just added?21:14
sdagueit's going to need a version of update that provides just the min versions21:14
jeblairclarkb: though specifically for logstash, that might be something that we could just handle with heat21:14
sdaguethat should provide the contents of the list21:14
sdagueanyway, I'll start digging on it tomorrow21:14
jeblairclarkb: though if we start spinning up single-use workers that aren't attached to jenkins, we'd need that21:15
clarkbthe etherpad looks good21:16
clarkbwe can key off the jenkins-url and have a gearman-server uri21:16
jeblairclarkb, mordred: i added a 'cleanup' line to the provider-image section -- so maybe we do something like that to tell it to try to run kexec to cleanup21:16
clarkbsomething like I added21:16
*** vijendar1 has joined #openstack-infra21:17
*** dkranz has quit IRC21:17
mordredjeblair: yeah. maybe so21:17
fungisdague: one more wrench in the works... what's the minimum version to test for a package with no specified minimum version in the requirements list?21:17
jeblairclarkb: presumably a logstash node would know how to connect to the logstash gearman server though21:17
*** anteaya has quit IRC21:17
clarkbjeblair: maybe? I am not exactly sure how much data you plan to push into the node from here21:18
clarkbI think either way works21:18
jeblairclarkb: i don't want to be heat.  :)21:18
clarkbthats fair21:18
jeblairand when/if heat can maintain pools, i'm happy to start using that21:19
jeblair(though our case is more subtle -- we want pools of _ready/idle_ nodes, not just pools of live nodes)21:19
fungisdague: also the minimum version specified may provide a lower bound while not specifying an actual released version number explicitly (for example >=2 where 2.0.0 was the actual released package version)21:19
*** vipul is now known as vipul-away21:20
*** vijendar has quit IRC21:20
jeblairfungi, sdague: at least for the case of gating a new requirement, we can reject ones that don't match or don't have a min specified.21:21
jeblairfungi: the issue still holds for something that aimed to run all the requirements at min; so that may be a cleanup item.21:21
fungijeblair: fair. and maybe we go back and clean up the existing ones lacking a lower bound or without a lower bound on a real released version21:21
jeblairmordred, clarkb, fungi: ok, thanks.  back to my hacking hole.21:22
mordredjeblair: in the hole!21:22
fungithe hacking hole. has a nice ring to it21:22
mordredjeblair: actually - I think our case isn't too subtle...21:23
fungimight be a good name for a beer bar/coffee house/cyber cafe/hacker space venture21:23
mordredjeblair: or different. "live" for us means "in a state ready to run devstack"21:23
clarkbfungi: progress! pdb is my friend21:23
clarkbfungi: in _generate_report_test there is string/byte decode/encode happening21:23
clarkband that breaks under py3k with an AttributeError. Not sure why that doesn't bubble out or what is causing it yet21:24
lifelesskindy run,back soon21:24
jeblairmordred: i don't either, but the thing that checks the low water mark needs to know that something that has started to run a devstack test is not live anymore (despite it needing to exist as a nova server for another hour at least, and possibly might become live again after kexec)21:24
mordredjeblair: +21:25
*** nicedice_ has joined #openstack-infra21:25
fungiclarkb: huh--good catch21:27
*** nijaba has quit IRC21:27
*** nijaba has joined #openstack-infra21:28
*** nijaba has quit IRC21:28
*** nijaba has joined #openstack-infra21:28
clarkbfungi: I think I can just remove that section. I am testing21:29
mordredjog0: ping21:31
jog0mordred: pong21:31
fungii hadn't thought about it this way before, but our devstack pools are managed in much the same algorithm as apache's worker process pools... try to maintain at least x available workers and don't let it exceed y total workers21:32
fungiand also don't keep more than z available workers21:32
fungithought i guess we don't do that last bit (yet at least)21:33
*** UtahDave has joined #openstack-infra21:33
mordredjog0: wanted to do high bandwidth about requirements sync for hacking21:33
fungibut would probably want to if we ever start moving nodes back from the used pool into the available pool21:33
mordredjog0: two things - a, we're close to landing a script that will auto-propose requirements patches when requirements changes21:34
mordredjog0: b- I can do a larger commit message this time, but moving forward, it's going to be a bot making these, that ok with you?21:34
jog0mordred: yes and yes21:34
mordredjog0: ossum21:35
jog0mordred: I figured as much, I am just want to make sure got logs are readable by someone who doens't know the context21:35
fungiafter the first one, he can simply berate the bot for not providing sufficient context in the commit message ;)21:36
openstackgerritMonty Taylor proposed a change to openstack-dev/hacking: Synced with global requirements
jeblairfungi: we have bots that can berate our bots.  :)21:36
* jeblair loves the idea of jog0's avatar in hacking -1ing monty's avatar in requirements-update.21:37
mordredjeblair, clarkb: could I get a hell yeah on :
openstackgerritKhai Do proposed a change to openstack-infra/jenkins-job-builder: ignore instead of fail when reading empty config files
jeblairmordred: does it work?21:38
jog0jeblair: :)21:38
mordredjeblair: it will once we land the pep8 job patch to requirements21:38
*** jerryz has joined #openstack-infra21:38
mordredjeblair: which I've been asked to hold off on until we land this patch21:38
jeblairmordred: which is ready?  the intent is to block in on that.21:38
jeblairi aprvd21:39
*** avtar has joined #openstack-infra21:40
zarofungi: would you mind triaging bug 1208901 ?21:40
uvirtbotLaunchpad bug 1208901 in openstack-ci "deploy jenkins plugin pom.xml file " [Undecided,New]
fungizaro: you bet21:41
zarofungi: i've already got a patch on gerrit but comments don't show up on the bug.21:41
fungizaro: that seems wrong. how about instead i track down why that didn't get updated21:41
zarojeblair: got time to review  ?21:42
fungizaro: which was the patch?21:42
zarofungi: ohh yeah. that would be nice.21:42
zarofungi: one i just mentioned to jeblair21:42
fungiahh, the one you just linked21:42
openstackgerritA change was merged to openstack-infra/config: Added pep8 checks to requirements
zarofungi: thought it was because bug needed to be triaged first, but that does seem wrong.21:43
zarofungi: you can review that one too if you've got time.21:44
fungizaro: right, does not need the bug to be triaged21:44
*** datsun180b has quit IRC21:44
*** sarob has quit IRC21:44
fungii'm going hunting in the gerrit logs again21:44
*** dina_belova has joined #openstack-infra21:44
*** sarob has joined #openstack-infra21:45
*** avtar has quit IRC21:45
*** dina_belova has quit IRC21:47
*** krtaylor has quit IRC21:47
openstackgerritClark Boylan proposed a change to openstack-infra/config: Make py3k and python2 compatible.
clarkbfungi: ^ I used the wrong topic branch for that >_> but that shouldn't affect anything21:48
*** sarob has quit IRC21:49
*** vipul-away is now known as vipul21:50
*** avtar has joined #openstack-infra21:51
*** woodspa has quit IRC21:51
markmcclainok.. new neutronclient in pypi that should enable the nova unittests to pass21:53
mordredmarkmcclain: woot!21:53
mordredjust so you all know - I'm currently testing the update-requirements script by running it by hand and havingit submit changes21:55
*** rfolco has quit IRC21:56
mgagneDo you mind if I tag bugs related to JJB with jjb?21:57
mgagnebut I don't think there's more than 1 bug tbh21:58
clarkbmgagne: fine wiht me21:58
fungimgagne: works for me. doesn't look like we have an existing tag for jenkins job builder21:58
fungiso jjb's as good as anything there, i suppose21:58
mgagneIs jjb fine with you or do people prefer the long form?21:58
*** nayward has quit IRC21:59
nati_uenosdague: jeblair: neutronclient fix is merged. Nova is fixed now?21:59
*** vijendar1 has quit IRC22:01
openstackgerritClark Boylan proposed a change to openstack-infra/config: Make py3k and python2 compatible.
clarkbfungi: Alex_Gaynor ^ now with proper topic and file closing22:02
*** dkranz has joined #openstack-infra22:02
Alex_Gaynorclarkb: Awesome, I'm no longer qualified to review :)22:03
clarkbfungi: I have generated html files with one subunit log under python 2.7 and python3.2 and under python2.7 using the old version22:03
clarkbthere is no diff between new and old python27 outputs, There is a slight diff with py3k and I think it is related to how classes are repr'd as strings22:03
fungimakes sense22:04
*** mriedem has quit IRC22:04
fungialso, neat use of "with" to effect the equivalent of close()22:04
clarkbfungi: I prefer it in the general case so that you don't have to worry about exceptions and branching logic22:05
fungiahh, yeah leaves it up to runtime gc to take care of for you. very neat22:06
fungii'll have to remember to leverage that trick in the future22:06
*** mrodden has quit IRC22:08
*** fbo_away is now known as fbo22:08
*** _TheDodd_ has quit IRC22:09
*** carlp-away is now known as carlp22:09
*** arosen1 has quit IRC22:11
*** arosen1 has joined #openstack-infra22:14
pabelangerNot sure the place to ask, hopefully somebody will point me in the right direction.  Am I correct is seeing that test repository (the test runner specifically) launches unit tests in parallel by default?22:16
zaroclarkb, fungi : talk about
clarkbpabelanger: yes22:17
clarkbpabelanger: well not by default, but by how we run it (with the --parallel flag)22:17
clarkbpleia2: it will fork X test runners where X is the number of available cpus giving each runner some partition of the test set22:17
zarofungi: sorry but i'm not a fan of your suggestion. putting a job in a time window doesn't seem good to me.22:17
clarkbpabelanger: ^ whoops22:17
clarkbzaro: I don't like the suggestion either as 24 hours off isn't major imo22:18
zaroclarkb: would it be better to create a project with python-grizzly-bitrot-jobs and folsom-bitrot-jobs then call the job in layout.yaml22:19
clarkbzaro: that said, I think this is a less problematic issue when we push all of this extra requirements checking down into the stable branches22:19
pabelangerclarkb, okay, I'm actually trying to get them to run sequentially but actually seeing some run in parallel for some reason22:19
pabelangermoving to nosetest works as I expect22:19
clarkbpabelanger: we want them to run in parallel :) if you really need sequential for debugging reasons you can source the venv and do `testr run`22:19
*** fbo is now known as fbo_away22:20
mordredpabelanger: why you want sequential?22:20
clarkbzaro: I am beginning to think we may be able to just let this die on the vine22:20
*** burt has quit IRC22:20
zaroclarkb: ohhh??22:20
clarkbzaro: because we will end up testing this stuff through other mechanisms22:20
clarkbzaro: see the discussion that began at about 1pm today about the mirror and stable branches22:20
clarkbzaro: basically we will add extra machinery around the stable branches to keep them from bitrotting against the requirements22:21
zaroclarkb: cool.  how do we let stuff die in gerrit?22:21
*** sarob has joined #openstack-infra22:22
clarkbzaro: I would make sure fungi agrees as he has all of this stuff in his head right now. But you can abandon that change and update the bug if this is the route we want to take22:22
pabelangerclarkb, mordred: I should have prefaced, this is for my own app.  Dealing with an race condition to an external redis server (yes, integration testing using unit tests)22:23
pabelangermoving sequential is the simple solution for now22:23
zarofungi: d'you get that?22:24
clarkbpabelanger: I would solve that by adding different redis queues for each test runner22:24
clarkbpabelanger: each runner can create one called runner_$PID or something22:24
clarkbpabelanger: then they won't trample each other22:24
clarkbs/queues/channel/ or whatever redis calls it22:24
*** carlp is now known as carlp-away22:25
clarkbjeblair: do you want to do any additonal testing on that before we approve it?22:25
clarkbjeblair: I am not terribly worried about it, maybe we merge it during a quiet time22:25
clarkbpabelanger: that is the approach we have been working towards when tests need to talk to DBs. each process gets its own schema22:26
jeblairclarkb: agreed22:26
pabelangerclarkb, Ya, thats likely what I will end up doing.  Just got side tracked with testr and the race condition.  Thanks for the pointers22:27
*** nijaba has quit IRC22:27
mordredsdague: we have a bug in the update script22:28
*** jhesketh has joined #openstack-infra22:28
*** nijaba has joined #openstack-infra22:29
fungizaro: sorry, had stepped away for just a moment22:29
*** changbl has quit IRC22:30
fungizaro: i'm not sure putting the bitrot jobs as children of the periodic mirror jobs in layout.yaml will work. i suspect that zuul's timed triggers may be more naive than that, but jeblair will know for sure22:31
jog0anyone seeing this kind of error:
fungizaro: clarkb: i'm not convinced that backporting requirements changes to the stable branches will solve the bitrot-job-catches-upstream-release-breakage issues which the bug in question was filed to address by running bitrot jobs sooner after mirror updates22:33
clarkbjog0: yes, it was caused by a recent neutronclient release22:33
*** prad_ has quit IRC22:33
*** weshay has quit IRC22:33
clarkbjog0: neutron has since released a new version of neutronclient that reverts the changes made to the client that break nova testing22:33
clarkbjog0: tests after the new version is in the mirror (which is now I think) should pass22:34
* clarkb double checks the mirror22:34
clarkb verion 2.2.6 is the one we want and it is present22:34
fungizaro: clarkb: basically my interpretation of that bug report was an attempt to shorten the window between when the mirror pulls in new upstream releases and when the bitrot jobs report that something broke because of it, so having them not run almost 24 hours after the mirror update would give more of a leg up on timely response and fix22:35
clarkbfungi: correct, but as part of the no stable mirror branches we will be directly testing this stuff22:36
jog0clarkb: gah thanks22:36
clarkbfungi: so we don't need to run the bitrot jobs after $thing happens, there will be dedicated jobs that servce this purpose22:36
Alex_GaynorDo we keep track of how many fails occur in the gating queue and on which projects/test suites/etc.?22:36
fungiclarkb: i can probably be convinced that we'll be running the equivalent of bitrot jobs more often than every 24 hours, and that makes the bitrot jobs themselves no longer necessary22:37
clarkbfungi: at least that was my understanding of this afternoons conversation22:37
clarkbAlex_Gaynor: we do22:37
*** thomasbiege1 has joined #openstack-infra22:37
clarkbAlex_Gaynor: let me get you a link22:37
clarkbfor example22:37
jeblairAlex_Gaynor: you can go spelunking in for more22:37
fungideath by url22:37
clarkbfungi: sorry22:37
Alex_Gaynorhas anyone looked at setting up a gdash instance with common graphs?22:38
* fungi has never heard of such a thing22:38
clarkbI haven't but it looks shiny22:38
fungiho, nifty22:39
jeblairAlex_Gaynor: the graphs at the bottom of the zuul status page come from graphite; i think it would be nice to add them to other pages there22:39
*** thomasbiege1 has quit IRC22:39
openstackgerritlifeless proposed a change to openstack-dev/pbr: Fix python-ldap mirroring.
fungiit does indeed seem full of fancy22:39
jeblairAlex_Gaynor: the jquery module it uses makes it pretty easy22:40
clarkbjeblair: I am ferrying people from the airport tonight/this evening and don't expect to have much keyboard time until Sunday... I can approve 39995 on Sunday if no one beats me to it22:44
jeblairclarkb: roger22:44
fungizaro: your change for bug 1208901 seems not to have resulted in any exceptions from so i wonder if maybe your bug mention in the commit message is insufficient to trigger it to update the bug at all22:46
dansmithclarkb: so we should be rechecking now?22:46
uvirtbotLaunchpad bug 1208901 in openstack-ci "deploy jenkins plugin pom.xml file " [Undecided,New]
Alex_Gaynormordred: I assume it's not intentional that has 4 commit-ids?22:47
zarofungi: that would be odd, because that's my typical message. i've written the same message on many other patches.22:49
clarkbdansmith: I think so, maybe try one change first and if that passes do them all22:49
dansmithclarkb: was there a bug filed?22:49
clarkbdansmith: I didn't file one but I think nati_ueno did22:49
zarofungi: it usually goes like "This is a fix for bug ..."22:49
uvirtbotLaunchpad bug 1209364 in python-neutronclient "AttributeError: 'MockClient' object has no attribute 'EXTED_PLURALS' " [Critical,Fix committed]22:49
dansmithclarkb: thanks22:49
nati_uenodansmith: could you share if it working now?22:50
nati_uenodansmith: code is fixed22:50
clarkbmirror is updated too so all the pieces should be in place22:50
dansmithnati_ueno: just reverified this:
dansmithnati_ueno: so we'll see :)22:51
zarofungi: however.. i do notice a period at the end of the bug number, not sure if that makes a difference but i don't usually end with a period.22:51
nati_uenodansmith: Thanks!22:51
clarkbzaro: that may break the regex22:52
clarkbzaro: also you should use the new syntax :)22:52
fungizaro: with _TheDodd_'s recent header implementation in the script, it might no longer match on precisely the same sentences it did before22:52
clarkbcloses-bug: #XXXXXX22:52
fungiwhat clarkb said22:52
fungii suspect the "to" between "fix" and "bug" might be the culprit there22:53
fungibut need to look back over the new patterns more closely22:54
fungii think <prefix> is one word bounded by whitespace22:56
*** rfolco has joined #openstack-infra22:56
fungiso if you had "Fixes bug bug 1172419." at the start of a line (that's what i used to have in mine as a one-liner paraghraph) it would match22:57
uvirtbotLaunchpad bug 1172419 in openstack-ci "build bitrot jobs after mirror update" [Low,In progress]
fungier, s/bug bug/bug/ there (paste error on my part)22:57
zarofungi: check this one out:
zarofungi: that one worked with period and all. message didn't even start with "Fixes.."22:59
zarofungi: but in any case i will start using new syntax.22:59
fungizaro: yeah, remember that the new regular expressions went into last week. that change you just linked is a couple months old22:59
zarofungi: i see.23:00
fungiwhat i'm trying to say is that _TheDodd_'s implementation was only mostly backward compatible, not 100% backward compatible23:00
clarkbpleia2: fungi: any reason to not go ahead and approve that?23:00
pleia2clarkb: that would should be fine23:01
pleia237794 will require baby sitting (if it messes up the gerrit replication file for git, we will all be unhappy)23:01
clarkb40253 approved23:02
clarkbpleia2: just people that use github >_>23:02
clarkbI suppose the local replica would be affected too...23:02
fungii'm okay with babysitting and manually fixing gerrit this evening if 37794 causes issues23:02
pleia2clarkb: heh, yeah23:02
clarkbfungi: let me review 37794 then23:02
clarkband you can babysit it :)23:02
pleia2fungi: that would be great, I'll be around most of the evening (have a 12:40 flight tonight to philly, so I have my evening before airport)23:02
mgagneclarkb: XML element is empty as in <description/>23:03
openstackgerritA change was merged to openstack-infra/config: Add httpd ssl support to
clarkbmgagne: is that not equivalent?23:03
clarkbmy knowledge of the intricasies of xml is not good23:03
mgagneclarkb: a pedantic would say no, but in the end, I guess it makes no difference23:04
fungibut i can try pulling 37794 onto review-dev first for good measure23:04
pleia2fungi: if you have time, it may be worthwhile23:04
mgagneclarkb: I'll test and see if the UI is impacted23:04
fungipleia2: i'm making time to do that right now23:05
clarkbmgagne: perfect23:05
pleia2fungi: thanks :)23:05
*** nayward has joined #openstack-infra23:06
*** pentameter has quit IRC23:06
*** lcheng has quit IRC23:08
fungipleia2: bad news :(23:08
clarkbpleia2: 37794 reviewed23:08
fungipleia2: "23:08
fungierr: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter replication at /opt/config/fungi/modules/openstack_project/manifests/gerrit.pp:176 on node review-dev.openstack.org23:08
clarkbpleia2: fungi: I think one of my comments is half the story there23:08
clarkbpleia2: you have $giturl whcih should be removed and need $replication23:08
pleia2clarkb: ah, I missed one!23:09
fungipleia2: while i'm testing, if you can get that change uploaded fairly soon i'll re-test23:09
*** nayward has quit IRC23:10
fungiwe'll get the configuration applying cleanly and then i'm happy to approve and keep an eye on it23:10
*** wu_wenxiang has joined #openstack-infra23:10
* fungi will brb23:11
clarkbdstufft: I am going to put my mordred hat on23:15
clarkbdstufft: how is not a bug if you may need to require new pip23:15
clarkbdstufft: specifically mordred wants to require new pip everywhere. `pip freeze` should reflect that23:15
clarkbalso pip list and pip freeze seem redundant23:16
dstufftpip list does more than just show the current installed version23:16
dstufftwe talked about making freeze be pip list --freeze23:17
openstackgerritMathieu Gagné proposed a change to openstack-infra/jenkins-job-builder: Fix handling of optional parameter description
dstufftbut nobody ever did it and those names already existed23:17
clarkblist is new23:17
clarkbI had to upgrade to 1.4 to get it23:17
clarkbanyways meh23:17
dstufftclarkb: er I mean pip freeze already existed23:18
clarkbmordred: ^ that may potentially be a problem if you want ot require pip 1.4 everywhere23:18
dstufftclarkb: I'm not sure how that'd work though23:18
dstufftif you're on pip 1.3, and you did pip install -r requirements.txt and it has pip==1.4.123:18
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add replication of git from gerrit to git.o.o
dstufftwhat is pip suposed to do?23:18
clarkbdstufft: upgrade pip23:18
dstufftand then exit?23:18
dstufftit can't upgrade the currently running code23:19
clarkbno, I think pip has already established it cannot handle upgradin setup requires and setting up a thing in the same process23:19
clarkbdstufft: the current behavior that requires you to run pip twice is probably fine23:19
clarkbdstufft: eg if you require new setuptools23:19
dstufftclarkb: I don't think there's a sane way of doing it otherwise, we'd have to somehow completely unload pip, evict it from the sys.modules cache, uninstall and install pip, then reimport pip and finish processing the requirements.txt file23:20
dstufftall without closing the process23:20
clarkbdstufft: the sane way is you fork and exec yourself23:20
clarkbbut the current behavior is established23:21
dtroyerare we still requiring new setuptools?  I thought I saw yesterday that the distro setuptools was generally ok now that d2to1 and friends are gone.23:21
clarkbdtroyer: we are not requiring new setuptools, but mordred does still want to require new pip iirc23:21
clarkbdstufft: and by sane I mean probably full of problems that will need work arounds23:22
dtroyerclarkb: that's what I thought.  btw, I've switched to installing pip 1.4 from source in tools/ and it appears to be working cleanly everywhere23:22
dstufftoh yea, I need to make a new release of pip in half an hour23:22
clarkbdstufft: but today if you want to upgrade anything in the path of installing stuff (pip,setuptools,distribute,etc) you must do two passes23:22
clarkbdstufft: which is essentially a fork and exec23:22
clarkbI just happen to do it with my shell instead :)23:23
dstufftclarkb: well upgrading setuptools you only needed to because of a bug in pip23:23
dstufftwell sort of a bug23:23
clarkbdstufft: or when one of your dependencies thinks you need it eg mysql-python23:23
dstufftclarkb: no that should be fixed in pip 1.423:23
clarkbI should upgrade pip on my local machine then23:24
clarkbmaybe after you push that new release with the bug fixes :)23:25
notmynamemordred: is supposed to have 2 change ids?23:25
Alex_Gaynornotmyname: you only got 2 change-ids? I saw one with 5 of them :P I assume it's not intentional, but I have no idea if it causes problems23:26
notmynameAlex_Gaynor: heh23:26
dstufftclarkb: the problem was sort of a perfect storm of things, basically because distribute was empty and forced install to setuptools, and the way pip handles upgrading setuptools and such it meant that if you were using distribute already and you tried to upgrade distribute (which means either setuptools or distribute becasue distribute <= 0.7 internally rewrote setuptools to distribute) it would uninsall distribute (which you were using)23:26
dstufft and attempt to install the "blank" distribute (which fails)23:26
dstufftfails because you no longer have a ``import setuptools`` to handle the installation23:27
dstufftnormally upgrading ditribute/setuptools was handled just by making the CWD the build directory so it ``import`` the setuptools package from the to be installed setuptools/distribute, but the blank one didn't have that23:27
funginotmyname: Alex_Gaynor: that definitely looks like scriptfail (on its way to botfail if you hadn't pointed it out)23:27
dstufftclarkb: if that makes sense? It's kind of crazy23:28
clarkbnotmyname: Alex_Gaynor: if you look at patchset one it has a single change id. I bet it was unintentional. It won't break anything other than looking up the change in gerrit might take a couple tries :)23:28
clarkbdstufft: I am still trying to grok. It sounds crazy :)23:28
*** nijaba has quit IRC23:28
clarkbdstufft: I see. Using the CWD broke because it was empty23:29
*** nijaba has joined #openstack-infra23:29
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Fix double-declaration of cgit class
pleia2^^ pretty sure that fixes what puppet is hiccuping on for ssl23:29
uvirtbotpleia2: Error: "^" is not a valid command.23:29
* pleia2 pets uvirtbot23:29
clarkbpleia2: you fixed one resource, where is the second?23:30
clarkboh there it is23:30
clarkbpleia2: you need to remove line 3023:30
dstufftclarkb: yea; pip 1.4 does some hackish shit to fix that. But it should get even better in the future. But upgrading setuptools setuptools should basically work (otherwise it's a bug) because we basically do fork/execute (subprocess)23:31
fungipleia2: still, yay though!
clarkbdstufft: it does mostly basically work23:31
pleia2clarkb: hm, how does that work?23:32
pleia2fungi: that makes me happy!23:32
clarkbdstufft: the issue is when a thing you are installing requires a newer version of setuptools. pip will upgrade it but pip has already imported the old version so the new one doesn't get used and dependency bombs out23:32
dstufftclarkb: pip doesn't import setuptools23:32
dstufftat all23:32
clarkbdstufft: sorry not pip, the of the things being installed23:32
fungipleia2: though i think you need to wrap whole lines in the template rather than just the values?23:33
pleia2clarkb: actually, I get it, nm :)23:33
pleia2fungi: ooh, yeah23:33
pleia2fungi: fixing23:33
dstufftclarkb: yea, that'll bomb because it got uninstalled before installing the new one (fixed in 1.4 supposidly, I wasn't the one working on that issue though)23:33
clarkbdstufft: when I noticed it it bombed because the version was wrong and not because the old one got uninstalled23:34
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Fix double-declaration of cgit class
clarkbdstufft: eg it upgraded just fine but the old stuff was loaded into memory23:34
dstufftclarkb: old stuff shouldn't be loaded into memory, each invocation is it's own subprocess so it's own process and it's own memory23:34
fungipleia2: actually, it looks like maybe you did wrap the entire line for each of those in a conditional check, but for some reason != "" is matching even when the value is nonempty23:34
pleia2fungi: yeah23:35
clarkbdstufft: yeah but there is only one invocation23:35
fungier, even when the value is empty23:35
dstufftclarkb: no, it does egg_info to get a list of deps, then later on it does install in a seperate process23:35
pleia2copied that logic from gerrit.config.erb23:35
*** changbl has joined #openstack-infra23:35
clarkbdstufft: but dependencies install in the install process right?23:35
fungipleia2: yeah, i'm not sure what rubbet devilry is afoot there23:35
dstufftclarkb: no23:36
dstufftclarkb: well they do for openstack23:36
dstufftbecuase you guys call pip yourself23:36
pleia2heh, rubbet :)23:36
dstufftinstead of letting pip handle things23:36
clarkbdstufft: well we saw this with tox before the pbr madness23:36
pleia2fungi: ooh, I think I have some syntax trouble, fixing up23:36
pleia2hm, no23:36
clarkbtox calls pip -r file and mysql-python cannot install because it requires distribute >= 06.28 abut virtualenv bundled 0.6.lessthan2823:36
clarkbpip was installing 0.6.28 it just wasn't imported into the process doing the execution23:37
fungipleia2: oh, yes i think you have -%> in some places you shouldn't?23:37
clarkbthe eventual work around there was to tell everyone to upgrade virtualenv to get a newer distribute bundle23:37
pleia2fungi: that's what I thought, but it actually seems consistent23:37
dstufftclarkb: i'd have to look to see closer.  gotta run for a few minutes right now. But if you get it again and can reproduce it let me know23:38
pleia2-%> should be in the if statement, %> otherwise23:38
clarkbbut another workaround was to upgrade distribute first in the virtualenv then run tox23:38
clarkbs/tox/pip or tox/23:38
clarkbdstufft: let me try23:38
dstufftclarkb: but that should be 100% supported IMO without hacks on your end (assuming the pbr stuff isn't making it happen)23:38
openstackgerritPaul Belanger proposed a change to openstack-infra/config: A few updates for RHEL based systems
dstufftclarkb: I'll be back in 10min or so23:38
pleia2fungi: I'm thinking maybe the variables aren't being expanded when it's <% rather than <%=23:38
*** rfolco has quit IRC23:39
fungioh, perhaps23:40
pleia2I'm just pattern matching though, I have no idea how this template magic works :)23:41
fungii'm actually wondering if != "" will ever match a hash value23:42
pleia2ah, yeah, what would an empty hash value look like?23:42
fungiit's possible it gets translated on insertion but doesn't straight up compare against a string23:44
* fungi is stabbing in the dark, being somewhat of a clueless ruby rube23:44
pleia2looks like ruby uses .empty .nil syntax23:44
pleia2so perhaps something like: <% if replication['replicatePermissions'].empty -%23:45
pleia2err, not empty :)23:45
*** rnirmal has quit IRC23:45
pleia2don't know how to do notempty23:45
fungii'll give it a try23:45
fungii'm trying if not replication['replicationDelay'].empty23:46
* pleia2 nods23:46
mgagnewhy is this not enough? <% if @replication['replicatePermissions'] -%>23:46
clarkbdstufft: reproduced in a horrid way. Install virtualenv 1.7.2 so that you get ancient distribute bundled (I did this in a venv), create a new virtualenv using version 1.7.2 using the --distribute flag. Deactivate initial venv, source new one made by old virtualenv, pip install mysql-python. It will bomb out and complain about the distribute version23:46
clarkbdstufft: now to test with pip 1.4 to see if it fixed it23:47
pleia2mgagne: oh, if that would work it would be nice23:47
clarkbnope pip 1.4 still breaks23:47
pleia2mgagne: including the @ ? (it's not used elsewhere23:47
pleia2mgagne is totally my puppet hero today23:48
fungipleia2: looks like it wants .nil instead of .empty anyway23:48
pleia2fungi: heh, naturally23:48
fungibut i'll try mgagne's suggestion23:48
clarkbFYI requirements approvals that don't pass pep8 will keep failing in the gate.23:48
notmynameanyone know why (ie migrate swift to pbr) has pbr v0.5.16 in requirements but 0.5.20 in
clarkb needs to merge before other requirements changes go in23:49
clarkbstill waiting on check tests to come back for it though23:49
clarkbnotmyname: because it took so long that things skewed. It should probably be 0.5.20 in both23:50
clarkbnotmyname: I would -1 that and ask mordred to correct it23:50
notmynameclarkb: ok, thanks23:50
mgagnepleia2: erb template variables now require @ notation (since Puppet 3.2):
mgagnepleia2: both methods were supported before.23:51
fungipleia2: mgagne: conditional matching on the lookup is also causing it to not pass the variables we want explicitly set = false23:51
pleia2fungi: *facepalm*23:51
*** rcleere has quit IRC23:51
mgagnefungi: true, if it's a bool value... ^^'23:52
fungipleia2: mgagne: so i suspect i really do need to match not lookup['key'].nil23:52
mgagnepleia2: @ refers to an instance variable in ruby. Without @, it's referring to a method. This way of accessing ERB variables got deprecated.23:52
pleia2mgagne: thanks, might as well update this template while I'm in it, since we'll go to 3.2 at some point23:53
fungimgagne: what am i doing wrong with the syntax there? getting undefined method `nil' for nil:NilClass23:54
mgagnefungi: your key does not exist. Can't call nil method on a nil.23:54
mgagnefungi: testing...23:55
fungihow do i match on nil? just != nil23:55
mgagnefungi: better only check if the key exists23:56
mgagnefungi: testing23:56
fungiwell, in this case != nil did exactly what we want, but if has_key?() is safer i'll give it a shot and see23:57
mgagnefungi: should work if your hash uses string for keys and not ruby symbols23:57
clarkbdstufft: I am going to walk home now. Will be AFK for ~20 minutes. Let me know if you need more info abuot the above mysql-python + virtualenv + pip + distribute stuff23:59
clarkbI will get back to it from hom23:59
mgagnepleia2: I can check to update puppet templates later this week if people don't mind to use @ notation23:59

Generated by 2.14.0 by Marius Gedminas - find it at!