Friday, 2013-08-02

*** bnemec has quit IRC00:00
*** bnemec has joined #openstack-infra00:01
*** michchap has joined #openstack-infra00:01
leifmadsenfor some reason some commands were very slow trying to update a branch, and now 'git review' and 'git remote show gerrit' just seem to be hanging. Is there anything going on with the network?00:02
clarkbleifmadsen: not that I am aware of00:02
* clarkb checks00:02
leifmadsenok, pabelanger told me he is having no issues00:02
leifmadsenso I'll re-setup locally00:02
leifmadsengit-review hangs too... heh00:02
leifmadsennot sure what is up00:02
clarkbleifmadsen: can you strace it to see what it is doing?00:02
clarkbleifmadsen: git review can do quite a bit of work behind the scenes00:03
pabelangerleifmadsen: what version of git-review?00:03
clarkbleifmadsen: it checks if rebases are necessary before pushing for example00:03
leifmadsenfstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 000:03
leifmadsenlseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)00:03
leifmadsenthat's where it just.... stops00:03
leifmadsenclarkb: ya I just ran a rebase on this feature branch00:03
*** changbl has quit IRC00:03
fungicrap, that's what i get for falling behind on reviews. i thought we'd discussed in here (before it got uploaded) and i pointed out that puppet does not default to root ownership on files, but instead defaults to copying the ownership from the source file as it exists on the filesystem00:03
clarkbssh connectivity is fine from here00:04
leifmadsenya, not sure what is up00:04
clarkbfungi: does it? --noop did not change the file00:04
leifmadsenodd, I think it is hanging on the ssh connection00:04
clarkbfungi: oh maybe it default to that when creating the file but if the file exists it leaves it be00:04
leifmadsenper git-review -v00:04
fungiat least the version we're running seems to do, last i tried it (which is why i had to chown my development puppet environment on the master so that it's root-owned)00:04
clarkbfungi: in that case I guess we need to change it to owner => root00:04
* leifmadsen checks firewall for odd forwarding00:04
pabelangerleifmadsen:  ssh -p 2941800:05
pabelangeryou might need to add username00:05
clarkbleifmadsen: ssh -p29418 gerrit ls-projects00:05
leifmadsenya, that's what it is hanging on00:05
leifmadsendoesn't complain, just sits there00:05
pabelangerdns issue?00:05
fungii'm also still three hours behind on scrollback, so it'll likely take me nearly that to read it all the way this week's been going00:05
mordreddtroyer: I would not be opposed to one giant venv00:05
leifmadsenpabelanger: I'd be shocked if it were00:05
leifmadsenunless is down00:06
mordreddtroyer: I'm also tired of dealing with it00:06
leifmadsenwebsites load fine00:06
mordreddtroyer: which is also weird, coming from a long-time debian guy00:06
clarkbleifmadsen: we do have AAAA records for it00:06
clarkbleifmadsen: could it be an ipv6 + ssh problem?00:06
mordredoh yeah - if you don't have ipv6 firewall stuff properly00:06
leifmadsenI have ipv6 tunnelling setup00:06
mordredtry ssh -4 -p29418 gerrit ls-projects00:06
leifmadsenwell, bitches about my key at least this time00:07
leifmadsenperhaps my ipv6 tunnel is down or osmething00:07
pabelangerleifmadsen: teksavvy right?00:07
leifmadsenpabelanger: no, I don't use them00:07
leifmadsenRogers cable with HE00:07
*** changbl has joined #openstack-infra00:08
openstackgerritlifeless proposed a change to openstack-dev/hacking: Revert "Add check for end of title line of commit messages"
pabelangerleifmadsen: Ya, I need to setup IPv6 again00:08
morganfainbergis there anyone who could tell me if there is an issue with pip and gerrit?  I'm seeing failures on dep installs for xattr(maybe cffi related?).  i didn't want to endlessly recheck nobug if it's a bigger issue or somehting i'm missing.00:09
morganfainberg(gate checks/pep8/etc)00:10
clarkbmorganfainberg: I have a story to tell you00:10
morganfainbergclarkb: oh boy00:10
clarkbmorganfainberg: cffi released version 0.7.1 today which xattr depends on. But 0.7.1 was broken00:10
leifmadsenbam fixed00:10
leifmadsenseems my IP changes00:10
morganfainbergclarkb: awesome.00:10
leifmadsenso had to update HE00:10
leifmadsenthanks for the ipv6 hint00:10
leifmadsenmordred: you get a beer00:10
clarkbmorganfainberg: Alex_Gaynor pinned cffi in xattr to the previous version and we reran our mirror scripts and at that point things should be beter00:11
morganfainbergclarkb: ok. cool.00:11
clarkbmorganfainberg: also cffi fixed their bug00:11
clarkbmorganfainberg: tl;dr all of our dependencies should get a gearman client to trigger our jobs with their new stuff before releasing because it turns out we test a lot of stuff :)00:11
morganfainbergclarkb: amazing right? :)00:11
clarkbmorganfainberg: let us know if things are still broken, but I believe everything should work at this point00:12
morganfainbergclarkb: thanks for the info.. maybe i should start lurking in here as well as the other openstack-* channels.00:12
morganfainbergclarkb: will do.  just issued the recheck.00:12
openstackgerritLeif Madsen proposed a change to openstack-infra/jenkins-job-builder: Add RVM (ruby version manager) support
leifmadsenclarkb: if you have a chance, in a few mins if 36386 passes the jenkins stuff, do you mind approving?00:14
*** UtahDave has quit IRC00:15
leifmadsenmaybe get this thing merged once and for all00:15
clarkbI'm assuming that is a rebase?00:15
leifmadsenyes sir00:15
clarkbwe should maybe start alphabetizing things to make conflicts less common00:16
*** pcrews has quit IRC00:17
leifmadsenah perhaps00:17
clarkbhmm check tests are going to take forver. I am just going to approve it00:18
leifmadsenit's been through the tests a few times now :)00:21
leifmadsenok, stepping away to watch some Suits and Big Brother ;)00:21
clarkbya and the diffs looked clean00:21
leifmadsenthanks for the help00:21
leifmadsenclarkb: +100:21
jeblairclarkb: i rechecked 39511, but it's at the bottom of the check queue, so....00:21
leifmadsenit was a simple rebase change00:21
jeblairclarkb, leifmadsen: yep, that's what the gate is for.  :)00:22
leifmadsenyay gates!00:22
* leifmadsen hops the fence00:22
leifmadsenpeas out yo00:22
jeblairclarkb: i mean, as long as you aren't in a queue with lots of changes, it doesn't really hurt to just aprv things and let the gate handle it.00:23
clarkbjeblair: ya00:23
clarkbjeblair: the one reason JJB is slightly different is we don't run the xml diff change00:23
clarkbbut like leifmadsen said it had been tested a few times and diffs of code were clean00:24
jeblairclarkb: there is that00:24
*** emagana has quit IRC00:25
jeblairclarkb: just in case you look at zuul, the keystoneclient change that is failing every test is legitimate: 38024,200:25
jeblairclarkb: apparently another change beat it to adding a test-requirement00:26
openstackgerritTom Fifield proposed a change to openstack-infra/config: Add build job for Japanese Operations Guide
dstufftDoes Fedora not have a server version00:28
dstufftI don't want a desktop :[00:28
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add RVM (ruby version manager) support
dstufftoh, guess I can spin up a cloud server00:29
dstufft<- worst openstack person ever00:29
jeblairdstufft: ha!00:29
clarkbwhy is weechat telling me I have one more line but cannot scroll down?00:30
clarkbooh there was a line there but why wouldn't it scroll00:30
*** Ryan_Lane has quit IRC00:30
clarkbanyways time for me to head home. Should be exciting we have protests00:31
*** bnemec has quit IRC00:32
*** bnemec has joined #openstack-infra00:32
*** michchap_ has joined #openstack-infra00:37
fungileifmadsen: worth noting, he will accept dyn-style updates for your endpoint addressing via ddclient and the like00:39
*** michchap has quit IRC00:40
*** mriedem has joined #openstack-infra00:42
jog0dumb question, I revised the wrong patch set and want to go back to an old revision00:42
jog0how do I do that?00:42
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Build a devstack-gate image with diskimage-builder
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Proof of concept of kexec/takeovernode
jeblairjog0: checkout the sha of the version you want and either (untested):00:42
jog0I tried checking outthe old version and doing git-review but gerrit said nothing changed00:42
jeblairjog0: (a) git review it and see i...00:43
clarkbyou must change it00:43
jeblairjog0: then (b) git commit --amend it to give it a new sha and git review that00:43
jog0that worked, but its sorta silly.  I guess I should be more careful next time00:44
fungijog0: you can git review -d changenum,psnum00:44
fungiand then git review again00:44
fungii think that's worked for me before, though it will munge your topic name so you might want to -t topicname00:45
jog0good to know for next time00:45
jeblairclarkb: that stable/grizzly change has started devstack tests; a good mix of az3 and others00:46
fungioh, i might also have done git commit -a --amend and improved the commit message while i was at it00:46
jeblairclarkb: checking to see if they are new-image nodes00:46
jeblairnew image timestamp is 00:05:40.00900:46
*** zul has joined #openstack-infra00:47
mordredclarkb, jeblair: the d-g patch above is what I've been using today for dev/test00:47
mordredalong with the script to build the image, and the script to reboot/takeover a node with the image00:47
mordredI want to make an element which includes the script and the kexec/qemu packages needed for takeover so that one can just have the image be a thing that knows how to takeover itself00:48
mordredbut the code above is totally workable and so far has been operating well00:48
jeblairclarkb: all the az3 nodes are newest-image00:49
jeblairmordred: that sounds excellent00:49
mordredjeblair: next time you have spare/extra time (hahahahaha) I'll step you through it :)00:50
jeblairmordred: did you see clark and i reverted the recent change to puppet to upgrade setuptools in pip...00:51
jeblairmordred: it broke stable/grizzly devstack-gate jobs00:51
mordredjeblair: awesome00:51
clarkbbut not master00:51
mordredI did not see that00:51
jeblairmordred: so i think we should proceed with extracting that stuff and moving it to jeepyb/zuul/jenkins-not-bare modules00:52
mordredclarkb: that should basically be because devstack on stable/grizzly doesn't have the pip install -U setuptools in it00:52
mordredjeblair: what state are we in now? reverted the revert?00:53
jeblairmordred: no, HEAD is Revert "Ensure latest setuptools after installing pip."00:54
jeblairclarkb: at least one az3 job has passed the point were we saw problems00:54
jeblairclarkb: so i think the new az3 image is good00:54
mordredcool. so current task on hand is fix stable/grizzly?00:54
clarkbjeblair: awesome00:54
clarkbmordred: no, I think current task is refactor pip in puppet00:54
clarkbonly use it where we need it then we can let devstack do its thing and it will test itself00:55
jeblairclarkb: +100:55
mordredoh ...00:55
mordredthe upgrade setuptools change broke stable/grizzly00:55
jeblairthis is not super-urgent because i think everything is working now00:55
mordredI read the sentences above in a different sequence00:55
mordredI thought you said the revert broke stable/grizzly00:55
mordredwith you now00:56
mordredand yes to the refactor00:56
jeblairmordred: correct.  the upgrade setuptools change from a few days ago broke stable/grizzly devstack runs when we finally built an image today00:56
jeblairmordred: so that change from a few days ago was reverted today, and now the images work again for stable/grizzly00:56
jeblairmordred: and then what clarkb said for actually getting that change in place in a better way00:56
jeblairfood now00:57
mordredyes. totally agree00:57
mordredback later00:57
clarkbya I am probably not going to be around much tonight. feeling blah00:58
*** pcrews has joined #openstack-infra00:59
*** erfanian has joined #openstack-infra01:10
anteayaclarkb: hope you are feeling non-blah soon, ghost tooth01:11
*** pcrews has quit IRC01:11
harlowjaqq, why are we needing cffi ?? :-/ for the ip address lookup stuffs?01:16
dkehnclarkb, you won't happen to have a localrc (devstack) from one of the runs handy that I could look and compare agains the one I'm using, I'm see a weird error that I haven't seen before, been using a modified forever01:17
*** markmcclain has quit IRC01:17
clarkbharlowja: xattr depends on it01:18
clarkbdkehn: if you open any tempet test runs logs you will see a localrc inthere01:18
dkehnclarkb, ok, thx01:19
harlowja if anyone is interested in how anvil is running these sorts of things :), able to fire it off from simple cloud-init script01:19
harlowjasource to rpms in no time :-P01:20
harlowja(and dependencies)01:20
dkehnclarkb, need a point to where that might be01:22
clarkbdkehn: its easier to start in gerrit01:22
clarkbdkehn: find any nova/swift/cinder/glance/etc change, click on the link to the build logs01:23
dkehnclarkb, ok, I'm an idoit I need a link to gerrit01:24
dkehnclarkb, that's what i was expecting sorry01:25
harlowjamore complete example (all fired off on y! cloud using cloud-init) - hope to get that into the upstream system (like smokestack someday)...01:26
clarkbharlowja: there are historical reasons for not using rpms/debs01:27
clarkbharlowja: mordred said earlier "We used them until the distros told us to stop" at some point today01:27
clarkbalso I think there is value in testing with builds from source. The fact that this breaks means it will break for others and is valuable for that reason01:27
mikalHeya, who here can explain how the MySQL db for migration testing are setup in the gate?01:28
mikaljhesketh: ^----01:28
clarkbmikal: I can01:28
mikaljhesketh: you should talk to clarkb!01:29
clarkbthats basically it, we provide mysql and postgresql servers running on localhost that fit that criteria01:29
jheskethclarkb: right, I'm having some trouble with running MySQL tests locally01:30
harlowjaclarkb i was thinking more of anvil could be like a smokestack system01:30
jheskethclarkb: are you in #openstack-nova per chance?01:30
clarkbjhesketh: no too much noise there01:30
clarkbI got my channel list to under 20 total channels recently :)01:30
jheskethnot as noisy in here, but that's okay01:30
harlowjasince anvil checks out repos, builds dependenies as rpms and the repos as srpms/rpms automatically, validates said dependencies are valid, can then install from said rpms and start the whole system, then tear it down01:30
clarkbjhesketh: ya but I have to be in here :)01:30
jheskethso if I run the tests with just sqlite it works fine01:31
jhesketh I'm just running: ./ db.test_migrations01:31
clarkbharlowja: yeah I think anvil doing like smokestack would be valuable01:31
jhesketh but if I change test_migrations.conf to just run MySQL tests it doesn't seem to set up the tests okay01:31
clarkbharlowja: second opinions are always good01:31
jhesketh for example: ProgrammingError: (ProgrammingError) (1146, "Table 'test_migrations.instances' doesn't exist") 'CREATE INDEX project_id ON instances (project_id)' ()01:31
jhesketh as in it doesn't create the database/tables correctly01:31
jhesketh but if I run the tests with test_migrations.conf with both sqlite and mysql set up, I get a different error01:31
jheskethspecifically: OperationalError: (OperationalError) (1242, 'Subquery returns more than 1 row') 'UPDATE services SET availability_zone=(SELECT aggregate_metadata.value \nFROM aggregate_metadata, aggregate_hosts \nWHERE aggregate_hosts.aggregate_id = aggregate_metadata.aggregate_id AND aggregate_metadata.`key` = %s AND = AND services.`binary` = %s)' ('availability_zone', 'nova-compute01:31
jhesketh (which I've lodged a bug for:
uvirtbotLaunchpad bug 1207309 in nova "Migration 147 does not downgrade on MySQL" [Undecided,In progress]01:32
clarkbjhesketh: we don't do anything to test_migrations.conf01:32
harlowjaeven the dependency validation is probably useful in itself01:32
clarkbjhesketh: so its possible those tests don't run and as a result have broken01:32
harlowja*cross repo dependency validation01:32
clarkbjhesketh: instead we run the opportunistic tests that are explained in the docstring I linked to01:32
jheskethah, I didn't realise they were different01:32
clarkbI don't know if they are, I just know that we don't change a conf file01:32
clarkbinstead the test suite detects the servers are available and uses them01:33
*** sarob has joined #openstack-infra01:33
jheskethclarkb: It looks like the opportunistic tests don't snake_walk the versions01:34
harlowjaanyways, me just rambling now, haha, gotta get someone to pony up the hardware for the smokestack like thing here01:35
clarkb:( thats unfortunate. I think when I fixed them for testr they did01:35
jheskethmost of the errors I'm encountering are downgrades on MySQL01:35
jheskethit skips the migrations.cfg01:35
clarkband sets snake walk to false01:36
clarkbso that is probably the issue01:36
clarkbit did snake walk at one point i wonder why it was changed01:36
jheskethso changing it now will probably block all the current gates01:37
*** sarob has quit IRC01:37
*** samalba has joined #openstack-infra01:38
clarkbwell the change won't pass the gate so it wont merge01:39
clarkbyou will have to fix the snakewalk to enable it01:39
samalbaIs there a place where I can find more info about the kernel used to run jenkins tests?01:39
jheskethah good point01:39
jheskethclarkb: I'll continue looking into it01:40
*** yaguang has joined #openstack-infra01:43
clarkbsamalba: what do you want to know?01:44
clarkbsamalba: it depends on the test as different tests run on different slave with different kernels01:44
*** yaguang has quit IRC01:44
*** yaguang has joined #openstack-infra01:44
clarkbsamalba: devstack tempest tests run on an up to date precise kernel, most other tests run on a relatively up to date precise kernel and python2.6 runs on a relatively up to date centos6 kernel01:45
samalbaclarkb: I see. Well I want to run Docker to add functional tests for Nova' docker hypervisor01:45
samalbaand Docker needs a kernel with aufs enabled and need to run a daemon as root01:45
samalbabasically if I add Docker's support to devstack to spawn a Docker daemon, can this daemon to be root?01:47
samalbaI guess it should not be a problem since each test runs in a VM, right?01:47
clarkball of our tests run in VMs devstack tempest tests are single use VMs and the unittests are on static slaves01:48
clarkbsamalba: looks like precise supports aufs so you shoudl be good?01:48
samalbaI think so, thanks01:48
openstackgerritDan Bode proposed a change to openstack-infra/config: Ensure python pip is installed before zuul
*** yaguang has quit IRC01:56
openstackgerritTom Fifield proposed a change to openstack-infra/config: Add build job for Japanese Operations Guide
morganfainbergclarkb: seems like everything is happy again.  thanks for the info earlier01:58
*** thomasm has joined #openstack-infra01:59
*** adalbas has quit IRC01:59
sdaguejhesketh: yes, we don't snake walk on opportunistic02:05
sdaguehonestly, downgrade with data is lossy02:05
jheskethsdague: right, but if an upgrade goes poorly and you want to rewind going back to as close as it was as possible is a good idea02:06
sdaguejhesketh: what models do you use then?02:07
*** anteaya has quit IRC02:07
jheskethsdague: what do you mean?02:07
sdagueyou need sqlalchemy models that match the db02:07
sdagueso if you are at a level of code that has new models, and the migration goes bad, how do you roll back your code to lock step?02:08
sdaguethat being said, if you can make the downgrade work, cool02:08
sdaguebut there was a reason we punted and instead worked on the data injection testing02:08
jheskethsure, valid points02:09
jheskethwould it work if you downgraded and then git reverted the models to before the migration was brought int02:09
sdagueprobably, but there are fun corner cases02:10
jheskethabsolutely, and they are what I'm trying to squash02:10
jheskeththis is to lead into a new CI tool to test against real data02:10
sdaguemore power to you :)02:10
jheskethso maybe I'll lead it up to rolling back models as well02:10
sdaguejust trying to give perspective on why I punted on it :)02:11
jheskethsure, makes complete sense02:11
jheskethI'm still wondering about needing to make them work myself02:11
sdagueright, my thinking is that no deployer would actually use them because they are lossy02:12
sdagueif things went that bad, they'd go to backup tapes02:12
sdagueand restore02:12
*** adalbas has joined #openstack-infra02:12
fungiheh... "tapes"02:12
jheskethagreed, and that was my next thought02:12
sdagueman... don't you diss tapes, they have awesome robot rooms02:12
*** nati_ueno has quit IRC02:13
* fungi wonders when we all stopped backing up our sensitive data onto stacks of greenbar02:13
jheskethsdague: so if we don't care about downgrading mysql or postgres, then why do we bother writing downgrades?02:13
sdaguejhesketh: ... for completeness?02:13
sdaguehoneslty, I have no idea, because they were there before I got here02:13
jheskeththey aren't particularly complete when they are broken ;-)02:14
sdagueyeh, well they work on sqlite :)02:14
sdaguewhich... you know... no one cares about02:14
sdaguesorry... it's late, I'm punchy02:14
fungisdague: it's not so much that nobody cares about it as we simply decided as a collective not to support its use in any way whatsoever? ;)02:16
sdagueyeh... like fungi said02:16
clarkbsdague: snake walk worked when testr went in02:16
clarkbsdague: did something change that made it not work?02:16
sdagueclarkb: it works with no data in the tables02:17
fungii got the impression zigo cared, because he didn't want to have to write database management maintainer scripts for the debian packages and had them default to sqlite...02:17
sdaguebut it's not very compatible with the data injection tests02:17
sdaguebecause a lot of downgrades are lossy02:17
sdaguelike drop columns02:17
sdaguethen how do you recover after?02:17
dkehnclarkb, hey thx for the pointer on the localrc, an update of the file has allowed me to run again, my old localrc I think was about 10 months old02:18
openstackgerritSean Dague proposed a change to openstack/requirements: one global-requirements.txt to rule them all
*** yaguang has joined #openstack-infra02:18
fungidkehn: that's like 20 in openstack-years!02:18
sdagueclarkb: where's the job we use to do requirements.txt testing on the projects?02:19
dkehnfungi, lol, I know02:19
sdagueas I'm sure that just broke it02:19
clarkbsdague: gate-$project-requirements iirc02:19
dkehnI'm surprised that its lasted this long02:19
sdagueclarkb: ok, you know what script it eventually calls out to?02:19
clarkbsdague: looking02:19
yaguangit seems something wrong with  static.openstack.org02:20
clarkbsdague: that I think02:20
clarkbyaguang: can you be more specific?02:20
fungiyaguang: how so?02:20
sdagueI can't believe I went around in circles for an hour today because I forgot to add test-requires to my test file....02:20
yaguangjenkins  fails  to upload  to static.openstack.org02:20
sdaguemuch thanks to mordred for unblocking me02:20
clarkbyaguang: the build was aborted which is why that happened02:20
dkehnhmm, maybe spoke too soon, I se I'm running the n-net not the neutron02:20
yaguangall jenkins  tasks  abort02:21
clarkbyaguang: the java there is being forcefully killed and the SCP plugin doesn't like it02:21
clarkbyaguang: yes, that is a zuul optimization. If something ahead in the queue fails everything behind it has its tests aborted02:21
clarkbyaguang: then when the thing that failed is removed from the queue tests are restarted for the changes behind it02:22
clarkbdkehn: neutron is broken02:22
clarkbdkehn: so good luck02:22
dkehnwith firewall group?02:22
dkehnclarkb, ^^^^02:22
clarkbsdague: that is the script02:22
clarkbdkehn: in general. We stopped voting on the devstack neutron tests02:22
clarkbso it could be in a giant state of disarray02:23
dkehnclarkb, ok, cuase I was see an error that I've not seen ion a couple of weeks02:23
fungiyaguang: it's about to get retested again as soon as pending tests on the two changes at the top of that gate queue finish02:23
yaguangfungi, get it, thanks02:24
dkehnclarkb, I'm actually running it in a VM that was my last successful run place to deal with a review02:24
openstackgerritA change was merged to openstack-dev/pbr: Use the same logic everywhere for requirements files
openstackgerritA change was merged to openstack-infra/devstack-gate: Copy tempest.log to logs dir after tempest run
fungiand those were the two02:25
openstackgerritSean Dague proposed a change to openstack-infra/config: support global-requirements.txt
sdagueclarkb: that look sane?02:27
clarkbsdague: will the other requirements files in openstack/requirements pollute global requirements or are the subsets?02:29
clarkbif subsets then yes sane02:30
sdaguehmmmmm.... right02:30
sdaguetest-requirements.txt will polute02:30
sdagueok, I'll circle on that02:30
*** rcleere has joined #openstack-infra02:30
sdaguebefore that, how do I add that new tox job to it's gate?02:30
sdagueI haven't actually added a new tox job in a while02:31
clarkbmaybe provide the list you care about when you create the requreiemts list object02:31
sdagueclarkb: yeh, I think that's a good call02:31
clarkbsdague: is it under the py27 tox env?02:31
clarkbin jenkins_job_builder/config/projects.yaml under openstack/requirements add gate-{name}-python27 to the job list02:31
clarkbthe in zuul/layout.yaml add gate-requirements-python27 under check and gate for openstack/requirements02:32
clarkbboth files rooted under modules/openstack_project/files02:32
*** pabelanger has quit IRC02:34
*** adalbas has quit IRC02:35
openstackgerritSean Dague proposed a change to openstack-infra/config: support global-requirements.txt
sdagueclarkb: yeh, it's a py2702:35
*** dims has quit IRC02:36
*** dims_ has quit IRC02:36
*** bnemec has quit IRC02:37
*** bnemec has joined #openstack-infra02:38
sdagueclarkb: so there is no requirements stub in projects.yaml yet02:39
sdagueis there a good starting point?02:39
clarkbsdague: you can copy any of the openstack/* org items02:39
clarkbsdague: tempest should work02:39
dkehnclarkb, in the gerrit logs is there a place I can look at the /etc/neutron/neutron.conf I think something else is missing the firewall_driver02:39
clarkbdkehn: I don't think we captuer the actual conf file02:40
*** jrex_laptop has quit IRC02:40
clarkbdkehn: but if you run devstack with the same localrc you should get the same conf file as its based on the sample02:40
sdaguedkehn: if you look at the console logs you should see all the devstack lines that write the file out02:41
sdaguebut we don't actually capture it02:42
dkehnclarkb, ok, thx I'll keep digging02:42
*** nijaba has quit IRC02:42
*** mriedem has quit IRC02:42
*** pabelanger has joined #openstack-infra02:43
openstackgerritSean Dague proposed a change to openstack-infra/config: add unit test job for global-requirements
dkehnclarkb, looks correct02:43
sdagueclarkb: ok, like that?02:43
dkehnclark, might do a clean pull from truck build and see02:44
clarkbsdague: yup02:45
sdagueok, well I think all the magical ponies are up for review now02:46
sdagueI'll let mordred take a swipe at them in the morning02:46
sdaguethough it will be a funny day tomorrow and mtreinish and I need to work on a packstack cluster to figure out a networking thing so unless we nail that early, might not be able to do much code tomorrow02:47
*** nijaba has joined #openstack-infra02:47
*** nijaba has quit IRC02:47
*** nijaba has joined #openstack-infra02:47
*** dkliban has joined #openstack-infra02:49
*** nijaba has quit IRC02:51
*** nijaba has joined #openstack-infra02:51
sdagueclarkb: nice02:51
sdagueyeh, the person I was at dinner with pointed out that new pep8 discourages lambdas02:55
sdaguewhich makes me a little sad02:55
*** bnemec has quit IRC02:57
*** dkliban has quit IRC02:57
*** bnemec has joined #openstack-infra02:58
*** jhesketh has quit IRC02:58
*** jhesketh_ has joined #openstack-infra02:58
clarkbsdague: it discourages them if you name them02:59
clarkbsdague: in favor of using def02:59
sdagueoh, only named lamdas02:59
sdagueok, I can live with that02:59
*** nijaba has quit IRC03:00
*** nijaba has joined #openstack-infra03:01
*** krtaylor has joined #openstack-infra03:08
*** melwitt has quit IRC03:11
*** sandywalsh has quit IRC03:11
*** sarob has joined #openstack-infra03:18
*** bnemec has quit IRC03:20
*** erfanian has quit IRC03:21
*** bnemec has joined #openstack-infra03:21
SpamapS2013-07-25 22:05:43.826 | E: heat.conf.sample is not up to date, please run tools/conf/generate_sample.sh03:26
SpamapSwhere is tools/conf/ ?03:26
clarkbSpamapS: it isn't in heat/tools/conf/ ?03:27
SpamapSclarkb: there is no heat/tools/conf03:27
SpamapSeven in the branch that is introducing said change03:28
SpamapSguessing a step was missed03:28
SpamapSlooks like that tool is in oslo-incubator03:28
dstufftmordred: dtroyer thanks to you guys I'm possibly writing a PEP to make two different places for distro isntalled packages and pip isntalled packages, so thanks for that ;P03:32
dstufftalso whoever else, I forget :V03:32
SpamapSclarkb: Hrm, ok, I'm trying to debug this fail
SpamapSclarkb: it is just adding help options, but for some reason the gate is kicking it out for generated configs not matching03:37
clarkbSpamapS: I think you need to run the tool so that it matches and include the new sample in your commit03:38
SpamapSclarkb: but why is the pep8 gate doing anymore than 'tox -epep8' ?03:39
*** sandywalsh has joined #openstack-infra03:44
openstackgerritSteve Baker proposed a change to openstack-infra/config: Do not run python jobs for tripleo-image-elements
clarkbSpamapS: because it is running flake8 which runs hakcing03:46
*** sarob_ has joined #openstack-infra03:47
*** sarob has quit IRC03:50
*** dkliban has joined #openstack-infra04:00
*** sarob_ has quit IRC04:04
*** SergeyLukjanov has joined #openstack-infra04:12
mikalHas anyone seen "Timeout while waiting on RPC response" in tempest from conductor? I'04:20
mikalm having trouble seeing how it would be related to my change04:20
*** dina_belova has joined #openstack-infra04:26
*** dkliban has quit IRC04:26
SpamapSclarkb: hacking would have an H rule I could control though...04:33
SpamapSclarkb: seems like it is something else..04:33
SpamapS2013-07-25 22:05:26.356 | pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn bin/heat-api-cloudwatch bin/heat-cfn bin/heat-engine bin/heat-manage bin/heat-watch04:33
SpamapS2013-07-25 22:05:26.377 |   /home/jenkins/workspace/gate-heat-pep8$ /home/jenkins/workspace/gate-heat-pep8/.tox/pep8/bin/flake8 heat bin/heat-api bin/heat-api-cfn bin/heat-api-cloudwatch bin/heat-cfn bin/heat-engine bin/heat-manage bin/heat-watch04:34
SpamapS2013-07-25 22:05:41.990 | pep8 runtests: commands[1] | /home/jenkins/workspace/gate-heat-pep8/tools/conf/check_uptodate.sh04:34
SpamapS2013-07-25 22:05:42.008 |   /home/jenkins/workspace/gate-heat-pep8$ /home/jenkins/workspace/gate-heat-pep8/tools/conf/check_uptodate.sh04:34
SpamapSclarkb: tox.ini, in that branch, does not have a second command04:34
clarkbSpamapS: is doing it?04:36
clarkbI thought it was hacking, apparently not04:38
*** Ryan_Lane has joined #openstack-infra04:41
SpamapSOh... rebased on master .. but.. the patch isn't..... ??04:43
clarkbSpamapS: we rebase on master before testing anything04:43
clarkbSpamapS: s/rebase/merge/04:44
clarkbwhich is why you get a -1 if you cannot merge04:44
clarkband in the gate queue you get tested in the same manner that you would merge (this accounts for other changes belonging to the same project ahead of you in the queue)04:46
*** boris-42 has joined #openstack-infra04:47
SpamapSclarkb: ah, very unexpected. :-P04:48
clarkbthis is one of the things that makes zuul special and different than travis saying everything is good on a pull request04:49
*** dina_belova has quit IRC04:49
clarkba lot of care goes into dealing with a moving trunk04:49
SpamapSYeah I love it..04:49
SpamapSIt was just unexpected. :)04:50
*** rcleere has quit IRC04:52
*** bnemec has quit IRC04:54
*** bnemec has joined #openstack-infra04:55
*** michchap_ has quit IRC04:59
*** UtahDave has joined #openstack-infra05:06
*** dstufft has quit IRC05:17
*** dstufft has joined #openstack-infra05:18
openstackgerritSergey Lukjanov proposed a change to openstack-infra/jenkins-job-builder: Pathignore plugin support added
*** hub_cap has joined #openstack-infra05:24
hub_capquestion. what ubuntu release should i be runnin devstack on? the reason i ask is cuz python-netaddr in 12.04 is 0.7.5 and nova's requirements.txt specify >=0.7.605:25
*** pabelanger has quit IRC05:26
clarkbhub_cap we use precise05:28
hub_capclarkb: hmm.. i dont know how im getting this error and not the rest of openstack then lol. ill keep diggin05:29
clarkbI think it is related to the setuptools mess05:29
hub_capseems my python-netaddr is 0.7.5 and i have to apt-get remove it in order for the pip installed 0.7.10 to be happy05:29
hub_capword. is there something i can do to get around it? /me assumes you havent heard of this yet?05:30
clarkbmordred was tackling it. you can pip install -U setuptools before running devstack05:31
*** mestery has quit IRC05:34
*** nicedice has quit IRC05:35
hub_caphmm clarkb no dice.05:39
clarkbcan you pip install netaddr?05:40
clarkbI dont have any other ideas right now05:40
hub_capthats the thing. it is pip installed05:42
hub_capbut devstack is apt installing python-netaddr as well :o05:42
hub_capi gotta see wahts going on .... cuz its obvi not happening in gates05:43
clarkbare you using neutron?05:43
clarkbor is it nova network? because neutron tests don't vote05:44
*** dina_belova has joined #openstack-infra05:46
*** michchap has joined #openstack-infra05:47
*** dina_belova has quit IRC05:48
*** UtahDave has quit IRC05:48
hub_capso im just pulling down latest devstack and running ./stack.sh05:49
hub_capbut let me isolate it first b4 i steal yer time05:49
* hub_cap assumes hes not using neutron05:50
hub_capis it default setup?05:50
hub_capits nova thats failing to start due to the netaddr conflict05:50
*** zehicle has quit IRC05:55
*** michchap has quit IRC05:56
*** michchap has joined #openstack-infra05:57
*** SergeyLukjanov has quit IRC06:08
*** mkoderer has quit IRC06:11
*** mkoderer has joined #openstack-infra06:15
*** Alex_Gaynor has quit IRC06:17
*** michchap has quit IRC06:33
hub_capclarkb: around? im not sure if im doing something wrong but here is what i did06:36
hub_capim on 12.04.1 could that be an issue?06:36
*** Alex_Gaynor has joined #openstack-infra06:39
*** zehicle has joined #openstack-infra06:43
* hub_cap tries .0206:44
*** boris-42 has quit IRC06:48
*** cody-somerville has quit IRC06:53
*** cthulhup has joined #openstack-infra06:56
*** jjmb has joined #openstack-infra07:04
*** cody-somerville has joined #openstack-infra07:05
*** SergeyLukjanov has joined #openstack-infra07:18
*** psedlak has joined #openstack-infra07:20
*** cody-somerville has quit IRC07:24
*** Ryan_Lane has quit IRC07:26
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Add support for the Summary Report publisher
*** cody-somerville has joined #openstack-infra07:34
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added some more scm options
*** odyssey4me has joined #openstack-infra07:42
*** hashar has joined #openstack-infra07:45
*** mestery has joined #openstack-infra07:47
*** fifieldt has quit IRC07:55
*** mkoderer has quit IRC07:55
*** jpich has joined #openstack-infra07:58
*** mestery has quit IRC07:58
*** dina_belova has joined #openstack-infra07:59
*** dina_belova has quit IRC08:00
*** boris-42 has joined #openstack-infra08:02
*** derekh has joined #openstack-infra08:03
openstackgerritTom Fifield proposed a change to openstack-infra/reviewstats: Add requirements to note that paramiko is needed
openstackgerritTom Fifield proposed a change to openstack-infra/reviewstats: Add requirements to note that paramiko is needed
*** vsergeyev has left #openstack-infra08:07
*** dina_belova has joined #openstack-infra08:13
*** odyssey4me has quit IRC08:28
*** SergeyLukjanov has quit IRC08:28
*** odyssey4me has joined #openstack-infra08:35
*** jhesketh_ has quit IRC08:41
*** jhesketh_ has joined #openstack-infra08:41
*** odyssey4me has quit IRC08:50
*** thomasbiege has joined #openstack-infra08:51
*** thomasbiege has quit IRC08:52
*** bnemec has quit IRC08:53
*** bnemec has joined #openstack-infra08:54
openstackgerritEvgeny Fadeev proposed a change to openstack-infra/askbot-theme: a fix to the language switch menu
openstackgerritA change was merged to openstack-infra/askbot-theme: a fix to the language switch menu
*** odyssey4me has joined #openstack-infra09:00
*** odyssey4me has quit IRC09:06
*** dina_belova has quit IRC09:07
*** jjmb has quit IRC09:11
*** odyssey4me has joined #openstack-infra09:14
*** cthulhup has quit IRC09:26
*** jjmb has joined #openstack-infra09:28
*** cthulhup has joined #openstack-infra09:40
*** hashar has quit IRC09:41
*** hashar has joined #openstack-infra09:45
*** cthulhup has quit IRC09:46
*** dina_belova has joined #openstack-infra09:47
*** mestery has joined #openstack-infra09:52
*** mkoderer has joined #openstack-infra09:52
*** dina_belova has quit IRC10:00
*** fbo has quit IRC10:07
*** fbo has joined #openstack-infra10:13
*** jjmb has quit IRC10:17
*** jjmb has joined #openstack-infra10:23
*** yaguang has quit IRC10:29
*** cthulhup has joined #openstack-infra10:35
*** dina_belova has joined #openstack-infra10:39
*** yaguang has joined #openstack-infra10:42
*** luisg has joined #openstack-infra10:47
*** luis_ has quit IRC10:48
*** cthulhup has quit IRC10:50
EmilienMHi Folks, I need one more review for puppet-heat in stackforge >  &
*** jjmb has quit IRC11:17
*** luis_ has joined #openstack-infra11:17
*** luisg has quit IRC11:18
*** michchap has joined #openstack-infra11:21
*** michchap has quit IRC11:22
*** michchap has joined #openstack-infra11:23
*** yaguang has quit IRC11:30
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added pre-scm-buildstep wrapper
*** nijaba has quit IRC11:43
*** bnemec has quit IRC11:49
*** jjmb has joined #openstack-infra11:49
*** bnemec has joined #openstack-infra11:50
*** ogelbukh has quit IRC11:52
*** dkranz has joined #openstack-infra11:58
*** ogelbukh has joined #openstack-infra11:58
*** CaptTofu has quit IRC11:58
*** zaro0508 has quit IRC12:05
*** weshay has joined #openstack-infra12:07
*** nijaba has joined #openstack-infra12:08
*** nijaba has quit IRC12:08
*** nijaba has joined #openstack-infra12:08
*** dina_belova has quit IRC12:08
*** pabelanger has joined #openstack-infra12:08
*** bnemec has quit IRC12:13
*** bnemec has joined #openstack-infra12:14
*** adalbas has joined #openstack-infra12:19
*** anteaya has joined #openstack-infra12:21
openstackgerritA change was merged to openstack-infra/config: Add heat puppet module to stackforge
openstackgerritA change was merged to openstack-infra/jeepyb: Add puppet-heat in launchpad project list
*** woodspa has joined #openstack-infra12:29
openstackgerritAnita Kuno proposed a change to openstack-infra/gitdm: Add anteaya to hp.
*** pabelanger has quit IRC12:33
*** dina_belova has joined #openstack-infra12:34
*** SergeyLukjanov has joined #openstack-infra12:35
*** lcestari has joined #openstack-infra12:36
*** openstackgerrit has quit IRC12:37
*** openstackgerrit has joined #openstack-infra12:37
*** pabelanger has joined #openstack-infra12:39
*** dkliban has joined #openstack-infra12:39
*** dims has joined #openstack-infra12:40
*** cthulhup has joined #openstack-infra12:41
*** pabelanger has quit IRC12:52
*** SergeyLukjanov has quit IRC12:54
*** afazekas has joined #openstack-infra12:55
*** burt has joined #openstack-infra12:59
dhellmanngood morning, any cores up yet?13:03
anteayamorning dhellmann haven't seen any yet13:04
dhellmannhi, anteaya! thanks, I'll check back after breakfast, then.13:04
fungidhellmann: yep13:05
anteayamorning fungi13:05
dhellmannah, the never-sleeping-fungi ;-)13:06
fungianteaya: morning! i usually just don't say anything unless i have something which needs saying13:06
fungiwhat's up, dhellmann?13:06
dhellmannI am hoping to get approved today so I can work on the basic project layout before taking off for vacation next week13:06
anteayafigured as much13:06
dhellmannI think we submitted that when there was a firestorm of other stuff going on, but it just needs one more +213:07
fungiahh, yeah... taking a look over it real quick like13:07
*** jjmb has quit IRC13:07
dhellmannI'm about to answer clarkb's question: we don't have a repo to prepopulate from13:07
*** dina_belova has quit IRC13:08
*** mriedem has joined #openstack-infra13:09
fungiheh, i was about to ask the same, but now i see he did already13:09
dhellmannthere are plans to reuse this code in other projects, so we thought we'd just start it out as a library. Also, most of ceilometer core doesn't want to "own" it right now, so we set up a separate team to include the devs who are working on it13:10
fungiworth noting, your first and subsequent commits to that project will have to be able to run (and pass) all the gate pipeline jobs you defined13:10
*** prad_ has joined #openstack-infra13:11
dhellmannI'm planning to either add an empty tox file, or write one dummy test for now13:11
dhellmanngordc will drop in the real code13:11
dhellmannmaybe "commands = true"13:11
fungiaside from the initial .gitreview file, which the manage-projects script will prepopulate13:12
fungiyeah, that should work as a stub13:12
*** rfolco has joined #openstack-infra13:12
dhellmannthe instructions for setting up a stackforge project say to open a bug to add the initial member to the new teams, is that still the best way to handle it?13:13
anteayadhellmann: depending on the tests that are run you may need to write a minimum of 4 for the tests to pass13:13
anteayaone test won't pass, unless that has changed13:13
anteayayeah, first rounds with storyboard failed since it only had one test13:13
dhellmannwhat's magic about 4?13:14
anteayaso I wrote 3 more13:14
fungianteaya: i think that may already be fixed, but we'll see soon enough13:14
anteayaclarkb knows13:14
anteayahe told me but I have forgotten13:14
dhellmannthanks for the heads-up, then, I'll keep that in mind13:14
fungisomething to do with the script which checks that tests ran... there are some additional lines of boilerplate being counted when they shouldn't be13:15
openstackgerritA change was merged to openstack-infra/config: Add pycadf project
anteayaas for storyboard, tests failed for random django reasons so we backed them off for now, just using pep8 currently13:15
dhellmannah, interesting13:15
dhellmannfungi: so, next I open a bug on launchpad to ask to be added to the gerrit teams?13:15
*** cppcabrera has joined #openstack-infra13:16
*** jjmb has joined #openstack-infra13:16
*** changbl has quit IRC13:16
fungidhellmann: i can add initial members for those groups if you just tell me what you want. the bug suggestion was just to provide some async solution for those requests so we don't forget if we're not around13:16
dhellmanncool, didn't want to assume you had all the spare time to do it now13:16
fungiit's fine13:16
dhellmannlet me get some gerrit ids13:16
dhellmanndo you need numbers, logins, or emails?13:17
fungione gerrit id for each group should be fine, e-mail preferred. that person can add others13:17
fungithe groups are self-managed13:17
dhellmannok, I'm doug-hellmann, so if you add me I can take care of it from there13:17
fungiwill do. it'll be a few minutes before manage-projects runs to create the groups and then i'll pop you into them13:18
dhellmannI expect I'll help them out with releases until they're comfortable (the main devs are new to python, git, etc. -- where I was a year ago)13:18
*** michchap has quit IRC13:18
dhellmannperfect, thanks, fungi !13:18
fungiand i'll give you a heads up in here when it's ready13:18
dhellmanncool, thank you13:19
* dhellmann wanders off to find caffine13:19
*** prad_ has quit IRC13:22
*** markmcclain has joined #openstack-infra13:23
*** SergeyLukjanov has joined #openstack-infra13:28
*** dina_belova has joined #openstack-infra13:28
anteayaif anyone cares I have a sneaking suspicion that the tempest.api.volume tests are not completing their teardown properly, and I am going with it isn't being caught in the test runs since it is the last round of tests to run13:29
anteayaI am going to see if I can find any evidence to back this up13:29
anteayaor more than made me suspicious13:29
*** dolphm_ has joined #openstack-infra13:29
dolphm_wishlist item! as a core reviewer, i really want to have a Rebase button on other people's changes in gerrit13:31
dolphm_(or, really any way to signal to other reviewers and the change author that this will no longer merge as-is)13:32
dolphm_i guess the use case would be better served by a new jenkins job13:33
*** krtaylor has quit IRC13:34
*** openstackgerrit has quit IRC13:36
*** openstackgerrit has joined #openstack-infra13:36
*** hashar has quit IRC13:36
sdagueman, I was hoping mordred would be around13:41
sdagueI've got all these ponies lined up for him to make magical, that we need to land in a specific order accross projects13:42
*** vijendar has joined #openstack-infra13:43
*** sandywalsh has quit IRC13:45
*** burt has quit IRC13:46
fungidolphm_: there may be good reasons not to add the rebase button for non-change-owners, but it looks like it's just a gerrit acl permission13:47
fungiso should be trivial to set for any group on a per-project basis13:50
*** tian has quit IRC13:52
fungianteaya: are you seeing this under traditional tempest or testr tempest?13:52
*** _TheDodd_ has joined #openstack-infra13:52
*** shanewang has quit IRC13:52
anteayatestr tempest13:52
fungiahh, fun13:52
anteayaI haven't been running traditional so I can't say it is or is not there as well13:53
anteayawhat do you consider traditional tempest, by the way13:53
*** bnemec has quit IRC13:55
*** bnemec has joined #openstack-infra13:55
fungiif i were awake enough i'd say something about the kind of tempest which puts candles up around the winter holidays, but instead... running without DEVSTACK_GATE_TEMPEST_TESTR_FULL=113:55
*** HenryG has joined #openstack-infra13:56
fungiusing DEVSTACK_GATE_TEMPEST_FULL=1 instead13:56
anteayaah okay13:56
sdagueanteaya: I could believe volume teardown is racey13:56
sdaguewe've seen that before13:56
anteayasdague: great glad I'm not tilting at windmills then13:57
sdagueafazekas has been working on resource leak detection13:57
anteayais it worth my time to confirm?13:57
sdaguebut it's probably post havana13:57
sdagueanteaya: yes definitely13:57
anteayaah okay13:57
sdagueit's one of the reasons we've got the ./ script in devstack13:58
*** mriedem has quit IRC13:58
fungialso #openstack-qa may find you a more focused audience for that topic13:58
anteayaah /me looks at ./clean.sh13:58
anteayafungi: true that13:58
*** prad_ has joined #openstack-infra13:58
*** sandywalsh has joined #openstack-infra13:59
*** mriedem has joined #openstack-infra13:59
afazekassdague: BTW:
sdaguefungi: any chance I could get you to look at and and ?13:59
fungisdague: in about 20 minutes i can crack into those14:00
* fungi puts them at the top of the list14:00
sdaguethey will need to land in that order, and mordred should have final say, but they could use another +2 if they are right14:00
fungiyou bet14:00
sdagueI think we definitely need an icehouse summit about how to have zuul do cross project deps14:01
*** adalbas has quit IRC14:01
*** prad_ has left #openstack-infra14:01
*** dina_belova has quit IRC14:02
fungidhellmann:,members and,members14:03
dhellmannfungi: thanks!14:03
*** dina_belova has joined #openstack-infra14:03
*** cppcabrera has left #openstack-infra14:04
*** vijendar has quit IRC14:05
mordredmorning sdague14:06
*** jeffa has joined #openstack-infra14:06
mordredsdague: I actually think that should land first14:07
mordredsdague: so that when we land it will in fact verify that the gate works14:07
mordredsdague: or - rather - I thnk we should land that list of things in the opposite order that you have it listed14:09
mordredsdague: land both of the infra config jobs, then the requirements job14:09
sdaguemordred: sure14:09
sdagueI think the last one doesn't work though14:10
mordredoh no?14:10
sdagueoh, wait14:10
mordredwhy not?14:10
sdagueright, the last one will ensure that no other requirements change can land before the first one14:10
sdagueno, you're right14:10
mordredI'm going to land it14:10
*** hashar has joined #openstack-infra14:13
openstackgerritA change was merged to openstack-infra/config: add unit test job for global-requirements
*** vijendar has joined #openstack-infra14:15
thomasmHey all14:17
*** adalbas has joined #openstack-infra14:17
mordredsdague: the other one - the projects requirements check - that can land any time, can't it?14:18
sdaguemordred: this one -
*** krtaylor has joined #openstack-infra14:18
sdaguethat actually needs the global- change first14:19
* mordred waiting for the job change to land in jenkins14:19
sdaguebecause clarkb pointed out that if I looped all the files, which was the first attempt14:20
*** rfolco has quit IRC14:20
sdaguethat I'd actually pick up requirements/test-requirements.txt14:20
sdaguewhich would polute global requirements14:20
*** mkoderer has quit IRC14:23
*** changbl has joined #openstack-infra14:23
*** pabelanger has joined #openstack-infra14:24
*** rcleere has joined #openstack-infra14:28
*** burt has joined #openstack-infra14:33
dhellmannmordred: pbr fails to generate an authors list for a repo without any real commits. is that intentional, or a bug?14:34
*** russellb is now known as rustlebee14:35
sdaguemordred: ok, so how about this next -
*** anteaya is now known as anteayah14:36
anteayah'cmon ghost tooth14:36
*** pentameter has joined #openstack-infra14:38
*** dansmith is now known as Steely_Dan14:42
*** markmcclain has quit IRC14:44
*** jjmb has quit IRC14:46
*** SergeyLukjanov has quit IRC14:47
fungidhellmann: is the expectation that you should get an empty authors file, or are you hitting an exception?14:48
dhellmannI get an error because it's trying to treat None like a string14:48
dhellmannan empty AUTHORS file would be fine, I think14:48
*** pabelanger has quit IRC14:49
dhellmannI'll work on a fix, I just wanted to know if this was expected behavior or a known issue someone was working on already14:49
fungidhellmann: i would definitely call that a bug. i'm not sure what to expect as a behavior, but i'd wager an exception is not it14:50
dhellmannor at least an exception with a better message :-)14:50
fungieither silently refuse to create an authors file or create an empty one with/without some boilerplate comments or whatever14:50
*** jjmb has joined #openstack-infra14:51
*** pabelanger has joined #openstack-infra14:51
dhellmannsounds good14:52
*** mrodden has quit IRC14:53
mordreddhellmann: uh - wow... I believe I have just simply never tried that case14:54
dhellmannmordred: that's what I figured. New pycadf repo exposed it.14:55
dhellmannI'm working on a fix14:55
fungithat was my suspicion as well. generally speaking, repos don't stay empty for long and rarely get packaged until they have a commit14:55
mordredfungi, clarkb: could one of you take a peek at ?14:55
mordredit's ready to land, but needs a second +@14:55
fungion it14:55
*** hashar has quit IRC14:56
fungii have no more +@s in my bucket, but plenty of +2s14:56
*** SergeyLukjanov has joined #openstack-infra14:56
*** UtahDave has joined #openstack-infra14:56
fungiand also appreciate the tolkien reference14:57
anteayahmorning jeblair14:58
*** dina_belova has quit IRC14:58
fungimordred: so one of the arguments against doing this early on was that we no longer give downstream packagers an indication of which requirements they need to package as run-time dependencies and which are only testing requirements. what is the current thinking on that?14:58
mordredfungi: the gate breakages are, I think, more important to engineer for14:59
fungiahh, the comment and separate sections14:59
*** dina_belova has joined #openstack-infra14:59
mordredfungi: and honestly - I think that downstream packagers _should_ package all of our test requires too :)14:59
fungii see we've kept them independent in the file, so i guess that's good enough to support that case14:59
jeblairwas there a reason they couldn't be two files?15:00
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added Robot Framework reposts publisher
mordredsdague: do you remember the details of why we did hte switch? I only remember things sucking and that one-file-to-rule-them-all was the answer15:02
jeblairthe commit msg says it 'massively simplifies'15:02
*** dina_belova has quit IRC15:02
jeblairi guess with all the scripts we have touching these files now (unit tests, update, etc) there's a lot of 'read both files and combine them'...15:03
jeblair(or would be otherwise)15:03
fungii got the impression there were a couple of issues being addressed, one of which was that having the lists in requirements.txt and test-requirements.txt confused some of the tooling into thinking those applied directly to the requirements project15:03
jeblairsure, but that's addressed by moving it to global-requirements (and there could be a global-test-requirements)15:04
*** changbl has quit IRC15:04
*** afazekas has quit IRC15:04
sdaguemordred: yes, otherwise you are running around reading both files all the time15:04
*** dina_belova has joined #openstack-infra15:04
jeblairmordred, sdague: is there a corresponding change to the mirror to read global-?15:04
fungithough i note that the rename issue is being quietly solved in this change without an actual mention15:04
jeblairmordred: also, this will probably need backporting to the stable branches of the requirements repo15:06
mordredjeblair: good catch...15:06
jeblair(which are still mostly unused, so probably nbd)15:06
sdaguejeblair: I didnt' touch the mirror script, where is it at15:07
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Add global-requirements.txt to the mirror list
jeblairsdague: ^ there :)15:07
mordredsdague: ^^ right there :)15:07
jeblairsdague: gate-tempest-devstack-vm-neutron is succeeding in master according to a quick scan of zuul status page15:07
fungiquick lock it down?15:08
*** mrodden has joined #openstack-infra15:08
mordredand disallow any further commits to neutron!15:08
jeblairin a few mins, i'll see if our new graphs can show us data on that15:08
mordredjeblair: I, for one, welcome our new graphs overlords15:09
jeblairafter i make an omelette15:09
mordreddoes everyone else feel like this week was a cauldron?15:09
fungiomelette du fromage15:09
fungimordred: yes, and the remaining bits of my flesh are still boiling off as far as i can tell15:10
mriedemsdague: is there still a running list somewhere (etherpad) of the neutron tests that fail in tempest?15:10
mordredmriedem: it's passing it's passing!15:10
sdaguejeblair: so the reason we pulled it originally is a race15:10
mordredoh right15:10
sdagueso we need a couple days of it working before we turn it back on15:10
*** toddmorey has joined #openstack-infra15:10
fungithus, graphsgraphsgraphs15:10
mriedemmrodden: hmm, i can't get this one to work: tempest/api/compute/security_groups/
mriedembut i couldn't find a bug for it15:10
mriedemmordred: ^  - didn't mean mrodden15:11
mordredmriedem: there's a lot of m's and r's in these parts...15:11
*** mordred is now known as TheRonin15:11
TheRoninit is casual nick friday ...15:11
fungiTheRonin: i'm wearing my casual pants instead15:12
sdaguehonestly, I think the reason it broke really hard was them not having the right oslo version in there for a bit15:12
sdaguewhich was fixed by oslo.config git in devstack going in15:12
TheRoninheck yes15:12
sdaguewell I bicycled to work, so I'm very casual today15:13
TheRoninsdague: have we added oslo.config into the gate yet? or just its git repo into the mix?15:13
*** rfolco has joined #openstack-infra15:14
sdagueTheRonin: it's waiting +A15:14
sdagueso pull the trigger. It's got 2 +2s plus markmc's +115:14
*** markmcclain has joined #openstack-infra15:15
anteayahfungi: and your casual Hawaiian shirt?15:15
TheRoninjeblair: any reason I should hold off on pulling the trigger on the oslo.config gating change?15:17
fungianteayah: my hawaiian shirts are anything but casual. causal maybe, but not casual15:18
openstackgerritA change was merged to openstack-infra/gitdm: Add anteaya to hp.
fungianteayah: congrats on the new patron!15:19
*** changbl has joined #openstack-infra15:19
anteayahfungi: yay, thank you15:19
anteayahah the causal hawaiian shirts, they must cost extra15:20
fungibut they always ensure an outcome15:20
anteayahha ha ha15:21
*** boris-42 has quit IRC15:21
ttxTheRonin: stable/folsom is unfucked so we can implement capping. I could do it I guess but I'd probably screw it up. Crae giving it a try ?15:22
anteayahttx are you back for storyboard questions or are you gone again?15:24
ttxanteayah: I'm almost done for the day, but you can shoot questions in the remaining time :)15:24
anteayahokay let me fire it up15:25
burtmordred: any progress on devstack building on Precise?15:25
jeblairsdague, mordred:
anteayahttx okay so currently if I have at least one project, one branch and one milestone in the db15:27
anteayahif I go to a project and create a new bug15:27
openstackgerritA change was merged to openstack-infra/config: make oslo.config gating on devstack/tempest runs
anteayahI get this error: Milestone matching query does not exist. Lookup parameters were {'undefined': True, 'branch__status': 'M'}15:27
TheRoninjeblair: that is an excellent graph15:27
anteayahI don't know if this is just me and I need to do something different for configuration or if this is an actual bug and I need to edit code15:28
anteayahttx are you able to confirm my findings?15:28
*** jeffa has quit IRC15:28
ttxanteayah: you need at least two branches defined, a master and release branch15:28
ttxSee the bottom of README.rst15:29
ttxMaster must have status='M'15:29
anteayahokay thanks, I will do that15:30
ttxmilestone-proposed (the release branch) must be status=R15:30
*** mrodden has quit IRC15:30
anteayahI had both a master and a release branch already15:31
clarkbdid neutron merge a fix? eg is this expected?15:31
*** rfolco has quit IRC15:31
anteayahI had branch_01 as master and branch_02 as release15:31
anteayahthey are already in the db15:32
fungiclarkb: 0000z this morning, looks like15:32
anteayahdo I need to rename branch_02 to milestone-proposed?15:32
ttxanteayah: no, but the branch_types need to be set right15:32
anteayahthey are15:32
anteayahbranch_01 is master and branch_02 is release15:33
anteayahdoes there need to be an additional relationship?15:33
ttxAh. er.15:33
anteayahor does the datetime play a role?15:33
sdagueTheRonin: ok so we're just waiting on the mirror script to land?15:33
ttxYou'll need one milestone for the master branch15:33
anteayahI have just been hitting today and now to set the time and I created them yesterday15:33
TheRoninsdague: the jenkins job? it has landed15:33
anteayahumm, let me look15:34
ttxanteayah: think "havana-3"15:34
ttxwe should definitely come up with initial fixture for the db to simplify15:34
anteayahI do, milestone_01 is on branch_0115:34
ttx(release_date is just used for ordering)15:34
jeblairTheRonin, sdague: the mirror script will run on both global-requirements and the requirements repo's own requirements... is that ok?15:34
anteayahI had that15:34
anteayahttx this is where my lack of knowledge about releases and branches is bringing me down, sorry about this15:35
*** nicedice has joined #openstack-infra15:35
anteayahso milestone_01 was already on branch_01, the master branch15:35
TheRoninjeblair: yes. I believe that the requirements repo's test-requirements.txt should be a subset of global-requirements15:35
ttxanteayah: you need one milestone with status=M and undefined=True, although I don't remember why right now :)15:36
jeblairok.  should the requirements repo get a requirements check job? :)15:36
sdaguejeblair: yes, probably :)15:36
*** afazekas_zz has quit IRC15:36
jeblairmirror change aprvd15:36
* ttx checks on dev instance15:36
sdagueok, so I think this is next -
openstackgerritA change was merged to openstack-infra/jeepyb: Add global-requirements.txt to the mirror list
sdaguethat makes sure no other requirements changes land15:37
ttxanteayah: ok, so you need an "Undefined" milestone on master branch with undefined=True15:37
ttxanteayah: and I failed to document /that/ :)15:38
sdagueactually, it's tightly coupled with the actual requirements change15:38
sdagueso I think they both basically need to go at the same time15:38
sdaguewe'll have a window of failing nova jobs until they do15:38
ttxanteayah: see doc @
*** zul has quit IRC15:38
anteayahttx that was it, once I clicked the undefined box on the milestone_01 I was able to create a bug on a project without an error15:38
jeblairsdague: but we don't normally run the requirements jobs unless they change15:39
sdaguejeblair: oh, right15:39
ttxanteayah: "Milestones are mandatory, there is a15:39
ttxdefault 'undefined' milestone for when we don't really know when that15:39
ttxwork will hit in the master branch."15:39
sdagueright, so next15:39
jeblairsdague: so as long as no one changes a req, we can land 39853 safely15:39
*** zul has joined #openstack-infra15:39
ttxanteayah: feel free to turn that commit message into proper doc :)15:39
sdagueman, this feels like aircraft traffic control :)15:39
anteayahttx okay thanks, I'll try to get my head around this documentation15:39
anteayahttx I might actually be able to land a patch then15:40
ttxanteayah: yeah, sorry about that15:40
anteayahha ha ha15:40
anteayahI have to have something to bug you about15:40
jeblairsdague: i think we may want: branch in ['master', 'milestone-proposed'] ?15:40
anteayahusually you are teflon man15:40
anteayahI will over use this one chink in the armour, never fear15:41
jeblairsdague: especially if we are going to change this to run on every change15:41
sdaguejeblair: well it wasn't there before, can we do that as a follow on?15:41
sdaguejeblair: so this doesn't actually make it run on every change15:41
sdaguethis just changes the source file15:41
clarkb is that the thing making neutron better?15:41
uvirtbotLaunchpad bug 1206330 in nova "nova list failure when quantum_url points to HTTPS endpoint" [Undecided,New]15:41
sdagueall the "run on every change" is phase 2 stuff, once we get this all in the gate15:41
jeblairsdague: yeah, you're right that doesn't impact this (yet; though it may when we start running on every change)15:42
clarkb_TheDodd_: you about? seems to have done the correct thing to
jeblairas currently written, if we run it on every change, then the first commit to m-p would have to be to remove anything in dev-requirements.15:43
uvirtbotLaunchpad bug 1207047 in openstack-ci "The logstash gearman workers should properly handle gzipped files." [Low,In progress]15:43
jeblairsdague: that may not be wrong.  :)15:43
TheRoninjeblair: ++15:43
clarkb_TheDodd_: I think I am ready to announce the new bug awesomeness15:44
clarkbjeblair: TheRonin fungi ^15:44
TheRoninI think we'll need to sort oslo pre-release releasing before we can do that for m-p15:44
TheRoninclarkb: ++15:44
fungiclarkb: yes, seems to be working well from what i've seen15:44
jeblairclarkb: _TheDodd_: +115:44
_TheDodd_clarkb: Well, that sounds great to me.15:44
_TheDodd_I'm glad everything is in order.15:45
fungithanks again for an excellent piece of work, _TheDodd_!15:45
jeblair_TheDodd_: you want to send the announcement out?15:45
_TheDodd_We've not received any complaints about back compat?15:45
_TheDodd_fungi: THX15:45
_TheDodd_jeblair: sure.15:45
_TheDodd_jeblair: how should I go about making the announcement?15:46
jeblair_TheDodd_: we usually send something to openstack-dev and tag with [Infra]15:46
openstackgerritDoug Hellmann proposed a change to openstack-dev/pbr: Ensure _run_shell_command always returns same type
sdagueok, 3 +2s -
sdaguecan we land? :)15:48
* fungi mistakenly sent his last announcement with [CI] since it was ci-specific and not just general infrastructure15:48
*** cthulhup has quit IRC15:48
_TheDodd_jeblair: Tell you what, I will let one of you pros do it this time. I'll watch and learn, so that when another opportunity comes up for me to make an announcement, I'll be all over it.15:48
dhellmannfungi, mordred: there's the fix for the "package in a new git repo" issue ^^15:48
jeblair_TheDodd_: sure, no pressure.  :)15:49
jeblairanyone remember where that etherpad is?15:49
*** mrodden has joined #openstack-infra15:49
*** reed has joined #openstack-infra15:50
fungisdague: approved and will keep an eye on running requirements jobs if any for potental breakage (though not expecting any)15:51
sdagueok, cool15:51
sdagueso as soon as that merges, then we've got the other one15:51
openstackgerritA change was merged to openstack-infra/config: support global-requirements.txt
clarkbhow are mail queues doing this morning?15:54
*** cthulhup has joined #openstack-infra15:55
fungi1248  4462KB      4d      0m  TOTAL15:55
funginot too bad15:55
sdagueok, so for our next trick -
sdaguethat's the big one15:56
jeblairi'm having a devil of a time getting snmpd to report that so we can graph it, btw, due to permissions nightmares.  :(15:56
clarkbsdague: you used my LoTR reference :)15:56
fungiclarkb: sudo exim -bp | exiqsumm15:56
clarkbfungi: ty15:56
fungiclarkb: note that most everything reported there is 0m in the queue15:56
sdagueclarkb: yeh, honestly, last night writing that commit message, it's all I could think of15:56
jeblairclarkb: 'a few hundred' is probably baseline for that server.15:57
jeblairlike 600-80015:57
clarkbfungi: is that the delay?15:57
clarkbjeblair: what about collectd into graphite?15:57
fungiclarkb: it's how long the delivery has been queued15:57
clarkb(adds a new tool but possibly easier to fight than snmp)15:57
fungiclarkb: the "delay" depends to a great extent on the recipient's mta15:57
*** burt has left #openstack-infra15:57
*** burt has joined #openstack-infra15:58
jeblairclarkb: in the current configuration, almost everything is going to end up getting queued for at least a minute.15:58
fungiclarkb: so having a handful in the multiple minutes, hours or even days is expected. it's more a question of calculating the statistical mode of queue times for deliveries15:58
*** jeffa has joined #openstack-infra15:59
jeblairmsg sent16:00
clarkbfungi: I see.16:00
sdagueok, I'm running off to lunch - still needs a +A, as no one can merge requirements changes until it lands16:01
jeblairsdague: gee haven't we done enough today?  (aprvd) :)16:03
clarkbthe day is just starting :)16:04
*** _TheDodd_ is now known as TheDodd16:05
*** emagana has joined #openstack-infra16:06
TheDoddDitched the underscores for Casual nick Friday!!!16:06
anteayahvery sharp16:07
jeblairclarkb: when would you like the periodic jobs run?16:07
*** mrodden1 has joined #openstack-infra16:08
*** jeffa has quit IRC16:08
clarkbjeblair: I believe our "quiet" period is 11pm PDT to 2am PDT ish16:09
clarkbjeblair: so 0600 UTC?16:09
*** TheDodd is now known as thedodd16:09
jeblairclarkb: sure.  i couldn't tell looking at our graph.  :)16:09
clarkbjeblair: there is a slight dip on the graph around 10-11pm16:09
* fungi is not so sure we have a "quiet period" any more16:09
fungiat least not on weekdays16:10
clarkbfungi: we still have a dip in the middle of the night pacific time, but it isn't a full stop16:10
fungifriday mornings are nice and quiet because much of the world has already started its weekend16:10
jeblairsince these are going in zuul, they'll be low gearman priority anyway16:10
clarkbjeblair: good point16:10
*** mrodden has quit IRC16:10
fungiahh, yes, low priority for periodic jobs sounds ideal16:10
clarkbwe didn't have that before so it was important to plan ahead and try to avoid starvation16:10
jeblaircan still do that of course16:11
jeblairfungi: is there a python 3.3 mirror slave?16:15
fungijeblair: there is a
*** nijaba has quit IRC16:17
fungijeblair: i'm eager to build it as soon as that gets merged16:18
fungiand then i'll add the jobs for it16:18
jeblairfungi: I'm eager to have it built!16:18
TheRoninjeblair, fungi, sdague: while we're gating requirements on things:
jeblairTheRonin: i already reviewed that.  :)16:19
clarkbjeblair: I intend on spinning up a new tx.slave today and fixing setuptools by hand if necessary16:19
TheRoninjeblair: I know16:19
TheRoninjeblair: I was just including you in the conversation16:19
jeblairTheRonin: thanks, it worked.  :)16:19
TheRoninclarkb: should we re-name that slave?16:19
clarkbTheRonin: sure, what do you want to call it?16:19
TheRoninclarkb: weren't we talking about having it start to do something else too?16:19
*** nijaba has joined #openstack-infra16:19
*** nijaba has quit IRC16:19
*** nijaba has joined #openstack-infra16:19
jeblairTheRonin: i think you're thinking of pypi16:19
* TheRonin trying to remember ...16:19
clarkbTheRonin: that is pypi.slave16:20
clarkbbecause it talk maven now too16:20
TheRoninno - I htink I'm thinking of a new things for tx.slave16:20
TheRoninwhich is tx.slave submits the translations changes to gerrit for review, right?16:20
fungiclarkb: your comment on 39561... i can take a stab at that after we merge that one, if there's consensus that we don't need future differentiation which might be out of place in the site manifest16:20
TheRoninso wouldn't that be a good place to submit the requirements changes to gerrit for review?16:20
clarkbI will probably spin up elasticsearch6 at the same time so that adding it to the cluster tomorrow is as simple as merging a puppet change and kicking es616:20
openstackgerritA change was merged to openstack-infra/config: Add a mirror33 slave node definition.
clarkbTheRonin: it does submit the changes for review16:21
clarkbTheRonin: is the act of submitting code to gerrit priveleged?16:21
jeblairTheRonin, clarkb: we could call it "jenkins.slave". since it interacts with gerrit as 'jenkins'.  that's not at all confusing16:21
jeblairclarkb: most slaves do not gave jenkins' private key for gerrit installed, that's the priv part.16:21
TheRoninjeblair: oh - btw - in the beta version of hpcloud, they have a new super really big xtralarge machine type with 128G of RAM and a boatload of processors16:21
TheRoninjeblair: :)16:22
*** pabelanger has quit IRC16:22
clarkbjeblair: ah16:22
*** mkoderer has joined #openstack-infra16:22
clarkbTheRonin: ya someone mentioned that and it made me want to spin up elasticsearch over there >_>16:22
clarkbjust have a two node cluster16:22
clarkbTheRonin: jeblair I can call it something else when I spin it up. I don't like jenkins.slave though16:23
TheRoninwhat about "steve"16:23
jeblairTheRonin: we would have to rename Jenkins in gerrit to "Steve".16:23
TheRoninjeblair: we should do that16:24
jeblairthat would really, really, really screw with the people tracking stats16:24
jeblairsuddenly some guy named Steve is leading in patches proposed and merged!16:24
*** ^d has joined #openstack-infra16:25
*** ^d has joined #openstack-infra16:25
jeblairclarkb: git.slave, gerrit.slave?16:26
jeblairprop.slave, review.slave16:27
fungisteve.slave still has a great ring to it16:27
TheRoninwhat about slave.slave16:27
*** ladquin has quit IRC16:27
TheRoninor mr.slave16:27
anteayahthen steve would be the new guy to beat16:27
clarkbjeblair: review.slave I like that one16:28
jeblairtx.slave does also have the tx creds on it.16:28
*** odyssey4me has quit IRC16:28
jeblairso it's not quite 1:116:28
jeblairi mean, these are still probably safe to stack.16:29
fungithat does a fairly good job of capturing its intent. i'm still not too keen on x.slave.o.o when we also have an x.o.o (purely from a confusion standpoint)16:29
clarkbmaybe we should go with steve and use a TXT record to explain it :)16:29
fungifor example, i always had to tread lightly around the differences between pypi.slave.o.o and pypi.o.o (before we collapsed the latter into static.o.o)16:29
*** comstud is now known as bearhands16:30
fungiTheRonin: kindly requests a rebase16:31
fungioh, and mdehaan's presentation last night was really good. i also finally got some clarification on the relationship between ansible and rpath linux... apparently while rpath used it, he developed ansible off the clock while working there and so got to take it with him after the sas buyout16:34
*** jjmb has quit IRC16:35
fungiit's compelling from the standpoint of being agentless+ssh+yaml16:35
clarkbfungi: and it has "fireball" mode for when it needs salt like performance16:35
HenryGIf I clone devstack in a freshly installed Ubuntu 12.04 server, then checkout the stable/grizzly branch of devstack, then fails on installing keystone client.
*** dina_belova has quit IRC16:36
fungiclarkb: yep. it can also do serialized small-batch operations for grouped rolling changes and the like16:36
fungiHenryG: that's pretty interesting... i will try it myself and see what happens16:37
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Trigger periodic jobs with Zuul
clarkbTheRonin: that looks like more setuptools madness16:38
clarkboh actually nevermind16:39
jeblairclarkb, TheRonin, fungi: ^ i did try to use project templates, btw, but multiple templates don't merge with each other in the way i'd expect (nor do they merge with additional jobs for a project), so i think we need to enhance that feature a bit.16:39
clarkbmore https madness16:39
clarkbHenryG: are you behind a proxy?16:39
*** ladquin has joined #openstack-infra16:39
*** ladquin has joined #openstack-infra16:39
HenryGclarkb: yes16:39
*** afazekas has joined #openstack-infra16:40
jeblairgrizzly devstack-tempest is good (a change merged to it a couple hours ago)16:40
*** jeffa has joined #openstack-infra16:40
clarkbya, sdague had mentioned proxy issues, I thought they were specifically related to git thuogh and this doesn't appear to be related to git16:40
uvirtbotLaunchpad bug 1207830 in openstack-ci "404 on some questions at" [Undecided,New]16:41
*** SergeyLukjanov has quit IRC16:41
clarkbHenryG: if you try to install d2to1 in a virtualenv does it fail with the same error?16:42
HenryGclarkb: lemme check ...16:42
clarkbI have a hunch this is related to proxy madness and not devstack16:42
reedclarkb, ah, interesting, thanks16:44
clarkbI am going to AFK for a bit to hunt down food and stuff16:44
dstufftproxy is flat out broken ins etuptools16:45
dstufftwhich setuptools installs setup_requires16:45
*** derekh has quit IRC16:45
* dstufft isn't paying attention16:45
reedclarkb, I think it's because we're in the middle of the installation of the new ask version, supporting multiple languages16:45
*** afazekas has quit IRC16:46
HenryGclarkb: yes16:48
jeblairclarkb, fungi, TheRonin: if you could review within the next 30 mins or so, that would be great (then i'll spin up new jenkinses)16:48
HenryGclarkb: wait, I tried to install keystone client.16:49
HenryGclarkb: in a virtualenv. It gave a similar error as d2to1, but on pbr16:49
*** gyee has joined #openstack-infra16:52
*** cthulhup has quit IRC16:54
TheRoninjeblair: looing16:54
TheRoninjeblair: I like that patch a lot16:55
TheRoninHenryG: are you creating the venv by hand?16:55
HenryGTheRonin: not when running stack.sh16:56
TheRoninah. gotcha16:56
TheRoninwell... ah. ok. I know what's giving you issues16:56
TheRoninand there is a change coming soon that will fix it foryou16:57
*** jpich has quit IRC16:57
HenryGTheRonin: for me? or for one of the other issues being discussed here? :)16:58
TheRoninHenryG: for one of the issues being discussed here.16:58
TheRoninHenryG: here are some underlying problems with how we're processing dependencies that are causing other work we're doing to make the pip/setuptools better to not show up immediately16:59
TheRoninso, as soon as we land the change - it will allow us to better assess the root cause and a fix for any remaining issues you have16:59
TheRoninsdague: do you have the run change done yet? or shall I do it?16:59
* HenryG deploys some more brain cells to process the parsing of TheRonin'17:00
TheRoninsdague: oh. duh. we wrote it together17:01
*** prad_ has joined #openstack-infra17:03
*** prad_ has quit IRC17:03
HenryGCan I run devstack latest master branch to deploy stable/grizzy of openstack? Will that work better?17:03
TheRoninnope. stable/grizzly of devstack should deploy stable/grizzly of openstack17:03
HenryGthought so17:04
HenryGany tips on a temp workaround?17:04
clarkbuse tsocks and a socks proxy? (I hate that I just suggested that)17:05
TheRoninpip install d2to1 ; pip install setuptools-git ; pip install pbr17:05
TheRoninthen run stack.sh17:05
*** zaro0508 has joined #openstack-infra17:05
HenryGMy pip is alread tsocksified17:06
TheRoninyah. pip will do the right thing, it's when setuptools calls easy_install that things go bad17:06
*** markmcclain has quit IRC17:06
TheRoninso the workaround is to use pip to pre-install anything that setuptools is going to attempt to install if it's not there17:06
clarkbyeah so you want to enable tsocks for all the things17:06
*** markmcclain has joined #openstack-infra17:07
*** burt has quit IRC17:08
*** markmcclain has quit IRC17:09
clarkbjeblair: in the periodic jobs change, are minute and hour transposed?17:09
*** mriedem has quit IRC17:11
openstackgerritDoug Hellmann proposed a change to openstack-dev/pbr: Ensure _run_shell_command always returns same type
thedoddjeblair: Was an announcement already made?17:11
clarkbthedodd: yes17:12
HenryGTheRonin: clarkb: trying with the pip preinstalls -- is looking much better so far ...17:12
*** mriedem has joined #openstack-infra17:12
sdagueTheRonin: I think it's under your id17:13
TheRoninsdague: yup17:13
TheRoninsdague: I just re-kicked it to see if the recent changes have fixed it yet17:13
sdagueoh, merge conflict on global requirements17:13
sdaguelet me update17:13
thedoddclarkb: Oh, I didn't see it. Was it broadcast on -dev?17:13
clarkbthedodd: it was
sdagueheh, it was my own fault, my other reqs changes landed17:14
openstackgerritSean Dague proposed a change to openstack/requirements: one global-requirements.txt to rule them all
sdagueTheRonin: ok, if you want to send that through17:15
*** vijendar has quit IRC17:17
TheRoninsdague: done17:17
TheRoninsdague: that explains why I didn't see that change when I pulled17:17
TheRoninclarkb, sdague: btw- I just realized - now that we do single-versoin-externally-managed by default17:17
TheRoninwe could move the locale dirs out of the package again if we wanted and actualy configure things to install them to /usr/share/locale ...17:18
TheRoninor /usr/local/share/locale17:18
openstackgerritA change was merged to openstack/requirements: one global-requirements.txt to rule them all
clarkbTheRonin: moving setuptools discussion here, after the giant discussion yesterday have we gotten anywhere?17:19
TheRoninclarkb: sort of ...17:19
TheRoninclarkb: right now I'm waiting to get sdague's work in17:20
*** reed has quit IRC17:20
*** prad_ has joined #openstack-infra17:20
sdagueTheRonin: so it's in17:20
TheRonincan we land this real quick:17:21
TheRoninclarkb, fungi, jeblair ?17:21
TheRoninI think without it we're still going to run in to some potential weirdness with devstack and requirments because of setup_requires17:22
TheRoninand things are going to want to suck in d2to1 which we will not have pre-installed anywhere17:22
sdagueTheRonin: you should make sure it passes the new unit tests :)17:22
sdagueor write some of your own17:22
TheRoninsdague: yes!17:22
sdagueI just rechecked it17:23
TheRoninlemme add a unittest...17:23
*** vipul is now known as vipul-away17:24
*** cthulhup has joined #openstack-infra17:24
*** zul has quit IRC17:24
*** nati_ueno has joined #openstack-infra17:25
fungiyeah, it's failing to merge, TheRonin17:25
sdagueTheRonin: ok, I sent the devstack and grenade bits off to recheck17:26
fungiHenryG: i'm going to guess you're hitting
*** zul has joined #openstack-infra17:27
fungiif nobody's pointed you in that direction yet (just skimmed scrollback)17:27
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Trigger periodic jobs with Zuul
*** cybertron_ has joined #openstack-infra17:31
jeblairi approved that ^ based on earlier feedback from clarkb and TheRonin17:31
HenryGfungi: thanks. she is up now17:31
*** bnemec has quit IRC17:32
jeblairi have confirmed the resize of graphite17:32
*** cthulhup has quit IRC17:32
*** vipul-away is now known as vipul17:33
openstackgerritA change was merged to openstack-infra/config: Trigger periodic jobs with Zuul
fungijeblair: yeah, was just leaving comments but i'll follow up with a fix for it instead17:33
jeblairfungi: let me respond to your comments17:34
fungijeblair: oh, those weren't for the same time anyway, so i guess this is backward-compatible17:34
jeblairfungi: those jobs are going to to be per-jenkins-master17:35
jeblairfungi: so i don't want zuul to trigger them17:35
jeblair(that's in the commit msg)17:35
fungiahh, right-o17:35
fungiand yes, now i see it in the commit message too, sorry! :/17:35
openstackgerritMonty Taylor proposed a change to openstack/requirements: Update ourselves to ourselves
openstackgerritMonty Taylor proposed a change to openstack/requirements: Update to also write a stock
clarkbjeblair: patchset2 lgmt17:38
TheRoninok - there's the change update with a test17:38
clarkbTheRonin: the assert in that test seems pretty weak17:41
clarkbTheRonin: could you do a multiline assert like we do in JJB?17:41
clarkbit gives you nice diff output too17:41
* TheRonin goes to look17:41
marunso once this setuptools upgrade is working, I can haz wheel?17:43
marunbinary python packages, how I've missed you!17:43
TheRoninclarkb: not without some further refactoring17:45
TheRoninclarkb: because I don't have a good wy to import to pull the exptected string out17:45
TheRoninand copying it would be ugly17:45
TheRoninclarkb: can we take that as a note for improvement?17:45
clarkbthe change did fail tests btw17:46
clarkbTheRonin: is -1 verified17:47
TheRoninclarkb: that makes no sense!!!17:48
fungiTheRonin: i find it a little odd that a script which is Copyright 2012 OpenStack, LLC is generating files which are Copyright (c) 2013 Hewlett-Packard Development Company, L.P.17:48
TheRoninfungi: well, the code was copied in from things that already had that header on them17:48
TheRoninalthough, honestly, the file is so small, I'd also be ok with just skipping the header17:48
TheRoninexcept I think that'll cause hacking to complain17:48
jeblairTheRonin: then disable the check17:49
fungii am on the fence about whether generated files should be copyrighted, but it's a gray area (especially since it's a static blob)17:49
TheRoninjeblair: in all of the projects?17:49
clarkbTheRonin: jeblair fungi the check is only that the license header exists iirc17:49
clarkbit does not enforce (C) lines17:49
TheRoninthat is true17:49
TheRoninI could remove the (c) line17:49
clarkbso you can remove the (C) line or cp the (C) line to the top of the script doing the generation17:49
TheRoninI don't understand why there is a feeling that the copyright lines need to match17:50
jeblairTheRonin: i do not think they need to match17:50
fungii think it's okay as-is, just feels odd17:50
fungiand wanted to make sure there was some thought behind that choice17:51
jeblairTheRonin: i was being partly facetious, in as much as i have not actually reviewed the change.  what i will say is that it sounds like we may be discussing a not-copyrightable piece of code.17:51
TheRonink. although I think I do take a point from clarkb ... the file contains code that is copyright something else, so that should be added to the file's header17:51
jeblairTheRonin: and if the hacking check is forcing us to have a header on a not-copyrightable piece of code, then hacking is wrong.17:51
jeblairbut as i said, that's just a general comment, i haven't looked at this change.17:51
TheRoninthe hacking check checks that all of our python files have apache license headers17:51
* fungi quickly goes and readds copyright and license lines to all the empty files17:52
jeblairTheRonin: empty ones too?17:52
TheRoninI do not know - I do not believe it tries to enforce on empty files17:52
TheRoninjog0: ?17:52
jeblairsadly, deciding whether something is copyrightable is not an automatable task.  in reality it requires a judge, in practice, we can safely assume that a 2-line boilerplate file is not copyrightable.17:53
jeblairi'm looking now.  this is 4 lines of boilerplate.  all bets are off.  :)17:54
fungijeblair: oh, well this one is four very, very short lines of code and one line of all-caps commentary17:54
*** markmcclain has joined #openstack-infra17:54
jeblairfungi: that commentary may push it over the edge.  it's transformative.17:54
*** vijendar has joined #openstack-infra17:55
*** vijendar has joined #openstack-infra17:55
*** fbo is now known as fbo_away17:55
clarkbwhat about
fungianyway, i do feel like there is something nonoptimal about having a check which is forcing us to inject license and copyright details into very small generated files, but my bigger point (similar to clarkb's) is that it'soff to have a copyright 2012 script generating copyright 2013 content17:55
clarkbfungi: it does not enforce copyright...17:56
jeblairTheRonin: i feel that the apache header and copyright is harmless, because it doesn't mean anything anyway.  if our tool really probits omitting that, then it's just being silly.  i will not -1 the change17:56
clarkbgranted the apache license does not make much sense outside the scope of copyright...17:56
jeblairclarkb: it's a copyright license.  :)17:56
clarkbI think the reason it ended up in the tool was to avoid discussions like this though17:57
clarkbparticularly in reviews17:57
clarkbbecause apparently a lot of misinformation and bad stuff had been going into some of them. Now at least we are consistently bad17:57
fungithough looking at it another way, the copyright line for the blob pretty clearly refers to the blob, including its representation within the containing script, so really it's probably appropriate just not all claimed in the same place in the file17:58
* fungi feels he has painted far too much of this bikeshed now17:58
clarkb(things like reviewers decrementing patches because they did not attribute copyright to the foundation for example)17:58
jeblairclarkb: yeah, yeah, i was there for that.17:59
jeblairclarkb: my point stands, though, this is wrong, but harmless.17:59
fungialso, IOError: [Errno 2] No such file or directory: 'tests/files/gr-base.txt'17:59
TheRoninright. I'm trying to figure that out right now18:00
*** Ryan_Lane has joined #openstack-infra18:01
*** pabelanger has joined #openstack-infra18:02
mesteryAnyone using the latest devstack with Fedora? I consistently this this bug I just opened:
uvirtbotLaunchpad bug 1207855 in devstack "On Fedora, devstack removes pip, rendering it unusable" [Undecided,New]18:03
mesterySeems as if it's removing pip using yum, and then it's game over. :(18:04
ekarlso-has mordred disappeared ?18:06
harlowjaso are we going to allow up to 99 char lines now ??? :)18:06
harlowja :)18:06
openstackgerritMonty Taylor proposed a change to openstack/requirements: Add pep8 checks
openstackgerritMonty Taylor proposed a change to openstack/requirements: Update ourselves to ourselves
openstackgerritMonty Taylor proposed a change to openstack/requirements: Update to also write a stock
TheRoninok. that should solve the test problem18:07
clarkbekarlso-: TheRonin is mordred18:07
marunmestery: devstack is in the middle of a mess18:07
clarkbharlowja: not now, we bump pep8 checking rules at the beginning of cycles18:07
TheRoninit turns out, ading the test increased the number of tests to greater than the nubmer of runners - so we got to the point where one runner ran two tests18:07
TheRoninand thus the lack of cleanup was exposed18:08
marunmestery: there is a proposed fix for fedora/rhel18:08
harlowjaclarkb awesome, hope that it will happen someday, 80 lchar lines and me don't get along :-P18:08
mesterymarun: Thanks, and I agree, it's awful right now.18:08
mesterymarun: I'll check that out, thank you!18:08
TheRoninmestery, marun: yeah - we're working on it :)18:08
*** sarob has joined #openstack-infra18:08
clarkbI tended to not get along with 80 character columns but then I started using a lot of vsplits in vim18:09
mesteryTheRonin: Thank you!18:09
TheRoninmarun: wow - that's a stellar patch18:09
sdagueTheRonin: hacking only cares if the apache license is there, not copyright18:10
TheRoninsdague: that is correct18:10
*** dims has quit IRC18:10
TheRoninclarkb, fungi: added the license line from the included snippet to the top of the including file - because I do believe that the whole file should indicate the total story within itself18:11
*** sarob_ has joined #openstack-infra18:11
openstackgerritClark Boylan proposed a change to openstack-infra/config: Rename tx slave to proposal slave.
clarkbjeblair: TheRonin fungi ^ with that in I can spin up the new node. Assuming the name is fine with everyone18:12
TheRoninsdague, clarkb, fungi, jeblair: so, if you guys could blow through the three above, ( being the most important) it would be a kindness18:12
dstufftharlowja: that got reverted18:12
dstufftPEP8 is back at 7918:13
harlowjadurn, ha18:13
dstufftBut PEP8 is techincally for the standard library and external projects are allowed to do w/e the fuck they want18:13
clarkbdstufft: was that because of Guido's last minute comments?18:13
dstufftso if openstack wants 99 they can just do it18:13
sdagueTheRonin: actually, can I get +2 on requirements ? :)18:13
dstufftPersonally I'm 79 for life :)18:14
dstufftclarkb: yea I think so18:14
TheRoninsdague: uhm. yeah18:14
jeblairdstufft: technically anyone can do anything, but pep8 bills itself as a standard for how people should write python, and standards are important so that people can write consistent tooling for them.18:14
fungidstufft: all my xterms are 80 characters wide ;)18:14
dstufftjeblair: no it doesn't18:14
TheRoninsdague: done18:14
dstufftjeblair: first life on PEP8 "This document gives coding conventions for the Python code comprising the standard library in the main Python distribution"18:15
jeblairdstufft: 4th line: Many projects have their own coding style guidelines. In the event of any conflicts, such project-specific guides take precedence for that project.18:15
jeblairdstufft: that could have said "no other projects should use this" but it did not.18:15
*** emagana has quit IRC18:15
*** sarob has quit IRC18:15
TheRoninI agree with jeblair. it does leave the door open to people using these in other projects18:15
TheRoninalso, regardless of author's intent in pep8, it has become the defacto community python style guide for all python code18:16
TheRoninwhich is a good case in point that you cannot control how someone will consume a thing that you write18:16
jeblairdstufft: at any rate, your point is taken that the primary audience is certainly the python stdlib itself, and that should be their continued concern.18:16
jeblairTheRonin: yeah; ideas are crazy that way.  :)18:17
dstufftWhy would it tell other projects *not* to use it? That's pointless to do. But it doesn't bill itself as how all people should write python code :) (ALthough I do Pep8 all my code, I'm jsut being pedantic)18:17
dstufftMostly because some projects don't do PEP8 and people cite the PEP8 at them and tell them they are doing it wrong18:17
openstackgerritChuck Short proposed a change to openstack-dev/hacking: python3: Fix python3 tox installation
clarkbto be fair the rule about spaces vs tabs should be a hard rule for all python >_> /me hides18:18
sdaguehmmm... TheRonin
sdagueany idea why that races in the test18:18
clarkbsdague: fixes it18:19
clarkbcurious that TheRonin went from print with parens to without in that diff...18:19
TheRoninsdague: yes18:19
TheRoninclarkb: I fixed the pep8 in the next patch18:19
TheRoninsdague: it's was lack of a cleanup for the cd18:20
dstufftclarkb: at least Python3 makes it a hard error to mix tabs and spaces in the same file (I believe)18:20
TheRoninI believe you are right18:20
jeblairlaunching jenkins01 jenkins02 devstack-launch01 devstack-launch0218:20
openstackgerritClark Boylan proposed a change to openstack-infra/config: Rename tx slave to proposal slave.
openstackgerritA change was merged to openstack/requirements: Update to also write a stock
clarkb^ it bothers me when languages require that the file name match some internal content18:21
uvirtbotclarkb: Error: "it" is not a valid command.18:21
*** rnirmal has joined #openstack-infra18:21
sdagueclarkb: there is some value in the "explicit vs. implicit" to do that18:22
jeblairTheRonin: what else do you need reviewed?18:22
TheRoninsdague: grenade doesn't like
sdagueTheRonin: is that because we are running on 2 node18:23
sdaguesorry 2 core18:23
TheRoninjeblair: I tihnk we're good for now18:23
clarkbI just realized has a -1 with no content. Is it possible to force gerrit to make you write something?18:23
*** thedodd has quit IRC18:23
TheRoninsdague: yeah - so, before I added the test, we had exactly as many tests as workers18:23
sdagueI guess I was running 8 core locally, so all the tests were on separate processes18:23
TheRoninwhich meant ath setup just happened to only run once per worker18:23
sdagueTheRonin: ok, gotcha18:23
sdagueyep, makes perfect sense18:23
sdaguejust wanted to make sure I fully grokked it18:23
TheRoninclarkb: probalby with 2.6 and prolog18:24
sdagueso I don't do it wrong again in the future18:24
jeblairclarkb: would it help if it said "don't merge this" or "aoeu" ? :/18:24
sdagueTheRonin: so that's expected18:24
TheRoninsdague: yes18:24
clarkbjeblair: no... but maybe it would prompt reviewers to think about why content is required. I am sure I have given the occasional negative feedback without actual comments18:24
sdaguesorry, the grenade fail, expected18:25
TheRoninoh - it is?18:25
sdaguenow I have to get grenade right18:25
fungiclarkb: i personally think it's reasonable to ignore or possibly delete -1 votes with no rationale provided18:25
sdague apparently wasn't quite it18:25
*** jog0 is now known as flashgordon18:25
openstackgerritChuck Short proposed a change to openstack-infra/config: Add python3 gate for openstack-dev/hacking
sdagueso it looks like pbr path fail
fungiclarkb: though if i deleted someone's vote i would certainly encourage them to re-add it with an explanation18:26
*** SergeyLukjanov has joined #openstack-infra18:26
*** vipul is now known as vipul-away18:28
clarkbfungi: it just occured to me that the ipv6 check in launch node won't play nice with hp. maybe we can set that with an env var and set the env var in the files we source?18:29
fungiclarkb: good point!18:29
fungiclarkb: i think that is precisely the right solution18:30
flashgordonTheRonin: we dont look for a license if file is under 10 lines
TheRoninflashgordon: oh great!18:30
*** _TheDodd_ has joined #openstack-infra18:30
TheRoninthen we could actually remove the header from the generated setup.py18:30
flashgordonalso we broke the license test into two tests: 1 if has license header at all. and 2 if the header is correct18:31
flashgordonas in no typos etc18:31
clarkbif I can get eyes on I can start spinning up that new slave18:33
fungisdague: bit of a bikeshed on the install_infra addition to devstack a couple days ago, but technically neither pbr nor requirements are in openstack-infra?18:33
sdaguedtroyer: you about?18:34
sdaguefungi: which program are they tracked under?18:34
fungisdague: hmmm, good question. i had assumed oslo but now don't remember where that thread ended up18:35
dtroyersdague: yup18:35
sdaguedtroyer: ok, I think the last magic key to the kingdom is in grenade18:35
sdaguedtroyer: in this -
*** melwitt has joined #openstack-infra18:36
sdagueit seems that when we get to target we don't have the PBR url18:36
sdagueor requirements18:36
sdagueis that because we don't source target's stackrc?18:36
sdagueand where should we pull that in?18:36
*** gyee has quit IRC18:37
dtroyerwe do source the target stackrc, but it is in each upgrade script so they can stand alone18:37
dtroyerI think that is what is missing18:38
dtroyerin upgrade-keystone it;s at line 60 where that sort of thing lives18:38
sdaguedtroyer: ok18:38
sdaguecool, let me go run at that18:38
dtroyerthat's the stuff that does that we can't otherwise get to18:39
openstackgerritChuck Short proposed a change to openstack-dev/hacking: python3: Use six.moves for ConfigParser
sdagueok, lets see how it goes -
*** _TheDodd_ has quit IRC18:43
sdagueTheRonin: so grenade has to go first here, and if it works, we can land the devstack bit18:43
TheRoninsdague: I'm excited18:44
*** _TheDodd_ has joined #openstack-infra18:44
sdagueit would be a pretty rocking way to end the week18:44
TheRoninsdague: then we get to backport this stuff in to devstack stable/grizzly right?18:44
sdagueI think18:44
sdaguewell, we should, it won't be any more broken than it currently is if we don't18:45
sdagueI've I remember how we changed all this :)18:45
sdaguebut, you know, let me celebrate the little victories here first18:45
*** cybertron_ has quit IRC18:47
*** bnemec has joined #openstack-infra18:47
TheRoninI'm so excited18:51
*** emagana has joined #openstack-infra18:51
ekarlso-TheRonin: Pecan + WSME or Pecan + JSONSchema ?18:51
TheRoninekarlso-: pecan + wsme is what dhellmann has been preaching18:52
clarkbdoesn't JSONSchema fit a different role?18:52
TheRonindhellmann: ekarlso- is working on some things and we've been telling him that he should use pecan/wsme for it - and to bug you about anything that doesn't work18:52
dhellmannTheRonin: ack18:53
TheRonindhellmann: where did the wsme->stackforge thing ever fall?18:53
TheRonindhellmann: was it the hundreds of tox envs thing?18:54
dhellmannTheRonin: I ran out of time this week, but plan to get back on it after my vacation.18:54
zulfungi: i dont think i missed anything
TheRonindhellmann: ok. cool. let me know if I can be useful18:54
TheRonindhellmann: also - vacation? I'm not sure that should be allowed18:54
fungizul: i'm playing with your stack of hacking fixes now just to make sure18:54
zulfungi:  cool18:54
TheRoninzul: !!!18:54
zulfungi:  there is a whole of fixes that is needed for hacking i think still18:54
TheRoninzul: so, when are we going to get backports of setuptools 0.9.8 and pip 1.4 into cloud-archive?18:55
fungizul: oh, so not just the three out there right now?18:55
dhellmannalso, jeblair objected to a change we'll need for that, and I need to talk about it with him. I think the commit log message gave the wrong impression.18:55
clarkbfungi: are we ready to take the plunge with ?18:55
zulTheRonin:  eh?18:55
TheRoninzul: new setuptools and new pip. in cloud-archive. will make all the bunnyrabbits happier18:55
TheRoninyou like bunnies don't you?18:55
fungiclarkb: yeah i'm willing to fix whatever 39580 breaks (i played around with it and it seemed to work, but you never know what corner cases to look for i guess)18:56
zulfungi:  yeah i kind of just started it and some people wanted to the python3 gates before accepting python3 patches18:56
zulTheRonin:  bunnies are fine but why again?18:57
dhellmannTheRonin: You can let me know if you think I'm missing anything in or
clarkbfungi: you tested it with older python as well?18:57
clarkbfungi: my biggest concern is breaking stuff that almost works today18:57
clarkbfungi: but, I think the change is safe18:57
TheRoninzul: because old setuptools installed via distro packages on machines causes absolute living hell when one wants to install new via pip18:57
TheRoninbecause of the upgrade hell18:57
*** pcrews has joined #openstack-infra18:57
fungiclarkb: tried it on a 26 and a 27 machine18:57
TheRoninbut, dpkg, it turns out, is good at handling the upgrade18:57
clarkbfungi: perfect18:57
TheRoninzul: so if the distro actually updates, the breaking will go away18:57
clarkbTheRonin: can you review ?18:58
*** zaro0508 has quit IRC18:58
TheRoninclarkb: yup18:58
zulTheRonin: lemme think about it18:58
*** ^d is now known as ^demon|lunch18:58
clarkbTheRonin: feel free to approve if it looks good18:58
clarkbsounds like fungi and I can babysit18:58
zulTheRonin:  i dont want to go down the rabbithole per say ;)18:58
TheRoninzul: k. just saying - it's one of those things that will be easier for you18:59
TheRoninzul: because you don't have to figure out how to upgrade setuptools and pip using setuptools and pip without breaking setuptools and pip18:59
zulTheRonin:  yes that makes me uneasy18:59
clarkbTheRonin: new thought on the setuptools issue18:59
dstufftTheRonin: FWIW I've been digging into the Fedora vs Debian thing19:00
clarkbTheRonin: what if we snapshot some images where the only changes we have made to the image is fixing the problem19:00
clarkbTheRonin: and just boot those19:00
TheRonindstufft: oh god. how's that going19:00
dstufftTheRonin: I'm gonna be writing a PEP19:00
dstufftto make dual directories the standard19:00
TheRoninlet me know when/where I can vote on that19:00
dstufftMostly becaue I hate myself and love pain19:00
TheRoninclarkb: what?19:00
TheRoninuhm - I mean, it means that our puppet modules cease being self-contained19:01
jeblairclarkb: --19:01
TheRoninI'd rather get things into an apt repo19:01
jeblairclarkb: i thought we had a plan; is this solving a problem in that plan?19:01
clarkbjeblair: we do have a plan, I am throwing this out there because TheRonin is throwing ideas out there for working around the plan19:02
TheRoninI am?19:02
mrodden1dstufft: what do you mean by "dual directories" ?19:02
clarkbTheRonin: yes putting packages in a special PPA that work around the problem is similar to using prebaked images from a puppet perspective19:02
TheRoninclarkb: nono - I didn't mean zul doing that to solve our world19:02
jeblairdhellmann: about to run out, but i understand slightly more now that i know the context; could you leave a note about that, and what the envs are expected to look like?19:02
TheRoninclarkb: no. I'm asking him to put backports into cloudarchive so that people in general don't get screwed19:02
dhellmannjeblair: definitely19:03
dstufftmrodden1: using pip and apt-get together on debian more or less works because apt-get installs to one directory, and pip installs to a different directory19:03
*** luisg has joined #openstack-infra19:03
dstufftmrodden1: vs systems like Fedora, where they both install to the same directory19:03
dhellmannjeblair: your points about the testr stuff are valid, though, since these projects don't use it19:03
dstufftwhich means they stomp all over each other19:03
dhellmannjeblair: so the best solution may be a new script that does less than the current one19:04
mrodden1oh site-packages vs dist-packages19:04
mrodden1Fedora doesn't do that... ?19:04
mrodden1thats horrible19:04
clarkbTheRonin: if you are using cloud archive and installing via debs you don't need setuptools right?19:04
jeblairdhellmann: yeah, probably so.  and sadly, ideally it should be called 'run-tox'.  we should move the existing one to run-testr.  but that can all wait if you want.  :)19:05
dstufftmrodden1: for instance ``apt-get install python-pip && pip install --uphtade pip && apt-get remove python-pip`` works, but ``yum install python-pip && pip install --upgrade pip && yum erase python-pip`` looks like it works, but leaves random files laying in /usr/lib/.../site-packages that nobody owns19:05
dhellmannjeblair: I can stage the 2 commits, no problem19:05
dstufftmrodden1: Well sort of, in debian both use dist-packages, deb install to /usr/lib/../dist-packages, and pip installs to /usr/local/lib/../dist-packages19:05
clarkbdstufft: about the upgrading pip with pip, why does pip tromp over the system install?19:06
dstufftclarkb: because Fedora tells it too19:06
clarkbdstufft: if pip goes in /usr/local when installing itself all is well19:06
clarkbdstufft: no on debian19:06
clarkbits actually not 100% clean on debian either19:06
dstufftclarkb: it should isntall to usr/local?19:06
*** harlowja_ has joined #openstack-infra19:07
clarkbdstufft: I think the majority of it does but pip finds the existing install and tries removing some of it19:07
clarkbor something19:07
clarkbdstufft: it made our puppet unhappy at one point19:07
dstufftpip --version19:07
dstufftpip 1.4 from /usr/local/lib/python2.7/dist-packages (python 2.7)19:07
dstufftclarkb: oh it probably does remove it19:07
* dhellmann thinks someone needs to teach gerrit markdown19:08
dstufftbecause it doesn't have any concept of what is owned by the system and what isn't19:08
dstufftit queries the sys.path looking for pip to be installed, and removes it19:08
clarkbdstufft: that implies to me it would leave existing pip alone19:08
clarkband you rely on your PATH to figure out which one to use19:08
mrodden1oh yeah... guess it does go into /usr/local/lib/19:09
dstufftclarkb: It doesn't look at PATH at all, it looks at sys.path and queries those directories for a egg-info or dist-info directory that says "I have pip installed in this directory"19:09
clarkbdstufft: I meant from a user's perspective19:09
*** mgagne1 has joined #openstack-infra19:10
*** mgagne1 has quit IRC19:10
*** mgagne1 has joined #openstack-infra19:10
clarkbdstufft: pip doesn't seem to do this extra special stuff when you install something else19:10
openstackgerritChuck Short proposed a change to openstack-infra/config: Add python3 gate for openstack-dev/hacking
dstufftclarkb: what extra special stuff?19:10
clarkbdstufft: removing system pip19:10
dstufft--upgrade always uninstalls the existing thing19:10
clarkbdstufft: even if it is installed in a different root?19:10
dstufftclarkb: yes, because it's on the sys.path19:10
*** samalba_ has joined #openstack-infra19:11
*** nati_uen_ has joined #openstack-infra19:11
dstufftsys.path == the database of what's isntalled19:12
fungiyeah, for me the freaky is when i have a deb of python-foo installed to meet some dependency of another deb, but then need a newer version of python-foo and grab it from pip, then realize it's broken or whatever and decide to pull it back out, but pip uninstall python-foo then blows away files from the deb19:12
*** dolphm_ has left #openstack-infra19:12
dstufftfungi: techincally pip install foo is what blew it away19:12
fungiahh, point19:12
dstufftthe main problem is pip removes it, but doesn't tell apt-get (or whatever) that it did19:13
dstufftso apt-get thinks it's still there19:13
clarkbinteresting I thought pip was special cased for some reason, but probably only noticed it with pip because it is gets installed with apt and everything else gets installed with pip19:13
fungiat any rate, the workaround is to use equivs to provide python-foo from a dummy package and then do whatever i want for it with pip, basically the same as the rpm workaround proposed to devstack19:13
clarkbdstufft: yup19:13
*** cody-somerville_ has joined #openstack-infra19:13
*** SergeyLukjanov has quit IRC19:14
dstufftclarkb: I don't have a good solution for that yet, it's possible that pip can leave the theortical "distribution owned" directory alone and just shadow the install in the "pip owned" directory… but in that case /usr/bin/pip will point to the new pip anyways unless /usr/bin/pip excludes the pip owned directory19:15
clarkbdstufft: because /usr/bin/pip is an entrypoint console script?19:15
dstufftbecause only one thing will be importable at ``import pip``19:15
dstufftPart of my PEP might be to include a flag that explicitly only asks for the distribution owned directory19:16
dstufftso it might make sense for /usr/bin/pip to use that19:16
sdaguedtroyer: ok, new grenade fail19:18
dstufftso it might be ``python`` -> sys.path has /usr/../site-packages and /usr/local/../site-packages, ``python --only-distro`` -> sys.path has /usr/, and then the python tools become smart enough to leave the distro owned /usr one alone19:18
clarkbdstufft: that makes sense to me19:19
dstufft(totally spit balling FWIW, no idea how well it will work)19:19
clarkbya, but the generally idea makes sense from a don't break apt-get perspective19:19
clarkbwow *general19:20
*** _TheDodd_ has quit IRC19:20
*** sarob_ has quit IRC19:20
*** nati_ueno has quit IRC19:20
*** luis_ has quit IRC19:20
*** cody-somerville has quit IRC19:20
*** mgagne has quit IRC19:20
*** harlowja has quit IRC19:20
*** samalba has quit IRC19:20
*** _TheDodd_ has joined #openstack-infra19:20
*** samalba_ is now known as samalba19:20
dstufftyea it keeps pip and apt/get/yum/whatever from breaking each other, the big question is how will things work with two different versions installed to sys.path, one being shadowed by something earlier in sys.path19:20
*** sarob has joined #openstack-infra19:21
dstufftif all the import paths remain the same it should work just fine, it gets tricker if there are different import paths :[19:21
sdaguelooks like new nova isn't picking up new oslo -
sdaguedtroyer: anything thoughts on that?19:23
clarkbI ninja approved (JJB pathignore plugin) for the same reason I ninja approved rvm-env yesterday.19:24
clarkbsame thing with both changes were rebases with clean diffs and happy check jobs19:25
*** sarob has quit IRC19:25
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Pathignore plugin support added
sdagueTheRonin: actually, why do we install cli tools before we process requirements -
*** mrodden1 is now known as mrodden19:26
sdagueTheRonin: 2013-08-02 18:54:59.662 is the timestamp19:26
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add support for the Summary Report publisher
*** vipul-away is now known as vipul19:30
*** psedlak has quit IRC19:30
* TheRonin lookie19:33
TheRoninsdague: what do you mean by "before we process requirements" ?19:34
sdagueso before nova devel installs things, it installs the clis19:34
sdagueand we are blowing up trying to call nova-manage19:35
mtreinishsdague, clarkb: no more sqlalchemy logs in glance!
sdaguebecause of wrong oslo19:35
clarkbmtreinish: \o/19:35
clarkbhmm where did reed go19:35
sdagueunless I'm reading the logs wrong19:36
clarkbwe should get reed to at least glance at before we approve it
openstackgerritChuck Short proposed a change to openstack-dev/hacking: python3: Use six.moves for ConfigParser
mesterymarun: Thanks for pointing me at this patch to fix devstack with Fedora ( Works great for me!19:37
marunmestery: cool beans19:37
mesterymarun: Yes, very happy.19:38
marunmestery: I wonder if the patch shouldn't just install a good version of setuptools, though...19:38
TheRoninsdague: right - we've always installed the clis before the servers19:38
marunmestery: i.e. from rpm19:38
TheRoninsdague: so that git cli would be present and cause server to not attempt to install it19:38
mesterymarun: That may make more sense, yes.19:38
sdagueTheRonin: so is there a thing we need to do here to get past it then?19:38
TheRoninsdague: but we do the install in two steps - we pip install -r requirements.txt19:39
TheRoninthen the develop19:39
TheRoninso that we don't cause easy_install to do things instead of pip19:39
sdagueso why isn't it picking up the right oslo?19:39
TheRoninhow did a _different_ oslo get installed?19:39
sdagueon base19:39
sdaguewe install old oslo on base19:39
sdaguerun grizzly19:39
sdagueshut it down19:40
sdaguenew oslo, from git, installs correctly19:40
sdagueI can see that19:40
TheRoninbut old oslo is still around?19:40
sdaguewell.... I don't know19:40
sdagueit must be, because it's complaining about a missing option19:40
TheRoninsdague: perhaps instead of python develop we should just be doing pip install -U -e .19:40
*** ruhe has joined #openstack-infra19:41
TheRoninand get rid of the 'install requirements.txt first' thing19:41
sdagueI'm going to run locally and see if I can track it down19:41
clarkb is the answer ?19:43
clarkbthe state of that project is actually quite confusing19:43
TheRoninsdague: I'm trying out a pip install -e patch19:44
*** bnemec has quit IRC19:45
TheRoninsdague: oh! grendae is using an old version setup_develop19:45
*** bnemec has joined #openstack-infra19:45
sdagueTheRonin: we're running the unfubar though19:46
TheRoninthat's not it19:46
TheRoningrenade is still parsing the output of egg_info19:46
dtroyerThe entire functions file needs to be copied19:46
TheRoninwhich we stopped doing in devstakc a while ago19:46
TheRoninbut ...19:46
*** ^demon|lunch is now known as ^d19:47
TheRonintry that on for size19:47
TheRoninthat should always work19:47
TheRoninand should always do the right thing19:47
sdague$1 is a directory19:47
TheRoninpip install -e ../path/to/dir19:48
TheRoninis like cd ../path/to/dir; python develop19:48
TheRoninexcept, it's actually more like doing _way_ more things than that :)19:48
TheRonindstufft: ^^ I'm right about the above, right?19:49
TheRonindstufft: ?19:49
TheRonindtroyer: I agree though - perhaps a cp of the functions file or something ...19:49
clarkbdoesn't -e simply determine which python to install under?19:49
TheRonin-e means "editable"19:49
clarkboh I thought it was the env selection flag19:49
clarkb-E is the one I am confusing with -e19:49
sdaguedtroyer: we need all the function file updated as well/19:50
sdaguedtroyer: that's not going to mess up stable?19:50
dtroyeralthough…it might be better to specifically import the target devstack functions file19:50
sdagueyeh, that's what I was wondering19:51
dtroyerI don't think it affects base since we run there19:51
dtroyerthere are some functions that the higher-level scripts want to use but inside the upgrade scripts we should use target devstack functions19:52
sdagueok, well trying once again19:54
sdague sits on TheRonin's changes19:54
*** hashar has joined #openstack-infra19:55
TheRoninfingers crossed19:55
sdaguewait... I lost the actual updater....19:57
sdaguetooo many branches19:58
sdagueok, another attempt19:58
*** belliott has joined #openstack-infra20:03
belliottgerrit question.  this review has a successful jenkins run, but no Verified flag.  how do i fix it?
clarkbbelliott: you need to have someon approve the change, that will rerun the gate tests and if they pass merge the change20:05
belliottclarkb: great, ty20:05
clarkbfungi: you about? I am beginning to think we should just approve
clarkbfungi: because I don't want it to go in so late that we don't catch any potential issues20:06
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: More launch improvements
fungiclarkb: go for it20:07
fungiclarkb: also there's your ping6 fix idea ^ (i had a documentation fix i needed to add for the launch script anyway)20:07
clarkbcool, I will review that one first to give TheRonin a little more time to review the py3k change20:08
clarkbTheRonin: you did say you would review it :)20:08
* sdague watches jenkins console log to see how grenade is doing20:08
flashgordonsdague: can you review,n,z20:08
flashgordononce those merge I want to cut hacking 0.720:09
fungiclarkb: also, i have a missing word in my commit message on 39992... i'll fix it if it bothers you20:09
*** belliott is now known as weimguy20:09
clarkbfungi: "to enabled"? feel free to update if it bothers you, but I was able to understand20:10
sdagueflashgordon: ok... one sec20:10
fungiclarkb: yup20:10
*** weimguy is now known as bearhandssunkmyb20:11
flashgordonsdague: thanks20:11
fungithe documentation fix was because that just bit me on another machine (where we probably shouldn't be installing the mysql puppet module anyway, but that's another day)20:11
clarkbfungi: which machine?20:11
*** bearhandssunkmyb is now known as spawnofbelliott20:11
sdagueflashgordon: on this one
fungiclarkb: mirror33.slave20:12
sdagueare we missing the merge test case?20:12
clarkbfungi: ok, I am approving the testr thing20:12
flashgordonsdague: hmm, so right now its either custom or default20:13
*** ruhe has quit IRC20:13
fungiclarkb: cool, keeping an eye out for badnez20:13
sdagueflashgordon: seems like it should be default + custom20:13
flashgordonsdague: yeah20:13
flashgordonyour right20:13
flashgordonalthough merging makes things a little less obvios for the end user20:14
flashgordonso I can see it either way20:14
flashgordonthe idea of explicit + implicit can be confusing20:15
sdagueok, I'm going to -1 it with that feedback20:15
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Not sending description or disabled by default
flashgordonso I guess I am on the fence20:15
sdagueflashgordon: well everyone needs to ignore those modules20:15
*** weshay has quit IRC20:15
openstackgerritA change was merged to openstack-dev/hacking: Extract configs-related code to the configs helper
flashgordonsdague: right but  Ithink it may be better to be explicit when possible20:16
flashgordonalthough I cannot see a case where someone would want to not ignore those two20:16
sdagueyeh, I think because it's in requirements hacking should own it20:17
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added Robot Framework reposts publisher
openstackgerritA change was merged to openstack-infra/config: Pass testr and a PYTHON envvar.
sdagueanyway, I could be argued away from that pov20:17
sdaguebut that's my first go20:17
*** bnemec has quit IRC20:17
sdaguedtroyer: ok.... so I updated functions20:17
sdaguenow, the fun begins20:17
sdaguebecause functions needs REQUIREMENTS_DIR20:18
*** bnemec has joined #openstack-infra20:18
sdaguehow do I unwind this...20:18
dtroyerthat's in target stackrc right?  or has that not landed yet?20:19
flashgordonsdague: there is a case where someone would not want ot ignore thsoe two20:19
flashgordonif they just use teh #noqa on the offending lines20:19
sdagueit's in target stackrc20:19
flashgordonsdague: but we can cross that line when we get there20:20
sdagueflashgordon: sure20:20
sdaguedtroyer: we load functions really early in grenade though20:20
dtroyerthat shouldn't be an issue, it isn't needed until runtime20:21
*** jeffa has left #openstack-infra20:21
sdaguebut it's evaled right away20:22
flashgordonfungi: can we add the hacking py33 test?20:22
fungiflashgordon: yes, i already +2'd it20:22
fungiif nobody else wants to review, i can go ahead and approve20:22
*** hashar has left #openstack-infra20:23
flashgordonfungi: can you share a link20:23
sdaguedtroyer: L114220:23
sdaguethat's the current issue20:23
fungihacking py3k job addition
fungigoing once, twice...20:23
fungii also held off waiting for jenkins to +1 it after my vote20:24
fungibut looks like it was happy as well20:24
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add sixth elasticsearch node.
flashgordonfungi: LGTM let the py33 begin20:24
clarkbmerging ^^ and kicking the elasticsearch6 node tomorrow will add it to the cluster20:24
fungibam. approved flashgordon20:25
clarkb(I plan on spinning up the new node this afternoon and will merge 39997 tomorrow when I am ready to add it to the cluster)20:25
clarkbchange 40k is coming up20:25
clarkbI should've waited on 39997 >_>20:25
fungiflashgordon: as i mentioned to zul, once you land your remaining py3k fixes to hacking, you'll want to follow up with a change to take out the voting:false there20:25
flashgordonfungi: right20:26
openstackgerritA change was merged to openstack-infra/config: Add python3 gate for openstack-dev/hacking
fungiflashgordon: in about 30 minutes you should be able to recheck a hacking change and see a non-voting failure for gate-hacking-python3320:27
flashgordonfungi: awesome20:28
*** boris-42 has joined #openstack-infra20:30
clarkbI feel like jeblair must have shut out the outside world to spin up his preciouses20:30
jeblairclarkb: approved20:31
jeblairer i mean +2d20:31
jeblairclarkb: i'll let you aprv it.20:31
jeblairi am going to be afk for a bit now, though.20:31
*** bnemec has quit IRC20:32
clarkbI have approved it, will spin up new node once that goes in20:32
*** bnemec has joined #openstack-infra20:33
sdagueman, I've got a hard stop in 30 minutes and the magical ponies aren't quite there yet20:33
openstackgerritA change was merged to openstack-infra/config: Rename tx slave to proposal slave.
dtroyersdague: I think we goofed with REQUIREMENTS_DIR…it belongs in stackrc since _everything_ needs it20:35
sdaguedtroyer: ok20:36
dtroyerit'll break if someone redefines DEST but I'll live with that for now20:36
dtroyeroh wait…grenade does that20:36
sdagueso can I just add it in grenaderc?20:37
sdaguesorry, my brain is kind of fried on a friday20:37
dtroyeri was looking at that, I think that's a better move20:37
sdaguegrenaderc still sources late20:37
dtroyerdoesn't matter, as long as it is before setup_develop() is called20:38
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add Python 3.3 PyPI mirror jobs
fungiooh, SO close20:39
fungiquick somebody grab review (warhammer) 40k20:40
sdaguedtroyer: - like that20:41
clarkbfungi: do it20:42
dtroyerclose…use TARGET_RELEASE_DIR rather than DEST20:43
openstackgerritMonty Taylor proposed a change to openstack/requirements: Remove unneeded tests dir reference
TheRoninI missed it20:45
*** datsun180b has joined #openstack-infra20:46
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Branch-specific PyPI mirrors
TheRonindstufft: 20:45:00            juice | it appears that devstack is using apt-get install python-netaddr which puts the package in /usr/lib/python and then later uses pip install20:46
TheRonin                          | netaddr >= 0.7.6 which puts the package in /usr/LOCAL/lib/python20:46
TheRonin20:45:09            juice | when nova boots up it sees /usr/lib/python first and then fails because it cannot find the more recent version in /usr/LOCAL/lib/python20:46
clarkb40k is a draft too :(20:47
TheRonindstufft: we're obviously working on this in the general area20:47
clarkbwhy why would you do that to us :)20:47
fungi"Install os-apply-config in a virtualenv" ...the pen and paper rpg geek in me is not amused20:47
TheRonindstufft: but the description I just paste-bombed...20:47
TheRonindstufft: does that make any sense to you?20:47
clarkboh not a draft, just stupid firefox20:47
dstufftTheRonin: it should see /usr/local/lib first20:48
dstufftat least on ubuntu that's higher in the sys.path20:48
dstufftAnd pip install should (currently) uninstall the one in /usr/lib/python/dist-packages because reasons20:48
TheRoninSlickNik: juice: can you guys confirm that you've got an ubuntu machine with both /usr/lib/python netaddr from apt and also a /usr/local/lib/python netaddr from pip?20:50
SlickNikyes TheRonin that's exactly it20:50
juicebut the machine was clean of python-netaddr until the script was run20:50
juicethe script = devstack.sh20:50
TheRoninso this pure precise + devstack ?20:50
TheRoninjuice: can you check what version of pip is on the box?20:50
juicesorry 1.520:51
TheRoninor 1.4?20:51
juicevipul ran pure devstack on a clean box and has 1.420:53
juiceignore my previous answers20:53
TheRoninok. cool20:53
dstufftlink to the script?20:53
TheRonindstufft: it's devstack ...20:53
dstufftTheRonin: openstack has like 4 different repos ;P I don't know where all the stuff is tored ;P20:54
TheRonindstufft: haha20:54
TheRonindstufft: I'm gwanna see if'n I can get you a smaller script...20:55
juicevipul found that there are file in each of the projects that contain a list of apt-get python packages20:55
fungiclarkb: i think my run-tox change needs a revert... do you agree?
juicewe also verified in the output of devstack/ the installation of python-netaddr vi apt-get20:56
fungiclarkb: somehow i missed that it seems we're not getting subunit into the virtualenv on py27 but seem to do on py26?20:56
dtroyerthree projects have netaddr in their prereq files: horizon, neutron and nova20:56
juiceapt-get's latest version is 0.7.5 of netaddr just one bug fix version away from satisfying nova's requirements.txt :)20:56
dtroyertat's what does the package install20:56
clarkbfungi: I don't follow that was the py27 test20:57
clarkbfungi: I think we should remove that list count check at the end of the testr post tox section20:57
clarkbfungi: instead of doing a full revert20:57
fungiclarkb: doing now20:57
*** changbl has quit IRC20:57
*** thomasm has quit IRC20:58
dtroyerjuice: try this:  remove python-netaddr with dpkg and remove it from files/apts/* (there are 4 occurrances) and try again20:58
dtroyerthat should let it only install with pip20:59
dstufftwould something apt-get install after the pip install?20:59
dtroyernot that I know of; the only place does that is near the beginning based on what is in file/apts/*20:59
juicedstufft: I don't think apt-get cares about what is installed by pip20:59
juiceit looks like they are using different repos21:00
clarkbspinning up now21:00
dstufftjuice: yea that was my point ;) If something did an apt-get instlal <after> the pip install occured then i'd expect both to exist21:00
juicerepos == directories21:00
dstufftbut pip should uninstall the apt-get installed one first21:00
juicedstufft: this seems to be the case21:00
dstufftbecause the two systems fundamentally don't know about each other, but pip understands sys.path so it looks at sys.path for things to uninstall21:01
juicedstufft: we tried a pip uninstall of netaddr and it errors out because while it holds onto the latest version, it doesn't seem to be able to uninstall the older one installed by apt21:01
dstufftapt-get doesn't do that (obviously).21:01
TheRonindstufft: /usr/lib/python2.7/dist-packages/netaddr21:02
juiceso to dtroyers suggestion you need to use dpkg or apt-get remove to clear the error21:02
TheRoninstill exists21:02
TheRonindstufft: after I pip install -U netaddr21:02
clarkbTheRonin: fungi: tx.slave is a 2GB node. Should I stick to that size in rs nova?21:02
TheRoninclarkb: yah21:02
dstufftTheRonin: what's inside of it?21:02
dstufftis it the entire netaddr21:02
dstufftor only a few files?21:02
TheRoninwhole thing21:02
juiceTheRonin: what does pip freeze show as the version now21:02
fungiclarkb: i expect so. our mirror slaves are 2g too21:03
juice0.7.5 (apt) or 0.7.10 (pip)21:03
dstufftTheRonin: what steps to repro, apt-get install python-netaddr && pip install netaddr ?21:03
TheRonindstufft: yes21:03
TheRonindstufft: so - that's ? #121:03
dstufftany particular ubuntu version?21:03
juiceso now nova will work21:03
TheRonindstufft: question 2 is why would pkg_resources entrypoint loading find the/usr/lib/ version?21:03
TheRonindstufft: precise21:03
TheRonindstufft: also - we pip install -U pip first to get pip 1.421:03
TheRoninbut I do not know if that matters or not21:04
TheRoninjuice: what did you do to make it work?21:04
TheRonindtroyer: why are we installing some of the python things from apt?21:04
juiceso after running the current devstack and waiting for the nova failure, you can do an apt-get remove python-netaddr21:04
juicethis at least gets you to the next package with same problem21:05
juicewhich is com221:05
dtroyerwe've been back and forth on this… the argument for using packages is that is likely to more closely match what is going to be deployed21:05
juiceor something to that effect21:05
sdaguedtroyer: - so we're missing xslt-config21:05
sdagueI've got to call it a week, if anyone wants to run with it, feel free21:05
TheRonindtroyer: but we install a crap-ton of packages from pip - so I'm not sure it is any closer to a real deployment21:06
TheRonininstalling some from one and some from the other seems to be the source of endless problems21:06
dtroyerTheRonin: I know, like I said this has been a see-saw21:06
uvirtbotdstufft: Error: "^" is not a valid command.21:06
juiceI am relatively new to this but I would favor pip over apt21:06
TheRonindtroyer: :)21:06
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Stop comparing tests counts
fungiclarkb: ^21:06
juiceapt often seem terribly out of date21:06
TheRonindtroyer: I think there's so many of us who have backgrouds in distros and want to focus on the packages...21:06
juicewhat is the benefit of using apt for python packages?21:07
dstufftInstalling random python stuff is far more consistent via pip across various distros; I'm of the opinion that it's the distros job to work out integrationg it into the system21:07
TheRoninbut the reality is that we're tracking a global set of requiremnts now based on pip versions but teting on precise21:07
dtroyerstablility and not breaking other things on the system that use them21:07
dstufftthat's basically their value proposition21:07
dstufftwe integrate shit into the system for you21:07
*** toddmorey1 has joined #openstack-infra21:07
*** esp has joined #openstack-infra21:08
clarkbfungi: the set +e can be removed too21:08
TheRoninright - I think that ubuntu will likely take whatever versions of python libraries we wind up with, make sure they are in the next release of ubuntu along with the next version of openstack ... but they're never going to have the packages for the libraries we need on the old veversoin of the distro we're doing current dev on21:08
*** UtahDave has quit IRC21:08
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Stop comparing tests counts
fungiclarkb: yeah, just realized that ^21:08
*** toddmorey has quit IRC21:08
sdagueTheRonin / dtroyer got to run, out of town guests coming in, and it's my b-day this weekend, so no computing for me. It's close, but will have to tackle on monday to get us the last mile.21:09
flashgordonfungi: py33 is working21:09
sdagueunless someone else wants to run with it21:09
dtroyerthis is mostly why other stuff just installs their complete runtime with the app…see vendor-supplied-java-runtime utilization21:09
TheRoninsdague: ok. is there anything I can poke at when you're out?21:09
fungiflashgordon: awesome!21:09
flashgordonsdague: happy birthday21:09
dstufftdamn python has a bunch of shit installed by default on ubuntu21:09
TheRonindstufft: yeah it does21:09
sdagueflashgordon: thanks21:09
clarkbmorganfainberg: it would be super duper if you could review 4000621:09
dtroyersdague: happy bitthday!  time off well earned21:09
clarkbTheRonin: ^21:09
morganfainbergclarkb: sure let me take a look21:10
sdagueTheRonin: it's just sorting the grenade fails, that last patch is close, looks like we are actually just missing a dep somewhere21:10
sdagueanyway, later folks, see you monday21:10
TheRoninsdague: have a great weekend!21:10
clarkbmorganfainberg: thanks I meant to ping mordred and tab completion failed me21:10
morganfainbergclarkb: hehe sorry21:10
morganfainbergclarkb: i was like "weell… sure. but uh, not really my forte"21:11
fungisince mordred is TheRonin at the moment, it's likely to happen again ;)21:11
morganfainbergTheRonin: Silly name collisions21:11
*** morganfainberg is now known as needscoffee21:11
needscoffeefungi: :P solution! hehe21:11
needscoffeeit is friday, right?21:12
fungiin some parts of the world, for a little while still at least21:12
needscoffeehehe. yeah.21:12
datsun180b"It's Friday somewhere" would be a pretty catchy line for the Galactic Federation21:12
datsun180breasons for space wayfaring: snarky t-shirts21:12
needscoffeedatsun180b: i'm so stealing that when I make my own federation....21:13
fungiit's friday happy hour somewhere, even moreso21:13
*** needscoffee is now known as morganfainberg21:13
TheRoninjuice, dtroyer:
datsun180bpython-cmd2 and one other package get caught in the blast too21:13
dstufftTheRonin: i see why21:14
clarkbfungi: I think we should approve soon if TheRonin doesn't review it21:14
fungiclarkb: i agree21:14
dstufftTheRonin: python-pip and python-netaddr are packaged differently21:14
TheRonindstufft: oh yeah?21:14
TheRoninfungi, clarkb approved21:15
fungiTheRonin: thanks! was breaking unit tests for a lot of projects21:15
dstufftTheRonin: python-pip includes the python packaging specific file that tells pips what files were installed as part of that distribution, so pip knows what files to remove when it does the uninstall port of pip install --upgrade pip21:15
dstufftTheRonin: python-netaddr doesn't have that file, so pip sees it as an installed package with zero files installed, so it just removes the entry21:16
clarkbfungi: did building mirror33 take forever?21:16
clarkbI am still stuck on BUILD 0 with proposal.slave21:16
openstackgerritA change was merged to openstack-infra/config: Stop comparing tests counts
TheRonindstufft: so - specifically, we shoudl avoid python-netaddr like the plague21:16
fungiclarkb: not that i noticed, though i didn't time it21:16
dstufftlook at /usr/lib/python2.7/dist-packages/pip-*.egg-info* vs vs /usr/lib/python2.7/dist-packages/netaddr-*.egg-info when installed from apt21:17
fungiclarkb: i'm going to fast-track that patch onto the unit test slaves21:17
dstufftTheRonin: yea21:17
clarkbfungi: go for it21:17
fungisince it missed the 2115 pulse21:17
dstufftTheRonin: the files are SOURCES.txt, top_level.txt, RECORD, (Stuff like that) inside of an egg-info or dist-info directory21:19
dstufftthis probably explains why clarkb thought pip had special cased behavior too :)21:20
dstufft-> afk21:20
clarkbyay I am not crazy21:20
clarkbfungi: it is going now... I wonder if the queues just got backed up21:20
fungiclarkb: clogged tubes between the rack spaces21:20
*** sandywalsh has quit IRC21:22
*** dhellmann is now known as dhellmann_21:23
flashgordonfungi: something is funny with the py26 and py27 hacking tests now
fungiclarkb: also think i figured out the issue with the previous change. i had tested its effects on testr but not on and for some reason using #!/usr/bin/env python failed to spot $PYTHON defined i the calling environment21:25
fungiflashgordon: yes, they're being fixed now21:25
flashgordonfungi: thanks21:25
fungiflashgordon: fix is being fast-tracked onto the slaves and should subside within another minute or two21:26
clarkbfungi: the testr stuff that was pulled out wasn't working properly bceause it is too naive21:26
fungiyeah, that too21:26
fungiokay, all centos6, precise and precisepy3k slaves have that fix updated onto them now21:31
*** boris-42 has quit IRC21:32
fungii assume we're not doing voting testr-based jobs anywhere else at this point21:32
*** rnirmal has quit IRC21:33
*** sandywalsh has joined #openstack-infra21:34
*** krtaylor has quit IRC21:34
fungii need to fire up the grill but will brb and keep an eye on new jobs to make sure the issues subside21:35
*** nicedice has quit IRC21:35
*** nicedice has joined #openstack-infra21:36
TheRoninwe still install python-m2crypto from apt21:36
dstufftYou pretty much have too21:38
dstufftOn ubuntu anyways21:38
mroddenso... every one of the jenkins slaves runs mongodb?21:38
dstufftM2Crypto isn't maintained anymore21:38
dstufftit depends on SSLv2 existing or you get compile time errors21:38
dstufftUbuntu removes SSLv2 from OpenSSL21:38
dstufftand patches M2Crypto to not need SSLv221:39
dstufftthat patch doesn't exist on the unmaintained one on PyPI21:39
dstufftso pip install M2Crypto failes on ubuntu21:39
openstackgerritAnita Kuno proposed a change to openstack-infra/storyboard: Edit README to include amended initial data instructions.
dstufftatleast afaik it's not maintained anymore21:40
clarkbmrodden: no, all of the precise and centos slaves do21:40
dtroyeronly nova las it listed in devstack still, but not in requirements.txt21:40
clarkbmrodden: but the devstack slaves do not unless told by devstack to do so21:40
dtroyerI think I have some housekeeping to do...21:41
mroddenclarkb: oh its in the standard packages list21:41
dstufftTheRonin: bug 13024 in Build/Install "Unable to use with OpenSSL 1.0+ due to missing SSL v2 support PATCH INCLUDED" [Blocker,New]21:42
mroddenclarkb: we aren't worried about having a mongodb instance running all time time on those slaves?21:42
*** datsun180b has quit IRC21:43
mroddeni think the debian packages start it up as soon as it installs21:43
clarkbmrodden: not really. Do you have a particular concern?21:43
mroddenidk just seems a bit insecure21:44
mroddengranted... they are only jenkins slaves21:44
clarkbmrodden: they are insecure :)21:44
clarkbmrodden: but you can only get to the servers there from localhost so we have some accountability through jenkins and gerrit21:44
*** mriedem has quit IRC21:45
*** boris-42 has joined #openstack-infra21:45
dstufftTheRonin: how extensive is the use of M2Crypto21:45
dstufftTheRonin: you might have a better time removing it for something that actually works in a venv21:46
zuldhellmann_:  ping21:49
clarkbadding proposal.slave to jenkins now21:49
clarkbI will change the label on tx afterwards and next week we can delete tx21:49
dtroyerdstufft: I just grepped the nova sources and it's mentioned but never imported…so it appears to be unused21:50
*** mrodden has quit IRC21:50
dstufftdtroyer: oh goody21:50
dstufftdtroyer: that makes it easy to kill it with fire21:50
*** ^d has quit IRC21:51
fungiclarkb: i think the only project testing mongo is ceilo, and pretty sure it uses a temporary local instance rather than the daemon, so we might be able to just override it not to start21:52
fungiunless its use changed substantially from what i originally discussed with them21:53
clarkbfungi: is it any worse than mysql or postgresql?21:53
fungiclarkb: afaik, not really21:54
clarkbI think I am happy leaving it as is then21:54
openstackgerritClark Boylan proposed a change to openstack-infra/config: Replace tx node label with proposal in jobs.
clarkband that can go in at any time21:57
*** UtahDave has joined #openstack-infra22:01
*** rcleere has quit IRC22:01
fungisame story for
nati_uen_Hi Infra team, it looks like 20% neutron gate fails. How do you think this number?22:04
clarkbnati_uen_: its getting better for sure22:04
nati_uen_yeah, last week was 98% :X22:04
nati_uen_so which percentage we can enable the gating?22:05
nati_uen_may be I should check non-neutron gating22:05
*** _TheDodd_ has quit IRC22:05
*** prad_ has quit IRC22:05
clarkbnati_uen_: check is harder to depend on22:05
clarkbnati_uen_: because we don't expect them to pass22:05
funginati_uen_: yes, something close to the % for the equivalent non-neutron tempest jobs in the gate pipeline should be our target there22:06
*** bnemec has quit IRC22:06
nati_uen_I found analize page22:06
nati_uen_which can show the success and failure rate using this query22:07
nati_uen_BTW, I got "502 Proxy Error" in kibana many times. may be, we should increase timeout in the proxy22:07
*** bnemec has joined #openstack-infra22:08
clarkbnati_uen_: yes, that is a known problem but kibana is silly nad doesn't make it configurable by default22:08
clarkbnati_uen_: which is why we haven't changed it22:08
nati_uen_ah I got it22:08
clarkbnati_uen_: also part of the problem is that query is using all of the available memory on one of the nodes which slows things down22:08
nati_uen_ah  @tags:console.html  AND (@message:"Finished: SUCCESS"  OR @message:"Finished: FAILURE") AND @fields.build_name:gate-tempest-devstack-vm-neutron  This one ?22:09
fungishut down ALL the trash compactors22:09
clarkbnati_uen_: ya, I think it has to do with the length of time you are search for22:09
clarkbnati_uen_: the longer the timeframe the more content it loads into memory22:09
nati_uen_clarkb: gotcha22:10
*** mrodden has joined #openstack-infra22:10
clarkbnati_uen_: I have it set to evict things from cache after that happens so that the cluster doesn't die22:10
*** bnemec has quit IRC22:10
clarkbnati_uen_: in the past it would die once a week or so and it was from an expensive query that never had its cache flushed22:10
nati_uen_yeah, we have fuge logs..22:11
clarkbI am adding another node tomorrow, but I am happy that the cluster did die this time :) it means my changes are workign22:11
*** bnemec has joined #openstack-infra22:11
nati_uen_clarkb: cool!22:11
clarkbnati_uen_: also, there is a better way to count these things now if you can figure out graphite22:12
nati_uen_clarkb: What's graphite ?22:12
clarkbnati_uen_: zuul is keeping track of the passes and fails for each job in each pipeline22:12
nati_uen_you guys adding new features as the speed of light!22:12
clarkbnati_uen_: its a number statistical graph system thing. http://graphite.openstack.org22:12
clarkbnati_uen_: it draws the graphs on the zuul status page22:13
fungii can't figure out if is supposed to be a joke22:13
uvirtbotLaunchpad bug 1207801 in openstack-ci "gate-swift-python26 TestServer test_wait failed" [Undecided,New]22:13
*** UtahDave has quit IRC22:13
nati_uen_clarkb: do you have manual for it?22:13
clarkbnati_uen_: I don't and I can never figure it out...22:14
clarkbfungi: :)22:14
clarkbfungi: I would just change it to swift and acll it good :)22:14
nati_uen_clarkb: ha ha. I'll try to figure it out22:14
clarkbnati_uen_: when jeblair gets back you will want to ask him about graphite22:15
clarkbnati_uen_: but graphite is good at counting things and giving you graphs22:15
clarkblogstash is better for finding particular errors and the like22:15
nati_uen_clarkb: gotcha22:15
*** psedlak has joined #openstack-infra22:15
*** lcestari has quit IRC22:15
*** pcrews has quit IRC22:18
*** krtaylor has joined #openstack-infra22:20
*** psedlak has quit IRC22:20
*** cybertron_ has joined #openstack-infra22:20
clarkbspinning up elasticsearch6 now22:22
*** bnemec has quit IRC22:22
*** woodspa has quit IRC22:24
openstackgerritChuck Short proposed a change to openstack-dev/hacking: python3: Fix python3 tox installation
*** reed has joined #openstack-infra22:25
*** psedlak has joined #openstack-infra22:39
openstackgerritA change was merged to openstack-dev/hacking: python3: Fix python3 tox installation
hub_capso has anyone seen issue w/ netaddr+12.04 ubuntu in devstack?22:43
clarkbelasticsearch6 is up, I have stopped es on it, and will approve the puppet change related to it tomorrow after which I can restart es22:43
clarkbhub_cap: yes there was a big discussion earlier today22:43
hub_capi can pull down a fresh devstack on a fresh 12.04.2 vm and it wont run. no extra commands, just add ubuntu user to sudoers...22:43
nati_uen_clarkb: Hi it looks like neutron gate failure is same level as non-neutron now.  gate-tempest-devstack-vm-full 27.27%  gate-tempest-devstack-vm-neutron 28%22:43
hub_capclarkb: WOO YES22:44
hub_capany review i can track etc?22:44
hub_capclarkb: ^ ^22:44
nati_uen_clarkb: in 4 hours. (i got proxy error for 12 hour check)22:44
clarkbhub_cap: if you don't have it in scrollback there are logs at http://eavesdrop.openstack.org22:44
clarkbnati_uen_: cool good to know22:44
hub_capoh i stayed in room. thx clarkb22:44
nati_uen_Ah 12 hour check is also works.  gate-tempest-devstack-vm-full:18.75% gate-tempest-devstack-vm-neutron:13.21%22:45
clarkbnati_uen_: that looks good. Was there a particular fix that went in that made it better?22:46
nati_uen_so it looks like we can enable neutron gating22:46
nati_uen_clarkb: yesterday, olso fixed. but this is for 98% failure one22:47
clarkbnati_uen_: we probably won't to get a broader opinion as the decision to make it non voting involved a few people22:47
clarkbnati_uen_: but I agree with you22:47
hub_caplol clarkb it was a guy that works on trove (like me)22:47
nati_uen_clarkb: yeah I agree22:47
nati_uen_clarkb: I'll send mail for that mailing list22:48
clarkbnati_uen_: what was wrong with oslo?22:48
nati_uen_clarkb: it changes exception type. and it brokes all neutron agent22:48
*** vijendar has quit IRC22:48
nati_uen_clarkb: After the my first fix, I didn't compare failure late with non-neutron gating22:48
nati_uen_clarkb: so I just thought it looks same gating fails with same error..22:49
clarkbnati_uen_: so the first fix fixed one thing but at the same time another issues was introduced somewhere else?22:49
nati_uen_clarkb: yeah,, we got two issue after the first issue22:49
nati_uen_one was pbr issue22:50
nati_uen_For the first issue, I may see the same issue, but failure rate looks like same level as non-neutron one22:50
nati_uen_I'll keep investigation, but may be we can turn on the vorting..22:51
clarkbI agree. I have been watching it over the last day and a half or so and it has gotten noticeably better22:51
nati_uen_clarkb: ok let's me ask opnions22:51
fungithe graph jeblair linked earlier showed tempest neutron jobs near 0% failure in the gate after 0000 utc today22:52
clarkboh ya jeblair made a graph22:53
clarkbnati_uen_: that should give you an idea of how to use graphite. basically you edit the url and it draws a different graph for you22:53
*** esp has quit IRC22:55
nati_uen_I have created non-neutron one
clarkbnati_uen_: notice the recent blip of failures is consistent across the two22:58
*** vijendar has joined #openstack-infra22:58
clarkbnati_uen_: I think it looks good to me. I would vote to make the test voting again22:58
nati_uen_clarkb: yeah22:58
nati_uen_clarkb: Thanks22:58
nati_uen_is may be 0000 utc22:58
*** sarob has joined #openstack-infra22:59
*** zul has quit IRC22:59
clarkbnati_uen_: it merged at 0500 UTC23:00
clarkbI think23:00
clarkbunless gerrit is showing me utc timestamps23:01
nati_uen_That's wired.. Gating should fail without this fix23:01
nati_uen_may be, some timezone configrations?23:02
clarkbnati_uen_: oh, it was 0500 august 123:02
clarkbnati_uen_: then started passing at 0000 august 223:02
nati_uen_clarkb: ah23:02
nati_uen_so another fix?23:02
clarkbso probably sdague's change to use oslo config master in devstack gate went in around that time23:02
nati_uen_do you know reviews for it?23:03
clarkbnati_uen_: ya, sdague has been working to make sure we use tip of trunk across all of the projects in the devstack gate23:03
clarkbnati_uen_: I don't but I might be able to find them give me am inute23:03
clarkbnati_uen_: but that merged before the oslo.config change so I am not sure if that was it23:04
nati_uen_ ?23:04
clarkbnati_uen_: yes that one and the one I linked they are both working towards the same thing23:05
clarkbbut they merged well before the gate started passing23:05
*** mgagne1 has quit IRC23:05
*** vijendar has quit IRC23:05
clarkbso I don't think that was what made the difference23:06
nati_uen_clarkb: BTW, how do you checking when it merged?23:06
clarkbnati_uen_: on look at the updated timestamp23:06
clarkbusually that timestamp is when it merged23:06
clarkb(or close enough)23:06
nati_uen_Ah I din't know that :)23:07
*** pcrews has joined #openstack-infra23:09
nati_uen_may be it is some time lag?23:09
clarkbnati_uen_: I have a hunch that maybe those cahnges didn't work as expected and there is a "fix" commit that went in more recently23:10
nati_uen_yeah.. possible23:10
nati_uen_But i can't see any of it on neutron and pbr and olso23:10
*** sarob has quit IRC23:10
clarkbnati_uen_: I think it will be in devstack or devstack-gate23:11
*** sarob has joined #openstack-infra23:11
nati_uen_hmm I can't see which one fixes..23:12
clarkbyeah I don't see any either23:12
clarkbmaybe oslo did a release?23:13
clarkband we aren't using the source installed version but a pip installed version instead?23:13
nati_uen_I thinks Sean's fix is working in my local23:14
nati_uen_so it looks like we are using source installed version23:14
nati_uen_may be 3rd party lib update?23:14
clarkbcould be23:14
nati_uen_may be the run was waiting in the queue23:16
nati_uen_so it was running with old configrations23:16
nati_uen_Isn't this possible to the cause of the issue?23:17
*** changbl has joined #openstack-infra23:17
clarkbnati_uen_: it shouldn't be because we stack things in the gate queue23:17
clarkband test in parallel optimisticly23:17
nati_uen_I send the mail for the openstack-dev23:17
nati_uen_is UTC?23:18
clarkbyes I think so23:18
clarkbunless, is it using the browser supplied tz?23:19
nati_uen_gerrit shows me PDT23:19
*** sarob has quit IRC23:19
flashgordonmorganfainberg: it works, I anyone seeing a netaddr issue?23:19
clarkbnati_uen_: graphite looks like PDT to me23:19
morganfainbergflashgordon: did you mean someone else? :P23:19
clarkbnati_uen_: so I think the oslo changes in devstack and devstack gate match up23:19
flashgordonmorganfainberg: woops sorry two msgs gon munged23:20
morganfainberghehe all good.23:20
*** sarob_ has joined #openstack-infra23:20
morganfainbergjust don't want someone to miss an important message cause it was directed to the wrong person :)23:20
flashgordonanyone seeing issues with netaddr I get a 0.7.5 installed and nova needs >0.7.623:20
nati_uen_clarkb: I'm grad to figure out it! :)23:20
clarkbnati_uen_: now the full test :)23:21
nati_uen_clarkb: yeah.23:21
clarkbjeblair: hi there23:23
flashgordonlooks like a bad env (apt installed an old version for me)23:26
fungii suppose i never notice tz preference matching in apps except when it fails to show me the utc times i ask for23:31
clarkbjeblair: fungi: elasticsearch6 seems ready to go. I will do that tomorrow morning. shouldn't impact anything, it will just rebalance quicker with less index load23:33
fungiclarkb: great!23:33
fungii'll probably be around, except when i go for groceries23:34
clarkbproposal.slave has been running jobs and they are succeeding23:41
*** Ryan_Lane has quit IRC23:47
*** pentameter has quit IRC23:50
*** pcrews has quit IRC23:57

Generated by 2.14.0 by Marius Gedminas - find it at!