Tuesday, 2014-01-28

mroddenjog0: the git.openstack.org one is behind a commit00:00
emaganaInfra masters! Have a question about my latest gerrit commit00:00
emaganaThe dependency is gone and this is why Jenkins is failing.00:00
emaganaanyone could help me to add the dependency back?00:00
openstackgerritA change was merged to openstack-infra/devstack-gate: Add support for grenade-dsvm-rolling test  https://review.openstack.org/6419900:00
openstackgerritA change was merged to openstack-infra/devstack-gate: Add Trove testing support  https://review.openstack.org/6504000:01
*** ociuhandu has quit IRC00:01
jeblairjog0: i triggered replication and git.o.o is updated now00:01
*** openstack has joined #openstack-infra00:02
*** mdenny has quit IRC00:02
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Ensure a recent(48h) check result  https://review.openstack.org/6950800:04
jeblairclarkb, mordred, fungi, sdague, mikal: ^ that's where we can vote on (b).  :)00:04
clarkbemagana: git review -d thedependency then git review -d thefailingchange, then git rebase -i branchfordependency, git review00:05
clarkbmordred: right running manage-projects once I get through more review is on my list of things to do00:05
clarkbI approved at least 2 changes that need it to be run00:06
emaganaclarkb: let me try that00:06
openstackgerritA change was merged to openstack-infra/config: Switch whisper to master branch  https://review.openstack.org/6618900:07
emaganaclarkb: what would be "thefailingchange"?00:07
emaganaclarkb: My commit https://review.openstack.org/#/c/67312/ should depend on this one https://review.openstack.org/#/c/67210/00:07
clarkbemagana: the one that failed because it doesn't have a dependency00:07
*** jcooley_ has quit IRC00:08
jog0jeblair: thanks, too bad I didn't want the my patch to land00:09
zaromordred: how to fix that?00:09
*** jcooley_ has joined #openstack-infra00:09
zaromordred: i'm restarting review-dev00:09
*** thuc_ has quit IRC00:10
zaroclarify, just restarting gerrit00:10
mordredzaro: puppet. my current ssh key in puppet is different than the one that was there pre-christmas - so I can't see in00:11
*** ken1ohmichi has joined #openstack-infra00:11
mordredclarkb: would you prefer me to do the manage-projects run? I'm happy to do that?00:11
mordredclarkb: and while I do, I can verify that the config change doesn't break things - or I can not do that00:12
*** CaptTofu has joined #openstack-infra00:12
clarkbmordred: please do run it :)00:12
*** wenlock has quit IRC00:12
zaromordred: clarkb is gonna take care of you.00:13
*** yamahata__ has quit IRC00:13
mordredawesome. I got it00:14
openstackgerritlifeless proposed a change to openstack-infra/config: Use the openstack pypi mirror for tripleo images.  https://review.openstack.org/6930900:14
openstackgerritlifeless proposed a change to openstack-infra/config: Run gate-tripleo-deploy on devstack-gate changes.  https://review.openstack.org/6927300:14
openstackgerritlifeless proposed a change to openstack-infra/config: Run gate-triple-deploy on tripleo-ci changes.  https://review.openstack.org/6927200:14
clarkbmordred: is camelot the new key?00:14
mordredclarkb: yes00:14
clarkbmordred: you should beable to log in now00:14
mordredclarkb: thank you00:14
clarkbI removed your old key for completeness00:14
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add support for Fedora 20 to nodepool  https://review.openstack.org/6951000:15
openstackgerritlifeless proposed a change to openstack-infra/config: Set a ratelimit for tripleo-test-cloud.  https://review.openstack.org/6851000:16
openstackgerritlifeless proposed a change to openstack-infra/config: Set appropriate rate-limit for RAX clouds.  https://review.openstack.org/6850900:16
openstackgerritlifeless proposed a change to openstack-infra/config: Set HP cloud rate limits.  https://review.openstack.org/6851200:16
lifelessfungi: rebased ^00:16
openstackgerritA change was merged to openstack-infra/config: Add requirements gate for solum project  https://review.openstack.org/6634000:16
clarkbpleia2: do you have the ability to test that locally? last time we tried adding fedora to nodepool it broke because fedora, paramiko, and sudo do not play nice together00:16
*** markmcclain has quit IRC00:16
openstackgerritA change was merged to openstack-infra/config: Add manuals-upstream-translation-update to api-site  https://review.openstack.org/6651900:16
clarkbpleia2: fedora requires a tty for sudo and paramiko as used by nodepool doesn't allocate a tty00:16
openstackgerritA change was merged to openstack-infra/config: Fix CLI args for welcome-message  https://review.openstack.org/6662300:16
*** jcooley_ has quit IRC00:16
clarkbpleia2: though that was fedora 19 iirc00:16
openstackgerritMichael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter.  https://review.openstack.org/6588500:16
pleia2clarkb: the plan for now is for it to go to the tripleo cloud, which I'll test it on, and it's a WIP for now since I still have some puppet work to do00:17
*** dcramer_ has joined #openstack-infra00:17
pleia2clarkb: thanks though, I'll make note of those issues :)00:17
clarkbpleia2: you will test it against tripleo cloud without that change merging? (just clarifying as it was a bit of a mess last time, also the puppet work should be mostly done)00:18
pleia2clarkb: yeah, planning on keeping it in WIP until I test it00:19
pleia2working with some redhat folks on this, so I wanted something up on review as I go along00:19
*** SumitNaiksatam has joined #openstack-infra00:19
*** thuc has joined #openstack-infra00:22
openstackgerritJoshua Harlow proposed a change to openstack-infra/config: Name adjustments for taskflow jobs  https://review.openstack.org/6951300:22
clarkbsdague: there is a bug for the ipv6 thing with some of fungi's initial debugging00:23
harlowjahi guys, could i get a quick review of 69513, causing commits to fail due to the name change00:23
*** hogepodge has quit IRC00:23
harlowjamuch appreciated :)00:24
clarkbharlowja: how did that change merge if the names are different?00:24
clarkbalso you can make these changes in a backward compat manner if you add the new envs, merge the config change, remove the old envs00:25
harlowjahow did it merge, well failed on a recent taskflow review, so i guess config changes don't cause a check on the taskflow repo before merging?00:26
clarkbbut I am curious how the change that updated tox.ini managed to merge if the envs were different00:26
harlowjathat merged recently, before those envs were inplace (which afaik is recent)00:26
*** matsuhashi has joined #openstack-infra00:27
harlowjamerged @ 11am (pst) - https://review.openstack.org/#/c/65135/00:27
harlowjaso change before that to tox.ini didn't update that review, a mistake i guess, then review goes through in 65135, then next review hits wrong env names00:27
*** morganfainberg is now known as morganfainberg|z00:27
clarkboh it was a change to config not to tox.ini that broke00:28
harlowjaone or the other, changes are brand new, so thats part of it00:28
harlowjafrom here on out it shouldn't be possible to do said things (once the corret names are in place)00:28
openstackgerritA change was merged to openstack-infra/config: Add info about browser incompatibility to etherpad.rst  https://review.openstack.org/6726200:30
harlowjaso afaik clarkb  i got the names right this time :)00:32
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Split the config out into two files  https://review.openstack.org/6950300:33
openstackgerritAaron Greengrass proposed a change to openstack-infra/config: Enable PATH for exec functions  https://review.openstack.org/6942700:34
*** dims has quit IRC00:34
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Split the config out into two files  https://review.openstack.org/6950300:36
*** dims has joined #openstack-infra00:36
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Split the config out into two files  https://review.openstack.org/6950300:39
openstackgerritA change was merged to openstack-infra/config: Add Periodic Translation Updates to ops-guide  https://review.openstack.org/6702200:40
openstackgerritA change was merged to openstack-infra/config: gerritbot: Add API doc git notifications on #openstack-doc  https://review.openstack.org/6757300:41
mordredjeblair, clarkb: ok. https://review.openstack.org/#/c/69503/ has been emperically tested on review-dev. I'm comfortable with it now00:43
*** ICmonitor has joined #openstack-infra00:43
openstackgerritA change was merged to openstack-infra/config: Add gates for API projects and operations-guide  https://review.openstack.org/6739400:43
clarkbmordred: it is starred00:43
*** ICmonitor1 has quit IRC00:44
fungimordred: it worked with gerrit 2.8 on review-dev, huh? neat00:44
mordredfungi: yah. woot00:44
*** habdi has quit IRC00:45
*** zz_ewindisch is now known as ewindisch00:45
zaromordred: heading out until later.  do you think you can restore jeepyb to master on review-dev when you are done?00:45
*** virmitio has quit IRC00:45
harlowjalol, so another question, we recently merged the usage of 'openstack_citest' in taskflow for db testing00:47
openstackgerritA change was merged to openstack-infra/config: Remove non-voting documentation gate job  https://review.openstack.org/6770200:47
harlowjawe of course clean the db, but also just found that the following happens00:47
harlowjaOperationalError: (OperationalError) database "openstack_citest" is being accessed by other users00:47
harlowja2014-01-27 18:54:21.776 | DETAIL:  There are 1 other session(s) using the database.00:47
harlowjaif say nova is using that db00:47
harlowjaand taskflow is also00:47
harlowjaand taskflow and nova try to clean that db by doing, 'DROP DATABASE IF EXISTS openstack_citest' {}00:47
fungiharlowja: unit tests or devstack/tempest?00:48
harlowjaunit, taskflow is just a library, so doesn't go through tempest00:48
*** jcooley_ has joined #openstack-infra00:48
harlowjainteresting errors there, lol00:48
clarkbharlowja: only one tox process is run at a time00:48
fungiharlowja: we only run one jenkins worker per slave, so there is no possibility for nova and taskflow tests to run on the same slave at the same time00:49
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add storyboard-dev node with krotscheck on it  https://review.openstack.org/6951600:49
fungior what clarkb said more eloquently and in fewer words00:49
*** ICmonitor1 has joined #openstack-infra00:50
clarkbharlowja: fungi: I suppose there could be a unittest process for nova hanging around if jenkins doesn't clean it up nicely00:50
fungithat's not to say that a particular slave won't necessarily have a hung process left over from a previous job locking some resource... but we consider that a problem and we clean it up when we hear about it00:50
harlowjaya, i put a big process lock around the taskflow tests00:50
clarkbharlowja: does taskflow run testr and have multiple tests that touch the DB?00:50
* fungi will just stop typing--doesn't help that i'm repeating everything clarkb says ;)00:50
harlowjaright, but i put in process level locking to stop cross-process collisions00:50
*** ICmonitor has quit IRC00:51
clarkbfungi: no its good, makes me not crazy :)00:51
clarkbharlowja: please don't do that.00:51
clarkbharlowja: I wish that process level locking had never happened and that it would diaf00:51
*** mfer has quit IRC00:51
clarkbI worry that the ship has sailed00:51
fungioslo sort of set an unfortunate precedent with that00:52
*** ryanpetrello has joined #openstack-infra00:52
harlowjak, so how should then we avoid each test messing up each others db00:52
openstackgerritA change was merged to openstack-infra/config: Add build job for Japanese Install Guide  https://review.openstack.org/6748100:52
clarkbnow I am partially to blame because I didn't update mysql access quickly, but that is because there isn't a relatively secure way of doing that and I wanted people that knowwhat they are doing to help out00:52
harlowja*especially in testr00:52
openstackgerritA change was merged to openstack-infra/config: Remove link to puppet dashboard  https://review.openstack.org/6872400:52
clarkbharlowja: each test uses a different db00:52
harlowjak, is that possible?00:52
clarkbharlowja: for postgres it is00:52
harlowjathought only given access to openstack_citest00:52
harlowjak, so then for mysql?00:52
clarkbharlowja: for mysql, not currently because it requires giving that user add and drop00:52
clarkbdrop is problematic00:52
harlowjaright, so then still need process lock for mysql path00:53
clarkbharlowja: no00:53
clarkbharlowja: we need to give the user mysql perms to do it sanely00:53
harlowjak, whats missing to make that happen?00:53
sdagueclarkb: so, I had another thought on the real db thing recently00:53
clarkbharlowja: we have a horde of mysql experts around and I secretly hoped one of them would say do it this way or write the change00:53
clarkbharlowja: but I haven't managed to get that out of them yet :)00:53
sdaguewhich is kick them out of unit tests entirely, make people run functional tests on a d-g node00:54
harlowjawhy not just blow-away all dbs every time a test comes through?00:54
clarkbharlowja: thats the plan00:54
clarkbharlowja: but the tests need to be able to do it themselves00:54
clarkbharlowja: because infra shouldn't care00:54
clarkbsdague: I would be on board with that00:54
*** oubiwann_ has joined #openstack-infra00:54
*** jcooley_ has quit IRC00:54
clarkbsdague: but, that adds another slave to the mix for every patchset00:54
sdagueit would also massively simplify the tox install for devs00:54
sdagueclarkb: yes00:55
harlowjahmmm, so in the meantime, should i remove these tests, nice to have them to make sure sqlalachemy is working fine?00:55
clarkbsdague: which I am trying to reverse the tide on :)00:55
harlowja*with real dbs (not sqllite)00:55
jeblairclarkb, sdague: but does that really address the fundamental problem?  you'd still want to paralellize those db tests running under testr00:55
jeblairclarkb, sdague: regardless of what kind of node they are running on00:55
clarkbsdague: after the gate thrash last week I think it is insane that we need so many slaves per change00:55
clarkbjeblair: it addresses the mysql access problem00:55
jeblairclarkb: how?00:55
clarkbjeblair: unlike postgres mysql has no good way of saying give this account access to add and drop new DBs00:56
mordredclarkb: not true00:56
clarkbmysql can only say add and drop anything globally00:56
clarkbmordred: you keep saying that but no one will tell me how to fix this problem00:56
jeblairclarkb: why is that a problem?00:56
clarkbmordred: and all of the docs are ^00:56
mordredit's fixed00:56
mordredit's not a problem00:56
clarkbjeblair: if we give openstackci_test DROP in mysql it can drop anything00:56
mordredthe key is, you can grant those permissions on the db once it's been created00:56
clarkbmordred: right thats the problem00:56
clarkbmordred: we want the tests to create arbitrary DBs themselves00:57
mordredcreate db, add permission, drop db00:57
mordredoh ARBITRARY dbs00:57
harlowjaarbitary dbs FTW00:57
harlowja*to avoid name collisions00:57
clarkbpostgres does this just fine as you can say DB creator gets drop and openstackci_test can add00:57
clarkbmysql doesn't know how to do that00:57
mordredyeah. with you00:57
jeblairclarkb: there's no db we care about other than 'mysql' itself00:57
clarkbjeblair: the information schema00:57
jeblairclarkb: and all nova unit test slaves are single-use.00:58
clarkband mysql itself00:58
mordredyou can't drop i_s00:58
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/config: Add jenkins master name to the logstash records  https://review.openstack.org/6750800:58
jeblairclarkb: so i actually don't care if the tests drop mysql itself.00:58
clarkbmordred: I thought hub_cap managed to do it00:58
clarkbjeblair: good point00:58
mordredclarkb: nah. he managed to do the thing I was talking about00:58
clarkbjeblair: so we just accept that tsets can be silly and throw them away00:58
mordredwe already accept that I think00:58
jeblairclarkb: yep.00:58
clarkbmordred: not on the static hosts00:58
harlowjahmmm, bad hub_cap00:58
clarkbbut those are going away00:59
mordredclarkb: on dynamic hosts00:59
jeblairthe static hosts are dead to me00:59
mordredof course, I STILL think00:59
mordredthat the unittests shoudl use drizzle and start/stop a new db server when they need isolation00:59
jeblairmordred: ++00:59
mordredso that we don't need a globa db at all00:59
*** ryanpetrello has quit IRC00:59
mordredI mean, for the love of god, we spent the first year of the project fixing all the crap in mysql to allow for non-root installs to work sanely00:59
harlowjaor have a script that can run per project to do test-env setup, teardown01:00
*** habdi has joined #openstack-infra01:00
mordredharlowja: that's called a fixture01:00
harlowjabut i'd almost call setting up mysql something different01:00
harlowjaor drizzle, or whatever01:00
mordredand you want it per unit test - and all we need is a fixture that does "get me a new port; drizzled -p $new_port; add_cleanup(kill drizzled)"01:00
sdagueso the reason I'd like to change this is the fact that in order to pep8 nova, you have to install postgresql01:01
mordredthat's my pint - it's only different because mysql and postgres both assume they are a heavyweight install01:01
jeblairmordred: did you just write it?01:01
mordredjeblair: pretty much01:01
sdaguewhich is causing lots of people to not do that01:01
clarkbsdague: no, you have to install the client bindings its different01:01
clarkbvery different01:01
mordreddrizzle, on the other hand, does not need to be installed - it can run in local dirs01:01
harlowjaah, thats nice01:01
mordredclarkb: it's not if you run osx01:01
sdagueclarkb: fair... however, I think it's time to rethink that01:01
mordredand dont' have a package manager so C libraries freak you out01:01
sdaguebecause I think it makes sense to have a class of tests between the unit tests and tempest01:02
mordredsdague: I do too01:02
harlowjame too01:02
mordredswift has their functional tests, which I think make sense01:02
sdagueright, exactly01:02
mordredglance and keystone each do $CRAZY to try to do that01:02
harlowjacause the unit tests can really run with sqlite, but its a nice middleground to run with postgres/mysql (real dbs)01:03
fungianother project has recently added functional tests similar to swift's too01:03
clarkbsdague: I don't disagree, but I think it is insane to spin up 12 nodes to test nova delete them all and do it all over again when pep8 fails01:03
mordredscrew sqlite01:03
sdagueand I think the bulk of the real db stuff that nova does actually should go into functional01:03
*** mrodden has quit IRC01:03
sdagueclarkb: sure01:03
sdaguebut that's about early pre-flight testing01:03
jeblairmordred: you can't run most of this stuff on osx so i don't think it makes sense to optomize the whole dev environment around it.  i'll buy some of the other arguments, but not that one.01:03
sdaguehonestly, I don't think we should allocate any test resources before pep8 runs01:03
clarkbit is also about all of that extra testing we do that fails 100% fo the time...01:04
sdaguejeblair: this isn't really the osx case, at least not in my thinking01:04
mordredjeblair: I agree - I was just responding to clarkb's comment about "it's just a library"01:04
*** dcramer_ has quit IRC01:04
harlowjaclarkb should i leave the process locking code in, still trying to figure out if its ok to have that? disable those tests?01:04
jeblairsdague: that's fine.  but mordred's statement required a response.  :)01:04
sdaguejeblair: sure :)01:04
clarkbharlowja: I am about to push a change to 'fix' mysql01:05
clarkbharlowja: if we can get reviewers for that then I think you can remove it otherwise keep it until mysql is happy01:05
sdaguebut we do get kind of crazy with what gets installed in tox venv01:05
sdaguewhich means... oh let me pep8 this new tree.... oh, right, time to make a pot of coffee01:05
sdagueand lunch01:05
*** masayukig has quit IRC01:06
harlowjasdague do u know if nova or others have different venvs for different types of plugins (one for mysql, one for postgres...), i believe they do this for tempest01:06
sdagueharlowja: we don't have different venvs in tempest01:06
openstackgerritA change was merged to openstack-dev/hacking: Fix python 3 incompatibility in _get_import_type  https://review.openstack.org/6884901:06
*** thuc has quit IRC01:06
harlowja*sorry, not in that sense, a environment that uses postgres then01:06
sdagueconfiguration is done in separate step01:06
sdagueharlowja: yeh, look at a nova job01:07
harlowjaright, so then the configuration step has knowledge of the variations in the nova code/requirements01:07
sdagueyou'll see the pg variant01:07
sdagueharlowja: well... only by fiat01:08
sdaguethat's all done at a very different level01:08
*** alexpilotti_ has quit IRC01:08
openstackgerritA change was merged to openstack-infra/storyboard: Update ProjectGroups API to consume ID's rather than names.  https://review.openstack.org/6854001:09
mordredthat pep8 requires mysql or postgres to be installed is a bug and should be fixed01:09
harlowjasdague ya, so many levels :-P01:09
mordredperiod. there is no reason that should be the case. whatever is causing it is doing something wrong01:09
*** masayukig has joined #openstack-infra01:09
openstackgerritClark Boylan proposed a change to openstack-infra/config: Give test mysql user create and drop perms.  https://review.openstack.org/6951901:09
clarkbmordred: ^ how does that look?01:09
sdaguemordred: deps = -r{toxinidir}/tools/pip-requires01:10
sdague       -r{toxinidir}/tools/test-requires01:10
sdaguethat's what's doing it01:10
clarkbmordred: sdague hacking causes it01:10
clarkbbecause hacking imports stuff01:10
harlowjaclarkb while u are at it, https://review.openstack.org/#/c/69513/ would be superb if u don't mind :)01:10
mordredno - he's talking about the tox env01:10
clarkbmordred: yes, the tox env needs it because hacking01:10
mordredhacking importing stuff can be worked around01:10
sdagueclarkb: it needs it for pep801:10
*** yamahata has joined #openstack-infra01:10
clarkbif hacking didn't do import checks you can remove test-requires from the env01:10
clarkband explicitly state the pep8/hacking/flake8 deps01:11
clarkbsdague: pep8 doesn't import stuff iirc01:11
sdagueclarkb: hmmm... maybe?01:11
jeblairclarkb: yes, this is my understanding.01:11
mordredright. it does that to support "only import modules"01:11
sdagueoh, right, that is actually the osx problem01:11
mordredflake8 runs fine on nova with no libs - I just tested it01:12
fungidoes flake8 import stuff to do pyflakes-like analysis?01:12
sdaguebecause there is a way to check that without eval in linux01:12
fungiahh, retract my question01:12
mordredsdague: how about add in an attempt to do the linux version01:12
mordredand if the linux version doesn't work, skip the modules test01:12
clarkbmordred: nova is a bad test because site-packages are enabled01:12
clarkbwhich means you probably have libs somehwere01:12
mordredclarkb: I do not have ANYTHING installed on this machine01:12
sdaguemordred: yeh, that's an interesting idea01:12
fungiapt-get remove --purge coreutils01:12
mordredI just re-installed it :)01:12
*** CaptTofu has quit IRC01:13
sdagueI'd have to figure out where that fix is01:13
mordredsdague: and if that means taht osx people have a slightly higher annoyance rate with the gate catching pep8 things that local didn't catch, scrwe it01:13
sdagueif we did that, we'd definitely want early abort on pep8 in check01:13
*** CaptTofu has joined #openstack-infra01:13
sdaguebecause we'd get a bunch more bad pep801:13
sdaguebecause way too many people on this project are mac people :)01:13
fungipeople running non-free operating systems can be burdened to run a free operating system in a virtual machine for testing code meant to run on a free operating system, right?01:14
jeblairsdague: i agree that too many people on this project are mac people.01:14
mordredwe need to be able to express grouping and dependency differently in zuul to do early-fail-on-pep801:14
jeblairmordred: no we don't01:14
clarkbjeblair: mordred: not since flake801:14
mordredwith templates?01:14
clarkbsince it bundles all of the cheap static checks together01:14
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Simple round trip API integration with storyboard-api  https://review.openstack.org/6852801:14
jeblairmordred: we only need to do the thing i suggested the last time you brought this up: allow jobs to be marked as able to cause early aborts.01:14
jeblairmordred: at least, that's if you want to do 'abort' on failure, rather than wait-for-success01:15
harlowjaor i think jog0  was thinking that maybe someday in the future we can have a smart reformatter that fixes all hacking errors automagically, thats another solution to the pep8 problem :-P01:15
sdaguejeblair: so that would still allocated the d-g nodes01:15
harlowjareformat all the code automatically01:15
jeblairmordred: if you want to do wait-for-success, then yeah, you need another way of grouping01:15
jog0harlowja: I see that as a locally run tool only01:15
mordredI think we were looking for wait-for-success - wanting to avoid allocating d-g nodes until pep8 worked01:15
jeblairsdague: is this really a very pressing problem though?01:15
harlowjajog0 but u could imagine it being hooked into CI01:16
sdaguejeblair: I don't know01:16
jog0harlowja: and I can imagine that being confusing01:16
harlowjai believe google hooks in gofmt01:16
sdaguejeblair: which part being pressing?01:16
jeblairmordred: i would be happy to support that in zuul; i don't think it's trivial to implement; but i'm not sure i would want that for openstack.01:16
*** _ruhe is now known as ruhe01:16
jog0IIRC thats usually a precommit hook, anway that is future vs now01:16
sdagueI do think we should probably try to make the tox dev side of the equation simpler01:16
fungiharlowja: eventually zuul will just reimplement all our python mess in lisp on the fly and reassign commit authorship to itself, because we are fallible meatsacks01:17
harlowjaskynet ftw01:17
mordredmeatsacks ftw01:17
jeblairsdague: are we so short of check nodes that we really want to wait for a pep8 run to finish before starting anything else?  it will slow check results down, and sometimes we're waiting on nodes to run pep8 checks while we have devstack nodes.01:17
*** CaptTofu has quit IRC01:18
clarkbgah I started reviewing lifeless' nodepool stack in the wrong direction then found a -1 whcih needs addressing01:18
clarkbthat'll teach me01:18
sdaguejeblair: honestly, I think that's your call. But clarkb expressed some concern on the number of slaves back at the beginning of this thread01:18
fungiwow. somebody rechecked that ginormous series of nova patches that run for 3 hours and fail out on large-ops01:18
clarkbjeblair: sdague: I think one of the major reasons gate thrash was so horrible was node starvation01:18
sdaguefungi: it's not a recheck01:18
sdagueit's a dansmith patch series01:19
fungisdague: oh, rebased?01:19
clarkbjeblair: sdague: I think we avoid that problem entirely if we didn't consume so many nodes and throw them away01:19
*** VijayT has quit IRC01:19
*** VijayT has joined #openstack-infra01:19
jog0where do pep8 jobs get run, I'm surprised we sometimes wait for pep8 jobs to run01:19
jeblairclarkb: we just made MAJOR changes to how we use nodes, and we're making even more.  how about we try them out for a few minutes before we re-engineer again?01:19
sdagueI am going to nudge him tomorrow to not push the whole thing again until patch #2 is passing largeops01:19
sdaguejeblair: yeh, agreed, I think we can sort that one later01:20
clarkbjeblair: I am not suggesting a reengineer, you guys are :P01:20
lifelessclarkb: oh, there's a -1, doh.01:20
sdaguejeblair: so.... about to get called to dinner. Hows the "need valid check" logic?01:20
clarkbjog0: on single use slaves for some projects01:20
jeblairclarkb: clearly we're not having the same conversation then.  i'm going to stop participating in it.  :)  goodnight!01:20
dansmithsdague: nudge me?01:20
clarkbjog0: on all projects01:20
jeblairsdague: did you see my a/b/c thing above?01:21
clarkbjog0: er soon on all projects01:21
sdagueapparenlyt not01:21
jog0clarkb: we run  one pep8 on a node at a time01:21
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Changed default API base to match WSGI module  https://review.openstack.org/6952501:21
clarkbjeblair: I was responding to the idea of adding a completely new set of tests to the gate01:21
clarkbI agree we need the tests but should be careful about simply throwing nodes at the problem01:21
sdagueyeh, that was a chunk when I ran away for a bit01:21
jeblair23:53 < jeblair> (a) clarkb is looking into fixing gerrit so the timestamp is updated01:21
jeblair23:54 < jeblair> (b) we can have zuul clear the check vote (pro: easy; con: adds an e23:54 < jeblair> (c) we search for "Works for me" and "Doesn't seem to work" comments from jenkins01:21
jeblair23:55 < jeblair> (a) is a change to gerrit, (c) is a change to zuul, (b) is a zuul config change01:21
jog0clarkb: interesting, thanks01:22
mordredclarkb, sdague we could build a special pep8 hivemind ...01:22
jeblair(b) adds an extra entry in the gerrit log for each change01:22
clarkblifeless: I take it you are addressing that comment?01:23
jeblairsdague: clarkb is working on (a); it's uncertain and i would not expect resolution there for at least 2 weeks01:23
lifelessclarkb: I'll see if I can find someone to do so, and if not I will get to it when I get a chance to cycle back around01:23
clarkblifeless: ok01:23
jeblairsdague: (c) we can probably do fairly quickly, but is ugly01:23
lifelessclarkb: but I have a tonne of urgent to do right now01:23
jeblairsdague: (b) is here: https://review.openstack.org/#/c/69508/01:23
lifelessclarkb: so if you want to fix it, please go right ahead01:23
clarkbjeblair: fungi: for the nodepool stack before the -1, shoudl I go ahead and review that or is the value only in the full stack?01:24
jeblairsdague: i'm leaning toward (b) for now, (a) when ready, (c) if a fails.01:24
*** dkliban has quit IRC01:24
*** wenlock has joined #openstack-infra01:24
clarkbmordred: can you review https://review.openstack.org/#/c/69519/ ?01:24
openstackgerritJoshua Harlow proposed a change to openstack-infra/config: Name adjustments for taskflow jobs  https://review.openstack.org/6951301:24
jeblairclarkb: i'm not certain; i'd have to look at all the changes, but i think significant chunks of the stack would be dangereous to merge not-as-a-unit01:25
fungiclarkb: which was the -1?01:25
dansmithjeblair: was sdague referring to my patch #2 above, in regards to not pushing until largeops is working?01:25
sdagueok, well dinner time, I left a quick comment01:25
mordredclarkb:  on it01:25
clarkbfungi: https://review.openstack.org/#/c/67985/201:25
jeblairsdague: i'm trying to ramp-up.  we can change to 24 hours later01:26
sdaguedansmith: so until we figure out why your patch series is blowing up on that test, on every piece, can we just push the first one that fails :)01:26
sdaguejeblair: ok01:26
jeblairsdague: but i'd like to start with 48 just to exercise all the bits first01:26
mordredclarkb: wfm01:26
sdaguejeblair: sure01:26
sdaguethen you can ignore my -101:26
dansmithsdague: I don't know which piece you're talking about01:26
mordredclarkb: btw - the jeepyb changes are good01:26
sdaguebut, dinner... later01:26
jeblairsdague: for instance, when we first rolled it out, it spammed several thousand comments, and i'm very glad it only did so on 5 changes.01:26
dansmithsdague: the first several pass01:26
lifelessclarkb: jeblair: replied01:27
*** ekis_isa has joined #openstack-infra01:28
lifelessthe answer is 'we could, after the rest of the stack is merged'01:28
lifelessIIR the stack C01:28
jeblairlifeless: your response doesn't make sense to me :(01:29
fungilifeless: just wondering why the 'if not block_on_delete' is being inserted inside the for loop rather than immediately before it01:30
jeblairfungi: exactly01:30
fungiit's two new lines either way01:30
fungiso the patch isn't any smaller01:30
lifelessjeblair: the patch *after it* splits the calling function in two01:30
clarkbI am just going to go through the stack up to the -1 and review without approvals01:31
clarkbwe can sort out merging when ready to babysit01:31
lifelessjeblair: if we don't split the calling function in two, then we have to probe once, or everything will always get forced to periodic cleanup, which you wanted to avoid01:31
lifelessjeblair: once the calling function is split in two and periodic cleanup is tuned to expect this, we can indeed stop probing at that place altogether01:31
jeblairlifeless: i don't see how a single probe (which is going to fail to find the server) is going to make any difference01:32
lifelessjeblair: I am fine however you want this tweaked, you asked for my reasoning, there you have it01:32
jeblairlifeless: (and indeed, the fact that this is in place is one of those critical chunks of patches i was mentioning to clarkb -- it _would_ be bad to merge up to this patch but not beyond)01:33
lifelessjeblair: it may be faulty01:33
lifelessjeblair: but I tried rather hard to make the stack part-mergable01:33
jeblairlifeless: i like where it ends up; i'm not particularly interested in trying to run a version of nodepool with a partial stack; i think we should address that nit, then merge the whole thing and go from there.01:34
lifelessthats fine by me01:34
lifelesslike I say, I can't do the changes right now - but I'm very happy for someone to take over01:34
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add storyboard-dev node with krotscheck and ruhe on it  https://review.openstack.org/6951601:34
*** CaptTofu has joined #openstack-infra01:34
lifeless[making the code move is trivial, checking it still works is what would take me more time]01:35
*** jhesketh has joined #openstack-infra01:35
*** nosnos has joined #openstack-infra01:36
*** jcooley_ has joined #openstack-infra01:42
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Teach periodicCleanup how to do one provider.  https://review.openstack.org/6829901:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Move cron loading below provider loading.  https://review.openstack.org/6830001:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Move cron definition out of the inner loop.  https://review.openstack.org/6830101:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Decouple cron names from config file names.  https://review.openstack.org/6830201:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Run per-provider cleanup threads.  https://review.openstack.org/6830301:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Make cleanupServer optionally nonblocking.  https://review.openstack.org/6798501:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Split out the logic for deleting a nodedb node.  https://review.openstack.org/6800301:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Use the nonblocking cleanupServer.  https://review.openstack.org/6800401:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Include check in fake.yaml.  https://review.openstack.org/6829501:45
clarkbhow does that look01:45
*** senk has joined #openstack-infra01:45
clarkbhrm, I suppose return False should fall outside the unneeded else block01:45
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Teach periodicCleanup how to do one provider.  https://review.openstack.org/6829901:46
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Move cron loading below provider loading.  https://review.openstack.org/6830001:46
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Move cron definition out of the inner loop.  https://review.openstack.org/6830101:46
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Decouple cron names from config file names.  https://review.openstack.org/6830201:46
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Run per-provider cleanup threads.  https://review.openstack.org/6830301:46
clarkbsorry for the spam01:46
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Make cleanupServer optionally nonblocking.  https://review.openstack.org/6798501:46
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Split out the logic for deleting a nodedb node.  https://review.openstack.org/6800301:46
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Use the nonblocking cleanupServer.  https://review.openstack.org/6800401:46
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Include check in fake.yaml.  https://review.openstack.org/6829501:46
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Changed default API base to match WSGI module  https://review.openstack.org/6952501:46
*** VijayT has quit IRC01:47
*** jcooley_ has quit IRC01:48
clarkbanyways I don't have jeepyb configured in a way I can test that easily and it is getting close to dinner time01:49
*** ryanpetrello has joined #openstack-infra01:49
ewindischnow that we've settled down a bit... just want to get this on the radar: https://bugs.launchpad.net/openstack-ci/+bug/127326401:50
ewindisch(request for 3rd-partying testing service account)01:51
*** julim has quit IRC01:56
*** gokrokve has joined #openstack-infra01:57
*** ryanpetrello has quit IRC01:59
*** ianw has quit IRC01:59
*** ianw has joined #openstack-infra02:00
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Changed default API base to match WSGI module  https://review.openstack.org/6952502:00
openstackgerritA change was merged to openstack-infra/config: Record build_branch in logstash  https://review.openstack.org/6749802:01
*** ruhe is now known as _ruhe02:03
openstackgerritA change was merged to openstack-infra/devstack-gate: Remove configure_tempest.sh  https://review.openstack.org/6692202:03
*** dcramer_ has joined #openstack-infra02:06
*** gyee has quit IRC02:07
*** gokrokve has quit IRC02:08
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Updated grunt proxy server to rewrite to WSGI in dev  https://review.openstack.org/6952902:08
*** yaguang has joined #openstack-infra02:12
*** vkozhukalov has joined #openstack-infra02:15
openstackgerritA change was merged to openstack-infra/config: Enable ironicclient py33 tests voting  https://review.openstack.org/6809202:16
*** sarob has joined #openstack-infra02:16
harlowjathx clarkb02:17
*** SnowDust has joined #openstack-infra02:17
harlowja(for the review)02:17
harlowjashould unstick taskflow reviews02:17
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add support for Test Stability with Junit  https://review.openstack.org/6815202:18
*** ryanpetrello has joined #openstack-infra02:22
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Fix references to examples in api documentation  https://review.openstack.org/6771202:23
*** dkliban has joined #openstack-infra02:24
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for test_server_rescue race bug 1226412  https://review.openstack.org/6953002:24
*** chandankumar_ has joined #openstack-infra02:29
*** chuck__ has joined #openstack-infra02:29
*** chuck__ has quit IRC02:29
*** chandankumar_ has quit IRC02:31
*** jcooley_ has joined #openstack-infra02:36
*** jcooley_ has quit IRC02:42
*** ewindisch is now known as zz_ewindisch02:43
SlickNikfungi: around?02:45
*** ryanpetrello has quit IRC02:46
fungiSlickNik: sorta... what's up?02:47
SlickNikpypy tests are giving me a hard time for python-troveclient. Trying to figure out what's up with them.02:48
harlowjafungi would it be possible for u to approve https://review.openstack.org/#/c/69513/ so that i can retrigger this test to make sure the right tox names are matching02:48
fungiSlickNik: i saw that... testing manually earlier i got...02:48
fungiImportError: unable to load extension module '/home/fungi/python-troveclient/testing/site-packages/lxml/etree.pypy-20.so': /home/fungi/python-troveclient/testing/site-packages/lxml/etree.pypy-20.so: undefined symbol: PyUnicode_Concat02:48
fungiAlex_Gaynor might have pypy-relevant suggestions there02:49
clarkbpypy isn't compatible with CPython CFFI, could be related to that02:50
Alex_Gaynorlxml is a c-extension, and it looks like the etree module isn't supported. there's a pure python etree compatible API in the stdlib though, so you might be able to fallback to that?02:51
*** habdi has left #openstack-infra02:51
harlowjathx fungi02:51
Alex_Gaynorif it were cffi it would work fine :-)02:51
fungiSlickNik: seems you came to the right place ;)02:51
SlickNikfungi / Alex_Gaynor: thanks...02:51
*** rnirmal has quit IRC02:51
*** sarob has quit IRC02:51
SlickNikI'll look into that.02:52
*** sarob has joined #openstack-infra02:52
SlickNikWe're looking to pull out xml support in the short term, so I might just end up pulling the whole thing out (in case that proves easier).02:52
*** sarob_ has joined #openstack-infra02:53
openstackgerritA change was merged to openstack-infra/config: Name adjustments for taskflow jobs  https://review.openstack.org/6951302:54
*** rcleere has joined #openstack-infra02:55
harlowjaalright, now to see if it all goes fine :)02:56
*** krotscheck has quit IRC02:56
*** sarob has quit IRC02:56
harlowjafungi does it take a little while to push out that config?02:57
harlowjai think i remember it being hourly (or similar)02:57
*** sarob_ has quit IRC02:58
SlickNikfungi / clarkb: I think I'm going to make PyPy non-voting for troveclient, until we figure out what we're doing with xml. Once we have that done, I'll turn it on again. There're quite a few checkins blocked on that right now. What say?02:58
*** CaptTofu has quit IRC03:00
*** wenlock_ has joined #openstack-infra03:00
fungiharlowja: yeah, let's call it 45 minutes (puppet master updates its git clone every 15 minutes, jenkins masters update their puppet configuration every 10 minutes, jenkins-job-builder takes up to 10 minutes to run through all the current configuration)03:00
harlowjafungi thx, guess i'll head to gym then come back to retry this :)03:00
*** dstanek has quit IRC03:01
fungiSlickNik: sounds reasonable03:01
*** gokrokve has joined #openstack-infra03:01
*** nati_ueno has quit IRC03:02
*** dkliban has quit IRC03:02
* fungi is gonna disappear for a while03:03
SlickNikThanks! later.03:03
*** dkliban has joined #openstack-infra03:05
*** morganfainberg|z is now known as morganfainberg03:11
*** matsuhashi has quit IRC03:13
openstackgerritEli Klein proposed a change to openstack-infra/jenkins-job-builder: Added clone-workspace publisher  https://review.openstack.org/6953203:14
*** matsuhashi has joined #openstack-infra03:14
openstackgerritSlickNik proposed a change to openstack-infra/config: Make PyPy gate job non-voting for python-troveclient  https://review.openstack.org/6953303:16
*** zz_ewindisch is now known as ewindisch03:16
*** emagana has quit IRC03:17
*** ewindisch is now known as zz_ewindisch03:17
*** emagana has joined #openstack-infra03:18
*** rwsu has quit IRC03:18
*** VijayT has joined #openstack-infra03:18
lifelessare tripleo test nodes still allowed reuse ?03:19
*** mrodden has joined #openstack-infra03:20
clarkbnodepool doesnt reuse aiui. unless hmm iy may be a layout thing where you need dsvm in the name03:20
clarkbfor the function03:20
lifelessit was reusing yesterday03:21
lifelesswhich I quite liked ;)03:21
openstackgerritEli Klein proposed a change to openstack-infra/jenkins-job-builder: Added clone-workspace publisher  https://review.openstack.org/6953203:22
*** sarob has joined #openstack-infra03:22
*** SnowDust has quit IRC03:24
clarkblifeless: there may be a problematic bug here. IIRC nodepool will always do its best to delete nodes after they have been used, but to make them go into offiline mode we need to make sure the job name matches the string for offlineing in zuul03:24
clarkbjeblair: ^ is this a problem for all of the other single use test slaves too?03:25
*** emagana has quit IRC03:25
SlickNikhttps://review.openstack.org/69533 when you guys get a chance. (Making the pypy gate job non-voting for python-troveclient)03:26
clarkblifeless: its in the codepaths for nodepool that you touched :) you should know what it does. IIRC if state isn't READY or HOLD there is a timeout before it is deleted03:26
clarkbSlickNik: any idea how that snuck past the gate?03:26
lifelessthe node complete function deletes always03:26
clarkbor are you trying to get something in?03:26
clarkblifeless: ya03:26
lifelessclarkb: so it should be deleting, and my stuff hasn't landed yet :)03:26
lifelessclarkb: which means I don't know why it did some reuse yesterday03:27
clarkblifeless: there is a delay right?03:27
lifelesshttps://jenkins02.openstack.org/computer/tripleo-precise-tripleo-test-cloud-1220561/ has no reuse03:27
*** yamahata has quit IRC03:27
clarkblifeless: it may have reused within the delay?03:27
*** masayukig has quit IRC03:27
lifelessclarkb: no, it tries to delete directly03:27
lifelessclarkb: periodic cleanup had a delay03:27
*** sarob has quit IRC03:28
lifelessthere might be a race condition with unregistering the node from jenkins03:28
openstackgerritA change was merged to openstack-infra/config: Make PyPy gate job non-voting for python-troveclient  https://review.openstack.org/6953303:29
SlickNikclarkb: Not sure, seems like a disconnect. It's never actually succeeded.03:32
*** ICmonitor has joined #openstack-infra03:32
*** ICmonitor1 has quit IRC03:32
SlickNikclarkb: thanks much!03:33
clarkbmordred: https://review.openstack.org/#/c/68439/1/modules/openstack_project/templates/nodepool/nodepool.yaml.erb I was seeing cpu limits hit by nodepool in iad today. any chance you didn't bump those numbers with the ram bump?03:33
*** morganfainberg is now known as morganfainberg|z03:36
*** thuc has joined #openstack-infra03:40
*** emagana has joined #openstack-infra03:40
*** thuc_ has joined #openstack-infra03:41
mikalclarkb: you still around?03:42
mikalI need a sample of /home/gerrit2/projects.yaml post template substitution to test some code03:43
clarkbmikal: yes, reviewing all the things03:43
mikalIs that something that someone can grab for me?03:43
* clarkb looks at what is templated03:43
clarkbmikal: so I will just tell you waht the values are for http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/review.projects.yaml.erb03:44
mikalActually, all the substitutions are in the first doc, yes?03:44
mikalI only care about the second doc, so perhaps I am ok03:44
clarkbmikal: local git dir is /var/lib/git, gerrit-host is review.openstack.org, ssh_project_key is /home/gerrit2/review_site/etc/ssh_project_key_id_rsa03:45
*** thuc has quit IRC03:45
mikalclarkb: ta03:45
*** kraman has quit IRC03:47
*** matsuhashi has quit IRC03:49
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better  https://review.openstack.org/5142503:50
openstackgerritA change was merged to openstack-infra/jeepyb: Lump all defaults reading into one place  https://review.openstack.org/6950203:51
lifelesscan changes be approved which failed check ?03:52
wenlockclarkb: we've been testing on our enviornment with 51425 a bit.  Not on the slave side so much but more on gerrit, jenkins::master, pastebin, puppetmaster03:52
clarkblifeless: yes03:52
lifelesstechnically, not policy- cool03:52
clarkblifeless: but, make sure you intend they should pass03:52
wenlockhopefully it passes checks03:52
clarkbwenlock: cool03:52
lifelessclarkb: dprince is keen to see tripleo-gate doing check votes03:53
clarkblifeless: so jeblair and sdague intend on enforcing check passes before queuing in the gate, that isn't happening today03:53
clarkbbut will happen near future probably, just keep that in mind03:53
*** pcrews has quit IRC03:53
*** dstanek has joined #openstack-infra03:54
lifelessclarkb: this job isn't in the gate at all though03:54
*** praneshp has quit IRC03:54
clarkblifeless: won't matter once jeblair and sdague get that in03:54
clarkblifeless: https://review.openstack.org/#/c/69157/ is that purely syntax?03:54
clarkbI mean I don't care, but trying to understand why it matters03:54
lifelessits'a  url03:54
clarkblifeless: the change at that url :P03:54
lifelessoh, just easier to read03:54
lifelessfoo>bar == foo > bar in shell03:55
*** mriedem has quit IRC03:55
*** CaptTofu has joined #openstack-infra03:55
lifelesslike it says in the commit message03:55
lifeless'Just easier to read.'03:55
clarkbright, I never remember where whitespace matters and was sure it didn[t matter there03:55
lifelesslook up and down a couple lines03:55
*** emagana has quit IRC03:56
*** AaronGr is now known as AaronGr_Zzz03:57
*** jcooley_ has joined #openstack-infra03:57
mroddenat what point should i not feel bad about doing a --reset-author on someone elses patchset03:58
*** slong has quit IRC03:58
mroddenbecause i am pretty sure i have added more lines of code in unit tests than this entire patch was before i started hacking on it03:58
lifelessmrodden: when if you were them and putting however much input of theirs is left in, you wouldn't bother adding co-authored-by:03:58
lifelessmrodden: I would just add cab03:59
mroddenadd myself on coauthored?03:59
mroddenor add them on cab and reset03:59
mroddenwas going to do the latter04:00
*** CaptTofu has quit IRC04:00
*** harlowja is now known as harlowja_away04:01
*** jcooley_ has quit IRC04:02
lifelessmrodden: do you really care?04:05
lifelesscredit is shared w/c-a-b04:05
mroddeni mean, i dont really care04:05
mroddencredit is shared04:05
mroddenpeople keep asking for updates to the patch in review and i'm the only one updating it now...04:06
*** markwash has joined #openstack-infra04:06
*** harlowja_away is now known as harlowja04:06
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better  https://review.openstack.org/5142504:06
wenlockforgot my puppet-lint :<04:06
mroddenalso going from python to ruby is weird04:06
*** kraman has joined #openstack-infra04:07
lifelessoh don't do that :004:08
mroddenits dynamic and fun like python04:09
mroddenbut it really suffers from what i term "the perl problem"04:09
mroddenmeaning, there are a billion ways to do something and non of them really seem obvious04:10
openstackgerritA change was merged to openstack-infra/config: Move tuskar-ui to horizon program.  https://review.openstack.org/6826404:12
clarkband with that I am done reviewing for the night04:12
* clarkb &04:12
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better  https://review.openstack.org/5142504:13
wenlockpesky white spaces04:13
*** gokrokve has quit IRC04:16
*** gokrokve_ has joined #openstack-infra04:17
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better  https://review.openstack.org/5142504:20
harlowjahmmm, got closer on the tox.ini envs, is there a way to have a tox env use a specifc python version04:20
harlowjagot 'ERROR: InterpreterNotFound: python2.6'04:20
harlowjamaybe have to specify in a yaml file somewhere to use 2.6 for these envs04:21
harlowjaclarkb if i specify 'node: centos6' would that make this happen (usage of 2.6)04:21
*** gokrokve_ has quit IRC04:22
harlowja*for a specific job04:22
*** thuc has joined #openstack-infra04:22
clarkbyou want centis6 for py2604:22
*** sarob has joined #openstack-infra04:23
openstackgerritJoshua Harlow proposed a change to openstack-infra/config: Ensure taskflow py26 environments are using centos6 node  https://review.openstack.org/6953404:24
*** AaronGr_Zzz is now known as AaronGr04:24
harlowjathink that should do it04:25
*** thuc_ has quit IRC04:25
wenlockcool, checks passed04:25
*** thuc has quit IRC04:26
harlowjaclarkb fungi  if u guys don't mind checking that out, hopefully last change ;)04:27
*** sarob has quit IRC04:28
*** jcooley_ has joined #openstack-infra04:28
openstackgerritMichael Still proposed a change to openstack-infra/jeepyb: Allow configurable mappings to different LP projects  https://review.openstack.org/6207404:29
*** jcooley_ has quit IRC04:30
*** wenlock has quit IRC04:31
*** masayukig has joined #openstack-infra04:31
*** david-lyle has joined #openstack-infra04:34
*** talluri has quit IRC04:41
*** UtahDave has quit IRC04:43
*** markwash has quit IRC04:44
*** matsuhashi has joined #openstack-infra04:45
*** thuc has joined #openstack-infra04:45
*** gokrokve has joined #openstack-infra04:47
*** praneshp has joined #openstack-infra04:48
*** gokrokve_ has joined #openstack-infra04:49
*** weshay has quit IRC04:49
*** gokrokve has quit IRC04:51
*** gokrokve_ has quit IRC04:53
*** markwash has joined #openstack-infra04:56
*** kraman has quit IRC05:00
*** AlexF_ has joined #openstack-infra05:01
*** gokrokve has joined #openstack-infra05:02
*** thuc has quit IRC05:03
*** thuc has joined #openstack-infra05:04
*** ekis_isa has quit IRC05:04
*** jp_at_hp has quit IRC05:05
*** jcooley_ has joined #openstack-infra05:06
*** jcooley_ has quit IRC05:07
*** emagana has joined #openstack-infra05:07
*** rushiagr has joined #openstack-infra05:08
*** rushiagr has left #openstack-infra05:08
*** thuc has quit IRC05:08
*** gokrokve has quit IRC05:09
*** emagana has quit IRC05:12
*** thuc has joined #openstack-infra05:16
*** markwash has quit IRC05:22
*** talluri has joined #openstack-infra05:23
*** sarob has joined #openstack-infra05:23
*** kraman has joined #openstack-infra05:25
*** AlexF_ has quit IRC05:25
openstackgerritKei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for Ci Skip Plugin  https://review.openstack.org/6954005:28
*** sarob has quit IRC05:28
*** nicedice_ has quit IRC05:29
kashyapHeya, /me is new to infra. Trying to explore how to have Fedora as part of Devstack Gate jobs.05:29
* kashyap is cloning Nodepool to see if he can understand something05:30
clarkbfirst step is getting it into nodepool. pleia2 is working on it too05:31
clarkbthere are paramiko lack of tty sudo problems on fedora05:32
*** AlexF_ has joined #openstack-infra05:32
*** AlexF_ has quit IRC05:35
kashyapHi clarkb, thanks for the detail.05:36
kashyapclarkb, Is there a bug report or some such for the problem you outlined?05:36
clarkbno bug iirc.05:37
kashyapOr some who knows the nature of the problem?05:37
clarkbthe nature of the problem is sudo requires a tty on fedora. paramiko does not allocate a tty so the nodepool node preparation scripts fail05:38
openstackgerritKei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for Ci Skip Plugin  https://review.openstack.org/6954005:38
kashyapclarkb, Thanks for that. New to all of Jenkins and CI infra. /me comes from libvirt/kvm/qemu land05:39
*** Ryan_Lane has joined #openstack-infra05:40
kashyappleia2, Hi, when you're around - can you please note about your thoughts/or any issues you encountered (apart from what Clark outlined here) on getting Fedora into Gate?05:45
kashyapI just started this etherpad to capture things at one place, please add any issues  - https://etherpad.openstack.org/p/fedora-on-gate05:46
clarkbkashyap: pleia2 has a WIP change in gerrit to add fedora hosts to nodepool05:47
*** AlexF_ has joined #openstack-infra05:48
*** sarob has joined #openstack-infra05:48
kashyapNice. /me is available to test patches, feel free to bug me if I can be of any help there.05:49
*** oubiwann_ has quit IRC05:51
*** AlexF_ has quit IRC05:51
*** sarob has quit IRC05:53
*** zehicle_at_dell has joined #openstack-infra05:53
*** CaptTofu has joined #openstack-infra05:56
*** CaptTofu has quit IRC06:00
openstackgerritChristopher Yeoh proposed a change to openstack-infra/elastic-recheck: Adds query for bug 1273556  https://review.openstack.org/6954206:01
*** gokrokve has joined #openstack-infra06:02
*** thuc has quit IRC06:05
*** thuc has joined #openstack-infra06:06
*** gokrokve has quit IRC06:07
openstackgerritKhai Do proposed a change to openstack-infra/jeepyb: fix manage_projects with MySQL-Python 1.2.5  https://review.openstack.org/6954306:10
*** thuc has quit IRC06:10
*** harlowja is now known as harlowja_away06:10
*** jcoufal has joined #openstack-infra06:12
*** reed has quit IRC06:13
*** kraman has quit IRC06:14
*** AlexF_ has joined #openstack-infra06:20
*** mrda is now known as mrda_away06:34
*** NikitaKonovalov_ is now known as NikitaKonovalov06:43
*** kraman has joined #openstack-infra06:44
*** gsamfira has quit IRC06:46
*** rcleere has quit IRC06:47
*** gokrokve has joined #openstack-infra06:47
*** kraman has quit IRC06:49
*** emagana has joined #openstack-infra06:50
*** vkozhukalov has quit IRC06:51
*** gokrokve has quit IRC06:52
*** nati_ueno has joined #openstack-infra06:54
*** rcleere has joined #openstack-infra06:56
*** sarob has joined #openstack-infra06:58
*** VijayT has quit IRC06:59
*** sarob has quit IRC07:03
*** nati_ueno has quit IRC07:07
*** nati_ueno has joined #openstack-infra07:08
*** starmer has joined #openstack-infra07:11
*** yolanda_ has joined #openstack-infra07:13
openstackgerritA change was merged to openstack/requirements: Require httplib2>=0.7.5  https://review.openstack.org/6648607:15
*** coolsvap has joined #openstack-infra07:17
*** kruskakli has joined #openstack-infra07:17
*** afazekas has joined #openstack-infra07:17
*** starmer_ has joined #openstack-infra07:21
*** starmer has quit IRC07:24
*** katyafervent is now known as katyafervent_awa07:28
*** vkozhukalov has joined #openstack-infra07:33
*** senk has quit IRC07:35
openstackgerritIvan Melnikov proposed a change to openstack-dev/hacking: More portable way to detect modules for H302  https://review.openstack.org/6885807:37
*** flaper87|afk is now known as flaper8707:44
*** gokrokve has joined #openstack-infra07:47
*** AlexF_ has quit IRC07:50
*** rcleere has quit IRC07:51
*** gokrokve has quit IRC07:52
*** markmc has joined #openstack-infra07:56
*** CaptTofu has joined #openstack-infra07:57
*** CaptTofu has quit IRC08:01
*** nati_uen_ has joined #openstack-infra08:02
*** luqas has joined #openstack-infra08:04
*** nati_ueno has quit IRC08:05
*** praneshp has quit IRC08:12
*** yamahata has joined #openstack-infra08:13
*** sdake has quit IRC08:14
*** sdake has joined #openstack-infra08:15
*** nati_uen_ has quit IRC08:15
*** vjay has joined #openstack-infra08:15
*** hashar has joined #openstack-infra08:15
*** nati_ueno has joined #openstack-infra08:16
*** ICmonitor has quit IRC08:17
vjayWhat is the process to be followed when jenkins is voting a review submitted as "-1" for test failures not related to the actual fix?08:18
*** ICmonitor has joined #openstack-infra08:19
*** mrodden has quit IRC08:19
*** mrodden has joined #openstack-infra08:19
vjayIt is failing due to bugs 1249065 & 1254890. But "Elastic recheck" does not add an entry in the Gerrit.08:20
*** lttrl has joined #openstack-infra08:23
*** yassine has joined #openstack-infra08:23
*** david-lyle has quit IRC08:27
*** e0ne has joined #openstack-infra08:27
*** lttrl has quit IRC08:33
*** dizquierdo has joined #openstack-infra08:39
*** e0ne has quit IRC08:44
*** e0ne has joined #openstack-infra08:45
*** pblaho has joined #openstack-infra08:47
*** starmer_ has quit IRC08:47
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: test dequeue on abandoned changes  https://review.openstack.org/6547608:49
*** e0ne has quit IRC08:49
*** _ruhe is now known as ruhe08:51
*** vkozhukalov has quit IRC08:51
*** Ryan_Lane has quit IRC08:51
*** Ryan_Lane has joined #openstack-infra08:52
*** starmer has joined #openstack-infra08:52
*** Ryan_Lane has quit IRC08:52
*** DinaBelova_ is now known as DinaBelova08:53
*** starmer has quit IRC08:55
*** jpich has joined #openstack-infra08:59
*** vjay has quit IRC09:04
*** ken1ohmichi has quit IRC09:05
*** derekh has joined #openstack-infra09:05
*** SergeyLukjanov_ is now known as SergeyLukjanov09:08
*** nati_uen_ has joined #openstack-infra09:09
*** nati_uen_ has quit IRC09:09
*** matsuhashi has quit IRC09:09
*** nati_uen_ has joined #openstack-infra09:09
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: dequeue abandoned changes  https://review.openstack.org/6546709:10
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: dequeue abandoned changes  https://review.openstack.org/6546709:12
*** nati_ueno has quit IRC09:12
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Enable precise-backports on tripleo test nodes  https://review.openstack.org/6795809:13
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Add some dependencies required by toci  https://review.openstack.org/6768509:13
*** matsuhas_ has joined #openstack-infra09:14
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: test dequeue on abandoned changes  https://review.openstack.org/6547609:14
*** NikitaKonovalov is now known as NikitaKonovalov_09:15
*** rossella_s has joined #openstack-infra09:18
*** marun has quit IRC09:19
*** ruhe is now known as _ruhe09:20
*** e0ne has joined #openstack-infra09:28
*** wenlock_ has quit IRC09:30
*** amotoki has joined #openstack-infra09:33
*** johnthetubaguy has joined #openstack-infra09:34
*** jooools has joined #openstack-infra09:40
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Fix up `zuul --version`  https://review.openstack.org/6662909:44
*** gokrokve has joined #openstack-infra09:47
*** nati_ueno has joined #openstack-infra09:51
*** gokrokve has quit IRC09:52
*** nati_uen_ has quit IRC09:53
*** luqas has quit IRC09:57
*** nati_ueno has quit IRC09:57
*** CaptTofu has joined #openstack-infra09:57
*** NikitaKonovalov_ is now known as NikitaKonovalov09:58
*** che-arne has joined #openstack-infra09:59
*** CaptTofu has quit IRC10:02
*** e0ne_ has joined #openstack-infra10:06
*** wenlock has joined #openstack-infra10:08
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Zuul trigger no more send GERRIT_* env variable  https://review.openstack.org/6956610:09
*** e0ne has quit IRC10:10
*** luqas has joined #openstack-infra10:19
*** jhesketh has quit IRC10:20
*** jhesketh__ has quit IRC10:20
*** matsuhas_ has quit IRC10:22
*** lcestari has joined #openstack-infra10:27
*** max_lobur_afk is now known as max_lobur10:29
*** jp_at_hp has joined #openstack-infra10:29
*** matsuhashi has joined #openstack-infra10:31
*** jhesketh has joined #openstack-infra10:32
*** jhesketh__ has joined #openstack-infra10:32
*** vjay has joined #openstack-infra10:33
openstackgerritSergey Lukjanov proposed a change to openstack-infra/release-tools: Fix requirements installation for pip>=1.5  https://review.openstack.org/6957210:33
*** wenlock has quit IRC10:34
*** vkozhukalov has joined #openstack-infra10:37
SergeyLukjanovttx, ping10:39
openstackgerritSergey Lukjanov proposed a change to openstack-infra/release-tools: Fix requirements installation for pip>=1.5  https://review.openstack.org/6957210:39
ttxSergeyLukjanov: pong10:39
SergeyLukjanovttx, hi10:39
SergeyLukjanovttx, the reason why release-tools fails to install is lazr.auth10:40
ttxSergeyLukjanov: I figured that much but couldn't tell WHY that was failing10:40
SergeyLukjanovttx, lazr.auth is hosted on launchpad10:41
SergeyLukjanovttx, only download links on pypi10:41
ttxSergeyLukjanov: I wonder how that ever worked, then10:42
ttxSergeyLukjanov: did we recently add tests ?10:42
SergeyLukjanovttx, nope10:42
SergeyLukjanovpip was updated to 1.510:42
SergeyLukjanovthere were tons of problems with it :)10:43
ttxlooks like you need --allow-unverified10:43
ttxbit scary10:43
SergeyLukjanovcheck the previous patch set10:43
SergeyLukjanovit adds --allow-unverified10:44
SergeyLukjanovbut isn't working too10:44
SergeyLukjanovhttps://review.openstack.org/#/c/66364/8/pypi_mirror/cmd/run_mirror.py adds all needed packages to pypi.o.o mirror10:44
SergeyLukjanovpatch set #1 works ok on my laptop with pip 1.5.210:45
StevenKHm, I wonder if the pip 1.5 upgrade is the cause of git-review being so unhappy, then.10:45
*** andreaf has joined #openstack-infra10:46
*** coolsvap has quit IRC10:47
ttxSergeyLukjanov: ok.. it's not as if we were actually testing anything with it10:47
*** gokrokve has joined #openstack-infra10:47
ttxSergeyLukjanov: so i wonder if we should not strip requirements.txt10:47
openstackgerritSergey Lukjanov proposed a change to openstack-infra/release-tools: Fix requirements installation for pip>=1.5  https://review.openstack.org/6957210:48
SergeyLukjanovprobably we could add lazr.auth to the pypi.o.o/infra10:48
ttxthat would be the alternative10:49
SergeyLukjanovheh, it works ok with the patch set #310:50
SergeyLukjanovrandom combination of --allow-abcd10:51
ttxOK, let's merge this then10:51
*** gokrokve has quit IRC10:52
SergeyLukjanovttx, checking it now on my laptop with pip 1.5.210:53
*** ArxCruz has joined #openstack-infra10:53
openstackgerritA change was merged to openstack-infra/release-tools: Fix requirements installation for pip>=1.5  https://review.openstack.org/6957210:54
*** DinaBelova is now known as DinaBelova_10:58
*** coolsvap has joined #openstack-infra11:00
*** matsuhashi has quit IRC11:00
openstackgerritA change was merged to openstack-infra/storyboard: Update documentation  https://review.openstack.org/6921111:02
*** yamahata has quit IRC11:03
openstackgerritA change was merged to openstack-infra/storyboard: API tests for rest  https://review.openstack.org/6744711:04
*** NikitaKonovalov is now known as NikitaKonovalov_11:04
*** NikitaKonovalov_ is now known as NikitaKonovalov11:06
openstackgerritA change was merged to openstack-infra/release-tools: Added ignore to additional egg-info files  https://review.openstack.org/6847111:07
*** boris-42 has quit IRC11:11
*** matsuhashi has joined #openstack-infra11:14
*** matsuhashi has quit IRC11:17
*** DinaBelova_ is now known as DinaBelova11:18
*** gokrokve has joined #openstack-infra11:19
*** SergeyLukjanov is now known as SergeyLukjanov_a11:19
*** SergeyLukjanov_a is now known as SergeyLukjanov11:19
SergeyLukjanovttx, it works ok on 1.5.2 pip too11:19
ttxSergeyLukjanov: ok, all merged11:19
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Database fixture added  https://review.openstack.org/6938411:20
*** gokrokve_ has joined #openstack-infra11:20
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file  https://review.openstack.org/6628011:21
SergeyLukjanovttx, great11:21
sdaguettx: can we get the logging standards on the project agenda for today? I'd like to socialize with PTLs and figure out which ones are interested11:22
ttxsdague: sure, adding11:22
*** gokrokve has quit IRC11:23
*** gokrokve_ has quit IRC11:25
*** coolsvap has quit IRC11:30
openstackgerritJulien Danjou proposed a change to openstack-dev/pbr: package: read a specific Python version requirement file  https://review.openstack.org/6323611:30
*** rfolco has joined #openstack-infra11:36
*** gokrokve has joined #openstack-infra11:47
*** hashar has quit IRC11:49
*** gokrokve has quit IRC11:52
*** lttrl has joined #openstack-infra11:53
*** NikitaKonovalov is now known as NikitaKonovalov_11:53
*** masayukig has quit IRC11:57
*** CaptTofu has joined #openstack-infra11:58
*** NikitaKonovalov_ is now known as NikitaKonovalov12:02
*** yamahata has joined #openstack-infra12:03
*** CaptTofu has quit IRC12:04
*** vjay has quit IRC12:05
*** hashar has joined #openstack-infra12:05
*** vipul has quit IRC12:10
*** vipul has joined #openstack-infra12:12
openstackgerritMax Lobur proposed a change to openstack/requirements: Add futures library to global requirements  https://review.openstack.org/6634912:12
*** beagles has quit IRC12:15
*** emagana has quit IRC12:19
*** b3nt_pin has joined #openstack-infra12:20
*** jhesketh__ has quit IRC12:22
*** changbl has quit IRC12:22
openstackgerritYuriy Taraday proposed a change to openstack-infra/git-review: Bump hacking version in requirements  https://review.openstack.org/4948612:24
*** jasondotstar has joined #openstack-infra12:28
*** b3nt_pin is now known as beagles12:29
*** boris-42 has joined #openstack-infra12:31
*** paul-- has joined #openstack-infra12:32
*** lttrl has quit IRC12:33
*** YorikSar has joined #openstack-infra12:36
*** luqas has quit IRC12:36
russellbinfra is so l33t that it crashes vim: https://code.google.com/p/vim/issues/detail?id=19612:39
YorikSarHello. Can anyone help me with an issue with Jenkins node: I get "Cannot fork" on precisepy3k-4 node.12:39
YorikSarAnd it's about to get the same result again.12:40
*** max_lobur is now known as max_lobur_afk12:41
YorikSarTests for 2.6 and 2.7 were run on centos6-5 and precise22 with no such issue12:41
*** jhesketh has quit IRC12:43
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Added documentation for REST API layer.  https://review.openstack.org/6921212:43
*** paul-- has quit IRC12:43
*** dkranz has joined #openstack-infra12:45
*** CaptTofu has joined #openstack-infra12:46
*** yaguang has quit IRC12:47
*** smarcet has joined #openstack-infra12:47
*** gokrokve has joined #openstack-infra12:47
*** vjay has joined #openstack-infra12:49
*** hashar has quit IRC12:51
sdaguerussellb: that's pretty wild12:51
*** amotoki has quit IRC12:51
*** _ruhe is now known as ruhe12:51
russellbi started looking at the code and then .... yeah12:51
russellbit wasn't obvious12:51
*** gokrokve has quit IRC12:52
sdagueyeh, crazy. I wonder if their syntax highlighting is guessing the file wrong12:55
sdagueyeh, I think it's the yaml parser, if you rename the file to nodepool.erb, it works12:57
*** dprince has joined #openstack-infra12:59
*** markwash has joined #openstack-infra13:00
*** markmcclain has joined #openstack-infra13:00
*** heyongli has joined #openstack-infra13:00
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Database fixture added  https://review.openstack.org/6938413:02
*** afazekas has quit IRC13:02
*** max_lobur_afk is now known as max_lobur13:04
*** nosnos has quit IRC13:08
russellbsdague: cool, added that note to the bug13:09
russellbsdague: too bad the bug list looks like a ghost town, heh13:09
russellbi shouldn't talk ...13:09
*** luqas has joined #openstack-infra13:10
*** pblaho has quit IRC13:11
*** gokrokve has joined #openstack-infra13:12
*** gokrokve has quit IRC13:14
YorikSarrussellb: Have you tried more recent version of Vim?13:14
YorikSarrussellb: I've just tried it on 7.4.131 and it works fine.13:15
russellbno, i haven't tried any other version13:15
russellbYorikSar: that's good to know though13:15
*** afazekas has joined #openstack-infra13:15
*** dhellmann_ is now known as dhellmann13:16
*** chandankumar has joined #openstack-infra13:16
*** sandywalsh has joined #openstack-infra13:18
*** freyes has joined #openstack-infra13:18
*** luqas has quit IRC13:18
*** emagana has joined #openstack-infra13:19
*** markwash has quit IRC13:23
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Add a new node type for precise with saucy kernel  https://review.openstack.org/6944513:23
*** pblaho has joined #openstack-infra13:24
*** katyafervent_awa is now known as katyafervent13:27
openstackgerritJulien Danjou proposed a change to openstack-dev/hacking: Allow to have noqa on H202  https://review.openstack.org/6959613:31
*** afazekas is now known as afazekas|mtg13:31
*** eharney has joined #openstack-infra13:33
*** beagles has quit IRC13:33
*** zhiyan has joined #openstack-infra13:34
*** NithyaG has joined #openstack-infra13:35
*** chandankumar has quit IRC13:36
salv-orlandoHi, I have a question. I'm still chasing this kernel bug, or at least trying to understand the exact root cause (I'm talking about bug 1273386)13:36
*** dhellmann is now known as dhellmann_13:36
*** dhellmann_ is now known as dhellmann13:37
salv-orlandoBasically, kernel 3.2.0-58 was published on 2014/01/02 - How long would it take from there to be installed on all the instances we use for testing on the gate? idk how frequently they are recycled13:37
salv-orlandoI'm just trying to understand whether the kernel crashes started with this new build or whether the root cause is something else13:38
*** b3nt_pin has joined #openstack-infra13:39
*** miqui has joined #openstack-infra13:40
fungisalv-orlando: uname -r on a random slave says "3.2.0-58-virtual #88-Ubuntu"13:41
sdaguesalv-orlando: so we rebuild the nodes nightly13:41
sdaguefungi: I assume we'd have taken that update the day after it published?13:41
fungisalv-orlando: we sould have been using that for devstack runs within 24 hours of ubuntu releasing it into their repository13:41
fungisdague: yes13:41
sdagueor is that coming from the base image?13:41
fungioh, hmmm13:42
salv-orlandoso if the kernel was published on 2014/01/02 by 2014/01/03 it would have been on all our images13:42
salv-orlandoThe oldest log we have now is from Jan 15th13:42
fungirighth, ubuntu does funny things with making kernel packages require dist-upgrade. i'd have to check the base images to see what they're running, but i think we do upgrade the kernel when building the image (the image log would tell us too)13:42
fungisalv-orlando: i believe so, yes13:43
fungii have to run out to an unanticipated appointment for a few hours though, so i can't really check much deeper right this moment13:43
*** dkliban has quit IRC13:44
sdaguesalv-orlando: so that's the oldest ES log13:44
sdaguebut we have raw logs before then13:45
salv-orlandosdague: yup, I managed to find them13:45
salv-orlandoI'm now reading a log from jan 10th13:45
salv-orlandoshame I can't find the kernel version without a kernel crash dump! do we log it somewhere else?13:45
*** prad has joined #openstack-infra13:45
sdaguesalv-orlando: probably not13:45
salv-orlandosdague: it seems we do just uname -m in devstack, it won't print kernel version13:46
sdaguesalv-orlando: yeh13:46
sdaguethere is also a pending change to dump the entire package list, which would help13:46
*** b3nt_pin has quit IRC13:48
*** markwash has joined #openstack-infra13:49
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add Savanna screen file to logstash indexing  https://review.openstack.org/6960213:50
*** dims has quit IRC13:50
*** emagana has quit IRC13:51
*** dims has joined #openstack-infra13:52
*** thuc has joined #openstack-infra13:53
*** thuc_ has joined #openstack-infra13:54
*** oubiwann_ has joined #openstack-infra13:54
*** b3nt_pin has joined #openstack-infra13:55
*** julim has joined #openstack-infra13:56
*** markmcclain has quit IRC13:57
*** markmcclain has joined #openstack-infra13:57
*** thuc has quit IRC13:58
openstackgerritA change was merged to openstack/requirements: Added a requirement to COIN PULP LP Modeler module  https://review.openstack.org/5344613:58
*** oubiwann_ has quit IRC13:59
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Make cells job gating  https://review.openstack.org/6353713:59
*** oubiwann_ has joined #openstack-infra14:01
*** dims has quit IRC14:03
*** chandankumar has joined #openstack-infra14:03
*** heyongli has quit IRC14:03
*** mestery has quit IRC14:03
*** dims has joined #openstack-infra14:04
*** gokrokve has joined #openstack-infra14:05
*** yaguang has joined #openstack-infra14:06
*** gokrokve_ has joined #openstack-infra14:08
*** dkranz has quit IRC14:09
*** chandankumar has quit IRC14:09
*** dcramer_ has quit IRC14:10
*** gokrokve has quit IRC14:11
openstackgerritgodp1301 proposed a change to openstack-infra/jenkins-job-builder: Add best and worst status details for conditional-step plugin.  https://review.openstack.org/6960814:11
*** afazekas|mtg has quit IRC14:11
*** prad has quit IRC14:12
*** dkehn_ has joined #openstack-infra14:14
*** pblaho has quit IRC14:14
*** mestery has joined #openstack-infra14:15
*** emagana has joined #openstack-infra14:17
*** afazekas has joined #openstack-infra14:18
openstackgerritBob Ball proposed a change to openstack-infra/nodepool: Support nodes with launch condition  https://review.openstack.org/6526114:21
openstackgerritBob Ball proposed a change to openstack-infra/nodepool: Support install phase with nodepool  https://review.openstack.org/6146314:21
openstackgerritBob Ball proposed a change to openstack-infra/nodepool: Allow useage of server IDs as well as names.  https://review.openstack.org/6942414:21
*** chandankumar has joined #openstack-infra14:21
*** mriedem has joined #openstack-infra14:23
*** emagana has quit IRC14:23
*** zz_ewindisch is now known as ewindisch14:23
*** dkliban has joined #openstack-infra14:23
*** yamahata has quit IRC14:25
*** NikitaKonovalov is now known as NikitaKonovalov_14:25
openstackgerritgodp1301 proposed a change to openstack-infra/jenkins-job-builder: Add best and worst status details for conditional-step plugin.  https://review.openstack.org/6960814:25
*** b3nt_pin is now known as beagles14:26
*** DinaBelova is now known as DinaBelova_14:26
*** yamahata has joined #openstack-infra14:29
*** ryanpetrello has joined #openstack-infra14:30
*** thuc has joined #openstack-infra14:31
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Split the config out into two files  https://review.openstack.org/6950314:32
mordredfungi: fixed your comments I believe14:33
*** thuc_ has quit IRC14:35
*** chandankumar has quit IRC14:35
*** thuc has quit IRC14:36
*** senk has joined #openstack-infra14:36
*** DinaBelova_ is now known as DinaBelova14:37
*** gokrokve_ has quit IRC14:39
*** senk has quit IRC14:39
dansmithsdague: can we finish this conversation about not pushing my set?14:40
sdaguedansmith: sure14:40
*** mestery_ has joined #openstack-infra14:40
sdagueso something early in your set is driving the load higher again, which is why large ops is falling over14:41
sdaguebecause it's not falling over anywhere else right now14:41
dansmiththat's not what was causing the failure the first time I pushed,14:41
dansmithand the second time I did, I had a fix for the issue it was hitting (which seems to have worked)14:41
sdaguedansmith: ok, it's how it failed the 2 times it went yesterday14:41
dansmith*now* it may be load, although I don't really see how, since it's failing in init_host14:42
dansmithsdague: why do you say that?14:42
dansmithbecause you saw a timeout?14:42
*** mestery__ has joined #openstack-infra14:42
sdaguedansmith: right14:42
sdagueso "load higher" might not be the right base issue14:43
*** gokrokve_ has joined #openstack-infra14:43
dansmithsdague: but the timeout was because conductor had fallen over early during init, not because it was waiting too long14:43
dansmithand I fixed that with a new patch in front14:43
sdaguedidn't that whole patch series fail again yesterday?14:43
sdagueafter that14:43
*** meste____ has joined #openstack-infra14:43
dansmithsorry, after what?14:43
*** mestery has quit IRC14:43
sdagueafter you pushed again last night14:44
*** senk has joined #openstack-infra14:44
dansmithseveral passed in front14:44
dansmitheverything failed after this:14:44
dansmithfor that reason,14:45
dansmithwhich I don't understand yet,14:45
dansmithbut I don't have any reason to believe that when that is resolved the others wouldn't pass,14:45
dansmithand splitting that set is non-trivial and significantly adds to the housekeeping to keep it all straight14:45
dansmithso I just don't think it makes sense to not push the whole thing unless you have some other reason I don't understand14:45
*** mestery_ has quit IRC14:46
sdagueso we probably have the resources right now, but that patch series basically consumes half of all available test nodes14:46
russellbalso note this is a pre-req to revisiting the concurrency setting  ... so don't want to slow it down if we can14:46
*** mestery__ has quit IRC14:46
*** NikitaKonovalov_ is now known as NikitaKonovalov14:46
dansmithsdague: seems like a scheduling/quota thing to work out14:46
sdagueespecially when the fail condition makes everything go to the timeout14:47
russellbsdague: could be a possible zuul improvement14:47
dansmithsdague: telling folks to arbitrarily limit the size of their queues doesn't seem like a good approach to me14:47
russellbpay attention to dependent patches in check, and rate limit the check jobs going through them, something like that14:47
dansmithrussellb: or even throttle per user instead of per branch, but yeah14:48
russellbyeah per user may be easier, i dunno14:48
sdagueyeh, don't know, right now all our zuul brain power is going into other things14:48
*** senk has quit IRC14:49
sdaguedansmith: also realistically you are the outlier case :)14:49
dansmithsdague: this was a tanget branch to immediately help the gate, and is blocking me from working on actual things I need to work on, so artificially making it harder/longer is really a problem, IMHO14:49
*** senk has joined #openstack-infra14:49
sdaguedansmith: yeh, no worries, I was just trying to say could we trial balloon the patches just up to the first one that fails before pushing everything, grabbing a branch at that ref and pushing should be really not too bad14:50
*** flaper87 is now known as flaper87|afk14:50
sdagueand can put the whole series back together real fast after that14:50
dansmithsdague: but then I have rebase pain if I make several changes to fix the problem and slap the others on top of it14:50
dansmithsdague: these are highly interdependent14:51
*** jcoufal has quit IRC14:53
sdagueok, just a suggestion14:53
*** jcoufal has joined #openstack-infra14:54
*** prad has joined #openstack-infra14:55
*** ruhe is now known as _ruhe14:56
*** mestery has joined #openstack-infra14:57
*** hashar has joined #openstack-infra14:57
*** mestery_ has joined #openstack-infra14:58
*** chandankumar has joined #openstack-infra14:59
*** meste____ has quit IRC15:00
* mordred isn't REALLY following - but if I wanted to do what sdague was suggesting without screwing myself ...15:00
mordrednope. nevermind. it sucks15:01
* mordred likes working things out as he tries to actually type solutions :)15:01
openstackgerritsebastian marcet proposed a change to openstack-infra/config: Implements: blueprint openid-oauth2-infra-implementation-documentation  https://review.openstack.org/6962015:01
*** mestery__ has joined #openstack-infra15:01
*** ewindisch is now known as zz_ewindisch15:01
sdagueyeh, because of the new rax quota, we're actually kind of fine right now. Without that it would have starved the gate15:02
*** mestery has quit IRC15:02
*** mrodden has quit IRC15:02
*** NithyaG has quit IRC15:02
*** jnoller has joined #openstack-infra15:03
*** NithyaG has joined #openstack-infra15:03
*** dcramer_ has joined #openstack-infra15:03
*** mestery_ has quit IRC15:03
*** jnoller has quit IRC15:04
*** jnoller_ has joined #openstack-infra15:04
*** IzikP_Radware has joined #openstack-infra15:04
*** weshay has joined #openstack-infra15:05
*** mestery has joined #openstack-infra15:05
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add rtfd trigger jobs for climate project  https://review.openstack.org/6806215:05
*** chandankumar has quit IRC15:06
*** kraman has joined #openstack-infra15:06
*** mestery__ has quit IRC15:09
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Database fixture added  https://review.openstack.org/6938415:10
*** rwsu has joined #openstack-infra15:12
markmcdansmith, I totally agree with what you're saying, but you could push in batches if that would help15:14
markmcgit push gerrit HEAD~30:refs/publish/master15:14
markmcthen later after they've passed15:14
markmcgit push gerrit HEAD~20:refs/publish/master15:14
markmcgerrit will handle that fine15:14
dansmithmarkmc: okay, I've never pushed to gerrit without git review15:15
dansmithmarkmc: if I don't have to break my local branch, then that's what I want15:15
*** malini_afk is now known as malini15:16
markmcyeah, so - git-review just pushes to refs/publish/master15:16
markmcHEAD~30:refs/publish/master is a refspec15:16
*** mrodden has joined #openstack-infra15:16
markmcgit-push manpage has a nice explanation of refspecs15:16
*** coolsvap has joined #openstack-infra15:16
markmcbut that means, "everything up to 30 patches back on this branch"15:17
dansmithyeah, I'm familiar, I'm just trained to not have to know the mechanics of how my patches get there :)15:17
mordredyay tools!15:17
*** zz_ewindisch is now known as ewindisch15:19
*** jgrimm has joined #openstack-infra15:20
*** che-arne has quit IRC15:20
*** rcleere has joined #openstack-infra15:22
*** morganfainberg|z is now known as morganfainberg15:22
*** dkranz has joined #openstack-infra15:24
*** thedodd has joined #openstack-infra15:26
*** talluri has quit IRC15:28
*** talluri has joined #openstack-infra15:28
*** _ruhe is now known as ruhe15:30
*** yaguang has quit IRC15:30
*** senk has quit IRC15:30
*** yamahata has quit IRC15:30
ttxmordred: I was cleaning up milestone-proposed branches from icehouse-2 and I accidentally removed swift's milestone-proposed branch as well (the yet-to-be-released one for 1.12.0)15:31
ttxmordred: is there any convenient way to restore that ?15:31
*** talluri has quit IRC15:32
ttxnotmyname: worst case scenario I can recreate it and we can repush the backport15:35
*** senk has joined #openstack-infra15:36
ttxfungi: ^if you have any suggestions... I think we are stuck because it wasn't tagged yet15:36
mordredttx: do you know what the sha was supposed to have been?15:41
*** eharney has quit IRC15:41
ttxlooks like https://review.openstack.org/gitweb?p=openstack%2Fswift.git;a=commitdiff;h=78704010cc4c3b5c2bd050ee846323506ad3ed5b15:41
*** david-lyle has joined #openstack-infra15:41
ttxWas about to try to recreate at 78704010cc4c3b5c2bd050ee846323506ad3ed5b15:41
mordredttx: I think recreate is the right choice15:41
ttxwill download the tarball first to check if it generates the same content15:42
*** wenlock has joined #openstack-infra15:43
ttxOK recreated15:43
ttxman, that git thing is really magic15:43
ttxnotmyname: branch recreated, just doublecheck it's sane15:44
ArxCruzALL: Can someone help me with zuul pipelines ? I want to report back my results, but I don't want to vote +1 or -1 (for now). If someone have an example, that would be awesome :)15:45
mordredArxCruz: one sec - I'll paste a link15:46
ArxCruzmordred: thanks!15:46
dstanekhi all, how do we go about getting a non-voting python3 job for keystone?15:46
mordredArxCruz: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n23415:47
*** amotoki has joined #openstack-infra15:47
ArxCruzmordred: okay, but which pipeline should I add in my yaml to post the results ?15:48
*** NikitaKonovalov is now known as NikitaKonovalov_15:48
dstaneki was looking for our jenkins-job-builder yaml, but i have no idea where to find it15:50
*** jcoufal has quit IRC15:50
*** jcoufal-mobile has joined #openstack-infra15:51
*** andreaf has quit IRC15:52
ArxCruzdstanek: it is on zuul yaml file :)15:52
ArxCruzdstanek: by the way, I'm looking for some zuul example to report back (with no voting)15:53
dstanekArxCruz: is that in openstack-infra/config somewhere?15:54
*** rcleere has quit IRC15:54
ArxCruzdstanek: just a sec, let me check15:54
ArxCruzdstanek: same url mordred points me15:55
*** reed has joined #openstack-infra15:55
dstanekmordred: thx - i missed that15:56
* fungi is back but on a conference call for the next hour, so not really paying attention yet15:57
*** rcleere has joined #openstack-infra15:57
*** wenlock_ has joined #openstack-infra15:58
*** AlexF_ has joined #openstack-infra16:01
*** freyes has quit IRC16:01
*** starmer has joined #openstack-infra16:02
openstackgerritsebastian marcet proposed a change to openstack-infra/config: Implements:  blueprint openid-oauth2-infra-implementation-puppet-script [smarcet] - #5478 - Clean up puppet Scripts  https://review.openstack.org/6963616:04
notmynamettx: lgtm16:04
ttxnotmyname: checked that the tarball content was identical too16:05
*** hashar has quit IRC16:05
ttxsorry for the glitch.16:05
mordredsmarcet: hey! I don't know much about your codebase yet - but quick question - two different mysql hosts and a redis?16:06
fungimordred: apparently one db is the silverstripe backend, the other is the openid provider backend16:07
*** hashar has joined #openstack-infra16:07
smarcet@mordred: redis host will run most likelly locally, but mysql hosts will be external ones and would be probably hosted on trove16:07
*** boris-42 has quit IRC16:07
mordredsmarcet, fungi: ah - ok. makes sense16:07
mordredalso, what's a greggilbert? :)16:07
smarcetfungi: that is rite, one mysql for openstackid and another one for ss db16:08
fungiin theory, the databases might get combined in the future once the org site gets opened up16:08
smarcetmordred: is a laravel lib that provides recaptcha implementation16:08
*** jcooley_ has joined #openstack-infra16:09
*** mfer has joined #openstack-infra16:09
*** SergeyLukjanov is now known as SergeyLukjanov_16:09
*** eharney has joined #openstack-infra16:09
*** flaper87|afk is now known as flaper8716:12
*** lcheng has joined #openstack-infra16:12
*** virmitio has joined #openstack-infra16:12
*** pcrews has joined #openstack-infra16:13
*** emagana has joined #openstack-infra16:14
*** DinaBelova is now known as DinaBelova_16:14
*** jcoufal-mobile has quit IRC16:15
*** jcoufal-mobile has joined #openstack-infra16:15
*** che-arne has joined #openstack-infra16:16
*** emagana has quit IRC16:16
*** atiwari has joined #openstack-infra16:18
*** SumitNaiksatam has quit IRC16:18
*** vkozhukalov has quit IRC16:19
*** VijayT has joined #openstack-infra16:19
openstackgerritsebastian marcet proposed a change to openstack-infra/config: Implements:  blueprint openid-oauth2-infra-implementation-puppet-script [smarcet] - #5478 - Clean up puppet Scripts  https://review.openstack.org/6963616:22
*** jcoufal has joined #openstack-infra16:23
*** jcoufal-mobile has quit IRC16:26
*** thuc has joined #openstack-infra16:27
*** talluri has joined #openstack-infra16:28
*** thuc has quit IRC16:28
*** thuc has joined #openstack-infra16:28
*** thuc has quit IRC16:28
*** gyee has joined #openstack-infra16:29
*** thuc has joined #openstack-infra16:29
*** thuc has quit IRC16:29
*** thuc has joined #openstack-infra16:30
*** wenlock has quit IRC16:30
*** thuc has quit IRC16:30
portanteany logstash kibana query experts around?16:31
*** ruhe is now known as _ruhe16:31
*** thuc has joined #openstack-infra16:31
portanteI am trying to understand why for this effort, https://review.openstack.org/69068, I am not able to reduce the double hits to one.16:32
portantethis is for the uncat horizon bugs (top of the uncat page)16:32
*** dizquierdo has quit IRC16:32
jesusauruswhat do you mean by double hits?16:33
*** SumitNaiksatam has joined #openstack-infra16:33
portanteit is reporting the same log file twice in the results16:38
sdagueportante: that's ok16:38
sdaguewe do squashing during the reporting for failures in the overall hit page16:39
portanteokay, so then we can just take that commit as is then?16:39
portantesdague: see the comments on the commit16:39
portantesdague: I thought the posted query would reduce the double hits, but it does not match anything, which says I really don't understand what the query is doing16:40
sdagueportante: yeh, I think jog0 just wanted to figure out if there was a narrower one16:40
openstackgerritDavid Stanek proposed a change to openstack-infra/config: Adds a non-voting python33 check to Keystone  https://review.openstack.org/6964916:40
sdagueportante: where is the other proposed query?16:41
portantein the last comment16:41
dstanekmordred, ArxCruz: is this really all i have to do? https://review.openstack.org/#/c/69649/16:41
portanteI justed added another message clause for one of the tests that failed, which is in all the logs that I have sampled16:42
sdagueportante: so you can only match single log lines, because of the way things are indexed16:42
sdaguewhich is how logstash works16:42
ArxCruzdstanek: according mordred told me, yes16:42
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1268631  https://review.openstack.org/6906816:43
sdaguewhich is less than ideal, but we've gotten a lot of mileage on it so far, so we haven't implemented multiline matches yet16:43
portantemeaning, the two "message" clauses have to match on the same line?16:43
sdagueportante: yes16:44
sdaguebecause each line is a document to logstash16:44
portantek, then I don't see a way to narrow this one down further16:44
sdagueyep, agree16:45
portantethanks, I think I get it now16:45
sdagueI just approved it, it's in the repo now16:45
portantegreat, thanks16:45
*** chandankumar has joined #openstack-infra16:45
dstaneki don't see a lot of pypy checks - is that something worth doing for projects?16:45
dstanekArxCruz: thx16:46
*** lcheng_ has joined #openstack-infra16:47
*** hashar has quit IRC16:50
*** lcheng has quit IRC16:50
*** hashar has joined #openstack-infra16:51
*** whoops has joined #openstack-infra16:51
*** emagana has joined #openstack-infra16:56
anteayamorning clarkb16:58
*** DinaBelova_ is now known as DinaBelova17:00
*** afazekas has quit IRC17:00
zaroclarkb, mordred : found this issue with mysql and jeepyb.  simple fix, probably should go in soon.. https://review.openstack.org/#/c/69543/17:00
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add trove translation jobs  https://review.openstack.org/6430917:02
clarkbI am just going to approve that, it is a merge conflict resolution for a change approved yesterday17:02
*** NikitaKonovalov_ is now known as NikitaKonovalov17:02
clarkbzaro: is that tuple compatible with older versions too?17:02
notmynamemordred: swift 1.12.0 was tagged on m-p proposed today. can you check that the merge job will happen so that the versioning keeps working?17:03
*** e0ne_ has quit IRC17:03
zaroclarkb: it works with MySQL-python version 1.2.4 and 1.2.517:03
*** marun has joined #openstack-infra17:03
mordrednotmyname: it ran and failed... looking17:04
*** e0ne has joined #openstack-infra17:04
clarkbzaro: perfect thanks, approved17:04
notmynamemordred: thanks17:04
*** marun has quit IRC17:05
*** dcramer_ has quit IRC17:07
anteayamorning zaro17:07
*** boris-42 has joined #openstack-infra17:07
*** kraman has quit IRC17:08
openstackgerritA change was merged to openstack-infra/jeepyb: fix manage_projects with MySQL-Python 1.2.5  https://review.openstack.org/6954317:08
*** marun has joined #openstack-infra17:08
*** e0ne has quit IRC17:09
mordrednotmyname: https://review.openstack.org/6966617:09
*** chandankumar has quit IRC17:09
mordrednotmyname: I ran the script locally to do that, and it worked, so I'm going to need to investigate further why it was unhappy on the jenkins slave17:09
*** kraman has joined #openstack-infra17:10
notmynamemordred: interesting. (I thought) I had resolved that conflict when I backported the last patch onto m-p. probably the merge back to master is causing the issue17:11
*** NikitaKonovalov is now known as NikitaKonovalov_17:11
mordrednotmyname: it's _supposed_ to null-merge17:12
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Run the tests for the OpenStack SDK tests  https://review.openstack.org/6966717:12
vjayHi, I am new to OpenStack infra. Jenkins reported failures due to bugs which are listed in  http://status.openstack.org/rechecks/ in Gerrit. But there is no entry from Elastic Recheck in Gerrit.17:14
*** starmer has quit IRC17:14
vjayWhat could be the reason?17:14
anteayavjay: elastic recheck doesn't know about the bug17:14
anteayayou can run recheck bug #bug number17:14
anteayayou also can add the bug number to elastic recheck17:15
*** _ruhe is now known as ruhe17:15
anteayavjay: http://docs.openstack.org/infra/elastic-recheck/readme.html17:15
*** chandankumar has joined #openstack-infra17:15
anteayathanks for asking and welcome to -infra17:15
vjaylet me check17:16
*** gokrokve_ has quit IRC17:16
openstackgerritA change was merged to openstack-infra/git-review: Correct .Fl typo WRT --compare in the manual page  https://review.openstack.org/6618217:16
notmynamemordred: I've got to drive to San Jose now. I'll check in again in an hour or so17:16
*** pcrews has quit IRC17:17
* hashar teleports notmyname 17:18
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools gates for openstack-manuals  https://review.openstack.org/6966917:19
openstackgerritPeter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1266616  https://review.openstack.org/6967017:20
smarcetfungi: mordred: hi guys, i am getting a puppet syntax error here https://review.openstack.org/#/c/69636/17:20
smarcetbut http://logs.openstack.org/36/69636/2/check/gate-config-puppet-syntax/b8954d4/console.html is referring to files that were not included on my patch , like17:20
smarcet[0;33mwarning: Deprecation notice:  Resource references should now be capitalized on line 557 in file /home/jenkins/workspace/gate-config-puppet-syntax/modules/gerrit/manifests/init.pp17:20
openstackgerritA change was merged to openstack-infra/config: Add trove translation jobs  https://review.openstack.org/6430917:21
*** moted has joined #openstack-infra17:21
fungismarcet: i bet we started testing with a new puppet-lint release17:21
* fungi checks17:22
*** dkliban is now known as dkliban_afk17:23
fungismarcet: oh, those are only warnings... i forgot we don't currently fail on warnings, just errors17:25
vjayanteaya: thanks for the super quick response. Let me see if I can understand. once i have commented "recheck bug #bug_number", any further patchset submission failures due to the same reason/query will be identified by Elastic Search.17:25
fungismarcet: if you scroll down to the end of the log, you'll see some syntax errors in modules/redis/templates/redis.2.4.conf.erb17:27
smarcetfungi: ok i think that i know what the error is , tnx :)17:27
clarkbmordred: if I address jeblair's comment in https://review.openstack.org/#/c/62187/ and get the beginning of that stack merged then the jeepyb change can land?17:27
smarcetyeah just saw it, i overlooked it :(17:27
anteayavjay: not quite17:27
*** zhiyan has quit IRC17:27
fungismarcet: it's like looking for a needle in a haystack sometimes17:27
*** ruhe is now known as _ruhe17:28
anteayaelastic recheck bug numbers are submitted by hand: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries17:28
*** flaper87 is now known as flaper87|afk17:28
jnoller_sdague: did you get all set up?17:28
openstackgerritA change was merged to openstack-infra/config: Adds a non-voting python33 check to Keystone  https://review.openstack.org/6964917:28
clarkbwoah ^ is exciting17:29
clarkbdid keystoneclient ever get py33 sorted?17:29
anteayain order for elastic recheck to know about a bug, some wonderful contributor has taken the time to identify a logstash fingerprint for that bug and submit it as a commit to the elastic-recheck repo17:30
fungiclarkb: i figured that one was safe enough to only need one core review17:30
mordredclarkb: other way around17:30
mordredclarkb: we need to merge the jeepyb change, then we can land that change17:31
mordredthe jeepyb change is backwards compatible17:31
*** BobBall is now known as BobBallAway17:31
vjayprobably i should rephrase my question. In my case, there is already a bug and already a yaml submitted. Elastic Recheck entry is not adding an entry to my gerrit review submit.17:31
anteayavjay: using recheck no bug allows recheck to know about it, and also is the syntax zuul is looking for to all ow the patchset to be testing in the check queue again17:31
anteayavjay: let's get more specific17:32
anteayaplease post the url of your patch, the bug and the yaml file in question17:32
anteayathere might be an error in syntax17:32
clarkbmordred: jeepyb is backward compat but puppet isn't aiui17:32
vjayPatch: https://review.openstack.org/#/c/57524 is my code submit17:32
clarkbor are they both compatible in both directions?17:33
mordredso we want to land the jeepyb change, which will understand old and new format17:33
mordredthen we can land the puppet change, which will move the file to the new format17:33
clarkbmordred: gotcha17:33
clarkbmordred: I am about to push to that stack addressing jeblair's comments17:33
openstackgerritsebastian marcet proposed a change to openstack-infra/config: Implements:  blueprint openid-oauth2-infra-implementation-puppet-script [smarcet] - #5478 - Clean up puppet Scripts  https://review.openstack.org/6963617:33
*** chandankumar has quit IRC17:34
*** yassine has quit IRC17:34
anteayavjay: who is embrance ci?17:35
vjayyaml: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/diff/queries/1254890.yaml17:35
vjayi dont know :-)17:35
pleia2kashyap: left some notes in the etherpad re: fedora in the gate17:35
clarkbpleia2: thanks17:36
anteayaback in conf tutorial, I'll come back at next break unless someone helps you before that17:36
fungianteaya: embrane ci is one of the third-party test systems... for neutron i think17:36
*** SergeyLukjanov_ is now known as SergeyLukjanov17:36
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add Storyboard puppet module  https://review.openstack.org/6501717:36
openstackgerritClark Boylan proposed a change to openstack-infra/config: Unlaunchpadify projects.yaml  https://review.openstack.org/6218917:36
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add storyboard-dev node with krotscheck and ruhe on it  https://review.openstack.org/6951617:36
openstackgerritClark Boylan proposed a change to openstack-infra/config: Track direct-release projects in projects.yaml  https://review.openstack.org/6219017:36
openstackgerritClark Boylan proposed a change to openstack-infra/config: Split config from projects list  https://review.openstack.org/6218717:36
vjaybug: https://bugs.launchpad.net/nova/+bug/125489017:36
clarkbfungi: any chance you can rereview the jeepyb change for ini config then we can review the stack above?17:36
fungianteaya: i believe it went haywire a while back and we revoked voting for it, now the operators of it are waiting on markmcclain to okay allowing it to vote again17:37
kashyappleia2, Thank you, will check tomorrow. It's late here :-)17:37
fungiclarkb: i'm sure the chance is nonzero. i'll go find it now17:37
dimsfungi, when you get a chance, please consider blessing this - https://review.openstack.org/#/c/67508/ ("Add jenkins master name to the logstash records")17:37
anteayafungi: okay thanks, later I will checkin with markmcclain about it17:37
clarkbfungi: trying to kill as many changes in review as possible :)17:38
clarkbdims: I can bless that17:38
clarkbdims: would it be Ok to make that a WIP until we have the plugin installed everywhere?17:38
vjayThe failure log in : http://logs.openstack.org/24/57524/18/check/check-tempest-dsvm-neutron-pg/1a5c336/logs/testr_results.html.gz is matching http://git.openstack.org/cgit/openstack-infra/elastic-recheck/diff/queries/1254890.yaml17:39
dimsclarkb, code logs "UNKNOWN" if data is not available. but am ok to WIP as well17:40
clarkbdims: I suppose we can merge it then get the data when available17:40
dimsclarkb, sounds good17:40
*** dcramer_ has joined #openstack-infra17:42
openstackgerritA change was merged to openstack-infra/jeepyb: Split the config out into two files  https://review.openstack.org/6950317:42
ArxCruzALL: I have my nodepool server running, it's adding the VM's on jenkins, jenkins is running tempest fine, however, after the tempest job finishes, the VM isn't automatically removed and a new one is added, it continues to try run other jobs in queue, and then it fails (due the garbage left from the last run)17:43
clarkbArxCruz: in our jobs list in layout.yaml for zuul we have a single use server function applied to *dsvm* jobs. I think you need that17:44
*** ICmonitor1 has joined #openstack-infra17:44
clarkbthat function tells gearman plugin on the jenkins side to mark the node offline when the test begins so that only one test will run on it17:44
*** _david_ has joined #openstack-infra17:44
ArxCruzclarkb: you know the function name ?17:44
_david_zaro, which plugin?17:45
fungiArxCruz: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n19817:45
ArxCruzfungi: thanks!17:45
fungiArxCruz: "parameter-function: single_use_node"17:45
*** gokrokve has joined #openstack-infra17:45
jerryzclarkb: sometimes, i still got confused. i used to have the same condition as ArxCruz, and i add the single use option in layout.yaml. But aren't the slaves labeled 'used' once a job ran on it? even it didn't get deleted, new jobs are not supposed to run a used slave in the first place.17:46
*** bookwar has quit IRC17:46
clarkbjerryz: they are marked used in the nodepool DB but not in jenkins17:46
zaro_david_: replication plugin17:46
*** ICmonitor has quit IRC17:47
clarkbso jenkins will happily run more jobs on the slave17:47
*** malini is now known as malini_afk17:47
*** hashar has quit IRC17:47
zaro_david_: https://groups.google.com/forum/#!topic/repo-discuss/vgDzcPjVmJ817:47
_david_zaro this is core plugin17:47
jerryzclarkb: so nodepool won't change the label in jenkins as d-g jobs used to do17:47
*** sandywalsh has quit IRC17:47
clarkbjerryz: I don't think so17:47
*** gokrokve has quit IRC17:47
clarkbjerryz: and even if it does that code was always racy17:47
fungijerryz: the label changing had unfortunate race conditions17:47
zaro_david_: ohh yeah, forgot.  but generally should 3rd party plugin bugs go into main gerrit issue tracker?17:48
fungijerryz: jenkins tended not to process the relabel request right away17:48
_david_zaro, i would say yes.17:48
fungijerryz: the jenkins-gearman plugin now has functionality to offline the nodes in jenkins instead17:48
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools gates for openstack-manuals  https://review.openstack.org/6966917:48
_david_(main project)17:48
_david_zaro, i would say you can't do that with alias., but why you ned to?17:49
_david_zaro, what is wrong with ssh gerrit replication start?17:49
_david_zaro, why adapt Gerrit and not your script?17:49
zaro_david_: from the documentation i thought that's what the ssh-alias allows you to do.17:50
_david_zaro, yeas, but inside core17:50
openstackgerritA change was merged to openstack-infra/config: Add jenkins master name to the logstash records  https://review.openstack.org/6750817:50
_david_zaro, not between core and plugins, because namespaces are different, (i think)17:51
jerryzfungi: great. the gearman plugin is updated?17:51
*** ICmonitor1 has quit IRC17:51
zaro_david_: i see.  i guess we only have option then?17:51
_david_zaro, can't parse your last statement?17:52
zaro_david_: i mean *one* option17:52
_david_zaro, still don get what is wrong with ssh gerrit replication start ?17:52
zaro_david_: nothing, except that it breaks backwards compatability with stuff that depends on the gerrit replicate command17:53
jerryzfungi: oh, i see it is a feature in 0.0.517:53
_david_zaro, what backwards compatibility? why it is needed? If we migrate to 2.8 who cares about backwards compatibility?17:53
zaro_david_: i would prefer to not change our scripts17:53
clarkbfungi: thank you for reviewing the jeepyb split config change. The companions to that change start at https://review.openstack.org/#/c/62187/17:53
fungizaro: _david_: i think it's probably fine if we update our manage-projects script to start doing a version check through the api and then branch on which commands it uses depending on the gerrit release it's talking to, assuming gerrit upstream is not concerned with api backward compatibility17:54
clarkb_david_: zaro: gerritlib is a gerritlib. not a gerrit 2.4 lib or a gerrit 2.8 lib17:54
clarkbit needs to be backward compat17:54
fungioh, right, we extracted the replication call out to gerritlib17:54
clarkb_david_: it is insane that a command name like that which honestly doesn't matter as long as it stays the same would change17:55
fungiso gerritlib should probably start doing version detection if there are no forward/backward compat guarantees for the gerrit api17:55
clarkbthat said, its not like we worry terribly much about backward compat around here17:55
zaroi'm looking into adding version command to gerritlib now17:55
_david_clarkb, I still don't understand that backward compatibility issue you mentioned with gerritlib17:56
*** gokrokve has joined #openstack-infra17:56
fungiright, i think version detection is likely to become necessary anyway since this is probably one of many breaking api changes we should expect from future gerrit releases17:56
clarkbgerrit seems to have realized this problem and added ssh aliases17:56
portantesdaque: I am looking at the gate-grenade-dsvm uncat jobs17:56
portantesdague: sorry17:56
clarkb_david_: we maintain python's gerritlib on pypi. That library should be able to talk to different versions without its users carring what the command is17:56
zaroohh yeah, i also patched python-jenkins with version command.  jeblair can you do your magic on that so we could use in jjb?17:57
clarkb_david_: it adds a bit of complexity to the lib to now need to do version detection and command lookups in a table17:57
portantethere is a high correlation between the "iSCSI device not found" errors in logs and about 6+ of those jobs failing17:57
*** ewindisch is now known as zz_ewindisch17:57
portantebut I am having a hard time finding the bug to hang that off of17:57
openstackgerritSergey Lukjanov proposed a change to openstack-infra/reviewstats: Add missed climate repos  https://review.openstack.org/6968017:58
*** IzikP_Radware has quit IRC17:58
*** zz_ewindisch is now known as ewindisch17:58
zaroclarkb, _david_ : just thought i could alias the command instead. i'll just patch our script.17:58
_david_clarkb, KISS principle. Replication feature was extracted to a plugin 4 years ago or something. I am not aware of any Gerrit installation that has 2.4 version. So just change it and forget the issue with version detection17:58
clarkb_david_: it wasn't 4 years ago.. I have only been working on this stuff for less than 2 years :P17:59
_david_clarkb, ,-)17:59
clarkb_david_: the problem isn't that it became a plugin it is that the command name itself changed17:59
zarowasn't it since 2.6?17:59
clarkbwhat was wrong with `gerrit replicate`17:59
clarkbnow it is `replications start`17:59
*** e0ne has joined #openstack-infra17:59
_david_clarkb, i try to explain: the command name didn't change at all ...17:59
_david_clarkb, it was start and is remained start18:00
*** gokrokve has quit IRC18:00
_david_replication is the plugin name18:00
_david_ro prevent clashes between different plugin an core18:00
fungi_david_: well, we're not upgrading overnight. we need to be able to test the new version of gerrit with our library while still supporting the version we have in production, at least until we've safely worked out all the changes we need to implement to upgrade and get it to a point where we know we don't need to roll back. so i think version detection or at least some fallback (try one way, then try18:00
fungithe other when it fails) will be necessary for at least a transitional period18:00
_david_so you can have plugin foo and bar and both can have SSH command baz18:00
_david_ssh gerrit foo baz18:01
_david_ssh gerrit bar baz18:01
clarkb_david_: maybe the alias needs to be installed by default then?18:01
clarkbanyways the ship has sailed, we will deal with it18:01
_david_clarkb, fungi zaro WIP plugin idea:18:02
openstackgerritSergey Lukjanov proposed a change to openstack-infra/reviewstats: Add murano with all subprojects  https://review.openstack.org/6968118:02
_david_i have an idea how we can my be not need that core change that was rejected:18:02
_david_Just use DRAFT status as replacement for WORKINPROGRESS status, and block draft workflow completely.18:03
_david_i still have to veriy it18:03
*** sandywalsh has joined #openstack-infra18:03
clarkbso overload that status and make the behavor different?18:03
_david_If it is working, we are done18:03
_david_clarkb, exactly18:03
_david_clarkb, https://gerrit-review.googlesource.com/#/c/5406218:04
fungithat's certainly an idea. as long as we can still control draft/undraft as non-change-owners via acl18:04
zarothought draft feature was going to get reworked?18:04
_david_in this change i disabled pushing to refs/drafts18:05
*** DinaBelova is now known as DinaBelova_18:05
fungialso it would be nice to still be able to draft/undraft through the api too (we want to make git-review capable of submitting changes and putting them into wip status immediately without button clicking in the webui)18:06
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools gates for openstack-manuals  https://review.openstack.org/6966918:07
_david_fungi, shoudn't it already doable with ssh command?18:07
clarkbmordred: still around? https://review.openstack.org/#/c/68439/ when you bumped RAM quota did you bump cpu quota too?18:08
*** pcrews has joined #openstack-infra18:08
clarkbmordred: there were cpu quota limit log error messages in the nodepool logs yesterday18:08
zaro_david_: draft/undraft is not part of draft workflow18:09
zaro_david_: how can you do it thru ssh?18:09
_david_zaro, not draft/undraft, but wip/ready18:10
mordredclarkb: I have no idea.18:10
_david_zaro, https://github.com/davido/gerrit-wip-plugin/blob/master/src/main/java/com/googlesource/gerrit/plugins/wip/SetCommand.java18:11
mordredclarkb: I just asked pvo for quota bump and he made magic happen18:11
fungi_david_: yes, our old wip patch made api extensions, but we ended up not implementing them long-term in git-review because we expected the implementation to change in later gerrit anyway18:11
fungimordred: vcpu quota?18:11
_david_fungi, in our plugin, where i have +2 we can do everything what you like ;-)18:11
openstackgerritSergey Lukjanov proposed a change to openstack-infra/reviewstats: Add mistral with subprojects  https://review.openstack.org/6968318:12
fungi_david_: right. assuming that doesn't change when switching to overloading the draft status field to track it, then it sounds like precisely the interface we should implement in git-review18:13
mordredfungi: I asked for it in terms of numbers of 8G instances18:13
fungimordred: that's a sane enough choice (we don't have many which aren't, and those are pretty much all in dfw)18:13
openstackgerritSergey Lukjanov proposed a change to openstack-infra/reviewstats: Add murano with all subprojects  https://review.openstack.org/6968118:14
zaro_david_: cool.  it's an idea worth exploring, however my concern is that draft feature might drastically changed so i don't know the impact on depending on it.18:14
*** dkliban_afk is now known as dkliban18:14
*** praneshp has joined #openstack-infra18:15
clarkbAaronGr: https://review.openstack.org/#/c/66072/2 did you plan on responding to jeblair there?18:15
*** mkerrin has left #openstack-infra18:15
_david_zaro, we just turn it off: nothing simpler as that, not matter how it is going to change, it is turn off and overloaded:18:16
AaronGrclarkb: i did .. but i wonder why it didn't save my change. will re-add18:16
clarkbAaronGr: thanks18:16
clarkbfungi: the other change (if I can be selfish) that I think could use review is https://review.openstack.org/#/c/69442/. That should simplify our use of puppet for slaves, fix sudo on single use unittest nodes, and fix the dpkg thing18:17
AaronGrclarkb: i see, it's showing as a draft. how do i un-draft a comment?18:17
_david_zaro, fungi https://gerrit-review.googlesource.com/#/c/54075 , i will verify this approach and let you know18:17
clarkbAaronGr: go to the top change page, and click review18:17
*** harlowja_away is now known as harlowja18:17
clarkbthen save the review. It will give you the option for a cover message and show you your inline messages18:17
zaro_david_: cool.  i will review that sometime soon.18:18
zaro_david_: btw, dborowitz says gerrit issue tracker should only be for core plugins.18:18
*** gokrokve has joined #openstack-infra18:18
_david_zaro, ok18:19
fungiclarkb: yeah, i started reviewing 69442 but had to take a break. it was brain-twisty. trying again in a bit18:20
*** gokrokve_ has joined #openstack-infra18:20
_david_clarkb, zaro, fungi folks want to know details about Zuul: https://groups.google.com/forum/#!topic/repo-discuss/Wica7iJ5DGk18:20
*** _david_ has quit IRC18:20
AaronGrclarkb: thanks18:21
clarkbfungi: np I spent a good 10 minutes or so building a model of includes in my head18:21
fungiit looks well-thought out. i just need to be able to repeat most of the well-thinking ;)18:21
clarkbwe had managed to cthulu it a bit18:21
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Use openstack-doc-tools gates for openstack-manuals  https://review.openstack.org/6966918:21
*** VijayT has quit IRC18:22
*** gokrokv__ has joined #openstack-infra18:22
*** vkozhukalov has joined #openstack-infra18:22
openstackgerritAndreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools to 0.4  https://review.openstack.org/6968518:23
*** gokrokve has quit IRC18:23
fungineat! that zuul thread on the gerrit list is a great opportunity for expanding the userbase18:24
*** gokrokve_ has quit IRC18:24
fungiwho do we have planning to attend the gerrit summit?18:24
*** vjay has quit IRC18:24
clarkbI think jeblair and zaro submitted talks18:24
reedany suggestion for what is going wrong here? http://paste.openstack.org/show/62034/18:25
*** nati_ueno has joined #openstack-infra18:25
*** jp_at_hp has quit IRC18:25
zaroi have submitted a general talk fro CI infra.  but will probably not go indepth on zuul alone.18:26
zarofungi: you wanna go?18:26
*** gokrokv__ has quit IRC18:26
clarkbreed: I think we need to see your template18:27
mordredneat! on zuul on gerrit list18:27
clarkbreed: I am guessing there is somethign in the template the substituter is falling over on18:28
*** lcheng_ has quit IRC18:28
reedclarkb, the settings.py file http://paste.openstack.org/show/62035/18:29
anteayaI think markmcclain hit the bug that was mentioned about dealing with the new rax nodes: http://logs.openstack.org/53/69653/1/check/check-tempest-dsvm-neutron-pg-isolated/48bc4c6/console.html18:29
anteayait looks to me like this was executed on one of the new rax nodes, was it not?18:29
anteayado we have a bug number for that yet?18:29
*** jpich has quit IRC18:30
*** VijayT has joined #openstack-infra18:30
fungireed: what's in line 11 column 51 of the list you're processing?18:30
clarkbreed: $600-off18:30
mgagneis there a way to abandon a change that's not mine?18:30
fungimgagne: one of the gerrit admins can. what change number?18:31
clarkbreed: need to escape the $ I thinnk18:31
reedeven inside the  """ ?18:31
clarkbreed: $$600-off is what you need18:32
clarkbreed: yes because it is substituting $name and $code and so on, it things $600 needs substituting too18:32
*** jooools has quit IRC18:33
fungiaha, yes makes sense. that was line 11 of the EMAIL_TEMPLATE format string18:33
*** DinaBelova_ is now known as DinaBelova18:33
reedthat was it, thanks clarkb18:33
clarkbanteaya: I am not familiar with that bug18:33
clarkbreed: np18:33
anteayaclarkb: fungi asked mordred about his request when he requested the rax nodes18:34
fungizaro: if we already have two reps from os-infra going to the gerrit summit, i'm not sure it makes a ton of sense for me to fly to california in march for it18:34
anteayamordred mentioned he requested 8G nodes18:35
anteayadid I mis-understand the conversation?18:35
anteayathat is entirely possible18:35
harlowjahi guys, could i get https://review.openstack.org/#/c/69534/ approved (followup from yesterdays changes) to make sure py2.6 works correctly18:35
clarkbanteaya: what is the bug though?18:35
harlowja*instead of currently failing (with not found)18:35
fungianteaya: well, he requested enough resources (cpu et cetera) to accomodate a specific number of 8g performance flavor nodes18:35
clarkbharlowja: done18:36
harlowjathx clarkb18:36
openstackgerritA change was merged to openstack-infra/config: Split config from projects list  https://review.openstack.org/6218718:36
harlowjau da man!18:36
anteayaclarkb: you asked the question: 2014-01-28T18:08:3218:36
anteayaah okay then I did misunderstand18:36
clarkbby the way if no one has noticed I have been approving things with one +2 if I deem them low impact. I apologize in advance if any of my judgements are unsound18:37
openstackgerritPeter Portante proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1266616  https://review.openstack.org/6967018:37
anteayaso looking at this log file, ServerFault: Got server fault would that be a neutron failure or a host failure18:37
fungiclarkb: i've been doing that for weeks (not that you can tell from the remaining backlog) so i wholeheartedly endorse that method18:37
anteayaI don't know18:37
fungianteaya: you have a log that's from, for greater context?18:38
clarkbanteaya: the line below the ServerFault tells you18:38
anteayafungi: sorry: https://review.openstack.org/#/c/69653/18:38
zarofungi: i don't see any talks on the schedule from jeblair18:39
fungianteaya: you probably already said and i just missed it. thanks18:39
anteayaah a setupclass failure18:39
clarkbzaro: I may have been mistaken about jeblair submitting one then18:39
anteayafungi: no I hadn't, I was jumping in in the middle again18:39
*** krotscheck has joined #openstack-infra18:40
*** rossella_s has quit IRC18:41
*** che-arne has quit IRC18:42
harlowjaclarkb fungi another followup question, for these custom tox jobs, it appears the testr results don't showup, but only console.html, any idea18:45
harlowjaexample @ http://logs.openstack.org/05/69205/5/check/gate-taskflow-tox-py27-sa8-mysql/7dd7044/18:45
harlowjamaybe some other config missing?18:45
notmynamemordred: looks like had the same problem you (or bot) did https://review.openstack.org/#/c/69666/18:45
notmynamemordred: works locally, git-review seems to be doing something odd18:45
*** ewindisch is now known as zz_ewindisch18:46
fungiharlowja: we probably need a separate publisher to collect them at the end of the job18:46
clarkbharlowja: the jobs use the other build script18:46
clarkbnot the run testr script iirc18:46
mordrednotmyname: git review -R right? (you definitely don't want it to rebase)18:46
notmynamemordred: ya18:46
harlowjafungi is there some special config option to make that happen?18:46
fungiharlowja: i'm looking18:47
openstackgerritSergey Lukjanov proposed a change to openstack-infra/reviewstats: Add murano with all subprojects  https://review.openstack.org/6968118:47
harlowjakk, thx fungi18:47
*** jcoufal-mobile has joined #openstack-infra18:47
*** e0ne has quit IRC18:47
notmynamemordred: ok, I don't have any more time to spend on it now. I'll try to find time tonight, if you haven't figured anything out before then18:48
fungiharlowja: the test-results publisher macro seems to grab '.testrepository/tmp*' and '**/*subunit_log.txt.gz' https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/macros.yaml#n29718:49
mordrednotmyname: wait - what are you trying to fix?18:49
mordrednotmyname: it's possible that you and I may be talking about different things18:49
clarkbharlowja: fungi: only the run-testr script puts those files in the appropriate places18:51
notmynamemordred: I'm trying to get the 1.12.0 tag on master18:51
mordrednotmyname: right. and there is a change up in gerrit for that - what's the problem?18:51
*** sdague has quit IRC18:51
fungiclarkb: harlowja: oh, right. so your test runner wrapper would need to do those same things itself18:51
harlowjak, let me double check where our testr is writing to18:52
mordrednotmyname: the left-hand diff shows what the 'conflict' of merging the tag in would have been, the right hand shows the solution, which is to turn this into a merge which does not do anything18:52
*** mrda_away has quit IRC18:52
notmynamemordred: ah, ok. I guess my issue was not understanding what the left hand was showing then. ok. I'll approve/merge it18:53
harlowjafungi clarkb which is just i guess testr, maybe a special command neeeded @ https://github.com/stackforge/taskflow/blob/master/tox.ini#L5118:53
*** smarcet has left #openstack-infra18:53
notmynamemordred: seems like it's removing some lines that aren't in master, so that's what's weird18:53
*** alexpilotti has joined #openstack-infra18:54
*** e0ne has joined #openstack-infra18:55
*** sdague has joined #openstack-infra18:56
*** mrda_away has joined #openstack-infra18:58
harlowjafungi thinking that i can have that tox.ini just output the files in the right locations, and that should maybe just do it :-P19:00
fungiharlowja: yep, ought to19:00
mordrednotmyname: yeah - the display of merge commits in gerrit is interesting - and since we never work with them, it's even weirder19:00
fungimeeting time!19:01
*** pblaho has joined #openstack-infra19:02
openstackgerritA change was merged to openstack-infra/config: Ensure taskflow py26 environments are using centos6 node  https://review.openstack.org/6953419:03
*** jcoufal-mobile has quit IRC19:03
*** rcleere has quit IRC19:05
openstackgerritAlex Gaynor proposed a change to openstack/requirements: Update python-openstacksdk from the global-requirements file  https://review.openstack.org/6970119:05
*** mrodden has quit IRC19:07
*** devkulkarni has joined #openstack-infra19:08
openstackgerritPeter Portante proposed a change to openstack-infra/elastic-recheck: Update fingerprint for bug 1270608  https://review.openstack.org/6970219:10
devkulkarniHi there. have a question about running zuul related tests. When I run tox, it doesn't seem to complete (at least it has not finished for a long time now for me). any pointers/suggestions as to what might be going on?19:10
*** VijayT has quit IRC19:11
clarkbdevkulkarni: we are in our meeting right now, can answer in about 50 minutes19:11
devkulkarniclarkb: sounds good. thanks!!19:12
*** mrodden has joined #openstack-infra19:12
*** alexpilotti has quit IRC19:13
*** rcleere has joined #openstack-infra19:13
*** thedodd has quit IRC19:14
ArxCruzfungi: besides the single_user_node, is there something else to configure? right now the nodes are disconnected from my jenkins, but there's no vm's being deployed in nodepool19:14
*** alexpilotti has joined #openstack-infra19:16
ArxCruzfungi: also I'm getting a lot of erros like this one http://paste.openstack.org/show/62037/19:16
ArxCruzwhich i have no idea why, because if I call the get_info manually, I'm able to retrieve the information19:17
openstackgerritA change was merged to openstack-infra/config: Unlaunchpadify projects.yaml  https://review.openstack.org/6218919:17
*** max_lobur is now known as max_lobur_afk19:17
anteayadevkulkarni: running tox produces no output (or very little) and takes quite a while depending on which project's tests you are running and the amount of resources available19:18
*** dolphm is now known as dolphm_afk19:18
anteayaalso if you are running the tests in parrallel or not19:18
fungiArxCruz: are you testing get_info using the same credentials (is your jenkins configured to allow the correct account)? also, we're in our weekly meeting, so i can't easily look into it for another 40 minutes or so19:19
ArxCruzfungi: yes19:19
ArxCruzfungi: oh, okay, sorry, we can talk later :)19:19
fungiArxCruz: have you checked the jenkins.log for corresponding errors?19:19
ArxCruztomorrow np :)19:19
openstackgerritA change was merged to openstack-infra/config: Track direct-release projects in projects.yaml  https://review.openstack.org/6219019:19
*** gokrokve has joined #openstack-infra19:20
*** alexpilotti has quit IRC19:21
*** dkliban is now known as dkliban_afk19:22
*** gokrokve has quit IRC19:24
*** hemna has joined #openstack-infra19:24
*** hashar has joined #openstack-infra19:25
hemnado we have any example logstash configs19:25
hemnafor pushing logs into logstash in a devstack env ?19:25
*** VijayT has joined #openstack-infra19:28
fungihemna: we have a client which relies on a zeromq plugin in jenkins to retrieve logs from completed jobs off our log publication server and feed them into logstash19:28
clarkband we have logstash filters19:28
hemnahmm ok.19:28
clarkbhemna: openstack-infra/config/modules/openstack_project/templates/logstash/indexer.conf.erv iirc19:29
hemnaI was just trying to logstash my normal devstack env (development machine)19:29
*** sandywalsh has quit IRC19:29
mikalHave any of you guys seen weird unit test failures on zuul? I can't explain the failure on https://review.openstack.org/#/c/65885/19:29
*** dkliban_afk is now known as dkliban19:29
clarkbmikal: that may be the thing that usually affects python2.619:30
pleia2clarkb: post meeting, I could use a quick sanity check re: fedora images defined in nodepool19:30
sdaguemikal: there is a zuul unit test race19:31
clarkbmikal: it appears to be a job timeout, see if it goes away with recheck? jeblair is aware of it and working it as of lca iirc19:31
mikalclarkb: as in I should recheck and see if that helps?19:31
clarkbpleia2: sure19:31
sdaguemikal: yeh, I hit it on one of mine. I think lifeless had an idea on it19:31
mikalI've never seen it before, but apparently its a thing!19:32
*** jerryz has quit IRC19:33
fungidevkulkarni reported something similar just above19:34
*** _ruhe is now known as ruhe19:34
*** pblaho has quit IRC19:34
*** rwsu has quit IRC19:36
*** hashar has quit IRC19:39
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Updated grunt proxy server to rewrite to WSGI in dev  https://review.openstack.org/6952919:40
*** rwsu has joined #openstack-infra19:40
*** sandywalsh has joined #openstack-infra19:41
*** david-lyle has quit IRC19:42
*** beagles is now known as beagles_brb19:42
*** lcheng has joined #openstack-infra19:43
*** hashar has joined #openstack-infra19:45
devkulkarnianteaya: so I am running zuul tests (via tox) on my setup (ubuntu 13.10, 1GB ram, single core). btw, how do I run these in parallel?19:45
anteayadevkulkarni: please paste your tox.ini file, paste.openstack.org is a good option19:46
*** denis_makogon has joined #openstack-infra19:47
devkulkarnianteaya: http://paste.openstack.org/show/62040/19:48
sdaguemikal: yeh, it is sufficiently rare and obtuse no one has gotten to the bottom of it yet19:48
anteayadevkulkarni: so in your tox.ini you can see testr is invoked19:50
*** AlexF_ has quit IRC19:50
*** sandywalsh has quit IRC19:50
anteayatestr allows for parrallel testing (if the project you are testing is set up for parallel testing)19:50
*** dkliban is now known as dkliban_afk19:51
devkulkarnianteaya: ah I see.19:51
clarkbit also by default requires more than a single core19:51
lyxusHello you, I waas wondering how long does it get the service account created ?19:51
devkulkarniclarkb: I see. so if I use multi-core machine these tests should run faster. let me give it a try.19:52
anteayadevkulkarni: https://wiki.openstack.org/wiki/Testr19:52
openstackgerritA change was merged to openstack-infra/reviewstats: Add missed climate repos  https://review.openstack.org/6968019:52
krotscheckCan I get a few more opinions on https://review.openstack.org/#/c/67729/ ?19:53
devkulkarnianteaya: thanks for the link. don't know whether zuul is setup for parallel testing though (I am new to zuul)19:53
lyxus( Service Accounts for 3rd party testing)19:53
anteayadevkulkarni: fair enough, I have never run tests on zuul before so I don't know off the top of my head either19:54
anteayamikal: can you run zuul tests in parallel?19:54
*** lcheng has quit IRC19:54
clarkbanteaya: they run in parallel19:54
mikalanteaya: turbo hipster does this now, as does openstack's zuul19:54
devkulkarniclarkb: thanks for confirming.19:54
mikalanteaya: however, we wait until the end to report results19:54
mikalanteaya: you can however report results as they appear if you want to19:55
clarkbkrotscheck: after meeting and lunch and stuff19:55
clarkbkrotscheck: I am still head down in reviews when not doing the meeting or eating19:55
krotscheckclarkb: Thanks. Also, for your patch this morning.19:55
anteayadevkulkarni: wait until the end until you report, if you are submitting comments to our gerrit19:56
krotscheckWouldn't mind getting fungi's opinion either.19:56
fungikrotscheck: my opinions are a commodity it seems. i'll see if i have any spares between the sofa cushions19:56
anteayadevkulkarni: if this is for a 3rd party system, one comment per event and hold off on voting until your system is stable19:57
krotscheckfungi: If they're really a commodity, can I get one at walmart?19:57
anteayaclarkb: nice job chairing today, btw19:57
fungikrotscheck: i only distribute through the 7-eleven. keeps my hipster cred intact19:58
anteayayou still are, 2 minutes left19:58
devkulkarnianteaya: sure. although, I don't think I am at a stage to submit anything yet :D need to just run the tests. this is not a 3rd party system. I am trying to help kraman and coolsvap with some of the changes that they have made in Zuul for Solum's needs.20:00
*** freyes has joined #openstack-infra20:00
fungianteaya: right, i read his questions as "running zuul's tests" not "using zuul to run tests"20:00
krotscheckfungi: I dunno, for hipster cred you're going to need to like something that has potential to be cool. I don't think 7-11 will ever be that.20:00
SergeyLukjanovlifeless, hi20:00
lifelessSergeyLukjanov: hi20:01
fungikrotscheck: proof i'm no hipster i guess20:01
SergeyLukjanovlifeless, are you already testing dib builded images in tripleo?20:01
SergeyLukjanovlifeless, or publishing them as artifacts?20:01
krotscheckfungi: Or maybe you're a meta-hipster20:01
lifelessSergeyLukjanov: we're not publishing them yet, they are local to the test run20:01
*** jcoufal has quit IRC20:01
SergeyLukjanovlifeless, ok20:01
lifelesswe'd like to be able to publish and reuse and so on20:01
clarkbpleia2: give me a few minutes before we talk fedora20:01
clarkbwill ping in a bit20:01
fungikrotscheck: i'm so hip i can't see over my own pelvis20:01
*** melwitt has joined #openstack-infra20:01
*** Sukhdev has joined #openstack-infra20:02
krotscheckfungi: You're so hip, you're titanium.20:02
* fungi needs to take a quick break, and then jump on reviews20:02
fungilyxus: i'll try to catch up on open bugs and ml threads requesting gerrit service accounts sometime today20:02
pleia2clarkb: ok20:02
anteayadevkulkarni: great, I hope you create great things20:02
anteayafungi: yes, my warning trigger finger is a little nervous20:03
*** vkozhukalov has quit IRC20:03
*** rfolco has quit IRC20:05
clarkbanteaya: thanks20:06
clarkbpleia2: now good?20:06
clarkbI need to grab lunch and make dinner shortly20:06
clarkbbut can talk fedora first20:06
* clarkb is going to attempt to make adobo. I will probably fail but adobo is too good to not try20:07
* anteaya googles adobo20:07
clarkbdims: my plan is to put your version of the zmq event plugin on jenkins-dev and check the events coming out of it then upgrade20:08
clarkbI guess I should do that before restart 02 and 03 for SCP plugin and I can do both plugins at once20:08
anteayalooks worthy of the attempt20:08
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Don't blow up on an empty core team list  https://review.openstack.org/6971120:09
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Add murano with all subprojects  https://review.openstack.org/6968120:09
*** freyes has quit IRC20:09
*** rcleere has quit IRC20:09
*** sandywalsh has joined #openstack-infra20:09
pleia2clarkb: hey, yes20:11
*** yolanda_ has quit IRC20:11
pleia2clarkb: so I just need a quick update on when the fedora issues happend and what the current status is, looking at the config files yesterday it looks like they are still defined?20:11
*** ArxCruz has quit IRC20:12
clarkbpleia2: yeah I believe most of the stuff is still in place, it wasn't an issue until we asked nodepool to build images20:12
*** mrodden1 has joined #openstack-infra20:12
clarkbpleia2: ArxCruz and dprince had done the work the first time around, let me see if I can find the commits20:12
fungiclarkb: did it cause noepool to blow up? i don't think it did20:13
* krotscheck is lying in wait for any of the infra core reviewers to indicate that they're reviewing again.20:13
*** nicedice has joined #openstack-infra20:13
fungiclarkb: it looked like nodepool20:13
fungilike nodepool was blocking on not getting responses from jenkins0220:13
clarkbpleia2: ebd8cddd6e0d52ab681624fd09bdb7e7bb936e2e was the revert commit20:13
*** mrodden has quit IRC20:13
clarkbfungi: this was before the holidays20:13
fungioh that20:14
clarkbpleia2: and has links to the things reverted20:14
clarkbpleia2: it is possible Fedora 20 doesn't have this problem20:14
russellbspeaking of node types ... i put up another pass on the "precise with saucy kernel" patch, trying to stomp top gate bug20:14
pleia2clarkb: so late last week we added the fedora 20 tripleo thing, did that have an impact?20:15
clarkbpleia2: does that help? basically nodepool sshs into the VMs and sudos a bunch of stuff, but it uses paramiko which doesn't allocate a tty and sudo on fedora 19 didn't like that20:15
clarkbpleia2: not sure. Did it build images successfully? if so then I don't think it had the same impact20:15
lifelessfungi: https://review.openstack.org/#/c/69309/ would be particularly good to land20:15
pleia2so no, only fedora images are being built in the tripleo cloud20:17
pleia2err only precise20:17
pleia2but I thought https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#n212 would mean it would try to build precise too20:18
pleia2unless nodepool hasn't been restarted20:18
pleia2I keep saying the wrong thing :) "mean it would try to build FEDORA too"20:19
clarkbnodepool doesn't need to be restarted to get new config20:20
clarkband yes I think line 212 would mean images should be built20:20
clarkbI can do a nodepool list after lunch to see what is going on20:20
*** gokrokve has joined #openstack-infra20:20
pleia2I did a nova list on the tripleo cloud, only see precise20:20
pleia2lifeless: is there something I'm missing here?20:21
clarkbkrotscheck: so your tests will use selenium?20:21
krotscheckclarkb: Some of them, yes.20:22
fungirussellb: comments left on that one20:22
krotscheckclarkb: It's executed via grunt though, for easier orchestration.20:22
funginm. i see clarkb left the exact same comments basically20:22
*** mfer has quit IRC20:23
lifelesspleia2: context?20:23
clarkbkrotscheck: and dist/** will contain documentation?20:23
russellbfungi: clarkb thanks guys :)20:23
pleia2lifeless: so you made the changes last week to add fedora to nodepool, but the tripleo cloud doesn't seem to be building them20:23
krotscheckclarkb: dist/** contains the minified app. mordred thought it'd be a good idea to sanity-check upload it much like docs so we can proof it.20:23
pleia2(or even trying, it won't actually work fully yet)20:23
openstackgerritA change was merged to openstack-infra/reviewstats: Don't blow up on an empty core team list  https://review.openstack.org/6971120:23
clarkbkrotscheck: gotcha20:23
openstackgerritA change was merged to openstack-infra/reviewstats: Add murano with all subprojects  https://review.openstack.org/6968120:24
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Add a new node type for precise with saucy kernel  https://review.openstack.org/6944520:24
lifelesspleia2: it tries and fails20:24
pleia2lifeless: ok, do we have record somewhere of what is happening so I can tackle that?20:24
lifelesspleia2: nodepool log20:24
pleia2lifeless: right now I'm just running devstack scripts manually on an F20 image20:24
lifelesspleia2: you can run nodepool against a real cloud like cd-overcloud  :)20:25
pleia2lifeless: ok, so if I tried this myself with nodepool and the fedora config I'll see what errors it's getting?20:25
clarkblifeless: pleia2: my guess is it fails for the same reason20:25
lifelesspleia2: I know you know how to do that :)20:25
clarkbparamiko with no tty breaks sudo on fedora 2020:25
pleia2lifeless: right-o, thanks20:25
pleia2clarkb: yeah, probably so20:25
clarkbnodepool needs to have paramiko allocate a tty20:25
fungilifeless: clarkb: isn't 69272 going to be a problem because the only job invoked in the check pipeline is non-voting?20:25
*** beagles_brb is now known as beagles20:25
*** gokrokve has quit IRC20:25
clarkbis it insecure to run sudo without a tty? what is the logic behind that?20:25
*** devkulkarni has quit IRC20:25
fungialso, seems i can either read irc or review changes. wow there's a lot of conversation20:26
lifelessfungi: I guess we should keep noop ?20:26
clarkbfungi: lifeless: no I believe it works fine20:26
clarkbI think we inadverdently tested that and it worked20:26
fungiclarkb: okay. i'll look to see whether we have any precedent for it20:26
pleia2ok lunching now, will be back on nodepool after20:28
krotscheckclarkb: Thanks for the review. I'm guessing jeblair's comment on the previous patchset is not enough to get this merged?20:28
fungipleia2: if i get time, i can look through the image log and see where it's going bad20:29
*** mrodden1 is now known as mrodden20:29
*** dkliban_afk is now known as dkliban20:29
openstackgerritChmouel Boudjnah proposed a change to openstack-infra/config: Don't log openstack-gate anymore.  https://review.openstack.org/6971420:29
clarkbkrotscheck: typically we need two +2s. I have been breaking that rule for simple changes that are low impact. This adds a bunch of new stuff so having more eyes would be nice20:29
fungikrotscheck: i'm looking over 67729 now20:30
krotscheckfungi: Thanks20:30
openstackgerritlifeless proposed a change to openstack-infra/nodepool: Expose paramiko's get_pty parameter.  https://review.openstack.org/6971520:30
lifelesspleia2: clarkb: try this ^20:30
fungieek... i cringe when i see Xvfb mentioned in unattended test scripts. i will get over it though20:32
*** whoops has quit IRC20:32
clarkbfungi: :) it means we can run firefox on our machines and have tests interact with firefox. That isn't scary at all20:32
*** devkulkarni has joined #openstack-infra20:33
fungium. right. of course not20:33
fungithat's why i drink20:33
krotscheckfungi: Just don't look too hard at what's going on there, or you'll realize its... *gasp*... javascript!20:33
fungiyeah, thinking too hard is not one of my failings, thankfully20:33
*** e0ne has quit IRC20:34
*** dkliban has quit IRC20:34
*** malini_afk is now known as malini20:34
fungithis pattern has been tested on a headless system already, presumably20:35
fungiand without sudo privs20:35
krotscheckfungi: That's how I run my tests. Having a browser constantly pop up during the selenium runs is obnoxious, xvfb makes it not annoy me.20:36
*** NikitaKonovalov_ is now known as NikitaKonovalov20:37
*** lcestari has quit IRC20:37
*** gokrokve has joined #openstack-infra20:38
*** thuc has quit IRC20:40
*** whoops has joined #openstack-infra20:41
*** thuc has joined #openstack-infra20:41
fungikrotscheck: so the only thing which i think is probably unnecessary is that you have copy-on-failure for the draft build which results from the unit tests run, even though your failure message is going to be the standard pointer to the console log20:42
*** gokrokve has quit IRC20:43
fungiif you want to keep the drafts from failed unit test runs, then cool. just be aware you have to construct the urls yourself20:43
krotscheckfungi: We fix that two commits down the line: https://review.openstack.org/#/c/67520/3/modules/openstack_project/files/jenkins_job_builder/config/javascript.yaml20:43
fungi(i assume they will be tiny and few enough as to not really contribute to disk utilization anyway)20:43
fungioh, hah20:43
fungithis is what i get for reviewing changes one at a time ;)20:44
*** pcrews has quit IRC20:44
*** johnthetubaguy has quit IRC20:44
krotscheckfungi: Seriously. You should just load it all into your brain at once like clarkb.20:45
fungii'd blow a gasket20:45
*** thuc has quit IRC20:45
fungikrotscheck: were you super eager to address clarkb's one non-necessary comment request on that change (or is it also addressed in a later patchset)? if not, approving20:46
anteayafungi I'd imagine a disk space error20:46
fungianteaya: nope. i'm purely analog20:46
*** DinaBelova is now known as DinaBelova_20:47
anteayamore reliable20:47
fungialcohol combustion engine powered20:47
anteayadistilled in a chevy rad?20:47
fungiand stockpiled in mason jars, yes20:47
openstackgerritPhilippe Godin proposed a change to openstack-infra/jenkins-job-builder: Add best and worst status details for conditional-step plugin.  https://review.openstack.org/6960820:47
krotscheckfungi: Not really, though I can throw that in there if it means I the review won't take 3 weeks this time.20:47
fungikrotscheck: nah, you're all set20:48
pabelangerQuestion about the jenkins slaves, are you using SSH public keys for them to communication with jenkins masters? Versus over HTTP / HTTPS?20:48
krotscheckfungi: Awesome, thanks!20:49
fungipabelanger: we have the jenkins masters connect to the slaves via ssh20:49
openstackgerritA change was merged to openstack-infra/config: Use nodeenv via tox to do javascript testing  https://review.openstack.org/6772920:50
pabelangerfungi, Oh, so, the master initiates the connect, not the slaves?20:50
krotscheckOk, so now there's this one: https://review.openstack.org/#/c/67731/20:51
fungikrotscheck: yep. i always look at dependency chains when i review so i can keep my train of thought. already on it20:53
krotscheckfungi: Ok, I'll shut up.20:53
fungipabelanger: yes, master ssh's to each slave and launches the slave agent on demand20:54
*** ruhe is now known as _ruhe20:54
fungipabelanger: basically when jenkins is told to connect (or reconnect) to a slave it launches or restarts the slave agent there20:54
*** markmc has quit IRC20:54
*** e0ne has joined #openstack-infra20:55
fungipabelanger: however, be aware that jenkins is designed to allow slaves to request running arbitrary things on the master via that channel, so don't just assume it's providing suitable slave isolation that way20:55
*** UtahDave has joined #openstack-infra20:56
pabelangerfungi, interesting.  Any reason why we connect directly from the master, vs the slave connecting back?  I think there is a plugin for SSH support20:56
fungipabelanger: we isolate all our sensitive slaves on a separate master which doesn't run jobs containing arbitrary code to help mitigate that issue20:56
fungipabelanger: not sure if it was assumed to be more secure (at least we don't have to install sensitive private keys on all our slaves, so there's that benefit)20:57
fungipabelanger: but since we have public ip addresses on all slaves currently, it's not any more convenient to have slaves connect to the master (to get around nat for example)20:58
lyxusfungi, awesome ! Thanks !20:58
*** gyee has quit IRC20:58
*** hashar has quit IRC21:00
*** _ruhe is now known as ruhe21:00
fungikrotscheck: on 67731 it looks like it's still not removing the "copy-after-failure: true" from the dist publisher for that gate-{name}-js-unittests template. was there another change to address that, or do you see benefit to uploading docs-drafts artifacts for failed unit tests? (keeping in mind that you will have to construct the url for them on your own)21:02
*** markwash has quit IRC21:02
krotscheckfungi: Yeah, now that's one patch down the stream.21:03
krotscheckfungi: And no - there's no reason to upload the app on a failure.21:03
fungii guess the other patch got disassociated from 67731 in that case (it has no children in gerrit)21:04
*** talluri has quit IRC21:05
krotscheckfungi: It's outdated, I'm in merge hell at the moment. Give me 1021:06
*** hashar has joined #openstack-infra21:06
fungikrotscheck: no rush. i'm in review hell, so i understand ;)21:06
fungii think those spheres of hell are mostly adjacent21:07
* krotscheck says screwit and abandons the prvious patch21:07
krotscheckNuke from orbit, start over21:07
openstackgerritA change was merged to openstack-infra/config: Run gate-triple-deploy on tripleo-ci changes.  https://review.openstack.org/6927221:08
*** UtahDave has quit IRC21:10
openstackgerritA change was merged to openstack-infra/config: Run gate-tripleo-deploy on devstack-gate changes.  https://review.openstack.org/6927321:10
openstackgerritA change was merged to openstack-infra/config: Use the openstack pypi mirror for tripleo images.  https://review.openstack.org/6930921:11
*** yassine has joined #openstack-infra21:12
clarkbkrotscheck: :/ nuking from orbit removes old review context21:12
fungikrotscheck: stick with us for a few months and you'll be a bona fide git rebase -i ninja21:13
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Enable tempest/savanna gate tests  https://review.openstack.org/6806621:13
krotscheckclarkb: In this case, nobody's reviewed it.21:14
clarkbkrotscheck: so about that21:14
krotscheckclarkb: Wait - I lied.21:14
clarkbwhen you have really long patch series (again I know mostly mordred's fault) you are not likely to get reviews on the 7th change when the first has -1s21:15
clarkbI think that is partially why some of these have languished21:15
krotscheckclarkb: In contrast, if a review languishes I'm going to continue tweaking and iterating on it until it makes someone happy.21:16
anteayakrotscheck: if you inclined, if you spend some time doing reviews, that helps the reviewers21:17
krotscheckanteaya: I do. Mostly on storyboard though.21:17
clarkbkrotscheck: thats fair just pointing out that it helps to do two things. not have long patch series if possible (logically independent changes are great) and address comments in earlier changes21:17
krotscheckanteaya: I'll expand my scope.21:17
anteayasays the person whose review statts are currently dismal21:17
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Updated tarball/client dist upload for javascript  https://review.openstack.org/6972221:17
anteayakrotscheck: thanks21:18
*** AlexF_ has joined #openstack-infra21:18
fungikrotscheck: you'll get particularly adept at using rebase's edit feature to change patches in the middle of topic branches21:18
fungiit's actually quite slick once you get used to it21:19
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Updated tarball/client dist upload for javascript  https://review.openstack.org/6972221:19
krotscheckAlright, fungi: That patch addresses the copy-on-failure thing as well as clark's request for additional comments.21:20
*** e0ne has quit IRC21:20
*** ArxCruz has joined #openstack-infra21:21
devkulkarniclarkb, anteaya: fyi, I have been able to run zuul's tests successfully. on a 8 core 8 GB machine (12.10 ubuntu) it took about two and half minutes. had to install python-dev though before tox could run21:21
fungibut i agree with clarkb... unless the changes are complex enough to be separate commits yet interdependent enough to require one another, stacking changes is somewhat counterproductive to the review process (with one obvious exception being changes which are mostly baked but are likely to merge-conflict with one another)21:21
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Enable tempest/savanna gate tests  https://review.openstack.org/6806621:21
anteayadevkulkarni: congratulations21:21
anteayayes installing python-dev makes sense21:21
krotscheckfungi: I'm inclined to agree as well. I just happened to inherit some of mordred's patches.21:21
fungikrotscheck: you are experiencing it first hand without causing it. lucky you21:22
krotscheckEhn, he's just hazing the new guy21:22
krotscheckIn the meantime I've gotten javascript into infra without making anyone's head explode (too much). I call that a win.21:23
*** AlexF_ has quit IRC21:23
*** NikitaKonovalov is now known as NikitaKonovalov_21:24
clarkblifeless: quick question, for that nodepool non blocking delete change. is that something you had planned on retesting?21:24
clarkblifeless: also, thanks for the pty change. At some point I should get a nodepool setup against my hpcloud account to test stuff21:25
*** dprince has quit IRC21:25
*** hashar has quit IRC21:25
lifelessclarkb: its' pretty easy, see the nodepool README these days :)21:25
lifelessclarkb: no, I don't have time to retest nodepool atm21:26
clarkblifeless: ok, I will try spinning up a testbed for it myself then thanks21:26
pleia2lifeless: testing nodepool now, but I'm not getting very far because I immediately get piles of: OverLimit: Quota exceeded, too many key pairs. (HTTP 413) (Request-ID: req-2f64bc4b-84ab-4a7a-8bb4-848b81eba00f)21:27
anteayashopping for upvotes to remove logging from openstack-gate: https://review.openstack.org/#/c/69714/21:27
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Update colors, header  https://review.openstack.org/6972321:27
lifelesspleia2: are you using your account?21:28
*** dolphm_afk is now known as dolphm21:28
lifelesspleia2: I suggest testing wit yyour tripleo-cd credentials21:28
*** rcleere has joined #openstack-infra21:28
fungipleia2: also, you can get novacient to list/delete keypairs on that provider21:28
fungipleia2: we've found they have a tendency to accumulate and people don't realize there are quotas on them21:29
pleia2lifeless: was using the infra one21:29
pleia2fungi: ah :)21:29
*** devkulkarni has left #openstack-infra21:30
jog0sdague: where is the new home of the gate job failure gprahite logs?21:30
clarkbpleia2: any chance you want to test lifeless' nodepool stack of changes after testing the get_pty change21:30
fungipleia2: aborted server launches can often leave orphaned keypairs behind, and the pile up over time21:30
pleia2clarkb: will do21:30
fungithey pile up21:30
anteayakrotscheck: thank you21:31
jog0because I used those every day21:31
lifelessyah we should make just one keypair21:31
lifelessand use it21:31
lifelessrather than per-server. IMNSHO.21:32
lifelessmaybe one per image or something.21:32
krotscheckanteaya: Seemed simple enough21:32
SergeyLukjanovclarkb, fungi, sdague, I've made an async gate jobs for savanna, could you, please, take a look on config update? https://review.openstack.org/#/c/6806621:32
anteayakrotscheck: yes, some are21:32
sdaguejog0: it currently is homeless21:33
*** cody-somerville has quit IRC21:33
anteayafor the simple ones, it then allows one core to drive by and approve if there are a bunch of +1s21:33
jog0sdague: :( lets fix that21:34
pleia2fungi: thanks, that sorted it21:34
jog0can we put it back on status.o.o/e-r at the bottom for now21:34
jog0until the new home is built21:34
*** coolsvap has quit IRC21:35
*** JpMaxMan has quit IRC21:36
sdaguejog0: so you realize it's completely flawed right21:36
jog0sdague: compleatly flawed? how so?21:36
sdaguebecause of the interupt exception21:36
sdagueinterupt == failed21:36
sdaguewhich it isn't21:36
sdagueso all the failed numbers are impacted by gate depth21:36
sdaguewhich is why they were corolated so strongly before21:37
jog0bug Bug 1270309 - jenkins java.lang.InterruptedException21:37
sdagueand why the fail numbers seemed higher21:37
jog0which has had 43 hits in last 24 hours21:37
jog0sdague: that makes sense21:37
jog0in that case we should get that bug fixed21:37
jog0clarkb: ^21:38
jog0what is involved in fixing that bug21:38
jog0sdague: I forgot about that bug :/21:38
fungijog0: feel like contributing upstream development on jenkins?21:38
jog0fungi: thats where it is done?21:39
fungiit's full of javalicious fun times21:39
sdaguejog0: https://review.openstack.org/#/c/65700/ - so jeblair rightly kind of objects to that being a dedicated page. Honestly, I'm kind of out of juice on that one.21:39
sdagueI don't think it should be on the e-r page, because I think it's mostly a distraction21:39
fungijog0: yeah, the underlying problem is that when jenkins receives an abort/cancel sometimes it shuts things down in such an order that it reports a failure for the running job21:39
fungijog0: we've just assumed we'd live with it until we no longer use jenkins, i think21:40
jog0fungi: yeah that sounds reasonable sadly21:40
anteayajog0: I hear you though, I do like graphs21:41
jog0anteaya sdague: I just want to know if there was a spike in gate failures or not21:41
anteayabut it helps if they reflect the data accurately21:41
jog0if anyone has an idea how to do that ...21:41
anteayado the individual graphs give you enough of that?21:42
jog0not really21:42
*** zehicle has joined #openstack-infra21:42
anteayaI scroll the top 5 graphs21:42
*** eharney has quit IRC21:42
jog0well we do know the classification rate is still very high21:42
sdaguejog0: use the ES data, then you can filter out the Interupt fails21:42
jog0sdague: that doesn't deal with duplicates21:42
sdagueit does if you code it21:42
jog0oh you mean: total fail - bug 127030921:43
sdagueI mean get all the build_uuids that fail on that21:43
fungiyeah, i think from a data mining perspective, finding a way to filter out failures resulting from a java interrupt exception is going to get you the cleanest numbers21:43
sdagueand process all the rest of the fails21:43
sdagueand subtract those21:43
fungicount them as equivalent to abort/cancel (which i assume you currently ignore)21:44
jog0sdague: fungi: yeah21:44
sdagueyeh, we're looking for that Finished: line21:44
sdagueso those wouldn't have it21:44
clarkbfungi: yup properly aborted jobs are ignored21:45
*** zehicle_at_dell has quit IRC21:45
* jog0 is looking into updating elastic-recheck-graph21:48
*** dkliban has joined #openstack-infra21:49
sdagueit's going to get ugly21:50
*** zehicle has quit IRC21:51
sdaguejog0: honestly, I think a more useful thing to do would be to make a version of the graphs page which is *only* gate resets21:51
jog0sdague: what does that tell us?21:51
sdaguewe want to know what to fix21:51
sdaguewhat is killing us21:51
jog0how does knowing just the number of gate resets help with that?21:52
clarkbAaronGr: left a comment on that change, think you should update the change to have cron or no cron21:52
sdaguejog0: it would be the bug list with only the gate resets21:52
sdaguethat's what's actually killing us21:52
sdagueright now those numbers are all queues21:52
AaronGrclarkb: no problem. i'll re-submit with that change.21:53
*** dizquierdo has joined #openstack-infra21:54
jog0sdague: you mean have an option to only list gate queue failures on status.o.o/e-r21:54
*** packet has joined #openstack-infra21:57
clarkbfungi: any changes you have seen that you +2'd that could use a second +2?21:58
*** rcleere has quit IRC21:58
clarkbfungi: AaronGr: https://review.openstack.org/#/c/68771/ just adds the ability to diable a user but doesn't actually disable them correct?21:58
sdaguejog0: yeh21:58
clarkbfungi: AaronGr: eg I don't have to worry too much about approving that21:58
fungiclarkb: not sure. i need to take a break to cook dinner here in a bit... https://review.openstack.org/#/q/watchedby:fungi@yuggoth.org+-owner:fungi@yuggoth.org+status:open+-status:workinprogress+-status:draft+-Approved%3E=1+CodeReview=2+-CodeReview%3C=-1+-Verified-1,n,z21:58
fungioh, i guess the watchedby is probably wrong for you21:59
AaronGrfungi: correct. just adds a function to let you disable users, non-impacting.21:59
*** zz_ewindisch is now known as ewindisch21:59
clarkbAaronGr: ok I will approve21:59
*** SergeyLukjanov is now known as SergeyLukjanov_21:59
fungiclarkb: yeah, would have been nice to have a few weeks ago ;)22:00
openstackgerritSteve Baker proposed a change to openstack-infra/config: Open port 8000 for heat-api-cfn for public neutron net  https://review.openstack.org/6927622:00
stevebakerclarkb: let me know what you think of the rule in ^22:01
*** Sukhdev has quit IRC22:01
openstackgerritA change was merged to openstack-infra/config: Extend user module, add 'disable user'  https://review.openstack.org/6877122:02
clarkbstevebaker: looks great22:02
clarkbstevebaker: I can deal with rebasing with the other change depending on how they merge22:03
stevebakerclarkb: ok. what kind of propagation delay happens after it lands?22:03
clarkbstevebaker: good question and it isn't great. Will need an image rebuild cycle which typically happens once a day but can be babysitted by hand if we want to go quicker22:04
jog0sdague: POC coming shortly22:04
clarkbstevebaker: fungi, stevebaker's change above is anotehr good one to review if you have time :)22:04
stevebakerhmm, the checks failed22:05
fungiclarkb: yeah, saw it. i'm good approving it if you want to rebase 6944222:05
fungionce it's clean22:05
clarkbfungi: fine with me22:06
clarkbsilly rocketships22:06
clarkbstevebaker: ^22:06
openstackgerritSteve Baker proposed a change to openstack-infra/config: Open port 8000 for heat-api-cfn for public neutron net  https://review.openstack.org/6927622:06
openstackgerritAaron Greengrass proposed a change to openstack-infra/config: Remove hardcoded config assumptions, cleanup variables  https://review.openstack.org/6607222:07
stevebakerthats better, the check passes22:09
* fungi steps away for a bit to make with the food22:10
*** harlowja is now known as harlowja_away22:10
*** sarob has joined #openstack-infra22:11
clarkbby the way logstash kibana users. searching all time is bad juju22:11
fungidon't anger the tiki gods22:11
*** eharney has joined #openstack-infra22:12
clarkbanteaya: can you abandon https://review.openstack.org/#/c/69458/2 if it isn't needed22:12
krotscheckfungi: So, keep-hierarchy: false?22:12
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Make files in reviewday docroot accessible  https://review.openstack.org/6973422:13
*** amotoki has quit IRC22:14
fungikrotscheck: oh, good point, that may be the default, in which case you're fine22:14
*** gyee has joined #openstack-infra22:14
openstackgerritA change was merged to openstack-infra/config: Open port 8000 for heat-api-cfn for public neutron net  https://review.openstack.org/6927622:14
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Make files in reviewday docroot accessible  https://review.openstack.org/6973422:14
fungiclarkb: do you know what it defaults to in the scp publisher?22:15
clarkbhttp://ci.openstack.org/jenkins-job-builder/publishers.html#publishers.scp says false22:15
krotscheckfungi: It is the default, but it helps to be explicit. Will update.22:16
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Updated tarball/client dist upload for javascript  https://review.openstack.org/6972222:16
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Only plot gate failures in status.o.o/elastic-search  https://review.openstack.org/6973522:16
jog0sdague: ^ that just switches to gate failures only on status.o.o/e-r22:17
fungiclarkb: thanks. i should rtfm22:17
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Only plot gate failures in status.o.o/elastic-recheck/  https://review.openstack.org/6973522:17
wenlock_hey, im hoping a review can be done on this one : https://review.openstack.org/#/c/51425/   i have some time for updates if needed22:18
sdaguejog0: so I don't think we want to switch22:18
*** harlowja_away is now known as harlowja22:18
jog0sdague: I tend to agree22:18
sdaguewe need both, because gate volume is currently low22:18
jog0we want to show both22:18
jog0sdague: so to do so requires some UI magic that I don't have22:18
sdagueso now would be the time for subpages22:18
jog0I think22:18
sdaguejog0: just make them different pages with our own nav embedded22:18
sdaguethe uncat page could come in as well22:19
jog0sdague: you want to take a first pass at the own nav?22:19
anteayaclarkb: as of yesterday I felt it was needed, I haven't had time to go back and do my reading to bring it up with jog0 again22:19
jog0my web UI is horrible22:19
*** jcoufal has joined #openstack-infra22:19
jog0web UI foo22:19
*** thuc has joined #openstack-infra22:19
sdagueI'm actually kind of buried at the moment22:19
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added no_api env  https://review.openstack.org/6861022:19
jog0sdague: heh kk aren't we all22:20
clarkbanteaya: I don't think it is needed based on jog0's comment22:20
* anteaya tries to find the url for the unclassified failures again22:20
*** thuc has quit IRC22:20
stevebakerclarkb: OK, the change has landed. Any chance you could ping me when the rebuild cycle is done?22:20
anteayaclarkb: I disagree with jog022:20
* jog0 is right here22:20
clarkbanteaya: whyfor?22:20
jog0anteaya: ?22:20
anteayaand then had to get onto a plane before he and I could discuss it22:20
*** thuc has joined #openstack-infra22:20
clarkbstevebaker: sure, I will try to kick those off when I have a chance to sit with a terminal to nodepool open22:20
anteayabecause the fingerprint was not picking up unclassified failures with the same message22:21
anteayaas of yesterday22:21
anteayajog0: do you have the unclassified url again?22:21
clarkbanteaya: if you look at the results in the logstash query you link they come in pairs as jog0 describes22:21
anteayait would save me time trying to find it22:21
jog0use status.openstack.org/elastic-recheck/ to find it22:21
jog0ohh you men http://status.openstack.org/elastic-recheck/data/uncategorized.html22:22
anteayaclarkb: but there were failures that were unclassified with the exact same message as of yesterday22:22
*** SumitNaiksatam_ has joined #openstack-infra22:22
anteayajog0: yes thanks22:22
*** SumitNaiksatam has quit IRC22:22
*** SumitNaiksatam_ is now known as SumitNaiksatam22:22
clarkbanteaya: do you have an example of that?22:22
clarkbwant to make sure there isn't a larger underlying problem22:22
anteayaclarkb: I am looking22:23
jog0sdague: so when running the gate only patch and ignoring CI22:24
jog0sdague: Bug 1273386 is the top with 3 fails in 24hrs22:24
jog0top open bug22:24
jog0so woot22:24
jog0sdague: so if we can get someone to create the nav bar I can do the rest22:26
StevenKHuh, someone fixed git-review22:28
*** jcoufal has quit IRC22:28
clarkbkrotscheck: your comment on https://review.openstack.org/#/c/69204/3/modules/openstack_project/files/jenkins_job_builder/config/storyboard.yaml can you help me understand that? isn't the storyboard docs job for building sphinx docs and not related to javascript?22:28
anteayaclarkb: jog0 http://logs.openstack.org/71/65371/2/gate/gate-tempest-dsvm-neutron/27ab52f/console.html22:28
anteayafirst failure under gate-tempest-dsvm-neutron : 6 Uncategorized Fails. 97.6% Classification Rate (246 Total Fails)22:29
*** AlexF_ has joined #openstack-infra22:29
anteayathat looks like the same message to me22:29
jog0krotscheck: ping, have some UI work22:29
anteayabut it is still on the unclassified page22:29
*** zehicle_at_dell has joined #openstack-infra22:30
krotscheckclarkb: In the tox patches we moved all the storyboard jobs from storyboard.yaml into templates in javascript.yaml. I was thinking that would also be something we want to do for docs, but then I realized that ruhe's talking about the API, which is a python thing.22:31
krotscheckjog0: What's up?22:31
jog0anteaya: looking22:31
*** e0ne has joined #openstack-infra22:31
anteayajog0: thanks22:31
jog0its in n-api at info level. clarkb do we collect that?22:32
anteayaI am prepared to be wrong, but if so, I need direction about how to write the query to classify that failure22:32
clarkbjog0: we should I am digging into it22:32
jog0clarkb: thanks22:33
krotscheckclarkb: I just had a build fail looking for the nodeenv package - I'm guessing this is because it's not on our pypi mirror? If so, is that in openstack/requirements?22:33
jog0anteaya: ^ thats your answer22:33
jog0krotscheck: so we have this page http://status.openstack.org/elastic-recheck/22:33
*** AlexF_ has quit IRC22:33
anteayajog0: logstash doesn't collect n-api at the info level?22:33
anteayaso what is the next step to get that failure classified?22:34
jog0right now it queries all queues (check, gate, etc) - but we want an option to only list hits on the gate queue22:34
jog0krotscheck: I have a patch to do the elasticSearch changes https://review.openstack.org/#/c/69735/22:34
jog0krotscheck: but not sure what best way to update UI22:34
ruhekrotscheck: clarkb: i'll add a comment that we agreed in irc to keep storyboard-doc template in storyboard.yaml22:34
anteayae-r queries all queues? (trying to ensure I have the correct vision of what "it" is referencing)22:34
jog0either subpages with a nav bar, or a button or something22:34
jog0anteaya: see clarkb's comment22:35
clarkbbuild_uuid:"27ab52ffbfec4f32a1ab23ef0361360a" AND filename:"logs/screen-n-api.txt" it didn't get the file22:35
jog0clarkb: ahh22:35
jog0anteaya: ^ we hit a different bug22:35
krotscheckjog0: Ok, gimme a few moments to decipher what you mean by all that :)22:35
clarkbso for one reason or another it didn't index that file (logstash doesn't try to hard to work around derpage because there is so much going o22:35
clarkbjog0: anteaya: so I don't think we should update the query22:36
anteayajog0: so this log is referencing a different bug with a similar message22:36
jog0krotscheck: http://status.openstack.org/elastic-recheck/  has a bunch of graphs my POC https://review.openstack.org/#/c/69735/ changes the data that is graphed22:36
clarkbit won't fix the problem22:36
*** e0ne has quit IRC22:36
anteayaokay I can abandon this patch and offer a new patch with a brand new query22:36
jog0sdague: ^ perhaps you have a better vision of what the UI should be22:36
clarkbanteaya: no, it is the same bug from e-r perspective. But the reason it didn't match was we didn't index the file22:36
*** mriedem has quit IRC22:36
clarkbanteaya: a new query won't help22:36
anteayaclarkb: ah okay, standing by22:36
clarkbanteaya: we need to make logstash fail less22:36
anteayaclarkb: ah22:36
*** dkranz has quit IRC22:36
anteayaclarkb: so yes to abandon the patch?22:37
jog0anteaya: yeah22:37
clarkbanteaya: the problem here is we are indexing a ton of data. If some of it falls on the floor we say oh well and move on otherwise we would back up and bad things happen22:37
krotscheckjog0: I'm guessing you're looking for an incremental change?22:38
clarkbkrotscheck: can you link to the failure? and no you shouldn't be using the mirror22:38
clarkbat least not only the mirror22:38
jog0krotscheck: yes22:38
krotscheckclarkb: http://logs.openstack.org/10/68610/4/check/gate-storyboard-webclient-js-unittests/f9709ce/console.html22:38
anteayaclarkb: right22:38
krotscheckjog0: Tall order, let me sketch a few things out - 10 mins or so.22:39
jog0krotscheck: doing sub pages with a nav bar may be the easiest22:39
jog0where we can include status.openstack.org/elastic-recheck/data/uncategorized.html as well22:40
jog0krotscheck: thanks!22:40
krotscheckjog0: That's where I'm leaning.22:40
jog0krotscheck: awesome22:40
clarkbkrotscheck: so you want to know something funny22:40
clarkbkrotscheck: you will like this22:40
*** CaptTofu has quit IRC22:41
krotscheckclarkb: .....22:41
krotscheckclarkb: ..... that sounds like a trap.22:41
clarkbkrotscheck: I think what happened htere is you used the mirror because run-jsbuild.sh does not run the select mirror function which means you are stuck with whatever the previous jobs mirror selection was22:41
clarkbkrotscheck: to fix that add http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/run-tox.sh#n32 to run-jsbuild.sh and I can push it through22:41
*** CaptTofu has joined #openstack-infra22:41
krotscheckclarkb: Ok, let me help out jog0 first.22:42
*** stevemar has joined #openstack-infra22:44
*** jnoller_ has quit IRC22:45
*** CaptTofu has quit IRC22:46
fungihooray for cross-project pollination/cooperation! as harry tuttle said, "we're all in it together"22:48
ruheclarkb: one question about https://review.openstack.org/#/c/69204 . it expects "ci/storyboard" to exist on i believe this dir should be created manually?22:49
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Run the tests for the python-openstacksdk project  https://review.openstack.org/6966722:49
stevemarlookin for some help debugging an error i'm seeing in my jenkins results, any takers?22:50
*** yamahata has joined #openstack-infra22:50
clarkbruhe: the scp plugin will create it if necessary22:51
ruheclarkb: that's handy22:51
fungiruhe: it's handy until you want part of the path to that to be a symlink22:52
*** mattoliverau has quit IRC22:52
lifelessfungi: can I get a +A on https://review.openstack.org/#/c/69309/ ?22:52
fungiat which point it gets removed and recreated22:52
clarkbdhellmann: what is the project that clients use that isn't prettytable but does output of stuff, cliff?22:52
dhellmanncliff uses prettytable22:53
dhellmannit just wraps it differently so clients aren't using it directly22:53
fungilifeless: did you mean something else? that's approved/merged already22:53
clarkbdhellmann: thanks22:53
*** dizquierdo has quit IRC22:53
anteaya6 +1s and a +2 from fungi for removing logging from openstack-gate: https://review.openstack.org/#/c/69714/22:53
lifelessfungi: hah, gocool22:54
lifelessstale browser page22:54
anteayaready for a +A anytime22:54
lifelessfungi: thanks!22:54
fungianteaya: done. smells like consensus from the denizens to me22:55
clarkbhopefully a meeting doesn't start in 5 minutes :)22:55
anteayaha ha ha22:55
*** thuc has quit IRC22:55
*** prad has quit IRC22:55
fungieep, right22:55
*** thuc has joined #openstack-infra22:56
openstackgerritA change was merged to openstack-infra/config: Don't log openstack-gate anymore.  https://review.openstack.org/6971422:57
fungii'm starting to think we should get rid of the notify/subscribe on that config and just schedule a weekly reload midnight utc saturday night/sunday morning until someone gives it a graceful option22:57
clarkbfungi: or just manually kick it like manage-projects :/ I don't actually like that idea23:00
*** thuc has quit IRC23:00
krotscheckjog0: http://bootply.com/109172 as a "First blush"23:01
*** melwitt1 has joined #openstack-infra23:01
*** mattoliverau has joined #openstack-infra23:01
clarkbcyeoh: comment on https://review.openstack.org/#/c/69542/23:02
krotscheckjog0: Does that show for you?23:02
* krotscheck will have to tweak the typography a bit of course...23:02
clarkbI see a cat in the render. I assume it worked :)23:03
*** melwitt has quit IRC23:03
cyeohclarkb: thx - will fix it up - I think it should be possible to find this in n-cpu.txt...23:03
krotscheckclarkb: Yes, well, that's placekitten.com :)23:03
jog0krotscheck: awesome23:03
*** pcrews has joined #openstack-infra23:03
mattoliverauMorning everyone.23:03
*** openstack has joined #openstack-infra23:04
-wolfe.freenode.net- [freenode-info] channel flooding and no channel staff around to help? Please check with freenode support: http://freenode.net/faq.shtml#gettinghelp23:04
jog0krotscheck: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/web/share/index.html23:04
jog0sdague: ^^23:04
krotscheckanteaya: Thanks!23:04
mattoliverauanteaya: how are you going this fine day!23:04
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better  https://review.openstack.org/5142523:04
*** obondarev has quit IRC23:04
anteayamattoliverau: I have been absoring salt info, and am full atm23:04
clarkbmattoliverau: is jhesketh about?23:04
anteayasitting in a tutorial at salfconf23:04
krotscheckjog0: Is that a "Hey do this for me" or a "can you review this?" :)23:04
anteayamattoliverau: and thou?23:04
anteayamattoliverau: how go the boxes?23:05
jog0krotscheck: this is a, hey I don't know how to do this halp23:05
clarkbmattoliverau: I was hoping he would have an opinion on whether or not his zuul patches should come with tests23:05
krotscheckjog0: Righto23:05
jog0krotscheck: if you do the layout I can do the content23:05
*** obondarev has joined #openstack-infra23:05
mattoliverauclarkb: I'm in Sydney, jhesketh is currently on a plane to join us.. so he should be online at some point, but not for a few hours.23:06
jog0sdague: we need more kittens http://bootply.com/10917223:06
krotscheckjog0: Seems fair. I've got a few storyboard patches I need to get through  first.23:06
jog0krotscheck: no problem, thanks a ton23:06
clarkbmattoliverau: gotcha thnaks. no rush23:06
mattoliverauanteaya: good, in Sydney for the next few days.. so I have escaped the boxes for a while :)23:07
anteayamattoliverau: I'm sure your wife loves that you have left her with the boxes23:07
anteayahappy sydney23:07
anteayaany cooler there?23:07
sdaguekrotscheck: yeh, right now it's basically inheriting style from status.o.o.... so you'll probably want to cross check with jeblair on outer frame changes23:08
mattoliverauanteaya: yeah, I bet she does :P Yeah, only 28 today, nice and cool!23:08
anteayaavoid the 3023:08
clarkbkrotscheck: sdague: oh right, it tries to be consistent23:08
anteayamattoliverau: be sure to take a gift home, but don't pack it in a box23:08
mattoliverauanteaya: lol! great idea.23:09
clarkbpleia2: any luck with get_pty testing?23:09
krotschecksdague: I'll probably custom write the CSS for the tabs to get jog0 on his way.23:09
fungiclarkb: the status page style is eventually consistent23:10
krotschecksdague: I usually just use bootply as a quick sandbox.23:10
sdaguekrotscheck: that would be totally cool23:10
pleia2clarkb: I'm pretty sure it worked since I'm getting a new error now, but it's been tricky to debug23:10
*** ruhe is now known as _ruhe23:10
clarkbpleia2: new errors are the best23:11
* fungi will take a moment to look at the current fedora issue as it stands in production, and make sure that's what's wrong23:11
openstackgerritEdward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better  https://review.openstack.org/5142523:11
*** mrda_away is now known as mrda23:11
openstackgerritJulien Danjou proposed a change to openstack-infra/pypi-mirror: Whitelist posix-ipc  https://review.openstack.org/6974323:12
pleia2fungi: will need to do some key cleanup again, they keep piling up and not helping the situation :)23:12
* clarkb is sad that the number of changes merged today is far less than yesterday, but ESOMANYREVIEWS23:12
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Added select_mirror to javascript build.  https://review.openstack.org/6974423:13
*** oubiwann_ has quit IRC23:13
fungiclarkb: yesterday were the low-hanging fruit. now we're down to mostly dense change material23:13
krotscheckclarkb: ^^23:13
fungialso, i was way too busy with administrivia today. hopefully tomorrow i can pull my weight on that23:14
anteayais jeblair still sick?23:15
fungikrotscheck: lgtm23:15
anteayaI know mordred is in a car or on a plane or something23:15
krotscheckI think jeblair's on a plane to FOSDEM23:15
anteayagood, I am glad he is feeling better then23:15
fungianteaya: he's somwhere over the mariana trench23:16
anteayahopefully high in the air then23:16
fungino, wait, the atlantic rift23:16
fungiforgot he was going through dc23:16
krotscheckjog0: Ok, how do you want these changes. layered onto your own patch?23:16
clarkbkrotscheck: question about https://review.openstack.org/#/c/69722/3/modules/openstack_project/files/jenkins_job_builder/config/javascript.yaml why are we copying dist/** to logs/LOG_PATH?23:16
openstackgerritA change was merged to openstack-infra/config: Added select_mirror to javascript build.  https://review.openstack.org/6974423:16
anteayasome water filled abyss in any case23:17
clarkbkrotscheck: isn't all of that content available as a tarball on the tarballs site?23:17
jog0krotscheck: sure23:17
krotscheckclarkb: Mostly as a sanity check to catch situations like "Hey, this tarball upload didn't work, did we build the right thing?"23:19
clarkbkrotscheck: but in either case you will have a tarball and a list of files that match each other right?23:19
clarkbI guess I don't see the value in copying the data twice23:20
fungii concur with clarkb, we're not in the business of testing whether 'tar -czf' works as advertised23:20
krotscheckclarkb: Fair point. I dunno, we're currently dealing with a situation where the post gate tarball was never uploaded.23:20
*** dcramer_ has quit IRC23:21
clarkbkrotscheck: that won't be fixed or debugged by that change.23:21
clarkbat least not by that part of it23:21
fungikrotscheck: that should result in an error in the console og23:21
clarkbI updated my review23:22
fungikrotscheck: where/when did it fail?23:22
fungii'm happy to help troubleshoot23:22
krotscheckfungi: Actually, it never ran.23:23
*** alexpilotti has joined #openstack-infra23:23
fungiahh, that's an important detail... under what circumstances were you expecting it to run?23:23
krotscheckfungi: And that was a couple of patches ago, so we wanted to land the build updates first before diving in on why the trigger never executed.23:23
fungiin the post pipeline?23:23
krotscheckfungi: Yup23:23
krotscheckfungi: Let me amend that. It ran once. Then never again.23:23
fungii should be able to hunt the log from the last change which merged to that repo23:24
*** dims has quit IRC23:24
krotscheckafk- making changes in accordance to clark's comments23:25
*** julim has quit IRC23:27
fungialso, public service announcement, most everyone in this channel probably has a summit invite as of the past few hours. if you're coming, please sign up asap. this year there's a cutoff where if you don't sign up in the first few weeks you're technically supposed to come up with the difference between the early-bird pricing and normal pricing (so please don't procrastinate)23:27
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Updated tarball/client dist upload for javascript  https://review.openstack.org/6972223:27
clarkbfungi: we ready to approve https://review.openstack.org/#/c/69445/6 ?23:28
clarkbfungi: I think we have until march before that becomes a problem23:28
clarkbbut yes, sign up23:28
* clarkb does that23:28
fungiclarkb: right, march is just around the corner ;)23:29
fungikrotscheck: http://logs.openstack.org/9e/9e4411efab99438c21d6fd02cb0cae32bb3f7579/post/storyboard-webclient-release-branch/4c48dae/23:29
clarkbfungi: I need to step out in a bit for about half an hour. If I +2 69445 would you be able to babysit?23:30
*** thuc has joined #openstack-infra23:30
fungiclarkb: yes, for sure23:30
*** thuc has quit IRC23:30
fungii'll be here all night (until i pass out anyway)23:30
clarkbfungi: :P approve at will23:31
*** thuc has joined #openstack-infra23:31
fungiclarkb: do you need/would you want me to rebuild nodepool devstack-precise images, or should we just wait for the timer?23:31
clarkbfungi: oh I suppose if you are in there go ahead23:32
*** thuc has quit IRC23:32
fungii can do that23:32
*** thuc has joined #openstack-infra23:32
clarkbI was intending on doing it when I got around to testing zmq plugin on jenkins-dev23:32
clarkbgreat thanks23:32
*** denis_makogon has quit IRC23:32
*** melwitt1 is now known as melwitt23:32
fungihpcloud-az2 does a poor job of successfully building images. the most recent devstack-precise image there is almost 4 days old there now23:34
fungithe last time i looked at the logs for it, we were hitting ssh timeouts after launch before puppeting23:35
*** dims has joined #openstack-infra23:39
*** stevemar has quit IRC23:40
openstackgerritA change was merged to openstack-infra/config: Add a new node type for precise with saucy kernel  https://review.openstack.org/6944523:44
openstackgerritA change was merged to openstack-infra/config: Genericize javascript release artifact creation  https://review.openstack.org/6773123:44
*** oubiwann_ has joined #openstack-infra23:45
anteayaso now that russellb's patch is merged, can we run a check experimental job on that node type?23:46
*** paul-- has joined #openstack-infra23:46
*** pcrews has quit IRC23:46
anteayaits all there in the commit message23:47
* anteaya goes off to check experiemental on some neutron jobs23:47
anteayaor should I wait for this patch to propegate?23:47
openstackgerritJoe Gordon proposed a change to openstack-infra/devstack-gate: Add support to run nova-api-metadata as separate binary  https://review.openstack.org/6830423:48
russellbanteaya: before the jobs can run nodepool has to build the image for this node type ... i'm not sure how often it's configured to rebuild them23:48
*** sarob has quit IRC23:48
*** sarob has joined #openstack-infra23:49
anteayaevery 24 hours, not sure of the utc time23:49
anteayaunless it is kicked23:49
anteayaI will wait until tomorrow then23:49
anteayabut thanks russellb, very happy to have this change in23:49
russellbnot sure if it'll help or not, but it's worth trying to see what happens23:51
russellband i always have fun learning more infra stuff :)23:51
anteayayes, regardless of the outcome I am keen to see the logs23:52
anteayaand yes, it is fun to learn more infra23:52
*** sarob has quit IRC23:53
clarkbanteaya: russellb: fungi indicated he would kick it23:53
clarkbso possibly in another hour or so we can trigger those experimental jobs23:54
clarkbI need to test dims zmq plugin change now23:54
anteayaclarkb: ah cool, yes I saw the babysitting/kick conversation but didnt connect the dots23:54
*** whoops has quit IRC23:54
anteayaI lose23:54
clarkbthat moment when you 'G' and expect firefox to go to end of page. I should reinstall vimperator23:55
fungiclarkb: anteaya: russellb: nodepool builds images on demand, so it should already be trying to build them as long as it thinks it needs nodes of tat type23:56
russellboh, cool.23:56
anteayaI'll trigger some experimental jobs23:56
clarkbI am going to restart jenkins-dev nowish, you have been warned :)23:56
anteayamight as well see what happens23:56
russellbanteaya: great23:56
fungiclarkb: anteaya: russellb: and that's the problem--forgot to add nodes to a provider using that image :(23:57
fungipatch coming23:57
anteayaonly hit check experimental on one so far23:59
anteayaif a patch is in check, commenting "check experimental" will not interfere with check tests executing properly, correct?23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!