Tuesday, 2015-05-05

*** erlon has quit IRC00:01
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Allow Magnum to conform to global requirements process  https://review.openstack.org/17083000:02
openstackgerritMerged openstack/requirements: Remove lockfile from requirements  https://review.openstack.org/15122400:02
*** sdake has quit IRC00:03
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Simple script to setup and test log server  https://review.openstack.org/16911700:05
*** mtanino has quit IRC00:05
*** emagana_ has quit IRC00:06
*** stevemar has joined #openstack-infra00:06
*** emagana has joined #openstack-infra00:06
fungipbr tested with testtools installed by pbr... <singing>it's the circle of liiiife</singing>00:07
*** zz_jgrimm has quit IRC00:10
*** emagana has quit IRC00:10
*** zz_jgrimm has joined #openstack-infra00:11
*** otter768 has joined #openstack-infra00:13
*** spzala has quit IRC00:13
*** sputnik13 has quit IRC00:15
*** signed8bit_ZZZzz is now known as signed8bit00:16
*** zz_jgrimm has quit IRC00:17
*** otter768 has quit IRC00:17
*** zz_ja has quit IRC00:17
*** zz_ja has joined #openstack-infra00:18
*** zz_jgrimm has joined #openstack-infra00:18
jeblairfungi: bravo!  (i didn't want you to think the extended awkward silence was at all due to your singing!)00:22
fungijeblair: well, to be fair... it usually is00:25
fungii need to take lessons from SpamapS00:26
openstackgerritMerged openstack-infra/git-review: Push language override down into the cmd wrapper  https://review.openstack.org/17984100:27
*** zhiwei has joined #openstack-infra00:31
*** stevemar has quit IRC00:32
*** armax has joined #openstack-infra00:33
*** tiswanso has quit IRC00:40
*** hashar has quit IRC00:42
openstackgerritJames E. Blair proposed stackforge/gertty: Add files table  https://review.openstack.org/17997900:44
openstackgerritJames E. Blair proposed stackforge/gertty: Attach comments to files  https://review.openstack.org/17998000:44
openstackgerritJames E. Blair proposed stackforge/gertty: Add database pruning  https://review.openstack.org/17998100:44
*** mpaolino has joined #openstack-infra00:44
jeblairthose changes are the result of two weeks of plane flights :)  they could probably benefit from some careful review and testing -- but if you do feel like testing, it would probably be a good idea to back up your gertty database first00:45
*** mpaolino has quit IRC00:45
jeblairthe first two changes involve a schema change, and the third actually deletes things00:45
jeblair(also, all of them are likely to take some time to execute the first time they are run)00:46
mordredjeblair: what if I want to run them, but not test things, and also complain when it breaks something?00:46
jeblairmordred: you've got this 'user' thing down00:48
mordredjeblair: I'm trying to empathize00:48
jeblairmordred: so regarding 17589100:49
mordredjeblair: talk to me00:49
jeblairmordred: the deprecation note says "until it's reasonable to expect people to have upgraded to 1.10"00:49
jeblairmordred: we can remove that after crossing a shade release boundary, right?00:50
mordredjeblair: ish. the danger people are people who are running ansible and shade from pip install and who are just upgrading both with an unbounded instal00:50
mordredhonestly, that's likely a very small set00:50
jeblairmordred: does ansible have a reasonable semver dep on shade?00:51
mordredif you installed ansible to run things, and installed shade to support it, and you upgrade ansible, it's not unreasonable that you would also upgrade shade00:51
mordredjeblair: ansible has no way to actually depend on shade - it has a way to give an error if you don't have shade installed00:51
jeblairmordred: do i want to ask followup questions?00:52
*** shashankhegde has quit IRC00:52
mordredwhich is to say - you may be running this code on a machien other than the one you ahve ansible installed on00:52
mordredand it's on _that_ machine that you want shade installed00:52
jeblairoh, hrm.  i thought the ansible shade thing was only useful in ansible proper00:53
mordredjeblair: but yes - I do not htink we need to maintain it indefinitely - I think I more would want to wait until at least another ansible release00:53
mordredI think releasing a break to taht api while the only released bit is in a previous release would be rude00:53
jeblairmordred: okay.  glad i asked.  :)00:54
*** dprince has joined #openstack-infra00:54
jeblairmordred: i do feel like the dependency relationship here is problematic.  but i'm also okay with long deprecation cycles in libraries.  :)00:54
mordredjeblair: yeah - I agree- but also feel like supporting thigns for a decent stretch00:55
mordredbeacuse otherwise we're evil00:55
jeblairi would like it if we could specify the lib dependency that we need, and the lib never changed.  :)00:56
mordredjeblair: have I mentioned this language I like called rust ...00:56
*** nikita09 has quit IRC00:56
mordredjeblair: it solves none of these problems - but it's fun to talk about00:57
mordredjeblair: oh - I have a patch I need to send up for you and clarkb to look at00:57
mordredI wrote an ansible module in shell not python00:57
mordredwhich turns out to be both easy and non-insane00:57
*** freerunner has quit IRC00:58
openstackgerritSamuel de Medeiros Queiroz proposed openstack-infra/devstack-gate: Remove arbitrary devstack config options  https://review.openstack.org/17998201:01
openstackgerritSamuel de Medeiros Queiroz proposed openstack-infra/project-config: Pass through arbitrary devstack config options  https://review.openstack.org/17998301:02
samueldmqmordred, ^01:02
samueldmqmordred, I could find 3 options that could be passed through to devstack directly, instead of having d-g options for them01:02
openstackgerritJoshua Harlow proposed openstack/requirements: Add 'shared_lock' package  https://review.openstack.org/16366601:04
*** alexsyip has quit IRC01:08
*** jtriley has quit IRC01:09
*** tiswanso has joined #openstack-infra01:11
openstackgerritClark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering  https://review.openstack.org/17998801:14
clarkbjeblair: zaro ^ that passes mvn tests but I have not tested it with a real jenkins yet01:14
clarkbjeblair: zaro its also getting late so I probably made all sorts of mistakes, but I hope the general goals are apparent01:14
*** tnovacik has quit IRC01:15
*** asettle has joined #openstack-infra01:16
mordredclarkb: btw - I did a test on teh flamel question from earlier01:18
mordred>>> yaml.dump(dict(foo=u'100'))01:18
mordred"{foo: !!python/unicode '100'}\n"01:18
mordredclarkb: ^^ that's the thing I was talking about01:18
clarkbmordred: oh you are supposed to convert it to utf801:19
clarkbmordred: then its never an issue01:19
clarkbbecause yaml will assume utf8 reading it back01:19
mordredright. but I'm getting a giant dict from elsehwere01:19
mordredand it's showing up as u'' values01:19
mordredclarkb: I could rewrite teh unicode_representer to concert the u'' to utf-8 when it passes it to the yaml.ScalarNode constructor01:21
jogodebian packages point to github as the upstream: http://metadata.ftp-master.debian.org/changelogs//main/k/keystone/keystone_2015.1.0-1_copyright01:22
clarkbmordred: ya its odd to me that pyyaml doesn't do this by default01:22
clarkbmordred: since utf8 should cover the code point space right?01:22
mordredclarkb: it is EXCEPTIONALLY odd to me that it does not01:22
clarkbmordred: eg unicode perfectly serialized to utf801:22
mordredclarkb: one would think01:22
mordredclarkb: with as messy as the py2/py3 unicode transition is - I'd have hoped that pyyaml would have gotten it right01:23
clarkbjogo: file a bug with debian?01:23
jogoclarkb: don't even know where to start for that01:23
clarkbjogo: I say that somewhat sarcastically since debian bugs are basically a mail tracker ticket system from the dawn of computers01:23
clarkbjogo: but it apparntly works for them so ya01:24
*** sarob has quit IRC01:27
*** purp_2 is now known as purp01:31
openstackgerritHuang Rui proposed openstack-infra/project-config: enable python-jobs for networking-zvm  https://review.openstack.org/17973001:32
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate: DO NOT MERGE: Examples of doing bash library commands  https://review.openstack.org/17999001:33
mordredclarkb: that's just an example for reference ^^01:34
mordredclarkb: I think it we wanted to make use of that ability, we'd want to think abot structure more01:34
*** purp is now known as stealth_purp01:36
clarkbianw: is there a reason you pushed a new change for https://review.openstack.org/#/c/178905/1 ?01:37
clarkbianw: you could just push a new patchset to the existing change01:37
*** Longgeek has joined #openstack-infra01:46
*** jtriley has joined #openstack-infra01:47
*** unicell has quit IRC01:54
*** dprince has quit IRC01:56
ianwclarkb: i was just in a stack of other stuff to get -e working as it should01:58
*** Longgeek has quit IRC01:58
ianwi can modify the change-id if you feel strongly01:59
clarkbno, its just really confusing for me as the reviewer to go "wait I wrote this change"02:00
ianwclarkb: let's call it "great minds" syndrome, i think we both saw the issue for different reasons02:02
ianwat pretty much the same time02:03
*** julim has joined #openstack-infra02:03
*** patrickeast has quit IRC02:04
*** baoli has joined #openstack-infra02:05
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron  https://review.openstack.org/17968902:05
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade  https://review.openstack.org/17589102:06
mordredclarkb, jeblair: ^^ ok. I respun to fix the pep8 error, and added a verbose commend about why-flamel. I'm also happy to replace flamel with a copy of the relevant code if either of you would prefer02:06
*** dboik has joined #openstack-infra02:07
*** dboik_ has joined #openstack-infra02:09
*** hloeung has quit IRC02:09
*** pcrews has quit IRC02:09
*** cody-somerville has joined #openstack-infra02:11
*** achanda has quit IRC02:11
*** dboik has quit IRC02:11
*** otter768 has joined #openstack-infra02:13
*** zz_dimtruck is now known as dimtruck02:16
*** otter768 has quit IRC02:18
*** baoli has quit IRC02:24
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add script to split the secret hieradata file  https://review.openstack.org/17965302:25
mordredjhesketh: I think I addressed your concerns ^^ - thanks, good feedback02:25
greghaynesmordred: looks like the https://review.openstack.org/#/c/179616/ stack needs some rebasing, mind if I go for it?02:26
jheskethmordred: cool, thanks. I'll take a look later though sorry02:26
mordredgreghaynes: go for it02:26
mordredjhesketh: totally no rush at al- very unurgent02:27
*** samueldmq has quit IRC02:27
*** sarob has joined #openstack-infra02:28
*** shashankhegde has joined #openstack-infra02:31
*** asettle has quit IRC02:31
greghaynesmerge conflicts in autogenerated files = awesome02:33
*** sarob has quit IRC02:33
*** sigmavirus24 is now known as sigmavirus24_awa02:33
*** ivar-laz_ has joined #openstack-infra02:35
*** baoli has joined #openstack-infra02:36
mordredgreghaynes: yah02:37
*** yamahata has quit IRC02:37
*** stealth_purp is now known as purp02:37
*** ivar-lazzaro has quit IRC02:38
mordredoh! nice02:38
* mordred missed ianw's change02:38
greghaynesheh, yea02:38
greghaynesim about to rebase https://review.openstack.org/#/c/179612/2 onto master so you can abandon the two below it02:38
*** sarob has joined #openstack-infra02:39
*** ivar-laz_ has quit IRC02:39
mordredabandon is done02:39
openstackgerritgreghaynes proposed openstack-infra/system-config: Start building ubuntu-trusty nodes  https://review.openstack.org/17961202:41
*** dimtruck is now known as zz_dimtruck02:41
*** niska has quit IRC02:48
nibalizerclarkb: https://github.com/nibalizer/hiera-garbage02:48
nibalizerclarkb: so 1) install taht in apply-test, 2) no more need for defaults in site.pp02:49
nibalizerfor hiera() calls02:49
*** Kennan2 has joined #openstack-infra02:49
*** Kennan has quit IRC02:50
mordredI like nt having defaults in site.pp ... I find them very dangerous02:51
*** jyuso1 has joined #openstack-infra02:52
*** sputnik13 has joined #openstack-infra02:53
*** Longgeek has joined #openstack-infra02:56
*** wenlock has quit IRC02:58
*** jtriley has quit IRC03:01
*** niska has joined #openstack-infra03:03
*** shashankhegde has quit IRC03:03
*** Longgeek has quit IRC03:08
*** tiswanso has quit IRC03:14
nibalizermordred: ya i didn't even realize the danger until clark pointed it out03:17
nibalizerthen im like wat noooooooooo03:17
*** niska has quit IRC03:18
*** pcrews has joined #openstack-infra03:21
*** coolsvap|afk is now known as coolsvap03:23
*** dims_ has quit IRC03:23
*** niska has joined #openstack-infra03:23
*** yamahata has joined #openstack-infra03:24
*** jtriley has joined #openstack-infra03:25
*** otter768 has joined #openstack-infra03:25
*** signed8bit has quit IRC03:25
*** jtriley has quit IRC03:29
*** asettle has joined #openstack-infra03:30
*** beisner- has joined #openstack-infra03:31
*** asettle has quit IRC03:31
*** beisner has quit IRC03:31
*** beisner- is now known as beisner03:31
*** asettle has joined #openstack-infra03:31
*** links has joined #openstack-infra03:31
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Update Networking L2 gateway meeting  https://review.openstack.org/18001603:33
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Update Nova API Meeting time  https://review.openstack.org/18001703:33
*** otter768 has quit IRC03:33
*** otter768 has joined #openstack-infra03:35
*** sputnik13 has quit IRC03:38
*** Kennan has joined #openstack-infra03:39
*** shashankhegde has joined #openstack-infra03:40
*** Kennan2 has quit IRC03:40
*** zz_dimtruck is now known as dimtruck03:41
jamielennoxjhesketh: around https://review.openstack.org/#/c/172803/ - when we were discussing whether to use the _ or - for doa-kerberos nobody really cared03:42
jamielennoxjhesketh: there is no inherent reason that doa uses _ and we figured we may as well go with the more standard scheme03:42
jamielennoxbut the first person with any strong opinion is probably going to carry the argument as no one else really cares03:43
*** stevemar has joined #openstack-infra03:43
*** Kennan has quit IRC03:45
*** Kennan has joined #openstack-infra03:47
*** MarkAtwood has quit IRC03:47
*** armax has quit IRC03:47
*** sks has joined #openstack-infra03:48
*** sabeen has quit IRC03:49
*** sabeen has joined #openstack-infra03:49
*** achanda has joined #openstack-infra03:51
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron  https://review.openstack.org/17968903:52
*** baoli has quit IRC03:52
*** shashankhegde has quit IRC03:56
*** Somay has joined #openstack-infra03:59
*** yfried|afk is now known as yfried04:01
*** Longgeek has joined #openstack-infra04:05
*** ayoung-dadmode has quit IRC04:05
*** otter768 has quit IRC04:09
*** achanda has quit IRC04:12
*** gokrokve has joined #openstack-infra04:16
*** dimtruck is now known as zz_dimtruck04:19
*** ildikov has quit IRC04:20
*** gokrokve has quit IRC04:20
*** Kennan2 has joined #openstack-infra04:21
*** Kennan has quit IRC04:22
*** dims has joined #openstack-infra04:23
*** jtriley has joined #openstack-infra04:25
*** dims has quit IRC04:28
*** Longgeek has quit IRC04:29
*** sarob has quit IRC04:30
*** jtriley has quit IRC04:31
*** ildikov has joined #openstack-infra04:33
*** unicell has joined #openstack-infra04:35
*** sdake_ has quit IRC04:38
*** sks has quit IRC04:41
*** SumitNaiksatam has joined #openstack-infra04:43
*** maishsk has quit IRC04:43
*** mrmartin has joined #openstack-infra04:45
*** gokrokve has joined #openstack-infra04:45
*** ildikov has quit IRC04:45
ianwhmm, why does centos7 have eth0 & eth1, while ubuntu has only eth0 ...04:45
mordredianw because04:48
*** swat30 has quit IRC04:48
mordredianw: that actually should be fixed in the new minimal images04:50
*** sabeen has quit IRC04:51
ianwso it's hpcloud v rax difference04:51
clarkbmordred I dont think images fix that04:51
clarkbmordred we need both interfaces04:51
mordredit depends04:52
clarkbianw yes rax gives you private and publoc nets04:52
clarkbhpcloud only does private with nat04:52
*** swat30 has joined #openstack-infra04:54
*** yfried is now known as yfried|afk04:54
clarkbmordred we use both on rax04:55
*** sks has joined #openstack-infra04:56
*** sdake has joined #openstack-infra04:57
*** yfried|afk is now known as yfried04:58
mordredclarkb: yah. the thing I was poorly saying before beacuse I was on my phone ...05:00
*** sks has quit IRC05:01
mordredclarkb: is that some of the images have baked in eth0 and eth1 files regardless of whether the instance actually has 2 interfaces or not05:01
mordredclarkb: and I believe that default is different across ubuntu and centos05:01
mordredbut that has nothiing to do with what ianw was describing05:02
clarkbreally? on rax they write in network startup scripts for you, and on hpcloud I have only ever seen tge one interface05:02
mordredclarkb: yeah - possible I'm thinking about base distro images05:03
*** abregman has joined #openstack-infra05:03
mordredclarkb: in any case - chalk it up to travel brain :)05:03
clarkbwriting those scripts is what broke our local dns resolver setup because they just overrode everything05:03
mordredyah. so nice of them :)05:03
clarkbwhich is great because their dna doesnt work05:03
mordreddna too05:03
mordredI tried to boot their dna into a freshly cloned host organism and it did not get me anywhere05:04
clarkbthat would be a neat trick, next step jurassic park05:04
ianwwell, yeah, just clutching at straws trying to get some handle on these centos7 failures.  debugging neutron is very, very far down the fun->painful spectrum05:07
*** asettle has quit IRC05:07
SpamapSmordred: sorry if I missed the middle, but I noticed the yaml.dump thing from earlier while reading backscroll..05:09
SpamapS>>> yaml.safe_dump(dict(foo=u'100'))05:09
SpamapS"{foo: '100'}\n"05:09
*** achanda has joined #openstack-infra05:10
SpamapSmordred: you basically never want yaml.dump05:10
SpamapSunless you want the insanity of object serialization05:10
*** Somay has quit IRC05:12
mordredah - gotcha. that probably makes sense as to why clarkb and I were missing each other there for a second05:12
mordredSpamapS: dare I ask why "dump" isn't the corrent API call to call/05:12
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade  https://review.openstack.org/17589105:14
mordredSpamapS, clarkb: ^^ there. That should take care of the concerns05:14
*** emagana has joined #openstack-infra05:16
lifelessmordred: because we CANT HAVE NICE THINGS05:16
*** BharatK has joined #openstack-infra05:17
*** shashankhegde has joined #openstack-infra05:18
mordredlifeless: I hear that's a problem in the python world05:20
lifelessDo NOT get me started.05:22
SpamapSIt so should be the default05:22
SpamapSand the other one should be 'unsafe_dump' and 'unsafe_load'05:23
clarkbmordred pretty sure its a problem with computers05:23
clarkbspeaking of maybe I should tackle the jenkins thread leak next, I have already been writing java anyways05:24
SpamapSmordred: it really goes down to the fact that yaml can represent arbitrary things, and load will happily try to define them.05:24
* greghaynes looks at ruby05:24
SpamapS" Warning: It is not safe to call yaml.load with any data received from an untrusted source! yaml.load is as powerful as pickle.load and so may call any Python function. Check the yaml.safe_load function though."05:25
* mordred hands SpamapS a half-used jar of pickles05:26
SpamapSalso the most performant version is 'yaml.load(x, Loader=yaml.CSafeLoader)' .. but it doesn't allow all the nice things from flamel IIRC05:26
SpamapSmordred: delicious05:26
SpamapShow did you "use" them?05:26
greghaynesare those safe or unsafe pickles05:27
clarkbthere are no safe pickles05:27
mordredSpamapS: it's best if you don't ask too many questions05:27
greghaynesReviews on https://review.openstack.org/#/c/179209/ requested, its a pretty no-brainer05:28
greghaynesthe depends-on just merged05:28
purpCheck taint. That's all I'm sayin'.05:28
mordredgreghaynes: +205:29
SpamapSactually it looks like CSafeLoader does work fine05:31
clarkbSpamapS doesnt that require you to have libyaml available and linked against though?05:32
SpamapSclarkb: yeah, you have to do a conditional import05:32
SpamapSclarkb: it won't be present if libyaml isn't05:32
mordredSpamapS: we should make a library called yamlyaml which wraps pyyaml but renames dump and load to unsafe_ and makes safe_ the default - and also set default_flow_whatever to the opposite05:33
SpamapSHave oft thought of making a pypi project called 'cyaml' which just always fails if libyaml isn't available but otherwise is a passthrough to the C backed yaml calls05:33
SpamapSFor large yaml docs, the pure python one can use 400x more memory and CPU, no joke.05:33
*** tnovacik has joined #openstack-infra05:33
SpamapSmordred: we could call that library 'flamel' ;)05:34
mordredSpamapS: bah. clearly flamel should use yamlyaml05:34
SpamapSreduce all the things05:34
purpSpamaps: Ugh. That's pretty rough. How's performance?05:34
*** ildikov has joined #openstack-infra05:35
SpamapSpurp: not sure I know how to answer your question.05:35
mordredSpamapS: parsecs. always answer in parsecs.05:37
purpSpamapS: with large docs, how long does load/and parse require for pure python vs. c lib?05:37
lifelesspurp: 400x05:38
lifelesspurp: for pathological cases05:38
purplifeless: sorry, read the earlier as 400x mem, not time. I'm slow today.05:39
SpamapSIt's logarithmic IIRC05:39
SpamapSor exponential, can't remember05:39
SpamapSWe found the problem while developing juju which used yaml to serialize things in zookeeper.05:40
lifelesspurp: mem & cpu05:40
lifelesspurp: the cpu is where a direct time increase comes in05:40
SpamapSas soon as you got to like 30 services to deserialize suddenly juju would crumble.05:40
lifelesspurp: enough memory pressure can cause perf issues too, of course05:40
lifelessgc overhead05:40
lifelessthere's a C thing, and we can all use it. \o/ and lets move on wif life05:41
SpamapSwith the cdumper/loader we pushed it to a simulated 50,000 services before we just ran out of ram to store the result.05:41
SpamapSlifeless: it has deficiencies to the python one.05:41
lifelessSpamapS: box had < 50M of ram ?05:41
SpamapSlifeless: each service had a _VERY_ large document. :)05:42
SpamapSand relations were big too05:42
SpamapSAnd I think we were probably storing 3 or 4 copies of the result05:42
mordredSpamapS: it would have performed better if you'd kept the original name05:43
*** hdd has joined #openstack-infra05:43
mordredwell, now that the new ansible modules have landed - I suppose putting off writing the docs for it is bad form05:44
*** cody-somerville has quit IRC05:46
*** asettle has joined #openstack-infra05:46
*** HeOS has quit IRC05:48
*** SumitNaiksatam has quit IRC05:48
*** SumitNaiksatam has joined #openstack-infra05:49
SpamapSmordred: yeah ensemble was way faster than juju ;)05:49
*** beecee has quit IRC05:52
*** sushilkm has joined #openstack-infra05:56
*** sushilkm has left #openstack-infra05:56
*** gokrokve_ has joined #openstack-infra05:58
*** Longgeek has joined #openstack-infra05:59
openstackgerritMerged openstack-infra/project-config: Add django-openstack-auth-kerberos project  https://review.openstack.org/17280305:59
*** mrunge has joined #openstack-infra05:59
openstackgerritMerged openstack-infra/puppet-diskimage_builder: Remove invalid ensure parameter from apt::ppa  https://review.openstack.org/17961806:00
*** gokrokve has quit IRC06:01
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/18003906:02
*** gokrokve_ has quit IRC06:02
*** maishsk has joined #openstack-infra06:06
*** asettle has quit IRC06:06
*** otter768 has joined #openstack-infra06:10
SpamapSfound my yaml test code06:11
SpamapS 5420 clint     20   0 1627556 1.521g   3392 R  99.7  9.8   0:47.70 python06:12
*** SamYaple_ is now known as SamYaple06:13
SpamapShttp://paste.ubuntu.com/10988471/   results06:13
SpamapSpurp: ^ in case you were still curious.06:14
*** otter768 has quit IRC06:14
*** Longgeek_ has joined #openstack-infra06:14
*** jtriley has joined #openstack-infra06:15
SpamapSoh interesting, I think I just found a python3 bug in pyyaml06:15
*** Longgeek has quit IRC06:17
SpamapSpython3 performs worse for both C and pure python cases. Interesting.06:19
*** masayukig has quit IRC06:19
*** jtriley has quit IRC06:19
*** masayukig has joined #openstack-infra06:19
*** openstackgerrit has quit IRC06:23
*** openstackgerrit has joined #openstack-infra06:23
*** bradm_ has joined #openstack-infra06:24
SpamapSpypy matches C06:24
SpamapSlifeless: ^06:24
lifeless<3 pypy06:24
SpamapShad to make it flexible to run via pypy06:24
SpamapSbut wow06:25
*** vponomaryov has quit IRC06:25
lifelessSpamapS: so summary data for the different interpreters?06:25
*** sslypushenko has quit IRC06:26
*** sslypushenko has joined #openstack-infra06:26
*** scheuran has joined #openstack-infra06:27
SpamapSso, for a 100,000 key dict:   python2, 47s, 1.6G RAM. python3, 53s, 250MB RAM. pypy, 7s, 154MB RAM, and libyaml, 8s, 170MB RAM06:28
* SpamapS just ran each one 3 times and threw out highs/lows06:28
SpamapSso python3 fixed the RAM insanity but made CPU worse.. better gc?06:28
SpamapSpypy is actually _beating_ C06:29
clarkbSpamapS is libyaml there via pyyaml binding?06:29
SpamapSclarkb: yes06:29
*** woodster_ has quit IRC06:30
SpamapSslight inconsistency in those results, I did 3.10 for python/python3, but 3.11 for pypy. Retrying all w/ 3.11.06:31
SpamapS(3.11 claims to just be a rebuild release but lets just make sure)06:31
*** imcsk8 has quit IRC06:31
SpamapSyeah same outcome06:31
*** emagana has quit IRC06:32
* SpamapS wakes up from optimization death match to realize time has been wasted06:32
SpamapSso.. if you need to parse giant yaml documents, DEFINITELY... use... pypy? ;)06:32
*** mrunge has quit IRC06:33
*** emagana has joined #openstack-infra06:33
*** bradm_ has quit IRC06:36
*** bradm has quit IRC06:36
*** bradm_ has joined #openstack-infra06:37
*** mrunge has joined #openstack-infra06:37
*** shashankhegde has quit IRC06:38
greghaynesnow just to make all the openstack bits work with pypy :)06:38
*** emagana has quit IRC06:38
*** shashankhegde has joined #openstack-infra06:39
*** vponomaryov has joined #openstack-infra06:39
openstackgerritMerged openstack-infra/project-config: Minor translation cleanups  https://review.openstack.org/17964306:44
openstackgerritgreghaynes proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492706:44
*** jcoufal has joined #openstack-infra06:50
*** e0ne has joined #openstack-infra06:51
*** yfried has quit IRC06:56
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/18003906:56
*** HeOS has joined #openstack-infra06:59
*** shashankhegde has quit IRC06:59
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade  https://review.openstack.org/17589107:04
*** e0ne has quit IRC07:06
*** sandywalsh has quit IRC07:11
*** cody-somerville has joined #openstack-infra07:12
*** cody-somerville has quit IRC07:12
*** cody-somerville has joined #openstack-infra07:12
*** sandywalsh has joined #openstack-infra07:12
*** markus_z has joined #openstack-infra07:12
*** yamahata has quit IRC07:14
*** achanda has quit IRC07:16
mgedminis bindep ready for alpha testing yet?07:16
*** dizquierdo has joined #openstack-infra07:16
* mgedmin checks https://pypi.python.org/pypi/bindep07:16
mgedminlooks like not :(07:16
*** sushilkm has joined #openstack-infra07:17
*** sushilkm has left #openstack-infra07:18
lifelessmgedmin: hmm?07:20
lifelessmgedmin: we haven't cut a release is all07:20
mgedminwell I can't pip install bindep yet ;)07:20
mgedmina more on-topic observation: https://github.com/openstack-infra/bindep#writing-requirements-files doesn't mention comments07:20
mgedminare comments allowed?  because I really really want comments07:20
mgedmin# these are build-deps for Pillow:07:21
lifelessmgedmin: the grammar is in bindep/depends.py07:21
*** achanda has joined #openstack-infra07:22
*** stevemar has quit IRC07:22
mgedminlooks like the answer is "no"07:22
mgedminafaiu there's no separate lexer/tokenizer?  parsley works directly with a stream of characters?07:23
lifelessstream of objects07:23
lifelessthis particular grammar is a character based one yes07:24
*** achanda has quit IRC07:24
lifelessshould be easy enough to add07:25
lifelesscomment = '#' any '\n'07:26
lifelessadd a rule alternate for comment07:26
*** luqas has joined #openstack-infra07:29
*** luqas has quit IRC07:30
*** maishsk has quit IRC07:30
lifelessmgedmin: anyhow, as far as getting a release, I don't have the release acl for bindep for whatever reason (odd since I started it, but others are driving it so meh)07:30
lifelessmgedmin: chat to fungi, whose been driving it and should definitely have the release acl, and will be awake in oh, 4-5 hours07:31
*** maishsk has joined #openstack-infra07:32
*** luqas has joined #openstack-infra07:32
*** luqas has quit IRC07:37
*** Hal has joined #openstack-infra07:38
*** Hal is now known as Guest1022407:38
mgedminwhy are py34 and pypy in the default envlist in tox.ini if tests are known not to be passing on py34 and pypy?07:39
mgedmin(I assume they're known not to be passing)07:39
mgedmin(mox is a dependency and mox doesn't support python 3)07:39
mgedmin(the pypy error about lack of _bsddb is more mysterious to me)07:39
mgedminwhere's the bug tracker for bindep?07:41
mgedminI've a feeling I asked about this before :(07:41
mgedminif the README had a link, I wouldn't need to ask again...07:41
mgedminhuh, perhaps I should read CONTRIBUTING.rst instead07:42
mgedminwaaah https://bugs.launchpad.net/bindep is a 40407:42
lifelessI don't think we've made a bug tracker07:42
*** jlanoux has joined #openstack-infra07:43
*** luqas has joined #openstack-infra07:43
*** chlong has quit IRC07:44
*** maishsk has quit IRC07:44
*** maishsk has joined #openstack-infra07:46
*** hdd_ has joined #openstack-infra07:48
*** ajo has joined #openstack-infra07:49
*** btran has quit IRC07:50
*** sushilkm has joined #openstack-infra07:50
*** sushilkm has left #openstack-infra07:50
*** bradm_ is now known as bradm07:50
*** hdd has quit IRC07:51
mgedminare there tests for the grammar?07:53
*** aarefiev has joined #openstack-infra07:54
mgedminhm, okay07:54
Davieylifeless: Sorry, i sent my mail - then went to bed.  Hai o/07:57
mgedminAttributeError: 'Grammar' object has no attribute 'rule_any'07:57
* mgedmin accidentally discovers himself reading http://en.wikipedia.org/wiki/Parsley07:57
mgedminah, it's anything, not any07:59
*** Somay has joined #openstack-infra08:02
*** samuelBartel has joined #openstack-infra08:02
mgedminany constraints on the grammar accepted by parsley?  context-free?  LL(1)?  LR(1)?  LALR(1)?  (not that I remember what those are)08:03
*** arxcruz has joined #openstack-infra08:03
mgedminI'm just wondering why it's not matching my blank line rule ...08:03
mgedmingrammar from http://paste.ubuntu.com/10988797/ fails the test with http://paste.ubuntu.com/10988798/08:04
*** liam_ has joined #openstack-infra08:04
*** liam_ is now known as Guest7458508:05
* mgedmin reads about http://en.wikipedia.org/wiki/Parsing_expression_grammar08:06
lifelessmgedmin: its ometa08:07
lifelessmgedmin: which is a PEG with extensions08:07
lifelessDaviey: no wakka08:07
lifelessmgedmin: yes there are08:07
*** Guest74585 has quit IRC08:09
*** hashar has joined #openstack-infra08:11
*** hashar has quit IRC08:11
*** hashar has joined #openstack-infra08:11
*** otter768 has joined #openstack-infra08:11
*** fhubik has joined #openstack-infra08:14
*** _nadya_ has joined #openstack-infra08:15
*** otter768 has quit IRC08:16
fricklerhmm, zuul getting stuck again? all changes in the check pipeline have queued tasks and do not seem to progress08:21
*** derekh has joined #openstack-infra08:23
*** fhubik is now known as fhubik_afk08:24
openstackmgedmin: Error: "logs" is not a valid command.08:25
mgedmin(wifi loss == I don't know if I missed any replies)08:26
*** e0ne has joined #openstack-infra08:28
*** e0ne is now known as e0ne_08:28
*** vponomaryov has quit IRC08:28
*** fhubik_afk is now known as fhubik08:28
*** vponomaryov has joined #openstack-infra08:33
*** MaxV has joined #openstack-infra08:34
openstackgerritVictor Stinner proposed openstack-infra/jenkins-job-builder: Fix Builder.update_job() on Python 3  https://review.openstack.org/17223808:36
*** Somay has quit IRC08:38
mgedmina-ha!  anything eats the '\n', which makes the pattern not match08:38
*** tkelsey has joined #openstack-infra08:38
mgedminokay, pull request time!08:40
mgedmin"Pull requests submitted through GitHub will be ignored." how friendly08:40
openstackgerritMarius Gedminas proposed openstack-infra/bindep: Allow comments and blank lines in other-requirements.txt  https://review.openstack.org/18006008:41
mgedmincool, it worked08:41
*** e0ne_ is now known as e0ne08:44
*** SumitNaiksatam has quit IRC08:44
openstackgerritMerged openstack-infra/project-config: Enable a check job for stackforge/dragonflow  https://review.openstack.org/17515208:45
*** e0ne is now known as e0ne_08:45
*** SumitNaiksatam has joined #openstack-infra08:46
*** e0ne_ is now known as e0ne08:46
lifelessmgedmin: and you has a review :)08:46
mgedminI see "(2 comments)" and "One small change requested"; how do I view the 2 comments?08:47
lifelessclick on README.rst08:48
anteayafrickler: gerrit is still broadcasting events08:49
mgedmingot it08:49
mgedminBTW my patch is wrong but the test suite doesn't catch the bug08:49
*** dguitarbite has joined #openstack-infra08:49
*** ssam2 has joined #openstack-infra08:50
anteayafrickler: I see queued and aborted, and the only job I see running is the top job in the post queue08:50
frickleranteaya: if you look at the graph of test nodes, it seems like more and more are made available, but not used08:51
anteayafrickler: I don't see any jobs running in the gate, though some are queued08:51
anteayafrickler: agreed08:51
anteayaI wonder what is happening08:51
anteayaSergeyLukjanov: are you about?08:51
lifelessmgedmin: so fix it up, then commit --amend; then git review08:51
lifelessmgedmin: :)08:51
mgedminugh, gerrit08:52
mgedmindoes it support any kind of markup in comments?08:52
lifelessmgedmin: yes08:52
lifelessmgedmin: prefix lines with space to make them <pre>08:52
anteayajhesketh: you aren't still online are you?08:53
*** MaxV has quit IRC08:53
anteayafrickler: what time zone are you? I'm soon back to bed08:54
openstackgerritDavide Guerri proposed openstack-infra/shade: Add keystone services/endpoints methods  https://review.openstack.org/17762108:54
*** sergsh_ has joined #openstack-infra08:55
frickleranteaya: GMT+2, so its morning here still ;) not sure how I can help with this, though08:55
*** sergsh has joined #openstack-infra08:56
anteayafrickler: if you see SergeyLukjanov around share your observations08:56
anteayaif you would08:56
anteayahe has root access to the servers08:56
fricklerk, will do08:56
anteayaI don't and everyone else who does is fast asleep08:56
anteayafrickler: thank you08:56
*** jistr has joined #openstack-infra08:56
anteayanot sure what can be done, but at least he can get more information that I can08:57
anteayafrickler: if you don't see him, not to worry08:57
anteayaand thanks08:57
fricklernp, good night to you, then08:57
anteayathanks, good night08:58
openstackgerritMarius Gedminas proposed openstack-infra/bindep: Allow comments and blank lines in other-requirements.txt  https://review.openstack.org/18006008:58
*** deepakcs has joined #openstack-infra08:59
*** pelix has joined #openstack-infra08:59
lifelessmgedmin: thanks, needs a second reviewer - again, nab fungi09:02
*** scheuran has quit IRC09:02
*** fhubik is now known as fhubik_afk09:02
lifelessnight all09:03
*** niska has quit IRC09:12
mgedminhmm, bindep takes 4 seconds to compute that none of my 30 packages are missing09:13
*** shardy_ has joined #openstack-infra09:13
AJaegerclarkb, jeblair, fungi : Post queue is again at 6 hours ;(09:14
mgedminapt-get install --no-install-recommends $(sed e 's/#.*//' < other-requirements.txt) takes 0.8 seconds09:14
*** shardy has quit IRC09:14
*** HeOS_ has quit IRC09:14
AJaegeranteaya, that might be again the gearman bug that fungi, clarkb and jeblair are chasing for a week now.09:15
fricklerAJaeger: anteaya is gone already09:15
fricklerAJaeger: seems liek new nodes are getting prepared but never used09:16
AJaegerfrickler, once fungi is up, he can rerun a command to get this sped up09:16
*** scheuran has joined #openstack-infra09:16
*** HeOS_ has joined #openstack-infra09:16
AJaegerfrickler, http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-05-04.log - search for gearman09:17
*** cdent has joined #openstack-infra09:17
*** fhubik_afk is now known as fhubik09:18
*** hdd__ has joined #openstack-infra09:18
*** shardy_ has quit IRC09:19
*** shardy has joined #openstack-infra09:19
AJaegerfrickler, but this looks today worse than "normal" ;(09:20
openstackgerritMerged openstack-infra/release-tools: Include 'source_url' in notes and use readme parser function  https://review.openstack.org/17652809:20
*** hdd_ has quit IRC09:21
*** jyuso1 has quit IRC09:21
*** jyuso has joined #openstack-infra09:21
*** samueldmq has joined #openstack-infra09:25
*** alexpilotti has joined #openstack-infra09:25
*** maishsk has quit IRC09:26
*** maishsk has joined #openstack-infra09:26
openstackgerritMatthieu Huin proposed openstack-infra/infra-specs: Allow reviewstats to be used with any issue tracker  https://review.openstack.org/17288609:27
*** rlandy has joined #openstack-infra09:29
*** isviridov_away is now known as isviridov09:32
*** maishsk has quit IRC09:32
*** maishsk has joined #openstack-infra09:32
*** gilllliard is now known as gilliard_afk09:33
openstackgerritMerged openstack-infra/release-tools: Move highest_semver.py from oslo-incubator  https://review.openstack.org/17924309:34
openstackgerritMerged openstack-infra/release-tools: update make_library_stable_branch.sh to set defaultbranch  https://review.openstack.org/17391909:34
*** isviridov is now known as isviridov_away09:35
openstackgerritAndreas Jaeger proposed openstack-infra/release-tools: Add script to cleanup of translations  https://review.openstack.org/17282709:36
openstackgerritMerged openstack-infra/irc-meetings: Update Networking L2 gateway meeting  https://review.openstack.org/18001609:37
openstackgerritMerged openstack-infra/irc-meetings: Update Nova API Meeting time  https://review.openstack.org/18001709:37
*** markvoelker has quit IRC09:48
*** ociuhandu has joined #openstack-infra09:51
sambettsHey Infra, I have created a stable branch for the stackforge/networking-cisco repo, which all works great, however I have just cherrypicked a patch that I wanted from master and it doesn't appear that Jenkins is working with that repo, have I missed something in the configuration to make that work?09:51
*** fhubik is now known as fhubik_afk09:52
AJaegersambetts, our CI is a bit slow right now. Check http://status.openstack.org/zuul/ and search for your change. I see three queued for the repo09:53
*** niska has joined #openstack-infra09:53
SergeyLukjanovanteaya, frickler, hi, (was driving to office), what's up?09:53
AJaegersambetts, please have some patience09:53
*** sergsh has quit IRC09:54
sambettsAJaeger: Ah, I thought I might have missed something again, thanks for a quick response09:54
AJaegerSergeyLukjanov, seems we run again into the gearman issue where we basically loose all workers09:54
*** zz_dimtruck is now known as dimtruck09:54
*** sergsh has joined #openstack-infra09:54
AJaegerfungi has done some magic in the past for that - not sure what he did and I'm also not sure it's the same issue.09:54
AJaegerthe bug would lead to stale ready notes09:55
SergeyLukjanovAJaeger, heh, I don't know what fungi was doing to fix it... :(09:56
AJaegerSergeyLukjanov, I didn't ping you ;)09:56
*** scheuran has quit IRC09:57
AJaegerSergeyLukjanov, might be best to wait for fungi09:57
*** Guest10224 has quit IRC09:57
AJaegerSergeyLukjanov, Fungi mentioned "i've gone ahead and deleted any nodes in a ready state for more than an hour, which should hopefully get things moving again rsn"09:58
*** samueldmq has quit IRC09:58
AJaegerSergeyLukjanov, that was during the weekend09:59
*** zhiwei has quit IRC09:59
SergeyLukjanovAJaeger, looking for bash history on zuul server :)10:00
*** ajo has quit IRC10:01
BobBallAnyone know how many posts to /servers are made for each node created by nodepool?  I imagine at least 2, but if it's more I'm kinda interested.  I've got a limit for 1000 requests to /servers per 24 hour period which I exceeded.  Trying to understnd how many failed server builds that equates to.10:01
*** niska has quit IRC10:03
*** maishsk has quit IRC10:07
*** maishsk has joined #openstack-infra10:07
*** ajo has joined #openstack-infra10:08
openstackgerritprabhu murthy proposed openstack/requirements: Added pyVim & pyVmomi to global requirements  https://review.openstack.org/18008810:09
openstackgerritSean Dague proposed openstack-infra/system-config: use optimized url for zuul status  https://review.openstack.org/17093510:10
openstackgerritSean Dague proposed openstack-infra/system-config: add dbg function to hideci  https://review.openstack.org/17957310:10
openstackgerritSean Dague proposed openstack-infra/system-config: Serve up jquery-visibility.js for gerrit  https://review.openstack.org/17976510:10
openstackgerritSean Dague proposed openstack-infra/system-config: add jquery visibility  https://review.openstack.org/17936210:10
openstackgerritSean Dague proposed openstack-infra/system-config: create proxy for testing hideci  https://review.openstack.org/17936010:10
openstackgerritSean Dague proposed openstack-infra/system-config: Revert "Revert zuul status in gerrit"  https://review.openstack.org/17936110:10
*** dimtruck is now known as zz_dimtruck10:10
*** yfried has joined #openstack-infra10:11
*** _nadya_ has quit IRC10:11
*** otter768 has joined #openstack-infra10:12
*** fhubik_afk is now known as fhubik10:12
*** sushilkm has joined #openstack-infra10:14
*** niska has joined #openstack-infra10:14
*** sushilkm has left #openstack-infra10:14
*** vryzhenkin is now known as freerunner10:15
*** freerunner is now known as vryzhenkin10:15
*** vryzhenkin is now known as freerunner10:15
*** jyuso1 has joined #openstack-infra10:15
*** otter768 has quit IRC10:16
*** jyuso194 has joined #openstack-infra10:18
*** jyuso has quit IRC10:19
*** dims has joined #openstack-infra10:20
*** wznoinsk has joined #openstack-infra10:20
*** jyuso1 has quit IRC10:21
*** katyafervent is now known as katyafervent_awa10:22
*** dims_ has joined #openstack-infra10:22
*** samueldmq has joined #openstack-infra10:23
*** dims has quit IRC10:26
*** jlanoux_ has joined #openstack-infra10:27
*** maishsk is now known as maishsk_afk10:29
*** wznoinsk has quit IRC10:29
*** claudiub has joined #openstack-infra10:29
*** wznoinsk has joined #openstack-infra10:30
*** jlanoux has quit IRC10:31
*** armax has joined #openstack-infra10:34
*** e0ne is now known as e0ne_10:35
*** jyuso1 has joined #openstack-infra10:36
*** e0ne_ is now known as e0ne10:36
*** Hal has joined #openstack-infra10:36
*** Hal is now known as Guest2137310:37
*** katyafervent_awa is now known as katyafervent10:38
*** jyuso194 has quit IRC10:39
*** armax has quit IRC10:42
*** pc_m has joined #openstack-infra10:44
*** jyuso has joined #openstack-infra10:46
*** jyuso1 has quit IRC10:49
*** scheuran has joined #openstack-infra10:54
*** e0ne is now known as e0ne_10:56
*** yfried is now known as yfried|afk10:58
*** armax has joined #openstack-infra10:59
*** jlanoux has joined #openstack-infra11:00
*** fhubik has quit IRC11:01
openstackgerritSean Dague proposed openstack-infra/system-config: use optimized url for zuul status  https://review.openstack.org/17093511:01
openstackgerritSean Dague proposed openstack-infra/system-config: Serve up jquery-visibility.js for gerrit  https://review.openstack.org/17976511:01
openstackgerritSean Dague proposed openstack-infra/system-config: add jquery visibility  https://review.openstack.org/17936211:01
openstackgerritSean Dague proposed openstack-infra/system-config: Revert "Revert zuul status in gerrit"  https://review.openstack.org/17936111:01
*** fhubik has joined #openstack-infra11:02
*** jlanoux_ has quit IRC11:02
*** yfried|afk is now known as yfried11:04
*** niska has quit IRC11:04
*** sandywalsh has quit IRC11:12
*** sandywalsh has joined #openstack-infra11:14
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron  https://review.openstack.org/17968911:20
*** niska has joined #openstack-infra11:21
*** freerunn_ has joined #openstack-infra11:25
*** jyuso1 has joined #openstack-infra11:30
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron  https://review.openstack.org/17968911:32
*** jyuso has quit IRC11:33
*** niska has quit IRC11:36
*** woodster_ has joined #openstack-infra11:38
*** freerunn_ has quit IRC11:40
*** niska has joined #openstack-infra11:41
*** marcusvrn1 has joined #openstack-infra11:46
*** marcusvrn has quit IRC11:48
deepakcsmtreinish, when u get back here, pls take a look @ https://review.openstack.org/#/c/172841/ where i quoted ianw's opinion which seems to conflict with yours.11:50
*** sushilkm has joined #openstack-infra11:51
*** markvoelker has joined #openstack-infra11:51
*** e0ne_ is now known as e0ne11:51
*** sushilkm has left #openstack-infra11:51
jklarehi, could somebody give this one https://review.openstack.org/#/c/179721/ a quick push (we currently have no gates for our openstack-chef-repo)11:52
*** Somay has joined #openstack-infra11:54
*** dprince has joined #openstack-infra11:54
*** ldnunes has joined #openstack-infra11:56
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Adjust precedence levels of params  https://review.openstack.org/17945911:59
*** armax has quit IRC12:00
*** freerunner has quit IRC12:01
*** freerunner has joined #openstack-infra12:02
sdaguefungi: once you wake up, zuul seems stuck12:02
*** maishsk_afk has quit IRC12:04
*** fhubik has quit IRC12:05
*** fhubik has joined #openstack-infra12:06
*** freerunner has quit IRC12:06
*** pc_m has quit IRC12:07
*** _nadya_ has joined #openstack-infra12:12
openstackgerritSean Dague proposed openstack-infra/system-config: make zuul_inline = true  https://review.openstack.org/18011612:12
*** otter768 has joined #openstack-infra12:13
*** kgiusti has joined #openstack-infra12:13
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Support default string values for variables  https://review.openstack.org/18011712:13
*** deepakcs has quit IRC12:15
*** mwagner_lap has quit IRC12:16
*** dkranz has joined #openstack-infra12:16
*** otter768 has quit IRC12:18
*** freerunner has joined #openstack-infra12:19
openstackgerritMichal Vanco proposed openstack-infra/jenkins-job-builder: Support for image gallery publisher  https://review.openstack.org/17574712:19
*** niska has quit IRC12:21
openstackgerritJulien Danjou proposed openstack-dev/pbr: builddoc: allow to use fnmatch-style exclusion for autodoc  https://review.openstack.org/18011912:22
jheskethanteaya: sorry I missed you earlier, I'm around at the moment if you are12:23
openstackgerritEmilien Macchi proposed openstack-infra/system-config: Create rubygems mirror from rubygems.org  https://review.openstack.org/17802612:24
*** fhubik is now known as fhubik_afk12:25
*** dkranz has quit IRC12:26
*** niska has joined #openstack-infra12:27
*** fhubik_afk is now known as fhubik12:27
jheskethsdague: thanks for addressing my comments12:31
*** mwagner_lap has joined #openstack-infra12:33
*** gordc has joined #openstack-infra12:34
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: use optimized url for zuul status  https://review.openstack.org/17093512:34
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Serve up jquery-visibility.js for gerrit  https://review.openstack.org/17976512:34
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: add jquery visibility  https://review.openstack.org/17936212:34
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: make zuul_inline = true  https://review.openstack.org/18011612:34
*** jyuso has joined #openstack-infra12:37
*** freerunner has quit IRC12:37
*** Somay has quit IRC12:38
*** rfolco has joined #openstack-infra12:38
*** dkranz has joined #openstack-infra12:38
*** gyee has joined #openstack-infra12:39
*** jyuso1 has quit IRC12:39
*** niska has quit IRC12:40
*** baoli has joined #openstack-infra12:40
*** aysyd has joined #openstack-infra12:40
sergmelikyanfolks, is Zuul feeling well?12:41
*** dims_ has quit IRC12:41
sergmelikyanhttp://status.openstack.org/zuul/ to big available nodes number and some of the patches are hours in waiting queue12:41
*** dims has joined #openstack-infra12:41
sergmelikyanSergeyLukjanov ^12:42
SergeyLukjanovsergmelikyan, yup it's known, waiting for fungi12:42
fungihrm, looking12:43
*** isviridov_away is now known as isviridov12:43
*** hashar is now known as hasharAway12:43
jheskethyeah that looks bad12:44
sergmelikyanthx :)12:45
*** maishsk has joined #openstack-infra12:45
fungii suspect it's the jenkins-gearman bug clarkb's working on fixing, but manifsted in a much worse way than we'd seen yet12:45
*** niska has joined #openstack-infra12:47
fungiit'll take a bit to get these workers replaced and readded, but things should start picking back up again shortly12:47
*** aufi has joined #openstack-infra12:47
*** spredzy is now known as spredzy|afk12:47
jheskethfungi: if it's jenkins-gearman wouldn't that only be the jenkins masters?12:48
fungiyes, it does12:48
*** notnownikki has joined #openstack-infra12:48
*** mattt has joined #openstack-infra12:48
fungithe workers end up in an odd state where they don't start running new jobs even though they're ready to do so12:48
jheskethfungi: so possibly easiest to flush nodepool?12:49
fungiand as we cycle through workers, we slowly build up more and more in a ready state which never run jobs12:49
fungijhesketh: great idea--that's what i'm doing!12:49
jheskeththat would explain the high number of test nodes12:49
fungiit just takes a few minutes to delete them all, and a little more to boot replacements for them12:50
* jhesketh is confident in fungi 12:50
fungiwhen there are >600 to churn through12:50
jheskethlet me know if I can do anything to help12:50
fungithanks--also why are you awake?12:50
fungi's gotta be late there12:50
*** ssam2 has quit IRC12:51
jheskethnearly 11pm12:51
jheskethso not too bad12:51
jheskethI can do nights, it's the 5am mornings that I suck at12:51
jhesketh(the time of the infra-meeting)12:51
fungiindeed. and that's tomorrow for you... get some rest!12:51
*** ssam2 has joined #openstack-infra12:51
*** links has quit IRC12:52
jheskethnah this is when I'm productive ;-)12:52
jheskethfungi: is this the change you think is relevant https://review.openstack.org/#/c/179988/112:53
fungijhesketh: looks like it! he apparently uploaded it after i crashed for the night12:54
fungithat's a pretty significant refactor12:54
jheskethyeah and my java foo is weak12:54
jheskethbut I'll take a look12:54
fungithough a lot of it's just indents12:54
fungiyeah, mine too :/12:54
jheskethhas anything changed that might have caused this to manifest?12:56
AJaegerthe commit message looks fine ;)12:56
fungiwhat i find odd is that it's just really gotten obvious in the past couple weeks... i don't think we've upgraded or reconfigured anything which should have triggered it12:56
jheskethyeah that's what I'm wondering12:56
AJaegerMore load - or faster machines?12:56
fungiunless there were recent patches to gear or zuul which changed timing around some calls and i didn't spot them12:56
jheskethyeah I was also thinking possibly related to the network issues we had a week ago12:57
jheskethI try to keep an eye on those projects and don't recall anything12:57
fungigear's seen no new commits since late march12:57
fungieven zuul's been nearly a month with nothing merged to it12:57
*** vhoward has quit IRC13:00
*** NithyaG has joined #openstack-infra13:00
*** vhoward has joined #openstack-infra13:00
*** radez_g0n3 is now known as radez13:00
AJaegerfungi, for some time later, a few words of wisdom are needed for https://review.openstack.org/173408 - could you push this on your review queue, please?13:01
* AJaeger doesn't know what's the best way forward is13:01
fungihrm, jenkins03 also seems mildly unhappy13:01
fungior could be this browser13:02
*** Krinkle|detached is now known as Krinkle13:02
*** tiswanso has joined #openstack-infra13:04
*** pabelanger has joined #openstack-infra13:05
*** pabelanger has joined #openstack-infra13:05
*** armax has joined #openstack-infra13:05
*** eharney has joined #openstack-infra13:05
*** changbl has quit IRC13:08
*** miqui has joined #openstack-infra13:10
*** dustins has joined #openstack-infra13:11
*** sarob has joined #openstack-infra13:12
sdaguefungi: so... is there a flush on gearman now? or are things still stuck for a while13:16
sdaguejhesketh: thanks for the last round of reviews on the hideci stuff13:16
*** Krinkle is now known as Krinkle|detached13:17
jheskethyou're welcome13:17
*** BharatK has quit IRC13:17
mesteryGood morning infra folks!13:19
mesteryI have a question on this review (https://review.openstack.org/#/c/179305/), and similar ones for the other neutron service repos13:19
mesteryLooks like it's trying to merge stable/kilo into master13:19
mesteryI have no idea why, and this is really odd. Any ideas what would trigger the bot to do this?13:19
mesterydhellmann: ^^^^13:19
mesteryI'm just going to abandon those patches I think.13:20
claudiubhello. I have another question: how do I create a new branch on a stackforge project? (aka stable/kilo branch)13:20
mordredmestery: so - we started tagging on stable rather than proposed branches this time13:21
mesterymordred: Ack, yes, I recall that13:21
openstackgerritDavide Guerri proposed openstack-infra/shade: Add port resource methods  https://review.openstack.org/17996813:21
mordredmestery: looking at your repos real quick13:21
mesterymordred: Thanks!13:21
*** gyee has quit IRC13:22
*** bswartz has quit IRC13:23
*** freerunner has joined #openstack-infra13:23
mordredmestery: ok. I honestly have no clue - still waking up ... fungi ^^ ?13:24
*** sabeen1 has joined #openstack-infra13:24
mesterymordred: Thanks for looking for me! I'll see what fungi may know.13:24
mesterymordred: For now, I put -1s on them, I'll just put -2s for now to block them until we sort this out.13:24
mordredor ttx may remember the status13:25
dguerrisomething funny is happening to shade functional tests... check-shade-dsvm-functional-neutron and check-shade-dsvm-functional-nova just won't start13:25
fungisdague: i've deleted all the nodes which had been in a ready state for at least an hour. a lot of them are in a delete state now and we've got a lot of new ready nodes but only about 70 seem to be running jobs at the moment13:25
*** marcusvrn has joined #openstack-infra13:25
*** dkranz has quit IRC13:26
dguerrithey have been queued for more than 6 hrs now13:26
*** marcusvrn1 has quit IRC13:26
sdaguefungi: yeh, things seem to remain pretty gummed up13:26
fungimestery: that's a release tag merge. you see them on all our integrated release repos every 6 months13:26
*** bknudson has joined #openstack-infra13:26
*** gyee has joined #openstack-infra13:27
mesteryfungi: It was weird, because it looks like it's merging phantom changes, but if you say it's safe to go in, I'm with you :)13:27
fungimestery: gerrit displays them oddly. checkout that merge commit and diff it against the branch tip13:27
mesteryfungi: Cool, I'll try that!13:27
*** dkranz has joined #openstack-infra13:27
*** mikeit has joined #openstack-infra13:27
sdaguefungi: once things get cleaned, should jobs get scheduled again, or are we going to need to do a bunch of rechecks?13:29
*** mase_x200 has joined #openstack-infra13:29
sdaguetrying to figure out if approving patches at this point is useful at all13:29
fungisdague: they'll run. it's just right now we have very few workers picking up jobs. i'm going to start going through and quiescing jenkins masters and restarting them to see if that helps13:30
*** jtriley has joined #openstack-infra13:30
mesteryfungi: So just to be clear, it's ok to merge those then?13:31
*** BharatK has joined #openstack-infra13:31
*** prad has joined #openstack-infra13:32
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Move writexml fix for older minidom to same class as caller  https://review.openstack.org/18015013:32
fungimestery: yes, it's so that things like git describe will look right on master now13:32
mesteryfungin: Ack, thanks!13:32
fungimestery: it should be bringing in no code changes at all, just merging in the commit history of the 2015.1.0 tag13:32
*** prad_ has joined #openstack-infra13:32
mesteryfungi: Yes, that's what it's doing, it just looked odd. I pinged dhellmann earlier, even he was confused a bit :) Thanks for clarifying!13:33
fungiso that git tag --contains 2015.1.0 will return both stable/kilo and master rather than just stable/kilo (because we tagged it after we branched from master)13:33
*** signed8bit has joined #openstack-infra13:33
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Move writexml fix for older minidom to same module as caller  https://review.openstack.org/18015013:33
dhellmannoh, so we want that? it just looked weird13:33
jeblairfungi: pls delay restarting jenkins0313:34
fungijeblair: will do13:34
jeblairfungi: (i'm investigating on-host there)13:34
fungijeblair: awesome. i'll start up at 07 anyway13:34
*** prad_ has quit IRC13:36
*** pc_m has joined #openstack-infra13:36
*** BharatK has quit IRC13:36
*** peristeri has joined #openstack-infra13:36
fungii would just try toggling the jenkins-gearman plugin enable state but not sure if that's safe while jobs are running13:37
*** mugsie_ is now known as mugsie13:38
*** jyuso1 has joined #openstack-infra13:39
*** signed8b_ has joined #openstack-infra13:39
*** signed8bit has quit IRC13:40
*** tonytan4ever has joined #openstack-infra13:40
claudiubhello. I have another question: how do I create a new branch on a stackforge project? (aka stable/kilo branch)13:41
*** jyuso has quit IRC13:41
*** mase_x200 has quit IRC13:42
*** Ala has quit IRC13:42
*** mase_x200 has joined #openstack-infra13:44
fungiclaudiub: make sure your gerrit account is a member of a group granted create permissions in the refs/heads/stable/* section of your repo's acl13:45
jeblairfungi, jhesketh, mordred, zaro, clarkb: i have a theory for the triggering event: the updates around april 28 which changed gerrit-git-prep necessitated updates to all 5500 jenkins jobs.  that exceeds the puppet timeout for that exec, so it never completes.  therefore each time it's triggered (by a config change) it tries to update all jobs again13:46
*** dboik_ has quit IRC13:46
fungiooh, i didn't think to look at whether jjb is spinning constantly13:46
openstackgerritLouis Taylor proposed openstack-infra/project-config: Make glanceclient-dsvm-functional gating  https://review.openstack.org/17941513:47
fungithat could be putting the api under some strain13:47
jeblairfungi, jhesketh, mordred, zaro, clarkb: this suggests the problem has always been present, but we're seeing it now because we are triggering those updates so often which causes registerjobs to be called more often13:47
fungii can do manual jjb updates on each master to get that back under control13:47
jeblairfungi: yeah, i think that will help; i'm running one on 03 but nowhere else13:48
jeblairwe also should see about increasing the timeout there...13:48
jeblairfungi: be sure to run yours with "time" so we know the current runtime for a full update13:48
*** Ala has joined #openstack-infra13:48
fungiwill do13:48
*** BharatK has joined #openstack-infra13:49
jeblairfungi: (also, it's not constant, it only happens when project-config changes.  however, that is _many_ times per day, so is "constant" for some values of "constant")13:49
fungitiming one starting on jenkins06 right now13:49
*** claudiub_ has joined #openstack-infra13:50
jeblairfungi: and finally, this is going to make the immediate problem worse while this runs, so we'll need to keep deleting stuck ready nodes13:50
fungii am13:50
* jeblair goes back to reviewing clarkb's patch13:51
fungii've been periodically re-firing it to clear our more ready nodes13:51
*** mriedem has joined #openstack-infra13:51
fungithanks--i started looking at it but tend to go cross-eyed looking at java so haven't finished wrapping my head around it yet13:51
jeblairfungi: do you understand 179988 to supercede 17892913:51
*** claudiub has quit IRC13:51
*** yfried is now known as yfried|afk13:52
fungijeblair: yes, i believe it does. 178929 was based on a previous hunch which got ruled out but i think he hasn't remembered to abandon yet13:52
*** bswartz has joined #openstack-infra13:56
*** Sukhdev has joined #openstack-infra13:56
*** yamahata has joined #openstack-infra13:56
*** e0ne is now known as e0ne_13:56
*** niska has quit IRC13:57
claudiub_fungi: I see. So after than, I just git push gerrit stable/kilo?13:57
claudiub_after that *13:57
claudiub_(after git checkout -b stable/kilo)13:58
*** fawadkhaliq has joined #openstack-infra13:59
*** e0ne_ is now known as e0ne13:59
jheskethjeblair: good find13:59
jheskethis there anything I can do to help you guys?13:59
*** e0ne is now known as e0ne_13:59
*** e0ne_ is now known as e0ne14:00
fungiclaudiub_: then you should be able to create branches in the gerrit webui by browsing to projects->list->(filter for your repo name)->branches14:00
*** e0ne is now known as e0ne_14:00
*** e0ne_ is now known as e0ne14:00
anteayajhesketh: my motivation for my earlier ping was to get eyes on this fire when it started14:01
anteayaas I couldn't do anything myself14:01
*** wenlock has joined #openstack-infra14:01
fungiclaudiub_: due to quirks of gerrit acls i think pushing to create a branch requires push --force permission, which starts to get into foot-cannon territory14:01
*** ildikov has quit IRC14:01
jheskethanteaya: ah okay, sorry I wasn't around (although not sure I could have helped much anwyay014:02
anteayajhesketh: you can't _always_ be around, I took a chance14:02
*** pcm_ has joined #openstack-infra14:02
*** ayoung has joined #openstack-infra14:03
*** sigmavirus24_awa is now known as sigmavirus2414:03
anteayamoral support counts though14:03
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds refs/heads/stable/* to networking-hyperv  https://review.openstack.org/18016014:03
*** niska has joined #openstack-infra14:05
claudiub_fungi: I see, thanks for the info! :D14:05
*** pc_m has quit IRC14:06
*** zz_jgrimm is now known as jgrimm14:06
*** esker has joined #openstack-infra14:06
*** craigbr has joined #openstack-infra14:06
*** ajmiller has joined #openstack-infra14:07
*** dboik has joined #openstack-infra14:09
*** dboik has quit IRC14:09
mordredjeblair: ah! that explains why we're seeing it all of a sudden14:10
*** dboik has joined #openstack-infra14:10
*** gokrokve has joined #openstack-infra14:10
*** yfried|afk is now known as yfried14:12
*** mpavone has joined #openstack-infra14:13
*** otter768 has joined #openstack-infra14:14
*** asselin has joined #openstack-infra14:15
*** sabeen1 has quit IRC14:15
*** fhubik is now known as fhubik_afk14:16
*** ildikov has joined #openstack-infra14:17
*** freerunner has quit IRC14:18
fungiclarkb: when you're around, are you done poking at jenkins01? can i restart it and bring it back into operation now?14:18
*** armax has quit IRC14:18
*** otter768 has quit IRC14:18
*** lifeless has quit IRC14:20
*** wenlock_ has joined #openstack-infra14:20
* jhesketh heads off for some zzz's before the meeting14:21
*** tosky has joined #openstack-infra14:22
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Use plumbing rev-parse to get the branch name  https://review.openstack.org/17993714:22
openstackgerritAlexandre Viau proposed openstack-infra/project-config: New stackforge project: Bansho  https://review.openstack.org/18016614:23
*** panda has joined #openstack-infra14:23
jeblairclarkb: 179988 reviewed; i think it has a problem, but do agree the refactor is clearer.  :)14:24
pandawhat is the correct contact to notify a problem in gerrit ?14:25
jeblairalso, reviewing java takes 2x energy points14:25
jeblairpanda: right here14:25
*** lifeless has joined #openstack-infra14:27
pandajeblair: ok14:27
*** dangers_away is now known as dangers14:28
pandaI'm trying to update offline contact information, but I get a "server error: cannot store contact information"14:28
* mordred hands jeblair a box of extra energy points14:29
*** mrmartin has quit IRC14:29
*** annegentle has joined #openstack-infra14:29
jeblairpanda: see the instructions here: http://docs.openstack.org/infra/manual/developers.html#account-setup14:30
jeblairpanda: in particular, make sure your email address matches your foundation individual membership14:30
*** armax has joined #openstack-infra14:31
Shrewsjeblair: I still haven't recovered my lost energy points from writing the original WIP feature for gerrit14:31
* Shrews shudders at the memory14:31
*** mtanino has joined #openstack-infra14:32
*** aufi has quit IRC14:32
*** maishsk is now known as maishsk_afk14:33
*** amitgandhinz has joined #openstack-infra14:33
jeblairShrews: you might get them back in gerrit 2.11 which supports the WIP plugin :)14:34
SergeyLukjanovfungi, good morning, so, for the today's odd zuul behavior nodepool delete all ready slaves is the temp solution?14:34
*** nelsnelson has joined #openstack-infra14:34
fungiSergeyLukjanov: workaround, yes14:35
*** fhubik_afk is now known as fhubik14:35
openstackgerrityolanda.robla proposed openstack-infra/puppet-jenkins: Add option to send workers on jjb  https://review.openstack.org/18017314:35
fungiSergeyLukjanov: though as jeblair pointed out we're rerunning and then killing jenkins-jobs update over and over on the masters too, so that's probably what's causing it to expose this race so much recently. cleaning that up ought to limp us along while clarkb's jenkins0gearman fix gets polished14:36
anteayaI'm seeing more jobs running, gate queue doesn't have all the nodes it wants yet so check won't have any nodes available for a bit14:36
*** mattfarina has joined #openstack-infra14:36
fungithis is going to be a slow recovery but i suspect in the next few hours we'll have crawled back to normal throughput again14:36
anteayado you have to constantly restart jenkinses today?14:38
anteayaI still don't know the plural form14:38
*** btully has joined #openstack-infra14:38
openstackgerritAlexandre Viau proposed openstack-infra/project-config: New stackforge project: Bansho  https://review.openstack.org/18016614:39
anteayaall the gate patches have some jobs running now, hopefully the check queue will start to get nodes soon14:39
fungifirst declension nominative, jenkinsae14:40
anteayanever would have arrived at that14:40
anteayathank goodness for your latin14:40
* anteaya looks up declension14:40
*** emagana has joined #openstack-infra14:40
* fungi promises that will be his only latin grammar joke for today14:40
* anteaya bets $5 it won't be14:41
fungiitym $v14:41
fungi(didn't promise to obstain from roman number jokes on the other hand)14:42
fungiabstain either14:42
*** hichihara has joined #openstack-infra14:42
*** dboik_ has joined #openstack-infra14:42
*** gokrokve_ has joined #openstack-infra14:43
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Use 'http' to download war to allow urlopen use proxies  https://review.openstack.org/18017814:43
anteayaha ha ha14:43
*** sdake_ has joined #openstack-infra14:43
anteayaanyone have a sanskrit comment?14:45
*** dboik has quit IRC14:45
Davieycaput inter nubilia condo14:46
*** gokrokve has quit IRC14:46
anteayathat's sanskrit?14:46
anteayathe stars in the apartment have gone out14:47
*** sdake has quit IRC14:47
*** hasharAway is now known as hashar14:47
*** sputnik13 has joined #openstack-infra14:48
*** coolsvap is now known as coolsvap|afk14:48
pandajeblair: solved, thanks. (at the time of my first contribution foundation membership was not required)14:50
*** sputnik13 has quit IRC14:50
jeblairmordred: 175891 left me confused; see comment.14:52
clarkbjeblair: so I removed the comment about sleeping because in this case it goes directly on the queue that work() processes so should be handled 100% of the time. I believe that is what we want for the post initialize and post work fail due to unknown function cases but not the post register functions case14:53
clarkbjeblair: I can switch post register back to using the session event queueing which will check the state, and keep the other two as a regular enqueue14:53
*** hdd__ has quit IRC14:54
*** hdd__ has joined #openstack-infra14:54
*** mattfarina has quit IRC14:55
zxiirofor a matrix job http://ci.openstack.org/jenkins-job-builder/project_matrix.html14:55
zxiirois it possible to pass a variable to the value?14:55
jeblairclarkb: oh, then the other half of the comment is incorrect because it says that it's processed by the gearman session.  in that case, we'll end up sending lots of grab_job packets which means we'll get extra assigned jobs.14:55
*** notnownikki has quit IRC14:55
*** mattfarina has joined #openstack-infra14:55
zxiiroi want my "project" definition to pass a variable containing the values14:55
jd__if someone has 5 minutes, it'd be cool to git branch -d 1.0 in openstack/gnocchi which I created by mistake14:55
jd__no urgency though :)14:55
clarkbjeblair: ya so I think we need both cases, one for when we first init and post fail, and the other for the general hey we updated, do a grab job if necessary14:56
jeblairclarkb: ++14:56
*** hdd__ has quit IRC14:56
jeblairclarkb: (agreed; i can't think of a way to combine them, so will need minor duplication)14:56
*** fhubik has quit IRC14:56
anteayaand a check job just got a node14:57
*** imcsk8 has joined #openstack-infra14:57
*** abregman has quit IRC14:58
*** Sukhdev has quit IRC14:58
*** reed_ has joined #openstack-infra14:59
openstackgerritJerome Hourquebie proposed stackforge/python-jenkins: Adds Cloudbees folder plugin support This patch adds cloudbees folder plugin support. Folder should be specified in job name : <folder>/<job> This update also extend job_builder (http://ci.openstack.org/jenkins-job-builder/) tool without any modification  https://review.openstack.org/18018515:00
*** sdake_ is now known as sdake15:01
*** achanda has joined #openstack-infra15:01
openstackgerritClark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering  https://review.openstack.org/17998815:03
clarkbjeblair: something like ^15:03
*** hichihara has quit IRC15:03
*** vhoward has left #openstack-infra15:03
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds refs/heads/stable/* to compute-hyperv  https://review.openstack.org/18018715:04
*** erikmwilson is now known as Guest3297015:04
*** erikmwil_ has joined #openstack-infra15:04
*** Guest32970 has quit IRC15:04
*** erikmwil_ is now known as erikmwilson15:04
*** erikmwilson_ has joined #openstack-infra15:04
*** mpavone has quit IRC15:04
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Support copyartifact build selector param  https://review.openstack.org/13377415:05
*** achanda has quit IRC15:06
claudiub_AJaeger: Hello, If I may disturb you once more, could you take a look at another 2 commits for project-config? It adds rights for creating stable branches on networking-hyperv and compute-hyperv: https://review.openstack.org/#/c/180160/ https://review.openstack.org/#/c/180187/ Thanks!15:06
AJaegerclaudiub_, will do later...15:08
jaypipesclarkb: is something up with etherpad.openstack.org or is it just me? Keeps reconnecting...15:08
claudiub_AJaeger: ok. :)15:08
clarkbjaypipes: it can do that if you have the same pad open in multiple tabs, I would check that first, if that isn't the case let me know what our etherpad is and I can check logs15:09
anteayajaypipes: what browser are you using?15:09
jaypipesah, clarkb that may be it...15:09
jaypipesanteaya: chromium15:09
*** mrunge has quit IRC15:09
anteayareally, I never had a problem with chromium15:10
*** shashankhegde has joined #openstack-infra15:10
openstackgerritLouis Taylor proposed openstack-infra/project-config: Make glanceclient-dsvm-functional gating  https://review.openstack.org/17941515:12
*** jaypipes has quit IRC15:13
*** dboik_ has quit IRC15:13
*** gyee has quit IRC15:13
*** tonytan4ever has quit IRC15:13
*** dboik has joined #openstack-infra15:13
*** mase_x200 has quit IRC15:13
*** abregman has joined #openstack-infra15:13
*** gyee has joined #openstack-infra15:14
zxiiroFigured it out. the answer is to use {obj:value}15:15
*** asselin has quit IRC15:15
claudiub_sdague: Hello. If it's not much to ask, could you take a look at a few project-config commits? They're a few, only 1-5 lines long, similar in purpose, etc. Thanks! https://review.openstack.org/#/q/status:open+owner:%22Claudiu+Belu%22+project:openstack-infra/project-config,n,z15:16
*** yfried is now known as yfried|afk15:18
clarkbfungi: are you ok with holding 01 a bit longer? I want to get a thread dump from it, may try to fix the bug in that plugin now that I am writing java anyways15:18
clarkbI am going to toggle these ready nodes on jenkins nodes by disconnect/connect ing them15:19
*** arxcruz has quit IRC15:21
*** claudiub_ has quit IRC15:21
clarkbmaybe, that doesn't seem to have done much for a bare-(centos6|precise|trusty) node15:22
*** Ala has quit IRC15:24
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Switch to requests to support proxying of 'https'  https://review.openstack.org/18017815:25
fungiclarkb: yeah, i haven't done anything with 01 other than disabling puppet agent and starting an update of its job configs15:26
*** mpaolino has joined #openstack-infra15:26
*** nelsnelson has quit IRC15:26
fungiwhich already completed. i've reenabled puppet agent on it now15:27
fungijeblair: point of interest, on jenkins01 where the service was already quiesced, wall clock time for a full jjb update was a mere 9m14.676s15:27
fungithe rest are taking way, waaaay longer15:28
clarkbcool, going to take a thread dump now will let you know when that is done15:28
clarkbfungi: likely because it doesn't register any jobs when in shutdown mode15:28
*** abregman has quit IRC15:28
*** shashankhegde has quit IRC15:28
fungioh, great point15:28
clarkbwhen we do it with gearman enabled its a bunch of network data15:28
jeblairfungi: gtk15:28
*** freerunner has joined #openstack-infra15:28
fungii'll fire off a second one later when we restart the service15:28
jeblairclarkb: yeah, the regstration churn is huge15:28
*** yfried|afk is now known as yfried15:29
*** shashankhegde has joined #openstack-infra15:29
*** Swami has joined #openstack-infra15:29
*** yamahata has quit IRC15:30
clarkbjeblair: ya when I was tcpdumping I noticed that our window size would fall to zero regularly15:30
clarkblooked like registraration was to blame as we essentially fill the pipe for a bit15:30
anteayaso for anyone who cares, I just downloaded firefox 37.0.2 and don't have a reconnection problem so far with one tab in firefox and one tab in chromium at the same time15:31
anteayareconnected if I open a second tab in firefox though, which I expected15:32
*** dims_ has joined #openstack-infra15:32
*** dboik has quit IRC15:32
*** dizquierdo has quit IRC15:33
anteayathis is etherpad, sorry I didn't say that15:33
*** mpaolino has quit IRC15:34
*** dims has quit IRC15:34
clarkbanteaya: thats good news, so possibly related to 37.0.1?15:34
clarkbfungi: I have the thread dump now, feel free to restart 0115:34
fungiclarkb: thanks. doing!15:35
anteayaclarkb: that is the idea I'm tying myself to for the moment15:35
anteayaclarkb: only one datapoint so weak right now15:36
anteayano idea how to explain the safari and chromium reconnections15:36
*** dannywilson has joined #openstack-infra15:36
anteayaunless jaypipes did have more than one chromium tab open for the same etherpad15:36
openstackgerritSergey Kolekonov proposed openstack-infra/project-config: Add apps-catalog/ci group for apps-catalog project  https://review.openstack.org/18020115:36
*** maishsk_afk is now known as maishsk15:37
*** craigbr has quit IRC15:37
*** freerunner has quit IRC15:37
*** nelsnelson has joined #openstack-infra15:37
*** sdake has quit IRC15:37
*** freerunner has joined #openstack-infra15:38
*** yfried is now known as yfried|afk15:39
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade  https://review.openstack.org/17589115:40
*** samuelBartel has quit IRC15:40
*** sdake has joined #openstack-infra15:40
*** mwagner_lap has quit IRC15:40
*** dannywilson has quit IRC15:41
*** abregman has joined #openstack-infra15:41
*** dannywilson has joined #openstack-infra15:42
*** freerunner has quit IRC15:42
mordredjeblair: thank you- very helpful feedback - I added docstrings, renamed the param so that the boolean isn't confusing, and added unittests for it15:43
anteayathe 37.0.2 release notes leave me with the feeling I'm missing something: https://www.mozilla.org/en-US/firefox/37.0.2/releasenotes/15:44
mordredSpamapS: if you want to take a look at that ^^ and tell me how I should write the tests with mock instead of monkeypatch, I'd love to learn15:44
*** reed_ has quit IRC15:45
*** changbl has joined #openstack-infra15:45
*** dboik has joined #openstack-infra15:47
*** hemnafk is now known as hemna15:48
hemnahey guys, any chance I can get someone to +A this guy for me today?  https://review.openstack.org/#/c/177372/15:48
*** jcoufal has quit IRC15:48
*** smccully has joined #openstack-infra15:49
*** yamahata has joined #openstack-infra15:51
*** freerunner has joined #openstack-infra15:53
*** maishsk has quit IRC15:53
*** jlanoux has quit IRC15:54
*** whoops has joined #openstack-infra15:54
*** armax has quit IRC15:54
*** tosky has quit IRC15:55
anteayacan someone else look at the gate please? http://status.openstack.org/zuul/15:56
anteayawhy didn't the gate reset on that devstack failure?15:56
anteayathe whole gate is finished and I had been thinking it is calculating a reset15:57
cineramaAJaeger, around?15:57
anteayabut it has been for some minutes already15:57
*** bhunter71 has joined #openstack-infra15:58
*** scheuran has quit IRC15:58
*** MarkAtwood has joined #openstack-infra15:58
*** yfried|afk is now known as yfried15:59
anteayagerrit is still broadcasting events15:59
*** rloo has joined #openstack-infra15:59
*** tosky has joined #openstack-infra16:00
fungirerunning a full jjb update on jenkins01 a few minutes after it was restarted took 2m36.326s wall clock time, though that was after it had already been previously caught up16:00
anteayafungi: can you take a look at teh gate queue?16:01
*** chlong has joined #openstack-infra16:01
fungianteaya: yes, i have been since you mentioned it16:01
anteayaah thanks16:01
fungilooks like events and results are slowly climbing16:01
*** dboik_ has joined #openstack-infra16:01
fungiwhich may still indicate a problem16:01
anteayaI don't understand why the whole gate finished, it should have reset at the first failure16:02
*** mikeit has quit IRC16:02
gordc... is bob hansen a bot?16:02
anteayagordc: what channel?16:02
jeblairclarkb: have you run your latest patch?16:02
*** ildikov has quit IRC16:03
gordcanteaya: https://review.openstack.org/#/q/reviewer:hansenb%2540us.ibm.com+status:open,n,z16:03
*** tonytan4ever has joined #openstack-infra16:03
gordcanteaya: seems like it's mimicing jenkins.16:03
*** freerunner has quit IRC16:03
anteayagordc: yup, thanks for letting me know, will get that account disabled16:03
anteayathat kind of activity is verboten16:04
*** dboik has quit IRC16:04
*** hashar is now known as hasharMeeting16:04
*** freerunner has joined #openstack-infra16:04
gordcanteaya: ok, thanks! i wasn't sure what was happening. seems to be tracking a bunch of recent patches16:04
clarkbjeblair: not beyond `mvn package` which builds it then runs the built in tests. Seems to not be happy with the test start stop thread test, but I thought it was likely a race (traceback isn't too useful)16:04
*** craigbr has joined #openstack-infra16:05
clarkbstill waking up, need to go make breakfast soon, but will get back to it after that16:05
anteayagordc: thanks for asking, humans only to comment on human accounts, we have lots of pathways for bots and that isn't one of them16:05
*** craigbr has quit IRC16:05
*** freerunner has quit IRC16:05
rloothanks for handling Bob Hansen, I was wondering about it! :)16:05
jeblairclarkb: okay, i'm not sure there are any _actual_ tests on this; last time i used it, i ran it along with a geard and a simple script to trigger jobs16:06
jeblairclarkb: i think that might be worthwhile; i'm starting to feel like a java runtime :)16:06
anteayacan someone please disable gerrit account 15674: http://paste.openstack.org/show/215011/16:06
anteayarloo: thanks :)16:07
*** ddieterly has joined #openstack-infra16:07
*** freerunner has joined #openstack-infra16:08
mordredjeblair: yay for no tests16:08
ddieterlyis zuul sick?16:09
anteayaddieterly: yes16:09
ddieterlyok, thanks16:10
fungiwatching its logs, it seems to be starting jobs and getting results16:10
fungithe server itself is not loaded heavily16:10
anteayawe hope in a few hours to have a better flow, if we can catch up16:10
fungithe zuul-merger workers are basically idle16:10
anteayamordred: have you disabled a gerrit account before?16:11
anteayamordred: would you like to try with gerrit account id 15674?16:11
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add migration to convert append only tables to MyISAM  https://review.openstack.org/17996416:11
fungithe results count seems to be catching up some now16:11
mtreinishmordred: ^^^ I'm not sure how bad that is16:11
mtreinishI just played copy and paste with the one we put together to add the column16:12
mordredmtreinish: oh, crappit. I forgot to look at your thing last night16:12
mordredanteaya: I'm in a meeting thing, so doing things that require attention is a bad idea16:12
mtreinishmordred: no worries, it's not super important16:12
anteayamordred: ah yes, a bad idea okay thanks16:12
mtreinishwe still ahve plenty of space on the db (I think)16:13
*** unicell has quit IRC16:13
*** dguitarbite has quit IRC16:14
*** ashleighfarnham has joined #openstack-infra16:14
*** bknudson has quit IRC16:14
openstackgerritFabien Boucher proposed openstack-infra/puppet-zuul: Attempt to add an acceptance test with beaker for puppet-zuul  https://review.openstack.org/18021416:14
*** otter768 has joined #openstack-infra16:15
fungijeblair: clarkb: seeing quite a few of these tracebacks in the zuul log http://paste.openstack.org/show/215013/ collecting stats now to see if it's recent and to gauge their frequency16:15
jeblairclarkb: okay, i think i found 2 problems with simple solutions16:15
anteayapleia2: let me know if you are around to disable gerrit account id 1567416:16
fungilooks like we normally get a few a day, but today we have ~500 and counting16:16
EmilienMnibalizer: you may want to see https://review.openstack.org/18021416:16
*** dboik_ has quit IRC16:17
anteayafungi: that is a few more than usual16:17
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Reorder imports to match hacking guidelines  https://review.openstack.org/13346516:17
*** dboik has joined #openstack-infra16:17
cineramapleia2: hi16:17
fungilooks like it climbed steadily from when the logs rotated at 07:00 to present time16:17
fungiso possibly related to the bug under discussion16:18
*** chlong has quit IRC16:18
*** pblaho has quit IRC16:18
anteayajeblair: I'm curious about your problems and solutions16:18
*** rloo has quit IRC16:18
anteayaunless you want to wait for clarkb16:18
anteayathe gate seems stuck16:19
jeblairanteaya: feel free to look at the patch if you are curious16:19
anteayajeblair: ah okay thanks16:19
anteayafungi: do you think the gate will move at some point?16:19
*** otter768 has quit IRC16:20
anteayamorning zaro16:21
anteayazaro: you know java16:21
fungianteaya: at some point, yes. not sure yet whether the current situation is slowing things down to the point where zuul is having trouble handling its event queue as quickly as it's being filled or what. it does seem to be logging that new jobs are starting and running jobs are completing16:21
anteayazaro: this patch could use your review: https://review.openstack.org/#/c/179988/216:21
zaroanteaya: thanks.  i'm aware of the problem.  going thru backlog now.16:22
anteayazaro: k16:22
*** freerunner has quit IRC16:22
anteayafungi: the gate hasn't move in over 20 minutes16:22
mordredanteaya: yah - it's a bit of a bad one / all hands on deck to solve it16:22
fungianteaya: its waiting job queue was up over 2500 earlier and that has been steadily dropping (almost down to 1000 now)16:22
anteayafungi: okay16:23
*** isviridov is now known as isviridov_away16:23
*** annegentle has quit IRC16:23
anteayafungi: I guess I can't see that number16:23
fungianteaya: graph at the bottom of the status page16:23
fungialso in graphite16:23
*** maishsk has joined #openstack-infra16:23
*** sputnik13 has joined #openstack-infra16:23
fungii have a feeling the gate reset processing is slow because of sheer size of the backlog we accumulated16:24
anteayaI'll try my hand at graphite16:24
anteayafungi: okay16:24
*** annegentle has joined #openstack-infra16:25
*** jistr has quit IRC16:26
*** maishsk has quit IRC16:26
fungilooks like we haven't turned zuul status queries back on in hideci.js, so i don't think it's that16:26
*** salv-orlando has quit IRC16:27
*** salv-orlando has joined #openstack-infra16:27
*** annegent_ has joined #openstack-infra16:27
*** annegentle has quit IRC16:28
jeblairfungi: those jjb runs still continuing?16:28
jeblairfungi: because i think geard is cpu bound at this point16:28
anteayafungi: thank you now I can see the numbers you see16:28
*** baoli has quit IRC16:29
jeblairfungi: mine from earlier is still chugging along very slowly16:29
fungijeblair: yep, all but 01 which ran while jenkins was stopped and then very quickly reran once it was restarted16:29
*** tiswanso has quit IRC16:30
fungi~9m to run while jenkins01 was quiesced, and ~2 minutes to run again once it was started back16:30
*** alexsyip has joined #openstack-infra16:31
*** dboik_ has joined #openstack-infra16:31
*** tiswanso has joined #openstack-infra16:31
*** baoli_ has joined #openstack-infra16:31
fungijeblair: should we stop all but one, leave puppet disabled on them, and then update one at a time to completion before starting the next?16:31
*** tiswanso_ has joined #openstack-infra16:31
fungithat would keep us from beating up the geard quite so badly16:31
*** annegent_ has quit IRC16:33
jeblairfungi: not a bad idea; and you can do that without interrupting the process.  i'd probably do 2 at a time.16:33
jeblairor 4.16:33
fungijeblair: well, we have 6 underway, i can kill 3 of mine for noew16:34
*** dboik has quit IRC16:34
yolandahere, can i get some help? i need to know the meanings of the zuul times in status.openstack.org/zuul16:35
yolandafirst number is the estimated one, second one is the total elapsed?16:35
mordredyolanda: hey - in the middle of a bit of an outage - might be a few minutes16:35
*** tonytan4ever has quit IRC16:35
*** tiswanso has quit IRC16:35
fungi93m elapsed on jenkins07, 112m elapsed on jenkins02, 116m elapsed on jenkins04, now all stopped. still running on jenkins03 (yours), jenkins05 and jenkins0616:36
jeblairyolanda: yes, the second is time in queue16:36
yolandajeblair, it includes the waiting time as well?16:36
jeblairyolanda: yep16:36
*** jaypipes has joined #openstack-infra16:38
fungii'll resume 02, 04 and 07 once the others complete16:38
*** craigbr has joined #openstack-infra16:38
jeblairfungi: i accidentally restarted geard on zuul.16:38
fungioh, oops16:39
fungiis that going to need a scheduler restart as well?16:39
jeblairfungi: nah16:39
*** gokrokve_ has quit IRC16:39
fungiokay, just making sure16:39
*** ivar-lazzaro has joined #openstack-infra16:41
*** ivar-lazzaro has quit IRC16:41
*** ivar-lazzaro has joined #openstack-infra16:42
openstackgerritSwaminathan Vasudevan proposed openstack-infra/project-config: Make neutron-dvr job voting at the gate  https://review.openstack.org/18023016:42
jogois launchpad timing out for anyone else?16:43
nibalizerfungi: jeblair could I have a branch on openstack-infra/publications called 'puppet-overview' or similar ?16:43
anteayahttps://launchpad.net/ renders for me16:43
anteayajogo: do you mean login?16:43
jogoanteaya: filing a bug errors out16:43
nibalizerI have a deck from a talk I gave last night on the subject16:43
*** tim_o has joined #openstack-infra16:43
funginibalizer: branched from the tip of template?16:44
anteayajogo: just logged into the wiki16:44
anteayajogo: ah okay16:44
nibalizerfungi: yes16:44
*** freerunner has joined #openstack-infra16:44
fungianteaya: gerrit account 15674 is now disabled16:45
jogoanteaya: I think it was just launchpad failing as usual16:45
*** SumitNaiksatam has quit IRC16:45
funginibalizer: done16:45
anteayafungi: thank you16:45
*** sdake_ has joined #openstack-infra16:45
*** unicell has joined #openstack-infra16:45
*** Guest21373 has quit IRC16:46
anteayajogo: ah well then, don't let me interrupt your usual fun16:46
*** craigbr has quit IRC16:46
nibalizerfungi: thanks16:46
anteayaI expect that account to still be commenting for a while since we don't have close connection yet, but at least it is disabled now16:46
anteayaand we will soon have close connection16:46
*** freerunner has quit IRC16:46
jeblairfungi: i'm not sure my geard restart is happy; i'm thinking of doing a zuul restart16:46
*** freerunner has joined #openstack-infra16:47
anteayagordc rloo if you are able to check https://review.openstack.org/#/q/reviewer:hansenb%2540us.ibm.com+status:open,n,z tomorrow and let me know if you see anymore bot activity from bob16:47
*** freerunner has quit IRC16:47
fungijeblair: okay. it does look like zuul's job backlog has leveled off rather than continued dropping. i guess we're going to lose the last ~1.75 hours of gerrit events16:47
anteayagordc rloo the connection may still be open until it expires but it should be expired by tomorrow16:48
jeblairfungi: oh that's weird, it just bursted16:48
*** AJaeger_ has joined #openstack-infra16:48
*** sdake has quit IRC16:49
clarkbok back, I have cookies and hot tea16:51
anteayaclarkb: perfect for firefighting16:52
fungijeblair: looks like you went ahead and restarted the scheduler daemon?16:52
*** maishsk_ has joined #openstack-infra16:52
*** cody-somerville has quit IRC16:52
*** spzala has joined #openstack-infra16:52
*** doude has quit IRC16:52
anteayacheck is empty (well 2 now) but gate is still the same16:52
*** maishsk_ is now known as maishsk16:52
*** doude has joined #openstack-infra16:52
funginot any longer16:53
jeblairfungi: i put jenkins 03,05,06 in shutdown mode16:54
fungijeblair: okay, thanks. that will likely help speed this up16:54
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Puppet: don't run beaker jobs on previous stable branches  https://review.openstack.org/18023316:54
fungii'll delete all their ready nodes16:54
anteayathe gate just restarted but it didn't move anything16:54
jeblairfungi: thanks16:54
anteayathe top four patches are still the same as they were16:55
anteayaso I'm guessing the rest are too16:55
clarkbjeblair: I agree with your comments, what do you think about pulling the driveSessionIO out of submitFunction instead of using a continue after running the function?16:55
*** markus_z has quit IRC16:56
clarkbalso feel free it ignore me while fires are fought16:56
*** dustins has quit IRC16:56
*** mattfarina has quit IRC16:57
jeblairclarkb: that changes the ordering related to availability unlocking; unsure if that's important16:57
*** mattfarina has joined #openstack-infra16:57
*** mattfarina has quit IRC16:57
*** BharatK has quit IRC16:57
AJaeger_cinerama: I'm around now16:57
clarkbjeblair: good point, probably easiest for now to do what you suggest16:58
*** HeOS has quit IRC16:58
cineramaAJaeger_: oh awesome. did you see my question in my mail about the percentage completion? if it's per-project i can streamline some of the stuff i've done16:59
fungi?status notify zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing16:59
fungijeblair: ^ ?16:59
*** deepakcs has joined #openstack-infra16:59
*** ildikov has joined #openstack-infra16:59
fungiare you planning to requeue the prior check pipeline as well?17:00
fungiif not, i can adjust wording accordingly17:00
*** patrickeast has joined #openstack-infra17:00
deepakcsmtreinish, ping17:00
jeblairfungi: enqueing in progress17:00
AJaeger_cinerama: haven't checked mail again, will do now.17:00
fungithe jenkins prepare to shutdown has seriously sped up jjb updates17:00
jeblairfungi: lgtm (unsure about those times, but i don't think we can be)17:00
*** ivar-laz_ has joined #openstack-infra17:01
fungijeblair: i'm going on the age-in-queue of the oldest change in the check pipeline prior to the restart17:01
*** ivar-laz_ has quit IRC17:01
fungiwhich was about 2 hours ago now17:01
*** ivar-laz_ has joined #openstack-infra17:01
jeblairfungi: shouldn't it be newest in check?17:01
fungier, newest is what i meant17:02
jeblairfungi: sounds good then17:02
*** ivar-laz_ has quit IRC17:02
fungiyes. sorry for the brainbreak17:02
*** dustins_ has joined #openstack-infra17:02
fungibottom one in the pipeline on the status page17:02
*** ivar-laz_ has joined #openstack-infra17:02
fungi#status notify zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing17:03
openstackstatusfungi: unknown command17:03
*** maishsk has quit IRC17:03
fungi#status notice zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing17:03
openstackstatusfungi: sending notice17:03
*** dustins_ is now known as dustins17:03
reedare bugs for CI logged in Launchpad kept in sync with Storyboard?17:03
AJaeger_cinerama: I missed your question, thanks for asking back! I answered. Should I explain more?17:03
jeblairreed: the projects that moved to storyboard have moved; there is no syncing17:04
-openstackstatus- NOTICE: zuul has been restarted to troubleshoot an issue, gerrit events between 15:00-17:00 utc were lost and changes updated or approved during that time will need to be rechecked or have their approval votes readded to trigger testing17:04
*** dustins has quit IRC17:04
jeblairreed: but "openstack-gate" is still in launchad, and holds bugs related to CI17:04
reedjeblair, to be clear this for example: https://bugs.launchpad.net/openstack-ci/+bug/1265591 may already have been fixed, right? I should check on storyboard17:04
openstackLaunchpad bug 1265591 in OpenStack Core Infrastructure "jeepyb git url hard-coded to openstack" [Medium,Triaged] - Assigned to Sergey Lukjanov (slukjanov)17:04
*** ivar-lazzaro has quit IRC17:04
reedi'm looking for low-hanging-fruit bugs17:05
*** btully has quit IRC17:05
jeblairreed: yes, that should be in the jeepyb project on storyboard17:05
openstackstatusfungi: finished sending notice17:05
pleia2anteaya: sorry, had some other things come up this morning, available to do gerrit things now :)17:05
fungireed: it's possible. the bug numbers in storyboard should be a 1:1 correspondence to lp too17:05
*** e0ne has quit IRC17:05
anteayapleia2: fungi did it, thanks though17:06
jeblairreed: also, how did you find it?  https://bugs.launchpad.net/openstack-ci/ is fairly dead-endish17:06
*** hodos has joined #openstack-infra17:06
patrickeasti might be way late to the party, but is anyone else seeing or know about the devstack error with swift failing? is that related to the zuul restart?17:06
anteayapleia2: and I understand was just covering my options17:06
anteayapatrickeast: what error?17:06
pleia2anteaya: ok, next time! :)17:06
pleia2cinerama: hey17:06
patrickeastanteaya: ^17:06
anteayapatrickeast: if it occured in the last 8 hours, then yes probably it is17:06
hodosis ok for thirdparty ci not run all the 304 tests ?17:06
*** gokrokve has joined #openstack-infra17:07
cineramaAJaeger_: i did see that. the issue i'm having is that the transifex client itself also handles percentages. if you have a look at propose_translation_update.sh17:07
hodoscan you manually disable some of them ?17:07
anteayahodos: ask the project you are testing17:07
*** gokrokve has quit IRC17:07
cineramaAJaeger_: there's a minimum_perc flag17:07
patrickeastanteaya: ok, cool, i’ll wait and see if it goes away then17:07
anteayapatrickeast: so this is your ci system not ours17:07
fungianteaya: patrickeast: i can't imagine a third-party ci test error would be related to issues on our jenkins masters17:07
patrickeastanteaya: it is in mine, but http://bit.ly/1IdXRc7 shows it in jenkins too17:08
AJaeger_cinerama: brb17:08
anteayafungi: yeah, I didn't think so either17:08
*** harlowja has quit IRC17:08
fungipatrickeast: those in ours are potentially related. in yours they should not me17:08
fungier, should not be17:08
patrickeastfungi: makes sense17:09
*** harlowja has joined #openstack-infra17:09
anteayapatrickeast: you may have uncovered a new bug that is unrelated to our current fire17:09
fungipatrickeast: er, actually no even in ours they should not be. it looks like a bug encountered while testing, not a bug in uploading our logs to our swift container17:09
DuncanTCould anybody core for global requirements looks at https://review.openstack.org/#/c/177372/ please? It has one +2 and is holding up cinder work.... been there for a while with no objections, lots of +1s17:09
AJaeger_cinerama: we only sat the minimum_perc flag in file http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh17:10
AJaeger_cinerama: see line 108 (PERC variable) for openstack-manuals17:10
anteayaDuncanT: here is your list of people: https://review.openstack.org/#/admin/groups/131,members17:10
jeblairfungi: my jjb run on 03 is deleting lots of jobs, including ones in use17:10
fungithat sounds not right17:10
DuncanTanteaya: I pinged a bunch of them directly, only Monty got back to me (and +2d) :-(17:10
anteayaDuncanT: k, kind of fighting a fire in here17:11
cineramaAJaeger_: we also do in propose_* scripts17:11
anteayaDuncanT: ttx might still be around or markmcclain17:11
*** btully has joined #openstack-infra17:11
jeblairfungi: i'm removing the cache and restarting17:11
cineramaAJaeger_: are they not used?17:11
*** gokrokve has joined #openstack-infra17:12
AJaeger_cinerama: ah, now remembered details ;)17:12
jeblairjjb cache writing is unsafe for program interrupts :(17:12
AJaeger_So, for python projects we use tx pull --minimum-perc like in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/propose_translation_update.sh#n3717:12
*** gokrokve has quit IRC17:12
DuncanTanteaya: Thanks17:12
cineramaAJaeger_: so basically, if the percentages are per project rather than per file, i can rip out a bunch of potentially brittle code17:12
AJaeger_for openstack-manuals, we add to the config file17:12
fungiDuncanT: yeah, there's really not an irc channel for the openstack/requirements reviewers in aggregate. a lot of them may be hanging out in this channel, but the infra team generally refrains from weighing in on requirements changes unless they're infra-related17:12
AJaeger_cinerama: openstack-manuals is the only repo that uses per file, everything else is per project17:13
fungijeblair: yeah, i'm rerunning mine that just finished too17:13
cineramaAJaeger_: we have to do it a little differently for zanata because we don't have an equivalent of "tx set" and when we pull there is no built-in percentage option17:13
cineramaAJaeger_: so that has resulted in a bunch of work building a workaround for that17:14
cineramaAJaeger_: ah ok so for the manuals i will still need to calculate the percentage for each file?17:14
*** sdake_ has quit IRC17:14
*** freerunner has joined #openstack-infra17:14
*** yfried is now known as yfried|afk17:14
*** gyee has quit IRC17:14
AJaeger_cinerama: so, openstack-manuals has two files - the glossary and common - that use 8 %, the rest 7517:14
cineramaAJaeger_: the other hangup is that i don't seem to be able to pull an individual file from the project so i had to build that17:14
cineramaAJaeger_: but if it17:15
pleia2ooh, I didn't realize there was a 8% one17:15
*** SumitNaiksatam has joined #openstack-infra17:15
cineramas just for the two manuals then maybe i i can do it an easier way17:15
AJaeger_cinerama: let us talk in 30 mins or so, ok?17:15
cineramaAJaeger_: np17:15
anteayaDuncanT: welcome17:15
cineramapleia2: shenanigans!17:15
AJaeger_pleia2: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n11717:15
pleia2cinerama, AJaeger_ - also just sent out the email for meeting up at 0600 on May 6th17:15
*** achanda has joined #openstack-infra17:16
cineramapleia2: just trying to unblock so i can wrap up what i have17:16
*** btully has quit IRC17:16
fungiDuncanT: probably the most appropriate irc channel to use for requesting requirements reviews is #openstack-dev17:17
*** dboik_ has quit IRC17:17
pleia2cinerama: oh yes, please do!17:17
DuncanTfungi: I'll try there in future, thanks. anteaya and PMs has worked for now17:17
*** dboik has joined #openstack-infra17:17
*** shashankhegde has quit IRC17:18
jeblairfungi: running jjb on 03 again to make sure it's stable17:18
cineramapleia2: so i really want to get us out of the business of having to pull individual files because it's a bit hairy17:18
mriedemlifeless: according to http://docs.openstack.org/developer/pbr/semver.html#semantic-versioning-specification-semver item 9, shouldn't a pre-release version like 2015.1.1.a1 work on a project?17:18
AJaeger_cinerama: ok, that was a quick 30mins - I'm back...17:18
mriedemlifeless: like nova on stable/kilo17:18
gordcanteaya: will do.17:18
AJaeger_cinerama: percentages calculation is done here for example: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n35017:19
anteayagordc: thanks17:19
cineramapleia2: actually if you could cast a second set of eyes on "zanata-cli pull" just to double check i'm not missing somethign obvious17:19
AJaeger_So, really your call what is easier to do - filtering like that function after pulling or as part of zanata-cli17:20
*** kambiz has joined #openstack-infra17:20
pleia2cinerama: will do17:20
cineramaAJaeger_: yes, i've seen that code...i'm talking about this: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/propose_translation_update.sh#n3717:21
*** gyee has joined #openstack-infra17:21
*** liam_ has joined #openstack-infra17:21
jeblairfungi: it seems to be flapping and now deleting jobs again17:21
*** liam_ is now known as Guest9566017:21
AJaeger_cinerama: so, what's the issue with that parameter?17:21
cineramapleia2: so i don't think we can pull just the one file...it looks like we can start from a file and go in alphabetical order, or do includes17:21
cineramaAJaeger_: the problem is that we don't have it in zanata-cli17:22
cineramaAJaeger_: so i need to know exactly what it's doing so we can try to work around that17:22
*** baoli_ has quit IRC17:22
AJaeger_cinerama: then let's not do it. I do something like in the other function to replicate that.17:22
*** sergsh has quit IRC17:22
*** sergsh_ has quit IRC17:22
AJaeger_cinerama: If we just have "pull down all files", I can filter in bash like in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n35017:23
fungijeblair: yeah, i'm seeing similar behavior too17:23
AJaeger_cinerama: what that help you?17:24
cineramaAJaeger_: oh ok, i see - so you're saying we should switch to grabbing everything and using msgfmt?17:24
cineramaAJaeger_: rather than trying to do stuff with zanata17:24
AJaeger_cinerama: Yes, that's my suggestion.17:24
AJaeger_cinerama: Just offering suggestions since I hear that it's difficult with zanata...17:25
*** Krinkle|detached is now known as Krinkle17:25
cineramaAJaeger_: i think that's good actually - i was kind of getting focused on the zanata client's capabilities because i wasn't seeing the whole picture17:25
*** Somay has joined #openstack-infra17:25
pleia2it may even end up being faster than individual analysis through percentage statuses via REST pre-download17:26
cineramaAJaeger_, pleia2: yeah grabbing the stats from zanata is slow17:26
* pleia2 nods17:26
AJaeger_cinerama: we have the client and we have our scripts, together they form our framework and we can decide on where to do what we need17:26
cineramaAJaeger_, pleia2: simplifying the pull is a big win because it avoids us having to do crazy stuff to reassemble the po files17:27
AJaeger_cinerama: if grabbing the stats is slow, let's do a local msgfmt. It means more files to download...17:27
*** tnovacik has quit IRC17:27
cineramaAJaeger_, pleia2: so from my stuff we have the zanata.xml generation which is great17:27
AJaeger_cinerama: then let's go for that - and if we later figure out downloading is issue, we can change the architecture again17:28
cineramaAJaeger_, pleia2: so in between now and the meeting i will simplify what i have17:28
*** freerunner has quit IRC17:28
AJaeger_cinerama: go for it ;)17:29
AJaeger_cinerama: thanks a lot!17:29
pleia2cinerama: wonderful, thank you17:29
jeblairfungi: i'm resetting 03 and prepping to debug17:29
cineramaAJaeger_: no, thank you. your broader perspective on this has been hugely helpful. and i like deleting code that's going to be hard for others to maintain :)17:29
*** yfried|afk is now known as yfried17:30
*** Somay has quit IRC17:30
*** baoli has joined #openstack-infra17:30
AJaeger_cinerama: ;)17:31
*** tiswanso_ has quit IRC17:31
*** tiswanso has joined #openstack-infra17:32
*** pradk has joined #openstack-infra17:32
clarkbzaro: how do I get logs when running `mvn test`?17:32
*** dims_ has quit IRC17:33
*** mwagner_lap has joined #openstack-infra17:33
zaroclarkb: -X flag?17:34
*** dims has joined #openstack-infra17:34
zaroclarkb: reviewing your change now.  i ran build and didn't see any test failures.17:34
clarkbzaro: that appears to give me mvn's logs, I want logs generated by the tests17:35
*** Sukhdev has joined #openstack-infra17:35
*** Guest95660 has quit IRC17:35
*** freerunner has joined #openstack-infra17:35
clarkbzaro: the start stop thread fails for me, I think it may have to do with how my change handles interrupted exceptions, I could try jdb instead but logs seem so much simpler here17:37
*** maishsk has joined #openstack-infra17:39
*** dboik_ has joined #openstack-infra17:39
openstackgerritJames E. Blair proposed openstack-infra/gearman-plugin: Stop sending status updates  https://review.openstack.org/18024917:39
zaroclarkb: here are all the options for mvn test https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html17:40
jeblairclarkb, zaro: ^ while we're digging around in there, i think that would be a good change to land to help reduce zuul busy-work17:40
*** sdake has joined #openstack-infra17:41
*** dboik has quit IRC17:42
zaroclarkb: that failure existed before your change: http://logs.openstack.org/28/145828/1/check/gate-gearman-plugin-build/3b586a1/console.html.gz17:42
*** melwitt_ has joined #openstack-infra17:42
*** melwitt_ has quit IRC17:42
clarkbjeblair: I think you need to still loop until done right?17:43
zaroclarkb: i also get that failure on my build.17:43
clarkbzaro: ah ok, that makes me feel better17:43
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Make command line parsers available as modules.  https://review.openstack.org/18025217:45
jeblairclarkb: future.get should take care of that; the one with no args should mean "wait until complete"17:45
jeblairclarkb: (the call in the loop had a timeout)17:45
*** dustins has joined #openstack-infra17:45
clarkbjeblair: gotcha17:45
clarkbjeblair: lgtm17:45
jeblairclarkb: i think, we should double check that :)17:45
*** abregman has quit IRC17:46
zanebsdague: any chance of a re-review on https://review.openstack.org/#/c/174233/ ? Your comment was addressed and mordred gave +217:47
zanebwould be much appreciated :)17:47
fungijeblair: i cleared the cache and rebuilt the jenkins configs on 05 and 0617:47
openstackgerritDavide Guerri proposed openstack-infra/shade: Add floating IP pool resource.  https://review.openstack.org/18025517:48
*** AndroUser has joined #openstack-infra17:49
*** AndroUser has quit IRC17:49
jeblairclarkb: yeah, reread docs and i think what's there is sufficient17:49
jeblairfungi: does it flap if you re-run?17:49
*** shashankhegde has joined #openstack-infra17:49
jeblairfungi: or are you saying you are in a state ready for a bugfix? :)17:49
*** e0ne has joined #openstack-infra17:50
*** mrmartin has joined #openstack-infra17:50
*** e0ne is now known as e0ne_17:50
zarofungi: are we increasing timeout for jjb or want to make it upload jobs faster?17:50
fungijeblair: yeah, it flaps if i keep rerunning with --delete-old17:50
jeblairfungi: i believe i have confirmed a bug masked by an over-broad exception17:50
clarkbpossible new change to jjb breaking us?17:50
jeblairclarkb: yes, and i think i know the one... more soon17:50
*** e0ne_ is now known as e0ne17:51
fungialso 05 and 06 are available for further testing or to get thrown back to the gate wolves now17:51
zarohmm, big change that just landed was parallel uploading of jobs.17:51
*** chriscowley has joined #openstack-infra17:51
*** packet has joined #openstack-infra17:51
waynrruh roh17:51
*** maishsk is now known as maishsk_afk17:52
*** crc32 has joined #openstack-infra17:52
waynri was just preparing to test that out, hopefuly it's not too searious breakage17:52
*** ZZelle has quit IRC17:53
kambizjeblair: bswartz radez and I are working on a hardware refresh and will need assistance with DNS for trystack.org at some point in the not too distant future.  would you recommend emails to hostmaster@openstack.org for those sorts of things ?17:53
*** ZZelle has joined #openstack-infra17:53
chriscowleyHi all, first time in the infra meeting. Will be lurking to see what's up17:53
fungiwaynr: basic symptom is that if you run `jenkins-jobs update --delete-old /etc/jenkins_jobs/config` twice, the second run seems to delete all the jobs *at least while the jenkins service is quiesced)17:54
fungichriscowley: welcome--hopefully we won't disappoint17:54
waynrokay that doesn't seem like it will affect me yet...i haven't been using --delete-old17:55
lifelessmriedem: yes17:55
*** yfried is now known as yfried|afk17:55
lifelessmriedem: have you had an issue w/ it?17:56
mriedemlifeless: hmm, didn't seem to be working for me locally17:56
mriedemlifeless: yeah, couldn't use 2015.1.0.1 either17:56
lifeless2015.1.0.1 is definitely not supported17:56
mriedemyeah i expect not after reading the docs17:56
mriedemwould have to be like .a1 or something17:56
jeblairkambiz: probably better to ask folks in here, or send something to openstack-infra@lists.o.o17:56
lifelessmriedem: what error do you get?17:57
lifelessmriedem: or what symptoms are you seeing?17:57
*** tiswanso has quit IRC17:57
mriedemlifeless: http://paste.openstack.org/show/215036/17:58
*** deepakcs has quit IRC17:59
lifelessmriedem: oh, don't do that.17:59
lifelessmriedem: set the version in setup.cfg to 2015.1.1, if thats the release you are working *towards*17:59
*** tkelsey has quit IRC18:00
openstackgerritClark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering  https://review.openstack.org/17998818:00
mriedemlifeless: so do dev versions not work anymore?18:00
jeblairzaro, waynr: is david caro here?18:00
lifelessmriedem: they totally do18:00
clarkbjeblair: zaro ^ ok I am going to work on spinning up a jenkins and trigger jobs with that now18:00
*** ssam2 has quit IRC18:00
*** bknudson has joined #openstack-infra18:00
*** baoli has quit IRC18:00
clarkbgiving ./run-fast a go18:00
jeblairclarkb: yeah that should work18:01
clarkbhrm that exploded18:01
lifelessmriedem: there may be some disconnect here; I presume you want to make an alpha of nova?18:01
waynrjeblair: i'm not sure18:01
lifelessmriedem: if so, tag it.18:01
clarkbNPE and doesn't seem to have touched the gearman plugin so unlikely related to my changes18:01
fungiclarkb: any reason not to test on jenkins-dev?18:01
lifelessmriedem: 2015.1.1.dev1 is < 2015.1.1.a1 - pbr won't bump up to a1 until you tag it18:01
clarkbfungi: probably not, I can give that a go18:01
mriedemlifeless: ok18:02
clarkbzaro: any idea why run-fast fails? appears to be an error in the codehaus launcher18:02
*** baoli_ has joined #openstack-infra18:02
*** yfried|afk is now known as yfried18:02
mriedemi don't really want an alpha, it's kind of a weird situation18:02
lifelessmriedem: tell me more18:02
mriedemlifeless: we basically release our 2015.1 version of openstack a few weeks after community18:02
lifelessO\___ <- my couch18:02
mriedemso we branch off kilo at 2015.1 tag18:02
mriedemthen we backport fixes as needed until we release our 2015.118:03
openstackgerritAlexandre Viau proposed openstack-infra/project-config: Use surveil gerrit acl for python-surveilclient  https://review.openstack.org/18025718:03
*** dannywilson has quit IRC18:03
*** tiswanso_ has joined #openstack-infra18:03
mriedembut it's really like 2015.1.0.1 since this is before community releases 2015.1.118:03
mriedembut 2015.1.0.1 isn't a thing in pbr land18:03
jeblairpelix: ^ see conversation about jjb and parallelization18:03
zarojeblair: i have not seen him in here, i think yolanda might be able to get his attention somehow18:03
mriedemlifeless: anyway, our packages (rpms) are version 2015.1 but the source is like 2015.1 + some patches18:03
mriedemlifeless: and i'm trying to figure out how to versoin that18:04
lifelessok, so we haven't exposed it in the CLI yet18:04
lifelessbut there is a function for this18:04
*** freerunner has quit IRC18:04
zaroclarkb: no idea.  is this run-fast from gearman-plugin?18:04
yolandai can ping Darragh on hipchat, what happens?18:04
clarkbzaro: yup, I have also replicated using the debugging a plugin steps at https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial18:04
jeblairyolanda: no i pinged darragh here, i'm asking about david caro18:05
*** ddieterly has quit IRC18:05
yolandaah, i know David Caro, i can talk with him, and he is dcaro on freenode, he will answer if pinged in private18:05
yolandawhat's up with him?18:06
zaroclarkb: don't know, but 'mvn clean hpi:run' works for me18:06
jeblairi'm about to revert his patch18:06
*** ildikov has quit IRC18:06
yolandajeblair, why? is there some issue?18:06
jeblairyolanda: yes, it deletes all of the jobs18:06
lifelessmriedem: have a look at rpm_version on semantic version (and or the semver spec in oslo-specs)18:06
clarkbzaro: huh let me try that18:07
lifelessmriedem: rpm doesn't have a sort-before operator18:07
lifelessmriedem: and so its fugly :(18:07
clarkbzaro: that failed for me18:07
openstackgerritJames E. Blair proposed openstack-infra/jenkins-job-builder: Revert "Added parallelization options"  https://review.openstack.org/18025818:07
clarkbits fine, I have a container running jenkins somewhere I can just resurrect that and put my plugin in it, then migrated to jenkins-dev if that looks good18:07
jeblairi'm going to update the commit msg on that ^ gimme a min18:08
*** chriscowley_ has joined #openstack-infra18:08
yolandajeblair, deletes jobs from jenkins, as if they were old?18:08
zaroclarkb: what's not working about it? it seems to be ok for me.  maybe try 'mvn install' first?18:08
*** gokrokve has joined #openstack-infra18:09
*** patrickeast_ has joined #openstack-infra18:09
*** btully has joined #openstack-infra18:09
clarkbzaro: http://paste.openstack.org/show/215037/18:09
*** hdd has joined #openstack-infra18:10
zarojeblair: ohh that sux! i tested --delete-old and that seemed to worked for me.  maybe i did it with a clean cache.18:10
yolandazaro, i tested that as well, under your suite of tests, and it passed tests for me as well18:11
yolandawe need to be testing something incompletely18:11
*** patrickeast has quit IRC18:11
*** patrickeast_ is now known as patrickeast18:11
*** chriscowley has quit IRC18:11
*** chriscowley_ is now known as chriscowley18:11
*** gokrokve has quit IRC18:12
lifelessvishy: tsk! 3d1de2a breaks pbr..     <A0>* Remo18:12
lifelessvishy: *how* did you get that into git :)18:13
clarkbbtw https://github.com/cboylan/jjbhowto/tree/master/dockerstuff is the docker stuff for jenkins, not perfect but handy18:13
clarkbzaro: jeblair ^ I can copy/move that into the plugins if it helps testing too18:13
zaroclarkb: looks like maven went wacko there.  i would try to kill your local maven repo at ~/.m2 and rerun the command18:14
*** btully has quit IRC18:14
*** packet has quit IRC18:14
*** ddieterly has joined #openstack-infra18:15
*** otter768 has joined #openstack-infra18:15
*** dizquierdo has joined #openstack-infra18:16
jogoso mellanox CI is spamming nova18:16
jogoand they are a non-voting CI18:17
jogolast 30 comments http://paste.openstack.org/show/21503918:17
*** sparkycollier has joined #openstack-infra18:18
zaroclarkb: you mean gearman plugin ?  seems too specific for inclusion.18:18
*** ildikov has joined #openstack-infra18:18
anteayalennyb are you here18:18
clarkbzaro: any plugin, its pretty handy to say docker run and get a jenkins running with your plugin to test again18:19
clarkbzaro: considering that hpi:run seems to not work18:19
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Reorganize the configuration loading/munging code.  https://review.openstack.org/18026218:19
jogoanteaya: looks like he isn't around18:19
jogo can we email him and turn it off?18:19
zanebmarkmcclain: thanks for the review on https://review.openstack.org/#/c/174233/ ! Any reason we're holding back on the +A?18:19
jogobecause all the merge conflict messages are frustrating and misleading18:19
*** ivar-laz_ has quit IRC18:20
anteayajogo: well strictly speaking they are just commenting18:20
jogoanteaya: and commenting spam18:20
anteayajogo: right but we never told them they can't do that18:20
*** _nadya_ has quit IRC18:20
mriedemanteaya: people are thinking they are being told their patch is in merge conflict and has to be rebased18:20
*** ivar-lazzaro has joined #openstack-infra18:20
jogocommenting spam should never be allowed18:20
anteayajogo: I can disable but know they they didn't know they can't do this18:20
anteayamriedem: no argument here18:20
zaroclarkb: with a specific version of java and jenkins on ubuntu?18:20
*** otter768 has quit IRC18:20
mriedemi don't think this is intentional18:20
anteayajogo: sure but define what spam is for nova18:20
anteayajogo: so I can tell them18:20
jogoanteaya: thanks, should I contact the maintainer or what is the standard procedure18:21
zaroclarkb: i'm guessing not everyone's gonna want that18:21
jogoanteaya: what mriedem said, reporting merge failed on everything18:21
mriedemjust, it's busted so let's turn it off if they aren't going to have 24x7 converage18:21
anteayaI will have them disabled and post to third-party-announce and have them contact you18:21
mriedemcool, thanks anteaya18:21
anteayaI'm not arguing with you18:21
anteayaI'm agreeing18:21
jogoanteaya: great, thank you!!18:21
mriedemnow jogo can stop hurting all of us with his blue language18:21
jogomriedem: lol18:21
anteayabut I'm telling you from their perspective they might not know what they are doing isn't helpful18:21
anteayaas lennyb isn't a dev18:22
jogoanteaya: understood, I will be happy to walk them through the issue if needed18:22
clarkbzaro: ya its a specific version of java and jenkins, you don't have to use it and can edit it to use different things18:22
*** ivar-lazzaro has quit IRC18:22
anteayajogo: great thank you I will direct lenny to you18:23
*** ivar-lazzaro has joined #openstack-infra18:23
anteayajogo: he will show up, he is good for persisting, he just has no idea what he is doing but he means well18:23
jogoanteaya: thanks!18:23
*** gokrokve has joined #openstack-infra18:24
anteayathank you18:25
anteayacan we get gerrit account id 9732 disabled please: http://paste.openstack.org/show/215041/18:25
anteayapleia2: if you are about?18:25
pleia2anteaya: o/18:25
anteayacan you disable 9732 please?18:25
zaroclarkb: i'm fine with it. maybe just with a little more documentation.18:26
pleia2anteaya: sure, let's see...18:26
*** tosky has quit IRC18:27
*** ildikov has quit IRC18:27
*** hdd has quit IRC18:28
yolandajeblair, you were still running that change on single mode, right? without --workers=018:28
clarkbfwiw I am not seeing GET_JOB requests on the geard side18:29
pleia2clarkb: so when we disable an account we use --active, I assume disabling is --inactive? (that's what I get from reading the docs, just want to make sure I have the right one)18:29
*** gokrokve has quit IRC18:29
clarkbI see the RESET_ABILITIES and CAN_DOs though18:29
*** Ala has joined #openstack-infra18:29
pleia2clarkb: re: gerrit 3rd party accounts18:29
clarkbpleia2: disabling is --inactive, enabling is --active18:29
openstackgerritJames E. Blair proposed openstack-infra/jenkins-job-builder: Revert "Added parallelization options"  https://review.openstack.org/18025818:30
pleia2got it right \o/18:30
pleia2ok thanks18:30
jeblairyolanda, pelix, zaro: ^ see updated commit message18:30
*** fawadkhaliq has quit IRC18:30
jeblairclarkb, fungi: ^18:30
pleia2anteaya: done18:30
jeblairyolanda: yes, i have no interest in running multi-threaded.  i think it's a great way to kill our already over-burdened jenkinns masters.18:31
anteayapleia2: thank you18:31
fungijeblair: also worth noting, zuul still seems fairly bogged down throughput-wise even though we don't have jjb currently running anywhere now18:31
*** Somay has joined #openstack-infra18:31
*** AL34N1X has joined #openstack-infra18:31
*** shardy_ has joined #openstack-infra18:33
*** gokrokve has joined #openstack-infra18:34
*** shardy has quit IRC18:34
*** ivar-laz_ has joined #openstack-infra18:34
lifelessvishy: https://bugs.launchpad.net/pbr/+bug/1451976 <- I'm interested in how that got into nova18:34
openstackLaunchpad bug 1451976 in PBR "non-utf8 commits break getting version info" [Undecided,New]18:34
*** sparkycollier has quit IRC18:34
fungilooks like devstack/tempest may just be hosed18:34
*** ivar-lazzaro has quit IRC18:35
*** ivar-laz_ has quit IRC18:35
*** prad has quit IRC18:35
fungiyeah, image upload failures in devstack18:35
zaroclarkb: yeah, i don't see a GRAB_JOB after CAN_DO from the worker.18:35
*** hdd has joined #openstack-infra18:35
*** ivar-lazzaro has joined #openstack-infra18:35
jeblairfungi: asymmetric gate?18:36
*** prad has joined #openstack-infra18:36
fungijeblair: maybe, though there's an xattr-related traceback http://logs.openstack.org/56/177356/9/gate/gate-tempest-dsvm-full/d186960//logs/devstacklog.txt.gz#_2015-05-05_17_53_13_45018:36
fungimight be a new release of some dep18:37
*** cody-somerville has joined #openstack-infra18:37
fungidigging now18:37
jeblairfungi: my best guess for zuul slowness is geard slowness since job submission is synchronous18:37
openstackgerritlifeless proposed openstack-dev/pbr: Issue #1451976: handle commits with non-utf8 text  https://review.openstack.org/18026518:38
lifelessmriedem: ok, now I've shaved that yak...18:38
*** shardy_ has quit IRC18:38
*** packet has joined #openstack-infra18:39
mriedemlifeless: oh yeah?18:39
*** shardy has joined #openstack-infra18:39
lifelessmriedem: yeah, I went to poke at nova stable/kilo and found that pbr broke on it18:39
lifelessmriedem: for totally different reasons to your need18:39
jeblairfungi: are there any jjb runs still in progress?18:40
lifelessmriedem: anyhow, did you find the spec? we defined a mapping to rpm versions - its nonintuitive because we have to map *up* a version to deal with pre-releases18:40
lifeless>>> pbr.version.VersionInfo('nova').semantic_version()18:40
lifeless>>> pbr.version.VersionInfo('nova').semantic_version().rpm_string()18:41
fungijeblair: as i said, none for the past hour or so... so not sure that theory panned out after all18:41
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Return Bunch objects instead of plain dicts  https://review.openstack.org/18026918:41
mriedemlifeless: yeah i found the spec18:41
mriedemlifeless: so rpm-version is busted with pbr 0.11 and nova stable/kilo?18:42
mriedemi see the 2015.1.1 is missing18:42
lifelessmriedem: I think publishing dev versions should be fine for you - 2015.1.1.0a2.dev4 (what stable/kilo's semver version is right now) is > 2015.1.0 and < 2015.1.118:42
*** ivar-laz_ has joined #openstack-infra18:43
*** ivar-laz_ has quit IRC18:43
lifelessmriedem: the 1.1 in the rpm mapping is missing because there isn't a version that both starts with 2015.1.1 and is < 2015.1.1 in rpm land18:43
mriedemlifeless: ok18:43
*** sdake_ has joined #openstack-infra18:43
*** ildikov has joined #openstack-infra18:43
lifelessmriedem: (AIUI, harlowja did most of the work there :))18:43
mriedemlifeless: i have other issues anyway to deal with - like we are building the packages with pbr 0.6 on the build nodes,18:43
lifelessI just translated to code18:43
harlowjalifeless i thought u did most of the work18:43
lifelessmriedem: oh!. Heh :)18:43
*** fawadkhaliq has joined #openstack-infra18:43
mriedembut on our test nodes, we're pip installing tempest's requirements (we don't package tempest), which is putting pbr 0.11 on the test nodes18:43
mriedemso shit breaks18:44
*** ivar-laz_ has joined #openstack-infra18:44
mriedemb/c pbr 0.11 doesn't like pbr 0.618:44
lifelessyeah, 0.6 is, uhm, dated.18:44
* anteaya shakes a finger at mriedem18:44
mriedemlifeless: pbr 0.6 is still the min version in g-r18:44
zaroclarkb: seems to be blocked on ioAvailable.select() line 34418:44
mriedemso we haven't moved up18:44
mriedemanteaya: i learned it from watching jogo18:44
lifelessmriedem: whats the issue? we might be able to fix it in the next pbr release18:44
harlowjalifeless ah ya, i remember, i've done to much with rpm specs and sorting and crap prior, lol, so i guess i braindumped some of that to u :-P18:44
mriedemlifeless: we have a 2015.1.ibm.rc1 tag on our source18:44
* anteaya shakes a finger at jogo for good measure18:44
mriedemlifeless: so that's what's in our pkg-info18:45
mriedemb/c pbr 0.6 is cool with that18:45
lifelessfungi: mordred: https://review.openstack.org/180265 moderately important. Possibly critically :)18:45
clarkbzaro: I am trying to get logging set to allow debug lines but that is what I expected18:45
anteayaI'm having difficulty reading the graph did all our in use nodes just drasticly shrink?18:45
mriedemlifeless: but at runtime, pbr 0.11 is hating the 2015.1.ibm.rc118:45
lifelessmriedem: oh wow. Yes, we have no defined mapping for that at all.18:45
anteayanot that we had many to start off with18:46
mriedemlifeless: as in http://paste.openstack.org/show/215043/18:46
*** ivar-lazzaro has quit IRC18:46
lifelessI don't believe PEP-440 has a vendor labelling mechanism; we could possibly squash it into local versions18:46
*** sdake has quit IRC18:47
mriedemlifeless: well, fwiw, python setup.py build was giving us the userwarning: http://paste.openstack.org/show/215045/18:47
mriedemwe weren't heeding it18:47
mriedemb/c python setup.py build doesn't fail on that18:47
lifelessmriedem: we could in principle partse 2015.1.ibm.rc1 -> 2015.1.0.rc1+ibm18:47
lifelessmriedem: I was commenting in the nova patch the other day that the distro package override logic should move to pbr18:47
lifelessso we can offer it to all projects consistently. I don't know if vendor strings are part of that.18:48
*** maishsk_afk has quit IRC18:48
lifelessbut if they are, I suspect local identifiers will be needed18:48
lifelessmriedem: heh yes.18:48
mriedemwell, i think my short term answer is to not allow tempest to install pbr 0.11 from pip on our test nodes18:48
lifelessuhm, pbr 0.6 in g-r.18:48
mriedemfor kilo18:48
lifelessI think thats bogus.18:48
lifelessFor everything18:49
*** maishsk has joined #openstack-infra18:49
mriedemyeah, because you can't build with pbr 0.6 and run with pbr 0.11 if the versions aren't kosher with 0.1118:49
lifelessbecause: we had to fixup pep-440 issues in the versions generated about 4 months ago18:49
lifelessit was a mad scramble18:49
lifelessand the only fixed versions are 0.10 and 0.1118:49
mriedemlifeless: i'd be +1 on updating the min version on pbr to .10 in g-r on master :)18:49
lifelessthe reason g-r didn't change is oversight, because nothing has (and nothing can have) a versioned setup_requires18:49
clarkbzaro: I am just using a new build with log.debug replaced with log.info...18:50
lifelessmriedem: I'll propose a 0.11 bump, if we're bumping, we may as well get everything consistent18:50
mriedemlifeless: yeah https://github.com/openstack/nova/blob/master/setup.py#L2918:50
clarkbzaro: but I want to confirm that we process the NOOP event18:50
*** dboik has joined #openstack-infra18:50
mriedemlifeless: that used to have a min version in setup_requires18:50
*** dboik_ has quit IRC18:50
mriedemlong ago18:50
mriedemthat's why our pbr package was still 0.618:50
lifelessit breaks if it does18:50
lifelessmriedem: I linked the bug which documents why it breaks in my 1.0 thread yesterday, I can dig it up if you like)18:51
mriedemso short term for kilo i have to cap pbr < 0.11 in our stuff, for master we (ibm) need to update our build/test environments to use pbr 0.1118:51
mriedemlifeless: that's ok, i'm in enough rabbit holes at once as it is18:51
mriedem*and have to leave work in an hour18:51
zaroclarkb: i don't see NOOP from server.18:51
clarkbzaro: no the worker sends it to itself18:52
zaroclarkb: that's probably becuase no GRAB_JOB/PREE_SLEEP18:52
openstackgerritlifeless proposed openstack/requirements: Bump pbr versions.  https://review.openstack.org/18027118:53
lifelessmriedem: ^18:53
*** tnovacik has joined #openstack-infra18:53
*** dboik has quit IRC18:53
clarkbzaro: that is what sends the GRAB_JOB18:54
clarkbzaro: at least in theory, but its not happening18:54
*** dboik has joined #openstack-infra18:54
mriedemlifeless: thanks, +118:55
lifelessif folk have been using 0.6 this whole time18:56
lifelessI can understand distros going 'omg its broken patch it out' :)18:56
lifelessjhesketh: 5am?!18:56
pleia2infra meeting means early day for jhesketh18:56
jheskethPretty much18:57
lifelessthere's eary18:57
lifelessand there's EARLY18:57
*** pblaho has joined #openstack-infra18:57
anteayamorning jhesketh18:57
zaroclarkb: i was expecting a NOOP from server after CAN_DO but that seems to not be happening.18:57
jheskethlifeless: are you suggesting we move the meeting per chance? :-P18:57
jheskethanteaya: o/18:57
anteayamy CAN_DO graph just nosedived recently18:58
anteayaif I am looking at the correct graph18:58
lifelessjhesketh: Nope :)18:58
zaroclarkb: maybe NOOP from server is what's needed?18:58
pleia2anteaya: here's to graph collection18:58
pleia2err correction18:58
clarkbzaro: well in theory we are giving ourselves a NOOP to trigger the GRAB_JOB action18:58
anteayapleia2: I'm trying to follow along18:59
anteayait is tough18:59
lifelessjhesketh: since I'm not tracking tripleo so closely, the conflict doesn't bother me now, and 7am I can usually do18:59
*** btully has joined #openstack-infra18:59
*** annegentle has joined #openstack-infra18:59
*** bknudson has quit IRC19:00
*** mmedvede has joined #openstack-infra19:00
fungimeeting time19:01
fungikeystone have wrapped up19:01
*** gokrokve has quit IRC19:01
*** gokrokve has joined #openstack-infra19:01
*** Longgeek_ has quit IRC19:02
*** soren_ has joined #openstack-infra19:02
*** soren_ has quit IRC19:03
*** soren_ has joined #openstack-infra19:03
sdake_hey my gate is failing with NOT_REGISTERED19:03
sdake_what does that mean?19:03
*** soren_ is now known as soren19:04
anteayazuul is unhappy today19:04
sdake_thanks so not a bug on my part19:04
*** hdd has quit IRC19:04
anteayamight be hard to diagnose given today's zuul illness19:04
*** HeOS has joined #openstack-infra19:04
sdake_unfortunately it says "fiel not foudn" and there are no logs19:05
sdake_and its causing all our gating to fail :(19:05
clarkbits not your fault, there are no logs because there was no job19:05
fungiclarkb: jeblair: another interesting problem... we apparently have nodes in use which still look ready in nodepool. i suspect an unrelated zmq disconnect is part of the mess here. restarting nodepool momentarily19:06
*** baoli_ has quit IRC19:06
*** jistr has joined #openstack-infra19:08
funginodepoold has been restarted. i'll keep an eye out for nodes still stuck in ready long enough to no longer be running jobs19:08
*** shashankhegde has quit IRC19:09
*** ZZelle_ has joined #openstack-infra19:09
clarkbzaro: I figured it out sort of19:11
*** shivharis has joined #openstack-infra19:11
clarkbzaro: will need to trace the code paths to understand if my fix is sufficient, but basic problem is that when we start work() we are already initialized so we never add the NOOP to the queue19:11
*** dizquierdo is now known as dizquierdo_afk19:12
shivharisclarkb: hi19:13
*** gyee has quit IRC19:13
nibalizerclarkb: we didn't get to it last week, but this week lets talk about your plan to break my refactor up19:13
*** tiswanso_ has quit IRC19:14
*** tkelsey has joined #openstack-infra19:14
clarkbshivharis: currently in a meeting so mostly focused on that19:14
*** ayoung has quit IRC19:14
shivharisclarkb: np19:15
*** EmilienM is now known as EmilienM|afk19:15
*** gyee has joined #openstack-infra19:15
openstackgerritSergey Kolekonov proposed openstack-infra/project-config: Add apps-catalog/ci group for apps-catalog project  https://review.openstack.org/18020119:15
* waynr gets head caught in tangle of spaghetti code19:15
openstackgerritSergey Kolekonov proposed openstack-infra/project-config: Add apps-catalog-ci group for apps-catalog project  https://review.openstack.org/18020119:16
shivharisfungi: hi19:16
fungishivharis: hi. the meeting to which clarkb refers is the infrastreucture team meeting. we're _all_ focused on that19:18
*** rlandy has quit IRC19:18
fungier, infrastructure19:18
*** baoli has joined #openstack-infra19:18
shivharisfungi: np19:19
*** tkelsey has quit IRC19:19
*** pradk has quit IRC19:19
*** chriscowley has quit IRC19:21
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron  https://review.openstack.org/17968919:22
sdaguezaneb: +A19:22
markmcclainzaneb: was waiting on sdague to confirm19:23
*** subscope_ has joined #openstack-infra19:27
*** tiswanso has joined #openstack-infra19:27
*** annegentle has quit IRC19:28
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for swift-object-server bug 1451992  https://review.openstack.org/18027819:30
openstackbug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] https://launchpad.net/bugs/145199219:30
*** rloo has joined #openstack-infra19:31
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Return Bunch objects instead of plain dicts  https://review.openstack.org/18026919:33
*** mtanino has quit IRC19:34
*** Ala has quit IRC19:35
*** Sukhdev has quit IRC19:35
*** yfried is now known as yfried|afk19:35
*** doug-fish has quit IRC19:36
davidlenwellwhen adding a new project.. does the upstream: git repo have to have the same name as the new repo?19:37
davidlenwellsweet .. thanks19:38
*** weshay has quit IRC19:39
*** ildikov has quit IRC19:41
*** sdake has joined #openstack-infra19:42
*** emagana has quit IRC19:42
*** sdake has quit IRC19:42
*** emagana has joined #openstack-infra19:42
*** sdake_ has quit IRC19:45
*** sdake has joined #openstack-infra19:46
*** btully has quit IRC19:47
*** sdake_ has joined #openstack-infra19:47
*** mestery has quit IRC19:48
*** vhoward has joined #openstack-infra19:48
*** jistr has quit IRC19:49
zanebsdague, markmcclain: thanks!19:49
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Update server API for get/list/search interface  https://review.openstack.org/18029319:51
*** sdake has quit IRC19:51
*** weshay has joined #openstack-infra19:52
*** shivharis has quit IRC19:52
*** shivharis has joined #openstack-infra19:52
*** ildikov has joined #openstack-infra19:54
openstackgerritDavid Lenwell proposed openstack-infra/project-config: adding akanda horizon plugin to stackforge  https://review.openstack.org/18029419:54
*** shashankhegde has joined #openstack-infra19:54
*** yfried|afk is now known as yfried19:55
*** Rockyg has joined #openstack-infra19:59
*** maishsk has quit IRC19:59
*** maishsk has joined #openstack-infra20:00
anteayafungi: no storm like a perfect storm20:00
fungishivharis: what did you need?20:01
*** sdake has joined #openstack-infra20:01
*** ildikov has quit IRC20:01
AJaeger_fungi, are you storm fighting now or could you help me with a review where I have a question?20:01
jeblairfungi, clarkb, zaro: i do still think the zuul sluggishness is due to geard sluggishness, possibly because we have so many ready nodes (even though they may not be emitting as many registration events as before)20:01
jeblairfungi, clarkb, zaro: hopefully having fewer ready nodes sitting in pre-sleep (and therefore generating a lot of noop traffic) should help, and also perhaps my change to drop the work_status packets.20:02
fungijeblair: i'm unsure they're actually ready. i've ssh'd into some of the devstack-trusty nodes which claim to be ready for up to an hour and they had devstack running and doing stuff20:02
zarojeblair: yes, that was based on last version of your change: https://review.openstack.org/#/c/174607/20:02
*** mriedem has quit IRC20:02
yolandahere, i wanted to ask for some reviews on nodepool image testing spec, please: https://review.openstack.org/13959820:03
jeblairfungi: oh, that was why you mentioned zmq earlier20:03
yolandaand for the ton of downstream-puppet changes as well :)20:03
fungiAJaeger_: several fires at once. is it urgent?20:03
zarojeblair: incidentally that change is .23 on review-dev.20:03
shivharisfungi: wanted to find out how to add members to the <projectname>-core group - did once some time back but cannot seem to find it20:04
jeblairzaro: ah, that splains it20:04
anteayashivharis: are you currently a member of the core group?20:04
AJaeger_fungi: Then continue firefighting and after you rested, please see my last comment in https://review.openstack.org/#/c/173408/ ;)20:04
shivharisanteaya: yes, core and admin20:04
anteayaif you are signed into gerrit and go to people > groups > name of group-core20:05
*** sdake_ has quit IRC20:05
anteayayou should see a text field and an add button20:05
AJaeger_fungi: not urgent for me but the glusterfs guys are fighting some time already...20:05
*** annegentle has joined #openstack-infra20:05
*** _nadya_ has joined #openstack-infra20:05
shivharisanteaya: thanks20:05
*** yfried is now known as yfried|afk20:05
anteayashivharis: welcome20:06
*** doug-fish has joined #openstack-infra20:06
*** heyongli has quit IRC20:07
*** freerunner has joined #openstack-infra20:08
*** heyongli has joined #openstack-infra20:09
fungiAJaeger_: you mean mtreinish's comment on that review?20:09
AJaeger_fungi: no, my comment on name collision20:10
*** sdake has quit IRC20:10
*** sdake has joined #openstack-infra20:10
AJaeger_fungi: I prefer to rename the job so that it cannot collision with the plugin one20:10
*** kgiusti has quit IRC20:10
openstackgerritClark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering  https://review.openstack.org/17998820:11
clarkbI have run ~100 trivial shell jobs on the master executor using ^ with 10 clients20:11
* AJaeger_ wonders whether I'm insisting here on too much and should just say go for it - or really insist...20:11
*** achanda has quit IRC20:11
clarkbzaro: jeblair the needed fix was that when we start running in work() we are already initialized (which is weird but that can be a followup fix) so you have to queue the noop event first thing20:12
AJaeger_fungi: really put this on the back and continue fire fighting ...20:12
clarkbotherwise we never queue the noop event20:12
jeblairclarkb: ooooh20:12
* clarkb is going to do more manual testing of that20:12
fungiAJaeger_: seems like a fine suggestion to me. should we though rename other jobs which also use a similar naming pattern so as to avoid the same sorts of conflicts in the future?20:13
*** ildikov has joined #openstack-infra20:13
AJaeger_Like the ceph one? Yeah, I could propose that to make a point ;)20:14
AJaeger_any ideas on how to rename?20:14
ianwfungi / AJaeger_ : has anyone ever considered arguments to macros in jjb, so we could retain templates but pass through things like the regex for skipping tests?20:14
ianwi can't think of a way to do that as is20:15
fungiianw: you can pass parameters to a builder macro, for example20:15
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Update server API for get/list/search interface  https://review.openstack.org/18029320:16
AJaeger_ianw, so invoking it using: "variable: regex"? That should work today...20:16
*** otter768 has joined #openstack-infra20:16
clarkbhrm my minimal jenkins doesn't seem to support running jobs by label20:16
*** shivharis has quit IRC20:16
*** shivharis has joined #openstack-infra20:17
fungiAJaeger_: probably the plugins should have namespacing like ...-plugin-{plugin}{branch-designator}20:17
clarkbwant to test that the reset abilities works properly while jobs are running20:17
*** maishsk has quit IRC20:17
clarkbI actually should be able to just add a new job20:17
AJaeger_fungi: Let me propose that. thanks20:17
harlowjaperiodic ping to get someone to review https://review.openstack.org/#/c/171393/ :-P20:17
harlowja^ and followups that add tests20:18
harlowja^^^^^^ tests are good20:18
clarkbyup it seems to have done the correct thing, reset after work complete, then sent can dos then sent grab jobs20:18
*** dizquierdo_afk is now known as dizquierdo20:19
openstackgerritgreghaynes proposed openstack-infra/nodepool: WIP: Use shade to get the client objects  https://review.openstack.org/16860320:19
openstackgerritKhai Do proposed openstack-infra/system-config: Upgrade Gerrit from 2.8 to ver 2.10 on review.o.o  https://review.openstack.org/15546320:19
greghaynesmordred: test indicates that change doesnt work as expected :(20:19
greghaynesmordred: er, ^ that test20:20
*** mtanino has joined #openstack-infra20:20
ianwAJaeger_: ahh, ok.  so why not setup a builder like devstack-rally-gate for plugins that we can pass arbitrary test skips through to, and avoid having to create duplicate jobs?20:20
clarkbzaro: jeblair: I am going to go grab lunch now that those two basic things seem to work20:20
greghaynesyes, food20:20
zaroclarkb: will take a look20:21
clarkbzaro: thanks, independent testing is likely also useful20:21
clarkbzaro: since that should use different timings and stuff20:21
*** otter768 has quit IRC20:21
*** shivharis has quit IRC20:21
*** shivharis has joined #openstack-infra20:22
*** palendae has left #openstack-infra20:22
openstackgerritKhai Do proposed openstack-infra/project-config: Make gerritbot report changes from Gerrit 2.10.2 branch  https://review.openstack.org/15545020:22
*** maishsk has joined #openstack-infra20:23
AJaeger_ianw: So, doing something like if [ "{regex}" != "" ] ; then export DEVSTACK_GATE_TEMPEST_REGEX="{regex}" fi ?20:25
AJaeger_that would mean we would need to add to each job that uses this the regex.20:25
AJaeger_Let me propose that as separate step and then we can review...20:26
AJaeger_... and discuss20:26
ianwAJaeger_: yeah, something like that ... we have a nice template for plugin devstack jobs, but if you want to skip anything you can't use it20:26
ianwAJaeger_: i'm happy to work on that.  i want to clean up the centos7/fedora jobs in a similar way as well20:26
ianwAJaeger_: so feel free to ignore my -1 and i'll look at it soonish20:27
*** abregman has joined #openstack-infra20:27
*** gyee has quit IRC20:28
*** ildikov has quit IRC20:28
ianwi'm not sure i agree with tempest's position that it's happy to be broken for known issues with drivers, but anyway20:28
fungiianw: i believe it's position is that it's willing to remove testing and general support for drivers if they're broken or untestable, but not simply add overrides/skips20:28
fungithough mtreinish can certainly clarify20:29
mtreinishfungi: yeah that's a good high level explanation20:29
ianwI don't pretend to know the history.  it seems if there was a desire there are certainly ways you could keep skip lists in some common format within tempest itself20:30
ianwinstead of leaving it up to devstack-gate jobs20:30
*** AL34N1X has quit IRC20:31
ianwmy main point is that we should be striving for less & more templated jobs in jjb, not more20:31
mtreinishianw: that exact thing was proposed maybe 2 years ago and it wasn't accepted for the same reason20:32
mtreinishit's not tempest's job to maintain what works with what driver, it just tests that an openstack deployment works as expected through the rest api20:34
mtreinishit's a layer separation thing20:34
jlvillalWould this be the correct channel to ask about Stackaltyics?  When going here for Liberty: http://stackalytics.com/?release=liberty  I don't see an Ironic project listed.20:34
*** tiswanso has quit IRC20:34
mtreinishthere are already flags to say a cloud does or does not support a particular feature20:34
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename tempest-dsvm-full-{plugin}  https://review.openstack.org/18030820:34
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Parse regex to skip to tempest-dsvm-full-plugin jobs  https://review.openstack.org/18030920:34
mtreinishwhich is what should be leveraged to skip a test if a driver is broken for something20:34
AJaeger_ianw, fungi here're my two patches ^20:35
*** tiswanso has joined #openstack-infra20:35
ianwmtreinish: well, you could add a layer ontop in a yaml file or something that kept some centralised list of issues, without having to add @skip to every function.  at least people would know where to look20:35
*** achanda has joined #openstack-infra20:35
fungiokay, i'm deleting the devstack-trusty images from today with the expectation that will provide short-term relief for bug 1451992 while we continue digging into the underlying cause20:35
openstackbug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] https://launchpad.net/bugs/145199220:35
anteayabit of meeting bot lag in the meeting20:35
anteayalike that adds to all the other things20:35
*** miqui has quit IRC20:36
mtreinishianw: what exactly are you trying to solve? Is it just that using a selection regex with testr doesn't report on the console that the test isn't being run?20:38
*** e0ne has quit IRC20:38
*** Trozz has quit IRC20:39
*** Trozz has joined #openstack-infra20:39
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Optionally pass regex to skip to tempest-dsvm-full-plugin jobs  https://review.openstack.org/18030920:40
*** persia has quit IRC20:40
*** _nadya_ has quit IRC20:40
*** ildikov has joined #openstack-infra20:41
*** d34dh0r53 has quit IRC20:41
*** AJaeger has quit IRC20:41
*** AJaeger_ has quit IRC20:41
*** fungi has quit IRC20:41
*** persia has joined #openstack-infra20:41
*** pblaho has quit IRC20:41
dimsfungi: cffi version mismatch? (looks like tripleo guys saw it before - https://bugs.launchpad.net/tripleo/+bug/1340776/comments/3)20:42
openstackLaunchpad bug 1340776 in tripleo "Swift fails to start in overcloud" [High,Fix released] - Assigned to Eamonn O'Toole (eamonn-otoole)20:42
*** mwagner_lap has quit IRC20:42
ianwmtreinish: my beef, much as it is, is that devstack-gate isn't the only way to run tempest, so we know it doesn't work in various ways, but there's no way for anyone to realise that20:43
ianwthe information about what works and what doesn't is scattered in regex's to various jobs all over the jjb config20:44
openstackgerritDavide Guerri proposed openstack-infra/shade: Add port resource methods  https://review.openstack.org/17996820:45
mtreinishianw: well sort of. The issues is deeper than just tempest, if a tempest test doesn't work then that doesn't work for any cloud. How is an end user using the api supposed to know?20:45
*** d34dh0r53 has joined #openstack-infra20:45
*** ivar-laz_ has quit IRC20:45
mtreinishencoding that into tempest somehow doesn't solve the fundamental issue, I would argue that it would probably mask it even further20:45
*** amitgandhinz has quit IRC20:46
*** ivar-lazzaro has joined #openstack-infra20:46
*** ivar-lazzaro has quit IRC20:46
*** AJaeger has joined #openstack-infra20:47
*** r-daneel has joined #openstack-infra20:47
*** fungi has joined #openstack-infra20:47
*** ivar-lazzaro has joined #openstack-infra20:47
ianwmtreinish: well, i agree that say, hypervisor assisted snapshots don't work the same way with glusterfs and skipping the test in tempest doesn't help anyone trying to do that outside tempest20:48
*** sabeen has joined #openstack-infra20:49
fungibad time for freenode to be breaking20:49
*** tonytan4ever has joined #openstack-infra20:50
fungiclarkb: jeblair: i need to disappear for a couple hours unfortunately, but will check back in on things when i return20:50
clarkbfungi before you go anything I can take over?20:51
clarkbor are we mostly recovering at this point?20:51
fungideleting ready nodes >2 hours old is generally safe at this point i think, so i've done that20:51
fungiin another hour, deleting ready nodes >1 hour should be safe20:51
clarkbcan do20:51
*** fawadkhaliq has quit IRC20:51
fungithanks! that should get things moving faster i think20:52
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Create XmlBuilder class, split Builder.update_jobs  https://review.openstack.org/18031520:52
ianwmtreinish: but i still think that keeping this information within tempest would be better.  anyway, adding arguments to the plugin template as proposed i think at least is a good way to avoid job-explosion in project-config20:52
fungialso check in with #-dev and see how the xattr/cffi debugging is going in case they need more infra assistance besides the image removal i did20:52
fungiback as soon as i can be20:52
*** patrickeast has quit IRC20:52
*** aysyd has quit IRC20:52
*** patrickeast has joined #openstack-infra20:53
*** pabelanger has quit IRC20:53
mtreinishianw: btw, have you seen: http://git.openstack.org/cgit/openstack/os-testr/commit/?id=ff3dc87d6b496af625652838936288e0c3882853 that was my idea for doing what your describing20:53
mtreinishthe plugins could then just put an exclude file in a known place in the plugin tree and the jobs would use that to exclude if present20:54
*** gyee has joined #openstack-infra20:54
*** amitgandhinz has joined #openstack-infra20:54
openstackgerritTimo Tijhof proposed openstack-infra/zuul: Status: Don't raise "abort" as error to the user  https://review.openstack.org/17224120:55
ianwmtreinish: that's certainly going down the right path -- keep the skip information right there alongside the plugin.  i have to dash for a bit, i will take a closer look20:55
openstackgerritMerged openstack-infra/release-tools: Add script to cleanup of translations  https://review.openstack.org/17282720:56
openstackgerritTimo Tijhof proposed openstack-infra/zuul: Status: Don't raise "abort" as error to the user  https://review.openstack.org/17224120:56
*** sparkycollier has joined #openstack-infra20:56
*** shivharis has quit IRC20:58
*** frobware_ has joined #openstack-infra20:58
*** nelsnelson has quit IRC21:00
*** hdd has joined #openstack-infra21:02
*** shashankhegde has quit IRC21:03
anteayaI am going to get a walk in21:06
*** maishsk has quit IRC21:06
*** radez is now known as radez_g0n321:06
*** btully has joined #openstack-infra21:07
clarkbpleia2: re https://review.openstack.org/#/c/179840/1/doc/source/testing.rst can you try also granting perms for openstack_citest@ and @::1 ?21:07
clarkbpleia2: I bet its a localhost isn't resolved the way we expect it there issue21:07
*** craigbr has joined #openstack-infra21:08
jeblairclarkb: who else should we ask to review your change and mine?21:08
jeblairclarkb: or should we just merge them now?21:08
*** hdd has quit IRC21:08
clarkbjeblair: zaro, jhesketh maybe?21:09
clarkbjeblair: I am pretty confident in yours21:09
clarkbjeblair: mine is the scary one :)21:09
jeblairclarkb: but you actually ran it, right? :)21:09
clarkbjeblair: I did, but only against a local jenkins (of the same version) with a a test job and a handful of clients21:09
pleia2clarkb: will do, also testing the devstack one and have a few comments for that too21:09
clarkbjeblair: I also added new jobs to force reregistration21:09
clarkbjeblair: and that all looked good21:09
clarkbpleia2: cool, thanks21:10
jeblairclarkb: that seems good to me21:10
zaroclarkb, jeblair : i'm reviewing, if you want to wait a little.21:10
jeblairzaro: excellent, will do, thanks.  :)21:10
jeblairjhesketh: let us know if you want to review those gearman-plugin changes too21:11
*** annegentle has quit IRC21:11
*** annegentle has joined #openstack-infra21:12
*** bswartz has quit IRC21:12
dstufftwooo, cffi 1.0 is gonna be awesome21:14
*** ldnunes has quit IRC21:15
zaroclarkb: i don't see a GRAB_JOB after register and CAN_DO?21:16
clarkbzaro: with the latest patchset?21:16
zaroclarkb: yeah21:16
zaroclarkb: I guess we want NOOP right?21:17
clarkbzaro: no you want GET_JOB21:17
clarkber GRAB_JOB21:17
*** dkranz has quit IRC21:18
*** dprince has quit IRC21:18
zaroclarkb: so i'm doing set node offline, SUBMIT_JOB, bring node back online21:18
zaroclarkb: register happens, but not GRAB_JOB21:19
clarkbzaro: so that may be a code path I didn't cover, I did check the temporarily offline then online case21:19
clarkbis that what you mean by offline or are you disconnecting?21:19
*** claudiub has joined #openstack-infra21:20
zaroclarkb: temp offline/online _not_ disconnect21:20
*** panda is now known as panda|Zz21:20
clarkbok, I did test that, let me try again21:20
*** tiswanso has quit IRC21:21
clarkbzaro: huh this time I see what you see21:21
clarkblet me dig in21:22
openstackgerritDavide Guerri proposed openstack-infra/shade: Add keystone service resource methods  https://review.openstack.org/17762121:22
openstackgerritDavide Guerri proposed openstack-infra/shade: Add keystone endpoint resource methods  https://review.openstack.org/18032821:22
*** jtriley has quit IRC21:22
*** baoli has quit IRC21:23
clarkbgah I think that is because we call register jobs directly from that jenkins event listener thing21:24
*** baoli_ has joined #openstack-infra21:24
clarkbI see the bug I think21:27
*** gyee has quit IRC21:27
*** boris-42 has quit IRC21:28
clarkbits a logic bug around me trying not to send extra NOOPs21:28
clarkbbut this works around my serialization of register jobs and grab jobs21:29
*** dizquierdo has quit IRC21:29
clarkbmight be a good follow up fix to deal with that separately21:30
*** doug-fish has quit IRC21:30
*** shardy is now known as shardy_z21:31
*** ociuhandu has quit IRC21:32
clarkbwe should be calling setFunctions instead21:33
*** sparkycollier has quit IRC21:33
*** dguerri is now known as _dguerri21:33
*** _dguerri is now known as dguerri21:33
*** dustins has quit IRC21:33
*** sdake_ has joined #openstack-infra21:35
mordredclarkb, jeblair: I support you landing your gearman plugin changes21:36
clarkbok I have tracked this down we have two different registerJobs() methods that do different things21:37
clarkbwe have the one in the thread which sets setFunctions() on the worker whihc does the right thing21:37
clarkbbut you cannot call registerJobs() on the worker from an arbitrary thread21:38
jeblairmordred, clarkb, fungi, zaro: i believe gear on zuul.o.o has regressed which may explain the performance degredation there21:38
clarkbI think we are stuck with this because we implemented some interfaces21:38
*** dguerri is now known as _dguerri21:38
*** _dguerri is now known as dguerri21:38
*** sdake has quit IRC21:39
*** shashankhegde has joined #openstack-infra21:39
mordredjeblair: like we landed a regression in the code? or we accidentally caused an old thing to install?21:40
jeblairmordred: the second21:40
mordredoh awesome21:40
jeblairmaybe we fixed puppet installing things, and probably it installed the actual released version instead of a local thing21:40
clarkblike before our event loop performance changes?21:40
jeblairclarkb: yeah, it's at 0.5.521:40
openstackgerritClark Boylan proposed openstack-infra/gearman-plugin: Fix race between adding job and registering  https://review.openstack.org/17998821:41
clarkbzaro: ^ can you test that version, it fixes the logic bug21:41
mordredjeblair: so maybe we should release a gear?21:41
jeblairso this is probably my fault for not releasing gear and bumping zuul's dep21:41
jeblairmordred: sounds like a good idea :)21:41
jeblairin the mean time, i would like to just fix it on server21:42
clarkbjeblair: sounds reasonable to me21:42
jeblairzuul hit 0 events so i saved the queue and restarted it21:43
jeblairi didn't know how long it would take to catch up again, so i took my opportunity21:43
mordredjeblair: ++21:44
zaroclarkb: aren't those registerJobs syncronized?21:44
clarkbzaro: ya, they are because they are registerJobs against the thread not the worker21:44
clarkbzaro: which is really confusing for me but I believe its the interface we were given so meh21:45
openstackgerritFlavio Percoco proposed openstack-infra/project-config: Make zaqarclient functional gate non-voting  https://review.openstack.org/18033221:46
openstackgerritFlavio Percoco proposed openstack-infra/project-config: Extend Zaqar's gate jobs  https://review.openstack.org/18033321:46
jeblairgeard is not pegging a cpu now21:46
*** stevemar has joined #openstack-infra21:47
*** SumitNaiksatam has quit IRC21:47
zaroclarkb: trying again now21:47
clarkbzaro: ty21:47
clarkbzaro: fwiw it did work for me locally using your test of tempoffline node, submit job, online node21:47
*** doug-fish has joined #openstack-infra21:48
jeblairgear 0.5.6 tagged21:49
*** peristeri has quit IRC21:49
openstackgerritJames E. Blair proposed openstack-infra/zuul: Update to gear 0.5.6  https://review.openstack.org/18033421:50
*** sdake has joined #openstack-infra21:50
*** mriedem has joined #openstack-infra21:51
*** whoops has quit IRC21:51
jeblairnodepool 0.1.0 tagged21:52
jeblairhasharMeeting: ^21:52
jeblairclarkb: i think that takes the pressure off of the gearman-plugin changes21:53
*** sdake__ has joined #openstack-infra21:54
clarkbjeblair: ok21:54
*** pcm_ has quit IRC21:54
*** sdake_ has quit IRC21:54
jeblairclarkb: after the jjb change merges and we manually run it on all the jenkins masters, we should not see big ready node problems unless we merge another change that touches all jobs21:54
jeblairso we just need to get it fixed before we do that :)21:55
clarkbroger, that gets us back to where we were with the existing race in gearman plugin21:55
clarkbwhich while annoying wasn't nearly world shattering21:55
*** ptoohill is now known as pothole21:55
*** tonytan4ever has quit IRC21:55
*** annegent_ has joined #openstack-infra21:55
*** mrmartin has quit IRC21:55
jeblairyeah, we had a nice perfect storm of 3 distinct problems :)21:55
jeblair(jjb deleting jobs, gearman-plugin racing, and geard regressing)21:56
*** cdent has quit IRC21:57
*** asettle has joined #openstack-infra21:58
*** sdake has quit IRC21:58
*** asettle has quit IRC21:58
*** SumitNaiksatam has joined #openstack-infra21:58
*** esker has quit IRC21:58
pleia2clarkb: no luck with @ or @::1 (have those and @localhost in the database now), it's odd since it seems to create them fine (I have about a billion mysql databases now ;)), but some part it gets to where it gets permission denied for them21:59
* pleia2 confirms no typos and usernames/passwords21:59
*** annegentle has quit IRC22:00
clarkbpleia2: oh, maybe its failing to clean them up22:00
clarkbpleia2: did you do the grant ALL on *.*?22:00
*** Krinkle is now known as Krinkle|detached22:00
pleia2clarkb: yeah, and I thought it was a flush privileges issue at first, but that didn't help either22:01
clarkbpleia2: hrm22:01
clarkblet me look at my existing db22:01
*** Swami has quit IRC22:03
*** SumitNaiksatam has quit IRC22:03
BobBallAny suggestions for the best way to enable a local login for nodes built by nodepool with modified standard scripts?  Got an issue where a large proportion of images don't get their networking when built by nodepool and I need to login using the console to investigate as I can't ssh22:03
*** changbl has quit IRC22:04
clarkbBobBall: add an adduser to your build scripts22:04
clarkbBobBall: have it set a password for some dummy user22:05
clarkband possibly add it to the sudo group at the same time22:05
clarkbor make root have a password22:05
BobBallheh... yes... sorry I am clearly being thick... I was thinking the puppet scripts reset the root password22:05
BobBallbut clearly I could just add a new user *sigh*22:05
*** bnemec has quit IRC22:05
clarkbI don't think puppet touches root22:05
BobBallit's 11pm and I'm tired.  That's my excuse.22:05
clarkbbut it may, would have to double check22:05
BobBallOh... I could have sworn something was clearing my root password22:06
*** bnemec has joined #openstack-infra22:08
clarkbpleia2: my mysql user table also has openstack_citest@% which is wildcard match I think22:09
clarkbpleia2: maybe give that a go? looking at perms now22:09
clarkbalso I seem to have thousands of rows in the user table which has me worried :/22:09
clarkboh they are all randomized users? is nodepool creating those?22:11
pleia2I only have 10, so it's not doing something crazy22:11
pleia2randomized database names for me22:11
pleia2actually | localhost | iHiDbXEd_367222:11
sdake__willa a recheck fix a "Need Verified"state?22:11
pleia2have a couple of those22:11
* pleia2 headscratch22:12
clarkbpleia2: ya thats what I have too, I think maybe if it doesn't clean up properly then we may leak the users too?22:12
*** packet has quit IRC22:12
clarkbsdake__: if the verified state is -2 or -122:12
*** jtriley has joined #openstack-infra22:12
sdake__the verified state is +122:12
pleia2clarkb: yeah, it only seems to happen sometimes for me, I have dozens of databases but only ended up with 2 randomized users22:12
sdake__but its been in needs verified for 8+ hours22:12
*** amitgandhinz has quit IRC22:12
sdake__our gate takes about 2 minutes to run :(22:13
clarkbpleia2: so ya my accounts have all the permissions, make sure you do the grant for all of them and maybe try the wildcard %?22:13
*** erikmwilson has quit IRC22:14
pleia2clarkb: I'll give that a go22:14
clarkbsdake__: what you can do is reapprove it22:14
sdake__i'll try that22:15
*** soren has quit IRC22:15
clarkbsdake__: the problem is since its already +1 verified the check queue is going to come back as +1 which is not a delta to the old vote so gerrit won't emit an event with the +122:15
*** hodos has quit IRC22:15
clarkbzaro: ^ any progress with your chnage to gerrit to fix that?22:15
sdake__clarkb thanks22:16
sdake__clarkb now the gate says "NOT REGISTERED"22:17
*** otter768 has joined #openstack-infra22:17
clarkbsdake__: thats related to the current world is upside down issues we are having22:17
sdake__and verified went to -222:17
clarkbsdake__: we have no f21 nodes22:17
sdake__oh thats still a problem22:18
sdake__f21 was in the gate previosuly wasn't it?22:18
clarkbf21 has never been in the gate iirc22:18
sdake__the gate was working for ages prior with f2122:18
clarkbwe have run jobs on f21 but not in the integrated gate iirc22:18
zaroclarkb: ohh yeah, it's looking good, just wanted to try with jjb.22:18
clarkbsdake__: anyways I will see what I can do about adding some nodes, thats what the issue is22:19
*** freerunner has quit IRC22:19
clarkbsdake__: right now gearman knows of no f21 hosts so says go away22:19
clarkbusually this isn't an issue because typicaly we have had at least one f21 host attached to gearman at some point22:20
clarkbits just right now with the geard restarts and nodepool losing track of nodes things are silly22:20
sdake__oh so this problem will be resolved when gearman eventually restarts :)22:21
clarkbsdake__: no it will be resolved as soon as we build an f21 host and attach it to gearman22:21
sdake__ok, so that probably sin't goin to happen immediatey, shoudl I just revert the gate patch temporarily?22:21
*** gordc has quit IRC22:22
clarkbsdake__: what gate patch?22:22
sdake__the one where we added f21 gating to kolla22:22
*** otter768 has quit IRC22:22
clarkbis that new?22:22
clarkb(I totally missed that if that is the case)22:22
sdake__3 months ago22:22
*** freerunner has joined #openstack-infra22:23
jeblairactually should happen real soon22:23
pleia2clarkb: aha, I think we need 'with grant option' at the end of the GRANT ALL line22:23
*** stevemar has quit IRC22:24
pleia2  py27: commands succeeded22:24
pleia2  congratulations :)22:24
sdake__jeblair should be rectified today then?22:24
pleia2I'll run through these again with a clean database to be sure22:25
clarkbpleia2: aha!22:25
jeblairsdake__: yes, probably within an hour, possibly much sooner22:25
*** haypo has joined #openstack-infra22:25
haypohey. i'm investigating https://bugs.launchpad.net/swift/+bug/1451992 -- i just commented the issue22:26
openstackLaunchpad bug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New]22:26
*** Krinkle|detached is now known as Krinkle22:26
haypoit looks like the issue is that DevStack builds a local wheel package for xattr, whereas building a wheel package for cffi extensions doesn't work right now22:26
clarkbhaypo: you should talk to clayg and greghaynes and jogo in openstack-dev22:26
haypois anyone aware of a recent change in devstack or something else related to building wheel packages?22:26
clarkbhaypo: they have mostly figured it out I think22:26
*** anthonyper_ has joined #openstack-infra22:27
sdake__thanks jeblair22:27
*** jcooley_ has joined #openstack-infra22:27
clarkbjeblair: are you currently updating jobs with jjb?22:27
*** baoli_ has quit IRC22:27
* clarkb is trying to keep up22:28
jeblairclarkb: no22:28
jeblairclarkb: waiting for jjb change to merge so i don't have to install it manually on masters22:28
*** jgrimm_ has joined #openstack-infra22:28
clarkbjeblair: gotcha22:28
jeblair(or risk it reverting, etc)22:28
*** doude_ has joined #openstack-infra22:29
*** sweston_ has joined #openstack-infra22:29
*** rbradfor has quit IRC22:29
*** neoice_ has joined #openstack-infra22:29
*** darvon_ has joined #openstack-infra22:29
*** anthonyper has quit IRC22:30
*** jcooley has quit IRC22:30
*** jgrimm has quit IRC22:30
*** kun_huang has quit IRC22:30
*** darvon has quit IRC22:30
*** neoice has quit IRC22:30
*** tteggel has quit IRC22:30
*** JayF has quit IRC22:30
*** rwsu_ has quit IRC22:30
*** edwarnicke has quit IRC22:30
*** doude has quit IRC22:30
*** sweston has quit IRC22:30
*** morgabra has quit IRC22:30
*** dmellado has quit IRC22:30
*** jcooley_ is now known as jcooley22:30
*** Rockyg has quit IRC22:30
*** anthonyper_ is now known as anthonyper22:30
*** morgabra has joined #openstack-infra22:30
*** darvon_ is now known as darvon22:30
*** kun_huang has joined #openstack-infra22:30
*** tteggel has joined #openstack-infra22:30
*** JayF has joined #openstack-infra22:30
*** rwsu_ has joined #openstack-infra22:30
*** dboik_ has joined #openstack-infra22:31
*** dmellado has joined #openstack-infra22:31
*** sweston_ is now known as sweston22:31
*** edwarnicke has joined #openstack-infra22:31
mordredjeblair: wow. nodepool is up to 0.1.022:31
*** unicell1 has joined #openstack-infra22:31
*** unicell has quit IRC22:31
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add migration to convert append only tables to MyISAM  https://review.openstack.org/17996422:32
sdake__jeblair can you ping me when you think I hsould try again22:32
shashankhegdeI noticed that one of my patches was not merged even though gerrit says that it is22:32
sdake__or say something gerneral in the channel i'll be watching :)22:33
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds refs/heads/stable/* to compute-hyperv  https://review.openstack.org/18018722:33
shashankhegdeAny idea what could be going on?22:33
shashankhegdeThe patch in question is: https://review.openstack.org/#/c/177061/22:33
*** subscope_ has quit IRC22:34
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds refs/heads/stable/* to networking-hyperv  https://review.openstack.org/18016022:34
*** dboik has quit IRC22:35
*** erikmwilson_ is now known as erikmwilson22:35
*** dboik_ has quit IRC22:35
*** Krinkle is now known as Krinkle|detached22:35
*** Sukhdev has joined #openstack-infra22:36
*** ddieterly has quit IRC22:36
*** tkelsey has joined #openstack-infra22:39
*** craigbr has quit IRC22:41
zaroclarkb: for some the rest api on my local instance of jenkins doesn't seem to work.  i'm guessing it's because i've started with maven or something.22:42
*** tkelsey has quit IRC22:43
zaroclarkb: i can't seem to test with jjb right now and I gotta take off.  can test more later tonight if you want to wait otherwise i have no problem if you want to get it in.22:43
clarkbzaro: ok thanks22:43
zaroclarkb: most of everthing i've look at looks good22:43
clarkbI can probably update my local testing to do some jjb pretty easily22:43
*** salv-orlando has quit IRC22:43
*** craigbr has joined #openstack-infra22:44
clarkbbtu trying to make sure the other issues get addressed since they seem more pressing right now22:44
zaroyeah, i would just check gearman-plugin continues to work with changes from jjb.  we had issues with that before.22:44
zaroyeah, i checked online/offline of slaves and looks good22:45
*** craigbr has quit IRC22:45
zarodid lots of stuff with re-registering of jobs and looks good too.22:45
*** craigbr has joined #openstack-infra22:47
shashankhegdeSo anyone has an idea why my patch was not actually merged even though jenkins says it has?22:49
*** alex_xu has quit IRC22:49
clarkbshashankhegde: can you expand on that? jenkins shouldn't claim mergedness22:50
clarkbshashankhegde: maybe point at the change?22:50
shashankhegdeHi clarkb , here is the review: https://review.openstack.org/#/c/177061/22:50
*** alex_xu has joined #openstack-infra22:51
shashankhegdeIts a stackforge repository22:51
clarkbshashankhegde: so gerrit says it has merged, can you point out whee it has not merged?22:51
shashankhegdeThe patch should have merged here: https://github.com/stackforge/networking-arista/blob/stable/kilo/requirements.txt22:52
*** EmilienM|afk is now known as EmilienM22:52
tchaypoI cloned the repo, and the tip looks like22:52
tchaypo*   6f62ae6 2015-04-24 Merge "Adding jsonrpclib to requirements.txt"  (HEAD, refs/remotes/origin/master, refs/remotes/origin/HEAD, refs/heads/master) <Jenkins>22:52
tchaypo| * a8788ac 2015-04-23 Adding jsonrpclib to requirements.txt  <Shashank Hegde>22:52
tchaypoit looks fairly merged to me22:52
tchaypoGithub is not where the repo lives22:52
tchaypogithub is a convenience mirror that gets updated sometimes22:52
shashankhegdehmmm... I actually branched from stable/kilo branch and submitted my review from there22:53
shashankhegdeAny idea why it merged to master?22:53
shashankhegdetchaypo: I see22:53
*** sarob has quit IRC22:53
tchaypoIf you look up the top of the Gerrit page, it has a “project” heading; if you click on that it will take you to https://review.openstack.org/#/admin/projects/stackforge/networking-arista which says..22:53
clarkbshashankhegde: your chage was against master22:53
tchaypoactually it says to clone from review.openstack.org and I don’t think that’s right either22:54
tchaypobut it any case http://git.openstack.org/cgit/stackforge/networking-arista/log/ shows the merge as well22:54
shashankhegdeclarkb: If I checkout "stable/kilo" and submit a review, shouldn't the change merge to the "stable/kilo" branch?22:55
clarkbshashankhegde: not necessarily, it depends on what branch you pushed the chagne against22:55
clarkbin this case the chagne was proposed against master as the gerrit change screen shows22:55
jeblairsdake__: there is an f21 node ready now22:55
tchaypothe parent commit is the one at the tip of the stable/kilo branch22:55
tchaypobut that commit is also part of the master branch22:56
fungiskimmed scrollback... i _think_ things look more under control22:56
tchayposhashankhegde: FWIW, if you’d submitted to stable/kilo it should (I think) have been rejected anyway22:57
*** emagana has quit IRC22:57
tchaypoassuming that project follows the guidelines at https://wiki.openstack.org/wiki/StableBranch22:57
shashankhegdeclarkb:, tchaypo, sorry, I am following http://docs.openstack.org/infra/manual/developers.html#committing-a-change and I can't seem to find how to propose change against a different branch22:57
*** Swami has joined #openstack-infra22:57
*** ZZelle_ has quit IRC22:58
jeblairtchaypo, clarkb, shashankhegde: i think we're missing info in infra-manual about submitting changes to alternate branches22:58
openstackgerritMerged openstack-infra/jenkins-job-builder: Revert "Added parallelization options"  https://review.openstack.org/18025822:58
*** emagana has joined #openstack-infra22:58
clarkbjeblair: appears to e the case22:58
jeblairshashankhegde: the short version is "git review stable/kilo", sorry that's not in the manual22:58
tchaypojeblair: we definitely don’t have anything in infra-manual, I was just hunting for it. But my understanding is that people should be submitting (except for cherry-picks) to stable branches anway?22:58
shashankhegdejeblair, thanks!22:59
jeblairtchaypo: i believe that's generally the case22:59
tchayposo perhaps we need a mention of how to push to an alternate branch, with a note that most projects won’t accept new code on a branch named “stable/*”?22:59
shashankhegdejeblair, tchaypo, The change is basically to back port something in master to kilo22:59
*** IAX has joined #openstack-infra22:59
shashankhegdeSo how do I go about cherrypicking that change?23:00
jeblairtchaypo: yeah, we should document how the tool works but be careful not to overly document the workflow in infra-manual23:00
tchaypoor even better, a pointer to “If you’re submitting to a stable branch, please follow the process at https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes”23:00
tchayposhashankhegde: can you take a look at https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes and let us know if it’s helpful?23:01
*** IAX has quit IRC23:01
tchaypoGetting someone unfamiliar with the process is usually a good way to get useful feedback :)23:01
tchaypojeblair: thanks for putting the conversation back on track23:01
jeblairfungi: indeed.  i found that geard had reverted to an old version on zuul.o.o; i fixed that manually and proposed a change to zuul to fix it more permanently.  that has resolved the geard load issues.23:01
*** bswartz has joined #openstack-infra23:02
jeblairfungi: once the jjb change lands, i think we can continue our work of cleaning up the jobs on masters23:02
*** annegent_ has quit IRC23:02
shashankhegdetchaypo: Give me 5 mins , I'll let you know23:02
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron  https://review.openstack.org/17968923:02
fungiinterested to know which puppet change caused that23:02
*** emagana has quit IRC23:02
jeblairfungi: so there's no rush for clarkb's change, just we should probably get it into prod before the next 'change every job' change23:02
clarkbya I can take a stab at rolling it onto jenkins07 as soon as everything else stabilizes23:03
jeblair(in the interim, we might leak a few ready nodes, but we'll be back at our background level for that, we can live with that for a little while longer)23:03
jeblairfungi: i have not been deleting old ready nodes though, sorry.23:04
jeblairoh looks like the jjb change has landed23:05
ianwso i'm not quite keeping up, sorry ... the NOT_REGISTERED issues for jobs are still happening?23:05
clarkbI deleted a few but not the entirety23:06
openstackgerritDavid Lenwell proposed openstack-infra/project-config: adding akanda horizon plugin to stackforge  https://review.openstack.org/18029423:06
*** btully has quit IRC23:06
jeblairianw: should probably be settled out at this point for newly enqueued changes23:06
jeblairapparently that's not true for a change enqueued 6 mins ago though23:06
ianwok so my 9:02 change just above got NOT_REGISTERED for centos7 ... let me try again23:06
*** camunoz has joined #openstack-infra23:06
jeblairah, old ready nodes there23:07
jeblairi will delete them23:07
jeblairianw: that should correct in 15-30 mins23:07
shashankhegdetchaypo, jeblair, clarkb: Thanks guys! it helped23:07
fungii'll check for more stale nodes23:08
ianwjeblair: ok, thanks23:08
*** wenlock_ has quit IRC23:09
jeblairi just cleared out the bare-precise nodes too23:09
*** emagana has joined #openstack-infra23:09
jeblair(basically stuck-ready nodes + zuul restart means low-traffic nodes don't end up registering; it's an edge case for the problems we're seeing and should be resolved by the solutions in progress)23:09
fungiianw: also puppet agent is probably still stopped on the jenkins masters. i think i need to circle back around and finish jjb runs on 3 of them23:10
*** alex_xu has quit IRC23:10
fungiso if that's a new job, it really may not be registered yet23:10
ianwfungi: no, it's just the regular job.  sounds like the stuck nodes issue23:10
*** alex_xu has joined #openstack-infra23:11
openstackgerritMerged openstack-infra/elastic-recheck: Add query for swift-object-server bug 1451992  https://review.openstack.org/18027823:11
openstackbug 1451992 in OpenStack Object Storage (swift) "swift-object-server fails to start in devstack which kills the gate" [Undecided,New] https://launchpad.net/bugs/145199223:11
tchayposhashankhegde: woo :)23:11
openstackgerritColleen Murphy proposed openstack-infra/project-config: puppet: run beaker on both trusty & centos7  https://review.openstack.org/17543423:12
*** jtriley has quit IRC23:12
jeblairfungi: i think we have inconsistent versions of jjb installed on masters :/23:13
*** emagana has quit IRC23:14
timrcjeblair, Is running multiple jenkins masters really as simple as bringing up a new Jenkins server, installing the gearman plugin, and pointing it at the gearman host?23:15
timrcgearman server host23:16
clarkbtimrc: yes23:16
jeblairtimrc: there's a tarball of configuration that we have not automated that we copy onto the new masters23:16
clarkbtimrc: just keep in mind that when you do so things like periodic jobs need to go through zuultoo23:16
jeblairclarkb: i think that's still the case ^ ?23:16
clarkbjeblair: yup23:16
fungijeblair: that also doesn't entirely surprise me23:16
jeblairtimrc: but in spirit, yes :)23:16
jeblairfungi: i will ansible them into shape23:16
fungisince gear is reupgraded, no performance concerns with me kicking off full jjb updates on 02, 04 and 07 once jjb is upgraded everywhere?23:18
*** panda|Zz has quit IRC23:18
jeblairfungi: nope, i think we'll just expect some more stuck ready nodes23:18
clarkbfungi: I don't think so23:18
*** panda|Zz has joined #openstack-infra23:19
jeblairhow do i ansible on jenkins, jenkins0* but not jenkins-dev?23:19
mordredjeblair: !jenkins-dev23:19
jeblairmordred: how do i combine that with the specifier for jenkins*?23:19
clarkbjeblair: its a regex if you prepend with ~23:19
mordredjeblair: one sec23:19
fungi~ totally says "regex" to me23:19
clarkbso ~jenkins[0-9]*.openstack.org I think23:20
clarkber ~jenkins[0-9]*.openstack\.org23:20
jeblairoh :23:20
clarkbfirst . needs \ too :/23:20
jeblairokay, i regexed it this time23:20
mordredso, "jenkins.openstack.org:jenkins01*:!jenkins-dev*"23:20
jeblairbut good to know both things23:21
fungii already knew half a dozen distinct regex grammars. what's one more?23:21
jeblair~ instead of ^.  i tried "," instead of ":".  :)23:21
*** shashankhegde has quit IRC23:21
jeblairso apparently, the ansible authors and i don't quite see eye to eye on hostnames.23:21
*** heyongli has quit IRC23:21
* fungi needs to remember to make a new kind of regex for whatever his next project is. just to keep people on their toes23:21
* jeblair mumbles something about ipv6 and colons23:21
fungiyeah, um, don't regex ipv6 addys in ansible i guess23:22
*** heyongli has joined #openstack-infra23:22
mordredI don't think they expect the ipv6 address to be the hostname23:22
mordredor the name of the group in which the host is defined to be23:22
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron  https://review.openstack.org/17968923:22
mordredjeblair: in theory, we should perhaps have a "jenkins-prod" group23:22
fungibest news of the day... zuul seems to still be catching up. nearly down to 500 jobs waiting now23:23
*** abregman has quit IRC23:23
jeblairjenkins-job-builder==  # git sha 5be7b7623:23
jeblairjenkins-job-builder==1.2.0.post47  # git sha 5be7b7623:23
jeblairjenkins-job-builder==1.2.1.dev40  # git sha 5be7b7623:24
mordredjeblair: taht's spectacular23:24
jeblairfrom 3 different hosts :(23:24
clarkbthats different pbrs23:24
mordredjeblair: I believe we want to upgrade pbr to the latest everywhere23:24
*** signed8b_ is now known as signed8bit_ZZZzz23:24
clarkbmordred: yup23:24
*** Krinkle|detached is now known as Krinkle23:25
jeblair'pip install -U pbr' does not accomplish that; do i need to give it a min?23:25
timrcclarkb, Periodic jobs going through zuul as opposed to running directly on the Jenkins server? Just want to be clear on that point.  AFAIK all of our periodic jobs are configured through zuul.23:26
*** jtriley has joined #openstack-infra23:26
jeblairclarkb, mordred: oh, pbr is latest everywhere23:26
jeblairclarkb, mordred: maybe it wasn't latest when jjb was installed though?23:27
mordredjeblair: yah. that would be it23:27
clarkbtimrc: yes, because otherwise you get mulitple copiues of the jobs running across your masters23:27
jeblairanyway, i think it's not important since the git shas are all the same23:27
clarkbtimrc: using zuul gets you a single set23:27
jeblairso let's call it done23:27
jeblairfungi: i think you're clear to proceed with jjb runs23:27
Krinklejeblair: FYI the failure on https://review.openstack.org/#/c/172241/ seems a false positive since the file I modified afaik doesn't influence tests of any kind.23:28
*** Krinkle is now known as Krinkle|detached23:29
fungijeblair: thanks, firing up23:29
timrcclarkb, Cool.  Thanks.23:29
fungipkg_resources.DistributionNotFound: ordereddict23:30
fungionce again pip fails us23:30
*** shashankhegde has joined #openstack-infra23:30
fungiseems to be running okay on 02 and 04 but the jjb install is broken on 0723:31
fungirerunning `sudo pip install -U .` in /opt/jenkins_job_builder seems to have fixed it23:32
jeblairKrinkle|detached: quite likely, thanks23:33
*** ashleighfarnham has quit IRC23:33
fungionce these jjb runs complete, i'll reenable puppet agent on the jenkins masters23:35
*** wenlock has quit IRC23:35
*** abregman has joined #openstack-infra23:38
*** Kennan2 is now known as Kennan23:38
*** craigbr has quit IRC23:39
Kennanhi :fungi23:39
*** apmelton is now known as apmelton_away23:40
*** IAX has joined #openstack-infra23:40
Kennanjust short question, if we have a branched named  kilo-stable , is it the devstack gatejob knows it would run against kilo branch ?23:40
*** abregman has quit IRC23:40
Kennanwe(our project)23:41
*** _nadya_ has joined #openstack-infra23:41
*** ajmiller has quit IRC23:41
IAXhi guys, just a quick question, do you have configured a bash shell lint job in jenkins? if so, what do you use?23:42
*** sdake__ has quit IRC23:42
clarkbIAX: we use bashate23:43
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron  https://review.openstack.org/17968923:43
fungiKennan: devstack-gate will try to test a change against branches of the same name in other projects23:44
*** salv-orlando has joined #openstack-infra23:44
fungiKennan: so if your branch is "kilo-stable" then devstack-gate isn't going to magically figure out that it should test that with branches named "stable/kilo" in other projects, no23:45
fungiKennan: you should probably have a "stable/kilo"23:45
fungilike other projects do23:45
Kennan:fungi ok Thanks23:45
*** _nadya_ has quit IRC23:46
fungiKennan: you're welcome23:46
IAXawesome, thank you clarkb!23:46
*** annegentle has joined #openstack-infra23:47
pleia2clarkb: any thoughts on my traceback re: https://review.openstack.org/#/c/179499/ ?23:47
*** annegentle has quit IRC23:47
pleia2(still on these docs testing, need this dependencies)23:48
pleia2er, dependency23:48
Kennanit seems different project can have differnt names23:50
Kennanit need self config23:50
Kennancheck this https://review.openstack.org/#/c/176907/23:50
Kennanyou reviewed this before23:51
*** salv-orlando has quit IRC23:51
*** rfolco has quit IRC23:52
fungiKennan: yes, you have to manually define a mapping. like in that configuration you're linking. devstack-gate won't simply figure it out unless it matches (which is preferable, as that's automatic)23:52
cineramapleia2: zanata folks marked the percentage bug with high severity, cool23:52
*** IAX has quit IRC23:53
pleia2cinerama: yay!23:53
*** dangers is now known as dangers_away23:56
clarkbpleia2: only that maybe your nodepool code is broken?23:59
clarkbpleia2: that is odd23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!