Thursday, 2015-12-10

clarkbits not only bluebox but bluebox is a good number of them00:01
clarkblet me double check security groups and do some due diligence first00:01
jeblairclarkb: (a bad image might cause this too)00:02
clarkboh I need to do the oic thing first before I forget and approve the elasticsearch index expiration00:02
greghaynesianw: oh, I totally forgot we had glean-install00:04
greghaynesianw: that makes things a ton easier00:04
*** markvoelker has quit IRC00:05
ianwgreghaynes: that's what 50-simple-init calls00:05
*** amuller has quit IRC00:05
*** Sukhdev_ has joined #openstack-infra00:06
*** dimtruck is now known as zz_dimtruck00:06
openstackgerritMichael Krotscheck proposed openstack/requirements: Updated gabbi to 1.11.0
clarkbok I tried submitting the thing and it doesn't do anything00:08
clarkb:/ going to respond to them00:08
clarkbvia email00:08
*** chlong has quit IRC00:09
Sam-I-Amclarkb: did you figure the secgroup stuff out?00:10
clarkbSam-I-Am: ya I think mordred said its all working the way he wanted00:10
Sam-I-Amok, thats cool00:10
*** akshai has quit IRC00:11
Sam-I-Amany other things i can help with there?00:11
openstackgerritMerged openstack-infra/puppet-logstash: Escape % in cron command
clarkbSam-I-Am: if you work with the innovation center for openstack or bluebox clouds maybe otherwise no I think we are good00:12
*** ashleighfarnham has quit IRC00:12
notmorgancrinkle: bootstrap sanely00:13
*** gokrokve has joined #openstack-infra00:13
*** alexsyip has joined #openstack-infra00:14
clarkbok email sent now back to debugging clouds00:14
Sam-I-Amclarkb: that'll never be done :)00:15
openstackgerritJames E. Blair proposed openstack-infra/zuul: Load in-repo configuration
jeblairokay ^ is really cool -- with that, when zuul starts up it will ask its mergers to return in-repo configuration files to it so that it can load its own config.  now *that* is a distributed application :)00:16
clarkbjeblair: it doesn't appear to be an image issue, of two nodes that were just building in bluebox with the devstack-trusty label one went ready and the other had ssh timeout00:17
clarkbwhich also implies that glean is working00:18
* clarkb double checks security groups now00:18
clarkbsorry its not glean00:18
clarkbits cloud-init but ya00:18
*** ddieterly has quit IRC00:19
pabelangerjeblair: Woah, that's pretty nice.00:20
clarkbsecurity groups look good00:21
*** tonytan4ever has quit IRC00:21
*** Qiming has quit IRC00:24
*** dingyichen has quit IRC00:24
openstackgerritHaomeng,Wang proposed openstack-infra/project-config: skip ironic client dsvm and pypy gates for doc patches
*** dingyichen has joined #openstack-infra00:24
openstackgerritHaomeng,Wang proposed openstack-infra/project-config: skip ironic client dsvm and pypy gates for doc patches
clarkb this is the shade swift image upload traceback00:28
clarkbmordred: greghaynes Shrews SpamapS other shadey people ^00:29
greghaynesseems shadey00:29
Shrewsclarkb: oh, i think that's a version issue somewhere00:29
Shrewscapabilities wasn't available in all versions00:29
clarkbShrews: versions of the client or the server?00:30
*** rossella_s has quit IRC00:30
clarkbthough its failing when trying to auth (based on traceback)00:30
Shrewsprobably need to specify swift version in cloud.yaml (just a guess)00:30
clarkbI would expect fail after that if it was a capabilities thing00:30
*** rossella_s has joined #openstack-infra00:30
greghaynesYea, the no tenant specified makes me think otherwise but need to read swift client first00:30
notmynamecapabilities discoverability (via /info) was added to swift in the 1.11.0 release, which was Dec 12, 201300:31
clarkb(I believe that all rax image uploads are currently broken as a result of this)00:31
Shrewsnotmyname: ah, maybe a client version issue then. but clarkb raises a good point on the tenant thing00:31
notmynameFWIW rax cloud files is reporting (via the /info endpoint) that it's 2.2.100:32
clarkbnotmyname: thanks for checking00:32
*** thorst has joined #openstack-infra00:32
greghaynesThis seems a lot like another project-id vs tenant-id vs etc issue00:32
SpamapSI tested the image upload bits in shade against rax00:34
SpamapS(since they were the only ones with swift image upload capability...)00:34
SpamapSseems like a client config issue.00:35
clarkbSpamapS: well it was working00:36
clarkbbecause we had successful uploads for a while00:36
*** ParsectiX has joined #openstack-infra00:37
*** ParsectiX has quit IRC00:37
*** ParsectiX has joined #openstack-infra00:37
SpamapSclarkb: did os-client-config maybe change under it?00:38
clarkbya this is a shade bug00:38
clarkbits not passing any args to ObjectCapibilities task00:38
Shrewsclarkb: shade version?00:38
Shrewslatest shade works against rax for me (for get_object_capabilites)00:38
clarkbShrews: I am looking at master code but can check what nodepool is using in a minute00:39
*** thorst has quit IRC00:39
*** gokrokve_ has joined #openstack-infra00:39
clarkbshade==1.3.0  # git sha 5662df100:39
tmcpeaklifeless: you around?00:39
Shrewsclarkb: alsow, swiftclient version?00:39
*** gildub has quit IRC00:40
clarkbpython-swiftclient==2.6.0  # git sha 7cd2a0100:40
clarkbos-client-config==1.12.0  # git sha cc92c92 for complteness00:40
Shrewshrm, exactly what i have00:40
clarkb that doesn't pass any args00:41
*** gokrokve has quit IRC00:41
clarkband if using v2 or v3 keystone auth swiftclient expects a tenante/project_id/name in os_options dict in the kwargs00:41
Shrewsbut... that works for me. something else is different00:41
SpamapSun tenante es muy importante00:41
clarkbShrews: are you uploading a 16GB image?00:43
Shrewsclarkb: can you share the clouds.yaml with me? (redact auth info though)00:43
clarkbI wonder if its  a timing thing00:43
Shrewsclarkb: no, i'm just calling get_object_capabilities()00:44
clarkbya I bet this is a timing thing00:44
greghaynesThat sounds highly plausible - token timeout and we dont pass the creds in to the client to re-auth00:44
clarkbShrews: getting you the clouds.yaml link its in puppet00:44
SpamapSwhats the timeout on rax tokens?00:45
Shrewsgreghaynes: hrm, that could be00:45
clarkbSpamapS: I am not sure00:45
Shrewsthat seems fine00:46
clarkbreading the code we _get_client the one time to make self._swift_client not None00:46
clarkbthen we don't seem to ever reup it00:46
crinklenotmyname: omg you are awesome00:46
crinkleer notmorgan00:46
crinklesorry notmyname00:46
*** tmcpeak has left #openstack-infra00:46
crinkleyou are probably also awesome00:46
notmorgancrinkle: :)00:47
SpamapSeveryone is awesome00:47
notmorgannotmyname: seriously, there are only a few channels we overlap in00:47
notmynamenotmorgan: curse you and your new nick! ;-)00:47
Shrewsclarkb: yeah, so long lived clients haven't really been tested. the likely culprit00:47
notmorgannotmyname: still better than mor<tab> in *every-single-channel*00:47
notmorgancrinkle: please try it out, let me know if it is sufficient00:48
clarkbShrews: well I know I fixed this once before :)00:48
*** sigmavirus24_awa is now known as sigmavirus2400:48
notmorgancrinkle: toally worth the 1.5hrs of hacking today to get it done.00:48
Shrewsclarkb: in shade?00:48
clarkbfor this exact same issue :(00:48
SpamapSyeah I kind of have some deja vu around this00:49
notmorgancrinkle: cause if we can kill admin_token... wooooooot00:49
clarkbjammielennox and mordred and I worked through this exact thing several months ago00:49
SpamapSbut then again I"m used to disappointing clarkb as a general rule, so that could be it.00:49
crinklenotmorgan: yesssss00:49
notmorganclarkb: what issue is happening?00:49
Shrewsclarkb: did you add a test? b/c if you didn't, it's broken  :)00:49
clarkbI am going to dig up logs now bceuse it may hvae magicalness in it00:49
* notmorgan scrolls up00:49
clarkbShrews: ok00:49
notmorganthis looks like something i should stare at00:49
Shrewsclarkb: i think that's a ttx saying00:49
*** gongysh has joined #openstack-infra00:50
clarkbI think I86cdb3cdd2710ef302520184ccfcb1605384f706 is the change that broke it00:50
clarkbanyways its beer 30 here now00:50
*** thorst has joined #openstack-infra00:50
notmorganclarkb: ooh one of these days we need to do beer-30 in the same location00:50
clarkbbut basically it used to always make a session which would always reup the token00:51
clarkb(if it was within some percentage of expiry or expired)00:51
notmorganoh image uploads argh...00:51
clarkbbut now it only does it if the client doesn't exist00:51
*** pvaneck has quit IRC00:52
notmorganyeah we should be crafting a session smartly00:52
notmorganso it will refresh tokens properly00:52
* notmorgan needs to go dig into this more to figure it out.00:52
* notmorgan will try to do so if possible.00:52
notmorganbut i have to do run errands :(00:52
clarkboh wait thats close but not quite it, swift_service will use auth_token() to get fresh tokens00:53
clarkbbut not swift_client00:53
notmorganclarkb: oh yeah swift_client doesn't have session support00:53
notmorganbut.... i think we have notmyname's buy in to give it session support if we don't break the other stuff00:54
notmorgani *think* if i recall the last convo00:54
clarkbnotmorgan: ok, cool00:54
notmorganshouldn't be too bad to do, since session also can act almost completly as a Requests proxy-through00:54
clarkbShrews: so uses the auth_token propery which will reup the token if needed00:55
notmorganso, if we are smart we can do a zero impact + keystoneauth capable swift_client easily00:55
clarkbShrews: but since swift_cliet doesn't do sessions that won't reup00:55
clarkband greghaynes says that swift_session doesn't do object capabilities?00:55
* notmorgan throws things at notmyname to confirm00:55
Shrewsclarkb: ah, k00:55
notmorganclarkb: the hack [if this solves it] is to create a new swift_client as needed [ugh...]00:56
*** thorst has quit IRC00:56
* notmorgan is *not* a fan00:56
*** olaph has quit IRC00:56
clarkbya so we could make swift_clien a property that always reups itself00:56
notmorganyeah that is the best stopgap00:56
clarkbor otherwise introspects the token and tries to be smart00:56
notmorganthe session support is going to be more involved and require releases00:56
*** thorst has joined #openstack-infra00:56
notmorganso, expect session in swift client soon(tm) but not tomorrow.00:57
notmynameyeah, we talked about keystone session in swiftclient in tokyo. first step is to get it passed in to the connection object.00:57
notmynamedefinitely not tomorrow00:57
notmynamebut soon (we all hope)00:57
notmorgannotmyname: discussed w/ mordred we have a plan to make it all work00:57
notmorgannotmyname: since he and I are attacking the keystoneauth.Session support everywhere00:57
notmynameoh good. then I'll just wait for the patches ;-)00:57
notmorgannotmyname: hah, we learned most people haven't sat down to understand session00:57
notmorganeasier for us to just help things along00:58
greghaynesclarkb: SpamapS Shrews We only call get object capabilities to get the size we have to split the objects - we should be able to do that all upfont so we dont have to renew a token00:58
notmynamenotmorgan: cool00:58
notmorgannotmyname: the thought is make connection based on KSA session00:58
notmorgannotmyname: since it can act just as a requests proxy w/o all the extra lookup00:58
notmorganthen if you construct w/ client.Client() [or whatever] you get full session support00:58
notmorganand the connection thing is transparent for those using the old interface00:58
notmynamenotmorgan: yeah. so it duplicates everything we do now. the benefit is that if you already have a session you don't have to do it again00:59
notmorganand i think we wanted to add a swift_auth plugin so we can encourage everyone to move to client.Client()00:59
notmorganlong term, even if they don't use keystone00:59
notmynamenotmorgan: the challenge was going to be either keeping auth v1 support or soemthing with oslo.config00:59
notmyname(those are always the challenges in swift ;-)01:00
notmorganah so KSA has no ties to oslo_config01:00
notmorganand we can append argparse etc things directly now01:00
notmorganbut we *can* handle oslo_config if it's requested01:00
SpamapSgreghaynes: you are correct01:00
notmorganwas one of the key things we chose to do with keystoneauth when migrating01:00
notmorganensure we had minimal/no oslo deps [except in testing]01:00
*** thorst has quit IRC01:01
notmorgannotmyname: confirming you have no issues with these deps:
greghaynesSpamapS: Shrews So probaly on image create we need to kick the swift client so we make a new one01:01
openstackgerritSachi King proposed openstack-dev/pbr: PASSENV integration environment variables re-enabling integration tests
greghaynesannoying but uploads are hours long anyhow so it isnt that big of a deal01:01
notmynamenotmorgan: likely not. not sure what stevedore brings in. I'm not familiar with that one. otherwise, yeah, its probably fine01:02
*** eil397 has quit IRC01:02
notmorgannotmyname: not a lot.01:02
notmorgannotmyname: it lets us do some magic lookups for plugins *if* neeed01:02
notmorganit's pretty light01:02
timburkenotmyname: more of hte same:
notmorganbasically pbr01:03
notmorgantimburke: ++01:03
notmynamenotmorgan: oh! you should talk to timburke :-)01:03
notmorganhi timburke !!01:03
notmynamenotmorgan: timburke is swiftclient core and that's where he spends most of his upstream time when he's not hacking on the company's product01:04
notmorgannotmyname: so i'll likely propose a "use keystoneauth under connection" patch as simple passthrough then we'll expand for the rest of the support01:04
notmorgantimburke: ^01:04
notmynamecool. sounds good01:04
notmorganprobably in the next ~week or so. depends on how off in the weeds i end up tyring to fix nova to use keystoneauth01:04
notmorgan(nova server not client)01:04
*** chlong has joined #openstack-infra01:05
timburkelove it. i started looking at how to use Session, didn't have a clear idea, and tabled it in favor of other things a bit ago. having someone who knows how to use it submit a patch probably *is* more expedient :)01:05
*** markvoelker has joined #openstack-infra01:05
notmorgantimburke: and that is almost everyone trying to use it atm.01:06
notmorgantimburke: it's a bit opaque01:06
notmorgantimburke: it's why we have OCC to be honest01:06
*** annegentle has joined #openstack-infra01:08
*** yuanying_ has joined #openstack-infra01:09
*** Qiming has joined #openstack-infra01:09
*** yuanying_ has quit IRC01:09
*** markvoelker has quit IRC01:11
*** yuanying has quit IRC01:11
*** zhenguo has joined #openstack-infra01:13
*** aeng has quit IRC01:14
openstackgerritTim Hinrichs proposed openstack-infra/project-config: Make python34 job voting for Congress
*** ggillies has quit IRC01:14
*** ggillies has joined #openstack-infra01:14
*** Sukhdev_ has quit IRC01:15
*** dingyichen has quit IRC01:16
*** sigmavirus24 is now known as sigmavirus24_awa01:16
*** loaf is now known as olaph01:18
openstackgerritClark Boylan proposed openstack-infra/shade: Make a new swift client prior to each image upload
*** camunoz has quit IRC01:19
clarkbShrews: SpamapS mordred greghaynes ^ so thats the "cheap" workaround01:19
clarkbbut I think that this is a larger issue for any long term user of shade since the only way around this outside of shade is to make a new OpenStackCloud for every operation? (maybe you can check the token yourself and only make a new cloud object when token is old)01:20
*** pahuang has quit IRC01:20
clarkbin shade we could also pass the auth details to every swift_client call and I think swift client will reauth if it has to01:21
clarkbthat seemed like a lot more work for late in the evening so started with quick and easy and we can go from there01:21
*** dingyichen has joined #openstack-infra01:21
*** camunoz has joined #openstack-infra01:22
*** ajmiller_ has quit IRC01:22
clarkbtimburke: notmorgan can you confirm that swiftclient will attempt to reconnect on its own if it has the creds? looks like that is where we hit the exception because we didn't pass in the necessary cred info01:22
*** chlong has quit IRC01:23
notmynameclarkb: if you can pass the token and or creds to swiftclient, it can reuse them and reauth as necessary. that would be halfway between reauth every time and use session for everything01:23
notmorganexcept the token doesn't extend expiration01:23
*** mtanino has quit IRC01:23
clarkbright, I think right now we passi ntoken only01:23
notmorganso, the expiration on the token is fixed (except in HP Cloud)01:23
notmorganso, passing creds would be needed01:23
*** yuanying has joined #openstack-infra01:23
notmynameI've got to pick up a kid, but I leave it in timburke's good hands ;-)01:24
clarkbya I need to pop out nowish too01:24
notmorganclarkb: commented on the fix01:24
notmorganclarkb: basically confirming what you said01:24
notmorganit's the inexpensive fix for "now"01:24
notmorganand hopefully we will be much better off in the soon-ish timeframe01:24
*** pahuang has joined #openstack-infra01:24
clarkbnotmorgan: thanks and as for beer yes we should01:25
timburkeclarkb: it does, though it's worth noting that won't work well for streamed objects.  we need to be able to reset the data source when we encounter the 40101:25
notmorgantimburke: so, just create a new client is the easiest solution for today?01:25
timburkeprobably? looking at the patch...01:28
notmorgantimburke: /me nods.01:29
*** Sukhdev has joined #openstack-infra01:30
*** tonytan4ever has joined #openstack-infra01:33
*** Daisy has joined #openstack-infra01:34
*** Swami has quit IRC01:35
*** akshai has joined #openstack-infra01:36
*** bharathm has quit IRC01:36
*** EricGonczer_ has joined #openstack-infra01:37
*** gongysh has quit IRC01:39
anteayaseems to me congress is describing a net split on the mailing list01:39
anteayaany alternate opinions?01:39
notmorgananteaya: fwiw the openstackgerrit bot was 15-30 seconds delayed for a bit earlier01:40
notmorgananteaya: it might be related?01:40
anteaya[openstack-dev] [Congress] [infra] IRC broken01:40
* notmorgan has no insight into the bots so just hazarding guesses01:40
anteayanotmorgan: thanks01:40
notmorganit does sound net-split-y01:41
*** akshai has quit IRC01:41
notmorganfrom that email01:41
anteayathe meeting seems to have started fine, according to the logs:
*** armax has quit IRC01:43
*** yamamoto has joined #openstack-infra01:43
*** dewsday has joined #openstack-infra01:45
openstackgerritSachi King proposed openstack-dev/pbr: Handle markers to support sdist on pip < 6
*** jamielennox|away is now known as jamielennox01:48
*** ildikov has quit IRC01:49
*** yamamoto has quit IRC01:49
*** Daisy has quit IRC01:51
*** Liuqing has joined #openstack-infra01:53
*** yamamoto has joined #openstack-infra01:54
*** zhenguo has quit IRC01:54
*** boris-42_ has quit IRC01:54
*** fdegir has quit IRC01:54
*** dolphm has quit IRC01:54
*** nikhil_k has joined #openstack-infra01:54
*** yamamoto has quit IRC01:54
*** johnthetubaguy has quit IRC01:54
*** dolphm has joined #openstack-infra01:54
*** nijaba has quit IRC01:55
*** zhenguo has joined #openstack-infra01:55
*** fdegir has joined #openstack-infra01:55
*** nikhil has quit IRC01:56
*** nijaba has joined #openstack-infra01:56
*** nijaba has joined #openstack-infra01:56
*** boris-42_ has joined #openstack-infra01:56
*** Qiming has quit IRC01:56
*** Qiming has joined #openstack-infra01:56
*** johnthetubaguy has joined #openstack-infra01:57
*** Sukhdev has quit IRC01:58
*** ayoung has quit IRC01:58
*** zhurong has joined #openstack-infra02:01
*** rcernin has quit IRC02:01
*** ildikov has joined #openstack-infra02:01
*** alexsyip has quit IRC02:02
*** tqtran has quit IRC02:04
*** salv-orl_ has joined #openstack-infra02:07
*** ayoung has joined #openstack-infra02:08
*** rhallisey has joined #openstack-infra02:09
*** salv-orlando has quit IRC02:10
*** davidlenwell has quit IRC02:10
*** zhurong has quit IRC02:11
*** davidlenwell has joined #openstack-infra02:13
*** rhallisey has quit IRC02:13
*** zhurong has joined #openstack-infra02:16
HenryGWhat causes a bug to be filed by "OpenStack Infra"? e.g. bug 152451002:19
openstackbug 1524510 in neutron " Remove Neutron FWaaS static example configuration files" [Medium,Confirmed] - Assigned to Martin Hickey (martin-hickey)02:19
*** EricGonczer_ has quit IRC02:30
*** EricGonczer_ has joined #openstack-infra02:32
*** EricGonczer_ has quit IRC02:32
*** links has joined #openstack-infra02:33
*** dewsday has quit IRC02:33
*** zhurong has quit IRC02:35
*** EricGonczer_ has joined #openstack-infra02:37
*** thorst has joined #openstack-infra02:38
*** zhurong has joined #openstack-infra02:39
*** sridhar_ram has quit IRC02:41
*** baoli has joined #openstack-infra02:43
*** dewsday has joined #openstack-infra02:44
*** chlong has joined #openstack-infra02:44
*** Liuqing has quit IRC02:46
openstackgerritDi Xu proposed openstack-infra/jenkins-job-builder: Support deep merge of defaults
*** gildub has joined #openstack-infra02:48
anteayaHenryG: that is very interesting02:49
HenryGanteaya: I am glad you think so :)02:50
anteayaHenryG: something told launchpad to take the commit message of that neutron-fwaas patch and create a bug against neutron with the contents of the commit message as the body of the bug02:50
anteayaHenryG: why did you mark the status as confirmed?02:50
HenryGanteaya: At first I did not realize the patch was already merged02:51
anteayait appears to be created as a result of the merge event02:52
HenryGanteaya: why would that happen?02:52
anteayattx dhellmann ^^ might the existance of 1524510 have anything to do with the change in release tools bug reporting behaviour?02:52
anteayaHenryG: well I don't entirely know, however I do know that the release team recently changed their bug reporting behaviour, rather than fix-commited and the fix-released they now just do fix-released (I think)02:53
*** MarkAtwood has quit IRC02:53
anteayaand I wonder if this bug (and if there are any others like it) may be an artifact of that change02:54
*** FallenPegasus has joined #openstack-infra02:54
HenryGsorry cat on keyboard02:54
anteayabut I don't know how to evaluate that02:54
anteayaI was going to ask what that emoticon was02:54
anteayacrying while hiking in a canyon perhaps02:54
HenryG:) could be02:55
anteayamy cat fortunately stays off the keyboard02:55
anteayaapparently my lap it much more appealing02:56
anteayaso all I have at this point HenryG is, hmmmmm02:56
anteayaand I'm tired and soon calling it a night02:56
StevenKI've been sprinting with lifeless this week. His two cats have walked over my keyboard a few times. :-)02:56
anteayahopefully someone else will chime in02:56
anteayaStevenK: ha ha ha02:57
anteayaHenryG: or share cat stories :)02:57
StevenKanteaya: I had to run git commit --amend, since one of the cats managed to hit enter after some random keys while I was in the middle of a commit.02:57
anteayaha ha ha02:57
anteayatroll kitty02:57
anteayalearning from lifeless :)02:58
*** thorst has quit IRC02:58
anteayanext time offer a patchset02:58
lifelessanteaya: :)02:58
anteayaand be sure to add co-authored-by02:58
*** tonytan4ever has quit IRC02:58
HenryGWe have two cats, they usually stay off desks/tables. Florence is on my lap but for some reason reached up and swatted right at the enter key.02:58
anteayaha ha ha02:59
anteayathey are learning from each other02:59
anteayaI'm off to bed03:00
*** EricGonczer_ has quit IRC03:00
anteayahappy continuation of cat stories03:00
StevenKanteaya: Sleep well!03:00
*** Apoorva has quit IRC03:00
anteayaStevenK: thanks :)03:00
anteayaStevenK lifeless have a productive day03:00
lifelessthank anteaya03:00
lifelessok so I need some advice03:00
HenryGGood night anteaya03:00
lifelessI'm updating the infra manual03:01
lifelessto document contraints03:01
lifelessI've done the obvious bits: changing references to python-jobs to python-constraints-jobs etc03:01
lifelessbut I'm struggling to find a good place to put the actual description of how to add constraints jobs to existing projects03:01
lifelessperhaps a new section in creators.rst?03:01
lifelessor should we just send mail to the list, and not worry about permanent docs for this? It is demonstrated in cookiecutter now...03:02
harlowjaqq, i'm looking at anddddd its suspiciously empty03:03
harlowjathe last build that merged has
*** tlian has quit IRC03:03
*** boris-42_ has quit IRC03:03
harlowjaso is that likely just something that the next merge will fix (when the docs build reruns?) or are there logs that i can look at for that (i think there are doc build logs, i just always forget where)03:04
*** ajo has quit IRC03:04
*** Liuqing has joined #openstack-infra03:05
*** markvoelker has joined #openstack-infra03:07
*** yamamoto has joined #openstack-infra03:10
*** markvoelker has quit IRC03:12
openstackgerritSachi King proposed openstack-dev/pbr: DNM: Testing - passenv integration environment variables re-enabling integration tests
*** ParsectiX has quit IRC03:13
*** dewsday has quit IRC03:14
*** unicell1 has quit IRC03:19
*** yuanying has quit IRC03:19
openstackgerritSachi King proposed openstack-dev/pbr: passenv integration environment variables re-enabling integration tests
*** aeng has joined #openstack-infra03:20
*** flepied has joined #openstack-infra03:22
*** annegentle has quit IRC03:24
openstackgerritAnusha Ramineni proposed openstack-infra/system-config: Add statusbot/eavesdrop to #congress channel
*** tjones has joined #openstack-infra03:27
*** eliqiao has quit IRC03:27
openstackgerritlifeless proposed openstack-infra/infra-manual: Update the manual to reference -constraints jobs.
*** Qiming has quit IRC03:28
*** Qiming has joined #openstack-infra03:29
openstackgerritSachi King proposed openstack-dev/pbr: Handle markers to support sdist on pip < 6
*** Sukhdev has joined #openstack-infra03:31
*** thorst has joined #openstack-infra03:31
*** gongysh has joined #openstack-infra03:31
*** thorst has quit IRC03:32
*** tsg has joined #openstack-infra03:35
*** yuanying has joined #openstack-infra03:37
*** gongysh has quit IRC03:37
*** baoli has quit IRC03:39
*** EricGonczer_ has joined #openstack-infra03:40
*** gyee has quit IRC03:41
*** EricGonczer_ has joined #openstack-infra03:41
*** gokrokve_ has quit IRC03:42
*** EricGonczer_ has quit IRC03:42
*** gongysh has joined #openstack-infra03:52
*** flwang1 has quit IRC03:55
*** rguillebert has quit IRC03:57
*** yuanying has quit IRC04:02
*** tjones has quit IRC04:04
*** davidlenwell has quit IRC04:06
*** yuanying has joined #openstack-infra04:06
*** gongysh has quit IRC04:06
*** timothyb89 has quit IRC04:08
*** timothyb89 has joined #openstack-infra04:08
*** davidlenwell has joined #openstack-infra04:09
*** yamahata has joined #openstack-infra04:11
*** gongysh has joined #openstack-infra04:13
*** jerryz has quit IRC04:17
*** gongysh has quit IRC04:17
*** flepied has quit IRC04:19
*** gongysh has joined #openstack-infra04:22
*** amotoki has quit IRC04:25
*** tjones has joined #openstack-infra04:28
*** vilobhmm11 has joined #openstack-infra04:30
*** rossella_s has quit IRC04:30
*** rossella_s has joined #openstack-infra04:30
*** kozhukalov_ has joined #openstack-infra04:37
*** tsg has quit IRC04:47
*** fawadkhaliq has joined #openstack-infra04:48
*** Ramanjaneya has joined #openstack-infra04:52
*** ParsectiX has joined #openstack-infra04:56
*** ParsectiX has quit IRC04:56
*** ParsectiX has joined #openstack-infra04:57
*** _denisra has quit IRC05:04
*** markvoelker has joined #openstack-infra05:07
*** Qiming has quit IRC05:08
*** armax has joined #openstack-infra05:11
*** markvoelker has quit IRC05:12
*** oomichi_away is now known as oomichi05:12
openstackgerritSachi King proposed openstack-dev/pbr: passenv integration environment variables re-enabling integration tests
*** FallenPegasus has quit IRC05:20
*** ashleighfarnham has joined #openstack-infra05:20
*** ildikov has quit IRC05:20
*** ashishb has joined #openstack-infra05:20
*** kozhukalov_ has quit IRC05:28
*** amotoki has joined #openstack-infra05:30
*** Qiming has joined #openstack-infra05:31
*** mohankumar has joined #openstack-infra05:36
*** harlowja_at_home has joined #openstack-infra05:37
*** Liuqing has quit IRC05:43
*** zhurong has quit IRC05:43
*** tjones has quit IRC05:43
*** zhurong has joined #openstack-infra05:44
*** Liuqing has joined #openstack-infra05:44
*** ParsectiX has quit IRC05:47
openstackgerritGal Sagie proposed openstack-infra/project-config: Add rally integration job to networking-ovn
*** red_trela is now known as red|05:51
*** yonglihe has quit IRC05:51
*** greghayn1 has joined #openstack-infra05:57
*** greghaynes has quit IRC05:58
*** greghayn1 is now known as greghaynes05:58
gsagieIf i want to call Neutron client in the gate for integration testing, which credentials/auth_url/tenant/name should i give it?05:58
*** greghaynes has quit IRC05:59
*** greghaynes has joined #openstack-infra06:00
*** links has quit IRC06:01
*** links has joined #openstack-infra06:01
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** ParsectiX has joined #openstack-infra06:03
*** ParsectiX has quit IRC06:03
*** ParsectiX has joined #openstack-infra06:03
*** Sukhdev has quit IRC06:04
*** greghaynes has quit IRC06:05
*** abregman has joined #openstack-infra06:05
*** greghaynes has joined #openstack-infra06:06
*** Qiming has quit IRC06:08
openstackgerritGal Sagie proposed openstack-infra/project-config: Add Kuryr Rally integration
*** gildub has quit IRC06:10
openstackgerritGal Sagie proposed openstack-infra/project-config: Add Dragonflow Rally integration job
*** aeng has quit IRC06:16
openstackgerritThomas Bechtold proposed openstack-infra/zuul: Fix documentation example
*** hdd has quit IRC06:22
*** unicell has joined #openstack-infra06:24
*** zhurong has quit IRC06:26
*** zhurong has joined #openstack-infra06:27
*** chlong has quit IRC06:28
*** jaosorior has joined #openstack-infra06:29
*** vilobhmm11 has quit IRC06:30
openstackgerritDi Xu proposed openstack-infra/jenkins-job-builder: Support deep merge of defaults
*** _nadya_ has joined #openstack-infra06:33
*** mgrosz has joined #openstack-infra06:34
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** vilobhmm11 has joined #openstack-infra06:40
*** red| is now known as red|trela06:42
*** red|trela has left #openstack-infra06:44
*** ildikov has joined #openstack-infra06:45
*** gildub has joined #openstack-infra06:47
*** unicell has quit IRC06:48
*** links has quit IRC06:49
*** unicell has joined #openstack-infra06:49
*** chlong has joined #openstack-infra06:49
*** armax has quit IRC06:57
*** ashleighfarnham has quit IRC07:01
*** vilobhmm11 has quit IRC07:01
openstackgerritDi Xu proposed openstack-infra/jenkins-job-builder: Support deep merge of defaults
*** chlong has quit IRC07:04
*** links has joined #openstack-infra07:05
*** pahuang has quit IRC07:06
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Added OpenID Apache configuration options
*** markvoelker has joined #openstack-infra07:08
*** flwang1 has joined #openstack-infra07:09
*** rcernin has joined #openstack-infra07:11
*** AJaeger has joined #openstack-infra07:12
*** markvoelker has quit IRC07:13
*** mgrosz has quit IRC07:17
*** chlong has joined #openstack-infra07:20
*** harlowja_at_home has quit IRC07:20
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** isaacb has joined #openstack-infra07:26
*** dims_ has joined #openstack-infra07:27
*** elo has quit IRC07:31
*** mohankumar has quit IRC07:31
*** nmagnezi has joined #openstack-infra07:31
*** kozhukalov_ has joined #openstack-infra07:32
*** ildikov has quit IRC07:39
*** _nadya_ has quit IRC07:41
*** _nadya_ has joined #openstack-infra07:43
openstackgerritMerged openstack-infra/project-config: Switch TripleO jobs to F22 jenkins nodes
*** salv-orl_ has quit IRC07:47
*** kozhukalov_ has quit IRC07:48
*** _nadya_ has quit IRC07:48
*** unicell has quit IRC07:48
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add yolanda to statusbot nicks
*** wshao has joined #openstack-infra07:52
ttxanteaya, HenryG: hmm... weird. this has nothing to do with the new release-tools thing (which is only run at release time)07:56
ttxCould be a side-effect of the 'direct-fix-released' change (where we set bugs to fixreleased instead of fixcommitted) but that code hasn't really changed and just posts a comment07:57
AJaegerrobcresswell: translation sync for horizon-cisco-ui are broken, I tried to fix it with but now the testsuite fails. Could you look at it, please?07:57
*** wshao has quit IRC07:57
*** fawadkhaliq has quit IRC07:58
ttxAlthough in that precise case it's the partial-bug thing, which doesn't set any status anyway, so that shouldn't have an effect07:58
*** fawadkhaliq has joined #openstack-infra07:58
*** mgrosz has joined #openstack-infra07:59
openstackgerritMerged openstack-infra/project-config: skip ironic client dsvm and pypy gates for doc patches
ttxanteaya, HenryG: apparently the infra user creates bugs for things like DocImpact tags:08:00
*** peterstac has quit IRC08:04
ttxYeah, that's the DocImpact code. Creates bugs where they are missing. Could be more explicit08:04
*** dansmith has quit IRC08:05
*** chlong has quit IRC08:05
*** jgriffith has quit IRC08:05
*** sirushti has quit IRC08:05
*** timfreund has quit IRC08:05
*** dougshelley66 has quit IRC08:05
*** mgrosz has quit IRC08:06
*** dansmith has joined #openstack-infra08:07
*** dansmith is now known as Guest1777908:07
*** matrohon has joined #openstack-infra08:08
*** jgriffith has joined #openstack-infra08:10
*** sirushti has joined #openstack-infra08:11
*** peterstac has joined #openstack-infra08:11
*** dougshelley66 has joined #openstack-infra08:11
AJaegerttx, HenryG, anteaya : neutron-fwaas has docimpact-group as neutron and thus we do not add the usual boilerplate for it.08:13
*** yamahata has quit IRC08:14
AJaegerHave a look at - where we add the boilerplate08:14
openstackLaunchpad bug 1523855 in kolla " Allow fetching plugins as part of the build" [Undecided,New]08:14
*** links has quit IRC08:14
*** markus_z has joined #openstack-infra08:14
*** egonzalez has joined #openstack-infra08:14
AJaegerttx, HenryG, anteaya should we add a boilerplate to each docimpact bug?08:15
* AJaeger cooks up a patch...08:16
openstackgerritAndreas Jaeger proposed openstack-infra/jeepyb: Always add prelude for docimpact bugs
AJaegerHenryG, ttx, anteaya ^08:18
*** timfreund has joined #openstack-infra08:19
*** fawadkhaliq has quit IRC08:21
openstackgerritMerged openstack-infra/glean: Basic logging
*** ParsectiX has quit IRC08:23
*** mpavone has joined #openstack-infra08:23
*** ParsectiX has joined #openstack-infra08:25
openstackgerritMerged openstack-infra/project-config: Make the ironic-pxe_ipa-ipxe voting
*** HeOS has joined #openstack-infra08:25
*** mgrosz has joined #openstack-infra08:26
*** arxcruz has joined #openstack-infra08:26
*** ParsectiX has quit IRC08:28
*** aopincar has joined #openstack-infra08:28
*** arxcruz has quit IRC08:28
*** arxcruz has joined #openstack-infra08:28
*** mgrosz has quit IRC08:31
*** ildikov has joined #openstack-infra08:31
*** oomichi is now known as oomichi_away08:32
*** oomichi_away has quit IRC08:32
*** e0ne has joined #openstack-infra08:32
*** elo has joined #openstack-infra08:35
*** _nadya_ has joined #openstack-infra08:39
*** ifarkas has joined #openstack-infra08:39
*** ifarkas has quit IRC08:39
*** ifarkas has joined #openstack-infra08:40
*** scheuran has joined #openstack-infra08:40
*** ParsectiX has joined #openstack-infra08:41
*** ParsectiX has quit IRC08:41
*** ParsectiX has joined #openstack-infra08:41
*** ParsectiX has quit IRC08:41
*** gongysh has quit IRC08:42
*** ParsectiX has joined #openstack-infra08:42
*** shardy has joined #openstack-infra08:43
*** mohankumar has joined #openstack-infra08:43
igorbelikovhey folks, could someone review/merge this please? It's just a simple switch to 'delay-release' option for fuel-* repos08:44
*** pblaho has joined #openstack-infra08:44
*** jlanoux has joined #openstack-infra08:46
*** salv-orlando has joined #openstack-infra08:48
*** sshnaidm has joined #openstack-infra08:48
*** kushal has joined #openstack-infra08:51
*** isaacb has quit IRC08:52
*** e0ne has quit IRC08:52
*** jordanP has joined #openstack-infra08:52
*** salv-orlando has quit IRC08:52
*** links has joined #openstack-infra08:52
*** kushal has quit IRC08:53
*** flwang1 has quit IRC08:53
*** kushal has joined #openstack-infra08:53
*** dingyichen has quit IRC08:59
*** fhubik has joined #openstack-infra09:01
gsagieHello, i want to use gate tests that uses openstack neutron client, what is the best way to retrieve the username/password and auth url to use?09:02
*** sfinucan has joined #openstack-infra09:02
*** mpavone has quit IRC09:03
*** hashar has joined #openstack-infra09:06
*** aopincar has quit IRC09:06
*** BobBall_AWOL is now known as BobBall09:07
*** hashar has quit IRC09:07
*** markvoelker has joined #openstack-infra09:09
*** jamielennox is now known as jamielennox|away09:09
*** yuanying has quit IRC09:10
*** hashar has joined #openstack-infra09:13
*** markvoelker has quit IRC09:14
*** zhurong has quit IRC09:14
*** zhurong has joined #openstack-infra09:15
*** mpavone has joined #openstack-infra09:15
openstackgerritMerged openstack-infra/project-config: Add python jobs for tacker client
AJaegergsagie: better discuss with #openstack-qa team09:17
gsagieAJaeger: k, thanks09:17
*** zeih has joined #openstack-infra09:18
*** zhurong has quit IRC09:18
openstackgerritMerged openstack-infra/project-config: Add watcher project to gerritbot
robcresswellAJaeger: Just saw those patches to cisco-ui, sorry about the incomplete setup09:25
*** ihrachys has joined #openstack-infra09:26
openstackgerritMerged openstack-infra/project-config: Add ansible-role-(diskimage-builder|jenkins-job-builder)
*** hogepodge has quit IRC09:26
*** elo has quit IRC09:27
robcresswellAJaeger: I'm confused by the secret_key change though09:28
*** hogepodge has joined #openstack-infra09:28
*** flwang1 has joined #openstack-infra09:30
*** openstackgerrit has quit IRC09:32
*** openstackgerrit has joined #openstack-infra09:32
*** ihrachys has quit IRC09:33
jd__#help for said AJaeger :]09:35
openstackgerritMerged openstack-infra/project-config: add seperate job for monasca plugin in devstack install
*** flwang1 has quit IRC09:36
*** dizquierdo has joined #openstack-infra09:37
*** hichihara has quit IRC09:40
*** links has quit IRC09:42
*** fawadkhaliq has joined #openstack-infra09:44
*** jaypipes has joined #openstack-infra09:49
*** jcoufal has joined #openstack-infra09:51
*** e0ne has joined #openstack-infra09:51
*** mpavone has quit IRC09:52
*** deepakcs has joined #openstack-infra09:52
deepakcsjohnthetubaguy: around09:52
*** cdent has joined #openstack-infra09:53
*** salv-orlando has joined #openstack-infra09:53
*** links has joined #openstack-infra09:55
*** kozhukalov_ has joined #openstack-infra09:57
*** ParsectiX has quit IRC09:58
AJaegerrobcresswell: might be a fall-out from adding django-babel to requirements?09:58
AJaegerrobcresswell: I'm lost on how to fix this properly, any advice is welcome on how to do it properly.09:58
*** Guest76732 has quit IRC09:59
*** jistr has joined #openstack-infra09:59
AJaegerHi jd__ , better ask during US morning...09:59
*** Hal has joined #openstack-infra09:59
*** Hal has quit IRC09:59
*** Hal has joined #openstack-infra10:00
*** Hal is now known as Guest7575310:00
*** ashtokolov has quit IRC10:01
*** briancurtin has quit IRC10:01
*** ashtokolov has joined #openstack-infra10:01
jd__AJaeger: ah I should move :p10:02
*** smcginnis has quit IRC10:02
robcresswellAJaeger: I'm just looking now :)10:02
AJaegerjd__: No - just ask questions when other experts are around ;)10:02
*** mpavone has joined #openstack-infra10:03
*** smcginnis has joined #openstack-infra10:03
jd__AJaeger: this sounds too easy ;)10:03
AJaegerthanks, robcresswell. I looked myself and got parts working but not all.10:03
*** briancurtin has joined #openstack-infra10:03
*** aopincar has joined #openstack-infra10:03
*** links has quit IRC10:04
*** rguillebert has joined #openstack-infra10:05
*** zhenguo has quit IRC10:06
*** placko has joined #openstack-infra10:06
*** abregman is now known as abregman|shawarm10:07
dims_AJaeger : what does gate-grenade-dsvm-forward do? (i see it for example in python-novaclient)10:08
*** samuelBartel has joined #openstack-infra10:08
AJaegerdims_: introduced in Iee18bf97a42dab3cdbef3066446b540e77072d73 , ask sdague if the commit message does not help...10:09
dims_AJaeger : y didn't help :) will ask10:10
*** k4n0 has joined #openstack-infra10:15
*** k4n0 has quit IRC10:15
*** links has joined #openstack-infra10:17
*** electrofelix has joined #openstack-infra10:19
*** Liuqing has quit IRC10:19
tsufievhello there!10:19
tsufievany idea why a patch doesn't get to zuul queue?10:19
*** kozhukalov_ has quit IRC10:20
tsufievspecifically looking for at -> nothing there10:20
*** erlon has joined #openstack-infra10:21
*** fhubik is now known as fhubik_brb10:23
*** sheeprine has quit IRC10:24
*** markvoelker has joined #openstack-infra10:25
*** bexelbie_ has joined #openstack-infra10:26
*** sambetts-afk is now known as sambetts10:27
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Experimental grenade jobs for oslo.messaging
dims_AJaeger : decided to just do an experimental for now to kick the tires10:28
*** markvoelker has quit IRC10:29
*** dtantsur|afk is now known as dtantsur10:30
AJaegerdims_: talk to sdague about 255782, syntax looks fine..10:31
dims_ack will do thanks!10:32
AJaegertsufiev: zuul queue shows "Queue lengths: 133 events, 121 results. "10:32
AJaegerAs long as that is not both 0, it's not complete. So, wait a bit...10:32
*** kozhukalov_ has joined #openstack-infra10:33
*** sheeprine has joined #openstack-infra10:33
*** fhubik_brb is now known as fhubik10:34
AJaegertsufiev: queue is down to zero but job is still not there ;(10:35
hasharsomehow Zuul managed to flush its results :D10:35
*** ildikov is now known as ildikov_afk10:36
AJaegerinfra-root, any ideas why 224756 gets ignored?10:36
tsufievAJaeger, this is IMO at least 2 time it happens with this patch10:37
AJaegertsufiev: we have to wait for somebody with access to log files to check this...10:37
*** _nadya_ has quit IRC10:39
tsufievAJaeger, ok. I'd prefer to wait and find a root cause than resubmit the patch with a different change id10:39
*** _nadya_ has joined #openstack-infra10:41
*** Liuqing has joined #openstack-infra10:42
AJaegertsufiev: agreed10:43
AJaegertsufiev: we just need to wait for one of the other admins, seems nobody is around right now... Should be different in a few hours - and I expect this can wait10:44
tsufievAJaeger, sure10:44
yolandaAJaeger, i cannot see obvious reason10:44
yolandai am here now, i was busy with other things10:44
*** jyuso1 has joined #openstack-infra10:44
yolandai can check zuul logs10:44
AJaegerthanks, yolanda !10:45
AJaegeryolanda: I checked project-config and don't see any obvious reason either10:45
yolandayes, cannot see anything, also no related dependency that could cause it10:46
* AJaeger check zuul debug output, looks all fine there as well10:48
AJaegerthe patch on top of it has the same problem:
AJaegerother changes for the repo are tested10:51
yolandayep, nothing obvious in zuul10:51
AJaegerlet's wait for fungi or clarkb...10:52
*** dizquierdo has quit IRC10:53
*** flepied has joined #openstack-infra10:54
yolanda2015-12-10 08:23:11,184 DEBUG zuul.Gerrit: Getting commit-dependent change 222129,1010:54
yolanda2015-12-10 08:23:11,185 ERROR zuul.GerritEventConnector: Exception moving Gerrit event:10:54
yolandaTraceback (most recent call last):10:54
yolanda  File "/usr/local/lib/python2.7/dist-packages/zuul/trigger/", line 112, in run10:54
yolanda    'approval': toList(variable_dict),10:54
yolanda  File "/usr/local/lib/python2.7/dist-packages/zuul/trigger/", line 103, in _handleEvent10:54
yolanda    'comment_filter': toList(str),10:54
*** jaypipes has quit IRC10:55
yolandaand also10:56
yolandaException: Dependency cycle detected: 222131 in [u'222132', u'222131']10:56
openstackgerritIgor Malinovskiy proposed openstack-infra/project-config: Use Manila specific job templates for building images
yolandaAJaeger, seems that the chain of patches is wrong10:56
AJaegerit gets confused...11:00
AJaegertsufiev: can you unabandon 222131, please?11:01
AJaegerThen remove the depends-on from that commit and abandon again?11:01
AJaegerThe change has the same change-id as yours and Zuul stumbles over this.11:01
AJaegerand 222131 has a cyclic dependency on itself which is forbidden11:01
AJaegerfungi, clarkb, jeblair: Is the above something you want to fix in Zuul: change in Project A with cyclice dependency, abandoned; change in Project B with same change-id as the first patch; change on top of the project B one....11:02
*** nelsnelson has joined #openstack-infra11:03
AJaegertsufiev: are still around?11:03
AJaegeryolanda: great detective work, thanks!11:03
*** dims_ has quit IRC11:04
*** _nadya_ has quit IRC11:04
*** nelsnels_ has quit IRC11:06
*** kozhukalov_ has quit IRC11:07
*** binbincong has quit IRC11:08
*** binbincong has joined #openstack-infra11:08
openstackgerritMerged openstack-infra/project-config: Add 'delay-release' option to fuel projects
hasharZuul doc states:  "If a cycle is created by use of CRD, Zuul will abort its work very early. There will be no message in Gerrit and no changes that are part of the cycle will be enqueued into any pipeline. This is to protect Zuul from infinite loops."11:10
hashar(ref )11:10
*** claudiub has joined #openstack-infra11:10
hasharwhich is fairly annoying :-}11:10
hasharwell done yolanda !11:10
*** abregman|shawarm is now known as abregman11:12
*** yamamoto has quit IRC11:15
tsufievAJaeger, here, doing as you said11:17
*** aysyd has joined #openstack-infra11:18
*** nelsnels_ has joined #openstack-infra11:18
*** cdent has quit IRC11:18
tsufievyolanda, thanks from my side as well )11:19
yolandaso did it help? you'll need to refactor the chain of patches?11:19
gsagieAJaeger: just a question, if i use in my gate job export DEVSTACK_GATE_TEMPEST=1 , does it mean it will run all tempest tests? i want a job that runs only tests i have in my repository11:19
tsufievyolanda, yes, it did, the change has appeared in queue11:20
*** nelsnelson has quit IRC11:21
openstackgerritEvgeniy L proposed openstack-infra/system-config: Add statusbot/eavesdrop to #openstack-bareon channel
*** kozhukalov_ has joined #openstack-infra11:22
gsagieanyone else can also answer :)11:25
johnthetubaguydeepakcs: sorry, missed your ping for some reason, but I am around today11:26
*** aopincar has quit IRC11:28
*** lucas-dinner is now known as lucasagomes11:28
deepakcsjohnthetubaguy: just wanted to check with you, about the NV job i added, its been running good ( see
deepakcsjohnthetubaguy: if it continues to work fine, i plan to enable the same for Nova and Glance too, hope thats fine with you11:32
deepakcsjohnthetubaguy: you had raised concern about not adding the job for all 3 at the same time, hence following up with u11:33
johnthetubaguydeepakcs: I think so, what was the main difference between the old job and the new job again?11:33
*** mwagner_lap has quit IRC11:34
johnthetubaguydeepakcs: I was more worried about infra capacity, but now we are past the milestone week, there should be less pressure on those, I assume.11:35
openstackgerritMerged openstack-infra/project-config: Add reno jobs for python-saharaclient
openstackgerritChangBo Guo(gcb) proposed openstack-dev/cookiecutter: Don't include by default
AJaegertsufiev: glad to see it working!11:44
tsufievAJaeger, yep, thank you for the quick response! Will keep the thing about cyclic deps in mind11:44
openstackgerritGal Sagie proposed openstack-infra/project-config: Change tempest jobs for Kuryr
deepakcsjohnthetubaguy: the difference was the devstack plugin methodology, old uses hook-based approach which is beign deprecated, new uses devstack-plugin approach11:49
johnthetubaguydeepakcs: ah, right, thanks, I remember now11:50
deepakcsjohnthetubaguy: ok then, i will work towards enabling the new job on the other 2, thanks11:52
*** markus_z is now known as markus_z_meeting11:52
*** flepied has quit IRC11:55
*** sputnik13 has quit IRC11:57
*** fhubik is now known as fhubik_brb11:59
*** ppai has joined #openstack-infra12:00
*** fhubik_brb is now known as fhubik12:00
*** fawadkhaliq has quit IRC12:01
odyssey4meafternoon all12:04
odyssey4meyolanda would you mind adding to your review list please12:04
yolandayep, it's on the queue :)12:05
odyssey4methanks yolanda :)12:06
*** gildub has quit IRC12:08
openstackgerritPrashanth Pai proposed openstack-infra/project-config: Allow branch creation for swiftonfile project
openstackgerritIgor Belikov proposed openstack-infra/project-config: Bring back python27 checks for fuel-specs
openstackgerritIgor Belikov proposed openstack-infra/project-config: Bring back python27 checks for fuel-specs
*** _nadya_ has joined #openstack-infra12:10
*** _nadya_ has quit IRC12:11
ppaihello infra folks! could any of core reviewers please review this: Thanks :)12:12
*** fhubik is now known as fhubik_brb12:13
*** fhubik_brb is now known as fhubik12:14
*** kozhukalov_ has quit IRC12:14
*** igorbelikov has quit IRC12:16
openstackgerritMerged openstack/requirements: Updated from generate-constraints
*** fhubik is now known as fhubik_brb12:19
hasharyour poor zuul is deadlocked :(   Queue lengths: 942 events, 518 results.12:20
hasharthe gate has a bunch of changes that had all jobs ran but are not processed :/12:20
AJaegerhashar: the numbers are still changing, I doubt it's a deadlock...12:21
*** igorbelikov has joined #openstack-infra12:21
*** salv-orlando has quit IRC12:22
hasharwell it is pilling up for sure12:23
*** eliqiao has joined #openstack-infra12:23
AJaegeryeah, kind of busy ;)12:24
hasharQueue lengths: 2060 events, 658 results.12:24
hasharyeah your install is very busy12:24
AJaegerhashar: peanuts ;)12:24
*** fhubik_brb is now known as fhubik12:25
*** weshay_xchat has joined #openstack-infra12:25
*** markvoelker has joined #openstack-infra12:26
*** kozhukalov_ has joined #openstack-infra12:27
*** rossella_s has quit IRC12:30
*** markvoelker has quit IRC12:30
*** rossella_s has joined #openstack-infra12:30
*** ociuhandu has joined #openstack-infra12:31
*** Guest75753 has quit IRC12:31
*** doug-fish has joined #openstack-infra12:32
*** fhubik is now known as fhubik_brb12:35
*** Daisy has joined #openstack-infra12:36
*** kozhukalov_ has quit IRC12:37
*** weshay_xchat is now known as weshay12:39
*** sambetts has quit IRC12:40
*** tlian has joined #openstack-infra12:40
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Don't take into account commit message for skip-if filter
*** Daisy has quit IRC12:41
*** sambetts has joined #openstack-infra12:41
*** deepakcs has quit IRC12:42
*** Daisy has joined #openstack-infra12:44
*** baoli has joined #openstack-infra12:45
openstackgerritMerged openstack-infra/project-config: Added new projects for the OSA role break out
*** jaosorior has quit IRC12:47
*** baoli_ has joined #openstack-infra12:47
*** openstackgerrit has quit IRC12:47
*** jaosorior has joined #openstack-infra12:47
*** Hal has joined #openstack-infra12:47
*** Daisy has quit IRC12:47
*** thorst has joined #openstack-infra12:48
*** openstackgerrit has joined #openstack-infra12:48
*** Hal is now known as Guest2101912:48
*** Daisy has joined #openstack-infra12:48
odyssey4methanks yolanda12:49
*** baoli has quit IRC12:50
*** Daisy has quit IRC12:52
*** 7F1AA6H41 has joined #openstack-infra12:52
*** ddieterly has joined #openstack-infra12:52
*** EricGonczer_ has joined #openstack-infra12:53
*** kozhukalov_ has joined #openstack-infra12:53
*** fhubik_brb is now known as fhubik12:55
*** amuller has joined #openstack-infra12:57
*** notmyname has quit IRC12:57
*** flwang has quit IRC12:57
*** dmellado has quit IRC12:57
*** crinkle has quit IRC12:57
*** flwang has joined #openstack-infra12:57
*** calebb has quit IRC12:57
*** notmyname has joined #openstack-infra12:57
*** crinkle has joined #openstack-infra12:57
*** dmellado has joined #openstack-infra12:57
*** yamamoto has joined #openstack-infra12:59
kozhukalov_guys, could you please add me to the group,members I am the author of the patch Thanks a lot in advance.13:00
kozhukalov_AJaeger, yolanda ^^13:01
*** yamamoto_ has joined #openstack-infra13:01
*** gongysh has joined #openstack-infra13:02
*** skraynev has quit IRC13:02
kozhukalov_and into this group as well,members13:02
*** jffische has joined #openstack-infra13:03
*** flepied has joined #openstack-infra13:03
*** Qiming has joined #openstack-infra13:03
*** skraynev has joined #openstack-infra13:03
*** markus_z_meeting is now known as markus_z13:04
*** ParsectiX has joined #openstack-infra13:04
*** flepied1 has joined #openstack-infra13:05
*** yamamoto has quit IRC13:05
*** mwagner_lap has joined #openstack-infra13:06
*** flepied has quit IRC13:07
*** markvoelker has joined #openstack-infra13:08
*** bexelbie_ is now known as bexelbie13:09
*** ParsectiX has quit IRC13:10
*** ildikov_afk is now known as ildikov13:10
*** EricGonc_ has joined #openstack-infra13:14
*** yamamoto_ has quit IRC13:14
*** yamamoto has joined #openstack-infra13:15
*** ParsectiX has joined #openstack-infra13:15
*** gordc has joined #openstack-infra13:17
*** EricGonczer_ has quit IRC13:17
*** ParsectiX has quit IRC13:20
*** fhubik is now known as fhubik_brb13:20
*** ashishb has quit IRC13:21
*** dprince has joined #openstack-infra13:21
yolandakozhukalov, added13:21
*** bcourt has joined #openstack-infra13:22
*** markmcclain has quit IRC13:23
*** trown|outttypeww is now known as trown13:24
*** fhubik_brb is now known as fhubik13:26
*** mriedem_away is now known as mriedem13:26
*** ppai has quit IRC13:27
*** jaypipes has joined #openstack-infra13:28
*** jaypipes has quit IRC13:28
openstackgerritThierry Carrez proposed openstack-infra/system-config: Discontinue Release status page
*** pradk has joined #openstack-infra13:29
*** markmcclain has joined #openstack-infra13:34
*** fhubik has quit IRC13:35
*** kozhukalov_ has quit IRC13:37
*** regXboi has joined #openstack-infra13:38
*** jyuso1 has quit IRC13:41
*** calebb has joined #openstack-infra13:42
claudiubAJaeger: hello. I have a question. How do I disable py3 jobs on older branches? For example, for networking_hyperv, we've introduced python 3 compatibility in liberty, but the jobs still run on kilo and they shouldn't.13:43
*** dims has joined #openstack-infra13:43
*** Ramanjaneya has quit IRC13:43
AJaegerclaudiub: something similar to
*** bcourt has quit IRC13:44
AJaegerclaudiub: see also
*** mohankumar has quit IRC13:45
claudiubAJaeger: interesting. thanks!13:45
*** derekh has joined #openstack-infra13:45
sdaguemriedem and I have a mystery around git proposal bot things13:46
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Don't take into account commit message for skip-if filter
*** dingyichen has joined #openstack-infra13:47
*** ldnunes has joined #openstack-infra13:47
*** Daisy has joined #openstack-infra13:47
*** kozhukalov_ has joined #openstack-infra13:48
mriedemnova was synced with this reqs change
mriedembut failed tests because it wasn't in the requirements master that was cloned13:48
*** dims has quit IRC13:48
mriedemit pulled in the commit right before the change was merged to the repo13:48
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Disable stable/kilo python3 jobs for networking_hyperv
*** zhiyan has quit IRC13:51
AJaegerclaudiub: " branch: branch:" is wrong, once is enough ^13:52
*** zhiyan has joined #openstack-infra13:53
*** kozhukalov_ has quit IRC13:54
AJaegersdague, mriedem: Looking at the time stamps, 254911 merged at 21:33 but the marked change is "20:09:34.592"13:54
*** ddieterl_ has joined #openstack-infra13:54
claudiubAJaeger: ah, sorry, didn't see that.13:55
*** flepied1 has quit IRC13:55
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Disable stable/kilo python3 jobs for networking_hyperv
*** dims_ has joined #openstack-infra13:57
*** openstackstatus has quit IRC13:57
*** links has quit IRC13:57
*** Daisy has quit IRC13:58
*** sc68cal has quit IRC13:58
*** Daisy has joined #openstack-infra13:58
*** ddieterly has quit IRC13:58
*** sigmavirus24_awa is now known as sigmavirus2413:59
*** bcourt has joined #openstack-infra13:59
*** nikhil_k is now known as nikhil14:00
*** 7F1AA6H41 has quit IRC14:00
*** sc68cal has joined #openstack-infra14:01
*** sc68cal has quit IRC14:01
*** sc68cal has joined #openstack-infra14:01
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch to F22 jobs
*** Daisy has quit IRC14:01
*** Daisy has joined #openstack-infra14:01
*** mohankumar has joined #openstack-infra14:01
*** julim has quit IRC14:03
*** ddieterl_ has quit IRC14:03
*** peristeri has joined #openstack-infra14:03
*** julim has joined #openstack-infra14:03
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Disable stable/kilo python3 jobs for compute_hyperv
openstackgerritGal Sagie proposed openstack-infra/project-config: Add rally integration job to networking-ovn
*** rlandy has joined #openstack-infra14:05
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version
*** alivigni has joined #openstack-infra14:06
*** lykinsbd has joined #openstack-infra14:07
*** julim has quit IRC14:08
*** kgiusti has joined #openstack-infra14:08
AJaegerclaudiub: there's a special Python 3.4 jobs section strting around 1894, could you put both of your changes there, please?14:10
claudiubAJaeger: sure14:10
*** gongysh has quit IRC14:13
*** esker has quit IRC14:13
sdagueAJaeger: but I don't understand how that would happen, given that it was supposed to have been triggered by that14:14
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Disable stable/kilo python3 jobs for compute_hyperv
sdagueAJaeger: basically there seems to be a really weird timing issue here14:14
AJaegersdague: I'm confused as well...14:15
*** mpavone has quit IRC14:15
*** lykinsbd_ has joined #openstack-infra14:15
AJaegersdague: could I bother you with a trivial nova review, please? - caught while translating nova for Liberty...14:16
*** fawadkhaliq has joined #openstack-infra14:16
openstackgerritMerged openstack-infra/project-config: Experimental grenade jobs for oslo.messaging
*** fawadkhaliq has quit IRC14:17
*** xyang1 has joined #openstack-infra14:17
*** scheuran has quit IRC14:17
*** yamamoto has quit IRC14:17
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Disable stable/kilo python3 jobs for networking_hyperv
*** fawadkhaliq has joined #openstack-infra14:17
openstackgerritJulia Kreger proposed openstack-infra/shade: Add wait support for ironic node [de]activate
*** lykinsbd has quit IRC14:18
claudiubAJaeger: done. :)14:18
*** amitgandhinz has joined #openstack-infra14:19
AJaegerclaudiub: thanks, will review in a few minutes...14:19
*** yamamoto has joined #openstack-infra14:20
AJaegerclaudiub: for the other one as well, please14:21
*** tlian has quit IRC14:21
*** tlian has joined #openstack-infra14:22
*** yamamoto has quit IRC14:22
claudiubAJaeger: compute-hyperv is at L1897 and networking-hyperv is at L1922, which is still in the same section14:23
*** annegentle has joined #openstack-infra14:23
*** tlian has quit IRC14:23
AJaegerclaudiub: ah, great...14:23
*** yamamoto has joined #openstack-infra14:24
*** flepied has joined #openstack-infra14:24
mriedemmordred: re:
*** yamamoto has quit IRC14:26
dims_thanks AJaeger and sdague14:26
*** flepied1 has joined #openstack-infra14:26
*** annegentle has quit IRC14:27
*** flepied has quit IRC14:29
*** salv-orlando has joined #openstack-infra14:30
*** tlian has joined #openstack-infra14:30
*** ajo has joined #openstack-infra14:31
*** jcoufal has quit IRC14:34
*** mohankumar has quit IRC14:34
*** jcoufal has joined #openstack-infra14:34
*** lucasagomes is now known as lucas-hungry14:35
*** ddieterly has joined #openstack-infra14:35
*** Guest17779 is now known as dansmith14:36
*** dansmith is now known as Guest1812414:37
*** rhallisey has joined #openstack-infra14:37
*** Guest18124 is now known as dansmith14:37
*** yamamoto has joined #openstack-infra14:38
*** dkranz has joined #openstack-infra14:39
*** melania has joined #openstack-infra14:40
yolandahi melania14:40
*** daemontool has joined #openstack-infra14:46
*** mtanino has joined #openstack-infra14:48
*** salv-orl_ has joined #openstack-infra14:49
clarkbAJaeger yolanda, I think that zuul behavior is mostly by design. Do we know why the changid was used across projects? wonder if there is a use case here we havent considered14:49
* regXboi wanders in with a few questions :)14:49
AJaegerclarkb: the change was first done for horizon, abandoned and done for django_openstack-auth, reusing the changeid14:49
regXboiin the build node string - ovh = ????14:49
*** kushal has quit IRC14:50
clarkbregXboi ovh is the cloud provider14:50
AJaegerregXboi: ovh ;)14:50
AJaegerregXboi: ovh.com14:51
*** mohankumar has joined #openstack-infra14:51
clarkbAJaeger gotcha, so probably more a corner case than a regular workflow14:51
AJaegerclarkb: yes.14:51
*** psc_test has quit IRC14:51
*** salv-orlando has quit IRC14:51
regXboiI ask because I think I'm seeing live migration tests fail on that provider only14:51
*** EricGonc_ has quit IRC14:51
regXboinote *think* - I'm trying to run it down14:52
AJaegerwith the extra discussion we had afterwards, I consider it as a corner case and would leave it as is for now.14:52
clarkbregXboi recently or did it stop failing in the last day or so?14:52
regXboithis is over the last seven days14:52
clarkbregXboi there was a failure to set host names properly14:52
clarkband nova uses that as a unique identifier so when it isnt unique nova breaks14:52
*** mmedvede has quit IRC14:53
clarkbbut shouldve been fixed a day or so ago14:53
*** mmedvede has joined #openstack-infra14:54
*** julim has joined #openstack-infra14:54
regXboiclarkb: so, if I see an error caused by a difference between devstack-trusty-2-node-ovh-gra1-6457320 and devstack-trusty-2-node-ovh-gra1-6457320-73372 ??14:55
clarkbthe issue was both haf a hostname of "ubuntu"14:55
clarkband when that happens nova gets confused because nova requires unique hostnames14:56
*** bcourt has quit IRC14:56
*** j_king_ is now known as j_king14:56
regXboiso... I think we are talking about two different errors - because the one I'm looking at looks to be from less that an hour ago14:57
*** pradk has quit IRC14:57
*** pradk_ has joined #openstack-infra14:57
*** pradk_ is now known as pradk14:58
clarkbok then itis probably different14:58
regXboiclarkb: if you have a cycle, the error can be seen in
*** boris-42_ has joined #openstack-infra14:59
regXboiclarkb: sorry, let me get more specific:
*** amotoki has quit IRC15:02
clarkbthe compute node is saying it doesnt know about itself, so possibly stil name related but it at least seems to get the name right15:04
*** kozhukalov_ has joined #openstack-infra15:04
regXboiclarkb: so it is name related, but we aren't quite there yet15:04
*** dingyichen has quit IRC15:05
clarkboh I may have misread the logs that is weird though15:07
clarkbit errors saying I dont know about myself, then later creates a record15:07
*** agentle has joined #openstack-infra15:07
*** kozhukalov_ has quit IRC15:08
*** yonglihe has joined #openstack-infra15:08
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Improve test coverage: container/object list API
*** bcourt has joined #openstack-infra15:09
clarkbregXboi I would grep for that request if through the nova logs and see if there is any more info on ehat failed15:09
clarkbat the error level it isnt very verbose15:10
regXboiclarkb: ack - I'll go looking and maybe wander over to the qa channel as well15:10
*** tonytan4ever has joined #openstack-infra15:11
regXboiclarkb: I'm thinking maybe the test is truncating the host name15:12
clarkboh did we get to hostname limits again? /me counts bytes15:12
openstackgerritMerged openstack-infra/tripleo-ci: Switch to F22 jobs
regXboibecause the host name shows the full string, but this shows a truncation:
clarkbregXboi I font think that is truncated15:14
clarkbthe primary test node has a single id in the name15:15
clarkbthe subnodes rach have the primaryid-theirid15:15
regXboiclarkb: the something is getting confused between the two nodes15:15
regXboior I should say I *think* something is getting confused in the test15:16
openstackgerritEvgeny Antyshev proposed openstack-infra/zuul: Connection names for legacy configs
*** eharney has joined #openstack-infra15:16
clarkb there is the traceback15:18
openstackgerritEvgeny Antyshev proposed openstack-infra/zuul: Connection names for legacy configs
*** dgonzalez_ has joined #openstack-infra15:19
*** dgonzalez_ has quit IRC15:20
clarkblibvirt log says migration unexpectedly failed15:20
eantyshevjhesketh, asselin: found a small but annoying issue with current zuul and default config 3rd-party CI15:20
*** dgonzalez__ has joined #openstack-infra15:21
*** dgonzalez__ is now known as dgonzalez_15:21
*** tsg has joined #openstack-infra15:21
*** rbrndt has joined #openstack-infra15:21
asselinwhat's that?15:21
eantyshevjhesketh, asselin:
*** zeih has quit IRC15:21
*** dgonzalez_ has quit IRC15:21
clarkbregXboi maybe someone from nova can make more sense of the libvirt logs15:22
*** tsg_ has joined #openstack-infra15:22
regXboiclarkb: ack15:22
*** zz_dimtruck is now known as dimtruck15:22
*** trown is now known as trown|brb15:23
asselineantyshev, what are legacy configs? I must've missed something15:24
eantyshevasselin: just ordinary zuul.conf created by puppets, with "gerrit" or "smtp" sections, not aware of recent connection concept15:25
*** mohankumar has quit IRC15:25
*** tsg has quit IRC15:26
*** dizquierdo has joined #openstack-infra15:26
eantyshevasselin: it is not well supported. I've updated zuul and got problems15:26
*** lykinsbd has joined #openstack-infra15:26
*** trown|brb is now known as trown15:27
asselineantyshev, so what happens when this occurs?15:28
asselincbader, that might be the issue you're running into ^^15:28
eantyshevasselin: merge jobs failing, authentication errors15:29
*** dgonzalez__ has joined #openstack-infra15:29
*** lykinsbd_ has quit IRC15:30
*** Daisy has quit IRC15:30
asselinI don't see any updates to puppet-zuul15:30
eantyshevasselin: it doesn't find '.ssh_wrapper__legacy_gerrit' (there is only  .ssh_wrapper_gerrit), and doesn't set GIT_SSH15:30
*** Daisy has joined #openstack-infra15:31
eantyshevasselin: this is caused by recent zuul changes, not puppets15:31
*** daemontool has quit IRC15:32
asselineantyshev, yes, but if puppet-zuul updates the config, then there shouldn't be any issues (assuming you're using those)15:32
*** tonytan4ever has quit IRC15:32
*** zeih has joined #openstack-infra15:33
asselinso why isn't upstream having any issues?15:33
*** mpavone has joined #openstack-infra15:33
*** mpavone has quit IRC15:34
eantyshevasselin: I don't see any signs of connections in zuul/templates/zuul.conf.erb15:34
eantyshevasselin: how often the upstream updates Zuul?15:35
*** Daisy has quit IRC15:35
asselingood point. not often15:35
eantyshevasselin: I'm not sure, just guessing15:36
clarkbthough I believe jeblair restarted zuul just for this change? I wpuld check with him15:36
*** fawadkhaliq has quit IRC15:37
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Switch ironic to use devstack plugin
jrollthere's no way to apply a project-config in a way that a depends-on can use the result of that, right?15:39
jroll^ e.g. with that patch, there's no way to confirm it works in the gate?15:40
clarkbright you cant run jenkins jobs against the proposed job config15:40
clarkb(even if we had the machinery in place its not very secure15:40
jrollhrm, need to figure out how to unwind that template and apply it to one job15:41
* jroll digs around15:41
clarkbjroll what you can do is run that script locally and confirm the plugin works15:41
*** jordanP has quit IRC15:41
*** jordanP has joined #openstack-infra15:42
jrollclarkb: sure, that doesn't give me enough confidence that it won't break down in the gate though15:42
clarkbwhy not?15:42
jrollbecause I don't believe in myself :P15:42
clarkbif ypu follow the steps to run devstack-gate it should be a very good analpg15:43
jrollI'll give it a shot though15:43
clarkb(devstack-gate readme has directions)15:43
*** esker has joined #openstack-infra15:43
cbadereantyshev, asselin, that fixed my issue I have been having for the last six days. thank you.15:43
jrollright, thanks clarkb15:43
*** esker has quit IRC15:43
cbadereantyshev, I will +1 it and would +2 if I could15:44
eantyshevcbader: glad to help you!15:44
*** esker has joined #openstack-infra15:45
*** armax has joined #openstack-infra15:45
rbrndtIf anyone has a moment for a review, I'm looking to add some check jobs to monasca components
cbadereantyshev, good fix I have been hunting this down since Monday.15:45
eantyshevcbader, are you working on some downstream CI?15:46
cbadereantyshev, I have four downstream CI systems for hpeclient testing.15:47
cbadereantyshev, I took over for someone that left so still getting hang of systems. my last straw was creating new Common-ci system to replace failed one. I will still do it but takes pressure of for a day.15:48
*** bcourt has quit IRC15:49
cbadereantyshev, traced it down to was using no key instead of key I was trying to pass, but couldn't figure out what was changing it.15:50
*** claudiub has quit IRC15:50
*** armax has quit IRC15:50
clarkbeantyshev: asselin zuul was started on november 6th15:50
eantyshevcbader: it explains how you faced that problem, I'm refactoring 2 CI robots to use commonci as well15:50
clarkbthat was 4 days ago, which I think is new enough for the new connections stuff15:50
*** EricGonczer_ has joined #openstack-infra15:50
eantyshevclarkb: november?15:51
*** lucas-hungry is now known as lucasagomes15:51
*** tzn has joined #openstack-infra15:51
clarkbgah its early in the morning :)15:51
clarkbso nevermind then you are right it hasn't been restarted since the code merged15:51
clarkbI should get caffeine15:51
tznGuys, where do you track bugs15:52
*** tonytan4ever has joined #openstack-infra15:52
tznlooks like wiki docs are outdated, they suggest storyboard15:52
clarkbcurrently still storyboard15:52
eantyshevclarkb: then I would say it should be dealt sooner then zuul is restarted again15:52
clarkbeantyshev: ya going to take a look at the change now15:52
*** tonytan4ever has quit IRC15:53
clarkbeantyshev: basically looks like the name used in the Connection object should match the name used in the connections dict key?15:54
clarkbotherwise there is a name lookup failure?15:54
*** bcourt has joined #openstack-infra15:54
eantyshevclarkb: i think so, yes. Don't know are there any other places relying them being equal15:55
*** agentle has quit IRC15:55
*** armax has joined #openstack-infra15:55
eantyshevclarkb: not a name lookup, but file .ssh_wrapper* not found. Silently.15:56
clarkbcool +2 and thank you for bringing that up15:56
clarkboh the wrapper itsself15:56
*** agentle has joined #openstack-infra15:56
*** agentle has quit IRC15:57
eantyshevclarkb: I could also propose to issue a warning if wrapper is not found, it would save some time for me or cbader15:57
*** tonytan4ever has joined #openstack-infra15:58
*** gmmaha has left #openstack-infra15:58
*** spzala has joined #openstack-infra15:58
clarkbeantyshev: I think this is good since zuul writes the wrapper itself. Warning doesn't hurt but zuul shouldwrite the wrapper to a location that it can find the wrapper too15:59
*** mdenny has joined #openstack-infra15:59
*** daemontool has joined #openstack-infra15:59
*** rbrndt has quit IRC16:01
*** Qiming has quit IRC16:03
*** _nadya_ has joined #openstack-infra16:03
cbadereantyshev, I agree with the warning this was a pain to run into. lucky only one of my four system updated.16:04
openstackgerritEvgeny Antyshev proposed openstack-infra/zuul: Warning if .ssh_wrapper not found
eantyshevclarkb, cbader: ^^16:05
*** tsg__ has joined #openstack-infra16:08
*** dtantsur is now known as dtantsur|brb16:08
*** _nadya_ has quit IRC16:09
AJaegerclarkb: could you review, please ? That would have avoided some questions here by HenryG and anteaya last night.16:10
*** tsg_ has quit IRC16:10
cbaderclarkb, eantyshev, I agree a warning would really help to fix this if it happens again.16:11
*** eantyshev has left #openstack-infra16:11
*** armax has quit IRC16:11
*** eantyshev has joined #openstack-infra16:11
*** zeih has quit IRC16:13
*** pblaho has quit IRC16:14
*** denisra_ has joined #openstack-infra16:15
*** devananda has quit IRC16:15
*** lykinsbd has quit IRC16:16
*** yamamoto_ has joined #openstack-infra16:17
*** ajmiller has joined #openstack-infra16:18
sc68calmordred: ok, I'm now online again - something about default security group?16:18
*** yamamoto has quit IRC16:18
*** zul has quit IRC16:18
*** denisra has quit IRC16:18
clarkbsc68cal: I believe mordred was looking for a way to set the source of the default security group so that new users would get a default that works by default16:18
clarkbsc68cal: I think he said that the neutron channel said thats not possible by design in order to make a consistent user experience across clouds16:19
clarkb(where that consistency is "you need to set this up to meet your needs")16:19
*** dprince has quit IRC16:19
*** zul has joined #openstack-infra16:19
sc68calclarkb: ah. bummer16:19
*** fawadkhaliq has joined #openstack-infra16:19
sc68calsounds like a job for a heat template ;)16:19
*** e0ne has quit IRC16:20
*** dprince has joined #openstack-infra16:21
*** devananda has joined #openstack-infra16:21
clarkbI also mentioned that its odd that our default default security group rule which is not configurable uses group rules which aiui don't scale well past a small number of VMs16:22
clarkbbut thats a separate issue and probably not one worth bikeshedding (though as the recipient of the 3am phone call from my cloud provider demanding I delete those rules that the cloud gave me by default maybe I care a little more than other people)16:23
*** rossella_s has quit IRC16:30
*** rossella_s has joined #openstack-infra16:30
yolandainfra-root can you take a look at and ?16:31
yolandafirst one for automatic apt-get autoremove, second one to add me to statusbot, before it gives merge comflicts again16:31
*** spzala has quit IRC16:32
jeblairclarkb, eantyshev, asselin: i have not restarted zuul since connections went in16:33
*** rkukura has quit IRC16:34
gordchi folks, we've run into a blocker with unit test timeouts.
*** EricGonc_ has joined #openstack-infra16:34
jeblairclarkb, eantyshev, asselin: jhesketh is also working on a performance regression with that as well i believe16:34
gordcwe have multiple drivers in gnocchi and we were told to have it all tested under one job. but that job is too long now because we have too many tests across the drivers16:35
*** zeih has joined #openstack-infra16:36
*** sputnik13 has joined #openstack-infra16:36
*** EricGonczer_ has quit IRC16:37
*** zeih has quit IRC16:40
*** sshnaidm has quit IRC16:40
*** dprince has quit IRC16:41
*** rcernin has quit IRC16:42
*** mhickey has joined #openstack-infra16:43
*** jlanoux has quit IRC16:43
mhickey Hey All. I am getting an error in 'gate-devstack-bashate' as follows: "ERROR: InvocationError: '/bin/bash -c find /home/jenkins/workspace/gate-devstack-bashate ........". Anyone got any idea whats up?16:44
mordredsc68cal: yah. clarkb has you up to date - I have my info16:46
mordredsc68cal: and yeah - heat template ... or ansible playbook :)16:46
*** sputnik13 has quit IRC16:48
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Add an Ironic job to use devstack plugin
jrollclarkb: ^ we decided we'd rather see it in the gate, I guess16:49
*** agentle has joined #openstack-infra16:49
*** agentle has quit IRC16:50
*** bpokorny has joined #openstack-infra16:50
openstackgerritTomasz 'Zen' Napierala proposed openstack-infra/project-config: Create new project Solar
*** melania has quit IRC16:52
*** sabeen has joined #openstack-infra16:53
*** david-lyle has quit IRC16:53
*** sridhar_ram has joined #openstack-infra16:55
*** sabeen3 has joined #openstack-infra16:55
*** AJaeger has quit IRC16:56
*** sabeen has quit IRC16:58
*** hashar has quit IRC16:59
*** dtantsur|brb is now known as dtantsur17:01
*** rbrndt has joined #openstack-infra17:01
*** ddieterl_ has joined #openstack-infra17:02
clarkbgordc: did you see AJaeger's comment?17:02
*** dprince has joined #openstack-infra17:02
clarkbgordc: you are going to need to stop using the python27 template for gnocchi and make a new job in jjb with that name. In that job set the timeout to be >4017:02
*** ociuhandu_ has joined #openstack-infra17:03
*** salv-orl_ has quit IRC17:03
*** armax has joined #openstack-infra17:03
gordcclarkb: kk. so we should create it under a new unique template...17:04
*** zeih has joined #openstack-infra17:04
*** ociuhandu has quit IRC17:04
*** ociuhandu_ is now known as ociuhandu17:04
clarkbgordc: well you won't need a template since its just your one job, but ya17:04
gordcclarkb: but i take it we still prefer it to be a single job?17:04
*** ddieterly has quit IRC17:04
*** yamahata has joined #openstack-infra17:04
*** matrohon has quit IRC17:05
*** christiana_ has joined #openstack-infra17:05
*** Shrews has quit IRC17:05
clarkbthat I don't know, not sure what the history is on that17:06
openstackgerritPaul Michali proposed openstack-infra/project-config: Make VPN constraints jobs voting
gordcclarkb: just for background, we have mysql|postgres+file|ceph|swift. all running under that job because we were told we shouldn't create six separate jobs? (i don't know exact history either)17:06
clarkbgordc: ^17:06
*** dims_ has quit IRC17:06
openstackgerritTomasz 'Zen' Napierala proposed openstack-infra/project-config: Create new project Solar
*** Shrews has joined #openstack-infra17:07
gordcclarkb: you think maybe it'll be cool to group by mysql and postgres so we have 2 jobs with (3 separate ut runs) rather than 1 job with (6 separate ut runs)17:07
gordcmaybe i'll just propose taht and see if someone yells :)17:08
*** zeih has quit IRC17:09
*** hdd has joined #openstack-infra17:09
*** HeOS has quit IRC17:11
*** ig0r_ has joined #openstack-infra17:11
*** EricGonczer_ has joined #openstack-infra17:12
*** dims has joined #openstack-infra17:12
openstackgerritTomasz 'Zen' Napierala proposed openstack-infra/project-config: Create new project Solar
*** EricGonc_ has quit IRC17:15
*** samuelBartel has quit IRC17:15
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version
*** dims has quit IRC17:16
*** ashleighfarnham has joined #openstack-infra17:16
*** bardia has joined #openstack-infra17:16
*** dims has joined #openstack-infra17:17
clarkbgordc: if the entire test runs are partitioned well in that way that seems to make sense to me (mysql and postgres jobs)17:17
*** alexsyip has joined #openstack-infra17:17
*** Sukhdev has joined #openstack-infra17:17
*** ashishb has joined #openstack-infra17:17
gordcclarkb: kk, i'll give that a try17:18
*** mohankumar has joined #openstack-infra17:18
*** dprince has quit IRC17:22
*** Sukhdev has quit IRC17:23
*** dims has quit IRC17:24
*** dizquierdo has quit IRC17:25
*** burgerk has joined #openstack-infra17:25
openstackgerritMerged openstack-infra/system-config: Add yolanda to statusbot nicks
yolandawoot, thanks, i had to rebase several times..17:27
*** gyee has joined #openstack-infra17:27
*** Piet has joined #openstack-infra17:28
*** sigmavirus24 is now known as sigmavirus24_awa17:29
openstackgerritgreghaynes proposed openstack/diskimage-builder: Allow grub2 to build with opensuse
openstackgerritDoug Wiegley proposed openstack-infra/irc-meetings: Add neutron-lib subteam meeting
openstackgerritTomasz 'Zen' Napierala proposed openstack-infra/project-config: Create new project Solar
anteayamorning zaro17:32
anteayattx: thanks for tracking down the source17:32
*** tsg__ has quit IRC17:32
*** sfinucan has quit IRC17:33
openstackgerritDoug Wiegley proposed openstack-infra/devstack-gate: Add neutron-lib repo to list of projects
*** jordanP has quit IRC17:34
*** daemontool has quit IRC17:35
*** openstackstatus has joined #openstack-infra17:36
*** ChanServ sets mode: +v openstackstatus17:36
*** tonytan4ever has quit IRC17:37
*** xarses has joined #openstack-infra17:39
regXboiclarkb: ping with a project-config question17:39
clarkbregXboi: pong17:40
regXboiclarkb: I have a job foo that is currently non-voting, and is part of some project's check pipeline and other project's experimental pipeline17:40
regXboiI'd like to create an -nv version of the job for the experimental pipelines, so that I can see the failure rate in the check pipeline as a prep to making the job voting17:41
*** esker has quit IRC17:41
regXboiis that the right way to go about things?17:41
*** mhickey has quit IRC17:41
regXboior is that not necessary?17:41
*** dprince has joined #openstack-infra17:41
*** weshay has quit IRC17:41
clarkbregXboi: you want to see the fail rate of the check job independent of the fail rate for the job running in experimental?17:42
regXboiclarkb: yes, but I don't want to cause experimental pipelines angst by having a voting job in them17:42
clarkbgraphite should group them by pipeline so I don't think youneed a newly named job, just restrict to the pipeline (check) that you want17:42
regXboiclarkb: ack, and is it ok to have a voting job in an experimental pipeline?17:42
clarkboh voting jobs in experimental are fine since gerrit doesn't +/- experimental results, it just reports them17:42
jeblairseriously, every time i search for real in hound, i get war and peace as a result17:43
regXboiok - then I'll go double check my graphite fu17:43
regXboiclarkb: thanks for the time17:43
clarkbregXboi: np17:43
anteayajeblair: agreed it would be handy to be able to scope results17:44
anteayafuel results take me out everytime17:44
anteayasince it is basically just a copy of openstack17:45
*** agentle has joined #openstack-infra17:45
jesusauruszaro: I've commented on you need to use "Depends-On" instead of "depends-on"17:46
*** weshay_xchat has joined #openstack-infra17:46
*** dims has joined #openstack-infra17:47
*** weshay_xchat is now known as weshay17:47
jesusauruszaro: oh, wait, nevermind, we are using re.IGNORECASE, so that is misinformation17:48
pabelangerfungi: jeblair: do you think we could tag a new release of nodepool?  Last one we did was back in May17:48
jeblairhowever, please do use "Depends-On" to set a good example :)17:49
*** trown is now known as trown|lunch17:49
eantyshevjeblair: Hello! Did you see these: and ? Or this is going to be fixed some other way?17:49
jeblairpabelanger: that's probably an excellent idea17:49
pabelangerjeblair: Actually, here is something that is pretty crazy17:49
*** Sukhdev has joined #openstack-infra17:49
pabelangerwe actually are missing a LICENSE file for nodepool17:49
jeblaireantyshev: yes, thanks -- let's ask jhesketh to look at it17:50
*** Liuqing has quit IRC17:51
*** tsg__ has joined #openstack-infra17:51
*** eharney has quit IRC17:51
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Add LICENSE file
*** eharney has joined #openstack-infra17:51
pabelangerjeblair: ^ Think we can safely add that.17:52
pabelangerwill help with packaging efforts17:52
pabelanger\r for the fail17:52
*** egonzalez has quit IRC17:52
jeblairpabelanger: copy from one of our other projects, they should be good17:52
pabelangerjeblair: that was actually from zuul17:52
jeblairpabelanger: then maybe we should leave it be :)17:53
*** Apoorva has joined #openstack-infra17:53
openstackgerritTomasz 'Zen' Napierala proposed openstack-infra/project-config: Create new project Solar
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Add LICENSE file
clarkbI want to say the apache license source has funny line breaks in it17:53
jeblairpabelanger: so the md5sum matches17:53
jesusauruszaro: ohh, you are adding allow_drafts to ::gerrit and then to ::openstack_project::review but you missed ::openstack_project::gerrit17:53
pabelangerjeblair: okay, so safe with zuul then?17:53
*** dims has quit IRC17:53
jeblairpabelanger: yeah, that's what i'd do17:53
pabelangerYa, ^M everywhere17:54
pabelangerfor zuul17:54
*** yamamoto_ has quit IRC17:54
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Add LICENSE file
clarkbfwiw nodepool seems to have the proper source file headers for the license17:55
clarkbso putting it in the repo should be fine17:55
clarkb(its not like we are changing the licensing or anything)17:55
pabelangerYa, that was my thoughts too17:55
*** jaypipes has joined #openstack-infra17:57
*** dims has joined #openstack-infra17:57
*** rkukura has joined #openstack-infra17:58
*** SumitNaiksatam has joined #openstack-infra17:58
*** agentle has quit IRC17:58
*** ifarkas has quit IRC18:00
openstackgerritJon-Paul Sullivan (j3p0uk) proposed openstack-infra/zuul: Make the cloner strategy clearer
*** Swami has joined #openstack-infra18:01
*** bardia has quit IRC18:01
*** jaypipes has quit IRC18:02
*** nmagnezi has quit IRC18:04
*** dgonzalez__ has quit IRC18:05
*** shardy has quit IRC18:05
*** vilobhmm11 has joined #openstack-infra18:06
fungihuh, weird that we have so many infra projects with dos format line endings in the license files18:07
fungithe one in the cookiecutter repo and in most of the other openstack projects i've sampled is regular unix line endings18:07
clarkbI checked apache's license copy and its unix line endings18:08
clarkbI wonder if that is based on something your browser/httpclient sneds in a header18:08
*** dtantsur is now known as dtantsur|afk18:08
fungioh, likely yes18:08
fungias far as whether you're getting unix format text when you check what they're serving on a website for it18:09
fungisimilar to the "text" mode supported by most ftp clients/servers18:09
fungiwhich does automatic line ending conversions for you18:09
*** Piet has quit IRC18:10
*** Piet has joined #openstack-infra18:10
* fungi can't begin to count the number of bug reports he fielded years ago from people downloading source code without putting their ftp client into "binary" mode first18:10
*** salv-orlando has joined #openstack-infra18:11
*** dims has quit IRC18:12
anteayazaro: is review-dev down?18:13
*** rockyg has joined #openstack-infra18:14
clarkbfungi: my favorite ftp debug was windows ftp server sets a window size that is tiny and you must edit registry to fix18:14
openstackgerritJon-Paul Sullivan (j3p0uk) proposed openstack-infra/shade: Get the status of the ip with ip.get('status')
*** amit213 has quit IRC18:14
clarkbwe spent too much time looking at the network patrh before realizing this18:15
fungiclarkb: heh, you had me at "windows ftp server"18:15
*** amit213 has joined #openstack-infra18:15
clarkblike oh a window of 512 is just what it does ok wontfix18:15
*** jerryz has joined #openstack-infra18:16
*** arxcruz has quit IRC18:17
fungimy only memories of the ftp server in microsoft iis was clueless dedicated server users leaving anonymous upload enabled to non-sandboxed file trees and ending up with warez servers (if they were lucky, we found out via dmca takedown notices; if less lucky, they found out when their server crashed because someone filled up the drive for them)18:17
*** sabeen3 has quit IRC18:17
*** hdd has quit IRC18:18
openstackgerritJon-Paul Sullivan (j3p0uk) proposed openstack-infra/zuul: Make the cloner strategy clearer
fungialso there were ways to create "undeletable" directories on ntfs via ftp by using unsupported characters and/or really long pathnames18:18
openstackgerritJon-Paul Sullivan (j3p0uk) proposed openstack-infra/zuul: Add vim swap files to .gitignore
fungiwhich made the warez sites a pain to clean up18:19
Clintsounds like some quality software18:19
*** amuller is now known as amuller_afk18:20
fungilet's just say that i don't put any windows sysadmin experience on my resume for very good reasons18:22
*** trown|lunch is now known as trown18:24
jerryzfungi: is there a way to sync a git repository to a newly created project on review.o.o, other than submitting a giant init patch?18:28
*** tsg__ has quit IRC18:28
*** lykinsbd has joined #openstack-infra18:28
*** tsg has joined #openstack-infra18:28
fungitonyb: (if you're awake) from the related thread on the -docs ml it looks like there's some possible quick turnaround desired on their patch to irc-meetings. any objection to me approving?18:28
*** markus_z has quit IRC18:28
*** jaypipes has joined #openstack-infra18:28
jerryzfungi: the repo is not public accessible, so i didn't included it as upstream when creating the project on review.o.o18:29
fungijerryz: if there's no public commit history you're concerned about maintaining, then i'd recommend proposing one big patch like you describe. alternatively you could repropose all the commits in series to gerrit and review them individually to maintain the distinct commits themselves18:30
*** Guest21019 has quit IRC18:31
jerryzfungi: got it, thanks.18:33
openstackgerritDerek Higgins proposed openstack-infra/project-config: Remove tripleo-f21 nodes from nodepool
*** sridhar_ram has quit IRC18:36
openstackgerritDerek Higgins proposed openstack-infra/project-config: Remove tripleo-f21 nodes from nodepool
*** dims has joined #openstack-infra18:36
*** agentle has joined #openstack-infra18:37
*** bexelbie has quit IRC18:37
*** bharathm has joined #openstack-infra18:38
*** jistr has quit IRC18:38
openstackgerritClark Boylan proposed openstack-infra/infra-manual: Clarify project import options
fungiclarkb: yep18:40
*** agentle has quit IRC18:40
jeblairtypo; otherwise lgtm18:42
openstackgerritClark Boylan proposed openstack-infra/infra-manual: Clarify project import options
clarkbnow with fewer topos18:43
greghaynesclarkb: less maps?18:43
*** arxcruz has joined #openstack-infra18:43
fungiless tapas18:43
*** placko has quit IRC18:44
* fungi can no longer see that word without hearing maurice moss say "tape-ass"18:44
*** dizquierdo has joined #openstack-infra18:44
clarkb"I am stuck under a desk"18:45
*** tzn has quit IRC18:45
*** fawadkhaliq has quit IRC18:50
*** dizquierdo has quit IRC18:51
clarkblooks like jenkins04 is unhappy18:53
* clarkb wonders if any other rooter wants to run the restart jenkins safely playbook against it18:53
clarkbI can walk you thorugh it :)18:54
*** mohankumar has quit IRC18:54
*** amuller_afk is now known as amuller18:55
jeblairclarkb: if you write it up in i'll run it :)18:55
* clarkb does this18:56
*** harlowja has quit IRC18:56
*** harlowja has joined #openstack-infra18:56
*** jcoufal has quit IRC18:58
*** esker has joined #openstack-infra18:58
*** dprince has quit IRC18:58
*** dprince has joined #openstack-infra18:59
openstackgerritMerged openstack-infra/nodepool: Add LICENSE file
*** Sukhdev has quit IRC19:00
*** Sukhdev has joined #openstack-infra19:00
fungii'm happy to take turns running it too if it's in a public document (to the earlier discussion with yolanda about the location of our "infra runbook")19:01
*** asselin__ has joined #openstack-infra19:04
*** tsg has quit IRC19:04
openstackgerritMerged openstack-infra/infra-manual: Clarify project import options
*** asselin has quit IRC19:05
pabelangerjeblair: I think nodepool is ready for a tag when you are.  c8c680912384f041dd1a824e9970c98dc74c7ef0 just landed19:06
openstackgerritClark Boylan proposed openstack-infra/system-config: Add doc on using jenkins restart playbook.
clarkbthere we go19:06
jeblairclarkb: want to tag nodepool while i run that ^ ?19:08
anteayaclarkb: the last sentence isn't the easiest for me to comprehend19:09
clarkber ya its not worded well let me fix that19:09
openstackgerritClark Boylan proposed openstack-infra/system-config: Add doc on using jenkins restart playbook.
jeblairclarkb, mordred: is it possible to use the on-host hiera info to remove the password from the command line in ?19:10
clarkbanteaya: ^ is that better?19:10
clarkbjeblair: we might be able to make ansible do a hiera lookup in the playbook19:11
clarkbjeblair: something like register var: hiera $key then use var directly19:11
*** tonytan4ever has joined #openstack-infra19:11
anteayaclarkb: yes, thank you19:11
*** sigmavirus24_awa is now known as sigmavirus2419:11
jeblairclarkb: i give it the launchpad password?19:11
clarkbjeblair: I have been using what nodepool uses19:12
jeblairclarkb: oh, we stopped using gerrig?19:12
*** e0ne has joined #openstack-infra19:13
clarkbjjb and nodepool both use the hudson-openstak user19:13
jeblairapparently so :)19:13
jeblairclarkb: so apikey= from secure.conf?19:13
fungii hope it really is hudson-openstak19:13
fungiso that we replaced one typo with anoter19:13
clarkbfungi: nope my typo :)19:13
fungitypos all 'round!19:13
*** mriedem is now known as mriedem_haircut19:14
clarkblooking at nodepool history I think next tag should be 0.2.019:14
clarkbpabelanger: jeblair ^ that sound right?19:14
jeblairclarkb: ya19:15
*** ociuhandu has quit IRC19:15
fungisounds good19:16
*** sridhar_ram has joined #openstack-infra19:17
adam_ganyone know if the target git branch of a landed patch is exposed to post jobs in any way via environment vars or something else?19:17
jeblairclarkb: oh, thought of an easier way to snarf the password; left comment on
*** SumitNaiksatam has quit IRC19:17
jeblairclarkb: got jenkins.JenkinsException: node[bare-trusty-hpcloud-b5-6468725] does not exist19:18
jeblairclarkb: running in screen as root if you need to look at it19:18
*** SumitNaiksatam has joined #openstack-infra19:18
*** SumitNaiksatam has quit IRC19:18
clarkbneat on the hiera read thing19:18
pabelangerdo we still need python-glanceclient, keystoneclient, novaclient in requirements now that we are using shade?19:19
clarkbfor the fail that may mean the server is too far gone to have useable http api :/19:19
pabelangerfor nodepool19:19
zaroanteaya: no, review-dev is up.  you just need to refresh your browser few times.19:19
clarkbpabelanger: if we directly consume them then yes (we directly consume at least novaclient)19:19
jeblairjhesketh: is there a reason we need to stop() the triggers/sources/reporters?  that method isn't implemented anywhere, and i can't think of a reason to -- only to stop the _drivers_.19:19
anteayazaro: thanks, that is what I did :)19:19
openstackgerritgordon chung proposed openstack-infra/project-config: split gnocchi unit tests by indexer
jeblairclarkb: should we spin up more masters?19:19
*** agentle has joined #openstack-infra19:20
*** sputnik13 has joined #openstack-infra19:20
*** Sukhdev has quit IRC19:20
jeblairclarkb: and should i give the playbook one more chance?19:20
clarkbjeblair: ya I would give the playbook another chnace as it may have just been super racy due to slow api19:20
clarkband things should slow down now that it is in shutdown mode19:20
zarojesusaurus: thanks for the info in 25505019:21
*** agentle has quit IRC19:21
clarkbas for more masters maybe? jenkins04 actually seems to do this more regularly than the other masters. A rebuild to a new hypervisor may help?19:21
*** mrmartin has joined #openstack-infra19:21
clarkbpabelanger: jeblair 0.2.0 nodepool pushed19:21
*** ashishb has quit IRC19:21
pabelangerclarkb: heh, only grep'd keystone and glance. Okay, will get a patch up to drop them from requirements19:21
pabelangerclarkb: danke19:21
openstackgerritClark Boylan proposed openstack-infra/system-config: Add doc on using jenkins restart playbook.
clarkbnow with fancy hiera reading19:25
jeblairclarkb: 502 this time... running it once more so i can double check my hiera thingy19:25
fungiadam_g: it can be divined yes. if ZUUL_PIPELINE==post then ZUUL_REFNAME should contain the branch name (e.g., stable/kilo)19:26
*** pvaneck has joined #openstack-infra19:26
clarkbunfortunately once the api stops being reliable it is really hard to manage this externally :(19:26
jeblairclarkb: actually, was a problem, need to change ' to "19:26
clarkbjeblair: oh right19:26
*** Apoorva is now known as apoorvad19:26
fungiadam_g: see for the parameters jenkins inserts into the job environment on a stable/kilo post job19:26
openstackgerritClark Boylan proposed openstack-infra/system-config: Add doc on using jenkins restart playbook.
adam_gfungi, ah, that params list is exactly what i was lookin for. thanks19:27
*** tzn has joined #openstack-infra19:27
*** r-daneel has joined #openstack-infra19:27
fungiadam_g: the zuul documentation also covers this at but sometimes an example is useful to make sense of docs19:28
openstackgerritJames E. Blair proposed openstack-infra/zuul: Remove stop method from reporters, sources, triggers
jeblairjhesketh: ^19:29
jeblairthat will simplify the reconfiguration logic a bit19:30
jeblairclarkb: it's... doing something this time.19:30
clarkbjeblair: it will poll running jobs every minute19:31
*** mrmartin has quit IRC19:31
clarkbso it should hopefully sit there then restart when jobs are done19:31
jeblairoh, so it'll still have plenty of chances to die :)19:32
clarkbya :/19:32
*** alexsyip has quit IRC19:32
clarkbI wonder if we can easily poll thread counts in the jvm19:32
clarkbthen just have a watchdog type service that runs the playbook whenever they go above 1500 threads or so19:33
*** tzn has quit IRC19:33
openstackgerritKhai Do proposed openstack-infra/system-config: Disable Gerrit drafts feature
*** agentle has joined #openstack-infra19:34
*** agentle has quit IRC19:35
*** jaypipes has quit IRC19:36
jeblairclarkb: failed again after a while; running one last time just to exercise the command as i copy/pasted it from the latest revision19:39
jeblairand that's failed now too19:39
jeblairbut the command looks good19:39
jeblairso +2 there :)19:39
clarkbcool, I think we just manually restart it when its done running jobs then :(19:39
jeblairhttplib.IncompleteRead: IncompleteRead(6146 bytes read)19:39
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add dib-python element
*** electrofelix has quit IRC19:39
clarkband maybe put in a large number of retries in the script19:40
clarkblike try to poll every minute and don't actually quit until you fail 5 times in a row19:40
jeblairso i'll check back in after lunch19:40
jeblair(there's at least one 59 minute job in there)19:40
*** tqtran has joined #openstack-infra19:47
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Drop glanceclient and keystoneclient requirements
*** hdd has joined #openstack-infra19:51
*** aysyd has quit IRC19:53
*** baoli_ has quit IRC19:53
*** mrmartin has joined #openstack-infra19:54
xarsesif I want to add a test to my project, and want to start it as non-voting do I add it into jobs filters and add voting false, call the template/ job -nv (so it matches the ^.*-nv$ jobs filter), or something else?19:54
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Publish per-branch astara-appliance images
clarkbError: Your application used more memory than the safety cap of 500M. <- yay new logstash /me tries 1GB and keeps an eye on it19:55
clarkbxarses: usually we use the -nv suffix if you want to have two versions of the same job, one voting and one not voting. If the goal is to transition form non voting to voting typically we use an explicit voting: false which is removed once ready to vote19:55
clarkbbut either way works19:55
*** hashar has joined #openstack-infra19:56
clarkbjesusaurus: fyi on the logstash 2.0 memory issue. Though I think the default was 1GB on 8GB nodes before since we didn't specify19:59
*** Piet has quit IRC20:01
*** rcernin has joined #openstack-infra20:01
*** sridhar_ram has quit IRC20:02
openstackgerritgreghaynes proposed openstack/diskimage-builder: Run package-intalls with py3k if we must
clarkbgreghaynes: that commit message sounds so sad about it20:03
clarkb"if we must"20:03
openstackgerritKhai Do proposed openstack-infra/system-config: Add script to cleanup Gerrit 2.8 data
greghaynesclarkb: Yea, I was not terribly happy about the first rev of that20:03
zarofungi: ^ that contains the fix for the akanda rename typo20:05
*** bpokorny has quit IRC20:07
*** bpokorny has joined #openstack-infra20:07
fungizaro: thanks--looks great!20:08
*** rockyg has quit IRC20:08
*** armax has quit IRC20:09
fungizaro: one minor adjustment marked inline20:09
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add a new element hpdsa
zarofungi: should that just be changed to a comment? or should it be a seperate bash file?20:13
zaro*be/be in20:13
*** mriedem_haircut has quit IRC20:13
fungizaro: a comment is fine, we just need to remember to perform that step on the git servers20:14
*** ldnunes has quit IRC20:14
*** HeOS has joined #openstack-infra20:15
clarkbif logstash worker20 makes it through to monday with logstash2.0 would be great to get the service upgraded next week20:16
fungii concur20:18
*** mriedem has joined #openstack-infra20:18
fungithough hopefully not on or immediately before wednesday20:18
clarkbya can work with gerrit :)20:18
fungiright, want to make sure we're not distracted by other problems of our making in the middle of major gerrit stuffs20:19
clarkbagreed, speaking of I really meant ot pull up that topic and review changes20:19
*** lucasagomes is now known as lucas-beer20:19
*** Swanson has quit IRC20:20
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Better duplicate detection for project requirements check
*** Sukhdev has joined #openstack-infra20:22
*** sridhar_ram has joined #openstack-infra20:22
openstackgerritClark Boylan proposed openstack-infra/puppet-logstash: Install logstash 2.0
clarkbthat reflects the new heap sizing on logstash-worker2020:24
*** sabeen1 has joined #openstack-infra20:24
*** ddieterl_ has quit IRC20:27
jrollif someone has a minute to help me add an ironic job, that would be great. trying to get out of devstack's hair :)
openstackgerritKhai Do proposed openstack-infra/system-config: Add script to cleanup Gerrit 2.8 data
*** unicell has joined #openstack-infra20:29
*** alex_xu has quit IRC20:30
clarkbjroll: if you invert the dep order you could test the change ot add the plugin20:30
*** rossella_s has quit IRC20:30
jrollclarkb: oh, I should just drop the depends-on there20:30
jrollforgot I had it20:30
*** rossella_s has joined #openstack-infra20:31
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Add an Ironic job to use devstack plugin
jrollclarkb: ^ done20:31
jrollthanks man20:32
openstackgerritMerged openstack-infra/project-config: Disable stable/kilo python3 jobs for compute_hyperv
sc68calanteaya: soo... stable/liberty is kind of borked, and it looks like linuxbridge may be wedging the gate20:32
sc68calanteaya: I'm trying to get a fix in for devstack to unwedge, but it now blows up on an APT dependency error20:32
*** mrmartin has quit IRC20:32
sc68calanteaya: so, this is a long winded background for the following question - can I just update the branch regexp and just make linuxbridge voting on master/mitaka only?20:33
*** alex_xu has joined #openstack-infra20:33
*** lucas-beer has quit IRC20:34
clarkbsc68cal: assuming that keeping linuxbridge tested (and transitively working) on kilo for the next six months or so isn't something we care about sure20:35
*** dims has quit IRC20:35
clarkbI am not sure what the state of linuxbridging is there20:35
*** dims has joined #openstack-infra20:35
clarkbits not uncommon to only run new things on recent branches. If you want to make it nonvoting on kilo you can add a -nv job for it and only run that one on kilo20:36
sc68calclarkb: I'll explain in commit message20:36
sc68calclarkb: basically we've got a clusterF@#&*( where we can't get things merged to fix LB in the branch20:36
sc68calbecause of APT stuff20:36
clarkbsince both the voting flag and branch flag apply to a job as a whole you can't selectively vote on one branch or another. We deal with that by having separate jobs for each20:36
clarkbsc68cal: are packages missing?20:37
sc68calneutron grenade that installs kilo then tries to upgrade to liberty explodes on an apt dependency20:37
clarkbyou grenade with linuxbride?20:37
*** dprince has quit IRC20:37
sc68calclarkb: no - it's just to get the patch into devstack to fix linux bridge on liberty, we're hitting a neutron grenade issue20:38
clarkboh fun20:38
clarkbso fix one thing breaks another?20:38
sc68calclarkb: no - it's like, to fix a we need to fix b - which is broken due to C (apt)20:38
*** sabeen3 has joined #openstack-infra20:39
sc68caland now I want to go home and crawl into bed20:39
clarkblooking at that change linuxbridge tes tisn't run against it?20:39
sc68calclarkb: no20:39
*** sabeen1 has quit IRC20:39
clarkbso you should be able to fix the other thing, then fix linuxbridge?20:39
openstackgerritMerged openstack-infra/system-config: Allow Gerrit to use 4 threads for indexing changes.
*** lucasagomes has joined #openstack-infra20:40
sc68calclarkb: possibly. but I don't know anything about fixing apt dependency issues20:40
clarkbsc68cal: do you have an example of the apt fail?20:40
sc68calclarkb: yeah let me get it for you20:40
sc68calI'm just trying to get OUT OF THE WAY since it seems to be making people grumpy20:40
openstackgerritAndrew Woodward proposed openstack-infra/project-config: Create a seperate job for puppet-ceph integration
*** e0ne has quit IRC20:41
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Switch all ironic jobs to use devstack plugin
xarsesclarkb: it's my first attempt, let me know how badly I did20:42
clarkbunderstood, just making sure we don't remove testing if there is alternative to just fix it20:42
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Switch all ironic jobs to use devstack plugin
clarkbsc68cal: I do notice that apt is failing to update some indexes due to hash mismatch (which is apt repo mirroring brokeness)20:42
*** jffische has quit IRC20:43
sc68calis that a symptom of mirroring brokeness20:43
clarkbsc68cal: the hash sum mismatch a bit up from there is20:43
clarkbnot sure if that can trigger the error you linked to yet20:43
*** _nadya_ has joined #openstack-infra20:44
*** Swami has quit IRC20:44
clarkbinternet seems to say an apt-get update can correct the problem20:44
clarkbso its possible that is why its fialing20:45
clarkbsc68cal: is this a persistent problem or did it fail just the once?20:45
sc68calclarkb: rechecking now20:45
openstackgerritgordon chung proposed openstack-infra/project-config: split gnocchi tests by indexer
openstackgerritSean M. Collins proposed openstack-infra/project-config: Linux Bridge - disable voting on stable branches
*** jaosorior has quit IRC20:47
*** jaosorior has joined #openstack-infra20:47
*** aeng has joined #openstack-infra20:48
clarkbxarses: ya that looks good except for one thing, you will need to instantiate that template on the puppet-ceph project in jjb. At the end of the puppet module jobs file you can see job groups that do it for the existing job or you can add the single job directly to puppet-ceph project20:49
*** ddieterly has joined #openstack-infra20:50
xarsesclarkb: isn't that what i did in zull/layout.yaml:754020:51
*** ddieterly has quit IRC20:52
nibalizercrinkle: I have successfully logged into a node booted on infra cloud20:52
openstackgerritSean M. Collins proposed openstack-infra/project-config: Linux Bridge - disable voting on stable branches
nibalizermuch excite20:52
clarkbxarses: no, jenkins job build configures jenkins only. The only relationship between jjb and zuul is that zuul triggers the jobs that end up in jenkins20:52
clarkbxarses: so zuul/layout.yaml is telling zuul, when these events happen run these jobs. You still need to tell JJB take this job template and apply some parameters to it to make a job20:53
crinklenibalizer: omg20:53
clarkbnibalizer: now change your mtu20:53
funginibalizer: so full of win!20:54
clarkbzaro: if wrks on 2.8 as well do  you wan tto remove the depends on and we can go ahead and merge it now?20:56
clarkbfungi: ^20:56
clarkbjust thinking that reducing the set of changes we have to make during the upgrade itself might help things go smoother20:56
fungiif it's safe for 2.8.x then i agree20:56
fungimerge as much as we can safely merge in advance20:57
clarkbzaro: and is safe to merge too since there are defaults for those params ya?20:57
xarsesclarkb: thanks, I think I get it now20:57
clarkbxarses: they are decoupled like this beause if you use jenkins you don't need to use zuul and if you use zuul you don't need to use jenkins20:58
clarkb(and we know of other people/groups that run jenkins and zuul both independnetly of each other)20:58
* clarkb approves that jeepyb change and the firefox text copy in gerrit change21:00
*** gordc has quit IRC21:01
openstackgerritMerged openstack-infra/jeepyb: Add gerrit change hook parameters for Gerrit 2.11
*** Sukhdev has quit IRC21:01
clarkbnotmorgan: similar question about I think that will work on old gerrit too right?21:01
openstackgerritKhai Do proposed openstack-infra/system-config: Fix toggle ci button for CS2 on Gerrit 2.11
clarkbnotmorgan: can we remove the depends on and merge it?21:01
notmorganclarkb: oh it should work on old gerrit21:01
notmorganclarkb: i mean... i have not tested it21:02
clarkbnotmorgan: :)21:02
notmorganclarkb: but there should be no reason why it wpould be broken21:02
clarkbnotmorgan: I agree21:02
clarkbapache verion doesn't change21:02
zaroclarkb: removed depends-on in 235079 but sadly Gerrit removed all the votes21:02
anteayasc68cal: sorry I wasn't here when you needed me, looks like clarkb was better help anyway21:03
clarkbzaro: I just reapplied my vote, I bet we can get fungi to appy a second and approve21:03
notmorganclarkb: exactly21:03
clarkbmy only concern is if older gerrit working with mod rewrite proxies is somehow inversely broken than newer gerrit21:04
zaroclarkb: yep, I'm pretty sure 236220 is ok for 2.821:04
clarkbzaro: ya me too wnet ahead and approved it :)21:05
*** bcourt has quit IRC21:05
openstackgerritlifeless proposed openstack-infra/infra-manual: Update the manual to reference -constraints jobs.
zaroohh, nice!21:05
jeblairclarkb: all builds are finished; i'm going to go ahead and run the playbook again and see if slightly reduced load might help21:05
openstackgerritgreghaynes proposed openstack/diskimage-builder: Run package-intalls with py3k if we must
clarkbjeblair: rgr21:05
clarkbnotmorgan: +2'd your change, happy to get that in sooner if you think its safe (and it sound slike you do)21:06
*** tonytan4ever has quit IRC21:06
notmorganclarkb: yes21:07
notmorganclarkb: i would go on record and say it should be safe21:07
clarkbzaro: for changes like it looks like upstream merged that to their 2.11 stable branch so this is just us catching up right? we aren't adding any forked changes?21:08
*** lykinsbd has quit IRC21:08
*** baoli has joined #openstack-infra21:09
*** tsg has joined #openstack-infra21:09
*** sigmavirus24 is now known as sigmavirus24_awa21:10
clarkbI almost prefer that we use a merge commit for those21:10
clarkbbut separate changes will work21:10
*** sigmavirus24_awa is now known as sigmavirus2421:10
zaronope, it's an unreleased fix but still stable-2.11 branch so should be in the next 2.11.x release21:11
*** burgerk has quit IRC21:11
clarkbnotmorgan: want t oremove the depends-on then?21:11
openstackgerritPaul Belanger proposed openstack-infra/project-config: Fix project description for ansible-role-diskimage-builder
openstackgerritMerged openstack-infra/system-config: Add doc on using jenkins restart playbook.
notmorganclarkb: sure. i can or you can21:12
notmorganclarkb: give me a moment21:12
clarkbnotmorgan: you can just edit commit in gerrit if thats easiest21:12
zaroclarkb: ohh wait, i'm talking got that change mixed up with another change.21:12
clarkbor I can do it21:12
notmorganok will do now21:12
openstackgerritMorgan Fainberg proposed openstack-infra/puppet-gerrit: Update Gerrit proxy configuration.
notmorganclarkb: done21:12
clarkbzaro: well there are about 4 change to our 2.11.4 branch and they all have upstream equivalents that appear to be from their stable 2.11 branch21:13
clarkbzaro: so I think they are all from 2.11 upstream but possibly unreleased?21:13
*** Swami has joined #openstack-infra21:13
*** julim has quit IRC21:14
zaroclarkb: yes, that is correct. i think there is only 1 that was unreleased.21:14
zarothis one
mordredjeblair, clarkb: yah - I think we can probably do something with ansible and the hiera files21:15
*** tzn has joined #openstack-infra21:15
clarkbzaro: gotcha21:16
*** _nadya_ has quit IRC21:16
*** kgiusti has left #openstack-infra21:18
openstackgerritTim Hinrichs proposed openstack-infra/project-config: Make python34 job voting for Congress
*** Sukhdev has joined #openstack-infra21:18
ianwgreghaynes: devuser doesn't give sudo access by default?  I thought when it said "passwordless sudo" in the var, that would mean it would make a password enabled one21:18
*** Sukhdev has quit IRC21:19
greghaynesianw: Correct - I wouldnt mind it making password enabled by default21:19
*** Sukhdev has joined #openstack-infra21:20
greghaynesianw: The reason it isnt there is because I tend to use password disabled users21:20
*** baoli has quit IRC21:20
ianwgreghaynes: yeah, that only works when networks is working and you can ssh in :)21:20
greghaynesianw: yeee21:21
xarsesclarkb: so, could I instead duplicate the job in the jbb projects yaml and just change the options to it so that it runs this other set of tests as another job?21:21
*** baoli has joined #openstack-infra21:21
*** tzn has quit IRC21:21
clarkbxarses: what you have is fine, you just have to tell jjb make a job from this job template21:22
*** ddieterly has joined #openstack-infra21:23
jeblairmordred: cool; i removed the immediacy of doing that by doing a lookup on the puppetmaster since we know we are running there, but figuring out how to do that would probably make playbooks a more generally useful thing for us21:24
xarsesclarkb: but at that point I've duplicated an entire job template to change just one thing, the filter of which tests will be executed21:24
openstackgerritMerged openstack/diskimage-builder: Add proliant-tools element
mordredjeblair: I think, amusingly enough, that what we should do is a hiera module21:25
openstackgerritMerged openstack/diskimage-builder: Follow up patch for 25d3ee547176528e86d42eb026c99a134dff9452
mordredthat actually shells out to the hiera command21:25
mordredand gets the value you've requested21:25
*** ddieterly has quit IRC21:25
mordredwhich would mean we could do what clarkb was suggesting and have the lookup in the playbook21:25
*** ddieterly has joined #openstack-infra21:26
clarkbxarses: oh I didn't realize that is all yo uare changing, in that case ya I would update the template that exists to accept a parameter that changes the filter21:26
clarkbor use a different rake task like EmilienM suggested21:26
EmilienMI just wrote a comment21:26
*** igormarnat has quit IRC21:26
mordredjeblair: I had to push 3 things onto my stack - I'll write the hiera lookup module when I get back to fixing the ansible inventory21:27
*** dteselkin has quit IRC21:27
*** HeOS_ has quit IRC21:27
*** kzaitsev_ws has quit IRC21:27
*** katyafervent has quit IRC21:28
*** rakhmerov has quit IRC21:28
*** akuznetsova has quit IRC21:28
*** tsufiev has quit IRC21:28
*** asilenkov has quit IRC21:28
*** HeOS_ has joined #openstack-infra21:29
*** kzaitsev_ws has joined #openstack-infra21:29
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] job for sahara-dashboard integration tests
zaroclarkb: what does 'interdiff between two changes' mean?21:29
jeblairmordred: okay, no rush :)21:29
*** rakhmerov has joined #openstack-infra21:29
clarkbmordred: here is random gist that claims to do it
clarkbzaro: I diffed the diffs21:29
zaroahh ok.  thanks21:30
clarkbzaro: to confirm we didn'tsneak in any changes that diverge from upstrema21:30
clarkbmordred: of course its a gist with no licensing so maybe don't look at it21:30
*** asselin has joined #openstack-infra21:31
*** akuznetsova has joined #openstack-infra21:31
*** armax has joined #openstack-infra21:31
*** ddieterl_ has joined #openstack-infra21:31
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] job for sahara-dashboard integration tests
*** katyafervent has joined #openstack-infra21:32
*** dteselkin has joined #openstack-infra21:32
*** baoli has quit IRC21:32
*** asselin__ has quit IRC21:32
*** tsufiev has joined #openstack-infra21:32
*** baoli has joined #openstack-infra21:33
*** asilenkov has joined #openstack-infra21:33
*** tonytan4ever has joined #openstack-infra21:34
*** ddieterly has quit IRC21:34
*** igormarnat has joined #openstack-infra21:34
*** gordc has joined #openstack-infra21:35
*** ddieterl_ has quit IRC21:35
*** hashar has quit IRC21:36
*** ddieterly has joined #openstack-infra21:37
*** tlian has quit IRC21:38
clarkbzaro: for does our jeepyb default of visible to all not work in all cases? (Just want to make sure I understand if this is fixing and issue or just taking advantage of new features)21:41
xarsesEmilienM: it's not that I don't want to run the beaker tests, I just want to change some of the options to it (to start with, which jobs are run, maybe later ostype) and then have them vote separately from each other21:43
*** rbrndt has quit IRC21:43
*** tlian has joined #openstack-infra21:44
zaroclarkb: i didn't realized jeepy already did that.  do we ever create groups manually?21:44
*** baoli has quit IRC21:44
*** baoli has joined #openstack-infra21:45
clarkbzaro: occasionally we do, so I think the change is good, I just also wanted to fix jeepyb if it wasn't working21:45
clarkbI am not approving it htough because I odn't remember if gerrit gets mad about config itdoesn't understand21:45
*** ZZelle_ has joined #openstack-infra21:45
EmilienMxarses: I don't have strong opinions21:45
zaroclarkb: ohh, i'm don't know if jeepyb is working or not, i'm assuming that it is.21:45
openstackgerritMerged openstack/diskimage-builder: Add a new element hpdsa
xarsesEmilienM: Ok21:46
zaroclarkb: gerrit will usually just ignore any config it doesn't know about.21:46
jeblairclarkb: playbook still failed, and didn't clean up many nodes before doing so... what's next?21:46
zaroclarkb: gerrit code needs to read in the configs so if doesn't know about it the config never gets read in.21:46
clarkbjeblair: manual stop, remove slaves from config.xml, manual start21:46
openstackgerritSachi King proposed openstack-dev/pbr: passenv integration environment variables re-enabling integration tests
jeblairclarkb: all slaves and just let the nodepool 24h timeout catch the valid ones?21:47
*** abregman has quit IRC21:47
clarkbjeblair: ya I usually don't worry about the slaves as I hav efound most of them tend to be gone from nodepool but jenkins delete failed because api was unhappy21:47
clarkbjeblair: so you have offline slaves in jenkins that don't exist anymore21:48
jeblairclarkb: oh, yeah, i was talking about the online ones though21:48
clarkband even if they do exist they go away soon enough, if you really want to be good about it you can stop jenkins, remove from config.xml, delete all jenkins04 nodes in nodepool, start jenkins21:48
jeblairthere are 61 of those on jenkins0421:48
clarkb^^ is the shotgun appraoch but it is "correct"21:48
jeblair61 ready nodes that is21:48
clarkboh in that case I would do the shotgun appreach unless you want ot write an xml parser to remove only those not read :)21:49
jeblairdo you think 61 nodes is unusual?  how many do you normally see?21:49
clarkbI want to say normal is closer to a handful, 5ish21:50
*** Hink has joined #openstack-infra21:50
jeblair(i mean that's 0.6% of our quota; doesn't seem outlandish)21:50
clarkbzaro: ok I think I have gotten through that set of changes, let me know if I missed something21:51
*** ddieterly has quit IRC21:51
zaroclarkb: Add script to cleanup Gerrit 2.8 data21:51
*** ddieterly has joined #openstack-infra21:51
zaroopps it's
*** baoli has quit IRC21:52
*** dimtruck is now known as zz_dimtruck21:52
*** Swami has quit IRC21:52
*** ddieterl_ has joined #openstack-infra21:53
zaroclarkb: would be good to merge all the openstack/gerrit changes so i can prep a new version for
*** baoli has joined #openstack-infra21:53
*** lykinsbd has joined #openstack-infra21:53
*** Sukhdev has quit IRC21:54
clarkbzaro: I am your first +2 on them21:54
zarocool, ty!21:54
jeblairclarkb: jenkins04 restarting21:55
StevenKclarkb: and for jeepyb goodness whenever you have a chance.21:55
*** ddieterly has quit IRC21:56
*** Sukhdev has joined #openstack-infra21:56
*** gildub has joined #openstack-infra21:59
zaroclarkb: this one too
*** peristeri has quit IRC21:59
zaroohh wait, that one already has 2 +222:00
clarkbzaro: I skipped that one because it ya that22:00
clarkbbut I can look it over to get mor eeyes on it22:00
*** baoli has quit IRC22:00
*** baoli has joined #openstack-infra22:01
clarkbjeblair: looks like its back up and doing stuff22:02
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] job for sahara-dashboard integration tests
*** ddieterl_ has quit IRC22:03
*** ddieterly has joined #openstack-infra22:03
*** Hink has quit IRC22:05
*** tlian has quit IRC22:05
*** Hink has joined #openstack-infra22:05
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for oslo.utils netifaces bug 1524991
openstackbug 1524991 in oslo.utils "moving netifaces to extras in 3.1.0 breaks the world" [Critical,Fix released] - Assigned to Joshua Harlow (harlowja)22:05
*** eharney has quit IRC22:06
*** alex_xu has quit IRC22:07
*** ddieterly has quit IRC22:07
* zaro does some errands, bbl22:07
*** derekh has quit IRC22:07
*** Hink has quit IRC22:08
*** Hink has joined #openstack-infra22:08
*** dewsday has joined #openstack-infra22:08
*** alivigni has quit IRC22:08
*** alex_xu has joined #openstack-infra22:09
*** Piet has joined #openstack-infra22:09
openstackgerritMatt Riedemann proposed openstack/requirements: Block oslo.utils 3.1.0
clarkbzaro: if you haven't gone yet, you did end up testing hte rollback and it worked right?22:09
*** tzn has joined #openstack-infra22:10
*** hogepodge has quit IRC22:10
*** tlian has joined #openstack-infra22:11
*** jogo has quit IRC22:11
*** gyee has quit IRC22:12
*** hogepodge has joined #openstack-infra22:13
openstackgerritMerged openstack-infra/elastic-recheck: Add query for oslo.utils netifaces bug 1524991
*** baoli has quit IRC22:13
openstackbug 1524991 in oslo.utils "moving netifaces to extras in 3.1.0 breaks the world" [Critical,Fix released] - Assigned to Joshua Harlow (harlowja)22:13
*** gordc has quit IRC22:13
*** baoli has joined #openstack-infra22:13
*** Hink has quit IRC22:13
*** dgonzalez__ has joined #openstack-infra22:13
*** lykinsbd has quit IRC22:14
*** Swami has joined #openstack-infra22:14
*** trown is now known as trown|outttypeww22:14
*** jogo has joined #openstack-infra22:14
*** jogo has joined #openstack-infra22:14
*** tzn has quit IRC22:15
dgonzalez__hi sdague, you here?22:16
*** Hink has joined #openstack-infra22:16
*** Hink has quit IRC22:17
lifelessdims: - its passed the thing it was failing on now, just waiting on tempest22:18
*** moravec has quit IRC22:18
lifelessdims: care to +2A?22:18
*** moravec has joined #openstack-infra22:19
*** moravec has quit IRC22:19
openstackgerritSachi King proposed openstack-dev/pbr: Handle markers to support sdist on pip < 6
*** ddieterly has joined #openstack-infra22:20
*** moravec has joined #openstack-infra22:20
*** gyee has joined #openstack-infra22:20
*** david-lyle has joined #openstack-infra22:20
*** ddieterl_ has joined #openstack-infra22:21
*** dkranz has quit IRC22:21
*** jamielennox|away is now known as jamielennox22:22
*** derekh has joined #openstack-infra22:24
*** yamahata has quit IRC22:24
*** moravec has quit IRC22:24
*** ddieterly has quit IRC22:24
*** baoli has quit IRC22:24
*** yamahata has joined #openstack-infra22:25
*** baoli has joined #openstack-infra22:25
*** ddieterl_ has quit IRC22:25
*** sdake has joined #openstack-infra22:26
*** Swami has quit IRC22:27
*** apoorvad has quit IRC22:28
*** gongysh has joined #openstack-infra22:28
tonybfungi: I'll approve it and then push a change to get Sam-I-Am and emagana's +1's22:29
*** mwagner_lap has quit IRC22:31
clarkbNakato: quickly reading that diff does it only affect extras?22:32
*** ddieterly has joined #openstack-infra22:32
*** ddieterly has quit IRC22:33
clarkbNakato: oh wait it iterates over all requirements then treats them as extras if they match?22:33
Nakatoclarkb: It iterates over all markers and evaluates them emediately, unless we're building a wheel.22:34
clarkbNakato: and what happens when it can't evaluate them?22:35
clarkbthe pass case there22:35
clarkbis that what happens when old pip is used?22:35
Nakatoclarkb: No, that means the marker isn't valid, because it's something like 'pythan_version' or something that pkg_resources doens't handle, so we leave it alone.22:36
*** rkukura_ has joined #openstack-infra22:36
*** ddieterly has joined #openstack-infra22:36
*** baoli has quit IRC22:36
clarkbNakato: then how is this helping old pip?22:36
*** ddieterly has quit IRC22:36
clarkbis it old pip + new pkg resources?22:37
*** baoli has joined #openstack-infra22:37
*** ddieterly has joined #openstack-infra22:37
* clarkb checks when pkg_resources added that support22:37
Nakatoclarkb: Because old pip doesn't even try evaluate them because the issue is with pip deciding if it's going to pass them to pkg_resources or not first.22:37
*** rkukura has quit IRC22:37
*** rkukura_ is now known as rkukura22:37
*** ddieterly has quit IRC22:37
lifelessclarkb: its all in the bug22:37
lifelessclarkb: old pip + current openstack packages, or anything using pbr with markers in fact== silently not installing dependencies22:38
clarkblifeless: no the bug says do what we do22:39
clarkbit doens't really say why dstufft suggests this22:39
clarkber do what the change does22:39
openstackgerritMerged openstack-infra/irc-meetings: Add APAC meeting for networking guide team
lifelessclarkb: the second paragraph gives the external constraints on this22:39
clarkbit isn't clear why pkg_resources would be any better at this than pip but sounds like thats because pkg_resources knows how to do this well before pip decided to leverage it22:39
*** ddieterly has joined #openstack-infra22:39
lifelessclarkb: I don't know why you are talking about pkg_resources here22:40
clarkbbceause pkg_resrouces is being used to evaluate the markers22:40
*** pradk has quit IRC22:40
lifelessclarkb: so the thing isn't that pkg_resources is being used. Its that we're evaluating them at all.22:40
clarkband I had thought that old pip and old pkg_resources had an equivalent level of marker parsing but sounds like I am wrong22:40
lifelessclarkb: marker support came in very sporadically into pip22:40
lifelessclarkb: first in wheels, then in general support22:41
lifelessclarkb: pkg_resources got it way before that22:41
clarkbright that was the peice I was missing22:41
lifelessclarkb: it may be that we're going to have an issue with very old pkg_resources in the wild, in which case we may end up with some horrid hack or something: but it will at least change from a silent fail to an explicit fail in that case22:41
lifelesswhich is strictly better22:42
*** ddieterly has quit IRC22:42
clarkbI mean if you have ubnutu lts pip and ubuntu lts pkg_resources I would expect this to still fail or I did when I frst started reading the change22:42
clarkbI have not testd this or looked at the versions in use22:42
*** amitgandhinz has quit IRC22:43
*** sdake_ has joined #openstack-infra22:43
*** slogan621 has joined #openstack-infra22:43
*** sdake has quit IRC22:44
lifelessclarkb: so precise will fail (markers were addedin 0.7) but trusty will work22:44
lifeless(and an explicit fail is better than a silent fail)22:45
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add job script
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add a gate-bindep-fallback-ubuntu-trusty job
slogan621quick question: after I commit all of the initial changes for my unofficial openstack project, on average, what is the wait time for the project to appear in the git repo? Not currently at that point, but like to know for planning. A week, a day?22:45
*** rcernin has quit IRC22:45
lifelesswe can make than work too but it will require vendoring packaging I suspect: I'd rather say we don't care about the combination of markers-on-precise22:45
clarkblifeless: and here the explicit fail would be pkg_resources does not have method evaluate_marker() yada yada?22:45
clarkblifeless: instead of just ignoring the dep?22:46
lifelessclarkb: yah22:46
clarkbya thats definitely an improvement22:46
clarkbslogan621: changes appear as soon as they are merged22:46
*** sdake_ has quit IRC22:46
*** andrewar has joined #openstack-infra22:48
*** baoli has quit IRC22:48
*** christiana_ has quit IRC22:48
*** baoli has joined #openstack-infra22:49
*** terriyu has joined #openstack-infra22:49
clarkbslogan621: do you mean once the project add changes merge?22:49
clarkbslogan621: for that it should all be done iwthin an hour (takes a puppet run which happens every 15 minutes but sometimes takes longer than 15 minutes so there is some variance there)22:50
*** hichihara has joined #openstack-infra22:50
*** flepied1 has quit IRC22:52
*** abregman has joined #openstack-infra22:52
*** flepied has joined #openstack-infra22:53
*** abregman has quit IRC22:53
slogan621clarkb: talking about that initial project add commit to infra:
*** david-lyle has quit IRC22:55
slogan621after that there is a "wait" step :-) It's that wait I wonder about.22:55
clarkbslogan621: for that step usually no more than an hour and should be less than half an hour22:55
slogan621in otherwords, no wait :-)22:56
slogan621perfect, thanks clarkb22:56
*** armax has quit IRC23:00
*** lykinsbd has joined #openstack-infra23:00
*** baoli has quit IRC23:01
*** baoli has joined #openstack-infra23:01
*** openstackstatus has quit IRC23:01
*** armax has joined #openstack-infra23:01
*** lykinsbd_ has joined #openstack-infra23:02
*** lykinsbd_ has quit IRC23:03
*** lykinsbd_ has joined #openstack-infra23:04
*** flepied1 has joined #openstack-infra23:04
*** tzn has joined #openstack-infra23:04
*** lykinsbd has quit IRC23:06
*** baoli has quit IRC23:07
*** flepied has quit IRC23:08
andrewarhello! I'm setting up a 3rd party cicd system, and I have a quick question if anyone is on23:08
jeblairanteaya, clarkb: any ideas for a word to describe "files that are not germane to a job"?  I would like to replace "skip-if: all-files-match-any:" with a one-word attribute for jobs (the new job structure in zuulv3 makes the project/branch parts of skip-if unecessary, so it can be a lot simpler)23:09
*** tzn has quit IRC23:09
mgagneirrelevant? :D23:10
asselin_jeblair, quick review of this. it's very important for 3rd party ci folks:
asselin_andrewar, just ask23:10
*** dgonzalez__ has quit IRC23:11
*** baoli has joined #openstack-infra23:11
jeblairmgagne: yeah, i started there and am looking at synonyms; kind of has a negative aspect.  though that could be fun.  :)  ("pointless")23:11
jeblairclarkb: inconsequential might be hard enough to type/spell that we'll get fewer complex skip-if changes :)23:12
mgagneI guess you already looked at the antonym dictionary23:12
*** rhallisey has quit IRC23:12
antonymmgagne: wat?23:12
andrewarok, I'm running the community supported puppet scripts on my local system, and I was getting the gerrit feed until yesterday. Now I get authentication errors, and my ssh keys seem to be gone from my launchpad account... I haven't seen anything that suggests I was spamming the system, and I'm still pointing at sandbox23:12
clarkblifeless: Nakato does fixtures honor self._setUp() in addition to self.setUp()?23:12
andrewarbut I wanted to check here23:13
clarkblooking at line 19023:13
clarkbandrewar: there is a patch let me dig it up23:13
jeblairasselin_: yeah, i was hoping jhesketh would look at it...23:14
jeblairbut i haven't seen him around today23:14
*** mriedem has quit IRC23:14
jheskethjeblair: I just popped up ;-)23:14
jheskethjeblair: which thing is that?23:14
andrewarah! Ok. I was worried I had gotten myself blocked...23:14
asselin_breaking zuul23:14
asselin_or rather, fixes broken zuul on master23:15
clarkblifeless: Nakato nevermind docs seem to explain it, is that new?23:15
jheskethasselin_: okay thanks, looking23:15
lifelessclarkb: newish23:16
*** tonytan4ever has quit IRC23:16
*** baoli has quit IRC23:17
*** yuanying has joined #openstack-infra23:18
openstackgerritMatthew Treinish proposed openstack/os-testr: Add support to ostestr to use subunit-trace color
*** spzala has joined #openstack-infra23:18
slogan621when does the tree effectively lock down, at feature freeze at R-5, or a later date? I'm guessing after that it is string changes and bug fixes only, unless the PTL allows it?23:18
clarkbslogan621: dpeends on which tree23:19
*** Hink has joined #openstack-infra23:19
slogan621does being official or non-official matter? What I mean by tree is doing git review via gerrit23:19
slogan621I'm going to be an unofficial project, if that matters23:20
*** Hink has quit IRC23:20
*** regXboi has quit IRC23:20
clarkbgerrit does not enforce any feature freeze (un)activity23:20
clarkbso you are free to do what you want. For most projects its get slushy a couple weeks before the last milestone so that yo ucan get outstanding changes in then actually freeze at the mielstone and work on bug fixes and so on23:21
slogan621ok, so, I can basically go up to the wire for Rc1 (not asking if that is smart, just possible)23:21
slogan621yeah, understood23:21
*** gildub has quit IRC23:21
*** sabeen3 has quit IRC23:21
*** ddieterly has joined #openstack-infra23:23
jheskethasselin_, jeblair: I left a comment on
*** thorst has quit IRC23:24
*** timothyb89 has quit IRC23:25
asselin_jhesketh, puppet-zuul hasn't been it's still the active configuration mechanism23:25
mordredclarkb: that is pretty much exactly what I was going to do - so I won't copy that code, but the approach is I think similar23:25
jeblairjhesketh: heh, you messed up adding your extra vowel in 'favor' ('faovur').  ;)23:25
jheskethasselin_: exactly, so I think your patch may need reworking slightly sorry23:26
jheskethjeblair: where's that?.. also not overly surprising that I can't spell :p23:27
asselin_jhesketh, (not my patch) but cbader and eantyshev have it working on their" legacy" configurations23:27
clarkbNakato: commented but didn't think any of that was worth a -123:27
jheskethjeblair: thanks, I might move some of the remaining connection stuff out from under the sql reporter and fix that up23:28
jheskethasselin_: hmm okay... I would have thought it wouldn't be able to find the connection its for23:28
asselin_jhesketh, also I didn't think this was related to the layout.yaml, but the zuul.conf23:28
jeblairjhesketh: yeah that sounds like a good prioritization now that it's out in the wild23:28
jheskethasselin_: right, but with connections when you use a reporter or trigger etc, instead of having "gerrit:" in the layout.yaml you have "connection_name:"23:29
asselin_jhesketh, I think we're disagreeing on which file is being referenced....I thought this was zuul.conf, is it really layout.yaml? ....???23:30
jheskethasselin_: it's both. They are related. zuul.conf defines the connections and gives them a name, layout.yaml uses the connections by said name23:31
*** Sukhdev has quit IRC23:32
clarkbStevenK: re the only one I worry about is swiftclient since we pin it on the log server23:32
openstackgerritKen Dreyer proposed openstack-infra/jenkins-job-builder: document workspace-cleanup wrapper "dirmatch" default
clarkbStevenK: so if jeepyb and log server were to ever mix (like in the case of an all in one third part ci setup) that could cause problems23:33
clarkbStevenK: maybe we can leave that unbound and comment about it?23:33
StevenKclarkb: Out of interest, does the log server have >=2.2.0?23:33
clarkbStevenK: I don't think so but I will double check23:33
clarkb oh hey it does23:34
asselin_jhesketh, are you suggesting line 59 be changed to  'gerrit' from _legacy_gerrit?23:34
clarkbso I am maybe overly worried23:34
clarkb(though dpeending on order of installs pip can still get confused but at least its resolvable23:34
jheskethasselin_: yes23:35
clarkbStevenK: so this should be fine23:35
*** Qiming has joined #openstack-infra23:35
StevenKThe pip resolver will fix everything ever.23:35
*** derekh has quit IRC23:35
clarkbStevenK: but only in another universe23:36
lifelessclarkb: worried about?23:37
lifelessclarkb: [I saw pip, resolver, confused, ...'23:37
*** agentle has joined #openstack-infra23:37
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Set keepalives for gerrit connections
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Tidy up loggers
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Update python-daemon
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Log a warning when zuul.conf is misconfigured
clarkblifeless: we have two different projects with slightly different deps on swiftclient, one has a cap the other doesnt23:38
*** ddieterly has quit IRC23:38
lifelessclarkb: caps bad :(.23:38
*** ddieterly has joined #openstack-infra23:38
clarkblifeless: ya unfortunately swiftclient made a non backawrd compatible api change23:38
jheskethjeblair: commented on as requested overnight23:39
*** EricGonczer_ has quit IRC23:39
clarkbStevenK: lxml say it isn't so :P23:39
lifelessI am very surprised, I would have thought swift would be the last client to do that23:39
*** tsg has quit IRC23:39
notmynamelifeless: me too ;-)23:39
clarkbnotmyname: we have discussed it previously23:39
notmynameclarkb: oh the generator thing?23:39
clarkbyes that23:39
StevenKclarkb: Hah23:39
ianwmordred / greghaynes : do you know offhand what the ubuntu-minimal image situation with /etc/resolv.conf in the built image?  is a blank one left behind?23:39
*** weshay has quit IRC23:40
greghaynesianw: you mean for the infra nodes? AIUI there are odities there but that is not part of ubuntu-minimal, it is because they use unreal resolver or w/e it is called23:40
greghaynesya, that23:40
clarkbit should point at
clarkbwhich is where unbound listens and is a forwarding caching resolver23:41
ianwyeah, a lack of /etc/resolv.conf indicates to systemd to start up it's magic resolver stuff23:41
greghaynesof course23:41
clarkbwe should just s/systemd/universald/23:42
ianwhowever, with configdrive & glean, there seems to be a chicken/egg issue of the network actually working before starting the resolver23:42
jeblairjhesketh: i'm expecting zuul to be very dynamic and constantly reconfiguring itself23:42
*** ddieterly has quit IRC23:42
ianwgreghaynes: so far as i can tell, we get a dangling symlink /etc/resolv.conf to a /run/systemd location that doesn't exist, because the systemd resolver died, but then that screws up glean23:42
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version
greghaynesianw: :/23:43
greghaynesianw: can we disable the systemd resolver stuff somehow?23:43
jeblairjhesketh: so "stopping" those every time a reconfiguration happens is... well, i mean since they aren't doing anything, i can't say what it is.  but i kind of think the idea of "stopping" something like that for a reconfiguration is not a good idea23:43
clarkbianw: is thta using the infra elements too or just a minimal set of elements?23:43
ianwgreghaynes: also, why does "glean" not have any logging or debugging about what it's doing?  very hard to work with23:44
clarkbI just wonder if unbound stuff is making it worse23:44
jeblairjhesketh: basically, whatever a future author might thing "stop" might mean is probably not what i want to have happen everytime a change merges.23:44
ianwclarkb: just trying to get glean+configdrive working23:44
clarkboh right doesn't systemd hardcode google dns too or something23:44
clarkbmaybe I am confusing it with docker23:44
mordredianw: just hadn't added any - I agree, logging would be great23:45
greghaynesianw: Yesplz, logging welcome :)23:45
*** andrewar has quit IRC23:45
*** dingyichen has joined #openstack-infra23:45
*** ddieterly has joined #openstack-infra23:46
*** flepied1 has quit IRC23:46
openstackgerritSteve Kowalik proposed openstack-infra/jeepyb: Fix pep8 failure by adding a space after a #
clarkbStevenK: comment on the translations jeepyb change though I hvaen't finished reviewing hte actual code moves23:47
clarkbStevenK: I think what you want to do is add those package installs to puppet-jeepyb and depend on that change23:47
*** sabeen1 has joined #openstack-infra23:47
ianwmordred: yeah, a review will be forthcoming :)23:49
*** ddieterly has quit IRC23:50
jesusaurusclarkb: what was setting the 500M memory safety cap? logstash 2.0 imposed that on itself?23:50
clarkbjesusaurus: yup it is part of the pakaging defaults23:50
jesusauruswas that hardcoded or % of total?23:50
clarkbjesusaurus: hardcoded23:50
clarkbjesusaurus: except now that I check again it OOM'd23:51
clarkbso I think maybe I can't run as many workers23:52
StevenKclarkb: The global requirements changed failed, so I pushed up
StevenKclarkb: How jeepyb got into that state first, I have no idea :-(23:52
jheskethjeblair: sure, I can see your point. The timer trigger isn't written like this, but needing to unload the cron jobs that it does could be an example where stop is used. There are two ways we could approach this. 1) Wait until we have a need for a stop() - whatever that may be - and then add the code in to support it. Or 2) Keep the code supporting stop() and review carefully to not let heavy tasks enter there.23:52
*** gildub has joined #openstack-infra23:53
*** earlephilhower has quit IRC23:53
jheskethjeblair: I have a preference for #1 since I'd prefer for the code to not be lost (ie if we do ever find a need for stop() it's unlikely a dev will know or remember to go dig up the code)23:53
clarkbStevenK: I went ahead and approved since its trivial23:53
jheskethjeblair: but I can understand your hesitance and happy for you to do what you think is most sensible23:53
openstackgerritMerged openstack-infra/jeepyb: Fix pep8 failure by adding a space after a #
*** lykinsbd_ has quit IRC23:55
pleia2water is coming out of the sky23:55
greghaynesianw: So I am curious what glean is doing here - is it trying to write the resolv.conf, failing, and as a result failing to set up networking?23:56
clarkbgoing to try 3 worker daemons23:56
ianwgreghaynes: i haven't yet established, but i think the issue is mostly the dangling symlink resolv.conf23:57
mordredso ...23:58
greghaynesianw: ok. That is wierd because I dont think the infra stuff which sets up the resolv.conf nor glean make a symlink for that so I assume systemd is doing it23:58
*** tzn has joined #openstack-infra23:58
mordredyeah. I didn't think we were adding any resolv.conf files at all23:58
ianwgreghaynes: oh yeah, it is systemd making this symlink.  it sees no resolv.conf, and takes that as a sign it can use systemd-resolver23:59
ianwbut then systemd-resolver dies, i think because networking is not up, and ... dangling symlink23:59
greghaynesmordred: glean does, infra puppet in dib does too and it was a workaround for something unbound related23:59
greghaynesianw: Gotcha, and I bet glean barf's on the symlink23:59
mordredyah. sorry, I mean in dib23:59

Generated by 2.14.0 by Marius Gedminas - find it at!