Monday, 2015-05-11

*** ddieterly has quit IRC00:01
*** dimsum__ has joined #openstack-infra00:06
*** sdake_ has joined #openstack-infra00:08
*** sdake_ has quit IRC00:08
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron
*** sdake_ has joined #openstack-infra00:09
*** zz_dimtruck is now known as dimtruck00:10
*** camunoz has joined #openstack-infra00:11
*** sdake has quit IRC00:12
*** dimsum__ has quit IRC00:14
*** yamamoto has joined #openstack-infra00:18
*** amotoki has joined #openstack-infra00:19
openstackgerritlifeless proposed openstack-infra/project-config: Check resolver pip on global-requirements changes
*** otter768 has joined #openstack-infra00:23
lifelessoh ugh00:24
lifelesswe do have a complete copy of the same build-everything script in requirements00:25
*** jogo has quit IRC00:25
*** ddieterly has joined #openstack-infra00:28
*** weshay has quit IRC00:28
*** Alexandra_ has quit IRC00:32
*** dimsum__ has joined #openstack-infra00:39
*** hdd has quit IRC00:40
tchaypoWhich script?00:40
*** ivar-laz_ has joined #openstack-infra00:40
*** woodster_ has quit IRC00:40
*** ivar-la__ has joined #openstack-infra00:42
*** ivar-lazzaro has quit IRC00:42
*** ivar-laz_ has quit IRC00:42
*** trey has joined #openstack-infra00:43
openstackgerritlifeless proposed openstack/requirements: Allow testing with specific pips
lifelesstchaypo: oh hai, so lets talk environment markers00:45
lifelesstchaypo: tools/ in pbr and in requirements.00:45
tchaypolifeless: also, would you have some time  this afternoon (ie, at least 2 hours from now) to talk about...00:46
tchaypoHah, you beat me to it00:46
lifelessI will have time anytime until 3.5hrs from now00:47
lifelessat which time i need to go out and pick up C etc00:48
*** dimtruck is now known as zz_dimtruck00:48
tchaypoOkay good, I'll ping you as soon as I'm free. Hopefully it will be less than 2 hours00:49
*** yamahata has quit IRC00:55
*** zz_dimtruck is now known as dimtruck00:55
*** craigbr has joined #openstack-infra00:56
*** craigbr has quit IRC00:58
*** fhubik_lunch has joined #openstack-infra01:01
*** mriedem has joined #openstack-infra01:02
*** dtantsur|pto has quit IRC01:03
*** dtantsur has joined #openstack-infra01:03
*** fhubik_afk has quit IRC01:04
*** mriedem1 has quit IRC01:04
*** julim has joined #openstack-infra01:05
*** dtantsur has quit IRC01:10
*** dtantsur has joined #openstack-infra01:11
*** fhubik_lunch has quit IRC01:11
*** fhubik_lunch has joined #openstack-infra01:11
*** fhubik_lunch is now known as fhubik_afk01:11
*** otter768 has quit IRC01:12
fungilifeless: there used to be one script, in the pbr repo. sdague forked them so that he could adjust the requirements integration job to not test things which were only relevant to pbr's integration testing01:19
lifelessfungi: well, sure - I know that history01:20
lifelessfungi: but - I don't see what value testing that everything can build successfully on a global requirements change brings01:20
fungii suspect the one in the requirements repo simply hasn't been stripped down as far as it could be01:21
*** zhiwei has joined #openstack-infra01:25
*** zhiwei has left #openstack-infra01:30
*** markvoelker has quit IRC01:35
*** markvoelker has joined #openstack-infra01:36
*** banix has quit IRC01:37
*** markvoelker has quit IRC01:37
*** baoli has joined #openstack-infra01:39
*** baoli has quit IRC01:44
*** ddieterly has quit IRC01:45
*** julim has quit IRC01:48
*** hdd has joined #openstack-infra01:50
*** mriedem has quit IRC01:53
*** achanda has joined #openstack-infra01:55
*** banix has joined #openstack-infra01:56
*** mahito has quit IRC01:57
*** dimtruck is now known as zz_dimtruck01:59
*** Shrews has joined #openstack-infra02:00
*** oomichi has joined #openstack-infra02:01
*** oomichi has quit IRC02:01
openstackgerritMonty Taylor proposed openstack-infra/shade: Straighten up keystone endpoint management
mordredlifeless: the reasoning is that sometimes a requirements bump causes a knock-on effect elsewhere which causes pain02:03
lifelessmordred: I presume its done so in the past?02:04
mordredlifeless: like, "I bumped the six version and it caused me to not be able to install eventlet" - I believe is the thing that was trying to be trapped for02:04
mordredlifeless: yah02:04
mordredlifeless: now, whether the current state can still fail in that manner is an excellent question02:04
lifelessso we have the coinstallability test02:04
lifelesswhich is up the top of the script02:04
lifelessand separate to the map over projects02:04
mordredyah. which is likely a better test from what we had before02:04
lifelessI'm not questioning the coinstallability test02:04
lifelessjust the map over projects02:05
mordredlifeless: does coinstallablility of the total list imply coinstallability of all arbitrary sub-lists?02:05
lifelesscertainly the multiple variations of per project02:05
lifelessmordred: with my resolver branch yes02:05
mordredbut not with non-resolver02:05
lifelessI can imagine corner cases with non-resolver02:06
mordredI think that's the thing - without resolver, a sub-list may exhibit a different coinstallability solution because of ordering02:06
mordredand the brute-force solution is to try each likely sublist to be sure02:06
lifelessif two components A and B depend on (bad, good) versions of C02:06
mordredand then to not trust different invocations to not alter the ordering02:06
*** mahito has joined #openstack-infra02:06
lifelessand C was in the global list02:06
lifelessbut not in the sublist02:06
mordredlifeless: have I mentioned how excited I am for resolver?02:07
lifeless(I hope)02:07
*** ddieterly has joined #openstack-infra02:07
lifelessoh man, this virus has my body messed up. Sitting and shaking now..... hope I'm well by Vancouver time02:07
mordredlifeless: I recommend not getting viruses02:08
lifelessmordred: while you're here, I'd like to remove the requirement of pip from pr02:08
lifelesssince we no longer call pip ourselves02:08
mordredI think we don't need it anymore now that we're not ... yeah02:08
mordredand that gets us back down to 0 runtime depends, which is nice02:08
*** camunoz has quit IRC02:08
lifelessthough I still see02:08
lifelesspbr/    cmd = [sys.executable, '-m', 'pip.__init__', 'install']02:08
lifelesswhats that about I wonder02:09
mordredlifeless: it's vestigal and used in a branch that never gets touched really02:10
lifelesscan we nuke it?02:10
mordredlifeless: the attempt to install test requirements before running tet02:10
mordredlifeless: the attempt to install test requirements before running test02:10
lifelessso same situation02:10
mordredyah - need to remove _PipInstallTestRequires02:10
mordredand then you can remove _pip_install02:10
mordredit can really be one big happy patch02:11
lifelesswe could fold the test requires in our projects into an extra02:11
lifelessand then install .[test]02:11
mordredwe could - but we do not use that feature02:11
lifelessok, so if we don't use it02:11
mordredlifeless: last I checked, extra is not additive02:11
lifelessextras is a set02:11
mordredlifeless: so it would need to be pip install . .[test] no?02:11
lifelessand its additive02:11
mordredit is? ok. that's way less useless than I thought02:11
mordredI though each extras list was separate02:12
*** dimsum__ has quit IRC02:12
lifelesstry pip install requests[security]02:12
lifelessin a clean venv :)02:12
lifelessthey are02:12
mordredin any case- I support removing _pip_install, its consumers, and the requirement02:12
lifelessthe install is the union of the thing and its extras02:12
mordredthat's what I want it to be02:12
lifelessoh nice02:13
lifelesswe can nuke more stuff02:13
mordrednuking stuff from pbr is always a happy02:14
*** otter768 has joined #openstack-infra02:14
mordredit means something in life has improved02:14
lifelesshooks/backwards uses parse_dependency_links02:15
mordredyah. there is a bunch of stuff in there that can die02:15
lifelesscomplex enough I'm shelving that aspect02:15
mordredyah - but I'd like to come back and nuke a large chunk of that02:15
mordredit's leftover from like 3 years ago and we don't support the thing it was written to support anymore :)02:16
lifelessbut one bite02:16
mordredlifeless: I can send you a patch for the removes if you want02:17
openstackgerritlifeless proposed openstack-dev/pbr: Finish removing invocations of pip.
lifelessmordred: sure, ^ has the pip bit02:19
lifelessI'll do a 1.0.0 tomorrow (so that infra folk are around), with whatever we have in trunk02:19
*** mmedvede_ has quit IRC02:20
mordredlifeless: you misssed _pip_install - comment left - also, there's a SKIP_PIP_INSTALL reference still in the docs02:21
*** jogo has joined #openstack-infra02:21
*** camunoz has joined #openstack-infra02:21
openstackgerritMerged openstack-infra/project-config: Remove n-novnc from enabled services
mordredlifeless: also, 181777 is unhappy with pypy, fwiw02:24
*** zz_dimtruck is now known as dimtruck02:27
lifeless181777 needs the config change done02:27
lifelessI mean, it may have more issues02:34
lifelessbut there's no point fiddling until the new job is up and done - merging it first would be bad :)02:34
*** dimtruck is now known as zz_dimtruck02:40
*** yamamoto has quit IRC02:40
*** Alexandra_ has joined #openstack-infra02:40
lifelessmordred: - updated02:40
openstackgerritlifeless proposed openstack-dev/pbr: Finish removing invocations of pip.
*** yamamoto has joined #openstack-infra02:41
*** amotoki has quit IRC02:43
*** zz_dimtruck is now known as dimtruck02:44
*** sputnik13 has joined #openstack-infra02:46
tchaypolifeless: okay, I’m settled.02:49
*** yamamoto_ has joined #openstack-infra02:49
*** mpopow has joined #openstack-infra02:49
lifelesstchaypo: ok so hi02:50
* tchaypo loads backscroll from friday02:51
*** yamamoto has quit IRC02:52
*** patrickeast has quit IRC02:52
*** mpopow has quit IRC02:54
lifelesstchaypo: also add this thread to your reading list02:55
*** amotoki has joined #openstack-infra02:56
lifelesstchaypo: may be useful too02:56
*** boris-42 has quit IRC02:58
*** dimtruck is now known as zz_dimtruck02:59
*** camunoz has quit IRC03:00
*** achanda has quit IRC03:01
tchaypoOh goody03:03
tchaypoI end up at which was filed by haypo03:03
*** camunoz has joined #openstack-infra03:04
*** fedexo has quit IRC03:04
*** achanda_ has joined #openstack-infra03:04
tchaypoCurrently reading
lifelessoh don't03:04
lifelessstay shallow for now03:04
lifelesswhats your understanding so far03:04
*** sdake has joined #openstack-infra03:05
lifeless(and do you know what environment markers even are?)03:05
*** sdake has quit IRC03:05
tchaypobut from the release notes on 6.1.1 and your thread it sounds like this is already implemented in pip/setup_tools/distutils03:05
*** fedexo has joined #openstack-infra03:05
*** sdake has joined #openstack-infra03:05
tchaypomy understanding so far is that environment markers are the thing to say “if the environment looks like X, i require Y"03:05
tchaypoobviously useful for python2 vs python3, but I suspect possibly also useful for “if redhat require blah, if debian require other blah"03:06
*** sdake_ has quit IRC03:06
lifelessless so, but perhaps03:06
tchaypoah, the pep has “if windows” and “if i386” as examples03:06
*** sputnik13 has quit IRC03:07
lifelessso - have a look at my pastebin03:07
openstackgerritSteven Dake proposed openstack-infra/project-config: Add python-cloudpulseclient project for CloudPulse
tchaypoI’m noticing lots of mentions of wheels here. My memory says that wheels only have very limited ways to indicate what they’re compatible with03:08
*** sdake has quit IRC03:08
lifelessyou'll need a python2.6 virtualenv. is a good way to get that03:09
lifelesswheels have all the same requirements capabilities as the rest of python packages03:09
*** sdake has joined #openstack-infra03:09
lifelessyou're remembering the applicability discusssion about used library files03:09
lifelesse.g. was it compiled on rhel vs debian03:09
lifelessthats a related but distinct topic, ignore for now03:10
tchaypoyes, that sounds right - it’s an issue for wheels but not for source, right?03:10
lifelessyes, but what we're interested in has to apply for both03:10
lifelessok, so - get my pastebin applied to a new git tree and try installing it in a clean venv of 2.603:12
lifelessditto a clean venv of 2.703:12
*** sputnik13 has joined #openstack-infra03:12
*** sputnik13 has quit IRC03:13
*** sputnik13 has joined #openstack-infra03:13
*** panda|afk has quit IRC03:13
*** panda|afk has joined #openstack-infra03:14
*** achanda_ has quit IRC03:14
*** ddieterly has quit IRC03:14
*** mahito has quit IRC03:16
*** alex_xu has quit IRC03:20
*** baoli has joined #openstack-infra03:21
*** alex_xu has joined #openstack-infra03:21
*** marcusvrn has joined #openstack-infra03:21
lifelesstchaypo: <crickets>03:22
*** yamamoto_ has quit IRC03:22
tchayposorry. adding the ppa and setting up the venvs.03:23
*** shashankhegde has joined #openstack-infra03:24
lifelesstchaypo: ok, for clarity:I'm doing nothing other than discuss this with you atm03:24
*** mahito has joined #openstack-infra03:24
lifelesstchaypo: if I do something else i'll context switch and be gone for an hour03:24
* tchaypo waits for packages to install03:25
tchaypoi hope you have some music playing or something03:25
lifelesslupe fiasco03:25
*** baoli has quit IRC03:26
*** mahito has quit IRC03:29
*** mahito has joined #openstack-infra03:30
tchaypomy first attempt at applying the patch gave me a file called -.rej03:34
tchaypogood work james03:34
tchayposecond attempt worked better03:34
tchaypookay, installing in clean 2.6 venv03:35
tchaypothat seems fine. trying 2.7..03:35
tchaypoand that works fine. didn’t install argparse, as expected03:35
lifelessthe thing we want03:37
lifelessis some machine mergable requirements thing03:37
lifelessrequirements.txt gets machine gardened with human approvals03:38
lifelessvia in requirements - have you looked at that?03:38
lifelessextras_require in is not machine gardened today03:38
lifelessand since its editing-python-source less easily machine gardened than either requirements.txt or setup.cfg03:39
lifelesssince the setup.cfg stuff is still in nightmare territory03:39
lifelessis the way of the future03:39
lifelessI'm going to suggest an impl strategy, which is to teach pbr to read extra requires from setup.cfg03:39
*** sputnik13 has quit IRC03:40
*** marcusvrn has quit IRC03:40
lifelessand inject that like requirements.txt is injected into the Distribution metadata03:40
tchaypostepping back03:40
tchaypowhy are the extras_require in
tchaypooh. because pbr doesn’t know how to read them from setup.cfg03:41
lifelessbada bing bada boom03:41
tchaypoI might be confused about what extras_require is then03:41
*** sputnik13 has joined #openstack-infra03:41
lifelessits a dict of conditional requirements03:42
lifelessprior to environment markers the condition was only activated one way: by someone requesting one or more of the conditions03:42
tchayponope, I’m not confused, I just hadn’t seen it used for this03:42
lifelesse.g. requests[security]03:42
tchaypothe usage I’m familiar with is..03:42
tchaypoexactly, requests[security]03:42
lifelessthe extra '' is == install_requires03:43
lifelessenvironment markers make it possible to automatically activate additional requirements03:43
tchaypoand it makes sense to put the env-marker stuff in extras_requires03:43
lifelessby having an extra of '' with the condition <whatever>03:43
lifelessexpressed as ':condition'03:43
tchaypobecause the stuff that’s in install_requires always needs to be installed, the stuff in extras only needs to be installed sometimes03:44
lifelessits batshit insane as far as UI goes03:44
lifelessbut there it is03:44
tchayposo, next question03:44
*** yamamoto has joined #openstack-infra03:44
tchaypo"The micro-language behind this is a simple subset of Python: it compares only strings, with the == and in operators (and their opposites), and with the ability to combine expressions"03:44
tchaypoyou’re proposing to teach pbr to understand a simple subset of python?03:45
lifelessI'm proposing to teach pbr about extras03:45
lifelessin a way that will be compatible with the broad needs of the requirements update process03:45
tchaypoand if pbr can read the extras, it can just pass the strings through as-is and let the existing machinery handle the parsing03:46
lifelessyes, that is what pbr does for requirements files too, FWIW03:47
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Update Kolla Meeting
*** hdd has quit IRC03:49
tchaypookay. So what would these look like in setup.cfg..03:51
*** banix has quit IRC03:51
tchaypohrm. This sounds like it depends on the work to make pbr read install_requires from setup.cfg rather than requirements.txt03:53
lifelessI had an implementation03:53
lifelesslet me push it back up (its forpip, but the schema and input code is what you'll need03:53
*** sdake_ has joined #openstack-infra03:56
lifelesstchaypo: just having to deal with some conflicts03:57
lifelesstchaypo: no, install_requires moving to setup.cfg is entirely separate03:57
*** camunoz has quit IRC03:58
*** rkukura has quit IRC03:59
*** rkukura has joined #openstack-infra03:59
*** sdake has quit IRC04:00
*** sputnik13 has quit IRC04:02
tchaypookay. that seems backwards to me04:02
lifelesstchaypo: (the discussion is irrelevant for now)04:02
tchaypohaving extras_requires in setup.cfg but not install_requires04:02
lifelessthats not a goal04:02
lifelessits just not a dependency04:02
tchaypo*thinks* on the other hand, pbr can already handle install requires somewhat, but it can’t handle extras at all04:02
*** sputnik13 has joined #openstack-infra04:03
lifelessis the format bit04:03
lifelesswas the parsing bit04:04
lifelessyou probably want to keep things in string format04:05
lifeless- I'll leave that to you to explore :)04:05
lifelesstchaypo: and yes, install_requires works - its reflected into by pbr already.04:05
*** mpopow has joined #openstack-infra04:05
lifelessmordred: around ?04:06
tchaypowhat I’m not seeing there is an example of an env marker inside setup.cfg04:06
lifelesstchaypo: of course, since I wasn't working on env markers04:06
lifeless:python_version=='2.6' =04:07
lifeless  argparse04:07
tchaypoyep, that’s what I was just coming to04:07
lifelessit might fall into a screaming heap04:07
lifelessdepending on iniparse04:07
tchaypothere’s really no difference between the env-marker and just-a-label case as far as pbr is concerned though04:07
tchaypoeither way it’s a string label to pass on, and The Machinery will handle it appropriately04:08
lifelessConfigParser.ParsingError: File contains parsing errors: setup.cfg04:09
lifeless        [line  7]: ':foo = \n'04:09
lifeless        [line  8]: '  bar\n'04:09
lifelessquotes work04:09
lifelessoh wow04:10
lifelessthats messed up04:10
*** camunoz has joined #openstack-infra04:10
lifeless":foo" =04:11
lifeless  bar04:11
lifelessgets me04:11
lifeless>>> r.items('extras')04:11
lifeless[('"', 'foo" =\nbar')]04:11
lifelessyes, a key of " ....04:11
lifelesssingle quotes fail similarly04:12
lifeless; is a comment04:12
lifeless| works04:12
lifeless>>> r.items('extras')04:13
lifeless[('|foo', '\nbar')]04:13
tchayposo to extend on that, it sounds like we don’t have to teach pbr about env markers as such04:13
lifelesstchaypo: we will need to04:14
tchaypowe need to teach it about reading extras in setup.cfg, and that should mean it now supports env markers04:14
lifelesstchaypo: because markerlib / pkg_resources know : as the separator04:14
lifelessso we'll need to translate '|foo' to ':foo'04:14
*** ddieterly has joined #openstack-infra04:15
lifelesswe don't need to write an interpreter or anything, if thats what you were speculating about04:15
tchayposo env markers have to be specified with a : in but a | in setup.cfg? I don’t like that much.04:16
lifelessI suspect that this may have cascading failure modes once we get it out there with the use of ini file keys as extra keys04:16
lifelesstchaypo: they are specified with a ; in requirements.txt04:16
lifelesstchaypo: so don't let it both you too much04:16
lifelesstchaypo: you know the three step program :)04:17
tchaypoAt least it should be safe to assume that we’re not going to worry about a package called “|foo”, right04:17
tchayporight? Please say I’m right04:17
lifeless| is not a legal package element TBBOMK04:17
tchaypoI’m just going to go with that.04:18
lifelessthough I can make an import a package called |04:18
lifelessbut, I'm evil.04:18
lifeless>>> __import__('|')04:19
lifeless<module '|' from '|/'>04:19
* tchaypo looks at birds in the park04:19
*** ddieterly has quit IRC04:19
tchaypomuch prettier04:19
lifeless>>> __import__('#|').name04:20
tchayporeading it seems like pkg_resources.parse_requirements already handles the requirements04:20
lifelessyes, the problem is marshalling04:20
tchaypoand it should be fairly trivial to iterate over them and push the result into dm04:20
lifelessand verification04:20
lifelessthe _dep_map thing I do there is irrelvant for pbr04:20
lifelesslook at pbr's existing code to see how marshalling is done on pbr's side04:21
tchaypoI understand “marshalling” to mean converting a data type from the form used by one language into the form used by a different language04:21
lifelessand yes04:21
lifelesssetuptools doesn't know about requirements.txt. pbr does. pbr marshalls the install_requires from requirements.txt into a form setuptools understands04:22
tchayponot sure why that’s relevant here - doesn’t setuptools already know about setup.cfg?04:23
*** coolsvap|afk is now known as coolsvap04:23
lifelessit doesn't process dependency data from it04:23
*** sks has joined #openstack-infra04:23
lifelessyou can make the argument that you should do this in setuptools04:24
tchaypopip/2603 was an attempt to add it in there04:24
lifeless2603 is about pip, defining something I hope to also add to setuptools04:24
lifelessthere's a mess of chicken egg around this04:24
lifelesspbr needs to kep doing requirements.txt indefinitely for backwards compat04:25
lifelessI am not sure that you could sanely do requires in setup.cfg and not have pbr stomp over stuff by mistake04:25
lifelessI know the pbr codebase well04:26
lifelessand can support you04:26
tchaypoam hunting for the bit where the marshalling happens in pbr04:26
lifelesssetuptools I have not yet tackled04:26
lifelessand I'm told its worse than pip was before I started cleavering04:27
*** yamamoto has quit IRC04:28
*** ildikov has quit IRC04:28
lifelessthere was a thing call d2to104:29
lifelesspbr is sortof a fork of that04:29
lifelessit may be that somewhere in its guts it supports extras already04:30
tchaypomy memory is a bit hazy04:30
lifelessthe author claimed that but I could not figure it out04:30
tchaypobut memory says there was a thing called setuptools, which got forked to make distutils, and they developed seperately; and then distutils2 was formed by merging distutils with setuptools04:30
*** yamamoto has joined #openstack-infra04:30
lifelessthose all other different things than d2to104:31
*** ildikov has joined #openstack-infra04:31
lifelessdistutils came first, setuptools layers on top of distutils, distribute was a fork of setuptools, distutils2 was a short lived thing before they went back to the name setuptools04:31
tchaypoin unrelated news, the birds in the park have stopped being pretty04:32
tchaypoone of them seems to be emulating a car alarm04:32
tchayposo d2 is distutils2?04:32
lifelessthats scary 1327 downloads in the last month04:33
lifelessof distutils204:33
tchaypod2to1 (the ‘d’ is for ‘distutils’)04:33
tchaypoit’s using distutils2 setup.cfg in a distutils1 setup.py04:34
lifelessnot quite04:34
lifelesssetuptools setup.py04:34
lifelessand actually we're in the 'pbr' context04:34
*** sdake has joined #openstack-infra04:34
lifelessif you think too hard on this, you'll be ready for our tentacled master to rise from the cthonian deeps04:35
*** otter768 has quit IRC04:35
tchaypodid lovecraft ever play with using extra spacing between paragraphs for dramatic effect?04:36
tchaypoI started reading a book yesterday that has two newlines between paragraphs, and it turns out that’s enough space to have me mentally reset and ready for a new scene, and so I keep getting surprised when the new scene has the same setting and the same characters continuing the same conversation04:36
*** sputnik13 has quit IRC04:36
tchaypoit makes me feel like I (or maybe the characters) keep passing out in the middle of the action04:37
lifelesssure thats not your sleep apnea?04:37
tchaypoit’s a very similar feeling to what I used to get04:37
*** ivar-la__ has quit IRC04:37
*** markvoelker has joined #openstack-infra04:37
*** sputnik13 has joined #openstack-infra04:38
*** ivar-lazzaro has joined #openstack-infra04:38
*** sdake_ has quit IRC04:38
tchayposo I’m ignoring d2to1 for now04:38
tchaypoit looks like extras is going to need to be added to MULTI_FIELDS04:39
mordredlifeless: sup?04:39
lifelessmordred: the pbr change failed because I deleted requirements.txt AFAICT04:39
lifelessmordred: I wanted a quick touchstone on WHY WE CAN"T HAVE NICE THINGS.04:39
tchaypois the comment at about this possibly evolving with distutils2 still current? or is that a hangover from the d2to1 legacy?04:40
lifelessmordred: I note that
mordredlifeless: well, we can't have nice things because eek04:40
lifelessmordred: now specifies requirements.txt04:40
mordredit does04:41
lifelessmordred: I think we do need to specify where, because of update.py04:41
lifelessmordred: but I don't think our tooling should ever need to install *install_requires* via -r requirements.txt of a repo04:41
mordredtchaypo: hangover04:41
mordredlifeless: well, we currently do in tox.ini04:41
mordredlifeless: although I largely agree with your premise04:41
lifelessmordred: - concretely, since I want move us to having deps in setup.cfg anyhow04:41
*** stevemar has joined #openstack-infra04:42
mordredlifeless: yah - once we're ready for that, I think we can take a PTI change to the TC04:42
lifelessmordred: so, there are two things04:42
mordredwhee! two things!04:42
lifelessmordred: one we need a PTI change for (and changes)04:42
lifelessmordred: which is 'how do we introspect in the requirements-management tooling to do updates/consistency checks'04:43
lifelessmordred: the other one is 'how do we install stuff' - and that should just be pip install .[test] everywhere04:43
lifelessmordred: which, once I've validated (and fixed probably) pip for that, isn't a PTI change04:43
lifelessmordred: - IMO anyhow04:43
mordredlifeless: yah. once we're to the point where that works - I think that's a great change to the places where we're doing -r requirements.txt04:44
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add in extra connections tests
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Refactor sources out of triggers
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Document the new connections in zuul
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add gerrit reviews into patchset approvals
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for negative requirements
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for triggers
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for sources
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for reporters
lifelessmordred: so -r requirements.txt should be replaceable by pip install . --only-dependencies04:44
lifelessmordred: -r test-requirements, pip install .[test] --only-dependencies04:44
lifelessmordred: great04:44
lifelessshoved onto my stack04:44
mordredlifeless: right - but we generally don't need to install the deps without the thing04:44
mordredlifeless: EXCEPT04:45
*** ivar-lazzaro has quit IRC04:45
lifelessmordred: in test :)04:45
*** ivar-laz_ has joined #openstack-infra04:45
mordredlifeless: for setup_requires of pbr04:45
lifelessmordred: thats not in requirements.txt today anyhow04:45
*** ddieterly has joined #openstack-infra04:45
mordredlifeless: which is one of the reasons we do -r requirements -r test-requirements in tox.ini - because tox does it before it touches the setup.py04:45
mordredwhereas pip install . touches the before it touches test-requirements.txt04:46
mordredwhich can trigger easy_install04:46
lifelessbad news for you04:46
mordredI love bad news04:46
lifelessmany of our projects don't have a runtime dep on pbr04:46
lifelessso we're vulnerable to easy-install of pbr unless something in their deps happens to bring it in as a runtime dep04:47
mordredthis is true - I'm not saying it's a great plan or even correct :)04:47
lifelessso, I'm going to ignore it and treat that as something to explicitly handle04:47
mordredthat would be hacking04:47
lifelesspip install pbr .[test]04:47
mordredhacking is in everything's test-requirements.txt04:47
mordredit has a runtime on pbr04:47
mordredI like your plan in general04:47
mordredjust making sure I say all the things I know about past context in case there are things underneath rocks04:48
lifelessthats why I pinged you04:48
lifelessbecause I hate rocks.04:48
mordredI have rocks in my head :)04:48
lifelessand cthulu04:48
mordredlifeless: so - are you going to make pbr grok test-requirements.txt into an extras_require test as a stepping stone?04:49
lifelessmordred: no04:49
lifelessmordred: I'm going to drink heavily04:49
mordredlifeless: that's the right choice04:49
tchaypolifeless: so it looks like the bulk of the marshalling is going to happen around; and it looks like a way to test this will be to look at the kwargs that function returns04:50
*** ddieterly has quit IRC04:50
tchaypomordred: while he’s drinking, I’m going to try to make pbr grok extras_require from setup.cfg04:50
tchaypoI think04:50
mordredtchaypo: neat04:50
mordredtchaypo: you'll need alcohol after taht04:50
lifelesstchaypo: sounds plausible04:51
lifelesstchaypo: oh yeah, and about 1000 folk are going to owe us drinks after this.04:51
mordredtchaypo: btw - there are many mentions in the codebase about being compatible with various things - most of those mentions are hysterical04:51
mordredtchaypo: so, you know, if they seem odd - don't be wary of challenging their existence - many may have grown _other_ reasons for being important, but there is some deep dark cruft up in there04:52
* mordred hands tchaypo a headlamp and a pickaxe04:52
lifeless(or just ask me, I am fairly current)04:52
mordredwell, that's what I meant04:52
openstackgerritlifeless proposed openstack-dev/pbr: Finish removing invocations of pip.
lifelesslets see if thats happier04:53
lifelessmordred: so the roadmap thats slowly forming in my head is04:53
lifelessmordred: pbr extras support in setup.cfg04:54
lifelessmordred: extras support, using setup.cfg04:54
lifelessmordred: pip --only-dependencies and pip .[test] support.04:54
mordredlifeless: sounds good - what does "pip --only-dependencies" support mean?04:54
lifelessmordred: pull the trigger on removing all references to *requirements*.txt outside of global-requirements processing04:55
mordredas in - what does supporting that look like other than just typing --only-dependencies ?04:55
lifelessmordred: it won't install the thing itself04:55
lifelessmordred: pip install -r requirements.txt04:55
lifelessmordred: == pip install --only-dependencies .04:55
*** afazekas has joined #openstack-infra04:55
lifelessmordred: (for us)04:55
mordredyes. grok that04:55
mordredjust saying - is there _work_ to be done to support that?04:55
lifelessyes, there's an open, approved in principle issue for it04:56
lifelessits fairly shallow04:56
mordredok. cool04:56
lifelessthere's already --ignore-deps04:56
lifelessso this probably becomes --only-deps04:56
lifelessand is symmetric04:56
mordredthe work is "add an option to pip called --only-dependencies"04:56
mordredI wasn't sure if the work was in pbr or in pip - thanks - I am with you now04:56
lifeless(or $*(bikeshed))04:57
*** yamamoto has quit IRC04:57
lifelessa separate arc is to get environment markers in place04:57
lifelesswhich I've convinced tchaypo is contained enough and relevant enough he's poking at now04:57
lifelesswhen thats in place, we can start the crusade to remove requirements$version.txt04:58
tchaypofirst I’m making sure I can run tests on pbr04:58
lifelessanother arc is to get install_requires into setup.cfg04:58
tchaypothen I’ll add a test case that has a setup.cfg with extra_requires and looks for them in kwargs04:58
tchaypothen I’ll make that test not fail04:58
lifelesswhich will be the last one I think, thats the point where we can move everything into setup.cfg04:58
tchaypothen tomorrow we can see if this translates to something useful04:58
*** deepakcs has joined #openstack-infra04:58
lifelesstchaypo: coool04:59
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add in extra connections tests
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Document the new connections in zuul
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for triggers
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for sources
mordredlifeless: I'd say we should replace setup.cfg with setup.yaml so that we wouldn't have to play the awful parsing games ... except yaml isn't in the stdlib :(05:00
mordredand json is not for humans05:00
lifelessmordred: yaml is an awful parsing game05:00
lifelessmordred: stdlib inclusion is irrelevant here05:00
mordredlifeless: possibly - but it's a very well defined one05:00
lifelessmordred: with objects. Honestly.05:00
mordredlifeless: as opposed to the meaning we have to put into intra-line syntax in an ini file05:00
lifelessanyhow, we can use arbitrary libraries if we want to, once setup-requires is fixed05:01
mordredwell, let's ignore the abritrary object serialization madness05:01
mordredthis is try05:01
mordredvery true05:01
mordredI just don't live in that world in my head yet05:01
lifeless<- has vision05:01
mordredbecause I'm terrified that if I believe it it'll not come true05:01
tchaypo  Could not find a version that satisfies the requirement fixtures>=0.3.14 (from -r /Users/james/src/openstack-dev/pbr/test-requirements.txt (line 3)) (from versions: )05:02
tchaypo  No matching distribution found for fixtures>=0.3.14 (from -r /Users/james/src/openstack-dev/pbr/test-requirements.txt (line 3))05:02
mordredthat sees unlikely05:03
tchaypoit says it couldn’t find fixtures on http://localhost:3141/root/pypi/+simple/fixtures/ (my local devpi instance)05:03
tchaypobut I can see it. I’m assuming it was a timeout..05:03
tchaypoyep, running it again and it works fine now.05:03
mordredbtw - I've been uploading an image to a cloud in china for the last 7 hours or so - impressive thing is - it's still uploading05:03
mordredtchaypo: devpi's transparent grabbing of missing things in the background can occasionally lead to timeouts05:04
*** sputnik13 has quit IRC05:05
tchaypousually when I’ve seen it fail it has been pretty opaque05:05
tchaypoas in, pretty evident what happened.05:06
mordredspeaking of bandersnatch - we really need to get jeblair's patch to that landed05:06
tchaypolifeless: doesn’t do external packages, and even if it did, I’m not wasting >90Gb of laptop space on packages I don’t use.05:06
mordrednext time anyone complains about patch velocity in openstack ...05:06
tchaypobandersnatch has its place, but my laptop isn’t that place05:06
lifelessdo you have an SSD ?05:07
lifelessmordred: oh? link me05:07
tchaypoyes, i have an ssd05:07
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Add nodejs to the source modules
*** sputnik13 has joined #openstack-infra05:08
lifelessmordred: how often does HP buy its staff laptops?05:08
mordredlifeless: almost never05:08
lifelessmordred: 3 years? 4? I have a vague memory of 4.05:09
*** chlong has quit IRC05:09
mordredlifeless: yeah - I think the lifecycle length is actually 405:09
mordredwhich is, of course, _crazy_05:09
mordredI usually trash a laptop after about 1 year05:09
mordredthis one ahs been going for almost 1.5 and is about done05:09
lifelessChristian is normally pretty good05:10
tchaypomy previous laptop is 4 and a bit years old and just starting to act a little tired05:10
lifelessI suspect bitbucket UI fail05:10
openstackgerritJoshua Hesketh proposed openstack-infra/puppet-zuul: Use the latest jquery on zuul
lifelessjeblair: have you tried pinging ctheune (e.g. via the bitbucket send message feature) ?05:11
mordredlifeless: yah. the last time christian responded, jeblair responded in 2 days. that was in december - since then the codebase has changed and produced conflicts :)05:11
mordredlifeless: which is fine - it happens to the best of us05:12
mordredit's just on the TDL and you said bandersnatch which reminded me05:12
*** sdake_ has joined #openstack-infra05:13
openstackgerritJoshua Hesketh proposed openstack-infra/nodepool: Remove the need for zmq in proper demand calcs.
mordredokie. I will now go to the bed05:13
*** emagana has joined #openstack-infra05:15
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Assume trailing slashes want to load indexes
*** sdake has quit IRC05:16
*** stevemar has quit IRC05:17
*** yamamoto has joined #openstack-infra05:17
tchaypoho hum.05:17
tchaypo… actually, no ho.05:18
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Assume trailing slashes want to load indexes
*** BharatK has joined #openstack-infra05:19
*** unicell1 has joined #openstack-infra05:20
*** unicell has quit IRC05:21
lifelessmordred: night05:21
*** woodster_ has joined #openstack-infra05:24
*** Alexandra_ has quit IRC05:24
openstackgerritJoshua Hesketh proposed openstack-infra/os-loganalyze: Create devstack plugin for testing os-loganalyze
openstackgerritMerged openstack-infra/zuul: Update the description for grey icons on status pg
*** sputnik13 has quit IRC05:28
*** ibiris_away is now known as ibiris05:29
*** heyongli has quit IRC05:29
openstackgerritJames Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras from setup.cfg
*** yamamoto has quit IRC05:30
*** sputnik13 has joined #openstack-infra05:30
*** heyongli has joined #openstack-infra05:31
tchayposo let me go flag that as WIP properly05:32
tchaypobut lifeless does that file format look right to you?05:32
*** ildikov has quit IRC05:32
*** camunoz has quit IRC05:33
*** yamamoto has joined #openstack-infra05:33
lifelessbut right05:33
lifelessplease put a blank line between the sections though05:33
*** achanda has joined #openstack-infra05:37
mgedminugh, won't the first '=' be interpreted by the ini parser?05:37
*** yamamoto has quit IRC05:37
*** koolhead17 has joined #openstack-infra05:39
*** jogo has quit IRC05:39
lifelessmgedmin: sufficiently not, that it works05:39
lifelessmgedmin: its ninth circle of hell territory, but wtf, deal the cards we have05:39
mgedminI can't parse that05:39
*** jogo has joined #openstack-infra05:39
*** ibiris is now known as ibiris_away05:40
lifelessmgedmin: its not interpreted by safeconfigparser05:40
lifelessmgedmin: others might interpret it05:40
mgedminah, you mean pbr doesn't use configparser05:40
lifelesswe use it05:40
mgedminmy experiments contradict your assertio05:43
*** ibiris_away is now known as ibiris05:43
*** camunoz has joined #openstack-infra05:46
*** ddieterly has joined #openstack-infra05:46
tchaypomgedmin: i dislike your experiment05:47
*** btully has joined #openstack-infra05:47
tchaypobut all the same I’d rather find out now than later05:47
mgedminit's not the most elegant code I've ever written05:47
*** yamamoto has joined #openstack-infra05:47
lifelessmgedmin: use items, not options05:48
lifelessmgedmin: your experiment is broken :)05:48
mgedminI get the same result if I replace for k in p.options(s): with for k, v in p.items(s)05:49
openstackgerritJames Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras from setup.cfg
lifelessmgedmin: so I tested this :)05:49
*** shashankhegde has quit IRC05:49
* tchaypo re-marks as wip, since we can’t have nice things05:49
tchaypolifeless: the bits you pasted in channel didn’t seem to have == in05:49
* mgedmin would like to register an objection to calling that syntax "nice"05:49
lifelessut perhaps not enough05:49
lifelessmgedmin: thank you for being paranoid05:50
*** ddieterly has quit IRC05:50
*** mpopow has quit IRC05:51
mgedminI've used 'foo = value that contains embedded = signs' in the past, I know what to expect05:51
lifelessmore grist for the approach of putting these things on extras being a tad fugly05:53
*** sdake has joined #openstack-infra05:57
*** mahito has quit IRC05:58
*** shashankhegde has joined #openstack-infra05:59
*** Longgeek_ has joined #openstack-infra05:59
tchaypohrm. My results differ.06:00
*** rmcall has quit IRC06:00
lifelesstchaypo: from mgedmins?06:01
lifelessI've verified his06:01
*** tnovacik has joined #openstack-infra06:01
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** sdake_ has quit IRC06:01
tchaypoi get different results with options and items06:01
tchayponeither particularly useful06:02
tchaypo… no I don’t. I was just reading them wrong06:02
lifelesswe could possibly use ConfigObj06:03
lifelessif we vendor it06:03
mgedminquestion is: do you want to push something like this to be a Python-wide standard?06:03
mgedminor just a pbr thing06:03
lifelessmgedmin: gosh no06:03
*** Longgeek has quit IRC06:03
lifelessmgedmin: I want something to unbreak the logjam we're in06:03
*** mahito_ has joined #openstack-infra06:04
lifelessmgedmin: which we can preserve as ugly back-compat here indefinitely06:04
lifelessmgedmin: and learn about the problems its going to have (like this!)06:04
*** luqas has joined #openstack-infra06:04
mgedminmaybe namespace the section name then?  [pbr:extras] |py26 = ...06:05
lifelessmgedmin: thats a not-unreasonable idea, though we're not namespaced for metadata either06:05
lifelessand thats much more intrusive and widely used06:05
tchaypoit’s not unreasonable, but06:06
tchaypoif we want to support all the flexibility of
*** yamamoto has quit IRC06:06
lifelessI've done some experiments06:06
lifelessand interpolation doesn't work in keys06:06
lifelessso we seem to have the following options to me:06:06
lifeless - embed all the extras in one text field06:07
tchaypoit’s easy enough to map “|py26” to “python_version==2.6”, but we need to support an arbitrary subset of python there..06:07
lifeless - make the keys be ignored, put everything in the text fields06:07
lifeless - use a different container format06:07
*** yamamoto has joined #openstack-infra06:08
tchaypocan that test field be json? I can’t imagine anyone being upset with json embedded in an ini</sarcasm>06:08
lifeless - use a different ini file parser [e.g. configobj with nested things]06:08
lifelesstchaypo: it probably would be yes06:08
tchaypoI don’t like the idea of relying on the quirks of one particular ini file parser06:08
*** koolhead17 has quit IRC06:08
lifelesstchaypo: the reasons to embed in a text field would be to avoid the dependencies implied by having a nice format for a new file (such as yaml)06:08
tchaypoone of the reasons i get so upset every time someone says “the config file is a standard ini file” is because N developers have N+3 ideas of what “standard ini file format” is06:09
mgedminhaha yes06:09
mgedminat least in Python land we have ConfigParser in the stdlib06:09
*** luqas has quit IRC06:09
*** mrunge has joined #openstack-infra06:09
tchayposo if we’re going to use ini files, I’d prefer to stick with ConfigParser.06:10
openstackgerritSirushti Murugesan proposed openstack-infra/project-config: Add grenade jobs for Heat
mgedminwhat was the pep that defined environment markers?06:10
lifelessthe markerlib impl is a little incomplete06:11
lifelessits missing <= and < for instance06:11
lifelessbut, if we can get the dots joined so things tunnel together properly06:11
lifelesswe'll be in good shape06:11
lifelesstchaypo: I think we should get something working06:11
tchaypoyaml is out because using yaml would mean you can’t reliably run without having a yaml parser installed first.06:11
lifelesstchaypo: and then do a spec and get input on the right way to make it nice06:12
lifelesstchaypo: no, thats not why yaml is out ;)06:12
tchaypolifeless: you mean, it’s not the only reason yaml is out, or that it’s not a a reason why yaml is out?06:12
lifelesstchaypo: yaml is out because having pbr install_require other libraries leads to the corner cases of easy_install affecting other things06:12
lifelesstchaypo: we already have this dep on easy_install for pbr06:13
tchaypoyep. if yaml was in core it wouldn’t be an issue06:13
lifelesstchaypo: I may be splitting hairs. But your statement was incorrect.06:13
lifelessYou can run it reliably. Just not safely.06:13
mgedminwhat if you invert the syntax?  instead of [extras] env_marker = pkg you do [metadata] requires = foo [env_marker] bar [env_marker] ... ?06:14
*** zhiwei has joined #openstack-infra06:14
*** rmcall has joined #openstack-infra06:14
mgedmin[ and ] are not valid inside env markers (unless inside strings)06:14
lifeless[] define extras06:14
mgedmindarn, they're used for extras06:14
mgedminuh, {env_marker}?06:15
lifelessthe setuptools UI for this is terrible IM06:15
*** sdake_ has joined #openstack-infra06:15
lifelessbecause its not per requirement06:15
lifelessbut I think thats been done for graceful backwards comapt06:15
lifelessbecause parse_requirements has no mechanism defined to make a requirements list contain markers06:16
tchaypoputting them as a single key seems to work…06:16
openstackgerritSteven Dake proposed openstack-infra/project-config: Add python-cloudpulseclient project for CloudPulse
lifelessI don't recall if there's a inter requirement separator06:17
lifelessfoo, bar06:17
*** sdake__ has joined #openstack-infra06:18
*** sdake has quit IRC06:18
*** koolhead17 has joined #openstack-infra06:18
lifelesstchaypo: you can parse that as a human. Its complex enough I'd want to write a proper parser for it I think06:19
lifelesstchaypo: we avoid most of the complexity today by one-line-per-requirements, and use parse_requirements06:19
tchaypothat’s slightly different to my idea06:20
tchaypoone moment06:20
lifelesstchaypo: that was representing one extra06:21
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
lifelesstchaypo: with a marker and four dependencies06:21
clarkbwhat happened to $requirement>=x.y ; python == 2.6 ? I want to say that was supposed to be the thing? (or is it just too hard to make pip honor that right now)06:21
*** sdake_ has quit IRC06:21
lifelessclarkb: thats the syntax for requirements.txt06:21
lifelessclarkb: which is not that for setuptools06:21
lifelessclarkb: also its python_version=="2.6" :)06:22
clarkbugh so we get two different overlapping things that are both bad?06:22
lifelessclarkb: three or four eventually probably06:22
lifelessso there are two bad things here06:22
lifelessfirstly, setuptools doesn't support environment markers on requires06:23
lifelessyou can't say 'requirement[EXTRAS] VERSIONSTUFF ; marker'06:23
BharatKHi, After running GlusterFS CI I am getting all "/var/log/glusterfs" logs of ZERO size.
*** yfried_ is now known as yfried|afk06:24
lifelessyou can currently only talk to setuptools via setup.cfg06:24
*** yfried|afk is now known as yfried_06:24
lifelesssetup.cfg [metadata] and requirements.txt are both all pbr things06:24
tchaypo*checks* yep, my version seems to work06:24
*** jamespage_ has quit IRC06:24
lifelesswe need to do related but distinct things06:25
tchaypoby work, I mean that ‘ordinary’ extras can be simple key=value lines, it’s only env markers that need special handling06:25
lifelessa) we need to make it possible for users to define extras in the pbr world outside of, including environment markers.06:25
clarkbyes a should be my example06:25
lifelessb) we need to reflect environment markers in requirements.txt requirements into the extras06:25
clarkbpbr can inject whatever setuptools wants06:26
lifelesspbr has no means to define extras today at all06:26
clarkbbuts use my example as a source then you dont write this twice in two terrible languages06:26
lifelessthe subset of extras that is environment markers on install_requires06:26
lifelessclarkb: there is no defined language for extras for pbr today06:26
*** scheuran has joined #openstack-infra06:26
clarkbIm not talking extras, just markers06:26
lifelessclarkb: we're talking extras06:27
clarkbthen why all the effort with markers?06:27
clarkbyou will just end up with two places defining the same info06:27
lifelessclarkb: because a new thing to do extras that can't also do markers will be immediately obsolete06:27
lifelessclarkb: no we won't06:27
lifelessclarkb: I think there's a disconnect here. I don't want to have a back-and-fro debate while we tease it out06:28
clarkbyes any pip markers will go in requirements.txt then again in setup.cfg for setuptools06:28
lifelessclarkb: can I ask you let us roll with it and assume some clue?06:28
lifelessto answer that concern, you wouldn't put an install_requires entry in via markers if you're using pbr06:28
lifelessbecause pbr will be getting taught to marshall that smartly06:29
lifelessbut we have a feature request for pbr to do markers for things like backends06:29
lifelesse.g. sqlite deps if you ask for the sqlite backend06:29
lifelessmysql for mysql etc06:29
lifeless*those* also need to support markers.06:29
lifelessSo we need a good solid proper implementation of extras that supports markers.06:30
tchaypoAre we still planning to have pbr be able to read install_requires from setup.cfg instead of requirements.txt06:31
lifelesstchaypo: thats a separate arc of work06:31
*** jcoufal has joined #openstack-infra06:31
lifelesstchaypo: its marginal benefit, long term alignment as a goal there.06:31
clarkbI would say its desireable for consistency but also separate06:31
tchaypobut if it happens, we’d need to end up being able to read the markers out of setup.cfg anyway, right?06:31
lifelesstchaypo: yes, and that suggests another option06:32
tchaypoI agree that it’s seperate, but also it seems desireable to end up with a format that’s still going to make sense when we’re reading install_requires from setup.cfg06:32
lifelesstchaypo: which is to do the somewhat convluted logic of translating things upfront06:32
clarkbtchaypo only if we assume extras go in setup.cfg06:33
lifelesstchaypo: the install_requires 'format' of one-per-line is much easier than extras which is a dict of one-per-line things06:33
tchaypoagreed. ConfigParser seems to be pretty insistent on splitting on = though06:34
lifelesstchaypo: not in values06:34
clarkbwhich is a decent assumption except for how the initial work around this happened06:34
tchaypoargparser>1.0 ; python_version==2.6 <<- splits on the first =06:34
lifelessclarkb: there was a thread about moving to setup.cfg on the list; if you disagree, could you chime in there?06:34
lifelesstchaypo: yes, but thats only if you give it as a key=value06:35
lifelesstchaypo: when its all a value its fine06:35
tchaypooh sure06:35
tchaypowhen it’s *all* value there’s no problem06:35
tchaypoexcept that we need to manually re-parse later06:35
clarkbI dont disagree06:35
lifelesstchaypo: install_requires = \n argparse>1.0;python_version=="2.6"06:35
clarkbI am just saying that oslo wrnt in a very different direction to start06:35
tchaypo… oslo has done work on this already?06:35
lifelessclarkb: oslo.config?06:36
clarkblifeless oslo in general, the $foothing-requirements.txt06:36
*** otter768 has joined #openstack-infra06:36
StevenKlifeless: Not trying to distract you, but you say your unpolished resolver "chews away at the problem" -- is it actually slow at calculating everything?06:36
lifelessclarkb: AFAIK thats all pbr06:36
lifelessStevenK: 60 seconds06:36
clarkbthey even wrote a change to pbr that was rejected to better support ^ as a general standin for extrad06:36
clarkblifeless oslo was using it for oslo06:36
StevenKLonger than I thought, yikes.06:36
lifelessclarkb: yes, oslo designed that in isolation06:37
clarkbI know06:37
lifelessclarkb: I'm aware of it, and meeting their needs for extras is why we're doing this...06:37
*** soren_ has joined #openstack-infra06:37
lifelessclarkb: I'm not sure what your point is.06:37
clarkbmy point is that we assume install reqs should go in setup.cfg but only because we are putting extras there and that isnt required06:38
clarkbits merely how we have chosen to solve this06:38
lifelessif you look at the backscroll06:39
lifelesswe're not assuming that06:39
lifelesswe were talking around different tradeoffs just before you joined06:39
lifelessno, where were we06:39
lifelesstchaypo: so - let me put a sketch together06:40
*** shardy has joined #openstack-infra06:40
*** otter768 has quit IRC06:41
*** shashankhegde has quit IRC06:41
*** Ala has joined #openstack-infra06:43
*** yamamoto_ has joined #openstack-infra06:45
lifelessmgedmin: ^06:46
*** ddieterly has joined #openstack-infra06:47
mgedminthis is nicer than the original suggestion06:47
mgedminone thing to check: doesn't ConfigParser intepret ; as comments and strip everything from ; until EOL?06:48
mgedmin(hopefully not)06:48
tchaypoi tested that06:48
*** yamamoto has quit IRC06:48
tchaypomgedmin: that last string there isn’t useful, but it at least shows configparser didn’t treat things after the ; as a comment06:49
tchaypolifeless: we’re going to have to de-dupe keys when building that dict06:51
lifelesstchaypo: union06:51
lifelesstchaypo: yes06:51
lifelesstchaypo: we need to do that for the install_requires marker handling in requirements.txt too06:51
lifelesstchaypo: we might want to do that patch as the first in the series06:51
mgedmintchaypo, I really hate to contradict you ...06:51
*** ddieterly has quit IRC06:52
*** mpopow has joined #openstack-infra06:52
lifelessmgedmin: does : work ?06:52
*** mpaolino has joined #openstack-infra06:52
mgedmin: more or less the same as = to configparser, i.e. you can have key: value instead of key = value; additional : inside value aren't special06:52
mgedminapparently ; is valid inside a key but is treated as a comment inside the value06:53
lifelessmgedmin: but : should work06:54
* mgedmin tests06:55
mgedminreqs = foo : marker came through unscathed06:55
*** mpopow has quit IRC06:56
*** ildikov has joined #openstack-infra06:57
tchayponaturally I only tested with it in a key.07:00
lifelessso we map requirements to extras_require={'': requirements_txt_contents}07:00
lifelessand the transform to expandout markers from inside that, taking either : or ; to delimit07:00
lifelessthen we can add native extras support trivially07:00
*** markus_z has joined #openstack-infra07:02
*** dizquierdo has joined #openstack-infra07:04
*** sergsh has joined #openstack-infra07:05
*** luqas has joined #openstack-infra07:05
lifelesstchaypo: right, will pickup discussion tomorrow - I think we've broken the back of this. Thanks mgedmin and clarkb07:09
*** hashar has joined #openstack-infra07:09
*** luqas has quit IRC07:10
*** panda|afk has quit IRC07:13
*** ibiris is now known as ibiris_away07:14
tchaypoin a moment, my review will be updated with an example of setup.cfg that I think is what we want07:14
*** panda|afk has joined #openstack-infra07:14
*** fedexo has quit IRC07:14
*** achanda has quit IRC07:15
openstackgerritJames Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras from setup.cfg
*** jtomasek has joined #openstack-infra07:15
tchaypoI’ve given it it two test+env_marker and one plain test, just to check that we’re making the union properly, and I’ve added another example of an extra that has several requriements but no env markers to make sure we handle that too07:15
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Propose translations: Fix handling of deleted files
lifelesstchaypo: so I suggest you use made up names07:18
lifelesslike python26_only_one and python26_only_two07:18
lifelesstchaypo: (also see my review)07:18
tchaypowhich review?07:19
*** jamespage_ has joined #openstack-infra07:20
tchaypooh, your comment on my review07:21
*** ibiris_away is now known as ibiris07:22
lifelessStevenK: resolving is NP-C in the limit, so slower is expected.07:24
lifelessStevenK: its not been profiled yet07:24
StevenKlifeless: Yeah, I'm not sure what number I was expecting, but ~60 seconds surprised me.07:25
StevenKMaybe I'm spoiled rotten by apt/aptitude and their C++-ness07:25
openstackgerritJames Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras from setup.cfg
lifelessStevenK: see my tweet07:26
lifelessStevenK: I put comparison data up07:26
lifelessStevenK: they also have a much more constrained DB07:26
lifelessStevenK: and cheap local data07:26
StevenKI don't use twitter, but let's see if I can scrape it07:26
lifelessStevenK: pip has hundreds of versions of some packages07:26
StevenKHah, when pip develop already takes 2 minutes ...07:28
*** sdake__ has quit IRC07:28
*** lennyb has joined #openstack-infra07:29
*** mpavone has joined #openstack-infra07:30
*** arxcruz has joined #openstack-infra07:31
StevenKlifeless: However, that tweet doesn't show up on
*** ildikov has quit IRC07:31
*** koolhead17 has quit IRC07:31
StevenKBut clearly that page is not what is supposed to be consumed.07:31
*** spredzy|afk is now known as spredzy07:32
*** yamamoto_ has quit IRC07:33
*** soren_ has quit IRC07:33
*** soren_ has joined #openstack-infra07:33
*** luqas has joined #openstack-infra07:33
*** soren_ is now known as soren07:33
lifelessStevenK: yeah that page is a lie07:34
*** AJaeger has quit IRC07:35
*** yfried_ is now known as yfried|afk07:36
*** koolhead_ has joined #openstack-infra07:37
lifelesstchaypo: ok, so ping me when you get up tomorrow07:37
*** mahito_ has quit IRC07:38
*** imcsk8 has joined #openstack-infra07:38
*** luqas has quit IRC07:38
*** yfried|afk is now known as yfried_07:38
*** jamespage_ has quit IRC07:39
lifelesstchaypo: yes, that looks plausible07:40
*** woodster_ has quit IRC07:40
*** ihrachyshka has joined #openstack-infra07:43
*** achanda has joined #openstack-infra07:45
*** ivar-laz_ has quit IRC07:45
lennybHi, We've started to get 'ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. ' after recent Gerrit update and our zuul update. any ideas07:45
*** ivar-lazzaro has joined #openstack-infra07:46
*** mahito has joined #openstack-infra07:46
*** yfried_ has quit IRC07:46
*** jlanoux has joined #openstack-infra07:46
*** ildikov has joined #openstack-infra07:47
*** AJaeger has joined #openstack-infra07:48
*** AJaeger has joined #openstack-infra07:48
*** ddieterly has joined #openstack-infra07:48
*** rmcall has quit IRC07:48
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron
*** dizquierdo has quit IRC07:50
*** rmcall has joined #openstack-infra07:52
*** mpopow has joined #openstack-infra07:52
*** ddieterly has quit IRC07:53
mrdaHey infra, wondering if you could help me with a problem I'm seeing.  I added "appdirs>=1.4.0" to requirements.txt in python-ironicclient and now I'm getting an error that " No such file or directory".  See  Any ideas on what I'm doing wrong?07:53
mrdaOr has anyone seen anything like it in the gate before?07:54
*** yamamoto has joined #openstack-infra07:54
mrda(btw, works locally in a pypy tox job, but that's with pip 6.1.1 rather than 1.5.4 in gate)07:56
*** mpopow has quit IRC07:57
*** fifieldt has joined #openstack-infra07:57
*** fifieldt has quit IRC07:58
*** jistr has joined #openstack-infra07:58
*** achanda has quit IRC08:01
*** devvesa has joined #openstack-infra08:03
lifelessmrda: thats due to too-old setuptools in that venv08:03
lifelessmrda: it was discussed on the list08:03
lifelessmrda: they're looking for volunteers to debug and fix08:03
mrdaoh, missed that one08:03
mrdathanks for answering!08:04
lifelessmrda: and clarkb has put up a patch that may fix it (but may need some iterations)08:04
mrdalifeless: you are a very useful engine08:05
lifelessmrda: thanks, I think08:05
*** nithyag_ has quit IRC08:12
*** koolhead_ is now known as koolhead1708:13
*** koolhead17 has joined #openstack-infra08:13
lennybmrda: we solved a similar issue by removing cache in ~jenkins/.cache ...08:14
openstackgerritMerged openstack-infra/irc-meetings: Update Kolla Meeting
openstackgerritQin Zhao proposed openstack/requirements: Update sqlalchemy-migrate to 0.9.6
openstackgerrityolanda.robla proposed openstack-infra/jenkins-job-builder: Query all jobs once
*** derekh has joined #openstack-infra08:22
*** yfried has joined #openstack-infra08:24
*** Hal has joined #openstack-infra08:25
*** Hal is now known as Guest1382608:25
openstackgerrityolanda.robla proposed openstack-infra/jenkins-job-builder: Query all jobs once
*** AJaeger has quit IRC08:26
*** ibiris is now known as ibiris_away08:27
*** AJaeger has joined #openstack-infra08:28
*** AJaeger has joined #openstack-infra08:28
*** btully has quit IRC08:28
*** rossella_s has quit IRC08:28
*** fhubik has joined #openstack-infra08:28
*** ihrachyshka has quit IRC08:30
*** dguitarbite has joined #openstack-infra08:32
*** Ala has quit IRC08:34
*** MaxV_ has joined #openstack-infra08:35
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add in F21 nonha job
*** otter768 has joined #openstack-infra08:37
*** gilliard_afk is now known as gilliard08:37
*** rossella_s has joined #openstack-infra08:41
*** otter768 has quit IRC08:42
*** ajo has joined #openstack-infra08:42
*** ociuhandu has joined #openstack-infra08:44
*** yamamoto has quit IRC08:45
*** achanda has joined #openstack-infra08:45
*** ddieterly has joined #openstack-infra08:49
openstackgerritCyril Roelandt proposed openstack-infra/project-config: Enable a non-voting py34 check for Neutron
*** fawadkhaliq has joined #openstack-infra08:52
*** ddieterly has quit IRC08:53
*** Ala has joined #openstack-infra09:00
openstackgerrityolanda.robla proposed openstack-infra/system-config: Start using cgit::lb class
*** ociuhandu has quit IRC09:03
*** dizquierdo has joined #openstack-infra09:03
openstackgerritStuart McLaren proposed openstack/requirements: Bump eventlet version to 0.17.4
*** ssam2 has joined #openstack-infra09:05
openstackgerrityolanda.robla proposed openstack-infra/system-config: Start using puppet-cgit to manage gitrc
*** hichihara has quit IRC09:09
*** mrunge has quit IRC09:09
openstackgerrityolanda.robla proposed openstack-infra/system-config: Start using puppet-cgit to manage gitrc
*** samuelBartel has joined #openstack-infra09:13
*** mrunge has joined #openstack-infra09:16
*** koolhead17 has quit IRC09:22
openstackgerrityolanda.robla proposed openstack-infra/system-config: Upgrade selinux version to 0.5.3
*** koolhead17 has joined #openstack-infra09:22
*** koolhead17 has quit IRC09:22
*** koolhead17 has joined #openstack-infra09:22
*** yamamoto has joined #openstack-infra09:24
*** sergsh is now known as sshnaidm09:25
*** pblaho has joined #openstack-infra09:25
*** derekh has quit IRC09:27
*** ildikov has quit IRC09:27
*** koolhead17 has quit IRC09:27
*** derekh has joined #openstack-infra09:28
openstackgerritMerged openstack-infra/storyboard: Email Outbox
openstackgerritMerged openstack-infra/storyboard: Email base plugin.
*** cdent has joined #openstack-infra09:31
*** aswadr has joined #openstack-infra09:33
shardyHi all, we have a problem with gate-python-heatclient-pypy, it's failing with "error: option --single-version-externally-managed not recognized", appears to be distutils related09:34
shardyI was wondering if anyone knows of any openstack-wide issues similar to that, or how we may fix it09:34
shardyIt looks a lot like bug #1290562, but that was closed many months ago09:35
openstackbug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] - Assigned to Ildiko Vancsa (ildiko-vancsa)09:35
shardyI raised bug #1453095 against heatclient so I can track figuring out what's causing the failure09:35
openstackbug 1453095 in python-heatclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,New]
*** achanda has quit IRC09:40
*** mpopow has joined #openstack-infra09:41
*** ZZelle has joined #openstack-infra09:42
*** xnox has quit IRC09:42
*** ildikov has joined #openstack-infra09:43
openstackgerritMerged openstack-infra/storyboard-webclient: Update to search UI.
openstackgerritMerged openstack-infra/storyboard-webclient: Renamed result-set-size directive
openstackgerritMerged openstack-infra/storyboard-webclient: Result set paging update.
*** mpopow has quit IRC09:46
openstackgerritMerged openstack-infra/storyboard-webclient: Story Search Results now show loading spinner.
*** zhiwei has quit IRC09:48
*** xnox has joined #openstack-infra09:48
*** pdangur has joined #openstack-infra09:48
openstackgerritMerged openstack-infra/storyboard-webclient: Initial proposal for integration tests
*** ddieterly has joined #openstack-infra09:49
pdangurhello, i need some help with jjb: i'd like to delete jobs that were removed in the YAML conf09:50
pdangurcan anyone please help?09:50
openstackgerritMerged openstack-infra/storyboard: Email Send Cron Worker
openstackgerritMerged openstack-infra/storyboard: Redirect Browsers from API to Client
openstackgerritMerged openstack-infra/storyboard: Added email preferences.
*** rlandy has joined #openstack-infra09:53
*** ddieterly has quit IRC09:53
AJaegerpdangur: you want to remove jobs from jenkins?09:54
AJaegerjjb does not handle that09:54
openstackgerritMerged openstack-infra/storyboard: Added worker resource resolution helper.
openstackgerritMerged openstack-infra/storyboard: Updated oslo.config to 1.11.0
pdangurAJaeger: i know. i'd like to come up with a rather quick way to examine changes to infer which jobs to remove09:54
pdangurAJaeger: is there a way to get the list of jobs configured in yaml?09:55
pdangurAJaeger: (not from jenkins)09:55
AJaegerpdangur: sure, have a look at how we gate project-config.09:56
AJaegerfor example these tests:
*** fawadkhaliq has quit IRC09:56
AJaegeror - both generate the list of current jobs09:57
*** ihrachyshka has joined #openstack-infra10:00
pdangurAJaeger: not sure what i'm seeing here10:00
*** Guest13826 has quit IRC10:01
AJaegerpdangur: invocations that show you how to get the list of jobs10:01
pdangurAJaeger: is there a summary of the current job somewhere in this console?10:01
AJaegerpdangur: no10:01
AJaegerpdangur: you can run these locally using tox on project-config repo10:01
*** dimsum__ has joined #openstack-infra10:02
therveSo pypy jobs are failing accross the board, we've noticed 2 different errors10:04
therveThey started happening on thursday10:04
therveAny idea what might be going on?10:04
*** shardy_ has joined #openstack-infra10:04
*** shardy has quit IRC10:06
*** shardy_ has quit IRC10:10
pdangurAJaeger: can you please point me to the actual invocation? i can't find it10:10
*** shardy has joined #openstack-infra10:10
*** MaxV_ has quit IRC10:12
AJaegerpdangur: What exactly do you mean? cd openstack-infra/project-config;tox -e jjb will show you the jobs10:12
*** pc_m has joined #openstack-infra10:13
openstackgerritJulien Danjou proposed openstack-infra/git-review: Allow to specify which commit to send
*** Ala has quit IRC10:15
*** NithyaG has joined #openstack-infra10:20
*** dguitarbite has quit IRC10:24
*** dguitarbite has joined #openstack-infra10:27
*** pelix has joined #openstack-infra10:30
*** fawadkhaliq has joined #openstack-infra10:31
*** samueldmq has joined #openstack-infra10:33
*** mahito has quit IRC10:36
samueldmqhmm, now gerrit defines a size metric for reviews10:36
*** mahito has joined #openstack-infra10:36
samueldmqwhat are the thresholds ?10:36
*** baoli has joined #openstack-infra10:37
*** otter768 has joined #openstack-infra10:38
*** baoli has quit IRC10:41
*** moshele has joined #openstack-infra10:41
*** otter768 has quit IRC10:43
*** mahito has quit IRC10:44
moshelehi, how can I create branch in my stackforge repository? I tried to follow but is seem I don't have admin in the gerrit10:45
*** ams__ has quit IRC10:47
*** coolsvap is now known as coolsvap|afk10:48
*** ddieterly has joined #openstack-infra10:50
*** ddieterly has quit IRC10:55
openstackgerritMerged openstack-dev/pbr: Advertise support for Python3.4
*** tnovacik has quit IRC10:57
AJaegermoshele: which repo is this?10:58
mosheleAJaeger: networking-mlnx10:58
AJaegermoshele: you're right, there's no permission at,access10:59
*** ildikov has quit IRC10:59
mosheleAJaeger: ok, can you add me?11:00
*** _nadya_ has joined #openstack-infra11:00
AJaegermoshele: feel free to review my patch and send a patch for the project-config repo to update your repository11:00
mosheleAJaeger: ok thanks11:00
AJaegermoshele: and then create permission is for the release group - so talk with Nurit Vilosny (unless that's you ;)11:01
mosheleAJaeger: ok thanks11:01
*** ibiris_away is now known as ibiris11:03
mosheleAJaeger: also can you change that I will be ptl of networking-mlnx instead of Nurit Vilosny11:06
AJaegermoshele: Nurit can add you and remove himself11:07
mosheleAJaeger: ok thanks11:08
AJaegermoshele: just tell her/him to use the URL above and click on the group11:08
mosheleAJaeger: ok11:08
*** dalgaaf has joined #openstack-infra11:08
AJaegerand then add you11:09
*** panda|afk has quit IRC11:13
*** panda|afk has joined #openstack-infra11:14
*** ildikov has joined #openstack-infra11:15
*** mpaolino has quit IRC11:15
*** _nadya_ has quit IRC11:17
*** doug-fish has joined #openstack-infra11:18
*** fhubik has quit IRC11:19
*** fhubik_afk has quit IRC11:19
*** fhubik has joined #openstack-infra11:21
*** marcusvrn has joined #openstack-infra11:22
*** marcusvrn1 has joined #openstack-infra11:24
*** mrmartin has joined #openstack-infra11:24
*** fhubik is now known as fhubik_afk11:26
*** marcusvrn has quit IRC11:27
openstackgerritMoshe Levi proposed openstack-infra/project-config: Add create rule to networking-mlnx
*** EmilienM|afk is now known as EmilienM11:28
openstackgerritMerged openstack-infra/project-config: Check resolver pip on global-requirements changes
openstackgerritMerged openstack-infra/project-config: Enable a non-voting py34 check for Neutron
*** mpopow has joined #openstack-infra11:30
*** mmedvede_ has joined #openstack-infra11:30
AJaegersdague, fungi, pleia2: Could you review, please?11:31
*** jaypipes has joined #openstack-infra11:32
sdaguedoes that mean deletes are never sent?11:32
openstackgerritMerged openstack-infra/project-config: Add in F21 nonha job
sdagueoh, I think I understand now11:33
*** moshele has left #openstack-infra11:35
*** ociuhandu has joined #openstack-infra11:35
*** ldnunes has joined #openstack-infra11:36
openstackgerritMikhail Dubov proposed openstack-infra/project-config: Add Rally job to Murano
*** zul has quit IRC11:37
*** zul has joined #openstack-infra11:38
*** mpopow has quit IRC11:41
sdagueso, the pypy jobs are also exploding requirements at the moment11:45
mordredsdague: they seem to be exploding across the board11:49
*** deepakcs has quit IRC11:49
*** notnownikki has joined #openstack-infra11:49
sdagueis there a compelling reason to not just remove pypy from our tool chain?11:49
*** ddieterly has joined #openstack-infra11:51
hughsaundersHey, I'm getting a gerrit modal with the message "Intraline difference not available due to timeout." when viewing the diff for this change: Other diffs seem fine.11:52
*** dizquierdo has quit IRC11:52
*** masayukig has quit IRC11:53
openstackgerritSean Dague proposed openstack-infra/project-config: make pypy non-voting across the board
openstackgerritMerged openstack-infra/nodepool: Clarify project-id meaning for Rackspace deployment
sdaguemordred / AJaeger you guys want to fast approve that? Then we can stop blocking on pypy11:54
*** ddieterly has quit IRC11:55
mordredsdague: on it11:55
*** dkranz has joined #openstack-infra11:56
*** baoli has joined #openstack-infra11:56
jheskethsdague, mordred: looking too11:56
mordredsdague: I mean, ultimately getting pypy _working_ acrss everything has the potential to be pretty neat11:57
mordredespecially for the places where people say they want better performance11:57
mordredhowever, in general, nobody actually cares11:57
jheskethI'm a little torn here to be honest... disabling the gate when it doesn't work is quite dangerous11:57
jheskethbut it maybe true that nobody actually cares about pypy11:58
mordredyeah- in this case I'm a little more inclined to do it than in others11:58
sdaguejhesketh: so, the gate can't be a thing when I show up 2 years ago, get a test working once, and now make the community have to support it for the rest of existance11:58
sdaguethere is no pypy champion that I can tell11:58
*** claudiub has joined #openstack-infra11:58
mordredbecause I'm not sure we have a champion for pypy at the moment11:58
sdaguewhich means it should be deleted11:58
*** fhubik_afk is now known as fhubik11:58
mordredwe both said champion11:59
mordredI'm going to send a quick mailing list message11:59
sdaguemordred: there is a thread11:59
jheskethyep, I don't disagree11:59
*** masayukig has joined #openstack-infra11:59
jheskethbut that's a bigger call to make imo11:59
sdagueand I think, realistically, we should probably get ahead of everything where we feel like we don't have an usptream champion and start removing those jobs.12:00
jheskethgiven that list thread was started 2 days ago and nobody has cried out I'm more inclined to disable it12:00
jheskethmind you, it has been a weekend12:00
sdagueyeh, but honestly, no one is going to own this for real12:01
mordredI mean, I've been arguing that we should delete postgres for the same reason, I'm pretty sure I need to be consistent here :)12:01
*** gordc has joined #openstack-infra12:01
sdaguemordred: you haven't yet argued that at a global level :)12:01
sdaguebtw, I love the new size bars in gerrit ui12:02
*** dprince has joined #openstack-infra12:02
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Make all pypy jobs non-voting
*** dimsum__ has quit IRC12:03
*** woodster_ has joined #openstack-infra12:03
*** boris-42 has joined #openstack-infra12:04
sdaguejhesketh: I also think it's *far* more dangerous to not be fast about removing failing tests that are completely unrelated to code patches, because if there is a test remove penalty, which things take too long to delete when they are no longer relevant, that's a massive disincentive to creating new an interesting tests in the first place12:04
*** _nadya_ has joined #openstack-infra12:05
jheskethsdague: Yep, especially since most projects just have pypy because it was there when they set up their jobs12:06
openstackgerritMerged openstack-infra/project-config: Add stackforge/networking-6wind project
jheskethI would like to give the pypy folks more time to weigh in, but I think that's likely to hold up too much12:06
*** luqas has joined #openstack-infra12:06
jheskethperhaps the consequence here is that if there are people serious about pypy they'll need to fix the jobs and anything else that breaks while they are disabled12:07
*** BharatK has quit IRC12:07
*** mwagner_lap has quit IRC12:07
jheskethmordred, sdague: I'd like to see another infra person weigh in if possible before merging.. however if that doesn't happen shortly I'd be in favour of merging. (I'm definately not blocking if you want to merge between yourselves)12:08
*** wwwBUKOLAYcom has joined #openstack-infra12:08
openstackgerritMarton Kiss proposed openstack-infra/puppet-askbot: Askbot module refactor
sdaguejhesketh: treating failing jobs as punishment works pretty poorly12:09
sdagueuntil this merges, requirements are fronze12:09
*** ddieterly has joined #openstack-infra12:09
sdaguetons of interesting working can't proceed12:09
jheskethsdague: by shortly I meant < 1hr12:10
sdagueI'm not sure what you expect to change in the next hour, but so be it12:10
* wwwBUKOLAYcom Click I Come.12:10
* wwwBUKOLAYcom Click I Come.12:10
* wwwBUKOLAYcom Click I Come.12:11
* wwwBUKOLAYcom Click I Come.12:11
* wwwBUKOLAYcom Click I Come.12:11
* wwwBUKOLAYcom Click I Come.12:11
* wwwBUKOLAYcom Click I Come.12:11
* wwwBUKOLAYcom Click I Come.12:11
* wwwBUKOLAYcom Click I Come.12:11
* wwwBUKOLAYcom Click I Come.12:11
* wwwBUKOLAYcom Click I Come.12:11
sdagueoh hey, someone want to kickban ?12:11
jheskethwell I'm dubious too, and I'm also happy for it to merge, I'm just being somewhat cautious12:11
* jhesketh can't sorry12:11
*** weshay has joined #openstack-infra12:12
*** wwwBUKOLAYcom has quit IRC12:12
openstackgerritMerged openstack-infra/shade: Fix delete_server when wait=True
*** claudiub has quit IRC12:13
openstackgerritMarton Kiss proposed openstack-infra/puppet-askbot: Askbot module refactor
openstackgerritMerged openstack-infra/project-config: Cleanup openstack documentation Gerrit ACLs
*** _dguerri is now known as dguerri12:13
*** esker has quit IRC12:14
openstackgerritMerged openstack-infra/project-config: Update jenkins jobs for networking-ovn
openstackgerritMerged openstack-infra/project-config: Adding create permissions to networking-vsphere
*** claudiub has joined #openstack-infra12:16
*** rfolco has joined #openstack-infra12:16
*** koolhead17 has joined #openstack-infra12:16
openstackgerritMerged openstack-infra/shade: Return Bunch objects instead of plain dicts
*** fhubik has quit IRC12:18
*** fhubik has joined #openstack-infra12:18
openstackgerritMerged openstack-infra/project-config: Configure to use gem mirrors
openstackgerritMerged openstack-infra/project-config: Remove integrated-gate template from sahara
openstackgerritMerged openstack-infra/project-config: Explicitly enable all required services in rally
openstackgerritMerged openstack-infra/project-config: Stop hacking integration test from cloning across the network
openstackgerritMerged openstack-infra/project-config: Enable upload to pypi for Shaker project
mordredgreghaynes: responded to
*** sks has quit IRC12:21
*** banix has joined #openstack-infra12:22
mordredgreghaynes: and the next one in the stack12:23
openstackgerritMerged openstack-infra/devstack-gate: Return command status in tsfilter
*** TheJulia has joined #openstack-infra12:24
openstackgerritMerged openstack-infra/devstack-gate: Add errexit to setup & cleanup functions
*** _nadya_ has quit IRC12:26
*** fawadkhaliq has quit IRC12:26
*** banix has quit IRC12:31
*** yfried is now known as yfried|afk12:32
openstackgerritSean Dague proposed openstack-infra/project-config: make pypy non-voting across the board
sdaguefixed to pass layout tests, I missed the fact that there was a name clash12:33
*** Hal has joined #openstack-infra12:33
*** _nadya_ has joined #openstack-infra12:33
*** Hal is now known as Guest2137312:34
*** aysyd has joined #openstack-infra12:34
*** Ala has joined #openstack-infra12:35
sdaguegah, once more here12:36
*** sdake has joined #openstack-infra12:38
*** bswartz has quit IRC12:38
*** otter768 has joined #openstack-infra12:39
*** sdake_ has joined #openstack-infra12:40
*** mrmartin has quit IRC12:41
jheskethsdague, mordred: okay, doesn't look like others are around and I need to run12:42
jheskethsdague, mordred: you guys okay with me merging it and running? Worst case it can be undone12:42
sdagueyeh, though I need a new patch that will pass12:42
sdagueone second12:42
*** BharatK has joined #openstack-infra12:43
*** otter768 has quit IRC12:43
*** fawadkhaliq has joined #openstack-infra12:44
fungibusy night! almost 1k lines of scrollback in this channel. warning: i'm not going to read any of it aside from nick highlights12:44
*** sdake has quit IRC12:44
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Enable pacemaker via the resource registry
openstackgerritSean Dague proposed openstack-infra/project-config: make pypy non-voting across the board
sdaguefungi: yeh, so honestly, the kind of important one is probably ^^^12:46
sdaguewhich I'm still not sure I got right12:46
sdaguepypy ruins everything12:46
*** marcusvrn1 has quit IRC12:47
*** bcourt has joined #openstack-infra12:47
jheskethsdague: your latest change only disables pypy for icehouse?12:47
sdaguedoes it?12:48
sdagueI though that was 'don't run on icehouse'12:48
sdaguealso don't vote12:48
*** Longgeek_ has quit IRC12:48
sdaguebut, I could be wrong12:48
jheskethsdague: oh, I may have mis-read..12:49
*** Longgeek has joined #openstack-infra12:49
jheskethsdague: yep, you're right12:49
sdaguethe don't run on icehouse was there from a long time ago12:49
*** _nadya_ has quit IRC12:49
jheskethfungi: you normally read scrollback?12:49
*** koolhead17 has quit IRC12:49
*** marcusvrn has joined #openstack-infra12:50
*** fawadk has joined #openstack-infra12:50
*** mwagner_lap has joined #openstack-infra12:50
jheskethsdague: okay so lgtm.. I'll leave it for fungi and mordred to take from here12:50
jheskethnight all :-)12:50
*** fawadkhaliq has quit IRC12:50
fungisdague: i saw12:51
fungifun times12:51
sdaguewell, it's sort of to be expected from bit rot. Much like building a bridge, you have to occationally do some maintenance to it or it falls down.12:52
*** marcusvrn has quit IRC12:54
*** pdangur has left #openstack-infra12:54
openstackgerritTristan Cacqueray proposed openstack-infra/system-config: Use jenkins default file from puppet-jenkins
openstackgerritTristan Cacqueray proposed openstack-infra/puppet-jenkins: Adds jenkins default file
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Correct misleading comment about pypy jobs
fungisdague: approved, follow-on to correct typo ^12:55
*** xyang1 has joined #openstack-infra12:55
openstackgerritMerged openstack-infra/project-config: Propose translations: Fix handling of deleted files
*** sdake_ is now known as sdake12:55
*** mrunge has quit IRC12:56
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds pypi-jobs to cloudbase-init
fungii wouldn't normally be picky about comment text, but when it says the opposite of what it means, it can be confusing12:57
*** marcusvrn has joined #openstack-infra12:57
*** btully has joined #openstack-infra12:58
*** radez_g0n3 is now known as radez13:01
openstackgerritMonty Taylor proposed openstack-infra/shade: Straighten up keystone endpoint management
*** bknudson has quit IRC13:01
*** smarcet has joined #openstack-infra13:03
*** MaxV has joined #openstack-infra13:03
sdaguefungi: yep, good catch13:03
*** dkranz has quit IRC13:05
*** koolhead17 has joined #openstack-infra13:05
*** ddieterly has quit IRC13:06
yolandahi, any infra cores to help debug an issue with storyboard?13:06
fungiyolanda: what's broken?13:06
yolandafungi, we cannot login, it redirects to the authorize page, then redirects to home13:07
yolandaso we cannot see any error on xhr or in the html itself13:07
yolandawe'd need some logs to get a clue13:08
fungihrm, yeah i'm getting the same behavior... checking the logs now13:08
*** fawadkhaliq has joined #openstack-infra13:08
openstackgerritRussell Bryant proposed openstack/requirements: global-requirements: Add ovs
fungiyolanda: would errors end up somewhere besides the apache logs?13:10
*** fawadk has quit IRC13:11
*** amitgandhinz has joined #openstack-infra13:11
*** sigmavirus24_awa is now known as sigmavirus2413:11
openstackgerritTristan Cacqueray proposed openstack-infra/system-config: Use common jenkins master implementation
fungiit looks like!/auth/authorize is 303 redirecting back to
yolandafungi, they shoud be on apache, storyboard-error.log13:13
fungiyeah, no errors13:13
fungii'm guessing something has changed recently in the storyboard openid callout implementation or configuration13:13
yolandalet me check what landed today13:13
fungithat was going to be my next suggestion ;)13:14
yolandaa bunch of changes actually13:14
fungiyolanda: to help narrow it down, looks like 09:36:09 utc today was teh last successful login via launchpad openid13:15
fungiso you have fewer than 4 hours of potential changes breaking it13:15
yolandaall changes landed starting at 09:43utc13:15
*** Longgeek has quit IRC13:15
*** Longgeek has joined #openstack-infra13:16
mordredI saw a patch, I don't know if it landed, but something about redirecting API calls to storyboard13:16
fungithen it's almost certainly one of those13:16
yolandafungi, that looks as a potential guilty13:17
mordredttx landed that 4 hours ago13:17
*** fifieldt has joined #openstack-infra13:17
openstackgerritMonty Taylor proposed openstack-infra/storyboard: Revert "Redirect Browsers from API to Client"
*** fifieldt has quit IRC13:17
*** eharney has joined #openstack-infra13:17
yolandaok, let's start by it13:17
openstackgerritMerged openstack-infra/zuul: Status: Use $DEST_DIR instead of $BASE_DIR/public_html
openstackgerritMerged openstack-infra/project-config: make pypy non-voting across the board
openstackgerritMerged openstack-infra/project-config: Correct misleading comment about pypy jobs
*** bswartz has joined #openstack-infra13:20
fungiyolanda: it'll need a rebase to deal with conflicts on other changes touching nearby lines13:20
ttxmordred: agreed that's the most... likely suspect13:20
yolandaoh, bad... i don't have the time now, it will need to wait for a bit , unless some others come and help13:21
*** BobBall_AWOL is now known as BobBall13:21
fungii can do it13:21
sdaguefungi: so... that's a thing13:21
fungiunless mordred wants to. it's his patch13:21
yolandamany thx, fungi13:21
sdaguezuul retroactively applies voting rules?13:21
sdagueI don't think I've ever seen that before13:22
mordreduhm. I guess I'll do it13:22
*** bknudson has joined #openstack-infra13:22
*** chlong has joined #openstack-infra13:22
*** cinerama has quit IRC13:22
*** dkranz has joined #openstack-infra13:22
fungisdague: i believe those are evaluated real-time/on reporting back to the change. not positive13:22
sdaguewe landed the non voting change, zuul applied the new rules to the existing gate, it reset back into mostly green state13:23
sdagueyeh, I guess I shouldn't be totally surprised, I'd just never seen this edge case before13:23
openstackgerritMonty Taylor proposed openstack-infra/storyboard: Revert "Redirect Browsers from API to Client"
fungii can't say i've paid close enough attention to notice it before myself13:23
mordredthat's neat13:24
*** craigbr has joined #openstack-infra13:24
*** dustins has joined #openstack-infra13:24
*** tiswanso has joined #openstack-infra13:25
fungihuh... new in gerrit 2.10: ls-projects lists an "All-Users" project which can't be cloned13:26
*** cinerama has joined #openstack-infra13:26
*** ibiris is now known as ibiris_away13:27
sdaguelabel:Code-Review<=-1,group=xxxxxxxxxxx still doesn't work13:27
sdaguewhich makes me sad13:27
*** dizquierdo has joined #openstack-infra13:28
fungigitweb lists it too, but you can't browse it13:28
mordredwow. that's a fantastic degredation of functionality13:28
mordredso now we have to put a grep -v in everywhere we use ls-projects13:28
*** fawadkhaliq has quit IRC13:29
mordredzaro: ^^ I would contend that this is a bug13:29
*** peristeri has joined #openstack-infra13:29
fungiyep. in fact i noticed it because my daily remote update and clone new repos cron job on my workstation spat an error about it to my inbox13:29
fungii agree, you would think that since ls-projects doesn't list the All-Projects pseudoproject, it shouldn't list All-Users either13:30
*** baoli has quit IRC13:30
*** isviridov_away is now known as isviridov13:31
*** baoli_ has joined #openstack-infra13:31
*** jgrimm is now known as zz_jgrimm13:32
sdaguefungi: if you get to a slow point, I'd be kind of curious to figure out if there are errors being thrown in the gerrit logs on the group queries13:32
*** esker has joined #openstack-infra13:32
*** koolhead17 has quit IRC13:33
*** amitgandhinz has quit IRC13:33
*** koolhead17 has joined #openstack-infra13:33
*** koolhead17 has quit IRC13:33
*** koolhead17 has joined #openstack-infra13:33
*** koolhead17 has quit IRC13:34
*** koolhead17 has joined #openstack-infra13:34
*** koolhead17 has quit IRC13:34
*** koolhead17 has joined #openstack-infra13:34
*** amitgandhinz has joined #openstack-infra13:34
*** isviridov is now known as isviridov_away13:34
*** koolhead17 has quit IRC13:35
*** ddieterly has joined #openstack-infra13:36
*** mrmartin has joined #openstack-infra13:36
*** craigbr has quit IRC13:37
*** mpaolino has joined #openstack-infra13:38
*** yamamoto has quit IRC13:39
*** yamamoto has joined #openstack-infra13:39
openstackgerritMerged openstack-infra/zuul: Status: Move external dependencies into public_html/lib
*** dimsum__ has joined #openstack-infra13:39
*** prad has joined #openstack-infra13:39
sdakeis 30 minutes enough for the centos puppet apply job?
sdakeseems to be timing out13:40
*** teran has quit IRC13:40
openstackgerritMerged openstack-infra/storyboard: Revert "Redirect Browsers from API to Client"
*** ivar-lazzaro has quit IRC13:41
*** pabelanger has joined #openstack-infra13:41
*** ivar-lazzaro has joined #openstack-infra13:41
sdakealso this error looks suspicious:
fungisdake: centos 6.x git is broken. ever since the "ghost" vulnerability update we've seen random hangs of remote git operations on our bare-centos6 workers13:42
sdakefungi thanks, so recheck is  the solution for the moment then?13:42
fungii've so far been unable to reproduce it on demand, so haven't managed to get it to happen under a debugger13:42
*** mriedem has joined #openstack-infra13:42
*** julim has joined #openstack-infra13:42
openstackgerritMerged openstack/requirements: add analayzer
fungisdake: yes, and soon probably some timeout+retry logic in zuul-cloner (and somewhat longer term, stop caring about centos 6)13:43
*** emagana has quit IRC13:43
sdakeya, I was going to suggest just moving to centos713:43
sdakeits pretty stable13:43
fungisdake: i hear our python 2.6 jobs would run great on centos 713:44
sdakelol ;)13:44
fungi(which is the only thing we keep centos 6 workers around for)13:44
sdakei see13:44
fungias soon as we can drop python 2.6 testing, we will be dropping centos 6 like a hot potato13:44
*** yamamoto has quit IRC13:44
*** craigbr has joined #openstack-infra13:44
*** yfried|afk has quit IRC13:44
*** yfried|afk has joined #openstack-infra13:45
*** MaxV has quit IRC13:47
*** prad has quit IRC13:47
*** Ala has quit IRC13:48
*** prad has joined #openstack-infra13:48
openstackgerritMerged openstack-infra/project-config: Add python-cloudpulseclient project for CloudPulse
*** dizquierdo has quit IRC13:52
*** dimsum__ has quit IRC13:53
*** mattfarina has joined #openstack-infra13:54
openstackgerritMerged openstack/requirements: Bump futures up to 3.0 or greater
*** markvoelker has quit IRC13:56
openstackgerritMerged openstack/requirements: Add Bandit as a global test requirement
*** kgiusti has joined #openstack-infra13:57
*** Mmike has quit IRC13:57
sdaguezaro: is there an updated set of instructions somewhere to get a working gerrit eclipse environment? I'd like to try to solve this groups thing as this would really help our review flow13:58
openstackgerritMarton Kiss proposed openstack-infra/system-config: Disable puppet-askbot resources temporarily
openstackgerritMerged openstack-infra/system-config: Enable logging of #openstack-defcore
sdaguethe existing instructions seems to die for me after running buck on the failure to find watchman13:58
*** Mmike has joined #openstack-infra13:58
*** Mmike has joined #openstack-infra13:58
fungisdague: how does eclipse fit into the picture?13:58
sdaguefungi: because navigating large sets of java code without eclipse is... more challenge then I'm looking for13:59
yolanda<yolanda> fungi, how much time shall puppet take?13:59
yolanda<yolanda> it's still failing13:59
fungiyolanda: up to an hour13:59
fungiprobably between 30-45 minutes after the change merges though13:59
*** Longgeek has quit IRC14:00
yolandaok, let's hope is that14:00
*** Longgeek has joined #openstack-infra14:01
fungii'm testing workarounds for the openid callback double-slash problem in new gerrit, and not having any luck. i had hoped to convince apache to rewrite double-slashes to single slashes, but it seems not to do so14:01
*** stevemar has joined #openstack-infra14:01
*** ibiris_away is now known as ibiris14:01
*** wznoinsk has joined #openstack-infra14:02
*** smccully has quit IRC14:02
yolandamordred, i'm supper happy with your review speed for my changes , it's great :)14:02
*** smccully has joined #openstack-infra14:02
*** yamamoto has joined #openstack-infra14:02
*** Longgeek_ has joined #openstack-infra14:03
sdaguefungi: so... gerrit 2.10 means we can remove the merge check facility in zuul, right?14:03
sdagueas gerrit computes it directly itself14:04
*** Longgeek has quit IRC14:04
fungisdague: in theory, yes. in practice we need a little time to confirm it's working as advertised14:04
sdagueok, cool14:04
*** ibiris is now known as ibiris_away14:05
*** ibiris_away is now known as ibiris14:06
*** ssam2 has quit IRC14:06
*** openstackgerrit has quit IRC14:06
mordredfungi: we may have lost gerritbot - I just pushed two patches and no notification14:06
*** Guest21373 has quit IRC14:07
*** openstackgerrit has joined #openstack-infra14:07
*** craigbr has quit IRC14:07
mordredoh. nm. he just left and rejoined14:07
sdague - fwiw, that's the list of things gerrit now doesn't count as mergable that Jenkins does. Going the other way will be a little hard to auto compute14:07
mordredoh! is:mergable is a thing14:07
sdagueit is now14:08
mordredI was looking for it over teh weekend14:08
sdagueand the Status column in gerrit now displays Merge Conflict as a status14:08
mordredyah. well, that I knew14:09
mordredI just couldn't find the flag to search for it over teh weekend14:09
*** rossella_s has quit IRC14:09
mordrednot sure why14:09
*** mpaolino has quit IRC14:09
*** rossella_s has joined #openstack-infra14:09
openstackgerritMarton Kiss proposed openstack-infra/system-config: Disable puppet-askbot resources temporarily
sdagueI always just go and read - once the new release is out there14:10
openstackgerritMerged openstack/requirements: Require flake8 2.4.0
sdaguethe other nice add is size:<5014:10
sdagueso less than 50 lines of change14:10
*** Hal has joined #openstack-infra14:10
*** yamamoto has quit IRC14:11
*** Hal is now known as Guest2831114:11
sdagueand... the secondary indexes for full text search on message seem to actually work now14:11
*** dboik has joined #openstack-infra14:11
*** Ala has joined #openstack-infra14:11
*** hashar has quit IRC14:11
openstackgerritMarton Kiss proposed openstack-infra/puppet-askbot: Askbot module refactor
*** zz_dimtruck is now known as dimtruck14:11
fungioh, that's marvellous14:12
mordrednibalizer, EmilienM: I reworked EmilienM's patch that nibalizer had ameneded, basically reverting the patch back to PS9 and then maing nibalizer's changes a new patchset14:12
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Add kilo compat jobs to tempest
mordredthe combined patch was too big for me to grok what was going on - just to let you know why I did that14:13
sdaguebut I still really want reviews by group filter14:13
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements
sdaguewhich the documentation says works, but the fact that all the examples reference ldap means I bet they only have tested the ldap case14:14
EmilienMmordred: ack14:15
*** sdake_ has joined #openstack-infra14:15
*** dims has joined #openstack-infra14:16
openstackgerritMonty Taylor proposed openstack-infra/system-config: Rework mirrors to be more general
openstackgerritMonty Taylor proposed openstack-infra/system-config: Create rubygems mirror from
mordredhrm. ok. there we go14:17
*** openstack has joined #openstack-infra14:20
openstackgerritMerged openstack/requirements: Add cachetools requirement
*** ddieterly has joined #openstack-infra14:21
openstackgerritMerged openstack/requirements: Add ironic-discoverd
*** mpaolino has joined #openstack-infra14:25
*** bnemec has joined #openstack-infra14:25
*** kmartin has quit IRC14:26
yolandamordred, fungi, SB is working, many thanks14:27
*** wenlock has joined #openstack-infra14:28
* krotscheck likes the new gerrit14:29
*** miqui has joined #openstack-infra14:30
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use jenkins default file from puppet-jenkins
*** ddieterly has quit IRC14:31
*** sdake has joined #openstack-infra14:32
mordredkrotscheck: ++14:32
krotscheckI have to say though, there's now four columns I have to read to figure out the status of a patch.14:33
mordredkrotscheck: you could fix that with a fifth column which summarizes most of the info of the other columns14:33
mriedemwas there a specific bug that fixed this?
krotscheckmordred: You'd need a sixth to explain the rationale.14:34
mordredkrotscheck: or maybe an animation that bounces and the frequency of the boucing combined with the shade of the fill color would tell you the most important column to look at14:34
krotscheckmordred: It'd need to blink.14:35
*** mtanino_away has quit IRC14:35
mordredI can think of no reasons to not implement that14:35
notnownikkihave you seen the new change screen that cyanogenmod are using on their gerrit?14:35
* krotscheck runs off to StoryBoard to file a strory.14:35
*** mtanino has joined #openstack-infra14:35
mordrednotnownikki: nope! I shall go look14:35
*** sdake_ has quit IRC14:35
notnownikkiit's a bit rearranged and has the size graphs by each file14:36
notnownikkia little jarring to look at, at first14:36
mordrednotnownikki: you can have that change on our gerrit too14:36
mordrednotnownikki: go to your user preferences and select "new screen" (we have it turned off by default)14:36
notnownikkiah, i see it now :)14:37
*** prad has quit IRC14:37
*** zz_jgrimm is now known as jgrimm14:37
fungiyeah, i'm not a fan. then again, gertty is way better than either the old or new webui14:37
*** prad has joined #openstack-infra14:37
*** deepakcs has joined #openstack-infra14:38
*** mestery has quit IRC14:38
*** ddieterly has joined #openstack-infra14:38
*** otter768 has joined #openstack-infra14:40
*** annegentle has joined #openstack-infra14:40
*** emagana has joined #openstack-infra14:42
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for pypy job bug 1290562
openstackbug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] - Assigned to Ildiko Vancsa (ildiko-vancsa)14:42
*** anteaya has joined #openstack-infra14:43
*** koolhead17 has joined #openstack-infra14:43
*** otter768 has quit IRC14:44
*** amotoki_ has joined #openstack-infra14:45
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for pypy job bug 1290562
openstackbug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] - Assigned to Ildiko Vancsa (ildiko-vancsa)14:46
*** hemnafk is now known as hemna14:51
*** teran has joined #openstack-infra14:51
*** banix has joined #openstack-infra14:52
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for pypy job bug 1290562
openstackbug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] - Assigned to Ildiko Vancsa (ildiko-vancsa)14:52
*** teran has quit IRC14:55
*** asselin_ has joined #openstack-infra14:55
*** koolhead17 has quit IRC14:56
*** sarob has joined #openstack-infra14:58
*** koolhead_ has joined #openstack-infra15:00
krotscheckStoryBoard Meeting in #openstack-meeting-315:00
*** mpopow has joined #openstack-infra15:01
*** dkranz has quit IRC15:02
*** tiswanso has quit IRC15:02
*** markvoelker has joined #openstack-infra15:03
openstackgerritMerged openstack-infra/elastic-recheck: Add query for pypy job bug 1290562
openstackbug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] - Assigned to Ildiko Vancsa (ildiko-vancsa)15:04
*** sabeen1 has joined #openstack-infra15:04
fungiugh. i consider myself relatively adept at apache redirects and rewrites, but cannot for the life of me get a rewrite of to working15:05
*** dboik has quit IRC15:05
anteayafungi: do you know if jeblair has posted the infra schedule to the main schedule yet?15:06
mordredfungi: I'm going to blame systemd15:06
anteayathe third party folks are asking15:06
mordredanteaya: I believe so - I have subscribed to infra events15:06
anteayamordred: you are the one behind the colourful logs15:06
mordredanteaya: yes I am15:06
anteayamordred: have you a link, so far I can't find a url for it15:06
anteayathank you15:07
jeblairfungi: maybe apache/mod_rewrite is too smart and is normalizing that for you before processing?15:08
fungijeblair: that's all i can figure15:08
*** tim_o has joined #openstack-infra15:09
fungianteaya: btw don't direct normal conferencegoers to that url. the one omits the design summit sessions so that they don't go wandering in unless they know they need to be there15:10
anteayafungi: yes thank you15:11
*** nelsnelson has joined #openstack-infra15:11
openstackgerritRussell Bryant proposed openstack-infra/project-config: networking-ovn: disable n-novnc
anteayathis is a part of the third party meeting (now in progress in -meeting)15:11
anteayato direct and support those operators willing to learn to contribute and help the effort15:11
*** dims has quit IRC15:12
fungisounds great15:12
mordredfungi: whoever I should thank should be thanked for adding the conference sessions to teh design summit sched.org15:12
mordredfungi: I actually can deal with my schedule non-insanely for the first time EVER15:13
fungimordred: ttx and/or lsell i think15:13
mordredttx: thank you15:13
nibalizermordred: okay15:13
*** panda|afk has quit IRC15:13
mordredalso, tell lsell I'd thank her but she's not in here15:13
anteayafungi: thanks15:13
mordrednibalizer: the patch looks good - it was just too much at once for my reptile brain15:13
*** aswadr has quit IRC15:14
*** yamahata has joined #openstack-infra15:14
*** panda|afk has joined #openstack-infra15:14
ttxmordred: that was a very selfish move15:15
annegentleI'm getting a 503 for and also says "it looks down from here" -- is this a known issue?15:15
ttxbut I guess my problem was also the others problem15:15
mordredannegentle: it was up for me a second ago ... it seems unhealthy this instant15:15
annegentlettx: definitely a good move15:15
mordredttx: yah. it's too hard of a schedule merge15:16
annegentlettx: call it altruistic, it's ok15:16
mordredI mean, I'm still 95% in the summit - but the 3 things I'm doing in the conference show up well15:16
annegentlemordred: ok thanks for looking15:16
sdagueannegentle: yeh, seems down atm15:16
*** baoli_ has quit IRC15:16
fungithere must be a conference going on15:16
annegentlemordred: ttx: sdague: next step to notify <insert name>?15:16
fungior they're testing out their auto-crashing feature in preparation for next monday15:17
sdagueannegentle: no idea, it's a 3rd party service15:17
ttxannegentle: I'm on it, although it's just sched15:17
*** hashar has joined #openstack-infra15:17
annegentlettx: yeah sure it's out of our hands directly (sched being sched)15:17
sdagueoh, it's back15:17
*** unicell has joined #openstack-infra15:18
ttxoh, and it's fixed too. Cool15:18
sdaguealso, reseting my password15:20
*** dannywilson has joined #openstack-infra15:20
*** mtanino has quit IRC15:20
*** unicell1 has quit IRC15:20
sdaguethe captchas are so scrambled at this point it was 10 captchas before I found one I could do15:20
nibalizermordred: you're not the only one rewriting cloud init :P15:20
openstackgerritJaume Devesa proposed openstack-infra/project-config: Define MidoNet v2 devstack-gate jobs
*** bhunter71 has joined #openstack-infra15:20
nibalizermordred: ya good feedback on size i've been making epic code bombs so I need to get better at breaking it up15:21
*** Krinkle|detached is now known as Krinkle15:21
mordrednibalizer: I have the same tendency myself15:21
mordrednibalizer: yeah - I'd show them my version that's in rust - but somethign tells me they're obsessed with go :)15:22
*** fifieldt has joined #openstack-infra15:22
*** fifieldt has quit IRC15:23
annegentlewhat's rust good for?15:23
*** sabeen3 has joined #openstack-infra15:23
annegentle(not trolling)15:23
mordredannegentle: same things as go15:23
annegentleah ok15:23
fungiwriting software15:23
mordredannegentle: it statically compiles, is designed for high-performance high-concurrency15:23
* fungi _was_ sort of trolling15:24
morganfainbergannegentle: rust is pretty cool.15:24
mordredannegentle: main difference being that go is written by google and rust is written by mozilla15:24
clarkbmordred I think they forget reseting host keys too :) we are all doomed to repeat the same mistakes15:24
mordredand rust reminds me more of python/ruby while go reminds me more of java15:24
* morganfainberg likes some of the rust-isms more than go-isms personally.15:24
*** sabeen1 has quit IRC15:24
mordredI _really_ like the matcher stuff they borroed from haskell15:24
mordredannegentle: the mozilla folks seem to have taken the best features from all the langauges people love and made a nice language15:25
fungiand like most languages, it contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of common lisp ;)15:26
*** bhunter71 has quit IRC15:26
openstackgerritDavide Guerri proposed openstack-infra/shade: Rewrite extension checking methods
*** harlowja_at_home has joined #openstack-infra15:27
*** bhunter71 has joined #openstack-infra15:28
openstackgerritMarton Kiss proposed openstack-infra/system-config: Initial commit of ask-staging.o.o site
*** sputnik13 has quit IRC15:29
*** sks has joined #openstack-infra15:29
clarkbdguerri: question, why would checking if extensions exist be a private method?15:30
mordredclarkb: because it's an implementation detail15:31
*** mtanino has joined #openstack-infra15:31
mordredit's not related to managing a resource ina  cloud15:31
clarkbyes it is...15:31
mordredand honestly, that anyone has to ever call it is a bug in openstack15:31
dguerriyup, thanks mordred15:31
clarkbif that resource is part of an extension15:31
clarkbeg floating IPs15:31
mordredtotally disagree15:31
mordredclouds have some features and not others15:32
mordredwhether they are implemented as api extensions or not is immaterial15:32
mordredeither a cloud can perform a function or it can't15:32
clarkbsure so that I agree with, except that openstack has decided to have these things15:32
clarkband its part of openstack service apis15:32
*** ivar-lazzaro has quit IRC15:32
mordredright. but openstack has no UI15:32
mordredit's terrible15:32
mordredI'm not going to write a library that exposes the mistakes of openstack's insanity15:33
mordredI'm going to write a library that works15:33
*** ivar-lazzaro has joined #openstack-infra15:33
*** armax has joined #openstack-infra15:33
clarkbI don't think you are exposing a mistake, you are exposing info to the user15:33
*** hdd has joined #openstack-infra15:33
*** Ala has quit IRC15:33
mordredI disagree that the user should ever need to know that15:33
mordredthe user wants a floating ip15:33
mordredwe should give them one15:33
clarkb99% of the time the happily don't need that info, but the first time a shade user needs that info and they find its hidden away they will decide shade is just as broken as openstack15:33
mordredwhether we need to talk to an api extension to do so is who cares15:33
mordredclarkb: that's fine15:33
mordredthey can use something else15:33
clarkb(I mostly bring this up because you shouldn't change the publicit of a method like that fwiw)15:34
mordredyah. I agree15:34
clarkbmordred: or did we not tag that release yet?15:34
clarkbmordred: you said you were tagging a thing last week I thought15:34
*** scheuran has quit IRC15:34
mordredyup. I did15:34
mordrednone of our known consumers use that method15:34
mordredunknown consumers are using an 0.x and should know what they are getting15:35
*** koolhead_ has quit IRC15:35
*** tmcpeak has joined #openstack-infra15:35
*** tiswanso has joined #openstack-infra15:35
*** dboik has joined #openstack-infra15:35
*** ssam2 has quit IRC15:35
tmcpeakhey all - are there any provisions/processes for pruning global-requirements?15:35
mordredtmcpeak: with gusto!15:35
*** wenlock has quit IRC15:36
tmcpeakmordred: so we're fairly sure all of the things on there are used?15:36
clarkbmordred: ok, /me notes for the record that we are basically perpetuating problems that the clients have with shade by making changes like this.15:36
*** annegentle has quit IRC15:37
mordredclarkb: I'm not sure I follow you?15:37
*** markus_z has quit IRC15:37
clarkbmordred: the api changes, lack of documentation of the change/deprecation, and "no you can't do this because silly user"15:37
clarkbmordred: its basically everything I hate about using eg novaclient15:37
*** koolhead17 has joined #openstack-infra15:37
openstackgerritRussell Bryant proposed openstack-infra/reviewstats: Fix compat with gerrit 2.9
*** annegentle has joined #openstack-infra15:38
*** wenlock has joined #openstack-infra15:38
mordredclarkb: sure. right now I'm trying to get to a supportable 1.0 API contract as quickly as possible15:38
mordredclarkb: so that we can actually support everything with deprecations15:38
dguerriclarkb who is relying on a 0.x version?15:38
mordredclarkb: a few things have snuck in and we're trying to trim them out before we cut the 1.0 - which I'd like to do soon honestsly15:39
clarkbdguerri: you can't know, and thats sort of why thinking about this is important15:39
mordredbecause I agree with you15:39
dguerrithat method was not even documented clarkb15:39
*** yamamoto has joined #openstack-infra15:39
mordredcurrently there are 2 consumers I am willing to support- nodepool and ansible - outside of those I'll support everyone esle as soon as we cut 1.015:39
sdaguefungi: fyi, resume_sortkey was removed for gerrit cli api15:39
clarkbdguerri: it doesn't matter, it was part of the public api and now it is gone, anyone that has used it few they may be will break on your next release15:40
*** Swami has joined #openstack-infra15:40
*** soren has quit IRC15:40
sdagueso everywhere that was used we'll need to move to --start n15:40
dguerridefine part of public api please?15:40
sdaguelike -
mordreddstufft: no leading _15:40
clarkbdguerri: it was a public method has_extensions()15:40
mordreddguerri: no leading _15:40
*** gyee has joined #openstack-infra15:41
anteayaclarkb: can you join us in -meeting, have you seen the just posted gerrit error before?15:41
anteayaI have not15:41
mordredclarkb: so, yes - I agree with you - that breaks the public interface. but I quite explicitly do not care until the version is not 0.x - else 0.x does not mean anything either. "0" as a major number means "I make zero assurance about interface"15:41
mordredbut - I also agree with you that this is not friendly, thus want to make a 1.x as soon as possible15:42
dguerriThat's mean it is a public method. I don't know guys, to me a public API should documented.15:42
clarkbI wouldn't be so cranky about this if it wasn't shade15:42
*** dboik_ has joined #openstack-infra15:42
clarkbbut shade specifically exists to make life easier for users and these are specific problems users have with the existing clients15:43
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Have gate-ironic-python34 be a voting job
dguerrican be called != should be called15:43
clarkbdguerri: sorry, that isn't how python works15:43
dguerriclarkb ^ that's how humans work15:43
dguerriI can jump out of the window right now clarkb15:44
dguerribut I'd rather not15:44
*** dboik has quit IRC15:44
*** mpaolino has quit IRC15:44
mordreddguerri: sure. but I think the rule of thumb needs to be that if the method is not marked with a _ we need to assume it's being used even if not documented15:45
clarkbalso it will be documented if using doc generation15:45
mordredand we need to make a full pass through the library before we 1.0 to make sure that we're happy with everything that doesn't have a _15:45
* clarkb goes to find the shade docs15:45
*** markvoelker has quit IRC15:45
mordredclarkb: and totally agree re: being more cranky and pedantic with shade15:46
openstackgerritMerged openstack/requirements: Update python-cinderclient to 1.2.1
*** baoli has joined #openstack-infra15:46
clarkbdoesn't look like shade's doc api generation is working at all fwiw15:47
*** rwsu has joined #openstack-infra15:48
*** ssam2 has joined #openstack-infra15:48
mordredclarkb: you're looking in the wrong place:
clarkbmordred: you should clean up the old docs then15:49
mordredclarkb: I believe we have an outstanding patch to do that15:49
mordredthat fungi wrote I think?15:49
*** _nadya_ has joined #openstack-infra15:49
mordredto add the redirects from ci.o.o to docs.o.o/infra15:49
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Root the Change-Id commentlink explicitly
fungiclarkb: mordred: yeah we still have a couple of projects we need to trigger documentation jobs for and then we can turn on the redirects15:50
fungijust got swamped with other things in the middle of that15:50
clarkbfungi: for 181949 do we need to root the query below the one you have updated too?15:51
clarkbline 18115:51
mordredfungi: I can't find the change itself ...15:51
fungiclarkb: oh, so we do. i missed that because it was an html replacement. fixing15:52
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Root the Change-Id and commit commentlinks
fungianyway, tested and working on review-dev, so should get around the problem for now while we figure out what new behavior in 2.10 (or possibly even 2.9) caused it15:54
*** eharney_ has joined #openstack-infra15:54
*** wenlock has quit IRC15:54
*** hashar has quit IRC15:55
*** eharney has quit IRC15:56
*** ildikov has quit IRC15:56
*** ildikov has joined #openstack-infra15:56
*** wenlock has joined #openstack-infra15:57
*** e0ne has joined #openstack-infra15:57
*** e0ne is now known as e0ne_15:58
*** mpaolino has joined #openstack-infra15:58
*** e0ne_ is now known as e0ne15:58
clarkbmordred: dguerri properly reviewed now, one small issue with removing that import15:58
mordredthanks clarkb15:58
mgagneAm I the only one not able to load this change?
jeblairmgagne: you are not the only one15:59
*** baoli has quit IRC15:59
*** annegentle has quit IRC15:59
mordredme neither15:59
mgagneI didn't scrollback far in backlog, any known problem?15:59
mgagne(related to it, not in general)16:00
clarkbfungi: +2 thanks for editing the second query link thing16:00
jeblairthat change # shows up in the error log16:00
*** annegentle has joined #openstack-infra16:00 org.eclipse.jgit.errors.MissingObjectException: Missing unknown 3af29bcd3c88c49978e1dae55852b0a3fff1261616:00
fungigertty can't retrieve it either16:01
openstackgerritMerged openstack/requirements: Using django-nose>=1.2 needed by murano-dashboard
fungi"Error: Unable to sync change."16:01
mriedemyeah it's everything, git review -d fails too16:01
*** unicell has quit IRC16:01
jeblair    remote_revision['commit']['message'], remote_commit,16:01
jeblairKeyError: 'commit'16:01
jeblairgertty ^16:02
dguerriclarkb thanks16:02
jeblairit's a nova change?16:02
fboHi, I tried to add a task inside a storyboard story but the task never appears in the task list. And on the other side I can see the event in the events timeline list. Is it a known issue or I misuse the form ? here is the related story!/story/200010116:02
mgagnefound in bug #129354016:02
openstackbug 1293540 in OpenStack Compute (nova) "nova should make sure the bridge exists before resuming a VM after an offline snapshot" [Low,Confirmed] - Assigned to Luo Gangyi (luogangyi)16:02
jeblairgit show refs/changes/42/149942/116:03
*** lennyb has quit IRC16:03
jeblairthat works on gerrit's git repo16:03
*** asselin_ has quit IRC16:03
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Update server API for get/list/search interface
*** banix has quit IRC16:03
fungi| I43fd09c6c36aa401b53a1af77fcb6775313797c1 | 2015-01-26 07:54:26 | 2015-05-03 11:46:54 | 0034dae2000249b6 |            13630 | openstack/nova    | refs/heads/master | Y    | n      |                   12 | Ensure bridge exists before resuming a VM after an offline snapshot | Bug1293540 |         110 |    149942 | df04ec843a9d32d30ec99ba07b5e757281dfdbb5 | Y         |16:03
*** spredzy is now known as spredzy|afk16:03
fungifrom the changes table16:04
fungistatus is 'n'16:04
fungiso not merged yet16:05
clarkbI can fetch taht change from git.o.o just fine16:05
*** jlanoux has quit IRC16:05
*** craigbr has joined #openstack-infra16:05
clarkbI am guessing its jgit having a hard time16:05
jeblairclarkb, fungi: i kind of want to see if it persists after a gerrit restart16:05
jeblairsince we do repack the repo behind gerrit's back16:05
fungioh, yep16:06
clarkbjeblair: that seems like a reasonable check16:06
mgagnethanks for the link to git, will check now16:06
*** annegentle has quit IRC16:06
fungithat sounds familiar to the jgit bug about cgit replacing pack files while retaining the same name causing confusion16:06
openstackgerritMarton Kiss proposed openstack-infra/system-config: Initial commit of ask-staging.o.o site
jeblairclarkb, fungi: shall i execute?16:07
jeblair$ git fetch gerrit refs/changes/42/149942/116:07
jeblairfatal: git fetch_pack: expected ACK/NAK, got 'ERR want e3a4b7ea669b8905da337df3ceb68ca40996963e not valid'16:07
jeblairalso ^16:07
clarkbzuul merges are ~14 minutes out so now is a reasonable time to restart gerrit16:07
clarkbactualy wait16:08
jeblairthat's via ssh, so to gerrit's git repo via jgit16:08
*** baoli has joined #openstack-infra16:08
clarkbjeblair: maybe try flushing all gerrit caches16:08
*** fitoduarte has quit IRC16:08
clarkbperhaps that will get it to rebuild its internal data structures around git repos, and if it works we can do that post repack?16:08
jeblairclarkb: flushing16:09
clarkbthere are diff and projects caches according to the docs which may be at play here16:09
*** mpavone has quit IRC16:09
jeblairthis is slow16:09
zarosdague: gerrit eclipse setup,
jeblairthe error log is flooded with this:16:10
jeblairjava.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 6c337867abda3d53da25187dfdfaf948eb89461a16:10
fungizaro: to update you to where we are, there seem to be several known issues post upgrade. 1. account registration dates all got reset (presumably an oversight in a db migration?), 2. the openid callback url adds a redundant trailing slash (/) which confuses some relative hyperlinks, 3. gerrit fails to find commits for some changes (possibly a jgit bug triggered by repacking the git repos)16:10
*** yfried|afk is now known as yfried16:10
sdaguezaro: yeh, none of that worked for me16:11
sdagueor, more importantly, I'm stuck at
jeblairclarkb: flishing done16:11
sdaguebecause buck won't run16:11
jeblairi think that made things worse :/16:11
sdaguewon't complete successfully16:11
clarkbjeblair: :(16:11
sdaguebecause watchman is missing, even though not in the docs anywhere, which makes me wonder if I'm on the wrong buck version16:12
jeblairclarkb: there are no visible nova changes now16:12
mordredoh. that's not good16:12
jeblairready for me to stop gerrit?16:12
mriedemall my -workflow changes are not showing16:12
clarkbjeblair: yup16:12
zarofungi: anybody working on any of those issues?  i can take a look at the trailing slash issue.16:12
jeblairso all those entries in the log are due to all nova changes being impacted by the 'missing' object16:12
fungiso perhaps we need to emergency restart gerrit and temporarily disable our repack cronjob for gerrit's backend repos16:13
clarkbits kind of amazing that flushing caches makes a stale cache inside gerrit become more problematic16:13
jeblairfungi: if the restart fixes it, possibly so16:13
jeblairstarting now16:13
fungizaro: i've got a config change proposed to root the commentlinks which works around the openid callback being slightly wrong, but haven't opened an upstream bug about it yet, so please do16:13
jeblairokay, nova changes exist again :)16:13
clarkbwe do repack on sundays, so it would have popped up yesterday at some point16:13
clarkbI can write a change to disable that cron16:14
jeblair works now16:14
zarosdague: have you run the ./tools/eclipse/ script?16:14
clarkbI wonder what Gerrit suggests users do to repack then :/16:14
fungizaro: i did at least confirm that changing the canonicalweburl to not include a trailing "/" doesn't seem to prevent the extra one either16:15
jeblairwe could go all windows on this and repack && restart16:15
*** MmikeM has joined #openstack-infra16:15
*** ashleighfarnham has joined #openstack-infra16:15
jeblairclarkb, fungi, zaro: but maybe the "nfs repack" bug isn't actually nfs related; perhaps it is triggered on local fs's too16:16
clarkbjeblair: it is certainly looking like it16:16
fungiclarkb: background gerrit task to repack via jgit? no clue16:16
sdague/home/sdague/code/buck/bin/buck_common: line 64: watchman: command not found16:17
sdagueso... I'm assuming that watchman is thus a hard dep16:17
*** jistr has quit IRC16:17
sdaguethat's not called out?16:17
sdagueanyway, need to run an errand, back in a bit16:17
*** jcoufal has quit IRC16:18
openstackgerritClark Boylan proposed openstack-infra/puppet-gerrit: Disable Gerrit git repo repacking cron
*** jcoufal has joined #openstack-infra16:18
clarkbjeblair: fungi zaro ^16:18
fungitoward the end of!msg/repo-discuss/Qdmbl-YZ4NU/nMhm6IwQic0J the discussion is mostly just about jgit not handling repacks where the packfile name doesn't change, unrelated to choice of filesystem16:18
*** alexsyip has joined #openstack-infra16:18
clarkbalso this appears to be very similar to the issue GitPython has where it loads up the git tree layout in memory and if it every changes on disk you lose16:19
*** Krinkle is now known as Krinkle|detached16:19
zarohmm, i see that on our build of gerrit as well:
*** banix has joined #openstack-infra16:19
zarosdague: ^ but it seems to continue and finish ok16:19
*** jogo has quit IRC16:19
*** devvesa has quit IRC16:21
*** baoli has quit IRC16:22
*** kmartin has joined #openstack-infra16:22
*** koolhead17 has quit IRC16:22
*** claudiub|2 has joined #openstack-infra16:22
jeblairis there an issue in the tracker for the jgit thing?16:22
zarosdague: make sure you are cloning buck from same location as our build:
*** jcoufal has quit IRC16:23
*** jcoufal has joined #openstack-infra16:23
openstackgerritObulpathi proposed openstack-infra/project-config: Removes Pypy jobs for Poppy at Jenkins gate
*** claudiub has quit IRC16:25
openstackgerritDavide Guerri proposed openstack-infra/shade: Add floating IP pool resource methods
openstackgerritDavide Guerri proposed openstack-infra/shade: Rewrite extension checking methods
fungistill looking, haven't found one yet at least16:25
clarkbfungi: can I get a +A on I can rebuild a bare-trusty image this morning and babysit16:25
*** sdake_ has joined #openstack-infra16:25
*** ddieterl_ has joined #openstack-infra16:26
Clint is super old16:26
*** pcm_ has joined #openstack-infra16:27
clarkbsdague: do you know if jhesketh has written a change to run the new os-loganalyze devstack plugin against os-loganalyze changes?16:27
fungiyeah, fixed in 2.0.11 so at best a regression of that bug, but more likely some new bug with similar symptoms16:27
*** ddieterl_ has quit IRC16:27
dguerriclarkb, btw old implementation didn't work (self._extension_cache used before assignment). I get the general point of yours, but at least in this case I guess nobody were using that method16:27
*** obulpathi has joined #openstack-infra16:28
mordredor, couldn't have been, since it flat out did not work16:28
*** baoli has joined #openstack-infra16:28
clarkbfungi: Clint what is old is new again/16:28
*** ddieterly has quit IRC16:29
*** BharatK has quit IRC16:29
*** pc_m has quit IRC16:29
*** pfallenop has quit IRC16:29
*** antonym has quit IRC16:29
*** alaski has quit IRC16:29
*** gothicmindfood has quit IRC16:29
*** marun has joined #openstack-infra16:29
*** sdake has quit IRC16:29
*** tim_o has quit IRC16:29
* mordred cries16:29
*** smccully has quit IRC16:29
*** ldnunes has quit IRC16:29
*** mmedvede_ has quit IRC16:29
*** tmcpeak has quit IRC16:30
clarkbfungi: danke16:30
*** tiswanso has quit IRC16:30
fungigoing over that thread in more detail, there is indeed a suggestion of using jgit to do the repacking'16:31
*** pfallenop has joined #openstack-infra16:31
*** tiswanso_ has joined #openstack-infra16:31
*** gothicmindfood has joined #openstack-infra16:31
*** rmcall has quit IRC16:31
clarkbfungi: any indication if we can do that without writing a special java utility just to do it?16:31
clarkbeg java -jar jgit.jar repace /home/gerrit2/review_site/git ?16:32
*** rmcall has joined #openstack-infra16:32
*** _nadya_ has quit IRC16:32
mordredfungi: just java it. in the docker.16:33
clarkbdguerri: ah ok, missed that, commit message says it didn't work but not because of the NameError16:33
greghaynesmordred: re: I think simple-init should have that exact behavior16:34
clarkbwe should jython16:34
*** unicell has joined #openstack-infra16:34
zarofungi: Could that jgit thing be causing failures to view WIP changes?  i don't see that problem on review-dev.o.o16:34
jeblairsurely the problem is open files / stale internal cache, not a jgit-git incompatibility16:35
greghaynesmordred: and I think (or it should be) as easy as dependon on the cloud-init-datasources and exporting that var16:35
*** alaski has joined #openstack-infra16:35
greghaynesI can poke at that and see..16:35
*** Sukhdev has joined #openstack-infra16:35
clarkbzaro: the problem appears to be with repacking behind jgits back and doing so in a way that jgit does not expect16:35
mordredgreghaynes: so, I completely disagree with you16:35
mordredgreghaynes: but it's possible I'm missing something or understanding you wrong16:35
jeblair(eg, if you did something like "java -jar jgit.jar repack" i would expect it to still break)16:35
clarkbzaro: basically C git can reuse the pack name but make a new pack, jgit explodes when that happens16:35
jeblairclarkb: oooh, i see what you're thinking16:36
mordredgreghaynes: that behavior is about configuring how cloud-init works16:36
clarkbjeblair: I think the thread indicates ^ breaks gerrit, it expects a new pack name16:36
mordredgreghaynes: I dont' think simple-init should have that behavior, because cloud-init should not exist on images with simple-init16:36
jeblairclarkb: gotcha, so it is an incompatibility, of sorts.16:36
fungithat's my reading as well16:36
*** sdake has joined #openstack-infra16:36
clarkbI suppose its possible newer C git introduces this for us too ?16:36
fungi also discusses a possible workaround of not replacing packfiles16:36
clarkbsince we did upgrade to trusty16:36
greghaynesmordred: Since we build off cloud images for most of the distros I think it would be nice to work with that setup somehow (just the ubuntu and simple-init element as an example)16:37
greghaynesmordred: and its not hard...16:37
openstackgerritMerged openstack-infra/puppet-diskimage_builder: Update metadata to be diskimage_builder
greghaynesmordred: we just have to do exactly what that patch does16:37
mordredgreghaynes: right - but ... so you're saying you think that if someone installs simple-init _with_ cloud-init, that they should have simple-init configure cloud-init to only use config-drive?16:38
mordredgreghaynes: rather tahn having the simple-init element uninstall cloud-init if present?16:38
greghaynesoh what, sorry, I thought it did disable it16:39
fungitrying to figure out of git-index-pack's --keep option does this16:39
mordredgreghaynes: right - so if it disables it, then what is the point of putting something in the data sources config?16:39
fungihrm, perhaps not16:39
*** yamamoto has quit IRC16:39
greghaynesmordred: er, I thought that patch I was reading was there to disable it, youre right, it sets it to be configdrive16:39
greghaynesmordred: I think currently we dont disalbe cloud-init at all16:40
*** deepakcs has quit IRC16:40
greghaynes(in dib)16:40
mordredgreghaynes: I agree with you16:40
*** sdake_ has quit IRC16:40
mordredgreghaynes: so - a) I agree that simple-init should have a sane behavior on top of the distro-provided ubuntu element16:40
*** otter768 has joined #openstack-infra16:40
*** dangers is now known as dangers_away16:41
clarkbfungi: jeblair maybe the -F flag to repack would fix this?16:41
clarkbit will cost much cpu time though16:41
samueldmqhi, gerrit does not show any info for
fungiaha, that's what i was looking for16:41
samueldmqjust a blank page ^ : /16:41
*** BharatK has joined #openstack-infra16:41
*** kgiusti has quit IRC16:42
clarkbgah maybe repacking isn't causing the problems16:42
*** antonym has joined #openstack-infra16:42
fungiokay, maybe this isn't repack related if we've now got new failures since the restart16:42
*** mmedvede_ has joined #openstack-infra16:42
*** smccully has joined #openstack-infra16:42
jeblairorg.eclipse.jgit.errors.MissingObjectException: Missing unknown 906fca0ba6642bc2673b32f17cd31190b90805eb16:42
greghaynesmordred: ok, so im fine with the change, sorry about that16:43
jeblairnow the logs are full of that ^16:43
mordredgreghaynes: cool. I figured we were just missing each other somewhere16:43
*** ldnunes has joined #openstack-infra16:43
mordredjeblair: oh wow. thats so exciting16:43
samueldmqjeblair, oh, someone committed eclipse files  ?16:43
mordredquite literally exactly what we wanted to discover 2 days after the upgrade16:44
mordredsamueldmq: nah - that's the signature for jgit16:44
*** derekh_ has joined #openstack-infra16:44
*** ssam2 has quit IRC16:44
mordredsamueldmq: it's a library that exists within the eclipse org16:44
clarkbjeblair: looks the same client side too, gerrit responds with a 304 unchanged for the change details16:45
samueldmqmordred, ++ makes sense16:45
openstackgerritMerged openstack-infra/system-config: Use latest current virtualenv
samueldmqmordred, btw, do we have any known issues with gerrit today ?16:45
clarkbwhich is a really weird return code for "server failed"16:45
*** otter768 has quit IRC16:45
*** banix has quit IRC16:46
samueldmqI can't see any info for
*** mpaolino has quit IRC16:46
mordredsamueldmq: yup. we're seeing that problem ^^16:46
*** ZZelle is now known as ZZelle_16:46
mordredsamueldmq: which is what jeblair and clarkb are working on diagnosing right now16:46
clarkband fungi16:46
*** ericksonfgds has joined #openstack-infra16:46
jeblairi'm looking in the log for a triggering event16:46
samueldmqmordred, oh ... maybe what I said is a side-effect16:46
*** mleroy has joined #openstack-infra16:46
*** esker has quit IRC16:46
clarkbI am going to go get caffeine before more debugging, back in a few16:46
*** tnovacik has joined #openstack-infra16:47
*** derekh has quit IRC16:47
*** shashankhegde has joined #openstack-infra16:49
jeblair[2015-05-11 16:30:13,687] ERROR : Pack file /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack is corrupt, removing it from pack list16:49
jeblairorg.eclipse.jgit.errors.CorruptObjectException: Object at 87,706,216 in /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack has bad zlib stream16:49
jeblairthat happens right after the first "missing" error16:49
fungiseeing what git fsck reports for it16:50
jeblairfungi: maybe don't operate on the actual repo16:51
*** derekh_ has quit IRC16:51
jeblairfungi: i have one going on an rsync copy16:51
fungiahh, yours has probably been running longer16:51
fungii'm guessing jgit ran into some unexpected failure and didn't handle it well, and assumed a corrupt packfile16:52
fungiand that fsck will say everything's okay16:52
*** dguitarbite has quit IRC16:52
fungisimilar to what's discussed in the commit message for
*** harlowja_at_home has quit IRC16:53
jeblairwell, the fsck output exceeded my terminal buffer16:53
fungiand the bug linked from it16:53
*** banix has joined #openstack-infra16:54
*** afazekas has quit IRC16:55
jeblairthe fsck output is:16:56
jeblairerror: invalid parameter: expected sha1, got '.'16:56
jeblairfollowed by 1246 lines like:16:56
jeblairdangling blob 831000ca7365a61e700460703a6cc65d22355e0316:56
*** tmcpeak has joined #openstack-infra16:56
jeblairdangling commit 0a5380df6ccf793b87dd985c8b657d0b7d40d22c16:56
jeblairit's in review.o.o:~corvus/nova.git/fsck-out16:56
*** davideagnello has joined #openstack-infra16:58
*** yamahata has quit IRC16:58
*** banix has quit IRC16:59
*** yamahata has joined #openstack-infra16:59
*** eharney_ is now known as eharney16:59
jeblairbut it doesn't say anything about a bad zlib stream16:59
fungino mention of the 93ad57004de887eb835b2bd4df2d7c3f6a5c394b packfile there16:59
*** amotoki_ has quit IRC16:59
clarkbdoes git show on 906fca0ba6642bc2673b32f17cd31190b90805eb work from your repo copy?17:00
*** davideagnello has quit IRC17:00
*** davideagnello has joined #openstack-infra17:00
fungilooks like a normal commit17:00
*** sshnaidm has quit IRC17:00
clarkbso git porcelain commands are able to go find it17:01
*** _nadya_ has joined #openstack-infra17:01
*** whoops has joined #openstack-infra17:01
*** dizquierdo has joined #openstack-infra17:01
fungistarting to wonder if the issues fixed in the jgit upgrade that got rolled back are biting us, and whether we want to add that commit that brings in jgit 417:01
*** dkranz has joined #openstack-infra17:01
fungirevisiting the details from that now17:01
jeblairgit show-index < pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.idx|grep 8770621617:02
jeblair87706216 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 (b60fcf57)17:02
jeblairgit show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 works too17:02
fungibasically, we are in theory running a jgit affected by bug 422988 in JGit "checkReferencedIsReachable fails to find a tree which is not sent as part of a pack" [Normal,Reopened] - Assigned to jgit.core-inbox17:03
jeblairrunning git verify-pack pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.idx17:04
fungialso i didn't get "error: invalid parameter: expected sha1, got '.'" from my git fsck. might have had to do with invocation? i just ran `git fsck` while my cwd was in my raw copy of the repo17:04
clarkblooks like is the newly proposed fix?17:04
jeblairfungi: oh, yeah, that was probably it, sorry17:04
*** annegentle has joined #openstack-infra17:05
jeblairgit verify-pack pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.idx returned without error or output17:05
*** shashankhegde has quit IRC17:05
fungiseems like jgit is confused and thinks the packfile is corrupt when in fact it probably hit some other transient problem17:06
fungiand then defensively stops retrying to read it on subsequent requests, persisting in its belief it's corrupt17:06
clarkbfungi: so restarting the server may help just to reset that "corrupt" flag17:07
clarkband I suppose its possible for packing to move things such that we hit the bug, but is not the cause of the bug17:07
zaroi've restored the update to jgit 4.0 in case we need it:
*** aduarte has joined #openstack-infra17:07
clarkbzaro: well I think we will want once it is ready since that is supposed to fix the performance issues17:08
jeblairfungi: that patch seems to be related to receiving packs, so i wonder if it's relevant to current situation (it may be through some mechanism i don't understand, but on the surface, i would guess not?)17:08
clarkbzaro: but starting with the bad performing patch is likely a good start17:08
clarkbjeblair: thats how you push code, so could happen when new patchsets are pushed?17:08
fungijeblair: yeah, seems at best tangentially related17:08
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Grant sahara-ci group permissions to verify all sahara repos
* fungi was just grasping at the one known jgit issue we had discussed in our upgrade planning17:09
jeblairclarkb: ah, hrm.17:09
*** banix has joined #openstack-infra17:09
fungi was apparently the landed revert for the performance problem17:10
clarkbjeblair: `git upload-pack` is the C git thing that does it iirc17:10
*** baoli has quit IRC17:10
mleroyHi all. The changes for the creation of the networking-6wind project have been merged. Can you please add me to both groups (networking-6wind-core, networking-6wind-release) in gerrit? Or should I send an email to the mailing list for that ? Maxime Leroy17:11
clarkboh wait, send-pack17:11
*** _nadya_ has quit IRC17:11
*** baoli has joined #openstack-infra17:12
*** claudiub has joined #openstack-infra17:12
*** ivar-lazzaro has quit IRC17:12
*** e0ne has quit IRC17:13
*** sputnik13 has joined #openstack-infra17:13
jeblairthe action that appears to have triggered the zlib-related failure appears to be an ssh query operation -- so the direct cause does not appear to be an upload, but i suppose it could have had a delayed impact?17:13
*** btully has quit IRC17:14
*** Guest28311 has quit IRC17:14
*** prad_ has joined #openstack-infra17:14
clarkbmleroy: we are currently debugging issues with the gerrit upgrade, sending email might be best17:14
clarkbmleroy: so it doesn't get lost in scrollback17:15
*** jcoufal has quit IRC17:15
mleroyclarkb: ok thanks17:15
*** claudiub|2 has quit IRC17:15
*** turul has joined #openstack-infra17:15
clarkbjeblair: perhaps, I can imagine that a push via send-pack would make the repo unhappy, then the next query goes to look in the repo and throws the exception17:15
*** turul is now known as afazekas17:15
*** dguitarbite has joined #openstack-infra17:15
*** ddieterly has joined #openstack-infra17:16
jeblairthere's the relevant log snippet; the main error is in the middle17:16
KiallIn case anyone hasn't noticed it already, the review merged -> launchpad updates stuff seems flaky -
openstackLaunchpad bug 1437699 in Designate "mDNS should Handle Bad File Descriptor" [Medium,Fix committed] - Assigned to Kiall Mac Innes (kiall)17:16
*** prad has quit IRC17:17
*** tiswanso_ has quit IRC17:18
*** cdent has quit IRC17:18
*** jogo has joined #openstack-infra17:19
*** jogo has quit IRC17:19
*** jogo has joined #openstack-infra17:19
fungiKiall: looks like that may be unrelated to the gerrit upgrade. it's from a tag merge change, so probably enumerating all the commits between stable/kilo and the tag merge point on master17:19
fungiKiall: if you find other examples which aren't from a merge change, then i agree we may have a deeper issue17:20
clarkbjeblair: reading that it appears the diff timed out because it couldn't get the object because the pack was "corrupt" ? is that how you read the log?17:21
*** jamesmcarthur has joined #openstack-infra17:21
jeblairclarkb: i suspect the diff timeout is unrelated, they show up fairly frequently normally17:21
*** tiswanso_ has joined #openstack-infra17:21
*** ddieterl_ has joined #openstack-infra17:22
fungi bug 1085328 in Business Central "Git: cannot clone repositories from business-central to different machine" [High,Closed: currentrelease] - Assigned to porcelli17:22
*** ddieterly has quit IRC17:22
fungihas to do with jgit timeouts causing "packfile truncated" errors17:22
Kiallfungi: ah, Okay.. that makes sense.. I assumed upgrade related ;)17:22
fungiso possible that a timeout reading could cause jgit to get confused and think it's corrupt17:23
jamesmcarthurclarkb: fungi: - clarkb alerted me one of our server IPs being blocked over the weekend for excessive Gerrit queries. I took a look at the queries and we are paging. However, they could use a little refining. We are pulling merged changes for local stats for the Foundation (# of countries merging, % merging, etc…). We’re going to work to clean up the queries a bit so we’re not hitting you all so frequently.17:24
*** patrickeast has joined #openstack-infra17:24
*** emagana has quit IRC17:24
*** ihrachyshka has quit IRC17:24
*** krtaylor has quit IRC17:24
*** arxcruz has quit IRC17:24
clarkbjamesmcarthur: thank you, I think we can refine by adding an age parameter so that we aren't getting the entire history of all merges17:25
fungijamesmcarthur: thanks. it looked like the frequency of the queries looking for merges was the biggest drain. it was causing a fair amount of drag on gerrit performance too17:25
jamesmcarthurfungi: clarkb: One thing I noticed is that we’re not limiting by project. Am I correct in limiting it to only projects starting with openstack/17:25
jamesmcarthurAnd yes clarkb: - we’re adding an age param as well17:25
jamesmcarthurfungi: My apologies for the drag on Gerrit :| No bueno.17:25
*** emagana has joined #openstack-infra17:26
fungijamesmcarthur: not really. the repository naming is only mostly indicative of openstack officialness. there's actually a structured data file maintained by the technical committee which enumerates them
jeblairclarkb: there are 3 pack file removals in the log; two of them are near a diff timeout, one is not.17:26
*** dizquierdo has quit IRC17:27
fungijamesmcarthur: so checking all projects hosted in gerrit is likely necessary17:27
jeblairclarkb: the first removal is:17:27
jeblair[2015-05-11 02:59:02,155] ERROR : Pack file /home/gerrit2/review_site/git/stackforge/fuel-stats.git/objects/pack/pack-64b011fb58533468215ff41155731039b6cd56e1.pack was deleted, removing it from pack list17:27
fungijamesmcarthur: depending on what you're trying to determine with that query anyway17:28
jamesmcarthurfungi: thanks jeremy, I found that this monring. However a lot of those we don’t need data on. For instance stackforge projects.17:28
clarkbjeblair: huh, thats not when we repack, weird that it would decide the pack was deleted17:28
jamesmcarthurJust trying to figure out if there is some common thread we can use to pull a more limited data set.17:28
fungiclarkb: unless that was the first time since the repack that it tried to access that file17:28
clarkbfungi: good point17:28
jeblairjamesmcarthur: depending on what you want, you may want to include stackforge projects.  also, if you have in the past, i certainly would not stop now, especially not with the big-tent direction the project is taking.17:29
jeblairclarkb: the second and third times were for the same nova pack file (and that was across our restart)17:29
jeblairorg.eclipse.jgit.errors.CorruptObjectException: Object at 89,109,081 in /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack has bad zlib stream17:29
*** baoli has quit IRC17:29
jeblair[2015-05-11 02:59:02,155] ERROR : Pack file /home/gerrit2/review_site/git/stackforge/fuel-stats.git/objects/pack/pack-64b011fb58533468215ff41155731039b6cd56e1.pack was deleted, removing it from pack list17:29
jeblairoops, sorry wrong paste17:29
jeblairmeant to say:17:30
jeblairorg.eclipse.jgit.errors.CorruptObjectException: Object at 89,109,081 in /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack has bad zlib stream17:30
jeblairorg.eclipse.jgit.errors.CorruptObjectException: Object at 87,706,216 in /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack has bad zlib stream17:30
*** craigbr has quit IRC17:30
jeblairso same pack file, different indexes17:30
clarkbjeblair: and one before restart one after right?17:30
*** sks has quit IRC17:31
*** moshele has joined #openstack-infra17:31
jeblairclarkb, fungi: i'm willing to discount the fuel one for the moment since it was after a repack17:31
*** vipul has quit IRC17:31
jeblairbut i'm starting to wonder if there really is something about the pack-93 file that jgit doesn't like?17:31
clarkbjeblair: seems possible17:32
jogoI am getting a fatal internal server error when I call 'ssh -p 29418 gerrit query --format=JSON is:watched status:merged17:33
jeblairjogo: likely related to our current investigation17:33
jeblairit might be time for an alert17:33
clarkb jgit does have a cli interface but no fsck (was hoping we could compare fscks between C git and jgit)17:33
zarofungi: that '//' issue appears on review-dev.o.o as well but the link seems to work fine.  wondering what's different17:33
*** baoli has joined #openstack-infra17:34
*** shashankhegde has joined #openstack-infra17:34
*** notnownikki has quit IRC17:34
fungizaro: restart gerrit there. i was testing my proposed fix17:35
jogojeblair: thanks17:35
zarofungi: ahh ok.17:35
jeblairstatus alert We have discovered post-upgrade issues with Gerrit affecting nova (and potentially other projects).  Some changes will not appear and some actions, such as queries, may return an error.  We are continuing to investigate.17:35
jeblairdoes that look good ^ ?17:36
clarkbjeblair: yup17:36
pleia2thumbs up17:36
jeblair#status alert We have discovered post-upgrade issues with Gerrit affecting nova (and potentially other projects).  Some changes will not appear and some actions, such as queries, may return an error.  We are continuing to investigate.17:36
openstackstatusjeblair: sending alert17:36
*** krtaylor has joined #openstack-infra17:36
*** yfried has quit IRC17:37
fungipost-facto lgtm17:38
clarkbI have installed jgit-cli locally and am trying to see if it gives us any interfaces into sanity17:38
-openstackstatus- NOTICE: We have discovered post-upgrade issues with Gerrit affecting nova (and potentially other projects). Some changes will not appear and some actions, such as queries, may return an error. We are continuing to investigate.17:38
*** ChanServ changes topic to "We have discovered post-upgrade issues with Gerrit affecting nova (and potentially other projects). Some changes will not appear and some actions, such as queries, may return an error. We are continuing to investigate."17:38
jeblairclarkb: ++ thx17:38
fungiooh, i didn't realize jgit-cli was a thing17:38
fungigood call17:38
jeblairi need to afk for about 10 mins.  i have emitted our problem into the #gerrit channel, no response so far.  back asap.17:39
clarkbbut confirmed no fsck is available, do have a gc though17:39
fungigerrit seems to do its own gc as well, if i'm interpreting the files in ~gerrit2/review_site/logs correctly17:40
*** moshele has quit IRC17:40
mordredis that new since we added our cronjob gc?17:40
clarkbmordred: we cronjob repack not gc which happens to also prune17:40
fungi[2015-05-11 16:13:36,603] INFO  : Ignoring missing gc schedule configuration17:40
openstackgerritJoe Gordon proposed openstack-infra/project-config: Add pep8 job to nova-specs
openstackstatusjeblair: finished sending alert17:40
fungilooks like maybe it _would_ do them if we configured it to17:40
*** mestery has joined #openstack-infra17:41
fungiseems to add that to the log each time gerrit starts17:41
tmcpeakhey guys, what's the best way to find out which OpenStack projects are using a certain requirement?17:42
clarkbtmcpeak: clone all of the projects then find -name \*requirements.txt -exec grep $requirement {};17:42
tmcpeakclarkb: *all the projects*?17:42
fungitmcpeak: eventually something like livegrep. for now i have a cron job which keeps updated clones of everything from gerrit and then i loop over them ang git grep17:42
fungier, and17:43
tmcpeakfungi: ahh ok, I guess where's a good comprehensive list to point it at?17:43
fungitmcpeak: the gerrit api17:43
tmcpeakfungi: cool, thanks17:43
*** annegentle has quit IRC17:43
*** baoli has quit IRC17:43
fungirm_work: ssh -p 29418 gerrit ls-projects17:44
tmcpeakfungi: cool, thank you17:44
*** mestery has quit IRC17:44
openstackgerritSteve Leon proposed openstack-infra/project-config: Add devstack gate job for Cue project
fungitmcpeak: my hackish script i run from cron is if you want ideas17:45
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Update images API for get/list/search interface
fungirm_work: sorry, meant to highlight tmcpeak and my tab-complete failed me17:46
*** mestery has joined #openstack-infra17:46
tmcpeakfungi: perfect, thanks!17:46
clarkbdo we know what change fc05ab545c014f31078b3d53f843e299e95b8a51 belongs to in nova so I can fetch it?17:47
fungitmcpeak: obviously beware of hardcoded dragons, like my callouts to keychain and embedded username in my git remote17:47
clarkb that actualy worked surprisingly well17:47
tmcpeakfungi: yep, I don't copy and paste *anything* ;)17:48
tmcpeakfrom untrusted sources anyway, no offense17:48
fungitmcpeak: here, wget this and pipe it through sudo bash ;)17:48
fungitmcpeak: i expect nothing less!17:48
tmcpeak+1 for a good time17:48
clarkbusing the jgit-cli shipped by trusty I can `jgit fetch --fsck refs/changes/63/111563/4` and that works, now to see if I can build the same jgit gerrit is using17:49
openstackgerritgreghaynes proposed openstack-infra/nodepool: Use shade for image uploads
openstackgerritLouis Taylor proposed openstack-infra/project-config: Add experimental glance_store functional test job
*** Krinkle|detached is now known as Krinkle17:52
fungizuul is dealing interestingly with one of the broken changes... (see top of integrated gate change queue in status_17:53
*** moshele has joined #openstack-infra17:53
fungior i should say not dealing well at all17:53
fungilooks like that has gummed it up entirely17:53
openstackgerritSwaminathan Vasudevan proposed openstack-infra/project-config: Make neutron-dvr job voting at the gate
mordredfungi: you know I have a code indexer running, right? just waiting on reviews of the puppet before I start handing end-users the url ...17:53
*** achanda has joined #openstack-infra17:54
fungimordred: yep, just couldn't remember the name of it17:54
greghaynesmordred: openstack cross reference?17:54
mordredfungi: I say that not because it's important right now, but because you just said "eventually something like livegrep" - but there is actually one running17:54
*** cdent has joined #openstack-infra17:54
mordredgreghaynes: yes17:54
mordredwell, an index of all of the source code in gerrit is what it actually is - but yeah17:54
fungimordred: more specifically, that's why i said "something like" (because i couldn't remember what the new thing you got working is called)17:55
mordredfungi: :)17:55
openstackgerritLouis Taylor proposed openstack-infra/project-config: Add experimental glance_store functional test job
*** ibiris is now known as ibiris_away17:55
fungioh, right, hound17:55
jeblairi think it would be a good use of time to start preparing for a downgrade17:56
clarkbzaro: I am thinking we should try to reproduce these issues on review-dev. Maybe we put nova on that server. Then we upgrade to gerrit-v2. and see if it fixes17:56
clarkbjeblair: not ^17:56
*** esker has joined #openstack-infra17:56
mordredjeblair: oh wow17:56
*** ivar-lazzaro has joined #openstack-infra17:56
jeblairnova is basically dead in the water17:56
*** mestery has quit IRC17:56
mordredmaybe we do the two in parallel?17:57
jeblairmordred: yes, that's what i'm suggesting17:57
mordredthen yes - I totally agree17:57
*** prad_ is now known as prad17:57
jeblairclarkb: what's .24?17:57
clarkbjeblair: that change17:57
*** baoli has joined #openstack-infra17:58
jeblairi believe the main thing we need to do for a downgrade would be to reverse the schema changes17:58
*** pelix has quit IRC17:58
clarkbjeblair: sounds right, we have to reindex which will build that from scratch17:58
jeblairclarkb: agreed17:59
fungiyeah, i'm on board17:59
jeblairwould someone like to go and prepare the sql to reverse the schema changes?17:59
mordredso - worth looking at the schema changes to make sure we actually do need to schema downgrade?17:59
jeblairmordred: i'm pretty sure we need to do the work to create the downgrade ourselves; i don't know of a way to ask gerrit to downgrade to a certain version18:00
fungii'm sort of wondering how we reproduce still, because i get the impression that gerrit restarts clear up the changes it's thinking are broken18:00
mordredjeblair: yes. I agree that we will need to do that18:00
clarkbI think mordred is suggesting that the curernt schema may be compat18:00
clarkband we should check18:00
*** dguitarbite has quit IRC18:00
fungiat least that's what we saw so far18:00
mordredbut yes ^^18:00
mordredwe may not need to manipulate the db18:00
* mordred is going to read source code18:00
jeblairmordred: so the next step after writing the sql to downgrade is to see what the delta is and think about ti18:00
*** btully has joined #openstack-infra18:01
jeblairmordred: but step 1 regardless, is to figure that out18:01
mordredI agree with you - and I'm looking at it now18:01
jeblairmordred: and that's what i'm asking for a volunteer for; is that you? :)18:01
jeblairmordred: we were on schema 86 before18:02
jeblairmordred: thanks18:02
*** ajmiller has joined #openstack-infra18:02
clarkbzaro: so jgit 4.0 is basically are you suggesting that there are other fixes in 4.0 on top of our current 3.x state which does not include that patch?18:02
clarkbzaro: if so, do you expect those changes to fix this problem?18:02
* clarkb is trying to figure out if he should spend time with review-dev to reproduce and apply the .24 war18:03
jeblairclarkb: i think we'll need part or all of the database on review-dev too, so that we can look at changes in gerrit and cause it to try to access git18:03
mordredwhere are the migrations?18:03
*** dprince has quit IRC18:03
mordredthank you18:04
jeblairmordred: and just to confirm, we're currently on schema 98.18:05
*** btully has quit IRC18:05
mordred13 java schema migrations18:05
zaroclarkb: yes, i believe 4.0 contains more than that fix but i have no idea what the extra fixes are so don't know if it will fix our particular problem.18:05
*** obulpathi has left #openstack-infra18:05
fungii'm currently reading through the jgit changelog to see if anything looks relevant18:06
clarkb is one fix that may be handy that is in 4.018:07
clarkbfungi: me too :)18:07
mordredjeblair: so - the first migration, 87, seems like it might be hard to revert18:07
mordredjeblair: as it deletes SYSTEM groups18:07
jeblairmordred: so we might restore those from our dump18:08
mordredjeblair: yah18:08
clarkbfungi: thats the only one I am seeing between 3.7.x and 4.0.0 that looks helpful18:09
fungihow can i easily tell which jgit we've currently built on? the build log or is there a way to ask gerrit?18:10
clarkboh but we use 3.7.0 and there is 3.7.218:10
clarkbfungi: old side is what we use now18:10
fungiahh, yep, just ended up back tehre myself. thanks18:10
fungiso basically 65c379e..v4.0.0.201505050340-m218:12
*** annegentle has joined #openstack-infra18:12
clarkbthat looks correct18:13
clarkbI really hope that --0700 is meant to be +0700 but that doesn't seem likely18:15
*** emagana has quit IRC18:15
fungino idea if that plays into this at all18:16
clarkb is fun18:17
*** emagana has joined #openstack-infra18:17
fungiclarkb: the irony is in the end of the last file in that diff18:18
*** achanda has quit IRC18:18
mordredjeblair: ok -I've gone through the schema versions18:19
mordredjeblair, zaro clarkb: what's up with the empty ones? there are SEVERAL of them that just seem to increment the version and do nothing else18:19
*** yfried has joined #openstack-infra18:19
*** yfried has quit IRC18:19
jeblairmordred: i have no idea; maybe git blame and check out the commits?18:19
clarkbmordred: perhaps thats how they reserve them for older point releases?18:19
mordredclarkb: that's right18:20
*** arielb has joined #openstack-infra18:20
*** arielb has joined #openstack-infra18:20
fungiyeah, solution for later migration injection18:20
*** tonytan4ever has joined #openstack-infra18:20
clarkbfungi: zaro so I am not seeing any bugfixes that standout as fixing this (except possibly the first one I linked)18:20
*** baoli has quit IRC18:21
clarkbfungi: zaro but there does seem to be a decent sized set of bug fixes in general18:21
*** baoli has joined #openstack-infra18:21
*** funzo_ is now known as funzo18:21
mordredah. nope. it's more isidious18:22
clarkbso probably worth trying out on review-dev if we can reproduce the failures with nova there18:22
clarkbmordred: darth insidious?18:22
mordredthe java infers all the columns that need to exist18:22
mordredthe migration code only handles other things18:22
fungioh, ew18:22
fungithat's going to be a pain to unwind18:22
clarkbmordred: so you have an empty migration if the java changed?18:22
mordred97 is a good example18:23
mordredit adds a column in teh java model18:23
mordredand then has an empty migration18:23
fungiso basically empty migration means "java magic happens here"18:23
jeblairwell, all migrations mean that, right?  empty ones mean that no transformations were necessary18:23
fungier, right18:23
mordredwell, I think what will happen is when you go to run migrations, it will look at the current data model, compute needed additional columns and apply all of them18:23
mordredthen run through the schema migrations one at a time to handle things that aren't in teh java data model18:24
mordredlike indexes or data transforms18:24
fungiso basically any migration might also have unspecified implications in the java source18:24
clarkbmordred: so additions are ok as long as it doesn't also remove things?18:24
*** hemna_ has joined #openstack-infra18:24
*** unicell has quit IRC18:25
*** alaski has quit IRC18:25
*** dboik_ has quit IRC18:25
*** mpopow has quit IRC18:25
*** sarob has quit IRC18:25
*** bnemec has quit IRC18:25
*** kaisers has quit IRC18:25
*** Guest64787 has quit IRC18:25
*** smallbig has quit IRC18:25
*** morganfainberg has quit IRC18:25
*** russellb has quit IRC18:25
*** adreznec has quit IRC18:25
*** mfink- has quit IRC18:25
*** b^_^d has quit IRC18:25
*** SotK has quit IRC18:25
*** tsufiev has quit IRC18:25
*** mrda has quit IRC18:25
*** ianw has quit IRC18:25
mordredyah. I mean I think we can run on a half-migrated db  - but for reference:18:25
jeblairone thing to keep in mind is that if we run the old version without a schema downgrade, when we upgrade again, we may be in a pickle.18:25
mordredjeblair: yup18:25
*** samueldmq_ has joined #openstack-infra18:26
mordredlines 156 and 157 got added in the same commit18:26
*** larsks|alt has joined #openstack-infra18:26
mordredas added
*** mjg has joined #openstack-infra18:26
*** radez_g0n3 has joined #openstack-infra18:26
mordredso 156 and 157 add the column18:26
*** emagana has quit IRC18:26
*** gilliard has quit IRC18:26
*** larsks has quit IRC18:26
*** samueldmq has quit IRC18:26
*** rcarrill` has quit IRC18:26
*** dmellado has quit IRC18:26
*** tlbr has quit IRC18:26
*** hemna has quit IRC18:26
*** slagle has quit IRC18:26
*** phschwartz has quit IRC18:26
*** claudiub|2 has joined #openstack-infra18:26
*** claudiub has quit IRC18:26
*** yfried has joined #openstack-infra18:26
*** Shrews has quit IRC18:26
*** zaneb has quit IRC18:26
*** bkero has quit IRC18:26
*** hamzy has quit IRC18:26
*** waynr has quit IRC18:26
*** radez has quit IRC18:26
*** hamzy_ has joined #openstack-infra18:26
mordredand Schema_97 does nothing18:26
*** samueldmq_ is now known as samueldmq18:26
*** waynr has joined #openstack-infra18:27
*** zaneb has joined #openstack-infra18:27
*** slagle has joined #openstack-infra18:27
*** alaski has joined #openstack-infra18:27
*** unicell has joined #openstack-infra18:27
*** dboik_ has joined #openstack-infra18:27
*** mpopow has joined #openstack-infra18:27
*** sarob has joined #openstack-infra18:27
*** bnemec has joined #openstack-infra18:27
*** kaisers has joined #openstack-infra18:27
*** smallbig has joined #openstack-infra18:27
*** b^_^d has joined #openstack-infra18:27
*** morganfainberg has joined #openstack-infra18:27
*** russellb has joined #openstack-infra18:27
*** adreznec has joined #openstack-infra18:27
*** mfink- has joined #openstack-infra18:27
*** SotK has joined #openstack-infra18:27
*** tsufiev has joined #openstack-infra18:27
*** mrda has joined #openstack-infra18:27
*** ianw has joined #openstack-infra18:27
*** dmellado has joined #openstack-infra18:27
*** ddieterl_ has quit IRC18:27
*** tlbr has joined #openstack-infra18:27
*** radez_g0n3 is now known as radez18:27
*** slagle has quit IRC18:27
*** slagle has joined #openstack-infra18:27
*** emagana has joined #openstack-infra18:28
*** rcarrillocruz has joined #openstack-infra18:28
*** phschwartz has joined #openstack-infra18:28
*** yamamoto has joined #openstack-infra18:28
*** larsks|alt is now known as larsks18:28
mordreddevananda, SpamapS: isn't there a useful schema diff tool in maatkit?18:28
*** Shrews has joined #openstack-infra18:28
zarojeblair: i don't think you can even run old version on current schea18:28
*** tmcpeak has quit IRC18:29
*** annegentle has quit IRC18:29
*** dan| has joined #openstack-infra18:29
jeblairzaro: mordred is looking into the differences and evaluating whether it's possible/advisable and working on constructing a downgrade plan18:29
*** annegentle has joined #openstack-infra18:29
*** bkero has joined #openstack-infra18:29
mordredI've got a list of the data transformations - I'm going to grab the db backup, and a current dump and see if percona-toolkit can make me a schema diff18:29
jeblairzaro, fungi, clarkb: what jgit version were we running with 2.8?18:30
mordredat least one of the migrations invovled cross-referencing with file information from git though ... so I'm not going to be 100% happy with doing this18:30
* clarkb checks18:30
*** bkero has quit IRC18:30
*** bkero has joined #openstack-infra18:30
mordredbut - working on it18:30
clarkbjeblair: VERS = '' appears to be the version on 2.8 HEAD18:30
zaroyup, i see the same18:31
fungii wonder if downgrading jgit with gerrit 2.10 would work18:31
*** jodah- has joined #openstack-infra18:31
*** Krinkle is now known as Krinkle|detached18:32
*** jodah- has quit IRC18:32
zaroi can try to do a build with older version to see if it even builds18:32
*** Krinkle|detached is now known as Krinkle18:33
fungizaro: i think that would be a useful datapoint18:33
*** ibiris_away is now known as ibiris18:33
zarook trying now18:33
*** yamamoto has quit IRC18:33
*** jodah has joined #openstack-infra18:33
*** ildikov has quit IRC18:34
*** e0ne has joined #openstack-infra18:35
*** Sukhdev has quit IRC18:35
fungi#gerrit is a complete ghost town18:36
*** ZZelle has joined #openstack-infra18:36
zaroyou'll get quicker response from mailing list.18:36
jeblairi started looking into where the zlib-related exception is coming from18:39
jeblairthere is a difference in that function between jgit 3.2 and 3.718:39
*** annegentle has quit IRC18:40
*** yamahata has quit IRC18:40
*** annegentle has joined #openstack-infra18:41
*** otter768 has joined #openstack-infra18:41
*** ibiris is now known as ibiris_away18:41
fungiintroduced in v3.5.0.201409071800-rc118:42
fungifrom what i can see18:42
jeblairi just added a link to the EP with the commit that changed that function18:42
clarkbjeblair: big difference I see is old code checked that dstoff is less than length18:43
clarkbjeblair: current code seems to assume everything is aligned and will == length to end18:43
fungiyeah, that's the commit i was looking at. so definitely new in 3.518:43
clarkbbut it only checks that if headerOnly is set18:43
mordredI just installed something from CPAN18:44
mordredI havent' done that in so long18:44
jeblairclarkb: have you gotten jgit-cli to barf yet?18:46
*** otter768 has quit IRC18:46
SpamapSmordred: maatkit is percona toolkit now18:46
mordredSpamapS: yah. and best I can tell there is no schema diff in the version that's in ubuntu at least18:47
clarkbjeblair: no, sorry haven't built a local copy of the exact version will try that now (I got completely confused by BUCK)18:47
SpamapSmordred: yeah I don't remember one.18:47
mordredjeblair: note for next time - when we do the backup - we should make 2 backups - one that's the backup, and one that's a schema-only backup18:47
mordredjeblair: as I currently have no choice but to restore all the data on my laptop as well18:47
jeblairmordred: ++18:47
SpamapSgrep -v ^INSERT ?18:48
mordredSpamapS: not a bad idea!18:48
radezanyone know if there is openid connect support on
* mordred gives SpamapS a jellybean18:48
SpamapSmordred: unfortunately backups were done with systemd so they're binary and you're screwed </troll>18:48
*** ddieterly has joined #openstack-infra18:48
fungiradez: smarcet or mrmartin may know18:49
clarkboh jgit uses maven18:49
radezthx fungi18:49
fungiradez: i think the answer is "no" but i'll defer to them18:49
*** dan| has quit IRC18:49
radezsmarcet: mrmartin: does support openid connect?18:49
fungi(or rather, "not yet")18:49
*** tmcpeak has joined #openstack-infra18:50
clarkbjeblair: I have a jgit build going now18:50
radezI've got Kilo happy on the new TryStack gear and wanted to integrate instead of facebook for the auth18:50
*** BharatK has quit IRC18:50
*** baoli has quit IRC18:51
jeblairclarkb: i'm curious if you cat jgit-show 4ad6d89dc9ea435eef448a15a301c9ae4e7e311818:51
jeblairclarkb: s/cat/can/18:52
clarkbfor those following along gerrit apparently uses buck to build jgit but jgit itself uses maven18:52
*** harlowja has quit IRC18:52
*** harlowja has joined #openstack-infra18:52
mordredoh wow18:52
mordredof course it does18:52
jeblairclarkb: what version of jgit-cli did you install earlier?18:53
mrmartinradez: hi, as I know, not yet18:53
*** Krinkle is now known as Krinkle|detached18:53
radezmrmartin: kk, thx18:53
mrmartinradez: just plane openid and oauth18:53
clarkbjeblair: the version shipped by trusty18:53
*** soren_ has joined #openstack-infra18:53
mordredok - that's what mysqldiff says is the difference between the old and new dbs18:54
mordredthat does NOT include the data transformations18:54
radezmrmartin: and only oauth2 correct? not v1?18:54
jeblairclarkb: is there a jgit show command?18:54
*** dan_ has joined #openstack-infra18:54
jeblairclarkb: if so, can you verify that 'jgit show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118' works?  (also, are you doing this on an exact replica of the repo on review?)18:55
*** dan_ is now known as Guest2558018:55
zaroclarkb: gerrit builds jgit?18:55
jeblairclarkb: i would expect it to work since that version of jgit is pretty close to what we ran with 2.8; i will be very happy if you are able to build a jgit >= 3.7 and 'jgit show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118' fails18:55
mrmartinfungi, all of the askbot-staging patches available now in the gerrit queue:
zaroclarkb: doesn't only reference it as a dependency with buck?18:56
clarkbjeblair: I was using a git clone from git.o.o18:56
clarkbjeblair: I can try on a copied replica too18:56
zaroi'm having a hard time building gerrit with older ver
jeblairclarkb: yeah, i think an exact replica would be best18:57
*** moshele has left #openstack-infra18:57
mrmartinbasically we did a guerrilla staging site setup, and the deployment works well there, it is consuming askbot directly from github repo and we can fine-tune the branches18:57
clarkbjeblair: that ref doesn't seem to exist, is that an object for a ref?18:57
mrmartinbut the ideal way would be to move it entirely under infra and also upgrade the ask.o.o to match the new branching model18:57
jeblairclarkb: i think it's a blob18:58
clarkbjeblair: ok, git show only works against commits and refs right? probably need to figure out what the blob belongs to and go from there18:59
jeblairclarkb: git show works on blobs, when i do that with plain git on an rsync of the nova repo, i get a file18:59
jeblairclarkb: something about ec2 :)18:59
clarkbah ok18:59
fungimrmartin: excellent news. i'll try to have a look once we have our gerrit upgrade issues settled19:00
mrmartingreat, thank you19:00
clarkbjeblair: ~corvus/nova.git is your copy?19:01
*** kgiusti has joined #openstack-infra19:02
jeblairclarkb: yep19:03
jeblairgit cat-file -t 4ad6d89dc9ea435eef448a15a301c9ae4e7e311819:03
*** Rockyg has joined #openstack-infra19:03
*** jamesmcarthur has quit IRC19:03
*** tmcpeak has left #openstack-infra19:03
jeblair(so yes -- confirmed it's a blob)19:03
*** yamahata has joined #openstack-infra19:04
*** emagana has quit IRC19:05
*** emagana has joined #openstack-infra19:05
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Remove custom taskflow sqlalchemy environments
clarkbjeblair: `org.eclipse.jgit.pgm/target/jgit --git-dir ~/tmp/jgit-debug/nova.git show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118` works with my local jgit version that was built using the version 2.10 is using19:06
pabelangerjeblair, are you fine with grafyaml living under openstack-infra like JJB does?19:07
jeblairclarkb: :(19:07
jeblairpabelanger: yep19:07
pabelangerjeblair, Roger19:08
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Remove extra unused environments from tooz jobs
jeblairclarkb: i think i want to restart gerrit again and see if we get the same error in the same file at the same offset19:08
clarkbjeblair: ok19:09
openstackgerritJulia Kreger proposed openstack-infra/project-config: Add bifrost to OpenStack
clarkbjeblair: also, considering jgit show works it is possible this is a bug in how gerrit uses jgit19:09
fungisort of wondering the same. if we can confirm this is persistent vs random we might be slightly closer to finding the conditions under which it emerges19:09
jeblairi copied the nova.git i have been using to:
mordred <-- this is the latest - I stripped the autoinc changes out, and I added in or accounted for the data transformations19:10
jeblairso if anyone else wants to download that and try things out locally ^19:10
mordredwith 3 exceptions that are noted at the bottom, which I'm now working on19:10
fungithanks--that looks quite concise19:10
fungii was expecting something hairier19:11
jeblairadded to etherpad19:11
jeblairrestarting gerrit19:11
clarkbgerrit itself does not seem to use the WindowCursor or inflate method, so unlikely a gerrit api consumption issue19:11
*** eharney has quit IRC19:12
*** dguerri is now known as _dguerri19:12
*** soren_ has quit IRC19:12
*** BharatK has joined #openstack-infra19:13
*** panda|afk has quit IRC19:13
jeblairso now we just need to wait (probably 20 mins at most) for something to trigger it19:13
clarkbalso if anyone else builds jgit you will need maven3 not maven219:13
*** panda|afk has joined #openstack-infra19:14
jeblairjava.lang.StringIndexOutOfBoundsException: String index out of range: -1719:15
jeblairthose exceptions are curious.  i am assuming they are not related.19:15
mordredI agree. -17 is out of range19:15
clarkbmordred: can you check the old db dump for SYSTEM groups?19:17
fungii'm working on identifying what commit is responsible for the 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 blob, to see if we can retrigger it faster19:17
clarkbmordred: I want to say we have just a couple and could add them back easily enough19:17
pabelangerjeblair, So, I assume I would open a review into openstack/governance first adding grafyaml under the openstack-infra namespace?19:17
mordredclarkb: yes - that's what I'm doing now19:18
clarkbmordred: registered users and administraors?19:18
*** smccully has quit IRC19:18
fungipabelanger: yep, and then mention it in the project-config change19:18
jeblairpabelanger: you can create the repo in project-config first and have the governance change depend on it19:18
mordredclarkb: (I did SpamapS's trick to only load schema before - loading data now so that I can poke at those data changes)19:18
pabelangerjeblair, fungi: okay19:18
jeblairpabelanger, fungi: we reversed that for project additions to existing programs19:18
AJaegerpabelanger: all explained in
pabelangerAJaeger, thanks, that's what I was looking for19:19
fungijeblair: yep, so no need to use a depends on in the project-config change, but having a pointer to the governance change is still useful19:19
AJaegerpabelanger: jeblair's comment is in
*** freerunner has joined #openstack-infra19:19
*** hamzy_ is now known as hamzy19:22
*** e0ne has quit IRC19:22
clarkbI am going to grab lunch while we wait on post restart results19:22
*** btully has joined #openstack-infra19:25
*** spzala has joined #openstack-infra19:26
*** BharatK has quit IRC19:26
*** jtomasek has quit IRC19:27
fungifwiw, which was blank before the restart has content again19:27
*** BharatK has joined #openstack-infra19:27
*** rlandy_ has joined #openstack-infra19:28
*** btully has quit IRC19:29
*** rlandy has quit IRC19:30
mordredok. I updated the migration script19:30
mordredI believe it's complete now19:30
mordredand not particularly scary19:30
mordredI believe we'll also need to manually delete the All-Users.git repo19:30
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Templatize zuul: Use python*-jobs
jeblairmordred: just delete it from disk?19:32
mordredyeah. I don't believe it has entries anywhere else19:32
jeblairmordred: or are there any database tendrils... ok19:32
mordredjeblair: now - obviously running that script without proper testing is mildly terrifying19:33
mordredso I'm still hoping that you find jgit solutions that make it not needed19:34
fungiwell, running that script shouldn't be entirely terrifying. we'll backup the database before we try it19:35
fungithough testing separately first would still be good19:35
*** koolhead17 has joined #openstack-infra19:35
*** koolhead17 has quit IRC19:35
*** koolhead17 has joined #openstack-infra19:35
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Stop leaking server objects
*** _dguerri is now known as dguerri19:38
*** j^2 has joined #openstack-infra19:42
fungiaha! my recursive digging has turned up "b286872 Allow volume filtering" as the commit responsible for that blob19:43
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Single event email worker.
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Factory subject is now a template file.
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Revert "Email Send Cron Worker"
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Revert "Email Outbox"
j^2what is the process to open up #openstack-meeting-5 ?19:43
fungij^2: we'd need to register it. we've had some backpressure on people sticking to somewhat compatible meeting times by limiting the number of channels, but as many meetings as we've added it may be time to consider a 5th channel for them now19:45
j^2fungi: yeah, i’m trying to get the openstack-chef meeting at 1500UTC and it seems that all the meetings are full on Mondays:(19:46
*** hashar has joined #openstack-infra19:46
j^2there is no reason why we could have N nummber of meeting rooms right?19:47
fungij^2: other than then everyone would schedule their meetings in the same timeslot and it would become a lot harder to participate in multiple meetings because of increased conflicts19:47
j^2ahh makes sense19:47
fungiand also it increases the number of channels we need to lurk in should something come up, though that's a lesser issue19:48
j^2ohh yeah that is a challange19:48
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add grafyaml into openstack-infra
fungij^2: if there aren't any great compromise options open for a new meeting time, i'd recommend starting a thread on the openstack-dev ml to gauge interest in/concern over growing the count of meeting channels again19:49
fungisince it's a bit of a balancing act, but we do acknowledge that we need to increase them from time to time19:49
*** oomichi has joined #openstack-infra19:49
j^2fungi: the challange is that getting sacntioned we are requeride to use the meeting rooms, which we havent been doing, but we have a community that is used to meeting at a specific time19:50
j^2if that makes sense19:50
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: NPM-Run jobs are now voting.
*** oomichi has quit IRC19:50
fungithe infra team is not the (sole) decision maker over that, so it's useful to get some additional input as it affects the whole community to a degree19:51
j^2hmm ok19:51
*** mrmartin has quit IRC19:52
mordredI just did a scan - I will say that the overlap betweenthe current 1500UTC Monday meetings and chef is pretty low19:52
mordredin terms of exclusion of people who'd want to be in both19:52
clarkbI wonder, what if we are looking at the wrong level. Could it be that zlib has the bug?19:52
jeblair[2015-05-11 19:52:42,639] WARN : Cannot update gitlinks for openstack/openstack,refs/heads/stable/grizzly:python-openstackclient follows openstack/python-openstackclient,refs/heads/master due to The branch was probably deleted from the subscriber repository19:53
jeblair[2015-05-11 19:52:42,642] INFO : Deleted incorrect submodule subscription(s) [openstack/openstack,refs/heads/stable/grizzly:python-openstackclient follows openstack/python-openstackclient,refs/heads/master]19:53
*** koolhead17 has quit IRC19:53
mordredbut - let me tell you something I don't want to get into the business of19:53
krotscheckclarkb: Did vinz ever get backed?19:53
j^2mordred: yeah it’s a handful of meetings, all the offical channels are taken up :)19:53
jeblairmordred: ^ maybe file that for later19:53
mordredjeblair: oh my19:53
fungiokay, so in theory trying to access commit b286872 (Change-Id: Icd915f30be8120a9c3fde602be66b442e4460709) should cause gerrit to rerequest our previously failing blob 4ad6d89dc9ea435eef448a15a301c9ae4e7e311819:53
fungishould i try it?19:53
jeblairfungi: go for it19:53
*** tvander has joined #openstack-infra19:54
clarkbhrm that comes from so likely not at issue and I should have the same version as review.o.o19:54
clarkbkrotscheck: no :/19:54
*** Krinkle|detached is now known as Krinkle19:54
krotscheckclarkb: sadpanda19:54
fungino dice
mordredclarkb: as in - they chose to not back it? or they have not chosen?19:54
crinkleis the infra meeting agenda the authoritative list for infra's priority efforts? i found but it seems a bit out of date19:54
jeblairfungi: thx19:54
jeblairclarkb: yes, the agenda is the current auth list19:55
jeblaircrinkle: ^ sorry19:55
crinklejeblair: thank you19:55
*** tvander has left #openstack-infra19:55
clarkbmordred: we did not get selected19:55
clarkbmordred: I think even college seniors are smart enough to avoid the javascript :)19:55
fungicrinkle: we sort of semi-intentionally allow it to dwindle as the summit approaches, so that there's room to fill it back up with the new priorities we settle on19:56
greghaynesyeaa :/19:56
clarkbfungi: so I take it that means gerrit hasn't fallen over on nova yet?19:57
fungikrotscheck: in good news, we have an outreachy intern starting soon. you could try to convince them that's an interesting project ;)19:57
*** freerunner has quit IRC19:57
*** pabelanger has quit IRC19:57
zaroi think i might have figured out how to build with older version of jgit if we are still interested in that.19:57
clarkbzaro: perhaps put details on ?19:58
fungiclarkb: seems not. so viewing the change,patchset responsible for that blob, including looking at diffs of the files changed by it, doesn't seem to have retriggered the failure we saw earlier19:58
fungiclarkb: suggesting that it'19:58
fungis not a problem with that particular packfile i think?19:58
*** jamesmcarthur has joined #openstack-infra20:01
jeblairfungi: it seems that way; so maybe something about gerrit's internal state causes that method to fail...20:01
*** spredzy|afk is now known as spredzy20:02
*** Krinkle is now known as Krinkle|detached20:03
*** dims has joined #openstack-infra20:03
*** pabelanger has joined #openstack-infra20:04
*** pabelanger has joined #openstack-infra20:04
*** sabeen3 has quit IRC20:05
*** tiswanso_ has quit IRC20:06
krotscheckclarkb: I will shiv you20:06
*** mwagner_lap has quit IRC20:07
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Add non-voting coverage job to sahara check pipeline
openstackgerritKhai Do proposed openstack-infra/gerrit: Use jgit ver 3.2
krotscheckWe should probably just kill the project then, so that it doesn't suck up resources.20:08
jeblair[2015-05-11 20:08:00,161] WARN : Plugin </home/gerrit2/review_site/plugins/javamelody.jar> was disabled, because another plugin </home/gerrit2/review_site/plugins/javamelody-4744bfb.jar> with the same name <javamelody> already exists20:08
jeblairwhat ^?20:08
jeblairzaro: maybe a puppet bug there ^.  no idea why it emitted that _now_.20:09
clarkbkrotscheck: I don't think its sucking up any resources currently beyond what they will suck up if moved to the attic20:10
*** sdake_ has joined #openstack-infra20:10
krotscheckclarkb: Well, it's currently in my way because it's using js-build templates that I want to get rid of.20:10
krotscheckAnd, well, the repo is empty.20:10
zaroohh odd, i would expected to see that during the upgrade as well.  i'll take a look.20:10
jeblairkrotscheck: just remove the jobs20:10
zarobut quick fix is to just remove older one from plugins folder20:10
krotscheckjeblair: That's what I'm doing right now20:11
*** doude has quit IRC20:11
jeblairokay heads up, i'm seeing missing commits20:11
*** soren_ has joined #openstack-infra20:11
openstackgerritMonty Taylor proposed openstack/requirements: Add appdirs requirement
*** doude has joined #openstack-infra20:11
*** _nadya_ has joined #openstack-infra20:11
mordredjeblair: ooh!20:11
sdagueso.... I'm going to assume this is all part of the jgit bug, but my search dashboards are returning a dramatically small number of hits (mostly missing stuff from April 28th forward)20:12
zarojust built gerrit with older jgit, pushed a change, updated etherpad, and now testing it out.20:12
jeblair[2015-05-11 20:10:48,957] ERROR : Pack file /home/gerrit2/review_site/git/openstack/neutron.git/object20:12
jeblairs/pack/pack-5d1fdaf97301469c0d4cd0233b3dee590fed72bc.pack is corrupt, removing it from pack list20:12
jeblairorg.eclipse.jgit.errors.CorruptObjectException: Object at 88,537,491 in /home/gerrit2/review_site/git/openstack/neutron.git/objects/pack/pack-5d1fdaf9730146920:12
jeblairc0d4cd0233b3dee590fed72bc.pack has bad zlib stream20:12
sdaguein case that is a useful piece of information20:12
clarkbinteresting thing I notice about the jgit code, is it keeps a long start position and long current position in its windows but then when it takes the delta between them casts to an int20:13
jeblairsdague: thx, we're seeing a lot of failures; i would actually expect your dashboard to completely fail.  it might be related or there might be something else going on.20:13
clarkbI suppose if the pack file is gigantic we could be running into 2^31 rollovers in java ints20:13
sdaguejeblair: it mostly works, just a lot of stuff missing when I try to filter by label:20:13
jeblairclarkb: when it outputs that error, is that the long or the int?20:13
*** sdake has quit IRC20:13
sdaguethat seems to be the trigger for the filters dumping a ton of stuff into the void20:14
jeblairso the latest error is in a different repo, but has a similarly high index20:15
clarkbjeblair: when it does position += window.setInput(position, inf); The return value of setInput is an int20:16
*** _nadya_ has quit IRC20:16
clarkbif it rollsover we could in theroy be adding a negative value, this moves position backwards and the inflator may return n = 0 at that point20:16
* clarkb is reading docs on the inflator20:16
fungiso far we've seen this hit nova and neutron which are "really big" (commit, change and history wise) repos so that seems like a possible correlation20:17
clarkboh and inflate can throw its own DataFormatExceptions directly20:18
jeblairclarkb: the length in bytes of the nova pack file is much smaller than 2^31 -- 32572022820:19
*** ildikov has joined #openstack-infra20:19
jeblairclarkb: i assume you mean Inflater.inflate can throw those, yeah?20:21
clarkbjeblair: ya it can throw, but rereading the stacktrace I don't think it did20:21
jeblairclarkb: agreed, it was WindowCursor.inflate that threw20:21
zarolooks like the build with older jgit seems to generally work, i can upload a new patch and merge it thru.  was wondering if anybody wanted this on review-dev for further eval?20:25
*** teran has joined #openstack-infra20:28
jeblairthe neutron error was also on its largest pack file20:29
jeblair(the nova error was on its largest pack file)20:30
clarkbjeblair my java is bad but pretty sure the issue is in the headerOnly check, need to see if that is not set in places20:30
clarkbjeblair: we read dstbuf.length - dstoff bytes each time then update dstoff20:30
clarkbjeblair: eventually that will go to zero, at which point inflating should return zero bytes except if headerOnly is false that means we can be unfinished and not need more input20:31
clarkbnot sure what a good way to test/confirm that is other than maybe dumping the values of dstbuf.length and dstoff somehow (debugger, logger)20:31
jeblairclarkb: yeah, or possibly constructing an artificial test case if we think we know what kind of input may trigger it20:32
clarkbjeblair: note that the old code properly ended when that buffer was full without raising20:32
*** pcrews has quit IRC20:33
*** mtanino has quit IRC20:34
openstackgerritJoe Gordon proposed openstack-infra/project-config: Add pep8 job to nova-specs
mikalAm I correct in assuming that we don't bump the versions of dependancies in stable releases?20:34
* krotscheck just found stackforge/merlin20:34
krotscheckZee Javazcreeept, eet ees eeeverywhere!20:35
clarkbjeblair: if (curs.inflate(this, position, dstbuf, false) != sz) is the call whee we raise from20:35
clarkbjeblair: so its not setting true there20:35
mordredmikal: we try our best not to - I imagine we would if we needed to for a security update20:35
clarkbI am not quite sure what spearce added the headerOnly flag for, trying to grok that now20:36
*** radez is now known as radez_g0n320:36
clarkbI mean, if you pass in a buffer and don't grow the buffer for the caller then ou can only ever read in that many bytes...20:36
jeblairclarkb: he said that the buffer needs to be able to support the full decompressed data size20:37
jlvillalDoes anyone know if in project-config/zuul/layout.yaml  if I can use a template with a branch restriction?20:37
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: NPM-Run jobs are now voting.
jlvillalIt was suggested I change my patch to use a template, but I also have a branch restriction.20:38
*** craigbr has joined #openstack-infra20:38
clarkbjeblair: that seems like a really bad way to fail when you don't then...20:38
AJaegerjlvillal: which change?20:38
clarkb"Something wrong with your zip file except not really you actually need a bigger buffer"20:38
jlvillalAJaeger: You were the suggester :)20:38
AJaegerjlvillal: A template just expands, so no restrictions20:39
*** ams__ has joined #openstack-infra20:39
jeblairclarkb: so if you assume that the conditions in the javadoc hold (the supplied buffer is large enough), then i don't think there's a logic error, as inf.finished() should return true20:39
clarkbjeblair: correct20:39
jlvillalAJaeger: I don't understand if that means I can use it with a branch restriction???20:39
mikalmordred: ahhh, ok cool. That's kind of where I was going.20:40
*** baoli has joined #openstack-infra20:40
jeblairclarkb: i agree that if the supplied buffer is not large enough, we'll see the error we got (and that could be because of a programming error or because the stream actually is corrupt and is spewing data)20:40
jlvillalAJaeger: Or should I update my patch to put the check in both the gate and check steps?20:40
clarkbjeblair: there is a lot of masking to determine the value of the buffer size20:41
fungijeblair: clarkb: is that buffer filled with compressed or decompressed data?20:41
AJaegerjlvillal: You should run the voting python34 jobs in both check and gate. And instead of adding it manually in the file, use the template.20:41
clarkbfungi: decompressed (its receiving the inflated data)20:41
*** rmcall has quit IRC20:41
jlvillalAJaeger: What about the branch restriction?20:41
AJaegerThe branch restriction you have will work on the template the same way as it works on the jobs. Just assume that we do template expansion first and then the branch restrictions.20:42
jlvillalAJaeger: Thanks!20:42
*** ildikov has quit IRC20:42
fungiclarkb: okay, that's what it seemed like. so we need to compare the raw sizes i guess, not the current on-disk size20:42
AJaegerThe templates are just macros - syntactic sugar - they don't limit you20:42
*** otter768 has joined #openstack-infra20:42
jeblairwe also have erorrs in ironic-specs, keystonemiddleware, murano-agent and os-collect-config now20:43
jeblairlooking to see if they follow the pattern20:43
*** openstackgerrit_ has joined #openstack-infra20:43
*** baoli has quit IRC20:44
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Have gate-ironic-python34 be a voting job
*** tonytan4ever has quit IRC20:44
jlvillalAJaeger: Done I think :)  ^^^^20:44
openstackgerritKhai Do proposed openstack-infra/gerrit: Use jgit ver 3.2
jeblairthe last two have tiny pack files, which seems to suggest pack file size is not related20:44
*** baoli has joined #openstack-infra20:44
AJaegerjlvillal: that won't work. Leave the branch restriction where it was20:45
AJaegerjlvillal: and remove gate-ironic-python3 from the check queue20:45
jlvillalAJaeger: Okay.  working....20:45
jeblairalso, ironic-specs and keystonemiddleware had 2 errors apiece at different offsets but with the same log timestamp20:46
jeblairpossibly different threads?20:46
*** craigbr has quit IRC20:46
clarkbfungi: yes, and it appears to be doing it per deleta20:46
jeblairin all cases, it's still the largest pack file (also meaning it's probably the most used)20:46
clarkbbut its completely undocumented so hard to follow20:46
*** otter768 has quit IRC20:47
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Have gate-ironic-python34 be a voting job
*** achanda has joined #openstack-infra20:48
jlvillalAJaeger: The layout.yaml file is starting to make sense to me.  Thanks for your help.20:48
openstackgerritSean Roberts proposed openstack-infra/project-config: update akanda acls to support branching
AJaegerjlvillal: LGTM, +220:49 is where the size calculation happens20:49
jogoanywhere I can file UI complaints for the new gerrit?20:49
clarkbjogo: upstream against gerrit20:49
*** tonytan4ever has joined #openstack-infra20:49
* jogo checks config first to see if he can turn off the ugly20:49
jogolisting conflicts with by default is distracting to me20:49
jlvillalAJaeger: Thanks20:49
fungijogo: example?20:50
clarkbfungi: thats new change screen iirc20:50
fungijogo: is that just in the new change screen, or old change screen but new behavior in 2.10?20:50
fungiahh, the former. got it20:50
clarkbfungi: it tells you what other changes you conflict with20:50
ZZelleHi everyone20:51
ZZellei get the following error in gerrit:20:51
ZZelleorg.eclipse.jgit.errors.MissingObjectException: Missing unknown 8cbfa31e60d5d202da4fa89f942fb17704f52fea20:51
clarkbjeblair: but basically the only way I can make it raise that error while not being finished or needing more input is if the buffer provided is not large enough20:51
jogofungi: only checked the new screen20:51
*** Sukhdev has joined #openstack-infra20:52
jeblairjogo: can it wait until we have dealt with the git problems?20:52
jeblairZZelle: yep, that's the error we're working on (referenced in channel topic)20:52
ZZellejeblair, neutron is also concerned20:53
jogojeblair: very much so yes, that is why I was hoping there was some place I could put a comment20:53
*** claudiub has joined #openstack-infra20:53
jeblairjogo: upstream gerrit issue tracker?20:53
clarkbI hvae a doctors appointment I need to go to now, back after20:54
*** carl_baldwin has joined #openstack-infra20:55
jogojeblair: doing that now20:55
*** tnovacik has quit IRC20:55
*** claudiub|2 has quit IRC20:55
*** sks has joined #openstack-infra20:56
sdaguejogo: honestly, I really like having conflicts with. That's going to help a lot in pruning duplicate fixes in the nova backlog.20:56
fungiso, i'm seeing that gerrit has a core.packedGitWindowSize config option "Default on JGit is 8 KiB on all platforms."20:56
fungishould we be considering tuning that?20:56
*** ddieterly has quit IRC20:56
sdagueit's also good to see that every vmware patch is a 20 way conflict with all the other ones.20:57
zarojogo: what do you want to see differently?  turned off?20:57
jeblairzaro: can we try to keep the discussion related to the current problem please20:57
openstackgerritRuslan Kamaldinov proposed openstack-infra/project-config: Add functional test job for python-muranoclient
*** ildikov has joined #openstack-infra20:58
jeblairi will set up #openstack-infra-incident for the next time this happens, but it is not logged now, and i think it's important that this be logged20:58
fungialso core.packedGitLimit, core.deltaBaseCacheLimit, core.packedGitOpenFiles, core.streamFileThreshold, core.packedGitMmap...20:58
fungiwondering if maybe android's gerrit has those tuned to larger values and so isn't running into this20:59
jeblairfungi: if i understand the windowcursor correctly, i think the windowsize just means the size of reads that it performs within the inflate loop20:59
jogozaro: yeah or at least minimized to just a number that you can click on to expand20:59
fungiahh, okay20:59
*** moshele has joined #openstack-infra21:00
*** ociuhandu has quit IRC21:00
jogosdague: yeah it should help a lot for duplicates, I just don't know if I want to see a massive list every time21:00
* jogo wishes gerrit had a better UX21:00
jeblairsdague, jogo, zaro: please take that discussion to another channel21:00
jeblairit's seriously distracting21:00
jogojeblair: sorry21:00
*** marcusvrn has quit IRC21:01
anteayajeblair: I'm sort of out of the loop but trying to catch up, tell me if there is anything I can do to help21:01
jeblairfungi: i don't understand any of the other constants you mentioned; do they look likely to be related?21:02
*** mwagner_lap has joined #openstack-infra21:02
jeblairfungi: i think the fact that we haven't seen this in exactly the same place twice suggests that there's some transient operational aspect involved and maybe not a simple "constant not big enough" error21:03
openstackgerritMerged openstack-infra/project-config: Adds pypi-jobs to cloudbase-init
*** eharney has joined #openstack-infra21:03
*** sabeen1 has joined #openstack-infra21:04
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added NPM-based unit tests for merlin
*** sdake has joined #openstack-infra21:04
openstackgerritSean Roberts proposed openstack-infra/project-config: update akanda acls to support branching
jeblairfungi, mordred, zaro: when should we make a decision to revert or not?21:05
fungijeblair: agreed, since the impact is seemingly random i don't suspect this is intended/expected behavior from using default tuning values. just wondering if of the large gerrit implementations out there running 2.10+ others haven't hit this issue because they have some of these tuned to non-default values21:05
*** sabeen2 has joined #openstack-infra21:05
mordredjeblair: I think we should do it with enough time that we have time to do it before west coast EOD21:05
*** hichihara has joined #openstack-infra21:05
*** dustins has quit IRC21:05
mordredotherwise we're going to be trying to do it completely burned21:05
zarono interest using gerrit with older jgit?21:06
*** shardy is now known as shardy_z21:06
openstackgerritSean Roberts proposed openstack-infra/project-config: update akanda acls to support branching
openstackgerritMonty Taylor proposed openstack-infra/project-config: Register openstack-infra-incident with accessbot
openstackgerritMonty Taylor proposed openstack-infra/system-config: Log openstack-infra-incident
fungiif gerrit 2.10 with jgit 3.2 works (zaro's test build) it might be worth giving that a go to get us through until we can get some feedback from gerrit devs, but there's a good chance that they bumped the included jgit because newer gerrit depends on newer jgit features21:07
openstackgerritRuslan Kamaldinov proposed openstack-infra/project-config: Add functional test job for python-muranoclient
jeblairfungi, zaro: yeah, i'm worried there a design issue lurking in there, even if it appears to work...21:07
*** sdake_ has quit IRC21:08
fungiif i were more confident we might reliably reproduce this condition on review-dev i'd be inclined to start the rollback immediately21:08
jeblairfungi: i think to do that we will need to make -dev a full production staging environment21:09
*** sabeen1 has quit IRC21:09
fungimy worry is that even with the same data and configuration and some sort of fabricated load generation we'll be missing a key element to get it to happen there21:09
jeblairfungi: very possible21:09
mordredbut - it does motivate making it a real staging env21:09
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Removed old js-based test job.
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Deprecated js-test build.
*** moshele has quit IRC21:10
*** mtanino has joined #openstack-infra21:10
*** annegent_ has joined #openstack-infra21:10
*** tonytan4ever has quit IRC21:11
*** esker has quit IRC21:11
fungiso, that said, i'm at a loss as to what else to try to get this back on track. unless anyone else has ideas, we should probably start the rollback21:13
*** pcrews has joined #openstack-infra21:14
fungiis there any additional data about the failure we should be considering collecting which would be destroyed when we downgrade?21:14
* anteaya stands by to keep the channel clear21:14
*** annegentle has quit IRC21:15
*** baoli has quit IRC21:15
pleia2anteaya: yeah, -incident will be nice moving forward to avoid the interruptions :)21:15
fungialso, what's the jgit version in latest gerrit 2.9.x? (not that we should likely consider trying to downgrade to 2.9, nor immediately upgrade from 2.8 to 2.9 before we have more time to plan/test)21:15
mordredfungi, jeblair: I'm the more insane of us - but I'm tempted by the new-gerrit-old-jgit - purely because java is strongly typed ... and I can't imagine it being much worse21:15
anteayapleia2: agreed21:15
anteayapleia2: didn't want to side tracking making it now though21:15
*** baoli has joined #openstack-infra21:15
pleia2anteaya: need to wait until it's logged anyway21:15
mordredreviews welcome on those two patches...21:16
jeblairfungi, zaro: fwiw, review-dev has logged errors relating to packs being deleted (which we think is related to repack and we understand), but not zlib errors21:16
pleia2they look fine, just waiting on jenkins21:16
pleia2feeble human eye review21:16
mordredjeblair: I've lost track - which combo is review-dev running now?21:17
*** rlandy_ has quit IRC21:17
jeblairzaro: ?21:18
zaroi haven't changed anything.21:18
fungizaro: so gerrit 2.10 and jgit 3.7 still, i think21:18
jeblairmordred: but that was a few weeks ago, so i'm not actually sure what was running at the time21:18
fungisort of wishing the person who reported!topic/repo-discuss/fTx5piOjcJ4 had followed up in the past week21:18
*** sks has quit IRC21:18
fungijkt: ^ you commented on that post as well21:19
zaroit was 2.10 & jgit 3.7 as well21:19
zaro2.10 has been running on review-dev for a while21:19
*** tonytan4ever has joined #openstack-infra21:19
*** mpopow has quit IRC21:20
mordredso - maybe start 2.10 + 3.2 on review-dev real quick and make sure it doesn't just flat delete things21:21
*** mpopow has joined #openstack-infra21:21
mordredand if it doesn't delete things, try running it on review instead of 3.7 and see if the error triggers, if not, rollback?21:21
jeblairi think i can live with that if that doesn't terrify anyone else21:22
mordredbasically aLL of of the options terrify me21:22
jeblairfungi: thoughts?21:22
fungii'll just make sure to have a teddy bear and a security blanket handy21:22
fungibut i'm in21:22
fungii'm here all night21:22
fungiwe have backups21:22
fungiwors that happens is people reupload changes from the past couple days21:23
zaromy build is on review-dev.o.o ready to go21:23
jeblairzaro: go for it21:23
jeblairzaro: do you have a change ready for production?21:23
*** smarcet has quit IRC21:25
*** vhoward has left #openstack-infra21:26
*** bswartz has quit IRC21:27
*** rbradfor has quit IRC21:27
mordredit's got 2 +2s now - +A will build and upload an artifact somewhere, yeah?21:27
jeblairmordred: i +Ad and yes21:28
jeblairhow about i start on a message to the repo list?21:28
*** craigbr has joined #openstack-infra21:29
zarodone.  ver 2.10.2-23-g039a170-dirty21:30
*** dkranz has quit IRC21:30
*** julim has quit IRC21:31
mordredI confirm that that is the dirty version running there21:31
openstackgerritlifeless proposed openstack-dev/pbr: Finish removing invocations of pip.
mordredI can browse thigns at least21:31
*** peristeri has quit IRC21:32
anteayaI can sign in and leave a comment21:33
*** emagana has quit IRC21:34
*** dangers_away is now known as dangers21:34
jeblairpush up a new change; new patchset, etc?21:35
*** emagana has joined #openstack-infra21:35
fungidone and done21:35
fungimerged a couple too21:35
*** waynr has quit IRC21:36
*** jamesmcarthur has quit IRC21:36
zaroi also tested creating new projects on the UI as well.21:39
openstackgerritMerged openstack-infra/project-config: networking-ovn: disable n-novnc
*** Rockyg has quit IRC21:39
openstackgerritMerged openstack-infra/project-config: Add create rule to networking-mlnx
*** cdent has quit IRC21:41
adam_ganyone know where to configure the list of channels that the 'openstack' lp bug bot hangs out in?21:41
*** mattfarina has quit IRC21:42
*** hashar has quit IRC21:42
adam_gfungi, ah, cool21:43
*** achanda has quit IRC21:43
adam_gfungi, while ive got you: how does the gerrit bot resolve a gerrit patch's 'Closes-bug' to LP project? does it just go by $repo_name->$lp_project?  wondeirng how i can get several sub-projects updating bugs in a single place, similar to neuron  and the neutron-* sub-projects21:44
*** baoli has quit IRC21:44
fungiadam_g: it's configurable. come back when we're not trying to work through gerrit upgrade issues21:44
greghaynesfungi: Hypothesis maybe for why you couldnt re-trigger the pack corrupt error - when gerrit "removes from packlist" does gerrit keep the ObjectDirectory instance alive with the removed reference to that pack?21:44
adam_gfungi, sure, thanks21:45
bhunter71a question; what's the basic process upstream follows when creating a new stable release branch?   or, more specifically, are the branches created in gerrit first, and then replicated to
jeblairgreghaynes: i think when fungi was trying to retrigger it, it was after a restart?21:45
greghaynesah, if so then there goes that theory21:45
anteayabhunter71: our workflow is that gerrit is the canonical repo and git.o.o and github are mirrors21:46
fungiyeah, so far i'm not seeing any problems using review-dev with older jgit21:46
bhunter71anteaya; thanks.  Are there cases where you have to manually 'gerrit replicate', or is it smart enough to hook the branch creation event?21:47
fungierrors in its log seem to just be due to missing/misconfigured hooks, so nothing alarming so far21:48
fungibhunter71: it happens automatically. same with tags21:48
anteayabhunter71: I don't follow, our toolchain tests and merges to the canonical git repos on gerrit, once a merge has taken place in a repo the change is replicated to the mirrors21:48
zarofungi: anything issues with replication?21:48
anteayabhunter71: are we talking hypothetically or do you have a problem?21:49
fungizaro: no replication errors that i can see either, no21:49
jeblairfungi, zaro, mordred: draft message in etherpad21:49
*** funzo has quit IRC21:49
fungizaro: hrm... though changes that i merged aren't showing up in github
*** sdake has quit IRC21:50
mordredjeblair: lgtm21:51
anteayahave we decided to downgrade our jgit on review.o.o?21:51
anteayaI didn't see the draft email mention that21:51
anteayadid it and I missed it?21:51
mordredjeblair: maybe ... when you mention the pack file, you make it sound like it's related to one pack file - rather than that it can happen in any repo but when it does the file is the same?21:51
*** baoli has joined #openstack-infra21:52
jeblairHost not found: 3(NXDOMAIN)21:52
*** annegent_ has quit IRC21:52
jeblairready scripts are failing with that error21:52
*** pcrews has quit IRC21:52
fungijeblair: to disambiguate this from recent discussion about stale file handles on nfs, maybe mention that this is a regular (ext4) filesystem?21:52
jeblaircan someone fix that please? ^21:52
* ianw still catching up ... is pushing changes, etc, generally making things worse, or is this only affecting some git trees?21:52
*** mpopow has quit IRC21:52
zaroanteaya: msg is to gerrit mailing list.21:52
bhunter71anteaya: I think you're saying that if I create a branch in the gerrit UI, it will just replicate automaticaly (whereby my replication is currently explicitly only the master branch, but when I lift that, it should just work)21:52
*** sdake has joined #openstack-infra21:52
anteayazaro: ah sorry, I thought it was to -dev21:53
*** annegentle has joined #openstack-infra21:53
anteayamy mistake21:53
*** jamesmcarthur has joined #openstack-infra21:53
jeblairmordred, fungi: ack will address21:53
fungiianw: completely unknown21:54
openstackgerritMonty Taylor proposed openstack-infra/project-config: Revert "Configure to use gem mirrors"
fungimordred: thanks21:54
anteayamy replication is currently explictly only master branch, where did you specify that gerrit is to replicate only your master branch to the mirrors?21:54
zarofungi: IIRC there's a bit of a delay from when the change merges before it appears on github.21:55
mordredsorry. that was my bad - I left off teh depends-on21:55
*** kgiusti has quit IRC21:55
*** mriedem is now known as mriedem_away21:56
morganfainbergfungi, clarkb, do we have a definitive list of repos that are under investigation post gerrit upgrade?21:57
fungimorganfainberg: it doesn't seem to be repo-specific21:58
* morganfainberg isn't seeing a ML topic...but might be blind.21:58
mordredmorganfainberg: current theory is gerrit and/or jgit issue that's fundamental but not immediate in nature21:58
morganfainbergfungi: ah, ok. Keystonemiddleware is generating 500s for me in the gerrit ui in some cases now which is why i was asking21:58
morganfainbergmordred: ah21:58
fungimorganfainberg: yep, we saw errors related to that repo since the latest restart21:58
morganfainberglbragstad, ^^21:58
morganfainberglbragstad: there is your answer.21:58
fungimorganfainberg: i think so far the only message to the ml was from me in reply to a reply to the maintenance completion e-mail21:59
*** stevemar has quit IRC21:59
morganfainbergok cool.21:59
bhunter71anteaya; sorry- I have some replication config in modules/<openstack_project>/manifests/review.pp21:59
lbragstadah, cool, just checking21:59
bhunter71      push                 => [21:59
bhunter71          '+refs/heads/master:refs/heads/master',21:59
bhunter71          '+refs/changes/*:refs/changes/*',21:59
bhunter71          '+refs/meta/config:refs/meta/config',21:59
bhunter71      ]21:59
morganfainbergjust checking on things.21:59
lbragstadthanks fungi!21:59
anteayabhunter71: use a paste service please21:59
fungiwe don't have any great answers yet, just more questions21:59
morganfainbergfungi, mordred: thanks. will keep an eye on convos here as well21:59
jeblairbhunter71: we're dealing with an incident right now, can you hold your questions for another time please?21:59
bhunter71anteaya: I think the point of clarification is you're saying "once a merge has taken place", and I'm wondering where this merge happens when I'm creating a branch22:00
anteayamy fault sorry22:00
anteayabhunter71: jeblair has asked this discussion to take place later22:00
jeblair(and i imagine we're probably about a day or two away from having our incident channel logged because we can't use any nodes right now)22:00
*** emagana has quit IRC22:00
bhunter71gotcha, sorry, ttyl.  thanks anteaya, out.22:00
*** annegentle has quit IRC22:00
fungizaro: so... the commits i merged 25 minutes ago, not on github and no replication tasks oending in the show-queue output22:01
mordredjeblair: perhaps we should back that change out on nodepool - we're not going to get a ready node to land the read node fix22:01
jeblairmordred: i think if you force-merged your commit it would fix the problem22:01
jeblairmordred: i think you should do that22:01
mordredsorry - that's what I meant - I'm not using english22:01
mordreddoing now22:01
fungizaro: not sure if things are just weird with replication on review-dev for unrelated reasons, or something with this gerrit+jgit version combo22:01
openstackgerritMerged openstack-infra/project-config: Revert "Configure to use gem mirrors"
*** waynr has joined #openstack-infra22:03
mordredjeblair: ok. I kicked puppet on nodepool to boot - the gem change revert ahs been applied22:03
mordredso we should get nodes again22:03
zarofungi: taking a look22:04
openstackgerritMonty Taylor proposed openstack-infra/project-config: Revert "Revert "Configure to use gem mirrors""
mordredok. that can be left for later22:04
*** baoli has quit IRC22:05
*** emagana has joined #openstack-infra22:05
*** ddieterly has joined #openstack-infra22:06
*** bhunter71 has quit IRC22:07
zarofungi: hmm, the replication logs used to be pretty noisy. but don't see anything in the logs now.22:07
*** aysyd has quit IRC22:08
zarofungi: also i remember afs not working so it would throw a bunch of errors, unless that's been fixed of course.22:08
fungizaro: my bad, they are queued22:08
fungii was looking at show-queue from review not review-dev22:08
fungilots of queued up retries to push to github22:08
jeblairmordred, fungi: msg updated; changed paragraphs in bold22:09
fungiso maybe we just don't have the keys puppeted properly there yet22:09
zaroyup, ran into that problem last time22:09
*** Krinkle|detached is now known as Krinkle22:09
zaroi remember there was a change to add the key. but maybe invalid still?22:09
fungijeblair: looks great--hopefully that will get a faster response than the non-response in irc22:10
*** amitgandhinz has quit IRC22:10
mordredjeblair: looking22:10
zarofungi: change to setup keys
mordredjeblair: yes. that's excellent22:11
anteayajeblair: did you want to give timelines? that the upgrade took place Saturday, and perhaps how many restarts today22:11
*** banix has quit IRC22:12
jeblairanteaya: i don't think it would help22:12
fungiwell, today was the first day there was any significant activity on the server since the upgrade22:12
anteayathen everything else makes sense to me22:12
anteayayes was thinking that might be worth conveying22:12
fungiso for all intents and purposes, "immediately after upgrade"22:12
anteayaokay fine, just thought I would mention it22:13
zarojeblair: lgtm22:13
*** moshele has joined #openstack-infra22:13
jeblairanteaya, yeah, it just means i have to spend a paragraph explaining why "two days later" is "immediately", so i'd like to avoid the extra data.  main thing is that when under load, it happens shortly after restarts, and that's in there.22:13
*** whoops has quit IRC22:17
fungizaro: the ssh key on the old review-dev still works, and is not the same as the one on new review-dev22:17
fungii'll fix it real quick so we can make sure it's working as intended with gerrit 2.10 and jgit 3.722:18
jeblairfungi: thx22:18
*** bswartz has joined #openstack-infra22:18
openstackgerritJulia Kreger proposed openstack-infra/project-config: Add bifrost to OpenStack
jeblairmordred: Host not found: 3(NXDOMAIN)22:18
jeblairis still happening22:18
jeblairmordred: oh, it's because ready scripts are built into images, yeah?22:19
jeblairso this is a delete-current-image situation22:19
*** rkukura_ has joined #openstack-infra22:19
mordredon it22:19
mordredwhy did I think ready scripts were copied at run time?22:19
jeblairbecause we keep wishing they were :)22:20
mordredone of these days...22:20
jeblair(i think we can fix that, but later)22:20
* mordred stops hacking on the fix to nodepool22:20
jeblairmordred: so you're deleting <24h old images?22:20
*** claudiub has quit IRC22:20
mordredWOW there are some old images up in here22:21
jeblairyeah, and i think they are the only reason anything is working :)22:21
fungitry not to think about it22:21
*** tim_o has joined #openstack-infra22:21
fungii think gerrit caches this key too, so restarting it on review-dev real fast22:21
*** dimtruck is now known as zz_dimtruck22:22
fungiokay, it's back up and running again22:22
*** rkukura has quit IRC22:22
*** rkukura_ is now known as rkukura22:22
mordredfungi: fwiw, bare-centos6 seems to have not built in a very long time22:22
fungimordred: in hpcloud? rax? both?22:23
fungimordred: nevermind--i'll think about it later ;)22:23
mordredhp for now22:23
*** pcm_ has quit IRC22:23
fungibut thanks for the heads up22:23
fungireplication success
mordredthought for later - graphs tracking image age per image22:24
fungiafter restart it queued up some new replication pushes, but those have completed22:24
*** jgrimm is now known as zz_jgrimm22:24
fungimordred: maybe just a table of images available and their ages as part of the new nodepool status api22:25
*** rwsu has quit IRC22:25
mordredfungi: nodepool delete-all older-than:1day22:25
mordredalso - I'm a moron. why have I not thought about nodepool image-list | sort before now22:25
mordredsince, you know, image ids are autoinc ints22:26
jeblairmordred: i hope you are deleting all that are _newer_ than 1 day22:26
fungiokay, i have no identified usage problems with gerrit 2.10 and jgit 3.2 on review-dev22:26
jeblairokay.  there is never a reason to delete images _older_ than 1 day22:26
mordredjeblair: well, when you want to cause havoc22:26
*** tonytan4ever has quit IRC22:26
jeblairokay, should we just ninja the same build onto production?22:27
jeblairbecause at this point we might be waiting a while to do it the right way, due to the bad images22:27
*** arielb has quit IRC22:27
fungizaro: get up with me later to check that the replication key for review-dev is getting puppeted successfully (since that change to add it merged a couple weeks ago) and that it's entered correctly in hiera22:27
zarofungi: ok22:27
*** soren_ has quit IRC22:27
fungithanks. i'll likely forget otherwise22:27
zaro++ ninja22:28
fungiright now i just copied it from the old review-dev manually to make sure it worked22:28
jeblairalso, i'm thinking we should change the puppet manifest to not reindex on upgrade; i think that's causing us more problems than it's worth.22:28
lifelessanyone have objections to a pbr release happening now ?22:28
fungilifeless: you are a funny one22:28
fungiin for a penny, in for a pound. if we've tested this dirty build of gerrit, then we might as well roll it into production while we're in fact-finding mode22:29
fungiespecially if we're talking an hour or more to get workers to build a clean one from the proposed patch22:29
lifelessfungi: ?22:29
mordredlifeless: we're in the middle of a day-long incident22:29
fungilifeless: channel /topic22:30
anteayalifeless: yes, now would not be a good time22:30
lifelessfungi: ack22:30
mordredlifeless: however, THANK YOU for asking22:30
jeblairfungi: sounds good to me.  you want to do it?22:30
fungilifeless: yes, sorry, i actually thought you were making a joke ;)22:30
zarojeblair: hmm, i think reindex is required *sometimes*22:30
lifelessfungi: nope, twas not :)22:30
fungizaro: going between essentially the same gerrit code just with different jgit lib version?22:31
jeblairzaro: it is sometimes, it's just that whenever we want to make bugfixes like this, we almost never want it.  so it hinders our ability to use puppet the way we want and forces us to circumvent it.22:31
zaroohh i see, you mean just for this time.22:31
jeblairor the last time22:31
jeblair(last 3 times actually)22:31
jeblairanyway, can hash it out later22:32
*** ociuhandu has joined #openstack-infra22:32
fungiso i think as long as i stop gerrit, copy the dirty build to ~gerrit2/review_site/etc/gerrit.war and then start it again, we'll avoid teh reindex for now22:33
fungiand shouldn't need to disable puppet agent?22:33
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: post_test_hook is not executed when the gate test returns an error
jeblairfungi: agreed22:33
fungier, bin not etc22:33
fungibut ykwim22:33
fungicopying it to the server now22:34
zarofungi: you need to run init22:34
fungizaro: ahh, thanks. what's the easiest way to do that? just copy the comment from the gerrit exec?22:35
jeblairwhy run init?22:35
fungidoes it do anything besides redownload libs?22:35
fungiand run migrations and whatever?22:35
fungilike, anything we'd need to actually have updated between these two wars?22:35
jeblair(because if we run init, we'll need to clean up after it before starting)22:36
zarofungi: cmd is 'java -jar $jar_file init --batch --no-auto-start -d review_site'22:36
zaroneed to run init because it will copy jgit libs into review_site22:36
zaroohh wait, maybe it just copies the gerrit.war to review_site22:37
zaroohh so, maybe just copy the war to review_site/bin22:37
fungithat's what i'm doing22:37
zarosafer just to run the init command.22:38
jeblairnot safer22:38
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Allow for git base override
zaroohh ok.22:38
jeblairagain, if you run init, you need to clean up after it22:38
jeblairlet's just go with fungi's original plan, unless someone can state why init needs to run22:38
zaroi'm ok with that. but make sure it's the bin folder not the etc folder as you mentioned earlier22:39
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/system-config: Initial manual for infra-cloud
fungii've initially copied it to ~gerrit2/gerrit-wars/gerrit-v2.10.2.23.g039a170.dirty.war for ease of access22:39
fungiin case it comes up later22:39
fungishould we do it asap, or do we need to send out an update in irc first?22:40
jeblairfungi: let's just do it22:40
fungistoppign gerrit now22:40
jeblairit will not make things much worse for many people :)22:40
*** pcrews has joined #openstack-infra22:40
zaroi need to catch a carpool home, should be back online in 40 mins.22:40
mordredzaro: hopefully there will be nothing interesting going on at that time22:41
fungidirty gerrit build is on its way up now22:41
jeblairtbh, i sort of expect that this will not solve the problem22:41
mordredjeblair: I REALLY hope that you're wrong22:42
fungiwith our luck22:42
jeblairme too.  it's just a hunch.22:42
mordredwell, sadly you tend to have good hunches22:42
fungiconfirmed gerrit is up and working on "2.10.2-23-g039a170-dirty" per the webui22:42
jeblair(based on where we got to looking at jgit changes and possible causes)22:42
*** moshele has left #openstack-infra22:42
*** esker has joined #openstack-infra22:42
*** otter768 has joined #openstack-infra22:43
jeblairthat did not take long22:44
jeblair[2015-05-11 22:14:40,084] ERROR : Internal server error (user openstack-project-creator account 6199) during git-upload-pack '/opensta22:45
jeblairorg.eclipse.jgit.errors.PackProtocolException: internal revision error22:45
mordredwell, at least we didnt' have to wait22:45
fungihuh. didn't see any of that on review-dev22:45
mordredso - revert to normal 2.10 while we ready ourselves for reverting?22:45
jeblairmordred: i think so22:45
mordredoh god22:45
fungistopping gerrit again now22:45
mordredI'm so unhappy about this22:45
*** achanda has joined #openstack-infra22:46
mordredjeblair: I feel like there was a config patch we applied that will need to be reverted ...22:46
*** mpopow has joined #openstack-infra22:46
fungigerrit's on its way back up with normal 2.10 again22:47
*** nelsnelson has quit IRC22:47
anteayado we need an etherpad for the rollback?22:47
jeblairi don't see the same zlib triggering error22:47
jeblair[2015-05-11 22:43:51,636] WARN : Error computing PatchListKey[openstack/nova BASE..b3ee6a14c2dc1ced7ce775a6806da74a0f84b423 IGNORE_NONE]22:47
jeblairjava.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 19a0c41a3fb38875259a6353c76c90ee3433c19522:47
*** otter768 has quit IRC22:47
jeblairis first error22:48
jeblairi wonder if we should clear the cache?22:48
jeblairso two thoughts:22:49
fungidoes the cache persist between restarts?22:49
mordredI didn't think so?22:49
jeblair1) did not clearing the cache cause the -dirty rollback to fail?22:49
jeblair2) did not clearing the cache cause the zlib problem we're seeing?22:49
jeblairit does in review_site/cache22:49
jeblairit caches diffs22:49
*** dangers is now known as dangers_away22:50
fungii can restart it, moving that out of the way22:50
jeblairi have no idea if cache inconsistency could relate to missing object errors22:50
mordredme either22:50
jeblairand i guess while we're at it: 3) could the repack have caused the cache to get out of sync causing the zlib errors22:50
jeblairfungi: let's try -dirty again with no cache22:50
*** trey has quit IRC22:50
jeblairor should we try -clean again with no cache?22:51
*** mpopow has quit IRC22:51
fungiokay, so move ~gerrit2/review_site/cache to cache.old? or just delete the files in it?22:51
jeblairfungi: lets move22:51
*** Sukhdev has quit IRC22:52
jeblairand let's try -dirty22:52
jeblairsince it's likely to fail quickly if the problem is not cache related :)22:52
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add SQL file for emergency rollback
jeblair(it's just that if dirty works, we need to try clean again with no cache)22:52
fungiokay, doing now22:53
fungiit's starting again on dirty with no cache22:54
fungiconfirmed, running what i think it should22:54
jeblairit has not immediately blown up22:56
*** trey has joined #openstack-infra22:56
*** bknudson has quit IRC22:56
*** esker has quit IRC22:57
jeblairer, my stream events just failed and now i'm getting rejected connections22:57
fungiyep, big traceback in the error log22:57
anteayamorning jhesketh22:57
fungijetty seems to have gone belly-up22:58
*** craigbr has quit IRC22:58
anteayajhesketh: troubleshooting the gerrit upgrade:
fungiblow away the cache and switch to the clean 2.10 build now?22:58
jheskethYeah just saw some rollback. This doesn't look fun22:58
jeblairfungi: sure.  i don't know what to make of this.22:58
anteayajhesketh: not so much22:58
jheskethLet me know how I can help (although I'm only partially here today)22:59
zarothat's weird i would expect something like that to happen on review-dev as well.22:59
jeblair1) No index versions ready; run Reindex22:59
*** reed_ has joined #openstack-infra23:00
fungiStarting Gerrit Code Review: FAILED23:00
fungistart a reindex?23:00
zarowe still on dirty?23:00
fungior put the original cache back?23:01
jeblairfungi: yeah, maybe move the index dir out of the way just to be sure23:01
*** gordc has quit IRC23:01
jeblairfungi: i would not put the cache back; i'd also delete the current cache23:01
jeblairso clear cache, clear index, reindex, start 2.10-clean23:01
fungiyep, done and reindex underway23:01
fungithe original cache from before we started this particular experiment is still cache.old and index is moved to index.old23:02
fungiclean war is in place and reindex is currently running23:02
anteayazaro: you home now, that was only 20 minutes?23:02
fungiso starting from empty cache and empty index23:03
jeblairfungi: think we ought to do a git fsck on nova while that's running, just to make sure it wasn't damaged?23:03
zaroanteaya: neat thing about carpool is that you can use your computer as a passenger23:03
anteayazaro: k23:03
fungijeblair: on it23:03
fungii'll fsck a cp -ax of it as a test23:03
*** emagana has quit IRC23:03
*** funzo has joined #openstack-infra23:04
jeblairfungi: ok23:04
fungireindex is about 1/3 done23:04
*** yamamoto has joined #openstack-infra23:05
*** ashleighfarnham has quit IRC23:06
fungigit fsck shows nothing out of the ordinary besides the dangling blobs and commits we saw earlier23:08
*** reed_ has quit IRC23:08
fungireindex is about half done now23:08
*** reed_ has joined #openstack-infra23:08
funginearly there23:10
*** amitgandhinz has joined #openstack-infra23:10
*** jamesmcarthur has quit IRC23:12
*** dboik_ has quit IRC23:12
*** reed_ has quit IRC23:12
*** ldnunes has quit IRC23:13
*** panda|afk has quit IRC23:13
*** panda|afk has joined #openstack-infra23:14
*** ddieterly has quit IRC23:14
fungiokay, clean gerrit started now with empty cache and fresh indexes23:14
jeblairit seems to be able to receive patchsets23:17
mordredthat'sa  good start23:17
*** ivar-laz_ has joined #openstack-infra23:18
*** hemna_ is now known as hemnafk23:19
*** ivar-laz_ has quit IRC23:19
*** ivar-laz_ has joined #openstack-infra23:19
*** Longgeek_ has quit IRC23:20
*** Daviey has quit IRC23:21
*** carl_baldwin has quit IRC23:21
*** Daviey has joined #openstack-infra23:21
*** ivar-lazzaro has quit IRC23:21
*** lhcheng has joined #openstack-infra23:22
*** esker has joined #openstack-infra23:22
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Add support for keep-records parameter for plots-plugin
*** rrepaka has joined #openstack-infra23:24
jeblair[2015-05-11 23:24:15,695] ERROR : ERROR: Exception caught while accessing pack file /home/gerrit2/review_site/git/stackforge/fuel-web.git/objects/pack/pack-a814e22337395838b039d1e8ef3eef47ff3cd9fe.pack, the pack file might be corrupt23:24 Short compressed stream at 25,794,93323:24
jeblairthat's different23:24
rrepakalifeless: Hi23:24
jeblairand it did not remove the pack file23:24
jeblairi'm going to check out the file23:25
* mordred keeps fingers crossed23:25
lifelessrrepaka: hi?23:25
rrepakalifeless: Pbr is returning a wrong version string
openstackLaunchpad bug 1453996 in PBR "pbr-0.11.0 Returns wrong version string for oslo-config" [Undecided,New]23:25
zarojust noticed that label queries now need to be set with 'label', so this '-Verified-1' now needs to be '-label:Verified-1'23:25
rrepakalifeless: Can this be fixed quickly ?23:26
rrepakalifeless:Just checking if it takes longer i will go ahead with patching my build to use pbr 0.1023:26
jeblair[2015-05-11 23:26:12,079] ERROR : Pack file /home/gerrit2/review_site/git/openstack/glance.git/objects/pack/pack-f23:27
jeblairb9c038839e5238ba128a173570ee7787905edfd.pack is corrupt, removing it from pack list23:27
lifelessrrepaka: it works for me23:27
jeblairorg.eclipse.jgit.errors.CorruptObjectException: Object at 26,164,967 in /home/gerrit2/review_site/git/openstack/glance.git/objects/pack/pack-fb9c038839e5238ba128a173570e23:27
jeblaire7787905edfd.pack has bad zlib stream23:27
jeblairthere's our usual error23:27
lifelessrrepaka: what does 'git describe' output?23:27
mordredlifeless, rrepaka: if y'all could move the pbr discussion elsewhere, that would be helpful23:27
fungiyep, a ton of MissingObjectException errors now23:27
rrepakalifeless: its 1.4.023:27
mordredwell, at least we learned that it's not the cache23:27
fungiso i think we've probably done all we can short of rollback23:28
lifelessrrepaka: -> #openstack-oslo please.23:28
fungino reply on the repo-discuss ml yet either23:29
mordredso - has the script23:29
mordreddo we need to revert ?23:30
fungii believe so. zaro? ^23:30
mordredas well as the version rollback and the database rollback?23:30
*** teran has quit IRC23:31
fungioh, wait, that's version-detecting now23:31
fungii think we can just roll back the war version and not have to touch 17253423:31
jeblairi'm seeing a lot of 'short compressed stream' errors now23:31
fungiyeah, we landed it well before the upgrade23:31
jeblairi haven't found any actual git problems yet; i'll continue to look23:31
jeblairfeel free to proceed with prepping downgrade23:32
fungiwhere was our upgrade etherpad? anyone have the url handy?23:32
*** wenlock has quit IRC23:32
fungithanks, just found it in my history too23:32
anteayahopefully zaro should be home soon, 50 minutes since his alert he was afk23:33
fungii'll propose a revert of 155463 now23:33
jeblairthere are 'short compressed stream' errors in the log from before we switched to -dirty, so i don't think they are entirely new23:33
mordredoh - I just thought of something else that shoudl be in the migration23:34
mordredthe schema version is recorded in it isn't it?23:34
*** dims has quit IRC23:34
jeblairyeah, is a single row in a table23:34
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Revert "Upgrade Gerrit from 2.8 to ver 2.10 on review.o.o"
zaroshould not have to rever that change. made it so that it will install correct libs for either 2.8 or 2.1023:35
mordredyah. I think the script should set it when it's done23:35
jeblairmordred: can you double check the version table itself didn't change?  it has 2 columns now, and i can't remember if it did before23:35
fungizaro: thanks for verifying. i came to the same conclusion after i thought about it23:35
*** lhcheng has left #openstack-infra23:35
mordredjeblair: it did not show up in the schema diff23:35
fungishall i go ahead and disable puppet on review.o.o?23:36
anteayaso I'm clear, are we going back to 2.8?23:36
fungifor now, yes23:36
anteayathank you23:36
*** tim_o has quit IRC23:37
fungiuntil we come up with additional evidence through testing on review-dev as to the cause or get some feedback from the gerrit devs23:37
*** esker has quit IRC23:37
*** marun has quit IRC23:37
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add SQL file for emergency rollback
anteayafungi: right23:37
ianw"This did not work as planned"23:38
anteayayeah :(23:38
*** dims has joined #openstack-infra23:39
fungiianw: i am in some universes known as "cap'n obvious"23:39
anteayajhesketh: want to learn something about how to downgrade a gerrit?23:39
jheskethanteaya: what's that?23:40
* jhesketh has been watching the conversation 23:40
anteayajhesketh: we are downgrading to gerrit 2.823:40
anteayaah cool23:40
anteayawasn't sure23:40
jheskethI just don't have much to add but if there is anything I can do to help let me know23:41
anteayajhesketh: this was the upgrade etherpad:
anteayajhesketh: well was poking you along those lines, as it will likely be we're done and offline23:41
anteayaover to you jhesketh23:41
anteayawanted to make sure you were playing along at home23:42
jheskethYep as best I can23:42
mordredI DON'T want to learn something about downgrading a gerrit23:42
mordredbut I'm going to anyway23:42
anteayawell that is a good point23:42
fungieverything we never wanted to know about gerrit downgrades but have been forced to find out anyway23:43
anteayawill you have a book signing at summit?23:43
*** dims has quit IRC23:43
jheskethfungi: there's a conference talk topic23:43
fungidid we update the puppet exec timeout for the reindex? do we want to see about doing this with a puppet dev env so that we can patch in a change to avoid automatic reindex exec?23:44
anteayado we have a list of steps for the downgrade?23:44
anteayaso we can keep track of done things?23:44
fungianteaya: i'm working on it in
*** miqui has quit IRC23:45
anteayathank you23:45
greghaynesDo we want to do another db dump before the downgrade migration?23:46
mordredgreghaynes: yes23:46
greghaynesheh, the author of the downgrade seems keen on that, I wonder why ;)23:46
*** yamamoto has quit IRC23:46
*** ams__ has quit IRC23:47
fungi*quaking in virtual boots*23:48
mordredthese virtual boots are not keeping my feet warm23:48
mordredthese pretzels are making me thirsty23:50
*** gmann has joined #openstack-infra23:50
anteayazaro are you about and able to offer thoughts on the downgrade plan unfolding on
jeblairstatus alert Gerrit is going offline while we perform an emergency downgrade to version 2.8.23:51
jeblairlook good ^ ?23:51
fungijeblair: lgtm23:51
jeblair#status alert Gerrit is going offline while we perform an emergency downgrade to version 2.8.23:51
openstackstatusjeblair: sending alert23:51
fungimordred: if you're doing the db commands on the gerrit server, you want to do the other local tasks there so that we don't step on one another's virtual boots?23:52
mordredfungi: sure23:52
*** oomichi has joined #openstack-infra23:52
jeblairi'm writing an email update for the -dev list; we can finish and review it later, just didn't want anyone else to start on one23:52
mordredI'm going to to ahead and disable puppet on review.o.o23:53
-openstackstatus- NOTICE: Gerrit is going offline while we perform an emergency downgrade to version 2.8.23:53
*** ChanServ changes topic to "Gerrit is going offline while we perform an emergency downgrade to version 2.8."23:53
fungiseems to not be running either. that's goof23:54
fungier, good23:54
mordredfungi: actually - my mysql things are as root - you can do the things that happen as gerrit2 perhaps?23:54
zaroanteaya: i've comment on the changes23:54
fungimordred: sure thing23:54
anteayazaro: thank you23:55
mordredfungi: especially since you've been doing those the last few times23:55
clarkbI am almost home can help once thete23:55
*** dboik has joined #openstack-infra23:55
fungimordred: which is basically nothing except handling the cache and reindex, but happy to do so23:55
anteayaclarkb: thank you, travel safe23:55
*** ZZelle has quit IRC23:55
fungiclarkb: we have plenty of hands on deck, and there will almost certainly be aftermath, so there's no rush23:56
openstackstatusjeblair: finished sending alert23:56
fungicarnage awaits us all23:56
*** dboik_ has joined #openstack-infra23:56
mordredexcept Exception: raise Aftermath()23:56
fungianybody want to +2 ? then i'll submit it and make sure it's showing up on the puppetmaster23:57
*** sdake_ has joined #openstack-infra23:57
jeblairfungi: done23:57
mordredtwo of us even23:58
*** banix has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!