Monday, 2014-06-02

*** yamahata has quit IRC00:20
*** e0ne has joined #openstack-infra00:20
*** Sukhdev has quit IRC00:23
*** e0ne has quit IRC00:24
tchaypol/me fights the git00:25
* tchaypo fights the irc00:25
*** matsuhashi has joined #openstack-infra00:27
*** nosnos has joined #openstack-infra00:43
*** sarob has joined #openstack-infra00:47
*** sarob has quit IRC00:52
*** bknudson has left #openstack-infra01:01
tchaypoAll my reviews are failing after 7-8 minutes. If I hadn't seen the chatter in here this morning I'd be very sad.01:05
*** yamahata has joined #openstack-infra01:06
*** fifieldt has joined #openstack-infra01:15
*** e0ne has joined #openstack-infra01:18
*** yamahata has quit IRC01:21
*** yamahata has joined #openstack-infra01:22
*** e0ne has quit IRC01:23
*** otter768 has quit IRC01:23
openstackgerritA change was merged to openstack-infra/devstack-gate: Capture *.failure logs from $BASE/status/stack
bodepdmordred: I've got some inconsistent comments on I tend to gesticulate wildly when I get excited about something.01:42
bodepddamnit, that was a failed copy/paste :)01:43
*** sarob has joined #openstack-infra01:47
*** otherwiseguy has joined #openstack-infra01:47
*** sarob has quit IRC01:52
*** otherwiseguy has quit IRC01:52
*** dims has quit IRC01:57
*** dims has joined #openstack-infra01:59
*** dims has quit IRC02:00
*** trinaths has joined #openstack-infra02:05
*** otter768 has joined #openstack-infra02:06
*** achuprin has quit IRC02:08
*** trinaths has quit IRC02:10
harlowja_hi guys, i just tagged a taskflow release, is there usually a > 10 minute delay on the push to pypi?02:11
nibalizerbodepd: got my +102:12
*** amotoki has joined #openstack-infra02:14
*** e0ne has joined #openstack-infra02:18
*** sarob has joined #openstack-infra02:20
*** achuprin has joined #openstack-infra02:23
*** e0ne has quit IRC02:23
*** oomichi has joined #openstack-infra02:25
*** sarob has quit IRC02:25
*** bhuvan has quit IRC02:31
bodepdnibalizer: is there an official policy on what requires CLAs?02:33
*** otherwiseguy has joined #openstack-infra02:42
*** weshay has quit IRC02:42
nibalizerbodepd: not that I'm aware of, but I'm still comming on to the project, others are too, probably why you're getting inconsistency02:44
*** sarob has joined #openstack-infra02:47
*** fifieldt has quit IRC02:47
*** e0ne has joined #openstack-infra02:48
*** e0ne has quit IRC02:50
*** sarob has quit IRC02:51
*** otherwiseguy has quit IRC02:52
nibalizerbodepd: so infra/confing doesn't require a CLA so a request like yours wouldn't require one02:52
*** bhuvan has joined #openstack-infra02:58
*** gokrokve has joined #openstack-infra03:09
*** gokrokve has quit IRC03:14
*** harlowja_at_home has joined #openstack-infra03:15
*** e0ne has joined #openstack-infra03:18
*** e0ne has quit IRC03:20
*** harlowja_at_home has quit IRC03:23
*** lbragstad has joined #openstack-infra03:23
openstackgerritElizabeth K. Joseph proposed a change to openstack-infra/config: Update elastic-recheck link on index.html
openstackgerritElizabeth K. Joseph proposed a change to openstack-infra/config: Update elastic-recheck link on index.html
pleia2sneaky tabs03:31
*** nosnos has quit IRC03:34
*** talluri has joined #openstack-infra03:35
*** mmaglana has joined #openstack-infra03:39
*** lcheng_ has joined #openstack-infra03:42
*** harlowja_ is now known as harlowja_away03:47
*** sarob has joined #openstack-infra03:47
*** otter768 has quit IRC03:48
mordredbodepd: sorry for the confusion - I think we probably need to get jeblair to weigh in03:50
*** chuckC has quit IRC03:50
*** miqui has quit IRC03:51
*** talluri has quit IRC03:51
*** lcheng_ has quit IRC03:52
*** sarob has quit IRC03:52
*** chuckC has joined #openstack-infra03:52
*** talluri has joined #openstack-infra03:53
*** zhiyan_ is now known as zhiyan03:54
*** harlowja_at_home has joined #openstack-infra03:57
*** Sukhdev has joined #openstack-infra04:01
*** Ryan_Lane has joined #openstack-infra04:09
*** Ryan_Lane has quit IRC04:09
*** harlowja_at_home has quit IRC04:13
*** e0ne has joined #openstack-infra04:18
mordredsdague: when you're around, could you help explain this to me: ?04:18
mordredsdague: namely, what's up with the python -m nova.openstack.common.lockutils - if that's a thing that's needed in the test suite, why isn't it in a fixture?04:19
mordredsdague: oh - nevermind. I read the code04:19
mordredI really wish we didn't have to paste that into tox.ini - but also, fair04:20
*** e0ne has quit IRC04:23
*** nosnos has joined #openstack-infra04:26
clarkbmordred: its a terrible terruible thing that should diaf04:28
clarkbthere is no reason for it to exist04:28
openstackgerritA change was merged to openstack-infra/config: Add nodepool_prod.pp to documentation
*** trinaths has joined #openstack-infra04:35
*** harlowja_at_home has joined #openstack-infra04:36
*** mmaglana has quit IRC04:37
*** yfried has quit IRC04:38
* StevenK stabs setuputils 4.004:39
StevenKTypeError: 'unicode' does not have the buffer interface is now impacting local tests in newly created environments04:39
openstackgerritFathi Boudra proposed a change to openstack-infra/jenkins-job-builder: authorization property: add extended read support
*** gokrokve has joined #openstack-infra04:41
*** sarob has joined #openstack-infra04:44
openstackgerritClark Boylan proposed a change to openstack-infra/config: Don't delete tarballs in
openstackgerritClark Boylan proposed a change to openstack-infra/config: Generate sha1 and md5 sum file for wheels
clarkbmordred: dhellmann ^ that will fix at least one bug with the wheels change04:45
sarobMordred: I'd like to create storyboard project training-guides04:47
*** sarob_ has joined #openstack-infra04:47
harlowja_at_homeclarkb, thx for the quick fix04:47
*** e0ne has joined #openstack-infra04:48
sarobMordred: what's the process to get up and running?04:48
*** sarob__ has joined #openstack-infra04:49
*** sarob_ has quit IRC04:51
*** sarob__ has quit IRC04:54
*** e0ne has quit IRC05:02
*** W00dy has joined #openstack-infra05:03
*** W00dy is now known as Guest3756005:03
*** gokrokve has quit IRC05:06
*** yamahata has quit IRC05:06
*** basha has joined #openstack-infra05:06
*** yamahata has joined #openstack-infra05:06
*** vkdrao has joined #openstack-infra05:08
openstackgerritFathi Boudra proposed a change to openstack-infra/jenkins-job-builder: general: add support for childCustomWorkspace
*** harlowja_at_home has quit IRC05:26
openstackgerritFrederic Lepied proposed a change to openstack-infra/config: glanceclient: make the Python 3.3 gate voting
*** ildikov has quit IRC05:35
*** sarob has quit IRC05:36
*** masayuki_ has joined #openstack-infra05:39
openstackgerritRadomir Dopieralski proposed a change to openstack-infra/config: Add XStatic-* projects with packaged static files for Horizon
*** sarob has joined #openstack-infra05:47
*** Guest37560 has quit IRC05:50
*** sarob has quit IRC05:51
trinathsHi Good Morning..05:55
*** Sukhdev has quit IRC06:01
mattoliverauMorning trinaths06:02
*** _nadya_ has joined #openstack-infra06:07
*** mkoderer has joined #openstack-infra06:09
*** ildikov has joined #openstack-infra06:09
*** bhuvan has quit IRC06:13
*** sarob has joined #openstack-infra06:19
*** jlibosva has joined #openstack-infra06:23
*** sarob has quit IRC06:24
*** yfried has joined #openstack-infra06:25
*** _nadya_ has quit IRC06:26
*** _nadya_ has joined #openstack-infra06:29
*** _nadya_ has quit IRC06:30
*** Guest22618 has quit IRC06:31
*** Sukhdev has joined #openstack-infra06:33
*** _nadya_ has joined #openstack-infra06:34
*** lcheng_ has joined #openstack-infra06:36
*** Sukhdev has quit IRC06:39
*** jcoufal has joined #openstack-infra06:39
*** _nadya_ has quit IRC06:39
*** andreykurilin has joined #openstack-infra06:39
*** _nadya_ has joined #openstack-infra06:41
*** _nadya_ has quit IRC06:42
*** otherwiseguy has joined #openstack-infra06:44
*** yfried has quit IRC06:44
*** sarob has joined #openstack-infra06:47
*** otherwiseguy has quit IRC06:48
*** afazekas has quit IRC06:49
*** hashar has joined #openstack-infra06:51
*** sarob has quit IRC06:51
*** marun has joined #openstack-infra06:52
*** salv-orlando has joined #openstack-infra06:55
*** e0ne has joined #openstack-infra06:56
openstackgerritAntoine Musso proposed a change to openstack-infra/gerritlib: Apply cookiecutter defaults
*** wenlock has quit IRC06:59
openstackgerritA change was merged to openstack-infra/zuul: cleaning up index.rst file
*** praneshp has joined #openstack-infra07:06
*** marun is now known as marun_afk07:09
*** basha has quit IRC07:11
*** Longgeek has joined #openstack-infra07:16
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Fix up fetching jquery.visibility
*** e0ne has quit IRC07:18
*** rgerganov has joined #openstack-infra07:18
LonggeekJoshua Hesketh:ping07:22
mattoliverauLonggeek: if you want to ping him it'll probably be better to ping his irc nick so he gets an alert :) jhesketh: your being pinged!07:25
jheskethoh hiya07:25
jheskethLonggeek: mattoliverau's tactic got my attention.. what's up :-)07:26
fifieldt_fungi, jeblair - just saw your pings here. assume you'll read the patch response (in essence: it's intentional, and desired by docs team too)07:26
*** fifieldt_ is now known as fifieldt07:26
hasharjhesketh: it is great to see you get added to the Zuul core team (i.e. +2 / approve).  Your reviews have always been very helpful.07:26
Longgeekmattoliverau: ok07:26
jheskeththanks hashar :-)07:26
*** shardy has joined #openstack-infra07:27
Longgeekjhesketh: Your patch the url incorrect.07:28
jheskethLonggeek: How so?07:29
Longgeekjhesketh: The correct url is
*** salv-orlando has quit IRC07:31
jheskethLonggeek: Okay, but if you then click on "Raw" in github it takes you to the URL I was using07:31
jheskethwe want to wget the raw file, otherwise we'll pull in all of githubs html07:31
Longgeekjhesketh: See
mattoliverauNight all!07:34
*** andreykurilin has quit IRC07:35
jheskethLonggeek: I ran the same command and it works for me
jheskethLonggeek: what happens if you run curl without --silent?07:36
jheskethseeya mattoliverau!07:36
*** otherwiseguy has joined #openstack-infra07:38
Longgeekjhesketh: ok, here is my question, sorry07:38
*** otherwiseguy has quit IRC07:42
*** matrohon has joined #openstack-infra07:44
jheskethLonggeek: what was your question sorry?07:45
Longgeekjhesketh:My https proxy server is not working properly07:46
jheskethah ok07:46
openstackgerritA change was merged to openstack-infra/zuul: Add require-approvals to EventFilter representation
*** ociuhandu has quit IRC07:46
*** sarob has joined #openstack-infra07:47
*** sarob has quit IRC07:51
*** ihrachyshka has joined #openstack-infra07:52
*** thomasbiege has joined #openstack-infra07:54
trinathsmattoliverau: good night07:54
*** Guest22618 has joined #openstack-infra07:58
*** yfried has joined #openstack-infra07:59
*** ildikov has quit IRC08:02
*** thomasbiege has quit IRC08:04
*** ildikov has joined #openstack-infra08:10
*** SlickN1k has joined #openstack-infra08:11
*** spiffxp_ has joined #openstack-infra08:12
*** lcheng_ has quit IRC08:14
*** vponomaryov has joined #openstack-infra08:15
*** SlickN1k has quit IRC08:15
*** spiffxp_ has quit IRC08:15
therveI think there is an issue with the latest setuptools release?08:17
therveThe gate is about to explode08:17
*** apevec has joined #openstack-infra08:19
apevechi, this gate failure:
apevechappened after setuptools 4.0 was released08:21
*** dizquierdo has joined #openstack-infra08:22
therveapevec, Yeah. I think we need < 3.608:23
therveOr 3.708:23
therve3 releases in 1h sounds suspicious08:24
*** _afazekas has joined #openstack-infra08:24
*** basha has joined #openstack-infra08:25
therve fwiw08:26
*** yfried has quit IRC08:26
*** ominakov has joined #openstack-infra08:28
apevecyeah, also releasing on Sunday evening...08:29
zigoFun fun fun ...08:29
mordred#status alert setuptools upstream has broken the world. it's a known issue. we're hoping that a solution materializes soon08:29
openstackstatusmordred: sending alert08:29
-openstackstatus- NOTICE: setuptools upstream has broken the world. it's a known issue. we're hoping that a solution materializes soon08:31
*** ChanServ changes topic to "setuptools upstream has broken the world. it's a known issue. we're hoping that a solution materializes soon"08:31
zigoAlso, 3 releases in 57 minutes ...08:31
*** otherwiseguy has joined #openstack-infra08:32
zigomordred: Is there a way to blacklist setuptools >= 3.7 globally?08:32
mordredzigo: nope08:32
hashardon't you use a manually updated pypi mirror?08:32
*** habib has joined #openstack-infra08:33
mordredzigo: we don't express dependencies on setuptools because it breaks things08:33
mordredhashar: we don't - and setuptools is, I believe, special anway08:33
openstackstatusmordred: finished sending alert08:33
mordredwe kinda have to exist in a world where setuptools works08:33
hasharthat sounds reasonable08:33
mordredwell, except for when it breaks ;)08:34
zigomordred: How about hacking our pypi mirror to not include blacklisted versions?08:34
tchaypoHack hack hack08:34
tchaypoThat sounds nasty08:35
mordredwe could - but the thing is that openstack is broken in the wild08:35
mordredlike, it's actually broken currently08:35
mordredso we could hack aroudn it - but it will be broken on people's laptops08:35
*** salv-orlando has joined #openstack-infra08:35
*** _afazekas is now known as afazekas08:36
mordred(also, I'm at a conference and don't have tons of attention, so I don't want to start hacking things without others around)08:36
*** pblaho has joined #openstack-infra08:36
*** andreaf has joined #openstack-infra08:37
openstackgerritPeter Belanyi proposed a change to openstack-infra/config: Add jshint job for tuskar-ui
afazekasmordred: Isn't simple global-requirements change enough to prevent these issues both on the gate and on the laptops?08:41
therveafazekas, " we don't express dependencies on setuptools because it breaks things"08:41
*** otherwiseguy has quit IRC08:42
apevecyeah, it is not in global-requirements08:42
*** fbo has joined #openstack-infra08:43
afazekasIs one of the local pip.conf  (something like that) allows to disable installing a particular version of a package ?08:44
*** srenatus has joined #openstack-infra08:45
*** sarob has joined #openstack-infra08:47
*** sarob has quit IRC08:52
*** yfried has joined #openstack-infra08:53
*** salv-orlando has quit IRC08:53
openstackgerrityolanda.robla proposed a change to openstack-infra/storyboard-webclient: Fixes misaligned comments on very long lines
openstackgerrityolanda.robla proposed a change to openstack-infra/storyboard: Added new priority changed event for tasks
*** yfried has quit IRC09:06
openstackgerrityolanda.robla proposed a change to openstack-infra/storyboard-webclient: Added priority on task row
*** dizquierdo has quit IRC09:10
*** IvanBerezovskiy has joined #openstack-infra09:11
*** yolanda has joined #openstack-infra09:12
yolandahi, getting this error on a job i try to check, on every test script: TypeError: 'unicode' does not have the buffer interface09:13
yolandais that something related with the bug you talk about in the status of the chat?09:13
tchaypoyolanda: yep, that's it09:14
*** dkranz has joined #openstack-infra09:14
SlickNikyolanda: Yes, I believe it's related to the setuptools bug that's breaking us.09:15
SlickNikyolanda: FYI09:15
uvirtbotLaunchpad bug 1325423 in openstack-ci "Setuptools 3.7 and newer breaks installation of many packages" [Undecided,New]09:15
yolandayes, looks the same09:15
*** praneshp has quit IRC09:16
*** ociuhandu has joined #openstack-infra09:16
*** skolekonov has joined #openstack-infra09:18
*** salv-orlando has joined #openstack-infra09:19
*** mrda is now known as mrda-away09:21
*** SlickN1k has joined #openstack-infra09:34
*** SlickN1k has quit IRC09:34
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Add docs-on-rtfd to rally
*** SlickN1k has joined #openstack-infra09:35
*** yfried has joined #openstack-infra09:36
KiallInteresting... Upstream setuptools author is doing dev on Windows and Python 3, doesn't seem to be able to test linux / python 2.. I'm surprised it doesn't break more often ;)09:36
*** spiffxp_ has joined #openstack-infra09:37
tchayposo I hate to throw a spanner into the works, but I'm wondering if all the failures we're seeing are down to setuptools09:38
*** otherwiseguy has joined #openstack-infra09:38
tchaypobut happily I think I'm wrong.09:39
tchaypo  does seem to have failed because "E: Unable to locate package ubuntu-cloud-keyring"09:39
tchaypobut that seems to have been a one off, the other failures are because of setuptools09:39
*** yfried_ has joined #openstack-infra09:40
tchaypodo we have anything to show it happening in packages other than pbr?09:40
*** yfried has quit IRC09:42
Kialltchaypo: they are, it's a setuptools bug09:43
*** otherwiseguy has quit IRC09:43
KiallI've seen that error on tripleo builds recently too09:44
tchaypoyeah, but as far as i can tell they all break installing pbr09:44
*** yfried_ has quit IRC09:44
tchaypoI'd love to find more examples of the breakage to add to the bug09:45
tchayposo far it's just two packages, pbr and python-daemon09:45
srenatus chef cookbook stuff09:46
srenatuscould this be related?09:46
srenatus(I wonder, the tests shouldn't fail...)09:46
*** sarob has joined #openstack-infra09:47
Kiallsrenatus: they don't look obviously related, and being ruby I'd expect setuptools isn't involved09:47
*** rlandy has joined #openstack-infra09:48
KiallOh, I don't see any reference to that in the logs09:50
srenatusKiall: true, just all three explode with internet access failing... bundler, too.09:51
*** _nadya_ has joined #openstack-infra09:51
*** dizquierdo has joined #openstack-infra09:51
*** sarob has quit IRC09:51
KiallSo, in the python-daemon, I get some funky characters in my editor:09:54
*** ildikov has quit IRC09:54
KiallJust the 1, on a line of it's own in the middle of the file.09:54
*** ildikov_ has joined #openstack-infra09:54
Kiallpbr doesn't have anything similar though..09:55
Kiall(the again, pbr does all sorts of magic)09:55
*** ildikov_ has quit IRC09:56
*** ildikov_ has joined #openstack-infra09:56
*** Longgeek has quit IRC09:57
*** Longgeek has joined #openstack-infra09:58
tchaypoI'm hoping to find one of the pypa packages that does it09:59
tchaypovirtualenvwrqpper is as close as I've managed so far :p09:59
*** _nadya_ has quit IRC09:59
KiallYea - Installing all packages from global-requirements now, at least, those that don't depend on PBR09:59
KiallHopefully another one that he can test with bails10:00
*** _nadya_ has joined #openstack-infra10:00
*** mrmartin has joined #openstack-infra10:03
*** ildikov_ has quit IRC10:03
*** talluri has quit IRC10:05
*** talluri has joined #openstack-infra10:05
*** ildikov has joined #openstack-infra10:06
*** matrohon has quit IRC10:07
*** talluri_ has joined #openstack-infra10:08
*** habib has quit IRC10:09
tchaypodangnabbit. virtualenvwrapper uses pbr?10:09
*** talluri has quit IRC10:10
tchaypoI don't think it's that though.10:10
KiallEverything that depends on pbr is failing10:10
tchaypoI'd expect to see it getting to the point where it looks at the requirements and tries to install them10:11
tchaypoif it was just a transitive error from the pbr failure10:11
KiallYea, pbr get's loaded *really* early, so you won't see it as a normal dep10:11
tchaypodangnabbit. I don't suppose python-daemon depends on pbr?10:11
KiallI think MySQL-python just triggered it, not sure, got the same error, but it also failed to find mysql_config10:12
KiallNo, python-daemon does not use PBR10:12
Kialldamn .. MySQL-python passed after install libmysql10:13
KiallLiterally every package in global-requirements.txt that depends on PBR is failing ;)10:13
*** talluri_ has quit IRC10:13
KiallEverything else is so far succeeding (slowly making my way through the list, every fail means re-downloading all the previous packages ;))10:14
KiallOkay - every global-requirements.txt entry bar those that depends on PBR install and work just fine10:16
tchaypowell that certainly suggests another thing we can attack10:17
KiallYea, if PBR can work around it...10:19
tchaypoI feel bad about saying "we" now10:20
tchayposince it's 8:20pm here and I'm close to signing off and leaving this for other people to solve10:20
tchaypoI hate it when people say "we" when they mean "anyone but me"10:21
tchaypoI'm running through to see if i can find another package that has a problem10:22
tchaypobut about half of them are external and I cbfed doing the --allow-external --allow-unverified dance10:23
*** thomasbiege has joined #openstack-infra10:23
*** salv-orlando has quit IRC10:24
*** talluri has joined #openstack-infra10:25
openstackgerrityolanda.robla proposed a change to openstack-infra/storyboard-webclient: Added angular-momentjs to dependencies
*** salv-orlando has joined #openstack-infra10:26
*** Guest22618 has quit IRC10:27
*** talluri has quit IRC10:28
tchaypoI give up.10:29
*** talluri has joined #openstack-infra10:30
*** e0ne has joined #openstack-infra10:35
KiallInterestingly - rebuilding python-daemon's sdist and it installs perfectly10:35
KiallActually, pip install'ing the downloaded python-daemon tar.gz works, just not when downloaded from pypi10:37
*** thomasbiege has quit IRC10:38
*** yamahata has quit IRC10:39
*** thomasbiege has joined #openstack-infra10:39
openstackgerritAntoine Musso proposed a change to stackforge/python-jenkins: Speed up job existence tests by fetching less info
*** sarob has joined #openstack-infra10:47
*** e0ne has quit IRC10:51
openstackgerritStephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue
*** sarob has quit IRC10:52
*** e0ne has joined #openstack-infra10:52
*** jistr has joined #openstack-infra10:55
openstackgerritStephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue
*** viktors has joined #openstack-infra11:00
mrmartinSergeyLukjanov are you here?11:02
*** yfried_ has joined #openstack-infra11:14
*** Longgeek has quit IRC11:18
*** zhiyan is now known as zhiyan_11:19
*** zhiyan_ is now known as zhiyan11:20
*** zzelle has joined #openstack-infra11:23
*** _nadya_ has quit IRC11:25
*** salv-orlando has quit IRC11:25
*** salv-orlando has joined #openstack-infra11:27
*** thomasbiege has quit IRC11:28
*** sandywalsh has joined #openstack-infra11:29
*** markmc has joined #openstack-infra11:32
*** hashar is now known as hasharLunch11:32
*** Longgeek has joined #openstack-infra11:37
*** otherwiseguy has joined #openstack-infra11:39
*** yamahata has joined #openstack-infra11:41
*** mburned_out is now known as mburned11:42
*** otherwiseguy has quit IRC11:43
*** sarob has joined #openstack-infra11:47
*** e0ne has quit IRC11:49
openstackgerritThierry Carrez proposed a change to openstack-infra/infra-specs: Added specification for storyboard story tags
*** Guest22618 has joined #openstack-infra11:50
*** sarob has quit IRC11:51
*** basha has quit IRC11:55
*** yfried_ has quit IRC11:56
*** basha has joined #openstack-infra11:56
*** BAKfr has joined #openstack-infra11:58
*** e0ne_ has joined #openstack-infra11:58
*** basha has quit IRC11:59
*** hasharLunch is now known as hashar12:01
*** ArxCruz has joined #openstack-infra12:03
openstackgerritStephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Add docs-on-rtfd to rally
*** pdmars has joined #openstack-infra12:09
*** mrmartin has quit IRC12:10
*** pdmars has quit IRC12:11
*** dims has joined #openstack-infra12:12
*** adalbas has joined #openstack-infra12:12
*** weshay has joined #openstack-infra12:13
*** pdmars has joined #openstack-infra12:16
*** jgallard has joined #openstack-infra12:16
*** sandywalsh has quit IRC12:18
*** matsuhashi has quit IRC12:22
*** ildikov_ has joined #openstack-infra12:22
SergeyLukjanovmrmartin, I'm here now12:22
*** dims has quit IRC12:23
*** ildikov has quit IRC12:23
openstackgerritRuslan Kamaldinov proposed a change to openstack-dev/cookiecutter: Added a doc build target to tox.ini
*** mbacchi has joined #openstack-infra12:26
*** aysyd has joined #openstack-infra12:26
vponomaryovSergeyLukjanov: hello, if you have sme time, could you please review this: ?12:27
sdagueSergeyLukjanov: any change you could punch the new setuptools out of the mirror?12:30
sdagueotherwise I think we're going to be broken until it's fixed12:30
SergeyLukjanovvponomaryov, I'll take a look12:30
SergeyLukjanovsdague, morning12:30
*** talluri has quit IRC12:30
*** dims has joined #openstack-infra12:30
SergeyLukjanovsdague, AFAIK we're not able to limit setuptools12:31
SergeyLukjanovfrom scrollback12:31
sdagueSergeyLukjanov: not in global requirements12:31
SergeyLukjanov[12:32:38]  <zigo> mordred: Is there a way to blacklist setuptools >= 3.7 globally?12:31
SergeyLukjanov[12:32:43]  <@mordred> zigo: nope12:31
*** talluri has joined #openstack-infra12:31
sdaguebut I thought we could pull it from the mirror12:31
Kiallsdague: mordred was saying that even if we remove it from our mirror, everyone's laptops etc are still broke12:31
*** Alexei_987 has joined #openstack-infra12:31
sdagueKiall: sure12:31
Kiall(But.. That's better than 100% broke ;))12:31
*** talluri has quit IRC12:31
DinaBelovaKiall, sdague - but it this case we'll have the gate fixed12:31
sdaguebut then you don't have the inability for anyone to do code reviews because jenkins will fail everything12:32
SergeyLukjanovsdague, I'm not sure that it will help for enough time, probably even all images will be needed to rebuild12:32
*** dims has quit IRC12:32
*** talluri has joined #openstack-infra12:32
sdagueSergeyLukjanov: well, we could do that as well :)12:32
sdaguebasically everything is currently on fire12:32
*** dims has joined #openstack-infra12:32
SergeyLukjanovoops, I see "/srv/static/docs-draft is using 91.0% of 1.32TB"12:33
SergeyLukjanovfungi, is it ok? ^^12:33
sdagueI actually think if we aren't going to fix it, we should turn off zuul, because right now, it's not doing anything useful, except burning nodes and failing people's tests12:33
*** salv-orlando has quit IRC12:34
*** dkliban_afk is now known as dkliban12:34
sdagueDinaBelova: is there an openstack-ci bug for this?12:34
sdagueso I can create an elastic recheck query on it12:34
SergeyLukjanovsdague, is setuptools 4.0 work ok?12:35
sdagueSergeyLukjanov: no idea12:36
sdagueI'm just trying to avoid a giant amount of recheck in the gate12:36
SergeyLukjanovsdague, I can remove >=3.7 from the mirror12:36
sdagueSergeyLukjanov: I think you are also right that you need to kick off the nodepool rebuild as well12:36
*** talluri has quit IRC12:36
SergeyLukjanovsdague, the only issue is that I'm not sure how to do it safe12:37
srenatusIf anyone could have a look at this before I'm publicly ridiculed in a hangout this afternoon, I'd be glad...
sdagueSergeyLukjanov: gotcha, well fungi should be up soon I think12:37
DinaBelovasdague, there was bug first created for the tempest -
uvirtbotLaunchpad bug 1325514 in tempest "'unicode' does not have the buffer interface" [Undecided,New]12:37
DinaBelovaafter that folks have added other projects to it too12:37
sdagueDinaBelova: ok, I'm going to remove all the other projects except openstack-ci12:38
DinaBelovasdague, ok12:38
*** salv-orlando has joined #openstack-infra12:38
*** maxbit has joined #openstack-infra12:39
*** habib has joined #openstack-infra12:39
SergeyLukjanovdone: sudo rm -rf setuptools-4.0*12:39
SergeyLukjanovthe latest available setuptools in mirror is 3.612:40
DinaBelovahehe, let's try to recheck no bug :D12:40
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: added bug for setuptools 3.7 breaking the world
sdaguesweet, I can't get an er bug through, because... setuptools12:41
*** AJaeger has joined #openstack-infra12:42
*** AJaeger has joined #openstack-infra12:42
mtreinishsdague: nice12:42
DinaBelovasdague, probably recheck will help? as SergeyLukjanov removed the 3.7+ from the mirror12:43
DinaBelovajust after you proposed the error I guess12:43
*** radez_g0n3 is now known as radez12:43
YorikSarI see first successful jobs for Neutron.12:44
*** mwagner_lap has quit IRC12:44
YorikSarYou can start recheck flood ;)12:44
samuelmz_AJaeger, ping12:45
AJaegerHi samuelmz_12:45
*** rfolco has joined #openstack-infra12:45
samuelmz_AJaeger, hi :)12:45
samuelmz_AJaeger, please take a look at this12:45
samuelmz_AJaeger, and thanks for having confirmed this bug12:45
DinaBelovasdague, it looks like SergeyLukjanov's fix works12:46
AJaegerSam-I-Am, will do later. Thakns for the patch!12:46
AJaegersamuelmz_, I need to check whether we need to add the information elsewhere...12:46
SergeyLukjanovjeblair, mordred, fungi, clarkb, I've removed setuptools >= 3.7 from mirror to tmp fix this issue12:47
*** sarob has joined #openstack-infra12:47
*** rlandy_ has joined #openstack-infra12:47
sdagueDinaBelova: not for me :(12:47
salv-orlandoextra day off for everybody!12:48
YorikSarsdague: Which change request failed?12:48
apevecDinaBelova, sdague - there was setuptools bug reported earlier
uvirtbotLaunchpad bug 1325423 in openstack-ci "Setuptools 3.7 and newer breaks installation of many packages" [Undecided,New]12:48
samuelmz_AJaeger, yes, could be at 'Install the clients >  Installing from packages'12:48
DinaBelovaapevec, oh, sorry - did not find it earlier12:48
*** dprince has joined #openstack-infra12:49
AJaegersamuelmz_, I'll check later tonight - thanks12:49
samuelmz_AJaeger, ok, thanks12:49
*** rlandy has quit IRC12:49
YorikSarHm... I guess not everybody uses pypi mirror...12:50
SergeyLukjanovonly projects defined in os/requirements/projects.txt could be fixe12:51
SergeyLukjanovsdague ^^12:51
*** russellb has joined #openstack-infra12:51
*** sarob has quit IRC12:51
sdagueSergeyLukjanov: gotcha12:53
sdagueis there a different mirror for infra jobs? I thought we controlled those as well12:53
*** doug-fish has joined #openstack-infra12:54
dhellmanngood morning, all12:55
anteayajeblair: regarding the gertty tracebacks, I was browsing into a patchset and looking at diffs12:56
anteayajeblair: I had entered no comments for review12:56
*** rlandy__ has joined #openstack-infra12:57
SergeyLukjanovdhellmann, morning12:57
*** ihrachyshka has quit IRC12:57
anteayajesusaurus: okay I will ask about puppet-livegrep, so far I have gotten crickets on it so hopefully this week folks will be willing to weigh in12:57
SergeyLukjanovsdague, no, we remove this code when migrated to the projects.txt mirror chooser AFAIK12:57
*** bauzas has joined #openstack-infra12:57
openstackgerritAna Krivokapic proposed a change to openstack/requirements: Add Pint for Horizon
fungilooking back over the setuptools discussion now12:57
sdagueSergeyLukjanov: gotach12:57
SergeyLukjanovfungi, short info12:57
anteayatrinaths: have you checked the logs on your server like I asked you too?12:58
SergeyLukjanovfungi, I've removed bad version from the mirror, it fix projects defined in projects.txt12:58
anteayatrinaths: did you find out why the server only renders pages sometimes and not consistently?12:58
SergeyLukjanovfungi, it works and we see some green jobs now in gate12:58
SergeyLukjanovfungi, non-related question, how can I force push something? :)12:58
bauzasSergeyLukjanov: thanks for your work :)12:58
anteayamorning dhellmann12:58
*** rlandy_ has quit IRC12:59
fungiSergeyLukjanov: you have to add your gerrit account temporarily to,members and then when you review a change, delete any verified -2 or workflow -1 votes, add verified +2 in addition to your workflow +1, and then hit the "publish and submit" button13:00
fungioh, and make sure there's a code review +2 vote as well13:01
SergeyLukjanovfungi, okay, thx, It could be needed someday13:01
*** rdopiera has joined #openstack-infra13:01
fungiand then remove yourself from Project Bootstrappers immediately once you're done, to avoid any unpleasant mistakes13:01
fungii also usually make sure i leave a review comment with my approval when i do that explaining the reason i had to bypass all gating13:02
SergeyLukjanovfungi, sure, that makes sense13:02
SergeyLukjanovfungi, will it auto reset related chain in gate?13:03
fungiSergeyLukjanov: it won't, that change will simply fail to merge since it's already merged13:03
*** miqui has joined #openstack-infra13:03
YorikSarfungi: btw, wouldn't Cerdic's --submit-immediately be useful in this case?13:03
fungiSergeyLukjanov: but it's assumed that basically everything in the gate is doing that anyway if you're stuck bypassing tests completely13:03
SergeyLukjanovfungi, it works too, I mean that force push will not break everything13:04
*** yfried_ has joined #openstack-infra13:04
fungiYorikSar: possibly, except that we have to grant ourselves permission to use it, and by the time we've done that having a special option to git-review to skip pressing the submit button isn't really much of a time saver13:05
fungiYorikSar: we don't leave any of our accounts with the necessary permissions to use that under normal circumstances13:05
YorikSarfungi: Oh, ok.13:06
fungiSergeyLukjanov: well, when you say force push i assume you're talking about merging a change directly from gerrit bypassing all gating, not actually git push --force (which is bad, bad, bad)13:06
YorikSarfungi: How do you add yourself to a group? Is it in UI?13:06
fungiYorikSar: i believe it can be done via an api call in the new version of gerrit, but i'm used to using the webui for that since it was the only way in 2.4 and we've not been on 2.9 long at all13:07
YorikSarfungi: Although even if it's done not from UI automation would do more harm than good here, I think.13:07
fungisomething. moar coffee13:07
*** ArxCruz has quit IRC13:07
fungiYorikSar: yes, at least for openstack infra, we make this sort of exception extremely deliberate, so having a few additional steps to prevent you from doing it accidentally is more of a feature than a bug13:08
sdagueon the upside -
sdagueso the fix was merged into upstream 13 minutes ago13:08
*** julim has joined #openstack-infra13:08
sdaguehopefully we'll get a new release shortly13:08
fungisdague: awesome! i was about to ask how, exactly, new setuptools was breaking us13:08
sdaguefungi: basically, doesn't look like it was tested well on python213:08
* dhellmann is confused to see a github pull request for a bug filed on bitbucket13:09
sdaguedhellmann: yeh, no idea13:09
fungisdague: srsly... nobody uses python 2 these days. we should have been pure py3k so long ago, totally our fault ;)13:09
*** julim has quit IRC13:09
*** mriedem has joined #openstack-infra13:09
dhellmannhave we already talked about helping pypa set up testing?13:09
YorikSarfungi: How about we migrate all infra stuff to 3.4?13:10
dhellmannas in, using stackforge?13:10
sdaguewell, from the thread it looks like the maintainer is basically on windows and python3. And the changelog seems to indicate python2 breaks a couple times in the last year on releases13:10
*** yamahata has quit IRC13:10
*** ihrachyshka has joined #openstack-infra13:10
fungidhellmann: yes, i believe dstufft is wanting to run his own copies (understandably) and has been working on getting some of it running recently13:10
*** ArxCruz has joined #openstack-infra13:10
dhellmannfungi: cool13:10
*** samuelmz_ is now known as samuelmz13:10
*** yamahata has joined #openstack-infra13:10
fungiif the current setuptools maintainer develops entirely in python 3 on windows, that actually explains a lot ;)13:11
SergeyLukjanovfungi, sure, nothing about git push -f, I'm never doing it :)13:11
dhellmannfungi: indeed13:11
dstufftI've messed around with it a little bit, but haven't spent a great deal of time on it yet13:11
*** smarcet has joined #openstack-infra13:11
dstufftsetuptools does use travis to test itself as well13:12
Longgeekfungi: Good morning13:12
fungidstufft: just not in a way that would have caught i guess13:12
dstufftyea well13:12
dstufftI think setuptools in the same boat as pip there, where a lot of stuff isn't tested and/or is hard to test13:13
dhellmanndstufft: If there's an existing test suite, would you be interested in patches that tried installing common packages?13:13
fungiLonggeek: good morning! (or good night where you live maybe?)13:13
*** trinaths has quit IRC13:13
*** mbacchi has quit IRC13:13
*** eharney has quit IRC13:13
Longgeekfungi, yes, night.13:14
dstufftdhellmann: to which project?13:14
dhellmanndstufft: setuptools, I guess?13:14
dstufftdhellmann: so jaraco is the person to aska bout that13:15
dstuffthe's the setuptools maintainer13:15
dhellmanndstufft: ok13:15
Longgeekfungi: Upstream of the ci system is mainly used openstack_projects this puppet modules is not it?13:15
dstufftdhellmann: hes in #pypa-dev fwiw13:15
dstufftdhellmann: also FWIW, one of the things I want to do is test a bunch of common/known good patches in a seperate test suite, eventually13:15
dstufftonce I get around to setting up our CI13:16
dhellmanndstufft: that sounds good, too13:16
Longgeekfungi: I found that although the upstream jenkins home directory is /home/jenkins, but many places are defined /var/lib/jenkins13:17
fungiLonggeek: if i understand your question correctly, is the module we use to configure our servers for the openstack developer and community infrastructure. that module uses some other modules in and some modules from puppetforge13:17
dstufftit's the common not enough time problem :[13:17
*** nosnos has quit IRC13:18
fungiLonggeek: so the other modules in are designed to be possibly used for more generic system configuration, for example to set up a gerrit server or a jenkins master the way gerrit or jenkins developers might expect rather than the way openstack infra needs to run its copies13:18
*** lcostantino has joined #openstack-infra13:18
rdopierasdague: hey, do you want to talk about that xstatic package?13:19
fungiSergeyLukjanov: are we ready to sound the all-clear signal with statusbot now?13:20
SergeyLukjanovfungi, stackforge/* projects aren't working still13:21
SergeyLukjanovfungi, openstack/* projects should work ok13:21
fungiSergeyLukjanov: ahh, okay it's just fixed on our mirror for now. got it13:21
SergeyLukjanovwe could add it to the notice13:21
*** dripton has joined #openstack-infra13:21
SergeyLukjanovI can try to make my first notice ;)13:21
fungiSergeyLukjanov: so we should probably update the topic to include a link to the setuptools bug for people wanting to track progress13:21
dstufftThere is a 3.7.113:21
dstufftand a 4.0.113:22
dstufftthat is supposed to fix pbr13:22
fungii guess let's wait a few minutes to see if stackforge tests recover, and then we can send an ok message to all channels with a bug number devs can recheck against13:22
SergeyLukjanovfungi, +113:23
SergeyLukjanovfungi, do you have a correct bug id?13:23
SergeyLukjanov looks marked duplicated13:23
fungii haven't started looking yet, though i suspect Alex_Gaynor opened one13:23
uvirtbotLaunchpad bug 1325423 in openstack-ci "Setuptools 3.7 and newer breaks installation of many packages (dup-of: 1325514)" [Undecided,New]13:23
uvirtbotLaunchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New]13:23
sdaguerdopiera: sure13:24
SergeyLukjanovsdague, Alex_Gaynor, what's the correct bug id for setuptools issue?13:24
SergeyLukjanovfor rechecks13:24
sdaguethough I think I left my concerns pretty clear in the review13:24
sdagueSergeyLukjanov: I duped it into 132551413:25
fungiSergeyLukjanov: looks like we've settled on 132551413:25
*** vkdrao has quit IRC13:25
fungiyeah, what sdague said13:25
sdaguebecause that's the ER13:25
rdopierasdague: I'm not sure what is the required action then13:25
SergeyLukjanovsdague, okay13:25
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Add rally-dsvm-netron job to rally job group
rdopierasdague: just forget about the whole horizon split thing?13:25
*** homeless has joined #openstack-infra13:25
sdaguerdopiera: so if there is a bug in that pypi package, who's going to fix it?13:25
sdaguecode that hasn't released since 2011 is very concerning13:26
rdopierasdague: it has been updated as recently as a month ago13:26
sdaguethat's what I found when I went looking13:27
rdopierasdague: it is being actively used by MoinMoin 2 wiki project, and maintained by the project lead13:27
rdopierasdague: yeah, that's because the changes were only in the docs, because that project contains almost no code13:27
rdopierasdague: it's a "done" project, it won't grow more features13:27
sdaguerdopiera: then why is it self proclaimed as alpha?13:28
sdaguethose things don't add up13:28
SergeyLukjanovfungi, notice message draft: #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on
uvirtbotLaunchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New]13:29
rdopierasdague: he just set it as alpha at the beginning and never changed it13:29
fungiSergeyLukjanov: i'd just use the bug number rather than the url (uvirtbot will happily provide people with the url right after that anyway)13:30
SergeyLukjanovfungi, ok13:30
sdaguerdopiera: well that also doesn't speak well to it being well maintained13:30
fungi"... recheck on bug 1325514"13:30
*** bknudson has joined #openstack-infra13:30
uvirtbotLaunchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New]
SergeyLukjanov fungi, notice message draft: #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on 132551413:30
sdaguewhen we specify a global requirement we are basically telling all distros everywhere that they need to package this code and support it13:30
thervedstufft, Is 4.0.1 published somewhere?13:30
sdagueand telling them that's the case with something self marked as alpha is not the business we want to be in :(13:31
SergeyLukjanovfungi, notice message draft, correction: #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on bug
uvirtbotLaunchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New]13:31
*** jcoufal has quit IRC13:31
sdagueso if the maintainer is active ask them to cut a 1.0, change the maturity flags, then that would be different13:31
fungiSergeyLukjanov: looks great13:31
rdopierasdague: I already did13:31
*** maxbit has quit IRC13:31
dstuffttherve: pypi13:31
rdopierasdague: thanks13:32
SergeyLukjanovfungi, ok, so, waiting for confirmation from stackforge projects13:32
fungitherve: is linked from
therveAh just needed some hardcore refresh13:33
fungitherve: also a wheel and a zip13:33
mordredfungi: ooh. upstream fixed?13:33
*** mrmartin has joined #openstack-infra13:33
fungimordred: seems that way13:33
fungitime will tell13:33
*** bogdando has quit IRC13:34
SergeyLukjanovmordred, status ok notice is ready too :) waiting for the changes in overloaded check queue13:34
gilliardhi -infra.  Just checking in to see if nodepool's still having trouble booting on HP cloud?13:35
gilliardI added a lot more info to bug 132493413:35
uvirtbotLaunchpad bug 1324934 in nova "Neutron port leak when connection is dropped during port create in instance boot." [Undecided,In progress]
mordreddstufft: I've been meaning to get stuff in our CI systems to let you guys know if/when your changes are going to break us too...13:36
mordreddstufft: but also, you know, ENOTIME13:36
dstufftmordred: heh13:36
dstufftI need a minion13:36
dstufftor two13:36
therveConfirmed working for me13:36
dstufftso much stuff is broken13:36
*** markmc has quit IRC13:37
dstufftand it's like, well do I work on some CI, or do I work on things that'll clean up some of the other brokeness13:37
*** pcrews has joined #openstack-infra13:42
*** rlandy_ has joined #openstack-infra13:42
*** signed8bit has joined #openstack-infra13:42
fungigilliard: i just reran the script looking for disassociated neutron ports and only turned up 5 since friday, so at least that particular problem seems to be fairly infrequent (though our weekend volume is also much lower than weekdays)13:44
*** wenlock has joined #openstack-infra13:44
*** rlandy__ has quit IRC13:45
*** rlandy__ has joined #openstack-infra13:46
*** sarob has joined #openstack-infra13:47
*** mbacchi has joined #openstack-infra13:48
openstackgerritAndrea Frittoli  proposed a change to openstack-infra/devstack-gate: Allow to configure git base URL
*** signed8bit has quit IRC13:48
gilliardfungi: that's good.  We haven't done anything to mitigate it yet.  So instances are booting OK?13:49
fungigilliard: i'm looking at some stats now to see what the overall situation is there13:49
*** rlandy_ has quit IRC13:49
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add md5 checksum to groups tarball publisher
fungigilliard: looks like we are still having some percentage of nodes booting into an error state in nova13:50
fungieek huge url13:51
mrmartinSergeyLukjanov: hi, what I need to do restart gating check of this failed patch: ?13:51
*** sarob has quit IRC13:51
gilliardfungi: OK thanks I will see what more I can see.13:52
gilliardit's not going to be "port allocation quota exceeded" any more I guess.13:52
*** zz_gondoi is now known as gondoi13:53
fungimrmartin: looks like the projects alphabetization check failed to retrieve a needed package from (probably due to a transient network error somewhere on the internet). i'll reapprove it13:53
mrmartinfungi: thanx13:53
fungimrmartin: it's rechecking and will head into the gate shortly if there are no new problems with it13:54
*** sballe has joined #openstack-infra13:55
*** sballe has quit IRC13:56
*** signed8bit has joined #openstack-infra13:58
SergeyLukjanovmrmartin, /me was afk, fungi thx for help13:58
*** jcoufal has joined #openstack-infra14:00
openstackgerritLonggeek proposed a change to openstack-infra/config: Modify jenkins home directory
*** mfer has joined #openstack-infra14:00
*** wenlock has quit IRC14:01
*** wenlock has joined #openstack-infra14:01
*** mrmartin has quit IRC14:02
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Generate sha1 and md5 sum file for wheels
anteayaLonggeek: why do you want to modify the jenkins home directory?14:03
anteayaLonggeek: and as I said Friday, we tend to discuss in channel whether or not something actually is a bug before creating a report14:04
Longgeekanteaya: Why jenkins :: master the use of the /var/lib/jenkins?14:04
anteayathat is a good question14:04
*** mrmartin has joined #openstack-infra14:05
anteayain future please start by asking a question, just like the question you just asked14:05
anteayasaves us a lot of time14:05
anteayathank you14:05
*** jaypipes has joined #openstack-infra14:05
anteayahey jaypipes14:06
anteayado you  have much on your agenda today?14:06
anteayaI could use some of your time before our third-party meeting today14:06
Longgeekjenkins::master use /var/lib/jenkins, but jenkins::jenkinsuser use /home/jenkins14:06
ruheAFAIK /var/lib/jenkins is a default home for the jenkins user14:06
jaypipesanteaya: unfortunately yes :(14:06
anteayajaypipes: okay, go you14:06
anteayaLonggeek: yes, you have asked the question, well done14:06
anteayaLonggeek: now we wait14:07
*** julim has joined #openstack-infra14:07
anteayaLonggeek: we wait for someone who knows the answer to read your question and respond14:07
anteayaLonggeek: that person is not me14:07
openstackgerritA change was merged to openstack-infra/config: Don't delete tarballs in
Longgeekanteaya: ok14:07
anteayaLonggeek: and we don't continually ask the same question, because that gets annoying14:07
anteayaLonggeek: thanks14:07
Longgeekanteaya: ok, I will not ask.14:08
anteayaLonggeek: ask by all means, but ask once14:08
*** BobBall has quit IRC14:08
*** wenlock has quit IRC14:09
*** jgallard has quit IRC14:09
*** jgallard has joined #openstack-infra14:10
ruheSergeyLukjanov: fungi: fyi i see at least two stackforge projects with green py27/pep8 jobs: and
*** afazekas has quit IRC14:12
ruheSergeyLukjanov: fungi: also local run of "tox -e py27" also was successful14:12
SergeyLukjanovruhe, nice, thanks for the update14:12
SergeyLukjanovfungi, time for notice?14:12
*** mriedem has quit IRC14:12
SergeyLukjanovfungi: #status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on bug 132551414:13
uvirtbotLaunchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,New]
openstackgerritA change was merged to openstack-infra/config: Generate sha1 and md5 sum file for wheels
*** basha has joined #openstack-infra14:13
fungiSergeyLukjanov: that's cool with me14:13
SergeyLukjanovheh, bracking statusbot14:13
SergeyLukjanov#status ok setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on bug 132551414:14
openstackstatusSergeyLukjanov: sending ok14:14
*** mriedem has joined #openstack-infra14:14
*** sballe has joined #openstack-infra14:15
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Add docs-on-rtfd to rally
*** mwagner_lap has joined #openstack-infra14:15
*** sarob has joined #openstack-infra14:15
*** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs | bugs |"14:15
-openstackstatus- NOTICE: setuptools issue was fixed in upstream in 3.7.1 and 4.0.1, please, recheck on bug 132551414:15
*** sarob_ has joined #openstack-infra14:16
fungiLonggeek: part of the problem is that there are two different places the jenkins account gets used. the jenkins service running on the master and the jenkins slave agent running on slaves. it seemed reasonable to let the slaves have the jenkins user's homedir in /home but the server tends to do a lot of things under its account on a master which fits better in /var/lib14:16
*** BobBall has joined #openstack-infra14:17
*** primemin1sterp has joined #openstack-infra14:17
fungiLonggeek: so we set the default to /home/jenkins in the jenkinsuser class, but then override it to /var/lib/jenkins in the master class14:17
*** mgagne has joined #openstack-infra14:17
openstackstatusSergeyLukjanov: finished sending ok14:17
fungiLonggeek: the correctness of those assumptions can certainly be debated, and regardless i think a comment in the master class manifest explaining that would probably be a good idea14:18
*** pdmars has quit IRC14:18
*** primemin1sterp has quit IRC14:19
*** blamar has joined #openstack-infra14:19
*** pdmars has joined #openstack-infra14:19
*** _nadya_ has joined #openstack-infra14:19
*** primeministerp has quit IRC14:19
*** sarob has quit IRC14:19
*** primeministerp has joined #openstack-infra14:19
fungiLonggeek: i'll update the bug with some additional details14:19
SergeyLukjanovfungi, hm, that's strange, but there is no openstackstatus bot in sahara channel :(14:20
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add #openstack-sahara to status bot
SergeyLukjanovfungi, please, take a look on it ^^14:20
ruhewere there any changes which would hide devstacklog.txt.gz from job results?14:20
Longgeekfungi: ok14:20
*** sarob_ has quit IRC14:20
ruhehere is an example:
fungiruhe: what do you mean hide from job results?14:20
anteayasdague: I gave a whirl, since i am not core, I have interesting results in the -2 and +2 sections14:21
ruhehere is results for the same patch . it has devstacklog.txt.gz14:21
ruhethis one is ~1 week earlier14:21
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Run tox by stack user in rally-cli job
anteayasdague: I got things I reviewed that anybody -2'd or +2'd14:23
fungiSergeyLukjanov: i think you rebased 97247 incorrectly or something14:24
*** Longgeek has quit IRC14:24
*** Longgeek has joined #openstack-infra14:25
SergeyLukjanovfungi, today isn't my day14:25
*** Longgeek has quit IRC14:25
*** otherwiseguy has joined #openstack-infra14:25
*** Longgeek has joined #openstack-infra14:25
*** Longgeek has quit IRC14:25
*** Longgeek has joined #openstack-infra14:25
ruhefungi: btw it seems like all devstack jobs miss actual devstack logs. here is another example
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add #openstack-sahara to status bot
SergeyLukjanovfungi, ^^14:26
SergeyLukjanovfungi, fixed one ;)14:26
sdagueanteaya: firefox?14:27
*** afazekas has joined #openstack-infra14:27
fungiruhe: i wonder if something changed in the cleanup function which is causing it to fail to copy the logs... looking at change history in devstack-gate now for clues14:27
sdagueanteaya: can you jump to -qa, we're discussing there14:27
*** jgrimm has joined #openstack-infra14:27
ruhefungi: thank you!14:27
openstackgerritA change was merged to openstack-infra/config: Fix translation setup for api-site
*** beekneemech is now known as bnemec14:28
fungiruhe: my guess is a regression introduced by
fungiruhe: er, that was the merge. actual commit was
fungisdague: ianw: ^14:29
fungiruhe: yep, that change dropped the 'sudo cp $BASE/new/devstacklog.txt $NEWLOGTARGET/' and none of what it added accounts for it either14:31
openstackgerritTim Kelsey proposed a change to openstack-infra/gerritbot: Adding the a configuration option for gerritbots PID file.
fungii think just readding that one line ought to solve it... doing now14:31
*** olaph has joined #openstack-infra14:32
phschwartzmorning infra14:33
openstackgerritJeremy Stanley proposed a change to openstack-infra/devstack-gate: Copy devstacklog.txt
fungiruhe: ianw: sdague: ^14:33
anteayamorning phschwartz14:34
ruhefungi: woohoo! thank you, you alwasy fix things :)14:34
fungiruhe: any day i fix more things than i break is a good day, in my opinion14:35
anteayafungi: have you had time to break anything today?14:36
anteayaI can't see that you have14:36
fungianteaya: probably plenty, i just haven't found out yet14:36
anteayaha ha ha14:36
anteayaI'm still reading weekend scrollback14:37
sc68calIs there an IRC channel for devstack?14:37
sc68calI feel bad asking in -infra or -qa14:37
anteayasc68cal: -qa14:37
openstackgerritStephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue
fungisc68cal: #openstack-qa is probably the more effective option14:37
sc68calthx :)14:37
anteayadon't feel bad sc68cal14:37
fungithough they might say the same thing about #-infra ;)14:37
*** maxbit_ has joined #openstack-infra14:38
*** afazekas has quit IRC14:38
anteayafortunately due to overlap, folks don't ping pong much before they find a chatty person14:38
fungisc68cal: these are the people you likely want to pester about devstack patches...,members14:38
sc68calfungi: thanks :)14:39
*** maxbit_ has quit IRC14:39
*** maxbit has joined #openstack-infra14:39
*** andreaf has quit IRC14:40
*** pblaho has quit IRC14:40
anteayafungi: turns out he needs to talk to me, atm14:41
SergeyLukjanovfungi, +2'd waiting for job results14:41
openstackgerritStephan Renatus proposed a change to openstack-infra/config: Fix Chef cookbook-related strainer issue
openstackgerritNikita Konovalov proposed a change to openstack-infra/infra-specs: StoryBoard search spec
Alex_GaynorAre there an absence of whichever buiders are used for ht pep8, 2.7, and docs jobs?14:42
*** mrmartin has quit IRC14:44
fungiAlex_Gaynor: not specifically, from the look of it, we're just generally starved for workers at the moment due to the earlier setuptools failures (everything's getting rechecked now)14:45
*** wenlock has joined #openstack-infra14:45
Alex_Gaynorfungi: makes sense, thanks14:46
*** mbacchi has quit IRC14:46
*** trinaths has joined #openstack-infra14:46
fungiAlex_Gaynor: looks like we're about an hour wait in the check pipeline for non-devstack workers at the moment14:47
*** w_ has joined #openstack-infra14:47
*** skolekonov has quit IRC14:47
Alex_Gaynorfungi: I'm imagining a butcher, and there's a bunch of folks just milling about with those tickets they give you, and complaining :-)14:47
fungiAlex_Gaynor: and just under that wait time for devstack workers too14:47
fungiAlex_Gaynor: yes, next up is #93576,514:48
openstackgerritTim Kelsey proposed a change to openstack-infra/gerritbot: Adding a configuration option for gerritbots PID file path.
*** [1]trinaths has joined #openstack-infra14:49
*** olaph has quit IRC14:49
*** afazekas has joined #openstack-infra14:50
*** w_ is now known as olaph14:50
*** tkelsey has joined #openstack-infra14:50
*** pblaho has joined #openstack-infra14:50
*** jlibosva has quit IRC14:51
*** mbacchi has joined #openstack-infra14:51
*** david-lyle has joined #openstack-infra14:51
*** trinaths has quit IRC14:51
*** [1]trinaths is now known as trinaths14:51
*** atiwari has joined #openstack-infra14:53
*** duran has joined #openstack-infra14:53
*** vhoward has joined #openstack-infra14:55
*** e0ne_ has quit IRC14:57
cody-somervilleHey. Is it possible to tell nodepool to pick an az?14:57
*** e0ne has joined #openstack-infra14:57
*** hogepodge has joined #openstack-infra14:58
fungicody-somerville: by using a node label which is specific to a particular az14:58
cody-somervilleHow do I make a node label specific to a particular az?14:59
fungicody-somerville: in nodepool.yaml, create a label which only references one provider name14:59
*** skolekonov has joined #openstack-infra14:59
cody-somervilleHow do I make that provider only use one az though?15:00
fungicody-somerville: each az is a separate "provider" in nodepool parlance15:00
openstackgerritLonggeek proposed a change to openstack-infra/config: Modify jenkins home directory
cody-somervillefungi: No it ain't.15:00
cody-somervillefungi: or at least I don't see one specified :)15:01
cody-somervillefungi: What is the configuration option?15:01
*** skolekonov has quit IRC15:01
fungicody-somerville: for instance, see the two tripleo "providers" in (each of which is actually a separate "region" in the tripleo cloud)15:01
krotscheckStoryBoard meeting in #openstack-meeting-315:01
*** e0ne has quit IRC15:02
anteayafungi: the status of nodes in delete state seems to be staying steady, rather than following the curve of nodes in use, this tends to be my indicator that some nodes may be stuck in a delete state15:02
fungicody-somerville: right now there are "tripleo-test-cloud" and "tripleo-test-cloud-rh1" providers defined, which are the hp and rh regions of the tripleo cloud respectively15:03
cody-somervilleI want to pick availability zone.15:03
fungicody-somerville: similarly, hpcloud-az1/az2/az3 "providers" for the three hpcloud 1.0 west availability zones15:03
cody-somervilleThat's for US East.15:03
cody-somervilleand those are actually regions15:04
cody-somervilleHP Cloud used different regions to provide different "availability zones" in 1.0.15:04
cody-somervilleHP Cloud has capacity issue in AZ2 of US East (1.1, aka 13.5) and it appears nodepool is only creating instances in that az because that's the default is not specified otherwise.15:04
*** e0ne has joined #openstack-infra15:05
fungicody-somerville: ahh, you're asking about hpcloud 1.1. you should have been more specific. we were informed that there was one tenant quota covering that entire cloud and that hpcloud would simply round-robin or otherwise distribute nova boot requests to where there was available capacity. so we just treat it as a single provider currently15:06
cody-somervilleIt turns out it actually just puts it all in az2 as it seems.15:07
*** w_ has joined #openstack-infra15:07
*** lcostantino has quit IRC15:08
fungicody-somerville: if that's not actually the case after all, then we probably need to start using multiple region-b providers representing different availability zones and set separate max-servers levels for each in nodepool15:08
fungimordred: clarkb: jeblair: ^ fun!15:08
cody-somervilleI don't see how to set what az it actually uses?15:08
tteggelfungi: cody-somerville is correct15:08
*** eharney has joined #openstack-infra15:08
* tteggel goe hunting for the docs15:09
*** yfried_ has quit IRC15:09
Alex_Gaynorfungi: I think we might be more than just behind, even the top of the check pipeline has unstarted jobs15:09
fungitteggel: it's possible this has changed since the original beta account we had before region b was ga15:09
*** olaph has quit IRC15:10
fungiAlex_Gaynor: those usually represent a job which got started, lost because of a jenkins-specific failure (node went offline or something) and is queued to be automatically retried15:10
Alex_Gaynorfungi: No, I mean, basically the entire pipeline, /even/ the top has unstarted jobs15:11
fungiAlex_Gaynor: due to the queueing order, however, those don't get serviced until other changes which were already waiting for workers when that failed15:11
*** mbacchi has quit IRC15:11
*** mbacchi has joined #openstack-infra15:12
fungiAlex_Gaynor: the top several in the check pipeline all seem to have workers, there are a couple nova changes behind it which look like they probably lost a devstack worker and are waiting on their turn to get the lost jobs retried. behind those are a lot of changes waiting on centos6 nodes where they can run py26 jobs15:13
*** talluri has joined #openstack-infra15:13
*** rfolco has quit IRC15:14
fungiAlex_Gaynor: looking in the gate pipeline, which gets priority over check, we also seem to be running low on available precise-py3k nodes to run pypy and python33 jobs15:14
cody-somervillefungi: Looking at the code, nodepool does not have the capability to specify an az.15:14
fungicody-somerville: yeah, i have no idea what the nova server parameter is for that in the python api, but i assume there is one we can add and then expose in nodepool configuration15:15
trinathsHi Stackers!, You are able to access the CI logs server at ... Kindly help me with your comments.15:15
fungicody-somerville: from nodepool's perspective, without some significant refactoring anyway, i assume we'd just call them separate providers (one for each az)15:16
Alex_Gaynorfungi: mm, I guess it is just centos and precise-py3k then, thanks15:16
fungicody-somerville: it doesn't help for terminology that hpcloud previously called its regions "availability zones" i guess, so i didn't realize there was a distinction15:16
*** esker has joined #openstack-infra15:17
Kiallfungi: Yea, we fixed the terminology with the new regions.. The old naming was painful15:19
*** w_ is now known as olaph15:20
openstackgerritRoman Podoliaka proposed a change to openstack-infra/config: Upload tripleo-heat-templates releases to PyPi
tteggelfungi: cody-somerville: so hpcloud 1.1 is using OS-EXT-AZ extension15:21
tteggelno longer the weird az-as-region with its own endpoint thing15:22
*** mmaglana has joined #openstack-infra15:24
*** morganfainberg is now known as morganfainberg_Z15:25
srenatusIs there any way to pass more information from a jenkins job to the gerrit message?  I.e. "FAILED: part 1" if the check consists of several parts?15:25
*** basha has quit IRC15:26
anteayasrenatus: hi what is the context for your question?15:27
anteayasrenatus: are you working with you own ci system seperate from openstack's?15:27
srenatusanteaya: I'm pondering the idea of moving the chef cookbooks' tests to a per-repo fashion:
srenatusanteaya: but then we'd lose (compared to before) the quick visual feedback of "chef-lint: FAILURE, chef-spec: SUCCESS"15:29
srenatusanteaya: I'd like to figure out if it'd be possible to get that back15:29
anteayaI don't know enough about your workflow to offer an opinion15:29
anteayafor instance I have no idea what strainers are15:30
anteayado you have a sample patch I can view?15:30
anteayafungi: I have to conclude we have nodes stuck in a delete state somewhere15:31
srenatusanteaya: strainer is a tool to specify tests suites (tools, rather) to be run against a cookbook:
srenatusanteaya: and I'd love to have this file be the one source of truth about which tests are run for this cookbook15:31
anteayasrenatus: do you have a sample patch in gerrit that I can view?15:31
* anteaya clicks15:32
srenatusanteaya: it's all failures for some other reason, but15:32
srenatusif we changed the jenkins-part (i.e. the job builder yaml) to only one job, "strainer test", running all three parts, it'd only be one, accumulated output: success/failure15:33
mordred_phonecody-somerville: you mean it doesn't round robin if not specified? that's assinine15:35
tteggelmordred_phone: nope. we lost that argument early on15:35
anteayasrenatus: yes I see the three tests, lint, style and unit:
tteggelmordred_phone: because cross AZ compute volume attachment doesn't work15:36
mordred_phonetteggel: whoever you lost to was wrong15:36
* mordred_phone cries15:36
anteayasrenatus: I'm failing to see the efficienty for rolling the three tests into one15:36
fungisrenatus: there is work underway to replace jenkins with a more flexible job runner which can return more specific status information on task completion. right now we get "pass" or "fail" so there's no easy way to break that out into a list of discrete states for sub-tasks15:36
fungianteaya: what makes you think there are nodes stuck?15:36
srenatusfungi: thanks!15:37
srenatusfungi: good to know15:37
anteayafungi: the purple line is staying straight15:37
srenatusfungi: any details? :)15:37
anteayafungi: which is consistent with all other times nodes are stuck in delete15:37
*** rgerganov has quit IRC15:37
anteayafungi: the in use line is fluxuating and the delete state portion of the graph is not15:37
hasharhello! Any Zuul guru ever heard of Zuul being stuck in a loop while reporting change?  It looped with an exception that a job already got reported (trace: )15:38
fungianteaya: the one doesn't follow from the other. i haven't had a moment to look yet, but i suspect that the large number of nodes in a building state are rapidly transitioning directly to delete state because they fail to complete15:38
mordred_phoneso I have a cross az quota, but I still have to point to azs15:38
mordred_phonethat's going to be ugly for quota awareness15:38
anteayafungi: yes that was your assessment the last time I saw the same thing15:39
tteggelmordred_phone: yup. we are trying to reopen this dicussion internally for precisely these reasons15:39
anteayafungi: since you had been busy with other things, I wanted it to be on your radar, since we do have such a backlog in the queues right now15:39
tteggelmordred_phone: although the likely outcome is that you will be forced to specify AZ15:39
anteayahashar: zuul will loop if it can't report the change and then will kick off a new test since there is no feedback reported15:40
mordred_phonetteggel: whee! :)15:40
anteayahashar: like if there is a comment on patchset 5 and patchset 6 is the latest patchset15:40
hasharanteaya: yeah because it received a new patchset so it dequeue the previous event15:41
anteayaright, so it won't report15:41
anteayabecause voting is closed on that patchset15:41
anteayahashar: is it happening now?15:42
jeblairannegentle: regarding I don't really understand the idea of publishing something from an unofficial (stackforge) project on the official openstack website15:42
jeblairannegentle: i think that's a line we should not cross15:42
hasharanteaya: in this case I had only one patchset but the change was in two pipelines (test and gate).  So might be a race condition, I will look at the debug log15:42
hasharanteaya: na I killed Zuul, it was stuck and no more processing any event hehe15:42
jeblairfungi, fifieldt, mordred_phone: ^15:42
tteggelmordred_phone: there's no good answer unless we have a scheduler that can work with cinder, nova and neutron15:42
anteayahashar: kk15:42
tteggelmordred_phone: otherwise people who want AZ affinity of resources (which you really do with a volume) are screwed15:43
hasharanteaya: was just wondering whether it has been a known issue. Will investigate. Thank you!15:43
openstackgerritAna Krivokapic proposed a change to openstack/requirements: Add Pint for Horizon
anteayahashar: an edgecase that has popped up occasionally, I think we have the known cases fixed but are interested to hear details of other triggers15:43
*** e0ne_ has joined #openstack-infra15:44
mordred_phonejeblair: looking15:44
mordred_phonejeblair: yes. I agree15:44
mordred_phoneif we want it to be a thing, it should get adopted by docs for real ... or it should grow another home15:45
mordred_phoneor a new program15:45
fungimordred_phone: that's what sarob seems to be implying in his comment... that they want to start a training program which this will eventually be incubated in15:46
*** SumitNaiksatam has quit IRC15:46
*** SumitNaiksatam has joined #openstack-infra15:47
*** amotoki_ has joined #openstack-infra15:47
fungiso not going to be part of the docs program ever, i guess, which would explain why they don't want to incubate in docs (in case the training program isn't approved by the tc?)15:48
*** e0ne has quit IRC15:48
*** salv-orlando has quit IRC15:48
openstackgerritA change was merged to openstack-infra/config: Fix groups-dev drush dependency problem
*** gyee has joined #openstack-infra15:53
dhellmannclarkb: are you still up for testing alpha releases today, or should we wait for a day when we don't have huge breaking issues like setuptools?15:54
anteayadhellmann: I do believe setuptools is actually fixed now15:55
*** social__ has joined #openstack-infra15:55
anteayadhellmann: we are straining under the weight of rechecks though15:55
dhellmannanteaya: yeah, that's what I meant :-)15:55
anteayamighty kind of you15:55
*** salv-orlando has joined #openstack-infra15:55
*** gokrokve has joined #openstack-infra15:56
*** rdopiera has quit IRC15:57
*** zhiyan is now known as zhiyan_15:57
*** amotoki_ has quit IRC15:59
mriedemanyone know what thstack is?15:59
*** amotoki_ has joined #openstack-infra15:59
mriedemLonggeek: ^16:00
Longgeekmriedem: hi.16:00
Longgeekmriedem: my ci.16:00
mriedemLonggeek: let's talk in #openstack-nova16:00
*** sarob has joined #openstack-infra16:02
*** talluri has quit IRC16:04
*** gokrokve has quit IRC16:04
*** e0ne_ has quit IRC16:04
*** IvanBerezovskiy has left #openstack-infra16:05
anteayamorning zaro16:06
wenlock /wave zaro16:06
*** salv-orlando has quit IRC16:07
sarobfungi: any other questions on
srenatusanteaya, fungi: thanks for your input above (the jenkins job feedback issue)16:08
*** ewindisch has quit IRC16:09
anteayasrenatus: np16:09
*** viktors is now known as viktors|afk16:12
fungisarob: at the moment, my primary concern is with the slippery slope of allowing stackforge projects, which by definition are not under the governance of the openstack technical committee, to publish documentation on an official openstack website16:13
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Don't gate pecan on barbican for now.
ryanpetrelloany chance I could convince one of you to push ^ through?16:14
ryanpetrelloI goofed up on the git repo URL for barbican and now pecan patchsets are failing -_-16:14
ryanpetrello<— dummy16:14
*** packet has joined #openstack-infra16:14
ryanpetrelloactually, I think I did *that* wrong, too...16:15
sarobfungi: we have been planning on becoming a separate project since we started last summer16:15
*** dizquierdo has quit IRC16:16
fungiryanpetrello: sure, but the patch is wrong. see inline comments16:16
ryanpetrellofixing now16:16
fungiryanpetrello: oh, you just noticed it anyway16:16
*** ominakov has quit IRC16:16
fungisarob: cool. do you have plans for a website you're going to publish it on?16:17
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Don't gate pecan on barbican for now.
sarobfungi: right where it is under docs.openstack.org16:17
sarobfungi: annegentle can attest to planning16:18
anteayajaypipes krtaylor so this is what I have so far for today:
fungiryanpetrello: see my second inline comment on the first patchset of that change. leaving it non-voting in check is fine, but there's no reason to have it in the gate pipeline since nobody will see the results16:18
sarobfungi: we will continue to work with the documentation team, as we are reusing much of the material16:18
fungisarob: i understand, i'm just wondering where we define the criteria for which unofficial projects publish on if it's no longer reserved for official documentation, so i know how to answer the other stackforge projects which ask that question16:19
jaypipesanteaya: looks good. I will not be attending today's meeting, however...16:19
anteayafungi: the delete line for nodes is dropping now, thanks16:19
anteayajaypipes: sorry to hear that16:19
ryanpetrellogot it16:19
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Don't gate pecan on barbican for now.
sarobfungi: i need to bring on new training developers and giving 5-6 new people core reviewing under docs wasnt considered16:19
anteayajaypipes: I was hoping you could help answer some questions during the highlight an account section16:19
anteayajaypipes: if I send some folks your way, when might you be available for questions?16:20
sarobfungi: its a chicken egg problem16:20
*** UtahDave has joined #openstack-infra16:20
sarobfungi: we are working towards incubated16:20
sarobfungi: but i need consistent contribution16:20
jaypipesanteaya: if someone can add questions to an etherpad somewhere, I can answer them later on.16:20
sarobfungi: more contributors mean more core reviews16:20
anteayajaypipes: great, I will set that up and attach it to the meeting agenda16:21
anteayathank you16:21
sarobfungi: more core reviewers mean i need a separate repo from docs16:21
*** dkliban is now known as dkliban_lunch16:21
*** sballe has quit IRC16:22
anteayaI am going to go for a walk before the third-party meeting16:22
anteayaback in about an hour16:22
openstackgerritAntoine Musso proposed a change to openstack-infra/gerritlib: Apply cookiecutter defaults
fungisarob: well, this might be the time to make an incubation proposal to the tc, because as a project we have generally accepted publishing documentation for incubated projects within official programs to the official publication sites16:22
sarobfungi: im good with that16:23
fungisarob: assuming the idea is to have "openstack training" as an official program?16:23
*** adalbas has quit IRC16:23
dhellmannsdague: I'm wondering if there's a way to disable "foreach" for a section in gerrit-dash-creator so I can have a section that links to reviews outside of oslo for syncs that are happening16:23
sarobfungi: official not so sure16:23
sarobfungi: one step at a time16:24
sarobfungi: i need to ramp up about another 6-8 developers16:24
sarobfungi: our contribution rate is probably off for incubated right now16:25
*** msabramo has joined #openstack-infra16:25
sarobfungi: id like to get these new developers, that know training, but not openstack ci16:25
sarobfungi: ramped up with as few changes in the near future as possible16:26
sarobfungi: can we focus on this first step first?16:26
fungisarob: ultimately, continuing to publish it to an official openstack site when it's not currently incubated is something which should be discussed at a higher level within the openstack project than this code review change. i think the proper venue for that discussion is the tc. if we drop the publish job from the new project request for now, i have no other significant concerns16:27
ryanpetrellothanks, fungi16:27
sarobfungi: between annegentle and tfifield, we should have pretty good coverage on if publishing short term from a stackforge repo is okay?16:27
sarobfungi: sure we can discuss with TC as well16:28
fungisarob: i think they can act as advocates to the tc for that exception if they feel that way, yes16:28
*** julim has quit IRC16:28
sarobfungi: okay, ill discuss at next tc meet16:29
jeblairsarob: i think you should aim for getting the program going first16:29
jeblairsarob: i think publishing unofficial docs on the official site is putting the cart before the horse16:29
*** zehicle_at_dell has quit IRC16:29
*** zehicle_at_dell has joined #openstack-infra16:30
jeblairsarob: but fortunately, the bar for creating a new program isn't that high16:30
sarobjeblair: its being published there now16:30
jeblairsarob: we've made it a lot harder for new projects to get on the 'core' track, but programs that aren't aiming for that don't have nearly as many hurdles16:31
sarobjeblair: we want to bring on new developers and expand the core reviewer16:31
*** jgallard has quit IRC16:31
jeblairsarob: yeah, and i think a new program will help that16:31
sarobjeblair: i have the new developers lined up already16:31
fungiit's being published there as part of the documentation program, but this is a request for a project outside of the documentation program publishing to the same place, and without an official openstack program to govern it16:31
sarobjeblair: im aiming to expand the people invovled without muddling docs16:32
*** marcoemorais has joined #openstack-infra16:32
*** mmaglana has quit IRC16:32
jeblairsarob: ++16:32
sarobjeblair: if annegentle is okay with adding new core reviewers into docs for over the summer16:32
sarobjeblair: then breaking off into a new repo can wait16:33
*** mmaglana has joined #openstack-infra16:33
fungisarob: well, it can be broken off into a new repo within the docs program too, if they agree for it to still fall under their umbrella16:33
sarobjeblair: i dont want to confuse the new people coming onboard16:33
sarobfungi: i thought that was the case16:33
*** _nadya_ has quit IRC16:34
fungisarob: but in that case it would be openstack/openstack-training-guides instead of stackforge/openstack-training-guides16:34
sarobfungi: there is no reason to start a new program outside of docs16:34
sarobfungi ah, okay16:34
*** BAKfr has quit IRC16:35
sarobfungi tfifield and i didnt think of that16:35
sarobfungi: so should i just adjust the patch?16:36
*** shakamunyi has joined #openstack-infra16:36
fungisarob: that's primarily a question for annegentle16:36
sarobfungi: add annegentle as additional reviewer?16:36
fungisarob: but yes that would be one way16:36
sarobfungi: /me sigh16:36
* sarob sigh16:37
sarobfungi: its never one way16:37
*** mmaglana has quit IRC16:37
sdaguedhellmann: am I doing it unconditionally?16:37
sdaguerealistically you probably always want status:open in there anyway16:37
sarobfungi: ill confer with tfifield and annegentle, and then come back to the patch16:38
dhellmannsdague: yeah, I wanted to override the projects list for one case (see but I decided to make that a separate dashboard16:38
sarobfungi: thanks for the feedback16:38
fungisarob: okay, great. let me know what you figure out16:38
*** adalbas has joined #openstack-infra16:39
sarobjeblair: sound about right to have training as separate project but under docs program?16:39
sarobfungi: roger16:39
clarkbfungi: cody-somerville I think we should avoid specifying AZs unless we can confirm that the scheduling given us is actually wrong16:40
sdaguedhellmann: lgtm, merged16:40
clarkbit is supposed to schedule things properly16:40
sarobany infra online that can speak to storyboard access?16:40
jeblairsarob: yep, that's an option.  programs can have as many projects as they need.16:40
clarkbdhellmann: I am up for testing it today. Found one bug yesterday after harlowja_away tagged taskflow16:40
clarkbcatching up this morning to see if the fix merged16:41
jeblairsarob: re storyboard: yes.  :)  for what project?16:41
sarobjeblair: training-guides ;)16:41
fungisarob: yep, for storyboard to add your project it's just an additional parameter in the review.projects.yaml file16:41
*** radez is now known as radez_g0n316:41
sarobfungi: ah, righto16:41
fungisarob: and then when the review does merge, you'll have a corresponding project in storyboard16:41
jeblairsarob: yeah, that should be okay.  two things:16:41
fungisarob: so that 96334 change is the appropriate place to add it16:42
dhellmannclarkb: I saw that change earlier and left some comments16:42
*** marun_afk is now known as marun16:42
jeblairsarob: a) if you stay in the docs program, can you run that by annegentle?  she may want all docs-related projects to use the same tracker16:42
jeblairsarob: b) we're not actually recommending storyboard for production use by openstack projects yet.  i'd actually strongly discourage it.16:42
sarobjeblair: i'm torn actually16:43
fungisarob: jeblair makes very good points ;)16:43
sarobjeblair: its bringing on non-openstack people16:43
jeblairsarob: we haven't managed to do significant dogfooding in infrastructure yet; i would recommend at least waiting until we can injure ourselves first16:43
clarkbdhellmann: thanks. I just copy pasta'd that code from the script16:43
sarobjeblair: these are training developers, not python developers16:43
dhellmannclarkb: yeah, not worth a -116:43
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Gate barbicanclient on PyPy and Python 3.3
dhellmannclarkb: I don't see that change any more, was it merged?16:44
fungisarob: storyboard is definitely still short a few features which a lot of projects are likely to want too16:44
sarobjeblair: storyboard will be alot easier to train them on16:44
jeblairsarob: i think i can anticipate many of the things you're going to say about launchpad, but i'm worried about storyboard being worse for new devs16:44
clarkbdhellmann: it was merged16:44
jeblairsarob: and storyboard is going to change _radically_ several more times before we really start to use it16:44
dhellmannclarkb: cool16:44
*** morganfainberg_Z is now known as morganfainberg16:44
*** Guest22618 has quit IRC16:44
sdaguedhellmann: cool16:45
sarobjeblair: i know launchpad well, but i have trained about 50 people so far as new openstack devs16:45
clarkbjeblair: did the gertty stack get that missing DB migration?16:45
sarobjeblair: launchpad blueprints is one of the biggest confusing parts16:45
jeblairsarob: so i actually think it might hurt more than it would help (as much as i want everyone to just use storyboard now (!!!) i don't want to put people off because we're asking them to use something unfinished and which will fundamentally change16:45
*** habib has quit IRC16:45
dhellmannclarkb: unfortunately I have a block of meetings starting shortly, so it may be a while before I can trigger any more tests. If harlowja_away is already tagging taskflow and there were problems, should we continue testing with that lib to "fix" the release?16:46
sdaguedhellmann: as you are a mac person, do you happen to have IE installed somewhere?16:46
dhellmannsdague: I'm not sure there's a version of IE that runs on OS X any more.16:46
sdagueI'd like to figure out if the qa review url that breaks in firefox works in IE16:46
sdagueoh, I thought there still was16:46
*** trinaths has quit IRC16:46
sarobjeblair: i though we were just going to add new features16:46
morganfainbergdhellmann, sdague , there isn't16:46
sdaguemorganfainberg: ok16:46
*** rwsu has joined #openstack-infra16:46
sarobjeblair: thats what i got from the session16:46
morganfainbergsdague, most people use VMs if they need IE16:46
sdaguebeing on Linux, it's never been here :)16:46
sarobjeblair: i misunderstand?16:47
morganfainbergsdague, yeah hear that.16:47
clarkbdhellmann: ya we can try retriggering taskflow jobs16:47
clarkbdhellmann: rather than tagging something new16:47
jeblairsarob: some of those new features may require significant changes.  the "priority" system is something that is still way up in the air16:47
sarobjeblair: hmm, okay. i will wait then16:48
Alex_GaynorSo this graph looks ungood:
sarobjeblair: i was leaning that way, but after refstack joined, i changed my mind16:48
clarkbAlex_Gaynor: we don't have enough slaves16:49
fungiAlex_Gaynor: yeah, at the moment we're using almost nothing in hpcloud, and trying to get by only on rackspace16:49
Alex_Gaynorclarkb: is one of our clouds having a bad morning? It seems way worse than normal16:49
Alex_GaynorI gather that's not going well.16:50
clarkbAlex_Gaynor: its two things. we turned off one cloud that will be turned off today16:50
*** blamar has quit IRC16:50
clarkband the replacement for said cloud is giving us a hard time16:50
*** trinaths has joined #openstack-infra16:50
Alex_GaynorIf there's anything I can do to help on the rackspace side, let me know.16:50
fungiAlex_Gaynor: hpcloud 1.0 is apparently finally being turned off, so we stopped using it over the weekend. hpcloud 1.1 is struggling to boot new nodes, supposedly because all requests are getting scheduled to one az16:51
*** hashar has quit IRC16:51
fungithough i'm not sure if that's the only reason, it's been brought up as a reason16:51
clarkbfungi: is that really why its failing?16:51
*** afazekas has quit IRC16:51
clarkbit wasn't clear to me from cody-somerville that tis was related16:51
*** jistr has quit IRC16:52
zigosdague: I'm having an issue with tempest (with the "1" tag). It's doing "Non-zero exit code (2) from test listing." before I even got a chance to run tests. What can I do to investigate where the issue is?16:52
fungiclarkb: may have been tangential, but i assumed given the timing that it was related16:52
jesusaurusclarkb: i believe that nodepool is hardcoded to use an az thats having extra trouble16:52
jesusaurusat least thats my understanding16:52
clarkbjesusaurus: it isn't hard coded to use any AZ16:52
clarkbjesusaurus: nova client/server are supposed to schedule an AZ for you16:53
zigoThe relevant output seems to be: unittest.loader.LoadTestsFailure.tempest.api.compute.v3.flavors.test_flavors_negative\x1eH\xd0J\xb3 `I\x02\x17text/plain;charset=utf8\rimport errorsH\xd1tempest.api.compute.admin.test_migrations16:53
clarkbjogo: ^ can you fill us in on that? iirc you explained it to us when we added these regions to nodepool16:53
zigoHowever, that file seems fine ...16:53
jeblairsarob: oddly enough, refstack is not part of an official program, so i have a bit less of an opinion there.  (that refstack is not in an official program is perhaps something that should be addressed)16:54
sdaguezigo: probably ping mtreinish in -qa16:54
*** harlowja_away is now known as harlowja_16:54
fungiis the os-ext-az extension one which comes standard in novaclient, or something we would need to add another piece of software to support?16:54
jesusaurusclarkb: oh, huh16:54
mtreinishzigo: that sounds like the testr import error thing, probably a config file issue16:54
clarkbAlex_Gaynor: looking at numbers rax is mostly happy. Other than bumping quotas I am not sure we need rax to do anything. and I am not sure we need to bump quotas16:54
*** andreaf has joined #openstack-infra16:54
openstackgerritA change was merged to openstack-infra/elastic-recheck: added bug for setuptools 3.7 breaking the world
tteggelfungi: supported in default novaclient16:54
zigomtreinish: So, by default, the tempest.conf.sample doesn't work?16:55
*** mmaglana has joined #openstack-infra16:55
tteggelfungi: need to pass availability_zone param to server create method16:55
fungitteggel: okay, so assuming we're using latest released novaclient we can in theory add support to twiddle that from nodepool16:55
mtreinishzigo: no you need to add things like creds so tempest knows how to talk to the cloud16:55
clarkbtteggel: well reality may be we need to but we shouldn't need to16:55
harlowja_clarkb is there anything i need to do to retrigger the pypi taskflow tag stuff/job?16:55
jogoclarkb: yeah if you don't specify an AZ you will be assigned one16:56
zigomtreinish: I did add keystone creds, yes, but that's not enough?16:56
Alex_Gaynorclarkb: how are the different worker types distributed between the different providers?16:56
clarkbfungi: yes we can twiddle it in novaclient but we specifically chose not to because nova will do it for you16:56
zigomtreinish: I wanted to try with only keystone stuff ...16:56
jogoclarkb: its something you can specify to get better spread accross failure domains16:56
*** _nadya_ has joined #openstack-infra16:56
*** atiwari has quit IRC16:56
fungiclarkb: well, at least we were told nova would do it for us16:56
fungiclarkb: but ultimately i think it's going to be up to hpcloud whether they make that happen or not16:56
clarkbfungi: yes, I just want to understand if nova is actually failing us here so that we can hopefully fix upstream nova16:57
zigomtreinish: I was running "testr run"16:57
clarkbit may be the the nova scheduler is scheduling us in the only az with room16:57
zigomtreinish: FYI, I'm trying to package tempest in Debian...16:57
clarkbor some other contraint is influencnig the system16:57
fungiclarkb: certainly entirely possible16:57
jeblairclarkb, fungi, mordred, SergeyLukjanov: can you review when you get a chance?16:57
tteggelclarkb: hpcloud explicitly chooses to schedule you to az2 if you don't specify16:57
mtreinishzigo: ok, yeah the thing is if the config is incomplete and we have a bug that requires something at import then when test discovery is run it'll fail16:58
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Fixes misaligned comments on very long lines
mtreinishwhich is probably what you're hitting16:58
zigomtreinish: Ok, then what is mandatory to edit?16:58
clarkbtteggel: oh well wtf16:58
tteggelclarkb: this is due to a constraint in our cinder backend that doesn't allow cross-az volume attach16:58
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Added priority on task row
tteggelclarkb: and without a scheduler that can do az affinity of volumes / compute then a key use case for a lot of users becomes painful16:59
*** tkelsey has quit IRC16:59
mtreinishzigo: the bare minimum should be the keystone creds, auth url and the enabled services. But, depending on the services enabled you have other things needed. For example nova tests require image and flavor ids17:00
mtreinishzigo: I don't think we've ever mapped out exactly what the min set of options was to run tempest17:00
zigomtreinish: Thanks, will try then.17:00
mtreinishzigo: that being said you probably hit a bug because we don't want to be dependent on a config file at import time17:01
mtreinishit makes things like unit testing hard17:01
clarkbjeblair: yup I can17:01
clarkbfungi: have we started working on a nodepool patch yet?17:01
*** srenatus has quit IRC17:01
fungiclarkb: not yet, pending further confirmation that was the necessary solution17:02
*** lcostantino has joined #openstack-infra17:02
fungiclarkb: though i don't think the actual feature would be hard to add17:02
*** srenatus has joined #openstack-infra17:02
openstackgerritA change was merged to openstack-infra/storyboard: Added new priority changed event for tasks
*** atiwari has joined #openstack-infra17:02
mtreinishzigo: what happens when you try to manually import the file testr spit out in the python interpreter?17:02
clarkbharlowja_: nope we will need to trigger it on the gearman bus directly. Once we get slave availability sorted I can stab at rerunning te jobs17:02
*** matjazp has joined #openstack-infra17:02
harlowja_clarkb sounds great, thx :)17:03
SergeyLukjanovjeblair, looking17:03
*** julim has joined #openstack-infra17:03
clarkbfungi: ya I don't think so either. More an issue of how we want to do the assignment of AZ17:03
clarkbfungi: also iirc networks are cross AZ so not really sure what benefit there is of having AZs at all17:03
*** arnaud__ has joined #openstack-infra17:03
*** ominakov has joined #openstack-infra17:03
zigomtreinish: It just works ...17:03
clarkbtteggel: ^ is that true or do we need to create routers and networks per AZ? when I created them it seemed to not care what the AZ was17:04
tteggelrouter and networks are cross-az. volumes are not.17:04
clarkbfungi: jeblair what if we specify a list of AZ to the nodepool provider and just pick them at random when booting nodes?17:05
fungiclarkb: so given that we don't use cinder for nodepool, yes i think that would make sense17:05
clarkbor would it be more flexible (for volume things) to have a different provider per AZ?17:05
clarkband go to 15 providers for 1.1 east?17:05
*** ArxCruz has quit IRC17:05
mtreinishzigo: yeah that definitely sounds like a config issue. The other thing you can do is try to manually run it with unittest or So something like 'python -m unittest module'17:06
jeblairclarkb: so images are shared across az's, volumes are not.  right?  what about networks?17:07
*** ihrachyshka has quit IRC17:07
clarkbjeblair: images and networks are cross AZ17:07
clarkbvolumes are not. As we use nodepool today without volumes we should be fine just picking an AZ at random17:07
fungiclarkb: and i guess if we later encounter a provider where these things are not cross-az, then we just create separate providers with singleton az lists17:07
clarkbfungi: ya17:08
*** rfolco has joined #openstack-infra17:08
clarkbthat gives us more flexibility imo17:08
*** ihrachyshka has joined #openstack-infra17:08
mtreinishlifeless: when you're around later what do you think about doing something like:
*** esker has quit IRC17:08
mtreinishlifeless: at least until we have a real fix for
*** amotoki has quit IRC17:09
*** bhuvan has joined #openstack-infra17:09
*** esker has joined #openstack-infra17:09
*** esker has quit IRC17:09
*** esker has joined #openstack-infra17:10
*** HenryG has quit IRC17:10
*** HenryG has joined #openstack-infra17:10
*** amotoki_ has quit IRC17:11
jeblairclarkb: yeah, i think on balance we should probably do your list suggestion then.17:11
*** _nadya_ has quit IRC17:12
*** talluri has joined #openstack-infra17:12
clarkbjeblair: left a comment on the specs change. can you see if that makes sense?17:12
jesusaurusclarkb: images are shared cross az?17:12
clarkbjesusaurus: yes17:13
*** ewindisch has joined #openstack-infra17:14
openstackgerritJames E. Blair proposed a change to stackforge/gertty: Support background sync of missing refs
openstackgerritJames E. Blair proposed a change to stackforge/gertty: Add dependency navigation
openstackgerritJames E. Blair proposed a change to stackforge/gertty: Remove stray debug line
jeblairclarkb: so sorry about the missing migration!17:14
clarkbjeblair: its ok I tested on the other computer :)17:15
clarkbfungi: jeblair: I suppose I should start working on an AZ list then?17:15
jeblairclarkb: do you want to do the np change or shall i?17:15
clarkbI can do it. One last question. Should subnodes live in the same AZ? I am leaning towards yes17:16
jeblairclarkb: sounds reasonable17:16
*** mmaglana has quit IRC17:17
fungiyeah, probably, for sanity more than anything i think (i expect it would make the change smaller)17:17
*** ihrachyshka has quit IRC17:19
*** mmaglana has joined #openstack-infra17:20
jesusaurusthough subnodes being in other azs would uncover more issues17:21
jeblairjesusaurus: i think on balance we probably want them topologically close.17:22
fungijesusaurus: well, with the current nodepool design a subnode is useless without the head node to which it's associated, so if an az goes offline then we17:22
fungi're better off losing them all together17:22
jesusaurusjeblair: good point17:23
clarkbjeblair: fungi: actually sharing AZs with subnodes may make the code more complicated. Nodepool._run() iterates in a way that decouples the two sets17:23
fungithinking to the distant future where we have these in use and doing something, the more distributed between azs a multi-node group is, the more groups we lose to sudden loss of a single az17:23
jesusaurusfungi: i wasnt thinking of errors from losing an az, i was thinking about assumptions of being on the same subnet17:23
clarkbactually wait, I have a node object there, if we assign an AZ to it that should be fine17:23
jeblairclarkb: yeah that.  :)17:23
bodepdjeblair: to CLA or not to CLA for openstack-ifra?17:27
*** arunkant has joined #openstack-infra17:27
fungibodepd: generally not17:27
bodepdfungi: ok, I'm not changing it again after this :)17:28
*** fbo is now known as fbo_away17:28
*** Ajaeger1 has joined #openstack-infra17:28
fungibodepd: there are a few projects in openstack-infra which specify receive.requireContributorAgreement but the majority do not17:30
jeblairapparently 33% of our repos require the cla.  we're a bit inconsistent17:30
*** dims has quit IRC17:30
*** atiwari has quit IRC17:30
jeblairthere are hysterical raisins for that.17:31
fungibodepd: and for something being split out of openstack-infra/config, we shouldn't add receive.requireContributorAgreement because also lacks it17:31
jeblairfungi: that's as good an argument as any.  :)17:31
* fungi would come up with other convincing arguments too if needed ;)17:32
*** nati_ueno has joined #openstack-infra17:32
*** gokrokve has joined #openstack-infra17:32
arunkanthi...seeing this build failure  when change was set to merge..  It appear to hit some known bugs as mentioned by elastic-recheck. How to address this?17:32
jeblairbodepd: that work for you?17:33
*** denis_makogon_ has joined #openstack-infra17:35
*** atiwari has joined #openstack-infra17:35
fungimordred_phone: dhellmann: if pbr>=0.6,<1.0 is already specified in setup_requires for a project, should we still be adding it to requrements.txt as well?17:35
*** amcrn has joined #openstack-infra17:35
*** timrc is now known as timrc-afk17:35
*** zzelle_ has joined #openstack-infra17:36
mordred_phonefungi: depends17:36
mordred_phoneif the project uses pbr at runtjme17:36
fungimordred_phone: ahh, okay. so the specs project definitely doesn't have a run-time dependency on that17:37
mordred_phonefungi: that said. setup_requires should not be versioned17:37
bodepdjeblair: yes. it sounds like a decision. I woudl have preferred CLAs, but I'm much happier with a decision17:37
anteayajeblair fungi that makes life difficult for me reviewing acl patches17:37
fungimordred_phone: adding a note about that as well17:37
*** denis_makogon_ is now known as denis_makogon17:37
*** maxbit has quit IRC17:38
openstackgerritDan Bode proposed a change to openstack-infra/config: Add puppet-pip as a unique project
fungibodepd: you prefer our cla? i'm moderately frightened17:38
jeblairanteaya: yeah, it's tricky and i don't think we know what the right answer is.  we may need to turn them all on for infra projects some day.  or, hopefully, we may get to turn them all off.17:38
anteayajeblair: stackforge projects have requireCLA = true17:39
*** dkliban_lunch is now known as dkliban17:39
jeblairanteaya: i don't consider the CLA a review criteria for stackforge projects17:39
anteayaI don't ask stackforge projects to change it17:39
anteayaif they have it or not I just leave it17:39
*** sandywalsh_ has quit IRC17:39
anteayabut bodepd had it = true and was -1'd for it17:39
fungihopefully only stackforge projects which are trying to become official components of openstack itself are bothering with applying our cla17:39
anteayaI don't do that to stackforge projects17:40
anteayathat is what I assume as well, fungi17:40
jeblairthat's not what i assume, but it is what i hope.17:40
anteayaeither way, whatever they have, I just leave it if it is stackforge17:41
fungiyes, i ASSUME many of then do it as a cargo-cult behavior17:41
anteayaanything under openstack, I request CLA = true in the acl17:41
bodepdfungi: CLAs are there to help resolve copyright concerns and other lagal nonsense.17:41
bodepdI'm happy with anything that makes code more legit from a legal perspective17:41
fungibodepd: some clas are, i'm unconvinced our cla is17:42
bodepdI'm also becoming a suit these days :)17:42
anteayaand i appreciate your quick turn around on your patches, bodepd17:42
jeblairour cla is legal nonsense17:42
anteayathat maybe the case and there is a process for that17:42
bodepdfungi: well, I guess part of that was me assuming that the OS CLA had some value other than generating legal revenue ;)17:42
anteayabut in regards to what projects should do about it, my divinding line has been the use of the openstack namespace17:42
*** _nadya_ has joined #openstack-infra17:43
jeblairbodepd: i'm pretty sure that's its only value  :/17:43
fungianteaya: yes, that's what i've been doing as well17:43
anteayaso getting -1'd for including it when using the openstack namespace confuses me17:43
anteayasince I now have no direction on how to review future patches17:43
anteayaand "ask in channel" become a bottleneck17:44
jeblairbodepd: in all honesty, in your capacity as a suit if you are considering CLAs for open source projects, i would be happy to share some resources and introduce you to some folks who hold very sensible, legally supportable, non-revenue-generating opinions on the subject.17:44
fungianteaya: what openstack namespace? or did you mean you were doing it for openstack-infra and openstack-dev too?17:44
anteayaI have been17:44
*** gokrokve has quit IRC17:45
jeblairbodepd: i care a lot about free software and i want the contribution barrier to be minimal, which is why i prefer no CLAs or something like the kernel DCO model17:45
*** gokrokve has joined #openstack-infra17:45
jesusaurusjeblair: is that the signed-off-by process?17:45
jeblairjesusaurus: yes17:45
fungianteaya: ahh, i'd only been highlighting it as considered necessary for actual runnable code and libraries (but not documentation-only projects) explicitly within the openstack/* namespace (treating it optional in stackforge and suggesting against it in openstack-dev and openstack-infra but still optional)17:46
jeblairjesusaurus: (dco == developer certificate of origin; section 12)17:46
anteayafungi: why suggesting against it in the -dev and -infra spaces?17:47
fungianteaya: i hadn't seen any implication that it was strictly required except for software components of openstack itself17:48
anteayawhy would it not be required for openstack-infra and -dev?17:49
fungiand we generally put things in openstack-dev or openstack-infra which are supporting cast (used to develop and test openstack or to support its community and ecosystem, but not which would be something people would run and call an openstack cloud or an official channel for controlling one)17:49
anteayayes, that is what I understood to be the case17:50
anteayathough the infra projects are all in the governance/programs.yaml repo17:50
anteayaor should be17:51
anteayaand openstack-dev/devstack is a program17:51
fungii fail to understand what being under the governance of the tc has to do with a contributor license agreement17:51
clarkbok tox is running on what I have so far.17:51
*** timrc-afk is now known as timrc17:51
anteayathat is the reference I use for what constitutes an official openstack program17:51
anteayaopenstack-infra is an openstack program17:52
fungisure, official openstack program is broader than "something i would run and market as an openstack cloud"17:52
*** _nadya_ has quit IRC17:52
anteayaso what is the scope of the cla?17:52
anteayaand its expected usage17:53
*** ihrachyshka has joined #openstack-infra17:53
anteayaI had thought it was to be used with openstack programs17:53
fungithe lawyers who drafted the cla expressed concern about the openstack product (the cloud software competing with amazon et al) being compromised by bad actors within the contributing community, effectively tainting it for other member companies who also want to run and sell services on it17:53
fungie.g. foocorp wants some additional assurance that barinc won't sneakily introduce intellectual property into nova which prevents them from being able to continue to make money off of it17:55
*** nati_ueno has quit IRC17:55
anteayahence the third-party party17:55
fungiand i'm as of yet unconvinced it even does that17:55
*** nati_ueno has joined #openstack-infra17:55
anteayait doesn't17:55
anteayaso in anycase, we have a set of rules which we are trying to change17:56
anteayaand in the meantime we are still stuck with17:56
anteayaand what I do, I would prefer to do consistently17:56
fungianyway, it's been my understanding that only components of an openstack cloud, maybe also dependent libraries and official client interfaces, are of interest to those who want the cla enforced17:56
*** dims has joined #openstack-infra17:56
*** apevec has quit IRC17:56
anteayaso that if I post a review to bodepd he doesn't get -1'd down the road for impliementing the thing I asked him to change17:56
*** ihrachyshka has quit IRC17:57
anteayafungi: what documentation could I read to validate that, do you know?17:57
*** salv-orlando has joined #openstack-infra17:57
openstackgerritYuriy Taraday proposed a change to openstack-infra/git-review: Add an option to mark uploaded patchsets as WIP
fungianteaya: you can reread the cla and form your own opinions as to what it says, or consult your legal counsel. you are free to disagree and/or -1 any changes you see fit17:58
fungii'm merely trying to explain my interpretation of the same document17:58
anteayaso there is no documentation outside the cla that indicates its scope or use, just the cla?17:58
anteayafungi: I understand17:58
anteayaand I appreciate it17:58
*** sandywalsh has joined #openstack-infra17:58
*** mbacchi has quit IRC17:59
fungianteaya: also
*** Ryan_Lane has joined #openstack-infra18:00
*** nati_ueno has quit IRC18:00
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists
clarkbjeblair: fungi ^18:00
*** dims has quit IRC18:01
*** dkranz has quit IRC18:01
fungianteaya: it's sort of an ouroboros, in that the bylaws define that the foundation will accept contributions under a cla, and the cla grants contributions to the foundation18:01
*** marcoemorais1 has joined #openstack-infra18:01
*** marcoemorais1 has quit IRC18:02
clarkbnote I don't haev amysql daemon on my laptop (and don't really want one) so we should make suer that passes unittests18:02
*** marcoemorais1 has joined #openstack-infra18:02
*** blamar has joined #openstack-infra18:02
*** marcoemorais has quit IRC18:02
*** dkranz has joined #openstack-infra18:04
anteayatrinaths: please join us in #openstack-meeting for the third-party meeting18:05
anteayatrinaths: we need to address some of your questions18:05
fungiclarkb: would it make sense to add an availability-zone list to one of the test yaml configs, at a minimum (or better, a test that they work)?18:06
*** srenatus has quit IRC18:06
jeblairfungi: i just got an interesting bounce message from static with output from the log archive maintenance script18:06
*** srenatus has joined #openstack-infra18:06
*** julim has quit IRC18:06
clarkbprobably. I suppose I should find a better test machine and do that18:07
* fungi looks18:07
clarkbfungi: though the mocking is rudimentary iirc18:07
fungijeblair: bounce for my address?18:07
sdagueanteaya: yeh, I kicked off the legal thread / hornet nest this weekend because of our conversation on it on friday18:08
jeblairfungi: i didn't pay attention to whose bounce; it was a msg too big bounce -- it was the contents of the original message i found interesting18:08
*** ihrachyshka has joined #openstack-infra18:08
anteayasdague: yeah, I felt that18:08
*** zul has quit IRC18:08
jeblairfungi: i think we all got the bounce though, so you should be able to see it18:09
*** julim has joined #openstack-infra18:09
jeblairfungi: the bounce is Message-Id: <>18:09
fungijeblair: yeah, reading it now18:09
fungijeblair: looks like it tried to remove directories which were not yet empty18:10
*** SumitNaiksatam has quit IRC18:10
fungirereading the current script now and checking some of the examples on disk to see if i can spot why18:11
fungi-type d -not -empty -mtime +1 -execdir rmdir {} \;18:12
sdagueanteaya: my -1s on the infra split out I thought were pretty well reasoned in that there was not a CLA in the parent repo, so it didn't seem like there should be on in the split outs18:12
*** mbacchi has joined #openstack-infra18:12
fungithat doesn't look right. maybe i inadvertently introduced/orphaned a stray -not in that18:12
*** yfried has joined #openstack-infra18:12
*** zul has joined #openstack-infra18:13
*** pblaho has quit IRC18:13
*** dims has joined #openstack-infra18:13
fungipatch on the way18:13
fungisomehow that wasn't exposed when i tested it manually18:13
mordredclarkb: nice patch18:14
openstackgerritDan Bode proposed a change to openstack-infra/config: Add project for zuul
mordredjeblair, clarkb: ^^ I know it doesn't happen often - but since we've got it in place other places now successfully .. should we alembic nodepool?18:14
*** nati_ueno has joined #openstack-infra18:14
openstackgerritDan Bode proposed a change to openstack-infra/config: Add puppet-pip as a unique project
openstackgerritDan Bode proposed a change to openstack-infra/config: Add project for jenkins
jeblairmordred: oh. hm.  probably not a bad idea.18:17
openstackgerritDan Bode proposed a change to openstack-infra/config: Add project for zuul
mordredjeblair: k. cool. I'll put that on my plane hacking list if nobody gets to it18:18
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Fix empty directory removal for log expiration
fungijeblair: ^18:18
lbragstadsdague: mrodden and I had a quick question for you on the whole gerrit-powered-agendas subject. You had some ideas about publish to the wiki correct?18:18
*** _nadya_ has joined #openstack-infra18:19
sdaguelbragstad: I was just going to suggest be ghetto and mechanize18:19
clarkbmordred: jeblair do you know if the sqlalchemy <-> mysql translation of None to and from the mysql type does the right thing?18:20
jeblairmordred: you'll note that i did not do it the same way in gertty as storyboard does.  i thought it would be worth starting from first principles in gertty because i found the storyboard implementation a bit hard to follow.  i have learned quite a bit in the process but have not yet had a chance to apply what i learned to storybard.18:20
clarkbI am assuming it will set teh value to null and everything will be fine18:20
jeblairclarkb: None==null yes18:20
mordredjeblair: awesome. I'll look at gertty then18:20
*** marcoemorais1 has quit IRC18:20
*** marcoemorais has joined #openstack-infra18:20
lbragstadsdague: ok, currently, the way the team has it, the publish functionality is something that is controlled by the end-user... which I don't think is necessarily correct since it should only publish to the openstack wiki on post jenkins jobs, after an approval... right?18:21
jeblairstorybard, btw, was an englishman who wrote a bunch of plays during the elizibethan era18:21
sdaguelbragstad: that seems reasonable18:21
jeblairclarkb: reviewing now18:21
lbragstadsdague: so would we do that with a python task or... any other good ideas on how to best accomplish that with a jenkins post job?18:21
*** e0ne has joined #openstack-infra18:22
mkodererjeblair: mtreinish: hi are you around for a short hangout session?18:22
mkodererabout the mid cycle summit..18:22
sdaguelbragstad: now you are beyond my general knowledge on how to do that best18:22
sdaguebut plenty of folks in here should know18:22
bodepdthe fact that there was a merge conflict b/c someone was able to get puppet-yum excepted before me really indicates I'm doing something wrong :)18:22
jeblairmkoderer: oh, sorry, i don't use google hangouts.  happy to irc or phone call though.18:23
lbragstadsdague: ok, sounds good, thanks!18:23
mkodererjeblair: ok, phone is also ok for me18:23
*** ihrachyshka has quit IRC18:25
*** _nadya_ has quit IRC18:25
*** ihrachyshka has joined #openstack-infra18:25
mtreinishmkoderer: yeah I'm around18:25
*** _nadya_ has joined #openstack-infra18:25
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists
clarkbjeblair: fungi ^ now with tests that I haven't run locally18:26
*** sarob_ has joined #openstack-infra18:26
*** ihrachyshka has quit IRC18:26
fungiclarkb: i know someone who will run them for you18:26
mkoderermtreinish: ok cool.. we can do it via IRC.. I just thought it would be faster with voice :)18:26
*** amcrn has quit IRC18:27
*** amcrn has joined #openstack-infra18:28
*** ociuhandu has quit IRC18:28
clarkbfungi: unfortunately that someone is so far behind :)18:28
*** matjazp has quit IRC18:28
*** sarob has quit IRC18:29
*** homeless_ has joined #openstack-infra18:30
*** _nadya_ has quit IRC18:30
mordredclarkb: here's hoping that better distribution across the three azs will help the errors18:30
openstackgerritA change was merged to openstack-infra/devstack-gate: features: do not drop ceilometer when using grenade
mroddenlbragstad: i'd suggest prototyping it as python18:32
mroddenand we can figure out what belongs where later18:33
openstackgerritA change was merged to openstack-infra/config: Add #openstack-sahara to status bot
mordredclarkb: WOW. the current error spike is super impressive18:33
lbragstadmrodden: ok18:33
*** homeless has quit IRC18:33
jeblairclarkb: okay, finally finished leaving comments on PS1 of your change :)18:33
mroddennot sure if moin moin has an actual API or if we're gonna have to write a web crawler18:33
clarkbjeblair: looking18:33
tteggelmordred: ;(18:34
tteggelmordred: resolved now18:34
*** ildikov_ is now known as ildikov_afk18:35
mordredtteggel: neat. honestly, the spike is still impressive :)18:36
*** matjazp has joined #openstack-infra18:36
fungimrodden: lbragstad: we've been on mediawiki for over a year now. moin moin is history18:37
mordredjeblair: can we april fools a storybard install?18:37
*** [1]trinaths has joined #openstack-infra18:37
*** matjazp has quit IRC18:38
*** zzelle has quit IRC18:38
openstackgerritSean Dague proposed a change to openstack-infra/config: add tempest jobs to dib and os-* tools
*** rlandy_ has joined #openstack-infra18:41
*** trinaths has quit IRC18:41
*** [1]trinaths is now known as trinaths18:41
*** ihrachyshka has joined #openstack-infra18:41
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists
clarkbjeblair: ^ that addresses your comments18:42
*** rlandy__ has quit IRC18:43
*** srenatus has quit IRC18:43
*** srenatus has joined #openstack-infra18:44
*** fbo_away is now known as fbo18:44
*** reed has joined #openstack-infra18:46
*** homeless_ has quit IRC18:49
jeblairclarkb: thanks, lgtm except can you add a sentence to the docs about the new option?18:49
*** andreykurilin has joined #openstack-infra18:49
jeblair(sorry i forgot that the first time)18:49
*** openstackstatus has quit IRC18:50
anteayaopenstackstatus has quit18:51
clarkbjeblair: sure18:51
*** openstackstatus has joined #openstack-infra18:51
*** ChanServ sets mode: +v openstackstatus18:51
*** rlandy__ has joined #openstack-infra18:51
*** bauzas has quit IRC18:52
*** homeless has joined #openstack-infra18:53
*** rlandy_ has quit IRC18:54
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists
clarkband done18:55
*** hogepodge has quit IRC18:55
openstackgerritA change was merged to openstack/requirements: Move oslotest to test catalog
*** nati_uen_ has joined #openstack-infra18:58
*** _nadya_ has joined #openstack-infra18:58
*** nati_uen_ has quit IRC18:59
*** mrda-away is now known as mrda18:59
*** hashar has joined #openstack-infra18:59
*** nati_uen_ has joined #openstack-infra18:59
*** esker has quit IRC19:00
*** nati_ueno has quit IRC19:02
anteayajeblair mtreinish so when the question came up, can third-party ci folks attend the infra/qa meetup, I told them the intent is to encourage new core reviewers and if they are willing to commit to learning how to review 30 patches in 30 days (as a minimum) they could request to attend19:02
anteayajeblair mtreinish and if they wanted to learn how to set up their systems, they needed to ask questions and participate in the meeting space on irc19:03
pleia2everyone should attend our meetings :)19:03
*** tristanC has quit IRC19:03
*** lttrl has joined #openstack-infra19:03
anteayapleia2: meetup19:04
pleia2anteaya: ooh, the one next month19:04
anteayatimestamp 18:10:0319:04
jeblairclarkb: AttributeError: 'TestNodepool' object has no attribute 'aseertEqual'19:04
anteayaI knew there was going to be confusion and there was19:04
fungiaseert! aseert!19:04
anteayabut i think I dealt with it clearly19:04
*** trinaths has quit IRC19:04
*** sanjiva has joined #openstack-infra19:05
*** lcostantino has quit IRC19:05
mtreinishanteaya: I'm not sure about putting hard restrictions on it. But, it's definitely not about learning how to setup a third party ci system.19:05
anteayamtreinish: I needed to put something around it19:06
anteayamtreinish: if you don't want to enforce that, that is your call19:06
mtreinishwe're going to have a day or 2 of bootstrapping to explain the architecture of infra and qa, and how they work around it19:06
*** esker has joined #openstack-infra19:06
clarkbjeblair: gah thank you19:06
anteayabut they read it as, we can show up and get our systems working19:06
pleia2speaking of meeting meetings, might be a little late tomorrow because I have an appointment in the morning19:06
anteayamtreinish: since that is how they see everythign19:06
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Support provider AZ lists
anteayapleia2: k, hope it is good news19:07
*** yfried has quit IRC19:07
*** yfried has joined #openstack-infra19:07
mtreinishyeah, that's definitely not what it is19:07
anteayamtreinish: nod19:07
anteayayeah these folks want things and have no immediate use for it19:07
anteayaI'm glad they are speaking up19:07
*** duran has quit IRC19:07
anteayasometimes it is just frustrating19:08
anteayaI'm going to take another walk before it rains19:08
bookwaranteaya: actually i am considering to participate, because I want to learn more about how openstack infrastructure works19:08
anteayabookwar: know that this is not designed to answer questions about how to set up your third party system19:08
anteayaas long as you are prepared to follow the agenda and sprit of the meetup you can request to attend19:09
Longgeekanteaya: I think I have closed the start job Comments.19:09
anteayaLonggeek: test your system on the sandbox repo19:10
Longgeekanteaya: ok19:10
*** bauzas has joined #openstack-infra19:10
anteayaI want to see at least 5 comments from your system on the sandbox repo before we discuss anything else19:10
anteayaI played all sorts of whack a mole with third party people demanding to attend the neutron tempest code sprint who had no interest in actually contributing to neutron or tempest that I have no interest in doing that again19:11
*** marcoemorais has quit IRC19:12
*** wenlock has quit IRC19:13
zarofungi: i was going to pick this up again.  wanted to double check with you whether there's anything larger required than just cleaning from your suggestions.
anteayadidn't even make it 10 metres (rain)19:13
Longgeekanteaya: Has confirmed the closure19:13
anteayaLonggeek: post the url of one of the sandbox patches19:13
*** otherwiseguy has quit IRC19:14
*** praneshp has joined #openstack-infra19:14
*** radez_g0n3 is now known as radez19:15
*** tristanC has joined #openstack-infra19:15
Longgeekanteaya: patch link:
*** mbacchi has quit IRC19:16
*** andreaf has quit IRC19:17
*** arnaud__ has quit IRC19:19
anteayaLonggeek: okay so well done you have successfully gotten yourself to one commet per patchset19:20
anteayaLonggeek: do you remember what you did to accomplish that?19:20
*** sarob_ has quit IRC19:20
Longgeekanteaya: yes19:20
anteayagood, write it down19:20
Longgeekanteaya: remember19:20
anteayayou are going to present it at next week's meeting19:20
clarkbjeblair: fungi: I am going to wander and find lunch then an outdoor hacking spot19:21
*** sarob has joined #openstack-infra19:21
clarkbwill be afkish for the next bit. feel free to edit my nodepool change if you need to19:21
anteayanoone spoke up to help you when I asked, so now you can teach them all how to do it19:21
Longgeekanteaya: No problem19:21
anteayaLonggeek: now what projects do you want to test?19:21
*** yfried has quit IRC19:21
anteayapick one to start19:21
*** yfried has joined #openstack-infra19:21
Longgeekanteaya: devstack19:21
*** _nadya_ has quit IRC19:21
clarkbalso will need to whip up the corresponding infra/config change to add the az's19:22
*** lcostantino has joined #openstack-infra19:22
anteayago into #openstack-qa and look for mtreinish19:22
Longgeekanteaya: Write more appropriate in where?19:22
*** marcoemorais has joined #openstack-infra19:22
anteayaLonggeek: make your personal notes and you will present them at next week's third-party meeting19:22
Longgeekanteaya: ok19:22
anteayaask mtreinish if he will allow you to test and comment on devstack19:23
anteayashow him your sandbox repo19:23
anteayaaddress any questions or concerns he has19:23
anteayaand if you don't know how, ask me19:23
Longgeekanteaya, Good, thank you.19:23
anteayaLonggeek: if he let's comment on devstack patches, stay with only that project before adding any others19:24
anteayacome and talk to me before adding any other projects to what you test19:24
Longgeekanteaya: ok, clear19:25
*** sarob has quit IRC19:25
*** sarob_ has joined #openstack-infra19:25
mtreinishanteaya: heh, I'm not ptl of devstack :)19:25
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add availability zones to hpcloud region b
*** nati_uen_ has quit IRC19:25
clarkbjeblair: fungi mordred ^ that is the corresponding config chaneg and now to find food19:26
*** kaitlin-farr has joined #openstack-infra19:26
anteayamtreinish: though dtroyer is away19:26
Longgeekmtreinish: Who i ptl of devstack.19:26
clarkbyou can check the az names with `nova availability-zone-list`19:26
anteayamtreinish: and I am getting him into the habit of going into another channel and seeking permission to test19:26
*** marcoemorais1 has joined #openstack-infra19:26
*** marcoemorais1 has quit IRC19:27
*** nati_uen_ has joined #openstack-infra19:27
*** marcoemorais2 has joined #openstack-infra19:27
*** marcoemorais has quit IRC19:27
fungizaro: 47937 may need some updating to reflect changes which were introduced as part of the gerrit upgrade. also it's been tabled long enough we should have infra and the vmt confirm this is still the direction we want to go (just so we don't waste more of your time). aside from that, yeah really just cleanup19:28
sdagueso my question is mostly I don't know what is19:28
anteayasdague: good question19:28
sdagueso I'm not sure why it would be running on devstack changes19:28
anteayanether do I19:28
*** xyang1 has joined #openstack-infra19:29
anteayabut at least it isn't commenting twice per patchset, which was my goal19:29
sdaguewhich was the reason I raised the question last week19:29
anteayanobody knows what it is testing19:29
anteayaand I have been so busy teaching ettiquette I haven't had time to ask19:30
*** talluri has quit IRC19:30
xyang1anteaya: hi, so my question is if we have multiple CI systems, should we have multiple upstream service accounts, or just one?19:30
anteayawell for our sake it is much easier if you can do just one ci account19:30
*** andreaf has joined #openstack-infra19:31
anteayaand it will be easier for you if you have a dedicated person to stay on irc to monitor your account and deal with issues and questions as they arise19:31
fungixyang1: multiple ci systems won't be able to vote independently of one another if they use the same account19:31
xyang1sure, so we should be able to submit test results from multiple CI systems, using the same account, right?  that19:31
anteayaxyang1: what projects do you want to test?19:32
xyang1the reason for multiple CI systems is that we have multiple cinder drivers19:32
xyang1each driver will be tested at a different CI system19:32
anteayahow many cinder drivers do you have?19:32
fungiif they're testing different things and aren't controlled by the same zuul or gerrit trigger plugin instance, then one succeeding on a patchset after the other fails on the same patchset may mask failure results19:32
anteayaah very true19:32
anteayathis is going to get nice and confusing19:33
xyang1we don't have one central place that can test all 4 drivers for Juno19:33
xyang1hopefully we can consolidate in the future19:33
zarofungi: what's the path to confirmation?19:34
xyang1each Cinder driver works with a different storage platform.  So results could be different19:34
fungion the other hand if they're just four jobs testing four different things, you should try to engineer it so that they can combine reporting if possible (for example upstream zuul runs multiple jobs on a change, possibly using different jenkins masters entirely, then waits for them all to complete before posting an aggregated vote based on the results of all the jobs it ran)19:34
*** homeless_ has joined #openstack-infra19:35
xyang1fungi: there will be 4 CI systems at different labs19:36
fungizaro: remind me tomorrow morning, and lets try to get it onto the infra team meeting agenda as a quick sync-up topic. i'll make sure to confirm with the rest of the vmt today, though ttx won't be around to answer until tomorrow i think19:36
fungixyang1: yeah, and if you can't run one zuul instance somewhere which has access to talk to systems at all 4 of those labs, then that would be a bit of a blocker for using a shared account for them19:36
fungixyang1: though it's perhaps something to shoot for as a future goal19:37
clarkbfungi: in theory if all 4 can talk to gerrit they can talk to a central zuul19:37
*** homeless has quit IRC19:37
fungiclarkb: yep19:37
clarkbfungi: they are both outbound connections19:37
clarkbI neglected to do a gertty sync this morning19:37
clarkbI may just give up on that and do lunch anyways19:38
sdagueare things just flushing in zuul at this point, and we just need to wait for backlog to clear?19:38
lifelessmtreinish: tht looks like a no-op to me :)19:38
lifelessmtreinish: that said, the layer to carry a local fix is testtools and there is a PR there19:39
clarkbsdague: we don't have enough nodes19:39
sdagueclarkb: gotcha19:39
lifelessmtreinish: I'd like to see someone put up the fix in Python and us carry a backport, rather than us carry a workaround without the upstream fix being proposed.19:39
clarkbsdague: and the slope on the waiting jobs graph is very positive19:39
fungiclarkb: true, if all the jenkins masters at xyang1's different labs can reach the gearman service on a common zuul, then that should be sufficient19:39
xyang1fungi: so we should go with 4 accounts for now?  and in the future we'd like to consolidate everything in one central place19:39
sdagueclarkb: yeh, is that reaction to everything exploding and having to recheck the 451 failures from setup tools?19:39
clarkbsdague: that + hpcloud 1.0 being turned off19:40
sdagueor are we looking generally over capacity again?19:40
sdagueah... yeh, that would be an issue19:40
clarkband 1.1 is unable to provide consistent capacity. A potential fix is up for review now19:40
lifelessbye bye diablo19:40
sdagueshould we start pruning jobs?19:40
clarkbsdague: not yet. if the nodepool change makes tis better we should be ableto catch up19:40
sdaguelike the fact that nova is kicking off 2 extra silent jobs on every rev19:40
clarkbwe will essentially double our node count19:41
sdagueok, cool19:41
*** gyee has quit IRC19:41
mtreinishlifeless: ok yeah that makes sense19:41
clarkbsdague: we might need to prune if this lasts a whiel19:41
lifelessmtreinish: so yeah, if you want to move it forward - work upstream, I'll happily review there (and help find python-cores) and once you have something /working/ and plausible, we'll carry a copy of it in testtools19:42
sdagueclayg: ok, let me know if you need some help selecting stuff to prune19:42
*** trinaths has joined #openstack-infra19:43
*** gabriel-bezerra has joined #openstack-infra19:43
*** wenlock has joined #openstack-infra19:44
*** [1]trinaths has joined #openstack-infra19:44
*** e0ne has quit IRC19:46
fungixyang1: if you think you can make it work with one account, then that's fine. otherwise request four accounts for the equipment running tests in each of your labs (preferably in a single request) and make sure to include a brief explanation of why you're currently unable to do it with one account (since that's almost certain to get asked again otherwise)19:46
fungisdague: tab-complete fail?19:47
xyang1fungi: sure.  We are still in the early stage of setting them up.  I think we are likely going to request 4 accounts for Juno.  moving forward, we hope to consolidate with 1.  Thanks!19:47
*** trinaths has quit IRC19:47
*** [1]trinaths is now known as trinaths19:47
sdaguefungi: yep19:49
sdagueit's unacceptable that there are 3 cla<tab> nicks in here :)19:49
anteayayour the only sd<tab>19:50
anteayalucky guy19:50
*** ominakov has quit IRC19:50
*** nati_uen_ has quit IRC19:51
*** nati_ueno has joined #openstack-infra19:51
fungithere are only 5 nicks here starting with f (and i'm the only fu, which sounds like a joke but i assure you it's not)19:52
*** _nadya_ has joined #openstack-infra19:52
anteayaha ha ha19:52
*** ArxCruz has joined #openstack-infra19:52
anteayait was worth you saying it though19:52
* fungi enjoys the thought that when anyone replies to him, they're probably starting by typing fu[tab]19:53
anteayafungi: I know I do19:53
*** ominakov has joined #openstack-infra19:54
dstufftfungi: lol I never noticed that19:55
*** nati_ueno has quit IRC19:56
pleia2fungi: I'm coming to debconf this year (yay!) so if you're planning on doing a "how openstack ci can make debian more awesomer" kind of thing, I'd love to help out19:56
fungipleia2: ooh, i hadn't even looked to see when the cfp opens/closes19:57
fungi(or whether i'm due to be somewhere else at the same time)19:58
*** openstackgerrit has quit IRC19:58
clarkbpleia2 at psu?19:59
pleia2clarkb: yep19:59
fungiclarkb: yup19:59
pleia2fungi: I don't think it has, but I don't keep as close an eye on it as I probably should :)19:59
jesusaurusclarkb: oh, is debconf the conf theyre gonna have at psu?20:00
fungijesusaurus: yes20:00
clarkblate august it will be nice then20:02
mroddendo we have any nice graphing/reporting tools that could help me answer the question "how long on average do the various chef cookbook jobs each take to run?"20:02
mroddeni could look at the jenkins reports but is there a way to get that aggregated over all jenkins masters?20:03
clarkbmrodden graphite.o.o20:04
mroddenwow cool...20:04
*** nati_ueno has joined #openstack-infra20:06
*** gondoi is now known as zz_gondoi20:06
fungijesusaurus: clarkb: how accessible is psu from downtown?20:06
jeblairclarkb, fungi: assuming the latest np tests pass, shall i ninja-merge it and the config change?20:07
jeblair(the latest np tests do pass)20:07
fungijeblair: i think so, given the wait time20:07
fungiand then we need a (probably not-so-graceful) nodepool restart and database update before we add a change to do the thing with the thing, yeah?20:07
tchaypoSo we have working setup tools again?20:07
jeblairfungi: yup20:08
clarkbjesusaurus it is downtown20:08
fungitchaypo: as of the 3.7.1 and 4.0.1 releases earlier today20:08
jeblairi'm guessing gerritbot is dead20:08
tchaypoThis is lovely news to wake up to20:08
fungigah, i can restart it20:08
*** amcrn has quit IRC20:08
clarkber fungi: it is downtown20:09
*** sarob_ has quit IRC20:09
jeblairthose changes have merged20:09
fungiclarkb: smith memorial student union specifically?20:09
*** mbacchi has joined #openstack-infra20:09
fungiheh, it does indeed say "downtown" (ignore me)20:09
*** ominakov has quit IRC20:10
jeblairi'm about to do an extremely ungraceful nodepool restart20:10
jeblairi will delete all of the currently used nodes20:10
clarkbfungi: yup its at broadway and hallish20:10
clarkbjeblair: k20:11
*** openstackgerrit has joined #openstack-infra20:11
*** zz_gondoi is now known as gondoi20:13
*** marcoemorais2 has quit IRC20:14
*** marcoemorais has joined #openstack-infra20:14
*** gyee has joined #openstack-infra20:15
jeblairmordred: there is a server called testmt in the openstackjenkins2 hpcloud account that i will delete20:16
fungipleia2: yeah, i don't see anything pressing on my schedule for late august, so maybe we should do that. i'll see if my employer will agree to cover my registration and travel20:16
pleia2fungi: yay, keep me posted :)20:17
fungipleia2: we should also talk to dds we know who have an interest in our infra tools and who might be attending20:17
pleia2fungi: if not, I could use your help in preparing something20:17
fungia few names come to mind20:17
pleia2great, I should probably get more friendly with them anyway20:17
*** pafuent has joined #openstack-infra20:18
*** primemin1sterp has joined #openstack-infra20:19
*** hogepodge has joined #openstack-infra20:20
*** sarob has joined #openstack-infra20:20
reedcan someone help this person here?
reedlooks like his git repo is very messed up20:20
pafuentHi. I'm from Blazar project and I want to add Tempest test to our gate jobs. What should I do? (Of course at Blazar we have some Tempests tests in a contrib directory)20:21
*** sarob is now known as sarob_20:22
*** sarob_ is now known as sarob20:22
openstackgerritKhai Do proposed a change to openstack-infra/config: puppetize installation of gerrit third party plugins
andreafhi - any infra core around for a tiny review on devstack-gate - it has already a +220:28
*** trinaths has quit IRC20:28
*** e0ne has joined #openstack-infra20:29
*** melwitt has joined #openstack-infra20:30
fungireed: it's the "Could not apply 7304ed8... Initial commit from fork***" which worries me there. i've suggested that they check and make sure they're not trying to submit changes from a clone of a fork of novaclient20:33
reedfungi, thanks20:33
*** julim has quit IRC20:34
*** mwagner_lap has quit IRC20:36
fungipafuent: it looks like you already have a devstack-based job for blazar at so perhaps you want to consider having it run tempest in that job as well? you may want to ask in #openstack-qa with regards to how to get tempest to run your contrib tests out of tree (from20:37
fungitempest's perspective). i'm not sure how pluggable it is yet20:37
*** Longgeek has quit IRC20:37
*** dims has quit IRC20:38
*** weshay has quit IRC20:39
*** miqui has quit IRC20:39
*** weshay has joined #openstack-infra20:39
*** sarob has quit IRC20:40
*** sarob has joined #openstack-infra20:40
fungiandreaf: looks fine to me, though if you plan to do anything with grenade, it's probably not deep enough of a change to support the additional overrides you'd need there20:40
fungispecifically where branch selection is concerned20:41
*** markmcclain has joined #openstack-infra20:41
fungiandreaf: oh, i do have one concern actually. your commit message doesn't really mention the git_branch variable you're adding at all, much less the use case for it20:42
jeblairfungi, clarkb: erm, i was overzealous and deleted the long-running slaves.  i've started rebuilding those20:42
fungijeblair: noted. they were long overdue for a refresh anyway. hopefully we didn't miss puppeting something critical on any of them20:43
pafuentfungi: OK, thanks. I'll check that with the Tempest group20:43
*** sarob has quit IRC20:45
*** andreykurilin has quit IRC20:45
andreaffungi: ok thanks - I will update the comment20:46
*** grantbow has quit IRC20:46
fungisdague: the reason behind https:// instead of git:// in devstack-gate (following up on your review comment in 95901) is that a lot of people seem to run this behind the corporate iron curtain where anything which isn't a web protocol must be a backdoor channel for criminal activity20:47
*** andreykurilin has joined #openstack-infra20:47
*** Sukhdev has joined #openstack-infra20:47
andreaffungi: about the use case: I want to be able to run gate in a setup where the "main" branch is not called "master" :)20:48
dstufftfungi: you mean git:// doesn't stand for good internet theieving?20:48
fungiandreaf: i was pretty certain i knew why you wanted it, just prodding you to also mention it in the commit message in case others are less able to guess correctly (also helpful when browsing the commit log)20:49
fungidstufft: it does now20:49
*** yamahata has quit IRC20:49
* fungi retcons it20:49
*** otherwiseguy has joined #openstack-infra20:49
*** yamahata has joined #openstack-infra20:50
fungijeblair: do you need help with any of the slave rebuilds?20:50
jeblairit takes a bit longer than i would have expected to delete 600 servers20:50
jeblairpuppet agent --environment production --server --no-daemonize --verbose --onetime --pluginsync true --certname mirror27.slave.openstack.org20:50
jeblairinfo: Retrieving plugin20:50
jeblairerr: Could not retrieve plugin: execution expired20:50
jeblairfungi: i'm afraid we may have a problem ^20:51
jeblairi don't know what that means, but it's happened twice20:51
jeblair3 times20:51
clarkbI think puppet is downlosding plugin from master and that times out?20:51
clarkbrelocating to hack space now20:51
jeblairwhich plugins and why, i wonder20:51
*** kaitlin-farr has quit IRC20:52
*** amcrn has joined #openstack-infra20:53
*** Ajaeger1 has quit IRC20:53
fungiyeah, that's a strange one... it's not every time then?20:53
fungiseeing if i can glean any details from the master end now20:53
jeblairthat's been followed up with:20:53
jeblairerr: Could not retrieve catalog from remote server: execution expired20:53
jeblairnotice: Using cached catalog20:53
jeblairerr: Could not retrieve catalog; skipping run20:53
TheJuliasounds like the puppetmaster is timing out20:54
fungiperhaps this is the "puppetmaster is broken" symptom we've seen a few times, where we end up restarting apache20:54
jeblairfungi: oh yeah20:54
jeblairfungi: i have 1 more run in progress, when it fails out, let's restart apache20:55
fungisounds like a good next step20:55
jeblairfungi: can you help me make sure all the jenkins masters have 0 slaves and restart them?20:56
fungijeblair: will do. logging into them now20:57
jeblairfungi: i'll take 1-4, you take 5-7 and jenkins.o.o?20:57
fungiworks for me20:57
clarkboh wait20:57
*** amcrn has quit IRC20:57
clarkbwe could upgrade the timeout plugin now if it isn't too much trouble20:58
*** amotoki has joined #openstack-infra20:58
jeblairclarkb: ok20:58
jeblairclarkb: where is it?20:58
clarkbthat will allow us to do the timeout variable export20:58
clarkbjeblair: from upstream20:58
*** sweston has joined #openstack-infra20:58
fungijeblair: also, keep in mind that some of the long-running slaves are in openstackci rather than openstack-jenkins, so no need to rebuild any which survived due to being there20:58
fungier, rather than in openstackjenkins20:58
sdaguejeblair / clarkb: preferences on this being openstack-infra or stackforge? -
jeblairsdague: later?20:59
sdaguealso fungi in that opinion question20:59
fungialso later ;)20:59
sdaguejeblair: works for me20:59
jeblairclarkb: mechanically, how do we make that happen?20:59
clarkbjeblair: I would just upgrade via the jenkins gui20:59
clarkbI find that easiest for upstream plugins20:59
jeblairclarkb: puppet doesn't do upgrades, right?20:59
clarkbbecause versions aren't a thing on disk just plugin presence21:00
*** jamespage_ has joined #openstack-infra21:00
*** ArxCruz has quit IRC21:00
jeblairrestarting apache on puppetmaster21:00
fungijeblair: on second look, no we have none of our long-running slaves in the openstackci account. no idea why i thought we still did21:01
*** yfried has quit IRC21:01
*** SumitNaiksatam has joined #openstack-infra21:01
*** mburned is now known as mburned_out21:01
*** sarob has joined #openstack-infra21:01
openstackgerritDoug Hellmann proposed a change to openstack-dev/hacking: Update examples for i18n
clarkbjeblair: fungi: have you done jenkins masters yet?21:02
clarkbI can help out now if not21:02
*** andreykurilin has quit IRC21:03
*** andreykurilin_ has joined #openstack-infra21:03
jeblairclarkb: yeah, i'm almost done with upgrading the plugin.  says avail version is 1.1321:03
clarkbthat sounds right21:03
clarkbzaro: ^21:03
clarkbI think 1.12 added the feature we want21:03
zaroyep, that's right21:03
jeblairpuppet is running on mirror27, will launch others now21:03
zarono, it was 1.13 i believe21:04
*** rfolco has quit IRC21:04
zaroactually it's whatever is specified in puppet21:04
*** dims has joined #openstack-infra21:05
zarojenkins.pp says 1.1321:05
jeblairfungi: keep in mind jenkins doesn't always really stop with the init script21:05
*** lttrl has quit IRC21:05
openstackgerritAndrea Frittoli  proposed a change to openstack-infra/devstack-gate: Allow to configure git base URL
clarkbya I usually do stop, ps, then start21:06
fungijeblair: yeah, i'm almost ready to start stopping them from the cli21:06
jeblairstop; ps; kill -9; ps; edit config.xml to remove slaves; start21:06
fungithey keep logging me out of the webui at odd moments21:06
*** e0ne has quit IRC21:07
fungiwere there any plugins i needed to upgrade on these before stopping?21:07
andreaffungi: so I updated the comment... but I was wondering now what the function of OVERRIDE_ZUUL_BRANCH is... it sets the branch for all projects in non-grenade case, but not for devstack-gate itself21:07
jeblairfungi: just the build-timeout plugin21:07
jeblairfungi: (also, you can leave the slave definitions on jenkins.o.o)21:08
*** dkliban is now known as dkliban_afk21:08
openstackgerritSean Dague proposed a change to openstack-infra/config: add gerrit-dash-creator to stackforge
*** Longgeek has joined #openstack-infra21:08
andreaffungi: so perhaps I could just use OVERRIDE_ZUUL_BRANCH and drop the GIT_BASE21:08
fungijeblair: i had already deleted them, but i guess they won't be too hard to re-enroll21:09
*** pdmars has quit IRC21:09
*** dims has quit IRC21:09
dhellmannclarkb: I've run out of day. What needs to be done to retest those taskflow release jobs tomorrow?21:09
clarkbdhellmann: I (or another infra rooter need to add the jobs back to the gearman queue21:10
jeblairokay, jenkins01-04 done21:10
clarkbdhellmann: which should happen once jenkins + nodepool are happy again21:10
clarkbdhellmann: I don't think you ened to do anything and I will try to do it today if things settle down21:10
fungijeblair: clarkb jenkins.o.o saw 1.14 of the build timeout plugin available, but jenkins 05-07 only seemed to see 1.13. should i do anything about that? refresh them?21:10
dhellmannclarkb: oh, cool, I really appreciate that21:10
jeblairfungi: erm, maybe best to manually install 1.13 on jenkins master21:11
clarkbjeblair: I agree21:11
clarkblets do 1.13 as the puppet change sets that value and we have installed it on a bunch of nodes already21:11
fungii take it 1.14 is newer than we want?21:11
fungiokay, got it21:11
fungii'll swap it out while the jenkins service isn't running21:11
*** atiwari has quit IRC21:11
*** Longgeek has quit IRC21:12
*** arborism has joined #openstack-infra21:13
*** ominakov has joined #openstack-infra21:13
*** pafuent has quit IRC21:14
clarkboh gah21:14
clarkbreading the release notes for build timeout plugin we may need 1.1421:14
clarkbthey put a global lock in there21:14
clarkbwhy do jenkins plugins always do this?21:14
fungiis it safe to just overwrite the build-timeout.jpi with the desired one?21:14
jeblairfungi: when it's stopped, yes21:14
jeblairfungi: it will unpack on next start21:14
jeblairclarkb: did 1.14 remove the global lock?21:14
clarkbthat claims to have fixed the sync point21:15
*** arnaud has joined #openstack-infra21:15
*** mfer has quit IRC21:15
clarkb for release notes21:15
fungiokay, well jenkins.o.o:/var/lib/jenkins/plugins/build-timeout.jpi should be 1.14 then since i haven't overwritten it yet, if we want to copy that one to the others21:15
clarkbhrm actually it may be in the log watching which we don't use so possibly we are ok21:16
jeblairclarkb: what version do you recommend we use?21:16
clarkbI think we are ok with 1.13 if I udnerstand the bug properly21:17
fungiokay, then i'll go ahead and put 1.13 on jenkins.o.o21:17
openstackgerritKiall Mac Innes proposed a change to openstack-infra/config: Add Designate DevStack/Requirements/Docs Jobs
clarkbbut I don't know how you enable captured log outputs21:18
clarkbzaro: ^ do you udnerstand that?21:18
clarkbwe don't seem to explicitly enable that feature but I can't figure out where you might do that21:18
jeblairclarkb: tbh, if you and zaro are not fairly certain that this will not cause problems for us, i'd much rather downgrade back to 1.10 and wait for some testing.21:20
jeblairclarkb: i don't think we need the system to blow up again.21:20
clarkbjeblair: that wfm21:20
jeblairfungi: ^?21:20
openstackgerritKiall Mac Innes proposed a change to openstack-infra/config: Add Designate DevStack/Requirements/Docs Jobs
zarosorry, i don't know about that watch log feature.21:21
*** esker has quit IRC21:21
fungisure. what's the easiest way for me to get the 1.10 jpi file?21:21
jeblairfungi: 1.10 is still installed on jenkins-dev.  i think we can get it from there.21:21
fungigot a url?21:21
fungiokay, getting21:21
*** esker has joined #openstack-infra21:21
clarkbzaro: after all of this settles down we should put 1.14 on jenkins-dev and test it21:23
clarkbzaro: iirc you tested the plugin but not on -dev21:23
jeblairor, like a private jenkins21:23
jeblairi think we need to test for this behavior21:24
*** johnsom has joined #openstack-infra21:24
clarkbjeblair: yeah zaro's previous testing was on a private jenkins. could do that again21:24
*** Guest70585 has joined #openstack-infra21:25
zarono, i did test on dev.21:25
*** jcoufal has quit IRC21:25
zaroi mean jenkins-dev21:25
*** AJaeger has quit IRC21:25
clarkbzaro: did you downgrade after testing? the currently instaleld version is 1.1021:26
zarobut i agree, we should test again with 1.1421:26
zaroyes, i did21:26
fungiokay, jenkins.o.o and 05-07 are all stopped, configs contain an empty slaves list, and they are all re-downgraded to build-timeout 1.10. anything else before i start them again?21:26
jeblairfungi: i think that's it21:26
*** esker has quit IRC21:26
clarkbfungi: sounds good to me21:26
*** doude has quit IRC21:26
johnsomI am getting "Server Error" and "Cannot store contact information" when I try to set my contact details in gerrit.  Known issue?21:26
fungistarting them21:26
anteayajohnsom: have you joined the foundation yet?21:27
jeblair01-04 are running 1.1021:27
zarojohnsom: i don't think so.  might be a trove disconnect?21:27
fungijohnsom: try to make sure you've followed the recommendation at
johnsomYes, I have joined.21:28
fungijohnsom: and your primary e-mail address in your foundation profile matches your preferred e-mail address in your gerrit account settings/contact information?21:29
clarkblooks like the build log capturing was something added in 1.13 too which is why I don't see it today. I will keep digging at this so that we can figure out how t otest it21:29
*** yfried has joined #openstack-infra21:29
zaroclarkb, fungi, jeblair : i'll test the ver 1.14 on jenkins-dev now.21:30
jeblairzaro: it has no slaves at the moment21:30
clarkbzaro: ya probably best to let things settle first21:30
jeblairzaro: you can probably test with local executors on the master i guess21:30
clarkbzaro: tere is an example test case in the jenkins bug though. Should be simple to reproduce using that21:30
*** lcostantino has quit IRC21:30
jeblairzaro: you'll just need to make several since we don't normally run in that configuration.21:31
jeblairzaro: though honestly, it might be easier to test locally first21:31
fungiokay, jenkins seems to be up and operable again on jenkins.o.o and 05-07. all report to currently be using build-timeout 1.1021:31
jeblairfungi, clarkb: i think that means we're read to restart nodepool21:31
jeblairi have dropped the node and subnode tables, installed the new version and its config file21:32
fungiand then i have an appointment to which i need to skedaddle21:32
*** arborism is now known as amcrn21:32
jeblair2014-06-02 21:32:25,313 DEBUG nodepool.NodePool:     <AllocationRequest for 1281.0 of bare-precise>21:33
jeblair2014-06-02 21:32:25,314 DEBUG nodepool.NodePool:     <AllocationRequest for 1084.0 of devstack-precise>21:33
fungithat is plenty lots21:33
funginodepool list reports 936 nodes in a "building" state now21:33
jeblair2014-06-02 21:32:25,317 DEBUG nodepool.NodePool:     <AllocationGrant of 45 of bare-precise from hpcloud-b1>21:33
zaroclarkb, fungi, jeblair : actually looks like i cannot test.  i don't have ssh access to jenkins-dev.o.o only the jenkins management ui.  the management ui says latest version of build-timeout is 1.1321:33
jeblair2014-06-02 21:32:25,318 DEBUG nodepool.NodePool:     <AllocationGrant of 38 of devstack-precise from hpcloud-b1>21:33
clarkbzaro: you need to refreash the management ui's plugin list21:34
zaroohh yeah, that's right.21:34
jeblairjobs are running21:34
clarkboh nodepool list doesn't show az. is that something I should patch? or meh?21:35
jeblairclarkb: i think so21:35
clarkbI will work on that now as it may help with future debugging21:35
jeblairmight also want to log and statsd it21:35
*** smarcet has quit IRC21:35
jeblair(those could probably be separate changes)21:36
clarkbjeblair: ok21:36
clarkbdoes nodepool list show subnodes?21:36
jeblairclarkb: i think so21:36
jeblairi have nothing to back that up with atm though :)21:37
clarkblooking at the list() function I am not so sure (mostly curious because subnode azs are set by the parent node21:37
clarkbjeblair: it calls session.getNodes()21:37
jeblairclarkb: if it doesn't handle subnodes, don't worry about that for now21:38
openstackgerritYuriy Taraday proposed a change to openstack-infra/git-review: Add an option to mark uploaded patchsets as WIP
*** e0ne has joined #openstack-infra21:40
jeblairoh hehe, we got a node counter reset because i dropped the table21:40
fungiokay, i'm heading out for a bit, but back relatively soon (probably not much more than an hour)21:40
johnsomThanks guys, got it to work.  I thought I had setup my foundation account, but I got it confused with another account setup.21:40
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Display node AZ in `nodepool list` output
clarkbsometing like that21:41
jeblair"4 million nodes since a lost-time schema change"21:41
jeblairfungi: have fun!21:41
openstackgerritA change was merged to openstack-infra/nodepool: Display node AZ in `nodepool list` output
*** aysyd has quit IRC21:42
jeblairclarkb: output looks good!21:43
*** ominakov has quit IRC21:43
*** johnsom has left #openstack-infra21:43
zaroclarkb: i tried upgrading build-timeout plugin on jenkins-dev, now it won't start.  can you help there?21:45
*** signed8bit has quit IRC21:45
Alex_GaynorDid something just make all the CI jobs start?21:45
jeblairAlex_Gaynor: me?21:46
Alex_Gaynorjeblair: Thanks :-)21:46
jeblairAlex_Gaynor: clarkb and fungi helped too, but I hit "Enter".21:46
Alex_Gaynorhitting enter is important21:46
jeblairAlex_Gaynor: we made a change to nodepool to accomodate hpcloud1.1 which required basically a full restart of everything (except zuul, which kept its queue)21:47
*** xyang1 has quit IRC21:47
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add suppressed fingerprint for pep8 failures in gate
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Support suppressing bugs in ES.classify
jeblairhopefully that change means we'll actually end up with enough nodes to keep up with tests21:47
*** miqui has joined #openstack-infra21:48
*** e0ne has quit IRC21:48
jeblairwe now have as many nodes in use as before the restart21:49
jeblairwith 600 still building21:49
*** gondoi is now known as zz_gondoi21:49
*** UtahDave has quit IRC21:49
Alex_Gaynoroh god we are using so much cloud21:49
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Record provider AZ info in graphite.
*** _nadya_ has quit IRC21:50
JayFyou can't build a cloud without breaking a few clouds21:50
clarkbjeblair: ^ note that doesn't do the aggregate anymore. I figure we can sum on the graphite side btu let me know if you want me to keep the old keys too21:50
jeblairJayF: haha21:50
clarkbI can update the chnage21:51
*** mriedem has quit IRC21:51
openstackgerritDan Bode proposed a change to openstack-infra/config: Add puppet-openstack project puppet-openstacklib
*** sileht has quit IRC21:52
zaroi believe the bug in build-timeout about capturing log is actually "Writing the build description when timeout occurred."  none of our jobs use this setting.21:52
*** adalbas has quit IRC21:53
*** pcrews has quit IRC21:53
clarkbzaro: right because the setting doesn't exist until 1.1321:54
zarocan someone please kick start jenkins-dev?21:54
*** anteaya has quit IRC21:54
*** Guest70585 is now known as anteaya21:54
*** david-lyle has quit IRC21:55
*** sileht has joined #openstack-infra21:59
*** bauzas has quit IRC21:59
mordredjeblair: re: testmt - I'm sure you already did, but deleting is fine22:00
*** rlandy__ has quit IRC22:01
mordredclarkb: the current graphs I've got should still work until your nodepool change above lands, yeah?22:01
jeblairmordred: heh, i deleted that and more.  :)22:01
clarkbmordred: I think they will need a new glob22:02
mordredjeblair: awesome22:02
clarkbwhich si why I ask if people are ok withthat22:02
*** andreaf has quit IRC22:02
mordredclarkb: a new glob once your thing lands?22:02
clarkbmordred: yeah to glob azs22:02
*** amotoki has quit IRC22:03
mordredclarkb: I don't personally need it - but also I'm fine with updating graphs22:03
*** dims has joined #openstack-infra22:03
jeblairokay, mirror26,27,33 and proposal exist again and are attached to jenkins.o.o22:03
jeblairclarkb, fungi: ^22:03
jeblairi did not recreate the dev slaves or salt-trigger22:04
*** sileht has quit IRC22:04
anteayapleia2: changed my vm just in time, I can't log into the old system anymore22:04
anteayawas about to delete my vms, but *shrug* they can do it22:04
pleia2anteaya: phew :)22:05
anteayalike 40 minutes difference22:05
pleia2anteaya: so the networking is all set up for good now, all you ever need to do to start new vms is just go to the compute instances page (or use nova commands if you feel like learning those :))22:05
anteayaI'll learn horizon22:05
*** ihrachyshka has quit IRC22:05
pleia2I think the hpcloud cli still exists now, but it was always kind of meh22:06
*** sileht has joined #openstack-infra22:06
anteayathen I can have stuff to talk to david-lyle about22:06
* pleia2 nods22:06
*** markmcclain has quit IRC22:06
jeblairclarkb: if i'm reading correctly, everything in hpcloud has been building for 30 minutes22:07
clarkbjeblair: yeah I think its bonkers22:07
clarkbmordred: ^22:07
*** andreaf has joined #openstack-infra22:07
mordredjeblair: that seems like an exra long time22:07
*** chuckC has quit IRC22:08
clarkbnova availability-zone-list says i have the correct names22:08
*** dims has quit IRC22:08
jeblairtteggel: ^ so we just started spreading our load across az1-az3.  we asked for 500 servers and they have all been in the building state for 35 minutes22:08
*** Longgeek has joined #openstack-infra22:09
*** masayukig has quit IRC22:09
*** fbo is now known as fbo_away22:10
jeblairoh, good news, 179 are now in delete!22:10
anteayawell that's a different state22:10
clarkbso hypothesis time. maybe images and networks are not actually cross AZ?22:11
jeblairClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-bafd0526-1bce-411a-b23b-d41ffccd1bc4)22:11
clarkbI am going to do testing with my personal account22:11
*** sanjiva has quit IRC22:12
*** andreaf has quit IRC22:12
Kiallclarkb: images and networks are cross AZ22:12
*** jamielennox is now known as jamielennox|away22:12
clarkbKiall: that is what we are told :)22:12
Kiall(I assume you're talking about HP Cloud 1.1 / 13.5...)22:12
*** dims has joined #openstack-infra22:12
clarkbKiall: yes22:12
jeblairso we have 100 that built, 200 that failed, and 200 that are currently building.22:13
clarkbjeblair: and it took 30 minuts for the 100 that built?22:13
* clarkb looks at nodepool22:13
*** Longgeek has quit IRC22:13
Kialljeblair: have an instance ID of one that's building handy?22:13
jeblair| 937  | hpcloud-b5             | az3  | py3k-precise           | jenkins02 |                 | py3k-precise-hpcloud-b5-937                 | fdaa2e3e-d66e-4cbb-b826-644970d93b7b | None            | building | 0.69        |22:14
jeblairKiall: fdaa2e3e-d66e-4cbb-b826-644970d93b7b22:14
*** bauzas has joined #openstack-infra22:14
clarkbjeblair: looks like they built in all AZs so not a cross AZ issue22:14
*** nati_ueno has quit IRC22:14
*** masayukig has joined #openstack-infra22:14
*** nati_ueno has joined #openstack-infra22:14
clarkbmaybe the next thing to check is that they actually built in different AZs22:14
Kialljeblair: bad choice I think, that one is ACTIVE22:15
jeblairclarkb: oh!22:15
Kialland it's in az322:15
*** e0ne has joined #openstack-infra22:15
*** hashar has quit IRC22:16
clarkbya  Isee them in different AZs22:16
clarkbso nodepool change worked22:16
clarkbI am using `nova show uuid` to see that22:16
*** praneshp has quit IRC22:17
jeblairKiall: if that's the case, probably nodepool is just catching up to the status changes on the ones that are building22:17
*** _nadya_ has joined #openstack-infra22:17
*** otherwiseguy has quit IRC22:17
jeblairKiall: is a failed instance of interest?22:17
Kialljeblair: Possibly - Depends on the failure, I might be able see more detail than `nova show`22:18
*** e0ne has quit IRC22:18
Kiall(But I don't have root access on the compute nodes either, so, somewhat limited ;))22:18
jeblair2014-06-02 22:09:38,770 ERROR nodepool.NodeLauncher: ClientException launching node id: 935 in provider: hpcloud-b5 error:22:19
jeblair    return self._create("/os-floating-ips", {'pool': pool}, "floating_ip")22:19
jeblairClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-2eceb7cc-96dc-467f-98ee-32208252872d)22:19
jeblairinstance id: 41963302-7eda-4519-84b1-094b1429b93322:19
jeblairso node #935 got a 500 when creating a floating ip22:20
KiallAlso ACTIVE, az2, but the log looks like a floating IP issue22:20
mordredjeblair: when we restarted things, did we maybe orphan any flaoting ips?22:20
*** bknudson has quit IRC22:20
* Alex_Gaynor idly wonders which does more to ensure OpenStack quality, the tests the CI system runs, or the CI system itself22:20
jeblairmordred: tbh, i don't know how to tell.22:21
Kialljeblair: could nodepool be masking FIP create/attach failures as instance build failures?22:21
mordredjeblair: I show a bunch of floating ips not attached to servers22:21
jeblairKiall: it's all a failure to us22:21
*** esker has joined #openstack-infra22:22
jeblairmordred: is that because they just haven't been attached yet, or are they really orphaned?22:22
clarkbjeblair: how did you delete the nodes?22:22
clarkbjeblair: if you just did nova delete that would orphan the IPs iirc22:22
jeblairclarkb: i did that for some of them22:22
clarkbnodepools has to special delete floating IPs22:23
mordredjeblair: there are a ton missing a fixed ip, and a ton with one - I'd say half and half22:23
mordredI think I should delete the floating ips that don't have fixed ips22:23
jeblairmordred: okay, thanks22:23
mordredand if that race-conditions a node creation, oops22:23
clarkbmordred: are we anywhere near quota though?22:23
clarkbwe should have ~600 IPs22:23
clarkbif quota is any good22:23
mordredyes. we are at quota22:23
*** andreykurilin_ has quit IRC22:24
openstackgerritMorgan Fainberg proposed a change to openstack-infra/gitdm: Update company affiliation for morganfainberg
mordred321 have no server22:24
KiallI wonder if being at quota is whats causing the 500? OR, if the 500 returned by the FIP create is leaving an orphan behind...22:24
mordredfor ip in $(nova floating-ip-list | grep '| -' | awk '{print $2}') ; do echo "Deleting $ip" ; nova floating-ip-delete $ip ; done22:25
*** BadCub_ has quit IRC22:26
*** adalbas has joined #openstack-infra22:26
*** NobodyCam has quit IRC22:26
*** zaro has quit IRC22:28
clarkbmordred: is that still running?22:28
mordredclarkb: oh yeah. it's going to be running for a while22:28
*** jgrimm has quit IRC22:28
*** mwagner_lap has joined #openstack-infra22:28
mordredI think it's done roughly 3022:28
mordredso estimate 30 minutes to complete22:29
Kiallmordred: add --os-cache22:30
KiallIt's keep and reuse the same Keystone token, rather than doing an API call to keystone for each22:30
KiallIt'll keep and reuse*22:30
clarkbwow you would think that would be the default22:30
openstackgerritA change was merged to openstack-infra/config: Don't gate pecan on barbican for now.
Kiallclarkb: I know...22:31
KiallOh, and install python-keyring, it needs it22:32
mordredoh. right. that's why it's not default22:33
clarkboh that thing22:33
KiallIt silently continues without caching if that's not installed22:33
clarkbya ok we can leave it off :)22:33
KiallSo.. I don't count that as a reason to be off22:33
Kiallanyway.. Install, add flag, half the time required to do the deletes ;)22:33
mordredmordred@camelot:~$ sudo pip install python-keyring22:33
mordredDownloading/unpacking python-keyring22:33
mordred  Could not find any downloads that satisfy the requirement python-keyring22:33
clarkbKiall: you might have missed it but it was on by default at one point (the keyring) and it brkoe everything22:33
clarkbkeyring is terrible22:34
Kiallmordred: ehh.. that's the .deb name I was thinking of22:34
clarkbit makes bad env assumptions iirc22:34
Kiallmight just be keyring on pypi22:34
clarkblike no controlling tty? better prompt for a passphrase anyways22:34
Kiallclarkb: oh.. Nice22:34
russellbyeah not so great for automation ... would be nice to have a better solution for that22:35
russellbwriting the scripts to the Python API directly is one solution22:35
russellbthat's certainly not the only reason the CLI commands aren't well suited to automation22:35
Kiallrussellb: yea, the python APIs for OpenStack (bar neutron, sorry neutron folks) are pretty good+intuitive.22:36
KiallSpeaking of which - mordred - You'd be better off going straight to neutron with those FIP deletes ;)22:36
clarkbKiall: you haven't used the keystone api :)22:36
mordredKiall: oh. you're right22:36
asselin_Hi nodepool experts. I'm trying to make sense of the various credentials since I'm getting this error on image update: INFO:paramiko.transport:Authentication (publickey) failed.22:36
clarkbasselin_: that is an ssh publickey auth error22:37
asselin_my nodepool.yaml proviers has     keypair: 'jenkins'22:37
asselin_ which is being used. and images:         private-key: '/home/stack/.ssh/jenkins_key'22:37
clarkbasselin_: which may happen when nodepool ssh's into a node to make sure it is ready to go22:37
clarkbasselin_: your image creation scripts need to install that public key22:37
clarkbor you need to configure cloud init or something to put it in place foryou22:38
asselin_clarkb, I can manually ssh into it...just confused on how to configure nodepool to do so.22:38
clarkbasselin_: using that key?22:38
clarkbis there a passpharse on the private key?22:38
asselin_clarkb, I manually updated the public key and set the name to keypair: 'jenkins'22:38
clarkbmordred: down to ~450ish22:39
asselin_I'm going through the code and it uses the keypair set to create the instance...but nothing at all to ssh into it...??22:39
*** salv-orlando has quit IRC22:39
asselin_clarkb, both lines 7 and 10 are 'None'. I tried adding line 8...but wanted to ask if I'm missing something before going down that route22:41
*** chuckC has joined #openstack-infra22:42
clarkbasselin_: it creates a key to ssh in as root iirc22:43
*** jamielennox|away is now known as jamielennox22:43
clarkbbut it only does that when creating the image22:43
*** sweston has quit IRC22:43
clarkbthen the image creation process is expected to put in place the key needed by nodepool to ssh into after a regular node boots22:43
asselin_clarkb, ok, yes I'm starting to remember from when I did this creates a throw-away key for image update...and uses the export NODEPOOL_SSH_KEY=`cat ~/.ssh/`22:45
asselin_ variable....22:45
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added parallelization options
asselin_clarkb, is that what infra is doing?22:45
mordred404-{u'QuantumError': u'Floating IP b684688c-8893-42f0-9473-df0b4f24af8e could not be found'}22:47
jeblairwhat's a quantum?22:47
mordredthat's so sexy. I love that the error message is still quantum22:47
*** camunoz has quit IRC22:47
*** Sukhdev has quit IRC22:47
*** zaro has joined #openstack-infra22:47
clarkbasselin_: we have the nodepool user setup with the correct key and paramiko uses that by default22:47
clarkbI believe that is how we do it22:47
clarkbmordred: well you know backward compat22:47
clarkbdown to ~250 now22:48
asselin_clarkb, thanks I'll investigate that a bit22:48
mordredclarkb: we are penny-wise and pound-short with our backward compat attempts22:48
*** hemna is now known as hemnafk22:49
anteayathat is the error when you can't associate a floating ip because teh router doesn't ahve an interface22:49
anteayasaw that several times this afternoon22:49
*** esker has quit IRC22:49
mordredanteaya: yah. I was mostly just amused because it wasn't called NeutronError22:50
anteayaI too found that odd22:50
mordredok. the script is done22:50
mordredfor the record, I ran:22:51
mordred for ip in $(neutron floatingip-list | grep -v '10.0' | awk '{print $2}') ; do echo "Deleting $ip" ; neutron floatingip-delete $ip ; done22:51
*** BadCub has joined #openstack-infra22:51
*** NobodyCam has joined #openstack-infra22:52
clarkbthere are still a ton of floating ips with no fixed ip22:52
mordredclarkb: _those_ may be legit in-progress builds22:52
*** denis_makogon has quit IRC22:52
*** eharney has quit IRC22:53
clarkbwow nova commant is floating-ip-list, neutron is floatingip-list22:53
clarkbwe should have an openstack uses openstack session at the next summit22:54
Kiallclarkb: and it's `neutron security-group-list`, for consistency ;)22:54
anteayamorning jhesketh22:58
anteayathanks for finding Longeek's https proxy difficulty22:58
anteayathat marked the end of a day and a half of trying to shoe horn in a min.js patch that I never could understand the motivation for22:59
*** nati_uen_ has joined #openstack-infra22:59
*** otherwiseguy has joined #openstack-infra22:59
clarkbmordred: that seems possible. the number of non associated floating IPs is slowly falling but the total number of floating IPs is rising22:59
*** camunoz has joined #openstack-infra22:59
*** nati_uen_ has quit IRC22:59
*** nati_uen_ has joined #openstack-infra23:00
*** e0ne has joined #openstack-infra23:00
anteayamorning mattoliverau23:00
clarkbfungi: jeblair may be the next step in nodepool fun23:01
clarkbfor completeness23:01
jeblairclarkb: lgtm23:02
*** nati_ueno has quit IRC23:03
*** harlowja_ has quit IRC23:04
asselin_clarkb, ok I got it working by taking out keypair: 'jenkins'23:04
asselin_ from the nodepool.yaml file. This puts the code in the path to use the throw-away key for image creation.23:04
*** sarob has quit IRC23:04
*** harlowja has joined #openstack-infra23:04
*** weshay has quit IRC23:04
stevebakerdstufft: hey, do you have any idea when pip-1.5.7 will be released? I'm waiting for a fix to allow building custom heat images23:04
*** sarob has joined #openstack-infra23:04
clarkbasselin_: good to know and glad you figured it out23:05
*** yamahata has quit IRC23:05
*** e0ne has quit IRC23:05
jheskethjeblair: ping?23:05
*** adalbas has quit IRC23:05
asselin_clarkb, I'm curious to know why use the throwaway key? security related?23:05
jeblairjhesketh: pong23:06
jheskethjeblair: PM okay?23:06
clarkbasselin_: I think for ease of implementation23:06
jeblairjhesketh: yep23:06
clarkbasselin_: the image build is a one shot thing so persistent keys don't help much23:06
*** zzelle_ has quit IRC23:06
asselin_clarkb, they help when debugging image creation :)23:07
clarkbasselin_: nodepool deletes the node when it is done23:08
clarkbso you would have to change that behavior too. easier to just run the scripts on a node by hand23:08
*** sarob has quit IRC23:09
*** Longgeek has joined #openstack-infra23:10
*** mmaglana has quit IRC23:10
asselin_clarkb, ok I see23:10
*** mburned_out is now known as mburned23:10
clarkbok nodepool looks much happier according to mordred's graphs23:11
clarkbwe are still seeing some really long build times though23:11
clarkbI need to do a grocery run back soon23:11
asselin_anyone else like to review/approve these 2 patches:    ?23:13
openstackgerritNachi Ueno proposed a change to openstack-infra/reviewstats: Add Review Stat WebUI
*** Longgeek has quit IRC23:15
*** pcrews has joined #openstack-infra23:17
*** wenlock has quit IRC23:19
*** doug-fish has left #openstack-infra23:22
*** nati_uen_ has quit IRC23:22
*** nati_ueno has joined #openstack-infra23:23
*** andreaf has joined #openstack-infra23:23
*** otherwiseguy has quit IRC23:24
*** terryw has joined #openstack-infra23:24
*** nati_uen_ has joined #openstack-infra23:24
*** camunoz has quit IRC23:26
*** nati_ueno has quit IRC23:27
*** terryw has quit IRC23:28
*** otherwiseguy has joined #openstack-infra23:28
*** andreaf has quit IRC23:29
*** terryw has joined #openstack-infra23:31
*** sarob has joined #openstack-infra23:31
*** otherwiseguy has quit IRC23:33
mordredjeblair: ^^ perhaps we should figure out what ACL we need to set so that the REST call to get the list of cores can be done un-authed?23:37
mordredjeblair: it seems like there are a couple of programs out there keeping lists of things or inferring it, and now taht we've got the REST API they could just, you know, query gerrit23:37
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Ignore non-voting jobs in gerrit
jeblairmordred: group listing requires auth? huh, i haven't tried it.23:37
mordredjeblair: I tried it briefly with curl the other day23:38
*** yjiang5 has joined #openstack-infra23:40
*** timrc is now known as timrc-afk23:40
*** camunoz has joined #openstack-infra23:43
*** timrc-afk is now known as timrc23:48
*** nati_uen_ has quit IRC23:48
clarkbya it does23:49
Alex_GaynorAre all the py3k-precise nodes in one DC/AZ?23:49
*** unicell has joined #openstack-infra23:49
clarkbAlex_Gaynor: no they are in rax and hpcloud23:49
*** ArxCruz has joined #openstack-infra23:49
*** terryw has quit IRC23:50
Alex_GaynorOk, it must be in my head then (for some reason they seem particularly absent)23:50
jeblairclarkb: 518 nodes in use now; much more than earlier23:50
clarkbasselin_: I thought I -1'd 9653923:50
clarkbasselin_: because the override shoudl just override the entire url prefix not just the protol23:50
clarkbapparnetly that comment never made it to gerrit23:51
*** arnaud has quit IRC23:51
clarkbjeblair: woot23:51
jeblairclarkb: gertty?23:51
clarkbjeblair: ya it must've been gertty I probably closed it wihtout leaving it long enough to sync23:51
jeblairclarkb: it should sync on the next start23:52
*** nati_ueno has joined #openstack-infra23:52
jeblairclarkb: (the first sync task is to push up review comments)23:52
clarkbhrm not sure where that comment went then23:52
clarkbin any case -1 is there now23:52
clarkbasselin_: ^23:52
*** otherwiseguy has joined #openstack-infra23:52
jeblair(at least, in my branch tip, it's like #3 in master)23:52
asselin_clarkb, ok will look at that23:53
jeblairclarkb: if you find that you have a draft comment for that change in your gertty, i'm interested in debugging23:53
*** nati_uen_ has joined #openstack-infra23:54
clarkbjeblair: actually I wonder if it was done in the old gertty whose db I deleted for your new changes23:54
asselin_clarkb, got it: GIT_BASE=${GIT_BASE:-git://}23:54
clarkbjeblair: I should've backed that DB up instead of deleting it :(23:54
clarkbasselin_: yeah that way you can override the protocol and location if necessary23:54
*** sarob has quit IRC23:54
jeblairclarkb: doh23:54
*** jaypipes has quit IRC23:55
*** sarob has joined #openstack-infra23:55
clarkbasselin_: its a bit more flexible that way23:55
asselin_clarkb, agreed23:55
clarkbdid mordred's unbound change merge which allows 96551 to work?23:55
*** W00dy has joined #openstack-infra23:56
*** W00dy is now known as Guest5470023:56
mordredthe one thing that gertty does not help me with is tracking the state of my in flight patches23:56
clarkbmordred: yeah its very reviewing all the things oriented23:56
mordredand it's AMAZING23:56
mordredI've been hovering at inbox zero on many repos all day23:56
*** nati_ueno has quit IRC23:57
clarkbjeblair: I am giving the tip of your proposed changes another shot23:58
clarkbsync is happening now23:58
*** matsuhashi has joined #openstack-infra23:59
*** sarob has quit IRC23:59
clarkbwoot hyperlinks open23:59

Generated by 2.14.0 by Marius Gedminas - find it at!