Monday, 2014-04-28

*** e0ne has quit IRC00:00
*** etoews has joined #openstack-infra00:07
*** adalbas has quit IRC00:07
*** dkehn is now known as dkehn_afk00:08
*** morganfainberg_Z is now known as morganfainberg00:09
*** etoews has quit IRC00:12
*** rfolco has joined #openstack-infra00:13
*** rfolco has quit IRC00:14
*** rfolco has joined #openstack-infra00:15
*** matsuhashi has joined #openstack-infra00:20
*** etoews has joined #openstack-infra00:24
*** fifieldt has joined #openstack-infra00:24
*** jhesketh has quit IRC00:33
*** talluri has joined #openstack-infra00:41
*** morganfainberg is now known as morganfainberg_Z00:41
*** aconrad has joined #openstack-infra00:42
*** talluri has quit IRC00:45
*** mattoliverau has quit IRC00:50
*** yamahata has joined #openstack-infra00:52
*** mattoliverau has joined #openstack-infra00:53
*** etoews has quit IRC00:57
*** e0ne has joined #openstack-infra00:58
*** unicell has joined #openstack-infra00:59
*** e0ne has quit IRC01:03
stevebakerHey, I wonder if could be fast-tracked into tempest to see if it helps with
uvirtbotLaunchpad bug 1297560 in tempest "*tempest-dsvm-neutron-heat-slow fails with WaitConditionTimeout" [Undecided,New]01:16
openstackgerritBoris Pavlovic proposed a change to openstack-infra/config: Bump check-rally-dsvm-{name} job timeout
*** yamahata has quit IRC01:20
*** yamahata has joined #openstack-infra01:21
*** gondoi is now known as zz_gondoi01:23
*** zz_gondoi is now known as gondoi01:24
*** etoews has joined #openstack-infra01:26
*** saschpe has quit IRC01:32
*** saschpe has joined #openstack-infra01:32
*** thomasem_ has joined #openstack-infra01:33
*** thomasem has quit IRC01:35
*** dkranz has quit IRC01:39
*** yaguang has joined #openstack-infra01:44
msabramoWith pbr, is it possible to express that a package needs unittest2 for Python ≤ 2.6 but nor for Python > 2.6?01:47
msabramonormally I would do that with python code in but with setup.cfg, it's declarative so that doesn't seem possible01:47
*** nosnos has joined #openstack-infra01:50
*** rfolco has left #openstack-infra01:55
*** dims has quit IRC01:56
*** rfolco_ has joined #openstack-infra01:57
*** e0ne has joined #openstack-infra01:58
*** e0ne has quit IRC02:00
mattoliveraumsabramo: Not that I know of, but you force it to check the version of python, i.e  requires-python >=2.602:02
mattoliverauBut I don't know how'd you do the split otherwise.02:03
*** alexpilotti has quit IRC02:08
*** dims has joined #openstack-infra02:09
mattoliveraumsabramo: On closer look, you might be able to do: Requires-Dist: unittest2; python_version == '2.4' or python_version == '2.5'02:10
mattoliveraumsabramo: based off PEP 345 (
*** msabramo has quit IRC02:12
*** etoews has quit IRC02:23
*** e0ne has joined #openstack-infra02:25
*** e0ne has quit IRC02:26
*** thomasem_ has quit IRC02:26
*** msabramo has joined #openstack-infra02:27
msabramothanks, mattoliverau - I'll take a look at that PEP!02:38
msabramoit doesn't look like pbr supports that from the source code02:40
*** gokrokve has quit IRC02:41
msabramobut it looks like it might support having requirements-py2.txt and requirements-py3.txt02:41
*** zhiyan_ is now known as zhiyan02:47
*** msabramo has quit IRC02:56
*** e0ne has joined #openstack-infra02:58
*** e0ne has quit IRC03:02
*** msabramo has joined #openstack-infra03:09
*** asettle is now known as alex-lunch03:11
*** gokrokve has joined #openstack-infra03:11
*** gokrokve_ has joined #openstack-infra03:14
*** gokrokve has quit IRC03:16
*** dims has quit IRC03:17
*** dims has joined #openstack-infra03:18
anteayamattoliverau: hey there03:18
anteayamattoliverau: if anyone asks we are upgrading gerrit at 1600 utc:
*** gokrokve_ has quit IRC03:19
anteayaand welcome back from your vacation03:20
anteayaI am about to hit the hay03:20
mattoliverauanteaya: hey! thanks, OK will let people know :)03:23
StevenKOh, that's today?03:39
StevenKWell, tomorrow at 2am for me, so eh. :-)03:40
*** gondoi is now known as zz_gondoi03:41
*** afazekas has quit IRC03:47
*** matsuhashi has quit IRC03:54
*** e0ne has joined #openstack-infra03:58
*** e0ne has quit IRC04:02
*** etoews has joined #openstack-infra04:05
*** alex-lunch has quit IRC04:07
*** asettle has joined #openstack-infra04:08
*** etoews has quit IRC04:09
*** nosnos has quit IRC04:10
*** asettle has quit IRC04:12
*** alexandra has joined #openstack-infra04:12
*** alexandra is now known as asettle04:13
*** gokrokve has joined #openstack-infra04:14
*** gokrokve has quit IRC04:19
*** sdake_ has joined #openstack-infra04:21
*** sdake_ has quit IRC04:21
*** sdake_ has joined #openstack-infra04:21
*** asalkeld has joined #openstack-infra04:24
asalkeldhi I added stackforge/solum-dashboard recently04:25
asalkeldit's here:
asalkeldbut not on github04:25
asalkeld <- 40404:25
asalkeldany thing I can check?04:26
mordredasalkeld: probably an error in the manage-projects script - I can take a look at it tomorrow04:26
asalkeldcool, thanks mordred04:26
*** msabramo has quit IRC04:32
*** msabramo has joined #openstack-infra04:32
*** asalkeld has left #openstack-infra04:40
*** starmer_ has joined #openstack-infra04:43
*** etoews has joined #openstack-infra04:43
*** etoews has quit IRC04:48
*** afazekas has joined #openstack-infra04:49
*** UtahDave has joined #openstack-infra04:52
*** e0ne has joined #openstack-infra04:58
*** e0ne has quit IRC05:01
*** matsuhashi has joined #openstack-infra05:04
*** chandan_kumar has joined #openstack-infra05:06
*** yfried_ has joined #openstack-infra05:07
*** gokrokve has joined #openstack-infra05:07
*** gokrokve has quit IRC05:10
*** matsuhashi has quit IRC05:10
*** matsuhashi has joined #openstack-infra05:11
*** nosnos has joined #openstack-infra05:13
*** matsuhashi has quit IRC05:17
*** matsuhashi has joined #openstack-infra05:17
*** amotoki has joined #openstack-infra05:20
*** zhiyan is now known as zhiyan_05:26
*** Longgeek has joined #openstack-infra05:31
*** chandan_kumar has quit IRC05:31
*** zhiyan_ is now known as zhiyan05:33
*** praneshp has quit IRC05:41
*** gokrokve has joined #openstack-infra05:41
*** rcarrill` has joined #openstack-infra05:41
*** chandan_kumar has joined #openstack-infra05:42
*** rcarrillocruz has quit IRC05:43
*** gokrokve has quit IRC05:46
*** lazy_prince has joined #openstack-infra05:51
yfried_clarkb: ping05:53
*** Longgeek has quit IRC05:56
*** e0ne has joined #openstack-infra05:58
openstackgerritMarc Abramowitz proposed a change to stackforge/python-jenkins: 100% test coverage
*** e0ne has quit IRC06:00
*** Longgeek has joined #openstack-infra06:02
*** aconrad has quit IRC06:02
*** Longgeek has quit IRC06:02
*** Longgeek has joined #openstack-infra06:02
*** ildikov has joined #openstack-infra06:12
*** gokrokve has joined #openstack-infra06:14
yfried_Can someone point me in the right directrion? I'm looking for the log-pusher script described here and I can't find it. can some06:16
*** gokrokve has quit IRC06:19
*** achuprin has joined #openstack-infra06:22
achuprinHi team!06:22
*** e0ne has joined #openstack-infra06:25
*** rcarrillocruz has joined #openstack-infra06:25
*** rcarrill` has quit IRC06:26
*** jamielennox is now known as jamielennox|away06:31
*** jlibosva has joined #openstack-infra06:40
*** flaper87|afk is now known as flaper8706:42
*** _nadya_ has joined #openstack-infra06:44
*** chandan_kumar has quit IRC06:46
*** skolekonov has joined #openstack-infra06:51
*** UtahDave has quit IRC06:51
*** _nadya_ has quit IRC06:53
*** jcoufal has joined #openstack-infra06:54
*** chandan_kumar has joined #openstack-infra06:54
*** maxbit has joined #openstack-infra06:57
*** _nadya_ has joined #openstack-infra06:59
*** matsuhashi has quit IRC07:01
*** matsuhashi has joined #openstack-infra07:03
*** ociuhandu has joined #openstack-infra07:03
*** e0ne has quit IRC07:05
*** hdd has joined #openstack-infra07:10
openstackgerritDavid Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Remove unused openstack common modules
*** gokrokve has joined #openstack-infra07:14
mattoliverauHey achuprin, sorry for late response :)07:17
mattoliverauyfried_: The scripts don't seem to be in the repository anymore, I guess the documentation needs to be updated. I'll have a poke around and see what I can find.07:18
*** ihrachyshka has joined #openstack-infra07:18
*** gokrokve has quit IRC07:18
*** ihrachyshka has quit IRC07:23
*** matsuhashi has quit IRC07:23
*** ihrachyshka has joined #openstack-infra07:23
*** matsuhashi has joined #openstack-infra07:25
yfried_mattoliverau: hi, thank you. I'm trying to set up logstash on my private ci, and I'm looking for any help there might be07:25
yfried_mattoliverau: do you know of any other documentation? I have all of my logs are collected into a tar file at the end of a job. I'd like to extract this tar and ship everything to logstash07:27
yfried_seems better that creating a logstash agent on every OS node in the deployement and another one on the jenkins, etc...07:28
*** ihrachyshka_ has joined #openstack-infra07:29
*** _nadya_ has quit IRC07:32
BobBallsdague: See when you're around - urgent fix to your change to get XenServer CI back running.  Waiting on tests results first, of course... ;)07:32
*** ihrachyshka has quit IRC07:32
*** shardy_afk is now known as shardy07:38
mattoliverauyfried_: The scripts still exist, it looks like they have moved to a different module in the puppet configuration (config repo). You can find them here:
*** jooools has joined #openstack-infra07:46
mattoliverauyfried_: they use the same file name as mentioned in the documentation07:46
mattoliverauyfried_: I'm not up to speed with the logstash stuff in infra. So sorry if I can't help too much. I'll start taking a look into though so I can better help people in the future :)07:47
mattoliverauWell I'm off for the night, have a great night/day y'all.07:53
*** jpich has joined #openstack-infra07:53
*** e0ne has joined #openstack-infra07:54
*** hashar has joined #openstack-infra07:54
*** jlibosva has quit IRC07:57
*** jlibosva has joined #openstack-infra07:59
yfried_mattoliverau: do you know of anyone who is up to speed with logstash?07:59
*** ildikov_ has joined #openstack-infra07:59
*** ildikov has quit IRC08:02
*** _nadya_ has joined #openstack-infra08:02
*** derekh has joined #openstack-infra08:03
*** yassine has joined #openstack-infra08:09
*** zhiyan is now known as zhiyan_08:11
*** gokrokve has joined #openstack-infra08:14
*** zhiyan_ is now known as zhiyan08:16
*** e0ne_ has joined #openstack-infra08:16
*** e0ne__ has joined #openstack-infra08:17
*** ildikov has joined #openstack-infra08:18
*** gokrokve has quit IRC08:19
*** e0ne has quit IRC08:19
*** reed has joined #openstack-infra08:20
*** e0ne_ has quit IRC08:20
*** pblaho has joined #openstack-infra08:21
*** jgallard has joined #openstack-infra08:21
*** ildikov_ has quit IRC08:21
*** jlibosva has quit IRC08:23
*** vponomaryov has joined #openstack-infra08:24
*** jlibosva has joined #openstack-infra08:26
*** Ryan_Lane has quit IRC08:32
*** starmer_ has quit IRC08:33
*** morganfainberg_Z is now known as morganfainberg08:34
*** darraghb has joined #openstack-infra08:38
*** chandan_kumar has quit IRC08:46
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Added test for migrations
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Fixed xml output generation for python 2.7.5
*** dizquierdo has joined #openstack-infra08:56
*** ihrachyshka_ has quit IRC08:57
*** lazy_prince has left #openstack-infra08:59
*** chandan_kumar has joined #openstack-infra09:05
*** jp_at_hp has joined #openstack-infra09:05
*** etoews has joined #openstack-infra09:07
*** gokrokve has joined #openstack-infra09:14
*** etoews has quit IRC09:15
*** gokrokve has quit IRC09:19
*** morganfainberg is now known as morganfainberg_Z09:30
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Added test for migrations
*** mkerrin has quit IRC09:45
*** maxbit has quit IRC09:47
*** maxbit has joined #openstack-infra09:48
*** _nadya_ has quit IRC09:50
*** ociuhandu has quit IRC09:53
*** _nadya_ has joined #openstack-infra09:54
*** _nadya_ has quit IRC09:57
*** alexpilotti has joined #openstack-infra09:58
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Add some more Trove classifiers
*** ihrachyshka has joined #openstack-infra10:00
*** matsuhashi has quit IRC10:05
*** nosnos has quit IRC10:05
*** _nadya_ has joined #openstack-infra10:06
sdagueBobBall: you shouldn't need L35210:12
*** ihrachyshka has quit IRC10:14
*** gokrokve has joined #openstack-infra10:14
*** jgallard has quit IRC10:14
BobBallsdague: because PRECACHE_IMAGES is only set for the official gate?10:15
sdagueit's actually not set anywhere10:16
BobBalloh :)10:16
sdagueexcept in the tool10:16
BobBallOK - I'll remove 352.10:16
sdagueit seems like we maybe want a 2nd case block around heat image10:17
sdaguebecause without a working heat image, you can't run a bunch of the heat tests10:17
BobBallhmmm - I wonder why it worked before.  Maybe all of the heat tests were exlcuded from the XS CI10:18
*** gokrokve has quit IRC10:18
*** tchaypo has quit IRC10:20
BobBallMaybe we should just disable heat for xenserver for now?10:21
*** rcarrillocruz has quit IRC10:23
*** rcarrillocruz has joined #openstack-infra10:28
BobBallsdague: I'm not sure I understand... was HEAT_FETCHED_TEST_IMAGE set somewhere else before?  Because if not, then surely there was no heat image in devstack before your change?10:28
sdagueit was10:29
sdaguethe conditional was a little different10:29
sdaguemy guess is the xen ci wasn't setting it at all10:30
BobBallah - I see it in d-g10:30
BobBallLet me check10:30
BobBallseems that it was only set if DEVSTACK_GATE_TEMPEST_HEAT_SLOW was enabled10:31
BobBallwhich it wasn't10:31
sdagueso we should put another hypervisor case block around this10:31
BobBallor shift it into the hypervisor block above?10:31
*** chandan_kumar has quit IRC10:32
BobBallI think (?) that libvirt is the only hypervisor that supports qcow210:32
BobBallso that image can only be valid for libvirt since glance will reject it for everything else if properly configured?10:32
*** tchaypo has joined #openstack-infra10:37
BobBallsdague: OK - I propose to block out the enabling of HEAT so that it is only enabled for libvirt, and move the qcow2 image for heat into the libvirt section of the existing VIRT_DRIVER case statement10:38
BobBalldoes that make sense?10:38
*** tchaypo has quit IRC10:41
*** zhiyan is now known as zhiyan_10:41
*** tchaypo has joined #openstack-infra10:43
BobBallsdague: make sense then?10:43
*** ihrachyshka has joined #openstack-infra10:46
*** andreaf has joined #openstack-infra10:47
lifelessBobBall: baremetal supports qcow210:48
lifelessBobBall: moderately sure lxc does as well10:48
BobBalldrat.  but thanks lifeless10:48
lifelessBobBall: ironic does too (buts its not merged yet)10:48
BobBallyeah - lxc is libvirt though isn't it?10:48
sdagueBobBall: ok, so that change can't go in. Because we don't want to not turn on heat10:49
BobBallOK - I'll take the wimps way out and just disable heat for xenserver to avoid this10:49
lifelessnot sure, gotta crash :). Another way to addrewss the issue woude to fix qcow support for other drivers.10:49
BobBalloh - sorry - I thought you said that heat couldn't work without that image?10:49
sdagueBobBall: some of the heat tests don't work10:49
BobBalllove to lifeless - but that's far too big a fix - we need the CI back running soon :)10:49
sdagueso let's just do another case block10:49
*** chandan_kumar has joined #openstack-infra10:49
sdagueif [[ "$ENABLED_SERVICES" =~ 'h-api' ]];10:50
sdaguecase (hypervisor)10:50
sdagueand add the images there10:50
BobBalland set the image for all non-XS hypervisors?  or just libvirt + baremetal?10:50
BobBallwell - maybe libvirt + * with xenserver setting to nothing10:50
sdagueright now libvirt + barebetal is probably fine10:50
sdagueHEAT_CFN_IMAGE_URL being set triggers a tempest variable later10:51
sdaguewhich turns on a set of tests which need a HEAT_CFN image10:51
BobBallsdague: Maybe then10:55
sdagueBobBall: yep, that's good10:56
BobBallOK - let's hope all the tests pass then10:56
BobBallthanks for the help10:56
*** mburned_out is now known as mburned10:57
* BobBall sits and twiddles his thumbs until they pass10:57
*** ildikov_ has joined #openstack-infra11:01
*** mkerrin has joined #openstack-infra11:01
*** yaguang has quit IRC11:02
*** e0ne has joined #openstack-infra11:02
*** ildikov has quit IRC11:04
*** yamahata has quit IRC11:04
*** e0ne__ has quit IRC11:05
*** fifieldt has quit IRC11:07
*** fifieldt has joined #openstack-infra11:08
*** dizquierdo has quit IRC11:10
*** gokrokve has joined #openstack-infra11:14
*** gokrokve has quit IRC11:18
*** jaypipes has joined #openstack-infra11:19
*** viktors has joined #openstack-infra11:32
*** rcarrill` has joined #openstack-infra11:33
*** maxbit has quit IRC11:33
*** rcarrillocruz has quit IRC11:34
viktorsHi folks! Please, could anyone review patch (Import oslo.db repository) ? Thanks!11:35
*** dizquierdo has joined #openstack-infra11:39
*** akscram has quit IRC11:43
*** akscram has joined #openstack-infra11:44
*** maxbit has joined #openstack-infra11:44
*** e0ne has quit IRC11:45
*** e0ne has joined #openstack-infra11:45
*** tristanC has joined #openstack-infra11:48
*** mbacchi has joined #openstack-infra11:48
*** jgallard has joined #openstack-infra11:50
*** e0ne has quit IRC11:50
*** zz_gondoi is now known as gondoi11:50
*** maxbit has quit IRC11:55
*** bcrochet has quit IRC11:56
*** bcrochet has joined #openstack-infra11:56
*** ArxCruz has joined #openstack-infra11:57
*** bcrochet has quit IRC12:00
sdaguefungi back today?12:01
sdaguenodepool looks odd12:01
*** bcrochet has joined #openstack-infra12:02
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: add support for python builder
*** Longgeek has quit IRC12:03
*** chandan_kumar has quit IRC12:06
*** rlandy has joined #openstack-infra12:08
*** gondoi is now known as zz_gondoi12:09
*** adalbas has joined #openstack-infra12:10
*** e0ne has joined #openstack-infra12:11
openstackgerritThomas Leaman proposed a change to openstack/requirements: Move to newer version of python-swiftclient
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Factor out common code between cli utilities
*** jaypipes has quit IRC12:12
*** yamahata has joined #openstack-infra12:12
*** pdmars has joined #openstack-infra12:14
*** gokrokve has joined #openstack-infra12:14
*** pdmars_ has joined #openstack-infra12:16
*** Longgeek has joined #openstack-infra12:16
*** pdmars_ has quit IRC12:16
*** gokrokve has quit IRC12:18
*** pdmars has quit IRC12:19
*** pdmars has joined #openstack-infra12:20
*** aysyd has joined #openstack-infra12:21
*** maxbit has joined #openstack-infra12:22
hashar ! [remote rejected] HEAD -> refs/publish/master (you are not allowed to upload merges)12:22
hasharseriously gerrit12:22
*** chandan_kumar has joined #openstack-infra12:24
*** yamahata has quit IRC12:24
*** jaypipes has joined #openstack-infra12:26
*** yamahata has joined #openstack-infra12:27
*** miqui has joined #openstack-infra12:28
*** markmc has joined #openstack-infra12:31
*** yfried_ has quit IRC12:32
*** yfried__ has joined #openstack-infra12:32
*** lazy_prince2 has joined #openstack-infra12:36
*** lazy_prince2 has left #openstack-infra12:37
*** maxbit has quit IRC12:37
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories
fungisdague: back, yes. trying to catch up12:40
fungihashar: are you trying to merge a feature branch in a project in gerrit?12:41
sdaguefungi: welcome back!12:41
hasharfungi: I wanted to merge two changes with master (an octopus merge)12:41
hasharthen put  a patch on top of it :]12:41
hasharsomething like:   git merge --no-ff  pending_change_1  pending_change_2 && git cherry-pick my_new_feature12:41
hasharbut I can wait :]12:42
*** smarcet has joined #openstack-infra12:42
fungihashar: normally pushing merge commits into gerrit is not what you want. it's designed to sequentially merge rebased/cherry-picked/squashed commits since anything else is very hard to review12:42
boris-42fungi hi12:42
fungihi boris-4212:42
boris-42fungi how are you?)12:42
hasharfungi: yup that is painful. The merge was merely to let me work on a future state of the repo to avoid painful rebase later on.12:42
hasharfungi: I am sure I will survive it :]12:42
fungiboris-42: not sure yet. trying to absorb two weeks of scrollback and ml archives, plus it's the openstack project vacation week so technically infra's not here (except that we're upgrading gerrit in a few hours)12:43
*** dims has quit IRC12:43
boris-42fungi prapering for upadte?)12:43
sdaguefungi: I don't think most of the openstack project is on vacation :)12:43
fungiboris-42: yes12:43
sdagueI think ttx needs to not suggest that next time12:43
fungisdague: heh, clearly they didn't listen ;)12:44
boris-42sdague fungi  ahaha=)12:44
sdaguefungi: well, once you get settled in, nodepool looks quite odd12:44
sdaguein terms of nodes running vs. what it believes should be running12:44
boris-42117 lol12:44
fungisdague: yeah, looking at nodepool i'm betting it's concept of what nodes are available doesn't match reality. probably rackspace rebooted it for us again12:44
fungibad guess... up 5 days12:45
boris-42fungi btw downtime should be just a couple of hrs?12:45
*** timrc is now known as timrc-afk12:46
boris-42fungi btw could you take a look at this patch (already 8x+1 and 1x+2) and only 4 LOCs12:46
fungiboris-42: probably at least a couple hours. we've done multiple dry-runs of the upgrade process with but chances are some features may not be working completely when we first get it running (we're skipping ~4 release versions, from 2.4.x to 2.8.x)12:47
boris-42fungi uhhh=)12:47
boris-42fungi dangerous=)12:47
fungialso, if we end up having to roll back and restore the database from backups, it may be longer downtime still12:48
*** etoews has joined #openstack-infra12:48
*** mfer has joined #openstack-infra12:48
fungianyway, i'm working on diagnosing nodepool's apparent schizophrenia for the moment so we can get jobs running again12:49
*** eharney has joined #openstack-infra12:49
boris-42fungi ok12:50
*** weshay has joined #openstack-infra12:51
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories
boris-42fungi btw successfully catch of issue with eventlet under load in keystone in gates !12:51
*** ociuhandu has joined #openstack-infra12:52
*** dprince has joined #openstack-infra12:54
*** timrc-afk is now known as timrc12:55
*** zhiyan_ is now known as zhiyan12:56
*** dims has joined #openstack-infra12:56
*** rfolco_ has quit IRC12:56
fungisdague: still looking, but initial analysis suggests that nodepool's numbers are accurate, however it's definitely maxxed out on quotas and looks like it's adding/deleting a lot of nodes... there could perhaps be build failures at play12:56
*** rfolco_ has joined #openstack-infra12:58
*** smarcet has quit IRC12:58
*** mark0n has joined #openstack-infra12:59
*** yfried has joined #openstack-infra12:59
fungiquite a lot of "Exception while testing ssh access...Connection timed out"13:00
*** dkehn_afk is now known as dkehn13:00
*** ociuhandu has quit IRC13:00
fungiand also "Exception launching node id...Timeout waiting for ssh access"13:00
*** smarcet has joined #openstack-infra13:00
fungiand also quite a few nova boot attempts are resulting in servers in error state13:01
*** ominakov has joined #openstack-infra13:03
*** sdake_ has quit IRC13:03
*** yfried__ has quit IRC13:04
viktorsfungi: Hi! If you have a time, could you please look at patch (Import oslo.db repository) ?13:05
sdaguefungi: that ratio of deleting / building / in progress just seems like way off13:06
sdagueunless building got very slow13:06
fungisdague: yeah, i think those are nodes which are building, then the ssh test to them fails, then they're deleted, but then the providers are slow to react to delete calls13:07
fungistill trying to see if it's a particular provider or image or what13:07
*** homeless has joined #openstack-infra13:09
fungiyeah, i think something's up with rackspace... all the "used" state nodes are in hpcloud with the exception of a couple in rax-dfw (way too few, and nothing in use in ord or iad)13:09
*** gokrokve has joined #openstack-infra13:14
*** dkranz has joined #openstack-infra13:16
fungino obvious limits maxxed out13:16
*** unicell has quit IRC13:17
*** saper_ has joined #openstack-infra13:18
*** gokrokve has quit IRC13:18
*** saper has quit IRC13:19
*** signed8bit has joined #openstack-infra13:21
*** sdake_ has joined #openstack-infra13:21
*** doug-fish has joined #openstack-infra13:22
fungii'm doing some nova boot tests now just to see if i can spot anything obvious13:30
fungitrying with saturday's image instead of sunday too, just to rule out a new image from yesterday being the culprit13:30
*** julim has joined #openstack-infra13:31
BobBallI'm using RAX nodes just fine btw13:32
BobBallnot that there is a huge amount of similarity13:32
BobBallbut it's RAX and it's nodepool - that's where the similarity ends ;)13:32
sdagueyeh, it just strikes me as odd that we can only managed 240 in use nodes13:32
yfriedwhat's the difference between logs files marked as "oslofmt" to keystone log files?13:33
fungitook about 30 minutes to complete a bare-precise build in rax-ord, and now that it's supposedly active, i can't reach it on 22/tcp13:33
fungiand that was using the nodepool image from saturday13:34
fungithough ord is in the middle of getting upgrades for the past several days (according to support tickets and their status site), so i should test it with iad to rule out the possibility this is different13:35
*** mriedem has joined #openstack-infra13:36
*** mriedem has quit IRC13:37
sdagueshould we pull ord out then? If we're backing up on throwing out nodes and having to restart them because we allocate them in bad regions, that will definitely cause challenges.13:37
*** mriedem has joined #openstack-infra13:38
fungisdague: no idea yet. i have a feeling what we're seeing isn't just in ord, but all of rackspace13:38
fungii can get to the virtual console of the vm and it's booted to a normal login prompt, but i can't get it to respond over the network. i'll need to try building one with a script to set up an account with a known password to actually do any troubleshooting from that end though13:40
*** lcostantino has joined #openstack-infra13:41
*** zns has joined #openstack-infra13:41
SergeyLukjanovfungi, hey, how was your vacation?13:43
SergeyLukjanovanteaya, morning13:43
anteayawelcome back fungi13:43
anteayacongratulations on getting married13:43
anteayaw00t sdague13:44
fungiSergeyLukjanov: wasn't entirely vacation. more wedding and house hunting13:44
fungianteaya: thanks13:44
anteayawelcome to your first day with the new colours13:44
SergeyLukjanovfungi, oh, congratulations!13:44
sdaguefungi: are there fun pictures from the wedding some where? :)13:44
anteayamostly jeblair has been streamlining activity to focus on gerrit upgrade and anything not gerrit upgrade gets put on the later list13:44
anteayain terms of figuring out how many plates you want to spin this morning fungi13:45
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Project Groups API
fungisdague: probably. christine showed me some previews the photographers put up on their facebook blog thingy, but i don't have any urls off hand. later this week probably13:45
anteayayfried: the best logstash person is clarkb who is in hawaii this week13:45
fungiSergeyLukjanov: thanks13:45
anteayasome folks did take an infra off week13:45
*** thomasem has joined #openstack-infra13:45
sdagueyfried: keystone used to not use oslo loging13:45
sdagueit had a custom format13:46
anteayaSergeyLukjanov: you are in california, are you not?13:46
sdagueanteaya: thanks, now I just need cody to get online to bootstrap me into the network :)13:46
anteayasdague: let me know if there is anything I can do to help13:47
anteayaI don't know cody's schedule so I have no estimate as to when he will be online13:47
SergeyLukjanovanteaya, not yet, I'll be in CA at this weekend13:47
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Project Groups API
SergeyLukjanovanteaya, I'm trying to finish things here and it's a bit too late I was starting doing it...13:48
fungisdague: it looks like booting from snapshots is taking waaay, waaaaaaay longer than booting from base images... does that sounds likely? different storage backends maybe?13:48
anteayaSergeyLukjanov: ah okay13:48
anteayaSergeyLukjanov: /me nods13:48
mark0nCan anyone help me to determine if fixes a bug? I'm not a Python programmer and thus not familiar with installation of Python modules...13:49
anteayamark0n: well Marc Abramowitz has given you some feedback13:50
mark0nhere is what happens on my Debian Wheezy machine:
anteayaany chance you can find out who this person is and ask them for some further guidance?13:50
msabramomark0n: As far as I can tell that is not necessary, but maybe I am forgetting some case13:50
anteayamsabramo: are you Marc Abramowitz?13:50
msabramoanteaya, mark0n: I think we just found out who he is :-)13:51
mark0nanteaya: right. And what he says is right. But I still cannot _run_ JJB13:51
msabramomark0n: Hmmm, let me look at your gist13:51
msabramohmmm, interesting13:51
msabramoit seems to have downloaded the pbr egg file but then it still complains13:52
sdaguefungi: interesting13:52
sdaguedefine "way longer"13:52
dhellmannmark0n, msabramo, anteaya: if jjb depends on having pbr installed at runtime, that's likely a bug in jjb's requirements specifications13:53
fungisdague: 3-4x13:53
sdagueyeh, that would explain the build time right?13:53
*** zehicle has joined #openstack-infra13:53
msabramodhellman: oh yeah13:53
msabramooops, dhellmann: oh yeah, you're right13:53
msabramodidn't think about that it was at runtime13:54
fungisdague: though the bigger issue is the machines not being reachable over the network once booted. i can ssh into one i built from a precise base image, but not the bare-precise snapshot from saturday or sunday13:54
msabramolooks pretty vanilla13:54
msabramo    setup_requires=['pbr'],13:54
msabramo    pbr=True)13:54
dhellmannmsabramo: check the requirements.txt -- the last line has pbr13:54
anteayafungi: might that be a dns issue?13:54
fungisdague: i probably should test taking a snapshot of a reachable server, and then deleting it and booting a new one from that13:55
msabramodhellmann: ah ha! That doesn't need to be there, eh?13:55
dhellmannmsabramo: right13:55
fungianteaya: nope, this is the ip address reported by novaclient not responding over the network13:55
dhellmannmsabramo: if you submit the patch, I can +2 :-)13:55
dhellmannmsabramo: oh, actually, I can't on jjb :-/13:55
anteayafungi: ouch13:55
anteayadhellmann: you can +1 though13:56
msabramodhellmann: why does this cause the problem mark0n has though? Shouldn't setuptools still use the egg that was downloaded?13:56
dhellmannanteaya: yeah13:56
dhellmannmsabramo: that's a good question13:56
*** enikanorov__ has joined #openstack-infra13:56
msabramodhellmann: that said, this will probably fix his problem, though I'm not sure why :-)13:56
*** zehicle_at_dell has quit IRC13:57
*** _nadya_ has quit IRC13:57
msabramodhellmann, mark0n: Could be a buggy version of setuptools perhaps?13:57
dhellmannmark0n: do you have any version of pbr installed at all?13:57
mark0nisn't the setup process supposed to install pbr into /usr/local/lib/python2.7/dist-packages/?13:57
msabramooh maybe if he has pbr installed at the wrong version?13:57
msabramomaybe that overrides the egg and then it complains13:58
dhellmannmark0n: the install-time requirements for jjb don't specify a version, so if you have an old version installed it won't install the one that is listed as the (invalid) runtime requirement13:58
mark0nnot before I run the commands in the Gist. I'm starting with a fresh Wheezy system13:58
openstackgerritMarc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Remove pbr from requirements.txt
dhellmannok, new theory: the install-time requirement for pbr isn't installed to dist-packages (because as an install-time requirement it's "temporary")13:59
*** zehicle has quit IRC13:59
*** enikanorov_ has quit IRC13:59
dhellmannhowever, since it is present while jjb itself is being installed, that temporary pbr also prevents the proper one from being installed13:59
dhellmannthen after installation, the temporary copy is removed, leaving the system broken13:59
msabramoyeah I think it's something like what dhellmann is saying14:01
msabramosetup_requires makes it download an .egg file to the current directory14:01
msabramowhich it uses for python stuff14:01
msabramobut when you run jjb, it doesn't pick that up, maybe because it's not looking in that directory14:01
msabramoanyway, should fix it14:02
fungisdague: off the top of your head, do you happen to know what14:02
fungiwhat "Instance building... 40% complete" corresponds to?14:02
fungithat's where nova boot seems to spend the majority of its time14:03
mark0nmsabramo: I can confirm that this fixes the issue!14:03
msabramomark0n: Awesome! Thanks for verifying!14:04
msabramomark0n: Give it a +1 at
mark0nmsabramo: done14:04
sdaguefungi: nope14:04
sdaguethat on rax?14:04
fungisdague: yep14:05
chmouelit would be nice to have this plugin installed for that shiny new 2.8 install we will have
fungisdague: bingo!14:07
fungisdague: ip address bound to eth0 is *not* the ip address reported by nova14:07
*** rwsu has joined #openstack-infra14:08
anteayachmouel: hey14:09
anteayachmouel: your review tools ux session got accepted, congratulations14:09
fungithe ip address reported bound to eth0 seems to correspond to an iface eth0 inet static stanza in /etc/network/interfaces. wonder how that's getting there (cloud-init?)14:09
openstackgerritIhar Hrachyshka proposed a change to openstack/requirements: Bump python-keystoneclient to >= 0.8.0
anteayachmouel: it is a bit late in the process for us to consider new gerrit plugins14:10
anteayasince we have had no time to test them14:10
anteayaI can suggest that jim take a look at it14:10
anteayabut I can't make any promises14:10
anteayachmouel: thanks for letting us know about it though14:10
*** ociuhandu has joined #openstack-infra14:11
fungianteaya: chmouel: once we're done with the upgrade, we can certainly test new plugins on review-dev (maybe next week?) and evaluate whether they're worthwhile to add14:11
anteayathere we go, that sounds like a plan14:11
openstackgerritAndreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools to 0.13
*** jgrimm has joined #openstack-infra14:13
Shrewsfungi! wb & congrats!14:14
*** andreaf has quit IRC14:14
fungiShrews: thanks14:14
Shrewsfungi: olaph and i have decided you must meet us at Dos Perros before you move14:14
*** gokrokve has joined #openstack-infra14:14
fungiShrews: i've still got about 6 weeks, so can definitely work that in14:14
Shrewsor, anywhere, really14:14
fungiand then you'll have to come meet me at the beach after i move ;)14:15
mordredlook, it's a fungi!14:15
Shrewsfungi: cool. as soon as olaph quits tending his goats, chickens, cows and ducks and makes his way back to irc, we'll work out something14:16
fungimordred: what did i miss of note? gerrit upgrade slated for an hour and change from now... did we also change how puppet agent is being triggered?14:17
fungithough i saw some mention of that in scrollback14:17
fungier, thought14:17
*** msabramo has quit IRC14:18
anteayafungi: yes14:18
mordredfungi: yup14:18
anteayait is being triggered via ssh14:18
mordredwhat anteaya said14:18
anteayahey mordred14:18
mordredhey anteaya14:18
anteayaI have to be afk for about 20 minutes14:19
*** mark0n has quit IRC14:19
anteayaback soon14:19
*** gokrokve has quit IRC14:19
mordredfungi: also, I've got a patch that's ready for review around using diskimage-builder in nodepool14:19
*** ociuhandu has quit IRC14:19
mordredfungi: and by review, I mean it should be possible for you to pull the patch, follow instructions ina  readme and produce a diskimage locally14:19
mordredwhich you can then mount or whatever and inspect14:20
fungimordred: exciting. right now i'm just trying to figure out why rackspace nodepool servers have the wrong ip addresses for eth0 in /etc/network/interfaces14:20
*** exz has quit IRC14:20
mordredfungi: wow. that's exciting14:20
fungidid we merge any changes which could have caused that, maybe fridayish?14:20
mordredI don't _think_ we did anything related to /etc/network/interfaces14:20
*** exz has joined #openstack-infra14:21
fungii'm booting a second one now to see if maybe they're all inheriting an ip address from the snapshot image14:21
*** _nadya_ has joined #openstack-infra14:21
fungior whether something else (file injection? cloud-init?) is just doing the wrong thing14:21
*** skolekonov has left #openstack-infra14:22
*** gokrokve has joined #openstack-infra14:22
*** rcarrillocruz has joined #openstack-infra14:22
sdaguefungi: I looked through devstack logs and didn't find anything suspect14:23
fungisdague: well, the one i'm testing with is the bare-precise snapshot, so devstack wouldn't have anything to do with that14:23
BobBallfungi/mordred: Any chance one of you could approve an urgent* fix for devstack at *the xenserver 3rd party CI is broken waiting on this change14:24
mordreddhellmann, anteaya: fyi- there is a bug in setuptools that was being tripped earlier14:24
mordredthe fix is to use "pip install ." instead of python install14:24
*** rcarrill` has quit IRC14:24
sdagueBobBall: I can push it in14:24
BobBallAh - great, thanks sdague! as you can see, tests all came back with green things.14:25
*** adalbas has quit IRC14:25
jeblairfungi: good morning, congratulations, and welcome back!14:25
*** annegentle has joined #openstack-infra14:25
fungijeblair: thanks14:25
mordredmorning jeblair14:26
mordredfungi: and yes, congratulations!14:26
jeblairmordred: good morning14:26
fungithanks mordred14:26
jeblairi see fungi has been briefed14:26
fungibriefly briefed anyway14:27
jeblairbriefly briefed is the best brand of briefed14:27
*** pcrews has joined #openstack-infra14:28
fungiokay, situation confirmed. building two virtual machines from the same nodepool snapshot, the're coming up with the same (wrong) ip addresses bound to eth0, which match what's statically set in /etc/network/interfaces14:29
dhellmannmordred: nice, I hadn't spotted that mark0n was using install14:29
*** gokrokve has quit IRC14:29
jeblair#status notice Gerrit downtime for upgrade begins in 90 minutes.  See:
openstackstatusjeblair: sending notice14:30
mordredjeblair: any reason we don't use in ?14:30
fungihow do we expect rackspace to normally set ip addresses? dhcp?14:30
*** david-lyle has joined #openstack-infra14:30
mordreddhellmann: I would love it if setuptools worked, btw14:30
*** david-lyle has quit IRC14:30
dhellmannmordred: join the club14:30
jeblairmordred: only a matter of time.  there may even be a review for that.  unsure.14:30
mordredjeblair: ok. so it's a thing we think is a good idea14:30
-openstackstatus- NOTICE: Gerrit downtime for upgrade begins in 90 minutes. See:
jeblairfungi: file injection of /etc/network/interfaces i believe.14:30
*** david-lyle has joined #openstack-infra14:30
jeblairmordred: very much so, i believe it was written for exactly that purpose14:31
fungijeblair: okay, seems to be busted in that case14:31
mordredjeblair: great.14:31
dhellmannmordred: to be fair, we do tend to go out to the rough edges for a lot of cases14:31
mordreddhellmann: :)14:31
*** david-lyle has quit IRC14:31
openstackstatusjeblair: finished sending notice14:31
*** david-lyle has joined #openstack-infra14:32
*** mdenny has joined #openstack-infra14:32
jeblairfungi: has the problem you are working on broken all our nodepool rax nodes?14:32
fungijeblair: yes14:33
fungijeblair: seems that way14:33
jeblairfungi: have you spun up a node from a base image?14:33
fungijeblair: just trying to get some confirmation before i open a support case with them,14:33
fungijeblair: i have, and it works fine. doing another now14:33
jeblairfungi: in those cases, does it appear it got the ip from file injection or dhcp (or some other magic?)14:34
fungijeblair: that's what i'm checking with the new one i'm spinning up14:34
fungishould know in a couple more minutes14:34
* jeblair feels like he is nearly caught up. :)14:34
*** rfolco_ has quit IRC14:34
jeblairfungi: how old are our nodepool rax images?14:35
fungijeblair: saturday and sunday14:35
*** annegentle has quit IRC14:35
fungijeblair: i've been testing with an image from saturday just to see if it was any more likely to work, but no such luck14:35
jeblairfungi: i believe on friday we merged a change to chattr +i /etc/resolv.conf14:36
fungiperhaps that breaks all their file injection when it can't overwrite that14:36
jeblairfungi: because apparently, somehow the rackspace images were managing to overwrite them; i was suspecting that they may have been directly injecting that too14:36
jeblairfungi: that's what i was just starting to think14:36
jeblairif so, then we have a choice: dns or network14:37
pleia2going to tweet about the upgrade one last time, all still on track? (I'm on vacation, but figured I'd pop in)14:37
fungijeblair: gigantic hosts files ftw!14:37
jeblairpleia2: yep14:37
pleia2cool, good luck :)14:37
jeblairmordred: we may want to think about either doing something in rc.local to fix resolv, or having nodepool do it14:37
jeblairbut also....14:37
mattymogerrit is taking for ever to git review (thankfully the upgrade will start soon and solve all problems!)14:38
mordredjeblair: I have a think in rc.local in the nodepool patch14:38
mordredjeblair: I could break that out and have puppet do it14:38
fungijeblair: yep, confirmed. working server booted from a base image instead of a snapshot has a (correct) static stanza in the interfaces file, so they do indeed seem to be trying to inject that14:38
*** sandywalsh has joined #openstack-infra14:38
jeblairmordred: but also, i still want to know how we will have our image configure network -- eg, will we need to do file injection on rax and dhcp on hp?  etc...14:38
jeblairmordred: (with dib)14:38
jeblairfungi: let's revert the chattr +i change and see if it fixes things14:39
fungijeblair: agreed. we'll need new rackspace images after that14:39
mordredline 3614:39
mordredis what I'm having the rc.local file do14:39
*** exz has quit IRC14:39
*** exz has joined #openstack-infra14:40
*** wenlock has joined #openstack-infra14:40
mordredjeblair: good question re: images and network - I think it should Just Work though - rax can file inject /etc/network/interfaces on rackspace14:40
jeblairmordred: that's not the worst thing in the world.14:40
mordredregardless of what our image content it14:40
jeblairmordred: but how will it know to do that?14:40
mordredjeblair: I don't think the image has to know anything14:41
mordredI think that's a reach-down-from-outside thing rackspace just does?14:41
*** mrodden has joined #openstack-infra14:41
jeblairmordred: how will rax know to write that file?  that would be useless on redhat.14:41
*** adalbas has joined #openstack-infra14:41
mordredI honestly have no idea how their fileinjection works14:41
*** prad_ has joined #openstack-infra14:41
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Replace with
jeblairright, this is the other prong of the dib problem i was hoping someone would work on in parallel with you14:42
mordredmaybe they have scripts that looks at the host and determines, rh vs. deb vs. windows/14:42
jeblairwhat would be much nicer is if, say, rax supported dhcp behind the scenes and we could just dispense with file injection on our images14:42
derekhI think nodepool templates are failing to build because of ^14:42
mordredjeblair: ++14:43
*** mwagner_lap has quit IRC14:43
*** exz has quit IRC14:43
*** thomasbiege has joined #openstack-infra14:43
jeblairmordred: i love how we still depend on github.14:43
mordredderekh: +214:43
mordredjeblair: we could, you know download that file and host it on one of our servers14:43
*** exz has joined #openstack-infra14:44
jeblairmordred: i assumed we wanted the latest version or something14:44
mordredjeblair: in general, yes. but we do cache pip releases in pypi.o.o14:44
jeblairderekh: aprvd14:44
mordredjeblair: so we could just cache a version of the script, reference that in our stuff, and have it install pip from our mirror14:44
openstackgerritA change was merged to openstack-infra/devstack-gate: Revert "Revert "update grenade branches""
mordredjeblair: you want me to cook up a puppet version of the rc.local patch?14:45
sdagueoh finally ^^^^ Revert "Revert "update grenade branches""
anteayasdague: \o/14:46
anteayait was quite the bug parade14:46
sdagueyou should look at the recheck comment list on that review, I managed to uncover 3 previously unknown races in it14:46
sdaguemostly nova issues14:46
jeblairmordred: i'm actually wondering if it would be better to leave that in nodepool14:46
jeblairmordred: we shouldn't need it on long running servers14:46
mordredjeblair: nod. good point14:47 then?14:47
jeblairmordred: yeah14:47
mordredwe already do unbound stuffs there14:47
jeblairmordred: and with that in place, all the chattr stuff can go away i believe14:47
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Revert "Always make /etc/resolv.conf immutable"
*** talluri has joined #openstack-infra14:47
jeblairmordred: actually it may still be necessary on redhat14:47
jeblairmordred: that's why we put it there in the first place14:48
anteayajeblair: chmouel brought this gerrit plugin to our attention: fungi suggests we might be able to test it on review-dev next week and then go from there14:48
*** salv-orlando has joined #openstack-infra14:48
jeblairmordred: because all of the places where we could disable the write-resolv-conf behavior are file-injected on rax14:48
*** talluri has quit IRC14:49
jeblairmordred: so actually, chattr probably can go away with the rc.local change14:49
*** rcarrill` has joined #openstack-infra14:49
*** talluri has joined #openstack-infra14:49
jeblairmordred: since it will fix the nodepool case, and the long-running-server case should similarly self-correct14:50
fungijeblair: do you still have an e-mail discussion thread going with rax engineering about the dns blacklist lag? would be nice if they could suggest a working solution to not using their resolvers but still having working network interfaces14:50
jeblairfungi: it died off14:50
*** rcarrillocruz has quit IRC14:50
jeblairfungi: i think i'd like to just hold off and see if we can use our own images to fix this14:51
fungii guess rc.local is a viable solution so long as we don't need dns resolution prior to rc.local being run14:51
jeblairfungi: and if it becomes a problem then, continue the discussion14:51
fungimakes sense14:51
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Overrwrite resolve.conf at boot time
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Remove chattr
*** talluri_ has joined #openstack-infra14:51
*** talluri_ has quit IRC14:52
*** talluri has quit IRC14:52
*** atiwari has joined #openstack-infra14:52
fungiso we want to try that stack as a fix? if so i'll abandon the revert in 9076114:53
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch review.o.o out of testmode
*** talluri has joined #openstack-infra14:53
*** talluri has quit IRC14:54
anteayamorning zaro14:54
jeblairfungi: yeah, i think we could try those 2 instead14:55
jeblairfungi: i +2d; feel free to aprv14:55
jeblairzaro: morning14:56
mordredmorning zaro14:56
*** gokrokve has joined #openstack-infra14:56
jeblairmordred: did your 'how to stop puppet' doc change land?14:56
*** _nadya_ has quit IRC14:57
fungimordred: does the empty line before the shebang in 90763 do something special?14:57
*** gokrokve has quit IRC14:57
*** gokrokve has joined #openstack-infra14:58
*** gokrokve has quit IRC14:58
*** sandywalsh has quit IRC15:01
*** Longgeek_ has joined #openstack-infra15:01
jeblairyes, the doc change did merge but it's not reflected at
*** sandywalsh has joined #openstack-infra15:02
mordredwell - simple story "puppet agent --disable"15:02
*** Ryan_Lane has joined #openstack-infra15:03
jeblairi'm guessing the docs build or publish job is broken15:03
mordredand then the thing in the docs that tells you how to run it by hand15:03
*** chandan_kumar has quit IRC15:03
jeblairfungi, mordred: upgrade plan here:
*** Longgeek has quit IRC15:03
*** rcarrillocruz has joined #openstack-infra15:04
*** beekneemech is now known as bnemec15:04
anteayahow far to run?15:04
fungijeblair: the upgrade will involve replacing review.o.o with a new server?15:04
mordredfungi: yes15:04
fungii guess so. my connection lagged15:05
jeblairfungi: yes!  i guess that was left out of your briefing! :)15:05
mordredfungi: the new server is a performance node - we tried upgrading to trusty too - but that's a bucket of fail right now, so we have to wait for puppet3-ification everywhere15:05
fungijeblair: it was merely left out of my reading comprehension going through the ml posts and meeting minutes, more likely15:05
*** rcarrill` has quit IRC15:05
jeblairfungi: shiny new performance2 node.  /home/gerrit2 is on a center volume.15:05
mordredor cinder15:05
jeblairthat too15:06
jeblairfungi: that tidbit might have just been buried in channel logs.15:06
fungiis old review.o.o going git replication to new review.o.o or is there an rsync step hidden in there?15:06
jeblairfungi: rsync15:07
jeblairfungi: also, db is trove15:07
fungioh, nice!15:07
mordredwhich, I believe, makes paste our last non-trove thing15:07
mordredwell, and puppetdb which runs postgres15:08
jeblairfungi: we moved the local apache git mirror and the jeepyb git cache to /opt/lib (from /var/lib) so that they live on the ephemeral volume15:09
*** yfried has quit IRC15:12
*** Ryan_Lane has quit IRC15:13
zarofor the upgrade today, I was wondering who will be running the db upgrade?15:14
jeblairzaro: one of mordred, fungi, or i15:14
*** Ryan_Lane has joined #openstack-infra15:14
jeblairmordred: can you fix before we shut gerrit down?15:16
mordredjeblair: yes15:16
fungimordred: did you see my question above for 90763? (soft -1 since it's wrong-but-not-broken and i'd love to get that merged before we shut down gerrit)15:17
*** dizquierdo has quit IRC15:17
jeblairwe can have image builds running once that merges15:17
fungioh, what jeblair said15:17
anteayawhat will zuul do when gerrit is down?15:18
fungianteaya: a lot of nothing15:18
*** marun has joined #openstack-infra15:18
anteayaso how will patches being tested get their reports?15:19
anteayaor will zuul just pick them up again later15:19
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Overrwrite resolve.conf at boot time
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Remove chattr
*** talluri has joined #openstack-infra15:19
openstackgerritA change was merged to openstack-dev/hacking: Cleanup hacking doc generation
anteayaI'm wondering if I should be advising people to recheck no bug if they are in the queue when gerrit goes offline15:20
jeblairanteaya: we could snapshot the queue when we shut down gerrit15:20
fungii assume the plan is to dump zuul's queues and then kill it and restore later15:20
fungiyeah, that15:20
jeblairit is now :)15:20
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Fix minor non-important error in cache_devstack
jeblairi've stopped puppet on both review hosts; that should mean it's safe to merge all the gerrit upgrade changes15:20
*** Ryan_Lane has quit IRC15:21
mordredjeblair: awesome15:21
jeblairfungi: would you care to get started on that?15:21
jeblairs/stopped puppet/disabled puppet/15:21
fungijeblair: get started merging the gerrit changes? sure15:22
jeblairfungi: looking at the zuul queue, i'm not sure they will make it through in time.  i think we may want to bypass zuul on these.15:22
*** talluri has quit IRC15:23
fungijeblair: i can enqueue them15:23
*** talluri has joined #openstack-infra15:24
jeblairfungi: i was suggesting that you join project bootstrappers and force-merge them15:24
*** ildikov_ has quit IRC15:24
*** talluri has quit IRC15:24
fungicould do that too. need to be in that group to add the new branch anyway15:24
*** viktors is now known as viktors|afk15:24
jeblairnew branch?15:25
mordredjeblair: further down in the etherpad is a list of existing changes15:25
mordredfungi: ^^15:25
fungiline 7 of
fungior has that already been done?15:25
*** thomasbiege has quit IRC15:26
jeblairfungi: oh that's done; whoever did that just didn't mark it15:26
fungigah. okay, so the ones which say "approve after upgrade" should actually be approved now before the upgrade?15:26
*** talluri has joined #openstack-infra15:27
jeblairfungi: yep.  i have no idea how we would approve them after the upgrade :)15:27
mordredjeblair, fungi: do you guys have any objection to a) installing vim-nox on new review b) adding it to the base puppet manifest? I have way too many muscle memories for typing "vim $blah"15:27
fungijeblair: i tend to agree ;)15:27
jeblairmordred: that's fine; it used to come installed on the rax image i think, but not on the pvhvm ones i guess15:28
fungimordred: no objection. seems like something which was stripped out of the new pvhvm images15:28
*** e0ne has quit IRC15:28
*** talluri has quit IRC15:28
*** e0ne has joined #openstack-infra15:29
jeblairthere's a lot of repetition in these upgrade steps, i'm trying to sort it out15:29
anteayaI'm standing by, let me know if there is something I can do to help15:29
jeblair#status notice Gerrit downtime for upgrade begins in 30 minutes.  See:
openstackstatusjeblair: sending notice15:29
*** dizquierdo has joined #openstack-infra15:30
*** markmcclain has joined #openstack-infra15:30
jeblairzaro: why is "# Enable copying of vote scores:" listed in and not in ?15:30
-openstackstatus- NOTICE: Gerrit downtime for upgrade begins in 30 minutes. See:
fungioh, crikey... merge conflicts15:31
mordredfungi: want help?15:31
openstackstatusjeblair: finished sending notice15:31
jeblairzaro: i moved it15:31
*** talluri has joined #openstack-infra15:32
fungimordred: jeblair: zaro: a lot of the changes linked from the end of (lines 30-38) seem to have merge conflicts, negative votes, and so on15:32
jeblairzaro: do you have a copy of the all-projects acl config as it should appear in production after the upgrade?  i just see steps about how to change it.15:32
fungii'll start trying to tackle them in order15:33
jeblairzaro: if not, would you please prepare one that we can simply install?15:33
*** e0ne has quit IRC15:33
zarojeblair: it's on the remaining-gerrit-upgrade-changes etherpad15:33
zarojeblair: manually configure review.o.o15:33
zarojeblair: don't understand what you mean about 69800, it looks like it already merged.15:34
jeblairzaro: that's a change, and it doesn't change anything called "All-Projects.config" which means it's not what we need15:34
YorikSarzaro, jeblair: Hi. Did you consider my proposal to make -1: WIP autocopied to new patchsets in our new Gerrit? ML thread seems to be ended with a question...15:34
mordredjeblair: I think he means it's in
jeblairYorikSar: this is not a good time to have that conversation15:35
mordredjeblair: the new version of the docs acl file has been updated15:35
zarojeblair: it's just a doc change that shows how the acls should be configured.15:35
jeblairzaro: can you extract that into a paste that we can just copy/paste?15:35
zarojeblair: the change for gerrit.rst15:35
YorikSarjeblair: Ok15:35
*** talluri has quit IRC15:35
zarojeblair: will do.15:35
jeblairzaro: thanks.  as for i haven't looked at the change, i just saw that the upgrade script said to merge it, so i asked about it15:36
jeblairzaro: we can't actually merge any changes while we're running the upgrade script, so i moved it to the etherpad of changes that need to merge15:36
jeblairzaro: oh, that change only applies to review-dev.  is there an equivalent change for review?15:37
*** talluri has joined #openstack-infra15:37
mordredzaro, jeblair: I can get the all-projects acl step -15:38
BobBallWhat'll happen with jobs in the gate queue when gerrit goes bye bye?15:39
jeblairi'm going to start the rsync's15:39
*** Longgeek_ has quit IRC15:39
anteayaBobBall: we will snapshot them15:41
zarojeblair: the equivalent change for review is 82416 and it's on the list to approve after upgrade.15:41
anteayaonce new gerrit is up, zuul should pick up the snapshotted queue15:41
anteayaBobBall: if that fails we will have suggestions15:41
BobBallfair enough :)15:41
jeblairzaro: cool, thanks15:41
anteayaBobBall: thanks15:41
zaromordred: i added all-project.config setting to the bottom of the etherpad.15:42
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Activate secondary index support for Gerrit
*** pblaho has quit IRC15:42
fungizaro: sanity check on those rebases? ^15:42
jeblairi'm manually editing sshd_config on old-review to allow root logins without-password to do the rsync15:43
*** gokrokve has joined #openstack-infra15:44
*** asselin__ has joined #openstack-infra15:45
*** ociuhandu has joined #openstack-infra15:45
*** atiwari_ has joined #openstack-infra15:45
anteayaoh in case we have a net split, is everyone also on oftc?15:45
*** stevebaker has quit IRC15:45
*** asselin_ has quit IRC15:46
*** atiwari has quit IRC15:46
*** dangers has joined #openstack-infra15:46
fungianteaya: yes15:46
anteayafungi: great15:46
fungi(i am anyway)15:46
anteayaI see you15:46
*** dangers_away has quit IRC15:46
anteayadon't know if it is a concern or not, hopefully not15:47
jeblairanteaya: i was not, thanks for the reminder15:47
jeblair(am now)15:47
anteayazaro? you on oftc #openstack-infra15:47
*** markmcclain has quit IRC15:47
openstackgerritA change was merged to openstack-dev/hacking: Move git checks into their own file
zaroanteaya: no i'm not15:47
fungijeblair: "merge all the changes" might also need puppet disabled on more than review.o.o, yeah? for example thinking could be disruptive15:47
anteayazaro: can you join easily?15:48
mordredjeblair, zaro:
mordredI've gota n all-projects git checked out locally with that patch applied15:48
anteayamordred: you are coming too, right?15:48
zaroanteaya: don't know how15:48
anteayazaro: k15:48
jeblairfungi: i'll disable puppet on zuul15:48
zarojeblair, fungi :sanity checking.15:48
*** gokrokve has quit IRC15:48
fungizaro: thanks15:48
fungijeblair: okay, i'll go ahead with zuul-affecting config changes too in that case15:49
jeblairfungi: done15:49
anteayaif we net split15:49
anteayaotherwise don't worry about it15:49
openstackgerritA change was merged to openstack-infra/config: update trigger to match new workflow label
jeblair rsync -a --delete root@ /home/gerrit2/review_site15:50
jeblairthat look about right ^ ?15:50
jeblair rsync -a --delete root@ /home/gerrit2/review_site/15:50
*** jaypipes has quit IRC15:51
zaromordred: ALCs LGTM15:51
*** stevebaker has joined #openstack-infra15:51
*** mwagner_lap has joined #openstack-infra15:51
jeblair Gerrit is unavailable until further notice for a major upgrade.  See:
jeblairanteaya: ^ look good for the alert announcement?  (will be the channel topic everywhere)15:54
*** gyee has joined #openstack-infra15:54
mordredjeblair: yes15:54
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: disable the jeepyb trivial rebase hook on review.o.o
fungizaro: and that rebase too ^15:55
anteayajeblair: yes15:55
openstackgerritKhai Do proposed a change to openstack-infra/config: upgrade review.o.o to gerrit ver 2.8.4
mordredfungi: lgtm15:56
zaronew change for approve after upgrade ^15:56
*** adalbas has quit IRC15:56
fungi79107 is going to be a fun, fun, fun rebase15:56
mordredfungi: seems like a good candidate for just pulling master and running a script on15:57
fungimordred: maybe... it's also adding acls for some projects (accidentally because of a prior broken rebase?)15:58
fungizaro: ^ ?15:58
mordredyou know what? perhaps in the future with bulk changes like that- we should also post the script we used to generate them in case of need to re-do15:59
mordredfungi: well - the main repeated things seem to be:15:59
jeblairand/or keep them up to date15:59
mordredfungi: s/Approved = +0..+1/Workflow = -1..+1/ ; d/^.*workInProgress =.*$/15:59
jeblairinitial rsyncs are complete15:59
*** jgallard has quit IRC15:59
mordredfungi: and s/label-Approved/label-Workflow/16:00
zarofungi: ohh sorry about that one, probably should have done that yesterday.16:00
zarofungi: probably better just to start over there.16:01
*** dkliban is now known as dkliban_afk16:01
mordredjeblair: yah - keeping up to date is good - but if someone is following us with their own system, scripts we used to generate a particular commit might be helpful either as docs for what changed or to use (also, reading the script might be an easier way for us to review the intent of the change_16:02
mordredjust a thought for next time16:02
anteaya16:02 utc16:02
jeblairmordred: ++16:02
jeblairanteaya: yeah, we're waiting on those changes to be fixed/reviewed/merged16:02
* anteaya nods16:02
jeblairbut yes, we are now starting late :(16:03
*** sweston has joined #openstack-infra16:03
anteayano no16:03
jeblairi don't think it will be late enough to warrant an announcement16:03
fungizaro: sed -e s/label-Approved/label-Workflow/ -e s/Workflow = +0..+1/Workflow = -1..+1/ -e /.*workInProgress.*/d16:03
anteayajust wanted to ensure we hadn't lost track is all16:03
zarook, i think i've sanity checked all so far.16:04
anteayanot rushing16:04
anteayasweston: hey16:04
mordredfungi: yes. looks right to me16:04
anteayasweston: and and
*** jaypipes has joined #openstack-infra16:05
anteayasweston: and if it is easy to join #openstack-infra in case of a net split, that is great too but low priority16:05
*** marcoemorais has joined #openstack-infra16:05
zarofungi: sorry, i no sed afficianado.16:05
SergeyLukjanovjeblair, mordred, fungi, zaro is it ok that review.o.o is still working?16:06
anteayaSergeyLukjanov: yes16:06
mordredSergeyLukjanov: yes. we have not turned it off yet16:06
anteayaSergeyLukjanov: we have some patches we need merged16:06
anteayafor the upgrade16:06
SergeyLukjanovoh, got it16:07
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add vim to the list of explicit things on servers
mordred(that above is just to get it out of my local stack so I don't forget about it - super low priority)16:07
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: update gerrit project configs to use new Workflow label
swestonanteaya: hello!  thanks for the links.  for what it's worth, I just finished the upgrade script testing, with no problems.16:08
*** darraghb has quit IRC16:08
*** afazekas has quit IRC16:08
*** adalbas has joined #openstack-infra16:08
anteayasweston: thank you16:08
fungizaro: see if looks right to you16:09
anteayait is heartening to hear16:09
zarofungi: looking16:09
*** marcoemorais has quit IRC16:09
swestonanteaya: ;-) always welcome16:09
mordredfungi: that patch looks good to me - do you want me to check every file? I've looked at a good pile of them16:10
fungimordred: nah, as long as it looks like it didn't have obvious unintended side effects16:10
mordredlooks consistent and I think I've seen an example of most of the forms it takes16:10
jeblairmordred: ditto16:10
zarofungi: spot checked, LGTM16:11
openstackgerritA change was merged to openstack-infra/jeepyb: Add change abandoned logic to jeepyb
*** marcoemorais has joined #openstack-infra16:11
*** darraghb has joined #openstack-infra16:11
openstackgerritA change was merged to openstack-infra/config: update gerrit core branch
openstackgerritA change was merged to openstack-infra/config: Switch review.o.o out of testmode
fungii assume that one ^ was intended to be merged with the rest16:13
*** UtahDave has joined #openstack-infra16:13
*** denis_makogon_ has joined #openstack-infra16:13
fungii can revert if we need it not applied until after we test things16:13
*** pblaho has joined #openstack-infra16:14
jeblairfungi: it should be merged now16:14
*** denis_makogon_ is now known as denis_makogon16:15
zaroi'm not sure whether this should be before or after upgrade can someone look?
jeblairzaro: before, otherwise we'll end up downgrading review16:16
*** hashar has quit IRC16:17
zarook, good thing i ask.16:17
fungizaro: is the change number on line 39 of correct? it doesn't seem to match the change subject, and was merged a while ago16:17
*** annegentle has joined #openstack-infra16:17
jeblairzaro: the puppet config for the future state needs to be finalized and merged before we can stop old review16:17
*** packet has joined #openstack-infra16:18
zarofungi: 69800 was already merged.  it was for review-dev.o.o16:19
openstackgerritA change was merged to openstack-infra/config: Gerrit-2.8: Add secondary index support
zarofungi: the equivalent chane for review.o.o is 8241616:19
zarofungi: it's already on the list16:20
fungizaro: yep, just noticed16:20
Alex_GaynorI'm assuming it's a known thing that the py3k/pypy builds (which share the same buidlers) either aren't staritng, or have an incredibly deep queue?16:20
anteayaAlex_Gaynor: hi16:20
anteayawe are in the middle of upgrading gerrit right now, sorry about that16:21
fungiAlex_Gaynor: yeah, apparently we can't stop rackspace file injection of resolv.conf without also breaking their file injection of the interfaces config16:21
fungiAlex_Gaynor: should be resolved in a little while, hopefully16:21
anteayaAlex_Gaynor: what fungi said16:21
zaroi'm removing 69800 from the list of changes for upgrading.16:21
openstackgerritA change was merged to openstack-infra/config: Activate secondary index support for Gerrit
Alex_Gaynorfungi: oh, uh, good work us, I suppose16:21
mordredAlex_Gaynor: :)16:22
jeblairzaro: just mark it merged16:22
zarojeblair: done16:22
openstackgerritA change was merged to openstack-infra/config: disable the jeepyb trivial rebase hook on review.o.o
openstackgerritA change was merged to openstack-infra/config: update gerrit project configs to use new Workflow label
fungiokay, i think that's the last of them16:23
zarofungi: don't forget 8162216:23
*** bauzas has joined #openstack-infra16:23
mordredoh. what zaro said16:23
fungiahh, someone moved it further up the file before i got to it16:24
*** alexpilotti has quit IRC16:24
zarofungi: was me, sorry i didn't mention that.16:24
openstackgerritA change was merged to openstack-infra/config: upgrade review.o.o to gerrit ver 2.8.4
*** jcoufal has quit IRC16:25
jeblairanyone need a break before we shut it down?16:25
*** aconrad has joined #openstack-infra16:25
mordredI'm ready to do16:25
mordredI have coffee sitting next to me and just finished a bio break16:25
* zaro is ready!16:25
fungii should probably force in those chattr changes which were approved before we shut down gerrit?16:26
jeblairfungi: oh yeah good idea16:26
zarojust double checked, looks like all changes have merged16:26
fungisince they haven't gotten to the gate due to the backlog they fix16:26
*** jlibosva has quit IRC16:27
zaroshould i join OFTC now16:27
*** msabramo has joined #openstack-infra16:27
mordredjeblair: I just put in a note - one of the things in the manual list can't be done until gerrit is started on new gerrit16:27
openstackgerritA change was merged to openstack-infra/config: Overrwrite resolve.conf at boot time
swestoni tried joining OFTC, looks like I'm the only one there16:27
anteayazaro: if you can easily16:27
openstackgerritA change was merged to openstack-infra/config: Remove chattr
mordredsweston: in #openstack-infra ?16:28
jeblairmordred: where's your note?16:28
anteayasweston: oftc #openstack-infra ?16:28
mordredjeblair: I just added it to the other list - let me put it in the  manual list too16:28
anteayasweston: you should see me fungi jeblair mordred16:28
anteayaand a few others16:28
jeblairmordred: we have 3 etherpads, i don't know which ones you are talking abou16:28
fungijeblair: okay, those are merged too now, so i can get some images rebuilding in rax when i get a breather16:28
mordredjeblair: good point16:28
zaroanteaya: i have connected, how do i join?16:28
mordredjeblair: I have put the node in all of the places I think it needs to be noted now16:29
jeblairfungi: why don't you go ahead and run puppet locally on nodepool and kick one off16:29
anteayazaro /join #openstack-infra16:29
fungijeblair: will do. if that seems to succeed, i should probably just delete all existing rackspace images since they're all broken16:29
zaroanteaya: i'm connected to both freenode and oftc, how do i let my client know that i want to join the oftc one and not the freenode one?16:30
*** jpich has quit IRC16:30
jeblairfungi: good point; actually you could just go ahead and delete them16:30
*** dizquierdo has quit IRC16:30
anteayazaro: not sure, what client?16:30
jeblairmordred: got it16:30
zaroanteaya: weechat16:30
mordredzaro: I do it by switching to the oftc server tab and typing /join #openstack-infra in there16:30
anteayazaro: go to the first window, click control-x16:30
jeblairmordred: zaro i think ctrl-x works in weechat and irssi16:31
jeblairwhat anteaya said16:31
anteayayou should see in the bar server weechat, server freenode, server oftc16:31
anteayawhen you see server oftc /join #openstack-infra16:31
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Correct test_remote to support branchs without upstream
*** eharney has quit IRC16:32
*** talluri has quit IRC16:32
jeblairfungi, mordred: ready for shutdown?16:33
zaroanteaya: thanks.  i'm on16:33
anteayazaro: well done16:33
fungijeblair: sure. are we already zuul queue dumped and killed?16:33
mordredjeblair: ++16:33
*** yfried has joined #openstack-infra16:33
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Correct test_remote to support branchs without upstream
*** mrodden has quit IRC16:33
jeblairfungi: no, will do after i shutdown; i'll do the next few tasks and let you know when we're ready to proceed16:34
fungigood by me then16:34
mordredjeblair: hrm. should we have an apache "down for maint" page?16:34
jeblair#status alert Gerrit is unavailable until further notice for a major upgrade.  See:
openstackstatusjeblair: sending alert16:34
jeblairmordred: maybe, but i don't think we're organized well enough for that this time16:35
jeblairmordred: i expect it's going to be up and down for a bit16:35
mordredjeblair: ++16:36
*** sahid has joined #openstack-infra16:36
-openstackstatus- NOTICE: Gerrit is unavailable until further notice for a major upgrade. See:
*** ChanServ changes topic to "Gerrit is unavailable until further notice for a major upgrade. See:"16:36
*** thomasbiege has joined #openstack-infra16:36
*** dkliban_afk is now known as dkliban16:36
fungistopping puppet on nodepool and applying from a dev env... the make-resolv-conf-immutable notify is broken16:36
jeblairfungi: oops16:37
mordredfungi: piddle16:37
*** thomasbiege has quit IRC16:37
jeblairokay, gerrit is stopped and zuul is saved16:37
mordredI'm going to tranfer the database contents16:37
jeblairi assigned tasks in
openstackstatusjeblair: finished sending alert16:38
bauzasfolks, I know you're in the middle of a major upgrade, but would it be worth pushing an haproxy reverseproxy in front of review.o.o so we could get a nice 404 page instead of a 503 error when looking now ? :)16:38
*** aconrad has quit IRC16:38
jeblairbauzas: you're welcome to contribute that next time16:39
bauzasI can propose a bp on that :)16:39
bauzasjeblair: coolio, will do16:39
jeblairbauzas: better to propose a change16:39
bauzasjeblair: good luck, folks, I'm with you !16:39
mordredERROR 2005 (HY000): Unknown MySQL server host '' (0)16:40
* mordred investigating16:41
jeblairdns is updated; 5 minute ttl16:41
mordredoh. there is no review db16:42
jeblairi rather thought i created on16:42
*** gokrokve has joined #openstack-infra16:44
mordredthere was a host and password entry in hiera -16:45
*** msabramo has quit IRC16:45
fungidid we recently change permissions on authorized_keys files? lots of puppet apply failures for all of them because they were 0400 so root thought it shouldn't write to them16:45
*** ihrachyshka has quit IRC16:45
mordredwe changed puppet to use the ssh_authorized_key method16:45
*** mrodden has joined #openstack-infra16:46
fungistrange. it seemed to not like it at any rate16:46
mordrednew cloud db host in hiera16:46
fungii don't suppose we ever heard back on why trove instances are reachable on 3306/tcp from other tenants16:48
jeblairThis is known and intentional. Just like Cloud Servers, Cloud DBs have an open connection to the datacenter which means all other devices in the datacenter. They don't have public IP addresses and as far as I know they aren't ever going to support Cloud Networks. If you need more control over the connection (or the my.cnf file) it would be much better to deploy MySQL on a Cloud Server.16:48
* fungi sighs loudly16:49
jeblairfungi: would you like to move back to server-local mysql?16:49
*** gokrokve has quit IRC16:49
mordredfwiw, the trove team IS working on neutron support for this cycle16:49
jeblairit should be a fairly small config change to switch back16:50
fungijeblair: it's worth strong consideration, but for now maybe we just hope there are no pre-authentication vulnerabilities for the mysql version in use at rax, keep our instance names hidden away in hiera and hope for the best :/16:51
fungior do you mean for this maintenance specifically16:51
jeblairfungi: either.  :)16:51
*** sweston is now known as sweston_16:51
jeblairfungi: sounds like proceed as planned for this maint, and then we can anguish over whether to move back later16:51
jeblair(it should be a trivial downtime to move back)16:52
fungiyeah, i don't want to unnecessarily extend the down-time for now, and it merits further discussion16:52
mordredvipul: ^^ just fyi - the above is a stupid conversation for us to need to have16:52
mordredSlickNik: ^^ (I'd ping hub_cap too - but he doesn't seem to be on)16:53
jeblairfungi: are you available for upgrade work or are you still on nodeppol?16:53
mordredmysql> select count(*) from changes \G16:53
mordred*************************** 1. row ***************************16:53
mordredcount(*): 9028716:53
mordreddatabase transferred16:53
fungijeblair: i'm futzing with nodepool... my latest image-update just met with 503 SERVICE UNAVAILABLE from bitbucket.org16:54
*** derekh has quit IRC16:54
jeblairfungi: gah ok, i'll take the thing i assigned to you for the upgrade16:54
fungibut i should be able to work on upgradey things now16:54
mordredfungi: btw - I've got a patch to remove the need for that bitbucket stuff16:54
jeblairfungi: np16:54
yfriedHi guys, I'm trying to setup logstash on my jenkins, according to how the gate is set, and it keeps failing
anteayayfried: hi16:54
jeblairfungi: i'll give it back to you :)16:54
yfriedanyone familiar with that?16:55
anteayaright now we are in the middle of a gerrit upgrade16:55
fungijeblair: working on it now16:55
anteayayfried: after the upgrade we can talk about your logstash16:55
fungialso, retrying the image update confirms bitbucket really seems to be toes-up16:55
jeblairfungi: see etherpad -- i looked up the hiera keys for you16:56
jeblairfungi: (we never got around to doing this independently for heartbleed, so we rolled it into this upgrade)16:56
*** sarob has joined #openstack-infra16:56
anteayafungi: yes bitbucket confirms performance issues:
*** hogepodge has joined #openstack-infra16:56
*** sdake_ has quit IRC16:57
jeblairmordred: are you interested in running the upgrade script?16:57
mordredjeblair: sure.16:57
jeblairmordred: all yours then; let me know if you need anything16:58
mordredwell, the script isn't goig to work as written...16:58
mordredlet me edit it real quick16:59
mordred(it's assuming local db)16:59
*** harlowja_away is now known as harlowja17:00
zaromordred: could you check to see if there's an 'account_group_by_id' table in the db?17:00
jeblairzaro, mordred: i'll do that17:01
sdaguefungi: bitbucket definitely had a big outage this weekend, it would not surprise me if they were still recovering17:01
*** SumitNaiksatam has quit IRC17:01
jeblairERROR 1146 (42S02): Table 'reviewdb.account_group_by_id' doesn't exist17:02
zaromordred, jeblair : ohh nice, then that TRUNCATE TABLE workaround is _not_ needed.17:03
fungijeblair: new host keypairs (rsa and dsa) committed in hiera now17:03
jeblairfungi: cool, thanks17:03
jeblairfungi: btw, when we're done, would you mind sending a gpg-signed msg to openstack-dev with the host keys in it?17:03
jeblairer, the fingerprints17:03
jeblairfungi: the docs we published mention the web verification process, but some folks asked for a gpg signed mail too17:04
fungijeblair: gladly! i get it drafted up in a few minutes so it's ready to fire when the time comes17:04
fungier, i'll17:04
*** melwitt has joined #openstack-infra17:04
*** moted has quit IRC17:04
jeblairfungi: i figured you probably have the larger web of trust in the community17:04
zaromordred: i just updated the GERRIT_WAR & GERRIT_DOWNLOAD_URL vars17:04
jeblairboris-42: :)17:05
davidlenwellboris-42: lol17:05
*** bauzas has quit IRC17:05
*** talluri has joined #openstack-infra17:05
*** BobBall is now known as BobBallAway17:06
*** lakshmi has joined #openstack-infra17:06
*** Ryan_Lane has joined #openstack-infra17:08
mordredjeblair: it says "GERRIT_ADMIN=corvus"17:08
zaromordred: yeah, that's for review-dev.  you'll have to change that17:08
yfriedanteaya: what do you mean "after upgrade"?17:09
mordredI think it will be better to run the aggreement bit from the end of the script by hand as a second step17:09
anteayayfried: I mean that any conversation about anything other than the upgrade is a distraction right now17:10
zaromordred: i made a few last minute edits to the script.  just updated the JAVAMELOODY_PLUGIN var as well.17:10
mordredyeah - I'll grab those in a second17:10
anteayayfried: so after the upgrade is finished we can talk about other issues, including your logstash questions17:10
*** esker has joined #openstack-infra17:10
anteayayfried: tomorrow might be a good time17:10
mordredzaro: why does gerrit need to be started if the all-projects git interaction is with the copy on the filesystem?17:11
zarowhich line?17:11
mordredgit clone $GERRIT_SITE/git/All-Projects.git17:11
*** sweston_ has quit IRC17:12
yfriedanteaya: tnx. didn't know about it. sorry. last question can you point me to anyone I can bother privatly who might know?17:12
zaromordred: i assumed you answered your own question there?17:13
anteayayfried: np, well clarkb is our best logstash resource, he would be helping with the upgrade if he were here but he is in hawaii this week17:13
anteayayfried: so try us tomorrow, we will do the best we can17:13
mordredjeblair: would you please double-check /root/ on new review for me?17:15
mordredjeblair: I've made a few edits and changes and I think they're good but would like second eyes17:15
zaroyfried: you might want to ping jesusaurus17:15
jeblairmordred: ack17:15
*** msabramo has joined #openstack-infra17:16
*** nati_ueno has joined #openstack-infra17:16
*** markmc has quit IRC17:16
yfriedanteaya: tnx17:16
yfriedzaro: tnx17:16
jeblairmordred, zaro: ^ ?17:16
zaroupdated on etherpad17:16
mordredjeblair: thanks- javamelody-4744bfb.jar17:16
jesusaurusyfried: you have logstash questions?17:16
mordredonce gerrit is running, I'll copy accepted_agreement_users.txt back to my machine and run it as me17:17
jeblairjesusaurus, yfried: would you mind taking the conversation to another channel?  perhaps #openstack-qa or #openstack-dev17:17
jeblairmordred: will you update
mordredjeblair: yes17:17
jeblairwith the javamelody change17:17
*** sdake_ has joined #openstack-infra17:18
*** sdake_ has quit IRC17:18
*** sdake_ has joined #openstack-infra17:18
jeblairmordred: cp -rp $GERRIT_SITE $GERRIT_SITE-backup17:18
jeblairmordred: suggest removing that line17:18
mordredjeblair: nod17:18
jeblairmordred: actuall rollback strategy is use old host17:18
*** zzelle has joined #openstack-infra17:19
*** zhiyan is now known as zhiyan_17:19
jeblairmordred: lgtm;  if i were you i'd actually just run that one line at a time in an interactive shell17:20
*** ominakov has quit IRC17:20
mordredjeblair: ++17:20
mordredjeblair: also, I just made one more change17:20
*** SumitNaiksatam has joined #openstack-infra17:21
mordredI made a /root/.my.cnf with the db stuff17:21
mordredso the script should now be ehterpad-able17:21
jeblairmordred: cool17:21
jeblairmordred: fire at will17:22
mordredok. I've put it into the etherpad17:22
mordredrocking and rolling17:22
mordredzaro: root@review:/home/gerrit2# /usr/bin/java -jar gerrit-wars/$GERRIT_WAR -d $GERRIT_SITE --batch --no-auto-start17:24
mordredfatal: unknown command -d17:24
*** gokrokve has joined #openstack-infra17:24
*** praneshp has joined #openstack-infra17:24
*** atiwari_ has quit IRC17:24
zarowhat? there's definately a '-d' option17:24
anteayazaro: is there a missing library?17:25
anteayaor dependency?17:25
*** yassine has quit IRC17:25
mordredthis expands to /usr/bin/java -jar gerrit-wars/gerrit-v2. -d /home/gerrit2/review_site --batch --no-auto-start17:25
mordredand the war is there17:25
zaromordred: can you run with --help?17:26
*** ArxCruz has quit IRC17:26
mordredit's missing the word init17:26
jeblairmordred: ++ just came to that conclusion17:26
*** chandan_kumar has joined #openstack-infra17:26
*** sdake_ has quit IRC17:28
*** SumitNaiksatam has quit IRC17:28
*** gokrokve has quit IRC17:29
*** aconrad has joined #openstack-infra17:29
*** jp_at_hp has quit IRC17:31
*** ArxCruz has joined #openstack-infra17:31
mordredit's doing the migration of data, btw17:32
*** amcrn has joined #openstack-infra17:32
*** yidclare has joined #openstack-infra17:32
mordredit's currently migrating to ... ah, it's one17:32
*** thedodd has joined #openstack-infra17:32
*** thomasbiege has joined #openstack-infra17:32
mordredit printed a set of sql it thinks I should run to delete old things17:32
mordredshoudl I do that?17:32
anteayacan you paste?17:32
mordredI have added to teh bottom of the upgrade script etherpad17:33
mordredjeblair: ^^17:33
mordredI'm going ot not do it now because it does not seem urgent17:33
*** ihrachyshka has joined #openstack-infra17:33
jeblairmordred: why don't i go ahead and do that while you continue?17:34
mordredzaro: there is now "javamelody-4744bfb.jar" in review_site/plugins - shoudl that be javamelody.jar ?17:34
*** prad_ has quit IRC17:34
zaroi don't tihnk filename matters.17:35
zaromordred: i would just leave as is17:35
jeblairsql cleanup done17:35
mordredroot@review:/home/gerrit2# mysql reviewdb -e "INSERT INTO account_group_names(group_id,name) VALUES('$ICLA_GROUP_ID','CLA Accepted - ICLA')"17:37
mordredERROR 1062 (23000) at line 1: Duplicate entry 'CLA Accepted - ICLA' for key 'PRIMARY'17:37
*** talluri has quit IRC17:38
fungimordred: i believe gerrit auto-creates those groups on upgrade17:38
jeblair|      324 | CLA Accepted - ICLA               |17:39
jeblair|      325 | Change Owner                      |17:39
jeblairlast two groups17:39
jeblairso they were recently created17:39
fungisince it switches all clas to be group-clas beginning with 2.5.x (when they added a mechanism to auto-accept by automatically adding users to the corresponding group)17:39
mordredin taht case, I believe the upgrade script is done17:40
mordredexcept for pushing in the cla signers17:40
*** eharney has joined #openstack-infra17:40
jeblairaccording to the bug, zaro decided it wasn't necessary after all17:40
zaroyeah that was an invalid bug17:41
mordredfor the record, I've been updating the in root  - so it reflects what was run17:41
*** sweston has joined #openstack-infra17:41
fungiwell, adding the accounts to that group is needed. it wasn't a bug upstream because their non-group cla implementation in 2.4.x was half-baked and thus unsupportable without patches we added to fix it17:41
jeblairmordred: so are we at line 144 in
jeblairfungi: yeah, i think that's next17:42
*** e0ne has joined #openstack-infra17:42
mordredjeblair: next step is "start gerrit"17:42
fungiso they never added a mechanism to upgrade from their broken implementation17:42
anteayafor those following along, this is our upgrade todo list we are working though:
jeblairmordred: pause for a sec17:43
jeblairmordred: it's not clear to me whether we should run puppet now, then start gerrit... or the other way around17:43
jeblairmordred: i think we should run puppet now, then start gerrit17:44
swestonin my experience, if puppet is working correctly, it is better to let puppet do it17:44
jeblairthen proceed with the upgrade script17:44
mordredjeblair: kk17:44
mordredjeblair: I agree17:44
mordredjeblair: would you like to do that?17:44
*** hemna has joined #openstack-infra17:44
jeblairmordred: i will do that17:44
*** thomasbiege has quit IRC17:44
jeblairfungi: how's the resolv.conf fix coming? :)17:45
mordredfungi: for the bitbucket ...17:45
mordredfungi: find any occurance of use of or download of and remove it17:45
jeblairi'm not asking in relation to nodepool; i'm asking because we need that fix for all servers17:46
fungijeblair: i'm all set to push that for review when there's a gerrit17:46
*** prad_ has joined #openstack-infra17:46
jeblairfungi: we're getting rid of managed resolv.conf?17:46
fungijeblair: an already merged change sets it via rc.local17:47
jeblairfungi: that was for nodepool17:47
jeblairmordred: right?17:47
fungioh, then we should just remove the notify => Exec['make-resolv-conf-immutable'], line17:47
jeblairfungi: yeah i think so17:47
fungiupdating my prepared patch in that case17:47
*** morganfainberg_Z is now known as morganfainberg17:48
jeblairi'm updating my environment on puppetmaster with that change17:48
jeblair puppet agent --test --puppetdlockfile=/tmp/alt-lock-file --environment=corvus17:49
jeblairrunning that ^17:49
*** arnaud__ has joined #openstack-infra17:50
funginot as easy to test for nodepool image updates since they want to pull the config repo themselves during build17:51
fungihowever bitbucket stopped failing at least17:51
*** darraghb has quit IRC17:51
anteayafungi: because it is up or because you removed
jeblairmordred: okay, puppet has been run; i think you can start gerrit now17:51
mordredjeblair: starting gerit17:52
fungianteaya: bitbucket seems to be returning the file again17:52
mordredStarting Gerrit Code Review: FAILED17:52
anteayafungi: k17:52
jeblairnothing in the error log17:53
*** markwash has joined #openstack-infra17:53
mordredI'm going to run by hand17:53
mordredor, rather17:53
mordredyup. tried again. still errror. still nothing in logs17:54
*** mrmartin has joined #openstack-infra17:54
fungimordred: one possibility... if you updated the all-projects acl config file but not the groups file (which lists the corresponding uuids from the db for groups mentioned in the acl) then it might be unhappy?17:54
anteayais there a shell debugger you can run it in?17:54
mordredfungi: I did not update the all projects acls yet17:55
swestonwhat about starting manually ?  java -jar ... gerrit.war daemon -d ... review_site17:55
fungimordred: okay, thinking harder in that case ;)17:55
mordredthere is no gerrit.war17:55
*** hogepodge has quit IRC17:56
jeblairmordred: there isn't?17:56
zaroohh i see the problem.17:56
anteayaisnt this the war? gerrit-wars/gerrit-v2.
zarodownload location is incorrect17:56
*** hogepodge has joined #openstack-infra17:56
mordredwait - I'm wrong17:56
*** mat-lowery has joined #openstack-infra17:56
jeblair-rw-r--r-- 1 gerrit2 gerrit2 35115174 Apr 28 17:56 /home/gerrit2/review_site/bin/gerrit.war17:56
swestonthere should be a gerrit war file in the gerrit2 home directory .. /home/gerrit2/gerrit-wars/gerrit-version.war17:56
jeblairif you start it manually, be sure to do so as the gerrit2 user17:57
mordredyup. trying that now17:57
*** e0ne has quit IRC17:57
mordredit is silent17:57
mordredis there a way to start it in non-daemon mode?17:57
*** nati_uen_ has joined #openstack-infra17:58
jeblairmordred: ~gerrit2 is owned by root17:58
jeblairmordred: i'm not sure if that's an issue but let's fix that real quick and retry17:58
mordredso it is17:58
*** ramashri has joined #openstack-infra17:59
*** aysyd has quit IRC17:59
mordredstill failed17:59
*** nati_uen_ has quit IRC17:59
mordredbut now we have error logs!17:59
mordredthe db settings are wrong17:59
mordredin gerrit.config17:59
mordredI believe we may have missed a puppet patch18:00
*** sandywalsh has quit IRC18:00
mordredI can fix manually18:00
*** marcoemorais has quit IRC18:00
jeblairmordred: before you fix manually18:00
*** sandywalsh has joined #openstack-infra18:00
*** e0ne has joined #openstack-infra18:00
jeblairmordred: what puppet patch do we need?18:00
*** afazekas has joined #openstack-infra18:00
mordredwe need something that sets the host to not be localhost18:00
*** packet has quit IRC18:00
mordredthe other settings are all correct18:00
jeblairah ok18:00
jeblairmordred: let me see if i can fix that in puppet real quick18:00
swestonyou do not have anything in the /home/gerrit2/review_site/logs directory?18:01
*** packet has joined #openstack-infra18:01
mordredsweston: we do now18:01
mordredjeblair: the correct host is in hiera, btw18:01
*** hashar has joined #openstack-infra18:01
*** nati_ueno has quit IRC18:01
hasharhave a good Gerrit upgrade folks!18:01
swestonmordred: ok, that's good at least we have an error to work off of18:01
mordredhashar: we're getting closer18:01
mordredsweston: it's a mysql config error - we're fixing18:02
*** marcoemorais has joined #openstack-infra18:02
swestonmordred: ok, gotcha18:02
*** nati_ueno has joined #openstack-infra18:03
*** dkliban is now known as dkliban_afk18:03
jeblairso... i changed it in my environment but puppet still didn't touch the file18:04
*** sarob has quit IRC18:06
mordredjeblair: in modules/openstack_project/manifests/gerrit.pp18:06
mordredjeblair: mysql_host is not plumbed through to ::gerrit18:06
jeblairer how did that work on review_dev?18:07
jeblairperhaps it did not18:07
*** bhuvan has joined #openstack-infra18:07
jeblairmordred: okay, fixed18:08
jeblairmordred: clear to proceed18:08
*** arnaud__ has quit IRC18:08
mordredgerrit starting18:08
mordredbut tok much longer this time18:09
*** sdake_ has joined #openstack-infra18:09 Guice provision errors:18:09
jeblair1) No index versions ready; run Reindex18:09
mordredoh - that's the secondary index stuff, right?18:09
zarolooks like it18:09
mordredon it18:10
mordred20% done18:12
*** pblaho has quit IRC18:13
*** thedodd has quit IRC18:13
*** nati_ueno has quit IRC18:13
*** gokrokve has joined #openstack-infra18:14
anteayamordred is wearing the progress bar hat18:15
*** zz_gondoi is now known as gondoi18:15
*** thomasbiege has joined #openstack-infra18:15
*** aysyd has joined #openstack-infra18:16
mordredok. it's up and running18:16
*** zehicle_at_dell has joined #openstack-infra18:17
zaroi'm in.18:17
anteayaso indexing is finished or gerrit is up?18:17
mordredI'm going to run the CLA setting script18:17
mordredgerrit is u18:17
anteayawell done18:17
mordredjeblair: any issues with me moving forward on steps on the manual list?18:17
jeblairanteaya: we're far from finished18:17
* anteaya nods18:17
jeblairmordred: go right ahead18:17
mordredI have three - run clas script, update acls, update dashboards18:17
jeblairmordred: do you think i can run manage-projects while you do that, or should i wait?18:18
jeblair(manage projects needs to run to update all the per-project acls)18:18
mordredjeblair: I see no reason why it can't run18:19
jeblairi will do that then18:19
*** gokrokve has quit IRC18:19
hemnagerrit good to submit reviews ?18:19
*** _nadya_ has joined #openstack-infra18:20
anteayahemna: not yet18:20
hemnaok :)   thanks18:20
anteayahemna: we just achieved one of the steps on teh list18:20
jeblairmanage-projects is using the local mysql db18:20
hemnaok np.  I just happened to hit refresh in my browser and saw it was "up"18:20
anteayayes, so the app is running now we need to fine tune settings18:21
jeblairfungi: do you know how manage-projects gets its database config?18:22
fungijeblair: hmm... good question--looking now18:22
*** thedodd has joined #openstack-infra18:23
jeblairfungi, mordred: i think jeepyb is hardcoded for a local database18:23
jeblairas it, it simply does not use the hostname argument18:23
mordredjeblair: spectacular18:24
jeblairi will start hacking on that18:24
fungiyep, seems to not specify a hostname, but could conceivably scrape that from the config too18:24
*** yfried has quit IRC18:26
*** doddstack has joined #openstack-infra18:26
*** arnaud__ has joined #openstack-infra18:28
*** thedodd has quit IRC18:28
jeblairokay, i have that change staged and ready to go; the original run is still going though and i'm a little afraid to interrupt it18:28
jeblairit has about 140 more projects to go18:28
mordredacl updates updated18:29
*** nati_ueno has joined #openstack-infra18:29
mordredimportant changes dashboard uploaded18:29
*** marcoemorais has quit IRC18:29
jeblairzaro: can you help mordred change the "CR" label to "R" ?18:30
zarojeblair: can't.  there's a bug in gerrit that prevents it.18:30
*** _nadya_ has quit IRC18:30
jeblairoh.  that's unfortunate.18:31
*** marcoemorais has joined #openstack-infra18:31
mordredI am not seeing an Approved state in workflow for things I should have approve on - oh, that's because we need per-project acls run in isn't it?18:31
jeblairmordred: yep.18:31
mordredI DO see work in progress where appropriate18:31
mordredhow do I see dashboards?18:32
anteayamordred:  Projects -> List -> All Projects -> Dashboards18:32
*** yfried has joined #openstack-infra18:33
mordredgreat. important changes works18:33
jeblairzaro: are you sure order doesn't matter on the review labels?18:34
zarojeblair: it's alpha order18:34
jeblairi could have sworn that when gerrit started, the order was V/CR/W18:34
jeblairthen mordred made a change that removed an extra Verified label from the config file (which was before code-review)18:35
jeblairand then the order changed18:35
morganfainbergjeblair, i've see that setting not honored in newer versions of gerrit... it just does whatever it wants18:35
zarojeblair: might have been on old important changes dashboard only18:35
jeblairno, was on the main screen18:35
morganfainbergjeblair, it seems to make it alphabetical?18:35
jeblairmorganfainberg: ya, that's what zaro just said.  i'm attempting to dispute that. :)18:35
zaromordred: steps to create dashboards is in the gerrit upgrade script18:36
jeblairsadly, i lost my evidence in a reload18:36
morganfainbergjeblair, no i've seen exactly what you saw.18:36
mordredjeblair, zaro: I have tested just now by moving the code review block in the all projects file - and now CR is second18:36
morganfainbergjeblair, it was "correct" then switched to alpha after gerrit was running? logged in? something18:36
zarojeblair: i've already verified that it's always alpha order.18:36
*** ildikov has joined #openstack-infra18:36
mordredzaro: I have just verified the opposite :)18:37
jeblairmorganfainberg: oh, we agree then -- the change wasn't logging in -- it was actually mordred making a config change18:37
morganfainbergjeblair, hm. odd. i'd love to know what setting that was.18:37
jeblairmorganfainberg: and he just changed it again so it should be back to the order we are accustomed to18:37
jeblairmorganfainberg: it's simply the order listed in the config file18:37
mordredmorganfainberg: it's the order of the sections in projects.config18:37
morganfainbergjeblair, oh good it's honoring that setting then18:37
morganfainbergjeblair, in .. 2.6 i think it just went alpha... or changed18:37
zaromordred: what?  i'm must be bonkers then.18:37
* mordred hands zaro a wet cat18:37
morganfainbergwait, mordred , you changed it back?18:38
jeblairmordred: i see cr/v/w now18:38
morganfainbergi still see it in alphabetical18:38
*** markmcclain has joined #openstack-infra18:38
mordredhrm. I see v/cr/w18:38
morganfainbergmordred, cr/v/w here.18:38
mordredhave you logged out and logged back in?18:38
anteayaeverybody refresh, no config changes18:38
fungiyep, lgtm now18:38
morganfainbergmordred, yep, just did18:38
zaroohhh now i remember.  i was testing on master, not 2.818:38
jeblair(i haven't logged in at all)18:39
*** nati_ueno has quit IRC18:39
mordredwhy are we seeing different things?18:39
mordredmorganfainberg: where are you seeing yours?18:39
morganfainberglet me try incognito18:39
morganfainberglogged in18:39
mordredwe're talking about different things18:39
mordredyou're talking about the status grid18:39
morganfainbergjeblair, yep18:39
mordredI'm taling about the order in which the review vote blocks are displayed18:39
mordredbecause, of course, those shoudl sort differently18:40
morganfainbergmordred, yep*18:40
morganfainbergmordred, ah yeah18:40
jeblairoh good grief18:40
mordredthe vote block order is based on order of block in config file18:40
morganfainbergwe might want to make those match...18:40
zaroi'm seeing cr/v/w18:40
mordredthe matrix order is alpha18:40
mordredyou'd think with all of those class factory factory class injection class factories there would be, you know, a sort method18:40
morganfainbergmordred, wait so i should see V/CR/W on a specific review?18:40
morganfainbergmordred, because... i see it alphabetical there too18:41
mordredif you click review18:41
zarowhat do we want to change order on?  i thought it was the columns on the dashboards?18:41
mordredzaro: it is - I was talking about something different, because I like to make things complex18:41
*** dkliban_afk is now known as dkliban18:42
morganfainbergmordred, ok so where should i see it V/CR/W? in the place where you sleect the score?18:42
zarook, good to know i'm not bonkers after all.18:42
morganfainbergzaro, nah we're all just talking about many different things :)18:42
morganfainbergzaro, hehe :)18:43
*** alexpilotti has joined #openstack-infra18:43
mordredmorganfainberg: you probably cannot see it at all- when I'm logged in as a project bootstrapper, it shows me V/CR/W as the order of the vote blocks18:43
zaroyeah, columns are always alpha order.  everybody agree?18:43
mordredzaro: yes18:43
morganfainbergmordred ah ok18:43
jaypipeswhat's this new "Workflow" column about? :)18:43
morganfainbergzaro, ++ yeah18:43
fungiyeah, when looking at the main view for a change i see the vote table with columns in alpha order18:43
mordredmorganfainberg: and I verified that changing the block order in the config file DOES affect it18:43
jeblairall right, i don't think the order particularly matters for the review box, but let's set it to what we want it to be anyway: v/r/w18:43
swestonzaro: ++ yes, verified over here as well18:44
morganfainbergmordred, cool. yeah that makes sense.18:44
jeblair(it doesn't matter because no one sees verified)18:44
mordredjaypipes: it's where Approved or Work in Progress go now18:44
mordredjeblair: ++18:44
mordredzaro: it would be neat if the boxes in the matrix matched the order from the config file, tbh18:44
jeblair(btw, acl updates have been running for a few mins)18:44
mordredjeblair: I'm STILL adding CLA users18:44
zaromordred: boxes in the matrix?18:45
zaromordred: what is that?18:45
mordredzaro: the thing you guys were talking about when I was talking about the wrong thing18:46
*** nati_ueno has joined #openstack-infra18:46
*** yfried has quit IRC18:46
zaromordred: ohh yeah, agreed.18:46
anteayajeblair: is line 27 done or still in progress?
jeblairanteaya: just updated18:46
anteayahere is fungi's gpg host key fingerprint announcement for review:
jeblairi'm going to stop zuul, run puppet to get the config change, then start it again18:47
morganfainberganteaya, that reads well and is straightforward.18:49
anteayamorganfainberg: thanks for looking18:49
dimsnew key looks good (verified using "ssh -p 29418 gerrit ls-projects")18:50
*** _nadya_ has joined #openstack-infra18:50
jeblairzuul is running18:51
anteayadims: thanks for verifying18:51
jeblairmanage-projects is finished18:51
*** nati_ueno has quit IRC18:51
jeblairspot checking project acls under looks good18:52
jeblairmordred: how's the cla thing coming?18:52
dimsanteaya, bonus for me i can get to it using ipv6 as well18:52
mordredjeblair: still running18:53
*** ativelkov has joined #openstack-infra18:53
fungianteaya: i updated the known_hosts entry removal command in the wiki since it needs the port number (based on my limited testing)18:54
anteayafungi: thanks18:54
*** msabramo has quit IRC18:54
jeblairi don't think zuul works18:54
fungianteaya: fwiw that's also roughly the same syntax openssh is recommending at me when i test connecting18:54
jeblair2014-04-28 18:53:48,506 DEBUG zuul.DependentPipelineManager: Change <Change 0x31fcfd0 73469,2> can not merge, ignoring18:54
jeblair2014-04-28 18:53:48,506 DEBUG zuul.DependentPipelineManager: Change <Change 0x31fcfd0 73469,2> is not ready to be enqueued, ignoring18:54
anteayafungi: great, bonus for consistentcy18:55
jeblairit has no workflow +1 vote18:55
jeblairthat sort of makes sense18:55
jeblairfungi: can you add approved votes to the following changes?18:56
jeblairfungi: 73469 86542 90660 90514 89277 89240 90086 90089 67657 83028 89484 86497 90380 84648 9039618:56
fungijeblair: on it now18:56
jeblairapparently the mergers are all failing too, i'm looking at that18:58
jeblairoh, heh18:58
jeblairthey need their host keys updated :)18:59
jeblairtheir known_hosts, that is18:59
*** praneshp has quit IRC18:59
mordredjeblair: want help with that?18:59
jeblairmordred: i got it18:59
jeblairmordred: i wonder if anything else will..3.18:59
jeblairmordred: yep19:01
jeblairsince it runs on review.o.o, probably not19:01
jeblair(we still need to fix that)19:01
mordredjeblair: it runs as gerrit user?19:02
mordredgerritbot user19:02
mordredI'm just going to go check19:02
fungiit may need it, yes19:02
mordredno - gerrit2 user19:03
jeblairit's on a new host, so it should not have a known_hosts file19:03
fungialso, approvals added to the list of changes from the previous gate queues19:03
*** praneshp has joined #openstack-infra19:03
*** ativelkov is now known as ativelkov_away19:03
fungijeblair: oh, good point. it won't need the old key removed, but may need the new one accepted19:03
mordredjust did the acceptance19:03
*** ativelkov_away is now known as ativelkov19:03
jeblairmordred: would you like to stop it on old review and start it on new review?19:03
mordredjeblair: yes I would19:04
*** dkehn__ is now known as dkehnx19:04
*** openstackgerrit has quit IRC19:04
dansmithis new gerrit expected to be faster, or is it just experiencing low load after being down for a bit?19:04
zaromordred: i don't have access to dashboards?19:04
mordreddansmith: possibly both19:04
zaromordred: did you already create the imporant changes dashboard?19:05
mordreddansmith: it's on a better server19:05
dansmithif it's faster, I might stop bitching about this +W business :)19:05
*** changbl has quit IRC19:05
mordredzaro: I did19:05
*** changbl has joined #openstack-infra19:05
zaromordred: must be an acl setting.19:05
mordreddansmith: until you said that - I totally did not realize we were changing a communication semantic19:05
dansmithmordred: yeah, it's pretty unfortunate actually19:05
mordredI'm still going to say +A btw19:05
dansmithmordred: +A is in our vernacular19:06
*** alexpilotti_ has joined #openstack-infra19:06
mordredjeblair: I started gerritbot but he hasn't joined here19:06
mordredjeblair: IOError: [Errno 2] No such file or directory: '/home/gerrit2/.ssh/gerritbot_rsa'19:06
jeblairthe value is still "approved" even though the column is "workflow".  +A still makes sense to me19:07
fungimust not have puppeted that account's keys19:07
*** alexpilotti has quit IRC19:07
*** alexpilotti_ is now known as alexpilotti19:07
mordredin fact - we could say +A and -W now19:07
jeblairzuul host keys updated19:08
fungii'll get started inserting the gerritbot rsa key into hiera19:08
jeblairand it's running tests19:08
jeblairfungi: thx19:08
jeblairi'm going to re-enqueue now19:08
*** nati_ueno has joined #openstack-infra19:08
anteayamordred: is the cla thing finished?19:08
*** ativelkov is now known as ativelkov_away19:09
mordredfungi: yup. gerritbot ssh key not in hiera19:09
zaromordred: ohh i found the dashboard.  had to click on 'all-projects' first19:09
* mordred putting in hiera - patch coming19:09
jeblairi've restored the gate jobs19:09
fungimordred: okay, all yours then19:09
*** bookwar has left #openstack-infra19:09
jeblairi'm not entirely sure i want to restore the check jobs right now19:09
anteayaI confirm the top 3 gate jobs are the same as when we stopped19:09
anteayaI'm guessing the rest are good too19:10
fungiStarting gate jobs.
jeblairit looks like the commentlink stuff that makes job results pretty is broken19:10
anteayayeah, there were 175 in check when we stopped19:10
*** sdake_ has quit IRC19:11
dimsfolks, clicking on "HTTP Password" i see a HTTP 404 in firebug (
jeblairdims: works for me19:11
jeblairdims: where did your url come from?19:12
fungidims: neat. we don't use http passwords for gerrit, so should probably figure out how to disable that19:12
jeblairfungi: we do now -- they are required for rest api usage19:12
dimsjeblair, click on the page with firebug enabled, you will see it19:13
* fungi stands corrected19:13
*** esker has quit IRC19:13
anteayafungi: I didn't know that either19:13
dimsthe page itself loads fine19:13
fungialso, the generate password and clear password buttons are working for me at least19:13
*** esker has joined #openstack-infra19:13
jeblairdims: oh neat, it's an ajax thing19:14
jeblairdims: that works for me19:14
jeblairdims: do you have a 'generate password' or 'clear password' button?19:14
*** gokrokve has joined #openstack-infra19:14
anteayadims: I don't know if the cla script has finished yet, what happens if you log out and back in again?19:14
dimsi do see a "generate password"19:14
jeblairdims: click that?19:15
*** nati_uen_ has joined #openstack-infra19:15
jeblairdims: i don't think you get one automatically19:15
jeblairdims: (i think it works more like an api key)19:15
YorikSardims: You get 404 because you have no password. It's REST19:15
mriedemare the previously marked WIP patches supposed to still show up under "my changes"?19:15
dimsjeblair, PUT / DELETE works fine19:15
mriedem* my > changes19:15
dimsYorikSar, cool that would do it19:15
anteayamriedem: they should be workflow -119:16
anteayaso under W you should see a red X19:16
fungioh, did we miss a step to update changes which were formerly wip to workflow=-1?19:16
jeblairanteaya: i do not believe there was any kind of data migration there19:16
anteayajeblair: ohhh19:16
jeblairanteaya: so all previously wip changes should now just be normal changes19:16
mriedemanteaya: not seeing them, only if i mark them as workflow=-1 after the change19:16
anteayamriedem: I'm wrong19:16
jeblairfungi: i bet we could pull that list out of the old db19:16
jeblairfungi: want to do that?19:17
jeblairmordred: cla status?19:17
mordredjeblair: still running19:17
jeblairmordred: eta?19:17
fungijeblair: yep, working on it now19:17
anteayawow that takes a while19:17
*** msabramo has joined #openstack-infra19:17
*** nati_ueno has quit IRC19:17
fungialso, the puppet fix is pushed up as
anteayafwiw that doesn't resolve for me19:18
anteayathe dns must not be updated around me yet19:18
mordredjeblair, fungi:
fungianteaya: you may need to stop and relaunch your browser session19:19
*** gokrokve has quit IRC19:19
jeblairzaro: the diff pages look VERY different on review and review-dev19:19
anteayafungi: I can try that, or I might clear my cache first19:19
mordredjeblair: just did 11113 out of 1128119:20
*** ramashri has quit IRC19:20
mordredjeblair: vim tells me 98%19:20
anteayacleared cache, no difference19:20
zarojeblair: possible there was some change to it on ver 2.8.419:21
zarojeblair: review-dev is running 2.8.319:21
mordredjeblair: CLA jobs done19:21
mordredjeblair: there were some users it complained on19:21
jeblairoh, i think it has something to do with wide diffs19:21
jeblairzaro: nm for now19:21
*** bauzas has joined #openstack-infra19:22
anteayafungi: yeah, that did it19:22
jeblairfungi, mordred: think we can sound the all-clear for now?  then i can grab some food and we can continue to clean things up?19:23
fungijeblair: sure. that's probably a good time for me to e-mail out the host key change announcement too19:23
mordredjeblair: yes. I think so - although if you could +2 this:
anteayahuh, I have one WIP patch that doesn't show up on my changes page at all19:24
mordredI can get gerritbot back up19:24
anteaya"create a seafile instance"19:24
anteayawas sure it was WIP19:24
mriedemanteaya: yeah, same, sounds like fungi is working on that19:24
jeblairmordred: done19:24
fungianteaya: yeah, i'm working on digging that out of the old db19:24
anteayak, sorry19:25
harlowjahmmm, whats up with the fonts :-P19:25
anteayathought they would still be in changes, just normal patches19:25
anteayaharlowja: can you be more specific?19:25
harlowjaseems like commit messages have different font then the rest of the page now :)19:25
harlowjaguess thats a feature :-P19:25
mordredharlowja: they're editable in browser now too - as are the topic19:26
fungii find in-browser commit message editing slightly weird19:26
harlowjahmmm, new-screen view also seems new19:27
*** ramashri has joined #openstack-infra19:27
mordredjeblair: thanks19:27
* afazekas I got 500 from js, (not important to fix) it was WIP change19:27
*** e0ne has quit IRC19:28
*** mrmartin has quit IRC19:28
morganfainbergediting commit messages in the browser... thats... kinda cool19:28
mordredjeblair: hrm - having the comment report: "Patch Set 1: Code-Review+2 Workflow+1" will take a little getting used to19:28
jeblairok Gerrit upgradet to 2.8 complete.  See: Some cleanup tasks still ongoing; join #openstack-infra if you have any questions.19:28
dkranzfungi: I see links in jenkins comments now show full url (2 lines per entry instead of 1). Is that expected?19:28
mordredjeblair: ++19:28
harlowjado u guys know if the ssh stream that gerrit emits is changed much by this, might have to update my tool :-P19:28
jeblairmordred, fungi, anteaya: ^ look good ?19:28
jeblairharlowja: not much; zuul still works19:29
anteayajeblair: yes19:29
*** mrmartin has joined #openstack-infra19:29
jeblair#status ok Gerrit upgrade to 2.8 complete.  See:  Some cleanup tasks still ongoing; join #openstack-infra if you have any questions.19:29
openstackstatusjeblair: sending ok19:29
fungijeblair: lgtm19:29
anteayaharlowja: stream-events now needs express permission, it isn't available to registered users anymore19:30
fungidkranz: it needs some work. something about the commentlinks configuration syntax probably changed19:30
jeblairanteaya: yes it is19:30
anteayais it?19:30
anteayaI'm wrong19:30
YorikSarjeblair: Will it be a good time now to ask to add one line to project.config?19:30
jeblairYorikSar: no19:30
fungianteaya: it isn't by default on a plain gerrit install, so we added it to the default in our config19:31
*** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs | Bugs | Code"19:31
-openstackstatus- NOTICE: Gerrit upgrade to 2.8 complete. See: Some cleanup tasks still ongoing; join #openstack-infra if you have any questions.19:31
bnemecHmm, getting an Internal Server Error trying to look at
anteayaI connected those dots, didn't want to add to noise atm though19:31
jeblairYorikSar: we still have quite a bit of work to do; you're asking for a behavioral change.  that will require quite a bit of discussion which would be distracting now.19:31
anteayabnemec: can you restart your browser and try again?19:32
jeblairanteaya: does bnemec's change work for you?19:32
*** gabriel-bezerra has joined #openstack-infra19:32
anteayaah no it doesn't19:32
morganfainbergjeblair, i get ISE on bnemec's change19:32
bnemecI get the same thing from afazekas's link above.19:32
anteayasorry, I was dispensing wrong advice19:32
*** ashishsnigam has joined #openstack-infra19:32
YorikSarjeblair: Do those changes get applied without Gerrit restart?19:32
jeblairYorikSar: what changes?19:33
openstackstatusjeblair: finished sending ok19:33
bknudson503 Service Temporarily Unavailable19:33
jeblairfungi, mordred: we need to accept replication host keys for the gerrit user19:33
YorikSarjeblair: I mean changes to project.config that affect labels behavior.19:33
anteayabnemec: which url is afazekas's link?19:33
mordredjeblair: k. I can do that19:33
jeblairYorikSar: please let's discuss this later.19:33
anteayabknudson: have you restarted your browser?19:33
YorikSarjeblair: Ok. Sure.19:33
fungimordred: jeblair: and then re-trigger replication afterward19:34
bnemecanteaya: I wonder if it's a WIP thing.19:34
bknudsonanteaya: nope, will do.19:34
mordredjeblair: next time you get a sec - I could use help with debugging gerritbot19:34
anteayabnemec: is your patch a WIP?19:34
jeblairmordred: i think i need to get food right now19:34
psedlakit seems WIP changes are lost19:34
bnemecanteaya: Not sure, it isn't my patch.19:34
anteayabnemec: if yes, then yes, WIP patches were not carried over in the migration19:34
mordredit seems unhappy with irc19:34
afazekasanteaya: I guess the WIP changes has an upgrade issue19:34
anteayapsedlak: yes19:34
mordredjeblair: ok - I'll keep poking - get food19:34
anteayafungi is retrieving WIP patches from old gerrit19:34
*** sdake_ has joined #openstack-infra19:34
jeblairafazekas, bnemec: oh were the erroring changes marked wip before?19:35
anteayasince the WIP behaviour is changed, we didn't know our migration wouldn't bring them across19:35
anteayafungi is getting them19:35
jeblairi wonder if the problem is that they still have that state in the db which is unknown to gerrit19:35
fungianteaya: i am in the process of retrieving the list of which changes were marked wip so we can add a workflow -1 on them19:35
anteayafungi: thanks19:35
fungianteaya: sounds like the issue may run deeper than that though19:35
* anteaya stands by19:35
bnemecYeah, probably WIP.  I'll try again once those have been brought over.19:35
afazekasjeblair: y19:35
anteayabnemec: thanks19:36
morganfainbergbnemec, yeah i noticed a bunch of reviews seem to be gone from Keystone as well.  Likely WIP ones19:36
jeblairfungi: okay, so we probably need to manually update the status on those to 'new' in the new db.19:36
bknudsonwhat's the status field for?19:36
anteayabknudson: what are you looking at?19:36
*** eharney has quit IRC19:36
*** ChanServ sets mode: +o jeblair19:37
fungijeblair: interesting. i'll figure out what you mean by that once i poke around in the new db schema, i'm sure19:37
boris-42don't touch WIP button lol19:37
anteayabknudson: right now I don't know19:37
jeblairfungi: zaro or mordred may be able to explain19:37
anteayastatus is open, merged, abandoned19:37
anteayabknudson: in the url status is open19:38
bknudsonok, abandoned and merged show up on those views19:38
*** jeblair changes topic to "Known problem with WIP changes. Gerrit upgraded to 2.8. See:"19:38
zarofungi: the WIP feature added a wip status.  that's no longer a valid status in the db19:38
hasharcongratulations folks!19:38
*** nati_uen_ has quit IRC19:38
anteayahashar: thanks19:38
* jeblair will grab food quickly and return19:38
anteayajeblair: /nod19:39
anteayastill some small fires19:39
mordredok. replication kicked19:39
hasharI know how long you spent preparing that upgrade. Seems it went fine (at least I can see my dashboard)19:39
anteayahashar: a happy customer19:39
anteayathank you19:39
mordredhashar: thanks! big ticket items done at least- still some cleanup to go19:39
anteayazaro spent the most time on the upgrade19:39
fungizaro: okay, so i may just be able to add a wip vote for any changes in an (invalid) wip status and reset the status to new at the same time19:39
bknudsonI think I had a few WIP patches. No big deal if they're gone19:39
anteayabknudson: k, we are working on getting them back19:40
zarofungi: ++19:40
fungibknudson: looks like they're still in the db, just need to be adjusted19:40
hasharmordred: you will love 2.8 :-)19:40
*** nati_uen_ has joined #openstack-infra19:40
*** thomasbiege has quit IRC19:40
*** chandan_kumar has quit IRC19:40
bknudson -- says Internal server error19:40
anteayabknudson: that might be a WIP patch url19:41
fungibknudson: yep19:41
anteayado you know?19:41
*** matrohon has quit IRC19:41
*** pelix has joined #openstack-infra19:42
bknudsonanteaya: I think it might depend on a wip?19:43
anteayabknudson: that might explain it19:43
bknudsonanteaya: and I don't see that other review in my list19:43
*** ramashri has quit IRC19:43
anteayabknudson: that other review?19:43
anteayathe wip review19:44
*** openstackgerrit has joined #openstack-infra19:44
YorikSarThat link works with new change screen.19:44
YorikSarFails for Old screen.19:44
*** jamespd has joined #openstack-infra19:44
*** openstackgerrit has quit IRC19:44
bknudsonanteaya: I stuck another review in front of it over the weekend... maybe I can find the link19:44
YorikSar(new change screen is cool though)19:45
anteayabknudson: well if there is a chain with a wip in there, might better make a cup of tea and let fungi move those wip patches over19:45
YorikSarbknudson: @Fix tests to use strings rather than ints for IDs" - thix one?19:45
bknudsonI'm not in a hurry19:45
anteayabknudson: since this all might be different colours of the same issue, missing wip patches19:45
bknudsonYorikSar: that's the one I'm thinking of19:45
anteayabknudson: k, thanks19:45
YorikSarFails even in new change screen...19:46
*** ociuhandu has quit IRC19:47
mordredjeblair: for when you get back - gerritbot seems quite happy talking to new gerrit - but is having problems talking to IRC19:47
bknudsonI've got the changes locally so no big deal if they're gone.19:48
anteayabknudson: k19:48
anteayamordred: did add CLA users ever finish?
YorikSarbknudson: They are definitelly not gone - they are accessible through Git.19:50
*** lcostantino has quit IRC19:51
*** nati_uen_ has quit IRC19:51
fungiand still in the db19:51
* anteaya nods19:51
anteayago you19:52
*** _nadya_ has quit IRC19:52
mordredfungi, jeblair:
mordredthat encompasses the changes jeblair made on the filesystem19:53
mordredwell, that and fungi's change19:53
*** dprince has quit IRC19:53
*** mrmartin has quit IRC19:53
*** sweston has quit IRC19:54
*** nati_ueno has joined #openstack-infra19:55
*** mrmartin has joined #openstack-infra19:55
fungiokay, we're in luck. changes with status="w" are all also open="Y" so i'm making a list (so that i'll know which ones to add a wip vote on) and then updating them to status="n"19:56
anteayafungi: yay19:56
*** sarob has joined #openstack-infra19:56
anteayayou can add a wip vote on them via ssh, yes?19:56
anteayaor via the db19:56
fungiif i give myself permission to do so, yes via ssh19:56
fungivoting via the db is more complicated since it would involve touching multiple tables, i think19:57
mordredthe important changes screen is not showing the right things for previously reviewed19:57
anteayafungi: let's do the less complicated thing19:57
anteayamordred: what it is missing or adding?19:58
anteayamordred: and is add CLA users finished running?20:00
mordredanteaya: it's showing me things that I have not reviewed before, nor am subscribed to20:00
YorikSarfungi: You can also suexec to the owner of a change to vote for it.20:00
anteayamordred: that's odd20:00
*** atiwari has joined #openstack-infra20:00
*** annegentle has quit IRC20:01
*** matty_dubs has joined #openstack-infra20:01
anteayayeah me too, I have under important-changes, changes needing review20:02
anteayanever seen it before20:02
anteayaoh hang on there's my name, I have so20:02
fungiYorikSar: yeah, that involves authenticating to gerrit with its own host key and the syntax on suexec is a little messy20:02
anteayaI did a recheck no bug20:02
anteayano, changes needing review in my important changes dashboard all have my name on them20:04
YorikSarfungi: I thought you only need "Run As" right for that...20:04
jogooh neat edit commit message20:05
fungiYorikSar: oh, maybe. in earlier versions you had to use the special "gerrit code review" account, but i'm reading the new api docs now20:06
*** jgrimm has quit IRC20:06
fungiYorikSar: indeed you are correct... "suexec can only be invoked by the magic user Gerrit Code Review, or any user granted granted the Run As capability."20:06
*** nati_ueno has quit IRC20:06
anteayabut reviewed changes all have the most recent timestamp, irrespective if I have seen them before or not20:06
fungiYorikSar: though also seems to require additional configuration20:07
jeblairfungi: if you want to grant that to project bootstrappers i'd be okay with that20:07
YorikSarfungi: I guess superadmins (infra-core) can have "Run As" right just in case.20:07
fungi"The run as capability is permitted to be used only if auth.enableRunAs is true."
anteayaand I can understand the error, since when clarkb was testing the dashboard the list of recent changes was small and he had probably reviewed them all20:07
fungiso maybe another time20:07
*** denis_makogon has quit IRC20:09
fungitesting now to see if a workflow -1 vote i leave while a project bootstrapper on a change i wouldn't otherwise control in that regard sticks after removing myself from the group, and it seems to20:10
fungiafazekas: can you test un-wip'ing now that i've set it to workflow -1?20:11
anteayamordred: at one point this was the query clarkb pushed for important changes:
fungii just want to make sure that i'm not about to leave a bunch of wip votes which only admins can remove ;)20:11
anteayamordred: I am not sure if there was editing or not20:11
*** jreypo has joined #openstack-infra20:11
zaroline number on right side on change screen is annoying :(20:11
zaroopps i mean diff screeen20:12
anteayayes, it is a small change though20:12
anteayawe can get used to it20:12
jeblairanteaya: speak for yourself20:13
anteayaokay, it would be something I would just say meh to20:13
zaroon master version of gerrit, it got put back to left side.20:14
*** gokrokve has joined #openstack-infra20:14
*** adalbas has quit IRC20:14
mordredanteaya: ok. I've updated the important changes dashboard with that20:14
jeblairmordred: what did you change?20:14
jeblairmordred: also, still need me to look at gerritbot?20:15
YorikSarfungi: You can test on one of my changes. E.g.
anteayamordred: no no, I had said that was one of the versions clark had suggested20:15
mordredjeblair: yes please -20:15
mordredanteaya: well, it did not help :)20:15
anteayaI wasn't saying that was the fix20:15
* mordred will go dig in to review-dev and see what's there20:15
anteayamy comment was that reviewed changes just show the most recent changes in gerrit20:15
mordredjeblair: the debug log says things about joining channels, but I do not see that happening20:16
fungiYorikSar: done. are you able to un-wip that now?20:16
anteayaand that clarkb probably made a logical mistake since he was working with review-dev, not many changes on it20:16
*** gokrokve_ has joined #openstack-infra20:16
jeblairzaro: the trivial rebase hook is running, was that supposed to be disabled?20:16
YorikSarfungi: Nope20:17
zarojeblair: yes, i believe so.20:17
YorikSarfungi: Every user have one's own labels.20:17
* psedlak is not happy about clicking the project name or that 'cool-fluffy-icon' thing next to it while looking at single change ... bringing me to the wierd web-form or to the view with all open, abandoned and merged changes ... really ... no direct link to open changes ... ok bookmarking wins20:17
fungiYorikSar: that's an interesting drawback to the new workflow design20:17
YorikSarfungi: So only you can remove your -1.20:17
YorikSarfungi: Well... Only owner is supposed to wip and un-wip.20:17
fungiYorikSar: we granted core reviewers wip control with this too20:18
fungiYorikSar: we might want to re-think that20:18
YorikSarfungi: Ah... Yeah.20:18
jeblairfungi: users can upload a new patchset to clear wip reviews20:18
fungijeblair: ahh, fair enough!20:18
*** ramashri has joined #openstack-infra20:18
jeblairfungi: that's an intentional balance to the design of allowing core to -1 wip20:19
*** gokrokve has quit IRC20:19
fungijeblair: which then makes a good argument for not having work-in-progress convey to subsequent patchets20:19
YorikSarjeblair, fungi: the feature users want to get rid of for a long time :)20:19
YorikSarI guess we'll have to patch git-review then.20:19
afazekasfungi: unwip - ed, it still gets a red X20:20
jeblairfungi: if you want to enable runas, we can restart gerrit for that20:20
fungiafazekas: yep, thanks20:20
*** gokrokve_ has quit IRC20:20
zarojeblair: just checked again, trival rebase disabled in puppet looks ok to me.20:20
anteayamordred: try this:
fungijeblair: probably worth doing. i'll draft up the config change for it20:21
zarojeblair: and it's not running on review-dev so puppet seems to not be running it there.20:21
psedlakafazekas: red X where? after unWIPing?20:21
fungipsedlak: if someone else marks your patchset as a work in progress, you can't clear it without uploading a new patchset20:21
fungipsedlak: though only the change owner or a core reviewer for the target project/branch can do that, so shouldn't be a huge deal20:22
*** lcostantino has joined #openstack-infra20:22
anteayajeblair: I don't know the status of add CLA users20:23
anteayajeblair: I am guessing it is finished but don't know for sure20:23
jeblairanteaya: check scrollback, mordred said it's done.20:23
psedlakfungi: thx, yeah while cores should not be doing it, it's better to not let them do it at all20:23
mordredjeblair, anteaya: ok. important changes dashboard should be fixed now20:24
anteayajeblair: sorry, thanks, I will update the etherpad20:24
psedlakfungi: i mean only the owner should have the right to use W=-120:24
anteayamordred: did you use my patch?20:24
jeblairpsedlak: no, cores should have it.  cores know when a change absolutely needs more work and no other cores should waste time reviewing it.  that's what it's for.20:24
mordredNOT label:code-review=0,self does not work when there are projects that you don't watch (Which is what we missed on review-dev)20:24
fungipsedlak: well, you can also clear it with a new patchset (rebase, et cetera) or ask them to remove their workflow -1 vote20:24
*** nati_ueno has joined #openstack-infra20:25
mordredanteaya: oh - I did not see that before I made mine ... I think it still might miss something, although yours would certainly be better20:25
mordredanteaya: I wound up just doing this: status:open (label:code-review=-1,self OR label:code-review=-2,self OR label:code-review=+1,self OR label:code-review=+2,self) limit:5020:25
anteayareplication kicked? I guess I didn't know what that meant20:25
mordredanteaya: I told replication to run again20:25
psedlakjeblair: well i think think WIP should be more personal, cores can use -1/2s for that anyway20:26
dansmithare things actually flusing from the check queue?20:26
jeblairpsedlak: let's discuss this some other time20:26
dansmithlooks like several are done, but aren't voting and dropping out20:26
jeblairdansmith: ack -- unknown problem.20:26
*** adalbas has joined #openstack-infra20:26
jeblairdansmith: thanks for reporting20:26
dansmithjeblair: okay20:26
*** mwagner_lap has quit IRC20:27
dansmithjeblair: did you mean you hadn't noticed, or you don't know why yet?20:27
psedlakfungi: but that's twisting the usage - if you want to clear someone else wip, just do bull.. rebase/touch-the-code-twice review upload ... that does not sounds ok20:27
anteayamordred: okay, well if your important changes shows you want you are looking for20:27
jeblairdansmith: heh, sorry.  i had not noticed; you are the first person to report.20:27
psedlakjeblair: sure, it's not a big issue20:27
dansmiththe growing backlog seems like maybe it indicates that the first thing to finish has deadlocked zuul or something20:28
dansmithlike, if the vote failed or something20:29
mordredjeblair: we don't seem to be replicating to github - i'm investigating20:31
jeblairdansmith: something just changed20:31
mordredjeblair: I thinkw e need to restart gerrit20:32
mordredjeblair: it seems to cache ssh host keys and flush-caches does not fix it20:32
jeblairmordred: that's correct20:33
mordredit's re-trying ssh-ing to git* in a very tight loop right now20:33
fungimordred: jeblair: if we're going to restart gerrit, let's add (either force submit or hand-patch it on the server)20:33
jeblairfungi: have the runas change ready so it can go in with mordred's restart?20:33
fungijeblair: that's the one20:33
mordredwe're going to have to hand apply20:33
mordredgerrit isn't replicating, so we're not pulling new changes into puppet20:34
mordredhand applying now20:34
mordredjeblair, fungi: I'm ready to restart now - you guys good with that?20:35
*** bookwar has joined #openstack-infra20:35
jeblairmordred: yep20:35
jeblairzuul is getting a lot of errors from gerrit because of this:20:36
jeblairfatal: "--force-message" is not a valid option20:36
fungimordred: ye[20:36
jeblairhashar: have you run into that ^ ?20:36
hasharoh men20:36
hasharyeah that got removed from gerrit review command20:36
hasharsorry should have thought about it20:36
lifelessreview.o.o is 503ing right now20:36
jeblairlifeless: topic20:36
fungilifeless: it's undergoing another restart20:36
lifelessthat might be relatde to zuul having troubl20:36
lifelessjeblair: ack20:37
hasharjeblair: basically force-message is not needed any more. You can gerrit review "some message" on a closed change.20:37
fungilifeless: cause<-effect ;)20:37
lifelesstopic stops at GerritUpg for me, unless I explicitly check.20:37
jeblairhashar: what about reviewing old patchsets?20:37
lifelessjeblair: home page btw, not WIP changes... or I wouldn't hav ementioned it.20:37
hasharjeblair: same that should work out of the box. Let me find my layout.yaml tweak :)20:37
*** sweston has joined #openstack-infra20:38
mordredsigh. restarting once more - missed a hostkey that gerrit didn't have yet20:38
hasharjeblair: workaround is to pass verified: 0,unified20:38
jeblairlifeless: oh, yeah, that's the restart then.  sorry.20:38
mordred(much quicker that time)20:38
hasharjeblair: I think that is Zuul code requiring an action when sending a message. Or to say it otherwise, Zuul dont let you simply:  gerrit review 1234,1 --message "hello world"20:39
hasharjeblair: it requires an action like verified / code-review20:39
jeblairhashar: i was thinking about when it votes verified=-1 on an old patchset, does that still work without force-message?20:39
mordredsomethign tells me this is a lie: [2014-04-28 20:39:23,961] INFO  com.googlesource.gerrit.plugins.replication.ReplicationQueue : Created remote repository:
jeblairmordred: that would be cool :)20:40
*** dkliban is now known as dkliban_afk20:40
hasharjeblair: the verified:0 vote is discard by Gerrit since the job is closed, but Gerrit 2.8 will record the message as a comment20:40
*** julim has quit IRC20:40
swestonanteaya: jeblair: mordred: fungi:  I had to step out for a few meetings .. I see you all are still at it?  Anything I can do to help?  Please let me know :-)20:40
jeblairhashar: yeah, verified=0 is fine, i'm concerned with verified=-1 on a previous patchset20:40
*** bhuvan has quit IRC20:40
hasharjeblair: isn't --force-message used on closed changes?20:41
hasharjeblair: once closed, the verify vote is ignored. So the verify-1 should stick20:41
*** jgrimm has joined #openstack-infra20:41
jeblairhashar: oh deal20:41
jeblairoh dear, i mean20:41
hasharyou might want to try it out with your use cases though20:41
jeblairhashar: if we have to do verified=0, we can't have experimental queues anymore20:41
jeblairthey _depend_ on leaving no votes whatsoever20:42
*** bhuvan has joined #openstack-infra20:42
hasharso just a message20:42
hasharwhich Gerrit let you do, but Zuul reject due to some logic madness20:42
* hashar digs20:43
jeblairhashar: yeah, it's so that there's _some_ action there20:44
hasharjeblair: yeah that is in zuul/reporter/  will returns early if it is missing a params20:44
jeblairhashar: i guess we'll have to patch zuul to no longer require that20:44
hashar(the function takes params and message)20:44
hasharso if you pass it only a message (and no params), the message is never sent20:44
hasharsorry should have left a note on the ether pad :( completely forgot about that one20:45
hasharthe pity is that I fixed it only a month and a half ago20:45
jeblairhashar: do you mean you fixed zuul?20:45
jeblairhashar: or you changed your layout.yaml?20:45
hasharna i fixed the layout by passing verified: 020:45
hasharthis way there is a param passed to Gerrit reporter20:46
fungidoes gerrit 2.8 now solve the issue with comments and no associated vote not counting as review activity?20:46
hasharand it executes the command20:46
greghaynesInteresting upgrade bug - people with quotes in their displayname cause ISE when clicking to go to their dashboard. e.g.,n,z20:46
*** eharney has joined #openstack-infra20:46
hasharjeblair: with Gerrit 2.4  that would have rejected the command because you can't vote, hence why we had to pass force-message20:46
anteayagreghaynes: thanks20:47
*** nati_ueno has quit IRC20:47
jeblairhashar: i think the check in there dates from an ealier syntax in layout.yaml.  where the lack of params was an indication that it should be silent....20:48
jeblairhashar: but now we have explicit reporters, so i think the lack of _reporter_ indicates the same thing20:48
*** ociuhandu has joined #openstack-infra20:48
hasharI guess:)20:48
jeblairhashar: so i think we should be able to say that if there's a reporter configured, even with no params, it should leave a msg20:48
*** afazekas has quit IRC20:48
jeblairmordred: (i've dropped the irc thing in favor of zuul)20:49
mordredjeblair: yes. I agree with you20:49
anteayazaro: click Clint's display name:
SpamapSI _must_ have my quotes.20:49
SpamapSAND YES20:49
mordredjeblair: it's my turn to need to step away for a sandwich20:49
anteayazaro: how do we fix that20:49
SpamapSI found another way to break Gerrit!!20:49
anteayaSpamapS: well done20:49
anteayaSpamapS: working on it20:49
SpamapSGerrit: pwned20:50
mordredSpamapS: what is it that doesn't work?20:50
anteayaclick his name20:50
anteayainternal server erro20:50
mordredthat gets me:,n,z20:50
zaroanteaya: fix what?20:50
mordredwhich works fine20:50
zarowfm too20:50
mordredif I click his name from that listing20:51
mordredI get failure20:51
hasharjeblair: got a patch to Gerrit reporter with some background in the commit summart20:51
jeblairhashar: link?20:52
*** SumitNaiksatam has joined #openstack-infra20:52
anteayadid 2.4.4 use quotes around owner?,n,z20:52
anteayaI didn't think it did20:53
*** sarob has quit IRC20:54
fungiYorikSar: success!
mordredSpamapS: can you try changing your double quotes to single quotes?20:54
*** marcoemorais has quit IRC20:54
funginow i just need to work on querying the latest patchset numbers and owner e-mails for the ~300 wip changes20:55
YorikSarfungi: Oh. I've put my -1 there now? Really? :)20:55
YorikSarfungi: unwiping works fine now20:56
fungiYorikSar: excellent, that's what i was hoping20:56
ArxCruzI'm getting BadHostKeyException: Host key for server does not match! in my zuul20:56
*** nati_ueno has joined #openstack-infra20:56
*** marcoemorais has joined #openstack-infra20:56
YorikSarfungi: You can use new shiny REST API.20:56
fungiArxCruz: the user under which your zuul daemon is running probably needs to clear the old gerrit host key from known_hosts20:57
hasharjeblair: sorry
ArxCruzfungi: anteaya ty guys :)20:57
hasharjeblair: was fighting with git-review :D20:57
fungiYorikSar: good idea20:57
anteayaArxCruz: let us know, you are the first of many 3rd party ci systems that will show up20:58
*** Sukhdev has joined #openstack-infra20:58
anteayaArxCruz: would be good to have your feedback20:58
SpamapSmordred: sure20:58
ArxCruzanteaya: sure :)20:58
anteayaArxCruz: thanks20:58
hasharjeblair: no test though :-(20:58
YorikSarfungi: It won't work with jq though.. You need to skip first 4 chars in every response to get parseable JSON20:58
SpamapSmordred: single quotes works20:59
jeblairhashar: i think it may need a little more than that; i'm working on a test, should have something in a bit20:59
fungiYorikSar: yeah, for this time around, i'm going to settle for join queries on the database21:00
ArxCruzanteaya: it seems to work now :)21:00
anteayaArxCruz: awesome, thank you21:00
jogoI am confused at why this failed to merge:
*** pdmars has quit IRC21:01
anteayakrtaylor ArxCruz do you feel like whipping up an email for the 3rd party ci people? I'm sure they have missed all of ours21:01
anteayakrtaylor ArxCruz perhaps they will listen to you21:02
*** pdmars has joined #openstack-infra21:02
anteayaand it will give me something to point to in the next few days21:02
zarojogo: i believe you need verified+1 before something can merge21:02
*** dkehn is now known as dkehn_afk21:02
jogozaro: right, and jenkins said it couldn't be merged21:02
jogoThis change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset.21:02
hasharthat is usually Zuul not being to merge the change with the tip of the branch21:03
hasharbut Zuul might fails for another reason though :/ Like not being to update the tip of the branch from Gerrit if it is dead21:03
jogohashar: I rebased the branch locally and its at master21:03
hashar(at least on my setup, which is not OS one)21:03
*** mfer has quit IRC21:03
anteayathe patch comments seem to be "open" by default, I don't see a "closed" or "mark as read" type appearance21:03
hasharjogo: I think it is on the server side :/21:04
jogoexactly why I am wondering what happened21:04
jogozaro: ^21:04
jeblair+    success:21:04
jeblair+      gerrit: {}21:04
jeblairhashar: ^ we'll have to do that unless we do something special to change the yaml syntax21:04
ArxCruzanteaya: done21:05
anteayaArxCruz: thanks21:05
*** sarob has joined #openstack-infra21:05
anteayaArxCruz: where did you post it to?21:05
ArxCruzanteaya: openstack-infra maillist21:06
anteayaokay great21:06
hasharjeblair: apparently :( since you experimental only has a message which is defined internally21:06
*** pdmars has quit IRC21:06
ArxCruzanteaya: and twitter right now :)21:07
jogojeblair: any ideas on why this is failign to merge: it works locally --
jeblairhashar: can you take a look at that?21:07
hasharjeblair: sure21:09
jeblairjogo: yeah, more host key problems21:09
jogojeblair: on my side or your side?21:09
greghaynesIm having a similar issue to jogo with
anteayaArxCruz: go you21:09
jeblairjogo: try again now21:09
jeblairfungi: are you done with cleaning up wip?21:10
jogojeblair: no dice21:10
YorikSarjeblair: We can remove {} if we tolerate None in params.21:10
jogo still failing to merge21:10
hasharjeblair: might want to update doc as well. But I guess it can be done later21:10
fungijeblair: still working on it. i have the api query list i need to run almost finished21:11
*** matty_dubs is now known as matty_dubs|gone21:11
hasharjeblair: sounds good :)21:12
jeblairhashar, YorikSar: yes, we could change the layoutvalidator to accept (None or dict) there...21:13
jeblairzuul assumes it's a dict though, so we'd need to change a few references to that...21:13
hasharI am trying to find the documentation for the reporters21:13
*** gokrokve has joined #openstack-infra21:14
jeblairYorikSar, hashar, mordred, fungi: ^21:15
jeblairYorikSar, hashar, mordred, fungi: preferences?21:15
*** mrmartin has quit IRC21:16
*** thomasem has quit IRC21:16
anteayajeblair: so option B would work now and option A would require a patch to zuul?21:16
fungijeblair: option b has less of a chance to get mis-represented by an indentation error21:16
jeblairjogo: try again21:17
jeblairanteaya: they both require a patch; option a requires a bigger patch, option b only needs
hasharjeblair: is that going to change the layout ?21:17
jogojeblair: :(21:17
jogojeblair: still nothing21:18
Alex_GaynorIf we've got approved patches which didn't land and aren't in the pipeline, should we recheck no bug, or is someone going to batch do that?21:18
YorikSarjeblair: Yes. On another look opion B looks better.21:18
ArxCruzanteaya: in my case I had to run ssh -i /var/lib/zuul/ssh/id_rsa -p 29418 gerrit ls-projects21:18
jeblairbtw, openstack sucks at taking a week off.21:18
jogojeblair: heh yeah21:18
ArxCruzperhaps is better add a note that it might need to pint the ssh key21:18
ArxCruzs/pint/point to21:18
jeblairthat's my feedback for the next release cycle session.  :)21:18
*** dkliban_afk is now known as dkliban21:18
*** gokrokve has quit IRC21:19
anteayagerrit server is working awefully hard between page loads on that patch, for me21:19
*** aysyd has quit IRC21:19
jeblairArxCruz: yeah, i've done that but am still seeing RSA host key for IP address '2001:4800:7815:104:3bc3:d7f6:ff03:bf5d' not in list of known hosts.21:19
jeblairi also did that with the ip addr21:19
ArxCruzjeblair: it's working for me now :)21:19
anteayahuh, must just be me:
hasharYorikSar: jeblair mordred fungi : got a third option which is to explicitly ask the reporter to report a message21:20
*** ociuhandu has quit IRC21:21
*** nati_ueno has quit IRC21:21
YorikSarhashar: Why would you want to set message: false?21:21
anteayaso far {} is winning in my books21:22
hasharYorikSar: there is no point indeed. Unless you just want it to just vote21:22
jeblairi think known_hosts on the zuul servers is managed by puppet21:22
mordredjeblair: back - reading options21:22
mordredjeblair: {}21:23
hasharYorikSar: the idea is to make it more obvious when reading the layout file.  gerrit: {}  is not that readable21:23
jeblairjogo: found the problem21:23
*** jeblair changes topic to "Known problem with WIP changes. Known problem with bogus merge failures. Gerrit upgraded to 2.8. See:"21:24
anteayahashar: I find {} more recongizable then no characters or true21:24
mordredjeblair: what can I do to be useful to you right now?21:24
jeblairmordred: gerritbot21:25
mordredjeblair: k. on it21:25
mriedemWIP is back, \o/21:25
fungiokay, owner-impersonated workflow -1 votes have now been successfully added to the ~300 former wip-status patchsets (except for a dozen where it looks like gerrit was unable to reverse the owner's accounts.preferred_email into an actual impersonatable account for some reason... working on those now)21:25
jogojeblair: cool, thanks21:25
YorikSarhashar: Can there be a list of reporters?21:25
hasharanteaya: yup probably :]21:25
hasharYorikSar: I think so. There is a smtp reporter as well21:26
anteayamy WIP changes are back, thanks fungi21:26
*** jooools has quit IRC21:26
*** imcsk8 has quit IRC21:26
mroddeni think i may have found a Gerrit bug... or an unexpected behavior.21:26
*** imcsk8 has joined #openstack-infra21:26
YorikSarhashar: let me add option D then...21:26
mroddenthis patchset was merged on milestone-proposed21:27
mroddenbut this patchset was dependent ( a child of the above patchset, merged on milestone-proposed) but was merged into master
*** afazekas has joined #openstack-infra21:27
jeblairfungi: can you force-merge please?21:28
mroddenwhich brought in code to master that was originally only supposed to be for milestone-proposed21:28
fungijeblair: done, and lgtm21:28
jeblairmrodden: agreed; would be interesting to test with gerrit 2.8 (since that happened under gerrit 2.4)21:29
mroddenjeblair: ok so it was a 2.4 thing21:29
mroddenthat was my guess21:29
mroddenbut its horrifying.21:30
YorikSarhashar, fungi, jeblair, mordred, anteaya: Added option D there.21:30
jeblairmrodden: it still may be in effect21:30
mroddentheoretically i could take master, put a patch on it, propose it to stable/icehouse21:30
mroddenand if someone merged that patch, it would look like master entirely21:30
hasharYorikSar: that would erase the vote made by another pipeline :D21:31
*** aysyd has joined #openstack-infra21:31
jeblairhashar: i think the main thrust of YorikSar's suggestion is the change from dict to list21:31
YorikSarhashar: I've added parameter just for example.21:31
*** nati_ueno has joined #openstack-infra21:31
jeblairYorikSar: that breaks backwards compat though; if we do that, i'd rather have a migration plan21:31
*** maelfius has joined #openstack-infra21:32
jeblairi'm leaning toward option B for now, it's simple and explicit, and it doesn't preclude us from doing any of the others later.21:32
hasharYorikSar: but yeah that was my workaround for my use case ( a pipeline that runs on merged changes. I have added verified: 0 to let zuul message, gerrit 2.8 ignore the score in such when voting on closed changes)21:32
YorikSarjeblair: We can just wrap dict into a list.21:32
mroddeni really hop that bug is fixed in 2.821:32
jeblairfungi: can you review and force-merge ?21:33
fungilooks like the handful of impersonations which failed were due to multiple accounts with the same e-mail address21:33
mroddenit might be a bug with Zuul though, since that is doing the merges21:33
*** adalbas has quit IRC21:33
*** morganfainberg has quit IRC21:33
jeblairmrodden: no, gerrit performs the actual merges, zuul just tells it to.21:33
*** maelfius is now known as morganfainberg21:33
msabramoI also have a problem like jogo: where merges are failing; despite successful rebase locally21:34
jeblairmsabramo: see topic21:34
jeblairfungi: all done with wip?21:34
fungijeblair: i need to manually untangle the dupe accounts which were preventing impersonation for a small percentage of them, but the rest are fixed21:35
jeblairoh ok21:35
jeblairjogo: try again21:35
fungireviewing the zuul change now21:35
mordredjeblair: 2014-04-28 21:35:44,691 DEBUG irc.client: FROM SERVER: ERROR :Closing Link: (Excess Flood)21:36
mordredjeblair: I believe at some point we were trying to reconnect too much and freenode is now unhappy with us21:36
mordredor something21:36
jeblairmordred: does it sleep between joining channels?21:36
mordred0.5 seconds21:36
jeblairmordred: how far does it get before it's closed?21:36
*** dkranz has quit IRC21:37
hasharjeblair: gotta sleep sorry.  gerrit: {}   looks good to me :]21:37
jeblairhashar: cool, thanks for your help!21:38
hasharhave a good post upgrade day21:38
jeblairmordred: ?21:39
*** hashar has quit IRC21:39
mordredtrying to sort that out21:39
jeblairmordred: if it's not immediate, then we should probably sleep 2 seconds between joins21:39
YorikSarGot to sleep now. Thanks to everybody for great long-awaited upgrade!21:39
*** afazekas has quit IRC21:39
jogojeblair: looks promising, didn't fail to merge yet21:40
*** mat-lowery has left #openstack-infra21:40
jeblairmsabramo: ^21:40
*** jeblair changes topic to "Gerrit upgraded to 2.8. See: Gerritbot is offline. Known problem with formatting of test reports."21:41
*** jeblair changes topic to "Gerrit upgraded to 2.8. See: Gerritbot is offline. Known problem with experimental pipelines. Known problem with formatting of test reports."21:41
msabramojeblair: yeah mine looks good so far; at least it didn't fail right away -
mordredjeblair: right now we're getting blocked from connecting due to excessive flood21:42
*** kiall_ has joined #openstack-infra21:42
JayFYou guys might be able to get an exemption for the new IP. I have a buddy on freenode staff I can ping if it'd be helpful.21:43
*** kiall_ is now known as Kiall221:43
*** luisg_ has joined #openstack-infra21:43
Kiall2Hey guys.. tarball job post tag-push just failed.. How would I retrigger that?
jogothanks jeblair21:44
mordredJayF: thanks - might ping you in a bit if we can't sort out rate limits directly21:45
sdagueI would like to extend a warm congrats on upgrade well done. Now I'll just have to get used to CR & V columns being flipped :)21:45
mordredjeblair: just fyi - I've got gerritbot logging and edited in place on review.o.o21:45
*** adalbas has joined #openstack-infra21:45
mordredsdague: zaro is going to fix that for us21:46
sdagueoh, cool21:46
*** UtahDave has quit IRC21:46
sdagueyeh, there are a couple other css tweaks that might be worth it in the process as well21:46
JayFmordred: I ping'd him and asked him to /j here when/if he gets a sec21:47
jeblairsdague: there is so much ui to fix21:47
jeblairmordred: may i run the bot?21:48
mordredjeblair: please21:48
mordredjeblair: I added a debug logger for the irc stuff21:49
sdaguejeblair: well at least on the old screen it's not total vomit :)21:49
mordredin /var/log/gerritbot/gerritbot_irc.log21:49
chmouelanteaya: cool thnaks (about the review tools session)21:49
mordredjeblair: and I changed the delay from 0.5 to 1.5 just to see if it would get further21:49
jeblairmordred: how are you editing?21:49
mordredjeblair: edited /etc/gerritbot/logging.yaml in place21:49
jeblairmordred: how did you change the time?21:49
chmouelanteaya: about the plugin no worries, i was just mentioning that i like the idea about this plugin :) whenever you guys would have time21:50
mordredjeblair: same thing- just touched the installed file in /usr/local/lib - planned on just re-running install when I was done21:50
mordredjeblair: /usr/local/lib/python2.7/dist-packages/gerritbot/bot.py21:50
*** weshay has quit IRC21:51
jeblairmordred: there is no sleep between joins21:51
mordredthere isn't?21:51
jeblairmordred: and the latest output in the log looks like it spews all the joins at the server immediately21:51
mordredjeblair: heh. thank you - I didn't think to actually look at the tiemstamps :(21:52
jeblairmordred: edited to add sleeps21:52
jeblairmordred: how do you run it?21:52
mordredjeblair: /etc/init.d/gerritbot start21:52
mordredjeblair: there's probably a better way to do that21:53
*** ihrachyshka has quit IRC21:53
kiallSo .. nobody knows how I can retigger a failed job triggered by a tag push? Looks like a replication race or something :)
mordredthat's looking better!21:53
mordredjeblair: where did you add the sleep?21:53
*** smarcet has quit IRC21:53
jeblairkiall: you chose an interesting time to push a tag.21:53
msabramojeblair: my change finally merged and passed. Thanks for everyone who fixed this!21:54
kialljeblair: really? Whoops.21:54
*** markmcclain has quit IRC21:54
*** openstackgerrit has joined #openstack-infra21:54
*** markmcclain1 has joined #openstack-infra21:54
jeblairkiall: yeah, it's openstack's week off and we're upgrading gerrit.21:54
mordredkiall: we're still working through issues with this morning's gerrit upgrade21:54
kiallRight. /me should have known.. didn't... heh21:54
jeblairkiall: one of us will have to do that manually later on; remind us when we're not putting out fires21:55
kialljeblair: sure :)21:55
fungiOKAY! _now_ all remaining wip patches have been updated with appropriate workflow votes21:55
mordredfungi: woot21:55
* fungi should go find breakfast. it's been a... busy... day21:56
*** zehicle has joined #openstack-infra21:56
jeblairmordred: can you make that into a change for gerritbot?21:56
mordredjeblair: yah21:57
sdaguearosen: what's up with your devstack changes that are in triplicate?21:58
*** markmcclain1 has quit IRC21:58
*** zehicle_at_dell has quit IRC21:58
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Remove force-message from zuul layout
jeblairyay gerritbot21:59
*** markmcclain has joined #openstack-infra21:59
mordredjeblair: uhm. so22:00
*** rlandy has quit IRC22:00
mordredjeblair: we run gerritbot from pip22:01
*** zehicle has quit IRC22:01
jeblairmordred: yeah, we'll need to cut a new release22:02
mordredjeblair: yeah - we haven't done that since 201222:02
anteayachmouel: thanks (was putting my homemade pizza in the oven)22:02
anteayafungi: yay and thank you22:03
jeblairmordred: we can also pip install locally22:03
jeblairmordred: or switch to cd22:03
mordredjeblair: I'm fine with either - just looking at the list of changes22:03
jeblairi'm restarting zuul to pick up the change we just merged22:04
*** packet has quit IRC22:04
mordredjeblair: also, I _don't_ think we're running gerritbot from pip for real on gerrit22:04
mordredor, oh, wait - yeah22:04
mordredignore me22:04
jeblair /ignore mordred22:04
*** packet has joined #openstack-infra22:05
mordredjeblair: there were sleeps in both places you added sleeps- you want me to just update the old value with the new?22:06
*** jeblair changes topic to "Gerrit upgraded to 2.8. See: Known problem with experimental pipelines. Known problem with formatting of test reports."22:06
zaromordred: label ordering might a little more tricky than meets the eye.  i don't think it's appropriate to match column order with definition from project.config22:06
mordredzaro: what order do you think it should have?22:06
jeblairmordred: nah, don't worry about it then.22:06
*** homeless has quit IRC22:07
mordredjeblair: well, I'm still confused as to why it wasn't sleeping between joins, since there was a thing there22:07
jeblairmordred: because there wasn't22:07
*** zzelle has left #openstack-infra22:07
jeblairmordred:  i promise i added it to the thing on disk22:07
mordredit was at the end of the block22:07
mordredwhat the heck ...22:08
zaromordred: it's because child projects can also define labels, so when you have a collection of labels from parent+children there's many ways to set that order.22:08
pelixAny thoughts on adding a rules file to projects to automatically set the approved to +1 if a change receives two +2 CR and no need for core reviewers to remember once the criteria is reached?22:08
mordredpelix: yeah - we don't do that on purpose22:09
JayFI wouldn't like if that changed the default22:09
zaromordred: i think the best solution would be to let the clients set the order because it's basically just the visual part of gerrit.22:09
*** dims has quit IRC22:09
pelixmordred: any particular reason why?22:09
zaromordred: gerrit gwt is all magic to me right now :(22:10
mordredpelix: because there are many times where a second +2 is not an automatic approve22:10
* jeblair adds "can display review columns in correct order" to gertty's feature list22:10
mordredjeblair: ++22:10
anteayajeblair: will gertty take care of where the line numbers are in the diff view?22:10
jeblairmordred: did i mention that gertty does single-page diff? i think i forgot to put that in the readme22:10
openstackgerritMonty Taylor proposed a change to openstack-infra/gerritbot: Make sure we sleep a little more
mordredjeblair: there is your change applied to current head22:11
mordredjeblair: we should probably land the gertty patch22:11
jeblairmordred: my change shouldn't be necessary22:11
jeblairmordred: there was no sleep in the join loop before22:11
jeblairso whatever was installed must not have been current head22:12
mattoliverauMorning all, how'd the upgrade go?22:12
*** hdd has quit IRC22:12
anteayamattoliverau: so far so good22:12
anteayamattoliverau: still doing clean-up22:12
anteayathe change in the gui is irksome to some22:12
anteayabut that will get smoothed out in the next while22:13
mattoliverauanteaya: it was a big jump, but worth it, nice work everyone!22:13
jeblairanteaya: gertty's diff view has always been line numbers on left.  ;)22:13
anteayajeblair: ah, yay gertty22:13
mattoliverauI assume there will be more features to work with in the future, and still the best move forward.22:13
anteayajeblair: have ween been using gertty and I didn't know22:13
fungioh, also, i was able to finally get a nodepool image to rebuild in rackspace with the current fixes, nova boot an instance from it and reach it over the network22:13
anteayathe way to talk about it makes me feel like I should already know it22:14
jeblairfungi: !!!22:14
anteayafungi: yay22:14
anteayamattoliverau: /nod22:14
*** msabramo has quit IRC22:14
mordredfungi: w00t22:14
*** gokrokve has joined #openstack-infra22:14
mattoliverauI get 503 as well22:15
mattoliverausorry wrong channel :P22:15
mordredjeblair: it's got 2 +2's - I can aprv it if you like22:15
*** dizquierdo has joined #openstack-infra22:15
jeblairmordred: so i'm assuming gerritbot was running something close to master on the old server, does that sound rightL22:16
jeblairmordred: (and it's just the tagged release on pip that lacks the sleeps?)22:16
* anteaya eats pizza while it is hot22:16
mordredjeblair: yeah22:16
jeblairmordred: so we can land the increase sleep change if you want, but it's not necessary -- the necessary thing is either to tag or switch to cd22:17
morganfainbergmordred, jeblair , i think i might have found an issue w/ gerrit config, +1 Workflow is "approve" right?22:17
mordredjeblair: ya. with you22:17
*** msabramo has joined #openstack-infra22:17
mordredmorganfainberg: that's right22:18
*** flaper87 is now known as flaper87|afk22:18
morganfainbergi'm seeing (new dashboard) the ability to +1 Workflow on devstack-gate22:18
morganfainbergi'm not core.22:18
morganfainbergnot just -1 workflow22:18
morganfainbergi can't +2 CR, but it looks like i can +1W22:18
*** gokrokve has quit IRC22:19
mordredyou can -1/0/+1 on devstack-gate?22:19
morganfainbergwell the UI says so22:19
morganfainberglet me try in a sec22:19
jeblairmorganfainberg: your own change or someone else's?22:19
mordredmorganfainberg: change number?22:19
morganfainbergjeblair, my own change22:19
*** mwagner_lap has joined #openstack-infra22:19
morganfainbergit's a UX bug22:20
morganfainbergit doens;'t let me22:20
morganfainbergit puts up a really big error saying "+1 Workflow restricted"22:20
morganfainbergmeh, poor UX design, but thats what i get for using the "new" screen22:20
jeblairzaro: can you look into that please ^22:20
*** sarob has quit IRC22:20
jeblairmorganfainberg: oh, it's only on the new screen?22:20
*** sarob has joined #openstack-infra22:21
mordredmight be the way they draw the selector box things22:21
zaroi don't think you can change back to old screen22:21
morganfainbergjeblair, yeah just noticed it wasn't on the old screen when i flipped back22:21
morganfainbergjeblair, sorry for raising the flag.22:21
morganfainbergzaro, you can, it's in perferences22:21
jeblairzaro: the old screen is the default and you can definitely change back and forth22:21
morganfainbergzaro, server default is "old screen"22:21
mordredmorganfainberg: no - that's good - it's good to know22:21
jeblairone more reason to be terrified of the new change screen22:22
mordredjeblair: I think we should tag gerritbot22:22
mordredif we want to move to CD later, cool22:22
morganfainbergjeblair, i actually like some of the changes in  the new screen.  but.....22:22
sdaguemorganfainberg: yeh, the new screen is kind of a mess22:22
mordredjeblair: you want me to do it/22:22
zaronew screen breakdown:
*** nati_uen_ has joined #openstack-infra22:23
jeblairmordred: just did it22:23
zaromorganfainberg: ^22:23
morganfainbergzaro, ah nice. will add my feedback22:23
jeblairthe new screen has some serious ui problems22:23
morganfainbergjeblair, the worst was "reply" = "review"22:24
morganfainbergtook me clicking around a bunch to figure out that was how to score things22:24
mordredjeblair: aweomse22:24
*** etoews has quit IRC22:25
*** sarob_ has joined #openstack-infra22:25
mordredbasically, the UI is the thing that gerrit has been historically the worst at - and in the new UI, it has gotten much better at being the worst at UI22:25
*** sarob has quit IRC22:26
*** reed has quit IRC22:26
morganfainbergmordred, ++22:26
mordredwhich is a shame - because the engine does some pretty amazing things22:26
*** nati_ueno has quit IRC22:26
jeblairthe whole screen is too wide (for users who don't have super-wide displays), and the comment lines are too long (long lines hinder reading), and then the information architecture is mostly about how much can be crammed into random places22:26
jeblairanyway, this is distracting... i'll save my new screen ranting for some other time22:26
*** jgrimm has quit IRC22:27
mordredjeblair: I think we're good to re-enable puppet on gerrit22:27
*** ociuhandu has joined #openstack-infra22:27
jeblairmordred: let me check what i have in my env on puppetmaster22:27
*** jdurgin has quit IRC22:27
mordredjeblair: (I put patches for that up a little while ago which I think have landed)22:28
mordredunless you've added new thingses22:28
mattoliverauTook me a while to figure out how to make a code review (vote) on the new view as well.22:28
anteayashould I still be following: or have we moved on?22:29
sdaguemattoliverau: there is a reason the old view is the default :)22:29
*** dizquierdo has quit IRC22:29
morganfainbergsdague, ++22:29
jeblairmordred: i don't see the mysql changes in config HEAD22:29
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Plumb mysql host through gerrit.pp
jeblairmordred: we need to land that ^22:30
openstackgerritJames E. Blair proposed a change to openstack-infra/jeepyb: Use configured mysql host
jeblairmordred: and that ^22:32
mordredboth are +2 from me22:32
jeblairmordred: also look at
zaromordred: what happened?  i reviewed 90837 earlier but it looks completly new22:33
anteayayeah I am looking at the same:
zaroopps i meant 9083722:34
mordredzaro: no - you reviewed the version I tossed up - look at jim's patch22:34
anteayamordred: what is the difference between jims and yours22:34
anteayabesides author22:35
mordredanteaya: jim wrote his - and mine - so it makes mre sense to land jim's22:35
*** doug-fish has left #openstack-infra22:35
*** sarob_ has quit IRC22:35
anteayaoh okay, I was looking for a difference in the code22:35
anteayawhat about zaro's question on 90837?22:36
*** dims has joined #openstack-infra22:37
mordredoh - wow. workflow WIP shows as a big red X in the UI22:37
mordredthat's gonna take some getting used to22:37
*** dkehn_afk is now known as dkehn22:37
anteayaand abandoned does the same, sometimes22:38
mordredzaro, anteaya: I think we don't need to do that in this patch - the gerrit module has it as a default - but none of ours do22:38
mordredwhich is, I think, correct22:38
anteayaI haven't figured out when abandoned does show up as a big red x22:38
jeblairmordred: that's the downside of gerrit's answer to our ui changes always being "but you can make your own searches and bookmark them".  the main program's ui never gets better.22:39
anteayaokay, I just didn't want to gloss over khai's question22:39
mordredanteaya: ++22:39
mordredjeblair: yah22:39
mordredjeblair: this is the reason I  can't wait to get a copy of gertty - so I can stop using the main program's UI22:39
jeblairi can't seem to push the tag for gerritbot22:40
anteayaI'll +1 jim's patch and zaro can do what he likes22:40
mroddeni find it sad that the direction they are going with the Gerrit UI isn't like the original "spartan" clean and simple design22:40
mroddenits such a great tool otherwise22:40
anteayazaro: I am not disagreeing with you, just saying it is up to you to decide22:40
mordredjeblair: I think that there isa  new "pushSignedTag" thing in gerrit now, isn't there?22:41
mordredjeblair: let me try adding that to the config so you can try again22:41
anteayamrodden: maybe java devs don't naturally consider ux? I don't know is that an incorrect sweeping generalization?22:41
*** msabramo1 has joined #openstack-infra22:41
jeblairkiall: when you pushed a tag earlier, was it signed?22:41
*** jdurgin has joined #openstack-infra22:42
mordredoh! neat!22:42
mroddenanteaya: after looking at the new UI mode in 2.9, i think they are going after some kind of "social coding" UI idea22:42
mordredacl changes can produce review snow :)22:42
mroddenit has avatars and crap...22:42
kialljeblair: my first attempt to push a signed tag was rejected, switched to an annotated tag after..22:42
mattoliverauanteaya: I'm a java dev in a past life... and nope it's not an incorrect sweeping generalization :P22:42
jeblairkiall: nice.  we're not going to publish that.  you should only ever push signed tags.22:42
mroddenlol or that...22:42
mordredjeblair: well, the review is neat - but I have power to approve it22:43
kialljeblair: hah .. I thought so, but.. Yea, it didn't click.22:43
mordredif you're ok with me trying that now, I'll just do it old school22:43
openstackgerritMorgan Fainberg proposed a change to openstack-infra/devstack-gate: Allow setting services up under apache mod_wsgi
jeblairmordred: yeah, then we'll batch change the whole system once we verify it.22:43
mordredjeblair: done. try pushing tag again22:44
jeblair * [new tag]         0.2.0 -> 0.2.022:44
*** msabramo has quit IRC22:44
*** aconrad has quit IRC22:44
jeblairmordred: want to whip up a batch acl change?22:44
mordredjeblair: oh do I ever22:44
zaroanteaya: works either way. imo one way seems more correct than the other.22:44
mordredjeblair: I wonder if you need pushTag as well22:44
*** pelix has quit IRC22:44
mordredkiall: wanna be a guinea pig for something/22:45
jeblairmordred: how about the sandbox?22:45
zaroanteaya: 'x' is for the largest negative vote.22:45
kiall^ sounds like a better idea :P22:45
mordredjeblair: yah. ok22:45
fungioh, wow. gerrit decided to add a new acl to limit who could push signed tags, and made the old acl only cover allowing annotated tags? that's so completely backwards22:45
jeblairfungi: :)22:46
mordredjeblair: btw - it doesn't look like the refs/tags/* entries are needed at all22:46
anteayamordred: so I can comment on but I have no vote22:46
mordredanteaya: same here. I'm just goign to abandon that22:46
anteayamrodden: ewww, why do i need a picutre of you, you are right here22:46
anteayamordred: but why did that patch do that?22:46
anteayamattoliverau: ah okay, didn't want to hurt feelings22:47
zaroanteaya: that's a project config change. i think only admins have voting rights there22:47
jeblairmordred: i added push signed tag only22:48
anteayazaro: hmmmm, okay22:48
anteayalearning new things, thanks22:48
sdaguezaro: if I wanted to try to submit some gerrit css fixes, which branch should that be on?22:48
zarosdague: openstack/2.8.422:49
sdagueok, cool22:49
jeblairmordred: wrong reference; updated22:49
openstackgerritA change was merged to openstack-dev/hacking: Move other checks into their own file
*** jhesketh_ has joined #openstack-infra22:49
*** jhesketh_ is now known as jhesketh22:50
jeblairmordred: the current config of sandbox allows signed tags and disallows unsigned tags according to my testing22:50
mordredjeblair: awesome22:50
sdaguezaro: which css file should I be proposing fixes into?22:50
mordredjeblair: mind if I try something else real quick?22:50
fungisdague: zaro: we want css changes to be external to the gerrit source, and only apply things which are already or nearly accepted upstream to the gerrit fork we've got, right? it would be nice for the openstack-infra/gerrit project to disappear soonish22:50
jeblairmordred: ++22:50
sdaguefungi: sure, just point me to where that should be22:51
jeblairfungi, sdague, zaro: gerrit's css is meant to be customized locally22:51
jeblairfungi, sdague, zaro: and we have customized versions, i think that's fine22:51
*** prad_ has quit IRC22:51
jeblairfungi, sdague, zaro: having said that -- making upstream look nicer would be nice (we have done that once before already -- we are the reason why it isn't lime-green by default anymore)22:52
fungijeblair: sdague: zaro: should we start forking and installing the css via openstack-infra/config instead?22:52
*** msabramo1 has quit IRC22:52
zarojeblair: where are those local customizations located?22:52
anteayajhesketh: morning22:52
jeblairfungi: it's not forking, it's customizing22:52
sdaguefungi: it looks like it's compiling and compressing all the css into 1 file22:52
*** Kiall2 has quit IRC22:52
jeblairfungi: ^22:53
jeblairsdague, fungi:
fungiahh, yep, so we're already doing that--all the better22:54
sdaguejeblair: where do you think that's getting picked up?22:54
jeblairsdague: that file is a config option for gerrit22:55
sdagueah... ok22:55
sdaguethen it compiles it in22:55
jeblairsdague: wouldn't surprise me, but i can't speak to that.22:55
jeblairsdague: it is a run-time config option though22:55
*** mriedem has quit IRC22:56
sdagueyeh "compile" is loosely defined here22:56
jeblairso we're talking about compile-on-startup not compile-at-compile-time :)22:56
* jhesketh looks like he's missed all the fun22:56
*** aconrad has joined #openstack-infra22:56
jeblairhardly; fun continues22:56
jheskethdarn, I was about to go back to bed22:56
sdagueyeh, it's being just dumped straight inline actually22:56
sdagueok, good22:56
*** sarob has joined #openstack-infra22:56
*** mbacchi has quit IRC22:57
*** Sukhdev has quit IRC22:58
*** prad has joined #openstack-infra22:58
jeblair    if self.waitForRefSha(change.project, ref, change._ref_sha):23:00
jeblairthat's taking a long time in zuul23:00
jeblairthat's why it seems to be sitting there for a while23:00
jeblairthat whole idea is still problematic and we need to figure out a way to get rid of it23:01
*** etoews has joined #openstack-infra23:01
jeblairthough i suspect at the moment we've missed something in the move23:01
*** aconrad has quit IRC23:02
jeblairyeah, that's always failing, which means the dependent queue is worst case23:02
jeblairnot to mention, it's very slow23:03
jeblairdansmith: that's what you saw earlier23:03
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Update acls to enforce pushing signed tags
mordredjeblair: ^^ there ya go23:03
jeblairi'm going to dig into that for a few mins; biab23:03
openstackgerritA change was merged to openstack-dev/hacking: Remove H901,H902 since it is a duplicate of H714,H713
jerryz_fungi: could you please take a look at this tiny patch, Thanks!23:06
mordredjeblair, sdague, fungi: we could put an entry into the html header which loads a CSS file from a url - and we could manage that without changing gerrit23:07
*** alexpilotti has quit IRC23:07
sdaguethough that css file is just straight dumped in23:07
sdagueso we sort of get the same results now23:07
sdaguewith one less http fetch23:08
jeblairmordred: this is the last time i'm going to say this23:08
mordredsure - but i the problem was that we can't change it without restarting gerrit23:08
jeblairmordred: we do not edit the css in the gerrit source tree23:08
mordredjeblair: yes. I know that23:08
jeblairis a file that is included23:08
jeblairgerrit does not need to be changed23:08
mordredyes. I did not say it did23:08
mordredand I know that we're doing that23:09
jeblairmordred: then why did you say "and we could manage that without changing gerrit" ?23:09
mordreds/changing gerrit/restarting gerrit/23:09
jeblairmordred: that is an very important distinction23:09
*** mbacchi has joined #openstack-infra23:09
mordredjeblair: yes it is -sorry for the confusion23:10
mordredI was responding to "jeblair | "so we're talking about compile-on-startup not compile-at-compile-time"23:10
openstackgerritMorgan Fainberg proposed a change to openstack-infra/devstack-gate: Allow setting services up under apache mod_wsgi
*** bauzas has quit IRC23:11
*** mriedem has joined #openstack-infra23:11
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Check for new files as well as changed files
openstackgerritSean Dague proposed a change to openstack-infra/config: limit the width of vote columns
*** gokrokve has joined #openstack-infra23:14
*** afazekas has joined #openstack-infra23:15
*** jhesketh__ has joined #openstack-infra23:15
*** jhesketh has quit IRC23:15
sdaguedo I remember that we might be able to get full diffs in change email?23:16
jeblairsdague: we still have several regressions, can we defer discussion a bit until we have those wrapped up?23:17
*** jhesketh__ is now known as jhesketh23:17
jeblairmordred: i'm having trouble figuring out why zuul isn't dealing with them correctly23:18
*** packet has quit IRC23:18
jeblairmordred: with the ref sha thing23:18
mordredjeblair: if I wanted to help you look - what woudl be a good first place?23:18
*** rcarrill` has joined #openstack-infra23:19
jeblairjhesketh: any thoughts?23:19
*** gokrokve has quit IRC23:19
mordredjust the code?23:19
jeblair2014-04-28 23:09:50,833 DEBUG zuul.Gerrit: Change <Change 0x7fc91439ac50 90269,3> did not appear in the git repo23:19
jeblair2014-04-28 23:09:52,005 DEBUG zuul.Gerrit: Waiting for <Change 0x7fc914d99510 89390,2> to appear in git repo23:19
jeblairoops sorry23:19
mordredjeblair:  are they there?23:19
jeblair2014-04-28 23:04:48,554 DEBUG zuul.Gerrit: Waiting for <Change 0x7fc91439ac50 90269,3> to appear in git repo23:19
jeblair2014-04-28 23:09:50,833 DEBUG zuul.Gerrit: Change <Change 0x7fc91439ac50 90269,3> did not appear in the git repo23:19
jeblairthose are the relevant log entries23:19
jeblairbasically, every change it has merged since the upgrade exhibits that behavior23:19
jeblairmordred: yes23:20
mordredjeblair: is the hex number the internal zuul ide?23:20
jeblairmordred: yes23:20
jeblairmordred: and afaict, the replica is correct23:20
jheskethjeblair: sorry, my connection dropped out.. which were you asking my thoughts on?23:20
*** rcarrillocruz has quit IRC23:20
sdaguejeblair: sure23:21
jeblairmordred: it's a pretty dumb system actually -- it just checks to make sure the sha for the ref is different after the change merges23:21
*** asettle is now known as asettle-bfast23:21
jeblair(doesn't actually care what it is, just as long as it's not the old value23:21
sdagueI need to make dinner anyway, I'll circle tomorrow23:21
jeblairsdague: thx23:21
*** tjones has joined #openstack-infra23:21
mordredjeblair: is it possible that gerrit is reporting back a different payload?23:21
mordredjeblair: and therefore we're seeing the orig sha and not the sha of the merge commit?23:21
*** hogepodge has quit IRC23:21
jeblairmordred: it uses the apache mirror23:21
anteayajhesketh: new gerrit and zuul are not speaking well23:22
jeblairmordred: that's why it odesn't care23:22
openstackgerritA change was merged to openstack-dev/hacking: Add noqa everywhere that it is easy to do so
tjoneshi - i have made changes to the branch and am trying to push them upsteam and am getting remote: (W) No changes between prior commit 25607b5 and new commit 079bf04.  anyone else have trouble with git review?23:22
anteayatjones: well we just upgraded gerrit23:22
anteayatjones: and we are still sorting things out23:22
*** hemna is now known as hemna__23:22
*** david-lyle has quit IRC23:22
anteayatjones: so you have local changes and you are having difficulty getting them onto gerrit?23:22
tjonesanteaya: yeah i  know - just checking if it's me or something else.  i'll wait patiently :-)23:22
anteayatjones: can you paste your error output, I'll take a look23:23
*** melwitt has quit IRC23:24
tjonesanteaya: thanks
mordredjeblair: is coming from zuul data model right? there's no chance gerrit is sending a url escaped / ?23:24
anteayamordred: I am reviewing 90842 the acl change and I have questions, but they can wait until gerrit and zuul are getting along better23:24
jeblairmordred: i think the apache mirror may be behind23:24
mordredjeblair: oh!23:25
anteayatjones: what does git diff show you?23:26
tjonesnothing at all23:26
mordredjeblair: icationQueue : Rescheduling replication to to avoid collision with an in-flight push.23:26
jeblairmordred: yeah, the apache mirror is out of date23:26
mordredjeblair: there's a BUNCH of that ^^ in the log23:27
tjonesi'l paste another.  i made a change to spacing, git diff saw that, i added, committed, and review says i changed nothing23:27
mordredoh. hahahaha. also - not related to this - launchpadlib is not on the server, so the change hooks are unhappy23:27
anteayatjones: what kind of change to spacing?23:28
anteayaif I remember correctly there was a patch somewhere to compress more than one whitespace to one whitespace23:28
anteayaclarkb was working on that23:28
mordredjeblair: something is WILDLY Wrong with replication23:28
tjonesanteaya: i removed a blank line23:28
anteayaI have forgotten context though23:29
anteayatjones: at the end of a file?23:29
mordredI've been scrolling up in the rror log file with y finger holding down the PAGE UP button for quite a while23:29
tjonesno - midway23:29
*** asettle has joined #openstack-infra23:29
mordredand it's all just lines of that line above23:29
tjonesit didn't add much value23:29
*** lakshmi has quit IRC23:29
tjonesi can change something like ordering of init variables to see if it is whitespace dependent23:29
anteayatjones: try that23:30
anteayatjones: what editor are you using?23:30
jeblair$ grep Rescheduling error_log|wc -l23:31
anteayaI don't recognize23:31
mordredjeblair: yeah23:31
tjonesanteaya: same issue
mordredjeblair: right before the first one is an error about github hostkey23:32
tjonessee the bottom23:32
jeblairmordred: did we restart since then?23:32
mordredjeblair: I fix the github hostkey and restarted - perhaps I did not do it fully properly?23:32
jeblairScheduling a retry to avoid collision with an in-flight push is differentiated from a retry due to a transport error. In the case of collision avoidance, the job is rescheduled according to the replication delay, rather than the retry delay.23:33
*** saper_ is now known as saper23:33
jeblairmordred: so 1 thing -- i think the reason we get so many is the replication delay is 0, right?23:33
mordredyes. that's right23:33
tjonesanteaya: this may be a hint.  I made more significant changes originally and when it said "do you really want to do this" i said no and then changed something else and then add/commit/review again and this time said yes23:33
jeblairso it's doing that in a tight loop23:33
mordredquite impressively23:33
*** jaypipes has quit IRC23:34
mordredjeblair: is it possible that new gerrit is better at something which is causing it to trip this?23:34
mordredjeblair: and since we do have waitForRef in zuul - something like replicationDelay=1 shouldn't be a functinoal issue, right?23:34
openstackgerritKhai Do proposed a change to openstack-infra/config: update docs and All-Project configuration for gerrit 2.8
jeblairmordred: yeah, i wonder how short we can make that?23:35
jeblairmordred: could we do 0.1?23:35
*** praneshp has quit IRC23:35
jeblairi'm trying to find the docs for it23:35
anteayatjones: can you submit a test patch to
anteayatjones: create a new patch with no dependencies23:36
tjonesanteaya: sure23:36
tjonesjust a sec to set up23:36
*** jhesketh has quit IRC23:37
tjonesanteaya: yes that worked fine23:38
*** jhesketh__ has joined #openstack-infra23:38
*** jhesketh__ is now known as jhesketh23:39
anteayatjones: okay so that indicates to me that your gerrit account has the correct permissions23:39
tjonesanteaya: do you want me to try to make another patch depenent on this one in the sandbox?23:40
anteayalet's try that23:40
jeblairmordred: i think it's an int23:40
mordredjeblair: it's an int23:40
jeblairmordred: let's set replicationDelay to 1 for our sanity23:40
mordredjust foudn it in the code23:40
jeblairmordred: you want to do that and restart?23:40
*** jeblair changes topic to "Gerrit upgraded to 2.8. See: Known problem with replication. Known problem with experimental pipelines. Known problem with formatting of test reports."23:41
*** wenlock has quit IRC23:41
*** jeblair changes topic to "Gerrit upgraded to 2.8. See: Known problems with replication, experimental pipelines, and formatting of test reports."23:41
tjonesanteaya: bummer - it worked23:41
mordredjeblair: restarting23:42
anteayatjones: okay now create another dependency and answer no to the first prompt23:43
mordredjeblair: restated23:43
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Increase replication delay to 1 second
openstackgerritA change was merged to openstack-dev/hacking: Fix py33, make sure skip_on_py3 is part of every check
*** mnaser has left #openstack-infra23:44
*** smarcet has joined #openstack-infra23:44
* zaro needs to catch a bus home, bbl23:45
tjonesanteaya: still worked.  i did the same steps exactly - including a rebase23:45
*** etoews has quit IRC23:45
*** miqui has quit IRC23:46
*** asettle has quit IRC23:46
anteayatjones: hmmmmm23:46
anteayawell we have ruled out problems with your gerrit account23:47
anteayanow we have to look more closely at the git log of the repo you want to offer patches to23:47
mordredjeblair: it doesn't seem to be particularly happy about the lack of All-Projects.git on the targets23:47
jeblairmordred: i think that's always been an ignorable error23:48
*** etoews has joined #openstack-infra23:48
*** jamielennox|away is now known as jamielennox23:49
mordredwow. the documentation has gotten worse23:49
mordredhow do I find docs on replication now?23:49
openstackgerritA change was merged to openstack/requirements: troveclient 1.0.4 released
anteayatjones: so this is the patch we are discussing?
jeblairmordred: i never found them23:49
jeblairzaro: ?23:49
jeblairoh he's gone23:50
jeblairmordred: mind if i apt-get install python-launchpadlib23:50
mordredjeblair: please do23:50
mordredjeblair: I'll make a puppet change23:50
tjonesanteaya:  it is actyally the top of the dependency chaing
anteayatjones: you are trying to make a change to shawn hartsocks patch?23:51
tjonesanteaya: yes i have taken them over from him23:51
anteayathis might be a case of patch ownership, I don't know23:52
mattoliverautjones anteaya: I wonder if the errors we are seeing (reported by Jim) is the cause, they seem to signify patches are failing to land on some repos, and the continuing resheduling of replicaitons based on collisions... I wonder if tjones repo is effected by this problem. The patch fails to land, so git-review does a diff with nothing, seeing no change, etc. Maybe it's worth waiting till jim and23:52
mattoliveraumordred can solve the current issues.23:52
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Add launchpadlib to review.o.o
*** signed8bit has quit IRC23:53
tjonesmattoliverau: sure - if you guys can ping me when done i can resubmit the review to see23:53
anteayamattoliverau: they certainly might be overlapping issues, that is true23:53
anteayatjones: can you check back tomorrow?23:53
anteayathis is feeling like a stubborn issue and I'm sense we are all pretty tired23:54
jeblairmordred: do we have a hook that does something with 'gerrit gsql'?23:54
*** praneshp has joined #openstack-infra23:54
mordredjeblair: yes23:54
mattoliverautjones: it's only the start of my day, I'll ping you once it's resoved :)23:54
*** etoews has quit IRC23:54
anteayamattoliverau: thanks23:54
mordredor, I thought we did23:54
jeblairmordred: i mean, we're getting errors related to that in the logs23:54
tjonesanteaya: sure. ok thanks mattoliverau.  otherwise if i do not hear from you I will try again tomorrow am23:54
mordredjeblair: jeepyb trivial_rebase23:54
anteayatjones: what is am for you?23:54
*** lcostantino has quit IRC23:54
jeblair[2014-04-28 23:54:26,365] ERROR : Cannot store encrypted contact information23:54 Connection failed: 40423:54
tjonesanteaya: im in PDT23:54
jeblairthat's delightful23:55
anteayaso 1600 utc23:55
anteayak thanks23:55
mordredjeblair: yay23:55
jeblairoh well, no new contributors23:55
tjonesanteaya: yes23:55
mordredjeblair: didn't we land a patch to disable trivial rebase?23:55
*** alexpilotti has joined #openstack-infra23:55
jeblairmordred: yes, no one has investigated why it isn't working23:55
*** aysyd has quit IRC23:55
fungijeblair: i tested updating my contact info earlier, after the upgrade, and it succeeded23:55
jeblairmordred: but we have other hooks23:55
jeblairfungi, mordred: that may be benign23:56
jeblairit happened before the upgrade too23:56
fungijeblair: so either the 404 may be intermittent, or an actual e-mail address mismatch or similar23:56
mordredjeblair: ok23:56
jeblairfungi: with your confirmation it worked after the upgrade, i'll just ignore that for now23:56
*** hogepodge has joined #openstack-infra23:57
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Actually disable trivial_rebase_hook
mordredjeblair, fungi: that should make trivial_rebase _ACTUALLY_ go away23:58
anteayaI'm fading, mattoliverau do you have enough of a sense of things you can take over answering random dev questions?23:59
mattoliverauanteaya: I'll do my best :)23:59
jeblair2014-04-28 23:49:54,289 DEBUG zuul.Gerrit: Waiting for <Change 0x7fc8e017db90 86542,2> to appear in git repo23:59
jeblair2014-04-28 23:54:57,512 DEBUG zuul.Gerrit: Change <Change 0x7fc8e017db90 86542,2> did not appear in the git repo23:59
anteayamattoliverau: tag, you're it23:59
jeblairmordred: that's since the restart ^23:59
mordredjeblair: blast23:59

Generated by 2.14.0 by Marius Gedminas - find it at!