Thursday, 2016-10-20

*** Apoorva_ has joined #openstack-infra00:00
*** cardeois has joined #openstack-infra00:03
*** Apoorva has quit IRC00:04
*** amitgandhinz has quit IRC00:05
*** cardeois has quit IRC00:10
kfox1111the job exited 0, but zuul then says the job completed with result FAILURE. any idea why that may happen?00:11
fungi(phrased in the form of a url)00:11
*** cardeois has joined #openstack-infra00:11
fungithis sounds similar to the issue mwhahaha linked. we're about to restart zuul with some extra debugging in place00:12
kfox1111ah. k.00:12
*** spzala has joined #openstack-infra00:13
*** tosky has quit IRC00:15
*** cardeois has quit IRC00:16
*** dfflanders has quit IRC00:17
jeblairkfox1111, fungi, mordred: that error is interesting because while it does include the error we're working on addressing, the module_stdout includes a lot of broadcast messages from systemd-journal:
*** spzala has quit IRC00:18
jeblairthat may be a second problem.  i wonder if it will cause ansible to fail to parse the output of the module it runs even after we fix the first error.00:18
kfox1111looks like a bunch of messages from the ceph the job sets up.00:19
kfox1111that part of the job hasn't changed in a while.00:20
kfox1111some recent change in the ansible code is looking at it and choking maybe?00:21
jeblairyeah, we changed the way we run ansible; it's not immediately apparent to me what's going on though00:21
jeblairoh, i think this might be okay00:23
*** tphummel has joined #openstack-infra00:23
jeblairi think we're only getting that output now because the correct exception handler isn't being run.  if it is, or if there is no exception, there should be no stdout/stderr in the ansible output.00:24
jeblairso i think that's a red herring00:24
jeblairwe're probably just looking at the same problem again as fungi said00:24
*** armax has joined #openstack-infra00:25
*** spzala has joined #openstack-infra00:26
jeblairrestarting now00:30
kfox1111should I resubmit then?00:31
kfox1111or wait for a bit?00:31
jeblairkfox1111: not yet00:31
*** spzala has quit IRC00:32
kfox1111k. I'm going to head home now. I'll check back later.00:32
kfox1111Thanks for the help.00:32
jeblairkfox1111: np, sorry for the inconvenience00:32
kfox1111no worries. thanks, as always for all the hard work you do. :)00:33
*** mtanino has quit IRC00:38
*** snarwade has quit IRC00:40
*** kashyap has quit IRC00:40
jeblairrestart complete00:41
*** Julien-zte has joined #openstack-infra00:41
*** Julien-zte has quit IRC00:41
*** Julien-zte has joined #openstack-infra00:42
*** edmondsw has quit IRC00:46
*** xarses has joined #openstack-infra00:47
*** david-lyle_ has joined #openstack-infra00:51
*** Julien-zte has quit IRC00:52
*** baoli has joined #openstack-infra00:53
*** Julien-zte has joined #openstack-infra00:53
*** david-lyle has quit IRC00:54
*** gouthamr has joined #openstack-infra00:55
*** ijw has quit IRC00:56
*** ijw has joined #openstack-infra00:56
*** sree has joined #openstack-infra00:56
*** spzala has joined #openstack-infra00:59
*** sree has quit IRC01:00
*** amitgandhinz has joined #openstack-infra01:01
openstackgerritRamy Asselin proposed openstack-infra/puppet-bandersnatch: Fix bandersnatch crons to support full sync
*** zz_dimtruck is now known as dimtruck01:02
asselinclarkb, finally got back to this ^^01:02
*** pahuang has quit IRC01:04
*** sflanigan has quit IRC01:05
*** armax has quit IRC01:06
*** ijw has quit IRC01:06
*** ijw has joined #openstack-infra01:06
*** armax has joined #openstack-infra01:06
*** armax has quit IRC01:08
*** armax has joined #openstack-infra01:10
*** dimtruck is now known as zz_dimtruck01:12
*** Apoorva_ has quit IRC01:12
*** armax has quit IRC01:14
*** Julien-zte has quit IRC01:15
*** Julien-zte has joined #openstack-infra01:15
*** chandanc has joined #openstack-infra01:17
*** gyee has quit IRC01:18
*** zhurong has joined #openstack-infra01:18
*** ijw has quit IRC01:19
*** ijw has joined #openstack-infra01:19
*** sflanigan has joined #openstack-infra01:20
*** sflanigan has joined #openstack-infra01:20
*** pahuang has joined #openstack-infra01:21
*** chandanc has quit IRC01:23
*** ijw has quit IRC01:24
*** thorst_ has quit IRC01:24
*** hongbin has joined #openstack-infra01:25
*** thorst_ has joined #openstack-infra01:25
*** ijw has joined #openstack-infra01:26
*** spzala has quit IRC01:26
*** sdake has quit IRC01:27
*** yanyanhu has joined #openstack-infra01:27
*** Julien-zte has quit IRC01:27
sc`for anyone that was following along about, a point in time snapshot weighs in at 265gb. no idea what a regular delta would be01:29
*** sdake has joined #openstack-infra01:29
*** mriedem_away is now known as mriedem01:33
*** thorst_ has quit IRC01:33
*** zz_dimtruck is now known as dimtruck01:35
*** amitgandhinz has quit IRC01:35
*** gildub has joined #openstack-infra01:36
*** kaisers_ has joined #openstack-infra01:37
*** maeker has quit IRC01:38
*** Julien-zte has joined #openstack-infra01:38
*** Julien-zte has quit IRC01:38
*** Julien-zte has joined #openstack-infra01:39
*** sdake has quit IRC01:40
*** mtanino has joined #openstack-infra01:41
*** kaisers_ has quit IRC01:41
*** ijw has quit IRC01:47
*** ijw has joined #openstack-infra01:47
*** r-daneel has quit IRC01:48
fungithat sounds on par with a pypi mirror01:48
*** aspiers has quit IRC01:53
*** ijw has quit IRC01:54
*** armax has joined #openstack-infra02:00
*** ijw has joined #openstack-infra02:03
*** yamamot__ has joined #openstack-infra02:05
*** yamamoto has quit IRC02:05
*** tqtran has quit IRC02:06
*** aspiers has joined #openstack-infra02:06
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Retire neutron-pd-driver
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Complete retirement for neutron-pd-driver
*** ijw has quit IRC02:11
*** sflanigan has quit IRC02:13
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Complete retirement for neutron-pd-driver
*** thorst_ has joined #openstack-infra02:14
*** thorst_ has quit IRC02:14
*** ijw has joined #openstack-infra02:14
*** thorst_ has joined #openstack-infra02:14
*** armax has quit IRC02:15
*** mriedem has quit IRC02:17
*** ijw has quit IRC02:18
*** thorst_ has quit IRC02:23
*** sflanigan has joined #openstack-infra02:25
*** pahuang has quit IRC02:26
openstackgerritMerged openstack-infra/irc-meetings: Remove old weekly ArchWG meeting duplicate
*** spzala has joined #openstack-infra02:27
*** amitgandhinz has joined #openstack-infra02:32
*** thorst_ has joined #openstack-infra02:32
*** thorst_ has quit IRC02:33
*** pahuang has joined #openstack-infra02:38
*** thorst_ has joined #openstack-infra02:39
*** thorst_ has quit IRC02:39
*** spzala has quit IRC02:43
*** baoli has quit IRC02:45
*** yuanying has quit IRC02:47
*** chandanc has joined #openstack-infra02:49
tonybDoes anyone know about "[Zuul] standard output/error still open after child exited.* | [Zuul] Task exit code: 0" failures?02:55
jamielennoxis there a reason that shade is not in g-r?02:56
tonybAs far as I can tell the devsatck/grenade/tox has passed but due to the message above the job fails02:57
tonybjamielennox: It's not used by anythign that cares about co-installability?02:57
tonybthe errors seems to have started about Oct 11:
jamielennoxtonyb: it's just interesting that shade can be top of the tree when there are ansible modules and a whole bunch of infra stuff that uses it02:58
*** yamahata has quit IRC02:59
openstackgerritIan Wienand proposed openstack/diskimage-builder: Don't set tracing in environment files
openstackgerrityatin proposed openstack-infra/project-config: Add diskimage-builder to project list
tonybjamielennox: I don;t have an objecttion but my limited understanding suggests that it wasn't really intented to be used inside OpenStack.  Isn't the point of it multi-cloud compat which I'm not certain we handle within an OpenStack03:02
*** knangia has quit IRC03:02
jamielennoxtonyb: ok, it was just interesting from an install from devstack perspective - it would seem like a conscious choice to not have it in03:03
jamielennoxbut yea, i guess nothing that is using it is being tested on openstack infra03:04
*** amitgandhinz has quit IRC03:06
tonybjamielennox: yeah.03:07
*** Julien-zte has quit IRC03:10
*** kashyap has joined #openstack-infra03:13
*** sdake has joined #openstack-infra03:14
*** rfolco has joined #openstack-infra03:17
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add -{node} to dsvm job names
*** reed_ has joined #openstack-infra03:21
*** reed_ has quit IRC03:23
*** david-lyle_ has quit IRC03:23
*** david-lyle has joined #openstack-infra03:23
*** kaisers_ has joined #openstack-infra03:25
*** dave-mccowan has quit IRC03:27
*** mountpoint has quit IRC03:27
tonybgiven this failure: does that imply that the rm of /etc/sudoers.d/jenkins-sudo failed and the job aborted?03:27
*** vikrant has joined #openstack-infra03:27
* tonyb may have used his $random_questions budget03:27
*** coolsvap has joined #openstack-infra03:29
*** kaisers_ has quit IRC03:30
*** ramishra has quit IRC03:31
openstackgerritJamie Lennox proposed openstack-infra/shade: Allow setting env variables for functional options
*** ramishra has joined #openstack-infra03:32
*** ramishra_ has joined #openstack-infra03:35
*** ramishra has quit IRC03:37
*** thorst_ has joined #openstack-infra03:40
mgagnetonyb: looks like it's running on which might not be ephemeral and sudo removed already or non-existant03:43
*** krtaylor has joined #openstack-infra03:47
*** thorst_ has quit IRC03:48
*** yuanying has joined #openstack-infra03:49
*** yamahata has joined #openstack-infra03:52
*** vikrant is now known as vikrant|brb03:53
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Introduce experimental dsvm jobs with xenial
*** sree has joined #openstack-infra03:56
*** Julien-zte has joined #openstack-infra03:57
*** Julien-zte has quit IRC04:00
*** Julien-z_ has joined #openstack-infra04:00
*** sree has quit IRC04:00
*** tuanluong has joined #openstack-infra04:01
*** amitgandhinz has joined #openstack-infra04:02
*** hongbin has quit IRC04:03
*** tuanluong has quit IRC04:03
*** rfolco has quit IRC04:03
*** tuanluong has joined #openstack-infra04:05
*** mtanino has quit IRC04:05
*** Julien-z_ has quit IRC04:06
*** Julien-zte has joined #openstack-infra04:06
*** sdake has quit IRC04:07
*** Julien-zte has quit IRC04:08
*** Julien-z_ has joined #openstack-infra04:08
openstackgerritJamie Lennox proposed openstack-infra/shade: Add a devstack plugin for shade
*** sdake has joined #openstack-infra04:13
*** maeker has joined #openstack-infra04:16
*** csomerville has quit IRC04:19
*** csomerville has joined #openstack-infra04:19
openstackgerritIan Wienand proposed openstack/diskimage-builder: Turn down yum install-packages
*** netsin has quit IRC04:22
*** links has joined #openstack-infra04:23
*** cody-somerville has joined #openstack-infra04:23
*** cody-somerville has joined #openstack-infra04:23
*** csomerville has quit IRC04:25
*** chandanc has quit IRC04:25
*** sdake has quit IRC04:26
*** links has quit IRC04:27
*** pgadiya has joined #openstack-infra04:28
*** ijw has joined #openstack-infra04:31
ramishra_ianw: hey around?04:34
*** ijw has quit IRC04:35
ramishra_ianw:, it seems all grenade jobs are failing.04:36
openstackLaunchpad bug 1635111 in heat "grenade jobs are failing with 'standard output/error still open after child exited'" [Undecided,New]04:36
*** amitgandhinz has quit IRC04:37
ramishra_not sure, but looks like this is due some recent zuul changes.04:37
ramishra_fungi: hi ^^^04:38
openstackgerritGhanshyam Mann proposed openstack-infra/devstack-gate: DNM: For Debugging only
*** spzala has joined #openstack-infra04:44
*** thorst_ has joined #openstack-infra04:48
*** ijw has joined #openstack-infra04:48
*** spzala has quit IRC04:48
*** markvoelker_ has quit IRC04:49
*** ijw has quit IRC04:53
*** thorst_ has quit IRC04:54
*** rajinir has quit IRC04:56
*** baoli has joined #openstack-infra04:57
*** baoli has quit IRC05:01
*** yamamot__ has quit IRC05:04
*** jaosorior has joined #openstack-infra05:09
*** caowei has joined #openstack-infra05:14
*** kaisers_ has joined #openstack-infra05:15
*** kaisers_ has quit IRC05:19
*** maeker has quit IRC05:20
*** bhavik has joined #openstack-infra05:24
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: don't close stdout in command module
*** bhavik has quit IRC05:27
jeblairramishra_, kfox1111, mwhahaha, fungi, mordred, ianw, infra-root: ^ my earlier change to fix the get_exception error ( ) revealed this error: "close() called during concurrent operation on the same file object." ( ).  i *think* that ...05:29
jeblair... will fix the problem.  however, i am not able to restart the launchers with that change now.05:29
*** markvoelker_ has joined #openstack-infra05:30
jeblairinfra-root: however, if the next person able would like to land that and restart, (or else land a revert change of the command module work) that would be great.05:30
ramishra_jeblair: I can do the revert if that helps.05:32
*** amitgandhinz has joined #openstack-infra05:33
openstackgerritRabi Mishra proposed openstack-infra/zuul: Revert "Ansible launcher: import get_exception in ansible command"
jeblairramishra_: mordred prepared though it is now out of date.  if you want to update it that might be helpful (but the root that decides to restart may decide to push forward rather than backward)05:34
jeblairramishra_: oh, that commit isn't the problem -- that commit just allowed us to actually *see* the problem05:35
jeblairramishra_: the commit that created the problem is Iae4769f923ecf74462e1fe43168ea93ff1c61d6e  (but probably all of those commits should be reverted because they were all tested together)05:36
ramishra_jeblair: yeah I undersatsand that, I thought we would revert it and then fix the issue and merge itagain.05:36
ramishra_jeblair: would not that help?05:37
jeblairramishra_: no, if we wanted to revert, we would do something like  (but as i said, it needs updating)05:38
sc`with the magic number for known, what would need to happen in order to get some momentum on ?05:38
ramishra_jeblair: ok, sorry, I've little understanding of these things. Ok, I'll try and push a change to revert all command module related changes.05:39
*** jtomasek has quit IRC05:39
jeblairanyway, i'm sorry i have to go (it's quite late here and it takes about 30-60 minutes to restart the launchers).  hopefully an infra-root in sunlight can fix this soon.05:40
sc`crap. it _is_ late. enough about computers for the night o/05:42
*** asselin has quit IRC05:45
*** yamamoto has joined #openstack-infra05:45
*** asselin__ has joined #openstack-infra05:46
*** markvoelker_ has quit IRC05:46
*** camunoz has quit IRC05:48
openstackgerritSamuel Cassiba proposed openstack-infra/system-config: Added Gem Mirror to Infra
*** thorst_ has joined #openstack-infra05:52
*** hichihara has joined #openstack-infra05:53
*** thorst_ has quit IRC05:59
*** hurgleburgler has quit IRC06:01
*** aeng has quit IRC06:01
*** amitgandhinz has quit IRC06:07
*** dimtruck is now known as zz_dimtruck06:09
*** Naeil has joined #openstack-infra06:11
mordredtonybm jamielennox: I don't think there are any issues with shade being in g-r - but also what you said is accurate, it's not intended to be used _by_ openstack as much as it's intended to be used _on_ openstack06:12
*** zz_dimtruck is now known as dimtruck06:12
mordredjeblair: dude. you were up way too late06:14
mordredjeblair: I think I'm caught up on scrollback now06:14
*** e0ne has joined #openstack-infra06:15
mordredinfra-root: it's 1am, so I'm not going to start a launcher restart since it's unlikely it'll finish before I fall asleep. I did approve jeblair's patch because I think rolling it out before reverting is the right step to take. I'll run a restart first thing in the morning if nobody has beaten me to it06:16
mordredbut as soon as and puppet rolls it out to launchers, running the restart playbook should be fine06:17
mordredthere is a copy of it in ~root on puppetmaster - ls -ltra should show it to you06:17
*** chandankumar has joined #openstack-infra06:18
*** pcaruana has joined #openstack-infra06:18
openstackgerritMerged openstack-infra/zuul: Ansible launcher: don't close stdout in command module
*** tqtran has joined #openstack-infra06:19
*** yolanda has quit IRC06:20
*** andreas_s has joined #openstack-infra06:20
*** tqtran has quit IRC06:23
*** e0ne has quit IRC06:29
*** e0ne has joined #openstack-infra06:29
jaosoriormordred: any estimate on how long will it take for this to come into effect ?06:30
*** tphummel has quit IRC06:32
*** florianf has joined #openstack-infra06:33
*** kaisers_ has joined #openstack-infra06:35
*** nherciu has joined #openstack-infra06:36
*** sree has joined #openstack-infra06:37
*** migi_ is now known as migi06:38
*** vsaienko has joined #openstack-infra06:38
*** sputnik13 has joined #openstack-infra06:40
*** yanyanhu has quit IRC06:42
*** vsaienko has quit IRC06:42
*** vsaienko has joined #openstack-infra06:43
*** zhurong_ has joined #openstack-infra06:45
*** zhurong has quit IRC06:45
*** tphummel has joined #openstack-infra06:46
mordredjaosorior: we're waiting on someone to be awake enough to run a zuul-launcher rolling restart06:47
*** yanyanhu has joined #openstack-infra06:54
*** aviau has quit IRC06:55
*** tphummel has quit IRC06:55
*** aviau has joined #openstack-infra06:55
*** dimtruck is now known as zz_dimtruck06:56
*** vsaienko has quit IRC06:57
*** thorst_ has joined #openstack-infra06:58
*** yanyanhu_ has joined #openstack-infra07:00
*** martinkopec has joined #openstack-infra07:02
*** gildub has quit IRC07:02
*** yanyanhu has quit IRC07:04
*** amitgandhinz has joined #openstack-infra07:04
*** thorst_ has quit IRC07:04
openstackgerritMasayuki Igawa proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow
*** automagically has quit IRC07:06
*** automagically has joined #openstack-infra07:06
*** flepied has quit IRC07:08
*** pahuang has quit IRC07:09
*** tqtran has joined #openstack-infra07:10
*** dtardivel has joined #openstack-infra07:11
*** tesseract has joined #openstack-infra07:11
*** tesseract is now known as Guest1406907:12
openstackgerritIsaku Yamahata proposed openstack-infra/project-config: networking-odl: use ubuntu-xenial for newton+
openstackgerritIsaku Yamahata proposed openstack-infra/project-config: networking-odl: add periodic tempest job for stable branches
*** claudiub has joined #openstack-infra07:16
*** vsaienko has joined #openstack-infra07:19
*** amoralej|off is now known as amoralej07:21
openstackgerritIsaku Yamahata proposed openstack-infra/project-config: networking-odl: add periodic tempest job for stable branches
*** tkelsey has joined #openstack-infra07:21
*** esikachev has joined #openstack-infra07:22
*** tnovacik has quit IRC07:25
*** oanson has joined #openstack-infra07:26
*** amitgandhinz has quit IRC07:27
*** ijw has joined #openstack-infra07:27
*** spzala has joined #openstack-infra07:29
*** spzala has quit IRC07:34
*** jpena|off is now known as jpena07:37
*** yaume has joined #openstack-infra07:37
*** vsaienko has quit IRC07:39
*** ijw has quit IRC07:41
*** Julien-z_ has quit IRC07:50
*** Julien-zte has joined #openstack-infra07:51
ianwummm, let me see ...07:52
*** vsaienko has joined #openstack-infra07:53
*** flepied has joined #openstack-infra07:53
*** flepied has quit IRC07:53
*** flepied1 has joined #openstack-infra07:54
*** sdake has joined #openstack-infra07:54
*** jordanP has joined #openstack-infra07:54
*** sdake has quit IRC07:55
*** sdake has joined #openstack-infra07:56
ianwjeblair / mordred / anyone: well, i'm running the playbook in a root screen session.  it seems to be working07:57
*** yamahata has quit IRC07:57
*** yanyanhu_ has quit IRC07:57
*** pilgrimstack has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** ijw has joined #openstack-infra08:00
*** vsaienko has quit IRC08:03
*** thorst_ has joined #openstack-infra08:03
*** Julien-zte has quit IRC08:04
*** jpich has joined #openstack-infra08:04
ianwp.s. i did check 388936 had rolled out, seems to be there on the launchers08:05
openstackgerritMasayuki Igawa proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow
*** Julien-zte has joined #openstack-infra08:05
*** ijw has quit IRC08:05
*** sree_ has joined #openstack-infra08:05
*** yatinkarel has joined #openstack-infra08:05
*** sree_ is now known as Guest7133608:06
*** david-lyle_ has joined #openstack-infra08:07
*** e0ne has quit IRC08:08
*** sree has quit IRC08:08
*** oanson has quit IRC08:09
*** david-lyle has quit IRC08:09
*** thorst_ has quit IRC08:09
*** mhickey has joined #openstack-infra08:11
*** qwertyco has joined #openstack-infra08:11
*** yanyanhu_ has joined #openstack-infra08:14
*** dizquierdo has joined #openstack-infra08:15
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add Barbican key order to scenario002
*** markvoelker has joined #openstack-infra08:19
*** sflanigan has quit IRC08:19
*** ccamacho|afk is now known as ccamacho08:19
*** tqtran has quit IRC08:21
*** amitgandhinz has joined #openstack-infra08:23
openstackgerritMasayuki Igawa proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow
*** dingyichen has quit IRC08:28
openstackgerritWaldemar Znoinski proposed openstack-infra/project-config: gerrit: add intel-nfv-ci-tests-ci group
*** ijw has joined #openstack-infra08:33
*** flepied1 is now known as flepied08:34
*** Julien-zte has quit IRC08:35
*** r1chardj0n3s has joined #openstack-infra08:36
*** Julien-zte has joined #openstack-infra08:36
r1chardj0n3shi folks. We're seeing pretty consistent failures in the gate for Horizon jobs, related to xvfb, I think. An example:
*** vsaienko has joined #openstack-infra08:38
*** ijw has quit IRC08:38
*** tosky has joined #openstack-infra08:39
openstackgerritMerged openstack/os-client-config: Update ECS image_api_version to 1
docaedor1chardj0n3s: Looks to me like the gate is broken (zuul change), bug here:
openstackLaunchpad bug 1635111 in Zuul "All grenade jobs are failing with error" [High,New]08:41
r1chardj0n3sthanks docaedo08:41
*** hichihara has quit IRC08:41
vsaienk0infra-team, could you please help with all tests are passed but the job is failed. It looks like it was caused by timeout, but it is strange job timeout is 180 min, and it took near 60 min so I'm confused08:41
fricklerr1chardj0n3s: docaedo: seems is the fix and iiuc ianw has just rolled it out, so maybe try a recheck08:42
r1chardj0n3sthanks frickler, will give it a go08:42
*** spzala has joined #openstack-infra08:43
*** tosky has joined #openstack-infra08:45
*** electrofelix has joined #openstack-infra08:45
*** derekh has joined #openstack-infra08:46
*** spzala has quit IRC08:47
*** pblaho has joined #openstack-infra08:48
*** vsaienko has quit IRC08:48
*** Julien-zte has quit IRC08:48
*** Julien-zte has joined #openstack-infra08:49
*** otherwiseguy has quit IRC08:50
ianwfrickler / r1chardj0n3s : it's rolling out ... i'm not sure how long to wait for each zuul-launcher to stop being my first time doing this ... i'll give it a bit before i manually intervene08:50
*** Julien-zte has joined #openstack-infra08:50
r1chardj0n3sok thanks ianw08:50
*** jbernard has quit IRC08:51
*** jbernard has joined #openstack-infra08:51
docaedoianw: thanks - at least one patch that was failing earlier for me (with the sudo issue) is good now08:51
*** otherwiseguy has joined #openstack-infra08:52
*** esikachev has quit IRC08:54
*** woodster_ has quit IRC08:55
*** dtantsur|sick is now known as dtantsur08:56
ianwjeblair / mordred : ok, i had to intervene in zl06 & zl02 which seemed to get stuck.  i'll leave the screen session on puppetmaster.  monitoring for now, but otherwise zl0[1-7] report they restarted ok08:57
*** amitgandhinz has quit IRC08:57
ajafohi, guys, who can help us with we need to add cross-repos core group or first core to the group?08:59
rcarrillocruzajafo: fuel-ccp-ceph-core has now fule-ccp-core included09:01
ajaforcarrillocruz: thanks09:01
*** Julien-zte has quit IRC09:02
*** Julien-zte has joined #openstack-infra09:03
*** e0ne has joined #openstack-infra09:03
*** Rockyg has quit IRC09:06
*** thorst_ has joined #openstack-infra09:07
*** sambetts|afk is now known as sambetts09:10
*** sdake has quit IRC09:11
*** jordanP has quit IRC09:12
*** jordanP has joined #openstack-infra09:13
*** tnovacik has joined #openstack-infra09:13
*** Julien-zte has quit IRC09:13
therveGrenade heat gate is failing bizarrely:
therveThe only error I can see is "standard output/error still open after child exited", does that remind somebody something?09:13
*** Julien-zte has joined #openstack-infra09:14
*** thorst_ has quit IRC09:14
docaedotherve: believe was the fix, which ianw has been rolling out to the zuul launches09:15
docaedotherve: also
openstackLaunchpad bug 1635111 in Zuul "All grenade jobs are failing with error" [High,New]09:15
therveAh yeah that's the one09:16
thervedocaedo, Is the fix taking some time to roll out?09:16
*** panda|Zz is now known as panda09:18
*** ihrachys has joined #openstack-infra09:19
docaedotherve: I think that last response on 388723 from an hour ago is right about when the fix was rolling out - I had 5 patches that passed re-check in the last half hour09:20
thervedocaedo, Ok, thanks a lot09:20
docaedotherve: no prob09:20
*** vsaienko has joined #openstack-infra09:21
*** jtomasek_ has joined #openstack-infra09:21
*** john-davidge has joined #openstack-infra09:22
*** mfedosin has joined #openstack-infra09:22
*** oanson has joined #openstack-infra09:23
*** john-davidge has quit IRC09:24
*** john-davidge has joined #openstack-infra09:24
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add Barbican key order to scenario002
gmannAJaeger: for you -
*** matrohon has joined #openstack-infra09:34
*** jtomasek_ is now known as jtomasek09:35
*** zhurong_ has quit IRC09:35
*** Hal has quit IRC09:38
*** Hal has joined #openstack-infra09:38
*** markvoelker has quit IRC09:39
*** spzala has joined #openstack-infra09:41
*** derekh has quit IRC09:43
pandadoes anyonw has any ides why build-timeout wrapper for tripleo job-template is not working ? all our jobs are started with a timeout of 110 instead of 18009:43
pandaanyone* idea*09:43
*** jaosorior has quit IRC09:44
*** r1chardj0n3s has left #openstack-infra09:44
*** jaosorior has joined #openstack-infra09:44
pabelangerpanda: it is possible there is a regression in our most recent zuul update09:45
pabelangerpanda: do you have a log file?09:45
*** spzala has quit IRC09:45
pandapabelanger: is this enough ?
*** derekh has joined #openstack-infra09:47
pandapabelanger: were you looking for a different type of log ?09:47
pabelangerpanda: that works, thanks09:47
pandapabelanger: all tripleo jobs are affected09:47
pabelangerI think we are no longer setting the timeout environment variable properly09:47
pabelangerwill know more in a few mins09:48
*** caowei has quit IRC09:49
pandapabelanger: thanks09:50
*** tuanluong has quit IRC09:50
pabelangerYa, looks like we are not longer exposing it09:53
pabelangerworking on a patch09:53
*** amitgandhinz has joined #openstack-infra09:53
pandapabelanger: great :)09:54
*** Julien-zte has quit IRC09:55
amoraleji'm still getting jobs failing with "standard output/error still open after child exited ..." is it expected?09:55
*** caowei has joined #openstack-infra09:56
pandaamoralej: is it the error just at the end of the job?09:56
*** ssbarnea has joined #openstack-infra09:56
amoralejjobs seems to have run ok, but i'm getting failure09:57
amoralejafter two rechecks i reduced from 4 failures to 2 failures, but still getting them09:57
amoralejthis is from 10 minutes ago09:57
pandaamoralej: it happens also when job succeeds, look pretty harmless error.09:58
*** caowei has quit IRC09:58
*** amotoki has joined #openstack-infra09:58
amoralejyeah, i'm not worried about the error message but about the job false failure09:59
ianwamoralej: it's not ?09:59
amoralejthat 2 is expected09:59
amoralejfrom a successfull one
*** Guest71336 has quit IRC10:03
ianwamoralej: hmm, interesting ... it seems that the zuul change 389009 is *not* rolled out on zl0110:04
slaglei'm seeing an error during the ansible run at the end of otherwise successful jobs that is causing them to fail:
openstackgerritJordan Pittier proposed openstack-infra/shade: Logging: avoid string interpolation when not needed
slagleamoralej: looks like the same thing you might be seeing10:04
ianwslagle: yeah, that's also zl01 launcher10:05
ianwi wonder if it's not puppeting10:05
*** ociuhandu has quit IRC10:06
amoralejok ianw , let me know when change is rolled out on it and i'll recheck10:06
*** lezbar has quit IRC10:08
*** winggundamth has quit IRC10:08
*** ldnunes has joined #openstack-infra10:09
*** winggundamth has joined #openstack-infra10:10
*** yanyanhu_ has quit IRC10:10
*** lezbar has joined #openstack-infra10:11
*** thorst_ has joined #openstack-infra10:13
openstackgerritPaul Belanger proposed openstack-infra/zuul: Add back timeout_var logic
pabelangerpanda: mordred: ^ that will fix devstack-gate timeout issues10:14
*** vsaienko has quit IRC10:19
*** thorst_ has quit IRC10:19
pandapabelanger: thanks, that was fast :)10:19
*** degorenko|afk is now known as degorenko10:20
*** dizquierdo is now known as dizquierdo_afk10:20
*** sflanigan has joined #openstack-infra10:22
pandapabelanger: does that change means that there is a different way to get a devstack timeout now ?10:22
*** ralonsoh has joined #openstack-infra10:22
pandapabelanger: I mean this one Ie51de4a135d953c4ad9dcb773d27b3c54ca8829b that removed the timeout_var10:22
*** jordanP has quit IRC10:27
*** amitgandhinz has quit IRC10:27
*** rossella_s has quit IRC10:28
*** rossella_s has joined #openstack-infra10:28
*** TomazVieira has quit IRC10:29
openstackgerritMerged openstack-infra/release-tools: ensure version numbers are always strings
*** derekh has quit IRC10:31
openstackgerritMerged openstack-infra/release-tools: ensure all proposed versions include major.minor.patch values
*** tlian has quit IRC10:31
*** gildub has joined #openstack-infra10:33
ianwTASK [puppet : set log filename] ***********************************************10:34
ianwERROR! Unexpected Exception: [Errno 12] Cannot allocate memory10:34
ianwto see the full traceback, use -vvv10:34
ianwpabelanger: ^ that doesn't look promising10:34
*** derekh has joined #openstack-infra10:34
ianwpabelanger: yeah, i think puppet run's aren't getting all the way through, that's in puppetmaster puppet_run_all_cron.log10:35
ianw[31016926.950503] Killed process 1794 (ansible-playboo) total-vm:1163856kB, anon-rss:122312kB, file-rss:252kB10:36
ianwumm, am i nuts or is it a 2gb host?10:37
*** markvoelker has joined #openstack-infra10:37
*** ijw has joined #openstack-infra10:38
rcarrillocruzit's a long standing item to replace it iirc10:39
ianwrcarrillocruz: i think i know what you're doing today :)10:40
* rcarrillocruz walks away misteriously10:40
ianwi'm not sure there's much to do other than upsize it10:40
ianwi'm manually running puppet for zl* hosts now ... hopefully just running on the reduced host list will be ok10:41
ianwthat should hopefully redeploy zuul on zl01, which i'll restart, and fix up the last of these issues10:41
*** ijw has quit IRC10:43
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add Barbican key order to scenario002
*** amotoki has quit IRC10:46
ianwahhh, zl01 is in the emergency file!  that's not helping10:47
rcarrillocruzi'm looking at cacti, the is broken, it pulls fields that are present on vanilla which are not for some reason on chocolate10:50
rcarrillocruzwhich makes the whole tree of chocolate to not appear10:50
*** jkilpatr has quit IRC10:51
ianwpabelanger / jeblair / mordred : i think i'm going to tap out.  i'm assuming one of you put zl01 in the emergency file, so 389009 is not applied there.  although it would probably be fine, i'm not up for debugging it ATM should it explode if i manually run.  otherwise change is rolled out10:51
*** gmann__ has joined #openstack-infra10:51
ianwpabelanger / jeblair / mordred : and yeah, the oom's cause by ansible-playbook on puppetmaster probably need corrective action ... looks like the cron job hits it frequently10:52
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Fix iface variable name on loop
rcarrillocruzianw: i believe mordred did (the placement of zl01 on emergency)10:52
*** ociuhandu has joined #openstack-infra10:55
*** tnovacik has quit IRC10:57
*** liusheng has quit IRC10:58
*** dprince has joined #openstack-infra10:58
rcarrillocruzbtw, i've added cacti01 to emergency, till we land ^10:59
rcarrillocruzas i put the hotfix on the script10:59
rcarrillocruzotherwise puppet will overwrite it10:59
*** Julien-zte has joined #openstack-infra11:02
*** dizquierdo_afk is now known as dizquierdo11:02
*** Rockyg has joined #openstack-infra11:03
rcarrillocruzthx ianw , i'll approve now11:05
*** kairat has joined #openstack-infra11:06
ianw#status jobs launched by (check console.html) may fail due to 38900911:06
*** markvoelker_ has joined #openstack-infra11:07
kairatfungi, hello, could you please look at It would allow us to test deployment of new app-catalog with Glare on staging.11:07
ianwalright, heading out for tonight, good luck all :)11:08
rcarrillocruzhave a good one ianw11:08
*** lucas-sick is now known as lucasagomes11:10
*** markvoelker has quit IRC11:11
openstackgerritMerged openstack-infra/system-config: Fix iface variable name on loop
*** jkilpatr has joined #openstack-infra11:14
*** thorst_ has joined #openstack-infra11:17
*** thorst_ has quit IRC11:17
*** sputnik13 has quit IRC11:17
*** thorst_ has joined #openstack-infra11:17
*** zeih has joined #openstack-infra11:19
*** admcleod- has joined #openstack-infra11:19
pabelangerpanda: Ya, we'll want to change the way we configure it for zuulv3, but that won't happen for a while yet11:20
pabelangerpanda: yes, we need to revert parts of Ie51de4a135d953c4ad9dcb773d27b3c54ca8829b11:21
pandapabelanger: ack thanks.11:21
pabelangerianw: that is, expected.  We have OOM issues on puppetmaster.o.o and need to deploy a new server11:21
*** mtanino has joined #openstack-infra11:23
*** amitgandhinz has joined #openstack-infra11:24
*** erlon has joined #openstack-infra11:29
*** claudiub|2 has joined #openstack-infra11:30
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge
*** claudiub has quit IRC11:33
*** baoli has joined #openstack-infra11:36
*** ccamacho is now known as ccamacho|lunch11:39
*** baoli has quit IRC11:40
*** qwertyco has quit IRC11:42
*** tiswanso has joined #openstack-infra11:42
*** baoli has joined #openstack-infra11:42
*** gouthamr has quit IRC11:43
*** krtaylor has quit IRC11:43
*** jpena is now known as jpena|lunch11:43
*** tiswanso has quit IRC11:47
*** dizquierdo has quit IRC11:47
*** nicolasbock has joined #openstack-infra11:49
*** ssbarnea has quit IRC11:51
*** ssbarnea has joined #openstack-infra11:52
*** gouthamr has joined #openstack-infra11:55
openstackgerritJens Rosenboom proposed openstack-infra/system-config: Added Gem Mirror to Infra
*** tiswanso has joined #openstack-infra11:58
*** amitgandhinz has quit IRC11:58
*** gouthamr has quit IRC11:59
*** baoli_ has joined #openstack-infra12:00
*** gouthamr has joined #openstack-infra12:00
*** gildub has quit IRC12:00
AJaegerpabelanger: arrived well in Paris?12:01
*** baoli has quit IRC12:03
*** tiswanso_ has joined #openstack-infra12:05
*** tiswanso has quit IRC12:07
*** edmondsw has joined #openstack-infra12:08
*** zhurong has joined #openstack-infra12:08
*** amoralej is now known as amoralej|lunch12:08
*** markvoelker has joined #openstack-infra12:12
*** tiswanso has joined #openstack-infra12:13
*** tiswanso_ has quit IRC12:14
openstackgerritMerged openstack-infra/project-config: Add diskimage-builder to project list
*** oanson has quit IRC12:15
*** markvoelker_ has quit IRC12:16
*** EricGonczer_ has joined #openstack-infra12:16
*** tiswanso_ has joined #openstack-infra12:19
*** tiswanso has quit IRC12:21
pabelangerAJaeger: yes, in the Red Hat office now12:22
pabelangerrcarrillocruz: crinkle: I've added a few more slides to our presentation12:22
*** weshay is now known as weshay_pto12:22
rcarrillocruzk, got to copy paste stuff to the one crinkle shared12:22
rcarrillocruzi made a copy of it for my drafting12:22
*** dansmith has quit IRC12:23
AJaegerpabelanger: enjoy!12:23
*** tiswanso has joined #openstack-infra12:23
*** kgiusti has joined #openstack-infra12:24
*** amitgandhinz has joined #openstack-infra12:25
*** trown|outtypewww is now known as trown12:25
crinklepabelanger: rcarrillocruz cool12:25
*** EricGonczer_ has quit IRC12:25
*** tiswanso_ has quit IRC12:26
rcarrillocruzi think we should meet on monday to sync up and prepare it12:26
rcarrillocruzcrinkle, pabelanger ^12:26
pabelangerrcarrillocruz: crinkle: that works for me, assuming everybody here on Monday12:27
pabelangerotherwise, we could get into pbx.o.o again tomorrow12:28
crinklepabelanger: rcarrillocruz i won't be in till late on monday12:28
pabelangerk, how about we sync up first thing tomorrow?12:29
crinkleworks for me12:29
pabelangerI'm in UTC+2 right now FYI12:30
*** abregman has joined #openstack-infra12:33
*** tiswanso has quit IRC12:34
*** Jeffrey4l has quit IRC12:35
fricklerEmilienM: pabelanger: nibalizer: I made a fix for , please check whether I did the right thing, at least it passes jenkins now. would be great if we could this mirror running, there are lots of rubygem timeouts on chef jobs12:36
EmilienMfrickler: oh nice, thanks for helping on this12:37
*** amitgandhinz has quit IRC12:37
*** thorst_ has quit IRC12:37
pabelangerfrickler: EmilienM: I hope to spend some time at the summit working on this12:38
EmilienMpabelanger: thanks, I would be happy to helpo12:39
*** rlandy has joined #openstack-infra12:42
*** ccamacho|lunch is now known as ccamacho12:42
pabelangerEmilienM: frickler: -1, easy fix. Once that is updated, I think we can land the patch12:44
*** tnovacik has joined #openstack-infra12:44
*** zeih has quit IRC12:45
*** tlian has joined #openstack-infra12:45
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Mistral to scenario003
*** zeih has joined #openstack-infra12:46
openstackgerritJens Rosenboom proposed openstack-infra/system-config: Added Gem Mirror to Infra
fricklerpabelanger: EmilienM: ^^12:46
EmilienMfrickler: thx12:46
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002
*** jpena|lunch is now known as jpena12:48
*** gordc has joined #openstack-infra12:49
*** derekh has quit IRC12:50
*** mtanino has quit IRC12:51
*** zeih has quit IRC12:51
*** rhallisey has joined #openstack-infra12:51
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: Add new openssh-server element
*** rcernin has joined #openstack-infra12:53
*** jcoufal has joined #openstack-infra12:55
*** xavierr has joined #openstack-infra12:56
*** pgadiya has quit IRC12:56
xavierrgood morning Infra12:57
*** esikachev has joined #openstack-infra12:58
xavierrhey, I uploaded a new tag to python-oneviewclient and like always it should be upload automatically to pypi, however it was not. any ideas?12:59
*** jaosorior is now known as jaosorior_brb13:01
robcresswellxavierr: IIRC new tags aren't actually automatically processed. There's a next step that has to be triggered. Also this is a question for openstack-release, not openstack-infra, isnt it?13:01
*** bin_ has joined #openstack-infra13:02
*** gmann__ has quit IRC13:02
*** esikachev has quit IRC13:03
*** vikrant|brb has quit IRC13:03
fricklerso if I have a grenade error from zl01, should I just recheck or will than one be fixed soon, too?13:04
xavierrrobcresswell: people from Ironic told me to ask here, but ty anyways13:04
*** vsaienko has joined #openstack-infra13:04
mordredyah. I just removed zl01 from the emergency file - will get it updated real quickly13:05
mordredpabelanger, robcresswell: morning13:05
fricklermordred: cool, thx13:05
robcresswellxavierr: I think you might have more luck asking in openstack-release :)13:05
robcresswellmordred: \o13:05
*** Goneri has joined #openstack-infra13:07
AJaegerxavierr: right now something is broken, please wait with tagging until it's fixed - see
*** jistr is now known as jistr|biab13:08
*** mkoderer has joined #openstack-infra13:08
*** derekh has joined #openstack-infra13:09
*** sree has joined #openstack-infra13:09
pabelangermordred: o/13:10
pabelangerxavierr: I'll take a look13:11
pabelanger is the reason13:11
*** jistr|biab is now known as jistr13:11
*** yamamoto has quit IRC13:13
*** adrian_otto has joined #openstack-infra13:14
xavierrpabelanger: thanks, but there is anything I can do, or only guys from release can? :)13:15
pabelangerxavierr: trying to find out why it failed13:18
pabelangerit looks like the job timed out13:19
pabelangermordred: have we seen this yet?
pabelangerlooks like ansible could finish the file task13:20
*** mriedem has joined #openstack-infra13:20
*** mdrabe has joined #openstack-infra13:20
pabelangeractually, the command13:21
*** sdake has joined #openstack-infra13:21
AJaegerxavierr: nothing you can do right now, just wait. Once the underlying problem is fixed, we can discuss what to do - we might be able to reenqueu the job...13:21
mordredpabelanger: that's weird13:21
pabelangerlooks like the command ran, cause there is output in console.html13:22
pabelangerbut didn't exit properly13:22
*** nicolasbock has quit IRC13:23
xavierrAJaeger: ok, I'll wait until the end of this day :)13:23
xavierrthank you infra!13:23
*** florianf has quit IRC13:24
*** xavierr has left #openstack-infra13:25
*** nicolasbock has joined #openstack-infra13:27
njohnstonI get "Code Review - Error   500 Internal server error" when I try and cherry pick to stable/newton.  I had another person try and they got the same issue.  Does anyone know what could be causing this problem?13:27
*** matt-borland has joined #openstack-infra13:27
*** cardeois has joined #openstack-infra13:27
*** rfolco has joined #openstack-infra13:30
*** zhurong has quit IRC13:30
*** vsaienko has quit IRC13:30
*** dizquierdo has joined #openstack-infra13:31
fricklernjohnston: hmm, that looks strange, I'm getting that error too, but a local cherry-pick works just fine. I guess someone might have to check the logs on gerrit13:32
dhellmannrobcresswell, xavierr: we're seeing errors on the signing node with the jobs that run when a tag is being processed
dhellmannoh, AJaeger beat me to it ;-)13:33
*** billiebobthorty has joined #openstack-infra13:33
*** kjackal_ has joined #openstack-infra13:33
*** sdake has quit IRC13:33
njohnstonThanks for looking, frickler13:35
kjackal_Hi there, I get a "Cannot store contact information" so I cannot push anything for review. Any idea why?13:37
*** zhurong has joined #openstack-infra13:37
*** yamahata has joined #openstack-infra13:37
mordredpabelanger: I have restarted zl01 - it was in the emergency file last night so didn't get puppet updates to get the git repo updated13:38
*** amitgandhinz has joined #openstack-infra13:38
mordredpabelanger: I've also removed it from the emergency file, so it should be back in the fleet properly13:38
*** florianf has joined #openstack-infra13:38
AJaegerkjackal_: This needs your gerrit preferred e-mail address to match a primary e-mail address for a foundation individual member account.13:38
*** nicolasbock has quit IRC13:39
kjackal_AJaeger: thank you, let me try to parse that.13:39
AJaegerkjackal_: If you already followed the instructions (all, in order!) at and still get that, see for additional troubleshooting tips.13:39
*** hurgleburgler has joined #openstack-infra13:39
wznoinskhi infra, would someone have a moment for trivial gerrit group add review  ?13:42
*** amoralej|lunch is now known as amoralej13:43
wznoinskthanks rcarrillocruz13:45
*** EricGonczer_ has joined #openstack-infra13:46
openstackgerritMerged openstack-infra/project-config: gerrit: add intel-nfv-ci-tests-ci group
*** inc0 has joined #openstack-infra13:50
*** sree_ has joined #openstack-infra13:51
*** jheroux has joined #openstack-infra13:51
dhellmannpabelanger : is there anything to report on those signing job failures? it seems weird for it to be stuck on a call  to "rm" like that.13:51
*** sree_ is now known as Guest4960713:51
*** sdague has joined #openstack-infra13:51
*** sree has quit IRC13:53
*** vsaienko has joined #openstack-infra13:54
*** rfolco has quit IRC13:54
pabelangerdhellmann: not yet, still looking. We're executing a new code path, now that we removed async support. I suspect the rm command is successful, but ansible didn't parse the return code properly13:55
dhellmannpabelanger : ah, ok13:56
dhellmannwe're compiling a list of the tag jobs that will need to be re-run in
pabelangergood idea, I'm sure there as been more failures13:57
*** makowals has quit IRC13:57
* dhellmann nods13:58
*** makowals has joined #openstack-infra13:59
openstackgerritAnshul Jain proposed openstack/diskimage-builder: dib element to create customized image with cinder local attach-detach functionality.
*** esikachev has joined #openstack-infra14:01
*** spzala has joined #openstack-infra14:01
mordredpabelanger: AJaeger was saying something about sudo config being messed up on zlstatic perhaps14:01
*** nicolasbock has joined #openstack-infra14:01
AJaegermordred: not me - that was in the email I referenced14:02
pabelangerAJaeger: which email was that?14:02
*** EricGonczer_ has quit IRC14:02
*** adrian_otto has quit IRC14:02
mordredpabelanger: oh! are our requiretty settings not set up properly?14:03
*** sdague has quit IRC14:03
pabelangermordred: let me check14:03
mordredpabelanger: there's something about that relatde to the pipelining change - and I thought we'd verified our settings we correct, but maybe they aren't?14:03
*** yamamoto has joined #openstack-infra14:04
*** jaosorior_brb is now known as jaosorior14:05
*** EricGonczer_ has joined #openstack-infra14:05
*** EricGonczer_ has quit IRC14:06
*** EricGonczer_ has joined #openstack-infra14:06
AJaegermordred: see
mordredAJaeger: nod14:07
kashyapclarkb: fungi: Hi, just curious -- wonder is it possible to enable KVM nested virt (Intel / AMD) on the Kernels on Gate host? Perhaps for "limited machines"?14:08
*** ijw has joined #openstack-infra14:08
mordredkashyap: it is not, sorry14:08
kashyapmordred: I realize, you have to elaborate more than that...14:08
kashyapmordred: Is the fear that it "breaks the world"?14:08
kashyapThat distro kernels haven't enabled it?  Stability concern?14:08
mordredwell, that's part of it - and it's not fear, the times in the past it's been enabled it has in fact broken the world14:09
mordredbut more importantly - we do not run the clouds we use for the gate14:09
mordredand it requires enablement at the cloud provider level14:09
mordredmost of our clouds do not have it enabled - and some cannot provide it in the first place14:09
pabelangermordred: requiretty looks good, but we could add Defaults:jenkins !requiretty14:09
pabelangerto be safe in to our sudoers.d file14:09
mordredpabelanger: darn. I was hoping it wouldthat14:09
kashyapmordred: If it's security.  FWIW, at the recent KVMForum in Toronto, security engineers (IIRC from Google) have talked about audit of the nested KVM code...and haven't found anything glaring14:10
*** xarses has quit IRC14:10
*** sdague has joined #openstack-infra14:10
kashyapmordred: Ah, reading your other comments14:10
mordredkashyap: and it wasn't really security we were concerned about - as much as the times it has been tried when clouds have enabled it, it has been much more unstable and jobs have failed for extremely hard to debug / weird reasons14:11
kashyapmordred: Hmm, I don't know when were these "times in the past past".  But upstream there have been improvements consistently.14:12
mordredbut that, combined with the fact that we simply don't have the control over our clouds to make such a choice anyway14:12
kashyapmordred: Right, that's a fair point14:12
kashyapI think part of them are on Rackspace, which run Xen14:12
*** amitgandhinz has quit IRC14:12
mordrednested kvm virt there - much harder to get :)14:12
kashyapTherefore, we're stuck in limbo.14:12
mordredyah. there is not a good nested virt story we have at our disposal currently14:13
*** yamamoto has quit IRC14:13
*** ijw has quit IRC14:13
kashyapmordred: The point to consider is (credit where its' due: dansmith raised it some week ago)14:13
kashyapWe use plain emulation (QEMU "TCG") through out in the Gate for testing.  However, that same configuration is are not "recommended" for operators (for performance reasons) to run what we test in the Gate14:15
*** rbrndt has joined #openstack-infra14:16
kashyapAnyhow...Thanks for the comment.14:16
mordredkashyap: totally! I wish we had a better option to address that14:17
mordredpabelanger: I have reproduced the hang14:18
pabelangermordred: does it have to do with sudo asking for a password?14:18
*** mtanino has joined #openstack-infra14:19
mordredpabelanger: that I don't know yet - but I have a _very_ simple playbook that is exhibiting it14:19
pabelangerwhen I run sudo rm -f /etc/sudoers.d/jenkins-sudo I'm prompted for password as jenkins (expect)14:19
*** abregman is now known as abregman|afk14:19
pabelangerwasn't sure if command would handle that14:19
openstackgerritMerged openstack-infra/shade: Add test for os_keystone_domain Ansible module
openstackgerritMerged openstack-infra/shade: Allow setting env variables for functional options
pabelanger2016-09-26 17:31:46.481247 | sudo: no tty present and no askpass program specified14:20
pabelangeris what we'd get before14:20
mordredpabelanger: I have removed our command module and the hang still happens14:21
fungipabelanger: mordred: sounds like it may be related to a change in how ansible is executing the job?14:21
fungijust now catching up, my workout ran a lot longer than expected this morning14:21
pabelangermordred: ack14:22
mordredpabelanger: ok. I'm not sure what's going on - but a playbook from zlstatic to proposal.slave hangs if I run sudo in  a shell command14:23
mordredit does not hang if I do not14:23
openstackgerritMerged openstack-infra/zuul: Add back timeout_var logic
fungias a quick fix we could just remove the revoke-sudo builder from jobs that run on static job nodes14:23
fungiit's not needed, it's just there for consistency14:24
pabelangermordred: okay, that is what I was thinking.  I wonder if the becomes logic is coming into play14:24
mordredpabelanger: I'm not running with becomes14:24
pabelangersince you'd not run command: sudo foo14:24
mordredpabelanger: well, we do in the revoke-sudo builder14:24
pabelangeryoud do command foo14:24
pabelangerbecome: yes14:24
mordredbecause we aren't actually writing ansible14:24
fungialternatively, we could tweak revoke-sudo to (somehow) drop the tty for stdin... maybe </dev/null or something14:24
mordredwe're translating from jjb to generated ansible14:24
pabelangerI've never done sudo with ansible that way, I'd have to test14:25
openstackgerritRamy Asselin proposed openstack-infra/puppet-bandersnatch: Fix bandersnatch crons to support full sync
mordredfungi: just tried that - it doesn't help14:25
* fungi grumbles at computers14:25
jeblairthere's lots of sudo commands we run -- does this happen with all of them or just this one?14:26
mordredjeblair: this is the only one we know about14:26
fungijust this one, because jenkins lacks sudo perms14:26
pabelangerYa, I think it is only an issue when we are prompted for a password14:26
*** Guest49607 has quit IRC14:26
fungii mean, it conceivably happens elsewhere if a job tries to sudo when it shouldn't, but we generally fail those anyway so they're hopefully rare14:26
jeblairanyone have a link to a failed run handy?14:27
fungiwe could add -A /bin/false maybe?14:27
mordredjeblair: I have a copy of a setup in root@zlstatic01:~/tmpK08vMM14:27
jeblairwe also do this:14:28
*** rossella_s has quit IRC14:28
jeblair! sudo -n true14:28
openstackjeblair: Error: "sudo" is not a valid command.14:28
*** lucasagomes is now known as lucas-hungry14:28
jeblairat the end of the revoke-sudo command14:28
*** zhurong has quit IRC14:28
fungisudo -n true doesn't prompt14:28
*** rossella_s has joined #openstack-infra14:28
fungiahh, right, we could add -n instead of -A /bin/false14:28
mordredjeblair: and a playbook called test_playbook14:28
jeblairfungi: ah, the -n?14:28
jeblairmordred: thx14:29
fungii mean, we likely _always_ want sudo -n in our jobs anyway. there is nobody/nothing around to interact with it ever14:29
pabelanger$ sudo -n rm /etc/sudoers.d/jenkins-sudo14:29
pabelangersudo: a password is required14:29
pabelangerfungi: agreed14:29
*** xarses has joined #openstack-infra14:30
jeblairfungi: right, but also, we can't guard against that perfectly (just like we failed to guard against adding revoke-sudo to jobs that don't need it)14:30
mordredI can verify that adding -n does not cause it to hang14:30
cardeoispabelanger I see in the history that you were talking about async support that was removed. I have some jobs failing since yesterday that seems to be related to that. Can you elaborate or point me to more info?14:30
mordredlet me put the new console module and pipelining back in place14:30
cardeois(My build failing
mordredcardeois: we may be just about done figuring it out - hang on just a little bit ...14:30
cardeoisalright thanks14:31
pabelangercardeois: Ya, we are working on them as they come up14:31
pabelangercardeois: let me look at the log14:31
mordredjeblair: ok - doing sudo -n instead of plan sudo works14:31
mordredI mean, it fails the task - but it does not hang14:31
*** knangia has joined #openstack-infra14:31
mordredjeblair: and that's with our command  module in place and with pipelining turned on (neither of those seem to actually be related)14:32
cardeoispabelanger sure thanks. It seems related to xfvb we run in background in order to launch a chrome or firefox later for JS tests14:32
fungiwe could also inject SUDO_ASKPASS=/bin/false into the calling environment maybe? though that won't make it through tox or user switching like devstack does14:32
jeblairyeah.  i agree this is an immediate solution to getting signing jobs to run.  but i don't think we can leave zuul in this state.14:32
pabelangermordred: since you can test, what happens when you remove -n and add become: yes to the task? Does ansible raise an exception?14:33
pabelangercardeois: Ya, we haven't see that one yet.14:33
jeblaircardeois: we beleieve the problem exhibited in that job has been fixed, can you recheck it?14:33
mordredfungi: SUDO_ASKPASS in the environment did not work14:33
jeblairpabelanger: that was the close() problem14:34
cardeoisjetblair will do14:34
pabelangerjeblair: Ah, thank you. I missed that one from yesterday14:34
rcarrillocruzcrinkle: did you have a sequence diagram about bifrost provisioning at all?14:34
mordredpabelanger: become: True fails and does not hang14:34
rcarrillocruzor maybe i'm confused and i had it in on  of my old tech talks i gave about it14:34
* rcarrillocruz confused14:34
*** hongbin has joined #openstack-infra14:35
fungimordred: indeed, the sudo manpage indicates SUDO_ASKPASS is honored, but testing confirms it doesn't seem to help14:35
*** Jeffrey4l has joined #openstack-infra14:35
mordredbut I agree, we don't want things to hang when someone uses sudo when they're not supposed to - we want those things to fail14:35
pabelangermordred: okay, that is what I would expect.14:35
*** sputnik13 has joined #openstack-infra14:35
fungialso obviously attacking the symptom rather than the root cause, but we could replace sudo with a wrapper14:36
*** nherciu has quit IRC14:36
AJaegerjeblair: do you have time to fix the zuul-merger problem with the periodic translation jobs before the OpenStack summit? Or should I revert my change that exposed it for now?14:37
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Adjust functional test filter for OpenStack-Ansible
crinklercarrillocruz: i don't think i have anything specific to bifrost, the diagrams i have a little more big-picture14:38
jeblairAJaeger: i'd like to get to it today, but i can't promise i will; if you need to revert, that's fine.14:38
rcarrillocruzyeah, i would have sworn i had something, i'll check my messy hard drive for old presentations, it could be i did myself14:39
jeblairmordred: is the zuul command module in place in your tmpdir or no?14:39
AJaegerjeblair: I can wait another day14:39
mordredjeblair:  it is - and I'm poking at it right now - you might want to make a copy of that dir if you want to poke too14:43
*** tpsilva has joined #openstack-infra14:43
*** ssbarnea has quit IRC14:44
AJaegerjeblair: thanks14:45
*** makowals has quit IRC14:45
*** amotoki has joined #openstack-infra14:45
pabelangerjeblair: mordred: should I start rolling restarts on zuul-launchers to pick up devstack-gate fix? Or hold off for some more potential fixes14:45
jeblairpabelanger: i wouldn't do a rolling restart anyway, it would take all day and we're certain to want another.14:49
jeblairpabelanger: i'd say hold off for a few more mins14:49
jeblair(but if we did want to restart, do a hard restart)14:49
pabelangerokay, hard works too.14:50
*** oanson has joined #openstack-infra14:50
pabelangerin that case, happy to wait14:50
fungiwhat was the recent ansible change? i saw something about enabling pipelining...14:53
*** flepied has quit IRC14:53
pabelangerfungi: ya, that was enabled14:53
pabelangerplus we removed zuul_runner and replaced it with a modified version of command14:53
*** spzala has quit IRC14:53
fungiit looks like that utilizes the remote interpreter's stdin, so i can sort of see how it might change behavior this way14:53
fungipipelining i mean14:54
mordredpipelining does not affect this14:54
mordredI have tested with it on and off14:55
fungigot it. i saw you mention reenabling pipelining above, but wasn't sure what the outcome had been of testing without14:56
fungiso suspicion at this point is that it has to do with the command module?14:56
*** vsaienko has quit IRC14:56
*** nicolasbock has quit IRC14:56
mordredyah - although not necessarily with our version - the problem manifests both with and without our version of the command module in place14:57
mordredI have just verified that running the task under async fails the command properly14:57
mordredjeblair: ^^14:57
*** marst has quit IRC14:58
*** nicolasbock has joined #openstack-infra14:58
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on same event multiple times on schedule fixed code to prevent that situation
fungiit looks like we could maybe do something like authenticate=no in sudoers, or add a greedy glob that sets NOPASSWD15:02
*** amotoki has quit IRC15:02
openstackgerritMerged openstack-infra/openstackid-resources: Fix on same event multiple times on schedule fixed code to prevent that situation
*** flepied has joined #openstack-infra15:02
fungithat might be an effective way to guard against similar situations in the future (but not immediately as it would need new images)15:02
*** esikache1 has joined #openstack-infra15:02
*** r-daneel has joined #openstack-infra15:04
*** esikachev has quit IRC15:04
*** makowals has joined #openstack-infra15:04
jeblairfungi: fundamentally, this is something that needs to be fixed in zuul/ansible.  we can't have a system that lets people shoot themselves in the foot like this.15:05
*** amotoki has joined #openstack-infra15:05
*** piet_ has joined #openstack-infra15:05
fungibut sounds like there's a good chance it may be a behavior-changing patch to ansible (or new option)15:05
mordredjeblair: fwiw, I have been trying various absurd things in the command module to try to get python to make the subprocess invocation happy15:05
mordredso far I have not been successful15:06
mnaserjust saw a failure of a job: "W: The repository ' xenial Release' is not signed."15:06
*** eharney has quit IRC15:06
jeblairmordred: yeah, i'm about to start doing that :/15:06
fungimnaser: that shouldn't be a failure, just a warning15:06
mnaserpython-yaml installation failed because it could not be authenticated15:06
mordredjeblair: my most recent forray has been poking at the pty module15:06
mnaseryou're right15:06
pabelangermnaser: that not signed is expected. We don't actually have gpg signed repos for debuntu15:06
*** sdague has quit IRC15:06
pabelangerjust seen this linked in #tripleo: 2016-10-20 14:54:20,868 p=16333 u=zuul |  fatal: [node]: FAILED! => {"changed": false, "cmd": "/tmp/", "failed": true, "msg": "close() called during concurrent operation on the same file object.", "rc": null}15:08
pabelangerhave we seen that before?15:08
jeblairpabelanger: that's the close() bug that was fixed earlier15:08
*** amitgandhinz has joined #openstack-infra15:08
pabelangerjeblair: okay, do we need to restart launchers?15:08
jeblairpabelanger: that timestamp is pretty recent though15:08
pabelangerit just happened15:09
jeblairpabelanger: i gathered from irc logs that was done.  it may be worth tracking down which zuul launcher that ran on and see if it was actually restarted with the change in place.15:09
pabelangerokay, I can do that15:09
*** marst has joined #openstack-infra15:10
*** sdague has joined #openstack-infra15:10
pabelangerlooks like zl01, checking now15:10
*** mdrabe has quit IRC15:10
mordredI restarted zl01 this morning with the change in place15:10
*** nicolasbock has quit IRC15:11
mordredat 13:38:0815:11
*** thorst_ has joined #openstack-infra15:11
fungidoes the shell module suffer the same problem? or are there reasons we shouldn't use that one?15:11
*** thorst_ has quit IRC15:11
mordredfungi: it's the same module15:11
jeblairfungi: it's actuall that15:11
fungithe ansible docs make it sound like the command module and the shell module are distinct15:12
mordredwell, basically, using the shell module causes a parameter to be set15:12
*** thorst_ has joined #openstack-infra15:13
mordredso if you use shell, it tells python.subprocess to spawn a subshell, if you use command, it does not15:13
clarkbfungi: I think they were for a long time but were collapsed in 2.0 (or other relatively recent version)15:13
*** yamamoto has joined #openstack-infra15:14
*** yamamoto has quit IRC15:14
*** yamamoto has joined #openstack-infra15:15
rcarrillocruzin the code, they're pretty much the same15:16
rcarrillocruzas mordred says, it pretty much differ on a param saying 'uses_shell=True'15:16
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Fix on PHP 503 error updated pm.max_children from 300 to 400  to avoid 503 http error.
jeblairmordred: zuul_runner does not pass stdin to its command, but command module does15:17
jeblair        #stdin=st_in,15:17
*** mdrabe has joined #openstack-infra15:18
jeblairwill 'fix' the command module15:18
mordredoh - really?15:18
mordredjeblair: it still hangs for me15:18
jeblairhrm, let me make sure i didn't contaminate my test15:18
mordredjeblair: st_in defaults to None, fwiw. it's only set to something if someone passes in "data" as a parameter to the command module15:19
jeblairmordred: yep, bad test, sorry15:19
*** jcoufal_ has joined #openstack-infra15:19
mordreddarn. I was hoping I'd screwed up mine15:19
*** spzala has joined #openstack-infra15:20
dhellmannjeblair , mordred : you're not running under python 3 by any chance, are you? there were some changes to the way subprocess starts new processes under py3.15:21
*** jcoufal has quit IRC15:21
mordreddhellmann: we are not15:22
dhellmannok, good15:22
*** eharney has joined #openstack-infra15:22
mordredthe ansible folks just this last cycle started making things py3 capable15:22
dhellmannI think those changes were just related to signal handling, but it doesn't matter15:22
*** kairat has left #openstack-infra15:22
*** yamamoto has quit IRC15:23
jeblairmordred: i'm leaning towards thinking this is a side effect of async15:24
jeblairi think zuul_runner suffers this as well15:24
*** nicolasbock has joined #openstack-infra15:24
*** amotoki has quit IRC15:24
*** esikache1 has quit IRC15:25
*** yamahata has quit IRC15:25
mordredjeblair: and we just didn't notice because we were running under async which was running in a whole other daemon subprocess?15:25
mordredyah. I agree15:26
*** amotoki has joined #openstack-infra15:26
*** panda is now known as panda|bbl15:26
pabelangerjeblair: mordred: Ya, we are still running the unfixed plugin in zl01. We need another restart15:26
mordredpabelanger: sigh15:26
jeblairpabelanger, mordred: maybe pip install failed again :/15:27
pabelangerpuppet updated zuul about 5mins after you restarted15:27
pabelangerjeblair: possible, I don't see puppet kicking off a zuul_install exec until build-var patch landed on disk15:28
jeblairpabelanger: ah, then we may have just suffered from the puppetmaster oom15:28
*** sbadia has quit IRC15:28
jeblairmordred: i made a simple copy of zuul_runner and it has the problem:
jeblairalso, trying 'stdin=PIPE' and 'proc.stdin.close()' in that doesn't help15:29
pabelangerjeblair: Oh, nice (well no). Didn't know puppet would be left in a broken state on the far end15:29
*** eharney has quit IRC15:30
jeblairpabelanger: er, i don't know about broken -- it sounds like you're saying it just didn't get around to running for a while15:30
jeblairlike, 6 hours15:30
pabelangerjeblair: Oh, i see. Ya, that makes more sense.15:31
pabelangerAlso, I confused project-config update with zuul15:31
*** lucas-hungry is now known as lucasagomes15:32
*** vhosakot has joined #openstack-infra15:32
*** makowals has quit IRC15:33
*** dizquierdo has quit IRC15:33
*** sbadia has joined #openstack-infra15:33
*** amotoki has quit IRC15:34
*** baoli_ has quit IRC15:35
*** sputnik13 has quit IRC15:36
*** spzala has quit IRC15:37
*** billiebobthorty has quit IRC15:38
*** priteau has joined #openstack-infra15:39
*** nicolasbock has quit IRC15:39
*** yolanda has joined #openstack-infra15:39
*** andreas_s has quit IRC15:42
*** eharney has joined #openstack-infra15:43
*** amitgandhinz has quit IRC15:43
mordredjeblair: woot! I made something which does not hang15:43
jeblairmordred: neat!  whadyado?15:44
mordredjeblair: I lost the process return code in the process, so I need to figure that out now :)15:44
mordredjeblair: I used pty.spawn instead of subprocess.Popen + Thread15:44
*** jaosorior has quit IRC15:44
jeblairmordred: ah15:45
mordredjeblair: amusingly enough, pty.spawn has the same semantics as our follow process - or basically takes a function that works just like follow15:45
jeblairi was trying to hook it up to a pty and not making headway15:45
mordredjeblair: oo! I got the return code back15:46
mordredthat was easy15:46
*** spzala has joined #openstack-infra15:46
mordredjeblair: let me copy what I've got somewhere so you can look at it and we can clean it up15:46
*** yamamoto has joined #openstack-infra15:49
*** sflanigan has quit IRC15:49
*** spzala has quit IRC15:50
openstackgerritMonty Taylor proposed openstack-infra/zuul: Use pty.spawn to spawn the subprocess
mordredjeblair: ok ^^ that's cleaned up a little from the garbage I had at first - but still is likely open for many improvements15:51
*** rcernin has quit IRC15:52
*** Guest14069 has quit IRC15:59
fungiwow, we actually get a significant simplification in the bargain16:00
openstackgerritMerged openstack-infra/project-config: Add new project called molteniron
mordredit's got some issues ...16:01
mordredanybody know why this: bash -c sudo ls  doesn't work?16:01
*** david-lyle_ is now known as david-lyle16:01
fungibash -c 'sudo ls'16:02
openstackgerritMerged openstack-infra/project-config: Add experimental ironic grenade multitenant job
*** srobert has joined #openstack-infra16:03
dhellmannmordred : the handling of args as a list and a string in that function is confusing. It seems to convert back and forth a couple of times.16:04
*** jpich has quit IRC16:04
mordredoh yeah - this code is currently _bad_16:04
*** tqtran has joined #openstack-infra16:06
openstackgerritMerged openstack-infra/project-config: Freezer: Fixed tempest regex
mordredjeblair, fungi: ok. it worked for a little bit, but now I'm back to it not working - trying to figure out what I broke16:07
* dhellmann is envisioning a CommandLine class to hide all of this back-and-forth and manipulation16:07
*** amotoki has joined #openstack-infra16:07
*** piet_ has quit IRC16:08
fungii get the impression some of the structure there is inherited from similar mess in the ansible module from which it's forked16:08
*** nicolasbock has joined #openstack-infra16:09
*** chandankumar has quit IRC16:09
*** armax has joined #openstack-infra16:10
fungi(note the copyright header and license there)16:10
fungialso the leading comment block explains16:10
mordredjeblair: did you just run a copy of something?16:11
jeblairmordred: i'm occasionally running a python script containing little more than 'pty.spawn(['sudo', 'ls'], follow, read)' as the jenkins user on the proposal slave.16:12
*** yamahata has joined #openstack-infra16:12
mordredjeblair: ok. cool16:12
mordredjeblair: I saw a log entry happen in console.log and wasn't sure if it was you or not16:13
jeblairah, nope.16:13
mordredso - I'm back to sudo prompting for a password - because it now has a tty and will happily prompt for password16:14
jeblairyeah, the most progress i've been able to make with pty.spawn is that i can capture output from the pty.16:15
mordredyah. I swear I actually saw this work, but I'm starting to doubt my own sanity16:15
jeblair     * Lookup controlling tty for this process via sysctl.16:18
jeblair     * This will work even if std{in,out,err} are redirected.16:18
jeblairfrom sudo ^16:18
*** openstackgerrit has quit IRC16:18
*** openstackgerrit has joined #openstack-infra16:19
*** maeker has joined #openstack-infra16:19
fungiindeed, in ttyname.c16:19
*** oanson has quit IRC16:19
jeblairthat at least explains why it's able to get a pty regardless of stdin/out.  i don't understand enough about python pty module yet to know if its fork tricks are enough to get around that16:20
*** cody-somerville has quit IRC16:21
*** cody-somerville has joined #openstack-infra16:21
*** cody-somerville has joined #openstack-infra16:21
*** simondodsley has joined #openstack-infra16:22
jeblair is relevant16:23
jeblair(ansible runs with 'ssh -tt')16:23
*** derekh has quit IRC16:25
*** vsaienko has joined #openstack-infra16:27
jeblairso... if we *are* running with pipelining, it *doesn't* use -tt ?16:27
*** nherciu has joined #openstack-infra16:27
mordredjeblair: I see -tt either way16:28
*** matrohon has quit IRC16:28
persiaCould this be worked around by using ssh-agent?16:28
*** mriedem has quit IRC16:30
*** mriedem has joined #openstack-infra16:30
jeblairmordred: if i run a simple popen test, i get the behavior we want16:31
jeblairmordred: if i remove -tt16:31
mordredjeblair: oh lovely16:32
*** baoli has joined #openstack-infra16:33
*** pilgrimstack has quit IRC16:34
mordredjeblair: so - looking at the ansible source16:34
mordredthe function that runs this passes -tt: if not in_data and sudoable:16:34
jeblairmordred: so if in_data is set, we get -tt.  but in_data is set if we pipeline?  that seems backwards.16:36
*** yolanda has quit IRC16:36
jeblairoh, no i said that backwards.16:36
jeblairif in_data is set we do not get -tt.  which does make sense.16:37
*** jtomasek has quit IRC16:38
*** dtantsur is now known as dtantsur|afk16:38
*** amitgandhinz has joined #openstack-infra16:39
mordredjeblair: I am not finding a great place to hook/override that behavior - other than making either an action plugin that does evil, or an ssh connection plugin that does evil16:41
*** makowals has joined #openstack-infra16:41
jeblairmordred: do you grok why setting pipelining=true doesn't trigger it?16:41
*** mkoderer has quit IRC16:42
*** Apoorva has joined #openstack-infra16:42
mordredjeblair: no, I do not16:42
*** tpsilva has quit IRC16:44
AJaegerclarkb: did you figure out the solum problem? Do we need to restart gerrit?16:49
clarkbAJaeger: I think it is beginning to look that way16:50
jeblairmordred: whether i set pipelining=true, i still see 4 ssh operations for a simple command (mkdir, put, chmod, exec).  i would expect that to be one with pipelining enabled, right?16:52
*** makowals has quit IRC16:53
jlkit should be yes16:53
jlkunless it's a special module16:53
mordredjeblair: is it actually 4 different commands? or is it just logging each of the actions it's taking separately16:53
*** BobBall is now known as BobBall_AWOL16:53
jlkbut yes, pipeline=true should not do the separated actions16:54
jeblairmordred: well, it shouldn't be doing the first the commands i think16:54
jeblairer 'first three commands'16:54
mordredoh right. good point16:54
jlkjeblair: where are you setting pipelining=true ?16:54
*** amitgandhinz has quit IRC16:55
jeblair[ssh_connection] section of ansible.cfg in CWD16:55
*** amitgandhinz has joined #openstack-infra16:55
jlkjust ruling that out.16:56
jlkand connection method/plugin is ssh ?16:56
mordredjeblair: C.DEFAULT_KEEP_REMOTE_FILES has an effect on this16:56
jlkIt does, if you are setting that16:56
jlkbut I found that if you have pipelining that setting KEEP_REMOTE_FIELS doesn't work16:56
*** zz_dimtruck is now known as dimtruck16:56
jlkat least I thought so16:56
jlkKeeping remote files defaults to off, so unless you're setting it during execution...16:57
mordredI just unset it and now pipelining seems to work16:57
jeblairyeah, disabling keep_remote_files fixes it16:57
*** vsaienko has quit IRC16:57
jeblairapparently that takes precedence over pipelining16:57
mordredgood to know16:57
jlkah interesting16:57
jeblair(we had it set because we started with our current config, where we have that set to avoid an async bug)16:57
mordredI guess it can't keep remote files if it never made any16:57
jlkguess it's been a while since I played with both16:57
fungithat's a surprising side effect16:57
jlkwith pipelining there are no remote files16:58
jlkso it almost makes sense16:58
mordredjlk: yah. exactly16:58
fungiright, so it's effectively disabling pipelining16:58
jeblairi mean "error! does not compute! printer on fire!" might be a better behavior16:58
mordredjeblair: so - with that removed, I no-longer see -tt in the args16:58
jlkI still think that should be a warning or error on conflicting configs16:58
mordredjlk: ++16:58
fungiagreed, conflicting options shouldn't result in undefined behavior16:58
*** e0ne has quit IRC16:58
mordredjeblair: WOOT!16:59
jeblair2016-10-20 16:59:48.408057 | sudo: no tty present and no askpass program specified16:59
jeblair2016-10-20 16:59:48.408755 | [Zuul] Task exit code: 116:59
mordredjeblair: turning off remote_files and putting our current command module back in place looks like it works17:00
fungithat's a relief17:00
jeblairmordred: agreed17:00
mordredthat was very hard - but thankfully is a very easy fix17:00
*** davidsha has joined #openstack-infra17:00
jeblairyeah, and it all boiled down to "we failed to turn pipelining on"17:00
fungiso basically we can disable an option that's no longer necessary anyway and things do what we wanted? best possible outcome17:00
mordredjeblair: you wanna do the honors or shall I?17:00
davidshawould this be the correct place to ask questions about making a new project?17:01
fungidavidsha: sure, assuming it starts with you saying you've read
jeblairmordred, jlk: i think there's two things to take back to the ansible community: 1) something about these two conflicting options.  2) we really want to keep the ability to run without a tty.  some folks have attempted to "fix" the pipelining module so it always runs with a tty, but we're going to want to be able to intentionally run without one.17:02
mordredjeblair: ++17:02
jeblairmordred: yeah, i'll push up a fix17:02
davidshafungi: Yes, I've just finished making the launchpad and I'm about to move onto pypi17:02
fungiawesome. what's the question?17:03
*** yamamoto has quit IRC17:03
*** yamamoto has joined #openstack-infra17:03
*** yamamoto has quit IRC17:03
openstackgerritGhe Rivero proposed openstack-infra/shade: Add external_ipv4_floating_networks
*** mhickey has quit IRC17:03
davidshaThe project I'm making isn't for something to publish, it's an example project for how to make an out of tree neutron extension. I was just wondering would I skip the pypi step?17:03
*** ralonsoh has quit IRC17:03
*** ijw has joined #openstack-infra17:04
fungidavidsha: yeah, if you're not going to have it build a package that's uploaded to and installable from pypi, then there's no need17:05
fungidavidsha: though that said, you ought to consider whether it's possible to make that a cookiecutter template17:05
anteayathere seem to be a number of cookiecutters on pypi:
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: remove keep_remote_files
fungidavidsha: we already have several cookiecutter template repos for things similar to what you may be doing (for example, general python projects, oslo libraries, puppet modules, et cetera)17:06
jeblairmordred, fungi, jlk, pabelanger: ^ i'm going to test 389280 real quick17:06
fungisounds good17:06
fungithanks jeblair17:06
mordredjeblair: I fully support that patch17:07
mordredjeblair: and I am VERY glad that the answer was not "write an ssh connection plugin"17:07
jlkBeen down that path, don't want to go back17:07
jlkactually, still on that path :(17:07
davidshafungi: True, that sounds like a good Idea. Though I'm giving a presentation with this and would like to have it hosted somewhere for people to download it and test it. Since it's extending Neutron and could be updated as new features are added for extensions I thought it might be better as a project.17:08
* jlk mutters about locking around host keys17:08
*** ihrachys has quit IRC17:08
anteayadavidsha: the project/repo could be a cookiecutter template17:08
anteayais I believe what the suggestion is that is being offered17:09
fungithough we do also have precedent for similar "example" repos17:09
mordredjlk: ugh hostkeyes17:09
anteayafor example:
mordredjlk: have I mentioned "why can't my cloud provide me a hostkey" recently?17:09
*** inc0 has quit IRC17:10
funginow that we have "just give me a network" we need "just give me a host key" ;)17:10
jeblairmordred, fungi, jlk: that patch looks good in my local testing so i have approved it17:10
*** jpena is now known as jpena|off17:10
mordredjeblair: ++17:10
*** inc0 has joined #openstack-infra17:11
*** martinkopec has quit IRC17:11
mordredjeblair: it jives with the results of my testing as well17:11
fungijeblair: thanks, it was a refreshingly simple patch after half a day of confusion17:11
* mordred goes to abandon the pty.spawn patch17:11
davidshaanteaya, fungi: would a cookie cutter template have the same bug filing systems as other openstack projects?17:11
fungidavidsha: sure, it's a repo just like any other17:12
anteayadavidsha: I don't see why it wouldn't17:12
jeblairpatches that remove code and add comments are my favorites17:12
mordredjeblair: ++17:12
*** trown is now known as trown|lunch17:12
fungii guess we can do that global launcher restart once it's everywhere17:13
fungiand after that i'll get together a list of what needs to be rerun17:13
davidshafungi, anteaya : cool, Is it ok if I come back some time tomorrow after looking up what I'd need to change to make it a cookie cutter template?17:13
fungidavidsha: there are people around in here all the time, so sure17:13
davidshafungi, anteaya: Cool, thanks for the input!17:14
anteayadavidsha: thanks for asking17:14
fungithough we will likely start getting scarce as we move into the weekend and some people start getting on flights17:14
mordredfungi: I thought you said "start getting sarcasm" - and I was like "when did we ever stop?"17:15
fungiat least now you have an appropriately sarcastic comeback already formulated, should i ever happen to say it17:16
openstackgerritMerged openstack-infra/zuul: Ansible launcher: remove keep_remote_files
dhellmannfungi, mordred : what's the shelf-life of canned sarcasm?17:17
mordreddhellmann: I think if properly bottled it can last indefinitely17:17
dhellmannjeblair, mordred, fungi : am I correct in reading the scrollback to mean that the config option change fixes the sudo issue in the release jobs?17:17
dhellmannmordred : or pickled, I suppose17:18
mordreddhellmann: yes.17:18
fungidhellmann: or at least it will once it's in place and restarts happen17:18
dhellmannok, that was my next question :-)17:18
*** flepied has quit IRC17:19
fungidhellmann: in parallel though, do you have a list of what failed so i can get ready to rerun stuff that needs it once the restart is behind us?17:19
dhellmannfungi : how long does the restart take? is it just zuul?17:19
dhellmannfungi :
fungiit'll be a restart of the zuul launchers, though i think we just need the zlstatic01 restarted for me to be able to start in on release jobs17:20
dhellmannlet me see if I can add version numbers to the repos that don't have them17:20
fungijeblair: ^ ?17:20
jeblairfungi: true, but it'll probably be easier to hard restart all at once17:20
clarkbfungi: assuming that all of the jobs you need rerun run on static hosts yes. But tarball builds etc run on general instances17:20
fungicool, didn't know if you were going to shoot for a graceful rolling restart instead17:20
jeblairshould be done in < 30 mins17:21
fungiclarkb: in tis case the only observed issue for release work was that jobs running on static nodes with the revoke-sudo builder were hanging and timing out17:21
jeblairfungi, clarkb, mordred: i will manually update zuul, install, and restart on the launchers17:21
jeblairrather than waiting for puppet to oom.17:21
fungibecause the jenkins user doesn't actually have permission to revoke sudo for itself on those nodes, as we don't ever grant it17:21
fungijeblair: thanks, that'll speed things along nicely17:22
*** e0ne has joined #openstack-infra17:22
clarkbsounds good (also its ansiblet hat ooms not puppet aiui)17:23
*** degorenko is now known as _degorenko|afk17:23
*** jcoufal has joined #openstack-infra17:23
*** sputnik13 has joined #openstack-infra17:24
*** baoli_ has joined #openstack-infra17:24
*** jcoufal_ has quit IRC17:24
mordredpansible ooms17:25
*** baoli has quit IRC17:25
mordredthat almost sounds like a band17:25
AJaegerteam, clarkb has been trying t ofigure out the many "Submitted, Merge Pending" in solum - and we think it's best to restart gerrit, seems there's some jgit corruption. When do you want to restart gerrit?17:25
hamzyHello, could someone please add me to molteniron-core and molteniron-release? Thanks!17:26
jeblair#status log restarted ansible launchers with 2.5.2.dev3117:26
openstackstatusjeblair: finished logging17:26
jeblairfungi, dhellmann: should be gtg17:26
fungithanks jeblair!17:27
*** davidsha has quit IRC17:28
dhellmannfungi : is the info in that etherpad clear and complete for restarting those jobs? I added version numbers to the lines that were missing them.17:28
dhellmannI think it's safe to ignore the wheel errors17:28
*** yaume has quit IRC17:29
fungidhellmann: yeah, i'll need to look up the shas of those tags, but it looks like i can just reenqueue the tag refs for all of them17:29
dhellmannI can start pulling shas for you17:30
fungior i can nab them from the log urls you're adding17:30
dhellmannok, the log urls are easy17:30
*** esikache1 has joined #openstack-infra17:31
*** esikache1 has quit IRC17:31
*** esikachev has joined #openstack-infra17:32
*** tkelsey has quit IRC17:33
*** mat128 is now known as mat128|afk17:33
*** jcoufal has quit IRC17:34
*** tiswanso has joined #openstack-infra17:35
AJaegerOOps, we have still 70 periodic jobs running - including the requirements one from yesterday ;(17:38
AJaegerharlowja: is failing ;(17:39
dhellmannfungi : for js-openstack-lib, are you the owner then or should I talk to someone else?17:40
*** sputnik13 has quit IRC17:40
fungidhellmann: okay, hopefully they're all reenqueued correctly now, except for js-openstack-lib which needs a 0.0.2 tag pushed (for some reason npm thought 0.0.1 was already released/uploaded and refused the upload)17:40
AJaegerfungi, before you enqueue, please check status of zuul - I'm surprised by those 70 periodic jobs...17:40
dhellmannfungi : it would be even better to go with 0.1.0, but sure17:40
fungiAJaeger: i'm betting those are due to lengthy timeouts for jobs with revoke-sudo on static nodes but looking now17:41
*** sputnik13 has joined #openstack-infra17:41
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: zypper: Do not pull recommended packages
fungidhellmann: yeah, i honestly don't know what the versioning conventions are for the nodejs/npm ecosystem, so am mostly relying on cardeois to tell me (and the packaging.json file in it needs a commit updating it to the next release version before we tag it)17:42
cardeoisfungi dhellmann I just merged the 0.0.2 version17:43
dhellmannfungi : ok, if they have their own conventions we should follow those. I discourage 0.0.x versions in our projects because it makes branching slightly more confusing and implies the first release has no features. I could be over-pedantic on that, though.17:43
cardeoiscan you maybe try to replublish please?17:43
AJaegerfungi: ah, that might be...17:43
openstackgerritZara proposed openstack-infra/storyboard-webclient: Hide arrows to expand task details if there are no details
fungidhellmann: looks like the reenqueued releases are succeeding rather than timing out now, so we're hopefully all set there17:44
*** SumitNaiksatam has joined #openstack-infra17:44
fungijeblair: mordred: jlk: ^ !!!17:44
dhellmannfungi : yep, I'll keep an eye on these and if they all run through then I'll approve some of the other outstanding items we put on hold17:44
dhellmannfungi, jeblair, mordred, jlk, pabelanger : thank you all for getting to the bottom of this issue today and fixing it!17:45
harlowjaAJaeger thx, will have to see if i can figure out why that's dying17:45
mordredfungi: yay!!!17:45
*** rtheis has joined #openstack-infra17:46
*** vsaienko has joined #openstack-infra17:48
AJaegerfungi, feel free to kill the periodic jobs if needed - before they run several days17:48
fungiAJaeger: yeah, looks like we're starved on translation update jobs, which all run on one static node so we probably timed out a bunch of them and periodic has a lower priority17:48
fungii expect it will catch back up quickly now that the fix is in place17:48
AJaegerfungi, it normally takes 3-4 hours to run them - so, let's see. Also, there are entries in release queue from yesterday. Hope they move forward as well...17:49
AJaegerharlowja: there are more failing like
AJaegerharlowja: but some of these might also be due to the problems that the team just fixed (not the keystone one)17:51
harlowjai also recently changed that jenkins script17:51
AJaegerharlowja: I know - that's why I point it out ;)17:51
*** dimtruck is now known as zz_dimtruck17:51
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Add StoryBoard integration for graph commands
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Add Jira integration for graph commands
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Wait until the most recent index is available
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Refactor launchpad code into bug_tracker
harlowjaAJaeger  :)17:52
harlowjasomething in i guess17:52
AJaegerfungi, clarkb has been trying to figure out the many "Submitted, Merge Pending" in solum - and we think it's best to restart gerrit, seems there's some jgit corruption. When should we restart gerrit? And who can do it?17:52
harlowjaseems to be getting past the 'diff' part, so that's good17:52
AJaegerharlowja: hope you figure it out, it's only 22 lines ;)17:52
*** onovy has quit IRC17:52
*** peterlisak has quit IRC17:52
*** zz_dimtruck is now known as dimtruck17:53
harlowjaAJaeger ya, i think i got it17:53
harlowjathe diff command seems to exit with non-zero if there is a diff17:53
harlowjadidn't expect that...17:54
*** tphummel has joined #openstack-infra17:54
AJaegerwe use bash -xe normally, don't we?17:54
harlowjaya, tox does also i think17:55
clarkbAJaeger: fungi everythign I can tell is that the git repo itself is fine so must be some issue with jgit's current state? Similar to what we saw with eg nova and that failed upgrade17:56
*** markvoelker has quit IRC17:56
fungiwe can probably restart gerrit any time. i'm mildly surprised that we haven't needed to restart recently for jvm gc reasons17:56
*** onovy has joined #openstack-infra17:57
*** peterlisak has joined #openstack-infra17:58
fungicardeois: looks good!
fungicardeois: and shows 0.0.2 as the latest release now17:58
cardeoisawesome thanks a lot fungi !17:58
fungihappy to help17:59
AJaegerfungi, clarkb , could either of you do it,please?18:00
*** amoralej is now known as amoralej|off18:00
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Ensure a diff does not cause a non-zero return code
harlowjaAJaeger ok ^ should help with the diff problem, lol18:01
*** amitgandhinz has quit IRC18:01
*** amitgandhinz has joined #openstack-infra18:02
fungiAJaeger: probably want to make sure dhellmann is done or paused approving more release changes first18:03
fungialso i need to step away for lunch/voting so will be gone for a few hours18:03
fungibetter if i'm not the one to do the restart18:03
dhellmannAJaeger, fungi : I'm about to tag the final releases for the cycle-trailing projects. Should I wait?18:03
fungii suppose i can do the gerrit restart right now before i head out18:04
*** yamamoto has joined #openstack-infra18:04
fungiclarkb: are you around to look into this after i restart gerrit?18:04
fungiinfra-root: any objections to a gerrit restart so we can hopefully clear the submitted-merge-pending solum changes?18:04
fungilooks like our most recent gerrit restart was 10 days ago18:05
dhellmannfungi : I'll stand by and wait for that restart18:05
AJaegerthanks, fungi!18:06
clarkbI am fairly distracted too18:06
fungii'll #status notice it and fire the restart now18:06
clarkbtrying to get pretrip stuff done today so that I have time for work tomorrow...18:06
*** tqtran has quit IRC18:06
fungiand then i'll make sure it comes back up and is working at least18:07
fungi#status notice The Gerrit service on is being restarted now in an attempt to resolve some mismatched merge states on a few changes, but should return momentarily.18:07
openstackstatusfungi: sending notice18:07
*** mountpoint has joined #openstack-infra18:07
*** flepied has joined #openstack-infra18:07
*** maishsk has joined #openstack-infra18:08
-openstackstatus- NOTICE: The Gerrit service on is being restarted now in an attempt to resolve some mismatched merge states on a few changes, but should return momentarily.18:08
fungiwebui is returning content again18:08
*** tphummel has quit IRC18:08
fungilooks like it's back to working order18:08
fungiAJaeger: have one of those broken solum changes handy?18:08
AJaegerdoes the solum team need to do anything to get the changes merged? recheck?18:08
AJaegerpick one ;)18:09
AJaegerthe first fungi^18:09
openstackstatusfungi: finished sending notice18:10
fungiyeah, looks like they're still showing as in a submitted, merge pending state18:10
jlkmordred: yeah, hostkey would be good to get from cloud. But that doesn't help Ansible.18:10
fungiAJaeger: not sure what the next step is18:10
*** e0ne has quit IRC18:10
AJaegerI could recheck one...18:10
bswartzgot a question for you infra guys -- I'm going to propose addition of a new repo which contains code for testing manila, and it's GPL licensed -- the questions is whether it belongs in the openstack or openstack-infra namespace -- it's not clear from reading
fungiAJaeger: have you looked to see if any of these changes are actually merged in the repo and just not reflected as merged in gerrit? we can probably fix that in the db or something if so18:11
* AJaeger checks18:11
*** yamamoto has quit IRC18:11
*** electrofelix has quit IRC18:11
AJaegerhead is from Oct 9 - so, not merged18:12
fungibswartz: good question, i think the purpose of the software is what matters, and not necessarily which project team controls it18:12
*** timello has quit IRC18:12
bswartzyeah but that doesn't answer my namespace question18:12
bswartzis the openstack-infra namespace only for stuff your team owns?18:12
bswartzor is it for stuff that's not released as part of openstack?18:13
fungioh, got it... well, we have infra team projects in the openstack namespace, and there's at least one non-infra project in the openstack-infra namespace18:13
AJaegerbswartz: openstack/ namespace is for everybody, just use it18:13
fungihonestly it's not entirely consistent18:13
bswartzAJaeger: that's what I was leaning towards18:13
fungibut i agree with AJaeger, openstack namespace makes sense18:13
*** trown|lunch is now known as trown18:13
bswartzokay thx18:14
jeblairi think the others should be retired, just like stackforge.  it's just a lot of work.18:14
fungiif it turns out that it _needs_ to be under the infra team for some reason, then that's not a reason to change namespaces on it anyway18:14
*** otherwiseguy has quit IRC18:14
bswartzjeblair: including openstack-infra and openstack-dev?18:14
fungibswartz: yes18:14
fungiideally we'd just have no namespaces at all, but for mirroring to github we need one18:14
*** otherwiseguy has joined #openstack-infra18:14
bswartzI for one welcome our own single-namespace overlords18:14
jeblairfungi: i think gerrit handles pending merges as soon as it starts up, but just in case, i'd probably wait until it flushes the queue to finally determine whether the restart was effective or not18:14
AJaegerfungi, I rechecked the oldest solum change now18:15
bswartzI for one welcome our new* single-namespace overlords18:15
fungijeblair: yeah, it may still update it18:15
fungiokay, i'm going to go run my lunch errands now. back in a while18:15
AJaegerjeblair: ah, soI was impatient - let's see...18:15
AJaegerfungi, thanks!18:15
dhellmannfungi, AJaeger : so it's safe to approve patches in gerrit?18:15
AJaegerdhellmann: yes18:15
*** sputnik13 has quit IRC18:15
dhellmannAJaeger : thanks18:15
njohnstonHi!  I was hoping the gerrit restart would fix this, but it hasn't.  I get "Code Review - Error    500 Internal server error" when I go to and try to cherry-pick it to stable/newton.  Please, could someone take a look at the gerrit logs and see what is up?18:16
*** kjackal_ has quit IRC18:16
*** kjackal_ has joined #openstack-infra18:18
*** dave-mccowan has joined #openstack-infra18:18
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: zypper: Do not pull recommended packages
AJaegerclarkb, fungi: shows still "Conflicts With (N/A) - 500 Internal Server Error". Let's see whether anything merges eventually...18:25
openstackgerritKen'ichi Ohmichi proposed openstack-infra/project-config: Remove stress jobs from the gate
*** timello has joined #openstack-infra18:27
*** rossella_s has quit IRC18:28
*** rossella_s has joined #openstack-infra18:29
dhellmannAJaeger, jeblair, fungi : I'm seeing a new release tagging failure :-(
dhellmannmaybe that's a git cache issue?18:30
dhellmannthe tag is there18:30
AJaegerdhellmann: strange, that code has not been changed for some time18:31
dhellmannyeah, I suspect an issue with getting the local copy of the git repo updated18:31
dhellmannthe repo for the project the job is running for, openstack/instack-undercloud18:32
dhellmannI'm getting that same error with *lots* of repos18:32
AJaegerdhellmann: a git update?18:33
AJaegergit branch -a --contains refs/tags/5.0.0 works for me on thta repo18:33
*** vsaienko has quit IRC18:33
dhellmannAJaeger : yes, same here18:33
AJaegerbut says "error: malformed object name refs/tags/5.0.0"18:33
dhellmannthat error message means that the tag isn't present locally18:33
dhellmannso the node running that job did not fetch the tags correctly18:34
AJaegerdhellmann: Ah, indeed - can get the error with another tag18:34
dhellmannat one point in the past jeblair added some extra logic to zuul-cloner to ensure that all tags were fetched18:34
AJaegerjeblair: zuul-cloner did not fetch the 5.0.0 tag ^18:34
dhellmannat least I think so? I know we had a work-around in some other release jobs18:34
AJaegerdhellmann: I have no idea and can't help you further, hope others can.18:35
dhellmannAJaeger : thanks18:35
*** tnovacik has quit IRC18:35
AJaegerinfra-root, could you help dhellmann, please? ^18:35
*** timello has quit IRC18:36
*** rtheis has quit IRC18:36
*** ociuhandu has quit IRC18:38
*** pilgrimstack has joined #openstack-infra18:39
*** mountpoint has quit IRC18:40
dhellmanninteresting, I see that some jobs did ok but some failed and of those it looks like most (all?) are on osic nodes18:41
dhellmannI have no idea if that's meaningful18:41
*** dimtruck is now known as zz_dimtruck18:42
irtermitedhellmann: does it say how/why failed?18:42
irtermitecloudnull: ^^18:43
dhellmannirtermite : I'm building a list of links to the logs for failures at the bottom of
dhellmannso far they all seem to be failing because they're not seeing the new tag18:43
dhellmanna lot of the failures are doc jobs, which aren't critical18:43
dhellmannbut there are 2 tarball jobs that we'll need to redo18:44
irtermitenot seeing the 5.0.0 tag specifically, or having trouble hitting the repo period?18:44
*** timello has joined #openstack-infra18:44
dhellmannirtermite : there is no error about trying to fetch from upstream, but after the fetch the tag is not present18:44
dhellmannirtermite : this is a representative example
irtermitedhellmann: "2016-10-20 18:28:09.764006 | error: malformed object name refs/tags/14.0.0 "18:45
irtermiteoops, wrong one... "2016-10-20 18:23:17.628628 | error: malformed object name refs/tags/5.0.0 "18:46
dhellmannright. that error means the local copy of the repo does not contain the tag that was just pushed to the upstream copy of the repo by another job, which then triggered the job that failed18:46
*** markvoelker has joined #openstack-infra18:46
*** vsaienko has joined #openstack-infra18:46
dhellmannIOW, the job that failed was triggered by a tag being pushed, but the job itself doesn't see that tag for some reason18:47
*** mountpoint has joined #openstack-infra18:47
*** mountpoint has quit IRC18:47
*** markvoelker has quit IRC18:47
*** markvoelker has joined #openstack-infra18:47
dhellmannthis job failed on internap, so I don't think it's related to the cloud where the job ran
jeblairdhellmann: can you describe the process flow in detail re "the tag that was just pushed to the upstream copy of the repo by another job, which then triggered the job that failed"18:48
irtermiteyea, doesn't seem like an error that would be cloud related anyway18:48
irtermitesounds like something is not checking out properly18:49
*** mountpoint has joined #openstack-infra18:49
*** mountpoint has quit IRC18:49
*** eharney has quit IRC18:49
*** e0ne has joined #openstack-infra18:49
dhellmannjeblair : when a patch is merged to openstack/releases the tag-releases job runs in the post-release queue. That job runs on the special signing node with privileges to create signed tags and push them to gerrit. That job ran, and pushed a bunch of tags. Those tags in turn triggered various jobs related to releasing whatever was being tagged. Some of those have failed.18:50
*** zzelle has joined #openstack-infra18:50
dhellmannso far most of the failures look like jobs to rebuild documentation (I think those run in the tag queue) or to announce new releases18:50
dhellmann(I think those run in the releases queue)18:50
*** Julien-zte has quit IRC18:50
*** tobias_ has joined #openstack-infra18:51
dhellmanntwo of the failed jobs were tarball jobs, though18:51
dhellmannunfortunately, this release included all of the ansible repos, and there are a zillion of those18:51
*** zz_dimtruck is now known as dimtruck18:52
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Retire neutron-pd-driver
*** mountpoint has joined #openstack-infra18:52
*** mountpoint has quit IRC18:52
*** mriedem has quit IRC18:53
jeblairi think i understand the problem18:53
*** Julien-zte has joined #openstack-infra18:53
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Retire neutron-pd-driver
dhellmannjeblair : we had a similar issue a while back, and I made some changes to a few of the release scripts to force fetching tags as part of cloning a repo18:54
dhellmannI think those were in validation jobs or something, I don't remember exactly18:54
jeblairdhellmann: yeah, and that's in zuul-cloner now.  that's working, believe it or not.  :)18:54
dhellmannbut I thought that at the same time you also made changes to zuul-cloner to do the same thing18:54
dhellmannI wasn't sure if that made it in18:54
jeblairthis happened because of the gerrit restart18:54
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Retire neutron-pd-driver
dhellmannjeblair : that's what I was afraid of18:55
AJaegerjeblair: really? dhellmann waited until gerrit was restarted18:56
*** mountpoint has joined #openstack-infra18:56
jeblairthe problem is that we tell zuul-cloner to use as the canonical location to update a repository from.  when a tag is pushed to gerrit, it takes some time to be replicated to all the remote repos (git.o.o and github).  normally that's pretty fast.  but in this case, it went to the back of a queue that is about 15,000 git push operations long.  it takes, i want to say, longer than 30 minutes for gerrit to fully sync ...18:57
jeblair... all the repos.18:57
AJaegerargh ;(18:57
AJaegersorry, dhellmann18:57
dhellmannok, so if I'd waited an hour or so everything would be fine?18:57
dhellmannthat's good to know for next time18:58
jeblairthe best solution to this is to have zuul wait for replication completed events for these.  that may be doable in the long run, but it's complicated (someone started on a change to do that, but i don't think the first cut was quite right)18:58
*** lucasagomes is now known as lucas-afk18:58
*** jkilpatr has quit IRC18:58
*** tobias_ has quit IRC18:59
jeblairthat will solve even the small race condition we have in the normal case18:59
*** mriedem has joined #openstack-infra18:59
jeblair(usually, it takes longer to get a job running on a node that it takes for gerrit to push the ref out everywhere, but theoretically, it could be faster)18:59
*** amitgandhinz has quit IRC18:59
*** kgiusti has quit IRC18:59
AJaegerjeblair: yeah, we have too many nodes ;(18:59
*** amitgandhinz has joined #openstack-infra19:00
*** woodster_ has joined #openstack-infra19:00
*** panda|bbl is now known as panda19:00
jeblairalternative, we may be able to have the cloner fetch from the zuul merger in this case... i will think about that as i work on the other issue.19:00
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Complete retirement for neutron-pd-driver
jeblairdhellmann: but yeah, in the mean time, i think adding "ask infra-root if the gerrit processing queue is empty if it was just restarted" to the human protocol might be good19:01
jeblairit is now empty.  :)19:02
*** kgiusti has joined #openstack-infra19:02
jeblair(and i think we can now say with some confidence, it did not fix the solum issue)19:02
dhellmannjeblair : ok. Can I get you to re-enqueue the 2 tarball jobs that failed?19:02
*** kgiusti has left #openstack-infra19:02
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Complete retirement for neutron-pd-driver
jeblairdhellmann: i can try :)19:04
jeblairdhellmann: what were they?19:04
dhellmannjeblair : see lines 31-34 of
*** zzelle has quit IRC19:04
dhellmannlet me get version info for you19:04
*** abregman|afk is now known as abregman19:05
jeblairmordred: where are you on the vars.yaml change?  i'm having a lot of trouble with this because i can't see them19:06
mordredjeblair: oh! crap. I'm nowhere. let me be somewhere with it real quick19:06
AJaegerjeblair, clarkb, fungi: Indeed, the solum issue is not fixed ;(19:07
jeblairzuul enqueue-ref --trigger=gerrit --pipeline=release --project=openstack/instack-undercloud-tarball --ref=refs/tags/5.0.0 --newrev=b3f4eb6d17737c58ec9374cf46d306ad18e7d86719:07
dhellmannjeblair : when you're thinking about zuul features, I would be happy to have some sort of interface to restart jobs like this without bugging infra-root about it.19:07
openstackgerritNate Johnston proposed openstack-infra/project-config: Make neutron-fwaas tempest jobs for legacy and v1 voting
dhellmannmaybe that exists and I just don't have permission, which is also ok19:07
AJaegerinfra-root, was just rechecked and is still in "Submitted, Merge pending" - and there are more of these for solum at
jeblairdhellmann: ack19:08
*** eharney has joined #openstack-infra19:08
*** Goneri has quit IRC19:09
openstackgerritMonty Taylor proposed openstack-infra/puppet-openstackci: Treat yaml files as plain text
mordredjeblair: ^^19:09
jeblairzuul enqueue-ref --trigger=gerrit --pipeline=release --project=openstack/kolla-tarball --ref=refs/tags/3.0.0 --newrev=cc3426a73346aa8b7bbdbe78de3849c0e0c8752b19:09
jeblairdhellmann: do those two commands look right?19:09
dhellmannjeblair : let me check the shas19:09
*** devkulkarni has joined #openstack-infra19:10
jeblairmordred: thx19:10
mordredjeblair: sorry for the delay19:10
*** maishsk has quit IRC19:10
dhellmannjeblair : yes, those look correct19:11
jeblairdhellmann: oops, those are job names, not project names19:14
*** jkilpatr has joined #openstack-infra19:14
dhellmannjeblair : oh, sorry, I didn't notice that19:14
jeblairinfra-root: i think i've done more than my share of firefighting this week.  i am starving and am *far* behind on summit prep.  i'm going to go work on those now, and hope that others are now prepped for summit and can handle anything that comes up.  i do not expect to be active further on irc this week.19:14
openstackgerritNate Johnston proposed openstack-infra/project-config: Make neutron-fwaas tempest jobs for legacy and v1 voting
jeblairdhellmann: those two are done19:15
dhellmannjeblair : thank you19:15
mordredjeblair: ++19:15
dhellmannjeblair : enjoy your meal, and see you next week!19:15
* dhellmann makes a note to avoid having a release deadline the week before a travel event19:15
dhellmannalthough for ocata the final release will be *during* the ptg19:16
*** Sukhdev has joined #openstack-infra19:18
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
njohnstondhellmann: perfect time for people to pitch in as part of the horizontal team sessions, perhaps?19:18
*** maishsk has joined #openstack-infra19:18
dhellmannnjohnston : it would be, if those were on thursday when our release date falls19:18
dhellmannwe have time to move that earlier in the week, I guess19:18
odyssey4methanks dhellmann, mordred, jeblair, AJaeger and many other names for helping get Newton done19:21
odyssey4meit's been a fantastic cycle19:22
odyssey4meand if you don't mind, I'm going to pour myself a rather stuff drink and leave my laptop alone19:22
dhellmannodyssey4me : good plan19:22
*** amitgandhinz has quit IRC19:23
*** Apoorva has quit IRC19:24
AJaegerodyssey4me: go for it! glad to hear that Newton is down for you finally ;)19:24
*** jheroux has quit IRC19:24
*** amitgandhinz has joined #openstack-infra19:25
anteayaodyssey4me: enjoy your drink19:26
*** ihrachys has joined #openstack-infra19:30
mordredodyssey4me: yay drinking! (and also getting cycles done)19:30
*** stream10 has joined #openstack-infra19:30
*** jkilpatr has quit IRC19:31
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** devkulkarni has quit IRC19:32
*** devkulkarni has joined #openstack-infra19:33
*** mhickey has joined #openstack-infra19:34
*** Apoorva has joined #openstack-infra19:36
*** dimtruck is now known as zz_dimtruck19:36
anteayajeblair: safe travels19:36
*** dizquierdo has joined #openstack-infra19:36
dhellmannI'm going to follow jeblair & odyssey4me's lead and drop offline to get ready for travel. I'll see you all next week!19:36
anteayadhellmann: safe travels to you too19:36
*** e0ne has quit IRC19:43
*** tnovacik has joined #openstack-infra19:43
*** pilgrimstack has quit IRC19:43
*** yolanda has joined #openstack-infra19:43
ianwmordred: just checking ... all the launchers are happy now?  When i left last night was just zl01 that needed to be unemergencied, but seems like that's all fixed now19:44
mordredianw: yup! we're in good shape19:45
mordredianw: found the hanging problem and fixed that too19:45
mordredso it _should_ all be operating normally19:45
*** jkilpatr has joined #openstack-infra19:46
ianwawesome, thanks19:48
*** tykeal has joined #openstack-infra19:48
*** ociuhandu has joined #openstack-infra19:52
hamzyHello, could someone please add to the group,members and,members please?19:55
pleia2hamzy: I'll have a look, what's the email address you use for gerrit?19:55
hamzypleia2, I think my Ubuntu One account is linked to mark.hamzy@gmail.com19:56
pleia2hamzy: it's what you have in gerrit, not lp/ubuntu one19:57
pleia2I see
pleia2ah, the gmail one is in there too19:57
hamzyyeah, I have both... don't know what is primary19:57
pleia2hamzy: can go here to see:
pleia2(while logged in)19:57
hamzyit says
pleia2ok, adding19:58
hamzyand Account ID1824219:58
pleia2hamzy: done :)19:58
*** abregman is now known as abregman|afk20:01
vsaienkodevstack-gate core team, please help to merge chain needed for ironic multinode job It is a blocker of ironic team. Thanks!20:02
*** Naeil has quit IRC20:05
*** Apoorva_ has joined #openstack-infra20:07
*** abregman|afk has quit IRC20:08
*** nherciu has quit IRC20:10
*** abregman has joined #openstack-infra20:11
*** Apoorva has quit IRC20:11
*** abregman is now known as abregman|afk20:11
*** mfedosin has quit IRC20:12
*** aeng has joined #openstack-infra20:13
*** ldnunes has quit IRC20:15
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge
*** claudiub|2 has quit IRC20:16
*** Goneri has joined #openstack-infra20:16
*** mfedosin has joined #openstack-infra20:20
*** devkulkarni has quit IRC20:21
*** ccamacho has left #openstack-infra20:22
wznoinskhi infra, would someone have a moment to add me to,members ?20:22
wznoinskor better yet, intel-nfv-ci account20:22
*** thorst_ has quit IRC20:22
wznoinsk"Intel NFV CI <>"20:23
pleia2wznoinsk: added you20:23
*** ihrachys has quit IRC20:24
wznoinskpleia2, merci20:24
*** ihrachys has joined #openstack-infra20:24
*** ijw_ has joined #openstack-infra20:25
*** vsaienko has quit IRC20:25
*** mdrabe has quit IRC20:25
*** mhickey has quit IRC20:26
*** flip214_ is now known as flip21420:27
*** dave-mccowan has quit IRC20:28
*** ijw has quit IRC20:28
*** timello has quit IRC20:30
*** florianf has quit IRC20:30
*** maishsk has quit IRC20:32
*** maishsk has joined #openstack-infra20:33
*** Goneri has quit IRC20:33
*** jkilpatr has quit IRC20:33
*** timello has joined #openstack-infra20:36
*** ijw_ has quit IRC20:37
*** maishsk has quit IRC20:37
*** Rockyg has quit IRC20:38
*** mat128|afk is now known as mat12820:43
openstackgerritNate Johnston proposed openstack-infra/project-config: Make neutron-fwaas tempest jobs for legacy and v1 voting
*** john-davidge has quit IRC20:44
*** mat128 is now known as mat128|gone20:44
*** john-davidge has joined #openstack-infra20:44
*** askb has joined #openstack-infra20:47
*** dprince has quit IRC20:48
*** john-davidge has quit IRC20:49
*** tphummel has joined #openstack-infra20:50
*** stream10 has quit IRC20:53
*** dave-mccowan has joined #openstack-infra20:54
*** amitgandhinz has quit IRC20:54
*** priteau has quit IRC20:54
*** amitgandhinz has joined #openstack-infra20:55
*** mfedosin has quit IRC20:55
*** thorst_ has joined #openstack-infra20:56
*** jkilpatr has joined #openstack-infra20:59
*** tphummel has quit IRC21:00
*** ijw has joined #openstack-infra21:01
openstackgerritElizabeth K. Joseph proposed openstack-infra/project-config: Bump entercloud max servers up from 0
pleia2clarkb: ^21:02
*** edmondsw has quit IRC21:02
*** raildo has quit IRC21:03
*** yolanda has quit IRC21:03
*** devkulkarni has joined #openstack-infra21:04
*** inc0 has quit IRC21:04
*** dave-mcc_ has joined #openstack-infra21:05
*** devkulkarni has quit IRC21:05
*** baoli_ has quit IRC21:05
*** srobert has quit IRC21:05
*** baoli has joined #openstack-infra21:05
*** ijw has quit IRC21:07
*** Goneri has joined #openstack-infra21:08
*** baoli_ has joined #openstack-infra21:08
*** baoli_ has quit IRC21:08
*** dave-mccowan has quit IRC21:08
*** baoli_ has joined #openstack-infra21:09
*** baoli has quit IRC21:10
*** trown is now known as trown|outtypewww21:11
openstackgerritMerged openstack-infra/project-config: Bump entercloud max servers up from 0
mordredclarkb: you got image uplodas to ecs to work I take it?21:13
pleia2we did!21:13
*** matrohon has joined #openstack-infra21:13
pleia2v1 api21:13
clarkbya glance v1 ftw21:13
mordredpleia2: was the container_format relevant at all?21:13
clarkbI +2ed oscc change21:14
mordredI didn't want to have to implement support for that :)21:14
pleia2hah, right21:14
pleia2it looks good though, 3 clouds, 7 regions :)21:14
anteayamight I inquire as to what ecs stands for?21:15
mordredanteaya: entercloudsuite21:15
mordredanteaya: it's a european openstack public cloud based in italy21:15
anteayahow wonderful21:16
mordredclarkb: Unable to create new object: /home/gerrit2/review_site/git/openstack/os-client-config.git/objects/e9/21010db6806926f4a6e5ef451b1f6ca6df349c21:16
*** gyee has joined #openstack-infra21:16
mordredclarkb: I got that trying to rebase
mordredclarkb: any immediate thoughts? I know you were poking at other merge issues earlier21:16
clarkbthat means jgit cant resolve the merge21:16
clarkbusually manual rebase fixes21:16
openstackgerritMonty Taylor proposed openstack/os-client-config: Add support for volumev3 service type
clarkbsolums problem was even that didnt help21:17
openstackgerritMonty Taylor proposed openstack/os-client-config: Clarify how to set SSL settings
mordredI have manually rebased21:17
mordredclarkb: also - apparently there is a volumev3 service type now21:17
mordredit's a pabelanger !21:18
pabelangerjust catching up on backscroll, sudo issues from this morning was ansible config issue?21:18
*** baoli_ has quit IRC21:19
*** r-mibu has quit IRC21:19
*** baoli has joined #openstack-infra21:19
*** r-mibu has joined #openstack-infra21:19
prometheanfirefor those wondering (I think mordred AJaeger dirk and pabelanger) gerrit doesn't know how to tell jgit to lower the context when merging patches21:19
dirkprometheanfire: thx21:20
mordredprometheanfire: ah21:21
*** zz_dimtruck is now known as dimtruck21:21
anteayamorning jhesketh21:21
pabelangerlooks that way:
mordredit's a jhesketh !21:22
openstackgerritMonty Taylor proposed openstack/os-client-config: Remove validate_auth_ksc
openstackgerritMonty Taylor proposed openstack/os-client-config: Fix a bunch of tests
prometheanfirepabelanger: ?21:23
prometheanfirepabelanger: sudo thing?21:24
pabelangerprometheanfire: there was an issue with ansible and sudo this morning, looks like we got the fix (ansible.cfg change)21:24
*** esikachev has quit IRC21:24
pabelangerwas just checking to see if it was addressed21:24
*** baoli_ has joined #openstack-infra21:25
*** baoli_ has quit IRC21:25
*** cody-somerville has quit IRC21:25
*** csomerville has joined #openstack-infra21:25
*** baoli has quit IRC21:26
jheskethmordred: there's been some gerrit restarts etc... anything left outstanding or are things ticking along again?21:26
*** baoli_ has joined #openstack-infra21:26
*** tnovacik has quit IRC21:26
*** baoli_ has quit IRC21:27
*** baoli has joined #openstack-infra21:27
*** gordc has quit IRC21:28
*** matrohon has quit IRC21:29
*** baoli has quit IRC21:29
*** baoli has joined #openstack-infra21:30
*** cody-somerville has joined #openstack-infra21:30
*** cody-somerville has joined #openstack-infra21:30
mordredjhesketh: things are in good shape at the moment21:30
fungiokay, i'm back and caught up on scrollback now21:31
mordredjhesketh: we also found the bug with the new v2.5 change that was causing sudo commands in release jobs to hang and that's rolled out21:31
anteayafungi: welcome back21:31
fungias for the stuck changes for solum, the only thing i know to do next is unset submitted for them in the gerrit db... i'm not sure if that requires a reindex or just a cache flush though21:31
*** baoli has quit IRC21:32
jheskethmordred: that sounds nasty, but good job :-)21:32
*** csomerville has quit IRC21:32
*** baoli has joined #openstack-infra21:32
*** vhosakot has quit IRC21:33
*** baoli has quit IRC21:33
clarkbfungi: is there a way to resubmit without doing that?21:33
clarkbtry to force it outside of zuul?21:33
*** dave-mcc_ has quit IRC21:33
zarofungi: doesn't any change to db, not going thru gerrit, require a reindex?21:33
*** Jeffrey4l has quit IRC21:34
zaroclarkb: +1, restart the change.21:34
fungizaro: not sure. for example, we make changes to the accounts and account_external_ids tables to resolve duplicate accounts and don't reindex21:34
*** yamahata has quit IRC21:34
*** ijw has joined #openstack-infra21:34
fungii'm not entirely sure how to go about forcing it to redo the submit operation when gerrit already thinks it's submitted21:35
*** matt-borland has quit IRC21:35
*** baoli has joined #openstack-infra21:35
zarois abadone and new change an option?21:35
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Update query for TLS bug 1630664 for neutron
openstackbug 1630664 in OpenStack Compute (nova) "Intermittent failure in n-api connecting to neutron to list ports after TLS was enabled in CI" [Medium,Confirmed]
fungizaro: maybe, but it's a lot of changes apparently21:36
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Update query for TLS bug 1630664 for neutron
anteayaI missed the bit about why only solum is affected by this21:37
*** baoli has quit IRC21:37
*** inc0 has joined #openstack-infra21:37
*** baoli_ has joined #openstack-infra21:37
fungiwell, half a dozen anyway21:37
fungianteaya: no idea really, best guess is that gerrit got confused about something with the repo21:37
anteayahow odd21:38
*** baoli_ has quit IRC21:38
*** baoli has joined #openstack-infra21:39
*** baoli has quit IRC21:39
zarofungi, clarkb
fungino new changes have merged since the 13th21:40
fungifor solum21:40
zarodborowitz says try submit button or update db21:41
fungizaro: slightly different. these aren't merged to the repo (or at least AJaeger checked and said they weren't)21:41
*** baoli has joined #openstack-infra21:42
fungihrm, i've just spotted another inconsistency that may be related21:42
mriedempretty please - we have a docs change fail in the gate b/c of the ceph job21:43
mriedemthat shouldn't happen21:43
*** baoli has quit IRC21:43
mordredinfra-root: could one of your +A this:
*** baoli has joined #openstack-infra21:43
*** claudiub|2 has joined #openstack-infra21:43
fungicompare to and note that there's a change gerrit says has merged that doesn't appear in cgit21:44
openstackgerritMerged openstack-infra/elastic-recheck: Update query for TLS bug 1630664 for neutron
openstackbug 1630664 in OpenStack Compute (nova) "Intermittent failure in n-api connecting to neutron to list ports after TLS was enabled in CI" [Medium,Confirmed]
zaronot merged?  maybe just force it?21:44
*** simondodsley has quit IRC21:44
fungiyeah, i think the problem is that gerrit thinks it merged 385892 into the repo but it didn't21:44
jheskethmordred: +w21:45
fungiso all the other changes it wants to merge are now stuck behind that21:45
mordredjhesketh: thanks!21:45
zarofungi: try pressing the submit button?21:46
fungizaro: on which change?21:46
*** baoli has quit IRC21:46
*** baoli has joined #openstack-infra21:46
fungi`git log master` in ~gerrit2/review_site/git/openstack/solum.git definitely doesn't include the commit for 385892 even though gerrit says that was the last one to merge to master21:46
fungino, nevermind. that's stable/newton21:47
zaroall the merge pending ones.21:47
fungiso scratch that theory21:48
*** baoli has quit IRC21:48
fungizaro: i'll elevate my privileges and see if it gives me a submit button, even though it claims it's already submitted21:48
*** baoli has joined #openstack-infra21:49
zaroahh right. probably won't have the button then. can you do force push?21:50
openstackgerritMerged openstack-infra/puppet-openstackci: Treat yaml files as plain text
*** cody-somerville has quit IRC21:51
*** cody-somerville has joined #openstack-infra21:51
*** yamahata has joined #openstack-infra21:51
fungistrange, even in project-bootstrappers, and with a force reload of teh tab, i only get code review -1..+1 if i try to review one of those changes21:51
fungii'm going to try abandoning and restoring
fungiconfirmed, abandoning and restoring got it back to a ready to submit state, but then the submit button put it back to submitted merge pending21:56
*** thorst_ has quit IRC21:56
*** thorst_ has joined #openstack-infra21:57
fungiorg.eclipse.jgit.errors.ObjectWritingException: Unable to create new object: /home/gerrit2/review_site/git/openstack/solum.git/objects/16/56fbc8b649744c2785403dda480ee5dcc5baee21:57
fungithe backtrace looks similar to the one clarkb found21:58
*** jordanP has joined #openstack-infra21:58
fungithough looking in the logs, it's not the only one21:58
clarkbwhich should be just like the one that we get on a normal needs rebase21:58
fungiorg.eclipse.jgit.errors.ObjectWritingException: Unable to create new object: /home/gerrit2/review_site/git/openstack/puppet-glance.git/objects/87/6d2b4b38f6a7007c4b652513b417e84887ff3421:58
fungithough is only showing solum changes, so that may be a benign one21:59
clarkbfungi: ya its a normal error for anything that can't merge22:00
clarkbfungi: thats how the merge checker works22:00
fungiahh, the is:mergeable query seems to trigger some of these22:00
zaroturn up the logging for more info?22:00
fungiyeah, i was considering that as a next step22:00
*** gildub has joined #openstack-infra22:03
*** baoli has quit IRC22:03
*** baoli has joined #openstack-infra22:03
*** thorst_ has quit IRC22:05
wznoinskhi infra, any chance someone allow me edit info on my barcelona ticket in eventbrite?22:07
fungimriedem: i don't see where 366933 would stop docs changes from triggering ceph jobs unless i'm missing something. that change is totally about skipping jobs for the (frequent?) occurrence of adding lines to .gitignore22:07
wznoinskcurrently it says: The organizer has elected to not make this information editable. Contact the organizer if you made a mistake.22:07
*** jordanP has quit IRC22:07
clarkbwznoinsk: like your name/employer etc? I think we can get an email address for who can help you with that22:07
wznoinskclarkb, I saw 'edit' button this morning, see only the msg above now22:08
wznoinskclarkb, (UTC morning)22:09
clarkbwznoinsk: summit@openstack.org22:09
fungiwznoinsk: it's possible they've locked it now because they're dumping the registration data into the systems for the venue22:09
mriedemfungi: that's true, it was a docs change for adding the policy sample to nova docs - and the policy sample was added to .gitignore22:09
wznoinskclarkb, fungi roger22:09
mriedemso not a docs thing, but still22:09
*** esikachev has joined #openstack-infra22:09
fungimriedem: fwiw, i count 4 total changes to that file this year. still really seems like over-optimizing22:11
*** baoli_ has joined #openstack-infra22:12
*** baoli has quit IRC22:12
*** tykeal has left #openstack-infra22:13
mriedemdoes it hurt anything?22:13
mriedemprecious test nodes....22:13
fungii guess i shouldn't talk to nova people about review backlog, but... we have a pretty substantial review backlog (in addition to trying to fix things that are perpetually broken), so some up-front consideration to avoid changes that will marginally improve the efficiency of four changes out of tens of thousands or more a year is appreciated22:15
*** baoli_ has quit IRC22:15
*** esikachev has quit IRC22:15
*** baoli has joined #openstack-infra22:16
mriedemsorry, it's a 1 line change that was out there for awhile, i asked auggy to add that b/c we've seen things fail when they shouldn't, like i know the .gitreview change for stable/newton failed too when it shouldn't have22:16
fungialso, many of us would rather revert the skip-if support entirely (it's fiddly and causes far more problems than it fixes), so most of us just don't review chanegs that alter skip-if blocks because we don't want to be the ones responsible when they cause, say, half nova's jobs to cease being run and end up getting broken changes into the tree or whatever22:17
*** r-daneel has quit IRC22:17
*** baoli_ has joined #openstack-infra22:18
*** baoli has quit IRC22:18
*** abregman|afk has quit IRC22:19
*** amitgandhinz has quit IRC22:19
fungithe people who approved that feature have since regretted it22:19
*** Goneri has quit IRC22:20
*** thorst_ has joined #openstack-infra22:20
mriedemthe negative logic in the skip-if thing sucks, but it seems pretty useful to not run big resource hog jobs like multinode grenade on docs/unit test changes22:20
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton
fungii believe the idea for the future is to have better per-project filtering mechanisms (as opposed to the file filter which is only per-job today so a bit clunky for jobs that run across multiple projects)22:21
fungithe rule implementation for skip-if though makes it far too easy to skip jobs you didn't mean to, or completely deadlock certain changes because there is no job that matches the files it touches22:22
*** EricGonczer_ has quit IRC22:23
*** thorst_ has quit IRC22:24
fungizaro: clarkb: i'm going to `gerrit logging set-level debug`22:26
fungithen i'll do the abandon/restore/submit dance again and see if we get any more useful detail22:26
*** rossella_s has quit IRC22:28
*** cardeois has quit IRC22:28
*** rossella_s has joined #openstack-infra22:28
fungizaro: is that supposed to increase logging detail in the error_log? i still just see the same old backtrace22:29
zaroit's supposed to.22:29
zaroget-log to see if it's set?22:30
fungino, scratch that, i'm seeing an earlier timestamp on this backtrace22:30
*** csomerville has joined #openstack-infra22:30
*** baoli_ has quit IRC22:32
*** baoli has joined #openstack-infra22:33
fungizaro: `gerrit logging ls-level` apparently, but yes it shows it's set22:33
*** cody-somerville has quit IRC22:33
*** ijw has quit IRC22:33
*** baoli has quit IRC22:34
fungiokay, i do see additional detail, like "Found 5 existing heads" and "Running submit strategy MergeIfNecessary for 6 commits"22:34
*** baoli has joined #openstack-infra22:34
*** baoli has quit IRC22:35
*** baoli has joined #openstack-infra22:36
fungiclarkb: zaro:
*** baoli has quit IRC22:36
fungiwhy is it trying a 6-way merge there?22:37
clarkbwhat change did you abandon restore?22:37
fungithe change on which i submitted (388334) doesn't look like it has any related changes22:38
*** baoli has joined #openstack-infra22:39
clarkband its not in that list either?22:39
*** xarses has quit IRC22:39
fungiright, this is getting strange...22:39
clarkbI wonder if this is anoptimization of merge if necessary to reduce merge commits?22:39
*** SumitNaiksatam has quit IRC22:39
clarkbbasically pile together all the changes that can merge but makes conflicts more likely22:40
*** baoli has quit IRC22:40
fungimaybe if i abandon them all and restore just this one, then try to submit it again?22:40
zaroi'm confused.  which change did you submit?22:40
clarkbfungi: ya thatd what i am thinking if it is a bad optimization22:41
zaroit's 388334 dependent on that same topic list?  why do you say it doesn't ahve any related changes?22:42
*** baoli has joined #openstack-infra22:42
fungithere are lots of changes in other projects with that topic. there are no changes in solum depending on that change, and its parent is already merged and in the repo22:42
zaroahh, ok. i misread that panel again.22:43
fungiyeah, i misread it constantly22:43
*** dtardivel has quit IRC22:44
fungi"Opened branch refs/heads/master: commit 726a78929943c1d1bdaf4fc115cb79694fd79a1c" looks good at least. that's the master branch tip and its the parent of the 388334 change i attempted to submit22:45
*** baoli has quit IRC22:46
zarocan any other new change be sumbitted on this project?  maybe the stuck changes is causing all changes to be stuck?22:47
*** baoli has joined #openstack-infra22:48
fungimaybe... the one in the backtrace at least is
fungiperhaps if i abandon that one temporarily then the others can merge?22:49
openstackgerritRamy Asselin proposed openstack-infra/ansible-role-puppet: Update README with info about puppet apply
fungithat's also the one that was submitted earliest of them all, so it's possible22:50
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Allow ARA to test different versions of Ansible in the gate
*** pahuang has joined #openstack-infra22:51
zarofungi: worth a shot but i'm guessing they all need to be unstuck before anymore changes will get merged22:52
*** baoli has quit IRC22:52
zarotry to abandone one, then try abandoning all stuck ones?22:52
*** baoli_ has joined #openstack-infra22:53
*** marst has quit IRC22:53
asselin__does anyone know why this would fail?  Collecting ansible== I see the package here:
fungizaro: yeah, that's next on the list. this was trying to submit 388334 again after abandoning 385893:
fungithis time 388334 did show up in the list of what it was trying to merge at least22:55
fungithough the one which failed to merge this time per the backtrace was the commit for 38419922:56
*** baoli_ has quit IRC22:57
zarothat one appeared on both list.  so you gonna abandone one by one or all?22:57
funginot sure yet. i know there are also just plain issues we've seen in the past with n-way merges and jgit22:58
*** baoli has joined #openstack-infra22:58
fungiso that could be manifesting here22:58
zarowouldn't we see this same thing in other projects if that were true?23:00
fungiabandoned one more, submit still failed but now the number of changes it's attempting to merge has dropped to 423:00
fungizaro: i'm not sure, i think something happened to get multiple changes into a submitted merge pending state, and now none can merge because they all end up in this state and cause issues for each other due to gerrit trying to merge them all when you ask to merge one (because they're all already submitted)23:01
clarkbwe may see it in other cases but usually one merges then you tebase yhe others like mordred did earlier23:02
clarkbI wonder if it would do better serializing the merges23:02
fungisuccess! once i got the set down to 3 in a submitted merge pending state, i was able to get one to merge23:02
*** baoli has quit IRC23:03
zarocool, good thing you went one at a time :)23:03
fungiin fact, supporting this theory, it merged the 3 remaining which were in that state all at the same time when i resubmitted one of them23:04
fungii've restored the other 3 now23:04
fungianother possibility is that we somehow ended up with 2 changes in submitted merge pending which conflicted with one another, and getting one of them abandoned allowed the others to merge (just didn't know which one)23:05
fungibut i think that's less likely since i saw several different commits show up as unable to merge out of the set as i went through abandoning changes23:06
clarkbya which would be the reason to serialize23:06
*** gyee has quit IRC23:06
fungioh, though 385893 is showing up in proper merge conflict now23:06
zaroohh nice!23:07
*** rlandy has quit IRC23:08
fungieh, something strange is still afoot. i tried to submit 384199 since it didn't show in merge conflict and it's now submitted merge pending23:08
*** ihrachys has quit IRC23:09
fungilog says it was only trying to merge 1 commit at least23:09
fungii'm still not sure what this line means: [openstack/solum,refs/heads/master@23:08:11]: Found 5 existing heads23:09
fungiwhy would refs/heads/master have 5 existing heads?23:10
zarono clue either23:10
clarkbmaster newton mitaka kilo and something?23:10
fungioh, maybe it's choosing master out of 5 available heads23:10
clarkbshows 5 branches/heads23:11
fungicookiecutter, master, readme-start, stable/mitaka, stable/newton make 5, yeah23:11
*** rbrndt has quit IRC23:11
fungion the assumption that 384199 needed a rebase even though it didn't claim a merge conflict, i've rebased it23:12
fungii was able to get 387050 to merge successfully23:12
fungiand i've rebased 385893 to clear its merge conflict now23:13
*** mriedem has quit IRC23:13
fungiboth ended up being trivial rebases23:13
zaro384199 still says cannot merge?23:14
openstackgerritKevin Fox proposed openstack-infra/project-config: Add kolla-kubernetes multinode job & remove dead code
funginope, i rebased it and it needs workflow now23:16
fungiwaiting to see if its ci jobs re-pass23:16
fungisame for 38589323:16
*** dimtruck is now known as zz_dimtruck23:16
zarofungi: yeah, i see needs workflow but why does 384199 say "Cannot merge"?23:18
*** Sukhdev has quit IRC23:18
zarodoesn't that mean it's detecting a conflict still?23:18
*** sflanigan has joined #openstack-infra23:19
zaromaybe it needs a manual rebase, from command line?23:21
fungioh, that was off the side of my browser so i didn't notice it23:22
*** hongbin has quit IRC23:22
*** dave-mccowan has joined #openstack-infra23:24
*** inc0 has quit IRC23:24
*** thorst_ has joined #openstack-infra23:24
fungianyway, i've externally rebased that one now23:25
*** sdague has quit IRC23:25
*** gouthamr has quit IRC23:27
*** r-daneel has joined #openstack-infra23:28
*** maeker has quit IRC23:28
*** dizquierdo has quit IRC23:28
*** verdurin has quit IRC23:30
*** Jeffrey4l has joined #openstack-infra23:31
*** toabctl has quit IRC23:33
*** thorst_ has quit IRC23:33
*** r-daneel has quit IRC23:33
*** verdurin has joined #openstack-infra23:33
*** eharney has quit IRC23:35
*** markvoelker has quit IRC23:37
*** gildub has quit IRC23:37
*** dingyichen has joined #openstack-infra23:37
*** mountpoint has quit IRC23:39
*** Julien-zte has quit IRC23:39
*** toabctl has joined #openstack-infra23:44
openstackgerritRamy Asselin proposed openstack-infra/puppet-bandersnatch: Add serveraliases to bandersnatch::mirror
*** rhallisey has quit IRC23:45
openstackgerritRamy Asselin proposed openstack-infra/puppet-bandersnatch: Add serveraliases to bandersnatch::mirror
fungii've also reapproved (the rebased) 385893 to make sure it merges naturally on its own through the ci23:53
prometheanfireanyone know jeremy stanley on irc, dunno his nick23:53
fungiprometheanfire: nobody here but us chickens23:53
fungier, i mean that would be me, yes23:53
prometheanfireoh lol23:54
fungiwhat did i do now?23:54
*** gildub has joined #openstack-infra23:54
kfox1111fungi: got a quick sec to review this: ?23:59
*** mriedem has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!