Friday, 2015-04-17

davidlenwellI'm getting a python failure on a commit that I do not understand.. trying to make sense of the console output..
fungiyamamoto: only if the client implements a keepalive. it's an optional part of the protocol but i'm pretty sure the ssh implementation in gerrit doesn't send keepalives (it's usually off by default in most ssh clients and servers)00:00
mtreinishdavidlenwell: switch April to Apr and March to Mar00:00
mtreinishdavidlenwell: it doesn't match the regex in the traceback:
yamamotofungi: ok then ryu ci side of the connections might be completely dead?00:01
*** mahito has joined #openstack-infra00:01
*** emagana has joined #openstack-infra00:02
fungiyamamoto: yes, it's possible your gateway has an aggressive timeout and keeps silently dropping it without forging a tcp/rst or similar to close our end, and then your client reconnects and these slowly build up over time00:02
fungiyamamoto: but in theory whoever's taking care of your ci system will probably be able to find evidence in debug logs of connections being restarted00:03
*** mahito_ has joined #openstack-infra00:04
*** dims__ has joined #openstack-infra00:04
*** ZZelle_ has quit IRC00:04
*** mahito has quit IRC00:05
*** dims_ has joined #openstack-infra00:05
zarofungi: getting this error: Error: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class mysql::client00:06
zarofungi: in
yamamotofungi: makes sense thank you00:06
zarocan we add mysql::client to our repo?  if so how?00:07
yamamotobtw iirc metaplugin ci uses very similar setup with ryu ci.  doesn't it have the same issue?00:07
openstackgerritEmilien Macchi proposed openstack-infra/jeepyb: New hook: cherry_pick
fungizaro: i'm looking now00:08
openstackgerritAlice Chen proposed openstack-infra/jeepyb: Let pull request message be customized via file
fungiyamamoto: if that's the "metaplugintest" account, it has 2 open connections as of a few minutes ago00:09
*** dims__ has quit IRC00:09
fungiyamamoto: i just polled again, metaplugintest is still at 2 open ssh connections, neutronryu is at 17 (so it's gone down a few, but still comparatively much higher)00:09
anteayafungi: yes that is the username of the metaplugin ci:
anteayatokyo is clean!
*** bknudson has joined #openstack-infra00:12
*** davideagnello has quit IRC00:12
yamamotook then nat used at the location of ryu-ci is a suspicion... i'll take a look later today00:13
fungiyamamoto: thanks again00:13
* zaro needs to head out.00:13
*** sputnik13 has quit IRC00:13
fungizaro: i wonder if you should just use an "include mysql::client" there instead of instantiating teh class with no parameters00:13
funginibalizer: ^ ?00:14
fungi is the relevant file00:14
*** markmcclain has joined #openstack-infra00:14
*** luqas has joined #openstack-infra00:15
yamamotoanteaya: i've met the tortoise when i was walking.  it's nearby my home.00:15
fungizaro: nibalizer: er, i mean
*** davideagnello has joined #openstack-infra00:16
anteayayamamoto: ha ha ha00:17
anteayayamamoto: that is so cool00:17
anteayayamamoto: that was from reddit00:17
anteayayamamoto: have you seen it in the pink dress?00:17
anteayayamamoto: tokyo looks like it just freshly vaccumed00:18
yamamotoanteaya: unfortunately not :(00:18
anteayayamamoto: :(00:18
yamamoto(pink dress)00:18
*** tkelsey has joined #openstack-infra00:18
nibalizerfungi: :shrug: i've seen it both ways00:18
nibalizerI kinda prefer no-args over include but that is seriously just personal pref00:19
funginibalizer: then any insights into that error?00:19
anteayayamamoto: seeing the turtle would still be awesome though00:19
*** luqas has quit IRC00:20
*** sdake has joined #openstack-infra00:20
*** dmellado has quit IRC00:21
nibalizerfungi: yea it could be that ::mysql::client is needed, it could be that the mysql module didn't get installed right, it could be that the mysql module we are using doesn't have a ::client in it00:21
yamamotoin case you come to tokyo (eg. tokyo summit!) and want to meet the tortoise, it's around "Tsukishima" station.00:21
anteayayamamoto: thank you00:21
* anteaya notes Tsukishima station00:21
funginibalizer: great point. occam would say check that it exists at all00:21
anteayayamamoto: any other things to note about tokyo do share00:22
fungizaro: nibalizer: think it should be mysql::client::install there?
*** tkelsey has quit IRC00:23
fungioh, wait, does exist though00:24
nibalizerfungi: right but thats master of mysql00:24
nibalizerwe use $OLD00:24
*** sdake_ has quit IRC00:24
yamamotoanteaya: as tokyo is very big city, i'm afraid that many of sight-seeing spots might be too far from the summit location00:25
fungi0.6.1 got it00:25
yamamototime to go to office, later00:25
*** yamamoto has quit IRC00:25
funginibalizer: good call00:25
fungizaro: doesn't have a client class00:26
fungiwe probably need to just install the client package and move on00:26
EmilienMI'm having some beaker jobs failing, I don't know why exactly:
EmilienMnibalizer: have you seen that one already?00:27
*** yamamoto has joined #openstack-infra00:27
*** radez is now known as radez_g0n300:27
EmilienMI have the feeling Vagrant can't create the box00:27
*** yamamoto has quit IRC00:28
fungiBeaker::Hypervisor, found some none boxes to create00:29
fungithat's a neat and cryptic error00:29
*** SumitNaiksatam has quit IRC00:29
*** dmellado has joined #openstack-infra00:29
*** SumitNaiksatam has joined #openstack-infra00:30
EmilienMfungi: yeah, I just don't have access to the machine to debug00:30
EmilienMfungi: I'm investigating 'when the job ran successfuly' so I can dig the diff00:31
*** marun has quit IRC00:31
EmilienMtoday actually:
fungiEmilienM: here's a successful run from 10 days ago
EmilienMtoday !00:32
EmilienMI have one:00:32
fungi9 better!00:32
*** davideagnello has quit IRC00:32
EmilienMfungi: this one is from today,
EmilienMfungi: anyway. I'll continue to dig :)00:33
fungiEmilienM: cool00:33
*** zz_dimtruck is now known as dimtruck00:34
EmilienMfungi: just read your email about the hook00:36
EmilienMfungi: you make my life easier :)00:36
nibalizerEmilienM: yup aeen that many timea00:39
nibalizerso beaker on infra doesnt use vagrant00:39
nibalizerit uses sah to talk to the local machinw00:40
fungiEmilienM: when you get a chance to dig into that, let us know if you have questions. zuul post pipeline jobs to propose changes to projects when other changes merge is already a solved problem, so hopefully very little you'll really need to develop. also we try not to rely much on gerrit hooks00:40
nibalizerssh sorry, phone is hard00:40
anteayanibalizer: I was hoping it was ssh00:41
nibalizeranteaya: :)00:42
nibalizerEmilienM: anyways yea its buggy, i cant now but maybe tomorrw i can help you debug it00:43
anteayanibalizer: kudos to you to still aim for real words when using the phone00:43
*** thingee has joined #openstack-infra00:44
nibalizerEmilienM: you can also try using latest beaker, supposedly they fixed the issue that causes that00:44
lifelessfungi: / clarkb: a +2 on would be nice00:45
lifelesserm and +A00:45
lifelessand if both of you could see your way to +2 and +A on
*** patrickeast has quit IRC00:50
*** achanda has quit IRC00:51
*** Prophidys2 has quit IRC00:52
openstackgerritDavid Lenwell proposed openstack-infra/project-config: New repos for Akanda
EmilienMfungi: I still need to see how to detect the tag in commit message00:57
*** zhiwei has joined #openstack-infra00:58
*** achanda has joined #openstack-infra00:59
*** mwagner_lap has joined #openstack-infra01:00
*** emagana has quit IRC01:02
*** emagana has joined #openstack-infra01:03
*** emagana has quit IRC01:03
*** weshay has joined #openstack-infra01:05
fungiEmilienM: right, probably what you'd need is a job that runs in the post pipeline for your projects and retrieves the commit which triggered it. if it's a merge commit this gets a little trickier since it'll need to look at the parents to determine which is the one that just got merged to the branch (it'll be the one with no other parents itself i think?)01:06
openstackgerritDavid Lenwell proposed openstack-infra/project-config: New repos for Akanda
anteayanibalizer: you're the best01:06
fungiEmilienM: at that point, do a pattern match on the commit message and the rest is fairly similar to i expect01:07
*** tsg has quit IRC01:07
EmilienMfungi: well, I was more thinking at a task that 1/ find tags in commit messages and list branches 2/ for each branch, do the cherry-pick in checking the patch is not a cherry-pick itself01:08
*** baoli has joined #openstack-infra01:08
EmilienMbut I'll see how can I use the existing01:08
fungiEmilienM: yep, so that sounds like a couple conditional pattern matches. check out $ZUUL_REF and see if it has multiple parents. if it does, figure out which of its parents has no other parents and check that out instead01:09
fungiEmilienM: then pattern match in the commit message for a cherry-pick header and short-circuit the job successfully if found01:09
fungiEmilienM: if it wasn't already a cherry-pick, then check for a header indicating cherry-picks are desired and short-curcuit successfully if not found01:09
EmilienMfungi: with tag, like "run-cherry-pick" ?01:10
*** yamamoto has joined #openstack-infra01:10
EmilienM(in the commit message, like docimpact does)01:10
fungiEmilienM: better not to use the word "tag" here01:10
fungitags are another concept in git01:10
fungiyou're talking about headers in the commit message01:11
fungi(which git conventionally places in the last lines of the commit message, so tempting to call them "footers" but whatever)01:11
fungianyway, you probably want a header like Cherry-Picks: icehouse juno kilo01:12
fungior something01:12
fungii'm sure reviewers will gladly bikeshed the header key on the change with you once you get that far01:12
yamamotoanteaya: fungi: hi.  it turned out that we have upto 5-6 instances of ci for various purposes.  backup/testing/etc.01:12
EmilienMI'm investigating that way now, I'll produce something with your suggestions01:12
EmilienMfungi: good to know.01:12
EmilienMfungi: thx again01:12
fungiyamamoto: okay, that makes sense then. i was just trying to make sure it wasn't indicating a problem on your end01:13
*** gyee_ has quit IRC01:13
*** krtaylor has joined #openstack-infra01:13
yamamotofungi: if it seems to keep increasing, please let me know01:14
fungiEmilienM: actually, to bikeshed on that a bit, in launchpad bugs we've used the tag "backport-potential" so maybe Backport-Potential: icehouse juno kilo01:14
fungiyamamoto: i've just checked again and it seems to be holding steady between 17 and 20 so that's probably fine01:14
EmilienMfungi: will do, for consistency.01:14
yamamotofungi: thank you01:15
fungiyamamoto: you're welcome!01:15
*** notnownikki has quit IRC01:16
*** mattfarina has joined #openstack-infra01:17
*** mattfarina has quit IRC01:17
anteayayamamoto: thank you for looking into it01:17
HenryGCould a vendor (stackforge) repo hook in to global-requirements? Or is it a bad idea?01:18
yamamotoanteaya: you're welcome!  thank you for letting me know.01:19
fungiHenryG: as in get your repo-level requirements list changes checked against teh global list and get updates proposed to your repo from the global list? there are quite a few stackforge projects doing so already01:19
HenryGfungi: yes, exactly that. Do you have a pointer for setting it up?01:20
fungiHenryG: however trying to get changes/additions made to the global requirements list solely in support of a stackforge project is a lot harder01:20
HenryGfungi: not interested in that (yet)01:20
fungiHenryG: add the requirements-check job for your repo in openstack-infra/project-config:zuul/layout.yaml and add your repo to the list in openstack/requirements:projects.txt01:21
fungiHenryG: you should see plenty of examples in those two files01:21
HenryGfungi: awesome, thanks!01:21
fungiany time!01:22
*** smoser has quit IRC01:22
*** smoser has joined #openstack-infra01:24
*** smoser has quit IRC01:26
*** smoser has joined #openstack-infra01:26
HenryGfungi: ok, so if our repo has a requirement that is not in global-reqs will that cause a problem?01:30
EmilienMnibalizer: well, I reverted the Gemfile to use an old version of beaker. It works now. Willd debug tomorrow.01:31
nibalizerthats the other option01:32
Clintnibalizer: i think you don't know me, but tell aedo i say hi01:33
fungiHenryG: yes, though i think a "soft" option was recently added to the projects.txt which may prevent that from causing an issue. might want to double-check with someone who isn't about to head to bed01:33
*** Sukhdev has quit IRC01:34
HenryGfungi: thanks. I am also heading in that direction. I will pursue further tomorrow.01:34
*** mbacchi1 has quit IRC01:37
*** tqtran has quit IRC01:38
nibalizerClint: will do01:40
*** claudiub has quit IRC01:41
openstackgerritMerged openstack-dev/pbr: Ignore non-release related tags
*** rfolco has quit IRC01:46
*** harlowja is now known as harlowja_away01:47
*** dims_ has quit IRC01:54
*** trey has quit IRC01:54
boris-42fungi: hi01:57
*** trey has joined #openstack-infra01:59
boris-42fungi: I am trying to make voting coverage job01:59
boris-42fungi: so basically I did it, and it works when I run it locally02:00
boris-42fungi: but when I am trying to run it in infra I am getting strange errors02:00
boris-42fungi: and here is the patch
boris-42fungi: I really can't understand what is the difference between local and gate in this case..02:01
*** ivar-laz_ has joined #openstack-infra02:03
*** ivar-lazzaro has quit IRC02:06
*** ivar-laz_ has quit IRC02:09
fungiboris-42: i think git stash will only operate on files which are tracked02:10
fungiboris-42: particularly if you're wanting to stash files i think they can't match patterns in your .gitignore?02:11
fungiboris-42: yeah, they actually need to be added as tracked files even02:11
fungiif i create new files and don't git add them, then git stash says i have no local changes to save02:12
fungiboris-42: anyway, hope that helps. i'm heading to sleep but there are other people here who can probably help you debug further if you need02:13
boris-42fungi: ok thanks02:13
*** pal has joined #openstack-infra02:13
fungianyway, that looks like the failed assumption in your script. you're trying to stash the coverage report and then restore it after another checkout, and the log is saying there were no local changes to shash and no shash to apply02:14
fungier, no sTash to apply02:15
*** achanda has quit IRC02:20
*** dkranz has joined #openstack-infra02:21
*** achanda has joined #openstack-infra02:22
*** dmellado has quit IRC02:22
*** tiswanso has joined #openstack-infra02:24
*** ddieterly has joined #openstack-infra02:25
*** tiswanso has quit IRC02:27
*** tiswanso has joined #openstack-infra02:28
*** Krinkle is now known as Krinkle|detached02:31
openstackgerritDavid Lenwell proposed openstack-infra/project-config: New repos for Akanda
*** pal_ has joined #openstack-infra02:33
*** pal has quit IRC02:34
*** dimtruck is now known as zz_dimtruck02:34
*** dkranz has quit IRC02:37
*** weshay has quit IRC02:37
*** dmellado has joined #openstack-infra02:42
*** liusheng has joined #openstack-infra02:46
*** baoli has quit IRC02:46
*** achanda has quit IRC02:49
*** woodster_ has quit IRC02:50
*** sputnik13 has joined #openstack-infra02:53
*** unicell has quit IRC02:54
*** yamahata has quit IRC02:56
*** baoli has joined #openstack-infra02:57
*** Longgeek has joined #openstack-infra02:59
clarkb560km later I am back home03:03
*** Longgeek has quit IRC03:04
*** Longgeek has joined #openstack-infra03:06
*** liusheng has quit IRC03:07
*** liusheng has joined #openstack-infra03:07
*** otter768 has quit IRC03:08
*** baoli has quit IRC03:17
*** crc32 has joined #openstack-infra03:20
*** achanda has joined #openstack-infra03:20
*** pal_ has quit IRC03:22
*** tiswanso has quit IRC03:29
*** pal has joined #openstack-infra03:31
*** davideagnello has joined #openstack-infra03:33
*** sdake_ has joined #openstack-infra03:34
*** wenlock has quit IRC03:34
*** wwwBUKOLAYcom has quit IRC03:36
*** davideagnello has quit IRC03:38
*** sdake has quit IRC03:38
*** Sukhdev has joined #openstack-infra03:44
*** spzala has quit IRC03:45
*** trey has quit IRC03:50
*** gyee has quit IRC03:52
*** achanda has quit IRC03:52
*** luqas has joined #openstack-infra03:52
*** gyee_ has joined #openstack-infra03:52
*** sputnik13 has quit IRC03:53
*** gyee_ has quit IRC03:54
*** ddieterly has quit IRC03:56
*** luqas has quit IRC03:57
*** trey has joined #openstack-infra03:58
*** yfried|afk is now known as yfried__04:06
*** achanda has joined #openstack-infra04:07
*** achanda has quit IRC04:11
*** MarkAtwood has joined #openstack-infra04:11
*** thingee has quit IRC04:15
*** achanda has joined #openstack-infra04:16
*** yfried__ is now known as yfried|afk04:17
*** achanda has quit IRC04:18
*** yfried|afk is now known as yfried__04:19
*** tkelsey has joined #openstack-infra04:19
*** AJaeger has joined #openstack-infra04:20
*** achanda has joined #openstack-infra04:21
*** tkelsey has quit IRC04:24
*** ddieterly has joined #openstack-infra04:27
*** achanda has quit IRC04:29
*** ddieterly has quit IRC04:32
*** mrmartin has joined #openstack-infra04:37
*** crc32 has quit IRC04:38
*** crc32 has joined #openstack-infra04:38
*** achanda has joined #openstack-infra04:42
*** crc32 has quit IRC04:45
*** teran has quit IRC04:46
*** mmedvede has quit IRC04:46
*** ajmiller_ has joined #openstack-infra04:51
mordredclarkb: nicely done04:52
*** ajmiller has quit IRC04:54
*** mmedvede has joined #openstack-infra04:57
*** MarkAtwood has quit IRC05:01
*** dhritishikhar has joined #openstack-infra05:01
openstackgerritMerged openstack-infra/project-config: New repos for Akanda
*** xyang1 has quit IRC05:07
*** oomichi has joined #openstack-infra05:08
*** otter768 has joined #openstack-infra05:09
*** sputnik13 has joined #openstack-infra05:09
*** sputnik13 has quit IRC05:11
*** otter768 has quit IRC05:13
*** crc32 has joined #openstack-infra05:15
*** openstackgerrit has quit IRC05:21
*** openstackgerrit has joined #openstack-infra05:21
*** stevemar has joined #openstack-infra05:21
*** sks has joined #openstack-infra05:27
*** ddieterly has joined #openstack-infra05:28
*** harlowja_at_home has joined #openstack-infra05:31
*** ddieterly has quit IRC05:32
*** stevemar has quit IRC05:32
*** stevemar has joined #openstack-infra05:33
*** yfried__ is now known as yfried|afk05:34
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Fix major update_user issues
*** Sukhdev has quit IRC05:43
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Invalidate user cache on user create
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for user_cache
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: MonkeyPatch time.sleep in unit tests to avoid wait
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Test that deleting user invalidates user cache
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Fix major update_user issues
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: create_user should return the user created
*** Sukhdev has joined #openstack-infra05:50
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Test flavor cache and add invalidation
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks.
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Test flavor cache and add invalidation
*** tnovacik has joined #openstack-infra05:55
openstackgerritIan Wienand proposed openstack-infra/project-config: Add experimental glusterfs centos7 job
*** deepakcs has joined #openstack-infra05:57
deepakcsAJaeger, (do we need Sean's agreement before merging this one? )05:57
deepakcsAJaeger, the glusterfs CI job is failing bcos of ^^ not being merged, it would be great to have it in asap pls05:57
*** oomichi has quit IRC05:59
AJaegerdeepakcs: let me +2 it - and then let's see what other core reviewers say. Having sdague review it would be great06:01
deepakcsAJaeger, thanks, its been > 1 week that gluster CI is failing just bcos of this 1 testcase, otherwise it was green for many weeks06:02
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** rm_work is now known as rm_work|away06:03
AJaegerdeepakcs: that's unfortunate ;(06:04
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for caching in list_images
SpamapSso many caches to invalidate :-P06:08
*** pal has quit IRC06:10
deepakcsAJaeger, hence the need to get this in asap :)06:10
AJaegerdeepakcs: please ping other infra core reviewers later today (US morning)06:11
*** stevemar has quit IRC06:12
deepakcsAJaeger, sure06:12
*** emagana has joined #openstack-infra06:16
*** ibiris_away is now known as ibiris06:17
*** harlowja_at_home has quit IRC06:18
*** tkelsey has joined #openstack-infra06:21
*** armax has quit IRC06:22
*** ildikov has quit IRC06:23
*** tkelsey has quit IRC06:25
sdake_can someone with github superpowers change github's descrption of kolla06:27
openstackLaunchpad bug 1437562 in kolla "Uncorrect description on github" [Undecided,New]06:27
sdake_A commit hit the infra repo but it didnt change the name in github06:27
*** mrunge has joined #openstack-infra06:27
*** ddieterly has joined #openstack-infra06:29
*** bnemec has quit IRC06:32
*** hdd has joined #openstack-infra06:32
*** ddieterly has quit IRC06:33
*** scheuran has joined #openstack-infra06:36
*** Longgeek_ has joined #openstack-infra06:37
*** yamahata has joined #openstack-infra06:38
*** mrmartin has quit IRC06:38
*** Longgeek has quit IRC06:39
*** jyuso1 has quit IRC06:40
*** jyuso1 has joined #openstack-infra06:40
*** Kennan2 has quit IRC06:41
*** Kennan has joined #openstack-infra06:42
*** Sukhdev has quit IRC06:45
AJaegersdake_: so, you changed it in project-config but it was not hcanged in github?06:46
AJaegerAFAIK we have nothing implemented to change this automatically, might be a missing API on github site.06:47
sdake_daneyon hansen changed it iirc06:47
sdake_its really confusing to people because we abaonded kubernetes for kolla06:47
AJaegerLet's see whether an infra admin can help with manually doing it - please ping clarkb or fungi later today06:48
sdake_will do thanks06:49
sdake_its actually midnight time to ptfo :)06:49
*** achanda has quit IRC06:53
*** achanda has joined #openstack-infra06:56
*** yfried|afk is now known as yfried__06:56
*** gilliard is now known as gillllliard07:03
*** gillllliard is now known as gillllllliard07:03
*** mpavone has joined #openstack-infra07:03
*** zz_avozza is now known as avozza07:04
*** sdake has joined #openstack-infra07:06
*** jyuso has joined #openstack-infra07:06
*** ildikov has joined #openstack-infra07:08
*** e0ne has joined #openstack-infra07:08
*** jyuso1 has quit IRC07:09
*** sdake_ has quit IRC07:09
*** otter768 has joined #openstack-infra07:10
*** jcoufal has joined #openstack-infra07:10
*** achuprin has quit IRC07:10
*** yfried__ is now known as yfried|afk07:12
*** dtantsur|afk is now known as dtantsur07:13
*** teran has joined #openstack-infra07:14
*** markus_z has joined #openstack-infra07:14
*** otter768 has quit IRC07:14
*** ibiris is now known as ibiris_away07:14
*** ibiris_away is now known as ibiris07:15
*** clayton has quit IRC07:17
*** amotoki has quit IRC07:18
*** tkelsey has joined #openstack-infra07:18
*** dvorak has joined #openstack-infra07:18
*** soren_ has joined #openstack-infra07:19
*** luqas has joined #openstack-infra07:22
*** e0ne has quit IRC07:23
*** dhritishikhar has quit IRC07:23
*** vlaza has joined #openstack-infra07:25
*** e0ne has joined #openstack-infra07:28
*** ibiris is now known as ibiris_away07:28
*** asettle has quit IRC07:28
*** devvesa has joined #openstack-infra07:29
*** achanda has quit IRC07:29
*** ddieterly has joined #openstack-infra07:29
*** Hal has joined #openstack-infra07:29
*** alexpilotti has joined #openstack-infra07:30
*** Hal is now known as Guest5081607:30
*** andreykurilin__ has joined #openstack-infra07:32
*** achanda_ has joined #openstack-infra07:32
*** chlong has quit IRC07:33
*** dims__ has joined #openstack-infra07:33
*** ddieterly has quit IRC07:34
*** yamahata has quit IRC07:34
*** hashar has joined #openstack-infra07:35
*** jlibosva has joined #openstack-infra07:35
*** vponomaryov has quit IRC07:37
*** aarefiev_ has quit IRC07:38
*** dims__ has quit IRC07:38
*** mrmartin has joined #openstack-infra07:39
*** jlanoux has joined #openstack-infra07:40
*** achanda_ has quit IRC07:40
*** asettle has joined #openstack-infra07:41
*** yfried|afk is now known as yfried__07:44
*** mahito has joined #openstack-infra07:46
*** alex_xu has quit IRC07:46
*** jyuso1 has joined #openstack-infra07:46
*** heyongli has quit IRC07:47
*** heyongli has joined #openstack-infra07:48
*** alex_xu has joined #openstack-infra07:48
*** jyuso has quit IRC07:49
*** mahito_ has quit IRC07:49
*** jistr has joined #openstack-infra07:50
*** abhiram_moturi has quit IRC07:50
*** abhiram_moturi has joined #openstack-infra07:50
*** dtantsur is now known as dtantsur|brb07:51
*** MaxV has joined #openstack-infra07:52
*** andreykurilin___ has joined #openstack-infra07:53
*** andreykurilin__ has quit IRC07:53
*** e0ne has quit IRC07:57
*** arxcruz has joined #openstack-infra08:00
*** HeOS has quit IRC08:00
*** ibiris_away is now known as ibiris08:01
*** e0ne has joined #openstack-infra08:01
ttxAJaeger: o/08:01
ttxAJaeger: translations updates and RC2s are delayed due to stable/kilo being on fire, so I won't ping you about that today08:02
ttxAJaeger: if you want to help, you can review which is on hte critical path to fix all things08:02
*** mrmartin has quit IRC08:03
*** achanda has joined #openstack-infra08:06
*** sks has quit IRC08:08
*** mase_x200 has joined #openstack-infra08:08
*** sushilkm has joined #openstack-infra08:08
*** sushilkm has left #openstack-infra08:08
*** e0ne has quit IRC08:09
*** emagana has quit IRC08:12
*** e0ne has joined #openstack-infra08:14
*** sushilkm has joined #openstack-infra08:14
*** sushilkm has left #openstack-infra08:14
*** jyuso has joined #openstack-infra08:15
*** jyuso1 has quit IRC08:15
*** notnownikki has joined #openstack-infra08:17
*** e0ne has quit IRC08:18
*** turul has joined #openstack-infra08:18
*** unicell has joined #openstack-infra08:19
*** achanda has quit IRC08:20
*** jyuso has quit IRC08:20
*** jyuso1 has joined #openstack-infra08:20
*** hichihara has quit IRC08:21
*** derekh has joined #openstack-infra08:22
*** achanda_ has joined #openstack-infra08:22
*** zz_johnthetubagu is now known as johnthetubaguy08:22
*** yfried__ is now known as yfried|afk08:22
*** e0ne has joined #openstack-infra08:22
*** crc32 has quit IRC08:23
*** mrmartin has joined #openstack-infra08:24
*** sdake has quit IRC08:28
*** mrmartin has quit IRC08:29
*** turul is now known as afazekas08:30
ttxjohnthetubaguy: o/08:30
*** ddieterly has joined #openstack-infra08:30
johnthetubaguyttx: hello08:30
ttxjohnthetubaguy: I think it's safer to wait for Monday to open the RC2 window08:31
ttxthe stable/kilo branch testing is still in a lot of flux08:31
johnthetubaguyttx: I was going to ask about that for other reasons, but that totally makes sense08:31
AJaegerttx, will review now. Thanks for the heads-up08:31
ttxalso we expect a couple libs once the mess is fixed, and we can't really close the RC2 window before we have them (so that the right requirements are shipped in)08:31
johnthetubaguyttx: oh, very good point, eek08:32
ttxSo at this point... keep identifying issues and proposing backpotrs08:32
*** ssam2 has joined #openstack-infra08:32
ttxwe can open-close te RC2 quite fast if all patches are lined up08:32
ttxThanks for your understanding :)08:32
johnthetubaguyyes, thats a good point, I need to check up on some of the backports, but yes, makes sense08:32
AJaegerttx, approved08:33
ttxAJaeger: cool thx08:33
johnthetubaguyttx: no problem, thank you for doing your usual amazing job at tidying up all the loose ends we leave around the place!08:33
AJaegerttx, any guess for translation updates? I'm travelling all of Tuesday, so won't be able to help then08:34
ttxnp, would be more efficient if I wasn't sitting in a hotel room, but dhellmann stepped up and covered for me08:34
*** ddieterly has quit IRC08:35
ttxAJaeger: Monday-Wednesday at this point08:35
*** achanda_ has quit IRC08:35
johnthetubaguyttx: I know that feeling, hotel rooms are bad places for fire fighting08:35
*** mase_x200 has quit IRC08:35
AJaegerthanks, ttx08:35
* johnthetubaguy has a strange vision of lots of developers walking around tripping up over their badly tied shoe laces…08:35
*** dizquierdo has joined #openstack-infra08:36
*** mahito has quit IRC08:37
*** yfried|afk is now known as yfried__08:37
openstackgerritTim Kelsey proposed openstack-infra/project-config: Experimental bandit job for anchor
*** teran has quit IRC08:40
*** e0ne has quit IRC08:42
*** andreykurilin___ has quit IRC08:43
*** GheRivero has quit IRC08:45
*** _nadya_ has joined #openstack-infra08:45
*** e0ne has joined #openstack-infra08:46
*** aarefiev has joined #openstack-infra08:47
*** sushilkm has joined #openstack-infra08:49
*** sushilkm has left #openstack-infra08:49
*** pcaruana has joined #openstack-infra08:50
*** mpaolino has joined #openstack-infra08:50
*** e0ne has quit IRC08:51
*** e0ne has joined #openstack-infra08:53
*** dhritishikhar has joined #openstack-infra08:53
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Calico the new Fuel plugin project
*** GheRivero has joined #openstack-infra09:00
*** e0ne has quit IRC09:00
*** pal has joined #openstack-infra09:00
*** claudiub has joined #openstack-infra09:01
*** tnovacik has quit IRC09:02
*** e0ne has joined #openstack-infra09:04
*** yamamoto has quit IRC09:06
*** jyuso1 has quit IRC09:07
*** ildikov has quit IRC09:08
*** e0ne has quit IRC09:09
*** HeOS has joined #openstack-infra09:09
openstackgerritMerged openstack-infra/project-config: Temporarily ignore requirements on master libs
*** ibiris is now known as ibiris_away09:09
*** otter768 has joined #openstack-infra09:11
*** yfried__ is now known as yfried|afk09:12
*** mfmcdonagh has quit IRC09:14
openstackgerritStuart McLaren proposed openstack/requirements: Update the minimum glanceclient version
*** otter768 has quit IRC09:15
*** yfried|afk is now known as yfried__09:15
*** ihrachyshka has joined #openstack-infra09:16
*** ibiris_away is now known as ibiris09:19
*** fhubik has joined #openstack-infra09:19
*** HeOS has quit IRC09:20
*** HeOS has joined #openstack-infra09:20
*** teran has joined #openstack-infra09:21
*** dizquierdo has quit IRC09:22
*** ildikov has joined #openstack-infra09:23
*** Nikkau has joined #openstack-infra09:27
*** dhritishikhar has quit IRC09:28
*** ddieterly has joined #openstack-infra09:31
*** dhritishikhar_ has joined #openstack-infra09:33
*** ibiris is now known as ibiris_away09:33
*** yfried__ is now known as yfried|afk09:34
*** ddieterly has quit IRC09:35
*** ibiris_away is now known as ibiris09:35
*** dhritishikhar_ has quit IRC09:37
*** dhritishikhar_ has joined #openstack-infra09:37
*** avozza is now known as zz_avozza09:39
*** claudiub_ has joined #openstack-infra09:39
*** claudiub has quit IRC09:42
*** dhritishikhar_ has quit IRC09:42
*** jamielennox is now known as jamielennox|away09:49
*** e0ne has joined #openstack-infra09:50
*** skolekonov has quit IRC09:50
*** sushilkm has joined #openstack-infra09:52
*** sushilkm has left #openstack-infra09:53
*** pelix has joined #openstack-infra09:56
AJaegerttx, patch for master libs has merged09:58
*** cdent has joined #openstack-infra10:01
*** zz_avozza is now known as avozza10:07
*** enikanorov_ has quit IRC10:09
*** BobBall_AWOL is now known as BobBall10:12
*** sputnik13 has joined #openstack-infra10:12
*** cdent has quit IRC10:12
*** cdent has joined #openstack-infra10:13
*** enikanorov_ has joined #openstack-infra10:15
*** sputnik13 has quit IRC10:15
*** kmartin has quit IRC10:20
*** yfried|afk is now known as yfried__10:22
*** dhritishikhar_ has joined #openstack-infra10:23
*** dhritishikhar_ has quit IRC10:27
*** pc_m has joined #openstack-infra10:28
*** MaxV has quit IRC10:31
*** ddieterly has joined #openstack-infra10:31
*** MaxV has joined #openstack-infra10:33
*** dtantsur|brb is now known as dtantsur10:35
*** ddieterly has quit IRC10:36
*** e0ne is now known as e0ne_10:38
*** yfried__ is now known as yfried|afk10:38
*** ajmiller__ has joined #openstack-infra10:39
*** dhritishikhar_ has joined #openstack-infra10:41
*** e0ne_ is now known as e0ne10:42
*** ajmiller_ has quit IRC10:43
*** sushilkm has joined #openstack-infra10:43
*** sushilkm has left #openstack-infra10:44
*** pal has quit IRC10:45
*** Guest50816 has quit IRC10:46
*** _nadya_ has quit IRC10:47
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: New Fuel plugin project for Calico
*** jlibosva has quit IRC10:53
*** jlibosva has joined #openstack-infra10:54
*** jlibosva has quit IRC10:54
*** jlibosva has joined #openstack-infra10:55
*** jlibosva has quit IRC10:56
*** jlibosva has joined #openstack-infra10:56
ttxAJaeger: thx. Will approve the uncaps now10:57
*** yfried|afk is now known as yfried__10:57
*** jlibosva has quit IRC10:57
*** jlibosva has joined #openstack-infra10:57
openstackgerrityolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks.
ttxI see sdague already sent the rechecks11:02
*** dims__ has joined #openstack-infra11:02
*** _nadya_ has joined #openstack-infra11:03
*** _nadya_ has quit IRC11:03
sdaguettx: yeh11:03
sdaguethings were idle11:03
sdagueand it looked like they should work now11:03
*** _nadya_ has joined #openstack-infra11:03
ttxcool, trying to attract +2s by leaving comments. About to hop on a train11:03
ttxI should proabbly go now11:04
sdagueI think you are going to need to email PTLs directly11:04
sdagueclients get overlooked a lot11:05
*** sks has joined #openstack-infra11:05
sdaguethese are the times were we should have some uber team that can approve things like this11:05
*** vponomaryov has joined #openstack-infra11:06
AJaegersdague: the infra admins have these permissions11:07
AJaegerso, if really needed, we can ask fungi, jeblair, clarkb,...11:07
sdagueand then you also have things like - where ceilometer client doesn't reliably pass it's own unit tests :(11:07
*** yfried__ is now known as yfried|afk11:07
*** annegentle has joined #openstack-infra11:07
AJaegerThat's sad11:08
*** yfried|afk is now known as yfried__11:09
*** otter768 has joined #openstack-infra11:11
openstackgerritNikolay Fedotov proposed openstack-infra/zuul: Put valid username in repository url string
*** otter768 has quit IRC11:16
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: New Fuel plugin project for Calico
openstackgerrityolanda.robla proposed openstack-infra/system-config: Bump stankevich-python version
*** mwagner_lap has quit IRC11:26
*** zhiwei has quit IRC11:29
*** ddieterly has joined #openstack-infra11:32
*** ldnunes has joined #openstack-infra11:35
*** prad has quit IRC11:36
*** ddieterly has quit IRC11:37
*** deepakcs has quit IRC11:40
*** zxiiro has quit IRC11:41
sdaguegah, jenkins02 went south?11:43
*** julim has joined #openstack-infra11:46
Shrewsfungi: shade reviews 173994 and 173998 seem to have gone into limbo since being approved yesterday11:47
*** dprince has joined #openstack-infra11:47
*** Hal has joined #openstack-infra11:48
*** Hal is now known as Guest7526111:49
*** prad has joined #openstack-infra11:51
*** prad has quit IRC11:54
*** cdent has quit IRC11:55
Shrewsfungi: nm. gave them another kick in the pants. seemed to work11:58
*** e0ne is now known as e0ne_11:58
*** cdent has joined #openstack-infra12:01
*** rfolco has joined #openstack-infra12:06
*** aysyd has joined #openstack-infra12:06
*** e0ne_ has quit IRC12:08
*** e0ne has joined #openstack-infra12:10
*** Krinkle|detached is now known as Krinkle12:11
jrollttx: when does the (ironic) rc2 window open?12:12
*** annegentle has quit IRC12:12
*** weshay has joined #openstack-infra12:13
*** bknudson has quit IRC12:15
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Allow name or ID for update_subnet()
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Allow name or ID for update_router()
*** kgiusti has joined #openstack-infra12:19
*** mwagner_lap has joined #openstack-infra12:19
*** woodster_ has joined #openstack-infra12:19
sgordonrefreshing my memory it only possible to "un-abandon" a change in gerrit if you are core for the given project?12:20
openstackgerritJulia Kreger proposed openstack-infra/shade: Add test of OperatorCloud auth_type=None
mescanefhello guys, can somebody retrigger jenkins in ?12:24
*** sks has quit IRC12:25
*** dvorak is now known as clayton12:25
AJaegersgordon: owner and core can unabandon12:27
AJaegermescanef: just add "recheck" as comment, for details see
*** yfried__ is now known as yfried|afk12:29
*** yfried|afk is now known as yfried__12:29
mescanefAJaeger: ok tnx :-)12:32
*** _nadya_ has quit IRC12:33
mescanefAJaeger: btw, with these double quotes?12:33
*** ddieterly has joined #openstack-infra12:33
*** cdent has quit IRC12:34
StevenKmescanef: Nope, without quotes.12:34
*** doude_ has quit IRC12:35
*** cdent has joined #openstack-infra12:35
*** bknudson has joined #openstack-infra12:36
*** ildikov has quit IRC12:36
*** doude has joined #openstack-infra12:37
*** dalgaaf has quit IRC12:37
*** ddieterly has quit IRC12:37
*** jaypipes has joined #openstack-infra12:38
*** rkukura has joined #openstack-infra12:39
*** rkukura has quit IRC12:39
*** cdent has quit IRC12:41
*** cdent has joined #openstack-infra12:41
*** rlandy has joined #openstack-infra12:42
*** gordc has joined #openstack-infra12:44
*** fhubik is now known as fhubik_afk12:44
*** dhritishikhar_ has quit IRC12:44
*** dhritishikhar_ has joined #openstack-infra12:44
*** dizquierdo has joined #openstack-infra12:47
*** fhubik_afk is now known as fhubik12:48
*** dkranz has joined #openstack-infra12:51
*** ildikov has joined #openstack-infra12:51
*** baoli has joined #openstack-infra12:52
*** mase_x200 has joined #openstack-infra12:57
*** baoli_ has joined #openstack-infra12:58
*** zz_dimtruck is now known as dimtruck12:59
*** scheuran has quit IRC13:00
*** baoli has quit IRC13:01
openstackgerrityolanda.robla proposed openstack-infra/system-config: Bump stankevich-python version
*** dmellado has quit IRC13:02
*** xyang1 has joined #openstack-infra13:04
*** tiswanso has joined #openstack-infra13:07
sdaguesgordon: yes, core's can 'restore'13:07
*** otter768 has joined #openstack-infra13:12
*** smccully has joined #openstack-infra13:14
*** otter768 has quit IRC13:17
*** marcusvrn has quit IRC13:19
*** hdd has quit IRC13:21
*** doug-fish has left #openstack-infra13:22
*** cdent has quit IRC13:23
*** dustins has joined #openstack-infra13:23
*** radez_g0n3 is now known as radez13:24
openstackgerritJulia Kreger proposed openstack-infra/shade: Add test of OperatorCloud auth_type=None
*** mriedem has joined #openstack-infra13:26
*** peristeri has joined #openstack-infra13:26
*** dhritishikhar_ has quit IRC13:27
*** ddieterly has joined #openstack-infra13:28
*** markus_z has quit IRC13:29
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add a cli subcommand for deleting old rows
sdaguefungi: so... jenkins02 is failing a bunch of stuff randomly it seems13:30
*** dhritishikhar_ has joined #openstack-infra13:31
sdague502 proxy errors13:32
*** fhubik has quit IRC13:33
mescanefjenkins is nasty today :-(13:34
*** fhubik has joined #openstack-infra13:34
*** prad has joined #openstack-infra13:35
*** dansmith is now known as superdan13:35
fungisdague: i'll take care of it13:36
*** ddieterly has quit IRC13:36
fungijenkins02 is quiescing now. o13:37
fungii'll give the others a once-over13:37
*** pblaho_ has quit IRC13:37
*** doug-fish has joined #openstack-infra13:38
*** ddieterly has joined #openstack-infra13:39
fungijenkins02 seems to be the same thread leak we've been plagued by recently. and swift log uploads rely on being able to reach the jenkins web interface to pull the job console log. so when jenkins throws a proxy error, console log can't be retrieved and the job fails13:39
fungithough we do have a retry mechanism in place. this is a great opportunity to see if it's helping13:40
*** fhubik has quit IRC13:40
*** ddieterl_ has joined #openstack-infra13:40
*** pblaho has joined #openstack-infra13:41
*** yfried__ is now known as yfried|afk13:41
fungiit looks like the retry is still not happening
fungii wonder if that latest fix made it into our nodepool images13:42
fungiohh, we've got outdated images13:43
*** yfried|afk is now known as yfried__13:43
fungiguess now i'll dig into why image updates are having a problem13:43
*** ddieterly has quit IRC13:44
*** ddieterl_ has quit IRC13:45
odyssey4mefungi is it an option for projects to prepare their own images for checks?13:45
*** sigmavirus24_awa is now known as sigmavirus2413:45
*** mattfarina has joined #openstack-infra13:45
*** ddieterly has joined #openstack-infra13:46
fungiodyssey4me: we're trying to condense the number of images we're using as it is. we'd like to get down to one per distro/release and have jobs modify the workers as needed at run-time13:46
fungiyep, we've done something in the past week to break image updates for all bare-.* labels, but devstack-.* seems fine13:47
odyssey4mefungi sure, the reason I'm asking is because we've had to do a fair amount of fudging to remove stuff that's in the images when all we hope to get is a bare base install image, similar to that which one would get from a cloud provider13:48
fungiodyssey4me: yep, that's my top priority at the moment (when i'm not trying to just keep stuff running)13:48
*** ildikov has quit IRC13:49
odyssey4mefungi fair enough, good to know and thank you guys for keeping the ship afloat!13:49
fungiodyssey4me: we want to make it easy to pre-cache as much as possible on our images but pre-install as little as possible, and set up a framework for projects/jobs to be able to easily describe what they'll need so we know what to cache and what of that to install during the job13:49
*** pblaho has quit IRC13:49
fungiit's getting very close13:50
*** eharney has joined #openstack-infra13:50
odyssey4mefungi yeah, that makes sense13:50
aderyuginhi, guys! there is a problem with gate-mrano-devstack-dsvm job. it fails if we enable neutron in murano scripts. there is a fix for this problem we need to merge it13:50
anteaya<-- not a guy13:50
* anteaya moves on if only guys are needed today13:51
*** pblaho has joined #openstack-infra13:51
fungiaderyugin: that range you've specified might also conflict with local interface addresses in one or more of our cloud providers where we run these jobs. i'm checking now13:52
odyssey4meI wonder if anyone knows whether it's possible to follow up on the status of the Visa invitation letter for the summit. It's getting a little close for comfort for me to get the visa appointment and all that and all I'm missing is the letter.13:52
*** mpaolino has quit IRC13:53
fungiodyssey4me: i'll find out who you need to contact13:53
odyssey4mefungi thanks, I appreciate it!13:54
aderyuginfungi: ok, thanks13:55
*** _nadya_ has joined #openstack-infra13:56
fungiaderyugin: looks like it might pose a problem with eth0 in hpcloud. for example
aderyuginfungi: ok. I will try to find a better solution...13:57
fungiaderyugin: also whatever you pick, you'll probably need to be similarly mindful of eth1 in rackspace, e.g.
*** dimtruck is now known as zz_dimtruck13:58
*** jaypipes is now known as leakypipes13:59
fungiodyssey4me: i just talked to shari and she said to follow up with eventvisa@openstack.org14:00
jeblairfungi, pleia2: i think we missed some steps related to backups recently -- review, review-dev, and ask were throwing errors in cron because they didn't have local ssh keys for root to use to shell into the backup server.  i generated keys on those hosts14:00
odyssey4meexcellent, thanks fungi14:00
*** zhiwei has joined #openstack-infra14:00
fungijeblair: good to know--thanks. i don't think i helped set any of those up, but i'll be sure to keep an eye out for that in the future14:01
fungiit's probable we simply added a puppet class for backups on ask but hadn't gotten around to configuring that yet14:02
jeblairyeah, i think we missed review and review-dev with the server move14:02
jeblaircause that private key isn't puppeted (perhaps it should be)14:02
jeblairand yeah, ask just wasn't done yet14:02
*** amitgandhinz has joined #openstack-infra14:03
fungiand i agree, server moves for review and review-dev are a likely cause. i'll remember to update those when we rebuild anything which is getting backed up14:03
*** bnemec has joined #openstack-infra14:03
*** mriedem has quit IRC14:03
*** yfried__ is now known as yfried|afk14:03
*** ildikov has joined #openstack-infra14:04
jeblairi have removed oubiwann from the root distribution list for lists.o.o; his mail is bouncing14:05
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: DO NOT MERGE: Example of new getter
fungiError: Invalid parameter manage_firewall on Class[Postgresql::Server] at /root/system-config/modules/openstack_project/manifests/slave_db.pp:10014:05
fungilooks like that's probably our culprit14:05
fungiseeing if i can work out a fix now14:05
*** wenlock has joined #openstack-infra14:05
jeblairfungi: i'm guessing we have no puppet apply test for that module :/14:06
*** abhiram_moturi has quit IRC14:06
openstackgerritAlexey Deryugin proposed openstack-infra/project-config: [gate-murano-devstack-dsvm] Fix bug with neutron misconfiguration
*** mriedem has joined #openstack-infra14:06
fungithat's entirely possible14:06
fungior it's shadowed by a --noop related dependency14:07
openstackgerritAlexey Deryugin proposed openstack-infra/project-config: [gate-murano-devstack-dsvm] Fix bug with neutron misconfiguration
*** sarob has joined #openstack-infra14:07
*** zz_dimtruck is now known as dimtruck14:07
fungithis is odd. git blame says we haven't touched slave_db.pp in months, nor changed the version of puppetlabs-postgresql any time recently14:09
*** cdent has joined #openstack-infra14:09
*** yfried|afk is now known as yfried__14:09
fungii wonder if a recent module dep change is pulling in a newer version14:10
fungiError: Could not install module 'puppetlabs-postgresql' (v4.0.0)14:11
fungiModule 'puppetlabs-postgresql' (v4.3.0) is already installed14:11
*** e0ne is now known as e0ne_14:11
*** signed8bit has joined #openstack-infra14:12
*** sdake has joined #openstack-infra14:12
jeblairfungi: so, funny story for why i'm reading system cron mail...14:12
*** bswartz has quit IRC14:13
fungii could use a funny story right about now14:13
yolandahi jeblair, i saw hpcloud was blocked yesterday from upstream. That means no more test on nodepool patch? how is status now?14:13
yolandabtw, i submitted the change for stop() and a test14:13
*** e0ne_ is now known as e0ne14:13
jeblairfungi: the vm that my mail server is on had a filesystem error, and after the fsck was complete, everything looked great except pretty much all of the files in /var ended up as orphaned inodes.14:14
fungiyolanda: we've stopped using hpcloud at their request and haven't added them back yet. afaik we've heard no status update from them, but it sounded like it could be next week at least14:14
fungijeblair: oh, ouch14:14
fungithat's got to be a pain to untangle14:15
jeblairfungi: yeah, so... i mean, the good news is that i now have a shiny new mail server running trusty (instead of lucid)14:15
mordredyolanda: I'm supposed to ask about a status on some things they're doing on Monday14:15
yolandamordred, i heard something abou database cleanups? it may be index?14:16
mordredyolanda: but I believe that will be me getting a completion date from them on Monday - not them being complete on monday14:16
jeblairfungi: the current ETA for having all of my mail restored onto it is 16:11 PDT today14:16
jeblairfungi: but there was a window where i had deliveries but no sieve script, so i'm sorting through that :)14:16
*** deepakcs has joined #openstack-infra14:17
yolandamordred, jeblair, did you see some significant update in the time it was running, or is not possible to tell?14:18
deepakcsHi All, can someone from infra core pls bless ( It has +2 from ajaeger already)14:19
deepakcsthe glusterfs CI job is failing bcos of ^^ not being merged, it would be great to have it in asap pls14:20
deepakcsits been > 1 week that gluster CI is failing just bcos of this 1 testcase, otherwise it was green for many weeks14:20
deepakcssdague, fungi anteaya ^^14:20
*** ayoung has quit IRC14:20
*** yfried__ is now known as yfried|afk14:21
jeblairyolanda: i have not looked yet, sorry.  i may not get a chance to for a while.14:22
jeblairfungi: so i might be a bit distracted today14:22
fungijeblair: got it14:22
clarkbfungi we should revert the puppetdb update change to fix postgres I think14:22
fungii've got a lunch errand and another brief mid-afternoon errand, but will work on the etherpad for the 22:00 utc gerrit maintenance after i get back14:23
*** signed8bit has quit IRC14:23
fungiclarkb: i agree. reviewing historical update logs, we've got something in our modules list which is pulling in puppetlabs-postgresql. 7 days ago it couldn't install 4.0.0 because 3.4.2 was already installed. after that patch merged we started failing to install 4.0.0 because 4.3.0 was now installed14:24
*** sabeen has joined #openstack-infra14:24
fungiso something ahead of that module in the modules.env is depending on it and when we bumped the version it increased its dep version to one incompatible with our openstack_project::slave_db class14:24
fungiclarkb: alternatively, we could just try moving puppetlabs-postgresql to earlier than puppetlabs-firewall or puppetlabs-puppetdb, whichever is the one pulling it in early14:25
clarkbya its puppetdb14:25
clarkbwell puppetdb is broken for other reasons so reverting it to start again seems sane14:26
fungisubmitting that now14:26
*** alexpilotti has quit IRC14:27
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Revert "Bump puppetdb version"
*** Nikkau has quit IRC14:28
fungiunfortunately our images for today are already building, but aside from the console log retry fix i don't know of anything else pressing which would demand going to the trouble of restarting them all by hand14:29
yolandaclakb, so i found problems with module dependencies sometimes, we have a patch downstream to install modules on the right order, if not, it was doing crazy things. But i got that rejected upstream, because there was an intentaion to install every module from source. How is that? can i help with it?14:29
*** mwagner_lap has quit IRC14:29
yolandaclarkb, not clakb :)14:31
clarkb is the change, it needs a rebase and an update to make modules use the same versions of what we currently install14:31
*** hashar has quit IRC14:32
*** sabeen2 has joined #openstack-infra14:32
*** sarob has quit IRC14:33
*** dims__ has quit IRC14:33
jeblairFrom: (Cron Daemon)14:33
jeblairSubject: Cron <root@ask> /bin/bash /srv/askbot-sites/slot0/cron/send_email_alerts.sh14:33
jeblair{'': (501, '<>: missing or malformed local part')}14:33
*** wwwBUKOLAYcom has joined #openstack-infra14:33
jeblairthe email has 2817 lines worth of that last bit ^14:33
*** zz_jgrimm is now known as jgrimm14:34
*** hashar has joined #openstack-infra14:34
jeblairfungi: i'll forward that to mrmarton and cc you14:34
fungii think reed, mrmartin and Evgeny were working on getting the askbot e-mail sender set correctly14:35
*** tonytan4ever has joined #openstack-infra14:35
jeblairfungi: oh, should i just discard for now, or still send it?14:35
*** annegentle has joined #openstack-infra14:35
*** otter768 has joined #openstack-infra14:35
fungijeblair: please forward it along. it'll keep me from digging out a copy14:35
*** otter768 has quit IRC14:36
jeblairdone.  now i will see if i have my outgoing email config set up correctly :)14:36
*** sabeen has quit IRC14:36
*** wenlock has quit IRC14:37
*** hashar is now known as hasharAway14:37
*** stevemar has joined #openstack-infra14:37
fungispeaking of, if anyone wants to play around with it we have a zuul configuration help request ongoing at
jeblairthat seems misleading14:39
*** mase_x200 has quit IRC14:39
fungii agree14:39
fungii seriously doubt the reason his jobs aren't triggering is his reporter14:39
jeblairoh, er, i meant zuul questions on ask14:40
*** MarkAtwood has joined #openstack-infra14:40
fungiwell, yes that too14:40
*** soren_ has quit IRC14:40
fungiwe get occasional infra project help requests pop up on there. i've subscribed myself to keywords likely to spot them and if they rathole too far i try to redirect them to irc or our ml14:41
*** marun has joined #openstack-infra14:41
jeblairwow.  i don't.  ok.14:41
fungiwhich i'm about to do to this one14:41
*** mtanino has joined #openstack-infra14:41
jeblairwoot, i can send mail14:42
*** Daisy has joined #openstack-infra14:44
*** carl_baldwin has joined #openstack-infra14:44
*** sdake_ has joined #openstack-infra14:47
*** sushilkm has joined #openstack-infra14:49
*** sushilkm has left #openstack-infra14:49
*** Daisy_ has joined #openstack-infra14:49
*** sdake has quit IRC14:50
*** annegentle has quit IRC14:51
*** annegentle has joined #openstack-infra14:51
deepakcsTrying 2nd time - can someone from core pls review - It has +2 from AJaeger , thanks14:52
*** Daisy has quit IRC14:52
*** Daisy_ is now known as Daisy14:52
*** dhritishikhar_ has quit IRC14:53
*** avozza is now known as zz_avozza14:54
*** Daisy_ has joined #openstack-infra14:55
*** dizquierdo has quit IRC14:56
*** Daisy has quit IRC14:57
*** Daisy_ is now known as Daisy14:57
*** prad has quit IRC14:57
*** sgordon has quit IRC14:58
*** nelsnelson has joined #openstack-infra14:58
sdake_jeblair happen to be around?  Ajeager suggested I consult you on an issue with project-config14:58
*** dhritishikhar_ has joined #openstack-infra14:58
*** yfried|afk is now known as yfried__14:59
clarkbdeepakcs: I don't think the regex is correct14:59
clarkbdeepakcs: it says not anything or not volume boot pattern14:59
*** dhritishikhar_ has quit IRC14:59
*** dhritishikhar_ has joined #openstack-infra14:59
*** markvoelker has joined #openstack-infra15:00
*** markvoelker has quit IRC15:00
clarkbdeepakcs: also sdague did comment after you asked earlier15:00
sdake_clarkb ajeager suggested you may be able to help too15:00
*** carl_baldwin has left #openstack-infra15:00
clarkbsdake_: what is the issue?15:00
*** markvoelker has joined #openstack-infra15:00
*** prad has joined #openstack-infra15:01
*** yamahata has joined #openstack-infra15:01
sdake_clarkb need someone with github superpowers to manually change somethiing15:01
openstackLaunchpad bug 1437562 in kolla "Uncorrect description on github" [Critical,Confirmed] - Assigned to Steven Dake (sdake)15:01
sdake_we changed project-config, but it did not update the github description15:01
sdake_and I get quesitons on it *all the time*15:02
sdake_our project has abandoned kubernetes15:02
sdake_naturally when people see "kubernetes and openstack" they get confused :)15:02
deepakcsclarkb, sdague thanks, will have BharatK add a new job then, if this method is not recommended15:03
*** radez is now known as radez_g0n315:03
deepakcsclarkb, I am not an expert in regex, will lookout for some doc to figure the syntax. BharatK did validate on tempest cmdline, so I thought the regex is correct.15:04
*** dims__ has joined #openstack-infra15:04
clarkbsdake_: ok, I want to say we cut out description updates because the github api would throttle us15:05
clarkbsdake_: I can update manually15:05
sdake_clarkb that woudl rock :)15:05
sdake_normally i wouldn't care but its impacting our ability to attract devs15:06
sdake_thanks :)15:06
sdake_that was fast15:06
*** radez_g0n3 is now known as radez15:06
*** zhiwei has quit IRC15:06
*** yfried__ is now known as yfried|afk15:09
deepakcssdague, clarkb thanks for the quick review.15:09
*** zhiwei has joined #openstack-infra15:10
mordredclarkb: perhaps we should stop setting the description field on gh repo creation15:10
mordredclarkb: because we don't change them later on updates to that file - so it's kind of misleading15:10
*** dizquierdo has joined #openstack-infra15:10
*** wenlock has joined #openstack-infra15:10
mordredor maybe put ina stock description "this is a read-only mirror of code hosted at review.o.o" or something15:11
clarkbmordred: it seems that all of our use of github is misleading15:11
mordredclarkb: yah - but sometimes it leads to funny rage comments15:11
*** ildikov has quit IRC15:11
sdake_sdague on this version uncap on python-magnumclient, what is happening with jenkins?15:12
sdake_the check is passing yet veritifed -1?15:12
*** sslypushenko has joined #openstack-infra15:12
clarkbfungi: I am +2 on the puppetdb bump revert15:12
*** imcsk8 has quit IRC15:13
*** imcsk8 has joined #openstack-infra15:13
clarkblooking at I have no idea how nova's dhcp server is working15:14
clarkbwhich makes me wonder if that is the actual reason people don't deploy with dhcp15:14
jesusaurusmordred: i think that some stock 'this is read-only' text would be good, but im also not sure why we mirror to github in the first place15:14
mordredclarkb: there's only one person out there who doesn't run dhcp15:14
mordredclarkb: and that's rackspace15:14
mordredall the other clouds run it15:14
mordredclarkb: that does look ugly though15:15
clarkbmordred: maybe they are more away of nova's oddities than others? In any case I see the discover requests from VMs but no responses15:15
clarkband that is tcpdump -i any15:15
mordredjesusaurus: discoverability - people expect to be able to go clone a repo from github15:15
sdaguesdake_: I don't know, have you looked at logs?15:15
sdake_sdague admittidely not15:16
mordredjesusaurus: and if we didn't publish them there, someone else would, and then we wouldn't have any control over it15:16
sdake_I assumed pass means jenkins liked the results :)15:16
*** jistr is now known as jistr|mtg15:16
mordredjesusaurus: those are my two reasons15:16
*** dhritishikhar_ has quit IRC15:17
*** dannywilson has joined #openstack-infra15:18
*** rm_work|away is now known as rm_work15:18
*** harlowja_at_home has joined #openstack-infra15:19
jrollclarkb: are you saying we maybe actually did something right? :o15:22
jesusaurusmordred: those both make a lot of sense15:22
*** armax has joined #openstack-infra15:22
*** dims__ has quit IRC15:22
clarkbjogo: after reading nova net docs, nova net hosts setup a local dnsmasq that assigns private IPs as assigned by nova. dnsmasq logs to syslog, so we should check syslog on the jobs that fail due to a VM not getting a lease. I would bet there is a race between booting the node and telling dnsmasq what IP to assign15:22
clarkbjroll: or more worked around problems by not ifxing them and instead making crazy system instead :)15:23
*** Ala has quit IRC15:24
*** tonytan4ever has quit IRC15:25
*** ZZelle has quit IRC15:25
*** ZZelle has joined #openstack-infra15:26
*** yfried|afk is now known as yfried__15:26
*** ildikov has joined #openstack-infra15:26
*** Adri2000 has quit IRC15:28
openstackgerritMonty Taylor proposed openstack-infra/shade: Update register_machine to use tasks
openstackgerritMonty Taylor proposed openstack-infra/shade: Update unregister_machine to use tasks
*** tonytan4ever has joined #openstack-infra15:28
*** dprince has quit IRC15:28
*** Adri2000 has joined #openstack-infra15:29
*** Adri2000 has quit IRC15:29
*** Adri2000 has joined #openstack-infra15:29
*** mpavone has quit IRC15:33
fungiokay, popping out to lunch, back in a bit15:34
*** zz_avozza is now known as avozza15:34
*** dtantsur is now known as dtantsur|afk15:37
*** nfedotov has joined #openstack-infra15:37
*** dangers_away is now known as dangers15:39
*** Swami has joined #openstack-infra15:39
fungialso, jenkins02 has been cleaned up and restarted15:39
*** thedodd has joined #openstack-infra15:39
*** tsg_ has joined #openstack-infra15:43
*** kmartin has joined #openstack-infra15:43
openstackgerritMehdi Abaakouk proposed openstack/requirements: Bump kombu version to 3.0.9
*** jistr|mtg is now known as jistr15:44
*** zhiwei has left #openstack-infra15:44
*** Longgeek_ has quit IRC15:44
openstackgerritMerged openstack-infra/shade: Add some unit test for create_server
*** Daisy has quit IRC15:48
openstackgerritMonty Taylor proposed openstack-infra/puppet-diskimage_builder: Fix .gitreview file
openstackgerritMonty Taylor proposed openstack-infra/puppet-diskimage_builder: Consume vhd-util from infra vhd-util PPA
openstackgerritMonty Taylor proposed openstack-infra/puppet-diskimage_builder: Fix metadata.json file
*** yfried__ has quit IRC15:48
*** enikanorov__ has joined #openstack-infra15:48
*** deepakcs has quit IRC15:49
*** esker has joined #openstack-infra15:50
*** ildikov has quit IRC15:50
*** enikanorov_ has quit IRC15:51
*** soren_ has joined #openstack-infra15:52
*** sslypushenko has quit IRC15:54
*** sslypushenko has joined #openstack-infra15:55
openstackgerritMonty Taylor proposed openstack-infra/pypi-mirror: Workflow documentation is now in infra-manual
mordredjeblair: not urgent - but the "only show projects with unreview changes" keeps things in the unreviewed count that I've marked as viewed15:59
mordredjeblair: so I can have a project with 3 unreviewed changes show up in the list - but when I go to it, there are no changes15:59
*** gyee has joined #openstack-infra15:59
jeblairmordred: math is hard16:00
ttxjroll: I'd say Monday, so that we have a bit more time polishing the stable/kilo situation16:00
*** r-daneel has joined #openstack-infra16:00
jrollttx: ok, thanks16:00
ttxjroll: in hte mean time, fix all issues in master and pile up backpotrs, we'll approve them in one go16:00
*** ivar-lazzaro has joined #openstack-infra16:00
mordredjroll: I agree16:01
*** devvesa has quit IRC16:01
mordredjroll: I mean, I agree with jroll and all ... but16:01
mordredjeblair: I agree16:01
*** armax has quit IRC16:01
*** garyk has quit IRC16:01
jrollttx: yep, already done :)16:02
*** ivar-laz_ has joined #openstack-infra16:02
*** dprince has joined #openstack-infra16:02
*** ivar-laz_ has quit IRC16:03
*** ivar-laz_ has joined #openstack-infra16:03
*** annegentle has quit IRC16:05
*** ivar-lazzaro has quit IRC16:05
*** tqtran has joined #openstack-infra16:06
pleia2good morning16:06
*** dkranz has quit IRC16:07
waynrpelix: ping16:07
pleia2jeblair: re: review-dev, I think that one was actually disabled before we moved the *backups* server to make more space and I didn't follow up to see about re-enabling it, sorry about that, the other two were server move + new server that weren't quite finished up16:08
pelixwaynr: pong16:08
*** tqtran is now known as tqtran_afk16:08
*** yamahata has quit IRC16:08
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding
waynri've got a jjb question for ya related to local_yaml.py16:08
*** harlowja_at_home has quit IRC16:09
waynri'm running into an issue in the trigger-parameterized-builds publisher where i can't seem to reorder the parameter settings for the downstream job (so the current job's build parameters will always override predefined or properties file parameters)16:09
*** pcaruana has quit IRC16:10
waynrin the jenkins web ui the order in which these settings are added to the trigger-parameterized-builds web element matters (presumably because it leads to a different ordering of xml elements)16:10
*** sabeen2 has quit IRC16:11
*** e0ne has quit IRC16:12
waynranyway, i noticed that in dictionaries seem to be created as OrderedDict but it's not clear to me if this means that the order in which keys are listed in a yaml file leads matters16:12
pelixwaynr: looking, unlikely to be the yaml because that was specifically switched to use OrderedDict to deal with tests, but also it was reasoned that the user should control the order of the elements when dealing with lists and dicts16:12
*** unicell has quit IRC16:13
zaropelix: fyi, will create a new jjb release today.16:13
pelixwaynr: the order of the keys in the yaml is important, but the module code converting it to XML may not be written in a way that respects the order16:13
openstackgerritMonty Taylor proposed openstack-infra/puppet-diskimage_builder: Fix .gitreview file
openstackgerritMonty Taylor proposed openstack-infra/puppet-diskimage_builder: Consume vhd-util from infra vhd-util PPA
waynroh i know where the parameter ordering problem is in jenkins_jobs/modules/ i'm just curious if i can rely on strict ordering as seen in the yaml file when i work on the trigger_parameterized_builds publisher16:13
openstackgerritMonty Taylor proposed openstack-infra/puppet-diskimage_builder: Fix metadata.json file
*** jlanoux has quit IRC16:14
greghaynesmordred: ^ do we have puppet installing a dib that installs puppets?16:14
waynrzaro: is there any particular jjb release cadence?16:14
waynrpelix: thanks, that answers my question :)16:14
pelixwaynr: :)16:14
pelixyeap, you can rely on the order of the yaml16:15
mordredgreghaynes: it's puppets all the way down16:15
*** tonytan4ever has quit IRC16:15
mordredgreghaynes: that's the "install dib and also the dependencies and also the ppa with vhd-util"16:15
greghaynesthat reminds me, I need to binutils hack16:16
*** claudiub_ has quit IRC16:16
zaroi try to do it at least every 3 months.16:16
zarowaynr: ^16:16
waynrokay, was just curious16:16
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Only query jenkins plugins if config provided
*** marcusvrn has joined #openstack-infra16:17
*** wenlock has quit IRC16:17
pelixzaro: I presume that means, please don't approve anything ;-)16:18
*** mbacchi1 has joined #openstack-infra16:18
*** luqas has quit IRC16:18
*** salv-orl_ has quit IRC16:19
*** mriedem1 has joined #openstack-infra16:19
*** subscope has joined #openstack-infra16:20
*** mriedem has quit IRC16:21
zaropelix: yeah, probably a good idea for today.16:21
*** wenlock has joined #openstack-infra16:21
*** kmartin_ has joined #openstack-infra16:21
*** kmartin has quit IRC16:21
*** marcusvrn1 has joined #openstack-infra16:22
*** marcusvrn2 has joined #openstack-infra16:22
*** marcusvrn has quit IRC16:23
*** marcusvrn3 has joined #openstack-infra16:23
*** dizquierdo has quit IRC16:23
*** claudiub has joined #openstack-infra16:24
*** salv-orlando has joined #openstack-infra16:24
*** dims__ has joined #openstack-infra16:24
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Add InvalidAttributeError and MissingAttributeError.
*** soren_ has quit IRC16:26
*** marcusvrn1 has quit IRC16:26
*** marcusvrn2 has quit IRC16:26
* fungi returns16:27
KrinkleHm.. is there a way to enqueue zuul events from the command line for ref-updated / tag creations? Would make it a lot easier to backfill generated documentation for recent releases, since we only recently added a pipeline for it16:27
*** patrickeast has joined #openstack-infra16:27
KrinkleI've used the enqueue command for change-merged like this:16:27
Krinklezuul enqueue --trigger gerrit --pipeline postmerge --project unicodejs --change 201944,216:27
*** thingee has joined #openstack-infra16:27
KrinkleBut not sure what paremeters to pass for --pipeline publish (listens to ref-updated)16:28
jeblairKrinkle: i think we may not have fleshed out the enqueue command enough to support those yet16:29
KrinkleI would trigger them via Jenkins, but then additional functions to create environment variables don't get run, so will need to compute them by hand.16:30
*** Guest75261 has quit IRC16:30
KrinkleExample of a regular ref-update job. Seems to have an odd "Old revision: 000000000000"16:31
KrinkleNot sure what's that for.16:31
*** yamahata has joined #openstack-infra16:31
jeblairKrinkle: that's what gerrit sends on a ref-update if the ref is being created16:32
jeblair(so tag creation in this case)16:32
KrinkleAh, there's a delete event as well16:32
jeblairKrinkle: the opposite is true if a ref is deleted16:32
openstackgerritSpencer Krum proposed openstack-infra/system-config: Pull o_p::server invocation out of afs roles
KrinkleOh shit, it actually listens to that by default as well16:33
KrinkleWhen I deleted my test tag using push :refs/tag/test-2 it triggered the job16:33
Krinklewhich failed on Git being unable to check out the candidate revision16:34
Krinkle      gerrit:16:34
Krinkle       - event: ref-updated16:34
Krinkle         ref: ^refs/tags/.*$16:34
*** jcoufal has quit IRC16:34
KrinkleHm.. hoping there's a way to filter those ou16:34
*** sdake has joined #openstack-infra16:35
fungiKrinkle: we _do_ also have which can do some of what we'd eventually like to move into the zuul enqueue cli subcomannd16:36
*** ivar-laz_ has quit IRC16:36
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Add neutron-lib repo
*** otter768 has joined #openstack-infra16:36
*** ivar-lazzaro has joined #openstack-infra16:36
*** derekh has quit IRC16:37
fungithough that script just acts as a gearman client and injects work requests directly into the gearman queue, bypassing zuul entirely16:37
*** nfedotov has quit IRC16:37
*** thingee has quit IRC16:37
fungiit's still a way to re-trigger jobs for things like tags and merges16:38
*** ivar-lazzaro has quit IRC16:38
*** dkranz has joined #openstack-infra16:38
*** ivar-lazzaro has joined #openstack-infra16:39
*** sdake_ has quit IRC16:39
KrinkleI'm looking for a filter property to add to the pipeline or jobs that would allow it to not run a build if oldrev != 000 or if newref === 00016:39
*** thingee has joined #openstack-infra16:39
thingee./join #openstack-dev16:40
*** otter768 has quit IRC16:41
jeblairKrinkle: i don't think that exists either, but could be added fairly easily in zuul/model.py16:41
*** unicell has joined #openstack-infra16:41
*** ayoung has joined #openstack-infra16:41
*** unicell has quit IRC16:41
*** unicell has joined #openstack-infra16:42
Krinklejeblair: Either by exposing oldref and newref (in addition to ref) or by abstracting into a tag-action: create/delete/alter property16:42
*** BobBall is now known as BobBall_AWOL16:43
KrinkleI'd prefer the latter16:43
KrinkleI imagine there isn't really a use for matching against the commit sha1 itself other than to match for or against 00016:44
*** armax has joined #openstack-infra16:46
*** jistr has quit IRC16:48
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding
*** dims__ is now known as dimsum__16:50
*** sgordon has joined #openstack-infra16:52
jeblairKrinkle: agreed16:52
jeblairi have to run some errands, back before maintenance16:53
*** ajmiller__ is now known as ajmiller16:53
KrinkleOK. I filed it for us at
*** pblaho has quit IRC16:54
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Add pleia2 to statusbot_auth_nicks
*** jlibosva has quit IRC16:55
openstackgerritDoug Wiegley proposed openstack-infra/project-config: Add neutron-lib repo
pleia2anteaya: ^^ thanks for the ping there :)16:55
anteayapleia2: welcome16:55
*** wwwBUKOLAYcom has quit IRC16:57
*** unicell has quit IRC16:57
*** unicell has joined #openstack-infra16:57
*** annegentle has joined #openstack-infra16:58
*** armax has quit IRC16:58
clarkbok now that I am caught up on the thing I was working yesterday, is there a tl;dr for the other fun that happned yesterday?17:00
*** ihrachyshka has quit IRC17:00
fungiwhich fun specifically? there was so much to go around17:01
fungirelease-related fun? systems-related fun?17:01
clarkbfungi: clouds went down, gerrit restarted, then nodepool something?17:01
fungiwe have documentation migration fun too17:01
clarkbfungi: systems, though if there was release related fun I should probably figure out what happneed there too17:01
annegentlefungi: so much fun!17:01
*** tonytan4ever has joined #openstack-infra17:02
fungiannegentle: that's why this feels more like being retired. it's all fun and no work17:02
*** prad has quit IRC17:02
fungiclarkb: so on the systems front, we have hpcloud effectively turned off in nodepool17:02
fungieta is early next week to get an update on when they're ready for us to reconnect them17:02
*** armax has joined #openstack-infra17:03
*** markvoelker has quit IRC17:03
fungithough the nodepool config change exposed a bug in the earlier version of yolanda's queue management patch which caused nodepool to fall over when it tried to reload its config17:03
clarkbso we rolled back that change?17:04
fungiand since that was mostly to help us deal with hpcloud's performance characteristics anyway, we've reverted while it can be worked through (hand-reverted, it didn't actually get merged yet anyway)17:04
*** dimtruck is now known as zz_dimtruck17:05
fungithe unrelated gerrit fun is our old friend, stream-events getting blocked17:05
clarkbexcept there was a twist that involved buck and jeblairs patch not working?17:05
*** prad has joined #openstack-infra17:05
*** kmartin_ has quit IRC17:05
fungijeblair's earlier patch to add usernames to the stream-events tasks in the gerrit show-queue output was incomplete, but he managed to fire up a local gerrit and test it out until he got a working version implemented17:05
*** ivar-laz_ has joined #openstack-infra17:05
clarkbdo we have the working version installed?17:06
fungithough on the way there he learned important lessons about gerrit and it's forked buck17:06
*** salv-orlando has quit IRC17:06
*** prad has quit IRC17:06
fungii think we're ready for it to be working when we restart for today's maintenance17:06
fungii don't _think_ we've restarted it since his latest patch for that merged17:07
fungion the release front, we may have made a mistake when we decided to cap libs on master before branching17:07
fungibecause then they went ahead and released new versions with those caps baked in17:07
*** tsg_ has quit IRC17:07
*** ivar-lazzaro has quit IRC17:08
fungiand now master basically needs them all uncapped again and new tags outside the stable/kilo caps before they can resume having working integration tests17:08
clarkbbut if the caps merged doesn't that imply things were working first?17:09
*** prad has joined #openstack-infra17:09
*** kmartin_ has joined #openstack-infra17:09
clarkbI am guessing pip is unahppy resolving a complicated cap set though17:10
fungirequirements check jobs for all of them have been blacklisted on the master branch for a couple days so that they can be updated/released and then we'll turn that back on again and work out a better path for next time17:10
fungiyeah, the problem actually comes when we want to _un-cap_ one17:10
fungibecause then you get an explosion of version conflicts all over the place17:10
*** ivar-laz_ has quit IRC17:10
*** tsg_ has joined #openstack-infra17:11
*** ivar-lazzaro has joined #openstack-infra17:11
fungiwell, either un-cap or raise a cap17:11
clarkband changing the requirements version which is sycned doesn't help because we test from releases17:11
fungianyway, i suspect it would have worked better if we'd added caps on the clients in the stable/kilo branch only17:12
fungiand made sure to tag a new release of each off master outside the stable/kilo cap range before tagging any versions with those caps in their source17:12
*** MaxV has quit IRC17:13
*** russellb has quit IRC17:14
fungibut the prevailing sentiment has started to swing toward ditching version constraints in the requirements of libraries entirely, and only constraining requirements on applications (the trick i think will be telling libraries and applications apart? it's sort of a fuzzy distinction in my opinion)17:14
clarkboh nice17:14
clarkbwell I think we should be mostly uncapped everywhere :)17:15
*** tkelsey has quit IRC17:15
clarkbbut anyways sounds like we have a plan to unbreak the release so thats good17:15
fungialso there's been a lot of hand-waving around the current behaviors of pip when it comes to multiple install runs within one environment, ordering of constraints in requirements lists, et cetera17:15
clarkbits pretty straightforward, top level wins, otherwise its depth first, and an existing install won't be modified17:16
fungii'm worried that we're continuing to focus on the "recommended practices" of the python community, some of which ignore the mismatch between those practices and pip's actual behaviors in the face of complex dependency lists17:16
*** harlowja_away is now known as harlowja17:17
*** russellb has joined #openstack-infra17:17
fungiespecially since end users seem to want something which "just works" when you pip install it, but the python developer community at large is fairly against the idea of having your package really depend on anything at all unless it absolutely needs to have it in place to be able to install, and leaving definition of the sets of packages to be installed up to the end user instead17:18
*** FallenPegasus has joined #openstack-infra17:18
*** luqas has joined #openstack-infra17:19
*** vlaza has quit IRC17:19
fungiat least that's the conclusion i'm starting to arrive at in the wake of post-pycon discussions in here, and on our dev ml, and on the distutils-sig ml17:19
clarkbI will admit to mostly ignoring those conversations17:19
clarkbI always get sucked in and get frustrated by the lack of non hand waving around the problem17:20
clarkbso better to just avoid them17:20
dprinceanteaya: might you have time for a quick review of this TripleO job
dprincewe are anxious to get HA working w/ puppet in TripleO17:20
AJaegerfungi, clarkb: Can I get some advice on requirements and doc repos from you, please? See
fungibasically you have a bunch of packages, some of which may use setup_requires/install_requires to be able to install at all. then you have these "requirements" lists which are really an entirely separate concept outside of the packages themselves. so if you want a working system you tell pip to install a requirements list, not a package17:21
AJaegerfungi, clarkb: Please first finish you discussion ;)17:21
AJaegerand I'm used to RPMs that have hard-requires - and that's enough in most cases.17:22
AJaegerWe've had them extended to soft-requires: Suggests and Recommends...17:22
AJaegerbut I'm confused by the python practice as well17:23
clarkbAJaeger: infra does not sync openstack requirements to any of its projects and I think it works fine17:23
fungiclarkb: so i think to translate to something concrete in our terms, you might have a compute services requirements.txt which lists nova and all the other packages nova needs to be able to operate sanely. so if you want a working compute node you pip install the requirements list for that, rather than pip installing nova (which would just get you the nova package but you would be missing the things it17:23
fungineeds to do anything)17:23
clarkbbut that doesn't address pip broken dep resolver or how caps are often a problem and which package version actually wins17:24
*** luqas has quit IRC17:24
fungiyour requirements.txt would have a bunch of == version constraints. no ranges17:24
clarkbfungi: then it will break even harder17:24
*** russellb has quit IRC17:25
clarkbbecause now how do you bump that version constraint in all of your cleints17:25
clarkbyou can't, first one to go will win, then the others will either be too high or too low and pkg_resources will explode17:25
fungiit seems like this workflow is more or less baked into the python community, or at least key members thereof, which explains why pbr's behavior is seen as such an anathema. because it's conflating their completely separate ideas of "package" and "application"17:25
clarkbthats really weird because it won't work17:25
AJaegerthanks, clarkb17:26
clarkbthe only way it will work is if everything is in its own venv17:26
fungii'm sure it works fine when you hand-curate a pip freeze which has between several and a dozen deps17:26
clarkbfungi: the issue is in installing two different requirements.txt for fooclient and barclient when the pins don't agree17:26
fungiand yes, when you only install one python "application" per system/environment17:26
clarkbfungi: say novaclient says I need A==1.0 and glanceclient says I need B==1.117:26
clarkbwhichever one that installs first will win17:26
clarkbthen the other will break17:27
fungiyou wouldn't have that though17:27
clarkbfungi: oh maybe I misread what you said17:27
clarkbfungi: I thought ou said each package has its install requires then a separate requirements that ou install separately17:27
fungiyou would have a requirements list encapsulating a known-working set of package versions of novaclient and glanceclient and all the things they depend on17:27
clarkboh thats bonghits17:27
*** mrmartin has joined #openstack-infra17:27
waynrpelix: initial testing seems to indicate that changing the ordering of keys in my job-template definition does not affect the xml element ordering17:27
clarkbwhat happens when I want to use neutronclient tomorrow17:27
clarkbor uninstall glanceclient17:27
fungiand that would be a novaclient+glanceclient application requirements set17:27
anteayadprince: 173255 +A17:27
waynroh wait i am working on the wrong job-template17:28
fungiclarkb: if you want all three in one environment, then you'd need a separate set of requirements to represent that17:28
dprinceanteaya: thanks, appreciate it17:28
clarkbfungi: and a new environment17:28
anteayadprince: you're welcome17:29
fungiclarkb: think of requirements lists as "these are all the things i'm going to install on my server to make the one application i run there work"17:29
waynrokay, testing on the appropriate job-template the xml element order is still not affected17:29
clarkbfungi: ya, I understand now. This sounds a bit stockholm syndromy17:29
*** russellb has joined #openstack-infra17:29
clarkbfungi: the tools have us captive so now we just do what they want17:29
*** salv-orlando has joined #openstack-infra17:29
clarkbbut everything is awesome17:29
clarkbnow I have that song stuck in my head17:30
fungiclarkb: key point being it's what _you_ believe you need to be able to run _your_ particular setup. you might also publish that requirements list so that other people can redeploy the same setup you're deploying17:30
clarkbfungi: right but its also inflexible for what _I_ need if that changes tomorrow17:30
clarkbbasically I am not the dep resolver17:30
clarkband the uninstaller17:30
fungithen you get to keep both pieces17:30
clarkbthe only way for me to affect chagne in the system is to manually reason about it when it should be completely automatable17:31
clarkband this results in everyone writing their own tools that don't place nice with other tools fracturing our ability to install different python applications17:31
fungiin some ways, i think it serves as an ecological limiter to prevent any one python-based tool from depending on too many libraries. because if you try, your users will just hunt you down and string you up17:31
*** claudiub has quit IRC17:31
mordredwow. the scrollback is discussing crazy things17:31
clarkbfungi: hahaha, but there isn't necessarily anything wrong with that. Its only "wrong" because pip can't deal with it17:32
fungimordred: mostly just me ranting on requirements lists17:32
fungiand trying to reconcile ml threads between distutils-sig and our recent pbr discussions17:32
mordredwell, for the record17:32
mordredI think everyone in distutils-sig is wrong17:32
mordredbecause there is a latent assumption that we need to keep something flexible in an arbitrary way for consumers who are theoretically going to want to make a set of choices17:33
mordredthat I simply do not believe people want to make17:33
mordredexcept for the 5 people on distutils-sig17:33
waynrpelix: i am guessing parameter interpolation or defaults application somewhere is converting the OrderedDicts to normal dicts17:33
mordredit's pedantic correctness getting in the way of One Obvious Way17:33
*** tsg_ has quit IRC17:34
mordredbut, then again, python has categorically rejected that tentant and really should re-brand itself TIMTOWTDI17:34
zaropelix: i created the jjb release tag so approve away.17:35
*** sushilkm has joined #openstack-infra17:35
*** sushilkm has left #openstack-infra17:35
clarkbanyone else want to review and otehrwise I am approving both so that we can try spinning up new working slaves17:36
waynri am just going to have to add a new trigger-parameterized-builds parameter to direct the ordering of parameter types17:36
*** e0ne has joined #openstack-infra17:37
*** mriedem1 is now known as mriedem17:38
fungiclarkb: switching those to the top of my pile now17:38
fungioh, i've already reviewed17:38
clarkbfungi: ya you are set :)17:38
fungii just always feel behind on getting anything done17:38
clarkbfungi: me too17:39
fungiand i'm about to pop back out on another errand. two races this weekend, need to pick up our registration packets17:39
clarkbhave fun17:39
fungiat least the registrations are happening about 4 blocks from my house, so shouldn't take long unless there's a line waiting on them to open17:39
clarkbmordred: any interest in reviewing 174160 and 174897?17:39
clarkbAJaeger: ^ 174160 is project-config17:40
AJaegerclarkb: will do soon17:40
harlowjamordred u so popular, :-P17:40
openstackgerritAndreas Jaeger proposed openstack/requirements: Remove doc repositories from projects.txt
harlowja'Monty Taylor is "somewhere in between" Coghlan and Klose/Warsaw.' , lol17:41
fungidepends on how you define "between"17:41
*** SumitNaiksatam has quit IRC17:41
fungievery point on a globe is technically between any two other points via an arc17:41
fungii am always somewhere between tokyo and seattle17:42
*** _nadya_ has quit IRC17:42
clarkbAJaeger: thank you17:43
clarkbAJaeger: I know its late on friday so no pressure :)17:43
openstackgerritMerged openstack-infra/system-config: Expose jenkins_gitfullname and jenkins_gitemail
clarkboh is pleia2 back?17:43
*** esker has quit IRC17:43
clarkbpleia2: 174160 and 174897 are related to hopefully making log uploads when jenkins has a sad more reliable17:43
pleia2clarkb: I am17:43
AJaegerclarkb: +A17:43
pleia2tad? :)17:44
AJaegersdague: care to +2 ?17:44
*** tonytan4ever has quit IRC17:44
clarkboh hrm17:45
clarkbexcept 174897 may not fix puppet17:45
clarkbI don't think puppet module install postgres==someversion is going to downgrade to that version17:45
*** prad has quit IRC17:45
*** emagana has joined #openstack-infra17:45
*** e0ne is now known as e0ne_17:45
*** prad has joined #openstack-infra17:46
clarkbso while we should revert for sanity17:46
nibalizeralso puppet module install puppetlabs/postgres --version=myversion17:46
*** sarob has joined #openstack-infra17:46
nibalizeris the command17:46
clarkbwe also need to manually install the right version either via my install all from source change or by hand17:46
pleia2well, approved for sanity17:46
clarkbpleia2: ty17:46
nibalizerclarkb: i want your 'install all from git' change17:46
clarkbnibalizer: I need to rebase it and update all the versions and sources and make sure they are correct17:47
nibalizerbut fixing it by hand to unbreak things is also good17:47
clarkbnibalizer: I can work on that today17:47
nibalizerclarkb: yea kindof a :( ammount of work17:47
nibalizerbut hey you only have to do it once, or maybe two more times max17:47
openstackgerritMerged openstack-infra/project-config: TripleO: Add f20puppet-ha check job
*** achanda has joined #openstack-infra17:48
*** annegentle has quit IRC17:50
clarkbnibalizer: :P17:50
clarkbhopefully just once17:50
clarkbI need to eat some food now then finally go get that tdap vaccine17:50
clarkbbut will make the install from source change my priority after17:51
fungiclarkb: why wouldn't 174897 work? remember where it's breaking is on a fresh install of that set of modules in a nodepool image17:51
*** ivar-lazzaro has quit IRC17:51
fungiclarkb: the only fallout we've seen thus far has been image building17:51
clarkbfungi: oh right, for some reason I had in my head that the version was already changed on the puppetmaster but we are using apply17:51
clarkbso thats good17:51
*** ivar-lazzaro has joined #openstack-infra17:51
*** sgordon has quit IRC17:51
*** sgordon has joined #openstack-infra17:51
clarkbpleia2: ^ fungi is correct that will fix it17:51
*** e0ne_ is now known as e0ne17:52
fungiwell, and whatever the deal with that was on the puppetmaster, i've already forgotten but it was not immediately impacting and i assume nibalizer is working on a solution17:52
clarkbthe issue on puppetmaster is we didn't actually bump the puppetdb version17:52
fungithe problem you spotted in the log shortly after we merged the original change we're reverting there17:52
fungiyeah, that17:52
clarkbmy isntall from source change will address ^17:52
clarkbby giving us direct control of what is installed17:53
clarkbso I will still pick that up after food17:53
*** FallenPegasus has quit IRC17:53
fungidisappearing now for a few minutes (30 or so i hope), but will brb to start writing up the gerrit maintenance plan for 22:00 utc17:54
*** annegentle has joined #openstack-infra17:54
*** isviridov_away is now known as isviridov17:55
pleia2ah, good17:55
*** radez is now known as radez_g0n317:56
*** radez_g0n3 is now known as radez17:57
*** e0ne is now known as e0ne_17:57
openstackgerritMerged openstack-infra/project-config: Fix retries in
*** vponomaryov has quit IRC18:01
yolandaah fungi, i amended that change18:02
yolandai added a stop method now18:02
openstackgerritMerged openstack-infra/system-config: Revert "Bump puppetdb version"
clarkbso I just had a thought with hpcloud down18:02
*** ddieterly has quit IRC18:02
clarkbwhat if we do a first pass of infra cloud deployment with
clarkbwe would just need to write the nodepool info on all of the nodes then run devstack-gate on one we choose as the controller18:03
*** ddieterly has joined #openstack-infra18:03
clarkb172614 works and we could have a cloud in about 25 minutes or so18:04
greghaynesclarkb: and just do a one time reimage of all the nodes?18:04
clarkbya basically next time you want to upgrade cloud blow that away and rerun d-g18:04
*** esker has joined #openstack-infra18:04
clarkbtotal outage is ~30 minutes18:04
greghaynesthat is one of the kickers - wed need a baremetal layer to redeploy18:04
greghaynesso just hope we can restack18:05
clarkbnevermind images18:05
clarkbno don't even restack18:05
*** armax has quit IRC18:05
clarkbjust rm -rf /opt18:05
clarkbreboot, d-g18:05
greghaynesnot pretty, but I dont see why not18:05
*** sergmelikyan has quit IRC18:06
mordredclarkb: we need to get networking changes made18:06
mordredwe have not reconfigured the network there yet18:06
clarkbmordred: what changes? because the nice thing about d-g is it makes its own networking as necessary.18:06
mordredclarkb: public ips18:06
clarkbmordred: so as long as each node has connectivity to public ips18:06
clarkboh ok18:06
mordredwe need to redo the ranges and vlans  in the switches18:06
mordredclarkb: btw - did you catch that ttx has arranged to get us 600 nodes of capacity from runabove?18:07
clarkbmordred: I did not18:07
mordredttx: what are our next steps on that, btw?18:07
mordredclarkb: ttx worked magic18:07
*** tonytan4ever has joined #openstack-infra18:08
greghaynesthats awesome18:08
clarkbbut ya I think if we seed the nodepool data on those servers, rm -rf /opt, reboot, d-g we have a deployment that should mostly work18:09
clarkbdo need public IPs though18:09
clarkband likely an option to disable compute on the controller18:09
*** hdd has joined #openstack-infra18:10
greghaynesor we could just ansible remove that compute node after we deploy18:10
*** ZZelle_ has joined #openstack-infra18:10
greghaynesbut probably not hard to actually make d-g DTRT18:11
*** HeOS has quit IRC18:15
*** notnownikki has quit IRC18:16
*** Sukhdev has joined #openstack-infra18:19
*** claudiub has joined #openstack-infra18:19
*** AJaeger has quit IRC18:19
*** armax has joined #openstack-infra18:20
*** sarob has quit IRC18:20
*** annegentle has quit IRC18:21
*** isviridov is now known as isviridov_away18:21
yolandamordred, clarkb, i've been doing some work on puppetmaster downstream, and i was thinking in propose a puppet-puppetmaster module to infra. What do you think?18:24
pleia2could it be part of puppet-puppet?18:24
clarkbyolanda: we are trying to delete the puppetmaster18:24
mordredyolanda: I don't have a problem with that - other than that I'm planning on deleting the puppetmaster18:24
pleia2or that18:24
clarkbyolanda: so I think our preference would be to spend time deleting the puppetmaster18:25
mordredyolanda: it's next on my list once dib-nodepool is done18:25
*** annegentle has joined #openstack-infra18:25
mordredit's actually not that far away18:25
yolandaah, then it's not worth the effort18:25
*** leakypipes has quit IRC18:25
mordredclarkb: speaking of 167625 is safe to land to start clearing down the dib-nodepool stack18:26
*** ildikov has joined #openstack-infra18:26
*** tonytan4ever has quit IRC18:26
*** dustins_ has joined #openstack-infra18:26
greghaynesmordred: does that have the urlgrabber fix?18:27
greghaynesmordred: where we had to install that from git18:27
*** teran has quit IRC18:27
fungimordred: you were previously talking about working on an ansible playbook for repo renames. is that close enough to working that we should try it in a few hours, or stick to our old methods and try that out next round?18:28
*** teran has joined #openstack-infra18:28
*** ashleighfarnham has joined #openstack-infra18:28
openstackgerritDan Prince proposed openstack-infra/project-config: puppet-keystone: Add tripleo check job
mordredfungi: let's stick to our old methods and try it out next round18:28
*** johnthetubaguy is now known as zz_johnthetubagu18:28
mordredgreghaynes: urlgrabber fix?18:29
fungimordred: sounds good. just making sure before i write the etherpad for this one18:29
mordredgreghaynes: oh - in dib18:29
greghaynesmordred: yes, yumdownloader was failing unless we had urlgrabber installed from git on the builder18:29
*** andreykurilin__ has joined #openstack-infra18:29
mordredgreghaynes: do we have the thing that needs to be done documented somewhere?18:29
*** dustins has quit IRC18:29
*** carl_baldwin has joined #openstack-infra18:29
mordredgreghaynes: also - do we have any sense of when that's going to be not broken?18:29
*** ssam2 has quit IRC18:30
greghayneswhenever they push a release to pypi18:30
greghayneswe could ask18:30
*** e0ne_ is now known as e0ne18:30
clarkbso urlgrabber is a thing that yum needs but isn't available on pypi or ubuntu apt repos?18:30
clarkbdoes that mean ubuntu's package is broken?18:30
mordredclarkb: the patched version isn't released yet18:30
greghaynesclarkb: theres a bug in something ssl related, its fixed but the pypi versoin is out of data18:31
greghayneser, date18:31
greghaynesIIRC they even have tagged new versions, just not uploaded them18:31
mordredI think we should nudge them to release18:31
openstackgerritDan Prince proposed openstack-infra/project-config: puppet-keystone: Add tripleo check job
mordredSpamapS: what's the chance we can get the patch backported into trusty? ^^18:31
*** carl_baldwin has left #openstack-infra18:31
greghaynesyea, 3.9.1 is on pypi but they have a 3.10.1 tagged right after the fix commit18:32
*** teran has quit IRC18:32
clarkbmordred: about as good as python3.4 backports making it to trusty? >_>18:32
mordredclarkb: *sob*18:33
*** avozza is now known as zz_avozza18:34
mordredgreghaynes: any chance you know what the "fix" patch is?18:35
greghaynesmordred: I bet its;a=commit;h=857af5ac1cec0a06755b75286ae2192e398cd34118:35
greghaynesnot 100%, but that commit msg looks likely18:35
*** cdent has quit IRC18:36
*** otter768 has joined #openstack-infra18:37
mordredgreghaynes: fedora has 3.10.118:38
fungisounds like this is yet another problem we'd be able to safely ignore if we implemented that nodepool dib worker spec?18:38
* greghaynes was debating starting hacking on the dib worker stuff today18:39
greghaynesnot sure if we wanted to wait until the shade + nodepool sutff gets merged?18:39
* mordred would like to be able to build these images locally18:39
mordredgreghaynes: yes please18:39
mordredthere is too much churn already18:39
fungimordred: does a container count as locally?18:39
mordredfungi: right now it works18:39
mordredthere is no problem with me just building them18:40
mordredI'd like to keep it that way18:40
fungimakes sense18:40
mordredbecause it's not broken right now - other than the debian version of urlgrabber is old18:40
fungithough i still feel like there is some class of issues where we're not going to be able to reasonably expect to build images for every platform we might want from one common distro/release18:40
mordredI think all of those are bugs18:41
mordredthere is no reason why we can't copy files around in a fedora chroot on ubuntu18:41
*** otter768 has quit IRC18:42
clarkbuntil you need tar to support extended attributes18:42
fungieh, for platforms which support the same set of filesystems anyway18:42
fungi(and filesystem features, point)18:42
mordredfungi: I disagree18:42
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Bump kombu version and add amqp version
mordredthese are chroots, no?18:43
mordredin any case- if we fail, neat18:43
*** tonytan4ever has joined #openstack-infra18:43
fungithough i guess if you really want to bootstrap freebsd on ufs from trusty, there's ufs support in the linux kernel18:43
greghaynesits a chroot we copy into a mouned loop device we mkfs'd18:43
mordredbut I would personally prefer to not give up on my being able to easily test things locally18:43
fungiyou wouldn't be able to chroot into that filesystem, but you'd be able to write files into it18:44
mordredsure. although at the moment we don't support freebsd at all - so I'm not too worried about that18:44
mtreinishmordred: I thought someone was working on that18:45
clarkbmordred: you cannot locally build centos/fedora images on ubuntu precise18:45
clarkbmordred: jus saying, and I don't think anyone considers that a bug18:45
mordredright. because I don't care about precise18:46
greghaynesclarkb: that might work with -minimal now18:46
fungior our friend who wants to nodepool a bunch of arm systems. with dib workers he could run nodepoold on x86 and the workers on arm18:46
greghaynesclarkb: since its doing rpm, not tar18:46
*** achanda has quit IRC18:46
mordredso - listen - don't get me wrong - I think the workers are a GREAT idea and we should do them18:46
mordredand it'll help some folks who want to do strange things like cross-compile onto arm and whatnot18:46
* fungi supports the workers18:46
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: hipchat bug fix
mordredbut I think taht the general case is taht for normal things we should expect to be able to build ubuntu images on fedora and vice versa18:47
* fungi doesn't want to be the first with his back against teh wall when the revolution comes18:47
mordredand if something happens to start preventing that from being teh case, we should consider it a bug18:47
mordredand should not just assume that the host platform is going to need to match18:47
fungiyep. i think both those viewpoints are valid18:47
*** mbacchi1 has quit IRC18:47
clarkbmordred: then why isn't precise a thing you care about?18:47
greghaynesrelatedly - there was talk from tripleo about splitting image building out of their devtest and create a script that an read yaml and call dib to make corresponding images18:48
clarkbmordred: I think you have to care about precise in your example18:48
mordredclarkb: I don't care about precise as a build host18:48
greghaynesI might have mentioned they should work to use the same code nodepool does18:48
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: hipchat bug fix
mordredclarkb: because I do not care about precise as a host for new things at all right now18:48
greghaynesso id kind of like to do the nodepool worker stuff with that in mind, if possible18:48
mordredgreghaynes: the could also just use my ansible module that already does that18:48
clarkbmordred: except that you likely want to build precise images18:48
mordredsince it exists today and works today18:49
mordredclarkb: yes - and I can18:49
clarkbmordred: just not on precise18:49
mordredclarkb: I can build precise images on my trusty laptop today18:49
greghaynesmordred: ooo, that would be neat18:49
mordredI do not care about building things on precise18:49
clarkbmordred: right I am saying each image should be self hosting too18:49
clarkbbecause if ubuntu should build fedora and vice versa, hopefully ubuntu can build ubuntu18:49
mordredclarkb: in general, sure - except that this is new development so I don't care if it doesn't support precise18:49
mordredyes. and it can. trusty, which is the first supported version, can build anything right now18:50
mordredit also doesn't work on lucid18:50
mordredalthough it can build a lucid image if you like18:50
clarkblucid is actually eol this month18:50
clarkbmaybe even this week :)18:50
clarkbI think18:50
clarkbshould ask blkperl18:50
clarkbmordred: it is just weird to me to say that ubuntu must build fedora and vice versa but not also say that builds are self hosting18:51
clarkbsince the idea seems to be that we should make it easy for users to use18:51
mordredclarkb: I could just as easily just say "screw precise it's not supported"18:51
greghaynesthe issue was fedora on precise builder right?18:51
clarkband if a user wants precise images they likely want to build on precise too18:52
greghaynesI can just kick off a build with fedora-minimal and see if its even an issue still18:52
mordredclarkb: I'm just saying that precise, specifically, is currently half-supported18:52
mordredclarkb: it is a target but not a host18:52
clarkbgreghaynes: it was any build that required tar to understand xattrs18:52
*** zz_avozza is now known as avozza18:53
mordredgreghaynes, SpamapS: so - I think we really do need to get the urlgrabber packages fixed18:54
mordredbecause yum-utils depends on urlgrabber, and if we install urlgrabber over top of that via pip, it's going to be a mess18:54
clarkbmordred: I totally agree that ubuntu should be able to build fedora and vice versa18:55
mordredclarkb: ++18:55
clarkbmordred: but as the person that was completely screwred by precise being unsupported I do not thinkyou can argue for that then say f precise18:55
mordredclarkb: I do - as long as I'm clear that precise is an unsupported platform18:55
SpamapSmordred: the Ubuntu release team is distracted with vivid18:55
clarkbmordred: but it is supported18:55
SpamapSmordred: they just have to click a single button to release it to trusty-updates .. not sure why they haven't.18:55
* SpamapS will poke them18:56
clarkbmordred: dib explicitly supported precise last I looked18:56
*** MaxV has joined #openstack-infra18:56
mordredSpamapS: urlgrabber?18:56
SpamapSmordred: yes18:56
*** achanda has joined #openstack-infra18:56
mordredclarkb: maybe dib should stop claiming precise support18:56
SpamapSmordred: oh, it's released!18:57
SpamapSmordred: trusty-updates has a working urlgrabber18:57
*** salv-orlando has quit IRC18:57
*** e0ne has quit IRC18:58
SpamapSas of Monday actually18:58
SpamapSI just missed the bug response18:58
SpamapSno, Tuesday. Anyway, it's up, so that should all work and stuff18:58
SpamapScan we get urlgrabber to release to pypi so Debian notices?18:58
SpamapSalso it looks like Debian is getting on its horse and pushing hard for python3 in stretch (jessie+1)18:59
SpamapSRelease goal will likely be that all leaf packages use python318:59
*** MaxV has quit IRC18:59
SpamapSAnd, love it or hate it, when Debian does something systematically, it usually ripples through the ecosystem.19:00
mordreddprince: you know everyone at redhat, right?19:00
*** marcusvrn3 has quit IRC19:00
SpamapSFeels a bit like maybe 2015 will be the year everybody starts caring enough that it gets done.19:00
mordreddprince: any chance you can get the urlgrabber maintainer to release 0.10.1 to pypi and not just to the urlgrabber download site?19:01
mordreddprince: it's been released for 2 years - just isn't on pypi19:01
mordredrussellb, ianw: ^^19:01
mordredSpamapS: you could also make a patch for debian to change the watch file to look at ...19:02
*** MaxV has joined #openstack-infra19:03
openstackgerritMatthew Treinish proposed stackforge/gertty: Explain how to install on Arch Linux
*** arxcruz has quit IRC19:03
mtreinishjeblair: ^^^ figured I should document that19:03
*** marun has quit IRC19:03
mtreinishalthough not that I ever recommend using anything python from packages in arch19:04
mtreinishtalk about a headache19:04
*** pelix has quit IRC19:07
mtreinishjeblair: I also figure it was better to not put the package from git in the readme:
mordredSpamapS: ok. bug 1212958 in python-urlgrabber "Please release 0.10.1 to PyPI" [Unspecified,New] - Assigned to packaging-team-maint19:07
zarowaynr: had questions for you in
*** mrunge has quit IRC19:07
zaroand 150598 as well19:08
SpamapSmordred: ah so upstream just stopped uploading to pypi, but they have tarballs.. good to know.19:08
*** peristeri has quit IRC19:08
mordredSpamapS: yah - although they do not  have an SSL download location19:09
mordredat least, not one with a working cert19:09
mordredso it would still be nice to get them to upload to pypi19:10
SpamapSmordred: nor gpg sig, but if tags are signed, that is sufficient19:10
*** MaxV has quit IRC19:12
SpamapSnope, no sigs19:12
clarkbok really going to run so that I can be back before 220019:13
clarkbfungi: any chance you can kick off some image rebuilds to test your puppetdb revert?19:13
fungiclarkb: sure. i'll fire one up now to verify19:13
SpamapSmordred: yeah, they need to get their **** together...19:13
SpamapSsuch a basic thing.. sign your tarballs people.19:13
fungiit's sad, but i spend far more of my time than should be necessary explaining pgp to developers19:14
clarkbfungi: I think that says more about the implementation of gpg than anything else19:15
clarkbhow to get confused very fast: man gpg19:15
clarkbeven signing has ~3 different methods that should all be used in different circumstances that arent well documented iirc19:15
fungii disagree. a much larger percentage of free software developers actively used pgp a decade ago than do now. maybe it says something about a new generation of free software developers not wanting to learn to use the tools of their trade?19:16
clarkbfungi: its very hard to learn19:16
clarkbI find gpg to be incredibly obtuse19:16
clarkbsome of it is probably hard to make simple but signing should be easier for example19:17
*** teran has joined #openstack-infra19:17
SpamapSno you're absolutely right clarkb19:17
SpamapSwhen _debian's packaging tools_ are simpler to use...19:17
fungimaybe the problem is that other tools have become easier to use over time (or been replaced with easier to use tools) and pgp implementations generally have not19:17
mordredthat's a scary bar19:17
mordredyah. nobody has really stepped up and made a better ui for it- other than keybase19:18
fungiSpamapS: you think they're using debian's packaging tools? and not just some other fire-and-forget script which makes a deb dpkg won't completely fail to install?19:18
SpamapSHonestly the problem is that nobody who started writing software 3 years ago sits in front of their terminal and thinks "How in the hell do I trust that software from this random FTP site on the internet?" They think .. well shit it's in git, that uses hashes, its on github... I'm good.19:19
*** annegentle has quit IRC19:19
SpamapSfungi: no I think it's easier to learn how to use debian's tools than gpg19:19
SpamapSespecially debsign. :)19:19
fungiSpamapS: debsign isn't going to work if you haven't used gpg to set up a key, is it?19:20
mordredspeaking of - I just uploaded a modification of the vhd-util package19:20
mordredthat does not install library files19:20
mordredbecause it doesn't use them19:20
mordredand they conflict with libxenstore19:20
mordredwhich qemu depends on in vivid19:20
SpamapSfungi: no, but making a key isn't that hard and happens once. There are, as clarkb said, 3 ways to sign.19:20
*** peristeri has joined #openstack-infra19:20
ttxmordred: next steps with RunAbove is to send them a memo on what we agreed on, then have them come up with the exact set up (how many, what type, under what brand)19:21
SpamapSfungi: but managing ones' own key... getting it signed.. is actually the important part and is actually the hard part. :)19:21
ttxIdeally I'd like to piggyback on your Paris trip so that you can meet some of them19:21
mordredttx: cool19:21
ttxmordred: know when you arrive and leave ? jeblair told me you'd rather limit the duration of your stay19:22
*** bnemec is now known as beekneemech19:22
*** radez is now known as radez_g0n319:22
mordredyah - but this is a worthwile reason to extend19:23
openstackgerritAndrew Laski proposed openstack-infra/project-config: Blacklist last failing Tempest tests for cells job
mordredttx: my current plan is to arrive morning of apr 29 and to leave on may 1 (since I hear paris will be shut down that day)19:23
ttxit is a holiday yes19:24
fungialso keybase makes me sad. it's actively encouraging people to upload their private key material to an on-line third party. does nobody remember hushmail?19:24
ttxand we kind of take those seriously19:24
*** hdd has quit IRC19:24
mordredfungi: it is? I do not remember any "upload private key" step ...19:24
mordred(which I would not have performed)19:24
*** hasharAway is now known as hashar19:25
fungimordred: at least when i tried it out, that was their "recommended" usage. though if you really wanted, it also let you do local operations with your key and upload the results19:25
fungithey wanted to make it easier by generating and storing a key for you instead, so that you didn't have to worry about the complexities of key management19:26
*** hdd has joined #openstack-infra19:26
SpamapSmordred: true story19:26
greghaynesWCPGW with that19:26
SpamapSIt made ma a little sad too, but I just do the cli operations and paste19:26
ttxwow, every vote counts19:27
SpamapSgreghaynes: it'll be fine as long as they don't cross the streams19:27
* ttx just glanced at the results19:27
mordredah - yes - I see the option to upload one now19:27
fungibut that gives people new to the topic the impression that letting someone else be in control of your key material is a safe thing19:27
mordredttx: yeah?19:27
*** dustins_ has quit IRC19:27
ttx89-87 on the nova election19:27
ttx34-32 on the Glace one19:28
*** ajo has quit IRC19:28
mordredttx: who won?19:28
ttxJohn Garbutt and Nikhil Komawar19:28
*** ajo has joined #openstack-infra19:28
fungifrom what i've heard they've in more recent months gotten a lot of concerned feedback and are since downplaying the on-line keys option a bit more than they were at the start19:28
mordredfungi: good19:29
mordredttx: is that the first time that someone has run against a sitting ptl and won?19:29
funginope, neutron a couple cycles back19:29
mordredoh right19:29
ttxno, Kye won over Mark19:29
mordredI forgot about that one19:29
* SpamapS wanders off to find food19:30
*** yamahata has quit IRC19:31
ttxtrain is moving again. See you on the other side19:31
dprincemordred: I don't claim to know anyone, but I can put a ping in for it :)19:31
mordreddprince: thanks! I also filed a bugzilla19:31
*** HeOS has joined #openstack-infra19:34
jeblairmordred, ttx: back19:37
mordredjeblair: while you were gone, all the things broke, except for the ones that didn't, and also all the things got fixed, except we missed some19:38
*** stevemar has quit IRC19:40
*** stevemar2 has joined #openstack-infra19:40
*** subscope has quit IRC19:40
hasharnodepool_0.0.1-104-gddd6003_amd64.deb  nodepool.NodePool: Loading configuration ....  ^o^19:40
hasharyeah slowly progressing!19:42
hasharmordred: I have a replied a bit to your Phabricator emails. I am not sure I will have time to reply ttx great analysis though :/19:43
jeblairhashar: i know i promised i'd tag and release some things for you, but i haven't gotten to it yet and there's enough broken stuff for a friday already, so would next week be okay? :)19:44
hasharjeblair: yeah take your time19:44
hasharjeblair: I am building my package from arbitrary points.  Just thought tags would help figure out something is going to break horribly19:44
hashargood luck on friday fixes!19:45
jrollrandom question of the day: is there a reason openstack projects use % and not .format() for string formatting?19:45
*** ibiris is now known as ibiris_away19:45
*** tsg has joined #openstack-infra19:45
mtreinishjroll: whimsy?19:46
hasharless keystrokes?19:46
jrollpython 2.6 came up as a possible reason19:46
jeblairjroll, hashar: yeah, i count 8 reasons ;)19:46
jrollI guess I wonder if we care about doing it as part of the road to python 319:46
jrollor if we care about the road to py319:47
fungijroll: i thought % still worked in python 3.x19:47
jeblairwe definitely care about the road to py319:47
* hashar giggles19:47
hasharthe Debian python team is actively talking about switching to py319:47
jrollfungi: I don't believe it does19:47
fungijroll: yeah, still seems to work in 3.419:48
*** tsg_ has joined #openstack-infra19:48
fungijroll: fire up an interpreter and try it?19:48
jrollmaybe they put it back at some point19:48
jeblairfungi, ttx: any (release-related) reason we should defer today's maintenance?19:48
openstackgerritMerged openstack-infra/project-config: Update gerritbot #openstack-infra from governance
jrolldefinitely didn't work in earlier 3.x19:48
*** arxcruz has joined #openstack-infra19:48
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: zuul: drop notifications
jrollfungi: TIL, thanks :)19:48
fungijeblair: not to my knowledge. unless there's another flurry of reviews on the way it looks like we'll likely catch up zuul-wise by 22:0019:49
jeblairjroll: i want to say 3.3 was the big "put everything that made it impossible to move to py3 back in" release?19:49
*** mrmartin has quit IRC19:49
clarkbjeblair ya19:49
jrolljeblair: yeah, you might be right, I haven't been following19:49
fungijeblair: jroll: yeah, it might have been lacking from 3.2, i don't recall19:49
*** Darkwan has joined #openstack-infra19:49
*** salv-orlando has joined #openstack-infra19:49
jeblairoh wow, imapsync is up to 3.56 m/s and expects to be done in 2 hours!19:50
*** tsg has quit IRC19:50
fungidhellmann: any release-related issues you're aware of that we should postpone our scheduled 22:00-00:00 gerrit maintenance today?19:50
*** achanda has quit IRC19:50
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: DO NOT MERGE: Example of get/list/search method
fungiclarkb: successfully rebuilt a bare-trusty in rax-iad, so we should be all set puppet-wise now19:52
*** arxcruz has quit IRC19:52
clarkband that should make console logs more reliable19:52
*** funzo has quit IRC19:53
*** patrickeast has quit IRC19:53
openstackgerritMerged openstack-infra/puppet-gerrit: Add properties, md and css files to direct download list
*** nelsnelson has quit IRC19:59
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: Adding hipchat publisher
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for the xcode builder plugin
mordredjroll: I try to use format20:03
mordredjroll: but I've been doing python for a long time, so I forget20:03
*** teran_ has joined #openstack-infra20:04
jrollmordred: right on, just wondering if there was a good reason, I think "3.4 supports %" is good enough for me20:04
clarkbfwiw transifex groks % but not sure if it groks format20:04
mordreddo we know if zanata20:04
clarkbunsure of trnasifex20:04
mordredwell, we should file a bug with them if it doesn't20:05
openstackgerritKyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for android emulator
*** teran has quit IRC20:07
*** funzo has joined #openstack-infra20:09
kevinbentonfungi: ping20:10
fungikevinbenton: pong20:10
*** mattfarina has quit IRC20:11
kevinbentonfungi: quick clarification question regarding the email thread20:11
kevinbentonfungi: "My takeaway from Tom's message was that there was a concern about20:11
kevinbenton"complexity" in all forms (not just of the API but also due to the20:11
kevinbentonlack of maturity, documentation and debuggability of the underlying20:11
kevinbentontechnology), and that the self-service networking model was simply20:11
kevinbentonone example of that"20:11
*** teran has joined #openstack-infra20:11
kevinbentonfungi: the self-service networking model isn't from the underlying technology20:12
kevinbentonfungi: or did i parse that incorrectly?20:12
*** MaxV has joined #openstack-infra20:12
kevinbentonfungi: oh, i think i did now that i re-read the parens20:12
openstackgerritMerged openstack-infra/project-config: Move cliff gerrit announcements to the sdk channel
fungikevinbenton: the self-service model of neutron is one form of complexity concerning operators, but not the only source of complexity concerning them20:13
kevinbentonfungi: ok20:13
clarkbalso note nova net flatdhcp doesnt give every node an fip20:13
kevinbentonclarkb: oh, how do they get connectivity out?20:14
*** jogo has quit IRC20:14
fungi(see rackspace for a good example of that)20:14
clarkband I wrote a list of ovs complaints that are not "its new"20:14
fungikevinbenton: globally routable ip addresses on your instance interfaces20:14
clarkbkevinbenton you put them on a network directly20:14
funginat is a disease20:14
clarkbcan fip in addition to that20:14
*** dkranz has quit IRC20:14
kevinbentonfungi: i saw clarkb's complaints, but is there somewhere else that has operator complaints about OVS? we should kick them back upstream to OVS20:14
*** teran_ has quit IRC20:15
*** stevemar2 is now known as stevemar20:15
kevinbentonclarkb: so they would just get connectivity via whatever router is on the private network?20:15
*** Krinkle is now known as Krinkle|detached20:15
fungikevinbenton: i've not really had much feedback directly from the operator community, so what i know comes from the user committee, user survey and the few who do show up on the operators ml20:15
clarkbyes direct connectivity20:16
fungikevinbenton: ideally get tom, tim et al to pass along more specific complaints20:16
kevinbentonfungi: what is a 'user' in the user committee sense?20:16
*** dprince has quit IRC20:16
fungikevinbenton: it's all over the map20:16
kevinbentonclarkb: that explains the email thread about people wanting to bypass the neutron router but still have floating IPs20:16
kevinbentonclarkb: thanks20:16
fungifrom what i can tell it means people who rely on openstack in some way and aren't active developers with direct influence on the design (or who don't know they can have direct influence on the design, if they have time to get involved)20:17
kevinbentonfungi: why i asked is because 'tenants' shouldn't see OVS at all20:17
kevinbentonfungi: that makes sense20:17
fungithe user survey is a product of the user committee, but it asks questions about operator-related topics depending on what paths you take in the choose-your-own adventure20:18
fungior perhaps it just seemed that way to me the last few times i've taken it20:18
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Support the use of lists in trigger-build publisher.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Allow explicit parameter override ordering in trigger_parameterized_builds
*** Sukhdev has quit IRC20:19
waynrzaro: I'm interested in your thoughts on
kevinbentonfungi: sounds good20:20
waynrthis parameter ordering is required for some work i am doing on my production CI jobs, or i have to rethink the way i am using build parameters20:21
*** kgiusti has left #openstack-infra20:21
waynralso, i am addressing your feedback on
*** openstackgerrit has quit IRC20:22
*** openstackgerrit has joined #openstack-infra20:22
kevinbentonfungi: sorry, that was an incompatible acknowledgement to the last thing you said. running on little sleep :)20:23
*** marun has joined #openstack-infra20:25
fungikevinbenton: heh. i totally understand ;_20:25
openstackgerritSpencer Krum proposed openstack-infra/system-config: Use envassert for integration tests
*** achanda has joined #openstack-infra20:27
*** Sukhdev has joined #openstack-infra20:28
*** andreykurilin___ has joined #openstack-infra20:28
*** soren_ has joined #openstack-infra20:29
*** andreykurilin__ has quit IRC20:29
*** kgiusti has joined #openstack-infra20:31
*** gyee has quit IRC20:32
*** amitgandhinz has quit IRC20:35
*** otter768 has joined #openstack-infra20:38
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Allow explicit parameter override ordering in trigger_parameterized_builds
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Support the use of lists in trigger-build publisher.
*** kgiusti has left #openstack-infra20:39
*** tqtran_afk is now known as tqtran20:41
*** thedodd has quit IRC20:42
*** otter768 has quit IRC20:43
*** tqtran has quit IRC20:43
*** tqtran has joined #openstack-infra20:43
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Enable parameter ordering in trigger_parameterized_builds
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Support use of lists in trigger-build publisher.
*** tiswanso has quit IRC20:45
*** soren_ has quit IRC20:45
*** emagana has quit IRC20:45
*** tiswanso has joined #openstack-infra20:47
*** emagana has joined #openstack-infra20:47
openstackgerritMerged openstack-infra/project-config: Add #openstack-neutron-ovn to gerritbot.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Support the use of lists in trigger-build builder.
clarkbok working on puppet module installs now, does anyone know why we have a memcached module?20:54
* clarkb guesses askbot20:54
*** sdake_ has joined #openstack-infra20:56
openstackgerritMerged openstack-infra/puppet-nodepool: Install additional depends for dib
*** baoli_ has quit IRC20:56
nibalizerclarkb: looks like wiki20:56
nibalizermodules/openstack_project/manifests/wiki.pp:  class { 'memcached':20:56
*** tonytan4ever has quit IRC20:56
*** luqas has joined #openstack-infra20:57
nibalizerclarkb: do ping me if there is ways i can help20:57
clarkbnibalizer: thanks, I am making a list of toplevel devs now and checking the versions against what we have on the master20:58
*** jpeeler has quit IRC20:59
*** jpeeler has joined #openstack-infra20:59
*** jpeeler has quit IRC20:59
*** jpeeler has joined #openstack-infra20:59
*** sdake has quit IRC20:59
jeblairstatus notice Gerrit will be unavailable between 22:00 and 23:59 UTC for project renames and a database update.21:00
jeblairfungi: ^ ?21:00
*** armax has quit IRC21:01
*** luqas has quit IRC21:01
*** esker has quit IRC21:02
*** bcourt has quit IRC21:03
jeblair#status notice Gerrit will be unavailable between 22:00 and 23:59 UTC for project renames and a database update.21:03
openstackstatusjeblair: sending notice21:03
fungijeblair: lgtm21:03
jeblairi think there's some lag at play21:03
fungijust finishing up the details for the maintenance plan now21:04
*** hdd has quit IRC21:04
-openstackstatus- NOTICE: Gerrit will be unavailable between 22:00 and 23:59 UTC for project renames and a database update.21:04
*** avozza is now known as zz_avozza21:04
fungithough additional reviews of,n,z over the next half hour or so would be good so we can adjust plans accordingly if something's amiss with any of them21:05
*** peristeri has quit IRC21:05
openstackgerritSwaminathan Vasudevan proposed openstack-infra/devstack-gate: DVR Multinode test patch to separate ctlr and compute
openstackstatusjeblair: finished sending notice21:05
fungii plan to go back through them all shortly, and then try merging them locally to check for conflicts21:05
*** Krinkle|detached is now known as Krinkle21:06
*** mriedem has quit IRC21:07
jeblairfungi: on it21:08
jeblairyou'd think i woulda done that when i was copying the project names out of them into the announcement :)21:08
*** tonytan4ever has joined #openstack-infra21:09
jeblairfungi: i'd actually just recommend going ahead and stacking them in new patchsets21:11
jeblairfungi: that way we avoid the risk of octomerge21:11
jeblair(which is a terryfying multi-headed beast)21:11
fungifair point21:11
fungii'll do that close to the maintenance so that we don't risk approving anything else ahead of them21:11
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Enable parameter ordering in trigger_parameterized_builds
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Support use of lists in trigger-build publisher.
dtroyerdo we have any jobs that do testing in other languages? ie, C or Go or something that needs compiler support?  I'm looking for a cargo-cult to join…21:12
fungidtroyer: how about a support group instead? ;)21:13
fungidtroyer: there is some java21:13
jeblairdtroyer: java.  ish.21:13
jeblairi mean, we turned the tests off.  but that's actually a minor incidental detail.  :)21:13
fungiwell, we do have gerrit tests enabled now i believe (or has that patch merged yet?)21:13
jeblairfungi: oh, we're re-enabling them?  cool.21:14
fungii know i reviewed it very recently and it seemed to dtrt according to the log21:14
fungiincluding actually passing ;)21:14
dtroyerMy goal is to get some useful tests in stackforge/golang-client21:14
dtroyerif there's examples or precedent I'm all ears21:14
jeblairdtroyer: basically, you've got a test node with root on it, so you can install whatever you need.21:14
dtroyerotherwise I'll need to work it out21:15
fungidtroyer: jobs run shell commands. if your stuff happens via shell commands, there's not much to it21:15
jeblairdtroyer: if you can do that with jenkins-job-builder macros in project-config, that makes it easier for other folks to set up a 'go' env in the same way21:15
fungishell commands and sane exit codes21:15
*** ldnunes has quit IRC21:15
*** tiswanso has quit IRC21:15
dtroyerok, cool. and if I'm only pulling in distro stuff there shouldn't be any network-related hiccups21:15
fungibasically the two requirements21:15
jeblairdtroyer: yeah, it'll come from the cloud provider mirrors which are _generally_ okay :)21:16
dtroyeralright, thanks21:16
zarowaynr: failed tests?21:16
jeblairdtroyer: fungi is working on bindep which will let you express what you need in a way that we can pre-cache on nodes like devstack.  and maybe just say "install the bindep requirements for this repo" at the start of the job21:16
jeblairfungi: but you're probably not quite ready for dtroyer to start using bindep for this, right?21:17
dtroyerI want to see how close to the automated golang testing setup I can get, they're using docker now but I was thinking that might be v2.021:18
fungijeblair: dtroyer: er, it's very close. we could add it as a guinea pig21:18
fungiit's basically ready for me to tag a 1.0.0 release so it gets to pypi21:18
*** pc_m has quit IRC21:18
jeblairfungi: oh cool, well, i think that'd be ideal if dtroyer is game21:19
dtroyerI'm good with trying that…ping me when youre ready to try it21:19
fungiand whip up the changes to our nodepool scripts to preinstall it onto our images and work it into the package caching we're currently doing21:19
fungibut otherwise it's in good shape now i think21:19
* jeblair is excited21:19
fungime too. if other stuff would just stop breaking i'd have fewer excuses to not have finished it already21:20
*** shardy has quit IRC21:20
* dtroyer puts away secret hammer21:20
openstackgerritClark Boylan proposed openstack-infra/system-config: Install all puppet modules from source
clarkbnibalizer: fungi jeblair ^ that should be correct as of our current module set21:21
jeblairif any projects have voting rally jobs, they may be disrupted for a while after the maint while jjb updates those defs21:21
fungiboris-42: ^ happen to have a list of who those are handy?21:22
jeblairclarkb: oh, did that bitrot?21:22
jeblair(i was like -- i thought we already did that, but i see the last thing is i +2d 2 months ago)21:22
*** hashar has quit IRC21:22
clarkbya it bitrotted, we have added a few more modules and a couple versions changed in places21:23
*** openstackgerrit has quit IRC21:23
clarkbbut the broken puppetdb update prompted me to resurrect that change21:23
*** openstackgerrit has joined #openstack-infra21:23
jeblairimapsync has 50 messages left!21:23
fungijeblair: you're almost a real person again21:24
fungiwith real e-mail and everything21:24
clarkbmy lunch has arrived, back shortly to participate in gerrit things21:24
boris-42fungi: hi21:24
clarkbdo we have an etherpad handy yet?21:24
boris-42fungi: there is no voting rally jobs21:25
boris-42fungi: as far as I know21:25
jeblairyaaay it's done!21:25
fungiclarkb: but i'm still fleshing out steps 12 and later21:25
*** sdake has joined #openstack-infra21:25
fungiboris-42: okay, thanks21:25
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: (hipchat) Accept list of hipchat rooms.
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Add support for hipchat notifier 0.1.8
*** dkranz has joined #openstack-infra21:26
*** stevemar has quit IRC21:29
*** sdake_ has quit IRC21:29
*** erlon has quit IRC21:31
*** emagana has quit IRC21:33
*** annegentle has joined #openstack-infra21:34
*** dangers is now known as dangers_away21:34
zarowaynr: gave you some feeback on 17504921:35
fungiokay, that etherpad is ready for review now too21:35
clarkband I have finished eating way too fast21:36
clarkbfungi: I don't see the db encoding conversion in there?21:37
openstackgerritGary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack
* clarkb reviews rename changes now21:37
fungiclarkb: oh, great point21:37
fungizaro: what was the etherpad you had the conversion steps in again?21:37
fungii'll copy those steps into here21:37
zarodidn't know we started already, thought it was in 1/2 hour21:38
fungizaro: we're still preparing and making sure everything's ready to go on time21:38
zaroahh, cool21:38
clarkbboris-42: doesn't look like you are moving rallyclient too?21:38
pleia2I'll review some more changes in a moment, chrome is being a bit weird on me21:39
* clarkb figures out if rallyclient is a thing21:39
boris-42clarkb: nope21:39
clarkbboris-42: what is ^21:39
boris-42clarkb: mistake=)21:40
clarkbboris-42: should that go to the attic?21:40
boris-42clarkb: yep it can21:40
clarkbwe don't have to do it today, just want to make sure I understand what it is21:40
boris-42clarkb: yep some day we can move it to attic21:40
*** emagana has joined #openstack-infra21:41
*** ArxCruz has joined #openstack-infra21:41
fungihrm, worth discussing, do we want to do renames first and then the database conversion, or the other way around? i'm thinking do the renames first, that way if we have to back out the encoding change by restoring the unmodified db dump we create, it's still got the renaming taken care of (rather than needing to break in between them to test the encoding behaviors)21:41
jeblairfungi: ++21:41
nibalizerclarkb: ill check it out21:41
clarkbfungi: agreed21:41
*** emagana has quit IRC21:41
fungiokay, i'll work in the encoding change to happen after rename-related database modifications21:42
*** emagana has joined #openstack-infra21:42
clarkbarosen: same question for you and moving congress to openstack/ should that move too?21:43
clarkboh it does but the acl doesn't move /me investigates more21:43
clarkbah its a sahred acl21:43
*** bknudson has quit IRC21:44
openstackgerritMonty Taylor proposed openstack-infra/shade: Add design for an object interface
openstackgerritMonty Taylor proposed openstack-infra/shade: Add skeleton for an object interface
jeblairmordred: will you be around for the maintenance in case we have any mysql questions?21:45
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Enable parameter ordering in trigger_parameterized_builds
clarkbarosen: ignore me, your change looks great :)21:45
mordredjeblair: I unfortunately need to afk in 30 minutes21:46
mordredjeblair: so, I'll be here for the first bit21:46
*** armax has joined #openstack-infra21:47
pleia2my internet connection is really lousy this afternoon, I don't think I'll be able to help this time around21:48
clarkbfungi: the murano rename and rally rename may conflict on the rally.yaml file21:48
jeblairpleia2: i thought you had two?21:48
*** jamielennox|away is now known as jamielennox21:48
clarkbfungi: they both modify within one line of each other21:48
fungizaro: so the last 4 lines of section 5 at look right?21:49
fungiclarkb: yep, i'm getting ready to stack up those changes now anyway21:49
pleia2jeblair: I do, the other one is doing an initial sync of some offsite backups21:49
pleia2so it's slooow21:49
fungijeblair suggested avoiding a gerrit multi-way merge if possible, and i agree it's a sane choice21:49
*** markvoelker has joined #openstack-infra21:49
clarkbfungi: and is unrelated correct? can I change the topic?21:50
fungiclarkb: you can. it's still outstanding from the last maintenance21:50
fungiits core review team is... somewhat slow21:50
clarkbtopic changed21:50
*** markvoelker_ has joined #openstack-infra21:50
clarkbnote is not merged yet and the saml2 rename depends on it21:50
clarkbso we won't get that in until either we remove the dep or the governance change is merged21:51
zarofungi: only need to execute convert because that will do the backup as well21:51
*** serg_melikyan has joined #openstack-infra21:51
fungiclarkb: i'm in favor of enacting jeblair's workflow change from 17346521:52
fungizaro: oh, thanks!21:52
fungii see that now21:52
clarkbfungi: fine by me particularly in this case as its a simple renmae of a project not adding anything new21:53
fungiclarkb: er, 173485 really, though i guess it depends on 17346521:53
clarkbfungi: when you rebaes you will want to remove the depends on21:53
*** Krinkle is now known as Krinkle|detached21:53
pleia2 running stackforge-specs-jobs21:53
pleia2I -1ed, but if it can be fixed afterwards maybe we just let it go21:54
clarkbfungi: other than that the stack lgtm21:54
*** gordc has quit IRC21:54
*** otter768 has joined #openstack-infra21:54
clarkbpleia2: it should work it will just run the jobs in that template21:54
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Add support for hipchat notifier 0.1.8
*** markvoelker has quit IRC21:54
clarkbpleia2: but it should be fixed at some point. If arosen is around to fix now that would be great21:54
fungii've added the utf-8 client config patch to step 14 if people could review that too
jeblairmordred: ^ if you have time21:56
*** hdd has joined #openstack-infra21:56
clarkb164264 is just changing how we connect to the db right? not changing the encoding in the db21:57
jeblairclarkb: and only for running "mysql" from the command line21:57
clarkbdoes that mean the encoding in the db is correct? I haven't been able to keep up with that21:57
fungispecifically changing how people manually invoking the client connect to the db21:57
mordredjeblair: +2 - I held off on +A21:58
fungiclarkb: i need to add steps to do the trove config switch and instance restart--thanks for the reminder21:58
*** markvoelker_ has quit IRC21:58
* clarkb checks what override_options does21:58
zarofungi: also you should not need to execute with the gerrit2 account, run with sudo should work.21:58
clarkbzaro: there is no override options at
clarkbI have -1'd for ^21:59
zarofungi: also i think i see why re-index complains complains about missing projects, there's some references in changes table that point to old projects.22:00
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Rename keystoneclient-federation to saml2
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Rename stackforge/murano to openstack/murano
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Move Rally to OpenStack
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Remove the tuskar-ui-extras repository from Stackforge
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Move congress projects from stackforge to openstack
clarkbzaro: what is the intent of the override options setting there?22:01
pleia2connection seems to have recovered some22:01
zaroclarkb: ohh i must have been looking at the latest docs.22:01
fungiokay, that's the new stack. please particularly check murano.yaml and rally.yaml where i resolved merge conflicts22:01
zaroclarkb: it's to allow you to override default settings.22:01
clarkbzaro: it won't affect us because we use trove, but that will break anyone using our puppet module install script along with the gerrit module22:01
clarkbso maybe the thing to do is split that out of this change and figure it out in a separate chnage22:02
zaroclarkb: ++22:02
serg_melikyanfungi: I've checked everything looks right22:02
fungithanks serg_melikyan!22:03
jeblair#status notice Gerrit is unavailable until 23:59 UTC for project renames and a database update.22:03
openstackstatusjeblair: sending notice22:03
jeblairer, when that's done i'll make it an alert22:04
-openstackstatus- NOTICE: Gerrit is unavailable until 23:59 UTC for project renames and a database update.22:04
zaroclarkb: i'll fix up, but here's the reference:
*** Swami has quit IRC22:04
openstackstatusjeblair: finished sending notice22:05
jeblair#status alert Gerrit is unavailable until 23:59 UTC for project renames and a database update.22:05
openstackstatusjeblair: sending alert22:05
clarkbzaro: I see maybe we can just update line 30 instead22:06
fungiokay, i added the trove config changes into step #522:06
*** annegentle has quit IRC22:06
zaroclarkb: yep.22:07
-openstackstatus- NOTICE: Gerrit is unavailable until 23:59 UTC for project renames and a database update.22:07
*** ChanServ changes topic to "Gerrit is unavailable until 23:59 UTC for project renames and a database update."22:07
clarkbfungi: looks good based on what I know22:07
fungii'm in place and ready on review.o.o once zuul is stopped22:08
clarkbthe implicit backup bugs me a bit, I like to see that be explicit but I will live22:08
jeblairfungi: puppetmaster has been made safe22:08
jeblairi will shut down zuul now22:08
clarkbI am ready to help wherever I can, just let me know22:08
jeblairfungi, clarkb: oh, one thing22:09
jeblairi think we have not installed a new gerrit war with my change?22:09
fungioh, we probably should22:09
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding
fungido we need an updated puppet change to switch the warfile name?22:09
openstackstatusjeblair: finished sending alert22:09
* clarkb grabs repo moves/renames22:09
jeblairfungi: yeah.  maybe let's just do that later.22:10
zaroclarkb: fixed up that change.22:10
fungiokie dokie. we can always do a quick gerrit restart when someone'22:10
fungis around over the weekend22:10
*** MaxV has quit IRC22:10
jeblairfungi: zuul is stopped, clear to stop gerrit22:10
fungion it22:10
*** MaxV has joined #openstack-infra22:10
* zaro pushed just in time22:11
fungiit's down. working on the database now22:11
clarkbmoving repos on the mirrors now22:12
fungii've finished the rename-related changes and am working on the encoding changes now22:13
*** sigmavirus24 is now known as sigmavirus24_awa22:13
*** andreykurilin___ has quit IRC22:13
jeblairokay, i wrote the puppet change to upgrade gerrit22:14
clarkbgit01-05 are done22:14
clarkbdoing github now22:15
*** MaxV has quit IRC22:15
*** hdd has quit IRC22:15
zarojeblair: upgrade?? is that for later?22:17
fungizaro: to upgrade the production gerrit to include his debugging patch for stream-events22:17
*** soren has joined #openstack-infra22:17
zaroahh, ok.22:18
fungiin hopes of having it running the next time they hang22:18
*** jgrimm is now known as zz_jgrimm22:20
fungii went ahead and did the filesystem updates while waiting on the database reencoding to finish22:20
waynrjeblair: just curious, was there any kind of prior notice for the gerrit downtime? like on a mailing list that i can subscribe to for future reference22:20
waynrthanks fungi!22:21
fungiwaynr: same message was also sent to openstack-infra for people not subscribing to openstack-dev22:21
waynrcool, i'll subscribe to both22:21
*** sdake_ has joined #openstack-infra22:24
*** soren has quit IRC22:24
fungithe database dump has been converted and it's been reapplying that to the db server for a few minutes now. not sure exactly how long to expect it to take but it's a 2.1gb dump (uncompressed)22:24
*** otter768 has quit IRC22:24
zarois it doing the import now?22:25
clarkbgithub is done22:26
fungiyeah, i have to assume it's nearly finished22:26
*** sdake has quit IRC22:27
zarofungi: as i recall testing with prod data it took quite some time, maybe 40 mins?22:28
jeblairi think it's safe to do step 16, the ansible workspace cleaning.  i'll do that now.22:28
fungijeblair: yep, should be22:28
*** Sukhdev has quit IRC22:28
fungizaro: okay, thanks. hopefully it will be faster. the backup and conversion only took a few minutes22:28
*** stevelle has joined #openstack-infra22:28
fungiah! it's moved on to table conversions now22:28
zaroohh ok that's much faster than i expected!22:29
cloudnullhi infra. seems to be 503'ing. known issue?22:29
zaromusta got lots O memory on that machine.22:29
jeblaircloudnull: see topic and
zarocloudnull: gerrit downtime for maintenance22:30
*** bswartz has joined #openstack-infra22:30
* cloudnull goes and reads. 22:30
cloudnullthanks jeblair zaro22:30
clarkbI should convert the git0X moves to a set of ansible commands22:31
clarkbalso the review.o.o moves22:31
jeblairreally the whole thing :)22:31
clarkbthat should be a good place to start the ansiblification of the whole thing22:31
zarofungi: should be done now, yeah?22:31
clarkbya but you have to start somewhere22:31
jeblairclarkb: agreed.  the mysql part sounds harder :)22:31
fungizaro: it's converting the change_messages table22:31
clarkb also that may present trouble22:33
clarkbtrying to find out if the api exposes that22:33
fungiokay, it just finished22:36
*** otter768 has joined #openstack-infra22:36
fungilucene reindex going now22:36
*** annegentle has joined #openstack-infra22:38
*** dboik_ has quit IRC22:41
*** davideagnello has joined #openstack-infra22:41
*** dboik has joined #openstack-infra22:41
clarkbyou can rename within an org, but I still find no way to transfer ownership via the api22:42
*** ArxCruz has quit IRC22:42
jeblairclarkb: so we could either not transfer (delete/add), or just do the next 400 by hand and never do another one again22:43
clarkbjeblair: ya22:43
clarkbits not the end of the world, just curious to see what we can actually do with the github api22:43
greghaynesSince its just a mirror is there any downside to just git clone; create; git push to new location?22:44
greghaynesoh, delete/add22:44
greghaynesI bet thats what that means ;)22:44
*** rlandy has quit IRC22:44
jeblairgreghaynes: yeah.  transfer creates internal redirects on github's side.22:44
clarkbgreghaynes: github sets up redirects when you transfer. and when we don't transfer people say "why didn't you transfer?" but thats about it22:44
clarkbwe did use add delete22:45
clarkbdelete add :)22:45
jeblairclarkb: yeah, they say that for a day or two and then they stop saying it.  :)22:45
clarkbfungi: how is the indexing going?22:45
fungialmost there22:45
jeblairanyway, all just goes to show that encoding the software development lifecycle into the name of your git repository is, er, not a sustainable best practice.  :)22:46
fungiokay, lucene reindex done! now we should be able to start gerrit again22:46
fungianything else we need to worry about before i start it?22:46
*** sdake has joined #openstack-infra22:46
jeblairfungi: not that i can think of22:47
clarkbI have nothing22:47
fungiit's on its way up now22:47
clarkbjeblair: we probably want ot wait on starting zuul to check that gerrit is happy with new db first22:47
jeblairclarkb: agreed22:47
jeblairanyone have a test change handy we can poke at utf8 stuff?22:47
zaromay time a little time to start.22:48
jeblairzaro: it's been up for a couple minutes; i'm already logged in22:48
fungii was going to leave a comment, appropriately, on
clarkb my comment there22:49
*** annegentle has quit IRC22:49
clarkbI see snowmen22:49
fungiseems to have worked for me22:49
jeblairwell, that's what this was all about, right? :)22:50
fungigraffiti time!22:50
clarkbwe should see if old comments render properly22:50
*** doug-fish has left #openstack-infra22:50
* clarkb looks up the change from that ml thread22:50
*** sdake_ has quit IRC22:50
fungithey'll just be ??????22:50
fungiit was doing lossy substitution when it stored them22:50
jeblairyeah, i think fungi is right22:50
clarkboh thats too bad22:50
jeblairstart zuul?22:50
clarkband yes confirmed
fungiso there's nothing for it to decode. it's ascii question marks in the database22:51
*** Sukhdev has joined #openstack-infra22:51
fungihave at it22:51
clarkbright the client was ocnverting because it was talking not utf822:51
jeblairfungi, clarkb: look on the bright side -- at least it's still ???s.  :)22:51
clarkbI am ready for zuul to start22:51
jeblairwe could have turned everything into snowmen22:51
jeblairreloading queue22:52
clarkbjeblair: we really should've22:52
clarkbjeblair: maybe we do that next winter22:52
fungilet me know when i should approve changes22:53
jeblairfungi: ready now.22:53
fungii guess they get to fight the queue22:53
jeblairoh, sorry about that.22:53
jeblairi don't think they'll wait for long.22:53
*** achanda has quit IRC22:54
openstackgerritJames E. Blair proposed openstack-infra/system-config: Update production gerrit to
fungiyeah, things weren't terribly busy when we started, so should be reasonably quick22:55
*** aysyd has quit IRC22:56
clarkbjeblair: we want to do 175068 real quick and restart again?22:56
*** achanda has joined #openstack-infra22:56
fungiprobably better to let it settle and then restart again quick tomorrow or something22:56
jeblairyeah, let's do it tomorrow morning22:57
fungior later tonight22:57
jeblairalso, i'm pretty sure puppet does everything.22:57
fungii'll be around most of tomorrow22:57
jeblairthough since it also does a reindex, it's not fast.  :/22:57
*** vilobhmm1 has joined #openstack-infra22:57
clarkboh right puppet should do it all22:58
vilobhmm1does this bug still makes sense since now magnum are part of openstack namespace ?22:58
openstackLaunchpad bug 1419766 in Magnum "Magnum is already taken by someone else in pypi" [Wishlist,Triaged]22:58
clarkbwfm to do it tomorrow, I too will likely be around22:58
zaropuppet does it all on review-dev last i tried22:59
sdakenot being a pypi expert, I'm not sure what the best answer is vilobhmm122:59
clarkbvilobhmm1: well you will need tofigure out a name you can publish too22:59
sdakeI suspect someone in here will know22:59
fungivilobhmm1: yeah, unless you can convince the current maintainer of to give it up, your options are to rename your project or publish it under a different name on pypi22:59
morganfainbergfungi, yay renamed repo!23:00
sdakeI never liked that name magnum anyway ;-)23:00
fungimorganfainberg: well, not completely renamed yet. we've still got config changes in flight which need to get applied23:00
morganfainbergeh, close enough... it's on it's way23:01
clarkbfungi: so we are ready to merge the projects.yaml and related update changes now?23:01
clarkber thats what you were talking about earlier23:01
fungigerrit's at least running again, and supports utf-8 comments now! (thanks zaro and pelix!)23:01
sdakeuploaded 201223:01
vilobhmm1ok…so nothing needs to be done as of now as the name will eventually change from magnum->openstack_containers ?23:02
fungiclarkb: yeah, they're approved, just waiting23:02
*** dboik has quit IRC23:02
vilobhmm1with these config changes is what you mean ?23:02
clarkbvilobhmm1: I wouldn't expect it to be renamed openstack_containers23:02
clarkbbut you can publish to a different name on pypi23:02
jeblair#status ok Gerrit is available again.23:03
openstackstatusjeblair: sending ok23:03
openstackgerritMerged openstack-infra/project-config: Drop neutron-pg  in favor of a voting neutron-pg-full
fungivilobhmm1: no, your project name has not changed. your repositories have moved from the stackforge namespace to the openstack namespace, but that's all (e.g. stackforge/magnum became openstack/magnum)23:03
jeblair(i kind of think the name of the git repo should be the name of the thing published on pypi)23:03
clarkbjeblair: that is definitely ideal23:04
fungii agree. we have some unusual exceptions to that, but generally should avoid a mismatch23:04
openstackgerritMerged openstack-infra/project-config: Move neutron-pg-full to periodic
clarkbI'm just not a fan of unnecessary renames. It isn't hard on just gerrit, it is hard on anyone that may have cloned or installed your software previously23:04
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source"23:04
-openstackstatus- NOTICE: Gerrit is available again.23:04
jeblairclarkb: i agree.  we should probably build a check into the creation process to catch this.23:05
morganfainbergfungi, so out of curiosity is the infra slides (e.g. ) a pattern that makes sense for other projects? e.g. if keystone had set of publications (outside of the sphinx docs) on identity; or is this a rabbit hole/can of worms? (cc clarkb, jeblair, mordred ) - random thought.23:06
fungiugh, we had some project-config changes already approved an in the queue apparently, so that stack of changes will all wind up as merge commits anyway23:06
jeblairfungi: double sorry :(23:06
morganfainbergfeel free to ignore my question until you all have time / less headaches [i see you're busy]23:06
openstackstatusjeblair: finished sending ok23:07
fungimorganfainberg: it was an arbitrary choice. in retrospect we probably should have named it openstack-infra/infra-publications but name vaguery aside if it's a pattern you want to copy then feel free23:07
jeblairmorganfainberg: only thing to keep in mind is that we have it set up as branch-per-publication.  there's a bit of overhead to creating a new publication, and the model may not be for everyone.  but of course you don't have to do it that way.23:07
fungiand if you come up with a better solution, we might copy yours instead ;)23:08
pleia2ours are also text-based files in git, which are not what a lot of folks write slides in23:08
jeblair(i'm actually happy with branch-per-pub, but i came up with the idea so...)23:08
morganfainbergjeblair, i'll poke at what you've done but I'd like to see this happen for keystone. I would really enjoy official publications- and yes if we have something better (or think it might be easier) happy to discuss it23:08
pleia2giant binary .odp/.ppt files would be a bit of a mess23:08
morganfainbergbut branch-per-pub doesn't sound bad23:08
fungirighth, it may not work nearly so well for people writing slides in non-plaintext markup23:09
*** MarkAtwood has quit IRC23:09
fungisince that's a lot harder to code review23:09
clarkbjeblair: I like it too, also helps that you can git checkout -b origin/template my-talk and get going23:09
jeblairyeah, i thought it seemed reasonable because they are often completely unrelated.  also, if you really want, you could set per-branch acls23:09
morganfainbergfungi, oh i expect that we would do [if anything] a "this needs to be converted to a sane format if it's an official publication"23:09
morganfainbergnot just dumping PPTs in23:09
morganfainberggod no.23:09
clarkber I got order on that wron23:09
*** patrickeast has joined #openstack-infra23:10
morganfainbergjeblair, yeah.23:10
fungimorganfainberg: but if your team settles on a presentation framework that takes plaintext markup with images/media included as separate files, then it could probably work well for you23:10
fungibeing able to fork a presentation and then make incremental reviewable changes to it is extremely convenient23:10
*** doude has quit IRC23:11
jeblairmorganfainberg: there's a job that generates automatically from the branches (current publications) and tags (previous publications)23:11
morganfainbergfungi, i don't care what people use for their personal presentations, but before it would go in a central repo i would require something [rst?] that could be reviewed.23:11
*** achanda has quit IRC23:11
morganfainbergfungi, and i do like the incremental change aspect.23:11
*** ashleighfarnham has quit IRC23:11
morganfainbergit's something i actually was thinking about when i was writing up the deck for the meetup I presented at23:11
fungialso it's convenient for us in that people newish to the team and to presenting at conferences can just grab one of those canned presentations, propose the abstract included with it, and then present it23:12
jeblairi'm a fan of rst -- you can generate slidy (what we mostly use), revealjs (mordred likes spinny things), or use it with presentty :)23:12
morganfainberg1st: I don't want to do this from scratch again, 2nd: i like that infra has official publications, 3rd: it would help our users and deployers understand things w/o needing to dig through blogs etc.23:12
nibalizerwait i can generate my presentations?23:12
morganfainberg(#3 is "potentially")23:12
serg_melikyanfolks, I got -1 from Jenkins in gate-project-config-layout23:12
nibalizeri've been manually writing html things23:12
funginibalizer: i thought your presentations were already generated from random data sources23:13
serg_melikyancan't understand what is wrong there23:13
jeblairnibalizer: markov chains? :)23:13
morganfainbergjeblair, cool, i'll poke at what you guys have. i really really do like it.23:13
*** ddieterly has quit IRC23:13
morganfainbergjeblair, and i think it would be a win for [at least] us to have similar reference / not-boring-plain-docs about identity23:13
nibalizerheh, i mean i can write rst and it will generate slidy? then why do we store the slidy in git?23:13
clarkbserg_melikyan: one of the other changes involved seems to have broken it?23:13
jeblairnibalizer: i don't think we've done an RST -> slidy/reveal (maybe mordred has?).  but pandoc can do it.  i verified it when i wrote presentty23:13
clarkbfungi: ^ I am guessing that rebasing didn't go right there?23:13
morganfainbergjeblair, and not needing to say "sooo look for some blog jamielennox wrote some time about something"23:14
*** sdake_ has joined #openstack-infra23:14
fungiclarkb: possibly. i'm looking closer now23:14
nibalizerahh okay23:15
nibalizernext time!23:15
*** funzo has quit IRC23:15
jeblairyeah, worth poking at23:15
*** davideagnello has quit IRC23:15
clarkbmorganfainberg: for #3 it sounds like you need better documentation not a place to host presentations23:15
*** achanda has joined #openstack-infra23:15
clarkbmorganfainberg: and imo that should go in the openstack docs and/or keystone docs23:16
clarkbbecause that is where people will go to look23:16
morganfainbergclarkb, nah it's things like "you could do X, Y, Z and here are examples of what has been done" not just the plain docs.23:16
clarkbmorganfainberg: why shouldn't that go in the plain docs?23:16
clarkbmorganfainberg: thats exactly where I would look for such things23:17
morganfainbergclarkb, difference between deployment specific choices and evaluations for these specific use-cases of benefit vs. the more raw "thing x is supported and this is how"23:17
*** sdake has quit IRC23:17
fungiJob gate-python-keystoneclient-federation-requirements not defined23:17
clarkbmorganfainberg: for example and
fungiagreed, probably a rebase error. seeing if i can spot it23:17
morganfainbergclarkb, "I chose memcache to store tokens because in my environment I use memcache, and we did this thing etc"23:18
clarkbmorganfainberg: yes I understand23:18
clarkbmorganfainberg: there is no reason they cannot both be hosted under docs/keystone23:18
morganfainbergclarkb, or "here is a working example of WebSSO and steps i used in my environment23:18
clarkbmorganfainberg: one would be docs/keystone/deployment_strategies the other docs/keystone/tutorials or whatever23:18
morganfainbergclarkb, sure. i see the publications as more white-paper / slides but i don't know if i'd put it in the keystone tree docs.23:19
morganfainbergclarkb, i'll talk it over with the team and i'll bring up your points too :)23:19
morganfainbergclarkb, it's worth mulling over before we do anything :)23:19
clarkbmorganfainberg: so I have long tried to push openstack docs proper this direction fwiw23:19
morganfainbergclarkb, you know, i don't care where it coes come to thing of it. i care that we have a clear publication list we can link to.23:20
morganfainbergclarkb, so i'll think it all over and also disucss with the team :)23:20
morganfainbergclarkb, you did make me realize it was "official place for these things" not "something like what infra does" :)23:20
* morganfainberg runs to the airplane.23:21
serg_melikyanfungi: looks like I am late with my -1 :) You found not renamed line sooner in layout.yaml )23:21
morganfainbergclarkb, thanks!23:21
*** wenlock has quit IRC23:23
fungiserg_melikyan: well, that's not the actual cause of the error. zuul ignores those if they don't match anything (though it will need to be corrected)23:23
fungiserg_melikyan: or if it is the cause of the issue then i'm not sure why the previous version passed before rebasing23:24
fungithe log, which hasn't posted to gerrit yet, is
clarkbfungi: the not defined should make the job fail23:25
fungiyeah, so why did it pass before?23:25
clarkbits saying that the job is in zuul layout but jenkins doesn't know about it23:25
fungioh! i bet i know why23:25
clarkbfungi: because of the rebasing23:25
clarkbI think23:25
serg_melikyan I don't +1 from jenkins after rebase23:25
fungithat probably was on the tip of master23:25
fungiand was newer than the original patch23:26
fungiokay, fixed stack on the way, re-rebased onto master23:26
fungilet's refrain from approving any more project-config changes until these land23:26
*** Swami has joined #openstack-infra23:28
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Rename keystoneclient-federation to saml2
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Rename stackforge/murano to openstack/murano
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Move Rally to OpenStack
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Remove the tuskar-ui-extras repository from Stackforge
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Move congress projects from stackforge to openstack
*** dboik has joined #openstack-infra23:29
fungithat oughta do it now23:29
clarkbSwami: was is the intent with ?23:29
*** jamielennox is now known as jamielennox|away23:30
clarkbSwami: we do not want to do that separation as many nova and neutron behaviors depend on having two compute hosts23:30
clarkbSwami: also I think the regular job isn't working so well currentkly23:30
Swamiclarkb:ok got it.23:30
clarkbs/two compute hosts/more than one compute host/23:31
Swamiwe are seeing some failures so I wanted to figure out what was causing those failures.23:31
Swamiright now I don't think the neutron tests require more than one compute node.23:31
clarkbSwami: it is likely that having more than one compute makes things worse23:31
clarkbSwami: they do23:31
Swamiclarkb: no it does not make things worse. we have our internal CI running with two or more compute hosts.23:32
clarkbSwami: we test things like live migration, shelve and unshelve, and others that have different behaviors with two compute hosts23:32
SwamiI just want to figure out if having every VM instance in a single node, will make things better.23:32
clarkbSwami: we already test neutron with a single compute host so we really don't want to make the multinode job a single compute host23:32
*** sdake has joined #openstack-infra23:32
clarkbSwami: we know it will make things better, that is how we test normally23:32
SwamiNO even though we have a single compute host, we do have a "network node".23:33
SwamiSo in our case we do have routers created on the network node and on the compute node.23:33
clarkbSwami: but in this case compute and network are colocated as on the single node test23:33
SwamiSo we will be basically testing the multinode scenarion irrespective of the number of compute nodes.23:33
clarkbSwami: we will not be testing the tests above23:33
fungie.g. does live migration work with multi-node dvr? it's anybody's guess because we won't be testing that23:34
Swamibut will the nova be scheduling the VMs on both the nodes.23:34
clarkbSwami: yes23:34
clarkbSwami: thats the point23:34
Swamiclarkb: yes I understand that it would work, but I was having a chat with armax today and we decided to do some more testing and see if I can force all the compute to be in the same node and see how the tests fair.23:35
clarkbSwami: I think thats fine, I am just saying we should not merge that even if it makes things better. The purpose of this test is to have two compute nodes23:35
Swamiclarkb: understood. I will not merge it.23:36
Swamithis is just for my learning and debugging.23:36
*** nikhil_k is now known as cpallares_23:36
* armax scrolls up23:36
*** sdake_ has quit IRC23:36
armaxSwami: by the way I was looking here:23:36
armaxSwami: it looks like we have a pass, I dug a bit deeper in the log and it seems like we’re landing VM’s across both compute hosts23:37
fungilooks like the layout jobs are passing on the new stack. one is in the gate pipeline now23:37
clarkbarmax: Swami yes we are putting compute hosts on both nodes, we know this if live migration test passes23:38
Swamiarmax: that is good. that's what clarkb was also saying that it would schedule on both nodes.23:38
clarkbarmax: Swami that is the specific intent of these tests23:38
*** flaper87 has quit IRC23:38
armaxclarkb: true, but we had seem tests where we had massive failures23:39
armaxclarkb: migration test included23:39
clarkbarmax: yes, I think dvr is broken23:39
clarkbarmax: now is the time to fix it before we release (if we can)23:39
armaxclarkb: well, not entirely23:39
clarkbbut fixing it doesn't involve "use one compute node" because in production there will be many compute nodes23:39
armaxclarkb: that’s right, there’s no point to use one compute node permanently in the test23:40
*** cpallares_ is now known as nikhil_k23:40
clarkbif I had to guess what is breaking it is the arp stuff that makes vxlan work or ovs flows breaking23:41
clarkbthe first one is easy to test by replacing vxlan with gre right?23:41
clarkbI have no idea how to debug ovs23:41
armaxclarkb: we could switch DVR to using GRE23:41
Swamiclarkb: no worries, for the dvr test failures, mostly we have seen issues just with the floatingIP not getting ready in time for the instances to test.23:42
clarkbarmax: its just a hunch but the replace arp with service to keep arp tables populated seems like a culprit for connection failures23:42
fungiespecially if that process gets resource-starved and lags23:42
armaxclarkb: just to be clear, the migration test does not validate networking23:44
clarkbarmax: it does23:44
armaxclarkb: this one?
openstackgerritMerged openstack-infra/project-config: Rename keystoneclient-federation to saml2
*** ajmiller has quit IRC23:45
clarkbarmax: I think so? maybe its a scenario test. But it does a libvirt migration which requires two hosts (libvirt won't migrate to itself) and libvirt requires working networking23:45
armaxclarkb: I am not the most expert in Tempest tests, but all I could see are API tests and Scenario tetss23:46
*** stevelle has quit IRC23:46
armaxclarkb: scenario tests that I am aware of do not test migration23:46
*** stevelle has joined #openstack-infra23:46
clarkbtempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration thats the one according to console logs23:47
clarkbits the one you link to23:47
armaxso I think that Swami’s troubleshooting attempt is only to rule out any possiblity that we see test passes just because we’re hitting the compute host colocated with the controller node where tempest is executed23:47
armaxclarkb: the test is there, but I can’t see anything in the test code itself that tests connectivity23:48
clarkbarmax: libvirt itself "tests" it23:48
openstackgerritMerged openstack-infra/project-config: Move Rally to OpenStack
clarkbarmax: live migration won't work if networking isn't working23:49
armaxclarkb: I mean FIP connectivity23:49
clarkbarmax: yes I don't think FIP connectivity is tested23:49
armaxwe do know that the VM’s do get DHCP address23:49
clarkbshelve unshelve is likely a better test for that since it will move a host offline (not live) and it does check FIP connectivity23:49
clarkbtest_shelve_instance under scenario testing somewhere23:50
*** patrickeast has quit IRC23:50
fungiin the future it might be interesting to try a three-node variant where tempest runs on one node with no devstack whatsoever, just the tunnels it needs to reach the "pub" side of the two devstack nodes23:50
armaxclarkb, fungi, swami and I were also considering the possiblity to write a scenario test where we force two VM’s to land on separate hosts and tess cross connectivity23:51
clarkbarmax: I think that test exists, but let me look23:51
armaxthat would be the ultimate test for dvr23:51
armaxclarkb: there’s a similar one23:51
armaxin test_security_groups_basic_ops iirc23:52
openstackgerritMerged openstack-infra/project-config: Remove the tuskar-ui-extras repository from Stackforge
armaxbut the VM’s can land on the same hosts23:52
fungii wonder if there are any anti-affinity tests?23:52
armaxclarkb: is this the one you’re thinking about?23:52
fungiwhere we explicitly schedule them on different compute nodes23:52
clarkbarmax: ah so we have a cross VM test but it doesn't enforce scheduling of the VMs23:52
clarkbarmax: I think23:52
armaxfungi: that’s right23:52
armaxthat’s what I am talking about23:53
openstackgerritMerged openstack-infra/project-config: Move congress projects from stackforge to openstack
armaxI am not 100% sure whether we have thouse23:53
armaxI would be pleasantly surprised if we did23:53
fungii am woefully unfamiliar with actual openstack features, so it's entirely possible anti-affinity is not even a thing yet. i really don't know23:53
clarkbI think we don't have them explicitly23:53
armaxbut if we don’t, they would make a fine addition for the multi-node jobs23:53
Swamiarmax: yes that would be go idea.23:53
clarkbthe way the nova shceduler works we should test that most of the time23:54
armaxfungi: nova’s scheduler can be passed hints23:54
clarkbbut it isn't enforced23:54
*** annegentle has joined #openstack-infra23:54
armaxthey work only as admin23:54
clarkbby default the scheduler will round robin across the compute nodes23:54
fungiarmax: okay, so is sounds doable anyway23:54
armaxand can be used to specify a host on which to do the placement23:54
armaxat least that used to be there ;)23:54
clarkbso we should get two VMs on two hypervisors when booting two VMs but that isn't enforced23:54
armaxclarkb: corrct23:54
armaxclarkb: the two VM’s might as well as land on the same host23:55
armaxclarkb: we’d need a way to be sure we always land on separate hosts23:55
openstackgerritMerged openstack-infra/project-config: Rename stackforge/murano to openstack/murano
fungiyeah, with tests running in parallel that probably mucks with any expectations we might have from the round-robin scheduling23:55
clarkbfungi: yup23:55
clarkbbut evidence is that the majority of the time we get that behavior23:56
armaxfungi: there might be a nova scheduler filter with a round robin placement strategy23:56
clarkbarmax: thats the default23:56
fungiokay, that was the last change we were waiting on to merge. should be safe to unleash puppet from its chains once more23:56
fungiclarkb: jeblair: objections?23:56
clarkbfungi: not from me23:57
armaxclarkb: at one point the default was to fill up hosts first23:57
*** dboik has quit IRC23:57
jeblairfungi: nope, i will do that now23:57
clarkbarmax: that isn't how it works based on my testing. at least not now23:57
fungijeblair: thanks!23:57
armaxclarkb: or at least the one I recall, for the round robin one, I am not sure how guaranteed the randomeness is23:57
clarkbarmax: if you take the multinode test env then boot 2 nodes you will get one node on each hypervisor23:57
armaxclarkb: fair point23:57
*** serg_melikyan has quit IRC23:57
clarkbarmax: I think it may account for available resources to concurrent tests can affect scheduling23:57
armaxclarkb: right23:57
armaxclarkb: it depends on teh flavors too23:58
jeblairfungi, clarkb: it's 2 mins till, so i just uncommented the cron and will let it run normally23:58
clarkbjeblair: wfm23:58
fungisounds fine23:58
armaxclarkb: as far as I can tell, the backbone for testing DVR on multi-node is solid23:58
fungii'll start prepping .gitreview changes23:58
clarkbarmax: the other thing we are seeing with nova network is dnsmasq isn't being updated with mac addr to IP mappings on some tests23:59
armaxclarkb: now we need to figure out where is the source of instability and kill it as fast as we can23:59
clarkbarmax: which breaks dhcp, but you said that dhcp is working there23:59
*** weshay has quit IRC23:59
armaxclarkb: is this instability recently observed?23:59
armaxclarkb: as far as I can tell, all the tests that failed, failed because of L3 glitches, not because of DHCP23:59
clarkbarmax: no, its also not as high an error rate as the dvr job23:59

Generated by 2.14.0 by Marius Gedminas - find it at!