Wednesday, 2014-12-10

anteayajogo: no longer blocked by me devstack-gate is your next port of call00:00
mtreinishclarkb: yeah I think checking >= 1 is the only path forward00:00
mtreinishsc68cal: ^^^00:00
*** nelsnelson has quit IRC00:00
*** koolhead_ has joined #openstack-infra00:00
*** _nadya_ has joined #openstack-infra00:00
*** camunoz has quit IRC00:01
sc68calJayF: CPE equiment will get at least 2 IPv6 addrs when doing PD - a /128 and a /6400:01
sc68calcustomer premises equip00:01
sc68calactually better example, renumbering00:01
openstackgerritMerged openstack-infra/system-config: Add new git haproxy frontends to cacti
sc68cala router with two prefixes - devices attached to router will get two prefixes and migrate over as renumbering00:02
jogoanteaya: ?00:02
fungiokay, back now00:02
sc68calmtreinish: yeah >=1 might be it00:02
jogoanteaya: you mean the next patch in the series is blocked on devstack-gate? if so correct00:02
fungiso much with the talking00:02
*** shakamunyi has joined #openstack-infra00:02
clarkbfungi: git.o.o round robin complete. jeblair has done more bisecting and narrowed it down to a couple packages00:03
*** shakamunyi has quit IRC00:03
*** ryanpetrello has quit IRC00:04
*** koolhead17 has quit IRC00:04
openstackgerritRamy Asselin proposed openstack-infra/project-config: Prepare project-config for puppet module split
anteayajogo: yes sorry that was my meaning00:04
*** sabeen1 has quit IRC00:05
*** ivar-lazzaro has joined #openstack-infra00:05
*** _nadya_ has quit IRC00:05
clarkbfungi: also will fix the hostfile things00:05
openstackgerritRamy Asselin proposed openstack-infra/project-config: Prepare project-config for puppet module split
*** asettle is now known as asettle-errands00:07
jeblairclarkb, fungi: one line in fact:00:07
jeblairin paramiko/rsakey.py00:07
jeblair-        rsa = RSA.generate(bits,, progress_func)00:07
jeblair+        rsa = RSA.generate(bits, os.urandom, progress_func)00:07
jeblairthis has a familiar ring to it somehow00:08
clarkboh hey is that like what salt was doing?00:08
*** melwitt has quit IRC00:09
clarkb4) it makes multithreaded python terrible00:11
*** david-lyle is now known as david-lyle_afk00:11
clarkber 5)00:11
*** ZZelle_ has quit IRC00:12
*** andreykurilin_ has quit IRC00:12
*** andreykurilin_ has joined #openstack-infra00:13
clarkbnow I am glad that I WIP'd yolanda's patch to update paramiko00:13
openstackgerritMerged openstack-infra/system-config: Use puppet inventory when running ansible.
*** gokrokve has quit IRC00:14
*** packet has quit IRC00:15
*** gokrokve has joined #openstack-infra00:15
*** kumartin has quit IRC00:16
*** dims has quit IRC00:17
*** camunoz has joined #openstack-infra00:17
*** camunoz is now known as camunoz_mtg00:17
openstackgerritRamy Asselin proposed openstack-infra/project-config: Prepare project-config for puppet module split
*** dims has joined #openstack-infra00:19
sc68calcommit message needs rework00:19
*** Masahiro has joined #openstack-infra00:19
*** sabeen has joined #openstack-infra00:20
*** gokrokve has quit IRC00:20
*** jerryz has joined #openstack-infra00:20
*** Sukhdev has joined #openstack-infra00:23
*** hdd has quit IRC00:23
*** pc_m has quit IRC00:23
openstackgerritMerged openstack-infra/reviewstats: Update neutron core
fungijeblair: ha!00:23
*** pc_m has joined #openstack-infra00:24
clarkbI need to run shortly for an early dinner. It would be asomeness if someone could make sure that cacti gets the new git-feXX nodes in it before tomorrow00:24
clarkbthat way we get the precious datas with tomorrows nodepool image updates00:24
fungijeblair: so it really is contention over /dev/urandom during parallel key generation calls?00:24
*** Masahiro has quit IRC00:24
* fungi is finally done catching up on scrollback00:24
openstackgerritSean M. Collins proposed openstack-infra/devstack-gate: Set IP_VERSION in DevStack to run dual stack
*** pc_m_ has joined #openstack-infra00:25
fungiclarkb: now that 140497 is in, i'll keep an eye on puppetboard and make sure things continue puppeting00:25
fungi(and that things which weren't previously in the static hostfile start puppeting too)00:25
*** spzala has quit IRC00:26
fungiclarkb: since the cacti change merged a while ago, what else needs to be done for that? or just keep an eye on cacti and make sure they show up?00:27
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Added cron plugin to clean old access tokens.
*** pc_m has quit IRC00:28
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Added cron plugin to clean old access tokens.
*** doug-fish has left #openstack-infra00:30
*** dims has quit IRC00:30
*** Sincler has quit IRC00:30
*** Sincler has joined #openstack-infra00:30
*** Sincler has quit IRC00:30
*** rkukura has quit IRC00:30
*** Sincler has joined #openstack-infra00:31
*** Sincler has quit IRC00:33
*** Sincler has joined #openstack-infra00:33
krotscheckBecause OMG my access token change showed me that I have 100’s of old access tokens.00:34
fungikrotscheck: you're such a hoarder00:34
krotscheckfungi:  I bet you have a lot too.00:35
jeblairclarkb, fungi: os.urandom is faster for small values, and increasingly slower for large values of data read00:35
*** Sukhdev has quit IRC00:35
jeblairso that change either made things faster or slower for us, depending on how much data paramiko reads00:35
jeblairit also introduces two GIL release/acquire events each time os.urandom is read00:36
*** rkukura has joined #openstack-infra00:37
*** amitgandhinz has joined #openstack-infra00:37
fungiideally the thread would be accessing its own secure prng, and seed it once from /dev/urandom, rather than fighting over /dev/urandom as a substitute prng00:38
jeblairfungi: this isn't locked, so i don't think there's contention here00:38
jeblairunless the kernel does that00:38
fungii think there are patches to make it possible to have independent streams reading urandom00:39
jeblairfungi: what happens if two processes open it?00:39
fungiresearching whether that's common in current kernels00:39
fungiat one point there was a rotating buffer and the kernel would take turns serving each caller00:40
jeblairhuh look at that00:40
*** rmcall has quit IRC00:41
fungiso, yeah00:42
funginot good00:42
*** ryanpetrello has joined #openstack-infra00:42
*** dims has joined #openstack-infra00:43
jeblairand indeed, the urandom manpage says to use it to seed a prng00:44
jeblairbut the paramiko commit removed just that00:44
*** Sukhdev has joined #openstack-infra00:45
*** Masahiro has joined #openstack-infra00:45
fungipretty sure i had seen literature about that being a terribly efficient pattern. disappointed to see that's still the case though00:46
fungiterribly INefficient i mean00:46
fungiet cetera00:47
*** otter768 has joined #openstack-infra00:48
fungiseems like *bsd solved it with multi-stream prng generators in-kernel a while ago00:48
*** jamielennox is now known as jamielennox|away00:48
openstackgerritSukhdev Kapur proposed openstack-infra/project-config: Add networking-arisa project to StackForge
* mordred comes back into channel and sees very sad things about paramiko00:50
*** gyee has quit IRC00:50
*** ivar-lazzaro has quit IRC00:51
*** SumitNaiksatam has quit IRC00:51
jeblairit was either doing the wrong thing before and doing the right thing now, or the other way around00:51
mordredjeblair: this is all in the post-boot-onfig stuff in nodepool via ssh? or are we using paramkio for something else?00:51
* mordred trying to tldr back in to context00:51
*** koolhead_ has quit IRC00:51
jeblairdepending on which cryptographer on the internet you are asking00:51
jeblairmordred: i believe this is in creating keys for the nodes (for use in multi-node setups)00:52
mordredgotcha, that is00:52
jeblairi think at this point we can say that new paramiko is far less efficient at generating 1000 rsa keys00:52
*** otter768 has quit IRC00:52
mordredwell, that's good to know00:52
jeblairwe could probably defer the problem by not generating keys for single-host nodes00:53
mordredwe coudl also shell to ssh-keygen and not use paramkio00:53
mordredwhich just moves the problem00:53
fungior reuse keys00:53
fungior short-circuit os.urandom by redefining it?00:54
jeblairfungi: i'd rather not worry about inter-node interference that way00:54
fungijeblair: true00:54
*** xyang1 has quit IRC00:54
mordredor - offload to nova keypair extension00:55
mordredwhich can create a key for you00:55
jeblairmordred: i'm pretty sure that would take many many orders of magnitude longer00:55
*** avozza is now known as zz_avozza00:56
fungiperhaps paramiko would take a patch upstream to make "cheap but insecure" randomness for key generation as a foot-cannon named parameter00:57
*** MaxV has joined #openstack-infra00:58
*** Longgeek has joined #openstack-infra00:59
*** mikedillion has quit IRC01:00
jeblairi'm doing some more tests01:01
*** mikedillion has joined #openstack-infra01:01
*** MaxV has quit IRC01:03
*** salv-orlando has quit IRC01:03
*** StevenK has joined #openstack-infra01:04
*** mestery has quit IRC01:04
*** Longgeek has quit IRC01:04
*** armax has quit IRC01:05
*** armax has joined #openstack-infra01:07
*** signed8b_ is now known as signed8bit_ZZZzz01:07
openstackgerritRamy Asselin proposed openstack-infra/project-config: Prepare project-config for puppet module split #2
*** dims has quit IRC01:11
*** CIBuilds has quit IRC01:11
*** mestery has joined #openstack-infra01:11
*** marun has quit IRC01:11
*** wenlock has quit IRC01:13
jeblairfungi, mordred: it seems like overall, key generation is faster with the new paramiko01:13
mordredjeblair: so, generation is faste, but less concurrency friendly?01:13
mordredbecause GIL?01:14
*** mikedillion has quit IRC01:14
jeblairit may actually be more concurrent because it may hold the gil less01:14
*** marun has joined #openstack-infra01:14
mordredoh. neat01:15
*** xianghui has joined #openstack-infra01:15
*** adalbas has quit IRC01:16
*** yamamoto_ has joined #openstack-infra01:16
jeblairthat might be giving us 500 threads all ready to run at much closer to the same time01:16
reedI'm still not sure what the recommended way is to take over someone else's changeset. Are there instructions somewhere?01:17
reedis it better to just pull from gerrit? or checkout is better?01:18
jrollreed: you can't transfer ownership, you just push patches to it and remember to check01:18
*** gokrokve has joined #openstack-infra01:18
jrollreed: I also tend to use "git review -d 12345", where 12345 is the change ID, that will check it out locally01:18
*** zhiwei has joined #openstack-infra01:18
reedjroll, that's what I meant01:18
mmedvedeasselin: were does the script to refresh split out modules run? I do not think it does properly update the seed repositories right now01:18
reedjroll, ah, great, I'll try that01:18
asselinmmedvede, it runs on sweston's server01:19
asselinmmedvede, do you have an example? I'd like to take a look?01:19
mmedvedeasselin: there were a few questions regarding the details today at the meeting. This is exactly the problem, we are relying on it to be running, without having any control.01:20
mmedvedeasselin: example - subunit2sql module01:20
openstackgerritNikhil Manchanda proposed openstack-infra/project-config: Add link for Trove RSS feed
asselinasselin looking01:20
* sweston is looking01:20
*** pc_m_ has quit IRC01:22
asselinmmedvede, yes, something is missing01:23
*** salv-orlando has joined #openstack-infra01:23
asselinmmedvede, I think we can work out the issues. And we can delay the split until we know it's correct.01:24
asselinmmedvede, another option is to have -infra run the script01:24
asselinmmedvede, but it would be better not to in order to avoid tasking them with one more thing01:25
swestonwhat is missing? I do not see it01:25
jeblairclarkb, fungi, mordred: i'll think about this some more, but we may want to just see how the trusty server behaves with the updated pycrypto and deal with things from there.  my test may be too artificial to tell us what we'll see in production.01:25
fungii'll buy that01:25
*** sabari is now known as zz_sabari01:25
*** ryanpetrello has quit IRC01:26
asselinsweston, has a commit that is 6 days old01:26
jeblairi'm going to eod now; night01:26
*** rkukura has quit IRC01:26
asselinsweston, your last commit is 21 days ago:
*** ddieterly has joined #openstack-infra01:27
*** obondarev has quit IRC01:28
openstackgerritMerged openstack-infra/project-config: Add new project packstack-vagrant to stackforge
asselinmmedvede, another option is to have multiple ppl running the script. e.g. I can run it too.01:28
mmedvedeasselin: can you push to his github?01:29
asselinmmedvede, probably not unless he adds me01:30
anteayaasselin: good call01:31
*** Masahiro has quit IRC01:31
*** amitgandhinz has quit IRC01:31
asselinmmedvede, either way, i'd like to get the script working & validated. it's a huge time saver not to have to do this01:31
anteayaalso we need the ability to trigger is manually as well01:31
*** dannywilson has quit IRC01:31
swestonok I will add those features in01:31
*** Sukhdev has quit IRC01:32
*** xianghui has quit IRC01:32
*** yamamoto_ has quit IRC01:32
*** xianghui_ has joined #openstack-infra01:32
*** mestery has quit IRC01:32
*** armax has quit IRC01:32
anteayasweston: thanks steve01:32
asselinsweston, are you ok to add a few ppl with write permissions? I think this well help as a good backup.01:32
* anteaya heads out to the restaurant01:32
asselinsweston, but hopefully we'll never need to01:33
anteayaasselin: agreed01:33
*** ddieterl_ has joined #openstack-infra01:33
*** xianghui_ is now known as xianghui01:33
*** ddieterly has quit IRC01:33
asselinanteaya, bon appetite01:34
swestonasselin: yup01:34
*** Masahiro has joined #openstack-infra01:34
*** markmcclain has quit IRC01:34
mmedvedeasselin, sweston: +1. I like the automation :)01:34
swestonanteaya: have a good dinner01:34
anteayaasselin sweston thanks01:35
anteayathe automation is a good idea, we just need to ensure we can respin on short notice01:35
swestoni'll look into the github api tomorrow01:36
mmedvedesweston: regarding on what is missing in the subunit2sql module, see commit 5941f83 in system-config01:36
*** marun has quit IRC01:38
*** shayneburgess has quit IRC01:42
*** wuhg has joined #openstack-infra01:43
*** andreykurilin_ has quit IRC01:44
*** shakamunyi has joined #openstack-infra01:53
*** yaguang has joined #openstack-infra01:55
*** ivar-lazzaro has joined #openstack-infra01:57
*** oomichi_ has joined #openstack-infra01:57
swestonmmedvede: yup, ok, found the error.  will fix it in the morning01:59
*** shashankhegde has quit IRC02:00
*** ivar-lazzaro has quit IRC02:00
mmedvedesweston: thanks. Your script is great, what was the issue?02:00
*** ivar-lazzaro has joined #openstack-infra02:01
swestonmmedvede: thanks :-).  when I built it, I didn't have the opportunity to test multiple module updates.  the issue is with the parsing of the updated modules, it's a simple fix02:01
*** weshay has quit IRC02:02
*** baoli has joined #openstack-infra02:02
*** signed8bit_ZZZzz is now known as signed8b_02:03
*** fandi has quit IRC02:03
*** asettle-errands is now known as asettle-doctor02:06
swestonmmedvede: thanks for finding the issue.  since some of the modules are still being updated, I didn't catch it02:06
mmedvedesweston: glad to help :)02:08
*** harlowja has joined #openstack-infra02:10
*** signed8b_ has quit IRC02:11
*** tnovacik has quit IRC02:13
*** yamamoto_ has joined #openstack-infra02:15
*** r-daneel has quit IRC02:15
*** tnovacik has joined #openstack-infra02:17
*** dkranz has quit IRC02:18
*** ChuckC has quit IRC02:18
*** Masahiro has quit IRC02:21
*** Masahiro has joined #openstack-infra02:27
*** SumitNaiksatam has joined #openstack-infra02:30
*** xyang1 has joined #openstack-infra02:30
*** amitgandhinz has joined #openstack-infra02:34
*** shayneburgess has joined #openstack-infra02:37
*** tnovacik has quit IRC02:40
*** camunoz_mtg has quit IRC02:41
*** oomichi_ has quit IRC02:44
*** salv-orlando has quit IRC02:47
*** ryanpetrello has joined #openstack-infra02:48
*** Longgeek has joined #openstack-infra02:48
*** otter768 has joined #openstack-infra02:49
*** Sincler has quit IRC02:51
*** amitgandhinz has quit IRC02:52
*** ChuckC has joined #openstack-infra02:53
*** Longgeek has quit IRC02:53
*** otter768 has quit IRC02:54
*** zz_sabari is now known as sabari02:54
*** camunoz_mtg has joined #openstack-infra02:54
*** ryanpetrello has quit IRC02:55
*** r-daneel has joined #openstack-infra02:56
*** signed8bit has joined #openstack-infra02:58
*** r-daneel has quit IRC03:00
*** signed8b_ has joined #openstack-infra03:03
clarkbfungi still about? looks like we need to bounce snmpd on git-fe*03:07
*** signed8bit has quit IRC03:07
clarkbI can do that when I get home03:07
*** ryanpetrello has joined #openstack-infra03:10
*** harlowja is now known as harlowja_away03:12
*** ryanpetrello has quit IRC03:14
*** r-daneel has joined #openstack-infra03:15
*** patrickeast has quit IRC03:19
*** armax has joined #openstack-infra03:20
*** mestery has joined #openstack-infra03:21
*** salv-orlando has joined #openstack-infra03:23
*** zz_dimtruck is now known as dimtruck03:23
*** salv-orlando has quit IRC03:23
*** salv-orl_ has joined #openstack-infra03:23
*** fandi has joined #openstack-infra03:24
*** marun has joined #openstack-infra03:26
*** jamielennox|away is now known as jamielennox03:26
*** changbl has joined #openstack-infra03:28
*** amotoki has joined #openstack-infra03:28
*** signed8b_ is now known as signed8bit03:31
*** camunoz_mtg has quit IRC03:34
*** oomichi_ has joined #openstack-infra03:35
*** dimtruck is now known as zz_dimtruck03:45
*** camunoz_mtg has joined #openstack-infra03:50
*** r-daneel has quit IRC03:50
*** ivar-laz_ has joined #openstack-infra03:55
*** Sukhdev has joined #openstack-infra03:55
*** ddieterl_ has quit IRC03:56
*** _nadya_ has joined #openstack-infra03:57
*** ivar-lazzaro has quit IRC03:58
*** talluri has joined #openstack-infra04:00
*** ivar-laz_ has quit IRC04:00
*** rkukura has joined #openstack-infra04:00
*** markmcclain has joined #openstack-infra04:02
*** asettle-doctor is now known as asettle04:02
*** pc_m has joined #openstack-infra04:03
*** pc_m has quit IRC04:04
*** pc_m has joined #openstack-infra04:05
*** _nadya_ has quit IRC04:09
*** talluri has quit IRC04:10
*** talluri has joined #openstack-infra04:11
*** signed8bit has quit IRC04:15
*** signed8bit has joined #openstack-infra04:16
*** mrmartin has joined #openstack-infra04:17
*** amotoki has quit IRC04:17
openstackgerritMerged openstack-infra/publications: Tell people to clone template branch using git://
*** signed8bit has quit IRC04:25
*** ddieterly has joined #openstack-infra04:29
*** dannywilson has joined #openstack-infra04:32
*** ddieterly has quit IRC04:33
*** amotoki has joined #openstack-infra04:34
*** achanda has joined #openstack-infra04:35
*** dannywilson has quit IRC04:36
*** Ryan_Lane has quit IRC04:37
*** Longgeek has joined #openstack-infra04:37
*** Longgeek has quit IRC04:42
*** otter768 has joined #openstack-infra04:50
*** fifieldt has joined #openstack-infra04:53
*** otter768 has quit IRC04:54
*** nikil22_ has joined #openstack-infra04:56
*** Sukhdev has quit IRC04:59
*** achanda has quit IRC04:59
*** achanda has joined #openstack-infra05:01
*** achanda has quit IRC05:05
*** markmcclain has quit IRC05:06
*** mestery has quit IRC05:06
*** harlowja_at_home has joined #openstack-infra05:06
*** armax has quit IRC05:06
*** markmcclain has joined #openstack-infra05:06
*** yfried|afk is now known as yfried_05:07
*** hdd has joined #openstack-infra05:07
*** wenlock has joined #openstack-infra05:07
*** ddieterly has joined #openstack-infra05:08
*** mrmartin has quit IRC05:11
*** markmcclain has quit IRC05:11
*** koolhead17 has joined #openstack-infra05:11
*** marun has quit IRC05:12
*** ddieterly has quit IRC05:13
*** marun has joined #openstack-infra05:13
*** radez is now known as radez_g0n305:15
*** craigbr has joined #openstack-infra05:15
*** mmaglana has joined #openstack-infra05:16
*** craigbr has quit IRC05:18
*** achanda has joined #openstack-infra05:19
openstackgerritJoshua Harlow proposed openstack/requirements: Bump up oslo.utils to 1.1.0
*** camunoz_mtg has quit IRC05:25
*** achanda has quit IRC05:26
*** wenlock has quit IRC05:28
*** harlowja_away is now known as harlowja05:28
*** harlowja is now known as harlowja_away05:29
*** mrmartin has joined #openstack-infra05:33
*** sabeen has quit IRC05:35
*** Longgeek has joined #openstack-infra05:38
*** rushiagr_away is now known as rushiagr05:40
jyusosweston: Thank you for your config file.It solved my problem.:)05:41
*** armax has joined #openstack-infra05:42
*** Longgeek has quit IRC05:43
*** Sukhdev has joined #openstack-infra05:46
*** KanagarajM has joined #openstack-infra05:46
*** yfried_ has quit IRC05:47
*** viglesias has quit IRC05:48
*** viglesias has joined #openstack-infra05:48
*** mrmartin has quit IRC05:49
*** talluri has quit IRC05:50
*** sarob has joined #openstack-infra05:50
*** talluri has joined #openstack-infra05:50
*** achanda has joined #openstack-infra05:52
*** wenlock has joined #openstack-infra05:52
*** gokrokve_ has joined #openstack-infra05:53
*** esker has joined #openstack-infra05:55
*** achanda has quit IRC05:55
*** achanda has joined #openstack-infra05:56
*** gokrokve has quit IRC05:57
*** ddieterly has joined #openstack-infra05:57
*** gokrokve_ has quit IRC05:58
*** achanda has quit IRC06:00
*** ddieterly has quit IRC06:02
*** hdd has quit IRC06:02
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules
*** camunoz has joined #openstack-infra06:08
*** gyee has joined #openstack-infra06:08
*** koolhead17 has quit IRC06:16
*** gokrokve has joined #openstack-infra06:23
*** gokrokve has quit IRC06:25
*** sarob has quit IRC06:25
*** gokrokve has joined #openstack-infra06:25
*** ddieterly has joined #openstack-infra06:27
*** salv-orl_ has quit IRC06:29
*** gokrokve has quit IRC06:30
*** rlandy has joined #openstack-infra06:31
*** ddieterly has quit IRC06:31
*** gokrokve has joined #openstack-infra06:32
*** marun has quit IRC06:32
*** salv-orlando has joined #openstack-infra06:33
*** gokrokve_ has joined #openstack-infra06:34
*** krtaylor has quit IRC06:36
*** gokrokve has quit IRC06:36
*** harlowja_at_home has quit IRC06:37
*** krtaylor has joined #openstack-infra06:38
*** salv-orlando has quit IRC06:38
*** gokrokve_ has quit IRC06:39
*** yfried_ has joined #openstack-infra06:39
*** reed has quit IRC06:41
*** salv-orlando has joined #openstack-infra06:45
*** Longgeek has joined #openstack-infra06:45
*** teran has joined #openstack-infra06:46
*** oomichi_ has quit IRC06:46
*** armax has quit IRC06:46
*** teran_ has quit IRC06:49
*** otter768 has joined #openstack-infra06:51
*** mrunge has joined #openstack-infra06:51
*** belmoreira has joined #openstack-infra06:55
*** otter768 has quit IRC06:55
*** esker has quit IRC06:56
*** esker has joined #openstack-infra06:56
*** esker has quit IRC07:01
*** jgallard_ has joined #openstack-infra07:02
*** yfried_ is now known as yfried|afk07:05
*** yfried|afk is now known as yfried_07:06
*** e0ne has joined #openstack-infra07:08
*** _nadya_ has joined #openstack-infra07:09
*** _nadya_ has quit IRC07:14
openstackgerritMerged openstack-infra/project-config: Add networking-arisa project to StackForge
*** Sukhdev has quit IRC07:16
*** yfried_ is now known as yfried|afk07:19
*** zz_avozza is now known as avozza07:24
*** k4n0 has joined #openstack-infra07:25
*** ddieterly has joined #openstack-infra07:27
*** koolhead17 has joined #openstack-infra07:28
*** yfried|afk is now known as yfried_07:29
*** salv-orlando has quit IRC07:31
*** ddieterly has quit IRC07:32
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Fix indention in gerrit.config.erb
*** koolhead17 has quit IRC07:33
*** masayukig has quit IRC07:43
*** masayukig has joined #openstack-infra07:43
*** teran_ has joined #openstack-infra07:45
*** teran has quit IRC07:48
tchaypoSorry for nagging about this, but seems like it’s ready to land. Is there anything I need to do so it can get an extra +2?07:51
*** mmaglana has quit IRC07:52
*** dtantsur|afk is now known as dtantsur07:55
*** fifieldt has quit IRC07:56
*** ddieterly has joined #openstack-infra07:57
*** jerryz has quit IRC07:58
*** gyee has quit IRC07:58
*** tnovacik has joined #openstack-infra07:59
*** markus_z has joined #openstack-infra08:00
openstackgerritJames Polley proposed openstack-infra/zuul: Remove "properly" from error message for clarity
*** ddieterly has quit IRC08:01
*** luqas has joined #openstack-infra08:02
*** teran_ has quit IRC08:02
*** arxcruz has joined #openstack-infra08:05
*** jpich has joined #openstack-infra08:05
*** jerryz has joined #openstack-infra08:07
*** wenlock has quit IRC08:10
*** ala_ has joined #openstack-infra08:12
*** [HeOS] has quit IRC08:13
*** avozza is now known as zz_avozza08:16
*** jistr has joined #openstack-infra08:17
*** ociuhandu has joined #openstack-infra08:20
*** amuller has joined #openstack-infra08:23
*** jerryz_ has joined #openstack-infra08:24
*** stevemar has quit IRC08:25
*** oomichi has joined #openstack-infra08:26
*** salv-orlando has joined #openstack-infra08:31
*** gokrokve has joined #openstack-infra08:32
*** jcoufal has joined #openstack-infra08:32
*** e0ne has quit IRC08:36
*** salv-orlando has quit IRC08:37
*** koolhead17 has joined #openstack-infra08:37
*** gokrokve has quit IRC08:37
*** mpaolino has joined #openstack-infra08:38
*** jlibosva has joined #openstack-infra08:39
*** zz_avozza is now known as avozza08:39
*** nikil22_ has quit IRC08:40
*** pblaho has joined #openstack-infra08:40
*** oomichi has quit IRC08:40
*** MaxV has joined #openstack-infra08:41
*** ociuhandu has quit IRC08:44
*** unicell has joined #openstack-infra08:47
*** andreaf has quit IRC08:47
*** andreaf has joined #openstack-infra08:48
*** otter768 has joined #openstack-infra08:51
*** yfried_ is now known as yfried|afk08:56
ekarlso-ahmmm, is oslo.context not yet in requirements?08:56
*** otter768 has quit IRC08:56
AJaegerekarlso-, it's in as far as I can see08:57
ekarlso-kk, just gotta refresh then here :)=08:57
*** ociuhandu has joined #openstack-infra08:59
omrimDevstack was broken due to services split..Is someone know what is the status?09:02
openstackgerrityolanda.robla proposed openstack-infra/infra-specs: Add nodepool disk image builder testing spec.
*** nelsnelson has joined #openstack-infra09:03
*** yfried|afk is now known as yfried_09:09
*** ociuhandu has joined #openstack-infra09:10
*** jedimike has joined #openstack-infra09:13
*** ociuhandu has quit IRC09:14
*** derekh has joined #openstack-infra09:17
*** yfried_ is now known as yfried|afk09:20
*** Hal_ has joined #openstack-infra09:23
*** mrunge has quit IRC09:24
*** [HeOS] has joined #openstack-infra09:25
*** sabari is now known as zz_sabari09:26
*** sarob has joined #openstack-infra09:26
*** mrunge has joined #openstack-infra09:28
*** sarob has quit IRC09:31
*** gokrokve has joined #openstack-infra09:32
*** berendt has joined #openstack-infra09:33
*** gokrokve has quit IRC09:37
*** mfink has quit IRC09:39
*** yfried|afk is now known as yfried_09:40
*** e0ne has joined #openstack-infra09:41
*** salv-orlando has joined #openstack-infra09:45
*** koolhead17 has quit IRC09:47
*** teran has joined #openstack-infra09:47
*** salv-orlando has quit IRC09:50
*** mrunge has quit IRC09:50
openstackgerritAaron Rosen proposed openstack-infra/nodepool: Update Readme config -> system-config repo
*** ihrachyshka has joined #openstack-infra09:51
*** jerryz__ has joined #openstack-infra09:54
*** jerryz__ has quit IRC09:54
*** jerryz_ has quit IRC09:54
*** mrunge has joined #openstack-infra09:56
*** mrmartin has joined #openstack-infra10:04
*** pelix has joined #openstack-infra10:07
*** Masahiro has quit IRC10:10
*** berendt has quit IRC10:11
*** Masahiro has joined #openstack-infra10:11
openstackgerritMarton Kiss proposed openstack-infra/system-config: Fix puppet module dependency order
*** hashar has joined #openstack-infra10:14
*** yfried_ is now known as yfried|afk10:14
*** nfedotov has joined #openstack-infra10:15
*** skolekonov has joined #openstack-infra10:16
*** dizquierdo has joined #openstack-infra10:17
*** jcoufal has quit IRC10:18
*** unicell1 has joined #openstack-infra10:20
*** unicell has quit IRC10:22
*** ociuhandu has joined #openstack-infra10:28
*** ihrachyshka_ has joined #openstack-infra10:28
*** ihrachyshka has quit IRC10:30
openstackgerritDerek Higgins proposed openstack-infra/system-config: Add hp1 back running tripleo ci
*** gokrokve has joined #openstack-infra10:32
*** Masahiro has quit IRC10:33
*** yfried|afk is now known as yfried_10:34
*** gokrokve has quit IRC10:36
*** _shaps_ has joined #openstack-infra10:37
*** jp_at_hp has joined #openstack-infra10:40
*** dizquierdo has quit IRC10:41
*** ociuhandu has quit IRC10:44
*** yaguang has quit IRC10:44
*** yaguang has joined #openstack-infra10:44
*** ldnunes has joined #openstack-infra10:45
*** pblaho has quit IRC10:48
*** ociuhandu has joined #openstack-infra10:49
*** amotoki has quit IRC10:51
*** otter768 has joined #openstack-infra10:52
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a tripleo juno overcloud jobs
*** ociuhandu has quit IRC10:54
*** yfried_ is now known as yfried|afk10:55
*** dizquierdo has joined #openstack-infra10:55
*** jgallard_ has quit IRC10:56
*** otter768 has quit IRC10:57
*** ZZelle has quit IRC10:57
*** ZZelle has joined #openstack-infra10:57
openstackgerritDerek Higgins proposed openstack-infra/project-config: Add a tripleo juno overcloud jobs
*** talluri has quit IRC11:06
*** jkraj has joined #openstack-infra11:07
*** talluri has joined #openstack-infra11:08
*** teran has quit IRC11:09
openstackgerritMarton Kiss proposed openstack-infra/system-config: Enable SSL for
*** jule has quit IRC11:10
*** teran has joined #openstack-infra11:11
*** jule has joined #openstack-infra11:11
*** hashar has quit IRC11:12
*** cnesa has joined #openstack-infra11:13
*** yfried|afk is now known as yfried_11:15
*** yamamoto_ has quit IRC11:16
*** jkraj has quit IRC11:17
*** mika has quit IRC11:17
*** jp_at_hp has quit IRC11:17
*** jp_at_hp1 has joined #openstack-infra11:17
*** jkraj has joined #openstack-infra11:17
*** mika has joined #openstack-infra11:19
*** cnesa has quit IRC11:20
*** jp_at_hp1 has quit IRC11:21
*** cnesa has joined #openstack-infra11:25
*** yfried_ is now known as yfried|afk11:28
*** yaguang has quit IRC11:29
*** salv-orlando has joined #openstack-infra11:32
*** gokrokve has joined #openstack-infra11:32
*** aysyd has joined #openstack-infra11:33
*** alexpilotti has joined #openstack-infra11:33
*** andreykurilin_ has joined #openstack-infra11:34
*** Masahiro has joined #openstack-infra11:34
*** cnesa2 has joined #openstack-infra11:34
*** cnesa has quit IRC11:35
*** salv-orlando has quit IRC11:36
*** berendt has joined #openstack-infra11:36
*** gokrokve has quit IRC11:37
*** kmartin has quit IRC11:37
*** jamielennox is now known as jamielennox|away11:38
*** Masahiro has quit IRC11:38
*** yfried|afk is now known as yfried_11:39
*** asselin has quit IRC11:43
*** _nadya_ has joined #openstack-infra11:45
*** vigneshvar has joined #openstack-infra11:46
*** jcoufal has joined #openstack-infra11:46
*** cdent has joined #openstack-infra11:47
*** adalbas has joined #openstack-infra11:49
*** isviridov_away is now known as isviridov11:50
*** fandi has quit IRC11:54
*** doug-fish has joined #openstack-infra11:54
*** talluri has quit IRC11:55
*** achanda has joined #openstack-infra11:59
*** mpaolino has quit IRC12:00
*** e0ne has quit IRC12:00
*** ddieterly has joined #openstack-infra12:02
*** bradjones has joined #openstack-infra12:03
*** zhiwei has quit IRC12:08
*** david-lyle_afk is now known as david-lyle12:09
*** david-lyle is now known as david-lyle_afk12:17
nikil22hi i have installed the jenkins master with zuul  . Now i see that as a zuul user i am not able to get the gerrit triggers . my gerrit_key file is in "/var/lib/jenkins/.ssh/id_rsa" . Any idea how to solve this12:17
*** david-lyle_afk is now known as david-lyle12:17
nikil22i have tried using the blog :
*** jistr has quit IRC12:20
*** david-lyle is now known as david-lyle_afk12:23
*** david-lyle_afk is now known as david-lyle12:23
*** yamamoto has joined #openstack-infra12:23
*** nfedotov has left #openstack-infra12:24
nikil22could some one should me the proper docs for setting up the Third party CI system. The standard docs is not having much information12:26
*** jp_at_hp has joined #openstack-infra12:27
*** Murad has joined #openstack-infra12:29
*** amotoki has joined #openstack-infra12:29
*** mrunge is now known as mrunge_brb12:31
*** david-lyle is now known as david-lyle_afk12:31
*** david-lyle_afk is now known as david-lyle12:31
*** gokrokve has joined #openstack-infra12:32
*** ryanpetrello has joined #openstack-infra12:36
*** gokrokve has quit IRC12:37
*** ociuhandu has joined #openstack-infra12:37
*** teran has quit IRC12:37
*** ctlaugh has quit IRC12:38
*** ctlaugh has joined #openstack-infra12:38
*** jistr has joined #openstack-infra12:40
*** Masahiro has joined #openstack-infra12:46
*** ociuhandu_ has joined #openstack-infra12:48
*** ociuhandu has quit IRC12:48
*** ociuhandu_ is now known as ociuhandu12:48
*** Masahiro has quit IRC12:51
*** luqas has quit IRC12:51
*** e0ne has joined #openstack-infra12:52
*** ctlaugh has quit IRC12:52
*** Hal_ has quit IRC12:52
*** otter768 has joined #openstack-infra12:53
*** dprince has joined #openstack-infra12:54
*** ctlaugh has joined #openstack-infra12:56
*** talluri has joined #openstack-infra12:56
*** otter768 has quit IRC12:58
*** yfried_ is now known as yfried|afk12:58
*** ctlaugh has quit IRC12:58
*** david-lyle is now known as david-lyle_afk13:00
*** talluri has quit IRC13:01
*** talluri has joined #openstack-infra13:02
*** kgiusti has joined #openstack-infra13:03
*** wuhg has quit IRC13:03
*** yfried|afk is now known as yfried_13:06
*** mwagner_lap has quit IRC13:10
*** ddieterly has quit IRC13:11
*** pblaho has joined #openstack-infra13:11
*** jpich has quit IRC13:12
*** dkranz has joined #openstack-infra13:13
*** garyk has joined #openstack-infra13:16
*** jpich has joined #openstack-infra13:16
garykAJaeger: hi, you have a sec for a quick question about
*** achanda has quit IRC13:17
AJaegergaryk, yes13:17
garykAJaeger: Do you know where I can read up on what to add to ensure that we get the file history (sorry I am really bad at this stuff)13:17
AJaegerIt's your call what you do - I just want you to be aware of the pros and cons of importing or not13:17
garykAJaeger: what you suggest i great. I would like to go down that path, just not exactly sure what to do to ensure that13:18
AJaegergaryk, explain all about setting up a repo.13:18
*** ctlaugh has joined #openstack-infra13:18
AJaegerThe importing is not yet good enough explained, see:13:18
garykAJaeger: brilliant! thank you13:18
garykAJaeger: thanks!13:20
AJaegergaryk, for "exporting", use git filter-branch on a copy of the neutron repo to filter out the history for only those files that are interesting for your new project13:20
*** bswartz has quit IRC13:20
AJaegerAnd then create from that a new repo, make it available and use the upstream directive to import it.13:20
garykok, thanks. i will check it out13:21
AJaegerIf you have questions, come back and ask the team. I'm AFK for an hour now...13:21
garykok, thanks13:23
*** mpaolino has joined #openstack-infra13:24
*** dkliban_afk is now known as dkliban13:26
*** andreykurilin_ has quit IRC13:27
*** vigneshvar has quit IRC13:28
*** amotoki_ has joined #openstack-infra13:28
*** _nadya_ has quit IRC13:30
*** zul has quit IRC13:30
fungiomrim: i believe there are changes landed in the past 24 hours which disable those in devstack-gate/tempest while the neutron advanced services developers get separate testing going for them13:30
*** teran has joined #openstack-infra13:32
*** mpaolino has quit IRC13:32
*** gokrokve has joined #openstack-infra13:32
funginikil22: if you look in your zuul.conf it's probably trying to use /var/lib/zuul/ssh/id_rsa to authenticate to gerrit?13:33
*** amuller_ has joined #openstack-infra13:34
*** amuller has quit IRC13:34
*** zul has joined #openstack-infra13:34
*** _nadya_ has joined #openstack-infra13:36
*** gokrokve has quit IRC13:37
*** teran_ has joined #openstack-infra13:37
*** weshay has joined #openstack-infra13:37
*** yfried_ has quit IRC13:39
*** teran has quit IRC13:40
*** Sukhdev has joined #openstack-infra13:41
nikil22fungi: yes it tries to use /var/lib/zuul/ssh/id_rsa13:41
nikil22fungi: the script already copied in that location during instllation time. But the issue is zuul doesn't have permission i think13:42
funginikil22: did you copy your /var/lib/jenkins/.ssh/id_rsa to there?13:42
nikil22zuul user13:42
funginikil22: which permissions? are you talking about your local filesystem permissions?13:42
openstackgerritMerged openstack-dev/specs-cookiecutter: Add RSS feed
*** garyk has quit IRC13:43
*** mrunge_brb has quit IRC13:43
nikil22yes local filesystem permission ; Because if i log in as different user and and use the same key with sudo then it works13:43
*** eharney has joined #openstack-infra13:43
nikil22fungi: works means like i am able to to ssh -i key_file to get gerrit stream13:43
funginikil22: all the contents of /var/lib/zuul need to be owned by your local zuul user13:44
*** mpaolino has joined #openstack-infra13:44
fungiincluding the /var/lib/zuul/ssh/id_rsa file13:45
nikil22fungi : ok i will take snapshoot and try now.13:45
*** jerryz has quit IRC13:45
*** garyk has joined #openstack-infra13:48
nikil22fungi: will chown -R zuul:zuul /var/lib/zull works? because still i get same permission error .13:49
*** _nadya_ has quit IRC13:50
*** amotoki has quit IRC13:51
*** amotoki_ is now known as amotoki13:51
*** mbacchi has joined #openstack-infra13:51
*** ddieterly has joined #openstack-infra13:51
*** MaxV has quit IRC13:52
*** xyang1 has quit IRC13:52
*** MaxV has joined #openstack-infra13:52
*** stevemar has joined #openstack-infra13:54
*** Longgeek has quit IRC13:54
funginikil22: to help you troubleshoot this, i'll need to know the actual errors you're getting, not your interpretation of what they might mean13:54
*** pblaho has quit IRC13:56
*** mwagner_lap has joined #openstack-infra13:56
mordredjroll: while I'm bugging you about certs ... has a bogus cert13:56
*** jp_at_hp has quit IRC13:57
mordredjroll: also, I know your job is not ACTUALLY to be our personal support contact - so feel free to tell us to go away13:57
*** mestery has joined #openstack-infra13:58
*** jcoufal_ has joined #openstack-infra13:59
*** nfedotov has joined #openstack-infra13:59
*** oomichi has joined #openstack-infra14:00
*** sressot has quit IRC14:00
*** jp_at_hp has joined #openstack-infra14:01
*** Longgeek has joined #openstack-infra14:01
*** jcoufal has quit IRC14:01
*** sressot has joined #openstack-infra14:03
*** talluri has quit IRC14:05
*** doug-fish1 has joined #openstack-infra14:06
*** doug-fish has quit IRC14:06
*** bswartz has joined #openstack-infra14:07
*** luqas has joined #openstack-infra14:08
ttxfungi: let me know when we can work on stable maint apocalypse14:08
*** berendt has quit IRC14:08
sdaguettx: what apocalypsed?14:08
*** viktors has joined #openstack-infra14:09
*** eharney has quit IRC14:09
*** berendt has joined #openstack-infra14:09
*** berendt has quit IRC14:09
ttxsdague: refactoring of openstack-stable-maint into $PROJECT-stable-maint and stable-maint-core.14:09
sdagueoh, process appocalypse pony14:09
*** kaeso has joined #openstack-infra14:10
*** rushiagr is now known as rushiagr_away14:11
kaesosorry to disturb, who should I (privately) contact for a couple of inquiries regarding infrastructure admin/security?14:12
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add debootstrap to the needed packages for dib
openstackgerritMonty Taylor proposed openstack-infra/system-config: Make apt skip grabbing translations
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add rinse to nodepool
*** koolhead17 has joined #openstack-infra14:14
*** julim has joined #openstack-infra14:14
*** esker has joined #openstack-infra14:15
*** obondarev has joined #openstack-infra14:15
*** KanagarajM has quit IRC14:17
*** pc_m has quit IRC14:19
*** annegent_ has joined #openstack-infra14:20
*** hashar has joined #openstack-infra14:21
*** berendt has joined #openstack-infra14:21
ttxkaeso: fungi seems like the right guy with the right intersection of skills. Contact details at
ttx(fungi being Jeremy Stanley in that list)14:22
ttxbut then, any of those addresses will do.14:23
*** amuller_ has quit IRC14:23
*** amuller_ has joined #openstack-infra14:23
*** bknudson has joined #openstack-infra14:23
*** mriedem has joined #openstack-infra14:23
*** mjturek has joined #openstack-infra14:23
kaesottx: thanks, mail in progress :)14:24
*** cpowell has joined #openstack-infra14:28
*** cpowell has quit IRC14:29
*** jaypipes has joined #openstack-infra14:29
*** cpowell has joined #openstack-infra14:29
nikil22fungi: i get this error in zuul log14:31
*** ryanpetrello_ has joined #openstack-infra14:31
*** mika has quit IRC14:31
*** armax has joined #openstack-infra14:31
*** ryanpetrello has quit IRC14:31
*** mjturek has quit IRC14:31
*** ryanpetrello_ is now known as ryanpetrello14:31
*** mjturek has joined #openstack-infra14:31
*** signed8bit has joined #openstack-infra14:32
*** mika has joined #openstack-infra14:32
*** gokrokve has joined #openstack-infra14:32
*** signed8bit has quit IRC14:33
*** amitgandhinz has joined #openstack-infra14:33
*** signed8bit has joined #openstack-infra14:34
*** Masahiro has joined #openstack-infra14:35
fungikaeso: yep, e-mail me at that address, encrypted to my openpgp key if you like14:36
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules
*** gokrokve has quit IRC14:37
*** mestery has quit IRC14:38
ttxclarkb: since you reviewed the previous patchset and had reservations on it, would appreciate a new review of before I approve it14:39
*** Masahiro has quit IRC14:39
*** dims has joined #openstack-infra14:39
jrollmordred: blah14:40
*** amuller_ has quit IRC14:41
*** amuller__ has joined #openstack-infra14:41
mordredjroll: :)14:42
mordredjroll: also, good morning14:42
*** zz_johnthetubagu is now known as johnthetubaguy14:42
openstackgerritMerged openstack-infra/release-tools: spec2bp v2 -- support in-review specs
*** koolhead17 has quit IRC14:43
funginikil22: the "connection refused" you see there is at the network layer. looks like zuul is receiving a tcp/rst or icmp rejection packet of some kind attempting to reach tcp port 29418 on review.openstack.org14:43
*** koolhead17 has joined #openstack-infra14:43
jrollmorning mordred :)14:43
* jroll needs more coffee for this14:44
*** berendt has quit IRC14:44
*** radez_g0n3 is now known as radez14:44
*** resker has joined #openstack-infra14:44
funginikil22: or else the paramiko library is being very opaque/misleading with its error messages14:45
*** tonytan4ever has joined #openstack-infra14:45
nikil22fungi: may be this is due to i have used ubuntu 14.04?14:46
nikil22do i need to use ubuntu 12.0414:46
funginikil22: can you su to the zuul user on that machine and run 'ssh -i /var/lib/zuul/ssh/id_rsa -p 29418 gerrit stream-events'14:46
nikil22fungi: let me check14:46
funginikil22: replacing yourusername with the gerrit username listed in your zuul.conf obviously14:46
*** esker has quit IRC14:47
*** koolhead17 has quit IRC14:48
*** rushiagr_away is now known as rushiagr14:48
nikil22fungi: yes the events is successfull14:48
nikil22as zuul user14:49
funginikil22: did it prompt you first to accept an unrecognized ssh host key?14:49
nikil22let me restart the zuul service once14:49
nikil22fungi: yes previously it asked14:49
nikil22fungi: this time it did not ask14:50
*** hdd has joined #openstack-infra14:50
fungioh, okay, so that wasn't it either (though usually when that's the problem it fails with a different error/exception)14:50
*** ociuhandu has quit IRC14:50
*** dmsimard_away is now known as dmsimard14:51
nikil22fungi: i have restarted the service still it gives me same error. But now as a zuul user i can confirm that it can communicate to stream-events as zuul user14:51
nikil22fungi: the command you gave works .14:52
funginikil22: and the [gerrit] section of your zuul.conf has and doesn't specify a port=14:52
*** talluri has joined #openstack-infra14:52
jrollmordred: turns out is a CNAME to a cloud files container, the ssl hostname for it is https://f97f271e9fde8f434e2a-fa73b8821af96cee574a861a54d04230.ssl.cf3.rackcdn.com14:52
jrollmordred: though I'm told that ipxe doesn't play nice with https14:52
fungijroll: oh! that container. i'd know it on sight ;)14:52
fungiit has a very attractive uuid14:53
*** ociuhandu has joined #openstack-infra14:53
clarkbttx that script echos the command to run but switches source branch before you can run it. Then deletes the source branch. Is that ok for your workflow14:53
nikil22fungi: yes the and doesn't specify any port.
fungijroll: anyway, that container's ssl cert ideally needs a serverAltName for any of its cnames (or a wildcard match or something)14:54
*** pblaho has joined #openstack-infra14:54
mordredjroll: well, I care little about ipxe - I was more trying to build images that contain nova-agent securely14:54
mordredjroll: is there another place to download nova-agent from that doesn't have an invalid cert?14:54
ttxclarkb: hmm, let me think about that a bit14:54
*** otter768 has joined #openstack-infra14:54
*** arxcruz has quit IRC14:54
*** doug-fish1 has quit IRC14:54
* mordred grosk why might be fine in this situation14:54
*** Murad has quit IRC14:54
*** oomichi has quit IRC14:55
*** doug-fish has joined #openstack-infra14:55
ttxclarkb: I don't really use that script myself, so I have to try to understand how to use it. It should probably patch README anyway14:55
jrollmordred: ah, you can just use whatever you were doing, just s/
mordredjroll: and
mordredjroll: can I assume that that swift container will never change?14:56
clarkbttx I think it should either run git review or not delete merge-branch but not sure yet14:56
ttxclarkb: that echo line might be a debug leftover14:56
jrollmordred: never is a fun word, I would think yes in general14:56
ttxespecially with the commented line just above14:56
mordredjroll: :)14:56
mordredjroll: thanks!14:56
*** arxcruz has joined #openstack-infra14:57
ttxclarkb: I'll -114:57
clarkbttx ok14:57
*** yfried_ has joined #openstack-infra14:57
jrollmordred: np, enjoy14:57
ttxclarkb: thx!14:58
*** otter768 has quit IRC14:58
*** dkranz has quit IRC14:59
*** obondarev has quit IRC15:00
annegent_how do I determine merge rights on any given repo? jeblair or mordred or ttx will know15:01
jrollmordred: btw, this would solve the problem:
ttxannegent_: you can look up the Gerrit ACL15:01
fungiannegent_: you have to look at the acl for that project in gerrit and see what group has workflow +1 permirrion15:01
annegent_ttx: is that in a repo or in Settings?15:01
annegent_fungi: "in gerrit" means in a web browser15:02
*** resker has quit IRC15:02
fungiannegent_: there are ways to determine it programmatically. i have a script15:02
fungiannegent_: but yes, web browser will also work15:02
annegent_fungi: oh ttx's URL works15:02
*** amuller__ has quit IRC15:02
annegent_fungi: ok and yeah I'll also need to know how to propose changes to it?15:02
ttxannegent_: gerrit/acls files in openstack-infra/project-config15:03
fungiannegent_: you need to change the group in the acl, or the members of the group?15:03
mordredjroll: nice15:03
fungiannegent_: group membership can be edited in that webui, but to change the acl itself (for example to switch it to a different group) you need a change like ttx describes15:04
annegent_fungi: looks like I need to change members of the group, possibly15:04
ttxfungi: got my ping from earlier re: stable-maint refactoring ?15:04
annegent_okay, thanks ttx and fungi!15:04
fungiannegent_: just click on the hyperlinked group name you see and if you're a member of the group (as it's likely self-owned) you can add/remove members15:05
*** dims has quit IRC15:05
ttxfungi: we can do it now before meeting, otherwise will have to wait until tomorrow15:05
funginikil22: i'm not seeing any obvious errors, and i see successful ssh connections from your account in the gerrit logs15:05
*** dims has joined #openstack-infra15:05
*** zz_dimtruck is now known as dimtruck15:05
fungittx: we can do it now. let me see where i missed the ping15:05
*** amuller has joined #openstack-infra15:06
fungioh, there it is15:06
*** oomichi has joined #openstack-infra15:06
fungittx: okay, so do you have a list of the affected projects?15:06
*** xyang0 has joined #openstack-infra15:06
ttxfungi: I think I should be able to propose the change if you give me an example (say for nova)15:06
fungittx: ahh, right-o. just a sec15:07
*** achanda has joined #openstack-infra15:07
ttxfungi: but no idea what triggers the creation of the groups themselves15:07
ttxand how to specify who owns them15:07
fungittx: it's automatic, but then i or someone has to seed the initial member(s)15:07
fungittx: and ownership will be a manual change i'll have to make in gerrit for them15:07
ttxok, so let's do it this wey, you can give me an example, I can prepare the change to cover all projects we cover..., and then when approved you can add base members and I'll add the other ones.15:08
fungittx: if you look at the openstack/manila acl, the [access "refs/heads/stable/*"] section is what you need15:08
fungittx: but instead of manila-core it would be nova-stable or something15:08
*** bswartz has quit IRC15:09
nikil22fungi: could you tell me that the successfull connection is keep on coming. Because the successfull connection might be the one which i tried manually15:09
funginikil22: yes, is zuul retrying to connect over and over? i'll check the log and see what the most recent connection was15:09
ttxfungi: openstack-stable-maint generic control over stable branches must appear somewhere too, need to switch it to stable-maint-core and inherit it15:09
*** salv-orlando has joined #openstack-infra15:09
fungittx: yep, i can add it as an inlcuded group in each of the project-specific groups15:10
ttxAh, that would work15:10
kaesofungi: oh sorry, had to go afk, can I privmsg you a sec before writing a longer mail?15:10
*** Longgeek has quit IRC15:10
fungikaeso: sure15:10
*** dims has quit IRC15:10
ttxfungi: ok, let me try this15:10
funginikil22: last connection from ib-ipam-ci at was 14:48:32 utc, about 23 minutes ago15:11
ttxfungi: any idea where openstack-stable-maint is defined for, say, nova ? It's not in the nova acl15:12
*** Hal_ has joined #openstack-infra15:12
*** Hal_ has quit IRC15:12
ttxfungi: or should I just overwrite it same style as in manila acl15:12
*** dkranz has joined #openstack-infra15:12
ttxI think we inherit some default from somewhere here15:12
ttxbut can't find it15:12
nikil22fungi: then i guess that was the manuall one. Because i see that zuul is keep on trying to pull and i see these errors15:13
ttxonly designate seems to explicitely mention openstack-stable-maint. Default config somewhere ?15:13
*** yamamoto has quit IRC15:13
*** esker has joined #openstack-infra15:13
*** oomichi has quit IRC15:14
*** mestery has joined #openstack-infra15:14
*** Longgeek has joined #openstack-infra15:14
clarkbttx in all-projects which isnot managed by puppet15:14
*** fandi has joined #openstack-infra15:14
*** armax has quit IRC15:15
clarkbyou can see those acls in gerrit directly15:15
ttxclarkb: hmm. maybe simpler to make stable-maint-core part of the explicitly-declared group then, rather than rely on inheritance15:15
ttxwill ensure transition15:15
*** eharney has joined #openstack-infra15:16
ttxok, let me see what I can come up with.15:16
*** lttrl has joined #openstack-infra15:19
*** sabeen has joined #openstack-infra15:20
*** salv-orlando has quit IRC15:20
*** rkukura_ has joined #openstack-infra15:21
*** fandi has quit IRC15:22
*** fandi has joined #openstack-infra15:23
fungittx: sorry, too many conversations at once. openstack-stable-maint is handled in the all-projects global act inherited by every project right now. we would lift that once we get the per-project groups added with that group included in each of them15:23
fungioh, clarkb answered anyway15:23
*** r-daneel has joined #openstack-infra15:24
*** dustins has joined #openstack-infra15:24
*** rkukura has quit IRC15:24
*** rkukura_ is now known as rkukura15:24
*** vdo has joined #openstack-infra15:25
derekhIf anybody has a chance could I get some eyes on adding hp1 back running tripleo ci -
*** _nadya_ has joined #openstack-infra15:25
*** _nadya_ has quit IRC15:26
*** talluri has quit IRC15:26
*** emagana has joined #openstack-infra15:26
*** mrmartin has quit IRC15:27
*** _nadya_ has joined #openstack-infra15:27
*** patrickeast has joined #openstack-infra15:28
funginikil22: yeah, so something is blocking those egress connections presumably (or is your zuul maybe using a different zuul.conf than the one you think it is?)15:28
fungiderekh: i don't think we explicitly removed it, but i'll check15:28
*** zz_jgrimm is now known as jgrimm15:29
derekhfungi: thanks, we set the max-servers: at 0 so it wasn't being used, this increases to 9015:30
*** obondarev has joined #openstack-infra15:30
*** gokrokve has joined #openstack-infra15:30
*** esker has quit IRC15:30
*** pc_m has joined #openstack-infra15:30
derekhfungi: note this is a new cloud15:30
fungiderekh: max-servers isn't 0 it's 60 at the moment, but yes we seem to have removed it from the labels15:31
SergeyLukjanovgood morning folks15:31
*** esker has joined #openstack-infra15:31
fungiderekh: aha, in hp115:31
SergeyLukjanovfungi, I just saw that switch to the self-controlled groups for 3rd party ci has not been applied to all projects, for example it's not applied for sahara :)15:32
*** teran_ has quit IRC15:32
fungiderekh: sorry, i misread and thought you said rh115:32
fungimoar coffee15:32
SergeyLukjanovfungi, should I try to re-run jjb on gerrit?15:32
derekhfungi: no prob15:32
*** arxcruz has quit IRC15:33
garykdhellmann: hi, any chance you can please look at
fungiSergeyLukjanov: i think clarkb queried gerrit to find which ci accounts were leaving verify votes and only added groups for those which had accounts voting on them15:33
clarkbSergeyLukjanov: it should be applied to sahara15:33
SergeyLukjanovfungi, clarkb, oh, I think I've used bad wording15:33
dhellmanngaryk: +2a15:33
SergeyLukjanovfungi, clarkb, in there is a rule "label-Verified = -1..+1 group sahara-ci"15:33
nikil22fungi: i am sure my iptables is not blocking any ports. Let me debug more and confirm that zuul uses proper conf file15:34
clarkboh right sahara didn't have anything vote on it15:34
garykdhellmann: thanks!15:34
clarkbSergeyLukjanov: oh I see now15:34
SergeyLukjanovfungi, clarkb, but there is no corresponding permission in gerrit -,access15:34
*** julim has quit IRC15:34
*** marcusvrn1 has joined #openstack-infra15:34
*** marcusvrn has quit IRC15:34
fungiacl didn't get applied? or is that a stale acl file and sahara uses a different one overridden in projects.yaml?15:34
SergeyLukjanovclarkb, looks like ACL file is just not applied15:34
SergeyLukjanovfungi, yeah15:35
clarkbya so I think you want to run `manage-projects -d openstack/sahara`15:35
*** rkukura has quit IRC15:35
*** hdd has quit IRC15:35
SergeyLukjanovclarkb, on gerrit node?15:35
openstackgerritThierry Carrez proposed openstack-infra/project-config: Introduce project-specific stable-maint teams
ttxfungi: ^15:35
sdagueclarkb: how much have you played with the fixtures.MonkeyPatch?15:35
*** Sukhdev has quit IRC15:35
SergeyLukjanovttx, yay!15:35
clarkbSergeyLukjanov: yup, if you look in roots histor you should see similar commands15:35
clarkbsdague: I think so15:36
*** julim has joined #openstack-infra15:36
SergeyLukjanovclarkb, thank you, will try15:36
sdaguebecause I found a case where it didn't seem to work at all like it was expected15:36
fungiclarkb: SergeyLukjanov: yep, that does appear to be the correct acl file, so something must still be racy on m-p group creation sometimes?15:36
*** arxcruz has joined #openstack-infra15:36
ttxSergeyLukjanov: Note that I aligned sahara, even if technically you own stable/icehouse.15:36
clarkbsdague: oh?15:36
ttxSergeyLukjanov: we'll make sure sahara-stable-maint contains the right people for you to be able to continue doing that15:36
fungittx: you're going to need the exlcusive line in those15:37
clarkbsdague: it basically monkey patches a name until the cleanup runs so you get sane monkey patching per test case15:37
*** wenlock has joined #openstack-infra15:37
SergeyLukjanovttx, I think it'll be the same list as for juno, so, it's ok, thank you15:37
sdagueyeh except, it does it through some magical imports, which might fail15:37
ttxfungi: why ? It's not on the proposed/* ACL15:37
sdaguethat it masks the failures15:37
clarkbsdague: is it importing to make sure that both things (dest and patch) are available?15:37
*** obondarev has quit IRC15:37
sdaguemaybe if I show you my example, you can help me figure out why it doesn't work15:38
clarkbsdague: sure15:38
fungittx: without that, it's still overridden from all-projects15:38
fungittx: also, when we remove the entry from all-projects, it will be necessary to prevent inheritence from higher-level ref matches15:39
*** obondarev has joined #openstack-infra15:39
ttxfungi: hmm, if the net effect is just that openstack-stable-maint still ahs rights during the transition, that sounds like a feature to me. Or is there more ?15:39
*** marcusvrn1 has quit IRC15:39
ttxoh, hm15:39
*** marcusvrn has joined #openstack-infra15:39
fungittx: the way gerrit's acls work by default is that permissions on refs/heads/* are cumulative with those on refs/heads/stable/*15:40
SergeyLukjanovclarkb, "invalid project configuration"15:40
fungittx: exclusiveGroupPermissions is the way around inheriting15:40
*** marcusvrn1 has joined #openstack-infra15:41
ttxok, that means we'll have to populate groups soon afetr we push the change15:41
clarkbSergeyLukjanov: is there more log involved? would be good to see where that happens15:41
* ttx adds exclusivity lines15:41
SergeyLukjanovclarkb, it's on the push to config stage15:41
fungittx: yep, i plan to do it immediately after that's merged and applied via manage-projects15:41
SergeyLukjanovclarkb, " ! [remote rejected] HEAD -> refs/meta/config (invalid project configuration)15:42
clarkbsdague: looking at the source that is what it appeas to be trying to do. It is making sure the thing to be monkey patched is already imported15:42
*** marcusvrn has quit IRC15:42
clarkbSergeyLukjanov: huh, there must be something wrong with the file then which is odd because saharaclient was fine15:43
SergeyLukjanovclarkb, saharaclient uses another ACL file15:43
SergeyLukjanovclarkb, I need to go to the PTL Webinar, will digg into it after the webinar15:43
*** achanda has quit IRC15:44
clarkbSergeyLukjanov: oh that may explain it then15:44
*** marcusvrn has joined #openstack-infra15:44
clarkbya I will look quickly to see if I see anything15:44
*** xyang0 has quit IRC15:44
SergeyLukjanovclarkb, thx15:44
*** teran has joined #openstack-infra15:45
*** teran has quit IRC15:45
*** dmsimard is now known as dmsimard_away15:45
*** teran has joined #openstack-infra15:45
openstackgerritThierry Carrez proposed openstack-infra/project-config: Introduce project-specific stable-maint teams
*** marcusvrn1 has quit IRC15:45
clarkblooks like it would've broken way back in august15:45
clarkbor at least that was the last time we successfully updated it15:45
ttxfungi: ^should be better15:45
*** marun has joined #openstack-infra15:46
fungittx: thanks--looking now15:46
fungiderekh: i'll check your nodepool change after this15:46
derekhfungi: cool, thanks15:47
*** dmsimard_away is now known as dmsimard15:48
*** _buzz_ has joined #openstack-infra15:48
*** obondarev has quit IRC15:48
*** marcusvrn has quit IRC15:49
sdagueclarkb: so -
sdaguethat doesn't do what you think it's going to do15:50
*** craigbr has joined #openstack-infra15:50
*** _buzz has quit IRC15:51
clarkbsdague: it should attempt to import that thing, fail, then import oslo.messaging if I read this correctly15:52
clarkbsdague: I think you just need to import the thing you want first?15:52
fungittx: change lgtm, but what's the group configuration plan look like? we make openstack-stable-maint (optionally renamed) an included group in each of those and set the group owner to openstack-stable-maint as well? and let someone in openstack-stable-maint add per-project members as appropriate?15:52
sdagueclarkb: so... I have tried many invocations of this with various imports15:52
clarkbsdague: ya I think that if you just import oslo.messaging._drivers.impl_fake15:52
fungittx: or will the groups be self-owned so that per-project stable teams can add/remove members themselves?15:53
clarkbthen that will be imported so it can get monke patched15:53
fungittx: oh, you outlined it in the commit message15:53
fungittx: nevermind. works for me15:53
clarkbsdague: the problem here is monkeypatch says "I will try to import this thing for you but not scream if I can't because maybe you gave me a class method" which you did15:53
clarkbtrying to think if monkey patch could scream about issues a bit better though15:54
fungittx: if clarkb or another project-config core wants to give it a once-over, then i can make the group edits as soon as it's applied15:54
ttxfungi: yes, we create a new group15:54
sdagueclarkb: so ... no one using this fixture understands that subtlety15:54
sdaguewell maybe not no one15:54
clarkbsdague: its how monkey patching works15:54
sdaguebut in my look on how it's used in the tree, I'm pretty sure lots of people are getting this wrong15:54
clarkbsdague: if the name isn't available then you monkey patch nothing15:54
*** salv-orlando has joined #openstack-infra15:55
sdagueunder the assumption that the fixture did the right thing15:55
clarkbthis is true regardless of fixture or other tool15:55
fungittx: is the new stable-maint-core group a renamed openstack-stable-maint with the original members, or a separate/smaller group entirely?15:55
clarkbpython lets you say x = y15:55
*** achanda has joined #openstack-infra15:55
ttxfungi: you can add me as initial member/owner of stable-maint-core, then stable-maint-core as initial member/owner of $PROJECT-stable-maint groups15:55
clarkbthat is how python works15:55
sdagueclarkb: yep15:55
fungittx: okay, can do15:55
ttxfungi: smaller group / new group. Not the same role, much smaller group15:55
ttxthe idea is to have that "core" group the guardians of the policy15:56
fungittx: but also include the old openstack-stable-maint in all of the per-project stable groups?15:56
*** garyk has quit IRC15:56
ttxfungi: no, we'll delete that one.15:56
ttxah.. for transition, I guess15:56
*** bswartz has joined #openstack-infra15:56
ttxyou can initially add it, I can remove it when i'm done editing the group15:56
fungittx: okay, stable-maint-core can of course remove it as an included group at your discretion15:56
clarkbsdague: this is similar to how oslo.config breaks unittests because no one understands you have to import the thing that defines a config value before it is available to you15:56
ttxthat way it shouldn't break waiting for me to edit group15:57
fungittx: also i can set stable-maint-core as the owner of openstack-stable-maint if desired15:57
clarkbsdague: if you try to use that thing before you have imported it sadness happens15:57
sdagueclarkb: sure15:57
ttxfungi: works for me15:57
*** btran has quit IRC15:57
sdagueexcept the fixture half heartidly works some times15:57
fungittx: right now it seems to be owned by the release managers group15:57
sdaguebecause if the thing is a top level function in a namespace, it manages the import15:58
*** armax has joined #openstack-infra15:58
fungittx: do you then want release managers as the owner of stable-maint-core or make stable-maint-core self-owned?15:58
clarkbright just like oslo.config half heartidly works some times...15:58
ttxfungi: self-owned with me as initial member15:58
clarkbboth could do a better job at making it known when stuff is not imported15:58
*** belmoreira has quit IRC15:58
fungittx: got it15:58
*** wznoinsk has quit IRC15:58
sdagueclarkb: yeh16:00
*** wznoinsk has joined #openstack-infra16:00
*** hashar has quit IRC16:01
*** Sukhdev has joined #openstack-infra16:01
*** hashar has joined #openstack-infra16:01
fungittx: okay, the stable-maint-core group is set up self-owned with you as initial member and switched to be the owner of the openstack-stable-maint group. i'll make the per-project stable-maint groups owned by stable-maint-core with openstack-stable-maint as an included group as soon as 140738 lands16:01
clarkbfungi: ttx: the change looks good though the centos6 apply job failed due to haproxy not applying cleanly for some reason. I don't think that is related to this change16:02
*** achanda has quit IRC16:02
clarkblooks like puppet concat had a sad16:02
*** mfink has joined #openstack-infra16:02
*** hashar has quit IRC16:02
clarkbI am going to recheck to see if this is consistent16:02
fungittx: after that, i'll remove the old refs/heads/stable/* sections from the all-projects acl and look for other acls which can be cleaned up now that they'll no longer need to override it16:02
ttxfungi: sounds like a plan16:03
*** hashar has joined #openstack-infra16:03
* dhellmann wishes sdague and clarkb would file bugs instead of trash-talking other contributors in this channel16:03
*** emagana_ has joined #openstack-infra16:04
* dhellmann walks away for a while16:04
clarkbdhellmann: I tried16:05
clarkbdhellmann: was explicitly told by markmc ~1.5 years ago that this was a design feature/expected/wontfix16:05
*** asselin has joined #openstack-infra16:06
*** wenlock has quit IRC16:06
clarkbdhellmann: and to a degree it is. This is how python works. Unfortunately it is a thing that also trips people up constantly16:06
*** hashar has quit IRC16:06
mordredsdague, clarkb: is fixtures not moved to git yet?16:07
clarkbmordred: not that I could see16:07
*** emagana has quit IRC16:08
sdaguemordred: yes it is16:09
mordredseems like it it should be possible to fix the import logic there to make sure it always manages the import16:09
sdagueI'm trying to develop a test case right now16:09
clarkbpypi still says launchpad which is where I got link to above code16:09
sdagueclarkb: yeh, they moved it to github16:09
*** jpich has quit IRC16:09
sdagueI put over a PR as soon as it landed to fix a log fixture16:09
*** dtantsur is now known as dtantsur|brb16:10
sdaguemordred: yeh, mostly I needed to understand what it was or was not doing correctly, hence poking clarkb16:10
*** jistr has quit IRC16:10
*** kmartin has joined #openstack-infra16:11
clarkbI think I would prefer it to throw an exception if the thing being monkey patched is not imported16:11
clarkbthen you don't have to rely on it to always import properly, it pushes that off on you to do what you need16:11
sdagueclarkb: yeh, possibly16:12
sdagueI think this gets into the crazy logic that hacking has over finding symbols16:12
*** dangers_away is now known as dangers16:12
openstackgerritIvan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (template)
sdagueI actually just deleted a nova specific setup function that did exactly this... poorly16:13
*** achanda_ has joined #openstack-infra16:13
*** jkraj has quit IRC16:13
sdagueand turned out to not be working when it was used16:13
clarkbbut it should be relatively simple to say "does this name exist"16:13
clarkband throw an error if not16:13
*** shashankhegde has joined #openstack-infra16:13
mordred++ to throwing error16:14
*** yamamoto has joined #openstack-infra16:14
*** skolekonov has quit IRC16:17
*** e0ne_ has joined #openstack-infra16:18
*** rlandy has quit IRC16:18
*** yamamoto has quit IRC16:19
*** spzala has joined #openstack-infra16:19
*** e0ne has quit IRC16:19
sdagueyeh well the test case will be mostly the same either way16:19
sdaguehmmm... running tests in fixtures repo....16:19
sdagueoh, it's got a makefile. I can get behind that16:21
*** dannywilson has joined #openstack-infra16:23
*** gokrokve has quit IRC16:23
*** _buzz_ has quit IRC16:23
*** dannywilson has quit IRC16:23
*** dannywilson has joined #openstack-infra16:23
*** Masahiro has joined #openstack-infra16:24
sdagueso the test class that should fail, isn't16:24
*** shashankhegde has quit IRC16:25
*** resker has joined #openstack-infra16:25
*** resker has quit IRC16:26
sdagueclarkb: so - should fail in the same way, right?16:26
*** resker has joined #openstack-infra16:27
*** ryanpetrello has quit IRC16:27
*** erikmwilson has quit IRC16:28
*** ryanpetrello has joined #openstack-infra16:28
*** Masahiro has quit IRC16:28
*** erikmwilson has joined #openstack-infra16:28
*** mpaolino has quit IRC16:28
*** obondarev has joined #openstack-infra16:29
*** esker has quit IRC16:29
*** annegent_ has quit IRC16:29
*** mikedillion has joined #openstack-infra16:29
*** e0ne_ is now known as e0ne16:30
*** gokrokve has joined #openstack-infra16:30
*** dims has joined #openstack-infra16:32
*** erikmwilson is now known as Guest3055416:34
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Check requirements for taskflow
clarkbsdague: maybe the test suite imports all that stuff implicitly? but yes at first glance I would expect that to fail too16:35
*** yaguang has joined #openstack-infra16:35
*** erikmwilson has joined #openstack-infra16:36
*** dims has quit IRC16:36
*** dims has joined #openstack-infra16:36 is all it's doing16:37
*** adalbas has quit IRC16:38
dimssdague: this one helps with running locally esp with project that have not signed upto be in global req sync. can you please take a look when you get a chance?
sdaguedims: I'm not sure I understand the --local really there?16:39
sdagueclarkb: yep, you are right16:41
sdaguethe way load tests work it invalidates things16:41
*** vigneshvar has joined #openstack-infra16:41
*** erikmwilson is now known as Guest2543416:42
*** [BNC]erikmwilson has joined #openstack-infra16:42
*** ihrachyshka_ has quit IRC16:44
*** mikedillion has quit IRC16:44
*** julim has quit IRC16:45
*** shakamunyi has quit IRC16:45
*** simonmcc has quit IRC16:45
*** jlibosva has quit IRC16:46
*** ctracey has quit IRC16:46
*** markus_z has quit IRC16:46
*** zhiyan has quit IRC16:46
dimssdague: the current code looks at the last changeset, i want it to run against the full requirements.txt file16:46
*** serverascode___ has quit IRC16:47
*** ameade_ has quit IRC16:47
*** jraim has quit IRC16:48
sdaguedims: oh, gotcha16:49
sdagueclarkb: - so that's a test case which does fail16:49
*** julim has joined #openstack-infra16:50
*** nikil22 has quit IRC16:51
*** armax has quit IRC16:52
*** zz_sabari is now known as sabari16:53
openstackgerritMerged openstack/requirements: Upgrade oslo.vmware>=0.7.0
*** otter768 has joined #openstack-infra16:55
*** yaguang has quit IRC16:56
*** emagana_ has quit IRC16:56
*** mpaolino has joined #openstack-infra16:57
*** arxcruz has quit IRC16:58
*** achanda_ has quit IRC16:58
*** amuller has quit IRC17:00
*** otter768 has quit IRC17:00
*** viktors is now known as viktors|afk17:00
*** amotoki has quit IRC17:02
*** achanda has joined #openstack-infra17:02
*** garyk has joined #openstack-infra17:02
*** julim has quit IRC17:03
*** doug-fish has quit IRC17:03
*** armax has joined #openstack-infra17:03
*** armax has quit IRC17:03
*** david-lyle_afk is now known as david-lyle17:06
*** julim has joined #openstack-infra17:06
*** asselin has quit IRC17:07
*** mpaolino has quit IRC17:08
*** emagana has joined #openstack-infra17:08
*** dmsimard is now known as dmsimard_away17:09
openstackgerritDmitry Teselkin proposed openstack-infra/zuul: Whitelist for gerrit events
clarkbsc68cal: mtreinish: wanted to catch up on ipv6. Is that tempest test counting IPs the only issue so far? (want to make sure I haven't just asked you guys to fix half a million bugs to get that d-g change in)17:10
*** shayneburgess has quit IRC17:11
sc68calclarkb: it's the first issue to crop up17:11
sc68calclarkb: no guarantee it won't be the last17:11
mtreinishsc68cal: I think some of the scenario tests exploded too17:11
clarkbsc68cal: there is a way to run d-g with a specific tempest patch. Let me dig that up so that we can rerun the tests with your fix prior to it merging17:11
*** e0ne has quit IRC17:12
*** k4n0 has quit IRC17:12
*** berendt has joined #openstack-infra17:12
jeblairclarkb, fungi: i think we should try trusty nodepool again17:13
jeblairshall we do that now?17:13
*** mmaglana has joined #openstack-infra17:13
clarkbjeblair: you will find no disagreement from me17:13
clarkbjeblair: can you give me a few minutes to get caffeine and breakfast?17:14
fungijeblair: i'm in favor17:14
fungiderekh: i think we should hold off adding that other tripleo region until we have worked out whether our nodepool migration is going to stick this time17:15
*** koolhead17 has joined #openstack-infra17:15
jeblairwe may find that the world has changed around nodepool and we need to adapt.  if so, i think clarb's change to gear may help (just because we have fewer cycles available to us), though it's not a long-term solution to the fundamental problem17:15
jeblairi do have some ideas about how we could be a bit more robust without drastic changes; i'll write up a spec while we're working on this in case we need it17:16
derekhfungi: ok, any idea who long that will be? don't want to pop in asking about it prematurely.17:17
fungiderekh: probably a couple hours17:17
clarkbgah I am out of yogurt. But I have caffeine so lets do this17:18
derekhok cool, I'll probably be gone by then but will check later to see if its merged and watch to make sure its going ok17:18
clarkbthe images from last time should be new enough so we don't need to rebuild images on nodepool-dev again17:19
*** nfedotov has quit IRC17:21
swestonjyuso: you're welcome :-).  You should thank anteaya as well, she organized the meeting this week.  We never would have converged on a solution without her.17:21
clarkbjeblair: any other prep type items we might have to do based on your debugging?17:22
*** mmaglana has quit IRC17:22
clarkbjeblair: you already rebuilt pycrypto?17:22
jeblairclarkb: yep, should be gtg17:22
*** mmaglana has joined #openstack-infra17:22
*** ZZelle is now known as zzelle_17:23
*** zzelle_ is now known as ZZelle_17:23
fungiwant me to stop nodepoold on precise and start deleting ready/building nodes?17:23
clarkbso stop old nodepool, delete ready/building, start new nodepool17:23
clarkbthen delete remainder on old nodepool17:23
*** salv-orlando has quit IRC17:23
*** emagana_ has joined #openstack-infra17:23
*** johnthetubaguy is now known as zz_johnthetubagu17:23
clarkbfungi: I am ready when you are17:23
*** salv-orlando has joined #openstack-infra17:24
fungi~300 building+ready nodes deleting now17:26
*** salv-orlando has quit IRC17:26
*** jaypipes has quit IRC17:26
*** emagana has quit IRC17:27
openstackgerritMerged openstack/requirements: Bump the minimum required version of retrying
openstackgerritMerged openstack/requirements: Update xstatic-font-awesome to 4.2.0
openstackgerritMerged openstack/requirements: Add nova-docker to tracked project list
*** dtantsur|brb is now known as dtantsur17:27
*** koolhead17 has quit IRC17:28
clarkbsc68cal: the new block of code on line 317 there shows how you can test a specific tempest change within d-g17:29
*** armax has joined #openstack-infra17:29
* tchaypo continues song+dance routine extolling the virtues of the most wonderful and marvellously exciting
clarkbsc68cal: changes to d-g like that should never be merged but the can help you ensure that the number of bugs is falling17:29
*** sabari is now known as zz_sabari17:30
clarkbtchaypo: ok :P17:31
sc68calclarkb: thanks17:32
* tchaypo cheers17:32
tchaypothank you :)17:32
tchayponow of course it’ll turn out to break everything, I’ll be sad, but you’ll be the person who approved it...17:32
clarkbsc68cal: and if I am way off base on how much of this should just work we can go back to thinking about the experimental job17:33
clarkbsc68cal: which is why I dind't want to abandon thechange to add the new job just yet17:33
clarkbtchaypo: yes that is how things go around here17:33
*** asselin has joined #openstack-infra17:33
clarkbtchaypo: we can fix it together when that happens :P17:34
fungievery morning when i wake up, i put on my flame-retardant pants17:34
fungiwe're down to ~150 ready/building nodes on precise nodepool now17:35
*** salv-orlando has joined #openstack-infra17:35
*** vdo has quit IRC17:36
*** garyk has quit IRC17:36
*** avozza is now known as zz_avozza17:36
*** ihrachyshka has joined #openstack-infra17:38
*** gokrokve has quit IRC17:40
*** jraim has joined #openstack-infra17:41
*** ihrachyshka has quit IRC17:42
*** armax has quit IRC17:42
clarkbgood morning17:44
clarkbfungi: we must be getting close to ready to start nodepool?17:45
openstackgerritKhai Do proposed openstack-infra/system-config: make gerrit commentlink case insensitive for bug and story
*** tonytan4ever has quit IRC17:46
fungidown to 30 remaining17:46
*** gyee has joined #openstack-infra17:46
fungiso nearly there17:46
*** annegent_ has joined #openstack-infra17:47
*** KurtMartin has joined #openstack-infra17:47
*** _nadya_ has quit IRC17:47
anteayado you make crinkle noises when you walk?17:48
fungiit's now down to 0. i'll start deleting all other nodes if you want to start nodepoold on nodepool-dev17:48
* anteaya reviews ttx's patch17:48
clarkbok I am starting new nodepool now17:48
*** amotoki has joined #openstack-infra17:49
clarkbit is doing things17:49
fungiremaining 550 nodes on precise nodepool are deleting now17:49
clarkbI really like doing this when we don't have to restart zuul17:50
clarkbso much easier :)17:50
*** amotoki_ has joined #openstack-infra17:50
clarkbfungi: hold on17:50
*** stevemar has quit IRC17:50
clarkbfungi: don't delete used quite yet17:50
clarkbsince those jobs may still be running17:50
fungiyeah, realized that and didn't do it17:50
fungiwaiting one hour first ;)17:50
*** kmartin has quit IRC17:51
clarkbthough 550 seems a bit high17:51
fungiwe were maxxed out on capacity when we started17:51
clarkbI guess we are currently running quite a bit of jobs17:51
jeblairif we wannted to be really clever, we could copy the database between the servers17:51
*** gokrokve has joined #openstack-infra17:51
*** doude has quit IRC17:51
*** armax has joined #openstack-infra17:52
clarkbjeblair: ya I thought about that. Ithink we haveto do it after deleting building nodes though17:52
clarkbotherwise we have to copy the ssh keys and other non DB state17:52
clarkbstill doable though17:52
*** shakamunyi has joined #openstack-infra17:52
fungistick the db in trove and connect nodepoold to it, but only run one at a time?17:52
ryanpetrelloanybody have any idea why the `gate-pecan-tox-ceilometer-tip` is stuck in queued and doesn't show up in ?17:53
*** dizquierdo has quit IRC17:53
ryanpetrellolooks like pecan reviews are wedged due to this, wondering if something changed related to nodes running ceilometer17:53
*** andreykurilin_ has joined #openstack-infra17:53
*** simonmcc has joined #openstack-infra17:53
*** amotoki_ has quit IRC17:54
*** amotoki has quit IRC17:54
anteayattx so ironic, tripleo, zaqar, manila, designate, barbican are not part of the <project>-stable-maint grouping?17:54
fungiryanpetrello: why would gate-pecan-tox-ceilometer-tip show up in
clarkbryanpetrello: we are changing the nodepool node that boots the test slaves17:54
clarkbfungi: gearman updates the jenkins job summary with other jobs that ran17:54
fungiclarkb: oh...17:54
ryanpetrellofungi, sorry, I meant the "All builds for this change set:"17:54
*** [HeOS] has quit IRC17:55
fungiright, that17:55
ryanpetrelloand if you look at Zuul, the ceilometer-tip job is there but just stuck17:55
clarkbryanpetrello: I think this is lilely just due to us changing the host that is building nodes17:55
*** doug-fish has joined #openstack-infra17:55
clarkbso there will be a slight delay as we transition17:55
ryanpetrellois there something I need to do other than just waiting?17:55
ryanpetrellookay, that's fine :)17:55
*** ctracey has joined #openstack-infra17:55
ryanpetrellojust wanted to make sure it was a delay and not something I needed to change/retry17:55
*** shakamunyi has quit IRC17:55
fungiryanpetrello: which ceilometer change specifically?17:56
ryanpetrelloit's a pecan change that gates against ceilometer17:56
fungioh, right17:56
*** timcline_ has joined #openstack-infra17:56
fungii bet the jenkins slave that job originally ran on met with an unfortunate end and zuul requeued the job but hasn't gotten a new node to assign to it yet17:57
fungiand we're currently node-starved while in the middle of swapping nodepool servers17:57
anteayaAJaeger: I would like some sort of indication that a job called project-config-gerrit is only evaluating stackforge acls:
*** ivar-lazzaro has joined #openstack-infra17:57
clarkbfungi: I am going to check used nodes on old nodepool against each master and delete the ones that are safe to delete17:57
anteayaAJaeger: since the name does imply to me that it would apply to all incoming acl config files but the script is not doing that right now17:58
clarkbfungi: I do this by nodepool list | grep jenkinsXX > file. then just removing the handful of ndoes tha shouldn't be deleted17:58
clarkbfungi: it works well enough. though if I was smart I would figure out python-jenkins and completely script it17:58
*** sarob has joined #openstack-infra17:59
ryanpetrellofungi: makes sense, if it doesn't complete I'll just give it another try sometime later this week when things calmd own17:59
ryanpetrello*calm down17:59
fungiclarkb: yeah. or maybe nodepool could grow a quiescent mode we can trigger similar to zuul's18:00
openstackgerritgordon chung proposed openstack/requirements: Up pymongo version to avoid memory leak
fungiclarkb: where it immediately starts deleting ready/building nodes and doesn't provision any new nodes, but continues to handle the used->delete transition and retry deletes as usual18:01
*** Sukhdev has quit IRC18:01
*** emagana_ has quit IRC18:01
*** dannywilson has quit IRC18:01
fungiand disconnect from gearman too18:02
*** derekh has quit IRC18:02
*** MaxV has quit IRC18:02
*** serverascode___ has joined #openstack-infra18:02
*** cpowell has quit IRC18:03
*** annegent_ has quit IRC18:03
*** mmaglana has quit IRC18:04
*** patrickeast has quit IRC18:04
*** koolhead17 has joined #openstack-infra18:04
*** patrickeast has joined #openstack-infra18:04
*** ameade_ has joined #openstack-infra18:04
*** dannywilson has joined #openstack-infra18:05
*** SumitNaiksatam has quit IRC18:05
*** shakamunyi has joined #openstack-infra18:06
jeblairi have suspected a problem with the "All builds for this change set" data, but have not had a chance to actually diagnose18:06
jeblairjust to say -- best not to rely on its accuracy18:06
*** marun has quit IRC18:07
*** wznoinsk has quit IRC18:07
ryanpetrellofungi: clarkb: yep, it was just a delay, the job seems to be running now18:07
*** emagana has joined #openstack-infra18:08
*** jgallard_ has joined #openstack-infra18:08
clarkbok I have deletions running for 04-0718:09
*** zhiyan has joined #openstack-infra18:09
fungiyep, i show numbers dropping rapidly18:10
*** charz has quit IRC18:10
jeblairi don't see any red flags in the logs; either timeouterrors or exceptionally long times in the main loop18:11
*** emagana_ has joined #openstack-infra18:11
*** emagana has quit IRC18:12
*** resker has quit IRC18:12
*** jgallard_ has quit IRC18:12
*** Masahiro has joined #openstack-infra18:13
*** gokrokve has quit IRC18:13
*** amitgandhinz has quit IRC18:15
*** _nadya_ has joined #openstack-infra18:16
openstackgerritMerged openstack-infra/python-storyboardclient: Initial Cookiecutter Commit.
*** Masahiro has quit IRC18:17
*** cnesa2 has quit IRC18:18
*** zz_sabari is now known as sabari18:18
*** charz has joined #openstack-infra18:20
openstackgerritMerged openstack-infra/storyboard-webclient: Eabled custom dev configuration.
*** _nadya_ has quit IRC18:25
*** e0ne has joined #openstack-infra18:25
*** sabari is now known as zz_sabari18:25
*** armax has quit IRC18:26
*** sarob has quit IRC18:27
*** markmcclain has joined #openstack-infra18:27
clarkbjeblair: still no timeouts18:28
clarkbI have us down to ~150 nodes on the old host18:28
clarkband that is still falling18:28
jeblaircool.  the graph is really spiky, but i think that's because it still has incorrect data about usage18:28
anteayadhellmann: thanks for the project creator's guide, I was just able to guide sukhdev to use it so hopefully he will guide others as well18:28
jeblairspiky's good though -- it means the loop is fast18:28
*** cdent has quit IRC18:29
*** armax has joined #openstack-infra18:30
*** SumitNaiksatam has joined #openstack-infra18:30
*** luqas has quit IRC18:30
*** cpowell_ has joined #openstack-infra18:30
*** luqas has joined #openstack-infra18:31
*** harlowja_away is now known as harlowja18:31
*** signed8bit is now known as signed8bit_ZZZzz18:31
*** cpowell_ has quit IRC18:33
*** cpowell has joined #openstack-infra18:33
*** annegent_ has joined #openstack-infra18:34
*** emagana has joined #openstack-infra18:34
jesusauruswhat does it mean when fetching from a zuul merger says "error: Unable to get pack index"?18:35
*** marun has joined #openstack-infra18:35
jesusaurusis that a race condition of some sort?18:35
clarkbjesusaurus: hrm it shouldn't be. I think that git asked for an index of the pack files and that index operation failed. Could be a number of things. I would expect a race happening after you get the index and ask for a pack that is no longer there18:36
*** pradk has joined #openstack-infra18:36
clarkbjesusaurus: check apache on the zuul merger side for any errors in the error log?18:37
*** emagana_ has quit IRC18:37
*** annegent_ has quit IRC18:39
openstackgerritKhai Do proposed stackforge/python-jenkins: Handle job and node names that contain spaces
*** pelix has quit IRC18:40
fungispeaking of "Unable to get pack index" i just discovered that cgit has the ability to turn off its default dumb http protocol endpoint support (enable-http-clone=off)18:40
fungipatch on the way18:40
*** Sukhdev has joined #openstack-infra18:41
Sukhdevfungi, clarkb: Hi18:42
jesusaurusclarkb: thanks18:42
*** jedimike has quit IRC18:43
Sukhdevclarkb: I got my new project approved - see
Sukhdevclarkb: need to add couple of members so that we can work on this project18:43
*** cnesa has joined #openstack-infra18:44
clarkbSukhdev: who should be the initial member of the -core and -release groups18:44
clarkbSukhdev: I can just add one and that person canfill out the remaining members18:44
*** teran has quit IRC18:44
Sukhdevclarkb: that will be myself - do you need my email ID ?18:45
*** teran has joined #openstack-infra18:45
clarkbno I can get it from the change18:45
*** pblaho has quit IRC18:45
*** Hefeweizen has joined #openstack-infra18:46
clarkbSukhdev: done18:46
Sukhdevclarkb: cool - thanks18:46
Sukhdevclarkb: what is the easiest way to check/verify for me?18:47
clarkbjeblair: there was a timeout error ~2 minutes ago, but that was the first and only one18:47
clarkbSukhdev: I am not sure I understand the question. You want to see that i added you to the groups?18:47
clarkbSukhdev: to do that go to which is under People -> List Groups18:48
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Disable cgit's dumb HTTP protocol endpoint support
clarkbSukhdev: then type into the filter box to searc hfor the group you are interested in. When it shows up click on it18:49
clarkbSukhdev: and that takes you to the membership page18:49
*** melwitt has joined #openstack-infra18:49
*** tonytan4ever has joined #openstack-infra18:49
*** kmartin has joined #openstack-infra18:50
*** Ryan_Lane has joined #openstack-infra18:50
*** teran has quit IRC18:50
*** shashankhegde has joined #openstack-infra18:50
sdagueclarkb: hmmm... I'm about to just put this fixtures bit down. Because, here's the odd thing, I'm tracing inside the fixture code and it looks like it's rebinding the symbol fine18:50
sdagueexcept then it doesn't after the import18:50
*** berendt has quit IRC18:51
clarkbsdague: doesn't the import happen first though?18:52
clarkbbefore it rebinds18:52
*** AJaeger_ has joined #openstack-infra18:52
*** esker has joined #openstack-infra18:53
*** KurtMartin has quit IRC18:53
fungilooks like all old nodepool nodes are vamoose now18:53
sdagueso, in MonkeyPatch setUp it actually seemed to find the symbols correctly, and looks like it set the attrs correctly18:53
AJaeger_anteaya: regarding - I understand your concern but adding this as comment to the voting=false is not the right thing IMHO18:53
sdaguebut then when I try to use the object, nope18:53
clarkbfungi: for the most part. and new nodepool marked all the things ready and now gear is catching up18:54
anteayaAJaeger_: fair enough18:54
anteayaAJaeger_: any way we can address my concern?18:54
*** MaxV has joined #openstack-infra18:54
sdagueI have the hackiest of print debug code if you are up for poking at this18:54
clarkbsdague: so by the time the test runs its no longer correct18:54
zaroclarkb: do you know if we are having this problem at all?!/story/200003018:55
clarkbsdague: I think that may be explained if you do properly import the patched module later18:55
clarkbsdague: as that import action should write over the existing name?18:55
sdague if i print debug inside of MonkeyPatch the attrs on C are changed18:55
anteayaAJaeger_: armax is sitting 10 feet away and is very amenable, we just have to agree on what to suggest to him18:55
AJaeger_anteaya: we could add a comment above the job in layout.yaml if that helps.18:55
anteayaI can live with that18:56
*** shayneburgess has joined #openstack-infra18:56
anteayacare to add the comment?18:56
clarkbzaro: I don't think we have seen that. It may be related to us having single use only nodes though18:56
zaroclarkb: someone else reported what looks like the same thing:
AJaeger_anteaya: physically 10 feet?18:56
*** otter768 has joined #openstack-infra18:56
anteayaAJaeger_: in the review, a review comment18:56
clarkbzaro: I wonder if wikimedia is using long lived nodes which increases the changes of deadlock18:56
anteayaAJaeger_: yes, neutron mid-cycle18:56
AJaeger_anteaya: understood, on my way...18:56
AJaeger_anteaya: Ah! Say hi to him!18:56
anteayaAJaeger_: will do18:56
zaroclarkb: yeah they do.18:56
clarkbsdague: so in that case you setup before your import18:57
clarkbsdague: I think the followup import is what breaks you there18:57
zaroclarkb: i was thinking it might also be the other plugins that they have installed.18:57
sdagueclarkb: right, clearly, but why18:57
clarkbsdague: when you construct the patch then run setup You are essentially saying x.that.does.not.exist.yet = thing. Then at import time you say x.that.does.not.exist.yet = the thing it really is18:58
AJaeger_anteaya: I haven't seen these kind of comments in layout.yaml and those also become quickly outdated.18:58
anteayaAJaeger_: right whereas I have seen them in the location where non-voting status is set18:58
anteayawhich is why I picked there18:59
clarkbsdague: and python lets you get away with that whereas other lnaguages would complain about unknown scopes or symbols18:59
anteayaI just want to remember that right now this non-voting job only evaluates stackforge acls18:59
anteayaAJaeger_: I'm afraid I will forget18:59
sdaguebrb... one handed with infant19:00
AJaeger_sdague, clarkb, fungi: Any suggestions on how to address anteaya's comment suggestion best? See
fungiclarkb: what's the possibility that the repeat gate-infra-puppet-apply-centos6 failures are an issue in module dependency ordering introduced when i approved adding puppet-httpd to the modules array yesterday?19:00
*** sabeen3 has joined #openstack-infra19:00
clarkbfungi: I think that must be what it is. I noticed that mrmartin's fix for the module ordering didn't seem affected19:00
AJaeger_anteaya: The current comments in the voting section explain why something is not voting or set this way19:00
*** nfedotov has joined #openstack-infra19:00
*** wenlock has joined #openstack-infra19:00
anteayaAJaeger_: fair enough, but there are comments there19:01
AJaeger_anteaya: yes, there are comments.19:01
*** timcline_ has quit IRC19:01
*** otter768 has quit IRC19:01
anteayaI'm willing to branch out on content19:01
*** sabeen has quit IRC19:01
clarkbanteaya: AJaeger_ I think you can run that non voting, then add openstack in at some point still non voting and when you are happy with it make it voting?19:01
anteayathat is the plan yes19:02
*** annegent_ has joined #openstack-infra19:02
*** amitgandhinz has joined #openstack-infra19:02
anteayajust trying to capture that somehow so I don't forget when we make it non-voting19:02
anteayaor whoever approves the patch once it is offered19:02
*** nfedotov has quit IRC19:02
AJaeger_clarkb: patchset 6 introduced a comment saying "# Right now this is only evaluating acl files on stackforge projects. "19:02
*** nfedotov has joined #openstack-infra19:02
*** MaxV has quit IRC19:02
AJaeger_And that one didn't work for me at all ;(19:02
fungianteaya: AJaeger_: yeah, having it run tools/ against the proposed acl and complain if the output differs from the input might be a good addition19:03
AJaeger_anteaya: please tell armax ;)19:03
clarkbstill just the one timeout so this is much happier nodepool than before19:03
*** reed has joined #openstack-infra19:03
*** timcline has joined #openstack-infra19:04
AJaeger_anteaya, armax: let me make a proposal for the patch with specific wording. I hope that addresses both our concerns...19:04
anteayaI'm sorry if I am thick but I think I missed fungi's suggestion19:04
armaxAJaeger_: standing by19:04
armaxAJaeger_: thanks for bearing with me!19:04
AJaeger_armax: Thanks for bearing with my bikeshedding ;)19:05
*** arosen has joined #openstack-infra19:05
armaxanteaya, clarkb, AJaeger_: btw do we have an ETA for ?19:05
*** reed has quit IRC19:05
armaxAJaeger_: no worries19:05
*** jistr has joined #openstack-infra19:05
*** bitblt has joined #openstack-infra19:05
anteayaAJaeger_: you are the best person to bikeshed with19:05
dimsAJaeger_: thanks for the heads up on reverify vs recheck!19:06
anteayafungi: sorry I see what you are saying now19:06
*** mrmartin has joined #openstack-infra19:06
*** jaypipes has joined #openstack-infra19:07
fungianteaya: basically ensure acl normalization prior to allowing it to merge, similar to our alpha-order checks19:07
*** zz_sabari is now known as sabari19:07
*** ayoung is now known as shhh_im_busy19:07
anteayamakes way more sense that piecemeal tests19:07
AJaeger_dims: you're welcome19:08
fungianteaya: armax: also, i think the future of this job, once it's proven as being non-annoying/problematic, is to absorb it into one of our existing syntax/format checking jobs for that project so that we don't waste spin-up/tear-down overhead and waste additional job worksers running it19:08
armaxfungi: agreed19:08
clarkbalso so that it is easier to remember to run it locally19:08
AJaeger_fungi: agreed19:09
*** sabari is now known as zz_sabari19:09
*** andreykurilin_ has quit IRC19:09
*** tsg has joined #openstack-infra19:09
AJaeger_armax, anteaya: Does my comment work for you?19:10
*** shhh_im_busy is now known as ayoung19:10
*** MaxV has joined #openstack-infra19:11
fungiSergeyLukjanov: did you get a chance to retry applying that acl update to sahara? it doesn't seem to have changed in gerrit19:11
clarkbfungi: ya there is an error in that acl file19:12
*** emagana has quit IRC19:12
clarkbfungi: not sure what it is though but it appears to have nt updated since late august19:12
dhellmannanteaya: I'll be happy to try to help keep that up to date, since I've rewritten our Oslo guide to point to it at the appropriate spot(s), so let me know if I can help answer questions or add detail19:12
*** wenlock has quit IRC19:12
*** zz_sabari is now known as sabari19:12
*** emagana has joined #openstack-infra19:12
*** _shaps_ has quit IRC19:13
*** ala_ has quit IRC19:14
AJaeger_dhellmann: could you review , please? Fungi mentioned that it's not linear with my change anymore.19:14
AJaeger_and I haven't found a good way to keep it in order, suggestions are welcome19:15
fungiclarkb: i'll have a look then19:16
*** yamamoto has joined #openstack-infra19:16
*** emagana has quit IRC19:16
*** erikmwilson has joined #openstack-infra19:17
*** KurtMartin has joined #openstack-infra19:17
*** signed8bit_ZZZzz is now known as signed8bit19:19
AJaeger_speaking about extra checks, I added a check for the existance of descriptions in gerrit/projects.yaml with . Like anteaya prefers a comment that the gerrit acls checks is only partial, I like to see descriptions for projects. is this something we should do moving forward - or should i abandon it?19:20
*** kmartin has quit IRC19:20
*** baoli has quit IRC19:20
*** baoli has joined #openstack-infra19:21
*** yamamoto has quit IRC19:21
fungiit seems reasonable that every project should have a description19:22
fungii am not opposed to making that a required field19:22
*** jp_at_hp has quit IRC19:23
*** reed has joined #openstack-infra19:23
openstackgerritMarton Kiss proposed openstack-infra/system-config: Initial commit of Ask website
*** wenlock has joined #openstack-infra19:23
AJaeger_anteaya: Are you opposed completely to 138749 or do you have a suggestion on what to change?19:24
AJaeger_fungi, thanks19:24
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Add nodepool-launcher spec
*** [HeOS] has joined #openstack-infra19:26
jeblairclarkb, fungi, mordred: ^ addresses recent nodepool issues19:26
*** annegent_ has quit IRC19:26
*** emagana has joined #openstack-infra19:26
*** annegent_ has joined #openstack-infra19:27
fungijeblair: thank you!19:28
*** baoli has quit IRC19:28
jeblair2014-12-10 10:12:29,509 ERROR gear.Client.unknown: Connection <gear.Connection 0x267c0d0 host: port: 4730> timed out waiting for a response to a submit job request: <gear.Job 019:28
jeblairx7feaf0043e50 handle: None name: unique: 92cb9ce3549f44d283d4a3cb9230d8de>19:28
jeblair2014-12-10 10:12:29,510 INFO zuul.Gearman: Gearman job <gear.Job 0x7feb6d80d190 handle: H: name: build:gate-tempest-dsvm-neutron-large-ops-juno:devstack-trusty unique: 3cabdc4d17284762bc79144fe316a743> lost due to disconnect19:28
jeblairclarkb: ^19:28
clarkbjeblair: thanks19:28
clarkbI greped for 'gear' and 'gearman' before quickly ^Cing :)19:28
jeblairclarkb: "isconnect" :)19:28
AJaeger_armax: you can enable openstack-dev as well
jeblairclarkb: looks like 3 times today19:29
AJaeger_armax: could you remove the -x? That is too much debug for me right now19:29
*** jistr has quit IRC19:30
*** ZZelle has joined #openstack-infra19:31
armaxAJaeger_: will do19:31
armaxAJaeger_: with openstack-dev there are some projects that do not comply, as far as I recall19:31
armaxI doublecheck19:31
*** KurtMartin is now known as kmartin19:32
fungiclarkb: following merging last night, did puppet need to get manually reapplied on puppetmaster.o.o for that to work?19:32
clarkbfungi: ya I did it19:32
*** wenlock has quit IRC19:33
dhellmannAJaeger_: looking19:33
harlowjahave u guys ever seen errors like '/opt/stack/new/devstack/functions-common:603 Cloning not allowed in this configuration' ?19:33
harlowjaseen in
*** shakamunyi has quit IRC19:33
*** esker has quit IRC19:33
fungiclarkb: doesn't seem to have taken effect then. we still have stuff listed in puppetboard which isn't updating19:33
mordredjeblair: cool19:34
clarkbfungi: hrm let me double check I thought puppetmaster was puppeting it self at the very least19:34
*** shakamunyi has joined #openstack-infra19:34
dhellmannAJaeger_: that looks like good info to add; I'll have to consider the order a bit. You seem to have found a reasonable place for it, though.19:34
*** esker has joined #openstack-infra19:34
*** signed8bit has quit IRC19:34
fungiclarkb: there might be another reason. i'll check the hosts where this is failing. maybe agent is disabled (or for the new git-fe servers maybe puppetmaster needs to update ~root/.ssh/known_hosts)19:34
*** timcline has quit IRC19:35
*** bdpayne has joined #openstack-infra19:35
clarkbfungi: /etc/ansible/ansible.cfg looks right and running that script as root spits out json with a list of nodes19:35
AJaeger_armax: Only add it if it passes for openstack-dev ;)19:35
*** pc_m_ has joined #openstack-infra19:35
fungiclarkb: okay, i'm going with puppet agent is still set --disable on the afs/kdc hosts, and git-fe hosts need ssh host keys accepted19:36
AJaeger_dhellmann: thanks, looking forward whether you have some great ordering idea ;)19:36
clarkbfungi: likely19:36
clarkbfungi: I can do git-feXX now19:36
dhellmannAJaeger_: well, how about as a sub-section of the very next block?19:36
clarkbok thats done19:36
dhellmannAJaeger_: because step 1 in that section is to test "git review"19:37
*** sarob has joined #openstack-infra19:37
clarkbjeblair: should I be worried about it restarting 3 times today?19:37
*** esker has quit IRC19:37
*** sarob has quit IRC19:38
dhellmannAJaeger_: we could replace each of those vague numbered steps with a little description of what to do (your git-review stuff; instructions to run tox; where to look in gerrit for acls; a link to instructions for tagging a release)19:38
fungiclarkb: i guess if those go to reported in puppetboard then it's working for sure19:38
*** emagana has quit IRC19:38
jeblairclarkb: yep.  i want to catch it in a tcpdump, and then i think we should probably increase the zuul timeout back to 3 minutes19:38
jeblairer 5 minutes19:38
*** salv-orlando has quit IRC19:38
*** sarob has joined #openstack-infra19:38
jeblairclarkb: tcpdump is running19:38
clarkbjeblair: in that spec we would handle rate limiting by not submitting jobs to launch/delete above the rate limits? or would we need some bus between workers to negotiate that?19:38
clarkbjeblair: rgr19:38
jeblairclarkb: what kind of rate limiting?  openstack?19:38
clarkbjeblair: ya openstack api limits19:39
*** pc_m has quit IRC19:39
jeblairclarkb: the workers use the provider managers we have now, whose only purpose is to serialize and rate limit api calls from multiple threads19:39
*** sabari is now known as zz_sabari19:39
AJaeger_dhellmann: do you want to take over my patch?19:40
clarkbjeblair: but wouldn't they run in different processes without shared memory?19:40
*** dtantsur is now known as dtantsur|afk19:40
jeblairclarkb: one worker per provider, that worker still spawns lots of threads for each launch19:40
clarkboh I see19:40
*** marun has quit IRC19:40
dhellmannAJaeger_: "want" is a strong word :-) How about just moving what you have down below the list for now, and I'll work on the other details in a follow-up?19:40
AJaeger_dhellmann: so, just one section further down? I can do that.19:41
jeblairclarkb: so this moves the thread contention out of the main loop, makes it smaller because we could run 10 workers in our configuration (for our 10 providers), and finally if that's not enough, we can limit the contention by saying "no more than X threads at a time for this worker"19:41
dhellmannAJaeger_: yeah, under "Verify That Gerrit and the Test Jobs are Working"19:41
*** emagana_ has joined #openstack-infra19:42
*** marun has joined #openstack-infra19:42
*** mbacchi has quit IRC19:42
clarkbjeblair: yup19:42
*** sarob has quit IRC19:43
*** timcline has joined #openstack-infra19:43
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Check that Gerrit ACL files are normalized
*** timcline has quit IRC19:44
AJaeger_fungi: is the above what you had in mind? ^19:44
*** zz_sabari is now known as sabari19:44
*** emagana has joined #openstack-infra19:44
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: Add .gitreview content to Creator's Guide
AJaeger_dhellmann: done, thanks!19:45
*** emagana_ has quit IRC19:46
*** cnesa2 has joined #openstack-infra19:46
openstackgerritMerged openstack-infra/project-config: Support running script against local changes
dhellmannAJaeger_, sorry, I meant to make your new section a *subsection* of that one, not just list it after19:46
*** cnesa has quit IRC19:47
*** nfedotov has quit IRC19:47
dhellmannAJaeger_: "make sure git review works, if not, these are the steps to make it work"19:47
*** _nadya_ has joined #openstack-infra19:47
*** nfedotov has joined #openstack-infra19:48
AJaeger_dhellmann: so, then we should rename it.19:48
dhellmannHmm, probably. How about "Configuring ``git review``"?19:49
*** tsg has left #openstack-infra19:49
*** tsg has quit IRC19:49
AJaeger_dhellmann: and the first paragraph needs to be changed as well, let me review it again...19:49
*** jcoufal_ has quit IRC19:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Normalize Gerrit ACLs
dhellmannAJaeger_, oh, yeah, I missed the tests bit. I think you can leave that out because it is step 2 in the list right above19:50
*** bitblt has quit IRC19:51
*** mmaglana has joined #openstack-infra19:53
*** MaxV has quit IRC19:53
*** sabeen has joined #openstack-infra19:53
*** sabeen3 has quit IRC19:54
*** nfedotov1 has joined #openstack-infra19:55
*** nfedotov has quit IRC19:55
*** timcline has joined #openstack-infra19:56
AJaeger_dhellmann: what about adding them as subsection as well? Here comes a patch and I fear it needs some more fleshing out when I'm a bit less distracted19:56
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: Add .gitreview content to Creator's Guide
*** sabari is now known as zz_sabari19:57
dhellmannAJaeger_: yes, that's exactly what I'm talking about -- I'll do the other sections as a follow-up to your patch19:57
*** zz_sabari is now known as sabari19:57
*** pblaho has joined #openstack-infra19:57
AJaeger_dhellmann: thanks!19:58
clarkbwe are now down to a very small list of problem nodes on old nodepool19:59
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Check that Gerrit ACL files are normalized
*** nfedotov has joined #openstack-infra20:00
clarkbI will dig into those after lunch20:00
clarkbstill just the one timeout on new nodepool so things look much better there20:00
*** nfedotov has quit IRC20:00
*** nfedotov1 has quit IRC20:01
*** nfedotov has joined #openstack-infra20:01
*** erikmwilson has quit IRC20:01
AJaeger_fungi, addressed your comment and also normalized all current files, see .20:01
*** Masahiro has joined #openstack-infra20:01
*** dims has quit IRC20:02
*** dannywilson has quit IRC20:03
ryanpetrellosince the OpenStack CI doesn't test py26 and 32 anymore, is there a way for me to set up other external CI for pecan myself?20:04
ryanpetrello(that affects the vote on reviews?)20:04
dhellmannryanpetrello: we're still testing 26 for oslo libs, so you should be able to as well20:04
ryanpetrelloyea, though I assume it's eventually going away20:04
clarkbdhellmann: ya I offered but ryanpetrello said meh :)20:04
clarkbit will go away when we eol juno20:04
ryanpetrellosure :)20:04
*** kgiusti has quit IRC20:04
dhellmannah, ok20:04
ryanpetrelloand I don't want you all to have to carry this20:04
ryanpetrelloit's not a major issue for me20:05
fungiryanpetrello: third-party testing20:05
clarkbryanpetrello: there is a way for you to run those tests though let me get u a link20:05
*** aysyd has quit IRC20:05
ryanpetrelloI'm just wondering if there's some way for me to affect this myself20:05
ryanpetrellookay, awesome20:05
ryanpetrelloI'd be glad to manage some separate CI myself20:05
fungiyep, we do indeed have instructions20:05
*** dannywilson has joined #openstack-infra20:05
ryanpetrelloawesome, thanks!20:05
clarkbryanpetrello: some of that won't strictly apply because you will just be voting against a stackforge project right?20:05
ryanpetrelloclarkb: yep20:05
clarkbryanpetrello: but it has info on creating an account and reading the event stream and so on20:05
ryanpetrelloI'll have a read-through20:05
openstackgerritDoug Hellmann proposed openstack-infra/infra-manual: Expand more of the repo verification steps
dhellmannAJaeger_: ^^20:06
*** Masahiro has quit IRC20:06
AJaeger_dhellmann: wow! Let me review ...20:06
*** kgiusti has joined #openstack-infra20:06
*** teran has joined #openstack-infra20:07
*** wenlock has joined #openstack-infra20:08
*** MaxV has joined #openstack-infra20:08
*** spzala has quit IRC20:08
*** annegent_ has quit IRC20:09
*** armax has quit IRC20:10
* AJaeger_ waves good bye for today!20:10
*** AJaeger_ has quit IRC20:10
fungiclarkb: SergeyLukjanov: i checked the gerrit error_log and it does indeed dislike that acl file20:10
fungiclarkb: SergeyLukjanov: though i'm not immediately spotting the cause20:10
*** timcline has quit IRC20:10
SergeyLukjanovfungi, clarkb, sorry, folks, just returned home (11pm, heh)20:10
*** armax has joined #openstack-infra20:10
SergeyLukjanovfungi, I'll try to understand the issue in ACL file, don't see any atm :(20:11
*** ryanpetrello has quit IRC20:11
*** _nadya_ has quit IRC20:11
*** timcline has joined #openstack-infra20:11
*** ryanpetrello_ has joined #openstack-infra20:11
*** ryanpetrello_ is now known as ryanpetrello20:12
*** marun has quit IRC20:12
*** marun has joined #openstack-infra20:12
*** pc_m_ has quit IRC20:12
*** _nadya_ has joined #openstack-infra20:13
*** pc_m has joined #openstack-infra20:13
*** ldnunes has quit IRC20:14
*** _nadya_ has quit IRC20:16
*** sabari is now known as zz_sabari20:16
*** markmcclain has quit IRC20:17
fungiSergeyLukjanov: aha, found it20:19
SergeyLukjanovfungi, ?20:19
SergeyLukjanovfungi, /me still don't see :(20:19
*** salv-orlando has joined #openstack-infra20:20
fungiSergeyLukjanov: just a sec20:20
*** jcoufal has joined #openstack-infra20:23
*** koolhead17 has quit IRC20:23
*** unicell1 has quit IRC20:23
*** koolhead17 has joined #openstack-infra20:24
openstackgerritMerged openstack-infra/devstack-gate: Add os-net-config to the list of packages we clone
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Revert "Use more strict regex for sahara tags acl"
fungiSergeyLukjanov: ^20:25
*** erikmwilson has joined #openstack-infra20:25
SergeyLukjanovfungi, omg, I've been testing it manually on my gerrit installation20:25
*** signed8bit has joined #openstack-infra20:25
fungiSergeyLukjanov: when i reverted that manually on review.o.o i was able to apply it with manage-projects just fine20:26
fungiSergeyLukjanov: it's probably an escaping problem?20:26
*** luqas has quit IRC20:26
SergeyLukjanovfungi, hm, yeah, I think it could be20:26
*** nfedotov1 has joined #openstack-infra20:26
SergeyLukjanovfungi, thank you very much!20:27
*** luqas has joined #openstack-infra20:27
*** nfedotov has quit IRC20:28
*** koolhead17 has quit IRC20:29
*** [HeOS] is now known as HeOS20:31
fungiSergeyLukjanov: yw20:31
*** tonytan4ever has quit IRC20:32
dougwighi infra.  what's the right way to redo a gate job?   failed for a transient error, passed in the check queue.
*** spzala has joined #openstack-infra20:34
openstackgerritJoshua Harlow proposed openstack/requirements: Bump up oslo.utils to 1.1.0
*** MaxV has quit IRC20:37
*** erikmwilson has quit IRC20:38
fungidougwig: recheck, assuming you know why it failed and that it's not related to the change20:38
*** lttrl has quit IRC20:38
dougwigwas a network failure, unrelated to the change.  thanks.20:38
*** timcline has quit IRC20:39
*** annegent_ has joined #openstack-infra20:39
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules
*** Sukhdev has quit IRC20:41
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules
armaxAJaeger: I tweaked the check logic first, let me know if you’d like me to revise your patch that depends on it20:42
SergeyLukjanovfungi, we're using sahara-ci to run jobs to verify it's own config located in stackforge/sahara-ci-config, is it ok if I'll add permissions for sahara-ci to vote on it?20:43
*** rushiagr is now known as rushiagr_away20:43
clarkbyup as the ptl that is entirely up to you20:44
fungiSergeyLukjanov: yes, that's the intent of the new design model20:44
SergeyLukjanovfungi, clarkb, ok, thx20:44
fungiproject ptls get to decide what accounts are allowed to vote on their projects now20:44
*** annegent_ has quit IRC20:44
gilliardHello - what code backs this page? - I want to feature request search-by-irc-nick there.20:45
gilliard(actually it is fungi's idea, I think)20:46
*** bdpayne_ has joined #openstack-infra20:46
clarkbopenstack-org is the bugtracker used on launchpad. The source is on github somewhere though .I wonder if the are using github bug tracker now?20:47
clarkbfungi: ^20:47
mordredclarkb: I thought the source was in gerrit now20:47
*** nfedotov1 has quit IRC20:47
clarkbjeblair: jhesketh SergeyLukjanov fungi mordred fyi. I am going to try and send warren email today before the long break20:47
jeblair ?20:47
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules
clarkbmordred: no, it has been imported into gerrit but that isn't the source20:47
fungiclarkb: gilliard: right, the openstack-org project on launchpad for now. and while we have imported the source into gerrit as openstack-infra/openstackweb they've apparently gotten out of sync20:48
mordredoh good20:48
*** nfedotov has joined #openstack-infra20:48
* mordred steps away from this conversation20:48
gilliardAlright I'll hit launchpad and raise something - I'm happy to code it if I know where it lives.20:49
* clarkb realized that the university basically shuts down for 3 weeks so I better hurry up and send mail if I want a response reasonbly soon20:49
openstackgerritMerged openstack-infra/project-config: Revert "Use more strict regex for sahara tags acl"
fungiwe approved the project-config change to import that repo even though they apparently weren't done updating it in github yet, and before i'd had a chance to corner the tipit folks to discuss new workflow options20:49
*** bdpayne has quit IRC20:49
*** MaxV has joined #openstack-infra20:49
jeblairfungi: none of that makes any sense :(20:50
*** kmartin has quit IRC20:50
*** lttrl has joined #openstack-infra20:51
fungijeblair: well, they published that as a "preview" export of code they were developing for the next iteration of the site (which is now in production since late last week)20:51
jeblairfungi: that's not what their readme said :)20:52
fungii think they may have a little trouble navigating the space-time continuum20:52
jeblairindeed, it's been a long 2 years since we agreed the import was the next step.20:53
*** shakamunyi has quit IRC20:53
*** erikmwilson has joined #openstack-infra20:53
*** whoops has joined #openstack-infra20:53
*** lttrl has quit IRC20:54
jeblairanyway, nothing is stopping them from proposing a commit with "Code dump to bring repo up to date since our last code dump"20:54
fungiagreed. i will pick that conversation back up again20:54
gilliardSo I raised the ticket, but the short answer for where the code lives is "ask again later", right ;)20:55
fungii think they were waiting until that code was actually in production, so now it's a good time to get them using gerrit20:55
fungigilliard: yeah, disappointing to most of us, but that's the present situation20:56
gilliardNo worries, thanks all!20:56
*** bdpayne_ has quit IRC20:56
*** shakamunyi has joined #openstack-infra20:56
*** teran_ has joined #openstack-infra20:57
*** otter768 has joined #openstack-infra20:57
*** Longgeek has quit IRC20:58
pleia2I hoped after a few years I'd stop saying "arrrgghhh puppet"20:58
*** teran has quit IRC20:59
*** nfedotov has quit IRC20:59
pleia2I think it's time for lunch :)21:00
*** mwagner_lap has quit IRC21:00
*** MaxV has quit IRC21:00
*** nfedotov has joined #openstack-infra21:01
*** erikwilson has joined #openstack-infra21:01
*** wenlock has quit IRC21:01
*** erikwilson has quit IRC21:01
*** otter768 has quit IRC21:02
*** tonytan4ever has joined #openstack-infra21:02
*** dims has joined #openstack-infra21:03
*** btran has joined #openstack-infra21:03
*** tonytan4ever has quit IRC21:03
*** tonytan4ever has joined #openstack-infra21:04
*** belmoreira has joined #openstack-infra21:04
*** annegent_ has joined #openstack-infra21:05
*** dims has quit IRC21:08
*** dkranz has quit IRC21:08
*** hashar has joined #openstack-infra21:09
*** erikmwilson has quit IRC21:09
*** [BNC]erikmwilson is now known as erikmwilson21:10
*** jp_at_hp has joined #openstack-infra21:10
*** mestery has quit IRC21:10
*** MaxV has joined #openstack-infra21:10
*** bdpayne has joined #openstack-infra21:10
*** ociuhandu has quit IRC21:11
*** armax has quit IRC21:12
anteayamorning mattoliverau21:13
*** esker has joined #openstack-infra21:13
*** weshay has quit IRC21:13
*** teran has joined #openstack-infra21:13
*** hdd has joined #openstack-infra21:13
*** zz_sabari is now known as sabari21:13
*** teran_ has quit IRC21:14
*** emagana has quit IRC21:14
*** achanda has quit IRC21:15
*** emagana has joined #openstack-infra21:15
*** dannywilson has quit IRC21:16
*** jp_at_hp has quit IRC21:16
*** dims has joined #openstack-infra21:17
*** esker has quit IRC21:17
*** dannywilson has joined #openstack-infra21:17
*** emagana has quit IRC21:19
*** mfink has quit IRC21:21
*** kgiusti has quit IRC21:23
anteayaAJaeger: 138749 I'm kind of concerned over how much oversite we are adding to a space that by definition is not overseen21:23
anteayaAJaeger: I am really not wanting to set ourselves up to have to make decisions or arbitrate on stackforge issues21:23
JayFI just had a pep8 job fail in an interesting way; a traceback in pep8 itself. -- wanted to see if anyone here has seen it and/or if there's a known brokenness before rechecking21:23
*** esker has joined #openstack-infra21:24
anteayaAJaeger: feel free to move ahead without me, I am comfortable with the patch merging with my concern reflected in my vote and the comments21:24
*** teran_ has joined #openstack-infra21:25
anteayadhellmann: awesome, thank you21:26
*** mfink has joined #openstack-infra21:26
anteayaAJaeger fungi sorry been talking in person, will get a moment with armax soon21:26
*** timcline has joined #openstack-infra21:27
openstackgerritMonty Taylor proposed openstack-infra/project-config: Add elements for Infra servers
openstackgerritMonty Taylor proposed openstack-infra/project-config: Move to using package-installs.yaml
mordredjeblair, clarkb, fungi: ^^ latest state of the art - those build ubuntu and centos hosts - I have one more thing I need to do for nova-agent on centos21:27
*** timcline has quit IRC21:27
mordredalso, all of the upstream dib modifications I've done have landed already21:28
mordredso a large part of taht patch will go away very soon21:28
*** teran has quit IRC21:28
*** tonytan4ever has quit IRC21:28
*** julim has quit IRC21:29
*** denis_makogon has quit IRC21:31
*** dmakogon_ has joined #openstack-infra21:32
*** markmcclain has joined #openstack-infra21:32
*** timcline has joined #openstack-infra21:32
*** teran has joined #openstack-infra21:32
*** timcline has quit IRC21:32
*** asettle is now known as asettle-bfast21:34
*** teran_ has quit IRC21:36
*** MaxV has quit IRC21:38
*** timcline has joined #openstack-infra21:38
*** baoli has joined #openstack-infra21:39
*** timcline has quit IRC21:39
openstackgerritMerged openstack-infra/infra-specs: Zuul could use different yaml files
*** dizquierdo has joined #openstack-infra21:41
*** tonytan4ever has joined #openstack-infra21:42
*** shashankhegde has quit IRC21:42
*** jcoufal has quit IRC21:43
*** dannywilson has quit IRC21:43
jeblairwow, did i really approve that?  oops.21:45
jeblairclarkb: you previously reviewed that; would you mind giving it a quick review and either +2ing it or reverting it?21:45
jeblairclarkb: i'm really sorry :(21:45
anteayadid you just mean to +2 it?21:45
jeblairmust have slipped21:45
*** armax has joined #openstack-infra21:46
clarkbjeblair: sure21:47
jeblairyep, hit wrong button.21:47
*** weshay has joined #openstack-infra21:47
*** hashar has quit IRC21:48
*** armax has quit IRC21:49
*** nfedotov has quit IRC21:49
*** MaxV has joined #openstack-infra21:49
clarkbjeblair: ya  Ithink yolanda addressed my concerns with newer patchsets21:49
clarkbso not necessary to revert21:49
*** Ryan_Lane has quit IRC21:49
jeblaircool, thanks (i thought so too -- i went back and checked)21:50
*** Masahiro has joined #openstack-infra21:50
anteayain case anyone wonders if it is useful to have an infra person at mid-cycles the answer is yes21:51
*** dangers is now known as dangers_away21:51
jeblairanteaya: cool.  we're going to need 500 infra people.  ;)21:51
anteayaI get a lot of questions introduced wtih "I've been wanting to ask this for a while, but I didnt' want to bug anyone in -infra channel"21:51
anteayaha ha ha21:51
fungijeblair: mordred: any opposition to me enabling puppet agent on the afs/afsdb/kdc servers? i've checked with --noop and the only things of significance being changed are not related to afs or kerberos21:52
*** shashankhegde has joined #openstack-infra21:52
anteayaoh heads up jeblair in case you ever see a spec about testing things as a group in the gate, where either all the patches merge or none of them21:53
jeblairfungi: wfm21:53
*** patrickeast has quit IRC21:53
fungijeblair: mordred: i can leave afstest untouched for now... though it looks like the only significant changes it would get are possible reversion of a few iptables rules you may have added21:53
*** Ryan_Lane has joined #openstack-infra21:54
mordredfungi: nope. I think we should21:54
fungijeblair: mordred: great--doing now21:54
*** sabari is now known as zz_sabari21:54
jeblairfungi: yeah, i'd leave afstest out for now; it will be useful for a little longer, but not for long.21:54
*** markmcclain has quit IRC21:55
mordredfungi: if you're poking at those sorts fo things,
*** Masahiro has quit IRC21:55
*** zz_sabari is now known as sabari21:55
mordredfungi: actually - let me go take another pass through that21:55
mordredso nevermind21:55
jeblairanteaya: thanks.  that's close, and related to "one-way cross-repo dependencies", but not the same thing.  we'll have to get into the nitty gritty on that (some things in this area are desirable, some are actively not desired)21:55
fungimordred: sure. i mostly wanted to get started playing with afs as a consistent backend for our git server farm, so that we can think about turning off ip persistence in haproxy for better load distribution21:56
mordredfungi: ooh. I like that idea21:56
anteayaso since the short term fix is to make carl baldwin merge all the affected patches (it is easier than editing zuul for now) it might be a bit before he writes the spec but he said the would21:57
anteayaso details can come out of that21:57
mordredanteaya: I like carl baldwin being teh short term fix to things21:57
mordredmorning jhesketh21:57
* anteaya looks at one-way cross-repo dependencies spec21:57
anteayamordred: he doesn't, fortunately his price is "be nice to me"21:57
*** baoli_ has joined #openstack-infra21:57
mordredanteaya: :)21:58
anteayaso we are going to be extra nice to carl for a bit21:58
anteayamorning jhesketh21:58
anteayathe good news is being nice to carl is easy21:58
*** armax has joined #openstack-infra21:58
*** pblaho has quit IRC21:58
*** Longgeek has joined #openstack-infra21:58
asselinmordred, you wanted to discuss something with me yesterday?21:59
anteayaso how are we marking specs implimented? because I do believe this one is completed:
mordredasselin: I did?21:59
*** baoli has quit IRC21:59
anteayamordred: you did21:59
mordredasselin: that was a long time ago, I'm certain I've forgotten by now ...21:59
anteayamordred: in the meeting yesterday21:59
*** patrickeast has joined #openstack-infra21:59
mordredasselin: oh! right21:59
anteayaI remember22:00
*** wenlock has joined #openstack-infra22:00
mordredI wanted to connect with you on my crazy plans to rework our launch-node script22:00
asselinok....I see. I was wondering what you were talking about.22:00
*** bswartz has quit IRC22:00
asselinso this is the nodepool image-creating script?22:01
mordrednot quite, but it all winds up being related22:01
mordredI'm currently working on using diskimage-builder to build our images in nodepool22:01
jeblairanteaya: we talked about it pre-specs.  sdague was working on it at some point; i think there may be some wip changes in zuul's review queue22:01
*** marun has quit IRC22:01
anteayaah, I will look there, thanks22:02
*** imcsk8 has quit IRC22:02
mordredasselin: once that's all happy, I want to consolidate the code we use to create nodes, because we use different code to create long-lived servers than we do to create nodepool nodes - but most of the cloud logic needs to be very similar22:02
*** imcsk8 has joined #openstack-infra22:02
jeblairanteaya: also, i don't think we have marked any as implemented.  i think there should be a new section/directory for it22:02
anteayathe only spec I was seeing that was close was the nodepool one22:02
anteayajeblair: would you like me to offer a patch for a new dir?22:03
asselinmordred, interesting22:03
*** doug-fish has quit IRC22:03
anteayajeblair: I'm thinking implemented/22:03
sdaguetrying to gather context?22:03
asselinI haven't looked into dib yet, but I guess I should start sooner than later22:04
*** emagana has joined #openstack-infra22:04
*** ryanpetrello has quit IRC22:04
anteayasdague: being able merge one-way cross-repo dependencies22:04
*** Longgeek has quit IRC22:04
jeblairanteaya: sounds great22:04
*** dprince has quit IRC22:04
anteayajeblair: thanks22:04
asselinmordred, quick example of what's different to setup log-lived server and short one for nodepool?22:05
*** annegent_ has quit IRC22:05
sdagueyeh, I got a partial patch, but I ended up abandoning it. I can dig it up22:05
*** dustins has quit IRC22:05
mordredasselin: it's just two differetn code bases22:05
*** bdpayne has quit IRC22:06
mordredasselin: like, we have a script called "launch-node" that we run by hand to add new servers22:06
jeblairsdague: oh, can you explain your last comment there?22:06
mordredasselin: and nodepool has similar but different logic inside of itself to spin up and tear down short-lived ones22:06
jeblairsdague: i think this is actually a really important feature22:06
jeblairi thought we all felt that22:06
asselinmordred, I see. yeah, we're more primite then. we manually launch servers22:07
*** sabari is now known as zz_sabari22:07
mordredasselin: for dib - - if you install latest dib and then grab that patch, the tools/ will build a very small ubuntu image22:07
sdaguejeblair: maybe. It turns out you have to solve a lot of the things that git solves with parents22:07
sdaguemultiply so, and worse, cycles22:07
mordredasselin: yah. well, our long-lived servers are essentially manual - which is one of the things I want to fix22:08
jeblairyeah, we can't have cycles (just kick them out with an error if one is detected)22:08
sdagueso if someone else wants to run with it, that's a starting point22:08
*** jamielennox|away has quit IRC22:08
asselinmordred, I start looking into dib and that patch. thanks!22:08
*** shakamunyi has quit IRC22:08
jeblairsdague: okay, well, "i have decided not to work on this anymore" and "i'm not convinced this is a good idea" are different.22:08
sdagueit's not really on my priority list any more though, because I think that in most cases the right option is to not have that tight commit wise coupling between projects22:09
mordredasselin: cool. I've got to run - but I'll point you at the other work around node spin up when I get back onine22:09
jeblairsdague: i do not think the former requires anything further, but i think the latter requires a conversation.22:09
*** charz has quit IRC22:09
mordredwoot. latest elements official do build both ubuntu and centos base images22:09
*** mattoliverau has quit IRC22:09
*** serverascode___ has quit IRC22:09
*** zz_sabari is now known as sabari22:10
jeblairsdague: this helps with "i'm adding a test of another project" or "i'm intending to use a new in-development feature in another project"22:10
mordredI have a meeting to run to - when I get back, I will test taht those images work on both clouds22:10
clarkbmordred: and we are still running trusty nodepool so we can likely start pushing on dib nodepool images again22:10
sdaguejeblair: right22:10
*** jamielennox|away has joined #openstack-infra22:10
clarkbmordred: tl;dr much excite22:10
pleia2dhellmann: you handled rss for specs, any thoughts on this change when you have a moment?
jeblairsdague: that's still a think we want to do, right?22:10
fungii do feel like the extra barrier we've added with testing against released dep versions has significantly diminished the majority of the need for cross-project review dependencies, but there are still cases where projects are tighthly-coupled and don't release or release in tandem22:10
*** jamielennox|away is now known as jamielennox22:10
*** cloudnull has quit IRC22:10
*** shakamunyi has joined #openstack-infra22:10
mordredclarkb: well, don't get too excited yet22:10
sdaguewell... I'm not convinced thing 2 is a thing we want to encourage22:10
mordredclarkb: I will still need to write the patch to nodepool to properly upload these to both clouds22:10
sdaguebecause it actually makes upgrades quite hard22:10
*** mattoliverau has joined #openstack-infra22:10
jeblairfungi: yeah, i think no amount of wishing in the world is going to cause some of these things not to be tightly coupled22:10
*** zigo has quit IRC22:11
clarkbmordred: ya I have a half dozen nodepool patches that need to get in too22:11
mordredclarkb: which I'm expecting to take a few days to get right22:11
fungisdague: for example, project-config and system-config repos22:11
*** cloudnull has joined #openstack-infra22:11
clarkbmordred: but at least they are not stalled based on where the server is22:11
fungiwe don't release those, and they're tightly coupled22:11
sdagueand thing one I'm actually working to get more of the the tests back owned by the project in question22:11
mordredclarkb: yes22:11
fungiat least sometimes22:11
mordredclarkb: totally22:11
jeblairfungi: or any of our 50 puppet repos22:11
mordredclarkb: do any of your patches affect the openstack api interactions?22:11
sdaguefungi: yeh, I'm not saying there aren't places where this would be useful22:11
*** serverascode___ has joined #openstack-infra22:11
sdaguebut I also see some draw backs22:12
*** mrmartin has quit IRC22:12
*** zigo has joined #openstack-infra22:12
clarkbmordred: no, but one of them does interact with dib to do multi image output. you may want to coopt that one22:12
*** armax has quit IRC22:12
*** bdpayne has joined #openstack-infra22:12
sdaguewhich leaves me at, I'm fine if someone else wants to do it (like I won't argue against it)22:12
fungijeblair: i could get on board with saying we're going to make releases of puppet modules and do so before we consume new features from them, which is why i didn't go there with the example ;)22:12
mordredI just wanted to make sure I don't step on you as I start looking at api operation changes22:12
jeblairsdague: okay, thank you for the work you have done so far22:12
*** axelddd has joined #openstack-infra22:12
*** charz has joined #openstack-infra22:12
sdaguewhat's there was about 95% of the base case IIRC22:13
sdaguewith just 1 test that wouldn't pass22:13
sdagueso hopefully a starting point for someone22:13
dougwighi infra, can I get another core to look at this devstack change to re-enable neutron with services?  it has one +2 and has passed jenkins.  thank you.22:15
*** r-daneel has quit IRC22:16
*** axelddd has quit IRC22:16
fungichmouel: dtroyer: ianw: ^22:16
sdaguemtreinish also has +2 now22:16
*** andreaf has quit IRC22:17
jeblairfungi: our testing of that would have to be fanstastic; and that's a long ways off22:17
*** andreaf has joined #openstack-infra22:17
fungijeblair: true22:17
jeblairbut yeah, in principle :)22:18
*** yamamoto has joined #openstack-infra22:19
jeblairclarkb, fungi: whichever one of you feels more affinity with subunit2sql, can you review 106584?22:19
anteayaclarkb: you able to address ?22:20
clarkbanteaya: ya now that nodepool is mostly done I should do that22:20
clarkbI am going to disable puppet on review.o.o now, rename that group, then merge the change and reenable puppet22:21
clarkbfungi: jeblair ^ fyi22:21
anteayaclarkb: thanks22:21
*** emagana has quit IRC22:21
fungiclarkb: k22:22
*** emagana has joined #openstack-infra22:22
SergeyLukjanovjeblair, clarkb, fungi, mordred, are you folks ok with setting up storyboard-dev instance to run drafts on top of it?22:22
SergeyLukjanov ^^22:22
SergeyLukjanovif yes, I'd like to mode forward and setup it tomorrow22:23
fungimordred: yeah, i've been working with krotscheck on that front22:23
fungier, SergeyLukjanov ^22:23
fungisorry mordred22:23
jeblairSergeyLukjanov: you mean to have docs-draft use storyboard-dev...22:23
SergeyLukjanovjeblair, yeah22:23
dtroyerdougwig, fungi, +A on 14057222:23
SergeyLukjanovbad wording again, heh22:23
jeblairyeah, makes sense22:23
mtreinishsdague: I need to blindly +A something?22:23
*** yamamoto has quit IRC22:24
krotscheckSergeyLukjanov: We’re waiting on recheck for
dtroyermtreinish: not today, I beat you to it…22:24
mtreinishfungi: what's the fungi-proxy IRC syntax? I've got a query for jogo:
fungiSergeyLukjanov: it helps mitigate a potential security leak where, with some social engineering, you could convince a storyboard admin to test out a malicious front-end which made api calls using their credentials22:24
clarkbanteaya: its renamed and I have approved the change. will reenable puppet once that merges22:24
SergeyLukjanovkrotscheck, fungi, so, no need in assistance?22:24
SergeyLukjanovfungi, exactly, that's why I'd like to move it forward22:25
fungimtreinish: i accept a very loosely-defined human language grammar22:25
anteayaclarkb: thanks armax is happy22:25
fungiSergeyLukjanov: assistance is always welcome22:25
*** armax has joined #openstack-infra22:25
*** andreykurilin_ has joined #openstack-infra22:25
SergeyLukjanovfungi, okay, so, I'd like to try setting up the node for it tomorrow if it'll have +2 from someone else :)22:26
dougwigdtroyer: thank you!22:26
*** teran_ has joined #openstack-infra22:26
*** emagana has quit IRC22:26
*** mestery has joined #openstack-infra22:26
fungiSergeyLukjanov: sure. i'm good with that if krotscheck is22:26
SergeyLukjanovfungi, I've been already doing it on our internal infra with launch_node, so, I think that I know how to do it :)22:26
krotscheckWorks for me.22:27
fungiSergeyLukjanov: i'll look over that review here in a few minutes then22:27
mtreinishfungi: oops actuall I made a mistake in that link, one sec22:27
*** wenlock has quit IRC22:27
*** teran has quit IRC22:27
*** ddieterly has quit IRC22:28
fungimtreinish: okay, i'm connected to the db now so can run whatever query you need once you finalize it22:28
SergeyLukjanovfungi, thx22:28
SergeyLukjanovfungi, oh, the only question I don't know answer for - is how to issue the ssl cert for it?22:29
*** dkliban is now known as dkliban_afk22:29
jogois a patch to stable22:29
*** emagana has joined #openstack-infra22:29
jogobut the test names being run don't have juno in the names22:29
fungiSergeyLukjanov: we'd want to use a self-signed cert. easiest to just point it at the snakeoil cert already on the server like we do with review-dev, et al22:29
clarkbjogo: correct22:30
SergeyLukjanovfungi, ack22:30
mtreinishfungi: ok, give this a try:
*** mfisch has quit IRC22:31
*** obondarev has quit IRC22:31
fungimtreinish: it's running now22:31
clarkbjogo: d-g by default sets up devstack to deploy whatever ZUUL_BRANCH version is of things22:31
clarkbjogo: when we put juno/icehouse in a job name that implies we are overriding d-g defaults and saying install ZUUL_BRANCH of everything but the current change22:32
jogothat makes it a little harder to detect when stable branches are broken though22:32
jogobut makes sense22:32
*** yjiang5_away is now known as yjiang522:32
openstackgerritMerged openstack-infra/project-config: Fix ACL groups for networking L2GW
clarkbjogo: so you see that with branchless tempest beacuse tempest only has master but needs to test against juno22:32
mtreinishfungi: fwiw, if the test is working it should have 0 hits and jogo will close the bug :)22:32
clarkbjogo: how so?22:32
jogoclarkb: in graphite22:32
clarkbjogo: the branch info should still be available. gerrit knows it. zuul provides it to jenkins whihc provides it to logstash22:32
*** Guest76485 has joined #openstack-infra22:32
clarkbjogo: hrm ya graphite job name based22:33
jogoclarkb: right, its not that big of a deal though22:33
*** Guest76485 is now known as mfisch22:33
clarkbjogo: though zuul knows it and reports the graphite data. it may actually split by branch22:33
fungimtreinish: jogo: Empty set (2 min 20.11 sec)22:33
*** tnovacik has quit IRC22:34
jogojeblair: good point, the periodic-stable queue22:34
clarkbdid mordred ever review the mysql proxy change? we should feed him pizza and beer until he does that22:34
mtreinishfungi: awesome thanks22:34
mtreinishclarkb: no, mordred hasn't done that yet22:35
*** nelsnelson has quit IRC22:35
jogofungi: woot, thanks22:35
mtreinishclarkb: I also want to start working on my LCA talk on it, but I'm kinda blocked because I can't get to the data...22:35
mtreinishfungi: 2min seems kinda slow, while you still have things open, how big are things now?22:36
clarkbjeblair: fungi jhesketh SergeyLukjanov any opposition to making mordred do post merge review?22:36
*** baoli_ has quit IRC22:37
*** baoli has joined #openstack-infra22:38
*** dannywilson has joined #openstack-infra22:38
jeblairclarkb: fyi i negated some of your early comments on 129255  :(22:38
*** dannywilson has quit IRC22:38
*** jedimike has joined #openstack-infra22:39
*** cnesa2 has quit IRC22:39
*** dannywilson has joined #openstack-infra22:39
*** tonytan4ever has quit IRC22:39
*** marun has joined #openstack-infra22:40
jeblairclarkb: i feel like mordred is mostly back from his being very afk recently though is at a meeting right now.... could we give him a couple more hours?22:40
fungimtreinish: specific count queries you want me to run?22:40
clarkbjeblair: sure22:40
jedimikejeblair, hi, thanks for your review of the fail-fast spec, with your feedback I'm not sure if it's something I should pursue, although clarkb and sdague were keen on it and it would help us in gozer with our limited resources. What do you think?22:40
clarkbjeblair: and re 129255 thats fine. I don't know that I was super happy with what I had come with either, but wanted to drive the discussion to keeping the yaml as a datastructure that made sense structurally22:40
mtreinishfungi: yeah if you don't mind running a count on the runs, test_runs, and run_metadata tables22:41
clarkbjeblair: oh good point on making it a job attribute22:41
clarkband yes in geneal I agree. Fail fast is a thing we attempted and it never really panned out so not entirely convinced it will help22:42
openstackgerritAnita Kuno proposed openstack-infra/infra-specs: Adds an implemented directory
jeblairjedimike: i had meant to say that i think it is something that you should pursue; i may not have communicated it that well.... let me try again :)22:42
*** groknix has joined #openstack-infra22:42
*** emagana has quit IRC22:42
clarkbbut I also don't think zuul should refuse to add the feature22:43
clarkbmaybe that is what jeblair meant22:43
*** emagana has joined #openstack-infra22:43
jeblairjedimike: i think it is a fine feature to have in zuul.  people keep asking for it, and it shouldn't be too hard to do, relatively speaking.  i think many of the times people think that it would be totally awsome they haven't thought through all the issues, so it's not a silver bullet for some of the problems people want to apply it to.22:43
*** hdd has quit IRC22:43
*** armax has quit IRC22:44
jeblairjedimike: for that reason, it's probably not going to be our highest priority review item -- so you should expect slower than normal review turnaround times22:44
jeblairjedimike: but i'd be very happy to have the feature, and if you want to proceed, addressing the specific comments i marked with (-1) is the way to go, i think22:44
*** mriedem has quit IRC22:44
openstackgerritAnita Kuno proposed openstack-infra/infra-specs: Adds an implemented directory
*** craigbr has quit IRC22:44
fungimtreinish: 27421 runs, 24074155 test_runs, 329052 run_metadata22:44
fungimtreinish: count on test_runs took the longest, at 1 min 39.44 sec22:45
jedimikejeblair, ah ok :) I was mainly thinking about our situation in gozer where resources are quite limited, it would certainly help there, and I'd be happy to start the development myself, seems like a good project to cut my zuul teeth on22:45
jedimikei'll work on a new patch with your suggestions22:45
jeblairjedimike: absolutely!  and thanks.  :)22:45
mtreinishfungi: well at 24 million tests it is kinda big :)22:45
openstackgerritKhai Do proposed stackforge/python-jenkins: Add a default http timeout for connections to jenkins
mtreinishI would have expected it to be a bit bigger though22:46
jeblaironce it exists, mordred, sdague, clarkb, and i can then all argue about where we should and should not use it in openstack.  which will be lots of fun for us.  ;)22:46
clarkbya 27k runs seems low22:46
*** bdpayne has quit IRC22:46
clarkbfungi: no the first number22:46
fungioh, that22:46
clarkbwhich is 27k tempest jobs aiui22:46
fungimight want to check that against something in graphite i guess22:47
mtreinishclarkb: yeah 27k tempest jobs in the gate since  we started this, back around summit seems kinda low22:47
clarkboh its just gate? that may actually be accurate then22:47
clarkbbut ya we can check against graphite22:47
*** jedimike has quit IRC22:47
*** emagana has quit IRC22:47
mtreinishclarkb: well it was gate with poor filtering until: landed22:48
*** cnesa has joined #openstack-infra22:48
jeblairmtreinish: also, it's not even january and you want to work on your lca talk already? ;)22:49
mtreinishfungi: one more thing can checkthe most recent run in the runs table22:49
sdaguejeblair: I believe we should have that conversation in a closed room with a few bottles of whiskey22:49
*** wenlock has joined #openstack-infra22:49
jeblairsdague: i was thinking the same thing :)22:49
sdagueand whoever comes out standing declares victory22:49
*** esker has quit IRC22:50
jeblairi will start practicing now22:50
jeblairTo Whiskey!22:50
sdaguewell I'm about to go to a brewery22:50
jeblairTo Beer!22:51
*** Sukhdev has joined #openstack-infra22:52
mtreinishjeblair: yeah, it's unusual for me. I'm sure I'll put most of it off until the plane :)22:52
*** mattoliverau has quit IRC22:53
*** mattoliverau has joined #openstack-infra22:53
clarkbmtreinish: that is what 29 hours in a plane is good for22:53
mtreinishI just want to make sure I have enough time to do something with the data22:53
mtreinishclarkb: heh, I'm normally sufficiently uncomfortable squeezed into my seat on a plane using a laptop can be difficult sometimes.22:55
fungimtreinish: 2014-12-10 22:53:0322:55
*** mmaglana has quit IRC22:55
*** markmcclain has joined #openstack-infra22:56
mtreinishfungi: thanks, it looks like it's keeping up22:56
fungirunning the query a few times, seems to be ~3min behind present22:57
*** cpowell has quit IRC22:57
openstackgerritIan Wienand proposed openstack-infra/system-config: Bring up Fedora 21 nodes on rackspace
*** otter768 has joined #openstack-infra22:58
fungioh, heads up i'm going to be absent the first half of tomorrow for some appointments22:58
*** pc_m has quit IRC22:58
openstackgerritJoe Gordon proposed openstack-infra/elastic-recheck: Remove queries for closed bugs
mtreinishfungi: yeah, I'm still surprised it's keeping up. The test_run inserts got slow in my local testing after a while22:58
clarkbI will be around. Though I am likely out for much of friday and monday22:59
*** pc_m has joined #openstack-infra22:59
*** vigneshvar has quit IRC22:59
*** dimtruck is now known as zz_dimtruck23:00
openstackgerritMerged openstack-infra/infra-specs: Remove Modulefile from puppet-module spec
anteayayes the view of poor mtreinish squished into economy is a sight23:01
anteayafungi: happy appointments23:01
openstackgerritKhai Do proposed stackforge/python-jenkins: Add a default http timeout for connections to jenkins
*** tnovacik has joined #openstack-infra23:01
clarkbanteaya: starting puppet on review.o.o now23:01
*** SumitNaiksatam has quit IRC23:01
clarkbor enabling it23:01
anteayaI'll be offline all tomorrow with travel expecting to be online Friday23:01
anteayaclarkb: kk23:01
clarkbso next puppet run will add the release group23:01
anteayacool too23:02
anteayaso 20 minutes ish23:02
*** otter768 has quit IRC23:02
anteayak great thank you23:02
anteayabtw anyone who hasn't yet had the pleasure of working with armax, I recommend it23:02
*** craigbr has joined #openstack-infra23:04
openstackgerritMerged openstack-infra/infra-specs: Workflow documentation is now in infra-manual
openstackgerritKhai Do proposed stackforge/python-jenkins: Add a default http timeout for connections to jenkins
*** wenlock has quit IRC23:05
*** dkliban_afk is now known as dkliban23:06
*** ianw has quit IRC23:06
*** ianw has joined #openstack-infra23:06
*** Rockyg has joined #openstack-infra23:07
*** tnovacik has quit IRC23:07
*** zz_avozza is now known as avozza23:08
jeblairclarkb, fungi: also, heads up, we are forecast to get a severe rainstorm tomorrow, so i might have power outages, flooding, etc to deal with23:10
fungijeblair: noted--thanks23:10
jeblairwe lose power here in storms that would be considered "a gentle spring rain" in NC.23:10
*** bdpayne has joined #openstack-infra23:10
jeblairbut 3-4 inches in not insignificant anywhere23:11
fungiwell, you are just a little ways north of a partially-reclaimed desert23:11
*** wenlock has joined #openstack-infra23:11
*** annegent_ has joined #openstack-infra23:11
*** esker has joined #openstack-infra23:11
*** emagana has joined #openstack-infra23:11
anteayajeblair: stay dry23:12
jeblairfungi: oddly enough, desert communities are usually better equipped to deal with torrential rain than we are...23:12
anteayajeblair: any risk to property?23:12
jeblairsomething about having hundreds of square miles of catch basin funneling it all toward you...23:12
fungigood point, when they get rain they get _hard_ rain23:12
*** resker has joined #openstack-infra23:12
fungiand major flash flooding23:12
*** asettle-bfast is now known as asettle23:12
jeblairanteaya: yeah, but probably less than the riots and looting that's been going on the last few days23:13
anteayastay safe from the riots23:13
anteayaare you in a high risk area?23:13
pleia2I avoided an event on monday in berkeley because of it :\ probably for the best, bart was expressing through some downtown stations anyway, not sure I would have made it anyway23:14
jeblairthey've generally been focusing on downtown business districts and highways23:14
*** thedodd has joined #openstack-infra23:14
*** groknix has quit IRC23:14
*** belmoreira has quit IRC23:14
anteayaglad that means you are lower risk, but still :(23:14
jeblairpleia2: yeah, would have been tricky.  i think you have to have up to the minute info to plan trips like that now23:15
anteayaand yay neutron is now unfrozen
anteayamestery and dougwig are very grateful for the help23:16
jeblair"avoid areas with helicopters overhead" is a good rule of thumb once you're over here; doesn't help planning a bart trip tho23:16
anteayathanks so much23:16
*** esker has quit IRC23:16
mesteryIndeed, thank you to all in infra for their awesome help this week!23:16
pleia2heh, no, there were helicopters here too and our protests stopped mostly23:16
mesteryI feel like we had great team work this week, it was really awesome!23:16
anteayamestery: yay23:16
jeblairyay people who worked on this!  :)23:17
mesteryIt was truly epic to be a part of jeblair. :)23:17
anteayaI find it epic to be a part of jeblair23:17
jeblairspoken like my right hand23:18
mesteryOK, I've got to run and catch my flight soon. I'll be back online tomorrow, thanks again folks!23:18
anteayaha ha ha23:18
anteayamestery: happy trails23:18
*** amitgandhinz has quit IRC23:18
mesteryYou too anteaya!23:18
*** mestery has quit IRC23:18
*** wenlock has quit IRC23:20
*** thedodd has quit IRC23:20
bknudsondoes the CI use MyISAM MySQL engine?23:21
bknudsonwondering about this commit message:
jeblairbknudson: we explicitly set the default to myisam because it's the wrong choice for openstack...23:22
jeblairbknudson: that way, programs that assume that the default is innodb but do explicitly set it break23:22
*** marun has quit IRC23:23
*** groknix has joined #openstack-infra23:23
bknudsonjeblair: ohhh... guess we need better testing.23:23
bknudsonI think better testing is coming.23:23
*** groknix has quit IRC23:23
*** craigbr has quit IRC23:23
jeblairbknudson: yeah, i guess if you haven't noticed it till now, you're probably not exercising it...23:23
jeblairit was a big problem in nova early on23:23
jeblairthus the counter-intuitive default setting23:24
*** kumartin has joined #openstack-infra23:24
bknudsonit hit us in keystone too... but then new tables get added and we forgot, and didn't test well on real system.23:24
jeblairat any rate, it should definitely not be taken as an endorsement of myisam -- it's just there so that our test systems imitate a "lazy sysadmin"23:24
bknudsonit's the default in older RHEL... I was just surprised to see that CI was forcing it.23:25
*** ayoung is now known as ayoung_dad_mode23:26
jeblairi'll leave a comment in that change letting people know why23:26
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: Revert "Temporarily disable testing lbaas, fwaas, and vpnaas"
*** signed8bit is now known as signed8bit_ZZZzz23:27
*** kmartin has joined #openstack-infra23:27
*** ChuckC has quit IRC23:28
*** dizquierdo has quit IRC23:28
*** ryanpetrello has joined #openstack-infra23:29
*** whoops has quit IRC23:29
*** pradk has quit IRC23:29
*** r-daneel has joined #openstack-infra23:33
bknudsonjeblair: thanks.23:37
*** r-daneel has quit IRC23:38
clarkbanteaya: arg we missed one... oh well23:38
clarkbthats on me. cleaning it up now23:39
*** Masahiro has joined #openstack-infra23:39
mtreinishdougwig: I'm assuming we should be able to do: now23:40
openstackgerritClark Boylan proposed openstack-infra/project-config: Stop using networking-l2gw group
clarkbanteaya: ^23:40
anteayawhat did I msiss?23:40
anteayasorry about that23:41
*** doug-fish has joined #openstack-infra23:43
*** Masahiro has quit IRC23:43
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu
*** doug-fish has left #openstack-infra23:44
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Allow root user to ssh in as stack
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Enable live block migration
clarkbanteaya: no problem. I "deactivated" that group and that change should make everything go23:45
clarkbalso seeded release group membership23:45
*** ildikov_afk has quit IRC23:45
*** craigbr has joined #openstack-infra23:46
*** ildikov_afk has joined #openstack-infra23:46
*** resker has quit IRC23:47
clarkbjeblair: my lights just flickered. we are getting bad weather too23:49
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: deprecate postbuildscript onsuccess and onfailure parameter names
*** r-daneel has joined #openstack-infra23:50
*** ryanpetrello has quit IRC23:50
*** dannywilson has quit IRC23:51
*** sabari is now known as zz_sabari23:51
*** zz_sabari is now known as sabari23:51
*** sabeen has quit IRC23:51
*** annegent_ has quit IRC23:51
*** esker has joined #openstack-infra23:52
*** dannywil_ has joined #openstack-infra23:54
dougwigmtreinish: driving back to Boise right now.  let me sanity test before we revert.23:55
*** dannywil_ has quit IRC23:55
*** esker has quit IRC23:56
mtreinishdougwig: ok sure, but won't it self test. (ie the gate will kick it out if we missed something)23:56
mtreinishanyway we can pick it up tomorrow23:56
*** SumitNaiksatam has joined #openstack-infra23:56
*** esker has joined #openstack-infra23:57
*** cnesa has quit IRC23:57
anteayaclarkb: kk, thanks23:58
*** SumitNaiksatam has quit IRC23:58
openstackgerritSteve Baker proposed openstack-infra/project-config: Make check-heat-dsvm-functional-mysql voting
*** straycat has left #openstack-infra23:58
*** esker has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!