Tuesday, 2014-10-28

fungiconfirmed heat-translator has indeed fixed their py34 unit test run. also the dev who picked up the python-heatclient bug has a stack of patches up to rework their mocks, though i haven't tested them quite yet00:00
*** MaxV has joined #openstack-infra00:00
fungino new movement on the two open ubuntu sru requests though, nor any signs of activity for the python-glanceclient bug00:00
clarkbprobably hung over from unicorn release00:00
* clarkb hopes next week will be better00:00
fungiwe'll be too hung over to notice00:01
*** msabramo has quit IRC00:02
anteayaianw: not so far, can we get it as an agenda item on tomorrow's meeting agenda?00:03
anteayaianw: I won't be here for the meeting but will go by what the group decides00:03
ianwanteaya: yep.  hopefully there is time :)00:04
*** MaxV has quit IRC00:05
anteayaI would like to hear more from nibalizer about puppet module names00:06
anteayaI'm still leaning toward getting rid of the underscore00:06
*** mmaglana has quit IRC00:06
nibalizeranteaya: so i dont like puppet-openstackci_apache either really00:09
nibalizerbut i dislike openstackciapache much more00:09
nibalizerother modules have underscores https://forge.puppetlabs.com/openshift/openshift_origin/dependencies00:10
nibalizeropenstackci_apache gives us a name that can be grepped on 'openstackci' 'apache' and can be split on '_'00:12
nibalizerand most of the interactions with it will be in the puppet code where the 'puppet-' prefix won't be visible00:12
mordredwhat about just calling it samuel00:12
mordredlet's bring back meaningless names!!!!00:13
waynrthis seems like a long shot but is there any way to interpolate the names of jobs in a job group in another part of the configuration?00:13
*** stevemar has joined #openstack-infra00:13
*** stevemar is now known as notstevemar00:13
waynr(jenkins job builder)00:14
waynri have a bunch of jobs that use the same template whose names are of course parameterized--it'd be nice not to have to put them into a buildstep trigger field manually00:14
lifelessdhellmann: have you tried virtualenvwrappers with jython?00:15
lifelessdhellmann: I get, at the end of mkvirtualenv,00:15
lifelessERROR: None00:15
lifelessERROR: this script requires Python 2.6 or greater.00:15
lifelessERROR: None00:15
lifelessERROR: this script requires Python 2.6 or greater.00:15
lifelesserm, just one of those :)00:15
*** oomichi has joined #openstack-infra00:17
*** unicell has joined #openstack-infra00:20
*** weshay has joined #openstack-infra00:24
*** msabramo has joined #openstack-infra00:24
ianw10 is the new cool version to be? puppet-apache1000:29
fungiversion 10 is so done even openstack has one. this needs to go to eleven00:29
*** zaphod42_ has joined #openstack-infra00:32
*** achanda has quit IRC00:32
*** zaphod42_ has joined #openstack-infra00:32
reedI think Samuel is a good name: In American the meaning of the name Samuel is: Heard of God; asked of God. (source:google)00:32
*** achanda has joined #openstack-infra00:33
*** amcrn has quit IRC00:33
*** ayoung-dadmode is now known as ayoung00:33
clarkbso rax-iad is predominantly preferred for 2-node builds00:35
*** zaphod42 has quit IRC00:35
clarkbI think the allocator might be built so that this happens00:35
lifelessunittest2 0.6.0 just released, wheel and sdist formats. If I've broken the world, please tell me soon:)00:35
clarkbwe should know as soon as bandersnatch can sync them :)00:36
*** zaphod42_ has quit IRC00:37
*** achanda has quit IRC00:37
openstackgerritAngus Lees proposed a change to openstack/requirements: Allow newer versions of pylint  https://review.openstack.org/13101300:39
openstackgerritAngus Lees proposed a change to openstack/requirements: Upgrade to pylint >=1.3.0  https://review.openstack.org/13131000:39
*** rkukura has quit IRC00:39
*** rmcall has quit IRC00:41
*** yamamoto has joined #openstack-infra00:41
anteayafungi: puppet-opache ++00:42
anteayamordred: samuel +00:43
fungiand failing that, i also like puppet-gazpacho00:43
anteayanibalizer: ewwww, but you do make a point00:43
anteayaI also like puppet-gazpacho00:43
anteayayou can grep on pach00:44
*** pc_m_ has quit IRC00:45
fungiwow, i didn't realize that appledouble encoding dates all the way back to a/ux00:45
fungithe things you learn on the 'pedia00:45
*** yamamoto has quit IRC00:46
*** david-lyle has joined #openstack-infra00:46
*** david-lyle has quit IRC00:51
*** sarob has joined #openstack-infra00:51
*** r1chardj0n3s_afk is now known as r1chardj0n3s00:51
*** sarob has quit IRC00:52
*** sarob has joined #openstack-infra00:53
*** yamahata has joined #openstack-infra00:53
*** koolhead17 has quit IRC00:54
*** sarob has quit IRC00:58
lifelessfungi: aieeee01:01
lifelessfungi: flashbacks01:01
*** rkukura has joined #openstack-infra01:04
*** Iron_Chef has joined #openstack-infra01:07
*** mriedem has quit IRC01:08
Iron_ChefHi openstack :)01:09
*** freemanbrandon has joined #openstack-infra01:10
anteayahello Iron_Chef01:12
*** Longgeek has joined #openstack-infra01:13
*** baoli has joined #openstack-infra01:16
*** julim has quit IRC01:17
*** Longgeek has quit IRC01:17
*** Iron_Chef has left #openstack-infra01:18
*** amitgandhinz has joined #openstack-infra01:24
*** signed8bit has joined #openstack-infra01:26
clarkbwell I cannot get 2-node builds in not rax01:27
clarkbso I am just going to wait on that for when nodepool is busy and allocator will possibly do what I want01:27
*** david-lyle has joined #openstack-infra01:31
anteayaclarkb: :(01:32
ianwclarkb: sorry, i had it in my head that puppet would restart nodepool on the git change.  but it's nodepool that checks itself for updated config files01:33
ianwtoo many automagically moving pieces :)01:34
fungiclarkb: or tweak the config temporarily to only build them in hpcloud long enough to build one, then switch it back01:35
jogoclarkb: you are testing out the two node build stuff?01:35
jogoclarkb: is there an etherpad or something tracking the pieces of that01:35
*** david-lyle has quit IRC01:36
clarkbjogo: no etherpad, just the change and its children01:36
clarkbjogo: it works fine01:36
clarkbtrying a specific variation of it (or attempting to)01:36
clarkbfungi: that is, hrm I should totally do that01:36
clarkbI am going to do that now01:36
fungislightly faster than rolling loaded dice over and over01:37
jogoclarkb: cool, so we can run any tempest config with this?01:40
clarkbjogo: I mean I don't expect them to all work01:40
clarkbright now its just tempest + nova net and tempest + neutron and not full tempest best I can tell01:40
jogoclarkb: don't expect all configs to work or not TEMPEST_FULL01:40
jogoclarkb: yeah your probably right, TEMPEST_FULL may have some kinks to sort out before we can try adding things like live migration01:41
clarkbwoot fungi++ it is building an hpcloud pair now01:42
clarkbI will recheck as soon as I have at least one set of nodes ready01:42
jogoclarkb: have you gotten far enough to run tempest on a two node?01:43
jogohmm I wonder if I can simulate with sdague's two node devstack vagrant thing01:43
clarkbyes afazekas_beer has had tempest running on two nodes for months01:44
clarkbjogo: https://review.openstack.org/#/c/106043/2201:44
jogoclarkb: ohh aiopcpu01:44
jogowow that is super cryptic01:45
jogomight as well call it ieaiaio01:45
jogowhich is a system of a down song01:45
clarkball in one plus cpu01:45
clarkbnot that I am a fan01:45
clarkbbut meh important part is making it go01:45
clarkband it goes01:45
jogoclarkb: agreed01:46
* jogo listens to system of a down01:46
clarkbsystem of a down will get me too excited01:46
*** signed8bit has quit IRC01:47
jogoclarkb: haha01:48
jogoso this isn't tempest-full01:48
clarkbno I don't think so01:48
jogowould it make sense to get one of those jobs added to so we can work on sorting out the kinks in that in parallel?01:49
clarkbI don't understand the question we have two of them01:49
jogoyeah that didn't make sense, let me try again01:49
*** Ryan_Lane has quit IRC01:49
openstackgerritA change was merged to openstack-dev/hacking: Add hacking check H105: don't use author tags  https://review.openstack.org/12699201:50
jogothoughts on adding a tempest-dsvm-full-aiopcpu job, and maybe a tempest-dsvm-neutron-full-aiopcpu as well, to shake out any tempest related issues in parallel to the current work01:51
clarkbI would like to get the base code merged first so that we can run those jobs against arbitrary changes01:51
*** weshay has quit IRC01:51
clarkbI think we are really close at this point (I am testing what I believe is the last point of contention)01:52
*** amitgandhinz has quit IRC01:52
jogoclarkb: cool, I wasn't sure how far away we were since https://review.openstack.org/#/c/106043/22 has a few TODOs in the commit message01:53
clarkbI think we can work on those todos in parallel with expanding testing01:53
clarkbhttps://jenkins02.openstack.org/job/check-tempest-dsvm-aiopcpu/14/ finally that is running on hpcloud01:55
clarkbonce the other job starts I will edit the nodepool.yaml and reenable puppet on nodepool.o.o01:55
clarkbjogo: also the neutron job takes much longer. Maybe it is running full?01:56
clarkbjust slower I guess01:56
*** nosnos has joined #openstack-infra01:59
clarkbI have returned nodepool to its normal state. puppet is enabled and yaml file matches git02:04
*** tsg has joined #openstack-infra02:06
clarkbthere are unwritten changes to he buffer02:07
*** freemanbrandon has quit IRC02:11
anteayajogo: you can never leave02:12
*** baoli has quit IRC02:13
*** rfolco has quit IRC02:19
*** VijayTripathi has quit IRC02:19
*** VijayTripathi has joined #openstack-infra02:20
*** bhuvan has quit IRC02:20
jogoanteaya: noooo02:21
jogoclarkb: how many n-cpus should that aiopcpu have?02:21
jogoI don't see two logs02:21
clarkbjogo: I think that is the todo fix logging02:21
clarkbjogo: but there should be two let me look02:22
*** bdpayne has joined #openstack-infra02:22
*** bdpayne has quit IRC02:22
clarkbhttp://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/692a986/logs/ and http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/692a986/logs/ each have an n-cpu log02:22
jogoohhh I see02:23
jogoI didn't realize that is how it was split up02:23
jogothat isn't very obvious02:23
jogologs and logs/$IP/02:23
*** notstevemar has quit IRC02:24
*** VijayTripathi has quit IRC02:24
*** emagana has joined #openstack-infra02:26
*** ivar-laz_ has joined #openstack-infra02:30
clarkbwe do have volume related failures when using the floating ips in hpcloud02:31
clarkbnot sure yet if due to floating IPs or hpcloud /me checks the base change02:31
*** rfolco has joined #openstack-infra02:33
*** ivar-lazzaro has quit IRC02:34
*** david-lyle has joined #openstack-infra02:36
*** ivar-laz_ has quit IRC02:39
*** david-lyle has quit IRC02:41
*** dims_ has quit IRC02:44
*** dims__ has joined #openstack-infra02:47
jogoif you could run E-R against the logs it would be able to tell you  pretty quickly02:48
jogoas I don't think we have any unknown failures02:48
jogoI spoke too soon, we didn't at lunch time02:48
jogowell this is fun GnuTLS recv error (-9): A TLS packet with unexpected length was received.02:49
*** dims__ has quit IRC02:49
clarkbwell this is completely brand new02:49
clarkbit would likely not be able to tell me if its node slowness or NAT or some other flavor variety between clouds02:50
jesusaurusheh, thats the same error i get from glowing bear when i try to connect my weechat relay </not_helpful>02:50
jogoclarkb: wow 13 hits in 24 hours02:51
* jogo is afraid02:51
jogohmm a lot of them on successfull runs02:52
jogooh right we do a retry02:52
*** patrickeast has quit IRC02:54
*** jerryz has joined #openstack-infra02:54
*** liusheng has quit IRC02:55
mwhudsoni'm seeing tests fail (locally) with ResponseNotReady02:55
mwhudsonfrustratinly i know i've seen this before, but can't remember what it means :(02:55
*** liusheng has joined #openstack-infra02:55
*** yamamoto_ has joined #openstack-infra02:56
*** sarob has joined #openstack-infra02:56
*** timrc is now known as timrc-afk02:59
*** sarob has quit IRC03:01
*** amitgandhinz has joined #openstack-infra03:03
*** emagana has quit IRC03:03
clarkbok the base aiopcpu change works on hpcloud03:04
clarkbjeblair: so the fixed IP thing may actually be required03:04
*** emagana has joined #openstack-infra03:04
*** spzala has joined #openstack-infra03:04
*** alexpilotti has quit IRC03:04
*** alexpilotti has joined #openstack-infra03:06
*** alexpilotti has quit IRC03:07
*** amitgandhinz has quit IRC03:07
*** unicell has quit IRC03:08
*** emagana has quit IRC03:09
*** mmaglana has joined #openstack-infra03:10
*** alexpilotti has joined #openstack-infra03:10
*** sunil_ has quit IRC03:11
*** alexpilotti has quit IRC03:11
*** alexpilotti has joined #openstack-infra03:12
*** alexpilotti has quit IRC03:12
*** alexpilotti has joined #openstack-infra03:14
*** armax has quit IRC03:14
*** spzala has quit IRC03:15
*** armax has joined #openstack-infra03:15
*** liusheng has quit IRC03:15
*** unicell has joined #openstack-infra03:16
*** alexpilotti has quit IRC03:16
*** liusheng has joined #openstack-infra03:17
*** ChuckC___ has joined #openstack-infra03:18
*** Longgeek has joined #openstack-infra03:21
*** freemanbrandon has joined #openstack-infra03:22
*** camunoz is now known as camunoz_mtg03:25
*** emagana has joined #openstack-infra03:25
*** Longgeek has quit IRC03:26
*** nelsnelson has quit IRC03:26
*** freemanbrandon has quit IRC03:26
*** ChuckC___ has quit IRC03:27
openstackgerritBrian Saville proposed a change to openstack-infra/jenkins-job-builder: Add support for multiple port allocations  https://review.openstack.org/13132803:27
*** harlowja is now known as harlowja_away03:27
*** doug-fish has joined #openstack-infra03:29
openstackgerritA change was merged to openstack/requirements: Update to newer requests-mock  https://review.openstack.org/13106403:30
*** harlowja_away is now known as harlowja03:30
*** tomoe has joined #openstack-infra03:32
*** mbacchi has quit IRC03:32
*** MaxV has joined #openstack-infra03:33
*** mattoliverau has quit IRC03:35
*** zns has joined #openstack-infra03:35
*** mattoliverau has joined #openstack-infra03:35
*** rmcall has joined #openstack-infra03:35
openstackgerritBrian Saville proposed a change to openstack-infra/jenkins-job-builder: Add doxygen builder support  https://review.openstack.org/13133003:35
*** nosnos has quit IRC03:36
*** nosnos has joined #openstack-infra03:37
*** MaxV has quit IRC03:37
*** mattoliverau has quit IRC03:38
*** miqui has quit IRC03:38
*** mattoliverau has joined #openstack-infra03:39
openstackgerritBrian Saville proposed a change to openstack-infra/jenkins-job-builder: Add SVN credential support  https://review.openstack.org/13133203:39
*** pcrews has quit IRC03:41
*** nosnos has quit IRC03:41
*** terryw has quit IRC03:43
openstackgerritBrian Saville proposed a change to openstack-infra/jenkins-job-builder: Add support for multiple port allocations  https://review.openstack.org/13133503:46
*** otherwiseguy has joined #openstack-infra03:50
*** dims__ has joined #openstack-infra03:50
*** VijayTripathi has joined #openstack-infra03:51
*** VijayTripathi1 has joined #openstack-infra03:54
*** dims__ has quit IRC03:55
*** VijayTripathi1 has quit IRC03:57
*** VijayTripathi has quit IRC03:57
*** achanda has joined #openstack-infra04:01
*** ayoung has quit IRC04:04
*** zns has quit IRC04:06
*** david-lyle has joined #openstack-infra04:06
*** zns has joined #openstack-infra04:06
*** gyee has joined #openstack-infra04:11
*** AzherKhan has joined #openstack-infra04:14
*** nosnos has joined #openstack-infra04:15
*** Khaazher has joined #openstack-infra04:16
*** Khaazher has quit IRC04:17
*** armax has quit IRC04:20
*** unicell has quit IRC04:22
*** MaxV has joined #openstack-infra04:34
*** unicell has joined #openstack-infra04:36
*** otherwiseguy has quit IRC04:37
*** MaxV has quit IRC04:38
*** links has joined #openstack-infra04:39
*** stevemar has joined #openstack-infra04:44
*** sarob has joined #openstack-infra04:54
*** mrmartin has joined #openstack-infra04:54
*** achanda has quit IRC04:54
*** achanda has joined #openstack-infra04:56
*** sarob has quit IRC04:58
*** rmcall has quit IRC05:02
*** sdake_ has joined #openstack-infra05:04
*** nelsnelson has joined #openstack-infra05:06
*** tomoe has quit IRC05:08
*** tomoe has joined #openstack-infra05:08
*** tomoe has quit IRC05:13
*** gyee has quit IRC05:19
*** fifieldt__ has quit IRC05:20
*** ildikov has quit IRC05:24
*** yfried has quit IRC05:30
*** tomoe has joined #openstack-infra05:32
*** nelsnelson has quit IRC05:32
jogodid something happen to the git servers05:33
jogothere has been a spate of failures05:33
*** mrmartin has quit IRC05:33
*** talluri has joined #openstack-infra05:33
*** mmaglana has quit IRC05:36
*** tomoe has quit IRC05:36
*** stevemar has quit IRC05:37
*** harlowja is now known as harlowja_away05:37
clarkbshould have a bot demand links05:38
*** tomoe has joined #openstack-infra05:38
*** tsg has quit IRC05:38
*** Longgeek has joined #openstack-infra05:39
linksclarkb, ^ was that intended for me ?05:39
clarkblinks no for jogo. sorry05:39
links:) okay05:39
*** emagana has quit IRC05:42
jogoclarkb: http://logs.openstack.org/66/130966/2/gate/gate-tempest-dsvm-neutron-full/baf4264/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-10-28_00_14_10_25205:47
jogoand I thought there was another but can't seem to find it right now05:48
*** chandankumar has joined #openstack-infra05:48
clarkbI think that is new05:48
clarkbwe dont https on the mirrors05:48
clarkboh wait05:49
clarkbits late. we do05:49
clarkbpypi mirror is no https. git does05:49
*** reed has quit IRC05:49
lifelesswe should totally https up the pypi mirrors05:52
*** sdake_ has quit IRC05:52
*** loki184 has joined #openstack-infra05:53
*** r1chardj0n3s is now known as r1chardj0n3s_afk05:58
*** koolhead17 has joined #openstack-infra05:59
*** yamamoto_ has quit IRC06:01
*** yamahata has quit IRC06:06
*** achanda has quit IRC06:08
*** afazekas_drunk is now known as afazekas06:08
*** ChuckC___ has joined #openstack-infra06:15
*** nikil89_ has joined #openstack-infra06:16
*** yfried has joined #openstack-infra06:16
*** jerryz has quit IRC06:17
*** ildikov has joined #openstack-infra06:18
*** david-lyle has quit IRC06:20
*** ChuckC___ has quit IRC06:21
*** talluri has quit IRC06:23
*** ivar-lazzaro has joined #openstack-infra06:24
*** talluri has joined #openstack-infra06:26
*** ivar-lazzaro has quit IRC06:32
*** ivar-lazzaro has joined #openstack-infra06:33
*** sdake_ has joined #openstack-infra06:34
*** akamyshnikova_ has quit IRC06:34
*** yamahata has joined #openstack-infra06:34
*** MaxV has joined #openstack-infra06:35
*** unicell has quit IRC06:37
*** ashaeron has joined #openstack-infra06:38
*** ivar-lazzaro has quit IRC06:39
*** MaxV has quit IRC06:40
*** sdake_ has quit IRC06:43
*** dims__ has joined #openstack-infra06:44
*** HeOS has joined #openstack-infra06:45
*** sdake_ has joined #openstack-infra06:46
*** talluri has quit IRC06:46
*** kingia has joined #openstack-infra06:47
*** kingia is now known as IaKing06:47
*** talluri has joined #openstack-infra06:47
*** dims__ has quit IRC06:49
*** david-lyle has joined #openstack-infra06:50
*** tomoe has quit IRC06:51
*** david-lyle has quit IRC06:54
*** HeOS has quit IRC06:54
*** k4n0 has joined #openstack-infra06:57
*** rakhmerov_ is now known as rakhmerov06:58
*** e0ne has joined #openstack-infra07:01
*** jerryz has joined #openstack-infra07:05
*** fifieldt has joined #openstack-infra07:06
*** achanda has joined #openstack-infra07:09
*** unicell has joined #openstack-infra07:14
*** cody-somerville has quit IRC07:16
*** cody-somerville has joined #openstack-infra07:16
*** _nadya_ has joined #openstack-infra07:16
*** achanda has quit IRC07:17
*** oomichi has quit IRC07:19
*** david-lyle has joined #openstack-infra07:21
*** david-lyle has quit IRC07:25
*** rushiagr_away is now known as rushiagr07:27
*** e0ne has quit IRC07:29
*** talluri has quit IRC07:30
*** tomoe has joined #openstack-infra07:35
*** leopoldj has joined #openstack-infra07:36
*** e0ne has joined #openstack-infra07:38
*** e0ne has quit IRC07:40
*** yamamoto_ has joined #openstack-infra07:40
*** andreykurilin_ has joined #openstack-infra07:41
*** sdake_ has quit IRC07:41
*** koolhead17 has quit IRC07:43
*** MaxV has joined #openstack-infra07:47
openstackgerritA change was merged to openstack-infra/project-config: Add the API working group guidelines to specs.o.org  https://review.openstack.org/13070107:47
*** links has quit IRC07:48
*** e0ne has joined #openstack-infra07:50
*** e0ne has quit IRC07:52
openstackgerritIan Wienand proposed a change to openstack-infra/project-config: Add openstack-infra/puppet-openstackci_apache  https://review.openstack.org/12949607:55
*** _nadya_ has quit IRC07:55
*** mrmartin has joined #openstack-infra07:58
*** viglesias has quit IRC08:09
*** HeOS has joined #openstack-infra08:10
openstackgerritMagnus Lundin proposed a change to openstack-infra/jenkins-job-builder: Added support for Rich Text Publisher Plugin  https://review.openstack.org/13045508:11
*** koolhead17 has joined #openstack-infra08:15
*** yfried is now known as yfried|afk08:17
*** yfried|afk is now known as yfried08:19
*** david-lyle has joined #openstack-infra08:21
*** yamahata has quit IRC08:22
*** david-lyle has quit IRC08:26
*** slashme has quit IRC08:26
*** talluri has joined #openstack-infra08:31
*** koolhead17 has quit IRC08:31
*** talluri has quit IRC08:35
*** doude has joined #openstack-infra08:36
*** rushiagr is now known as rushiagr_away08:40
*** jgallard has joined #openstack-infra08:40
*** Guest74700 has joined #openstack-infra08:45
*** talluri has joined #openstack-infra08:46
*** yamamoto_ has quit IRC08:51
*** jpich has joined #openstack-infra08:51
*** Longgeek has quit IRC08:54
*** Longgeek has joined #openstack-infra08:54
*** andreykurilin_ has quit IRC08:56
*** slashme has joined #openstack-infra08:57
*** parimala has joined #openstack-infra09:00
*** nikil89_ is now known as naggappan09:00
*** naggappan has left #openstack-infra09:00
*** naggappan has joined #openstack-infra09:01
*** links has joined #openstack-infra09:01
*** hdd has joined #openstack-infra09:02
*** liusheng has quit IRC09:02
parimalaHello All,09:03
naggappanHi is it possible to set devstack-gate to install devstack with my neutron repository insted of standard neutron repository? I have tried to set ZUUL_PROJECT=naggappan/test-neutron in jenkins job. Currently in my CI system if there any any changes in test-neutron repository then the zuul triggers a job and everything works fine. But it takes the standard neutron repository insted of my repositry .09:03
*** vdo has joined #openstack-infra09:04
parimalawhen I am doing  "tox -e checklinks", it is giving error as the link is not available , so what should I do?09:05
parimalaany clue?09:05
parimalait is the redhat stigs link which is broken09:08
*** achanda has joined #openstack-infra09:15
*** derekh has joined #openstack-infra09:16
*** rcarrillocruz has quit IRC09:17
*** links has quit IRC09:17
*** achanda has quit IRC09:19
ttxmorganfainberg: no you can't. If it's really so important that people can't be trusted to cut and paste it, I can turn them into links from within sched itself09:20
*** rcarrillocruz has joined #openstack-infra09:20
morganfainbergttx, <a ref> works09:20
ttxannegentle: the pods are at Le Meridien (like all the design summit)09:20
morganfainbergttx, btw.09:20
ttxmorganfainberg: heh. Sounds like a bug :)09:21
morganfainbergttx, :)09:21
ttxmaybe not actually. It allows a subset of "safe" html09:21
morganfainbergttx, ah. yeah href is pretty *safe*-ish09:21
morganfainbergttx, i wasn't going to ask you to go through and make them all links, if i couldn't do it via the tools copy-paste-it-was-to-be09:22
*** david-lyle has joined #openstack-infra09:22
morganfainbergoh gah, it's late.09:22
morganfainbergi need to sleep09:22
ttxor early09:22
ttxmorganfainberg: if you're training for the new timezone, you're doing it wrong09:23
*** viglesias has joined #openstack-infra09:23
morganfainbergttx, nope. was re-installing my desktop because of a complete fubar that nuked something in a way i couldn't fix.09:23
morganfainbergotherwise i'd be sleeping already09:23
ttxrule  #124 of sumits: do not upgrade or reinstall laptop in the week before09:24
*** eglynn-officeafk is now known as eglynn-office09:24
morganfainbergttx, i think i'd rather not upgrade / reinstall anytime if i could avoid it09:25
*** david-lyle has quit IRC09:26
*** jamielennox_ has joined #openstack-infra09:28
*** parimala has quit IRC09:29
*** jp_at_hp has joined #openstack-infra09:29
*** ZZelle has quit IRC09:29
*** ZZelle has joined #openstack-infra09:30
*** IvanBerezovskiy has joined #openstack-infra09:34
*** pelix has joined #openstack-infra09:36
*** andreaf has joined #openstack-infra09:36
*** andreaf_ has joined #openstack-infra09:39
*** zns has quit IRC09:43
*** brad__ has joined #openstack-infra09:48
*** Longgeek has quit IRC09:51
*** jamielennox_ has quit IRC09:53
*** slashme has quit IRC09:54
*** teran has quit IRC09:59
*** jedimike has joined #openstack-infra10:02
*** pc_m_ has joined #openstack-infra10:02
*** koolhead17 has joined #openstack-infra10:02
*** pc_m_ has quit IRC10:03
*** pc_m_ has joined #openstack-infra10:03
*** Longgeek has joined #openstack-infra10:04
*** _nadya_ has joined #openstack-infra10:06
*** hdd has quit IRC10:08
*** _nadya_ has quit IRC10:08
*** hdd has joined #openstack-infra10:11
*** brad__ has quit IRC10:11
*** dizquierdo has joined #openstack-infra10:14
*** rlandy has joined #openstack-infra10:16
*** mkerrin has joined #openstack-infra10:16
*** slashme has joined #openstack-infra10:18
*** dims__ has joined #openstack-infra10:19
*** yfried is now known as yfried|afk10:20
*** david-lyle has joined #openstack-infra10:23
*** dims__ has quit IRC10:23
*** koolhead17 has quit IRC10:27
*** david-lyle has quit IRC10:28
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Support additional events for HipChat plugin  https://review.openstack.org/12976010:29
sdaguejhesketh: you up?10:32
sdaguettx: can you kick the following into the req proj - https://review.openstack.org/#/c/131184/ ?10:33
*** e0ne has joined #openstack-infra10:34
*** markmcclain has joined #openstack-infra10:36
ttxsdague: looking10:36
*** slashme has quit IRC10:37
ttxsdague: approved10:37
openstackgerritSean Dague proposed a change to openstack-infra/system-config: only show most recent results per pipeline  https://review.openstack.org/13124910:37
*** teran has joined #openstack-infra10:39
*** loki184 has quit IRC10:41
*** koolhead17 has joined #openstack-infra10:41
*** _nadya_ has joined #openstack-infra10:41
*** dims__ has joined #openstack-infra10:43
ianwsdague: got 2 minutes to sync up on bashate?  it's causing me some grief10:44
*** teran_ has joined #openstack-infra10:44
sdaguesure, I'm on cup off coffee 1, so be forewarned :)10:44
*** teran has quit IRC10:44
ianwsdague: TOT doesn't work on devstack ATM, there are several reviews out10:44
sdagueok, I haven't found my way back to those yet10:45
sdaguelet me go look10:45
ianwsdague: i think we should beat it into shape any which way and do a release10:45
sdaguesure, is there a particular sequence the patches should go in in your opinion?10:46
ianwi now have a devstack review that is probably incorrectly triggering bashate failures : https://review.openstack.org/#/c/131334/10:46
*** koolhead17 has quit IRC10:46
ianwbut see no way to skip that10:46
ianwsdague: i *think* the ones current causing devstack to fail are https://review.openstack.org/127052  & https://review.openstack.org/12977110:47
ianwmerging the `` was probably the wrong thing to do, we'll need to add an ignore to devstack for that10:48
ianwthe `` check i mean10:48
ianwi feel like we should have the ability to turn it down to a warning, at least then we could be fixing it and see from the logs10:48
*** teran_ has quit IRC10:48
ianwhttps://review.openstack.org/129761 (add ability to ignore in-file) might be useful for situations such as the false-positive.  haven't got to the comments on that yet10:49
*** dmakogon_ is now known as denis_makogon10:49
ianwhttps://review.openstack.org/128807 (Strip comments before checks) is also required for devstack TOT I think10:49
sdaguehttps://review.openstack.org/#/c/127052/5 seems like a weird way to do that10:49
sdagueianw: https://review.openstack.org/#/c/128807/ is now in merge conflict, can you rebase?10:50
sdaguehttps://review.openstack.org/#/c/128807/4 seems like a good idea, and I think it removes the need for 12705210:51
*** AzherKhan has quit IRC10:51
*** jklare has joined #openstack-infra10:52
ianwsdague: 128807 should be rebased10:52
sdaguehmmm... have we seen mrodden on reviews recently?10:53
openstackgerritSergey Skripnick proposed a change to openstack-infra/project-config: Add hook for building rally docker image  https://review.openstack.org/13138910:53
*** markmcclain has quit IRC10:53
*** markmcclain has joined #openstack-infra10:53
sdagueso there is another way to do https://review.openstack.org/#/c/131334/ that's less tricksie, let me figure out where I did that10:53
ianwsdague: yeah, can probably abandon 127052 if 128807 goes in10:54
sdagueI just fast approved 12880710:54
jklarehi, can somebody help me with this issue http://logs.openstack.org/11/115711/4/check/gate-openstack-chef-repo/2237e9e/console.html10:55
*** talluri has quit IRC10:55
jklarethis should have been fixed with the dep-selector-libgecode gem but apparently its not10:55
jklareand i cant repoduce this locally10:55
*** andreaf has quit IRC10:55
*** andreaf_ is now known as andreaf10:55
ianwsdague: happy for a better way on that expansion thing, that code does my head in10:56
sdagueianw: so... https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L285-L286 works without having to subshell the heredoc10:57
sdagueI think it's the same thing you are trying to do, right?10:57
ianwsdague: hmm, that doesn't seem to keep quotes correctly10:59
*** teran has joined #openstack-infra10:59
ianwmerge_config_file test-quote: failed:10:59
ianwfoo = foo bar baz !=10:59
ianwfoo = "foo bar" "baz"10:59
*** IaKing has quit IRC10:59
ianwusing the "cat" I think does10:59
*** tomoe has quit IRC10:59
*** baoli has joined #openstack-infra11:00
*** tomoe has joined #openstack-infra11:00
*** teran_ has joined #openstack-infra11:01
*** teran_ has quit IRC11:01
ianwwell, more the heredoc than the cat i guess11:01
sdaguecan you give me an example of where this is failing in the code?11:01
sdaguelike a real instance of the iniset that's breaking, so I can wrap my head around it better11:02
*** teran_ has joined #openstack-infra11:02
*** BharatK has joined #openstack-infra11:03
ianwsdague: pci_alias = {"name":"cisco","vendor_id":"8086","product_id":"10ed"} <- from https://bugs.launchpad.net/devstack/+bug/137411811:03
uvirtbotLaunchpad bug 1374118 in devstack "use of local.conf doesn't preserve quotes in the string and can't configure multi-line items" [Undecided,In progress]11:03
*** teran has quit IRC11:03
*** yfried|afk is now known as yfried11:03
ianwsdague: i assume you mean breaking with your proposed fix?11:03
ianwsdague: we started single-quoting the argument to allow that type of line to work, however, in the process, broke expansion of variables in the line11:04
*** baoli has quit IRC11:04
ianwsdague: to keep everyone happy, we need to both keep quotes, and expand variables11:04
*** tomoe has quit IRC11:04
*** baoli has joined #openstack-infra11:05
sdagueianw: yeh, that's what I was looking for11:05
*** dizquierdo has quit IRC11:05
*** dizquierdo has joined #openstack-infra11:06
*** weshay has joined #openstack-infra11:07
sdagueso... honestly, when did oslo start taking lines like that?11:07
ianwsdague: not sure.  I assume it's just naturally heading toward Turing completeness11:08
sdagueyeh, so my feeling is we should revert the thing that supported pci_alias11:09
sdagueget old working behavior11:09
sdagueand have a conversation about nested dicts inside of config options11:09
sdaguebecause I think they are a terrible idea11:10
sdagueit's been in olso config for a year, but I still think it's a terrible idea11:10
ianwsdague: ok, that's probably a conversation best started by someone with significant brand recognition :)11:12
sdaguesure, but can you propose the reverts? I'll start the config conversation11:12
sdaguezomg, it's not even using the olso DictOption11:15
sdagueit's building a random custom in option parser11:15
ianwsdague: i won't get to it tonight, can tomorrow .au time11:17
sdagueianw: sounds good11:17
ianwsdague: ok, TOT bashate runs on devstack with -i E040 (the `` v $() checking)11:18
*** talluri has joined #openstack-infra11:18
ianwsdague: it may have been a mistake to merge that ... maybe we should revert, figure out a way to have warnings, and re-instate it as a warning-level message -- not fatal?11:19
sdagueianw: which thing should we revert now?11:22
ianwsdague: the check in bashate for `foo`, erroring out that it should be $(foo)11:22
*** david-lyle has joined #openstack-infra11:24
sdagueah, gotcha11:26
sdagueyeh, that's definitely not true everywhere11:26
ekarlsohey guys, is there a smart way to drop a column in sqla migrate and preserve the indexes after the migration when it's done in unit tests using sqlite ?11:26
sdagueis there a revert up for that?11:26
*** HeOS has quit IRC11:26
ianwsdague: so the end game, is that if we have bashate in a release state, we can put out a new version and then merge https://review.openstack.org/#/c/128809/ (Fix up file-matching in bashate tox test) which actually starts running bashate on a whole bunch of stuff that's currently missed11:27
*** david-lyle has quit IRC11:28
*** yfried is now known as yfried|afk11:31
*** yfried|afk is now known as yfried11:31
ianwsdague: https://review.openstack.org/#/c/131396/ (Revert "Add E040: $(COMMAND) preferred over `COMMAND`") <- with that, bashate should pass on devstack with https://review.openstack.org/#/c/128809/11:33
*** aysyd has joined #openstack-infra11:34
*** _nadya_ has quit IRC11:35
*** koolhead17 has joined #openstack-infra11:40
*** cdent has joined #openstack-infra11:41
openstackgerritA change was merged to openstack/requirements: implement -s / --soft-update flag  https://review.openstack.org/13118411:44
*** HeOS has joined #openstack-infra11:46
*** yamahata has joined #openstack-infra11:47
*** yamamoto has joined #openstack-infra11:50
*** pelix has quit IRC11:50
*** kgiusti has joined #openstack-infra11:50
*** pelix has joined #openstack-infra11:52
*** pelix has quit IRC11:53
*** yamamoto has quit IRC11:54
*** HeOS has quit IRC11:55
*** tomoe has joined #openstack-infra11:55
*** doug-fish1 has joined #openstack-infra11:57
*** koolhead17 has quit IRC11:59
*** yfried is now known as yfried|afk12:00
*** doug-fish has quit IRC12:00
*** markmcclain has quit IRC12:00
*** teran has joined #openstack-infra12:01
*** HeOS has joined #openstack-infra12:02
*** weshay has quit IRC12:03
*** dkranz has joined #openstack-infra12:03
*** teran_ has quit IRC12:05
*** dims__ has quit IRC12:07
*** dims__ has joined #openstack-infra12:07
openstackgerritmeza proposed a change to openstack-infra/jenkins-job-builder: Added slack support  https://review.openstack.org/13140412:08
mordredianw, sdague: wow. pci_alias = {"name":"cisco","vendor_id":"8086","product_id":"10ed"}12:08
mordredthat's crazypants12:08
sdaguemordred: right... agreed12:08
sdagueI started an ML thread on it12:09
mordred_maybe_ we should drop a bomb on people and suggest that we drop ini for yaml?12:09
*** doug-fish1 has left #openstack-infra12:09
sdaguewell, honestly, the name = value here I think is overkill12:09
sdaguewe're talking about a few well defined fields12:09
sdaguewe could just do a fixed record format12:09
* mordred tries to keep his uninformed head out of most of these things12:09
sdaguesee ML12:09
mordredsdague: oh - while you were out, I wrote a new library12:10
*** mwagner_lap has quit IRC12:10
sdagueI just assume you write a new library every weekend :)12:10
mordredsdague: this one is called "os-client-config" - it's a library to handle end-user/client cloud credential config stuffs12:10
mordredsdague: dtroyer has a patch up to add it to python-openstackclient12:11
*** yfried|afk is now known as yfried12:11
mordredhandles multi-clouds-in-config-file as well as env vars - and I think dtroyer has it doing something sensible with post-processing argparse namespaces12:11
mordredI kinda think we should add it to all of the client libs - mainly because I'm sad that the nova command line client doesn't grok my clouds.yaml file now :)12:12
*** bradjones has quit IRC12:12
*** bradjones has joined #openstack-infra12:13
sdaguethat seems reasonable... oh, I should push up the python client from pypi testing patch out there. Maybe heads up on the ML first to get some acks on approach.12:14
*** yamahata has quit IRC12:15
openstackgerritSergey Skripnick proposed a change to openstack-infra/project-config: Add hook for building rally docker image  https://review.openstack.org/13138912:15
sdaguealso, I'm going to be that guy at least once today - https://www.flickr.com/gp/51681448@N00/o5D079 - my coding buddy coming for a visit yesterday12:15
*** weshay has joined #openstack-infra12:16
*** BharatK has quit IRC12:16
*** unicell has quit IRC12:17
mordredsdague: you're keeping to a strict diet of python, right? :)12:18
sdagueyep :)12:18
sdaguethough, actually, I guess I was writing bash and javascript mostly yesterday12:18
mordredturns out that openstack is not all written in python12:18
mordredturns out it's turned in to "python and bash and javascript"12:19
sdaguehttps://review.openstack.org/#/c/131249/ should further simplify the test results table12:19
dims__hmm, none of the tests in python27 seems to be running?12:19
dims__Run: 0 in 0.0 sec.12:20
sdaguedims__: on what?12:20
mordreddims__: that does not happy making me12:20
dims__sample - http://logs.openstack.org/86/130086/6/check/gate-nova-python27/0f838dc/console.html12:20
* mordred cries12:20
* dims__ hopes he is wrong12:20
mordredlifeless: ^^ you still awake?12:20
mrmartindims__: epic fast12:21
mordredI get 106 hits for that in the past 48 hours - but it seems to have picked up significantly in the last 8 hours12:22
*** david-lyle has joined #openstack-infra12:24
mordreddims__: https://review.openstack.org/#/c/130870/ seems to be an early one that did it12:26
*** slashme has joined #openstack-infra12:27
dims__thanks mordred looking12:28
*** slashme has quit IRC12:28
*** slahsme has joined #openstack-infra12:29
*** david-lyle has quit IRC12:29
*** koolhead17 has joined #openstack-infra12:29
mordreddims__: but - you know, may not be related - that's just me trying to pretend like I know how to use elasticsearch12:30
dims__mordred: understood. staring at run_tests.sh12:30
dims__but we are not running that12:30
dims__tox.ini changes only the cover target12:31
dims__so can't be that12:31
mordredI'm wondering if something changed to cause testr to fail open12:31
sdagueyeh, that happens some times12:31
mordredlifeless: I know we've said this before, so I don't want to lay it on too much ... but I do want to reiterate that I think that failing open is never the right choice12:32
sdagueso there is work around code in tempest to catch that case12:32
* mordred cries12:32
sdagueI can bring that over to nova12:32
sdaguedims__: you replicating this locally?12:33
* sdague is kicking off a local run for it12:33
dims__sdague: ya, thats how i noticed it12:33
mordredsdague: is that workaround code in a wrapper script?12:33
dims__"./run_tests.sh -N" also shows 0 tests run12:33
dims__"cat: .testrepository/next-stream: No such file or directory"12:34
*** ryanpetrello has quit IRC12:34
sdaguemordred: it is12:34
*** ryanpetrello has joined #openstack-infra12:34
*** dprince has joined #openstack-infra12:35
sdaguemordred: https://github.com/openstack/tempest/blob/master/tools/subunit-trace.py#L237-L23912:36
sdaguemordred: also... what tz you in?12:36
sdaguebecause this is freaking early for west coast :)12:36
mordredsdague: I'm in paris12:36
mordredsdague: wow. how is all of that code not just in testr?12:36
sdagueoh, nice12:36
*** miqui has joined #openstack-infra12:36
*** doug-fish has joined #openstack-infra12:36
sdaguebecause... testr is still in bzr12:37
*** arosen has joined #openstack-infra12:37
arosenanyone know where this file now lives? https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/nodepool/scripts/prepare_node.sh12:37
mordredsdague: I believe it has moved finally12:37
*** koolhead_ has joined #openstack-infra12:38
*** koolhead17 has quit IRC12:39
*** mbacchi has joined #openstack-infra12:39
*** ihrachyshka has joined #openstack-infra12:39
*** freemanbrandon has joined #openstack-infra12:40
mordredok. I'm going to go find coffee12:41
arosennvm found it.12:41
*** hdd has quit IRC12:42
sdagueok, cool12:42
*** weshay has quit IRC12:43
*** hdd has joined #openstack-infra12:44
*** alexpilotti has joined #openstack-infra12:45
*** unicell has joined #openstack-infra12:45
*** alexpilotti has quit IRC12:45
*** alexpilotti has joined #openstack-infra12:46
*** k4n0 has quit IRC12:46
*** dkliban_afk is now known as dkliban12:46
sdaguedims__: it's related to the olso.concurency bit12:46
sdagueis this one of those goofy things where oslo.concurrency exits because the tmpdir isn't specified?12:47
*** jerryz has quit IRC12:47
openstackgerritmeza proposed a change to openstack-infra/jenkins-job-builder: Fixed style violation  https://review.openstack.org/13141412:47
*** talluri has quit IRC12:48
sdaguedims__: so, honestly, I think we need to fast revert the concurrency patch12:48
sdaguebecause not running unit tests is not really an option12:48
*** _nadya_ has joined #openstack-infra12:49
*** talluri has joined #openstack-infra12:51
*** vhoward has left #openstack-infra12:52
*** todd_dsm has joined #openstack-infra12:52
*** mwagner_lap has joined #openstack-infra12:52
openstackgerritmeza proposed a change to openstack-infra/jenkins-job-builder: Fixed style violations in the properties  https://review.openstack.org/13141712:53
*** mfink has joined #openstack-infra12:57
*** ryanpetrello has quit IRC13:02
*** weshay has joined #openstack-infra13:03
*** unicell has quit IRC13:04
*** aysyd has quit IRC13:05
*** bknudson has quit IRC13:05
*** ryanpetrello has joined #openstack-infra13:05
*** ryanpetrello has quit IRC13:06
*** dustins has joined #openstack-infra13:06
*** derekh has quit IRC13:06
*** rodrigods has joined #openstack-infra13:09
*** julim has joined #openstack-infra13:10
anteayaso mikal and I have successfuly arm-twisted jogo into being the nova third-party liaison13:10
anteayaI've asked jogo to reply to http://lists.openstack.org/pipermail/openstack-dev/2014-October/049335.html and let him know we are happy to support his knowledge base so he can reply13:11
*** ryanpetrello has joined #openstack-infra13:11
*** markmcclain has joined #openstack-infra13:11
*** markmcclain1 has joined #openstack-infra13:12
*** rushiagr_away is now known as rushiagr13:12
*** markmcclain1 has quit IRC13:12
dhellmannlifeless: I haven't tried it with jython, no13:12
*** unicell has joined #openstack-infra13:15
*** markmcclain has quit IRC13:16
*** hdd has quit IRC13:18
*** achanda has joined #openstack-infra13:18
fungimordred: sdague: dims__: still catching up on scrollback, but lifeless uploaded a new release around the time that picked up13:21
*** aysyd has joined #openstack-infra13:21
dhellmannsdague, dims__ : is there a bug for this thing with oslo.concurrency and nova not running tests?13:22
*** teran_ has joined #openstack-infra13:22
dims__fungi: lockutils caused problem, sdague has a review for a revert of oslo.concurrency in nova and a review to make sure we bail out when we see "0 tests"13:23
*** achanda has quit IRC13:23
fungimordred: sdague: dims__: unittest2 0.6.013:23
fungihe commented in here 2014-10-28 00:35:4313:23
arosenHi, I noticed that my 3rd party CI doesn't actually seem to be showing it's voting score on gerrit. Is this something I need to change or something that the infra team needs to set?  The CI name is "VMWare Congress CI"13:25
*** bknudson has joined #openstack-infra13:25
*** david-lyle has joined #openstack-infra13:25
*** teran has quit IRC13:26
*** ryanpetrello_ has joined #openstack-infra13:26
*** tsg has joined #openstack-infra13:27
fungiarosen: the project on which it will vote needs to request voting rights for the account, just so we don't have accounts voting on projects' changes uninvited13:27
*** mriedem has joined #openstack-infra13:29
*** david-lyle has quit IRC13:29
arosenfungi: i see, i guess i need to send an email to  third-party-requests@lists.openstack.org to get it enabled?13:29
fungiarosen: i'm not sure we've formalized that process, but it wouldn't hurt... on what project(s) will it be voting?13:31
arosenstackforge/congress, stackforge/python-congressclient for now.13:31
fungiarosen: does congress have a ptl?13:31
arosenfungi:  yes it does13:32
*** thedodd has joined #openstack-infra13:32
arosenTim Hinrichs13:32
arosenI'm set as the release manager though.13:32
fungineat, haven't seen him around. and yes, i just confirmed that in the gerrit group membership13:32
*** zns has joined #openstack-infra13:33
*** sarob has joined #openstack-infra13:34
fungii guess if you want to send couple sentence e-mail saying that as the release manager for teh stackforge/congress projects you're requesting voting rights for that account so that it can provide better feedback on your projects, and then i'll reply that i've done so13:34
fungiarosen: ^13:34
arosenfungi:  awesome i'll do that. Thanks. I'll CC tim too.13:34
*** changbl has quit IRC13:35
fungiarosen: and in the meantime i've already added voting rights for it... the e-mail is just so that we have a more easily searched public record of these sorts of status changes13:35
arosenfungi:  thanks :), sending the email now.13:36
dhellmannsdague: I noticed yesterday that some of the oslo libs still send patches through the integrated gate. I don't think that's the intent any more, but can't remember how exactly that works to fix it myself. Can we work on that together after the summit?13:36
sdaguefungi: can you gate queue this - https://review.openstack.org/#/c/131416/13:36
sdagueit's kind of important that no other nova changes land until that's in13:36
sdaguedhellmann: sure, do you have an example?13:36
*** dmsimard_away is now known as dmsimard13:37
fungisdague: 131416,1 is enqueued into the gate now13:37
openstackgerritmeza proposed a change to openstack-infra/jenkins-job-builder: Fixed style violations  https://review.openstack.org/13143113:38
fungisdague: along with its child change13:38
sdaguefungi: thanks13:38
*** homeless has joined #openstack-infra13:39
*** timrc-afk is now known as timrc13:39
*** aysyd has quit IRC13:40
*** marun has joined #openstack-infra13:41
fungisdague: oddly, i thought we already had precautions in the tox wrapper script to fail out if no tests get run13:42
sdaguefungi: so I think we do13:42
sdaguewe have a different wrapper for nova now13:42
sdaguethat gives us more info13:42
sdagueso we probably bypass that13:42
fungii guess this didn't catch it then? http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-unittests.sh#n5913:43
anteayarain in france http://www.france-weather.com/13:43
clarkbwe do and nova managed to bypass13:44
*** homeless1 has joined #openstack-infra13:44
clarkbthe check in our tests relies on .testrepository existing13:44
funginova's testr runs don't create a .testrepository file any longer?13:45
clarkbthey do but not when broken as in this case13:45
sdagueclarkb: passing --subunit makes .testrepository not exist?13:45
clarkbit was dying before any testr could run13:45
clarkbno oslo.concurrency had no main13:46
*** marun has quit IRC13:46
sdagueoh, gotcha13:46
clarkbso python -m lockstuff went "fail"13:46
*** homeless has quit IRC13:46
clarkbbut it returned 0?13:46
*** zns has quit IRC13:47
clarkbthats the bit that has me confused13:48
*** freemanbrandon has quit IRC13:48
clarkbI am guessing python -m importedjust fine13:48
sdagueyeh, I don't know13:48
clarkband its up to the lib to execute code13:48
*** amitgandhinz has joined #openstack-infra13:49
clarkbsince it didnt the import was successful and return 013:49
*** armax has joined #openstack-infra13:49
*** amitgandhinz has quit IRC13:49
*** marun has joined #openstack-infra13:49
*** amitgandhinz has joined #openstack-infra13:49
fungiright, the import was a no-op13:50
*** pelix has joined #openstack-infra13:51
*** dizquierdo has quit IRC13:51
*** katyafervent is now known as katyafervent_awa13:51
*** aysyd has joined #openstack-infra13:52
*** zns has joined #openstack-infra13:52
*** freemanbrandon has joined #openstack-infra13:52
*** zns has quit IRC13:53
*** yfried has quit IRC13:54
*** koolhead_ has quit IRC13:55
*** hdd has joined #openstack-infra13:55
openstackgerritMate Lakat proposed a change to openstack-infra/project-config: XenServer: Fix initramfs scripts  https://review.openstack.org/13143813:59
*** ryanpetrello has quit IRC13:59
*** ryanpetrello_ is now known as ryanpetrello13:59
*** zns has joined #openstack-infra14:00
*** eharney has joined #openstack-infra14:02
*** zz_dimtruck is now known as dimtruck14:02
naggappanhi what is git_base URL in devstack gate? its always pointing to local git_base=https://git.openstack.org. I wan to modify to my get server path. so that it takes neutron code from my server and others it takes from standard repository. currently its trying git clone https://git.openstack.org/naggappan/test-neutron and repository not found and it gets the the standard one. Now i want the devstack-gate to use my Repository how can i do that.14:04
*** dims__ has quit IRC14:04
*** parimala has joined #openstack-infra14:04
*** unicell has quit IRC14:05
*** nosnos has quit IRC14:06
*** jerryz has joined #openstack-infra14:06
*** nosnos has joined #openstack-infra14:07
*** vhoward has joined #openstack-infra14:07
*** talluri has quit IRC14:07
naggappanby default will devstack-gate clone all openstack projects only from https://git.openstack.org/ URL? is it possible to change it14:07
*** dizquierdo has joined #openstack-infra14:07
*** chandankumar has quit IRC14:07
*** talluri has joined #openstack-infra14:07
*** unicell has joined #openstack-infra14:08
*** nelsnelson has joined #openstack-infra14:09
*** nelsnelson has quit IRC14:09
*** nelsnelson has joined #openstack-infra14:10
*** nosnos has quit IRC14:11
*** mattfarina has joined #openstack-infra14:14
*** katyafervent_awa is now known as katyafervent14:14
*** DuncanT- has joined #openstack-infra14:15
*** JayJ has joined #openstack-infra14:21
*** otherwiseguy has joined #openstack-infra14:23
funginaggappan: if you look in devstack-vm-gate-wrap.sh you'll see you can define a custom pre_test_hook function and export it in the calling environment. it runs after setup_workspace has setup all the listed projects but before gate_hook runs the devstack-vm-gate.sh script14:23
*** msabramo has quit IRC14:24
*** spzala has joined #openstack-infra14:24
funginaggappan: your pre_test_hook could do something like call setup_project for your special project and override GIT_BASE to your git server url when doing so14:24
*** miqui has quit IRC14:25
*** sarob has quit IRC14:25
*** sarob has joined #openstack-infra14:25
*** david-lyle has joined #openstack-infra14:26
sdagueclarkb: on - https://review.openstack.org/#/c/125095/ - do you have a snippet for only getting supported branches? my brain wasn't quite coming up with it14:26
*** tsg has quit IRC14:26
sdaguenaggappan: you probably don't want to override GIT_BASE, override the url for the specific project instead14:27
*** jerryz has quit IRC14:27
*** amitgandhinz has quit IRC14:28
clarkbsdague https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#n57 something like that14:29
*** radez_g0n3 is now known as radez14:29
*** sarob has quit IRC14:29
fungisdague: setup_project honors GIT_BASE from the calling environment but i don't see where you can pass in the specific url for the project you want to set up instead14:30
*** david-lyle has quit IRC14:31
naggappan@fungi so for specifig project i can't override the url. Because i want to use only neutron from my repository others i can use the same git.openstack.org. If possible please tell the the yamal file configuration where i can specify the environment variable14:32
*** thedodd has quit IRC14:33
sdaguefungi: oh... right. I was thinking about the devstack case14:34
*** yaguang has joined #openstack-infra14:34
funginaggappan: i'm not sure what you're asking. there are no yaml files i'm aware of in devstack-gate or consumed by devstack-gate14:35
fungioh, right14:35
fungiso anyway, no yaml files which let you modify clone urls or git remotes14:36
clarkband no feature.yaml knows nothing about git14:36
*** zns has quit IRC14:36
*** miqui has joined #openstack-infra14:37
naggappansorry i said wrong. Not feature.yamal. Acutally I am setting up CI setup . where all repo should take from standard git.openstack.org but only neutron it should take from my repository.14:37
funginaggappan: one easy trick might just be to make pre_test_hook change the origin remote in the local copy of the neutron repo and then git reset --hard origin/master14:37
openstackgerritMarton Kiss proposed a change to openstack-infra/system-config: Update drush dsd-tool of groups portal  https://review.openstack.org/13145214:37
fungimrmartin: is that ^ urgent to approve?14:40
fungior are you working around it okay for now?14:40
mrmartinfungi: I feel so14:40
naggappan@fungi so i have to first download the standared neutron repo. Then set this pre_test_hook with my repo and reset. Is that right? If so then where should i change this because devstackgate script automatically do git clone. where i should specify all this changes14:40
fungiclarkb: very simple patch if you feel like adding your +314:40
mrmartinwriting software is hard :) and had a stupid permission bug that caused a White screen of death14:40
fungimrmartin: i've had my share of those too :/14:41
mrmartinwe need some monitoring, because this stupid bug affected the prod portal also, I fixed that manually to bring it back today morning14:41
funginaggappan: wherever you're executing the devstack-vm-gate-wrap.sh script from, you need to export your custom pre_test_hook function in the calling environment (are you using jenkins-job-builder?)14:42
*** ayoung has joined #openstack-infra14:42
*** amitgandhinz has joined #openstack-infra14:42
*** yamahata has joined #openstack-infra14:42
naggappan@fungi yes i am using JJB14:42
mrmartingroups-dev still suffering from that, but I don't want to fix manually, because I want to wait the fixed tool14:42
*** chandankumar has joined #openstack-infra14:43
funginaggappan: then it would probably need to be exported in the shell builder for the job you're running, before calling devstack-vm-gate-wrap.sh14:43
*** zns has joined #openstack-infra14:43
pleia2good morning14:43
*** koolhead17 has joined #openstack-infra14:45
*** koolhead17 has joined #openstack-infra14:45
naggappan@fungi i know to export the variables. But how can i export the complete pre_test_hook function. Could you point me some example link14:45
funginaggappan: in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/rally.yaml you can see examples of jobs and job-templates exporting a pre_test_hook function14:46
*** ayoung has quit IRC14:46
fungimrmartin: as for monitoring for situations like that, who/how would you expect to get notified so as to reduce the impact duration?14:48
naggappan@fungi thanks let me gothrought and try that14:48
*** garyh has joined #openstack-infra14:48
fungimrmartin: we've talked about monitoring before, but it's usually boiled down to that set of procedural questions (actually implementing a monitoring system is the "easy" part)14:48
*** achanda has joined #openstack-infra14:48
mrmartinfungi: I agree, I think we can discuss this on the summit14:49
mrmartinhttps://review.openstack.org/#/c/131452/ get -1 from jenkins14:49
mrmartina dns / networking issue in the centos6 node14:50
fungimrmartin: yep, i just rechecked it. should go through this time14:50
*** markmcclain has joined #openstack-infra14:50
arosenhrm, any ideas why i've been hitting this failure in my 3rd party ci related to the python-keystoneclient http://paste.openstack.org/show/125975/ ?14:51
mrmartinfungi, thnx14:51
arosen2014-10-28 14:47:22.818 | No distributions matching the version for stevedore>=1.1.0 (from python-keystoneclient==
*** signed8bit has joined #openstack-infra14:51
*** tonytan4ever has joined #openstack-infra14:53
fungimrmartin: as for preparing for a productive discussion about monitoring, "we need monitoring" is too nebulous, but something like "it would be nice if i got an e-mail when this url returns no content for more than 5 minutes" would be an example of a more actionable request/proposal14:53
*** wenlock has joined #openstack-infra14:53
fungiarosen: i see "pypi.openstack.org" in that log14:54
arosenfungi: I guess the upstream CI doesn't use that?14:55
fungiarosen: nope, we don't. were you in the third-party meeting yesterday by any chance? or did you see my e-mail to third-party-announce about why the citrix xenserver ci account was disabled?14:55
arosenfungi:  sadly no. I was on vacation for the last week. I just joined the 3rd party ci last14:55
aroseni can dig up the thread though so you don't have to rehash it for me.14:55
mrmartinfungi: of course :) I did some icinga / nagios projects before14:55
fungimrmartin: point being, it's a proposal which addresses the people and notification mechanism desired14:56
fungiarosen: in particular the package mirrors maintained by the upstream project infrastructure are not stable public interfaces, and also present to actual benefit to third-party systems. you should either run a mirror of your own or just use pypi.python.org14:57
arosenfungi:  yup found youre email http://lists.openstack.org/pipermail/third-party-announce/2014-October/000084.html14:58
*** zz_jgrimm is now known as jgrimm14:58
*** mattfarina has quit IRC14:59
fungiarosen: for pypi.openstack.org, that was abandoned some months ago when we switched to running individual mirrors within each provider/region, so it hasn't been updated in some time and is slated for removal14:59
arosenfungi:  should i update jenkins_jobs/config/macros.yaml: s/http://pypi.openstack.org/http://pypi.python.org/g ?14:59
*** dane_leblanc has joined #openstack-infra15:00
fungiarosen: your nodepool prep scripts or dib elements, if you're reconsuming ours without modification, may be setting a pypi mirror15:00
*** ayoung has joined #openstack-infra15:00
arosenfungi:  so you don't thinks it's being done via jenkins_jobs/config/macros.yaml?  This is the only place in etc i see this string.15:01
fungiarosen: it's possible that's where you're getting it from15:01
arosenfungi:  k let me try updating it here first15:01
openstackgerritSean Dague proposed a change to openstack-infra/project-config: enforce stable requirements  https://review.openstack.org/12509515:01
sdagueclarkb: how about that?15:02
*** carl_baldwin has joined #openstack-infra15:04
*** david-lyle has joined #openstack-infra15:05
nelsnelsonGreetings, channel.  I've got a little problem trying to set up devstack-gate on an ubuntu vm.15:05
nelsnelsonI'm following the instructions here https://github.com/openstack-infra/devstack-gate15:05
nelsnelsonBut I keep getting timeouts when I run the safe-devstack-vm-gate-wrap.sh script: https://gist.github.com/nelsnelson/f43b987a5d4e94a6927715:05
openstackgerritA change was merged to openstack-infra/system-config: Update drush dsd-tool of groups portal  https://review.openstack.org/13145215:06
*** talluri has quit IRC15:06
funginelsnelson: have you looked in the devstack log to see what it might have been doing when that timeout was reached?15:07
*** achanda has quit IRC15:09
nelsnelsonfungi, thanks for responding.  Yes, I checked the logs, but I can't really see anything conclusive.15:09
*** SumitNaiksatam has quit IRC15:09
*** msabramo has joined #openstack-infra15:09
nelsnelsonThe last thing I see is "Processing triggers for man-db ..."15:09
nelsnelsonPrior to that, thtere's a bunch of "Reading database ..." messages, but it indicates progress, and completes.15:10
fungidtroyer: if you override devstack.org and/or www.devstack.org to you should be able to test the redirect to http://docs.openstack.org/developer/devstack/$115:10
fungidtroyer: are you okay with me switching dns over to the redirect now?15:10
*** tsg has joined #openstack-infra15:11
funginelsnelson: sounds like it's still in the middle of package installation/updates when it times out15:11
nelsnelsonSounds about right.15:11
funginelsnelson: we're usually running it with pre-cached local packages, so perhaps it's taking a lot longer because you don't have any cached and need to up the timeout on that?15:11
openstackgerritMark Vanderwiel proposed a change to openstack-infra/project-config: Missed repo gate job for zuul layout filter.  https://review.openstack.org/13146715:12
nelsnelsonWell, prior to that error message about the timeout, it just sits there for over 30 minutes at least.15:12
nelsnelsonThe package installs and everything appear to complete...15:12
nelsnelsonReading package lists... Done15:12
nelsnelsonand so on15:12
funginelsnelson: oh, in that case it may be that package installation is not the issue, but rather whatever comes immediately after it in the script15:13
nelsnelsonThe last thing I see is this:15:13
nelsnelsonRunning devstack15:13
nelsnelson... this takes 5 - 8 minutes (logs in logs/devstacklog.txt.gz)15:13
nelsnelson./safe-devstack-vm-gate-wrap.sh: line 321:  1452 Killed15:13
nelsnelsonERROR: the main setup script run by this job failed - exit code: 13715:13
nelsnelsonbut then there is not logs/devstacklog.txt.gz file15:14
*** freemanbrandon has quit IRC15:14
nelsnelsonjust a logs/devstacklog.txt file15:14
fungiyeah, not a very helpful line for it to point out unfortunately, as that's the definition for the gate_hook function15:14
*** eglynn-office has quit IRC15:14
dtroyerfungi: checking…15:14
*** sdake_ has joined #openstack-infra15:17
marcusvrn_fungi: I saw that you disabled the CI that was failing because it was using  pypi.openstack.org that will be removed. You mentioned to change the mirror to  pypi.python.org or local mirror. I have the same problem....can you help me to say where I have to change it?15:18
*** pcrews has joined #openstack-infra15:18
clarkbsdague: one question on 12509515:18
fungimarcusvrn_: it depends a lot on how you're building your job workers and running jobs on them15:18
clarkbsdague: but I think that mostly works15:18
dtroyerfungi: go for it…and thanks15:19
clarkbmarcusvrn_: arosen: are you all using jaypipes' stuff?15:19
arosenclarkb:  yes15:19
aroseni tried updating the /etc/jenkins files but that didn't do the trick15:19
marcusvrn_fungi:clarkb   I'm using asselin repo (that is a fork of jaypipes repo)15:19
clarkbiirc devstack-gate copies a "base" set of files into all the places we need it15:19
clarkbso I think you just need to edit that location. give me a minute to look that up15:20
*** sdake_ has quit IRC15:20
openstackgerritSean Dague proposed a change to openstack-infra/project-config: enforce stable requirements  https://review.openstack.org/12509515:20
sdagueclarkb: ok, fixed15:20
*** sdake_ has joined #openstack-infra15:21
clarkbmarcusvrn_: arosen ^ so that is copying whatever you install into jenkins homedir15:21
*** mattfarina has joined #openstack-infra15:22
clarkbhttps://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/pip.conf is likely the source of all the confusion15:22
clarkbalong with https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/pydistutils.cfg patches welcome15:22
arosenclarkb:  ah these get built into my nodepool image from prepare_node.sh when it clones this repo?15:23
openstackgerritMate Lakat proposed a change to openstack-infra/project-config: Experimental patch to try overriding pip config  https://review.openstack.org/13146915:23
clarkbya you end up with those files in jenkins' homedir via puppet15:23
*** lttrl has quit IRC15:24
clarkbwe then override them with nodepool to have per region mirrors15:24
kashyapHi, LaunchPad OpenID seems to be arbitrary strings - https://login.launchpad.net/+id/Nb3wnff,  I wonder if it's possible to s/Nb3wnff/$USERNAME15:24
fungiarosen: so yes you should either modify those to use your own mirror, or if you remove them they'll default to pypi.python.org15:24
clarkbbut if you just set those files to pypi.python.org instead of pypi.openstack.org that is a reasonable value15:24
marcusvrn_clarkb: hmm..nace! I'll try it15:24
clarkbfungi: well I think we should fix upstream15:24
clarkband set to pypi.python.org15:24
fungikashyap: you're wanting to look up the lp username associated with an lp openid?15:24
*** doug-fish has left #openstack-infra15:25
kashyapfungi, Yes15:25
fungikashyap: if so, they have an api call to do that lookup15:25
fungikashyap: you can see where we do it in openstack-infra/jeepyb i think...15:25
fungigetting you a quick example15:25
fungikashyap: http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/update_bug.py#n12115:26
kashyapfungi, Hm, I wasn't thinking of doing it programmatically, but it's good to know it's possible to do it, thank you15:26
fungikashyap: afaik that;s the _olly_ way to do it15:26
fungier, _only_15:26
kashyapSomething a bit more human friendly like: USERNAME.id.openstack.org  or some such15:26
arosenclarkb:  is we fixed this upstream then you guys have to update youre puppet stuff to set this value right?15:27
kashyapBut I'm just making random requests, you can disregard me :-)15:27
arosenor update it via another means15:27
fungikashyap: well, https://openstackid.org/ is running now, though it's not in use yet15:27
clarkbarosen: we already update it via another method15:27
clarkbarosen: because we don't know what region a node will boot in until it boots (particularly in the dib world) we have nodepool write these files for us after boot15:27
arosenclarkb:  ah okay, want me to update it upsteam then?15:27
fungikashyap: and it uses the username in the url instead of a random hash15:28
kashyapfungi, Oh, so eventually, users _will_ have something like: USER.openstackid.org, cool.15:28
aroseni was just going to hack my prepare_node.sh script15:28
*** hdd has quit IRC15:28
kashyapfungi, Thanks for the quick response.15:28
clarkbarosen: ya I think we can just set to pypi.python.org upstream15:29
fungikashyap: https://openstackid.org/jeremy.stanley15:29
*** msabramo has quit IRC15:29
*** msabramo1 has joined #openstack-infra15:29
arosenclarkb: cool, want me to push a patch for this change or are you going to?15:29
clarkbarosen: go ahead, that way my +2 counts :)15:30
fungimrmartin: out of curiosity, do you know how smarcet planned to keep openid urls stable when people change their name?15:30
kashyapfungi, Nice, thank you.15:30
*** dimtruck is now known as zz_dimtruck15:30
mrmartinfungi: I don't know this part15:31
mrmartinjust trying to hack-in the prod one15:31
mrmartininto groups portal15:31
*** zns has quit IRC15:31
fungimrmartin: thanks... i'll try to remember to ask him later15:31
kashyapfungi, I have to separately register or should my launchpad credentials will work?15:32
fungikashyap: it's using the same database as www.openstack.org so the login is the same as you use for your foundation member profile15:32
mrmartinI think it will bring up some issues during the first few weeks15:32
fungimrmartin: i agree, but that's why it's in beta ;)15:32
mrmartinso we can stabilize this openid service before using for something must-work service.15:33
*** whoops has joined #openstack-infra15:33
fungimrmartin: right, i'm hesitant to tie it into, e.g., review.openstack.org or wiki.openstack.org until it's proven on other services for a little while15:34
openstackgerritAaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror so use pypi.python.org instead of pypi.openstack.org  https://review.openstack.org/13147415:34
arosenclarkb:  ^15:34
fungiarosen: there's a review out there for that i think... checking15:34
openstackgerritAaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror to use pypi.python.org instead of pypi.openstack.org  https://review.openstack.org/13147415:34
kashyapfungi, Cool, noted.15:34
clarkboh you know what15:35
clarkbarosen: fungi: our non nodepool built slaves use those files15:35
clarkbwe have two at this point iirc15:35
clarkbproposal.slave.o.o and pypi.slave.o.o15:35
fungiarosen: ideally the comments in https://review.openstack.org/119466 need to be addressed15:35
clarkbdo we want to default to one of our internal regions?15:35
fungiclarkb: yep, that's why i -1'd 11946615:35
clarkbso would pypi.dfw.openstack.org be a better fallback there?15:36
fungiclarkb: probably yes15:36
clarkbarosen: ^15:36
clarkbsorry that just occured to me15:36
*** zz_dimtruck is now known as dimtruck15:36
*** zns has joined #openstack-infra15:37
*** msabramo1 has quit IRC15:40
clarkbsdague: I am +2 on 125095 now. Will be interesting to see how many projects trip over that15:40
*** rmcall has joined #openstack-infra15:40
clarkbsdague: we likely don't want to rollback client changes but instead let the stable branches pull up?15:40
arosenclarkb: so update it to  pypi.dfw.openstack.org  ?15:41
clarkbarosen: ya  Ithink that is a reasonable default for upstream and downstream (as much as we don't want downstream relying on our mirrors)15:41
*** dizquierdo has quit IRC15:41
sdagueclarkb: yeh, I think so15:42
fungiclarkb: i think pypi.python.org is a reasonable default for downstream. pypi.*.openstack.org is not15:42
fungibut i also don't know how we stop them from using it15:43
fungiso meh15:43
clarkbfungi: right, that file has a reasonable default for us15:43
clarkband then others reconsume it...15:43
openstackgerritAaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror to use pypi.dfw.openstack.org  https://review.openstack.org/13147415:43
arosenclarkb: ^15:43
*** SumitNaiksatam has joined #openstack-infra15:44
clarkbafazekas_beer: fungi: did you guys see the results of multinode using floating ips?15:44
clarkbwasn't pretty15:44
clarkbso it is likely that we need to use the internal addresses15:44
jeblairclarkb: where can i find your conclusions?15:44
clarkbjeblair: I left a review on afazekas_beer's change15:44
fungiclarkb: so is it a nat triangulation issue with the vm trying to talk to its global address out in hpcloud's network?15:45
clarkbjeblair: basically ran floating ip change on rax and hpcloud. rax worked but hpcloud did not (you can see those results in the jenkins results on the child change of afazekas_beer's change). I made sure to run afazekas_beer's change against hpcloud as well to make sure this wasn't a common issue but the test that ran on hpcloud for that change passed15:45
clarkbfungi: I didn't debug the specific problem with NAT. merely got the tests to run there and checked the result15:45
*** unicell1 has joined #openstack-infra15:46
clarkbbut the failing tests appeared to deal with volumes15:46
clarkbso I think there is a good chance the iscsi issues afazekas_beer mentioned were hit15:46
jeblairclarkb: did the configuration set the cinder ip address?15:46
fungioh, so maybe this is a devstack/cinder bug?15:46
clarkbjeblair: no, I merely made the changes you suggested to cat the nodepool files15:46
fungiahh, or missed configuration15:46
jeblairclarkb: okay, so i'm not convinced fixed ip is the answer.  i believe that proper cinder configuration should be attempted.15:47
*** unicell has quit IRC15:47
clarkbso cinder needs special config?15:47
*** andreaf has quit IRC15:48
clarkbI think I missed that yesterday15:48
* clarkb checks scrollback15:48
jeblairclarkb: part of the problem with what hp is doing with not locating the actual public ip on their hosts is breaking the assumption that many programs have that they can determine the ip addresses of the machines they run on.15:49
*** bitblt has joined #openstack-infra15:49
jeblairthis affects a surprisingly wide range of programs, apparently including cinder15:50
clarkbso we need to explicitly tell cinder what its IP is even though the IP isn't on a local interface15:50
jeblairbut most of them, (including cinder, fortunately) have a method of saying "don't trust your ip address detection, just use this user-supplied one instead"15:50
clarkbI can look into supplying that in my change15:50
jeblairclarkb: https://github.com/openstack/cinder/blob/master/etc/cinder/cinder.conf.sample#L103915:51
*** msabramo has joined #openstack-infra15:51
fungiclearly the solution is for openstack projects to post a request to http://www.whatismyip.com/15:52
jeblairfungi: that would be the RESTful service way, yeah?15:52
fungiabsolutely... i'll save that cross-project spec proposal for april though15:53
*** dizquierdo has joined #openstack-infra15:54
clarkbjeblair: it looks like we set that value15:54
clarkbhttp://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/localrc.txt.gz SERVICE_HOST is the public IP15:54
clarkb(it is set twice :( but second one is public IP)15:54
*** otherwiseguy has quit IRC15:54
*** terryw has joined #openstack-infra15:54
clarkbthen in devstack we set cinder's my_ip to SERVICE_HOST https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n22415:55
clarkbnote CINDER_SERVICE_HOST defaults to SERVICE_HOST and we don't override15:55
*** groknix has quit IRC15:55
clarkband cinder config you link to implies that the iscsi IP defaults to my_ip. Though I am currently trying to cnfirm that15:55
*** groknix has joined #openstack-infra15:56
*** zaphod42 has joined #openstack-infra15:56
*** slahsme has quit IRC15:56
jeblairclarkb, sdague: left some -1 comments on 12509515:57
clarkbhttps://github.com/openstack/cinder/blob/master/cinder/volume/driver.py#L51-L53 seems to default to my_ip15:57
*** zns has quit IRC15:58
jeblairwow '$my_ip' is actually a thing in oslo.config?  i'm constantly amazed15:58
*** miqui has quit IRC15:59
*** miqui_ has joined #openstack-infra15:59
jeblairclarkb: just so you know what i'm looking for -- i want to know what the actual problem is.  i want to know if this is a cinder bug, or an hploud bug, or a tgtadm bug, etc...15:59
clarkbyes understood16:00
jogoanteaya: yup, I will be replying to it this morning16:00
jgriffithclarkb: clarkb clarkb jeblair anything I ca help with here?16:01
*** fesp has joined #openstack-infra16:01
clarkbjgriffith: possibly. We are trying to get multinode devstack + tempest testing working16:01
clarkbjgriffith: currently when cinder is configured to use a floating IP for my_ip in hpcloud tempest tests fail16:02
jgriffithjeblaihmmm... i tsees my connection is dieing16:02
openstackgerritAaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror to use pypi.dfw.openstack.org  https://review.openstack.org/13147416:02
clarkbjgriffith: they do not fail in rax which does not have NATed floating IPs16:02
clarkbjgriffith: http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/ is link to the logs16:02
*** fesp is now known as flaper8716:02
*** flaper87 has joined #openstack-infra16:03
arosenclarkb:  I updated the rest of the places pypi.openstack.org was used there now here:   https://review.openstack.org/13147416:03
*** ivar-lazzaro has joined #openstack-infra16:03
openstackgerritSergey Skripnick proposed a change to openstack-infra/project-config: Add hook for building rally docker image  https://review.openstack.org/13138916:03
clarkbarosen: I don't think we want wholesale replacement. at least not in the docs like that16:04
*** parimala has quit IRC16:04
*** ivar-laz_ has joined #openstack-infra16:04
clarkbor in bandersnatch configs16:04
clarkbarosen: I think we should start with just those two files16:04
*** gyee has joined #openstack-infra16:04
*** doug-fish has joined #openstack-infra16:04
arosenclarkb:  so my patchset 3?16:05
*** amotoki has joined #openstack-infra16:05
clarkbI think so16:05
clarkbfungi: ^16:05
fungiclarkb: i guess the doc updates can get tacked onto https://review.openstack.org/119466 instead and then rebase it on top of 131474?16:05
clarkbfungi: ya. I see this change as fixing the immediate issue of pypi.o.o doesn't work and some slaves use it16:05
fungiclarkb: fair enough16:06
jeblairsdague: did you test either your "rewrite everything" patch to hideci or your "only show most recent" patch with the new change screen?16:06
clarkbproper cleanup of pypi.o.o can be a subsequent change16:06
jeblairsdague: because it does not appear that the new change screen works right now16:06
openstackgerritAaron Rosen proposed a change to openstack-infra/system-config: Update pypi mirror to use pypi.dfw.openstack.org  https://review.openstack.org/13147416:06
arosenokay i repushed patch set 3 with just those two files being update.d16:06
jeblairfungi, jhesketh, SergeyLukjanov: you approved https://review.openstack.org/#/c/124632/ .  did you test it or verify that it was tested?16:07
sdaguejeblair: I did not16:07
openstackgerritJames E. Blair proposed a change to openstack-infra/system-config: Revert "refactor of hideci for readability and modularity"  https://review.openstack.org/13148316:07
*** ivar-lazzaro has quit IRC16:08
fungijeblair: oops, i tested it in-browser in the old change screen before approving, but forgot to test against the new change screen16:08
fungijeblair: we can revert for now i guess16:08
sdaguejeblair: let me take a look to see if it's an easy fix16:08
*** mmaglana has joined #openstack-infra16:08
*** jgriffit1 has joined #openstack-infra16:09
clarkbjgriffith: the ndoe should be wide open between each other but I will double check16:09
*** jgriffit1 is now known as jdg16:09
*** jgriffith has left #openstack-infra16:09
*** jdg is now known as jgriffith16:10
*** VijayTripathi has joined #openstack-infra16:10
*** jgriffith is now known as Guest5826716:10
VijayTripathiHello everyone16:11
*** carl_baldwin has quit IRC16:11
clarkbGuest58267: -A openstack-INPUT -s -j ACCEPT and -A openstack-INPUT -s -j ACCEPT in http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/iptables.txt.gz should cover us16:11
clarkbGuest58267: at least on the cinder control plane side which is being connected to? I am checking the other side for completeness16:11
clarkbhttp://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/ has the same rules so I don't think firewall is at play. I am still digging through service logs to see what I can see16:12
arosenfungi: clarkb thanks for the quick approval, hopefully I this should let me get the CI in ordering order again :)16:12
fungidtroyer: dns updates for devstack.org are implemented now. the old ip addresses were for the domain address record and for the www subdomain address16:12
sdaguejeblair: zomg, new screen provides no semantic markup?16:13
Guest58267clarkb: not sure, but looking at logs I'm not sure those hosts are talking at all... ie able to ping or ssh regardless of the iscsi tgt16:13
fungidtroyer: now the domain has a and aaaa address records matching static.openstack.org, and www is a cname to the same16:14
*** Guest58267 is now known as jdg16:14
jdgsorry... hopefully I have a connection that is stable now16:14
jeblairsdague: no.  it just might be possible that the complexity you objected to might have been required in order to function with the new screen.16:14
openstackgerritSergey Skripnick proposed a change to openstack-infra/project-config: Add hook for building rally docker image  https://review.openstack.org/13138916:15
sdagueyeh... I'm trying to figure out how we can have nice things with the new change screen at all16:15
*** emagana has joined #openstack-infra16:15
sdagueit's amazing how they stripped out all the semantically meaningful content there16:15
clarkbGuest20522: http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/ has errors currently looking at those now as they are the first bread crumbs that are useable16:15
clarkber jdg ^16:16
*** msabramo has quit IRC16:16
jeblairsdague: the previous version collected the comments from either change screen and put them in a rather simple, easy to work with data structure.  i may be biased, but i think that perhaps instead of changing everything, you could probably start with that.16:16
sdaguejeblair: so the complexity of that made a bunch of higher level manipulations really hard16:17
jeblairsdague: it literally just pulled out the comments and put them in a data structure16:17
jeblairsdague: it was a function called "ci_find_comments"16:17
sdaguejeblair: right but it didn't have the grouping16:18
dtroyerfungi: looks good from here, thanks again16:18
jeblairsdague: sure, build on top of that16:18
clarkbjdg: oh I see, for some reason nova is using the internal addresses and not the public addresses16:18
clarkbjdg: so the ssh key hasn't been added16:18
sdaguejeblair: that's actually what I did16:18
clarkbjeblair: ^ so it may be that nova is being smart here16:18
jeblairsdague: it output a list of comments, each item in the list was a hash with attrs like 'name', 'psnum', 'results', 'comment'16:18
* clarkb digs more16:18
jeblairsdague: you could add 'pipeline' to that pretty easily16:18
jeblairsdague: and then collaps16:19
sdagueso, I can probably rework ci_parse_comments to work in the new screen as well, we'll have less info, but maybe that's ok16:20
jeblairsdague: why would we have less info?16:21
jeblairsdague: the same information is provided in both change screens16:21
sdagueit's actually not, email isn't, for instance16:21
jeblairsdague: we need email?16:21
*** stevemar has joined #openstack-infra16:21
sdagueI don't know, I'm just saying they aren't actually exactly the same.16:22
*** zns has joined #openstack-infra16:22
*** ildikov has quit IRC16:22
*** ivar-lazzaro has joined #openstack-infra16:22
jeblairsdague: anyway, i don't care whether you extend what i wrote or completely rewrite it.  up to you16:22
jeblairhowever, we do need something that works on both change screens now16:23
jdgclarkb: yeah... also this may help going forward: https://review.openstack.org/13149116:23
jeblairand it does not sound like this is a quick fix16:23
sdagueyeh, give me an hour to see how bad this is going to be16:23
asselinclarkb, arosen marcusvrn_ I see where the openstack pypi setup is in my repo. I will delete it: https://github.com/rasselin/os-ext-testing/blob/master/puppet/modules/os_ext_testing/templates/jenkins_job_builder/config/macros.yaml.erb#L6416:23
jeblairsdague: okay16:23
*** freemanbrandon has joined #openstack-infra16:24
asselinclarkb, arosen marcusvrn_ this is where i set my own pip mirror for prepare_node.sh: https://github.com/rasselin/os-ext-testing/blob/master/puppet/modules/os_ext_testing/files/nodepool/prepare_node_devstack_http_proxy.sh#L2316:24
clarkbhttp://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/ nova's my_ip is the internal IP...16:25
clarkbcinders is confirmed to be correct16:25
*** ivar-laz_ has quit IRC16:25
jdgclarkb: cool...16:26
*** ashaeron has quit IRC16:26
*** miqui_ has quit IRC16:26
*** zns has quit IRC16:26
*** reed has joined #openstack-infra16:27
naggappan@fungi if i use pre_test_hook it tried to change remote url before downloading . But if i add in post_test_hook then i am able to set my repository url . Is this the right way? or we need to use pre_test_hook16:27
*** freemanbrandon has quit IRC16:28
funginaggappan: well, you need to update the version in the git repository cache devstack-gate sets up. devstack shouldn't re-clone if it's already in the cache i think, but i've not actually tried to override a repository that way. you may have more luck talking to people who run third-party ci systems to find out how they're actually solving this problem16:29
*** dangers_away is now known as dangers16:29
*** zz_gondoi is now known as gondoi16:29
*** doug-fish has quit IRC16:29
*** msabramo has joined #openstack-infra16:31
*** dimtruck is now known as zz_dimtruck16:32
clarkbjeblair: https://review.openstack.org/131492 fixes at least some of the problem16:32
clarkb(I think, I don't actually grok all of the reasons devstack might use one var or another in these cases)16:32
*** unicell1 has quit IRC16:32
*** yamahata has quit IRC16:32
*** mriedem has quit IRC16:32
*** unicell has joined #openstack-infra16:32
*** dims__ has joined #openstack-infra16:33
jeblairclarkb: but does that affect cinder.conf?16:33
*** thedodd has joined #openstack-infra16:33
*** jaypipes is now known as jaypipes-afk16:33
clarkbjeblair: no, but I think cinder tests may have been failing due to nova failures (tests were along the lines of boot from volume)16:33
jeblairclarkb: gotcha16:34
clarkbjeblair: I am not sure this will fix all of the problems, but it is definiately a problem16:34
naggappan@fungi what you mean by "update the version in git repository cache devstack-gate" is there any cache for all repositories and i need to update over there also ?16:35
openstackgerritA change was merged to openstack-infra/system-config: Update pypi mirror to use pypi.dfw.openstack.org  https://review.openstack.org/13147416:36
*** zz_dimtruck is now known as dimtruck16:37
*** ivar-lazzaro has quit IRC16:37
*** zns has joined #openstack-infra16:38
*** ivar-lazzaro has joined #openstack-infra16:38
funginaggappan: the loop over setup_project in setup_workspace checks out all projects listed in $PROJECTS into $DEST16:39
*** bhuvan has joined #openstack-infra16:39
fungier, clones them16:39
funginaggappan: that happens before the pre_test_hook is called16:42
*** Guest74700 has quit IRC16:43
gondoihow can I kick off tests again on a review?16:44
gondoiit passed initial tests, but then failed verification of an already passing test16:45
*** markmcclain has quit IRC16:45
*** harlowja_away is now known as harlowja16:46
clarkbgondoi: leave a comment of 'recheck'16:46
gondoiawesome, thank you16:46
*** Longgeek has quit IRC16:47
naggappan@ok fungi thanks. Let me try with the post_test_hook itself if its not working, let me look for other solution16:47
*** bhuvan_ has joined #openstack-infra16:47
funginaggappan: post_test_hook runs after your tests, so probably won't do what you want16:48
*** doug-fish has joined #openstack-infra16:49
*** bhuvan has quit IRC16:50
*** bhuvan_ is now known as bhuvan16:50
*** tnurlygayanov has joined #openstack-infra16:51
sdaguejeblair: ok, I have a version working with new change screen, got to fix one thing I broke on old in the process16:53
*** jgriffit1 has joined #openstack-infra16:54
sdaguethe date parse is annoying, because in old we get full date, and in new we only get display date, but such it is16:54
*** VijayTripathi has quit IRC16:54
*** jgriffit1 is now known as jgriffith16:55
*** jgriffith is now known as Guest1501816:56
*** sarob has joined #openstack-infra16:56
*** jgriffith has joined #openstack-infra16:57
asselinclarkb, I reconfigured my tests to not use hooks as suggested. I abandoned the change.17:00
asselinclarkb, sdague, let me know if there's still interest in having the devstack-gate  hook refactoring change: https://review.openstack.org/#/c/123578/17:01
marcusvrn_fungi: clarkb asselin arosen thx all! The CI is working now with python.org repo17:01
clarkbasselin: ya I think we can leave that one up17:01
*** ivar-laz_ has joined #openstack-infra17:01
*** zns has quit IRC17:02
*** freemanbrandon has joined #openstack-infra17:02
*** ildikov has joined #openstack-infra17:02
*** ivar-lazzaro has quit IRC17:04
openstackgerritSean Dague proposed a change to openstack-infra/system-config: fix hideci for new change screen  https://review.openstack.org/13150017:05
sdaguejeblair: ^^^17:05
*** msabramo has quit IRC17:07
*** msabramo1 has joined #openstack-infra17:07
*** zns has joined #openstack-infra17:07
clarkbmy devstack change doesn't seem to be broken during check testing so thats good17:09
*** carl_baldwin has joined #openstack-infra17:10
clarkbI wonder if I can get d-g to check out that change when running experimental tests17:10
jeblairclarkb: that's the idea behind adding cross-repo deps to zuul, but afaik that's not done yet17:10
jeblairclarkb: though i assume with devstack-gate, you could bash something in there17:11
clarkbya I think I can just bash something into setup_workspace17:12
*** jpich has quit IRC17:12
fungiyeah, an explicit checkout of that change into the cached devstack repo after or instead of the reset to master ought to do it17:13
*** IvanBerezovskiy has left #openstack-infra17:13
*** changbl has joined #openstack-infra17:14
*** yaguang has quit IRC17:17
*** zns has quit IRC17:20
*** zns has joined #openstack-infra17:20
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Test nodepool provided addresses in aiopcpu setup  https://review.openstack.org/13129717:21
clarkbthere it is bashed in17:21
clarkbI am going to do the trick to force nodes onto hpcloud for 2-node before I check experimental that change17:21
*** lttrl has joined #openstack-infra17:22
*** VijayTripathi has joined #openstack-infra17:23
*** bhuvan has quit IRC17:23
*** bhuvan has joined #openstack-infra17:24
*** MaxV has quit IRC17:25
*** Ryan_Lane has joined #openstack-infra17:25
*** MaxV has joined #openstack-infra17:25
*** dprince has quit IRC17:27
*** tomoe has quit IRC17:29
*** stevemar is now known as notstevemar17:29
*** patrickeast has joined #openstack-infra17:29
*** MaxV has quit IRC17:30
*** naggappan has quit IRC17:31
*** achanda has joined #openstack-infra17:32
*** amotoki has quit IRC17:33
clarkbhttps://jenkins02.openstack.org/job/check-tempest-dsvm-aiopcpu/15/ woot that is running now17:34
clarkbshould give us good info17:35
*** terryw has quit IRC17:37
*** jp_at_hp has quit IRC17:38
*** mriedem has joined #openstack-infra17:38
*** vhoward has left #openstack-infra17:40
*** achanda has quit IRC17:42
*** freemanbrandon has quit IRC17:45
*** amcrn has joined #openstack-infra17:46
*** morganfainberg is now known as alsonotstevemar17:46
*** alsonotstevemar is now known as morganfainberg17:47
*** achanda has joined #openstack-infra17:47
*** garyh has quit IRC17:49
*** achanda has quit IRC17:49
*** jgallard has quit IRC17:49
*** achanda has joined #openstack-infra17:49
*** sarob_ has joined #openstack-infra17:50
*** freemanbrandon has joined #openstack-infra17:50
*** markmcclain has joined #openstack-infra17:51
*** emagana has quit IRC17:52
*** emagana has joined #openstack-infra17:53
*** ZZelle_ has joined #openstack-infra17:54
*** markmcclain has quit IRC17:55
*** jdg has quit IRC17:56
*** emagana_ has joined #openstack-infra17:57
*** emagana has quit IRC17:57
*** sdake_ has quit IRC17:59
*** chandankumar has quit IRC17:59
*** zns has quit IRC17:59
*** dimtruck is now known as zz_dimtruck17:59
*** rmcall has quit IRC18:00
*** otherwiseguy has joined #openstack-infra18:00
*** zz_dimtruck is now known as dimtruck18:01
*** rmcall has joined #openstack-infra18:02
*** ivar-laz_ has quit IRC18:03
*** ivar-lazzaro has joined #openstack-infra18:04
SergeyLukjanovjeblair, re hideci - I've been thinking that I test it... heh18:08
SergeyLukjanovjeblair, sdague, how are you testing hidejs changes?18:08
openstackgerritSean Dague proposed a change to openstack-infra/project-config: enforce stable requirements  https://review.openstack.org/12509518:09
*** ociuhandu has joined #openstack-infra18:09
fungiSergeyLukjanov: paste the raw version of the js file contents into the browser console18:09
sdagueyeh, what fungi said18:09
*** sarob has quit IRC18:09
fungiand then browse around review.o.o and see what it does18:09
sdaguecopy / paste the whole file into the chrome js console (or ff I think)18:09
lifelessmordred: failing open? context? I am at a loss18:09
SergeyLukjanovfungi, hm, that what's I've done for this patch... sounds like there was lack of coffeine in my blood and I've done it incorrectly18:09
*** dprince has joined #openstack-infra18:10
SergeyLukjanovfungi, sdague, thanks18:10
*** _nadya_ has quit IRC18:10
sdagueSergeyLukjanov: did you use the new change screen?18:10
SergeyLukjanovsdague, yup18:10
sdaguethe issue was that it didn't work if you had new change screen preferences on18:10
*** tgohad has joined #openstack-infra18:10
sdague131500 fixes it18:11
SergeyLukjanovsdague, oh, got it (added note to test that changes on both screen prefs)18:11
mordredlifeless: succeeding when no tests are found18:11
clarkbmordred: is this the nova thing?18:11
lifelessmordred: strangely I got pushback from [some] cpython devs http://bugs.python.org/issue1823218:11
clarkbmordred: testr can't do anything about that18:12
fungimordred: lifeless: in this case it turned out that the module being imported had no main() so the import was a successful noop, and this we never got a testrepository file to inspect to see how many tests were run18:12
clarkbthe way nova (and a lot of projects run their tests) is something like python -m oslo.lockutils testr18:12
lifelessmordred: but as we've discussed I'm working on these things upstream18:12
clarkbif oslo.lockutil exits 0 then testr is never even invoked18:12
lifelessclarkb: -wat-18:12
clarkblifeless: don't get me started18:13
lifelessclarkb: that makes no sense to me18:13
fungiyeah, seems like a bad choice to me18:13
clarkblifeless: we should make a club18:13
lifelessclarkb: you don't wrap the orchestrator18:13
sdagueso... we're fixing that18:13
clarkblifeless: no no they do18:13
lifelessclarkb: _why_18:13
clarkblifeless: global locking18:13
funginot for much longer apparently ;)18:13
lifelessstill not following18:13
lifelesstestr is concurrent18:13
clarkblifeless: the wrapper sets up global locks18:13
sdaguebecause oslo lockutils doesn't work without extra setup18:13
*** tsg has quit IRC18:13
clarkband its silly18:13
sdaguehonestly, I think this is beer talk in paris :)18:14
sdaguethere are many layers of silly here18:14
clarkbwell I tried to -1 it back in the day18:14
sdagueI'm hoping to kill a few of them today18:14
dhellmannclarkb: the wrapper sets up the lock DIRECTORY not the locks themselves. It sets an environment variable.18:14
lifelesssdague: that should be per test backend though, with process long fixtures to isolate each backend, or you'll have terrible trrouble with N-context tests.18:14
clarkbdhellmann: but why18:14
clarkbdhellmann: just put it in the tests18:14
lifelessclarkb: I now understand why- see my comment^18:14
dhellmannclarkb: sure. I'm just explaining what it actually does.18:14
lifelesssdague: sure, it can be a beer discussion, or I'll just get a patch together when I finish this EPIC yak shave18:15
clarkbdhellmann: back when I was more angry about it I was told it was for db tests to have a global lock18:15
clarkbdhellmann: except you don't need a global lock for db tests18:15
dhellmannclarkb: the script doesn't create any locks18:15
clarkbok tempest is still failing with my bashed in devstack change. will need to look at logs more when it is done18:16
clarkbanyways we cargo culted a thing that should never have been written in the first place. It is very frustrating that we don't scrutinize the cargo when we cult18:16
lifelesssdague: but in terms of fitting in with the rest of the test architecture, there really isn't anything to discuss except 'how do we fix this'18:16
clarkbhttp://logs.openstack.org/97/131297/2/experimental/check-tempest-dsvm-aiopcpu/e49cb03/logs/ for the logs. looks like nova my_ip is correct now to the n-cpu logs18:20
sdaguelifeless: sure, mostly I think the beer talk is to look at some other silly things we've let creep in and figure out which bits we can unwind18:22
mtreinishclarkb: http://logs.openstack.org/97/131297/2/experimental/check-tempest-dsvm-aiopcpu/e49cb03/logs/ looks suspicious18:22
arosenFrom time to time i see my nodepool nodes go offline in jenkins. Right now the error message is: Disconnected by anonymous : Offline due to Gearman request18:23
mtreinishwhich matches up with the resize test failures18:23
arosenis there anything i can do about this?18:23
*** garyh has joined #openstack-infra18:23
*** carl_baldwin has quit IRC18:24
fungiarosen: that is the node being offlined by the jenkins-gearman plugin, generally because either a job completed or was aborted18:24
arosenthe jenkins log says: WARNING: Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information18:24
clarkbmtreinish: ya so I fixed nova's my_ip which got us past the ssh errors18:24
clarkbmtreinish: but now qemu-img is failing on both nodes and those operations look local18:24
lifelessclarkb: oh18:25
fungiarosen: that warning is generally benign... just means your job created one or more files outside the workspace (unavoidable if you're running devstack-gate)18:25
arosenfungi:  oh maybe you're right let me dig closer.18:25
lifelessclarkb: that failing open... what was the return code from the lockutils thing?18:25
clarkblifeless: 018:25
lifelessclarkb: waaaaat18:25
clarkblifeless: the reason is oslo.concurrency remove their if __main__ check18:25
clarkbso python -m happily imported the module and did nothing returned 018:26
lifelessclarkb: I'm just surprised and astounded at python right now18:26
*** zns has joined #openstack-infra18:26
arosenfungi:  also it's weird because nodepool list ; doesn't show my nodes are in 'use' even though jenkins is using them.18:26
lifelessclarkb: and then I remembered, this is a consequence of python 118:26
lifelessclarkb: and there being no formal 'this is executable' in python land18:26
fungiarosen: if they all continue to show "ready" then your nodepool probably isn't successfully getting events from the zeromq event publisher plugin on your jenkins master18:27
*** zns has quit IRC18:27
jeblairclarkb: my_ip =
jeblairclarkb: http://logs.openstack.org/97/131297/2/experimental/check-tempest-dsvm-aiopcpu/e49cb03/logs/
clarkbjeblair: yup18:27
jeblairclarkb: i dunno how to run an openstack....18:27
clarkbjeblair: so I think my devstack change fixes that bit18:27
jeblairclarkb: but i feel like saying "my ip is .254" on a host whose ip is ".255" may have problems18:28
*** zns has joined #openstack-infra18:28
clarkboh is the ip wrong?18:28
jeblairclarkb: yeah, i think it's correct on the .254 host but incorrect on the .25518:28
clarkbwell its set to SERVICE_HOST everywhere18:28
clarkbI think my_ip is really "where is the controller ip"18:28
clarkbbut I could be wrong18:29
jeblairclarkb: i mean, maybe "my_ip" should be .254 on .25518:29
jeblairclarkb: but i can not reason about such a thing, so i'm going to go back to what i was doing before :)18:29
clarkbjeblair: :) probably a good idea18:29
*** thedodd has quit IRC18:30
dhellmanngit experts: We may have a case where we want to take a module from oslo-incubator and put it in an existing library repository (oslo.utils). Is it possible to do that and preserve the history? I think I understand how to get the 2 commit trees into one repo and merged, but not with gerrit in the mix.18:32
*** garyh has quit IRC18:33
dhellmannit's not the end of the world if we can't, but I want to explore options before just copying the file and losing the history18:33
jeblairdhellmann: gerrit is just another git repo; we can push to a branch and then make a merge commit of that branch into master18:34
*** HeOS has quit IRC18:34
jeblair(and the merge commit can be reviewed)18:34
*** otherwiseguy has quit IRC18:34
*** packet has joined #openstack-infra18:34
clarkbI have held the neutron tests's pair of nodes to poke around on disk18:35
dhellmannjeblair: ok, cool. I'm happy with the "it's possible" answer for now but may come back for more detailed instructions after the summit.18:35
arosen fungi is rebooted the box and still seems like the node pool nodes aren't going to ready. Is there a specific log file i should look at for this?18:35
clarkbwell thats interesting18:37
clarkbit looks like n-cpu on the primary node is running qemu-img for a host on the sub node18:37
clarkband so the file does not exist because we are running on the wrong node18:38
clarkbmaybe my_ip is actually wrong?18:38
*** tonytan4ever has quit IRC18:38
clarkbwhy is nova ssh'ing to localhost then18:38
fungiarosen: you should check that your zeromq event publisher plugin on the jenkins master is enabled, and probably look for traffic for that address/port18:39
clarkbso maybe what I need to do is abandon the devstack change and set HOST_IP to the value in the appropriate /etc/nodepool file18:39
*** dane_leblanc has quit IRC18:40
clarkbjeblair: except that if we grow beyond 1 subnode then those files are not sufficient to know what the external IPs are18:40
arosenfungi:  ah okay it's started working again. I'm not sure if it was related to running this though:18:40
arosensudo jenkins-jobs --flush-cache update /etc/jenkins_jobs/config18:40
*** carl_baldwin has joined #openstack-infra18:40
arosenseems like it shouldn't be...18:40
clarkband in this case cinder works because it doesn't run on two nodes?18:41
jeblairdhellmann: cool.  we also did once merge a git history into a repo without even a common shared point on the dag (two completely independent repos merged).  if that's possible, anything is :)18:41
dhellmannjeblair: it's like magic!18:41
clarkbso why would nova ssh to itself though18:41
jeblairclarkb: not following: 18:39 < clarkb> so maybe what I need to do is abandon the devstack change and set HOST_IP to the value in the appropriate /etc/nodepool file18:41
clarkbjeblair: I believe my_ip is supposed to be the IP of the localhost18:42
clarkbjeblair: but in hpcloud we have no way of getting that info reliably18:43
clarkbfor the public floating ip18:43
*** sdake_ has joined #openstack-infra18:43
clarkbhttp://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/ is failing to ssh to the local ip18:43
jeblairclarkb: so you want to set my_ip to a value supplied by nodepool, but need that to scale to n-hosts that are not the primary18:43
clarkbjeblair: if we want to use the public floating ips on each node: yes18:43
*** tonytan4ever has joined #openstack-infra18:44
jeblairclarkb: i agree that is not possible now, that it is desirable, and that we should enhance nodepool to supply that information18:45
clarkbI am going to go ahead and try it with the sub_nodes file data now to see if I can get this to work if the IPs are correct18:45
clarkbalso I have no idea why nova does ssh like that18:45
jeblairclarkb: ++18:45
clarkbsdague: jogo ^ do you guys know why nova does that?18:47
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Test nodepool provided addresses in aiopcpu setup  https://review.openstack.org/13129718:51
clarkbhttps://jenkins04.openstack.org/job/check-tempest-dsvm-aiopcpu/9/ is running with external IPs as HOST_IP now18:52
*** cdent has quit IRC18:54
*** dane_leblanc has joined #openstack-infra18:55
*** baoli has quit IRC18:55
clarkbit looks like nova does that ssh for "migration" operations where that operation can be entirely local?18:56
*** andreykurilin_ has joined #openstack-infra18:56
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add current node IP to /etc/nodepool  https://review.openstack.org/13153218:58
clarkbjeblair: that should probably come with a doc update (the other files are documented)18:59
*** zns has quit IRC19:00
fungimeeting time...19:00
*** jaypipes-afk is now known as jaypipes19:01
*** cinderbld has joined #openstack-infra19:01
*** pelix has quit IRC19:02
*** emagana_ has quit IRC19:02
*** naggappan has joined #openstack-infra19:02
*** dgenin has joined #openstack-infra19:03
*** emagana has joined #openstack-infra19:03
dgenindtroyer: do you have a minute for a cinder lvm question?19:03
*** dane_leblanc has quit IRC19:04
*** emagana_ has joined #openstack-infra19:05
*** emagana has quit IRC19:05
*** sdake_ has quit IRC19:06
*** tonytan4ever has quit IRC19:06
*** dprince has quit IRC19:07
*** AJaeger has joined #openstack-infra19:07
*** mestery has quit IRC19:07
*** dane_leblanc has joined #openstack-infra19:09
*** mestery has joined #openstack-infra19:09
*** darvon has quit IRC19:15
*** darvon has joined #openstack-infra19:16
*** sdake_ has joined #openstack-infra19:16
*** mjturek has quit IRC19:16
*** andreykurilin_ has quit IRC19:17
*** arosen has quit IRC19:17
*** andreykurilin_ has joined #openstack-infra19:18
*** garyh has joined #openstack-infra19:19
*** jedimike has quit IRC19:21
*** mjturek has joined #openstack-infra19:29
*** teran has joined #openstack-infra19:29
*** mjturek has left #openstack-infra19:30
*** gyee has quit IRC19:31
*** tonytan4ever has joined #openstack-infra19:31
*** dizquierdo has quit IRC19:32
*** dgenin has quit IRC19:33
*** teran_ has quit IRC19:33
jogoclarkb: it uses ssh for libvirt based migration if no shared disk19:34
jogoclarkb: that doesn't have to be the public IP, that can be the private IP19:34
clarkbjogo: right but it does so even when talking to localhost for a disk resize apparently19:34
clarkbjogo: we want it to be the public ip19:34
clarkbwhat was confusing to me is nova basically does `ssh localhost mkdir`19:35
*** arosen has joined #openstack-infra19:35
jogoclarkb: a resize up needs to be rescheduled19:35
jogoclarkb: fewer code paths that way19:35
clarkbits a longer code path...19:35
jogoclarkb: why do you want it to be the public IP19:35
clarkbyou are relying on network services that are not required19:35
jogoclarkb: how much overhead would that incur?19:36
jogoi am not convinced its worth changing, but I do agree it is odd19:36
*** pack3t has joined #openstack-infra19:39
*** pack3t has quit IRC19:39
openstackgerritSergey Lukjanov proposed a change to openstack-infra/project-config: Add Juno stable-compat-jobs to trunk sahara client  https://review.openstack.org/13154219:40
*** e0ne has quit IRC19:42
clarkbjogo: Ithink the biggest overhead is for humans19:43
clarkbjogo: you immediately assume remote connection with ssh19:43
jogoclarkb: that is fair19:43
jogoI wonder what dansmith thinks ^19:44
clarkb(like I did and completely went down the wrong hole for a while)19:44
pleia2zaro: so this has been bothering me for quite some time: http://people.ubuntu.com/~lyz/openstack/OS-gerrit-screenshot-10.28.2014.png (see bottom)19:44
pleia2turns out chrome on mobile does this awful FontBoosting thing that seems mostly broke, but there is a hacky fix19:44
openstackgerritSergey Lukjanov proposed a change to openstack-infra/project-config: Use common integrated jobs for sahara  https://review.openstack.org/13154319:44
pleia2would this be upstream, or is there a way for us to modify our instance? I could only find the css I want to change in our gerrit repo :\19:45
dansmithjogo: clarkb: reading19:45
jogoclarkb: btw OT, we broke py26 in nova already so hopefully we can deal with any concerns complaints early in the cycle19:45
clarkbthat was fast19:46
jogoclarkb: it was intentional19:46
*** hdd has joined #openstack-infra19:46
dansmithjogo: what's the question? why we ssh to localhost?19:46
jogodansmith: yes, for resize19:46
jogodansmith: I thought it was to use the same code path to make things easier19:47
dansmithjogo: is it part of the libvirt connection or the disk stuff?19:47
dansmithjogo: right, I'm sure it's that19:47
jogodansmith: http://logs.openstack.org/97/131297/1/experimental/check-tempest-dsvm-aiopcpu/9ac7127/logs/
jogodansmith: clarkb found it counter intuitive making debugging much harder, which is a reasonable concern19:48
*** _buzz_ has joined #openstack-infra19:48
dansmithbut was it fixed by setting my_ip properly?19:48
*** julim has quit IRC19:48
clarkbdansmith: no, it was proper before19:49
clarkbI thought it was wrong because I didn't realize nova would ssh to my_ip19:49
clarkbbut I had to go around and do it wrong before I figured it out :)19:49
dansmithokay, I guess I'm missing something19:49
dansmithbecause this is what you were hitting before you changed it at all, right?19:49
clarkbour nodes have two IPs19:50
clarkba public IP and a private IP19:50
clarkbwe are trying to run stuff on the public IP19:50
clarkbbut my_ip was the private IP19:50
zaropleia2: not sure.  since we are behind upstream, it might have already been fixed upstream.19:50
clarkbso when nova did ssh my_ip it hit an address for which it did not have an accepted ssh host key19:50
*** mwagner_lap has quit IRC19:50
clarkbI thought it was ssh'ing to the private IP of the other host because why would you ssh to local ip19:50
dansmithI see, but don't we ssh to both?19:51
*** _buzz has quit IRC19:51
pleia2zaro: ok, I'll dig around and see, is there a vanilla upstream gerrit instance I can look at somewhere? (not openstack, just something some project is using)19:51
dansmithi.e.: ssh to localhost and ssh to the remote machine19:51
clarkbdansmith: not in this case, its just a local resize19:51
dansmithclarkb: there's no such thing19:51
clarkbyou have to move a node to resize it?19:51
dansmithclarkb: resize is expected to move the instance to another machine19:51
clarkbthats ok19:52
dansmithclarkb: the hack in the config file is just there to support single-node testing19:52
jogodansmith: don't we set a flag in devstack to turn on local resize19:52
jogodansmith: we should drop that for multi-node right?19:52
dansmithjogo: we used to, otherwise resizes would fail because there were no hosts19:52
openstackgerritRamy Asselin proposed a change to openstack-infra/project-config: Refactor puppet-apply jobs into template  https://review.openstack.org/13154819:52
dansmither, I read that wrong19:52
zaropleia2: gerrit-review.googlesource.com is pretty close to upstream.19:52
dansmithjogo: right, we should drop that in multinode19:52
pleia2zaro: thanks19:52
clarkbin any case ssh some local ip that isn't localhost but is effectively localhost is super duper confusing19:53
jogo    iniset $NOVA_CONF DEFAULT allow_resize_to_same_host "True"19:53
jogoin lib/nova in devstack19:53
dansmithclarkb: well, I guess I can see that, but I'm not sure I think it's worth changing just for the debugging case19:53
pleia2zaro: aha, not fixed!19:54
dansmithbecause I htink that code is designed to be able to be run from potentially a third party location19:54
dansmithwhere it's just ssh $src ; ssh $dst19:54
clarkbeven `if ip == my_ip then ssh localhost` would help19:54
clarkbbut that would potentially cause lots of deployments to need to go add the host key for localhost19:55
clarkbso probably not worth the pain of changing19:55
dansmithclarkb: right19:55
dansmithclarkb: was just going to say that :)19:55
dansmithclarkb: the way it is, makes it easy to distribute a known_hosts file19:55
*** naggappan has quit IRC19:55
*** otherwiseguy has joined #openstack-infra19:56
jogodansmith: just running the same tests but with  allow resize and allow migrate to same hosts off will be huge19:59
sdagueresize_to_same_host isn't a live op20:00
sdagueunless a thing changed20:00
dansmithjogo: I'm not sure what you mean20:00
*** HeOS has joined #openstack-infra20:00
jogodansmith: w.r.t. testing20:00
jogosdague: allow_resize_to_same_host is defined in nova.compute.api20:01
*** emagana_ has quit IRC20:01
sdaguejogo: right, but I thought everything that used that path didn't need ssh20:01
jogosame for allow_migrate_to_same_host20:01
AJaegerjeblair: could you review https://review.openstack.org/#/c/129435/ , please? I'd like your comments on python 2.6 removal from infra projects20:01
sdagueallow_migrate_to_same_host ... on the other hand20:01
clarkbjeblair: new thing I have discovered with this multinode stuff. we are using the pivate address in all hpcloud tests today20:01
clarkbalso down to 2 failures20:01
*** emagana has joined #openstack-infra20:02
clarkbin http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-aiopcpu/065b025/logs/ VM netowrking appears to be broken20:02
jeblairAJaeger: clarkb pointed out in the infra meeting we actually do need 2.6 for zuul; do you want me to -1 that and request removal, or add it back with a followup patch?20:02
clarkbdoes nova network require everything to be on the same layer 2 network?20:02
*** zns has joined #openstack-infra20:02
jogosdague: with allow_resize_to_same_host off it just adds the current host to the ignored_hosts list20:03
dansmithyeah, I don't think setting that has any effect on what we do and don't do re: ssh20:03
*** baoli has joined #openstack-infra20:03
jeblairfungi: i'm stopping by new york on my way to paris: https://www.softwarefreedom.org/events/2014/sflc-10th-anniversary-conference/20:03
pleia2zaro: so here's the issue: http://people.ubuntu.com/~lyz/openstack/OS-gerrit-screenshot-10.28.2014.png & http://people.ubuntu.com/~lyz/openstack/Gerrit-gerrit-screenshot-10.28.2014.png ; if anyone cares, it's a one line fix in our gerrit-gwtui/src/main/java/com/google/gerrit/client/gerrit.css20:03
jogosdague: what would be the best way to make it optional to turn on *_to_same_host20:03
pleia2zaro: but maybe it doesn't matter and I shouldn't be opening reviews in chrome on my phone :)20:04
jeblairpleia2: is your puppetconf talk available on video?20:04
fungijeblair: oh, that looks exciting!20:04
sdaguejogo: tc meeting now, can noodle on it after20:04
pleia2jeblair: yep, let me grab the link20:04
jogosdague: kk20:04
clarkbjeblair: I think that makes me somewhat partial to starting with those internal addresses on the first iteration of this20:05
AJaegerjeblair: Let me remove it and resubmit.20:05
*** zns has quit IRC20:05
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Added support for Rich Text Publisher Plugin  https://review.openstack.org/13045520:05
pleia2jeblair: https://puppetlabs.com/presentations/how-open-source-your-puppet-configuration-elizabeth-krumbach-joseph-hp20:05
pleia2puppet-monty loses his glasses20:06
*** emagana has quit IRC20:06
clarkbdansmith: jogo: http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-aiopcpu/065b025/console.html has no route to host and ssh timeouts when trying to hit the cirros VM20:06
*** ayoung has quit IRC20:07
dansmithclarkb: neutron or n-net?20:07
clarkbdansmith: nova network20:07
jeblairpleia2: oh no!20:07
*** freemanbrandon has quit IRC20:08
jogoclarkb: so how are you setting up the networking underneath20:08
*** thedodd has joined #openstack-infra20:08
jogoclarkb: as in you have the aio and the ncpu nodes20:08
clarkbjogo: lol whatever devstack does20:08
*** todd_dsm has quit IRC20:08
openstackgerritAndreas Jaeger proposed a change to openstack-infra/project-config: Remove python26 jobs from various projects  https://review.openstack.org/12943520:08
jogohow does aio know how to route which VM IPs  to npcu20:08
clarkbnote that neutron has the same failures too http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-neutron-aiopcpu/df58711/20:08
clarkbjogo: I have no idea20:08
jogoclarkb: I dont't think devstack knows how to do that20:09
dansmithclarkb: multihost or normal?20:09
clarkbdansmith: both are mutlihost20:09
*** markmcclain has joined #openstack-infra20:09
clarkbjogo: I do not know how nova net expects that to be configured20:09
clarkbdo they all have to be on the same layer 2 network?20:09
AJaegerclarkb, jeblair, fungi: 129435 should now be fine20:09
jogoclarkb: I don't remember off the top of my head20:10
clarkbhttp://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-aiopcpu/065b025/logs/ is the nova config on the sub node20:10
jogoclarkb: I don't think its a nova issue actually20:10
*** markmcclain1 has joined #openstack-infra20:10
clarkbit probably isn't20:11
*** freemanbrandon has joined #openstack-infra20:11
clarkbhrm nova net runs on the primary node but not the secondary?20:11
*** emagana has joined #openstack-infra20:11
jogoclarkb: without adding special routing info I am not sure if this could even work20:11
clarkbjogo: sure, but lets not think about that yet20:11
jogovishy: you around20:11
clarkbI am trying to understand what nova expects20:11
*** cinderbld has quit IRC20:11
vishyjogo: yeah but in tc meeting20:11
clarkbdoes nova-net need to run on every node?20:12
*** carl_baldwin has quit IRC20:12
jogoclarkb: in one mode yes, but I don't thin kthat is enabled20:12
clarkbok if it doesn't then I assume every node must have to be on the same l2 network20:12
*** markmcclain has quit IRC20:12
clarkbotherwise how does anything work20:13
*** _nadya_ has joined #openstack-infra20:13
jogodansmith: what config option is used to set nova network multihost mode20:13
jogoclarkb: I think the latter20:14
*** emagana_ has joined #openstack-infra20:14
jogoand there needs to be routing to the L2 network from the test node somehow20:14
dansmithjogo: think it's just multihost=yes somewhere in there20:14
jogodansmith: looking at nova.conf.sample20:14
pleia2going to go find a snack (and possibly a quick swim), bbiab20:14
*** emagana has quit IRC20:14
jogoahh multi_host20:15
*** aysyd has quit IRC20:15
jogovishy: we are trying to set up nova-network in a two node devstack gate job20:15
jogofor when you have a momment20:15
*** markmcclain1 has quit IRC20:15
clarkbI am reading nova network docs now20:16
vishyjogo: ok that should be prettty easy20:16
vishy* pretty20:16
jogoclarkb: yeah we aren't running multi_host20:16
vishyyou want no multi-host?20:16
*** mrmartin has quit IRC20:17
*** spzala has quit IRC20:17
jogovishy: I don't think  it matters much either way at least not right now20:17
jogowhichever is easier for now20:17
jogolong term we want the one we would like to test more (multi-host?)20:17
clarkbyes we already test not mutlihost20:17
clarkbthis test env should test the things we can only test on mutlihost20:17
jogoclarkb: what to get vishy up to speed20:17
jogoclarkb: like multi_node nova-network mode ;)20:18
*** otherwiseguy has quit IRC20:19
vishyok so multihost is pretty simple20:19
jogovishy: well right now this is the issue: http://logs.openstack.org/97/131297/3/experimental/check-tempest-dsvm-aiopcpu/065b025/console.html#_2014-10-28_19_23_31_23520:19
vishydo we always use flaoting ips in tests right now20:19
vishyor are the instances reachable via fixed?20:20
jogowe cannot ssh into the VMs nova is creating20:20
clarkblets backup20:20
jogoI don't think we are using floating IPs for the VMs at least not public ones20:20
clarkbeverything currently works using the "fixed" ips20:20
jogoclarkb: yes please20:20
clarkb(I really dislike that we have about 3 terms for these things)20:20
clarkbinternal, private, fixed20:20
jogoclarkb: lets clarify which level of 'VM's we are talking about20:21
clarkbthe host VMs20:21
clarkbso the things hpcloud and rackspace give us20:21
*** freemanbrandon has quit IRC20:21
jogoright, just wanted to make sure we are all on the same page20:21
clarkbjeblair has asked that we use the public IPs instead of the "fixed" IPs20:22
clarkbit simplifies some things on the infra side to do so (we basically run everything assuming it is public)20:22
clarkbin rackspace we get normal interfaces with the public IPs20:22
clarkbin hpcloud we get NATed interfaces with floating IPs20:22
fungi(or if we can't use the public ips then at least identifying why so we know where to file the appropriate bug reports)20:22
clarkbnow when running the current setup against "fixed" IPs it just works20:23
clarkbwhen running the current setup against public IPs in rackspace it just works20:23
openstackgerritRamy Asselin proposed a change to openstack-infra/project-config: Refactor puppet-apply jobs into template  https://review.openstack.org/13154820:23
clarkbwhen running the current setup against HPCloud public IPs it does not work20:23
*** kgiusti has quit IRC20:23
clarkbwe are configured to use the flat dhcp manager right now20:24
clarkbit appears that that requires everything be on the same layer 2 network20:24
clarkbso that packets bound for addresses just end up on the right bridge20:25
clarkbif that is the case it would explain why it does not work when we go to hpcloud public IPs20:25
clarkbhowever reading nova network docs it appears that a cloudpipe vpn can be set up if we use the vlanmanager20:25
clarkbwhat is not clear at all is what does mutli_host do?20:25
*** sdake_ has quit IRC20:25
jogovishy: ^20:27
*** JayJ has quit IRC20:27
clarkbis multi_host a way of saying I am running nova net in multiple locations please don't trample each other?20:27
clarkbthen we would have to run nova-net on both nodes?20:28
*** JayJ has joined #openstack-infra20:28
vishyclarkb: i was actually referring to tempest testing20:28
*** nelsnelson has quit IRC20:28
vishyagainst devstack20:28
*** whoops has quit IRC20:28
vishyclarkb: multi host means a nova-net on each node yes20:29
*** nelsnelson has joined #openstack-infra20:29
*** _nadya_ has quit IRC20:29
clarkband the cloudpipe vpn thing can cross layer 3 networks?20:29
*** r1chardj0n3s_afk is now known as r1chardj0n3s20:30
vishyclarkb: ignore the cloudpipe vpn20:31
vishyit is useless :)20:31
*** dkranz has quit IRC20:31
vishyclarkb: you really need the nodes on the same layer 2 network for this to work properly20:31
clarkbvishy: thats, unfortunate20:31
vishywell, assuming you want the instances to talk to each other20:32
clarkbso I don't think we can actually make this work without a VPN then (even in the "fixed" ip case)20:32
vishyif you don’t care about that20:32
vishyit will be fine20:32
fungiclarkb: having them in the same provider neutron net is insufficient, or just means that bouncing through the provider's nat is out of the question?20:32
*** todd_dsm has joined #openstack-infra20:32
clarkbbecause we don't have an garuntee from rax that we will alays have nodes on the same l2 network20:32
clarkbfungi: both20:32
vishyclarkb: how much do we care about vms in the multinode devstack being able to talk to each other?20:33
*** rmcall has quit IRC20:33
clarkbvishy: well we need tempest to be able to talk to all of them20:33
clarkband tempest runs on one node20:33
vishybut it uses floating ips?20:33
vishyor fixed?20:33
clarkbvishy: the floating ip vs fixed is actually not really important anymore20:33
clarkbbecause we cannot garuntee everything ends up on the same l2 network regardless20:33
vishyit has to be for tempest20:33
clarkbit isn't we can cross routers in both situations20:34
vishyah i see20:34
vishyso yeah you will need to create a virtual layer 2 network20:34
vishyto do this20:34
jogovishy: and we should be able to do that via neutron right?20:35
clarkbI think we can garuntee it in hpcloud20:35
clarkbbut rax doesn't expose neutron so we can't do it there20:35
fungiclarkb: they were working on making it available, right?20:35
clarkbvishy: we cannot run cinder in a container20:35
clarkbfungi: maybe?20:35
vishyclarkb: i wasn’t really referring to the container part, more the overlay network using udp20:36
vishywhich is the only thing that works across most clouds20:36
clarkbya so we can openvpn them together pretty easily20:36
vishyyeah openvp would work20:36
fungior ipsec with ike+nat-t20:36
fungior even just straight up gre20:37
vishyno gre no work20:37
*** markmcclain1 has joined #openstack-infra20:37
fungiit's not like we have security/privacy concerns20:37
*** zns has joined #openstack-infra20:37
*** markmcclain1 has quit IRC20:37
vishyits too bad that the udp encap never merged in openvswitch20:37
fungigre with a point-to-point l2 won't work?20:37
vishyoh actually gre might work in rax + hp20:37
vishyits blocked by default20:37
vishybut if they are both using neutron20:38
*** teran has quit IRC20:38
vishyyou can open it up20:38
vishynova-network you can't20:38
fungiyou mean the ip protocol for gre is likely blocked between networks. got it20:38
vishyit doesn’t have an option for allowing gre, just like amazon20:38
fungibecause, you know, nobody really needs ip protocols other than icmp, udp and tcp right?20:38
clarkbya so "fixed" ips would work in hpcloud today as we set one network per virtual provider thing in nodepool20:39
vishyyes amazon :)20:39
fungii mean, the other 50+ are just wasting space20:39
*** markmcclain1 has joined #openstack-infra20:39
clarkbjroll: JayF: do you have any idea how rackspace is doing neutron and if all our nodes in one tenant end up in the same private network?20:39
vishyspeaking of which I’ve been meaning to add other protos to security groups20:39
*** dkranz has joined #openstack-infra20:39
vishybut getting stuff into nova-network is sooo painful these days20:39
jogovishy: you should just use neutron ;)20:40
clarkbvishy: so one more qusetion. If/when we sort out this layer 2 stuff we should probably go to testing multi_node by default beause we don't cover that today?20:40
jrollclarkb: on the virt side, I have no idea20:40
clarkbvishy: and if we do that should we keep using flatdhcpmanaer?20:40
jrollclarkb: I'm not entirely sure what you're asking, are you asking if by default a tenant has its own private network?20:40
fungianyway, if it's just a concern over ip proto 49 getting blocked, we could forward it over a tcp socket20:40
fungier, ip proto 4720:40
clarkbjroll: if I nova boot 2 nodes using the same tenant/project credentails. Will those two nodes' private IP always be on the same private l2 network?20:41
JayFclarkb: define: private IP20:41
clarkbjroll: so that traffic between them never crosses a router (at least logically)20:41
JayFclarkb: a cloud network? servicenet?20:41
jrollclarkb: no20:41
clarkbJayF: the non globally routable address that every node gets20:41
* fungi also hates the term "private ip address"20:41
JayFclarkb: that's very likely to be routed, and is not a contiguous layer 2 domain20:41
fungirfc-1918 in this case20:41
JayFclarkb: I believe cloud networks will behave as a contiguous layer 2 domain20:41
JayFclarkb: but not servicenet20:42
clarkbwhat is cloud networks?20:42
clarkband what is servicenet? you are using words that have no meaning :P20:42
openstackgerritSahdev Zala proposed a change to openstack-infra/project-config: Rename the heat-translator project repo  https://review.openstack.org/13155820:42
jrollclarkb: cloud networks is private tenant networks20:42
fungi"cloud networks" is rax's productization of neutron?20:42
jrollclarkb: servicenet and publicnet are the two default neutron networks you get with a standard cloud server20:42
vishyso lets clarify the layer 2 requirement20:42
vishyif the nodes can communicate with each other20:43
vishyi think that is good enough20:43
clarkbvishy: it isn't20:43
jrollfungi: that yes was for you20:43
clarkbvishy: I think the failing tests prove at least that much20:43
clarkbvishy: because the inner openstack uses some 172.x.y.z address space20:43
fungijroll: i always assume all yeses are for me20:43
vishyclarkb: well it wouldn’t work for mult-host mode20:44
clarkbvishy: and packets with those addresses cannot cross routers without service provider help20:44
jrollfungi: :P20:44
*** baoli has quit IRC20:44
vishyactually let me think for a sec20:44
clarkbvishy: if however everything is on the same l2 network those packets will get to their homes because arp20:44
clarkb(and broadcast if necessary)20:44
vishyyeah you would need two different private networks20:44
vishywith routing rules to the other host20:44
vishyso you’re right probably not feasible20:44
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Support stracktracing on Geard embedded server  https://review.openstack.org/13155920:44
*** baoli has joined #openstack-infra20:44
clarkbjroll: JayF: and cloud networks is not exposed yet?20:45
fungiclarkb: actually, i think without a tenant-specific layer 2 the providers have a definite incentive not to allow instances to add arbitrary unassigned ip addresses20:45
fungisince they could easily conflict between tenants20:45
clarkbfungi: I mean they probably should but it works20:45
jrollclarkb: afaik it is20:45
* jroll looks for docs20:46
JayFclarkb: it is; I don't know how to do it via the API as I've only done it via our control panels before20:46
*** e0ne has joined #openstack-infra20:46
clarkbI think we have been "lucky" that this has worked in rax so far20:47
fungiclarkb: also, if we fire up two multinode jobs in the same layer 2 broadcast domain, will their test instances conflict on arbitrary ip address assignment? i suspect we really need a virtual layer 2 network for each multinode if layer 2 is needed between subnodes at all20:47
clarkbfungi: good point, yes I think they will20:47
clarkbfungi: that would be fun to arp poison each other20:47
clarkbso ya we should just do openvpn20:47
clarkb(or similar)20:47
*** david-lyle has quit IRC20:48
* fungi takes his dusty old network engineer hat back off as quickly as possible20:48
ianwclarkb / jeblair: can we do a nodepool restart soonish, so i can make sure the hpcloud centos7 images are building before it seems most "root" people disappear for a bit20:48
clarkbsdague: jeblair ^ you will probably want to review scrollback when the TC meeting is over20:48
clarkbianw: I can do it now20:49
*** IaKing has joined #openstack-infra20:49
ianwclarkb: thanks, i'll check in on the logs in a bit and make sure things are going20:49
ianwclarkb: glusterfs people are keen to use centos -- it can work on ubuntu but package support is better and more up to date on centos i believe20:50
*** radez is now known as radez_g0n320:51
*** teran has joined #openstack-infra20:52
*** hashar has joined #openstack-infra20:54
*** carl_baldwin has joined #openstack-infra20:54
*** zns has quit IRC20:55
*** baoli has quit IRC20:55
*** baoli has joined #openstack-infra20:56
jogocan we use this as pressure to make rax give us neutron20:57
*** koolhead17 has quit IRC20:57
clarkbjogo: it won't help us20:57
clarkbso no not likely20:57
jogoclarkb: why wouldn't neutron help us?20:57
* jogo is not a networking person20:58
clarkbjogo: I could be wrong but my understanding is that the 172.x.y.z addresses for the inner cloud work on a layer 2 network because of arp20:59
*** teran has quit IRC20:59
clarkbjogo: before you send packet to 172.something you say who has 172.something. When there are 50 inner clouds all on the same network there can be many 172.somethings20:59
clarkbthey will effectively arp poison each other20:59
jogoclarkb: if each 2 node devstack setup each had its own private overlay network though21:00
clarkbthen tests will likely fail when they shouldnt or pass when they should fail because the wrong thing answered21:00
*** spzala has joined #openstack-infra21:00
clarkbjogo: ya if they had that no problem. But neutron doesn't help us with that21:00
jogoclarkb: it doesn't? I thought that was the point of it21:00
clarkbjogo: except it doesn't work all that well...21:00
fungijogo: right, private overlay network would be something specifically negotiated between the subnodes in a given multi-node group21:01
clarkbjogo: there are so many pieces you have to manage in order to "give me a network"21:01
clarkbwe could say create a new router, a new network, a new set of ports and floating IPs for every pair of servers we spin up21:01
clarkbor we just start openvpn on the pair21:01
clarkbmy feeling is openvpn is going to be simpler and easier to debug21:01
fungioh, i see. right it could in theory be possible for nodepool to provision multi-node-specific ephemeral networks which then get deleted again when teh node group is deleted21:01
jogoclarkb: ahh, that is a good point. but this sounds like a failing of neutron21:01
clarkbjogo: yes it is21:02
fungiby asking neutron at the provider to create one, then asking it to attach to the nodes in the group21:02
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Updated & Expanded Subscription API.  https://review.openstack.org/13156421:02
jogomarun salv-orlando markmcclain1 mestery: ^ we are making fun of neutron21:02
jogomarun salv-orlando markmcclain1 mestery: and using openvpn instead21:02
marunjogo: that's an invitation to join in the fun?21:02
clarkbjogo: not to mention quota21:02
*** hashar has quit IRC21:02
clarkbI think we are allowed 10 netowrks and routes21:02
jogomarun: yes it is, this sounds like a massive shortcomming of neutron to me21:02
clarkbwe can ask for more but I do not know what is involved on that backend and if asking for 600 networks is likely to come back with a yes response21:03
* mestery is quadruple tasking21:03
jogoclarkb: I think long term we should use neutron for this. Short term using openvpn sounds like a really good idea21:03
* markmcclain1 is slooooooooooow pipe21:03
*** david-lyle has joined #openstack-infra21:03
jogoclarkb: for the long term its a good dog fooding thing21:04
fungiespecially if they're not using some sort of vlan aggregation protocol... they might be physically limited to direct 802.1q mapping or something insane21:04
*** sdake_ has joined #openstack-infra21:04
clarkbjogo: ya. mostly I want to see neutron abstract enough of these common operations that I don't hve to track quota and uuids for every single network, port, floating ip and router and subnet21:04
clarkbjust give me a netowrk21:04
mesteryclarkb: neutron net-create21:04
mesteryclarkb: There you go ;)21:04
clarkbmestery: does that do the right thing with routers and ports and external nets and so on?21:05
apmeltonHas anyone here ever seen devstack-gate hung trying to install some dependencies through apt?21:05
apmeltonlogs: https://gist.github.com/ramielrowe/96e8ff07e896783a5a3721:05
fungior they might be bridging systems through hardware with other limitations on vlan count, such that it's cost-prohibitive for them to significantly up our network quota or deal with the create/delete churn21:05
mesteryclarkb: Your definition of right thing may be different than other peoples. Define yours. :)21:05
clarkbseems like last time I did any of this stuff it all had to be in a very specific order with all of the object created in the proper sequence21:05
apmeltonseems to be hung after "Processing triggers for man-db ..."21:05
salv-orlandoI think the bottom line of what clarkb and jogo say is that the API is mostly pointless crap21:05
clarkbmestery: neutron should have a single command along the lines of `neutron give me a network called foobar`21:05
mesterysalv-orlando: Summarized brilliantly21:05
*** marun has quit IRC21:06
clarkbmestery: that would return a uuid for a network that is attached to a router with a subnet21:06
jogosalv-orlando: yup21:06
*** marun has joined #openstack-infra21:06
clarkbmestery: and that router would be attached to the enxternal network21:06
jogosalv-orlando: and that this would be a great use case for neutron21:06
clarkband I could just use this bigger network abstraction21:06
markmcclain1clarkb: the correct api call is mordred-wants-a-network-now21:06
clarkbbecause in reality I don't care about routers and ports and external networks21:06
clarkb(and if I really did I would use the lower level api to muck with that stuff)21:06
mesteryclarkb: Porcelain APIs.21:07
*** zns has joined #openstack-infra21:07
clarkbnow where this is really problematic isn't the creation portion21:07
clarkbits the accounting issues that arrise21:07
clarkbwe already leak floating IPs everywhere21:07
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Added Vagrant VM for development purposes.  https://review.openstack.org/13130621:07
*** dimtruck is now known as zz_dimtruck21:07
salv-orlandoI am already aware that the infra team as a user of neutron has realized that at the end of the day all the features that neutron provides are a hindrance for them21:07
clarkbbecause there is no atomic just kill the damn thing for me21:07
fungi(and ports apparently?)21:07
salv-orlandoand many other teams like them as well21:07
clarkbinstead you have to kill the node then the floating IP21:08
tsufievhi there! I've noticed a new Gerrit code review message 'The change requires a local merge to resolve...' recently in my commits (I've never met it before). Is it a new way to say 'There is merging conflict'?21:08
clarkbbut if you kill the node you lose your pointer to the floating IP21:08
salv-orlandopossibly all users except the ones which are forced into buying stuff from hw vendors?21:08
clarkbso if the floating IP delete fails the first time around we have a leak21:08
salv-orlandoclarkb: arosen did patch a while ago for that21:08
salv-orlandoboth in nodepool and nova21:08
clarkband I would be very worried about similar issues when it comes to routers, ports, subnets, and networks as in the api today21:08
clarkbsalv-orlando: ya we just have to wait for it downstream21:09
tsufievthe thing that confused me is that after I did 'Rebase change' in gerrit, Jenkins said +121:09
fungitsufiev: there are two different subsystems which can complain about merge conflicts. that one is gerrit itself realizing that its internal jgit will be unable to merge a commit, even though zuul's cgit backend had no problem with it21:09
openstackgerritAndreas Jaeger proposed a change to openstack-infra/project-config: Remove obsolete rtfd-id entry  https://review.openstack.org/13156921:09
fungitsufiev: there are unfortunately minor differences in how merge conflict resolution is implemented between jgit and cgit from what we've seen21:10
arosenclarkb:  yup we backported this issue here to nova: https://review.openstack.org/#/q/I53b0c9d949404288e8687b304361a74b53d69ef3,n,z21:10
tsufievfungi, does that mean that it's pointless to press 'Rebase change' in web-interface in that case?21:10
clarkbarosen: we are still seeing the leaks so I don't think hpcloud has deployed those changes21:10
fungitsufiev: jgit seems to have particular trouble with n-way merges beyond ~4, so if you had a patch series of a lot of changes gerrit likely threw that on a change partway into the stack21:11
clarkbarosen: or do we need to modify how we do deletions client side?21:11
*** yjiang5 has quit IRC21:11
fungitsufiev: you can try the "rebase" button, but in my prior experience i've found i needed to locally rebase the commit and push it up as a new patchset21:11
jeblairclarkb, fungi: istr that we thought openvpn would eventually be necessary to do some neutron testing anyway; openvpn sounds good to me.  i also like being protected from the vagaries of our cloud providers' environments.21:12
arosenclarkb:  i assume you have this patch applied to youre node pool: https://review.openstack.org/#/c/113659/21:12
clarkbarosen: so likely we just need to wait for downstream cloud providers to deploy those changes21:12
fungijeblair: agreed. i do worry that trying to leverage the provider's neutron to create and delete tons of tiny networks on the fly may not work out so well21:12
arosenclarkb:  that patch should fix the leaking issue if not we should need to investigate more what else could be causing the leak21:12
clarkbarosen: well we knew it wouldn't fix all of the leaks21:13
arosenmaybe jogo or armax could help us find out if hp has though changes deployed.21:13
clarkbit would just shorten the locations of leaks21:13
*** sdake_ has quit IRC21:13
clarkband were dependent on the clouds themselves deploying the server side code to make it more atomic and fix it properly21:13
tsufievfungi, thanks, will use it as a rule of thumb. Didn't understand the difference between jgit and cgit though (which acts when) :(. Is there some ML thread I'm missing?21:13
armaxarosen: I am afraid I wouldn’t know how to help21:13
arosenmeh :P21:13
armaxperhaps with jogo you’re luckier21:14
jogoarmax: I am not21:14
armaxarosen: sorry :O21:14
armaxarosen: oh well21:14
jogoarmax: I don't even know who is in charge of neutron in HP public cloud21:14
jogomestery: works for us now right?21:14
armaxmaybe carl_baldwin could be a good source21:14
arosencarl_baldwin:  maybe21:14
mesteryjogo: Indeed21:14
armaxarosen: I am a small fish in a big pond21:15
* mestery is a freshly caught fish21:15
armaxit takes me ages to swim from one side to the other and back21:15
* carl_baldwin just joining now ...21:15
clarkbjeblair: ya agreed, if we openvpn we shouldn't ever need to worry about it again21:15
fungitsufiev: zuul is our testing orchestration layer, and also has some pseudo-jobs which will opportunistically report on mergeability of proposed changes. the version of gerrit we're running right now only figures out whether a change can merge when asked to permanently merge it to the target branch21:15
jogoarosen: wait this is a nova patcH?21:15
arosenjogo:  yes.21:15
jogoarosen: I may  be able to help21:16
fungitsufiev: zuul uses the usual cgit implementation like most people use from the command line, while gerrit is a java-based service running in a jvm and uses a java-based git implementation called jgit21:16
* jogo sends an email21:16
jogoI emailed phil day and paul murray but both are in the UK21:18
tsufiev fungi, and java implementation is more fussy that the cgit? Very Java-like :)21:18
fungitsufiev: basically yes21:19
nelsnelsonGreetings, channel.  So, when trying to get a devstack-gate set up and following these instructions https://github.com/openstack-infra/devstack-gate, I can get as far as executing safe-devstack-vm-gate-wrap.sh, but then that process seems to hang.  The last thing I see in the new/devstacklog.txt is 'Processing triggers for man-db ...'21:19
*** yjiang5 has joined #openstack-infra21:19
funginelsnelson: hi again... apmelton also just commented he's running into the same issue21:20
nelsnelsonOh okay, thank you.21:20
tsufievfungi, thanks again for the explanation!21:20
apmeltonfungi: we're working together on that issue21:20
*** harlowja is now known as harlowja_away21:21
apmeltonfungi: the apt process seems to be stuck in Stopped state21:21
nelsnelsonfungi, I think you might have mentioned trying to pre-install all the dependencies prior to running safe-devstack-vm-gate-wrap.sh.  Are there instructions that include a list of those dependencies, or is there a sub script somewhere that handles those installations?21:21
fungiapmelton: nelsnelson: fwiw, the package installations you're seeing are likely coming from devstack itself, since devstack-gate doesn't apt-get install any packages aside from git and python-yaml21:21
*** unicell has quit IRC21:22
nelsnelsonOkay, that makes sense.21:22
apmeltonfungi: I've noticed in the past, I can get past thing, by removing the timeout wrapper21:22
*** hdd has quit IRC21:22
fungiapmelton: nelsnelson: if you do think it's just timing out on something time-consuming, you can adjust the timeout without removing it entirely21:22
apmeltonfungi: it's not actually timing out21:23
nelsnelsonI personally don't think it's actually doing anything.21:23
apmeltonit hangs way before the timeout would actually hit21:23
fungiright, so whatever devstack should do next after installing packages seems to not be proceeding, and not resulting in any output in the devstack log21:23
*** zz_dimtruck is now known as dimtruck21:24
clarkbstrace it?21:24
apmeltonfungi: when the gate is run, are most of the dependencies pre-installed on the image?21:24
apmeltonclarkb: root@devstack-tempest-1414396776:/opt/stack/new# strace -p 278821:25
apmeltonProcess 2788 attached - interrupt to quit21:25
apmelton--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---21:25
apmelton--- SIGINT (Interrupt) @ 0 (0) ---21:25
apmeltonProcess 2788 detached21:25
apmeltonI'm not very familiar with strace, but it basically killed the process21:25
apmeltonput it in zombie state21:25
*** AJaeger has quit IRC21:25
fungiapmelton: yes, we use something called nodepool to precreate images which we later use to boot our test systems. it does a download-only install of the debs or rpms (depending on platform) mentioned in devstack21:26
*** armax has quit IRC21:27
fungithat way devstack's install step only has to retrieve updated package lists and maybe a handful of new packages but should find most of the ones it wants already in the cache21:27
*** rushiagr is now known as rushiagr_away21:27
*** markmcclain1 has quit IRC21:27
apmeltonfungi: but ultimately, it's devstack is still installing those packages, right?21:27
fungiapmelton: but given that package installation seems to be completing well in advance of hitting the timeout, i doubt that's the cause of your current issue21:28
fungiapmelton: and yes, devstack still installs them because we need to be able to reliably self-test proposed changes to devstack's package lists21:28
apmeltonnelsnelson: how did you kick off this script? just running the wrapper script in an ssh session?21:29
nelsnelsonYeah.  Regular shell session.  Just replicated all the commands indicated by https://github.com/openstack-infra/devstack-gate.21:30
*** bcourt has joined #openstack-infra21:30
apmeltonfungi: how does the devstack gate script get kicks off normally? I'm wondering if maybe executing it from ssh could be causing issues21:32
apmeltonget kicked*21:32
fungiapmelton: via a jenkins slave agent. we define it as a shell builder in a jenkins job21:32
fungiapmelton: it's possible if you're doing noninteractive ssh, not allocating a tty, that it could have some bearing on the result21:33
*** armax has joined #openstack-infra21:33
funginelsnelson: apmelton: maybe compare your devstack log to a working one from our ci like http://logs.openstack.org/93/131193/4/check/check-tempest-dsvm-full/1da1ae9/logs/devstacklog.txt.gz and see what should be coming next after where your output ends21:33
*** armax has quit IRC21:34
apmeltonfungi: dunno if this is interesting or not, but it would seem our run is installing libsigsegv2 while that run didn't21:35
*** dustins has quit IRC21:36
apmeltonthough, there seem to be a couple other packages like that as well21:36
fungiapmelton: may have to do with what envvars you're passing to devstack-gate. it has some switches to turn different things on and off21:36
*** e0ne has quit IRC21:36
*** emagana_ has quit IRC21:37
*** emagana has joined #openstack-infra21:38
nelsnelsonfungi, I'm only setting the environment variables exactly as indicated in the https://github.com/openstack-infra/devstack-gate readme.21:38
*** zns has quit IRC21:39
funginelsnelson: so you're using DEVSTACK_GATE_TEMPEST=1 and DEVSTACK_GATE_TEMPEST_FULL=1 in this case? should be mostly similar to the job run i linked (tempest-dsvm-full)21:40
apmeltonfungi: what image are y'all using when running against rackspace?21:40
*** andreykurilin_ has quit IRC21:41
*** emagana_ has joined #openstack-infra21:41
apmeltonfungi: was it just Ubuntu 14.04 PVHVM?21:41
fungi"Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM)"21:41
fungifor that particular test run21:41
apmeltonnelsnelson: which image did you use?21:41
*** emagana has quit IRC21:41
*** packet has quit IRC21:42
fungiand an 8gb performance flavor21:42
annegentlewhat is the name of the open source project that ask.openstack.org is built on?21:42
clarkbannegentle: askbot21:42
nelsnelsonWell, I thought that I was using the Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM), but I now realize that I am mistaken.21:42
*** dkranz has quit IRC21:43
apmeltonnelsnelson: lets try with 14.04 PVHVM and see what happens21:43
annegentleclarkb: thank you!21:43
apmeltonthanks for the help clarkb and fungi, hopefully using that image will fix this for us21:45
*** harlowja_away is now known as harlowja21:46
*** dims__ has quit IRC21:46
nelsnelsonThank you, apmelton, clarkb, and fungi...  I'll let you know if this fixes my problem.21:46
*** marun has quit IRC21:49
*** mattfarina has quit IRC21:50
*** teran has joined #openstack-infra21:51
fungiokay, gonna go watch nasa retry to launch this rocket, then grab some dinner and pack. mostly afk for the evening21:52
*** david-lyle has quit IRC21:52
*** andreykurilin_ has joined #openstack-infra21:53
*** david-lyle has joined #openstack-infra21:53
clarkbwait nasa launches rockets?21:57
*** pc_m_ has quit IRC21:57
*** arosen has left #openstack-infra21:58
fungiwhen there's not a sailboat in the way, they do21:58
clarkbI thought they paid other people to do all that now21:58
*** dimtruck is now known as zz_dimtruck22:01
*** doug-fish has left #openstack-infra22:01
*** dangers is now known as dangers_away22:01
*** JayJ has quit IRC22:02
*** JayJ has joined #openstack-infra22:03
*** thedodd has quit IRC22:09
*** bknudson has quit IRC22:11
*** bcourt has quit IRC22:11
*** ZZelle_ has quit IRC22:13
greghaynesthey just dont do manned low earth orbit22:14
greghaynesor sailboats22:14
greghaynesor any combination thereof22:14
*** todd_dsm has quit IRC22:14
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Test Version Controller  https://review.openstack.org/13158422:14
*** e0ne has joined #openstack-infra22:15
*** armax has joined #openstack-infra22:16
clarkbrereading this stuff it looks like afazekas_beer was already setting up gre bridges between the nodes22:18
*** dizquierdo has joined #openstack-infra22:18
clarkbor trying to? I don't see the interfaces22:18
*** amitgandhinz has quit IRC22:18
*** ryanpetrello has quit IRC22:19
*** thedodd has joined #openstack-infra22:19
*** julim has joined #openstack-infra22:20
clarkbI am going to guess that maybe my IP manipulation didn't do the correct thing there22:20
*** Ryan_Lane1 has joined #openstack-infra22:21
*** Ryan_Lane has quit IRC22:21
*** lttrl has quit IRC22:22
nibalizeris this just weird dns failures or is there something wrong with my test? http://logs.openstack.org/86/126086/3/check/gate-puppet-storyboard-puppet-beaker-rspec/207fc61/console.html22:22
openstackgerritA change was merged to openstack/requirements: Add keystoneclient-kerberos repository to projects  https://review.openstack.org/13106522:25
*** changbl has quit IRC22:26
*** rlandy has quit IRC22:26
clarkbok I think I may know why this broke at least for nova network22:26
clarkbnot sure about neutron yet22:26
*** emagana_ has quit IRC22:26
*** sarob_ has quit IRC22:27
*** tonytan4ever has quit IRC22:27
clarkb`remote_command $node sudo -i ip link add ${if_name} type gretap local $node remote $host_ip key $key` the remote host_ip is 15.something22:28
clarkbbut none of the traffic will original from that address when it talks because hpcloud knows not about it22:28
clarkbjeblair: I think we should give consideration to using the "fixed" ips22:29
clarkbjeblair: since we are already testing everything with them today and it keeps the crazy routing and bridging sane22:29
ianwclarkb: so i'm not seeing hpcloud centos7 builds in http://nodepool.openstack.org/image.log ...22:30
jeblairclarkb: what do you think we should do on rax?22:30
clarkbjeblair: same thing, rax gives us a "fixed" ip ish thing too22:31
jeblairclarkb: well, they give us public and private ips22:31
jeblairclarkb: so which one do you think we should use there?22:31
clarkbthe private ip22:31
jeblairclarkb: why?22:31
clarkbbecause that is more consistent with what hp gives us22:31
jeblairin what way?22:32
*** dmsimard is now known as dmsimard_away22:32
mordredit's an ip that the node knows about22:32
clarkbthey are not globally routable addresses22:32
clarkband they are btoh ips the node knows about22:32
jeblairmordred: rax knows about both ips22:32
clarkband we won't have to special case anything to get both behaviors22:32
*** david-lyle_ has joined #openstack-infra22:33
jeblairclarkb: what 'both behaviors'?22:33
clarkbnode knows about the IP address and IP address is not globally routable22:34
mordredamusingly enough, I've got a function already which will return the fixed IP on hp and the private IP on rax22:34
mordredhttps://github.com/emonty/shade/blob/master/shade/meta.py#L21-L32  - called like: https://github.com/emonty/shade/blob/master/shade/meta.py#L81-L8222:35
mordredbecause "public" and "private" and "fixed" and "floating" are not consistent terms across clouds22:35
*** thedodd has quit IRC22:35
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add current node IP to /etc/nodepool  https://review.openstack.org/13153222:35
*** gondoi is now known as zz_gondoi22:36
jeblairclarkb: well, not globally routable sounds like an arbitrary decision at this point22:36
*** david-lyle has quit IRC22:36
jeblairmordred: i think afazekas proposed a change to nodepool for that.  perhaps you could review it?22:36
mordredjeblair: yes22:36
jeblairmordred: i only -1d it on its necessity, which clarkb is now making a case for22:36
clarkbjeblair: but it isn't necessary22:37
jeblairclarkb: oh, i thought you wanted to use fixed ips22:37
*** andreaf has joined #openstack-infra22:37
clarkbyes but we don't need nodepool for that22:37
jeblairor, let's say "node-aware rfc1918 addresses"22:37
jeblairclarkb: why not?22:37
clarkbafazekas_beer's current change figures it out just fine best I can tell22:37
clarkbeach node already knows those addresses so it doesn't need nodepool to tell it22:38
mordredon the host? why? we're already putting info into the file in nodepool - why perform ip address figuring out logic in two places22:38
jeblairclarkb: i can not tell that it does that, nor can i tell that it will do that in future cloud configurations.  i'd much rather nodepool be the source of this information.  i think he would too.22:38
jeblairclarkb: i imagine that's why he wrote the nodepool change and said "this gets rid of a lot of complexity in devstack-gate"22:38
mordredI would also prefer nodepool to be teh source of the information - its view of the world is richer22:38
clarkbjeblair: sure I am not saying don't put it in nodepool22:38
clarkbjust saying it isn't necessary22:38
clarkbthis is the nice thing about nodes knowing their own IPs22:38
mordredjeblair: I do not see a nodepool change from afazekas_beer22:38
mordredoh - it's abandoned22:39
mordredone sec22:39
*** zns has joined #openstack-infra22:39
*** gyee has joined #openstack-infra22:39
jeblairclarkb: also, i want to make devstack-gate smaller22:39
mordredjeblair: his change is incomplete22:40
jeblairmordred: can you link it?22:40
mordredit only handles fixed ips - doesn't handle a neutron-based cloud which has a 'private' network22:41
jeblairmordred: missing rax?  yah22:41
*** david-lyle has joined #openstack-infra22:41
mordredjeblair: lemme fix - one sec22:41
jeblairi love that we have servers that don't know their own ip, and i love that everything we do is still wrapped in an "if hpcloud/elif rax" clause  :(22:42
mordredwell ... it's on my list to port nodepool to shade so that none of our code has any cloud-differences-aware code ... but I haven't even gotten around to writting up the spec for that :)22:42
*** zns has quit IRC22:44
*** nelsnelson has quit IRC22:44
*** jgrimm is now known as zz_jgrimm22:44
*** zns has joined #openstack-infra22:44
*** ryanpetrello has joined #openstack-infra22:44
*** david-lyle_ has quit IRC22:45
*** dane_leblanc has quit IRC22:45
*** dims__ has joined #openstack-infra22:46
jeblairi restored that change and left a comment with my current understanding22:47
jeblairmordred: it may want to be rebased onto https://review.openstack.org/131532 and an equivalent file added22:48
jeblairalso, we should avoid the use of the word fixed22:48
mordredyah. since it's "fixed or private"22:48
*** mfink has quit IRC22:48
*** zz_dimtruck is now known as dimtruck22:49
mordredjeblair: how about just private?22:49
jeblairmordred: that works for me since our fixed addresses are also private22:49
*** dims__ has quit IRC22:52
*** amcrn has quit IRC22:52
*** tgohad has quit IRC22:52
*** ryanpetrello has quit IRC22:53
*** dims__ has joined #openstack-infra22:55
*** dims__ has quit IRC22:55
*** emagana has joined #openstack-infra22:55
*** dims__ has joined #openstack-infra22:56
jeblairclarkb, fungi: the antares rocket exploded just after liftoff; http://www.nasa.gov/mission_pages/station/structure/launch/orbital.html#.VFAe_4WVtph22:56
jeblairno one hurt according to nasa22:57
*** ihrachyshka has quit IRC22:57
*** dims__ has quit IRC22:57
openstackgerritMonty Taylor proposed a change to openstack-infra/nodepool: Add private ip to the /etc/nodepool  https://review.openstack.org/11484022:57
*** dims__ has joined #openstack-infra22:57
openstackgerritMonty Taylor proposed a change to openstack-infra/nodepool: Add private ip to the /etc/nodepool  https://review.openstack.org/11484022:59
*** msabramo1 has quit IRC22:59
mordredmissed updating the docs in the first patch22:59
jeblairmordred: how about a "node_private" file?23:00
jeblairmordred: private ip address of current node23:00
mordreddoes it not add one of those?23:00
*** timrc is now known as timrc-afk23:01
jeblairmordred: primary_node_private, and _sub_nodes_private.23:01
mordredAH - gotcha23:01
*** baoli has quit IRC23:01
openstackgerritMonty Taylor proposed a change to openstack-infra/nodepool: Add private ip to /etc/nodepool  https://review.openstack.org/11484023:03
*** tomoe has joined #openstack-infra23:03
*** msabramo has joined #openstack-infra23:04
mordredjeblair: these files just gave me an idea - bear with me, it'll take a few seconds to explain23:06
jeblairmordred: i will beer with you23:07
*** timrc-afk is now known as timrc23:07
mordredmuch of the info that we're writing out into /etc/nodepool is very similar to the hostvars stuff that we get in ansible inventory, which I have a function that returns as a dict of lists and dicts as you'd expect - which ansible then turns in to json when needed for reconsumtion23:08
*** andreaf has quit IRC23:08
mordrednow, we're not using ansible for this- so json/yaml files are not useful23:08
mordredthus all the individual files in /etc/nodepool - shell scripts23:08
*** JayJ has quit IRC23:08
mordredSO - the idea is - what if we wrote a small generalized tool that would take a json/yaml/simple datastructure23:08
mordredand splat it out into files and dirs under a top level output dir23:09
*** msabramo has quit IRC23:09
mordredso that we could plop down a yaml/json file for any things like python that can consume all the data in a structured form23:09
mordredbut then have an exactly analogous dir of simple files that shell scripts could use23:09
jeblairmordred: doesn't sound bad, though i don't think we have a current problem that necessitates it23:10
mordredno. we don't23:10
mordredjust made me think that the problem of "I have json/yaml and I want to use from shell scripts" does come up from time to time23:10
mordredand it usually gets solved in some one-off crazy manner23:10
mordredso maybe we can invent new goodness23:11
jeblairmordred: perhaps, though i think the rules about how to translate a complex data structure into a flat structure may vary23:11
jeblairso generalizing that may be hard23:11
*** andreaf has joined #openstack-infra23:11
mordredthey may - and it may be hard to get it "right" every time23:11
mordredbut I think I'll take a stab at an 80% good enough approach and see what that looks like23:11
mordredmay suck, in which case I may delete it :)23:12
*** teran_ has joined #openstack-infra23:12
jeblairthough i guess things like '../subnodes/0/ip ..subnodes/1/ip' would probably work23:12
mordredyah. just as understandable as the equiv in the yaml :)23:13
clarkbafter much investigating an attempts to further debug I am relatively certain that the gre tunnel stuff that afazekas_beer sets up is sufficient for our needs23:13
jeblairmordred: so actually, yeah, you could probably translate lists and dicts directly to directories23:13
openstackgerritKhai Do proposed a change to openstack-infra/storyboard: Add a rest endpoint to retreive system information  https://review.openstack.org/13159923:13
clarkbit creates a bridge on the primary node that all subnodes are attached to via gretap devices23:13
clarkbthen devstack/nova is told to use the resulting pub_if interface for the Flat netwrok23:14
clarkbso the VMs we spin up nested all end up on that bridge23:14
clarkband I am pretty sure that this doesn't work for tempest in the case of public floating IPs because the node is unaware of the 15.something addresses so linux cannot route our packets appropriately23:14
*** teran has quit IRC23:15
clarkbbut when traffic originates on the "fixed" private address it does know how to get those packets from source to destination23:15
jeblairclarkb: cool, so way forward: 1) land nodepool change, 2) combine both afazekas devstack-gate changes (the one that adds stuff along with the one that removes complexity), 3) restart nodepool, 4) test/merge change ?23:15
clarkbthat sounds about right23:15
clarkbnote I haven't quite investigated this for the neutron case yet23:16
jrollmordred: seems like a useful thing23:16
clarkbbut if my memory is not completely terrible neutron basically sets up the same gretap bridge situtation for you23:16
*** emagana has quit IRC23:16
*** andreaf has quit IRC23:16
clarkband neutron doing that would also explain the brokeness when we set the gretap device hosts as using the floating IPs23:17
clarkbso to recap23:17
*** emagana has joined #openstack-infra23:17
*** andreaf has joined #openstack-infra23:17
clarkbI don't think cinder specifically cares23:17
clarkbI do think that communication to arbitrary VMs does care23:17
clarkbbecause of the gretap bridge23:17
clarkbmordred: you need to update the nodepool test fakes23:19
*** emagana_ has joined #openstack-infra23:21
*** emagana has quit IRC23:21
clarkbianw: I am going to start a manual hpcloud centos7 image build23:22
*** emagana_ has quit IRC23:22
*** emagana has joined #openstack-infra23:23
ianwclarkb: ok, great.  has it just not got around to it?23:23
clarkbit had a bunch of older ones23:23
clarkbI am deleting those first23:23
clarkbok one just started23:24
clarkbyou should probably start seeing it in the log now?23:24
ianwclarkb: downloading ... thank heavens for gzip :)23:25
ianwclarkb: yay, building!  i expect that will go ok23:27
*** emagana has quit IRC23:27
*** emagana has joined #openstack-infra23:28
openstackgerritKhai Do proposed a change to openstack-infra/storyboard: Add a rest endpoint to retreive system information  https://review.openstack.org/13159923:29
*** amitgandhinz has joined #openstack-infra23:29
*** teran has joined #openstack-infra23:30
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Test label with snapshot and dib images  https://review.openstack.org/13087823:31
*** teran_ has quit IRC23:31
clarkbjeblair: ianw ^ I think that should better cover the case where a label has both snapshot and dib provided images23:31
*** emagana has quit IRC23:32
*** amitgandhinz has quit IRC23:34
mordredjroll, jeblair: done: http://paste.openstack.org/show/12620523:35
clarkbmordred: did you see comment about nodepool test fakes?23:36
mordredclarkb: yup. on it now23:36
* jroll saves for later23:36
*** rushiagr_away is now known as rushiagr23:36
mordredjroll: after I'm done with test fakes, I'm gong to cookiecutter that and publish to pypi just because why the heck not23:36
*** carl_baldwin has quit IRC23:37
*** andreykurilin_ has quit IRC23:40
*** marun has joined #openstack-infra23:40
openstackgerritMonty Taylor proposed a change to openstack-infra/nodepool: Add private ip to /etc/nodepool  https://review.openstack.org/11484023:41
*** amcrn has joined #openstack-infra23:42
*** andreaf has quit IRC23:43
*** sarob has joined #openstack-infra23:44
*** SumitNaiksatam has quit IRC23:48
lifelessdhellmann: so any ideas of the jython issue? Or is it terra incognita?23:50
lifelessmordred: 'that'?23:50
*** david-lyle has quit IRC23:50
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Test label with snapshot and dib images  https://review.openstack.org/13087823:50
clarkblifeless: http://paste.openstack.org/show/12620523:51
clarkbis the that aiui23:51
fungiyes, antares rocket go boom six seconds off the pad. not our space program's finest hour23:51
*** packet has joined #openstack-infra23:51
*** mwagner_lap has joined #openstack-infra23:51
jeblairclarkb, mordred, afazekas: i left some -1 comments on the nodepool change23:51
*** dizquierdo has quit IRC23:52
fungiwe were far enough away we couldn't hear the blast either23:52
* fungi gets back to packing23:52
jeblairfungi, clarkb, mordred, jhesketh, SergeyLukjanov: i think it would be good to either merge my revert change or review and merge sdague's changes to fix the test displays on gerrit asap.  that's a serious regression that new change folks are living with.23:53
mordredjroll: https://github.com/emonty/explode https://pypi.python.org/pypi/explode23:54
jheskethjeblair: links?23:55
jeblairjhesketh: 131500 and 13148323:55
fungithey looked okay to me, but i did not find time to test them23:56
*** Ryan_Lane1 is now known as Ryan_Lane23:57
*** Ryan_Lane is now known as Guest9188023:57
openstackgerritClark Boylan proposed a change to openstack-infra/devstack-gate: Enable nova multi host on aiopcpu tests  https://review.openstack.org/13160423:57
mordredjroll, jeblair: I believe that is my absolute fasted time from idea in IRC to 1.0 release of a thing :)23:58
*** rmcall has joined #openstack-infra23:58
*** Guest91880 has quit IRC23:59
jheskethjeblair: 131500 lgtm if you want to merge it23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!