Saturday, 2016-03-19

« Friday, 2016-03-18 Index Sunday, 2016-03-20 »
*** bmace has quit IRC00:07
*** bmace has joined #kolla00:07
*** achanda has joined #kolla00:28
*** ssurana has quit IRC00:36
*** apuimedo has quit IRC00:37
*** ntpttr has quit IRC00:37
*** ntpttr has joined #kolla00:40
*** unicell has quit IRC00:40
openstackgerritSam Yaple proposed openstack/kolla: Test reduce gate usage  https://review.openstack.org/29485400:45
*** unicell has joined #kolla00:47
*** unicell has quit IRC00:53
*** rstarmer has joined #kolla01:00
*** Jeffrey4l has joined #kolla01:05
*** dims has joined #kolla01:12
*** dims_ has quit IRC01:12
*** iceyao has joined #kolla01:17
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Add check if all the hostnames are resolvable  https://review.openstack.org/29471301:22
*** sbezverk has quit IRC01:24
*** sbezverk has joined #kolla01:25
*** inc0 has joined #kolla01:26
inc0hello01:26
*** inc0 has quit IRC01:29
openstackgerritHui Kang proposed openstack/kolla: Ceilometer needs mongoengine if build from source  https://review.openstack.org/29472501:29
*** rstarmer has quit IRC01:33
*** dims has quit IRC01:50
*** rhallisey has quit IRC01:53
*** rstarmer has joined #kolla01:57
*** weiyu has joined #kolla02:08
*** weiyu has quit IRC02:09
*** achanda_ has joined #kolla02:09
*** achanda has quit IRC02:12
*** Marga_ has quit IRC02:17
*** achanda_ has quit IRC02:19
*** achanda has joined #kolla02:20
*** achanda has quit IRC02:24
*** rstarmer has quit IRC02:34
openstackgerritSam Yaple proposed openstack/kolla: Allow external ceph journals  https://review.openstack.org/29486202:40
*** vhosakot has joined #kolla02:42
*** banix has joined #kolla02:58
openstackgerritSam Yaple proposed openstack/kolla: Allow external ceph journals  https://review.openstack.org/29486203:07
openstackgerritSam Yaple proposed openstack/kolla: Allow user to choose filesystem for osd  https://review.openstack.org/29486303:07
openstackgerritSam Yaple proposed openstack/kolla: Fix permissions for log extraction in gate  https://review.openstack.org/29486403:09
*** dave-mccowan has quit IRC03:31
openstackgerritSam Yaple proposed openstack/kolla: TEST debug glean in gate  https://review.openstack.org/29486803:32
openstackgerritSam Yaple proposed openstack/kolla: TEST debug glean in gate  https://review.openstack.org/29486803:43
SamYaple~.03:47
openstackgerritSam Yaple proposed openstack/kolla: TdEST debug glean in gate  https://review.openstack.org/29486803:47
*** Marga_ has joined #kolla03:53
*** Marga_ has quit IRC03:55
*** Marga_ has joined #kolla03:55
openstackgerritMaru Newby proposed openstack/kolla: Update quickstart doc to use drop-in unit files  https://review.openstack.org/29486903:56
openstackgerritMaru Newby proposed openstack/kolla: Update quickstart doc to use drop-in unit files  https://review.openstack.org/29486904:01
*** rstarmer has joined #kolla04:09
SamYapleJeffrey4l mandre nihilifer pbourke this patch is critical to fixing the UNSTABLE gate and should be merged ASAP https://review.openstack.org/#/c/294864/04:11
*** rstarmer has quit IRC04:12
*** banix has quit IRC04:24
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Add check if all the hostnames are resolvable  https://review.openstack.org/29471304:25
openstackgerritSam Yaple proposed openstack/kolla: TEST reduce size  https://review.openstack.org/29487204:25
*** salv-orlando has joined #kolla04:27
*** spisarski has quit IRC04:29
*** salv-orl_ has quit IRC04:30
*** rstarmer has joined #kolla04:43
openstackgerritSam Yaple proposed openstack/kolla: Fix 'optional' config.json values  https://review.openstack.org/29435004:45
*** Marga_ has quit IRC04:46
*** rstarmer has quit IRC04:59
*** Marga_ has joined #kolla05:00
*** achanda has joined #kolla05:03
openstackgerritMaru Newby proposed openstack/kolla: Update quickstart doc to use drop-in unit files  https://review.openstack.org/29486905:10
*** iceyao has quit IRC05:21
SamYapleccesario_: i see the mariadb issue. i do not have a fix yet05:21
*** vhosakot has quit IRC05:25
mandreSamYaple: what's happening exactly with copying the logs on the gate?05:43
mandredo you know why it started failing?05:43
*** Windir has joined #kolla06:33
*** unicell has joined #kolla06:43
*** akwasnie has joined #kolla07:21
*** akwasnie has quit IRC07:36
*** achanda has quit IRC07:38
*** achanda has joined #kolla07:40
*** jhyang has quit IRC07:41
*** akwasnie has joined #kolla07:47
*** achanda has quit IRC07:51
*** akwasnie1 has joined #kolla07:52
*** akwasnie has quit IRC07:55
*** stvnoyes has quit IRC08:02
*** stvnoyes has joined #kolla08:03
*** jmccarthy has quit IRC08:05
*** jmccarthy has joined #kolla08:06
*** akwasnie has joined #kolla08:13
*** akwasnie has quit IRC08:19
*** cfarquhar has quit IRC08:21
*** cfarquhar has joined #kolla08:24
*** cfarquhar has quit IRC08:24
*** cfarquhar has joined #kolla08:24
*** achanda has joined #kolla08:51
*** achanda has quit IRC08:57
*** akwasnie has joined #kolla09:05
*** mbound has joined #kolla09:08
*** mbound has quit IRC09:13
*** mbound has joined #kolla09:16
*** akwasnie2 has joined #kolla09:19
*** akwasnie has quit IRC09:19
*** salv-orlando has quit IRC09:21
*** akwasnie has joined #kolla09:24
*** akwasnie2 has quit IRC09:25
*** shardy has joined #kolla09:41
*** akwasnie has quit IRC09:42
*** shardy has quit IRC09:52
*** achanda has joined #kolla09:54
*** achanda has quit IRC09:59
*** SiRiuS_ has joined #kolla10:13
*** pbourke has quit IRC10:13
*** pbourke has joined #kolla10:13
*** mbound has quit IRC10:24
SiRiuS_mandre, can you please look at my patch: https://review.openstack.org/#/c/294340/10:26
SiRiuS_mandre, SamYaple requested it before the patch can be accepted10:26
SiRiuS_pbourke, dos you use virtualbox? can you please take a look at https://review.openstack.org/#/c/294340/10:28
*** ccesario has quit IRC10:42
*** athomas has joined #kolla10:42
*** athomas has quit IRC10:46
*** athomas has joined #kolla10:52
*** dave-mccowan has joined #kolla11:53
*** rhallisey has joined #kolla11:57
*** achanda has joined #kolla11:58
*** achanda has quit IRC12:03
*** athomas has quit IRC12:07
*** ccesario has joined #kolla12:09
*** ccesario has quit IRC12:16
openstackgerritJeffrey Zhang proposed openstack/kolla:  DO NOT MERGE  https://review.openstack.org/29490612:20
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE  https://review.openstack.org/29490612:24
*** dims has joined #kolla12:25
*** dims has quit IRC12:25
*** achanda has joined #kolla13:00
*** achanda has quit IRC13:04
*** dave-mcc_ has joined #kolla13:17
*** dave-mccowan has quit IRC13:18
*** signed8bit has quit IRC13:20
*** banix has joined #kolla13:38
SamYapleJeffrey4l: i have responded to you, please merge the patch and if you have a better solution later submit it later, but get needs to be fixed asap13:39
SamYapleremember, all the logs stuff is getting rewritten for multinode13:40
Jeffrey4lSamYaple, i will merge it if I can not catch up a solution in 1 hours. is this ok?13:41
Jeffrey4lSamYaple, btw, do you have any thoughts about the multinode logs?13:41
SamYapleJeffrey4l: the problem is we arent messin with the owner for the files from the host because that affects the owner of the files in the contaienr13:43
Jeffrey4lagreed.13:44
*** banix has quit IRC13:46
*** iceyao has joined #kolla13:49
openstackgerritSam Yaple proposed openstack/kolla: Test reduce gate usage  https://review.openstack.org/29485413:59
Jeffrey4lSamYaple, see the comments in https://review.openstack.org/29486414:03
*** eandersson has joined #kolla14:03
eanderssonHey14:04
SamYapleJeffrey4l: that wont work14:04
eanderssonI am trying to set up a all-in-one openstack instance using kolla14:04
SamYaplethe jenkins user is the one that doesnt have access to read14:05
eanderssonbut getting stuck at waiting for virtual ip to appear14:05
SamYaplechanging dest perms wont help14:05
SamYapleeandersson: waiting for virtualip appears is wiaiting on keepalived14:05
SamYapleeandersson: did you set your vip to an unused ip on your network?14:05
eanderssonyep, but I don't see any ips on any of the docker containers14:06
Jeffrey4lSamYaple, disagree. :( , i am runnnig some test. wait for a moment.14:06
eanderssone.g. /keepalived -14:06
SamYapleJeffrey4l: you disagree that its breaking on read? or do you think updating the DEST perms fixes read issues?14:06
eanderssonSo I am pretty sure I am missing something obvious14:07
SamYapleeandersson: docker ps -a show haproxy and keepalived running?14:07
Jeffrey4lSamYaple, I think change the DEST perms can fix the read issue.14:07
SamYaplethat makes no sense14:07
SamYaplewhat would that be the case?14:07
eanderssonYepo14:07
eanderssonUp 21 minutes14:07
eanderssonfor both14:07
SamYapleeandersson: can you post your globals.yml14:07
SamYaplealso make sure to check your keepalived and haproxy logs14:07
SamYaplethey are at /var/lib/docker/volume/kolla_logs/_data/*14:08
eanderssonAwesome SamYaple, so simple14:08
eanderssonKernel is reporting: interface enp0s20f1 DOWN14:08
eandersson:D14:08
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE  https://review.openstack.org/29490614:08
eanderssonLets plug that network cable in14:08
SamYapleeandersson: cool. that definetely could cause this issue14:09
openstackgerritSam Yaple proposed openstack/kolla: Test reduce gate usage  https://review.openstack.org/29485414:10
Jeffrey4lSamYaple, a test PS is pushed. If it failed, i will merge you PS.14:10
SamYapleJeffrey4l: ok, but aside from that. im curious. why do you think dest file permissions affect source file permissions?14:11
SamYapleim trying to follow your thought process14:11
Jeffrey4lSamYaple, the root cause is that the jenkins scp the /tmp/log/ to a remote location. but the mariadb.log file(now in /tmp/log/kolla/mariadb/mariadb.log) can not read by jenkins user. So it failed.14:12
eanderssonThat was simple. Would probably have taken me a while to figure out that I could even look at the logs using docker like that. :D14:13
SamYapleJeffrey4l: oh so youre just saying reorder the chmod operation?14:13
SamYapleyea thats reasonable14:13
SamYaplethat should work fine14:13
Jeffrey4lthe minimal change which can fix the issue should be `chmod 777 /tmp/logs/kolla/mariadb/mariadb.log` i think.14:14
SamYapleill so that real quick14:14
Jeffrey4lSamYaple, yea. thanks.14:14
eanderssoninput file not found at /etc/kolla/config/swift/account.builder or /etc/kolla/config/swift/account.builder14:16
eanderssoninteresting14:16
openstackgerritSam Yaple proposed openstack/kolla: Fix permissions for log extraction in gate  https://review.openstack.org/29486414:16
SamYapleeandersson: are you trying to deploy swift?14:17
eanderssonYea14:17
eanderssonNYI?14:17
SamYapleeandersson: there is a swift guide online14:17
SamYapleit requires additional work14:17
eanderssonah kk14:17
eanderssonI disabled it for now14:17
eanderssonWill try after.14:17
eanderssonYou guys going to Austin?14:18
SamYapleajafo: of course!14:18
SamYapleeandersson: *14:18
eanderssonawesome14:18
eanderssonwill be fun =]14:19
rhalliseyyo yo14:19
SamYaplehey rhallisey14:19
SamYaplegate all whackabusted14:19
rhalliseyactive this morning eh14:19
SamYapleindeed14:20
rhalliseygate fixing is perfect for weekends.. that's also what I'm doing :/14:20
rhalliseyjust not for kolla14:20
SamYapleawesome14:21
SamYaplehey i got external journals working!!! https://review.openstack.org/#/c/294862/14:21
SamYaplefor ceph14:21
rhalliseythats for newton though right?14:22
SamYapleno this was in mitaka14:22
SamYapleblueprint is from early early mitaka14:22
rhalliseydidn't our rc1 already go out14:23
SamYapleyup14:23
SamYaplethis is wrapping up the blueprints that are still open14:23
rhalliseywhere will this fit in though? Seems like a feature14:23
SamYaplethis will fit in as finishing up blueprints like we are doing with all the other ones?14:23
rhalliseythought only upgrade were give the priority here and the rest kicked to N14:24
SamYaplefeel free to no approve it, but this is to stablize the ceph bootstrap stuff, so i can easily make the case taht this is a bug14:25
eanderssonThis was a pretty smooth experience!14:25
SamYaplerhallisey: https://review.openstack.org/#/c/294862/2/docker/ceph/ceph-osd/extend_start.sh (see all that hackish stuff removed)14:26
SamYapleeandersson: im glad you liked it!14:26
eanderssonI am guessing that it dosen't create any RC files automatically as it is right?14:26
SamYapleeandersson: you have to run post-deploy14:27
SamYaplekolla-ansible post-deploy14:27
SamYapleit will geenerate one for you on the deploy now14:27
SamYaplenode*14:27
rhalliseySamYaple, nice that's good.  I'm just saying that's all.  I guess we'll see what sdake or other community members have to say14:28
SamYaplei never punted this blueprint, it was always landing in mitaka14:28
eanderssonawesome missed that14:29
rhalliseyI mean you can argue mitaka has passed though14:29
SamYaplerhallisey: and i can argue this is a crtical bug because my ceph osds dont bootstrap too14:30
SamYaple90% of this fixes the bootstrap, the other 10% also looks for external journals14:30
rhalliseyit will have to go to the jury then14:30
SamYaplesee if you look at the find_disks.py youll see its the one finding hte partitions now14:30
SamYaplewhereas before it was the ceph-osd script and it had all kinds of race conditions14:31
rhalliseySamYaple, how about I -1 for a bug.  And you just attach that ^ so it's clear14:33
SamYapleif you want me to create a bug for this i can i suppose14:34
SamYaplethats not problem14:34
rhalliseyI think it's more clear14:35
SamYaplerhallisey: https://bugs.launchpad.net/kolla/+bug/155885314:35
openstackLaunchpad bug 1558853 in kolla "Bootstrapping Ceph OSDs fails" [Critical,Triaged]14:35
SamYaplealready a bug14:35
rhalliseycool14:35
rhalliseythanks Sam14:37
openstackgerritSam Yaple proposed openstack/kolla: Allow external ceph journals  https://review.openstack.org/29486214:39
openstackgerritSam Yaple proposed openstack/kolla: Allow user to choose filesystem for osd  https://review.openstack.org/29486314:39
*** diogogmt has quit IRC14:41
eanderssonlooks like my rabbitmq server is crashing constantly14:46
eandersson> ERROR: epmd error for host eu-admin-guardian01: timeout (timed out)14:47
SamYapleeandersson: is your hostname resolveable to your ip (we have a fix for this in the queue)14:47
SamYapleif it doesnt resolve to your internface ip, but rather 127.0.0.1 this wont work14:47
eanderssonoh14:47
eanderssonit is, but it actually resolves as the wrong ip14:47
SamYapleyes14:48
SamYaplethe fix in the queue is a check for this because rabbtiqm requires it14:48
SamYapleccesario_: mariadb works when deploying AIO then scaling to multinode14:48
*** eandersson has quit IRC14:50
*** eandersson has joined #kolla14:50
*** diogogmt has joined #kolla14:55
eanderssonI really need to change my domain to an internal sub-domain :p14:56
*** banix has joined #kolla14:59
eanderssonAnother silly question! =] What is the easiest way to view service logs?14:59
eanderssonDo you have to exec bash to the container?14:59
eanderssonfor each service14:59
eanderssonor is there an easier way to view all logs?14:59
SamYaplethe logs are in /var/lib/docker/volume/kolla_logs/_data15:00
SamYaplethere is also elasticsearch/kibana15:00
eanderssonah nice15:00
eanderssonI did a quick find /var -name neutron but it didnt come up with anything15:01
eanderssonwhich makes no sense, must have had a typo15:01
*** achanda has joined #kolla15:03
SiRiuS_is cinder with lvm working?15:04
SamYapleSiRiuS_: that will not be finished for master15:04
SiRiuS_oh, ok15:04
SamYapleeandersson: im not sure i follow. cd to /var/lib/docker/volume/kolla_logs/_data15:05
SiRiuS_thanks15:05
eanderssonyep found it15:05
SamYapleSiRiuS_: it will merge early on in netwon tough it seems15:05
SiRiuS_SamYaple, does it work with ceph ?15:05
eanderssonreally feels like this is the future for deploys15:05
SamYapleSiRiuS_: cinder+ceph has worked since pre-liberty15:06
SamYapleeandersson: glad to hear you say that. weve spent alot of time to make it so15:06
SiRiuS_ok, thanks i'll give it a try15:06
*** banix has quit IRC15:07
SiRiuS_this is the first time I'm enabling cinder so i tried lvm, but it's good to know that it works with ceph15:07
*** achanda has quit IRC15:08
eanderssonWhat is the recommended way of restarting services?15:08
*** Jeffrey4l_ has joined #kolla15:08
*** Jeffrey4l has quit IRC15:09
SamYapledocker restart <container>15:09
SamYapleif you want to update configs we also have the reconfigure command for kolla-ansible15:09
*** rstarmer has joined #kolla15:09
SamYapleif the container is _really_ shot you canremove the container and run deploy again for a fresh copy15:10
*** Jeffrey4l_ has quit IRC15:13
SamYaplerhallisey: wierd bug with mariadb15:14
SamYapleARGS is set in the ENV, but start.sh isnt using the options15:14
*** vhosakot has joined #kolla15:15
rhalliseyinteresting15:15
SiRiuS_SamYaple, what is the main difference between running kolla-ansible deploy again and running kolla-ansible reconfigure ?15:16
openstackgerritSam Yaple proposed openstack/kolla: Test reduce gate usage  https://review.openstack.org/29485415:16
SamYapleSiRiuS_: reconfigure looks for see if the config has changes and restarts the service if it has15:17
SamYapledeploy actually does deployment15:17
eanderssonIs there a command to stop/start all containers? It's easy to do manually using docker, but after doing a restart all I realzied that its bad haha15:17
SamYaplelots of tasks are shared15:17
eanderssonsince db, rabbit etc needs to be started first15:17
SamYaplethere is not a command to stop all container15:17
SamYaplestarting all contaienrs in teh proper order can be done by rerunning deploy15:18
SiRiuS_SamYaple, ok,thanks15:19
SamYapleahhh rhallisey i found it. its getting blanked by start.sh15:20
*** Jeffrey4l_ has joined #kolla15:26
openstackgerritSam Yaple proposed openstack/kolla: Fix mysql bootstrap  https://review.openstack.org/29492115:30
*** vhosakot has quit IRC15:31
*** vhosakot has joined #kolla15:31
SamYaplerhallisey: this is critical https://review.openstack.org/29492115:32
openstackgerritSam Yaple proposed openstack/kolla: Fix permissions for log extraction in gate  https://review.openstack.org/29486415:36
SamYaplerhallisey: this one is also critical https://review.openstack.org/#/c/294864/15:37
SamYapleJeffrey4l_: i forgot to but kolla in the path aove ^15:37
*** rstarmer has quit IRC15:40
Jeffrey4l_SamYaple, yea. I saw that. But i have no idea why we have no permission to /tmp/logs folder?15:40
SamYapleits a symlink Jeffrey4l_15:40
SamYaplethe actual logs folder that jenkins pulls is in a long path name15:41
SamYapleso i made a symlink for our convince15:41
Jeffrey4l_SamYaple, no. check this `drwxr-xr-x. 11 jenkins jenkins 4.0K Mar 19 14:34 ..` the .. means /tmp/logs   root can not change the /tmp/logs permission.15:44
Jeffrey4l_root have no permission :D15:44
Jeffrey4l_SamYaple, http://logs.openstack.org/06/294906/3/check/gate-kolla-dsvm-deploy-centos-binary/86707e5/console.html#_2016-03-19_14_34_05_76415:45
SamYapleJeffrey4l_: perhaps i can just switchback to my first patch and we can merge that?15:48
SamYapleyou can figure out the specifics later15:48
SamYaplethis needs to be fixed now15:48
eanderssonNetwork does not look healthy. Is linuxbridge fully functional in the latest version?15:48
SamYapleeandersson: it should be, but mind you we do not gate on LB and most of us test OVS15:49
SamYapleso it *may* be messed up15:49
SamYaplelikely not15:49
SamYaplelets dig into your issue15:49
Jeffrey4l_SamYaple, no need. your PS3 will work.15:49
SamYaplewhats the problem15:49
SamYapleJeffrey4l_: ok15:49
eanderssonThe network is created etc15:49
eanderssonbut when I try to create a VM I get a MessagingTimeout15:49
Jeffrey4l_root have no permission for /tmp/logs, but he has permission for /tmp/logs/kolla folder.15:49
SamYaplethat is a rabbitmq issue eandersson15:49
eanderssonWhen nova tries to call the validate_networks function in the API.15:49
SamYapleis rabbitmq running?15:49
eanderssonYes.15:50
SamYapledocker exec -it rabbitmq rabbitmqctl cluster_status15:50
eanderssonIt's all fine.15:50
SamYapleyou fixed rabbit previously right?15:50
eanderssonYes15:50
SamYapleyoull need to restart your openstack services. they likely stopped trying to reconnect15:51
eanderssonI then restarted all containers to make sure the bindings were in a good state.15:51
SamYaplehmmm15:51
eanderssonI first brought it all down, then started rabbit, mysql, haproxy15:51
SamYaplewhat do the nova logs have to say on this?15:51
eanderssonIt's nova complaning that it isn't getting a reply.15:51
SamYaple(and the rabbitmq logs for that matter)15:51
eanderssonI'll check the logs, but really it looks like either the queue isn't created.15:52
SamYaplethat normally means you need to restart a service15:52
eanderssonYep, restarted Neutron a couple of times now.15:53
SamYaplei would be digging into the logs to see why its failing15:53
*** Jeffrey4l_ has left #kolla15:53
SamYaplemaybe bad rabbitmq auth15:53
rhalliseySamYaple, is mariadb the only case that bootstrapping needs ARGS15:53
*** Jeffrey4l_ has joined #kolla15:53
eanderssonNothing in RabbitMQ15:53
eanderssonAll good, only bad auth was me trying to login as guest/guest at first.15:54
eanderssonIf I do a grep on the req id nothing shows up, so no service is receving it.15:54
*** ccesario has joined #kolla15:54
SamYaplerhallisey: yes, AFAIK15:54
eanderssonRestarting all Neutorn services again to make sure.15:54
SamYaplei changed the ARGS thing in the lights out patch15:54
SamYaplefor mariadb15:55
rhalliseyI'm just wondering if this is the best way to do this15:55
SamYaplerhallisey: well i cant put it into config.json because its not a permemnat thing15:56
SamYapleim not sure there is a better way, at least not this late15:56
SamYaplebut im all ears15:56
SamYaplei agree is not the best, but now mariadb bootstrap is clearer too!15:57
SamYaplewe have a true bootstrap_mariadb container15:57
SamYapleshould help with the understanding of the process now15:57
rhalliseyby permanent do you  mean only used in bootstrap or that we'll remove later15:57
SamYapleonly used in bootstrap15:57
SamYapleso we cant have the option in config.json15:57
SamYaplebecause config.json is used for bootstrap _and_ normal operation15:57
rhalliseymaybe we could do like: exec ${CMD} ${ARGS} ${BOOTSTRAP_ARGS}15:58
SamYaplewell that affects the whole start.sh script which i wont want to touch so late in the game15:59
SamYapletechnically what we are doing is how its designed to work16:00
SamYapleoriginally16:00
SamYaplei can name NEW_ARGS to BOOTSTRAP_ARGS if you want16:00
eanderssonSo I stopped all Neutron services and removed the logs.16:00
SamYaplebut im opposed to chanigng start.sh16:00
vhosakotSamYaple: How about just skipping reading / copying mariadb.log to workaround temporary Jenkins read issue....16:00
rhalliseyya let's go with bootstrap args16:00
SamYaplerhallisey: outside of start.sh right?16:01
rhalliseyya16:01
SamYapleok that works for me16:01
SamYaplevhosakot: nah we have a fix, just bikeshedding on the "best" fix16:01
SamYaplevhosakot: its just a permissions issue16:01
eanderssonand I get no errors in Neutron16:01
vhosakotSamYaple: ah ok..16:02
SamYapleeandersson: do you ahve DEBUG turned on for your openstack services?16:02
SamYapleif not i would recommend you turn it on and recheck16:02
rhalliseySamYaple, I'll file a tech debt bug for n16:02
rhalliseyshould be good to good otherwise16:03
eanderssonIs the config files shared?16:03
eandersson*are16:03
eanderssonSimilar to etc16:03
eandersson*log16:03
SamYaplewow eandersson i legit dont knwo what you are trying to day16:07
SamYaplesay*16:07
eanderssonHow do you enable debug logging on all services?16:07
SamYaplein your globals.yml16:07
eanderssonI was asking if there is a shared volume that controls etc for all containers.16:07
SamYapleno16:07
SamYaplethere are some shared folders between containers, but its mostly seperate16:08
openstackgerritSam Yaple proposed openstack/kolla: Fix mysql bootstrap  https://review.openstack.org/29492116:11
eanderssonhttp://paste.openstack.org/show/3MP196z4PmRSwIBdlRiO/16:14
eanderssonThat's from the Nova-API16:14
*** rstarmer has joined #kolla16:14
*** iceyao has quit IRC16:16
vhosakotSamYaple: saw your 2nd patch set for "Allow user to choose filesystem for osd".. so, match_mode is not needed for ceph as it is a swift variable ?16:20
openstackgerritSam Yaple proposed openstack/kolla: Change deprecated options throughout Kolla  https://review.openstack.org/29493116:20
SamYaplevhosakot: matchmode defaults to strict, but in this case I am setting the match_mode16:21
SamYapleeandersson: yea nova got issue talking with rabbitmq16:21
vhosakotah I see it, it is used in the external ceph journals patch set.. cool, thanks!16:21
SamYapleeandersson: you _should_ see some rabbitmq logs on this16:21
eanderssonSamYaple, sure, but there is nothing in the RabbitMQ logs.16:22
eandersson> accepting AMQP connection <0.3647.0> (172.16.0.10:34663 -> 172.16.0.10:5672)16:22
eanderssonFilled with nothing but that16:22
eanderssonThere was a ERR 2 hours ago due to the hostname issue16:22
SamYapleis this multinode eandersson ?16:24
eanderssonall-in-one16:24
eanderssonGoing to capture the AMQP message to see where it is going.16:24
eanderssonhmm it's going to nova-network16:26
eanderssonlol16:26
SamYaplewierd. kolla doesnt do nova-network16:26
eanderssonlolk16:27
eanderssonfound it16:27
eandersson> linuxnet_interface_driver = nova.network.linux_net.BridgeInterfaceDriver16:27
SamYapleoh yea thats not right16:27
SamYapleno wait that is correct16:28
eandersson> linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver16:28
eanderssonThat is right16:28
SamYapleyea16:28
SamYapleso that sounds like you get to write your first kolla patch!16:28
eandersson=]]16:28
eanderssonThis has been a great learning experience!!16:32
openstackgerritSam Yaple proposed openstack/kolla: Fix linuxbridge driver  https://review.openstack.org/29493616:33
SamYaplethere you go eandersson16:33
SamYaplethanks for testing linuxbridge for us16:33
eanderssonThanks SamYaple16:33
SamYapleJeffrey4l_: rhallisey https://review.openstack.org/#/c/294864/16:34
SamYapleignore the gate fialure, that is an issue with infra glean. im aware and fixing it16:34
SamYaplethat patch needs to merge to get rid of those UNSTABLE gates16:34
Jeffrey4l_SamYaple, done.16:35
vhosakoteandersson: thanks for reporting the linuxbridge driver issue!16:37
eanderssonThanks for answering my (mostly stupid) questions! =]16:38
vhosakothaha.. there is no stupid q16:38
eandersson:D16:38
vhosakot:)16:39
openstackgerritMerged openstack/kolla: Fix permissions for log extraction in gate  https://review.openstack.org/29486416:39
eanderssonSamYaple: Re-applying with the patch is cauisng this fatal: [localhost] => error while evaluating conditional: container_state.Running == false16:42
eanderssonFor task [neutron | Ensuring the neutron_linuxbridge_agent container is up]16:42
eanderssonBetter to just re-create from scratch?16:42
SamYapleeandersson: you can remove the container and run redeploy yes16:47
SamYapleJeffrey4l_: reconfig is busted in ceph as well16:47
SamYapleJeffrey4l_: we need to fix reconfig16:48
SamYapleeandersson: reconfig is about 2 weeks old as a feature16:48
eanderssonah =]16:48
eanderssonWell it's pretty awesome already.16:48
openstackgerritSam Yaple proposed openstack/kolla: Add memcache_servers to keystone_auth section  https://review.openstack.org/29493716:49
SamYapleeandersson: this makes me very happy to hear16:50
SamYapledo you use ceph?16:50
eanderssonNope16:51
SamYapleok16:52
eanderssonCinder is disabled.16:52
ccesario_hey SamYaple16:54
ccesario_I saw your message now... did you gt problem with mysql!?16:55
Jeffrey4l_SamYaple, i will check the ceph reconfig stuff.16:58
SamYapleccesario_: yes16:58
Jeffrey4l_and try to fix it.16:58
SamYapleccesario_: https://review.openstack.org/#/c/294921/16:58
SamYapleJeffrey4l_: its broken when rgw is not enabled16:58
SamYapleJeffrey4l_: its an easy fix, i just dont have time16:58
Jeffrey4l_SamYaple, yup. if so, it is easy to fix.16:59
eanderssonSamYaple: https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/do_reconfigure.yml#L4017:05
eanderssonLooks wrong :D17:05
*** achanda has joined #kolla17:06
SamYaplerhallisey: what do you think about this patch https://review.openstack.org/#/c/294854 ? it was generated from a list of most installed packages in our gate. it reduces gate time ~5m and size about ~600MB17:06
vhosakoteandersson: please file a bug, I will fix it17:06
SamYaplerhallisey: i used a similiar method for what i used last time to generate that list17:06
vhosakoteandersson: for https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/do_reconfigure.yml#L40 thanks for pointing17:06
SamYapleeandersson: you keep breaking it, well keep fixing it17:06
vhosakoteandersson: linuxbridge container cannot be  reconfiuged.. is this the issue ?17:07
ccesario_SamYaple, nice... let me try apply it and test17:07
SamYapleccesario_: youll need contaienr rebuild for mariadb17:08
eanderssonvhosakot: Yea, that is the issue, but I found it odd that it had name: neutron_openvswitch_agent under linuxbridge.17:08
vhosakoteandersson: yep, saw it.. I will fix... caould file a bug please.. and thanks!17:09
vhosakoteandersson: do you use AIO or multi ?17:09
eanderssonAIO at the moment17:09
vhosakoteandersson: cool, can you also add the output of "brctl show" in the bug please17:09
*** achanda has quit IRC17:10
rhalliseySamYaple, what about cenots17:12
rhalliseycentos17:12
rhalliseySamYaple, you can leave that to me if you like17:17
SamYaplerhallisey: if you mean packaging, this is just for source for testing17:18
SamYaplerhallisey: the same applies for binary, but tracking down those packages is harder for me17:19
SamYaplerhallisey: do you think its a good idea? that list _was_ autogenerated for packages that got installed 10 or more times in our containers17:19
rhalliseyya I do17:19
rhalliseythat way containers are literally just getting very specific pacakges17:20
SamYapleok. do you want to do centos binary packages in a sperate patch or would you like to coauthor?17:20
SamYapleyour call17:20
rhalliseyI'll do it later17:21
rhalliseyin a sep patch17:21
eanderssonvhosakot: I already destroyed my enviornment unfortunately.17:21
vhosakoteandersson: cool, np.. I got the output in my env... just filing bug would be great17:21
SamYaplerhallisey: ok17:23
vhosakoteandersson: I can file the bug myself.. np... :) I reproduced the issue17:24
eanderssonThanks vhosakot17:24
vhosakotnp :)17:24
*** vinkman has joined #kolla17:28
*** rstarmer has quit IRC17:28
*** vhosakot has quit IRC17:33
SiRiuS_mandre, ping17:36
openstackgerritSam Yaple proposed openstack/kolla: Increase usefulness of openstack-base  https://review.openstack.org/29485417:41
*** jpeeler has quit IRC17:53
*** achanda has joined #kolla18:03
ccesario_SamYaple, patch applyed... and rebuilding mariadb image18:20
*** vinkman has left #kolla18:49
*** ccesario has quit IRC18:58
*** ccesario has joined #kolla19:00
eanderssonSamYaple: It looks like something else was missing.19:04
eandersson> network_api_class = nova.network.neutronv2.api.API19:04
eanderssonuse_neutron for Mitaka19:05
eanderssonnvm it was there19:05
*** achanda has quit IRC19:06
eanderssonbut all good now!19:06
*** vhosakot has joined #kolla19:20
*** achanda has joined #kolla19:32
*** Jeffrey4l__ has joined #kolla19:33
*** Jeffrey4l_ has quit IRC19:34
*** ccesario has quit IRC19:38
*** ccesario has joined #kolla19:46
*** mdnadeem has quit IRC19:51
*** jasonsb has quit IRC20:07
*** vhosakot has quit IRC20:17
*** achanda has quit IRC20:26
*** vinkman has joined #kolla20:35
*** sdake has joined #kolla20:52
sdakehey folks20:52
*** ccesario has quit IRC21:07
*** achanda has joined #kolla21:12
*** jasonsb has joined #kolla21:13
*** sdake has quit IRC21:15
SamYapleeandersson: glad you figured it out! we made that change a few weeks ago21:17
SamYapleccesario_: any luck?21:30
*** vinkman has quit IRC21:40
*** achanda has quit IRC21:44
ccesario_SamYaple, sorry by delay... I needed recreate my environment .....21:48
ccesario_SamYaple, yes.... it worked as expected... the deployt works without problem :)21:48
SamYaplewonderful21:49
ccesario_SamYaple, thank you man by  his :)21:49
ccesario_this*21:50
*** DuncanT has joined #kolla21:57
DuncanTHi. Quick question I can't find the answer to in the docker docs: what does setting MountFlags=shared do? (or indeed private, or slave, or any other valid values).21:57
SamYapleit allows us to create shared mountpoints21:58
SamYapleor docker specifically21:58
SamYaplethis is a requirement for our "thin" neutorn contaienrs21:58
SamYapleyou can access the containers network namespaces from the host and other contaienrs21:58
SamYaplebasically mounts inthe container propogate up to the host21:58
DuncanTThere's no particular downside to setting it on a docker host that might be used for other docker dev work?21:58
SamYaplenope21:58
DuncanTGreat, thanks21:59
SamYaplea potential security problem in some situations21:59
SamYaplebut thats are really uncommon (and require mounting things from within contaienrs)21:59
*** achanda has joined #kolla21:59
DuncanTNot bothered by that, thanks, I'm just teaching myself docker bit by bit and don't want to break the basics22:00
SamYaplethats a fairly advanced docker thing that _most_ people arent aware of, so you should be fine22:00
SamYapleit just landed in 1.1022:00
*** mbound has joined #kolla22:21
-openstackstatus- NOTICE: Gerrit is restarting to increase performance issues22:22
SiRiuS_How do22:32
SiRiuS_How do I deploy again with cep, after a cleanup-containers?22:32
SiRiuS_*ceph22:32
SiRiuS_I get an error at this task TASK: [ceph | Generating initial Ceph keyrings and monmap22:33
openstackgerritSam Yaple proposed openstack/kolla: Allow external ceph journals and fix bootstrap  https://review.openstack.org/29486222:36
openstackgerritSam Yaple proposed openstack/kolla: Allow user to choose filesystem for osd  https://review.openstack.org/29486322:36
openstackgerritSam Yaple proposed openstack/kolla: Add memcached_servers to keystone_auth section  https://review.openstack.org/29493722:36
openstackgerritSam Yaple proposed openstack/kolla: Change deprecated options throughout Kolla  https://review.openstack.org/29493122:36
SamYaplerhallisey: had a bad rebase on that ceph patch ^22:37
SamYaplerhallisey: 12 minutes!!! full container build in the gate in 12 minutes!!!22:38
SamYapleall kolla built in the gate in 12 minutes!22:38
*** sdake has joined #kolla22:53
openstackgerritSam Yaple proposed openstack/kolla: Change deprecated options throughout Kolla  https://review.openstack.org/29493122:58
openstackgerritSam Yaple proposed openstack/kolla: Rework heka log copying  https://review.openstack.org/29420623:01
openstackgerritSam Yaple proposed openstack/kolla: Fix 'optional' config.json values  https://review.openstack.org/29435023:01
*** achanda has quit IRC23:13
*** sdake has quit IRC23:14
openstackgerritSam Yaple proposed openstack/kolla: Change deprecated options throughout Kolla  https://review.openstack.org/29493123:21
*** sdake has joined #kolla23:28
openstackgerritSam Yaple proposed openstack/kolla: Test commit  https://review.openstack.org/29496523:32
*** sdake has quit IRC23:32
*** sdake has joined #kolla23:35
*** unicell1 has joined #kolla23:52
*** unicell has quit IRC23:54
« Friday, 2016-03-18 Index Sunday, 2016-03-20 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!