Friday, 2015-06-19

*** sdake_ has joined #kolla00:00
*** jasonsb has joined #kolla00:00
*** sdake has quit IRC00:03
sdake_jpeeler https://review.openstack.org/#/c/193369/00:03
*** sdake_ is now known as sdake00:06
*** dims has joined #kolla00:39
*** sdake_ has joined #kolla00:41
sdake_there is some anestaphine in the back of the medlock!00:41
*** sdake has quit IRC00:44
*** vinkman has quit IRC00:45
*** dims has quit IRC00:47
*** dims has joined #kolla00:47
*** openstack has joined #kolla00:55
bmacethat is a whole lot of failed.. interesting yum failure message too, yeah, not happy01:00
sdake_the fact that setup_gate.sh failed indicates its a fedora problem as wel las a centos problem01:01
sdake_and I know for a fact they are in separate dcs01:01
sdake_so must be an internet wide thing or infra thing01:01
sdake_because the gate was failing for epel as  well01:01
sdake_in other cases01:01
*** zhiwei has joined #kolla01:03
*** erkules_ has joined #kolla01:09
*** erkules has quit IRC01:11
*** ccrouch has quit IRC01:14
jpeeleroh wow01:22
jpeeleryeah that's some serious failure01:22
jpeelersdake_: so i'm not sure if we really need to remove the voting gate or not...01:24
sdake_we are reerting it for now01:25
sdake_and I asked mandre to add a retry01:25
sdake_and then we will revert hte revert01:26
sdake_atm, the gate is totally blocked01:26
jpeeleryeah01:26
jpeeleri don't think i've ever seen a reverted revert01:26
jpeeleri wonder if any other projects have so much trouble with the gate, though this doesn't seem to be our fault at all01:28
sdake_status.fedoraproject.org says all is good01:29
sdake_but I hae seen several irc netsplits today01:29
sdake_so possibly the internet is busted01:29
jpeeleri'm pretty sure something in the gate is wonky01:29
jpeeleror that01:29
sdake_but I want retries in the build script01:29
jpeelerguess we need to run a network precheck in our scripts to help us debug01:29
jpeeleri think i might tack that on that bug...01:30
jpeelerhrm, is that easier to do in python? maybe i'll just do it01:30
sdake_is what easier to do i npython01:31
sdake_its not tthat the network has completely failed, its that it is intermitently failed01:31
jpeelerthe last one sure does look like complete failure01:31
openstackgerritMartin André proposed stackforge/kolla: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338901:31
mandrelet's see if that makes things better…01:32
sdake_mandre its not just epel01:32
sdake_its everything01:32
sdake_i have seen failures in epel, setup_gate.sh at the fedora docker install stage, and in the mirrorlist retrieval for centos01:33
sdake_anyway off for a break ttyal :)01:33
jpeeleri kind of want to see that run, and i don't think it will unless a + vote is given. so here goes01:36
jpeelermandre: here's your run - https://jenkins07.openstack.org/job/check-kolla-functional-build-f21/24/console01:37
mandrethanks jpeeler01:38
mandrelooking good so far01:39
jpeelerif it does work, i would feel a lot better if we go ahead and had retries too. think that'd be difficult to do?01:40
mandrethinking more about the proposed patch, I don't think it solves anything01:41
mandreI've never seen the base image failing, and the next line does a full packages upgrade anyway01:41
jpeelerit's not a fix i would have ever thought of01:41
mandreaccording to the error message, there is an option to skip unavailable repos, that may increase our success rate01:43
mandreyum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true01:43
jpeeleri don't see how we can skip though01:43
bmaceif they are ones we need we will just get different kinds of failures, and i can't imagine we have a ton of repos configured that we don't use.01:44
jpeelercorrect01:44
jpeelerthe one repo i have experienced this personally with is the google chrome repo. i just do another update and then it works01:44
jpeelernever really could figure out what the deal was01:45
jpeeleri don't know how easy it would be to check the build output for yum failure01:48
jpeelerbut mandre, maybe modifying build_image to try again upon failure would suffice?01:48
mandrejpeeler, possibly, however that won't fix the root cause01:50
mandreyou may get debug information exporting the URLGRABBER_DEBUG=1 variable before doing the yum install01:51
jpeeleri know...01:51
mandresorry, I can't be of any more help right now, I have to run01:53
jpeelerthanks for looking at it!01:53
mandreit would be interesting to know exactly what it is that yum doesn't like01:54
jpeelerthis might a silly question, but can Dockerfiles read env vars?01:55
jpeelerotherwise, modifying every Dockerfile doesn't sound fun01:55
mandreyou can export ENV in the base image01:56
jpeelerah right01:56
mandrethe build is doing fine as far as I can tell, it's probably just a happy coincidence01:57
mandreoops, kollaglue/centos-rdo-neutron-base failed01:58
mandreso yeah, that didn't change anything01:58
*** britthouser has joined #kolla01:58
mandrebbl01:59
*** britthou_ has quit IRC02:02
jpeelerit's not fixed :/02:07
mandrejpeeler, if you have some time to look at the issue I suggest you try setting the URLGRABBER_DEBUG variable to 1 and analyzing the logs02:12
jpeeleryeah i'll do that02:12
*** daneyon_ has quit IRC02:13
mandresetting "yum-config-manager --save --setopt=epel.skip_if_unavailable=true" may help unstuck the gate as well02:13
jpeelercan't really skip epel though02:15
mandreadding a retry for all yum operation is not a trivial change, it may require changing all dockerfiles (unless you manage to alias yum to a function that does the retry)02:15
mandreare all openstack-* packges coming from epel?02:16
jpeelerthat's why i suggested modifying build_image in build-all-docker-images instead. but that can come later02:17
mandrein the latest build, it died trying to install openstack-neutron-ml202:17
mandreoh, i see02:17
jpeeleri don't know what the exact breakdown is for tje repos02:17
mandreyeah, adding the retry directly in the build script is also an option02:18
jpeelernot elegant, but could be effective02:18
jpeelermandre: looking for the ultimate debug here, because i'm giving it one go and calling it a night. anything else other than URLGRABBER_DEBUG?02:19
jpeeleri was thinking of setting yum debug level to 1002:20
mandreno idea, http://unix.stackexchange.com/questions/193480/cannot-use-epel-repo-get-nss-error-5990-pr-io-timeout-error only mentions URLGRABBER_DEBUG02:20
*** tobe has joined #kolla02:24
openstackgerritJeff Peeler proposed stackforge/kolla: Attempt to add maximum debugging to yum  https://review.openstack.org/19339702:29
jpeelerugh, Dockerfile noob02:31
openstackgerritJeff Peeler proposed stackforge/kolla: Attempt to add maximum debugging to yum  https://review.openstack.org/19339702:32
*** sdake_ has quit IRC02:34
*** prad has quit IRC02:35
openstackgerritJeff Peeler proposed stackforge/kolla: Attempt to add maximum debugging to yum  https://review.openstack.org/19339702:39
openstackgerritJeff Peeler proposed stackforge/kolla: Attempt to add maximum debugging to yum  https://review.openstack.org/19339702:50
*** sami__ has joined #kolla03:01
*** dims has quit IRC03:06
*** sami_ has quit IRC03:06
openstackgerritJeff Peeler proposed stackforge/kolla: Attempt to add maximum debugging to yum  https://review.openstack.org/19339703:19
*** sdake has joined #kolla03:33
sdakeevening03:33
* jpeeler is going to bed03:35
jpeelersdake: see if this debug output means anything to you https://review.openstack.org/#/c/193397/03:35
sdakejpeeler nothing jumps out at me03:37
sdakenight jepeeler03:37
*** zhiwei has quit IRC03:40
sdakeThe Most Popular Problem occured when your Centos Operation System CLOCK not be correct04:02
sdakeso https keys ( certificates ) goes down due to invalid time and date , you must set your clock correct then install EPEL repo.04:02
sdakebut when this problem occured you can edit the /etc/yum.repos.d/epel.repo and comment the mirrorlist url (that have https) and uncomment baseurl.04:02
sdakeby this method your downloads passthrough HTTP protocol and time miss mach cannot make yum install as crazy.04:02
sdakemandre ^^04:02
*** gfidente has quit IRC04:06
*** sdake_ has joined #kolla04:07
openstackgerritSteven Dake proposed stackforge/kolla: Test change  https://review.openstack.org/19340404:08
*** sdake has quit IRC04:10
sdake_2015-06-19 04:12:18.558 | Fri Jun 19 04:12:18 UTC 201504:15
sdake_date looks good04:15
sdake_that shoots down that theory04:15
bmace:(04:16
openstackgerritMartin André proposed stackforge/kolla: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338904:16
sdake_that leaves 443 firewalled as an option04:17
sdake_ mandre you only have "mirrorlis" not "mirrorlist"04:19
mandrebah, doesn't matter, it's commented out :)04:19
sdake_ok04:20
sdake_it could be time related, we just went through a dst change in the last couple of weeks04:20
mandrethe weird thing is, it's not reliably failing04:21
sdake_but some locals do differently adjust dst from the us04:21
sdake_the gate runs on hp clouds and rax clouds04:21
sdake_using baeurl seems like the wrong workaround to me :)04:22
sdake_http://www.timeanddate.com/time/dst/2015.html04:23
mandrewhere is jpeeler finding the url of the jenkins builds?04:26
sdake_status.openstack.org/zuul04:27
sdake_use browser find to search for changeset04:27
sdake_click big box (not the changset link)04:27
sdake_it will hwo you the gates04:27
mandrecool, took me a wile to realize i can click the blue box04:29
sdake_https://jenkins06.openstack.org/job/check-kolla-functional-build-f21/28/console04:29
sdake_hwo/show04:29
sdake_mandre i dont like that hcange it doesn't solve the root cause tho04:30
sdake_because wedon't know the root cause04:30
sdake_and loads everything up on one url rather then the mirror list04:31
mandreagreed, we should not disable https but fix it04:31
mandrei'm simply curious if that's a valid workaround04:31
mandrewe could do a s/https/http if mirrorlist responds to http04:33
sdake_https://review.openstack.org/#/c/193369/04:33
sdake_mandre read and add your comment in there plz04:34
sdake_i read something today on #openstack-infra that there was a change in networking04:37
sdake_or something like that04:38
sdake_id be satisified with a mirrorlist ofhttp rather then https04:40
sdake_we dont need https in this case04:40
mandrei may have screwed the patch, it's still trying to find a mirror04:41
sdake_sedblack magic to me ;)04:41
sdake_the intenet was busted today mandre04:42
sdake_if that helps :)04:42
openstackgerritMartin André proposed stackforge/kolla: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338904:43
sdake_https://downdetector.com/archive/04:43
sdake_look at the 18th04:43
mandrethe internet is seriously broken04:44
sdake_probably some country rerouting traffic through their switches to monitor my irc chats04:46
mandrethe cleaning lady pulled the cable again04:51
jugglerlol04:52
sdake_tcp/ip designed to withstand nuclear attack04:53
sdake_doubt a cleaning lady pulling a plug would dod it :)04:53
sdake_although the intenet is more fraglie then itused to be04:53
jugglerheh yep04:54
sdake_mandre where is yourzuul job - idon't see iti nthe list04:56
sdake_nm i see it04:56
mandrehttps://jenkins07.openstack.org/job/check-kolla-functional-build-f21/28/console04:56
sdake_looks like http is a valid mirrorlist server endpoint04:57
sdake_it could just be that centos epel mirrorlist servering via https is broken some of the time04:58
sdake_and since we access it 40 times or more per gate04:59
sdake_our chances are high that one of them will fail04:59
mandreeven the big tohoku earthquake was not able to break internet in japan, http://conferences.sigcomm.org/co-next/2011/workshops/SpecialWorkshop/papers/1569500743.pdf05:00
sdake_mandre http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n11805:04
sdake_note we are not running that code in our gate05:05
sdake_because we don't do ad evstack-gate setup operation05:05
sdake_perhaps we need something like it05:05
*** Haomeng|2 has joined #kolla05:10
*** Haomeng has quit IRC05:13
mandremy latest patch to use http mirror didn't work, am I modifying the correct file at least?05:15
sdake_it looks like its still running mandre?05:16
mandrelooking directly at jenkins console05:17
openstackgerritMartin André proposed stackforge/kolla: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338905:17
sdake_i was looking at it05:18
sdake_2015-06-19 05:17:46.129 | Finished: ABORTED05:19
sdake_it didnt' finish becuase you submitteed a new patch mandre05:19
sdake_did you see it fial a build it should have not fialed?05:19
mandreyeah, now it's gone because I submitted a new patch, it had plenty of "Cannot retrieve metalink for repository" errors05:19
mandrefor the keystone image, etc.05:20
sdake_mandre those aren't fatal, it just will try a new mirror in that case05:20
sdake_when its fatal it prints out an assay warning about how it gave up05:20
sdake_it is possible the mirrors are updating atm05:21
sdake_if only there were this tool to search the internet to find out what centos is doing with their mirrors ;)05:21
sdake_god my wife yelling at me, I have had it up to here with it05:22
sdake_sorry for tkaing it out on you mandre05:22
sdake_http://mirror-status.centos.org/05:23
sdake_changes in the last 1-7 hours05:23
sdake_perhps that is the cuase05:23
mandrehmmm jenkins didn't like the last patch05:24
sdake_the fact that the mirrors are syncing05:24
sdake_could be cause for the repomod fialures05:25
sdake_repomod not found is different from the 500 mirrorlist error05:25
mandreso you say this error is not fatal? http://logs.openstack.org/89/193389/4/check/check-kolla-functional-build-f21/f967e29/console.html#_2015-06-19_05_20_39_55305:25
mandrei think that's what I has in the last build05:26
sdake_another possibility is coprs05:27
sdake_does coprs install https or http repos05:27
sdake_see the wall of text, that is different from the repomod data not being correct or found05:28
*** inc0 has joined #kolla05:28
sdake_2015-06-19 05:20:39.550 | INFO:test_images:One of the configured repositories failed (Unknown),05:28
sdake_I dont even know what that means05:28
inc0good morning05:28
sdake_inc0 yo05:28
inc0wassup?05:29
mandrein /etc/yum.repos.d/sdake-pam.noaudit-epel-7.repo, we have baseurl=https05:29
sdake_the world is imploding05:29
inc0last time I heard it was exploding, so I guess that's an improvement?05:29
sdake_maybe coprs is busted05:29
sdake_no implosion = singularity05:29
sdake_did I ever mention i hate computers ;)05:30
inc0we kolla gained self-conciousness and now our containers are more intelligent than we are?05:30
sdake_inc0 i mean the black hole singularity not the technology singualirty :)05:31
sdake_if the containes were more self-ware they would cooperate or face deletion05:31
sdake_;)05:31
inc0and fact that they don't cooperate prooves that they aren't that intelligent after all?05:32
sdake_precisely05:32
sdake_ok well I've had enough fun for today05:33
inc0I'll post a blueprint "create a container head on the spike to promote cooperation"05:33
sdake_if mandre cang et the gate unhammed05:33
sdake_unjammed05:33
sdake_yay for mandre :)05:33
sdake_night all05:33
*** sdake_ has quit IRC05:33
inc0mandre, did you fix single interface problem?05:33
mandresdake_ I see you're not losing hope :)05:34
mandreinc0: nah, the gate is currently failing and we're trying to figure out what's going on05:35
mandreyum is unable to find a mirror, and that fails the build05:36
mandreit's happening more frequently now05:36
inc0any chances we could create our own mirrors?05:36
inc0somewhere inside infra?05:37
mandrewe first thought that was an issue with https, where it's unable to validate the certificate, but it doesn't appear to be the case05:37
mandrewe would have to mirror all distros, not sure this is a good idea05:38
inc0where can I see traceback?05:38
mandreavailable mirrors are supposedly reliable05:38
mandrehere is one: http://logs.openstack.org/89/193389/4/check/check-kolla-functional-build-f21/f967e29/console.html#_2015-06-19_05_20_39_55005:39
inc0hmm..it have installed package before, so there are repos available05:40
inc0question is which one05:41
*** nihilifer has joined #kolla05:44
inc0mandre, http://logs.openstack.org/89/193389/4/check/check-kolla-functional-build-f21/f967e29/console.html#_2015-06-19_05_20_02_533 this line changes protocol05:47
inc0maybe one of mirrors doesn't have http at all?05:47
mandrethe error message is not the most obvious05:50
inc0far from it05:51
inc0form obvious I mean05:51
openstackgerritMartin André proposed stackforge/kolla: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338905:52
mandrethis should add some debugging05:52
inc0so now we wait05:53
mandrehttps://jenkins06.openstack.org/job/check-kolla-functional-build-f21/29/console05:53
nihilifero/05:54
inc0cześć Michał05:54
nihiliferoh, you're already talking about failing gate?05:54
nihilifersiemanko :)05:54
inc0yup, for a good morning start!05:54
inc0that's a traceback out there! run for your lives!05:55
mandreinc0, if you don't mind I'll leave it to you, at least you have somewhere to start looking :)05:58
inc0well, I can't promise anything05:58
mandrei really need to work on what they're paying me for now, I'll be back later06:00
inc0sure, I'm in comfortable situation because *this* is my work;)06:00
mandreand don't forget sdake will be mad at you if you don't fix the gate :)06:00
inc0I'll let you know what I'll find06:01
mandreputting some pressure06:01
*** absubram has joined #kolla06:01
*** absubram has quit IRC06:01
mandrewe've pushed a patch to make the gate non voting https://review.openstack.org/#/c/193369/06:02
openstackgerritMichal Jastrzebski (inc0) proposed stackforge/kolla: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338906:06
bmaceis that actually in now mandre?06:18
mandrebmace: the gate is still voting06:20
*** Slower has joined #kolla06:24
bmacewell, sleep time, maybe the internet will be happier when i wake up ;)06:26
*** gfidente has joined #kolla06:27
mandre'night06:27
*** juggler has quit IRC06:43
*** dasm|afk is now known as dasm07:11
dasmgmorning guys07:14
openstackgerritMichal Jastrzebski (inc0) proposed stackforge/kolla: Keepalived container  https://review.openstack.org/18798107:16
inc0damn, I've lost 2*+2:(07:17
Haomeng|2one question, who can help me - encountered "ERRO[0052] Couldn't run auplink before unmount: exec: "auplink": executable file not found in $PATH" error for command "kolla start", any comments?07:22
Haomeng|2details - http://paste.openstack.org/show/304149/07:22
inc0seems like you're missing a package07:25
inc0try to install auplink (I'm not sure if that's a case, but well..)07:25
SamYaplehello07:27
Haomeng|2inc0: ok, thank you07:28
inc0good morning SamYaple07:28
openstackgerritMichal Jastrzebski (inc0) proposed stackforge/kolla: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338907:32
*** akwasnie_ has joined #kolla07:36
*** akwasnie_ is now known as akwasnie07:37
Haomeng|2inc0: I run "apt-get install aufs-tools" to install auplink command, and no previous auplink not found issue now, thank you07:37
Haomeng|2inc0: but new issue encountered - http://paste.openstack.org/show/304150/, any ideas?07:37
*** inc0 has quit IRC07:49
nihiliferHaomeng|2: which version of docker-compose do you have?07:49
nihilifermaybe just try to install the newest docker-compose in virtualenv07:58
Haomeng|2nihilifer: ok, let me try again, thank you08:01
Haomeng|2nihilifer: my version is - docker-compose version: 1.4.0dev, not sure if this latest one08:06
*** inc0 has joined #kolla08:20
*** inc0_ has joined #kolla08:22
*** inc0 has quit IRC08:25
*** inc0_ is now known as inc008:25
inc0SamYaple, why did you -2 on this? https://review.openstack.org/#/c/187225/12/docker/centos/binary/galera/start.sh08:29
SamYaplei explained in the comments inc008:30
SamYaplei only -2 it because sdake had +2 it and its a major issue08:30
inc0uhh...-2 basically means patchset goes to trash08:30
inc0anyway, how would you disable an ENV?08:32
inc0after initial use?08:32
SamYapleinc0: i means do no merge, the patchset still exists08:32
SamYapleinc0: unset the variable, change it to false something like that08:33
inc0well, technically yeah, but it's usually used as "this is bad idea, we don't want this stuff in at all"08:33
inc0but you can't do this from inside container08:33
inc0and we don't have anywhere to unset it08:33
SamYapleyou can unset a variable from inside the container, but that wont stop recreates08:33
inc0this has to be unset on host08:33
SamYaplethis is something that needs to be figured out08:34
SamYapleotherwise were going to be trashing the db08:34
inc0well, this is issue with pretty much everything08:34
inc0I've submitted several bugs for that08:34
inc0https://bugs.launchpad.net/kolla/+bug/146647408:34
openstackLaunchpad bug 1466474 in kolla "Nova inits database each time container is run" [Undecided,New]08:34
inc0in most of patches we don't even check anything08:35
SamYaplethats a different issue08:35
inc0similar problem, different severity08:35
SamYaplesetting --wsrep-new-cluster can trash the whole database08:35
inc0but we need to figure this out as policy for every container08:35
SamYapleno08:35
SamYaplethis is a different issue08:36
SamYapledb_sync is safe to rerun, this is not08:36
inc0not always08:36
inc0anyway, I too think we need to figure this out08:36
inc0there will be other problems like that08:36
SamYaplecan you point to one?08:37
SamYapleif you mean future problems, those arent in the codebase08:37
SamYapleim not merging this into the code base with a major issue like that08:37
inc0well, imho sync_db isn't safe to run08:37
inc0nto always anyway08:37
SamYaplewhy08:37
inc0it not AS severe as this08:37
inc0but alters can lock tables08:37
inc0locket tables can mean downtime08:38
inc0accidental downtime is bad08:38
SamYaplefair enough08:38
inc0if you have huge databases08:38
SamYaplei dont disagree with you on that in anycase08:38
inc0which in case of for example ceilometer you can08:38
SamYaplebut that is not a corruption issue, thats a downtime issue08:38
SamYaplethis is a lose all your data issue08:38
inc0I've ran ceilometer alter once...it hold out whole system for 24h+08:38
inc0I know, I'm saying we're facing this problem08:39
inc0in few cases, not just this one08:39
inc0we need to figure it out08:39
SamYapleagree, but th db_sync is NOT a data corruption problem08:39
SamYaplegalera container is08:39
inc0still, we need to figure it out08:39
inc0imho in several places08:39
inc0so lets just move to figuring it out;)08:39
inc0env can be set up08:40
inc0hmm...I have an idea08:40
SamYaplecool, but its going to hold up the container until we do since this is the only issue that can lose data08:40
inc0no, I don't. I thought about adding someting to .buildconf, but we need decision on deployment moment08:40
inc0ENTRYPOINT?08:40
SamYaplecmd/entrypoint doesnt solve this08:41
inc0how can we push parameter like "yes, I am sure I want to init this time"08:42
SamYaplewith docker-compose? I do not know. it needs outside orchastration08:42
SamYaplebasically init container08:43
SamYapledestroy init08:43
SamYaple(init container doesnt have to be a new image)08:43
inc0and outside orchiestration can use this env we've set08:43
inc0docker compose will require manual stuff anyway, so again, env we've set works08:44
SamYapleok. so ill go through the scenario.08:44
inc0if someone doesn't turn it off...well...all we can do is to add BOLD UPPERCASE RED warning to docs...08:45
SamYaplestart contaienr 1 with --wsrep-new-cluster08:45
SamYaplestart contaienr 2 adn 3 without it08:45
SamYaplerestart container 108:45
SamYapleit wont join the cluster08:45
inc0why?08:45
SamYaple--wsrep-new-cluster08:46
inc0(if you'd remove env before_08:46
SamYaplehow does that remove it from docker-compose?08:46
inc0changing openstack.env file won't update it on restart?08:46
inc0it won't reread this?08:47
SamYapleno no08:47
SamYaplewe have restart-always set to on08:47
SamYapleif the contaienr restarts it will run --wsrep-new-cluster again08:47
inc0it can do this automatically08:47
inc0yeah, I get it now08:47
inc0hmmm...08:47
inc0maybe additional check there?08:48
SamYaplebasically the container has to check to see if the cluster exists which is a whole lotta stuff that should be outside orchastration in my opinion. but we are where we are08:49
SamYapleso that logic will need to be in the container08:49
inc0how about lock file added on first init?08:50
inc0we'll mount some external folder08:51
inc0and add cluster_created file08:51
inc0and if you explicitly want to recreate cluster, you need to set env up AND remove cluster_created file from host08:51
SamYaplehow will you deal with a situation where all database nodes have stopped?08:51
inc0we'll still have env only on master node08:51
inc0and this node will also have lockfile08:52
mstachowmorning08:52
* mstachow ziew08:52
SamYaplelets put it this way, im ok with however yo uwant to solve it for config-internal08:52
SamYaplemorning mstachow08:52
inc0let's solve this for bindconf later? we don't have anything besides config-internal08:53
inc0right now08:53
inc0we can file an critical bug at once08:53
inc0anyway, I have meeting now, think about that plz, I'll ping you when I'm back08:54
SamYapleits solved for config-external08:54
SamYapleyou just need to solve it fr config-internal  before merge08:54
*** inc0 has quit IRC08:58
*** inc0 has joined #kolla08:59
*** athomas has joined #kolla09:09
*** juggler has joined #kolla09:14
pbourkeharmw: you there?09:26
*** jmccarthy has quit IRC09:28
*** jmccarthy has joined #kolla09:28
*** tobe has quit IRC09:28
openstackgerritPaul Bourke proposed stackforge/kolla: Fix 'unsupported locale setting' errors in init-runonce  https://review.openstack.org/18843209:55
openstackgerritPaul Bourke proposed stackforge/kolla: Minor doc fixups in dev-quickstart.md  https://review.openstack.org/19054309:56
openstackgerritSam Yaple proposed stackforge/kolla: WIP Change fat neutron container into thin contianers  https://review.openstack.org/19348010:02
*** inc0 has quit IRC10:03
*** inc0 has joined #kolla10:04
SamYapleHey guys, could use a review of https://review.openstack.org/#/c/193480/1/docker/centos/binary/neutron/neutron-base/ip_wrapper.py10:04
*** dims has joined #kolla10:09
*** inc0 has quit IRC10:09
harmwpbourke: yo10:10
harmwI'm reading your reply, you are right ofcourse on most of it :)10:11
harmwbut10:11
harmwthe multiple RUN statements, comments and mkdir's you've found in other Dockerfiles are on the 'to be fixed'-list :)10:12
harmwin fact, there is a review from fangfenghua that addresses those things10:13
harmwaligntment with docker best practices10:13
harmwre: hardcoded FROM, I think I'm wrong here since you target the -base container10:13
harmwpbourke: I hope that makes clear why I choose to -1 :)10:14
*** Haomeng has joined #kolla10:38
*** vbel has quit IRC10:40
*** vbel has joined #kolla10:41
*** vbel has quit IRC10:41
*** Haomeng|2 has quit IRC10:41
pbourkethanks harmw10:43
*** inc0 has joined #kolla10:47
harmwSamYaple: I'll gladly look at that tonight10:47
harmwpbourke: no problem, if I do weird reviews ever again that don't make sense - just let me know :p10:48
*** inc0_ has joined #kolla10:48
pbourkeit was just funny as for a moment I thought we were looking at two different code bases ;)10:49
harmwhehe10:49
*** inc0 has quit IRC10:49
*** inc0__ has joined #kolla10:50
*** jmccarthy has quit IRC10:51
pbourkewhat happened with the gates last night?10:51
inc0__pbourke, basically yum is raising errors10:52
nihiliferyum is failing with epel mirrors, afaik10:53
inc0__https://review.openstack.org/#/c/193389/ look here10:53
*** inc0_ has quit IRC10:54
*** jmccarthy has joined #kolla10:55
*** dims has quit IRC10:57
openstackgerritPaul Bourke proposed stackforge/kolla: Symlink all non Dockerfile resources in images  https://review.openstack.org/19011710:58
openstackgerritPaul Bourke proposed stackforge/kolla: Add base image for oraclelinux  https://review.openstack.org/19101310:58
*** dims has joined #kolla11:06
*** rhallisey_ has joined #kolla11:22
rhallisey_what's up with the gate failing everywhere...11:23
SamYaplethats what I want to know rhallisey_11:26
SamYaplei dont know anything about the gate11:26
rhallisey_looks like a few images are failing to build11:27
rhallisey_ERROR: failed to build kollaglue/centos-rdo-ceilometer-notification:f69aa8511:29
rhallisey_RUN yum install -y openstack-ceilometer-notification && yum clean all11:29
rhallisey_this fails11:29
*** rhallisey_ is now known as rhallisey11:30
SamYapleugh ceilometer!11:31
pbourkeSamYaple: here's a bug for it https://bugs.launchpad.net/kolla/+bug/146667711:39
openstackLaunchpad bug 1466677 in kolla "build script errors out in yum" [Critical,In progress] - Assigned to Martin André (mandre)11:39
pbourkesdake has also submitted patch to make the gate non voting for now11:40
SamYapleif this were source builds.... ;)11:40
SamYapledid you know the steam summer sale is going on?11:40
SamYaplei did not11:40
pbourkeyup11:40
rhalliseywhat's in the steam summer sale11:40
* rhallisey looks11:41
pbourkelet us know if anything good11:41
pbourke:)11:41
rhalliseyevolve is 1/2 off11:41
SamYaplei dont know but if you like the original metroid someone just sent me this list http://www.reddit.com/r/gaming/comments/3a47iy/if_only_nintendo_had_announced_a_metroid_like/cs9ajap11:41
SamYaplei know 3 of the games on there and they are awesome, so i bought the rest11:41
rhalliseyarma 3 is 20$11:42
rhalliseythat's sick11:42
SamYaplei havent played games in so long11:43
rhalliseyI don't have a good enough computer to play much on pc :(11:44
rhalliseybut I wish11:44
SamYapleive got a semi decent desktop for the games. its 2-3 years old at this point11:44
SamYaplehavent played much in a year11:44
*** jmccarthy has quit IRC11:47
*** jmccarthy has joined #kolla11:47
* jmccarthy hasn't done much proper gaming in ages :|11:50
pbourkehave we tried something like this I wonder https://serverfault.com/questions/637549/epel-repo-for-centos-6-causing-error11:50
pbourkeseems we did11:52
pbourke:(11:52
pbourkeI think they've pushed a new centos image upstream11:55
openstackgerritPaul Bourke proposed stackforge/kolla: DO NOT MERGE: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338911:57
pbourkedo you have to +1 to trigger a check?11:59
*** erkules_ is now known as erkules11:59
*** erkules has joined #kolla11:59
*** dasm is now known as dasm|brb12:04
openstackgerritPaul Bourke proposed stackforge/kolla: DO NOT MERGE: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338912:06
*** dasm|brb is now known as dasm12:08
*** athomas has quit IRC12:14
*** athomas has joined #kolla12:25
openstackgerritMichal Rostecki proposed stackforge/kolla: DO NOT MERGE: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338912:37
rhalliseydoes that work ^12:39
rhallisey?12:39
nihiliferwe'll see. just tried to get rdo repo package without ssl, because pbrouke's patch was failing on yum -y install https://rdoproject.org/blabblah.rpm12:41
*** dims has quit IRC12:46
nihiliferaww, failed again on systemd-container12:46
*** dims has joined #kolla12:46
*** dwalsh has joined #kolla12:48
openstackgerritMichal Rostecki proposed stackforge/kolla: DO NOT MERGE: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338912:50
*** sdake has joined #kolla13:05
sdakemorning13:05
sdakegate fixed?13:05
*** sdake_ has joined #kolla13:07
pbourkenot yet13:07
pbourkemy theory is something has changed in the centos image as there was a new build pushed yesterday13:07
sdake_diddn't know that!13:08
pbourkethe git repo for their(centos) dockerfiles is a little strange though, i cant seem to get a proper history for it13:08
sdake_well we ccan check the yum repo file contents13:09
sdake_they possibly got damaged13:09
sdake_or did you do that already13:09
nihiliferinc0__ tried with yum.repos.d at the morning13:10
*** sdake has quit IRC13:10
pbourkeanyone know what the story is with systemd/fakesystemd?13:11
pbourkethere's something about it here https://registry.hub.docker.com/_/centos/13:11
nihiliferpbourke: fakesystemd, afaik, is the package installed in centos image by default to prevent installing real systemd packages13:11
pbourkenihilifer: yeah just reading it now. but we're trying to install systemd13:12
pbourke?13:12
nihiliferjust tried to do13:12
nihiliferrpm -e --no-deps fakesystemd13:12
nihiliferlocally13:12
nihiliferbut got another error, with epel...13:12
pbourkeby the looks of it if you want to install systemd you need to all the steps outlined on the above page13:13
pbourkewhich we're not doing13:13
pbourkedo we need systemd once we have thin neutron containers?13:15
sdake_we dont need real systemd afaik13:15
pbourkeill try a build with that line removed, one sec13:15
sdake_the issue is libvirt installs systemd as a part of its dep chain13:15
sdake_so i had to uninstall it and reinstall it ofr some wonky reason13:16
sdake_because systemd-container13:16
pbourke:/13:16
sdake_whatever that is13:16
openstackgerritPaul Bourke proposed stackforge/kolla: DO NOT MERGE: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338913:17
*** diga has quit IRC13:17
sdake_the issue has more to do with the yum repos13:19
sdake_and less to do with the actual pacakgess installed I think13:19
sdake_brb connecting to dev boxes13:19
*** sdake has joined #kolla13:20
sdakeso the centos image was udpated 19 hours ago13:21
sdakethat would be the prime candidate for the breakage13:21
pbourkeyeah13:21
pbourkethe question is what they did13:22
pbourkehttps://github.com/CentOS/sig-cloud-instance-images/blob/0a6a7fa816e834b29222fce2df0b858ab1b97a87/docker/Dockerfile13:22
nihiliferI just did now locally13:23
nihiliferrpm -e --no-deps fakesystemd13:23
nihilifer+ sed stuff that inc0__ submitted at the morning13:23
nihiliferseems to be working13:23
*** sdake_ has quit IRC13:24
sdakedoes it work all the time :)13:24
nihiliferbut disabling https and using hardcoded baserepo instead of mirrors isn't the way to go :P13:24
pbourkeso pinning to the older version of centos 7.1 rather than latest has the same issue: https://jenkins04.openstack.org/job/check-kolla-functional-build-f21/37/console13:24
sdakedisabling https is ok13:24
sdakeusing hardcode dbaserepo is wrong13:24
sdakedo we know precisely waht the problem is13:25
sdakeit is that mirrorlist fails?13:25
pbourkethe problem is "Cannot retrieve metalink for repository: epel/x86_64", but I dont think anyone has pinned down what that means yet13:27
pbourkeclock was suggested but that seems correct13:27
sdakeare the yum repo files correct in the image13:28
openstackgerritMichal Rostecki proposed stackforge/kolla: DO NOT MERGE: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338913:29
pbourkesdake: I think its one of the ones we're adding as part of the build13:30
nihilifersdake: the yum repo file from epel is not in raw image - it's provided by epel-repo package13:30
pbourkeThe command [/bin/sh -c yum -y install pam+noaudit && yum clean all] returned a non-zero code: 1 -- we're adding a repo to do with pam?13:31
sdakeyes i mean correct in the base image13:31
sdakeyeswe are13:31
*** akwasnie has quit IRC13:31
sdakehttps://copr.fedoraproject.org/coprs/sdake/pam.noaudit/13:32
pbourkefound it13:33
pbourketry this locally:13:33
pbourkedocker run -it centos /bin/bash13:33
pbourkecurl https://copr.fedoraproject.org/coprs/sdake/pam.noaudit/repo/epel-7/sdake-pam.noaudit-epel-7.repo -o /etc/yum.repos.d/sdake-pam.noaudit-epel-7.repo13:34
pbourkeyum -y install pam+noaudit13:34
pbourkegoes splat13:34
sdakethe dockerfile removes the existing pam on the system13:34
sdakeso that is not the same test13:34
*** jtriley has joined #kolla13:35
sdakecentos-rdo-base is not failing to buidl13:36
sdakepbourke Step 12 : RUN rpm -e --nodeps pam13:37
pbourkeah I see now yeah13:37
pbourkedisregard13:37
sdakehowever step 13 just faied on centos-rdo-base for me13:37
sdakeCannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again13:39
sdakeneed to know what is contained in tha tfile13:39
nihilifersdake: try to add13:40
nihiliferRUN sed -i 's/^mirrorlist/#mirrorlist/' /etc/yum.repos.d/epel.repo13:41
nihiliferRUN sed -i 's/^#baseurl/baseurl/' /etc/yum.repos.d/epel.repo13:41
sdakei want to know contents of  epel.repo unchanged :)13:41
nihiliferah, ok13:41
sdakeit is possible rpm -e pam is triggering problems with epel13:41
larskssdake: highly unlikely. Sometimes repositories fail to answer, but work when you retry.13:42
pbourkesdake: contents fyi https://gist.github.com/brk3/ec38433a06f1efa3b92f13:42
pbourkenihilifer: you've tried s/https/http/ in that I take it?13:43
nihiliferyes, and after changint https to http in patchset 10, the gate at least didn't have problem with installing rdo-repo package13:44
nihiliferchanging*13:44
*** prad has joined #kolla13:44
pbourkenihilifer: you only changed rdo in ps 10 though. not the epel.repo13:45
sdake06:44:23]  <sdake>is there some wierd outage with epel7 mirrorlist server13:45
sdake[06:45:21]  <puiterwijk>sdake: we are having network issues.13:45
sdake[06:45:27]  <puiterwijk>this should not impact mirrorlist, but for some reason it does13:45
sdakestraight from #fedora-admin13:45
nihiliferpbourke: yes, I changes epel.repo in the newest patchset13:46
nihiliferchanged*13:46
pbourkeI guess we have our answer then13:46
*** nihilifer has quit IRC13:48
openstackgerritPaul Bourke proposed stackforge/kolla: DO NOT MERGE: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338913:51
sdakewell the gate is nonvoting now or should be in a short period13:52
*** dims is now known as dimsum__13:57
openstackgerritPaul Bourke proposed stackforge/kolla: DO NOT MERGE: Attempt at fixing yum failing on epel repo  https://review.openstack.org/19338913:57
inc0__sooo...guys14:00
sdake06:52:49]  <nirik>it should be a good deal better now...14:00
sdake[06:53:12]  <nirik>we isolated a network issue, but still have a db load issue.14:00
sdake[06:53:23]  <nirik>but the network thing is likely what was hitting mirrorlists.14:00
sdake[06:54:08] kilted1 (~scotts@42-10-255-199.rev.celito.net) joined the channel14:00
sdake[06:56:50]  <sdake>nirik hey hows it going:)14:00
sdake[06:56:54] Gibba (gwhite@nat/ibm/x-kvgdwhflkukgzmju) left IRC (Ping timeout: 245 seconds)14:00
sdake[06:57:00]  <nirik>just lovely.14:00
sdake[06:57:02]  <sdake>nirik I wanted to explain our use case a bit14:00
inc0__whats up with gates?14:00
sdake[06:57:23]  <nirik>well, I am trying to fix this stuff, so I can't really pay much attention.14:00
sdake[06:57:34]  <sdake>nirik we build about 40 containers on every commit to our repo pre-commit14:00
sdake[06:57:35]  <nirik>but can you retry whatever used mirrorlists? it should be fixed14:00
sdake[06:57:51]  <sdake>this validates the commit works14:00
sdake[06:58:08]  <sdake>this ends up doing about 100+ mirrorlist access operations14:00
sdake[06:58:16] plarsen (~plarsen@redhat/jboss/pdpc.professional.plarsen) joined the channel14:00
sdake[06:58:43]  <sdake>nirik ack - just read when you get a chance :)14:00
inc0__soo..they broke their mirrors right?14:00
*** inc0__ is now known as inc014:03
sdakeinc0__ it is unclear who broke what14:04
sdakebut the fact is something was broken and they have fixed half the problem14:04
inc0so, do we need to change anything locally?14:05
sdakewe need our build script to retry builds14:06
sdakeit needs to be intelligent - only retrying if yum was the cause of the failure14:08
sdakeand only optionally retrying based upon cmd line paramaeter14:08
sdakehttps://bugs.launchpad.net/kolla/+bug/146667714:09
openstackLaunchpad bug 1466677 in kolla "build script errors out in yum" [Critical,In progress] - Assigned to Martin André (mandre)14:09
sdakethe gate was changed to non-voting as well (I reverted jpeeler's change)14:10
sdakewe will make voting again once the above bug/feature is implemented14:10
sdakeso the gate should be unblocked shortly wrequiring rechecks14:10
openstackgerritMichal Stachowski proposed stackforge/kolla: Galera container  https://review.openstack.org/18722514:12
sdakejust use recheck operation - once the gate patch has been merged in gerrit14:12
sdaketo recheck, in the review log leave the comment "recheck"14:12
inc0also, can I ask you sdake and rhallisey to review keepalived, merge failed due to conflict and had to rabase14:13
sdakeinc0 got a link14:13
inc0I've never lost 2*+2+W this way;(14:13
sdakeya a rebase loses the acks14:14
sdakeannoying property of gerrit14:14
inc0https://review.openstack.org/#/c/187981/14:14
mstachowis SamYaple here?14:14
SamYaplemstachow: yea14:15
mstachowCould You review my new patchset, please?14:15
SamYaplewhich one?14:15
mstachowIt's lucky 13: https://review.openstack.org/#/c/187225/14:16
SamYapleok14:16
inc0we'll need to figure out good way for init containers tho14:22
mstachowwow, tahnks Sam :O14:23
*** nihilifer has joined #kolla14:25
*** nihilifer has quit IRC14:26
*** nihilifer has joined #kolla14:27
*** nihilifer has quit IRC14:31
*** nihilifer has joined #kolla14:32
sdakeas you have no doubt guessed, I am morpheus !14:36
inc0sdake, you should sleep more to achieve this title14:40
inc0or rather, sleep at all14:40
sdakei got a good 9 hours yesterday14:40
sdakeit was nice ;)14:40
nihiliferI see that build of image in patchset 13 worked http://logs.openstack.org/89/193389/13/check/check-kolla-functional-build-f21/f508375/console.html14:44
nihiliferat least before aborting ;)14:45
openstackgerritMerged stackforge/kolla: Galera container  https://review.openstack.org/18722514:47
sdakebtw guys14:47
sdakeour gate is unjammed14:47
sdakethe voting functional gate has been made nonvoting14:48
sdakeso if folks want to pull the trigger ona recheck or rebase, we can get em merged14:48
openstackgerritMerged stackforge/kolla: Cinder container  https://review.openstack.org/17096514:49
*** daneyon has joined #kolla14:52
*** absubram has joined #kolla14:52
*** absubram has quit IRC14:54
*** absubram has joined #kolla14:55
inc0since we have mergefest, another +2 anyone plz?;) https://review.openstack.org/#/c/187981/14:55
inc0allright, have a nice weekend guys14:59
inc0bye14:59
*** daneyon_ has joined #kolla14:59
mstachowhuh It was really lucky 13 :O14:59
*** inc0 has quit IRC15:01
*** daneyon has quit IRC15:02
sdakethe designate sink service here https://review.openstack.org/#/c/189393/15:04
sdakeneedsa a erbase plz15:04
jpeelersdake: this isn't funny, but it sort of is: http://status.fedoraproject.org/15:04
*** jasonsb has quit IRC15:04
sdakejpeeler ya they updated that page after i bitched that their status page was green :)15:05
jpeelerOH15:05
sdakeand complained we spent like 6 people over 8 hours jerking around figuring out its not our problem :)15:05
jpeelerthere's this too, https://lists.fedoraproject.org/pipermail/devel/2015-June/211584.html15:05
jpeeleryeah...15:05
sdakejpeeler the nonvoting patch went in15:06
sdakewe can revert it after mandre sorts out a retry operation15:06
jpeeleryeah i saw15:06
sdakeI think we need retry anyway15:06
jpeeleragreed15:06
sdakethat would have got our gate rolling een with the disruptions15:06
sdakei tlaked tohte fedora-admin guys, they said our use case (800 mirrorlist hits in short period) is a perfectly reasonable use case15:06
sdakeand explained how we gate on their binaries15:07
sdakeso they are not going to block us15:07
sdakeand said hwe are not the cause of the problem15:07
*** mstachow has quit IRC15:07
openstackgerritMichal Rostecki proposed stackforge/kolla: Add designate-sink service  https://review.openstack.org/18939315:09
nihilifersdake: rebased :)15:09
sdakejpeeler this is a rebase: https://review.openstack.org/#/c/189393/ can you ack plz15:09
jpeelerdone15:11
*** daneyon_ has quit IRC15:12
pbourkehate to hassle but might as well get re-ticked on these too https://review.openstack.org/#/q/owner:%22Paul+Bourke%22+status:open,n,z15:12
sdakepbourke use recheck on the ones that are marked -1 in the V field15:17
pbourkeok15:17
jpeeleryeah some of them were done before non-voting occurred15:17
pbourkei had to rebase a few of them15:17
jpeelerit seems a little unproductive to recheck when we know things are broken, but i guess maybe it'll get fixed faster than we think15:18
pbourkethe never ending rebase cycle15:21
*** nihilifer has quit IRC15:22
jpeelernow that the cinder review merged, is bashate good to go?15:22
bmacejust set recheck on mine.15:23
bmacewhich should be the last of bashate issues :)15:23
*** dasm has quit IRC15:25
*** absubram has quit IRC15:26
sdakehmm15:32
sdakerecheck15:32
sdakeI made the gate non-voting15:32
sdakeso a recheck will fix it15:32
sdakewill fix the gerrit -1 vote15:32
sdakeand you are?15:34
sdakeA smith.  Agent smith.15:34
openstackgerritPaul Bourke proposed stackforge/kolla: Symlink all non Dockerfile resources in images  https://review.openstack.org/19011715:37
openstackgerritMerged stackforge/kolla: Fixed indentation issues to meet bashate compliance  https://review.openstack.org/19332015:40
bmacehuzzah!15:43
*** daneyon has joined #kolla15:46
*** jtriley has quit IRC15:47
*** jtriley has joined #kolla15:54
openstackgerritMerged stackforge/kolla: Add designate-sink service  https://review.openstack.org/18939315:54
sdakebashate gate still not passing15:56
bmacei have one that isn't in yet..15:58
bmacein progress15:58
openstackgerritSteven Dake proposed stackforge/kolla: Add base image for oraclelinux  https://review.openstack.org/19101316:01
openstackgerritPaul Bourke proposed stackforge/kolla: Symlink all non Dockerfile resources in images  https://review.openstack.org/19011716:02
sdakepbourke when yu work on patch sets, take care to use an interactive rebae and resubmit all patches16:02
sdakethi will keep the "OUTDATED" from showing up in gerrit16:02
pbourkesdake: ok, thanks16:03
sdakegerrit doesn't know what to do with an outdated dependency16:03
sdakeso it requires a rebase16:03
pbourkehttps://review.openstack.org/#/c/190117/ is the one that is causing a lot of headache16:03
sdakegit review will ask you if you really want  to do that16:03
sdakethe answer is yes :)16:03
pbourkeit needs to be rebased often and outdates everything that depends on it. so if it passes and people want to give it one more once over Id appreciate it16:04
*** cbaesema has quit IRC16:04
sdakepbourke ya but if you use git rebase -i to rebase your patch set in one repo16:04
sdakeand use the edit option16:04
sdakeyou can edit just that patch16:05
sdakeand then git review iwill keep all the updates in place16:05
openstackgerritMerged stackforge/kolla: Fixed indenting issue to meet bashate compliance.  https://review.openstack.org/19331916:05
bmacealright, now bashate should be happy16:05
bmacehrm, new errrors? galera16:06
openstackgerritSteven Dake proposed stackforge/kolla: Test if the bashate gate is passing - DO NOT MERGE  https://review.openstack.org/19361516:06
openstackgerritPaul Bourke proposed stackforge/kolla: Symlink all non Dockerfile resources in images  https://review.openstack.org/19011716:10
openstackgerritPaul Bourke proposed stackforge/kolla: Fix 'unsupported locale setting' errors in init-runonce  https://review.openstack.org/18843216:10
openstackgerritPaul Bourke proposed stackforge/kolla: Minor doc fixups in dev-quickstart.md  https://review.openstack.org/19054316:11
openstackgerritPaul Bourke proposed stackforge/kolla: Add base image for oraclelinux  https://review.openstack.org/19101316:11
openstackgerritPaul Bourke proposed stackforge/kolla: Create keystone user in start.sh if it doesn't exist  https://review.openstack.org/19107116:11
pbourkehttp://vignette2.wikia.nocookie.net/simpsonstappedout/images/c/cc/Scorpio_flamethrower.png16:12
*** pbourke has quit IRC16:13
*** pbourke has joined #kolla16:13
sdakesamyaple or jpeeler or rhallisey can one of you review the rebase https://review.openstack.org/#/c/190117/16:14
openstackgerritBorneMace proposed stackforge/kolla: Fixed non-bashate compliant method signature  https://review.openstack.org/19362116:15
*** diga has joined #kolla16:17
openstackgerritBorneMace proposed stackforge/kolla: Fixed non-bashate compliant file, missing newline at end  https://review.openstack.org/19362216:19
openstackgerritPaul Bourke proposed stackforge/kolla: Fix broken cinder/galera symlinks for fedora & rhel  https://review.openstack.org/19362316:20
*** jtriley has quit IRC16:21
sdakepbourke those should be 4 commits16:22
*** jtriley has joined #kolla16:23
pbourkesdake: kk16:23
pbourkewas thinking maybe they should have been16:23
sdakeya different fixes = different commits16:24
sdakethat way reviewers dont look at i tand go "damn thats big, let me go watch tv instead"16:24
sdakethey may do that with a large patch stream16:24
sdakebut atleast some of the patches get  reviewed :)16:24
sdakemake sense?16:24
sdakeobjective as a patch submitter is to reduce reviewer fatigue wherever possible16:25
pbourkesure thing thanks :)16:25
sdakebrb16:26
*** sdake_ has joined #kolla16:27
openstackgerritPaul Bourke proposed stackforge/kolla: Fix broken cinder symlinks for fedora  https://review.openstack.org/19362516:29
sdake_jpeeler can you review https://review.openstack.org/#/c/190117/ so we can stop the rebase madness plz :)16:30
openstackgerritPaul Bourke proposed stackforge/kolla: Fix broken cinder symlinks for fedora  https://review.openstack.org/19362516:30
jpeelerdone16:31
*** sdake has quit IRC16:31
sdake_thansk :)16:31
sdake_bmace you may have to rebase your bashate changes, not sure :)16:31
sdake_pbourke this blueprint doesn't link  blueprint refactor-base-image-layout16:32
sdake_is there a blueprint for that?16:32
sdake_and if so, have a link16:32
pbourkesdake_: there is, let me check16:33
bmacesdake_: i had re-pulled right before i made those changes.. so i hope not :/16:33
bmacenot that it would be super hard if i do16:33
sdake_well pbourke's change is going to change things a bit16:33
pbourkestrange it seems to have disappeared16:34
bmacewell, stop it pbourke ;)16:34
sdake_pbourke can you make a new blueprint16:35
sdake_it doesn't have to be super detailed16:35
sdake_I have no idea where it went either16:35
sdake_i remember reading it in detail16:35
pbourkehttps://blueprints.launchpad.net/kolla/+spec/refactor-base-image-layout16:35
pbourkeits there16:35
pbourkebut for some reason its not searchable in launchpad16:35
openstackgerritPaul Bourke proposed stackforge/kolla: Fix broken cinder symlinks for rhel  https://review.openstack.org/19362316:37
sdake_well i'm stumped16:37
sdake_maybe its the length of the blueprint name16:37
sdake_its listed here https://blueprints.launchpad.net/kolla/liberty16:38
sdake_it is also listed in our liberty-1 blueprints16:40
sdake_so that wfm16:40
sdake_I don't know why it doesn't link which is irritating16:40
sdake_but oh well :)16:40
*** athomas has quit IRC16:41
pbourkeyeah Im hesitant to create a new one as we lose all the info and context of the first16:42
pbourkeI can do so though if you like16:42
digasdake_: Hi16:42
sdake_sup diga16:42
digaCan we split this work - https://review.openstack.org/#/c/189157/16:42
sdake_pbourke no action needed on the blueprint linking problem16:43
sdake_diga yes Id' like to split it up16:43
digayes16:43
digaBetter we will split this into two BP's16:43
sdake_diga we are working on a prototype keystone ansible implementaiton to set the standard for how this work should be done16:43
sdake_which two would you recommend16:44
sdake_there are 5 actions in the specification16:44
digasorry 5 work iteam16:44
sdake_can you file a blueprint to rename kolla to kolla-compose16:44
sdake_and file a blueprint to implement kolla-ansible16:44
digayes16:44
sdake_those are two that pop out immediatley16:45
digayes16:45
sdake_i'll approve them for liberty216:45
sdake_make suer to link to the spec the work item it represents16:45
digayes, sound good to me16:45
digalet me file those BP right way16:45
pbourkeSamYaple I was looking at your ip_wrapper change but got distracted with the gate problems. Will have have another look Monday16:46
sdake_after you finish with that we can decompoe the rest of them16:46
pbourkehave a nice weekend all :)16:46
sdake_enjoy pbourke16:46
*** jasonsb has joined #kolla16:49
*** sdake has joined #kolla16:52
*** sdake_ has quit IRC16:55
digasdake: take a look at this - https://blueprints.launchpad.net/kolla/+spec/implement-kolla-ansible-utility-for-multinode16:56
digasdake: you can make change if you want16:57
sdakediga i shortened name a bit16:57
sdakehttps://blueprints.launchpad.net/kolla/+spec/kolla-ansible-script16:58
*** bradjones has quit IRC16:58
digayes16:58
sdakewe need a rename-kolla-script blueprint as well16:58
diga:)16:58
digaokay16:58
sdakeit is more then renaming i t is changign the docs and a bunch of stuff16:58
*** bradjones has joined #kolla16:58
*** bradjones has joined #kolla16:58
sdakethe heat etmplate16:58
sdakeetc16:58
openstackgerritMerged stackforge/kolla: Symlink all non Dockerfile resources in images  https://review.openstack.org/19011716:59
openstackgerritMerged stackforge/kolla: Create keystone user in start.sh if it doesn't exist  https://review.openstack.org/19107116:59
sdakehot16:59
sdakejpeeler can you take a quick look at this - its a 1 liner https://review.openstack.org/#/c/193622/17:00
sdakeand https://review.openstack.org/#/c/193621/17:00
sdakealos a one liner17:00
*** mfalatic has joined #kolla17:03
*** sdake_ has joined #kolla17:05
digaokay17:06
*** sdake has quit IRC17:09
*** bradjones has quit IRC17:11
*** bradjones has joined #kolla17:12
*** bradjones has joined #kolla17:12
openstackgerritRyan Hallisey proposed stackforge/kolla: Add cinder into the docker compose setup  https://review.openstack.org/17350717:15
*** nihilifer has joined #kolla17:16
sdake_jpeeler yet another rebase https://review.openstack.org/#/c/173507/17:16
openstackgerritRyan Hallisey proposed stackforge/kolla: Access volume groups created by Cinder from the host  https://review.openstack.org/18731317:17
sdake_jpeeler yet another rebase https://review.openstack.org/#/c/187313/17:18
sdake_rhallisey can you ack this bashate bug fix https://review.openstack.org/#/c/193622/17:18
sdake_rhallisey and htis other 1 liner https://review.openstack.org/#/c/193621/17:19
rhalliseynice once we get all this stuff merged I think we're lookin good17:19
sdake_one would hope so17:21
sdake_bmace if your up for an adventure you can try to make a nonvoting bashate gate17:21
*** inc0 has joined #kolla17:22
sdake_yo inc017:22
inc0damn, you've broken me guys, it's been a while since I've logged to irc after work;)17:23
inc0not to mention gerrit17:23
bmacebleeeeh, i have to rebase my bashate changes.17:24
inc0sdake_, do we have concrede dates for kollapalooza?17:28
sdake_told ya so bmace17:28
sdake_inc0 waiting on facilities17:28
sdake_I proposed 28th and 29th of June17:28
sdake_but dont book yet, waiting on ack17:28
*** rhallisey has quit IRC17:29
*** jasonsb has quit IRC17:31
inc0btw do you have problems with building our base images on ubuntu?17:32
inc0selinux doesn't install for me:/17:32
*** rhallisey_ has joined #kolla17:32
inc0systemd**17:32
*** dwalsh has quit IRC17:37
*** vinkman has joined #kolla17:37
openstackgerritBorneMace proposed stackforge/kolla: Fixed non-bashate compliant file, missing newline at end  https://review.openstack.org/19362217:44
sdake_rather not june july17:46
sdake_inc0 if you do the work knock yourself out :)17:46
sdake_inc0 I proposed July 28th, July 29th to facilities, have to see if I can lock that up - but I'll let you know via ml as soon as I know17:47
*** loth has joined #kolla17:48
inc0cool, as long as its after 17th of July, I should be able to come17:48
inc0never been in USA, so that'll be a first;)17:48
sdake_inc0 i'm going to be fighting hard for 28th/29th17:48
bmaceugh.. more bashate issues checked in, now mariadb.. fix inc.17:49
*** rhallisey_ is now known as rhallisey17:49
openstackgerritBorneMace proposed stackforge/kolla: Fixed non-bashate compliant method signature  https://review.openstack.org/19362117:50
sdake_rhallisey or jpeeler can you ack that one https://review.openstack.org/#/c/193621/17:51
jpeelerdone!17:52
sdake_ta17:52
openstackgerritBorneMace proposed stackforge/kolla: Fixed non-bashate compliance indentation in mariadb script  https://review.openstack.org/19365917:53
bmaceunless someone sneaks another one in, that should be it for bashate :)17:54
jpeelerok these types of commits can be in one together17:54
sdake_jpeeler one more for review :)17:54
bmacesdake_ specifically asked me to keep them separate17:54
sdake_jpeeler I prefer separat echanges per item17:54
sdake_i mean i coudl have jammed my 18 bahate commits into one17:54
jpeelersdake_: they are related though17:54
openstackgerritMerged stackforge/kolla: Fixed non-bashate compliant file, missing newline at end  https://review.openstack.org/19362217:54
sdake_but would anyone have reviweed it :)17:54
sdake_unlikley17:54
openstackgerritMerged stackforge/kolla: Add cinder into the docker compose setup  https://review.openstack.org/17350717:55
jpeelerwell yeah, i mean if you have 20+ files17:55
sdake_he had 4 files17:55
sdake_i guess it doesn't matter17:55
jpeelermore clicking with more reviews! if the file count isn't that high and the changes are related, i say keep it in one commit17:56
bmacei don't really care one way or the other.  there could be a sort of "max files / review" guideline for this sort of thing, though files isn't terribly reflective also, if there are a ton of changes per file, for example.. max LOC17:56
sdake_i think we want to encourage smaller commits that are self contained to prevent reviewer fatigue17:56
sdake_but ymmv :)17:56
sdake_just trying to teach best practices jpeeler :)17:56
sdake_but I guess judgement matters17:56
jpeeleri mean, there's viewer fatigue in multiple reviews each containing one space change as well17:57
*** jasonsb has joined #kolla17:58
jpeelerbut perhaps not as much as 200 file review, so maybe it isn't worth hammering. but i just don't stay quiet heh17:58
sdake_bmace if you are in for an adventure, you might take a stab at making the bashate gate voting17:58
sdake_jpeeler its aok to make noies - I do it all the time ;)17:58
sdake_hoise17:58
sdake_nosie17:58
sdake_after I learn how to type17:58
sdake_NOISE17:58
jpeelerbmace: would you like to know where to look, or would that take away the fun17:59
bmacehrm, i have some guessts, based on all the gate chaos going on lately i saw some changes made in that area, especially to make the functional one non-voting18:00
sdake_one note, the infra team doen't require bugs or blueprints attached to their changes18:00
sdake_so don't create more work for them by filing one :)18:00
jpeelerok, it's a two line change (3 including removal of newline)18:00
*** prad_ has joined #kolla18:01
bmaceseems like that stuff is in a separate code repo, which makes sense :)18:01
sdake_openstack-infra/project-config18:02
sdake_shower time18:04
sdake_bbl18:04
harmwjpeeler: I can agree with the 200files patchset18:09
harmw+1 for keeping functional bits and pieces together18:10
nihiliferinc0: ubuntu has some bug with aufs18:12
nihiliferand because of that some yum packages (i.e. systemd) cannot be installed18:12
nihilifergive me a moment, I will search a link with solution :P18:13
nihiliferwhich worked for me18:13
inc0thank you nihilifer18:16
sdake_nhi* I think inc0 wants to use ubuntu as a container os18:17
sdake_which is unimplemented18:17
sdake_lunch time bbl18:17
inc0actually sdake_ no, as host:)18:17
sdake_that used to work - i tested it18:18
sdake_this is why pam+noauth is in the container image18:18
nihiliferinc0: http://logs.openstack.org/89/193389/13/check/check-kolla-functional-build-f21/f508375/console.html18:19
sdake_ok gotta jet18:19
nihiliferfuck, not this link18:19
bmacei think the changes are ready for the voting bashate but i should probably wait until the last of my fixes actually go in.  they are in-flight right now18:19
nihiliferinc0: https://github.com/docker/docker/issues/6980#issuecomment-6272347018:19
inc0o, thank you nihilifer18:20
*** smeisner has quit IRC18:23
*** smeisner has joined #kolla18:35
*** jasonsb_ has joined #kolla18:37
*** jasonsb has quit IRC18:37
*** nihilifer has quit IRC18:51
harmwgod I just love those big reviews18:59
harmwthanks SamYaple :p18:59
openstackgerritMerged stackforge/kolla: Fixed non-bashate compliance indentation in mariadb script  https://review.openstack.org/19365918:59
openstackgerritMerged stackforge/kolla: Fixed non-bashate compliant method signature  https://review.openstack.org/19362119:00
*** sdake_ has quit IRC19:19
*** bradjones has quit IRC19:20
*** daneyon has quit IRC19:22
*** bradjones has joined #kolla19:22
*** bradjones has joined #kolla19:22
jpeelerdocker-compose 1.3.0 was released yesterday, need to update all those git installs to use pip19:36
*** gfidente has quit IRC19:42
*** dwalsh has joined #kolla19:45
*** inc0 has quit IRC20:09
*** sdake has joined #kolla20:16
*** sdake_ has joined #kolla20:17
sdake_yo20:17
bmaceyoyo20:19
bmacejust need a workflow on the bashate review to get voting in action20:20
*** sdake has quit IRC20:21
jpeelerdoes anybody have a working kolla deploy handy? i'm curious if "docker ps" shows the mariadb-data container or not20:22
jpeelerthe note in the dockerfile makes it seem like it may exit20:22
sdake_jpeeler only ps -a will show that20:22
jpeelerso the data container starts and then exits?20:22
sdake_roger20:23
jpeelergood to know20:24
*** prad_ has quit IRC20:26
*** daneyon has joined #kolla20:43
*** daneyon has quit IRC20:48
*** rhallisey has quit IRC20:50
*** britthouser has quit IRC20:50
sdake_jpeeler the reason the data container starts is simply to create teh mount blobs in the filesystem21:10
sdake_that is why all the data contains execute bin/true (which on conclusion exit)21:10
jpeelerthe mariadb one does /bin/bash21:11
jpeelerso if it was an invalid command there, there mounts would fail?21:12
sdake_mariadb should be changed to /bin/true21:12
sdake_i am not sure on that point jpeeler21:13
jpeelerthat's the only data container right?21:13
sdake_tehre are sevearl21:14
*** daneyon has joined #kolla21:14
jpeelerah i found the nova-compute one21:14
*** jasonsb_ has quit IRC21:18
sdake_jpeeler this needs a rebase https://review.openstack.org/#/c/189974/21:19
*** bradjones has quit IRC21:20
*** bradjones has joined #kolla21:22
*** bradjones has joined #kolla21:22
openstackgerritJeff Peeler proposed stackforge/kolla: Fix Heat container env vars and dependencies  https://review.openstack.org/18997421:34
*** jtriley has quit IRC21:41
*** dwalsh has quit IRC21:46
*** daneyon has quit IRC21:57
*** jasonsb has joined #kolla22:11
*** dimsum__ has quit IRC22:19
*** dimsum__ has joined #kolla22:20
*** sdake_ has quit IRC22:25
*** dimsum__ has quit IRC22:32
*** mfalatic has quit IRC23:03
*** jasonsb has quit IRC23:06
*** juggler has quit IRC23:06
*** juggler has joined #kolla23:06
vinkmanSo, why is it not allowed to run the build script as root? I know in general it is bad form to run anything as root but docker is a pain running non root...23:13
*** juggler has quit IRC23:16
*** juggler has joined #kolla23:17

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!