15:06:28 <anteaya> #startmeeting third-party
15:06:29 <openstack> Meeting started Mon Jul 13 15:06:28 2015 UTC and is due to finish in 60 minutes.  The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:06:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:06:33 <openstack> The meeting name has been set to 'third_party'
15:06:37 <anteaya> sorry I'm late
15:06:42 <anteaya> I got distracted
15:06:47 <anteaya> how is everyone?
15:06:51 <rhe00_> hi
15:07:11 <mmedvede> hello
15:07:28 <anteaya> hi
15:07:40 <rhe00_> trying to make nodepool be my friend again
15:07:48 <anteaya> anyone who was at the spring last week want to give a report?
15:07:53 <rhe00_> not going so well right now. :(
15:07:57 <anteaya> mmedvede: you attended, want to share your thoughts?
15:08:03 <anteaya> rhe00_: we can get to taht next
15:08:23 <ociuhandu> hello all
15:08:34 <mmedvede> anteaya: yes, I did get a few patches submitted. The sprint was good
15:08:40 <mmedvede> a lot of participation
15:08:48 <krtaylor> o/
15:08:55 <anteaya> mmedvede: was that the first virtual sprint you attended?
15:09:10 <mmedvede> anteaya: the first one was puppet module-split
15:09:25 <anteaya> mmedvede: ah
15:09:41 <anteaya> any thoughts on how they were different or similar?
15:10:26 <mmedvede> the module-split was more 'mechanical', just following the template. This one is harder, as everything needs to be tested and has more chances to affect Infra
15:11:03 <anteaya> I agree
15:11:08 <anteaya> thanks for the summary
15:11:16 <anteaya> anyone with any other thoughts about the sprint?
15:11:52 <anteaya> well thanks to all those that attended
15:12:05 <anteaya> I don't have the exact stats but it was a great success
15:12:21 <anteaya> thanks asselin for putting it together and running it
15:12:35 <anteaya> rhe00_: you are having some nodepool issues currently?
15:12:55 <anteaya> rhe00_: have you a paste or log so we have something to look at and create a shared understanding?
15:13:29 <rhe00_> yes, however unfortunately nodepool has been very sparse with information about what goes wrong.
15:13:55 <rhe00_> first it would spin up new instances after completed tests. I attributed that to timeouts reading the database
15:14:12 <rhe00_> so I dropped the nodepool database and now it won't finish building images
15:14:23 <rhe00_> it just stops without any messages as to why
15:14:42 <rhe00_> so unfortunately I have no logs to share at the moment
15:14:52 <anteaya> dropped the nodepool database?
15:15:02 <rhe00_> yes
15:15:09 <anteaya> I would think a database would be sort of important
15:15:46 <rhe00_> someone in the infra group suggested that to force nodepool to rebuild it's view of the world from scratch
15:16:07 <anteaya> ah
15:16:15 <anteaya> so recreate the database
15:16:29 <rhe00_> yes, nodepool has recreated the database again
15:16:58 <anteaya> what verision or patchset of nodepool are you running?
15:17:26 <rhe00_> 0.0.1
15:17:51 <rhe00_> it worked great for months and then quit on me
15:18:45 <anteaya> quit on me
15:18:46 <rhe00_> I have cleaned up the debug prints a bit in hopes to catch why it doesn't finish building the images
15:18:47 <mmedvede> rhe00_: you should also try to manually update the image, with 'nodepool image-update', this way you would see what goes wrong if your update does not work
15:18:59 <anteaya> how did it quit on you
15:19:35 <rhe00_> anteaya: that's when it stopped spinning up new nodes
15:20:00 <anteaya> did you restart it at all?
15:20:01 <rhe00_> the old nodes where stuck in deleting state
15:20:04 <rhe00_> were
15:20:09 <rhe00_> yes, many times
15:20:43 <mmedvede> rhe00_: which cloud do you use for your VMs?
15:20:47 <rhe00_> mmedvede: I have never been successful running nodepool image-update
15:21:11 <rhe00_> it errors out. from what I recall it complains about permissions
15:21:16 <anteaya> there have been two more nodepool releases since 0.0.1 http://git.openstack.org/cgit/openstack-infra/nodepool/refs/
15:21:26 <anteaya> have you considered upgrading to 0.1.1 perhaps?
15:21:31 <rhe00_> mmedvede: I use a packstack Juno
15:22:47 <rhe00_> anteaya: I will try upgrading next. Everything was working with what I had so I did not want to touch it. Didn't think of looking for a newer version. Thanks for the idea!
15:23:04 <rhe00_> anyway. I don't have to take up your time as I have no logs to share.
15:23:24 <mmedvede> rhe00_: first thing I would do is to try manually booting the VM on your cloud, and taking a snapshot of it. If it works, then most likely nodepool image-update fails running the update scripts.
15:23:31 <anteaya> rhe00_: yes tough to figure out what is happening under the hood with no logs
15:24:38 <mmedvede> rhe00_: I assume you are using prepare_node.sh for update, and not disk-image-builder though. I have no experience with latter.
15:25:04 <rhe00_> mmedvede: I use the prepare_node.sh. No disk-image.
15:25:40 <rhe00_> I'll wait for the current image build to stop building and then upgrade if I don't see anything in the logs.
15:26:26 <rhe00_> Thanks for the feedback. Feel free to move on.
15:26:27 <anteaya> do you feel you have a way forward for the moment?
15:26:33 <anteaya> okay thanks moving on
15:26:43 <anteaya> anyone have anything else they would like to discuss today?
15:26:45 <rhe00_> I will wait for this build to either complete or silently stop
15:26:57 <anteaya> rhe00_: hopefully it will complete
15:27:01 <mmedvede> rhe00_: ping me in IRC later if you still have problems. I have to deal with nodepool often
15:27:27 <rhe00_> mmedvede: Thanks! I will if I get stuck.
15:28:19 <anteaya> if folks have time to look at the puppet-openstackci repo
15:28:36 <anteaya> asselin is working on a patch for a config for a simple common ci setup
15:28:40 <anteaya> #link https://review.openstack.org/#/c/200330/
15:28:57 <anteaya> it is WIP but it would be nice for folks to take a look at it
15:29:07 <janonymous_> o/
15:29:10 <mmedvede> I have one question myself about zuul logging actually
15:29:12 <anteaya> perhaps test it out and see what kind of results they get with it
15:29:19 <anteaya> mmedvede: go ahead
15:29:40 <mmedvede> I have started using common-ci and puppet-zuul logging.conf
15:29:49 <mmedvede> and my logs grew to 80G overnight
15:30:09 <mmedvede> Does Infra regularly purging logs on their zuul server?
15:30:27 <mmedvede> Or does it look like a glitch on my setup to you?
15:30:44 <anteaya> mmedvede: that's not so good
15:31:02 <anteaya> we dont' tend to purge logs anywhere
15:31:28 <anteaya> fungi: any thoughts on zuul log growth of 80G overnight?
15:31:41 <mmedvede> I know that offending part is gerrit logger, so I did disable it. So I wonder if puppet-zuul logging.conf needs to be patched.
15:31:44 <anteaya> what is the content of the logs?
15:32:17 <anteaya> that is good feedback yes, we don't want logs to explode for folks using our tools
15:33:07 <fungi> would be good to know what logging levels you have set
15:33:13 <mmedvede> anteaya: when I checked, it was mostly filled with gerrit patch information (all comments, etc)
15:33:41 <fungi> we run ours with fairly detailed debug logging enabled, but we also have a lot of room for /var/log to accomodate it
15:33:46 <fungi> er, accommodate
15:34:32 <mmedvede> fungi: I have used the current logging.conf in puppet-zuul. Is there any other config that affects zuul logging?
15:34:41 <fungi> but 80g is way more than what we end up with
15:35:16 <mmedvede> fungi: ok, this is good to hear. This hints on something being wrong for our instance
15:35:55 <fungi> mmedvede: that's where you adjust the log levels, yes
15:36:23 <fungi> mmedvede: for example gear is set to warning and higher right now. if you switch it to debug then expect your logs to explode
15:36:55 <fungi> mmedvede: anyway, if you analyze your logs to find out the bulk of what's in them, that will likely tell you what's causing them to be so large suddenly
15:38:30 <anteaya> fungi: thank you
15:38:43 <mmedvede> fungi: ok. I would look further. I was afraid to open a 20G log file :)
15:39:03 <mmedvede> fungi: thank you
15:39:48 <anteaya> any more thoughts on this topic?
15:40:12 <fungi> mmedvede: you could always do something like `tail -n1000 /var/log/zuul/zuul.log|less`
15:40:30 <fungi> so that you're only loading the last 1000 lines into the pager
15:41:56 <mmedvede> fungi: right. That is the good way to chekc
15:42:15 <anteaya> anything more here?
15:42:42 <anteaya> does anyone have anything else they would like to discuss?
15:43:15 <anteaya> does anyone have a good reason why I should keep the meeting open?
15:43:50 <anteaya> thanks all for your kind attendance and participation
15:44:02 <anteaya> enjoy the rest of the local time of day/night for you
15:44:06 <anteaya> see you next week
15:44:11 <anteaya> #endmeeting