15:02:28 <anteaya> #startmeeting third-party
15:02:29 <openstack> Meeting started Mon Jun 15 15:02:28 2015 UTC and is due to finish in 60 minutes.  The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:33 <openstack> The meeting name has been set to 'third_party'
15:02:37 <anteaya> hello
15:02:45 <akerr> hello
15:02:45 <patrickeast> hey
15:02:55 <anteaya> akerr patrickeast how are you?
15:02:56 <Alexey_Nexenta> Hi
15:03:02 <IlyaG> Hi guys, I am looking for hardware requirements/references for CI infrastructure - can someone point me into a right direction?
15:03:03 <marcusvrn> Hi
15:03:04 <anteaya> Alexey_Nexenta: hello
15:03:12 <anteaya> IlyaG: I'm not a guy
15:03:28 <anteaya> what shall we discuss today?
15:03:53 <eantyshev> hello
15:04:25 <anteaya> #link https://wiki.openstack.org/wiki/VirtualSprints#OpenStack_Common-CI_Solution
15:04:33 <wznoinsk> hi
15:04:37 <anteaya> asselin_'s sprint is coming up
15:04:46 <anteaya> please consider attending if you can
15:05:03 <anteaya> we will need people to conduct patch reviews and tests
15:05:11 <anteaya> so everyone can be helpful
15:05:28 <anteaya> wznoinsk: do you want to go into any more detail about your experience
15:05:33 <anteaya> thanks for posting ot the mailing list
15:06:12 <anteaya> #link http://lists.openstack.org/pipermail/openstack-infra/2015-June/002851.html
15:06:15 <mmedvede> hi
15:06:17 <wznoinsk> anteaya: no problem
15:06:27 <anteaya> wznoinsk: care to expand a bit?
15:06:29 <krtaylor> o/
15:06:31 <anteaya> what is pygerrit?
15:06:42 <anteaya> is that the name of your script you wrote?
15:07:08 <wznoinsk> anteaya: I wasn't able to drill down exactly what piece of the operation is causing the problem or how to reproduce it outside of a python script
15:07:35 <wznoinsk> pygerrit is python lib for gerrit: https://pypi.python.org/pypi/pygerrit/0.2.2
15:07:40 <anteaya> ah
15:07:47 <anteaya> is anyone else using pygerrit?
15:08:27 <anteaya> I'm not advocating for it, just curious
15:08:47 <anteaya> wznoinsk: it seems to be staying away from Control-Z is your recommended way forward?
15:09:43 <wznoinsk> it didn't affect my 'ssh ... gerrit stream-event' on the command line when I did ctrl+z so my python/pygerrit is one situation you want to avoid ctrl+z with... rather than avoid ctrl+z at all
15:10:08 <anteaya> ah okay thanks for that clarification
15:10:24 <anteaya> wznoinsk: any other thoughts worth sharing at this time?
15:11:28 <wznoinsk> I got this link from someone on -infra, can't remember who it was tho: https://gerrit-documentation.storage.googleapis.com/ReleaseNotes/ReleaseNotes-2.9.2.html
15:11:36 <wznoinsk> it seems like this is fixed in gerrit 2.9.2
15:12:04 <wznoinsk> anteaya: nope, that was the most interesting one
15:12:25 <anteaya> wonderful
15:12:35 <anteaya> thank you for helping us with this issue
15:13:40 <anteaya> if anyone is able to help review any puppet-openstackci patches this week
15:13:45 <anteaya> there are two right now
15:13:49 <anteaya> #link https://review.openstack.org/#/q/status:open+project:openstack-infra/puppet-openstackci+branch:master+topic:downstream-puppet,n,z
15:14:00 <anteaya> it would be great if you are able to take a look at them
15:14:23 <wznoinsk> on slightly other note, I think it would be beneficial for CI operators (especially the ones that have their accounts disabled) to attend these meetings, it's a opportunity to get the issues solved quicker than on ML I think, how do we attract them? ;-)
15:14:33 <anteaya> IlyaG: you had some questions
15:15:03 <anteaya> IlyaG: did this person leave?
15:15:28 <patrickeast> looks like it
15:15:33 <anteaya> wznoinsk: a very good point
15:15:41 <anteaya> wznoinsk: I'm open to suggestions
15:15:58 <anteaya> as one of the big things I need when re-enabling a system is trust
15:16:10 <anteaya> and open communication goes a long way to establishing that
15:16:31 <wznoinsk> it needs promoting this irc meeting during any disabled/naughty account discussino on ml
15:16:37 <anteaya> since if someone takes out our system, I have to trust they will pay more attention next time
15:16:51 <anteaya> wznoinsk: I'm open to that
15:17:03 <anteaya> I don't do it myself as it may appear to be self-serving
15:17:12 <anteaya> however I do not discourage others from doing so
15:17:35 <anteaya> I do link to common third party resources when I have the energy
15:17:54 <anteaya> but I do admit that sometimes I lag behind in linking all the things
15:18:12 <anteaya> I welcome help and support in providing links to all the things
15:18:30 <krtaylor> wznoinsk, system operators have to want to get involved, they just can't miss all the irc/email traffic on it if they are paying attention
15:19:23 <akerr> can the ML not inject a link into the footer of every message on a list-by-list basis?
15:19:30 <Alexey_Nexenta> anteaya: Ilya wanted to ask if you have any recommendation/requirements for CI hardware ?
15:19:43 <anteaya> akerr: I don't know I haven't investigated
15:19:44 <Alexey_Nexenta> I would forward it to him
15:19:54 <krtaylor> akerr, we use openstack-dev, it would have no way of knowing
15:19:56 <anteaya> akerr: I find footers tend to be ignored
15:20:02 <krtaylor> ++
15:20:18 <anteaya> Alexey_Nexenta: ah okay thanks, are you working on the same ci system?
15:20:39 <Alexey_Nexenta> yes, working on Nexenta CI
15:20:43 <akerr> well somehow the third-part-announce emails have a very specific link in their footer, so it's able to figure out
15:20:45 <anteaya> Alexey_Nexenta: we don't have a list, it depends entirely on what you are planning on testing
15:21:41 <anteaya> akerr: the footer is the third party announce mailing list information
15:21:56 <anteaya> some of the conversations happen on -dev and -infra too
15:22:13 <anteaya> if folks want a busier footer for third party announce, I can look into it
15:22:34 <akerr> anteaya: true, but we when an account is disabled it goes through the announce email.  Why not just also advertise this meeting there
15:22:40 <patrickeast> Alexey_Nexenta: you would be testing cinder, right?
15:22:49 <anteaya> akerr: I can look into it
15:22:52 <krtaylor> hm, for announce, it might make sense, ++
15:22:56 <Alexey_Nexenta> patrickeast: yes, cinder drivers
15:23:20 <anteaya> #action anteaya to look into adding link to third party meetings in -announce ml footer
15:23:30 <Alexey_Nexenta> Our main problem is that we keep getting timeout errors
15:23:34 <patrickeast> Alexey_Nexenta: ok cool, same here, for fwiw i’ve found that any less than ~3 build slaves able to run tests and our system gets too far behind
15:23:55 <patrickeast> Alexey_Nexenta: with a single jenkins slave it ends up ~12-24 hours behind on build very quickly
15:24:10 <patrickeast> with 3 it can stay only maybe 2-3 behind (with 35min builds)
15:24:37 <patrickeast> so any hardware that can acommodate that amount of devstack vm’s and the controllers should work
15:24:38 <ameade> Alexey_Nexenta: A good optimization is to only run on patches that Jenkins has +1'd
15:25:06 <Alexey_Nexenta> ameade: yes, I think we only run those
15:26:33 <anteaya> Alexey_Nexenta: what is your current setup?
15:27:06 <hodos|2> anteaya: we use john griffiths sos-ci
15:27:23 <anteaya> hodos|2: how many other team members do you have in this meeting?
15:27:36 <hodos|2> anteaya: 3 threads, fuel over vsphere
15:27:41 <Alexey_Nexenta> anteaya: just us two
15:27:45 <anteaya> okay thanks
15:28:06 <anteaya> hodos|2: are you using zuul and jenkins then?
15:28:17 * anteaya is not familiar with jgriffith's setup
15:28:33 <hodos|2> anteaya: nope, just python + ansible
15:28:35 <patrickeast> it replaces zuul, gearman, jenkins, all that
15:28:42 <anteaya> oh okay
15:28:45 <anteaya> no idea then
15:28:55 <anteaya> for timeouts I suggest you talk to jgriffith
15:29:06 <anteaya> it isn't an infra supported setup
15:29:18 <hodos|2> anteaya: it seems to do the job, timeouts are happening on the different side
15:29:19 <patrickeast> what kind of timeouts are you guys getting?
15:30:02 <patrickeast> we get ssh timeouts on some of the tempest tests when our system is stressed (still looking into why its happening)
15:30:40 <hodos|2> patrickeast: we probably using slow setup for openstack cloud: we have all-in-one node, but i guess, it's the same: backend or network is too slow
15:31:25 <hodos|2> patrickeast: most of those are happening with compute_with_volumes
15:31:32 <patrickeast> have you adjusted any of the timeout settings? iirc there are like 5 different ones to play with depending on what is timing out
15:31:57 <wznoinsk> it may be too radical but you may look into containers instead of vm's and stack inside
15:32:28 <hodos|2> patrickeast: can you point on how to change those, cause i couldn't find any iinfo
15:32:56 <marcusvrn1> patrickeast: same here (ssh timeouts), boot_pattern intermittently fails  by timeout. If you solve this problem, could you let me know? I'm still working on it too
15:33:16 <akerr> we change a lot of our timeouts to crazy high numbers... let me see if i can find the variables
15:33:53 <patrickeast> hodos|2: so the ones we change are env variables for devstack-gate and jenkins, which unfortunately are not going to be helpful for sos-ci
15:34:08 <patrickeast> hodos|2: but, at some point they should be ending up in the tempest config file
15:34:28 <patrickeast> or i suppose in the sos-ci code if it has any global timeout for test runs like jenkins does
15:35:10 <hodos|2> patrickeast: well, i meant tempest timeouts, but i couldn't find the variable names for local.conf
15:35:26 <patrickeast> marcusvrn1: will do, we have looked into them a bit in the past and didn’t figure it out, but we are going back in to deep dive this sprint
15:35:35 <marcusvrn1> hodos|2: those variables are in tempest.conf
15:36:02 <marcusvrn1> #link https://github.com/openstack/tempest/blob/master/etc/tempest.conf.sample
15:36:31 <akerr> In the [volume] section you can set build_timeout
15:36:36 <akerr> of tempest.conf
15:36:50 <akerr> you also might want to adjust [compute] build_timeout
15:37:00 <hodos|2> marcusvrn1: thanks, i thougt you're able to modify tempest.conf through local.conf with TEMPEST_ vars
15:37:10 <marcusvrn1> hodos|2:
15:37:17 <marcusvrn1> hodos|2: yep, you are able
15:37:26 <patrickeast> hodos|2: you can, just define those variables in a section for your tempest config
15:37:36 <akerr> add [[post-extra|$TEMPEST_CONFIG]] to local.conf to set tempest variables
15:37:53 <marcusvrn1> hodos|2: in the local.conf you have to add something like "[[post-extra|\$TEMPEST_CONFIG]]" and then add which sections and variables you want to add
15:38:32 <hodos|2> akerr marcusvrn1 patrickeast: thanks, will try
15:39:11 <marcusvrn1> patrickeast: thanks
15:39:25 <anteaya> have we reached a happy place?
15:40:00 <anteaya> anything more to say on this topic?
15:40:08 <wznoinsk> one thing to remember when using proxy is to specify no_proxy to have at least 127.0.0.1,ip of management interface when running commands that connect to keystone
15:40:47 <asselin_> o/
15:40:52 <anteaya> hey asselin_
15:41:08 <asselin_> anteaya, hi
15:41:53 <anteaya> Alexey_Nexenta hodos|2 thanks for coming and asking, I hope you have enough to move forward this week
15:42:24 <anteaya> does anyone have anything else they would like to discuss today?
15:42:55 <hodos|2> anteaya: yep, we're hoping to speed up our env, by removing bottlenecks in network, and providing SSDs to backend
15:43:13 <anteaya> hodos|2: okay sounds great
15:43:17 <wznoinsk> anteaya: I focus to get my CIs as good as possible, is not having CIs mentioned on mailing list enough to say they're ok assuming they follow the rules from wiki?
15:43:19 <Alexey_Nexenta> thanks for the help everyone, will try those advices
15:43:48 <anteaya> wznoinsk: we don't have any stamp that says "this is a good ci"
15:44:08 <anteaya> wznoinsk: the biggest thing I look for is if someone has a question is someone around to answer it
15:44:11 <anteaya> or will be soon
15:44:39 <wznoinsk> anteaya: ok, is there anything in works to have something like 'infra approved' ci kind of thing?
15:44:53 <anteaya> wznoinsk: the fact you show up to meetings regularly and help others when you can goes a long way for me to add my voice of confidence when someone has a question about your system
15:45:15 <anteaya> Alexey_Nexenta: welcome, I hope you find a solution
15:45:19 <anteaya> wznoinsk: no
15:45:28 <asselin_> wznoinsk, not infra approved, but close: https://github.com/rasselin/os-ext-testing
15:45:33 <anteaya> wznoinsk: as any system can go sideways at any time
15:45:55 <anteaya> wznoinsk: we have an infra supported workflowl
15:46:11 <krtaylor> wznoinsk, we'll have a CI monitoring dashboard, hopefully soon, that will help with whether "this is a good CI" question
15:46:11 <wznoinsk> anteaya: thanks for the link, that's what I was after
15:46:13 <anteaya> which is what asselin_ is helpfing folks with and is working on improving
15:46:24 <anteaya> wznoinsk: ah sorry I mis-understood
15:46:42 <asselin_> wznoinsk, this is the goal: http://specs.openstack.org/openstack-infra/infra-specs/specs/openstackci.html
15:47:06 <anteaya> wznoinsk: some folks are looking to have their system blessed so they can stop maintaining it, a turn it on a forget it solution
15:47:17 <anteaya> wznoinsk: sorry I mis-understood your question
15:48:01 <anteaya> anything else on this topic?
15:48:24 <anteaya> anyone with anything else to discuss today?
15:49:08 <anteaya> can anyone think of any reason why we shouldn't wrap up the meeting?
15:49:29 <anteaya> okey dokey
15:49:43 <anteaya> thanks everyone for your attendance and participation
15:49:46 <anteaya> have a good week
15:49:51 <anteaya> see you next monday
15:49:54 <anteaya> #endmeeting