00:01:41 <thinrichs> #startmeeting CongressTeamMeeting
00:01:42 <openstack> Meeting started Thu Sep  8 00:01:41 2016 UTC and is due to finish in 60 minutes.  The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot.
00:01:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
00:01:46 <openstack> The meeting name has been set to 'congressteammeeting'
00:02:03 <aimeeu> hello!
00:02:11 <ekcs> hi all
00:02:56 <thinrichs> Hi all
00:03:04 <thinrichs> Here's agenda for this week..
00:03:06 <ramineni_> hi
00:03:23 <thinrichs> 1. Testing for newton release
00:03:38 <thinrichs> 2. First project teams gathering
00:03:52 <thinrichs> 3. Status
00:04:12 <thinrichs> Anything else?
00:04:44 <thinrichs> We'll start with number 2
00:04:50 <thinrichs> #topic First project teams gathering
00:05:18 <thinrichs> Here's Thierry's announcement about how the OpenStack conference and design summit are being split
00:05:21 <thinrichs> #link http://lists.openstack.org/pipermail/openstack-dev/2016-September/102981.html
00:05:36 <thinrichs> Or at least, that's the announcement for the first occurrence of the design summit after the split
00:06:59 <thinrichs> One of the ramifications is that the Ocata session will be slightly shorter than normal.
00:07:18 <thinrichs> The bigger consequence is that there will be 4 OpenStack events per year instead of 2.
00:08:03 <thinrichs> We'll all need to figure out how travel budgets will impact our ability to congregate and coordinate which events we'll be attending.
00:08:21 <thinrichs> Questions, comments?
00:08:23 <aimeeu> Have you heard whether the Project Team Gatherings will be held outside of North America once a year?
00:08:46 <thinrichs> I have no info outside the mailing list.
00:09:01 <thinrichs> So no.  I don't know where they're planning on holding events.
00:09:04 <aimeeu> OK - there's nothing on the official FAQ either.
00:09:22 <thinrichs> I'd guess they'd rotate it like they do today, given how many international contributors there are
00:09:39 <aimeeu> that would be fair
00:10:04 <thinrichs> #topic Testing for newton
00:10:45 <thinrichs> Today I was trying to test HA/HT and realized I don't know exactly how to do it.
00:10:52 <thinrichs> The HA guides are a start.
00:11:11 <thinrichs> Has anyone set up the full HAHT of Congress and done manual testing?
00:11:29 <thinrichs> Multiple PE+API nodes, 1 DSD node, a load-balancer
00:11:45 <thinrichs> running with the other OpenStack components like Nova/Neutron
00:11:54 <ekcs> not yet. still working to finish automated testing.
00:12:22 <ramineni_> not yet
00:12:28 <ekcs> we need to add load-balancer instructions to guide.
00:13:02 <thinrichs> Do we know how to set it up with devstack?
00:13:05 <thinrichs> It'd be nice if we could spin up devstack to get all the other components.
00:13:19 <thinrichs> Then to spin up Congress on 2 other VMs
00:13:53 <thinrichs> Maybe 3 VMs (2 for PE+API and 1 DSD)
00:13:59 <thinrichs> And then 1 VM for a load balancer
00:14:27 <ekcs> should be relatively straightforward.
00:14:49 <ramineni_> devstack problem is we cant start 2 PE+API like we discussed before
00:15:04 <ramineni_> because they start on same node?
00:15:08 <ekcs> just need to start other instances of congress using the same configs, after devstack is done setting up.
00:15:25 <thinrichs> ramineni_: Even if we could start 2 PE+APIs inside devstack, we'd want to test on separate nodes.
00:15:34 <ramineni_> right
00:15:35 <thinrichs> Devstack only runs on 1 node, right?
00:15:55 <aimeeu> devstack can run on multiple nodes
00:15:57 <ramineni_> multinode is there i suppose..but i dont know how that works
00:16:08 <clarkb> you can also run multiple processes on a single instance too
00:16:11 <thinrichs> ekcs: If I remember right, a standalone Congress install isn't trivial.  And we'd need to add rabbitmq
00:16:15 <clarkb> might need to modify devstack to make that happen though
00:16:16 <aimeeu> http://docs.openstack.org/developer/devstack/guides/multinode-lab.html
00:17:16 <thinrichs> aimeeu: thanks for the link
00:17:27 <aimeeu> I haven't yet tried to do that
00:17:29 <thinrichs> clarkb: is that the right link to help us, would you say?
00:17:45 <thinrichs> Does anyone know how other teams test their HA deployments?
00:17:47 <ekcs> thinrichs: right…
00:18:07 <thinrichs> Do they automate the whole thing with puppet/ansible/…?
00:18:12 <clarkb> thats basically how the gate multinode works, we run devstack twice once on each node with different settings for each
00:19:08 <thinrichs> clarkb: naive question: do the 2 devstack nodes communicate correctly over rabbitmq?
00:19:17 <clarkb> thinrichs: yes
00:19:40 <clarkb> its a functionaing two node cloud. One node is controller + compute all in one and the second node is a compute host
00:20:52 <ekcs> hmm that’d be helpful to leverage devstack to set up additional nodes. cuz manual install in complex like thinrichs said. will have to slog through devstack settings I guess to do it correctly.
00:21:22 <ekcs> and add to the plugin piece.
00:21:31 <thinrichs> Seems we'd want 4 devstack nodes then: 1 with everything but congress, 2 with just Congress in PE+API mode, and 1 with just Congress in DSD mode.
00:21:53 <thinrichs> Then we'd just need a LB, and we could (manually) test.
00:22:10 <aimeeu> thinrichs: is this helpful? Manual validation of cinder a/a patches:  http://gorka.eguileor.com/manual-validation-of-cinder-aa-patches/
00:22:14 <thinrichs> That doesn't sound too bad, since we probably all have a devstack VM we can clone a few times
00:23:06 <thinrichs> aimeeu: that looks really helpful—exactly the kind of testing we're trying to do
00:23:34 <thinrichs> I'll take an action item to try to get something working and report back.
00:23:49 <thinrichs> Anyone else want to give it a try, so we don't have 1 point of failure?
00:24:30 <ekcs> I can take a look too, but atm I’m prioritizing
00:24:35 <ekcs> same node diff process testing.
00:24:44 <thinrichs> ekcs: you're busy getting the auto testing in place
00:24:53 <thinrichs> ekcs: right that that's higher priority
00:25:00 <thinrichs> I should be able to find the time to do this
00:25:25 <thinrichs> #action thinrichs will set up multi-node manual testing and report back on how to do it
00:25:48 <thinrichs> Moving on...
00:25:53 <thinrichs> #topic Status
00:26:05 <thinrichs> ramineni_: want to give your status update?
00:26:39 <ramineni_> sure
00:27:23 <ramineni_> added a non-voting multiple PE gate job, so we could test it properly
00:27:50 <ramineni_> and fixed some of the issues we came across the HA testing
00:28:06 <ramineni_> thats it from my side
00:28:19 <thinrichs> Any of those issues during HA testing worrisome?
00:28:56 <ramineni_> still no stable yet ..hopefully we will get it done.. eric is working on that too
00:29:16 <thinrichs> ok
00:29:30 <thinrichs> aimeeu: want to go next?
00:29:35 <aimeeu> ok
00:29:45 <aimeeu> I've been looking into configuring Congress for pydevd remote debugging. Looked at several OpenStack projects - I like the way Trove handles it. Tried implementing it in Congress - didn't work but I am learning quite a bit about how Congress starts up; also looking at historical code changes in github regarding congress installation on devstack. I have
00:29:45 <aimeeu> questions that I'll put into an email later after I've organized my thoughts.
00:30:04 <aimeeu> that's it for me
00:30:49 <thinrichs> aimeeu: thanks.  Let us know if you need any help digging through the startup code
00:30:58 <aimeeu> thanks - will do
00:31:13 <thinrichs> ekcs: want to go next?
00:31:17 <ekcs> sure.
00:31:24 <ekcs> Mostly resolving testing issues with HAHT. Getting the tempest test to stabilize. Still not sure why occassional errors occur with instances not converging.
00:31:27 <ekcs> Hoping that local multi-process testing will shed more light. I have a basic local multi-process test framework setup. #link https://review.openstack.org/#/c/358927/
00:31:28 <patchbot> patch 358927 - congress - WIP - local HA tests
00:32:29 <ekcs> it’s working on my machine, but somehow failing on gerrit. I’m hoping it’s not because of a system config that disallows spawning new processes and http’ing them.
00:33:15 <ekcs> but even if it ultimately doesn’t work in gerrit it’s still useful tests to have in place.
00:33:34 <ekcs> other than that, just identifying scattered issues and bugs as I go.
00:34:23 <clarkb> you cant bind to privileged ports in those jobs but you can use many of the other thousands of ports
00:34:35 <clarkb> running tox should not require root
00:35:07 <thinrichs> clarkb: spinning up new processes is allowed (e.g. with subprocess.Popen)?
00:35:12 <ekcs> clarkb: ah great.
00:35:23 <clarkb> thinrichs: yes
00:35:34 <ekcs> I’m using ports 4001 and 4002.
00:36:46 <thinrichs> We're passing cwd=helper.root_path().  Could that be a problem on a different machine?
00:37:08 <ekcs> thinrichs: shouldn’t be because it’s relative to the helper.py file.
00:37:28 <ekcs> thinrichs: but still worth checking at some point.
00:38:17 <thinrichs> I can do a quick status update
00:38:23 <ekcs> anyway the failure is that I Popen a new congress server, and then http it, but connertion fails exceeding retry limit.
00:40:06 <ekcs> thinrichs: didn’t mean to cut you off.
00:40:15 <thinrichs> Nothing jumps to mind, other than a firewall on the OS.
00:40:20 <thinrichs> ekcs: no problem
00:40:31 <thinrichs> Did some manual testing.
00:41:05 <thinrichs> Reasonably comprehensive within devstack.  Not trying corner-cases, but just hitting all the core functionality.
00:41:21 <thinrichs> Some testing across processes; seems good too.
00:41:39 <thinrichs> Getting a real HAHT env set up is next
00:42:01 <thinrichs> Did notice we're returning some stack-traces as part of the HTTP error message.
00:42:15 <thinrichs> That's some low-hanging fruit I'd say.
00:42:18 <thinrichs> #link https://bugs.launchpad.net/congress/+bug/1620868
00:42:18 <openstack> Launchpad bug 1620868 in congress "Stack trace returned to end-user" [Undecided,New]
00:42:42 <thinrichs> aimeeu: might be a good one to take on
00:43:00 <aimeeu> yep I was just looking at it
00:43:00 <thinrichs> That's it from me.
00:43:14 <thinrichs> aimeeu: cool!
00:43:33 <thinrichs> Just found out I need to leave the meeting early today (in 2 min).
00:43:39 <thinrichs> Anything else for me before I take off?
00:43:42 <thinrichs> #topic open discussion
00:44:08 <thinrichs> I'll try to make ekcs chair before I leave…
00:44:10 <thinrichs> #chair ekcs
00:44:10 <openstack> Current chairs: ekcs thinrichs
00:44:22 <ramineni_> im on vacation next week , wont be able to attend meting
00:44:33 <ekcs> clarkb: any suggestions for how to see if a port may be blocked on a gerrit vm?
00:44:39 <thinrichs> ramineni_: okay.  Thanks.  Enjoy your vacation!
00:44:53 <thinrichs> That reminds me.  I can't make the meeting either.  I'll be traveling.
00:44:58 <clarkb> ekcs: I can look at job logs in a minute
00:45:09 <clarkb> I'm over in #openstack-infra
00:45:20 <thinrichs> ekcs: can you run next week's meeting?
00:45:28 <ekcs> clarkb: got it thanks so much.
00:45:34 <ekcs> thinrichs: sure thing.
00:45:50 <thinrichs> Got to run.  See you all later!
00:46:01 <ekcs> later.
00:47:16 <ekcs> so… anything else to discuss? or should we just end early?
00:47:26 <ekcs> anyone followed why the design summit split?
00:48:00 <ekcs> 4 events a year is A LOT
00:48:21 <aimeeu> I heard it was to facilitate better inter-team collaboration
00:48:57 <aimeeu> as well as to provide developers with a better working atmosphere - something about too many parties at the regular Summit (gossip of course)
00:49:08 <ekcs> hahaha.
00:49:33 <aimeeu> the Foundation said it was going to look at boring cities for the gatherings
00:50:03 <clarkb> ekcs: I think that your pe1 and pe2 objects are local to setUp and will be GC'd so there isn't anything listening for the cleint to connect to
00:50:51 <aimeeu> but I agree - 4x/year is a lot. The FAQ said it's up to individual teams to decide which events to attend
00:51:55 <ekcs> clarkb: oh thanks a lot for the help! But Hmm I didn’t know that it mattered if the process handle got GC’ed.
00:52:59 <clarkb> if your process isn't backgrounding that means it gets killed I think. (and it probably shouldn't background as init won't know what to do with it)
00:53:01 <ekcs> clarkb: plus, the failure is occurring within setUp where I http to the new process just after starting up.
00:53:55 <clarkb> ah I see that now maybe you aren't waiting long enough for the listening process to actually bind?
00:54:44 <clarkb> also typically we will recommend you listen on port 0 then get the actual port somehow to avoid conflicts between test
00:54:46 <ekcs> clarkb: I could retry longer. But I tihnk it’s up to 20s now. should be plenty?
00:55:00 <clarkb> depends on the process you are starting. I don't know how long it will take
00:55:43 <ekcs> clarkb: oh that sounds like something I should learn more about. starting on port 0 then get port.
00:56:00 <ekcs> clarkb: is there an example of that somewhere? Thanks so much! I’m very new to this.
00:56:13 <clarkb> also both processes use the same sqlite db (not sure sqlite will like that) and the path is rooted to a path that may not exist? not sure how that gets translated to a disk path
00:57:41 <ekcs> clarkb: Right I’ll need to be careful with sqlite. Thanks!
00:58:07 <ekcs> clarkb: it may be that the path translation works differently between my machine and gerrit. thanks for the pointer!
00:58:26 <ekcs> anyway we’re coming up to time. anything last minute?
00:58:47 <clarkb> I don't recall the location of any good port 0 usage off the top of my head. But when you use port zero linux gives you a free port to listen on then you have to determine which port it is someway. YOu can ask the scoket object ni python. But this ensures you don't have conflicting port use between threads or processes
00:59:31 <ekcs> clarkb: got it thanks for the crash course!
00:59:47 <ekcs> Alright time’s up.
00:59:52 <ekcs> Thanks all!
01:00:03 <ekcs> #endmeeting