| *** ladquin has quit IRC | 00:00 | |
| *** koobs has quit IRC | 00:04 | |
| *** koobs has joined #openstack-infra | 00:04 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: new nodepool https://review.openstack.org/40952 | 00:09 |
|---|---|---|
| jeblair | mordred, fungi: ^ it's about 98% there; needs statsd, and i think that's it. | 00:10 |
| *** koobs has quit IRC | 00:15 | |
| *** ewindisch has quit IRC | 00:21 | |
| *** sarob_ has quit IRC | 00:31 | |
| *** sarob has joined #openstack-infra | 00:32 | |
| *** sarob_ has joined #openstack-infra | 00:36 | |
| *** sarob has quit IRC | 00:36 | |
| *** sarob_ has quit IRC | 00:45 | |
| *** sarob has joined #openstack-infra | 00:45 | |
| *** sarob_ has joined #openstack-infra | 00:50 | |
| *** sarob has quit IRC | 00:51 | |
| *** vipul is now known as vipul-away | 00:52 | |
| *** sarob_ has quit IRC | 01:00 | |
| *** sarob has joined #openstack-infra | 01:03 | |
| *** emagana has quit IRC | 01:12 | |
| *** sarob has quit IRC | 01:21 | |
| *** sarob_ has joined #openstack-infra | 01:22 | |
| *** mriedem has joined #openstack-infra | 01:23 | |
| *** anteaya has quit IRC | 01:31 | |
| *** sarob_ has quit IRC | 01:32 | |
| *** sarob has joined #openstack-infra | 01:33 | |
| *** sarob has quit IRC | 01:37 | |
| *** sarob has joined #openstack-infra | 01:37 | |
| *** sarob has quit IRC | 01:42 | |
| *** mjfork has quit IRC | 01:47 | |
| *** dkliban has joined #openstack-infra | 01:51 | |
| *** seagulls has quit IRC | 01:51 | |
| *** sarob has joined #openstack-infra | 01:53 | |
| *** jfriedly has quit IRC | 01:55 | |
| *** ewindisch has joined #openstack-infra | 02:06 | |
| *** mriedem has quit IRC | 02:12 | |
| *** jerryz has quit IRC | 02:12 | |
| *** sarob has quit IRC | 02:29 | |
| *** sarob has joined #openstack-infra | 02:30 | |
| *** koobs has joined #openstack-infra | 02:32 | |
| *** sarob has quit IRC | 02:34 | |
| *** jerryz has joined #openstack-infra | 03:12 | |
| *** sdake_ has quit IRC | 03:35 | |
| *** sarob has joined #openstack-infra | 03:40 | |
| *** jerryz has quit IRC | 03:41 | |
| *** sarob has quit IRC | 03:45 | |
| *** sarob has joined #openstack-infra | 03:49 | |
| mordred | jeblair: you are awesomepants | 03:54 |
| *** sdake_ has joined #openstack-infra | 04:03 | |
| *** changbl has quit IRC | 04:07 | |
| *** dkliban has quit IRC | 04:16 | |
| *** sarob has quit IRC | 04:17 | |
| *** sarob has joined #openstack-infra | 04:17 | |
| *** sarob has quit IRC | 04:21 | |
| *** erfanian has quit IRC | 04:23 | |
| *** Ryan_Lane has joined #openstack-infra | 04:27 | |
| *** ewindisch has quit IRC | 04:28 | |
| *** ewindisch has joined #openstack-infra | 04:32 | |
| *** vogxn has joined #openstack-infra | 04:33 | |
| *** ewindisch has quit IRC | 04:44 | |
| *** sarob has joined #openstack-infra | 04:51 | |
| *** sarob has quit IRC | 04:58 | |
| *** sarob has joined #openstack-infra | 04:58 | |
| *** ewindisch has joined #openstack-infra | 05:00 | |
| *** melwitt has quit IRC | 05:02 | |
| *** sarob has quit IRC | 05:02 | |
| *** Ryan_Lane has quit IRC | 05:03 | |
| *** ewindisch has quit IRC | 05:13 | |
| *** afazekas has joined #openstack-infra | 05:38 | |
| *** vogxn has quit IRC | 05:47 | |
| *** Ryan_Lane has joined #openstack-infra | 05:55 | |
| *** sarob has joined #openstack-infra | 06:09 | |
| *** sarob has quit IRC | 06:14 | |
| *** flashgordon is now known as jog0 | 06:16 | |
| *** Ryan_Lane has quit IRC | 06:46 | |
| marun | devstack broken...again | 06:51 |
| *** nayward has joined #openstack-infra | 06:57 | |
| *** sarob has joined #openstack-infra | 07:08 | |
| *** Ryan_Lane has joined #openstack-infra | 07:10 | |
| *** sarob has quit IRC | 07:12 | |
| *** koolhead11 has joined #openstack-infra | 07:25 | |
| *** Ryan_Lane has quit IRC | 07:29 | |
| *** sarob has joined #openstack-infra | 07:38 | |
| *** sarob has quit IRC | 07:47 | |
| *** odyssey4me has quit IRC | 07:59 | |
| *** odyssey4me has joined #openstack-infra | 08:00 | |
| *** sarob has joined #openstack-infra | 08:13 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added more options to the Build Timeout plugin https://review.openstack.org/39329 | 08:17 |
| *** sarob has quit IRC | 08:18 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Improved the gerrit trigger https://review.openstack.org/40644 | 08:29 |
| *** mikal has quit IRC | 08:41 | |
| *** nayward has quit IRC | 08:42 | |
| *** nayward has joined #openstack-infra | 08:42 | |
| *** mikal has joined #openstack-infra | 08:43 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added ignore-cache and flush-cache options https://review.openstack.org/40651 | 08:45 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added more options to the Build Timeout plugin https://review.openstack.org/39329 | 08:51 |
| *** sarob has joined #openstack-infra | 09:14 | |
| *** sarob has quit IRC | 09:19 | |
| *** nayward has quit IRC | 09:38 | |
| *** vogxn has joined #openstack-infra | 09:48 | |
| lifeless | dstufft: is there a way to make pip install -d <foo> where foo or one of foo's deps is already downloaded, not download it again? ATM it prompts (wipe/backup/ignore) | 10:00 |
| *** sarob has joined #openstack-infra | 10:14 | |
| *** sarob has quit IRC | 10:19 | |
| *** koolhead11 has quit IRC | 10:31 | |
| *** mjblack has joined #openstack-infra | 10:38 | |
| *** mjblack_ has quit IRC | 10:42 | |
| *** marun has quit IRC | 11:05 | |
| *** nayward has joined #openstack-infra | 11:10 | |
| *** sarob has joined #openstack-infra | 11:15 | |
| *** sarob has quit IRC | 11:20 | |
| *** nayward has quit IRC | 11:39 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack-dev/hacking: Reword H404 description to match check https://review.openstack.org/41240 | 11:59 |
| *** vogxn has quit IRC | 12:16 | |
| *** mjblack_ has joined #openstack-infra | 12:19 | |
| *** mjblack has quit IRC | 12:22 | |
| *** sdake_ has quit IRC | 12:32 | |
| *** cp16net_ has joined #openstack-infra | 12:43 | |
| *** mjblack has joined #openstack-infra | 12:45 | |
| *** krtaylor_ has joined #openstack-infra | 12:45 | |
| *** sarob has joined #openstack-infra | 12:45 | |
| *** sarob has quit IRC | 12:50 | |
| *** wayneseguin has joined #openstack-infra | 12:50 | |
| *** cp16net has quit IRC | 12:50 | |
| *** wayneeseguin has quit IRC | 12:50 | |
| *** krtaylor has quit IRC | 12:50 | |
| *** mjblack_ has quit IRC | 12:50 | |
| *** mordred has quit IRC | 12:50 | |
| *** wayneseguin is now known as wayneeseguin | 12:51 | |
| *** cp16net_ is now known as cp16net | 12:52 | |
| *** nayward has joined #openstack-infra | 12:55 | |
| *** mordred has joined #openstack-infra | 12:57 | |
| Alex_Gaynor | If I wanted to steal the apache header check from hacking, for use in a non-opnestack project, how much work would that be? | 12:59 |
| *** nayward has quit IRC | 13:00 | |
| *** rustlebee is now known as russellb | 13:05 | |
| *** vogxn has joined #openstack-infra | 13:11 | |
| *** jjmb has quit IRC | 13:14 | |
| sdague | Alex_Gaynor: you can pull the hacking project pretty easily, and just disable the parts you don't like | 13:14 |
| sdague | it's one of the reasons we made it a separate pip | 13:15 |
| sdague | it might be easier than gutting out the code | 13:15 |
| *** vogxn has quit IRC | 13:15 | |
| *** vogxn has joined #openstack-infra | 13:16 | |
| *** vogxn has quit IRC | 13:20 | |
| *** dkliban has joined #openstack-infra | 13:32 | |
| mordred | Alex_Gaynor: yup. what sdague said. that said - it's all just a collection of flake8 plugins | 13:41 |
| mordred | Alex_Gaynor: so it would likely be very little work | 13:41 |
| *** GheRivero is now known as GheRivero_CH | 13:44 | |
| *** sarob has joined #openstack-infra | 13:46 | |
| *** ewindisch has joined #openstack-infra | 13:47 | |
| *** dkliban has quit IRC | 13:50 | |
| *** sarob has quit IRC | 13:50 | |
| Alex_Gaynor | Thanks. | 13:51 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Add disagreement stats https://review.openstack.org/41243 | 13:57 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add disagreement stats https://review.openstack.org/41243 | 13:57 |
| *** ewindisch has quit IRC | 14:00 | |
| sdague | mordred: you understand what's wrong here? http://logs.openstack.org/95/40995/2/gate/gate-requirements-integration-no-mirror/64792bf/console.html.gz | 14:02 |
| sdague | that blocks this - https://review.openstack.org/#/c/40995/ | 14:02 |
| sdague | which blocks writing the devstack fix | 14:02 |
| mordred | sdague: yes. and I just noticed that the fix to the jenkins jobs hadn't applied because puppet was stuck | 14:02 |
| sdague | ok | 14:03 |
| mordred | I restarted puppet, applied the fix, and am re-rechking a pbr change | 14:03 |
| sdague | cool | 14:03 |
| mordred | sdague: however - reverify away - the fix removes that job (which was new) from requirements | 14:04 |
| sdague | ok, cool | 14:07 |
| *** sdake_ has joined #openstack-infra | 14:09 | |
| *** sdake_ has joined #openstack-infra | 14:09 | |
| *** russellb has quit IRC | 14:13 | |
| *** fungi has quit IRC | 14:20 | |
| *** jpeeler has joined #openstack-infra | 14:21 | |
| *** russellb has joined #openstack-infra | 14:22 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Refactor into a main() function https://review.openstack.org/41244 | 14:27 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Refactor into a main() function https://review.openstack.org/41244 | 14:27 |
| *** dkehn has quit IRC | 14:28 | |
| *** dkehn has joined #openstack-infra | 14:29 | |
| *** mberwanger has joined #openstack-infra | 14:30 | |
| *** fungi has joined #openstack-infra | 14:34 | |
| *** dkehn has quit IRC | 14:35 | |
| *** dkehn has joined #openstack-infra | 14:38 | |
| *** sarob has joined #openstack-infra | 14:47 | |
| *** sarob has quit IRC | 14:52 | |
| *** nayward has joined #openstack-infra | 14:55 | |
| *** nayward has quit IRC | 14:58 | |
| openstackgerrit | A change was merged to openstack/requirements: add option for adding a suffix to output files https://review.openstack.org/40995 | 15:00 |
| *** russellb has quit IRC | 15:09 | |
| *** russellb has joined #openstack-infra | 15:12 | |
| *** mberwanger has quit IRC | 15:15 | |
| *** mberwanger has joined #openstack-infra | 15:21 | |
| *** changbl has joined #openstack-infra | 15:30 | |
| *** changbl has quit IRC | 15:36 | |
| *** erfanian has joined #openstack-infra | 15:38 | |
| *** sarob has joined #openstack-infra | 15:47 | |
| *** sarob has quit IRC | 15:51 | |
| *** jpeeler has quit IRC | 16:02 | |
| *** sarob has joined #openstack-infra | 16:11 | |
| fungi | fyi, got impatient waiting to turn cacti back on and opened https://launchpad.net/bugs/1210822 | 16:20 |
| uvirtbot | Launchpad bug 1210822 in cacti "Please import 0.8.8b+dfsg-2 from Debian and backport security fixes to 12.04 LTS" [Undecided,New] | 16:20 |
| *** Protux has joined #openstack-infra | 16:25 | |
| *** sarob has quit IRC | 16:29 | |
| *** sarob has joined #openstack-infra | 16:29 | |
| clarkb | fungi: thank you, I miss cacti too | 16:31 |
| fungi | granted, the version containing the fix to the regression introduced by the version with the security fixes only just got uploaded to debian | 16:31 |
| fungi | like, yesterday sometime | 16:32 |
| clarkb | I hope to review the nodepool code tomorrow once I have recovered from everything that is going on | 16:33 |
| clarkb | we are thankfully not going to Mt. Rainier today | 16:33 |
| fungi | aww, that sounds like fun though | 16:33 |
| *** sarob has quit IRC | 16:33 | |
| clarkb | I am too tired | 16:33 |
| *** mberwanger has quit IRC | 16:37 | |
| *** Adri2000_ is now known as Adri2000 | 16:39 | |
| *** erfanian has quit IRC | 16:42 | |
| *** erfanian has joined #openstack-infra | 16:53 | |
| *** odyssey4me has quit IRC | 16:53 | |
| *** Steely_Dan is now known as dansmith | 16:55 | |
| *** nayward has joined #openstack-infra | 16:56 | |
| *** nayward has quit IRC | 17:00 | |
| *** boris-42 has joined #openstack-infra | 17:08 | |
| *** dkliban has joined #openstack-infra | 17:21 | |
| *** pabelanger has quit IRC | 17:30 | |
| *** sarob has joined #openstack-infra | 17:40 | |
| *** sarob has quit IRC | 17:45 | |
| jgriffith | fungi: clarkb Any idea why my change is running Python 3 pep-8? https://jenkins01.openstack.org/job/gate-cinder-pep8/96/console | 18:03 |
| jgriffith | fungi: clarkb fresh devstack setup with this patch it's not being checked so I'm having a hard time figuring out where the ignore isn't being set etc | 18:04 |
| jgriffith | fungi: clarkb hmm... wait, is it the auto-pulled test-requirements update to hacking perhaps? | 18:06 |
| jgriffith | appears that might be the case | 18:09 |
| clarkb | yes from hacking | 18:12 |
| clarkb | and I think the check is to avoid adding more incompatibility with py3k | 18:13 |
| clarkb | jgriffith you can # noqa the line or ignore the rule in flake8's tox.ini section | 18:13 |
| *** Ryan_Lane has joined #openstack-infra | 18:16 | |
| *** jpeeler has joined #openstack-infra | 18:18 | |
| jgriffith | clarkb: well.. it's not quite so simple :) | 18:19 |
| jgriffith | clarkb: so the problem is that the updates were committed with my patch and resulted in things falling apart | 18:20 |
| jgriffith | clarkb: I'm sending an email out to describe it and how to avoid it | 18:20 |
| jgriffith | clarkb: although adding the ignore to tox.ini will fix it | 18:21 |
| jgriffith | I'll do that now | 18:21 |
| jgriffith | clarkb: thanks | 18:21 |
| *** mriedem has joined #openstack-infra | 18:24 | |
| *** annegentle has quit IRC | 18:24 | |
| *** romero has joined #openstack-infra | 18:31 | |
| *** erfanian has quit IRC | 18:34 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Tweak and explain disagreements metric https://review.openstack.org/41253 | 18:40 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Tweak and explain disagreements metric https://review.openstack.org/41253 | 18:40 |
| *** marun has joined #openstack-infra | 18:48 | |
| *** Ryan_Lane has quit IRC | 18:51 | |
| fungi | right, jog0 said something about "new hacking release" | 18:54 |
| fungi | jgriffith: https://pypi.python.org/pypi/hacking says 0.7.0 circa yesterday | 18:55 |
| * fungi is back from groceries | 18:55 | |
| *** nayward has joined #openstack-infra | 18:56 | |
| *** annegentle has joined #openstack-infra | 18:59 | |
| *** nayward has quit IRC | 19:01 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Update requirements in integration test https://review.openstack.org/41098 | 19:01 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Added documentation for packagers https://review.openstack.org/41104 | 19:02 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Use wheel by default https://review.openstack.org/41255 | 19:03 |
| *** erfanian has joined #openstack-infra | 19:15 | |
| fungi | it's funny. the moment i saw those ^ patches, "a knife and a fork" ("...a bottle with a cork, that's the way to spell new york" from information society's "hack") came up in rotation. perhaps it's his theme song? | 19:18 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Start building wheels in the mirror https://review.openstack.org/41215 | 19:19 |
| *** Ryan_Lane has joined #openstack-infra | 19:22 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Gate jeepyb and openstack/requirements on pbr https://review.openstack.org/41256 | 19:31 |
| *** Ryan_Lane has quit IRC | 19:32 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Start building wheels in the mirror https://review.openstack.org/41215 | 19:37 |
| *** jpeeler has quit IRC | 19:52 | |
| *** jpeeler has joined #openstack-infra | 19:52 | |
| mordred | dstufft: you know the best thing about pip wheel? | 19:56 |
| mordred | dstufft: it normalizes -'s to _'s in a way that does not match sdist!!! | 19:56 |
| dstufft | mordred: I didn't have to write any of the code for it and people are thanking me for it? | 19:56 |
| mordred | dstufft: :) | 19:56 |
| mordred | dstufft: I'm real close to having out mirror build wheels for everything, btw | 19:56 |
| dstufft | mordred: awesome :D | 19:57 |
| *** mriedem has quit IRC | 19:57 | |
| *** changbl has joined #openstack-infra | 20:00 | |
| *** jpeeler has quit IRC | 20:00 | |
| clarkb | I am on a phone but the list question about bugs is related to the new changes around that in jeepyb | 20:08 |
| clarkb | if you use the old style string it uses the weakest association which is related iirc | 20:08 |
| clarkb | at least this is what seems to be happening | 20:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Start building wheels in the mirror https://review.openstack.org/41215 | 20:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Start building wheels in the mirror https://review.openstack.org/41215 | 20:15 |
| mordred | ok. that last one shoudl work | 20:16 |
| fungi | any objections to me restarting gerrit later on today to pick up the new replication config modifications? once i do i'll keep an eye on things for a while and roll back the config if it breaks badly for some reason | 20:19 |
| mordred | fungi: please do | 20:23 |
| mordred | jgriffith: your current project should not have picked up new flake8/hacking rules without a patch to update your project | 20:26 |
| mordred | jgriffith: I'm quite interested in learning more about how your env wound up breaking | 20:26 |
| mordred | jgriffith: because it's certainly not what it's designed to do :) | 20:26 |
| lifeless | mordred: maybe it wasn't pinned | 20:27 |
| *** sarob has joined #openstack-infra | 20:28 | |
| fungi | mordred: worth noting i saw similar issues with git-review, thus https://review.openstack.org/40799 (admittedly no requirements enforcement there, but still) | 20:28 |
| fungi | didn't manifest in the same way, but stale uncapped hacking dependency declared | 20:29 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Added documentation for packagers https://review.openstack.org/41104 | 20:30 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Update requirements in integration test https://review.openstack.org/41098 | 20:30 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Fix python-ldap mirroring. https://review.openstack.org/40732 | 20:30 |
| mordred | lifeless, fungi: nod | 20:32 |
| mordred | fungi, clarkb, lifeless: if you get bored: https://review.openstack.org/41215 works in my local testing and puts wheels in the created mirror | 20:32 |
| lifeless | are the wheels differentiated enough that different arch / distro versions won't be a problem ? | 20:34 |
| lifeless | e.g. the same wheel on centos and raring | 20:34 |
| mordred | lifeless: no. /home/mordred/pypimirror/openstack/pyOpenSSL/pyOpenSSL-0.13-cp27-none-linux_x86_64.whl is an example of a wheel with actual binary content | 20:35 |
| lifeless | so | 20:35 |
| lifeless | doesn't that imply that we can't use them ? | 20:36 |
| mordred | it doesn't imply that the gate can't use them | 20:36 |
| lifeless | unless we do a separate mirror per distribution ? | 20:36 |
| lifeless | arch, python are in there | 20:36 |
| mordred | purely that people wanting to consume the openstack mirror _and_ turn on "PIP_USE_WHEEL" can only do so in certain situations | 20:37 |
| lifeless | ok | 20:37 |
| lifeless | I still don't understand how wheels will ever actually work at the moment | 20:37 |
| lifeless | for pypi itself I mean. | 20:37 |
| lifeless | dstufft: ^ | 20:37 |
| mordred | I tihnk that wheels on pypi for anything other than pure-python are a bit a problem | 20:38 |
| mordred | until the wheel people start doing some of the binary depend manifest matching stuff we talked about a while back | 20:38 |
| lifeless | so to the extent that our mirror is used like pypi - e.g. by gate clients that have different so versions | 20:39 |
| lifeless | we'll have the same problem | 20:39 |
| mordred | but pure-python wheels on pypi are, I think, still helpful | 20:39 |
| mordred | we have 2 different gate envs | 20:39 |
| *** pabelanger has joined #openstack-infra | 20:39 | |
| mordred | centos / python 2.6 and precise 2.7 | 20:39 |
| mordred | but - I agree re: the potential for confusion - perhaps we should put wheels in a separate mirror dir with distro specified | 20:40 |
| mordred | so that one would need to explicitly follow a particular set of wheels we put up | 20:40 |
| lifeless | so we have two mirrors? or one mirror and two gate environments? | 20:40 |
| lifeless | [today] | 20:40 |
| mordred | we currently have one mirror for the openstack reqs and one for openstack-infra | 20:41 |
| mordred | but one mirror shared by both gate envs | 20:41 |
| lifeless | right | 20:41 |
| mordred | however, in that one specific case, since our mirror builders are the same as our consumers | 20:41 |
| mordred | and we know the distro on both | 20:41 |
| lifeless | so this will be a compat issue if we put both gate builds in the one mirror | 20:42 |
| mordred | the py26/py27 split in wheels will be good enough for the very specific case of the gate | 20:42 |
| lifeless | oh, I see | 20:42 |
| lifeless | right | 20:42 |
| lifeless | what about 3.3 ? | 20:42 |
| mordred | 3.3 are also on a separate distro | 20:42 |
| mordred | :) | 20:42 |
| mordred | but - I could totally see putting the wheels into, like pypi.o.o/openstack/precise and pypi.o.o/openstack/centos6 or something | 20:43 |
| *** mberwanger has joined #openstack-infra | 20:43 | |
| lifeless | I think that is a good idea | 20:43 |
| mordred | less chance for accidental harm ++ | 20:43 |
| mordred | sigh | 20:45 |
| lifeless | reviewed | 20:45 |
| mordred | it would be great if there was a sensible way to make an id for the distro level of that | 20:45 |
| lifeless | if you could tag the wheel and tell pip to look for a tag | 20:45 |
| lifeless | that would be great. | 20:45 |
| mordred | lifeless: so - the thing dstufft and I spoke about last time we talked - was to get wheel to include distro/release in the wheel name as well | 20:46 |
| mordred | and then also put a manifest of the actual ld reqs into the wheel | 20:46 |
| mordred | so that pip could attempt to download the best match it could find - but _also_ check that it was a sensible thing to install once it had it | 20:46 |
| mordred | and if not, bail and go get teh source | 20:47 |
| lifeless | yah | 20:47 |
| lifeless | the life of an early adopter... | 20:47 |
| mordred | lifeless: lsb_release -i -r -s | xargs | tr ' ' '-' | 20:48 |
| mordred | does that seem to be a sane enough id to use for the distro id? | 20:49 |
| lifeless | lsb_release -cs | 20:49 |
| mordred | on centos, that's "Final" | 20:49 |
| lifeless | ahahahahahaha | 20:49 |
| lifeless | dammit | 20:49 |
| mordred | right? | 20:49 |
| * mordred headbutts everyone | 20:50 | |
| mordred | lifeless: btw - have you seen the python sh lib? | 20:51 |
| lifeless | yes | 20:51 |
| mordred | I tried it out the other day - with _one_ small patch, it's totally usable for the type of shell stuff we tend to do in here | 20:53 |
| mordred | unfortunately, the author is taking a little bit of a break from doing side opensource projects | 20:53 |
| *** nayward has joined #openstack-infra | 20:56 | |
| lifeless | when I looked at it the DWIM aspect terrified me | 20:58 |
| *** mberwanger has quit IRC | 20:59 | |
| mordred | yeah. | 21:00 |
| mordred | lifeless: ooh! so - | 21:00 |
| mordred | it turns out that we're still producing html files in run-mirror | 21:01 |
| mordred | BUT | 21:01 |
| mordred | we do not use them in the actual openstack mirror, becaseu we use mod_autoindex | 21:01 |
| *** nayward has quit IRC | 21:01 | |
| lifeless | yes, I know. My q the other day was about file:/// use. | 21:02 |
| mordred | hrm. file use does seem to need them | 21:03 |
| mordred | dammit | 21:03 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Start building wheels in the mirror https://review.openstack.org/41215 | 21:19 |
| mordred | lifeless: ^^ there's a stab at per-distro wheel dirs | 21:19 |
| *** w_ has joined #openstack-infra | 21:23 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add support for per-distro wheel mirrors https://review.openstack.org/41268 | 21:26 |
| mordred | and there's a stab at consuming it | 21:26 |
| *** olaph has quit IRC | 21:26 | |
| *** sgviking has joined #openstack-infra | 21:29 | |
| mordred | lifeless: w00t. it even works | 21:31 |
| mordred | fungi: ^^ the run-mirror change I have tested and it seems to work well | 21:35 |
| mordred | fungi: the per-disto wheel mirrors in select-mirror is not directly tested | 21:35 |
| fungi | i saw the test results just come back on that. having a look after dinner | 21:36 |
| mordred | but I did test the actual distro=`` command | 21:36 |
| fungi | mmm, per distro. fun | 21:36 |
| mordred | on both Centos and Ubuntu | 21:36 |
| lifeless | mordred: reviewed | 21:36 |
| mordred | lifeless: thanks | 21:39 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Start building wheels in the mirror https://review.openstack.org/41215 | 21:46 |
| mordred | lifeless: ^^ I think I've addressed things - also, here's the output it produces: | 21:48 |
| mordred | paste.openstack.org/show/43806 | 21:48 |
| *** sarob has quit IRC | 21:48 | |
| *** sarob has joined #openstack-infra | 21:49 | |
| *** dripton has quit IRC | 21:50 | |
| jeblair | mordred, fungi: i'm not really here, but i'm going to put jenkins01 into shutdown so later on i can upgrade gearman plugin there | 21:51 |
| mordred | jeblair: I'm excited about that | 21:51 |
| mordred | jeblair: when you are really here, I believe I've got a solid wheel in mirror solution coded and tested | 21:51 |
| *** dripton has joined #openstack-infra | 21:53 | |
| *** sarob has quit IRC | 21:53 | |
| lifeless | mordred: what about stdout from the commands ? | 21:58 |
| lifeless | mordred: you do out = ... on some, not on others. | 21:58 |
| lifeless | mordred: I don't understand the difference. | 21:58 |
| mordred | lifeless: I tried to delete the out from the ones I'm not using anymore | 22:00 |
| mordred | the only real reason to do anything with the output of those command is, well, if you want to do something with the output | 22:00 |
| dstufft | lifeless: mordred Compiled wheels for windows work fine | 22:01 |
| dstufft | on PyPI | 22:01 |
| lifeless | mordred: but if it fails | 22:01 |
| lifeless | mordred: how does the user debug it ? | 22:01 |
| dstufft | there's a problem with pure python wheels if you use the console_scripts entry point though | 22:01 |
| lifeless | dstufft: it'll work fine for windows, as long as any dependent dlls are abi compatible :) | 22:02 |
| dstufft | lifeless: I think windows SxS solves that? I dunno I don't know fuck all about wndows | 22:03 |
| mordred | dstufft: what sort of problem? | 22:03 |
| dstufft | I just know I had people who should tell me that it'll work | 22:03 |
| dstufft | mordred: entry points are platformish specific | 22:03 |
| dstufft | e.g. for Windows there's an extra .exe that is generated | 22:03 |
| mordred | oh - gah. I thought you were talking about console script entrypoints on linux | 22:04 |
| dstufft | so a pure python wheel with a consule_script generated on Linux won't have the .exe on windows | 22:04 |
| * mordred does not care about windows even the tiniest bit here | 22:04 | |
| dstufft | yea if all you care about is Linux it shouldn't affeect you in the slightest | 22:04 |
| mordred | w00t! | 22:04 |
| mordred | then I _think_ we're close to good to go for our needs | 22:04 |
| mordred | lifeless: if it fails, run on verbose | 22:05 |
| mordred | that will print all commands and all of their output | 22:05 |
| lifeless | mordred: oh | 22:05 |
| mordred | only reason the _program_ needs the out return value is if it needs to process it for further operation | 22:05 |
| lifeless | mordred: so one common problem with that pattern is 'fuck, it errored, and now it doesn't when I run it again' | 22:05 |
| mordred | lifeless: indeed. we run all of our mirror jobs with verbose on :) | 22:06 |
| lifeless | mordred: similarly, 'fuck it errored and it took 50m to get here, so I have to wait 50m to fix that up' | 22:06 |
| lifeless | s/fix/find out/ | 22:06 |
| mordred | yeah. totally. the UI can totally do with some rework | 22:06 |
| *** erfanian has quit IRC | 22:06 | |
| lifeless | mordred: ok - thanks for explaining. Clearly a separate problem. | 22:06 |
| mordred | yup. I agree though that we should address it | 22:06 |
| mordred | being one of the people who does 'enjoy' running this locally a lot | 22:06 |
| lifeless | anyhow, I'd like a helper function to group the wheel thing | 22:06 |
| mordred | "group the wheel thing" ? | 22:07 |
| lifeless | and the odd "setuptools Bug." comment needs a fixup of some sort. | 22:07 |
| lifeless | mordred: see the review | 22:07 |
| mordred | looking | 22:07 |
| dstufft | mordred: lifeless I really want to make a transactional logging library | 22:07 |
| dstufft | for python | 22:07 |
| lifeless | dstufft: hah. I wrote one :) | 22:08 |
| dstufft | log tons of verbose stuff, and if a logical operation succeeds you discard the logged messages, if it fails you output them all | 22:08 |
| dstufft | :V | 22:08 |
| lifeless | dstufft: the 'oops' set of micro libraries | 22:08 |
| mordred | lifeless: yes. stdout from the commands is also printed with verbose | 22:08 |
| mordred | oh.crap. I was scrolled back | 22:08 |
| mordred | :) | 22:08 |
| dstufft | lifeless: awesome, I'll take a look | 22:08 |
| dstufft | right now I'm working on a new crypto library | 22:08 |
| dstufft | because I'm bad at life choices | 22:08 |
| mordred | lifeless: and yes on the function - I was going to do that | 22:08 |
| mordred | dstufft: wow. you suck at life choices, actually | 22:09 |
| * mordred does too | 22:09 | |
| *** sarob has joined #openstack-infra | 22:10 | |
| dstufft | mordred: techincally this is the second crypto library I've worked on | 22:10 |
| dstufft | So i'm also a slow learner | 22:10 |
| dstufft | :V | 22:10 |
| dstufft | first one is PyNaCl https://github.com/dstufft/pynacl second is cryptography https://github.com/alex/cryptography | 22:11 |
| lifeless | dstufft: so - https://pypi.python.org/pypi/oops | 22:12 |
| lifeless | dstufft: the concept was factored out from LP | 22:12 |
| dstufft | lifeless: misread LGPL as GPL and almost had to discount it as useless to me D: | 22:13 |
| lifeless | dstufft: so you accumulate as much state as you want - MB's or GB's even - and then spit it out to whatever sink you want | 22:13 |
| *** boris-42 has quit IRC | 22:13 | |
| dstufft | lifeless: awesome | 22:13 |
| lifeless | dstufft: in LP we captured every single SQL query, all web service back end calls, syscalls we expected to be slow etc. | 22:14 |
| lifeless | dstufft: then spat it out over AMQP, with a local cache if AMQP was down; and a cron job to empty that cache when AMQP came back | 22:14 |
| lifeless | dstufft: there is a twisted variant of it, with hooks into the twisted log module | 22:14 |
| lifeless | dstufft: and there is a thunk for python 'logging' too, so regular logging output gets captured. | 22:14 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Remove spurious uses of output return values https://review.openstack.org/41273 | 22:15 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Start building wheels in the mirror https://review.openstack.org/41215 | 22:15 |
| lifeless | dstufft: there is a js variant and a wsgi app too, so you can capture js exceptions from webapps back to the wsgi app and that injects into the same framework | 22:15 |
| lifeless | dstufft: and finally a django app to store and analyse in a db (but you can equally do any custom sink you want) | 22:16 |
| dstufft | lifeless: wow, a bunch of stuff :D | 22:18 |
| lifeless | dstufft: yah. | 22:22 |
| lifeless | dstufft: I think the django app might be GPL, the libraries are all LGPL from memory | 22:23 |
| lifeless | https://pypi.python.org/pypi?%3Aaction=search&term=oops&submit=search | 22:23 |
| lifeless | yeah, oops-tools is AGPL | 22:24 |
| lifeless | but you don't need it | 22:24 |
| lifeless | well, depends on the problem you're tackling, - but doing a different console is very doable | 22:24 |
| lifeless | see for instance whoopsie-daisy, the ubuntu error reporting system, which uses the same schema | 22:25 |
| lifeless | but uses a big-data analysis setup - errors.ubuntu.com | 22:25 |
| lifeless | mordred: so what about this sentence - | 22:27 |
| lifeless | # Need to do these separately to not trigger setuptools | 22:27 |
| lifeless | # Bug. | 22:27 |
| lifeless | I still can't make head or tails of the end of it. | 22:27 |
| *** UtahDave has joined #openstack-infra | 22:29 | |
| marun | i'm still seeing random devstack breakage | 22:50 |
| marun | mordred: i'm wondering if it might make sense to have a smoke build of devstack that isn't optimized for speed, but rather to ensure that a developer can build without error | 22:51 |
| *** nayward has joined #openstack-infra | 22:57 | |
| mordred | marun: is this still on redhat? | 23:00 |
| marun | nope | 23:01 |
| mordred | you're seeing problems running devstack on ubuntu? | 23:01 |
| marun | yes | 23:01 |
| mordred | that's very distressing | 23:01 |
| marun | first one is related to this fix: https://review.openstack.org/#/c/41209/1 | 23:01 |
| marun | I'm going to submit an updated patch that does the same for httplib2 as the patch currently does for pretty table | 23:01 |
| marun | i'm thinking this issue really needs to be fixed in pip, though | 23:02 |
| mordred | dstufft: have you seen this ^^ ? | 23:02 |
| marun | the second is related to pbr | 23:02 |
| *** nayward has quit IRC | 23:02 | |
| marun | when i do a clean devstack run, it fails due to a pbr-related error. stacking again fixes it. | 23:02 |
| marun | i'll get you the precise message shortly | 23:02 |
| *** dkliban has quit IRC | 23:03 | |
| marun | they are small things - nothing that can't be worked around - but still | 23:03 |
| marun | the fact that gate isn't catching them is worrying | 23:03 |
| mordred | yes. I agree. I would lke for the gate to catch them | 23:04 |
| mordred | so I would consider the gate not catching them to be a bug for sure | 23:04 |
| marun | i'm thinking it's related to gate becoming so specialized for the sake of speed/isolation | 23:04 |
| *** UtahDave has quit IRC | 23:04 | |
| marun | i'm not sure that's compatible with testing for general use | 23:04 |
| mordred | well, I mean, the only things we do in the gate that are different | 23:05 |
| mordred | have to do with caching packages (not installing them) and pointing to an external mirror | 23:05 |
| mordred | internal | 23:05 |
| mordred | not external | 23:05 |
| *** mjblack_ has joined #openstack-infra | 23:05 | |
| mordred | we actually don't care about speed - the only thing we care about is repeatability- because touching pypi or github or whatever has demonstrably failed at the volume of tests we run | 23:06 |
| mordred | BUT | 23:06 |
| mordred | if we're doing something to pollute the host we're running on | 23:06 |
| mordred | we need to fix it | 23:06 |
| mordred | because that would be 100% contrary to what we're trying to do | 23:06 |
| *** mjblack__ has joined #openstack-infra | 23:07 | |
| marun | mordred: ah, fair enough | 23:08 |
| * mordred just spun up a blank node and will try running devstack on it with no caching | 23:08 | |
| * mordred curious | 23:08 | |
| *** mjblack has quit IRC | 23:08 | |
| marun | mordred: pip/setuptools seem to be the major culprits lately. | 23:08 |
| mordred | well, the setuptools upgrade has been a complete and total nightmare | 23:09 |
| *** mjblack_ has quit IRC | 23:10 | |
| marun | mordred: It points out, at least, the importance of behavioral stability in a packaging system. | 23:10 |
| mordred | if you pip install -U setuptools and then pip install -U pip before you run stack.sh - it SHOULD work | 23:11 |
| mordred | marun: totally | 23:11 |
| mordred | although I think we are currently doing a thing that nobody thinks is a good idea - in that we are mixing pypi and distro packages and we are doing it without virtualenvs | 23:11 |
| dstufft | basically a bad idea | 23:11 |
| mordred | basically, there are 0 people in python, ubuntu or redhat packaging worlds who think it's a good idea | 23:12 |
| dstufft | You'll need to use symlinks to make M2Crypto work in a venv though | 23:12 |
| dstufft | I think you're using M2Crypto | 23:12 |
| marun | mordred: do you think the way forward can be virtualenv installation and no python packages from distro? | 23:12 |
| mordred | we don't use M2Crypto anymore | 23:12 |
| dstufft | well then | 23:12 |
| mordred | marun: YES | 23:12 |
| dstufft | venv it up | 23:12 |
| mordred | marun: I actually think there are only two choices | 23:12 |
| mordred | marun: 100% in a venv with no distro python packages (except for libvirt, because fail) | 23:12 |
| mordred | marun: OR | 23:12 |
| mordred | marun: 100% distro packages | 23:13 |
| dstufft | basically | 23:13 |
| mordred | which, from experience, we can both do, and will cause a different pile of heartache and confusion | 23:13 |
| dstufft | also | 23:13 |
| dstufft | why does libvirt not work in a venv? | 23:13 |
| mordred | because, it turns out, there are tons of python devs who do not know how to make debs | 23:13 |
| mordred | dstufft: because. it doesn't build with setuptools | 23:13 |
| mordred | it builds as part of the libvirt C code | 23:13 |
| dstufft | lol | 23:14 |
| marun | i vote no on distro packages | 23:14 |
| dstufft | W E L P | 23:14 |
| marun | developing python is best done pre-distro imho | 23:14 |
| dstufft | +10000 | 23:14 |
| mordred | marun: me too. I'm willing to go down the road, but it is fraught with peril, tbh | 23:14 |
| mordred | marun: ++ | 23:14 |
| dstufft | it's how most python packages are developed | 23:14 |
| mordred | I made a patch to devstack to install ina global venv | 23:14 |
| dstufft | so it takes you closer to veryone else, for better or worse :V | 23:14 |
| mordred | I cannot get it fully working | 23:14 |
| marun | why not? | 23:14 |
| mordred | https://review.openstack.org/#/c/40534/ | 23:15 |
| mordred | the services can't figure out how to find stuff | 23:15 |
| dstufft | mordred: I bet sooner or later Alex_Gaynor makes a lbirary to use CFFI to bind to libvirt | 23:15 |
| mordred | also, my patch seems to suck | 23:15 |
| *** mjblack__ has quit IRC | 23:15 | |
| mordred | dstufft: that would be great | 23:15 |
| mordred | dstufft: as long as it doesn't setup_requires on something crazy :) | 23:15 |
| dstufft | likely just cffi | 23:16 |
| mordred | marun: I just updated the patch based on the comments in the review | 23:17 |
| mordred | marun: if you feel like tyring to figure out where it breaks - I'd LOVE to have it as a real option | 23:17 |
| clarkb | cffi is the crazy :P | 23:17 |
| marun | mordred: really? | 23:17 |
| marun | mordred: Are you adding the env to site-packages? | 23:18 |
| dstufft | mordred: I disagree that it's not proving anything to install globally with pip… it's proving it's a bad idea ;] | 23:18 |
| marun | mordred: I used to configure mod-python to target envs so that web apps could be isolated. I'll take a look and see if the same logic applies. | 23:19 |
| marun | mordred: I'm thinking the fact that pbr generates scripts could allow those scripts to be generated with the appropriate site packages fixup. | 23:19 |
| mordred | marun: I _think_ it has something to do with the screenit function | 23:19 |
| marun | mordred: ah, so you're just activating the env prior to running it | 23:20 |
| marun | mordred: that's one way. | 23:20 |
| marun | mordred: the other way is ensuring the daemon scripts will automatically pick up the global env | 23:21 |
| mordred | marun: cool | 23:24 |
| mordred | marun: I just fixed merged conflicts - re-pushing | 23:24 |
| mordred | marun: is [ERROR] /home/ubuntu/devstack/lib/keystone:316 keystone did not start | 23:26 |
| mordred | marun: what you get when running devstack on a blank node? | 23:26 |
| marun | that's the one i'm fixing | 23:26 |
| marun | https://review.openstack.org/#/c/41209/ | 23:27 |
| marun | dean's fix was for pretty table, i've added the same fix for httplib2 | 23:27 |
| mordred | that's FANTASTIC | 23:28 |
| marun | I'm going to restart from a clean vm and see if i can replicate the pbr installation issue | 23:28 |
| mordred | cool | 23:28 |
| mordred | marun: Requirement already satisfied (use --upgrade to upgrade): httplib2 in /usr/lib/python2.7/dist-packages | 23:29 |
| mordred | Cleaning up... | 23:29 |
| mordred | chmod: cannot access `/usr/lib/python2.7/dist-packages/httplib2-0.8*/*': No such file or directory | 23:29 |
| marun | patch no worky? | 23:30 |
| mordred | marun: /usr/lib/python2.7/dist-packages/httplib2/ is what I've got on there | 23:30 |
| mordred | because, I believe, it was installed via apt | 23:30 |
| mordred | granted, I'm running an unstack/stack | 23:30 |
| mordred | so maybe I should start with clean node too | 23:30 |
| marun | huh, is the check for the directory not working then? | 23:30 |
| marun | oh, i see | 23:31 |
| marun | dean's check ensures that the library is installed from pip | 23:31 |
| * mordred is going to try the combo of my patches and that patch | 23:31 | |
| marun | mine just looks for the httplib2 directory, which could be from distro | 23:31 |
| marun | I guess I should be looking for the egg directory? | 23:32 |
| mordred | it depends | 23:32 |
| jeblair | mordred, fungi: jenkins01 has been restarted with new gearman plugin; i'm going to leave them both in the mix for a bit, and if it looks okay, do the same for jenkins02 | 23:34 |
| fungi | jeblair: awesome | 23:34 |
| mordred | jeblair: ossum | 23:35 |
| mordred | marun: I'm running devstack on a clean node with your and both of my patches applied | 23:36 |
| marun | mordred: I can't reproduce the pbr install issue I saw earlier. I guess it was temporal. | 23:44 |
| marun | mordred: I've updated the review so that the permission change for httplib2 can't ever fail. | 23:45 |
| mordred | marun: because the keystone error is actually one I was seeing when trying to do the 'use global venv' patch - so maybe your fixup patch will fix that too | 23:45 |
| fungi | found a moment when gerrit seemed particularly quiet and restarted it | 23:46 |
| marun | mordred: tangential - what's the best way to deduce failure in devstack-deployed openstack? | 23:46 |
| fungi | seems to be up and responding | 23:46 |
| marun | so many services that can fail, is the only option grepping the logs? | 23:46 |
| mordred | fungi: w0t! | 23:46 |
| mordred | marun: yes. although - we have a logstash system up and going that we haven't publicized much yet | 23:47 |
| mordred | marun: into which we are feeding all devstack logs | 23:47 |
| mordred | marun: we have some plans to try to do some smart things like baysian filtering and stuff, to try to show what's _different_ from normal in your logs from other logs | 23:47 |
| mordred | for now though - grpping | 23:47 |
| mordred | dammit | 23:47 |
| mordred | still [ERROR] /home/ubuntu/devstack/lib/keystone:316 keystone did not start | 23:47 |
| marun | i'll try out your patch | 23:48 |
| marun | on second thought, though, i'm going to stop working on a saturday :p | 23:48 |
| mordred | hehe | 23:48 |
| marun | I'll try out your patch next chance I get, though. | 23:49 |
| mordred | fungi: git.o.o is still unhappy | 23:49 |
| mordred | fungi: didn't we land the ssl patch? | 23:49 |
| fungi | mordred: we did | 23:49 |
| mordred | fungi: welp. that's different. | 23:50 |
| mordred | fungi: I restarted apache there - now it shows me self-signed cert, but does not show me the cgit anymore | 23:50 |
| mordred | pleia2: ^^ | 23:50 |
| fungi | huh, it has a self-signed cert now. it had a proper one before | 23:50 |
| fungi | yeah | 23:50 |
| fungi | i wonder if there's some conflicting configuration in the ssl.conf that patch added | 23:51 |
| fungi | also i suspect to start replicating into that server i need to prompt gerrit to replicate --all | 23:51 |
| fungi | will do here in a moment | 23:51 |
| mordred | fungi: found it | 23:51 |
| fungi | different keys specified in ssl.conf? | 23:51 |
| mordred | fungi: the if mod_ssl.c in the apache conf | 23:52 |
| fungi | yep, "SSLCertificateKeyFile /etc/pki/tls/private/localhost.key" | 23:53 |
| fungi | et cetera | 23:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Remove if module statements https://review.openstack.org/41278 | 23:53 |
| mordred | fungi: that fixes it | 23:53 |
| *** marun has quit IRC | 23:53 | |
| mordred | fungi: although there still isnt' content in openstack/nova there ... | 23:54 |
| mordred | but I think we're getting closer | 23:54 |
| fungi | mordred: well, i haven't told gerrit to force replication yet | 23:55 |
| mordred | ah | 23:56 |
| mordred | fungi: there don't seem to be repos at aall there though? am I just jumping the gun? | 23:56 |
| fungi | mordred: i think the assumption was that gerrit would create them | 23:57 |
| mordred | ah. interesting. I will be surprised if that is the case, because as far as I know, git cannot create a repo remotely | 23:57 |
| mordred | I thought we were going to have a thing which made sure there was an empty repo at /var/lib/git/$f for every repo in projects.yaml? | 23:58 |
| fungi | i just issued the replicate call | 23:58 |
| mordred | cool! | 23:58 |
| fungi | but yeah, doesn't seem to actually have started creating any git repos there | 23:58 |
| mordred | I do not believe that is possible | 23:59 |
| fungi | pleia2's assertion, when i asked that question, was that git ssh mirroring would create ne4w repositories | 23:59 |
| fungi | so i figured it was worth testing at any rate | 23:59 |
| mordred | it is definitely worth testing | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!