Monday, 2017-03-13

*** Guest45618 has joined #ara00:34
*** Guest45618 has quit IRC00:41
*** karimb has joined #ara07:42
*** Maximus86 has joined #ara08:49
Maximus86Hi all08:49
Maximus86I'm playing around with the new ara version08:50
Maximus86looking at the webapp, I received some weird errors08:51
Maximus86"DataTables warning: table id=stats_5fda38fb-b26d-4dc6-bada-ab604371c199 - Ajax error. For more information about this error, please see http://datatables.net/tn/7"08:51
Maximus86..combined with some stacktraces in my apache error log08:52
Maximus86"[Mon Mar 13 06:46:13.585698 2017] [:error] [pid 7682] [remote 192.168.7.127:80] KeyError: '004_ansible_version_py'"08:52
Maximus86(several key errors actually)08:52
Maximus86what can / should I do? Shall I report a bug? If you'd like to know more, I'll stay on this chat for the coming 6 hours or so08:53
Maximus86This is after upgrading from a previous version btw08:53
*** jclaret has joined #ara09:13
*** permalac has quit IRC09:29
*** permalac has joined #ara09:35
*** jclaret has quit IRC09:58
*** evrardjp has joined #ara11:08
dmsimardMaximus86: hmm, sounds like a sql migration that wasn't done properly11:37
dmsimardMaximus86: did you restart apache after upgrading ?11:37
*** sebie01 has joined #ara11:38
Maximus86I did11:59
Maximus86I did have an issue installing crypto with pip and had to install a dep12:00
Maximus86after which I reran the pip install --upgrade  command12:00
Maximus86could it be the migration scripts were not triggered?12:00
*** sebie01 has quit IRC12:17
dmsimardMaximus86: could you do a select * from alembic_version in the database ?12:20
Maximus86MariaDB [ara]> select * from alembic_version \G13:33
Maximus86*************************** 1. row ***************************13:33
Maximus86version_num: 22aa8072d70513:33
Maximus86using this on MariaDB 10.1.20-1 (Centos 7.2)13:34
*** DoWhatNow has joined #ara14:18
DoWhatNowMorning all.  Could use a bit of help with a new install, red hat enviroment.14:19
DoWhatNowHere is the error "ara generate html /var/www/html/ara/14:19
DoWhatNowGenerating static files at /var/www/html/ara/...14:19
DoWhatNow  freezer.freeze()14:19
DoWhatNowDone.14:19
DoWhatNow"14:19
*** leifmadsen has joined #ara14:48
Maximus86encountering a different issue after performing an upgrade of ara on my ubuntu workstation14:54
Maximus86meanwhile was able to reproduce the previous issue14:54
Maximus86sent a PM to dmsimard with details on how to contact me, so I can send logs14:55
*** Maximus86 has left #ara14:58
leifmadsenbefore I build my own, there isn't an ARA container image already being built is there?15:05
dmsimardDoWhatNow: hey, sorry was away -- that doesn't look like an error to me, what's up ?15:45
dmsimardleifmadsen: there isn't an "upstream" or "official" one15:45
dmsimardleifmadsen: there is this gentleman that recently did something using docker compose, I think he eventually wanted to write about it https://twitter.com/chmod666/status/83990538248716288215:46
DoWhatNowSorry, seems  the one relevant line didnt copy.15:50
DoWhatNow  freezer.freeze()15:50
DoWhatNowirc seems to be kicking it...  im gonna break it up a bit.15:51
DoWhatNow "/usr/lib/python2.7/site-packages/ara/cli/generate.py:46: MissingURLGeneratorWarning: Nothing frozen for endpoints reports.ajax_results, reports.ajax_records, file.show_file, "15:51
DoWhatNow"reports.ajax_stats, reports.ajax_plays, result.show_result, reports.ajax_files, host.show_host. Did you forget a URL generator? freezer.freeze()"15:52
DoWhatNowThere we go.15:52
dmsimardDoWhatNow: looks like the database would be empty -- did you run a playbook with the callback configured ?15:52
DoWhatNowyep,  let me log into my db and make sure im atleast close to sane here...15:53
dmsimardwhen you use the generate command, you also need to be have the necessary configuration to point to that database -- env variables or ansible.cfg15:54
dmsimardbrb15:54
DoWhatNowdatabase is populated with15:55
DoWhatNowseveral tables in ara i didnt put there, doing a select * from hosts ; is showind all the hosts i had in the playbook so it seems to have populated well.15:56
leifmadsendmsimard: ok cool, I might contribute a Dockerfile to the repo then15:56
leifmadsenI'll see if I can do that today15:57
leifmadsen(I'm working on breaking out ARA from our jenkins master in TOAD)15:57
DoWhatNow echo $ARA_DATABASE16:01
DoWhatNowmysql+pymysql://ara:password@localhost/ara16:01
karimbdmsimard can i ask you how you set this patternfly thing, really installing npm, bower and all those things ?16:20
karimbdoesnt make too much sense for me, installing bootstrap is just a matter of referencing two urls...16:21
dmsimardkarimb: ara currently bundles the necessary compiled files from patternfly which is a few css files, a few js files and a few font files16:32
karimbdmsimard yeah, but how did you come up with this minimal list ?16:32
dmsimardkarimb: I took only what I needed ? I'm not sure I understand the question16:33
dmsimardDoWhatNow: ok so you have a database, you have data in it, great. When you do the generate command, you also have the database configured ?16:34
DoWhatNowI believe i do,  echo $ARA_DATABASE dumps mysql+pymysql://ara:password@localhost/ara16:35
dmsimardDoWhatNow: what if you do "ara playbook list" does that work ?16:36
DoWhatNowYes,  looks like a sql table,  shows a single run.16:37
dmsimardif you start the development server and browse to it, does it work ?16:38
dmsimard"ara-manage runserver"16:38
DoWhatNowIt works, but it tells me when i run a playbook i will be able to see the contednt.16:38
dmsimardThat's odd. Do you think you could send me or let me download your database so I can have a look ?16:42
dmsimardUnless there's confidential stuff in there or something, of course16:42
dmsimardOtherwise what would probably be best is to have a way to reproduce the problem16:43
DoWhatNowsomewhat... let me drop it and then repopulate with a smaller dataset.16:45
dmsimardDoWhatNow: if it's reproducible every time, just let me know how I can do it and I'll reproduce it on my end16:45
karimbdmsimard forget it, it s just that i havent done web devellopment in way too long.... feels like a nightmare :)17:09
leifmadsendmsimard: am I using ARA_SQL_DEBUG wrong, or maybe I've found a bug...17:09
dmsimardkarimb: I'm not a frontend dev either, this is arcane :)17:09
leifmadsen(incoming pastebin)17:09
leifmadsenhttps://paste.fedoraproject.org/paste/h7vXmyG6YQf-nl5tzOweHF5M1UNdIGYhyRLivL9gydE=17:10
karimbdmsimard i remember suffering with jquery and ajax stuff17:10
dmsimardleifmadsen: looking in a few17:10
leifmadsenkk17:10
leifmadsenoh I think I know what I did wrong17:11
leifmadsenthis is probably a layer 8 problem, let you know in a minute17:12
DoWhatNowdmsimard:  I have a dump available for you . PM sent.17:13
DoWhatNowI tried adding in the "sqldebug = True" and  it broke ansible.17:15
DoWhatNow  File "/usr/lib64/python2.7/site-packages/sqlalchemy/log.py", line 94, in __init__17:17
DoWhatNow    if self._echo_map[echo] <= logging.INFO \17:17
DoWhatNowKeyError: 'True'17:17
DoWhatNowIs there a prefered SQLAlchemy version ?17:22
dmsimardDoWhatNow: whatever is latest should be fine, I replied to your PM btw17:23
-openstackstatus- NOTICE: restarting gerrit to address performance problems17:43
dmsimardDoWhatNow: hm, I'm not able to reproduce on a fresh install of centos7 with mariadb 10.1.2017:48
dmsimardI don't have a fedora vm nearby so maybe there's something else I'm missing17:48
dmsimardboth the webui and the generation works17:49
dmsimardDoWhatNow: quite literally the history of the VM: http://paste.openstack.org/show/602540/17:50
dmsimard(centos-release-openstack-ocata is just me cheating to get mariadb 10 on centos)17:51
dmsimardI added a "use ara;" at the top of the dump to import it, otherwise just copy/pasted the database creation procedure from the docs17:52
DoWhatNowdoes "ara playbook lis" access the DB using the same method that "ara generate html" does ?17:54
leifmadsendmsimard: ok so that's interesting... if I specify ARA_DATABASE and run "ara --help"17:54
leifmadsenit can't seem to figure out how to create the database17:54
leifmadsenbut if that ENV isn't defined, it'll create it in the same path no problem17:54
dmsimardleifmadsen: so the application will sort of aggressively attempt to bootstrap itself, even if just running ara --help.. I got rid of some of that but there's still some bits to take out to make sure it doesn't initialize needlesslyt17:55
dmsimardDoWhatNow: everything uses the same configuration mechanism, the callback, the CLI commands, the web application17:56
leifmadsenyea, I basically expected it to initialize the sqlite DB in the location I specified in ARA_DATABASE but it seems if I define that it won't do the bootstraping17:56
dmsimardleifmadsen: it doesn't bootstrap for every command (well, it used to, but I want to fix it so it only initializes when necessary) .. if you do an "ara playbook list", it should initialize the database and directories17:57
dmsimardjust have to be careful, the user using the application needs read and write access to the database location17:58
dmsimardsaw a user a bit confused around permissions when using apache17:58
leifmadsendmsimard: yea, what i'm saying is... is if ARA_DATABASE is defined, it won't bootstrap17:58
leifmadsenso python bombs into a traceroute17:59
dmsimardleifmadsen: haven't seen that happen before, can you show me how to reproduce ?17:59
leifmadsenif I unset ARA_DATABASE it'll bootstrap fine17:59
leifmadsendmsimard: very easy :)17:59
leifmadsendelete ansible.sqlite file17:59
leifmadsen* export ARA_DATABASE=sqlite:///data/ansible.sqlite17:59
leifmadsen* ara --help17:59
leifmadsen(or anything)18:00
* dmsimard looks18:00
leifmadsenbasically, it seems to skip the bootstrap methods when the envvar is defined18:00
leifmadsenat least that's what it looks like18:00
leifmadsenthis in in master fyi18:00
dmsimardleifmadsen: wfm -- I think you're missing an extra slash in there18:01
dmsimard(yes, it's a lot of slashes)18:01
leifmadsenshould be 3 right?18:01
dmsimardleifmadsen: http://paste.openstack.org/show/602541/18:01
dmsimard4 ?18:01
leifmadsenoh... hrm18:01
leifmadsenlet me try!18:01
leifmadsenoh well then18:02
dmsimardleifmadsen: If I just put three slashes, I get sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file18:02
leifmadsenyep18:02
leifmadsenok, so reproduced... layer 8 issue it seems :)18:02
dmsimard¯\_(ツ)_/¯18:02
leifmadsenwith 4 slashes it works for me too18:02
leifmadsenI thought URI was just sqlite://18:03
leifmadsenbut I guess its.... :///18:03
dmsimardyeah it's a very awkward smilie18:03
dmsimardlike :/18:03
dmsimardbut :///18:03
leifmadsen:D18:03
DoWhatNowthe url "sqlite:///~/.ara/ansible.sqlite" refrenced on treadthedocs gives me the errot: OperationalError: (sqlite3.OperationalError) unable to open database file18:03
dmsimardDoWhatNow: that's sort of a documentation-specific thing, it gets expanded in python18:04
dmsimardDoWhatNow: the "default" value is ~/.ara/ansible.sqlite but that gets expanded18:04
dmsimardI'm not sure sqlite knows how to expand that18:04
dmsimardDoWhatNow: doing the same in bash could be like: export ARA_DATABASE="sqlite:///${HOME}/.ara/ansible.sqlite"18:05
dmsimardif you check "env |grep ARA" after that, the HOME should've been expanded18:05
DoWhatNowYep.   but it looks from the readthedocs page that the config gile should contain the lines "dir = ~/.ara18:09
DoWhatNowdatabase = sqlite:///~/.ara/ansible.sqlite"18:09
dmsimardDoWhatNow: I should probably make that clearer18:10
DoWhatNowSo i killed my MariaDB. and moved over to the sqlite.   Reran a playbook to create the sqlite3 db,  tried to generate the html and /usr/lib/python2.7/site-packages/ara/cli/generate.py:46: MissingURLGeneratorWarning: Nothing frozen for endpoints18:16
dmsimardDoWhatNow: that's so weird :(18:17
dmsimardif "ara playbook list" works and shows your playbook, there is no reason at all why "ara generate" wouldn't work -- they all feed on the same configuration and database18:18
DoWhatNowFlask (0.11.1), Flask-Migrate (2.0.3), Flask-Script (2.0.5), Flask-SQLAlchemy (2.2), Frozen-Flask (0.13),   These all seem like good versions to you ?18:21
dmsimardDoWhatNow: yeah lgtm18:23
DoWhatNowWell i am about out of ideas.18:28
dmsimardDoWhatNow: you're doing configuration through env variables ?18:31
DoWhatNowNo,  ansible.cfg  I tried  once to  sue the env instead to see if that would help but no change.  I have unset all that i set.18:31
dmsimardDoWhatNow: did you ever set up any override ? i.e, ARA_PLAYBOOK_OVERRIDE ?18:32
DoWhatNowno.18:33
DoWhatNowplaybook_override = None18:33
DoWhatNow What are optopns for "loglevel" read the docs does not specify ?18:34
DoWhatNowI think i got it.  Looks like the playbook_override = None in the ansible.cfg was killing me.18:36
dmsimardDoWhatNow: oh hey at least we found it18:37
dmsimardloglevel are the usual python logging keywords so "INFO" "WARN" "ERROR" and "DEBUG" iirc18:38
dmsimardhowever we're not doing a really good job at logging right now, we need to improve that18:38
DoWhatNowSo it seems i misunderstood the documentation.  I made aan entry into ansible.cfg for each variable,  and specified its Default.   and this should not be done.18:38
DoWhatNowdmsimard: Thanks for all your help18:39
dmsimardDoWhatNow: ah, yeah, definitely just configure the things you need to override the default for18:39
dmsimardDon't need to specify the default, the defaults are embedded and assumed in ARA :)18:40
DoWhatNowyep.18:41
dmsimardSo anyway for context, ARA_PLAYBOOK_OVERRIDE is a way to restrict the playbooks shown in the web UI18:42
dmsimardYou can use it yourself but it's mostly an under-the-hood parameter that's passed from the "ara generate html" command -- it's to generate a report that would only contain one or many different specified playbooks18:42
dmsimardIt's useful if you have a centralized/aggregated instance and want to generate a static report for a specific playbook18:42
DoWhatNowYeah,, I see how that could be usefull.   Ohh i am loving this.18:43
leifmadsenARA ftw18:44
dmsimardleifmadsen: did you know we were getting stickers? :)18:44
leifmadsendmsimard: ARA_SQL_DEBUG values would be "False" or "True" ?18:44
leifmadsendmsimard: I did not :)  my wife could print them :)18:44
leifmadsenI got her to make me stickers for Browbeat18:44
leifmadsenI need to get some TOAD ones made up I suppose heh18:45
dmsimardleifmadsen: ah, getting them printed by stickermule https://twitter.com/dmsimard/status/83963814346060595218:45
leifmadsenah nice nice!18:45
dmsimardI haven't used ARA_SQL_DEBUG in a long, long time, let me look18:45
leifmadsenI think it might hate me18:45
leifmadsen:)18:45
leifmadsenwhen I set it, the bootstrap fails18:45
leifmadsenif I unset it, then it's fine18:45
leifmadsen(sorry for all the weird issues... I'm working on the Dockerfile :))18:46
dmsimardleifmadsen: it defaults to "False" i.e, https://github.com/openstack/ara/blob/master/ara/config.py#L2318:46
leifmadsenhttps://paste.fedoraproject.org/paste/HWYirQRAipovVjOes1xQpV5M1UNdIGYhyRLivL9gydE=18:46
dmsimardI have an idea of what the problem is18:46
leifmadsenif I set ARA_SQL_DEBUG=False in Dockerfile, it gives me that when I run "ara --help"18:46
dmsimardleifmadsen: let me see18:46
leifmadsencool18:46
dmsimardyeah18:47
dmsimardlet me test a fix18:47
leifmadsenrock18:48
dmsimardleifmadsen: fixed18:50
leifmadsendmsimard: pulling18:50
leifmadsendmsimard: btw remote should be openstack/ara ?18:51
leifmadsenin github?18:51
*** openstackgerrit has joined #ara18:52
openstackgerritDavid Moreau Simard proposed openstack/ara master: Properly use boolean Ansible types in configuration parameters  https://review.openstack.org/44505018:52
dmsimardleifmadsen: ^18:52
leifmadsenahh18:52
leifmadsenlet me patch18:52
dmsimardgithub.com/openstack/ara18:52
leifmadsenok cool18:52
dmsimardgithub.com/dmsimard/ara is sort of mirrored manually for historical purposes18:52
dmsimardSince it's original upstream I don't really want to delete it18:52
leifmadsenmakes sense18:52
dmsimardopenstack-infra creates the repo, it doesn't move it18:53
dmsimardleifmadsen: so anyway that patch tl;dr is18:53
leifmadsenyea I looked at it18:53
dmsimardoverriding the default would be a string, not an actual bool18:53
leifmadsenmakes sense18:53
leifmadsengotcha18:53
leifmadsendmsimard: should have a Dockerfile soon for review18:56
leifmadseninterested to see if you like my approach :)18:56
dmsimardleifmadsen: that guy I linked on twitter with docker-compose thing18:57
leifmadsenoh right, I need to check that18:57
dmsimardhe had some issues getting pymysql to work for some reason18:57
dmsimardI think he ended up going with mysql-python driver instead18:57
dmsimardbut then again, he ran on ppc64le so it might be that18:57
leifmadsenyea I was just noticing that18:57
dmsimardtold him to get back to me if he ended up figuring out18:58
leifmadsenpatch works!18:59
dmsimard\o/19:00
leifmadsen+1'd19:00
dmsimardnow what did you need super verbose sql alchemy for ?19:00
leifmadsenI didn't :)19:00
dmsimardoh okay19:00
dmsimardI mean finding and fixing bugs is cool too19:00
leifmadsenI'm just setting the default ENV values19:00
leifmadsenI set the default to False, I think I had just been trying values, or mistyped True for False :)19:00
leifmadsenthe truthy false is never wrong19:01
dmsimardThere's no big features in the pipe right now, kind of worn myself out with 0.11 and 0.12 -- big, big releases. Planning to just do stabilization/bugfixes for a bit :)19:02
leifmadsenthat's a good plan19:02
leifmadsenyou gotta have stabilization releases19:03
leifmadsenalways a fan of those19:03
openstackgerritLeif Madsen proposed openstack/ara master: [WIP] Add Dockerfile for building ARA  https://review.openstack.org/44505819:05
dmsimardthe python3 base image has all the other package requirements in it ?19:08
dmsimardlike openssl and xml and xslt and junk19:08
dmsimardalso ara is not full python3 yet :(19:08
openstackgerritMerged openstack/ara master: Properly use boolean Ansible types in configuration parameters  https://review.openstack.org/44505019:08
leifmadsendmsimard: it seems to build fine with that package19:09
leifmadsenI can switch to python:2 if you think that would be better19:09
dmsimardleifmadsen: yeah, unit tests aren't all passing on py3 yet, not really confident19:09
leifmadsenI need to integrate this into TOAD now via our docker-compose and see if I can use it all19:09
leifmadsendmsimard: ok np, easy fix19:09
dmsimardthe py3 base image, is it based off of ubuntu? :)19:09
leifmadsenprobably...19:09
leifmadsenI didn't look19:10
openstackgerritLeif Madsen proposed openstack/ara master: [WIP] Add Dockerfile for building ARA  https://review.openstack.org/44505819:10
dmsimardleifmadsen: debian :)19:10
leifmadsenthe worst19:11
dmsimardat least looking at https://hub.docker.com/_/python/19:11
dmsimard"3" currently points to https://github.com/docker-library/python/blob/7eca63adca38729424a9bab957f006f5caad870f/3.6/Dockerfile19:11
dmsimardIt installs python from source19:12
dmsimardwoah19:12
leifmadsenI mean, the alternative is I can switch to FROM centos:7 and install from packages19:16
leifmadsenhappy to do that19:16
dmsimardwell then you'd probably have to do an ubuntu-specific one :P19:18
dmsimardI mean ara is distro agnostic for the most part19:18
leifmadsenpffft ubuntu :)19:18
dmsimardlet's keep it simple for now, just use the python:2 one19:19
leifmadsen+119:19
dmsimardand we can reconsider later if there's demand19:19
dmsimardara is all about simplicity19:19
leifmadsen+119:19
*** DoWhatNow has quit IRC19:23
openstackgerritLeif Madsen proposed openstack/ara master: [WIP] Add Dockerfile for building ARA  https://review.openstack.org/44505819:28
dmsimardleifmadsen: what clones ara in your thing ?19:33
dmsimarddocker magic ?19:33
leifmadsennothing, just does an ADD from repo19:33
leifmadsenthat way you can build for dev purposes19:34
leifmadsensee line 1819:34
dmsimardah so it's expected that you'd clone ara and then use that dockerfile19:34
leifmadsenyea for now19:34
leifmadsenwe can probably create some sort of build in hub as well19:34
leifmadsenactually19:35
leifmadsenhrmmm19:35
dmsimardI wonder if it's better to do just "pip install ." instead of pip install -r requirements and then setup.py install19:35
leifmadsenknow what I should do, is make that an ENV19:35
leifmadsenmaybe, I'm not a python guru, so I used what I knew :D19:35
leifmadsenthere are likely better ways! :)19:35
dmsimardfrom inside the git repo, you can just do pip install .19:35
dmsimardMaybe some sort of toggle to install from pypi vs source19:37
leifmadsenhrmmm yea19:37
dmsimardand then do the necessary logic in the dockerfile ?19:37
dmsimardI don't know, I'm a docker noob too.19:37
leifmadsenall good, I'll figure something out, but I agree19:38
leifmadsenI'll bother dougbtv who is our docker guru :)19:38
dmsimardI have faith in him :D19:38
leifmadsen:D19:39
dmsimardalso make sure to test some features like junit export19:39
leifmadsenhe clued me into a few things already on this Dockerfile19:39
dmsimardI expect you'll need to install some openssl/xml/xslt libs19:39
leifmadsendmsimard: this is starting to sound like I should add some CI tests...19:39
leifmadsenlol19:39
dmsimardleifmadsen: feel free to hijack run_tests.sh for test purposes within your patch19:39
dmsimardwe can work something out later19:39
leifmadsenjunit export?19:40
leifmadsenI don't see that command....19:40
leifmadsenoh generate junit export ?19:40
dmsimardYeah19:41
leifmadsennice, seems to work already19:42
leifmadsenI haven't run it against a playbook yet though19:42
leifmadsenbut it does generate a mostly empty xml file19:42
leifmadsen4 lines19:42
dmsimard¯\_(ツ)_/¯19:52
leifmadsendoesn't freak out and give me a trace file :)19:55
leifmadsenwining!19:55
dmsimardleifmadsen: yeah it's sort of weird actually, because even ansible requires stuff like paramiko which needs some crypto libs and those usually require bindings from python-devel or libopenssl and such19:59
dmsimardI'm surprised it works out of the box19:59
leifmadsensame lol19:59
dmsimardleifmadsen: ah well there you go20:00
dmsimardleifmadsen: the python base image pulls from https://github.com/docker-library/buildpack-deps/blob/587934fb063d770d0611e94b57c9dd7a38edf928/jessie/Dockerfile20:00
dmsimardDOCKER MAGIC20:00
leifmadsenlol20:01
dmsimardfwiw purpose-built distro containers would probably be smaller and contain less fat but whatever20:01
leifmadsenyea, I just uploaded one for testing, and it was "only" 301MB20:01
leifmadsenbut I think that is the post-FROM size20:02
leifmadsenyea, on my machine it's more like 771.8MB :)20:02
dmsimardyeah that's probably just that one layer20:02
leifmadsenyep20:02
leifmadsenah, yea python image is 688MB20:02
dmsimardjesus20:02
leifmadsencentos is like 192MB20:02
leifmadsenI might see how much diff there is if I make it centos based...20:03
leifmadseneither way, you're biased towards debian or centos pretty much :)20:03
leifmadsenunless you use alpine20:03
dmsimardwe're in the process of doing an ansible role to install and set up ara20:04
dmsimardplanning to support fedora/centos/rhel and debian/ubuntu20:04
leifmadsenah nice20:04
dmsimardsafe to assume that's the majority of the market with the least effort involved20:04
leifmadsen+120:04
dmsimarddiminushing returns after that ...20:04
dmsimardbsd, osx, arch, gentoo, etc :p20:05
*** karimb has quit IRC23:42

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!