palica Posted December 9, 2017 Report Share Posted December 9, 2017 Hi all, I wanted to open a discussion about collecting some anonymous usage data about funtoo kits and maybe some more metrics from your computers. What are your thoughts about it? This is what for example Nextcloud is using and collecting: Quote We take your privacy seriously. The data is anonymized and you can enable/disable it at any time, by default it is always disabled. Below you can also adjust what kind of data is sent and always see the last report sent to us. When the server receives a new report of your instance, all entries from previous reports are removed. If you disable one of the settings below, you can send a new report, which will delete the data that is currently stored on the server. Actual data sent: { "id": "oc9f8tble2uv", "items": [ [ "server", "version", "12.0.3.3" ], [ "php", "version", "7.1.10" ], [ "php", "memory_limit", 536870912 ], [ "php", "max_execution_time", 0 ], [ "php", "upload_max_filesize", 10737418240 ], [ "database", "type", "pgsql" ], [ "database", "version", "PostgreSQL 9.6.4 64-bit" ], [ "database", "size", 82090520 ], [ "apps", "audioplayer", "disabled" ], ... You get the idea. We can discuss what types of metrics could make sense to collect. Let me know what you thing about it. Link to comment Share on other sites More sharing options...
haxmeister Posted December 9, 2017 Report Share Posted December 9, 2017 I think this is a good idea actually. It could improve user oriented focus of development by pointing developers to the most widely used softwares and hardware among fun too users. Link to comment Share on other sites More sharing options...
Funtoo Linux BDFL drobbins Posted December 11, 2017 Funtoo Linux BDFL Report Share Posted December 11, 2017 I like the idea. We can collect usage data on ebuilds and thus know what ebuilds we should prioritize for users. As long as there is an opt-out for people, I think it's ok. We can add this functionality to ego. klib.so 1 Link to comment Share on other sites More sharing options...
captaincrutches Posted December 11, 2017 Report Share Posted December 11, 2017 I'd rather it be opt-in, myself. One of the main reasons for my using Linux over other OS's is that data is specifically NOT sent by default. haxmeister, klib.so and jhan 3 Link to comment Share on other sites More sharing options...
palica Posted December 11, 2017 Author Report Share Posted December 11, 2017 let's discuss what metric should we track? COMPUTER related cpu cpuflags mem swap kernel epro arch subarch flavor mix-ins ego kit enabled non-default kits emerge --info portage version ego version python version gcc version glibc useflags (somehow removing the flags coming from flavor and mix-ins) world and sets entries Link to comment Share on other sites More sharing options...
haxmeister Posted December 12, 2017 Report Share Posted December 12, 2017 I think following extra repositories could be helpful as well. If a mass of users is pulling ebuilds from a repo that is not in main, we could use that to consider adding it to main. Also a simple tool for turning on/off reporting on each item individually to allow for "secret repos" and other such things. I agree with the above that it should be defaulted to "off" and that details about it should be on the installation page so that it's up front from the beginning. These things require transparency which includes probably letting the user know where the source code is located on his/her machine. klib.so 1 Link to comment Share on other sites More sharing options...
haxmeister Posted December 12, 2017 Report Share Posted December 12, 2017 For the record I'm excited about the idea since funtoo is open source, it makes it unoffensive to me. Should this happen, I will install and immediately begin reporting on all my machines (3). Link to comment Share on other sites More sharing options...
palica Posted December 12, 2017 Author Report Share Posted December 12, 2017 So this is how nextcloud is doing it. Allowing individual stats is a good idea. Opt-in is ok, but since the data is anonymous and not containing any secrets I would feel the same as haxmeister and would report from all of my machines. Link to comment Share on other sites More sharing options...
haxmeister Posted December 29, 2017 Report Share Posted December 29, 2017 have you done any work on this or began development of it? Link to comment Share on other sites More sharing options...
klib.so Posted December 30, 2017 Report Share Posted December 30, 2017 I'm not against the idea at all, but these things always start out meaning well using nice terms like metrics and having extremely valuable uses; like delivering an all round better experience, or freeing up devs for better features and so on. However, with the current feeling in the world today this is going to be a much bigger deal than you might originally think, especially considering the type of individual who ends up using a distro such as this—meaning smart not shady. To my mind it absolutely has to be opt-in, and extremely well advertised to the users. That way you send a very clear message of intent, and avoid the situation where user-x suddenly finds out one day, that his favourite distro—the last place he thought was sacred—started spying on him six months ago without his consent. I think in that situation all those extra stats that you gained would walk away with a bad taste in their mouth. After all opt-io is a metric in and of itself. It should also be said that that information is far from being anonymous. It is always the same idea: the information sent to microsoft is anonymised, but completely trivial to fingerprint in a heartbeat. One bug report or forum post including emerge --info and I have little doubt any one of us who loves to configure so much would be uided. I'm not intentionally trolling the idea, I'm highly aware of how important statistics are, but yes I realise I am trolling somewhat just the same :P So what's next? Why not go with the flow and move to systemd? ...and now that I think of it, binary packages would make things a little smoother... :D Link to comment Share on other sites More sharing options...
haxmeister Posted December 30, 2017 Report Share Posted December 30, 2017 It could be simply an ebuild that doesn't come in the stage3. In the installation instructions there could be a section explaining it and giving instructions for installing it with a link to a page describing exactly what it does. The user would not have it on their system at all without deliberately installing it. Having it on the installation page would make them aware that it's out there and gives a place for making the case that it is valuable to funtoo. The statistics captured by the reporting tool could be publicly available on the funtoo website since they are collectively not linkable to any individual. This is the most free, open, and unintrusive scenario. Link to comment Share on other sites More sharing options...
klib.so Posted December 30, 2017 Report Share Posted December 30, 2017 That's the way I was thinking too. Something similar to the old news bulletins. Also a great idea about making them publicly available. Would have the added benefit of letting aspiring contributors see where they might be useful by seeing what the less maintained and less used packages are. Link to comment Share on other sites More sharing options...
haxmeister Posted January 15, 2018 Report Share Posted January 15, 2018 I have this simple output.. this is from my container: { "kernel-info" : { "ostype" : "Linux", "osrelease" : "2.6.32-042stab123.3", "version" : "#1 SMP Sun Jun 4 03:35:16 2017" }, "world-file" : [ "app-admin/eselect", "app-editors/vim", "app-misc/mc", "app-misc/screen", "app-text/tree", "app-text/wgetpaste", "dev-db/phpmyadmin", "dev-lang/php:5.6", "dev-lang/php:7.0", "dev-lang/rakudo", "dev-libs/libmcrypt", "dev-perl/JSON", "dev-perl/Perl-Critic", "dev-perl/Perl-Tidy", "media-gfx/imagemagick", "net-ftp/vsftpd", "net-irc/irssi", "net-misc/dropbox", "sys-process/htop", "virtual/mysql", "virtual/perl6", "www-apps/joomla", "www-client/elinks", "www-servers/apache" ], "mem-info" : { "SwapFree" : "47412 kB", "MemFree" : "3770776 kB", "MemTotal" : "4194304 kB", "SwapTotal" : "262144 kB" }, "boot-dir-info" : { "available kernels" : [] }, "version-info" : { "python versions" : [ "3.4.6", "2.7.13" ], "portage version" : "2.3.18-r1", "ego version" : "2.3.3-r1", "gcc versions" : [ "5.4.0" ] }, "profile_info" : { "ruby-kit" : [ "3.4-prime" ], "core-kit" : [ "x86-64bit", "current", "intel64-westmere", "server", "no-systemd", "3.4-prime" ], "security-kit" : [ "3.4-prime" ], "desktop-kit" : [ "3.4-prime" ], "xorg-kit" : [ "3.4-prime" ], "ml-lang-kit" : [ "3.4-prime" ], "php-kit" : [ "3.4-prime" ], "perl-kit" : [ "3.4-prime" ], "python-kit" : [ "3.4-prime" ], "games-kit" : [ "3.4-prime" ], "science-kit" : [ "3.4-prime" ], "dev-kit" : [ "3.4-prime" ], "editors-kit" : [ "3.4-prime" ], "core-hw-kit" : [ "3.4-prime" ], "lang-kit" : [ "3.4-prime" ], "gnome-kit" : [ "3.4-prime" ], "media-kit" : [ "3.4-prime" ], "kde-kit" : [ "3.4-prime" ], "java-kit" : [ "3.4-prime" ], "net-kit" : [ "3.4-prime" ], "nokit" : [ "3.4-prime" ], "haskell-kit" : [ "3.4-prime" ], "lisp-scheme-kit" : [ "3.4-prime" ], "text-kit" : [ "3.4-prime" ], "xfce-kit" : [ "3.4-prime" ] }, "cpu-info" : { "flags" : [ "fpu", "vme", "de", "pse", "tsc", "msr", "pae", "mce", "cx8", "apic", "sep", "mtrr", "pge", "mca", "cmov", "pat", "pse36", "clflush", "dts", "acpi", "mmx", "fxsr", "sse", "sse2", "ss", "ht", "tm", "pbe", "syscall", "nx", "pdpe1gb", "rdtscp", "lm", "constant_tsc", "arch_perfmon", "pebs", "bts", "rep_good", "xtopology", "nonstop_tsc", "aperfmperf", "pni", "pclmulqdq", "dtes64", "monitor", "ds_cpl", "vmx", "smx", "est", "tm2", "ssse3", "cx16", "xtpr", "pdcm", "pcid", "dca", "sse4_1", "sse4_2", "popcnt", "aes", "lahf_lm", "ida", "arat", "dtherm", "tpr_shadow", "vnmi", "flexpriority", "ept", "vpid" ], "cpu cores" : "6", "model name" : "Intel(R) Xeon(R) CPU X5670 @ 2.93GHz", "cpu MHz" : "2926.192" } } Link to comment Share on other sites More sharing options...
haxmeister Posted January 15, 2018 Report Share Posted January 15, 2018 The config file it uses looks like this : ## Configuration for selecting and deselecting which data ## is reported by the funtoo anonymous reporting tool ## ## All options are defaulted to report, you can change an item ## by altering the "y" and "n" to indicate either yes (y) report it ## or no (n) do not report it. ## # To report cpu info which includes clock speed, model name, # and cpu cores cpu-info:y # To report memory info which includes the amount of free memory, # the amount of memory available, total amount of swap space, # and the amount of free swap space mem-info:y # To report kernel info including O.S. type, release and version kernel-info:y # Allows the reporter to search your /boot directory and list # any kernels it finds # (limited to kernel names that start with "kernel" or "vmlinuz") boot-dir-info:y # To report versions of key softwares on your system including # portage, ego, python, gcc, and glibc version-info:y # To report the contents of /var/lib/portage/world world-info:y # To report profiles information # the same as epro show-json profile-info:y # To report kit versions as reported by ego # extracted from ego kit show kit-info:y Link to comment Share on other sites More sharing options...
palica Posted January 15, 2018 Author Report Share Posted January 15, 2018 looks perfect Link to comment Share on other sites More sharing options...
palica Posted January 15, 2018 Author Report Share Posted January 15, 2018 can you package it? Link to comment Share on other sites More sharing options...
bcowan Posted January 15, 2018 Report Share Posted January 15, 2018 looks cool, all the kits seem to be only showing the python-kit version. Link to comment Share on other sites More sharing options...
haxmeister Posted January 18, 2018 Report Share Posted January 18, 2018 I will work on this more this weekend.. just got over the flu. I'll correct the profile/python thing or remove the listings for all the other kits other than the core kits. Writing the function to get the right version information for those is a bit of an enigma to me right now. Not sure where to get the right information from. I've also begun dabbling with a tool that will hopefully list all global USE flags that are enabled by your profiles, which is also a difficult trail to follow. Link to comment Share on other sites More sharing options...
palica Posted January 18, 2018 Author Report Share Posted January 18, 2018 OK, cool. ego kit list shows all sorts of information. maybe you can just call ego kit for the information about kits. Link to comment Share on other sites More sharing options...
haxmeister Posted January 22, 2018 Report Share Posted January 22, 2018 Ok this is new output from my laptop: { "kernel-info" : { "version" : "#1 SMP Fri Jan 12 08:45:32 CST 2018", "osrelease" : "4.14.12-2", "ostype" : "Linux" }, "version-info" : { "python versions" : [ "3.4.6", "2.7.13-r1" ], "gcc versions" : [ "5.4.0" ], "portage version" : "2.3.18-r1", "ego version" : "2.3.3-r1" }, "cpu-info" : { "cpu cores" : "1", "cpu MHz" : "2194.421", "model name" : "Intel(R) Celeron(R) CPU 900 @ 2.20GHz", "flags" : [ "fpu", "vme", "de", "pse", "tsc", "msr", "pae", "mce", "cx8", "apic", "sep", "mtrr", "pge", "mca", "cmov", "pat", "pse36", "clflush", "dts", "acpi", "mmx", "fxsr", "sse", "sse2", "ss", "tm", "pbe", "syscall", "nx", "lm", "constant_tsc", "arch_perfmon", "pebs", "bts", "rep_good", "nopl", "cpuid", "aperfmperf", "pni", "dtes64", "monitor", "ds_cpl", "tm2", "ssse3", "cx16", "xtpr", "pdcm", "xsave", "lahf_lm", "pti", "dtherm" ] }, "profile-info" : { "mix-ins" : [ { "shortname" : "xfce" }, { "shortname" : "no-systemd" } ], "subarch" : [ { "shortname" : "core2_64" } ], "build" : [ { "shortname" : "current" } ], "arch" : [ { "shortname" : "x86-64bit" } ], "flavor" : [ { "shortname" : "desktop" } ] }, "mem-info" : { "MemTotal" : "3944032 kB", "MemAvailable" : "3043968 kB", "SwapFree" : "2097148 kB", "SwapTotal" : "2097148 kB", "MemFree" : "2739700 kB" }, "world-file" : [ "app-editors/bluefish", "app-editors/gvim", "app-misc/screenfetch", "app-office/libreoffice", "app-office/orage", "app-text/wgetpaste", "dev-db/mysql", "dev-lang/php", "dev-perl/DBD-mysql", "dev-perl/JSON", "dev-perl/JSON-Parse", "dev-perl/Perl-Critic", "dev-perl/Perl-Tidy", "dev-perl/Tk", "dev-perl/XML-Dumper", "dev-util/geany", "games-arcade/frozen-bubble", "games-fps/nexuiz", "games-fps/worldofpadman", "games-strategy/warzone2100", "gnome-extra/nm-applet", "mail-client/thunderbird", "media-gfx/freecad", "media-gfx/gimp", "media-gfx/inkscape", "media-gfx/viewnior", "media-sound/pavucontrol", "media-sound/volumeicon", "net-fs/sshfs", "net-irc/irssi", "net-misc/dropbox", "net-misc/networkmanager", "sys-apps/inxi", "sys-apps/lshw", "sys-boot/boot-update", "sys-boot/grub", "sys-firmware/intel-microcode", "sys-kernel/debian-sources", "sys-kernel/linux-firmware", "sys-process/htop", "virtual/perl6", "www-client/dillo", "www-client/google-chrome", "www-client/lynx", "x11-base/xorg-x11", "x11-misc/lightdm", "x11-misc/menumaker", "x11-terms/xterm", "x11-wm/fluxbox", "xfce-base/xfce4-meta", "xfce-extra/thunar-archive-plugin", "xfce-extra/thunar-dropbox", "xfce-extra/thunar-media-tags-plugin", "xfce-extra/thunar-shares-plugin", "xfce-extra/thunar-vcs-plugin", "xfce-extra/thunar-volman", "xfce-extra/tumbler", "xfce-extra/xfce-theme-manager", "xfce-extra/xfce4-battery-plugin", "xfce-extra/xfce4-gvfs-mount", "xfce-extra/xfce4-pulseaudio-plugin", "xfce-extra/xfce4-screenshooter", "xfce-extra/xfce4-volumed-pulse" ], "kit-info" : { "perl-kit" : "5.24-prime", "xfce-kit" : "4.12-prime", "core-hw-kit" : "master", "python-kit" : "3.6-prime", "text-kit" : "master", "dev-kit" : "1.1-prime", "ruby-kit" : "1.1-prime", "lang-kit" : "1.1-prime", "nokit" : "master", "media-kit" : "1.1-prime", "security-kit" : "1.0-prime", "editors-kit" : "master", "xorg-kit" : "1.19-prime", "haskell-kit" : "1.1-prime", "java-kit" : "1.1-prime", "gnome-kit" : "3.20-prime", "games-kit" : "master", "kde-kit" : "5.10-prime", "php-kit" : "master", "desktop-kit" : "1.1-prime", "net-kit" : "master", "lisp-scheme-kit" : "1.1-prime", "ml-lang-kit" : "1.1-prime", "science-kit" : "master", "core-kit" : "1.0-prime" }, "boot-dir-info" : { "available kernels" : [] } } Also launching from command line without arguments produces this: haxmeister@cel900 ~$ ./reporter-1.3.pl Funtoo anonymous data reporting tool usage: report send "Send the report to funtoo's data collection" report show-json "Show the output that will be sent, in JSON format" report help "Show this help list" Output can be ommitted by modifying the /etc/report.conf file Link to comment Share on other sites More sharing options...
palica Posted January 22, 2018 Author Report Share Posted January 22, 2018 looks perfect, want it :) Link to comment Share on other sites More sharing options...
haxmeister Posted January 23, 2018 Report Share Posted January 23, 2018 so cpu-info is a shady business. I'm thinking maybe we should remove "cpu cores" because that doesn't mean what most people think it means. Instead replace this with "siblings" which should always show the number of threads the CPU can handle at once. Do you think this would be a more pragmatic metric? Link to comment Share on other sites More sharing options...
haxmeister Posted January 23, 2018 Report Share Posted January 23, 2018 https://github.com/haxmeister/funtoo-reporter palica 1 Link to comment Share on other sites More sharing options...
dlc Posted February 1, 2018 Report Share Posted February 1, 2018 I tried funtoo-reporter as an educational endeavor because I approve of the concept and I am curious about its internals. I got show-json to kick out something reasonable but send complains thusly: [Request] ** [http://elk.liguros.net:9200]-[400] [mapper_parsing_exception] object mapping for [profile-info.flavor] tried to parse field [null] as object, but found a concrete value, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at ./report line 40. With vars: {'body' => {'error' => {'type' => 'mapper_parsing_exception','root_cause' => [{'type' => 'mapper_parsing_exception','reason' => 'object mapping for [profile-info.flavor] tried to parse field [null] as object, but found a concrete value'}],'reason' => 'object mapping for [profile-info.flavor] tried to parse field [null] as object, but found a concrete value'},'status' => 400},'status_code' => 400,'request' => {'ignore' => [],'serialize' => 'std','method' => 'POST','qs' => {},'mime_type' => 'application/json','body' => {'profile-info' => {'flavor' => ['desktop'],'arch' => ['x86-64bit'],'subarch' => ['core2_64'],'mix-ins' => ['no-systemd','xfce','mediadevice-audio-pro','X','audio','print'],'build' => ['current']},'timestamp' => 'Thu Feb 1 23:17:21 2018','kernel-info' => {'version' => '#1 SMP Tue Jan 23 04:26:34 -00 2018','osrelease' => '4.14.12-2','ostype' => 'Linux'},'mem-info' => {'MemAvailable' => 5524172,'SwapTotal' => 1048568,'SwapFree' => 1047800,'MemFree' => 3874640,'MemTotal' => 8170200},'cpu-info' => {'flags' => ['fpu','vme','de','pse','tsc','msr','pae','mce','cx8','apic','sep','mtrr','pge','mca','cmov','pat','pse36','clflush','dts','acpi','mmx','fxsr','sse','sse2','ss','ht','tm','pbe','syscall','nx','lm','constant_tsc','arch_perfmon','pebs','bts','rep_good','nopl','cpuid','aperfmperf','pni','dtes64','monitor','ds_cpl','vmx','smx','est','tm2','ssse3','cx16','xtpr','pdcm','sse4_1','xsave','lahf_lm','pti','tpr_shadow','vnmi','flexpriority','dtherm'],'siblings' => 4,'model name' => 'Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz','cpu MHz' => '2000'},'boot-dir-info' => {'available kernels' => ['vmlinuz-4.8.0-53-generic','kernel-debian-sources-x86_64-4.14.12-2','vmlinuz-4.9.6_iso']},'kit-info' => {'python-kit' => '3.4-prime','text-kit' => 'master','games-kit' => 'master','lisp-scheme-kit' => '1.1-prime','desktop-kit' => '1.1-prime','editors-kit' => 'master','php-kit' => 'master','perl-kit' => '5.24-prime','gnome-kit' => '3.20-prime','nokit' => 'master','ml-lang-kit' => '1.1-prime','kde-kit' => '5.10-prime','dev-kit' => '1.1-prime','media-kit' => '1.1-prime','core-hw-kit' => 'master','ruby-kit' => '1.1-prime','xfce-kit' => '4.12-prime','security-kit' => '1.0-prime','java-kit' => '1.1-prime','lang-kit' => '1.1-prime','core-kit' => '1.0-prime','xorg-kit' => '1.19-primt','net-kit' => 'master','science-kit' => 'master','haskell-kit' => '1.1-prime'},'version-info' => {'gcc versions' => ['5.4.0'],'portage version' => '2.3.18-r2','ego version' => '2.3.3-r1','python versions' => ['2.7.13-r1','3.4.6-r1']}},'path' => '/funtoo/user/8f642ea9-9dc2-4a6c-93e3-d6ebb57cd4b1%20'}} FYI. If this isn't the place for this type of feedback, point me in the right direction, please. Link to comment Share on other sites More sharing options...
palica Posted February 2, 2018 Author Report Share Posted February 2, 2018 I think it is ok to discuss here. You can probably directly open an issue on github. Had to disable profile-info in /etc/report.conf and then it reports. I have elasticsearch with kibana running here http://elk.liguros.net there you can check the results or data being submitted. Link to comment Share on other sites More sharing options...
Recommended Posts