Anonymous usage reports for funtoo kits usage

[palica]

Hi all,

I wanted to open a discussion about collecting some anonymous usage data about funtoo kits and maybe some more metrics from your computers. What are your thoughts about it?

This is what for example Nextcloud is using and collecting:

Quote

We take your privacy seriously. The data is anonymized and you can enable/disable it at any time, by default it is always disabled. Below you can also adjust what kind of data is sent and always see the last report sent to us. When the server receives a new report of your instance, all entries from previous reports are removed. If you disable one of the settings below, you can send a new report, which will delete the data that is currently stored on the server.

Actual data sent:

{
    "id": "oc9f8tble2uv",
    "items": [
        [
            "server",
            "version",
            "12.0.3.3"
        ],
        [
            "php",
            "version",
            "7.1.10"
        ],
        [
            "php",
            "memory_limit",
            536870912
        ],
        [
            "php",
            "max_execution_time",
            0
        ],
        [
            "php",
            "upload_max_filesize",
            10737418240
        ],
        [
            "database",
            "type",
            "pgsql"
        ],
        [
            "database",
            "version",
            "PostgreSQL 9.6.4 64-bit"
        ],
        [
            "database",
            "size",
            82090520
        ],
        [
            "apps",
            "audioplayer",
            "disabled"
        ],
...
      

You get the idea.

We can discuss what types of metrics could make sense to collect.

Let me know what you thing about it.

[haxmeister]

I think this is a good idea actually.  It could improve user oriented focus of development by pointing developers to the most widely used softwares and hardware among fun too users.

[drobbins]

I like the idea. We can collect usage data on ebuilds and thus know what ebuilds we should prioritize for users. As long as there is an opt-out for people, I think it's ok. We can add this functionality to ego.

[captaincrutches]

I'd rather it be opt-in, myself.  One of the main reasons for my using Linux over other OS's is that data is specifically NOT sent by default.

[palica]

let's discuss what metric should we track?

1. COMPUTER related
   1. cpu
      1. cpuflags
   2. mem
   3. swap
2. kernel
3. epro
   1. arch
   2. subarch
   3. flavor
   4. mix-ins
4. ego kit
   1. enabled non-default kits
5. emerge --info
   1. portage version
   2. ego version
   3. python version
   4. gcc version
   5. glibc
   6. useflags (somehow removing the flags coming from flavor and mix-ins)
   7. world and sets entries

 

 

[haxmeister]

I think following extra repositories could be helpful as well. If a mass of users is pulling ebuilds from a repo that is not in main, we could use that to consider adding it to main. Also a simple tool for turning on/off reporting on each item individually to allow for "secret repos" and other such things. I agree with the above that it should be defaulted to "off" and that details about it should be on the installation page so that it's up front from the beginning. These things require transparency which includes probably letting the user know where the source code is located on his/her machine.

 

[haxmeister]

For the record I'm excited about the idea since funtoo is open source, it makes it unoffensive to me. Should this happen, I will install and immediately begin reporting on all my machines (3).

[palica]

So this is how nextcloud is doing it. Allowing individual stats is a good idea. Opt-in is ok, but since the data is anonymous and not containing any secrets I would feel the same as haxmeister and would report from all of my machines.

[haxmeister]

have you done any work on this or began development of it?

 

[klib.so]

I'm not against the idea at all, but these things always start out meaning well using nice terms like metrics and having extremely valuable uses; like delivering an all round better experience, or freeing up devs for better features and so on. However, with the current feeling in the world today this is going to be a much bigger deal than you might originally think, especially considering the type of individual who ends up using a distro such as this—meaning smart not shady. To my mind it absolutely has to be opt-in, and extremely well advertised to the users. That way you send a very clear message of intent, and avoid the situation where user-x suddenly finds out one day, that his favourite distro—the last place he thought was sacred—started spying on him six months ago without his consent. I think in that situation all those extra stats that you gained would walk away with a bad taste in their mouth. After all opt-io is a metric in and of itself.

It should also be said that that information is far from being anonymous. It is always the same idea: the information sent to microsoft is anonymised, but completely trivial to fingerprint in a heartbeat. One bug report or forum post including emerge --info and I have little doubt any one of us who loves to configure so much would be uided.

 

I'm not intentionally trolling the idea, I'm highly aware of how important statistics are, but yes I realise I am trolling somewhat just the same :P So what's next? Why not go with the flow and move to systemd? ...and now that I think of it, binary packages would make things a little smoother... :D

[haxmeister]

It could be simply an ebuild that doesn't come in the stage3. In the installation instructions there could be a section explaining it and giving instructions for installing it with a link to a page describing exactly what it does. The user would not have it on their system at all without deliberately installing it. Having it on the installation page would make them aware that it's out there and gives a place for making the case that it is valuable to funtoo. The statistics captured by the reporting tool could be publicly available on the funtoo website since they are collectively not linkable to any individual. This is the most free, open, and unintrusive scenario.

[klib.so]

That's the way I was thinking too. Something similar to the old news bulletins. Also a great idea about making them publicly available. Would have the added benefit of letting aspiring contributors see where they might be useful by seeing what the less maintained and less used packages are.

[haxmeister]

I have this simple output.. this is from my container:

{
   "kernel-info" : {
      "ostype" : "Linux",
      "osrelease" : "2.6.32-042stab123.3",
      "version" : "#1 SMP Sun Jun 4 03:35:16 2017"
   },
   "world-file" : [
      "app-admin/eselect",
      "app-editors/vim",
      "app-misc/mc",
      "app-misc/screen",
      "app-text/tree",
      "app-text/wgetpaste",
      "dev-db/phpmyadmin",
      "dev-lang/php:5.6",
      "dev-lang/php:7.0",
      "dev-lang/rakudo",
      "dev-libs/libmcrypt",
      "dev-perl/JSON",
      "dev-perl/Perl-Critic",
      "dev-perl/Perl-Tidy",
      "media-gfx/imagemagick",
      "net-ftp/vsftpd",
      "net-irc/irssi",
      "net-misc/dropbox",
      "sys-process/htop",
      "virtual/mysql",
      "virtual/perl6",
      "www-apps/joomla",
      "www-client/elinks",
      "www-servers/apache"
   ],
   "mem-info" : {
      "SwapFree" : "47412 kB",
      "MemFree" : "3770776 kB",
      "MemTotal" : "4194304 kB",
      "SwapTotal" : "262144 kB"
   },
   "boot-dir-info" : {
      "available kernels" : []
   },
   "version-info" : {
      "python versions" : [
         "3.4.6",
         "2.7.13"
      ],
      "portage version" : "2.3.18-r1",
      "ego version" : "2.3.3-r1",
      "gcc versions" : [
         "5.4.0"
      ]
   },
   "profile_info" : {
      "ruby-kit" : [
         "3.4-prime"
      ],
      "core-kit" : [
         "x86-64bit",
         "current",
         "intel64-westmere",
         "server",
         "no-systemd",
         "3.4-prime"
      ],
      "security-kit" : [
         "3.4-prime"
      ],
      "desktop-kit" : [
         "3.4-prime"
      ],
      "xorg-kit" : [
         "3.4-prime"
      ],
      "ml-lang-kit" : [
         "3.4-prime"
      ],
      "php-kit" : [
         "3.4-prime"
      ],
      "perl-kit" : [
         "3.4-prime"
      ],
      "python-kit" : [
         "3.4-prime"
      ],
      "games-kit" : [
         "3.4-prime"
      ],
      "science-kit" : [
         "3.4-prime"
      ],
      "dev-kit" : [
         "3.4-prime"
      ],
      "editors-kit" : [
         "3.4-prime"
      ],
      "core-hw-kit" : [
         "3.4-prime"
      ],
      "lang-kit" : [
         "3.4-prime"
      ],
      "gnome-kit" : [
         "3.4-prime"
      ],
      "media-kit" : [
         "3.4-prime"
      ],
      "kde-kit" : [
         "3.4-prime"
      ],
      "java-kit" : [
         "3.4-prime"
      ],
      "net-kit" : [
         "3.4-prime"
      ],
      "nokit" : [
         "3.4-prime"
      ],
      "haskell-kit" : [
         "3.4-prime"
      ],
      "lisp-scheme-kit" : [
         "3.4-prime"
      ],
      "text-kit" : [
         "3.4-prime"
      ],
      "xfce-kit" : [
         "3.4-prime"
      ]
   },
   "cpu-info" : {
      "flags" : [
         "fpu",
         "vme",
         "de",
         "pse",
         "tsc",
         "msr",
         "pae",
         "mce",
         "cx8",
         "apic",
         "sep",
         "mtrr",
         "pge",
         "mca",
         "cmov",
         "pat",
         "pse36",
         "clflush",
         "dts",
         "acpi",
         "mmx",
         "fxsr",
         "sse",
         "sse2",
         "ss",
         "ht",
         "tm",
         "pbe",
         "syscall",
         "nx",
         "pdpe1gb",
         "rdtscp",
         "lm",
         "constant_tsc",
         "arch_perfmon",
         "pebs",
         "bts",
         "rep_good",
         "xtopology",
         "nonstop_tsc",
         "aperfmperf",
         "pni",
         "pclmulqdq",
         "dtes64",
         "monitor",
         "ds_cpl",
         "vmx",
         "smx",
         "est",
         "tm2",
         "ssse3",
         "cx16",
         "xtpr",
         "pdcm",
         "pcid",
         "dca",
         "sse4_1",
         "sse4_2",
         "popcnt",
         "aes",
         "lahf_lm",
         "ida",
         "arat",
         "dtherm",
         "tpr_shadow",
         "vnmi",
         "flexpriority",
         "ept",
         "vpid"
      ],
      "cpu cores" : "6",
      "model name" : "Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz",
      "cpu MHz" : "2926.192"
   }
}

 

[haxmeister]

The config file it uses looks like this :

 

## Configuration for selecting and deselecting which data
## is reported by the funtoo anonymous reporting tool
##
## All options are defaulted to report, you can change an item
## by altering the "y" and "n" to indicate either yes (y) report it
## or no (n) do not report it.
##

# To report cpu info which includes clock speed, model name, 
# and cpu cores
cpu-info:y

# To report memory info which includes the amount of free memory,
# the amount of memory available, total amount of swap space,
# and the amount of free swap space
mem-info:y

# To report kernel info including O.S. type, release and version 
kernel-info:y

# Allows the reporter to search your /boot directory and list 
# any kernels it finds 
# (limited to kernel names that start with "kernel" or "vmlinuz")
boot-dir-info:y

# To report versions of key softwares on your system including
# portage, ego, python, gcc, and glibc
version-info:y

# To report the contents of /var/lib/portage/world
world-info:y

# To report profiles information
# the same as epro show-json
profile-info:y

# To report kit versions as reported by ego
# extracted from ego kit show
kit-info:y

 

[palica]

looks perfect

[palica]

can you package it?

[bcowan]

looks cool, all the kits seem to be only showing the python-kit version.

 

[haxmeister]

I will work on this more this weekend.. just got over the flu. I'll correct the profile/python thing or remove the listings for all the other kits other than the core kits. Writing the function to get the right version information for those is a bit of an enigma to me right now. Not sure where to get the right information from. 

I've also begun dabbling with a tool that will hopefully list all global USE flags that are enabled by your profiles, which is also a difficult trail to follow.

[palica]

OK, cool.

ego kit list shows all sorts of information. maybe you can just call ego kit for the information about kits.

[haxmeister]

Ok this is new output from my laptop:

{
   "kernel-info" : {
      "version" : "#1 SMP Fri Jan 12 08:45:32 CST 2018",
      "osrelease" : "4.14.12-2",
      "ostype" : "Linux"
   },
   "version-info" : {
      "python versions" : [
         "3.4.6",
         "2.7.13-r1"
      ],
      "gcc versions" : [
         "5.4.0"
      ],
      "portage version" : "2.3.18-r1",
      "ego version" : "2.3.3-r1"
   },
   "cpu-info" : {
      "cpu cores" : "1",
      "cpu MHz" : "2194.421",
      "model name" : "Intel(R) Celeron(R) CPU          900  @ 2.20GHz",
      "flags" : [
         "fpu",
         "vme",
         "de",
         "pse",
         "tsc",
         "msr",
         "pae",
         "mce",
         "cx8",
         "apic",
         "sep",
         "mtrr",
         "pge",
         "mca",
         "cmov",
         "pat",
         "pse36",
         "clflush",
         "dts",
         "acpi",
         "mmx",
         "fxsr",
         "sse",
         "sse2",
         "ss",
         "tm",
         "pbe",
         "syscall",
         "nx",
         "lm",
         "constant_tsc",
         "arch_perfmon",
         "pebs",
         "bts",
         "rep_good",
         "nopl",
         "cpuid",
         "aperfmperf",
         "pni",
         "dtes64",
         "monitor",
         "ds_cpl",
         "tm2",
         "ssse3",
         "cx16",
         "xtpr",
         "pdcm",
         "xsave",
         "lahf_lm",
         "pti",
         "dtherm"
      ]
   },
   "profile-info" : {
      "mix-ins" : [
         {
            "shortname" : "xfce"
         },
         {
            "shortname" : "no-systemd"
         }
      ],
      "subarch" : [
         {
            "shortname" : "core2_64"
         }
      ],
      "build" : [
         {
            "shortname" : "current"
         }
      ],
      "arch" : [
         {
            "shortname" : "x86-64bit"
         }
      ],
      "flavor" : [
         {
            "shortname" : "desktop"
         }
      ]
   },
   "mem-info" : {
      "MemTotal" : "3944032 kB",
      "MemAvailable" : "3043968 kB",
      "SwapFree" : "2097148 kB",
      "SwapTotal" : "2097148 kB",
      "MemFree" : "2739700 kB"
   },
   "world-file" : [
      "app-editors/bluefish",
      "app-editors/gvim",
      "app-misc/screenfetch",
      "app-office/libreoffice",
      "app-office/orage",
      "app-text/wgetpaste",
      "dev-db/mysql",
      "dev-lang/php",
      "dev-perl/DBD-mysql",
      "dev-perl/JSON",
      "dev-perl/JSON-Parse",
      "dev-perl/Perl-Critic",
      "dev-perl/Perl-Tidy",
      "dev-perl/Tk",
      "dev-perl/XML-Dumper",
      "dev-util/geany",
      "games-arcade/frozen-bubble",
      "games-fps/nexuiz",
      "games-fps/worldofpadman",
      "games-strategy/warzone2100",
      "gnome-extra/nm-applet",
      "mail-client/thunderbird",
      "media-gfx/freecad",
      "media-gfx/gimp",
      "media-gfx/inkscape",
      "media-gfx/viewnior",
      "media-sound/pavucontrol",
      "media-sound/volumeicon",
      "net-fs/sshfs",
      "net-irc/irssi",
      "net-misc/dropbox",
      "net-misc/networkmanager",
      "sys-apps/inxi",
      "sys-apps/lshw",
      "sys-boot/boot-update",
      "sys-boot/grub",
      "sys-firmware/intel-microcode",
      "sys-kernel/debian-sources",
      "sys-kernel/linux-firmware",
      "sys-process/htop",
      "virtual/perl6",
      "www-client/dillo",
      "www-client/google-chrome",
      "www-client/lynx",
      "x11-base/xorg-x11",
      "x11-misc/lightdm",
      "x11-misc/menumaker",
      "x11-terms/xterm",
      "x11-wm/fluxbox",
      "xfce-base/xfce4-meta",
      "xfce-extra/thunar-archive-plugin",
      "xfce-extra/thunar-dropbox",
      "xfce-extra/thunar-media-tags-plugin",
      "xfce-extra/thunar-shares-plugin",
      "xfce-extra/thunar-vcs-plugin",
      "xfce-extra/thunar-volman",
      "xfce-extra/tumbler",
      "xfce-extra/xfce-theme-manager",
      "xfce-extra/xfce4-battery-plugin",
      "xfce-extra/xfce4-gvfs-mount",
      "xfce-extra/xfce4-pulseaudio-plugin",
      "xfce-extra/xfce4-screenshooter",
      "xfce-extra/xfce4-volumed-pulse"
   ],
   "kit-info" : {
      "perl-kit" : "5.24-prime",
      "xfce-kit" : "4.12-prime",
      "core-hw-kit" : "master",
      "python-kit" : "3.6-prime",
      "text-kit" : "master",
      "dev-kit" : "1.1-prime",
      "ruby-kit" : "1.1-prime",
      "lang-kit" : "1.1-prime",
      "nokit" : "master",
      "media-kit" : "1.1-prime",
      "security-kit" : "1.0-prime",
      "editors-kit" : "master",
      "xorg-kit" : "1.19-prime",
      "haskell-kit" : "1.1-prime",
      "java-kit" : "1.1-prime",
      "gnome-kit" : "3.20-prime",
      "games-kit" : "master",
      "kde-kit" : "5.10-prime",
      "php-kit" : "master",
      "desktop-kit" : "1.1-prime",
      "net-kit" : "master",
      "lisp-scheme-kit" : "1.1-prime",
      "ml-lang-kit" : "1.1-prime",
      "science-kit" : "master",
      "core-kit" : "1.0-prime"
   },
   "boot-dir-info" : {
      "available kernels" : []
   }
}

Also launching from command line without arguments produces this:

haxmeister@cel900 ~$ ./reporter-1.3.pl 
Funtoo anonymous data reporting tool usage: 

report send         "Send the report to funtoo's data collection"
report show-json      "Show the output that will be sent, in JSON format"
report help          "Show this help list"

Output can be ommitted by modifying the /etc/report.conf file 
 

[palica]

looks perfect, want it :)

[haxmeister]

so cpu-info is a shady business. I'm thinking maybe we should remove "cpu cores" because that doesn't mean what most people think it means. Instead replace this with "siblings" which should always show the number of threads the CPU can handle at once. Do you think this would be a more pragmatic metric?

 

[haxmeister]

https://github.com/haxmeister/funtoo-reporter

[dlc]

I tried funtoo-reporter as an educational endeavor because I approve of the concept and I am curious about its internals.  I got show-json to kick out something reasonable but send complains thusly:

[Request] ** [http://elk.liguros.net:9200]-[400] [mapper_parsing_exception] object mapping for [profile-info.flavor] tried to parse field [null] as object, but found a concrete value, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at ./report line 40. With vars: {'body' => {'error' => {'type' => 'mapper_parsing_exception','root_cause' => [{'type' => 'mapper_parsing_exception','reason' => 'object mapping for [profile-info.flavor] tried to parse field [null] as object, but found a concrete value'}],'reason' => 'object mapping for [profile-info.flavor] tried to parse field [null] as object, but found a concrete value'},'status' => 400},'status_code' => 400,'request' => {'ignore' => [],'serialize' => 'std','method' => 'POST','qs' => {},'mime_type' => 'application/json','body' => {'profile-info' => {'flavor' => ['desktop'],'arch' => ['x86-64bit'],'subarch' => ['core2_64'],'mix-ins' => ['no-systemd','xfce','mediadevice-audio-pro','X','audio','print'],'build' => ['current']},'timestamp' => 'Thu Feb  1 23:17:21 2018','kernel-info' => {'version' => '#1 SMP Tue Jan 23 04:26:34 -00 2018','osrelease' => '4.14.12-2','ostype' => 'Linux'},'mem-info' => {'MemAvailable' => 5524172,'SwapTotal' => 1048568,'SwapFree' => 1047800,'MemFree' => 3874640,'MemTotal' => 8170200},'cpu-info' => {'flags' => ['fpu','vme','de','pse','tsc','msr','pae','mce','cx8','apic','sep','mtrr','pge','mca','cmov','pat','pse36','clflush','dts','acpi','mmx','fxsr','sse','sse2','ss','ht','tm','pbe','syscall','nx','lm','constant_tsc','arch_perfmon','pebs','bts','rep_good','nopl','cpuid','aperfmperf','pni','dtes64','monitor','ds_cpl','vmx','smx','est','tm2','ssse3','cx16','xtpr','pdcm','sse4_1','xsave','lahf_lm','pti','tpr_shadow','vnmi','flexpriority','dtherm'],'siblings' => 4,'model name' => 'Intel(R) Core(TM)2 Quad CPU    Q9550  @ 2.83GHz','cpu MHz' => '2000'},'boot-dir-info' => {'available kernels' => ['vmlinuz-4.8.0-53-generic','kernel-debian-sources-x86_64-4.14.12-2','vmlinuz-4.9.6_iso']},'kit-info' => {'python-kit' => '3.4-prime','text-kit' => 'master','games-kit' => 'master','lisp-scheme-kit' => '1.1-prime','desktop-kit' => '1.1-prime','editors-kit' => 'master','php-kit' => 'master','perl-kit' => '5.24-prime','gnome-kit' => '3.20-prime','nokit' => 'master','ml-lang-kit' => '1.1-prime','kde-kit' => '5.10-prime','dev-kit' => '1.1-prime','media-kit' => '1.1-prime','core-hw-kit' => 'master','ruby-kit' => '1.1-prime','xfce-kit' => '4.12-prime','security-kit' => '1.0-prime','java-kit' => '1.1-prime','lang-kit' => '1.1-prime','core-kit' => '1.0-prime','xorg-kit' => '1.19-primt','net-kit' => 'master','science-kit' => 'master','haskell-kit' => '1.1-prime'},'version-info' => {'gcc versions' => ['5.4.0'],'portage version' => '2.3.18-r2','ego version' => '2.3.3-r1','python versions' => ['2.7.13-r1','3.4.6-r1']}},'path' => '/funtoo/user/8f642ea9-9dc2-4a6c-93e3-d6ebb57cd4b1%20'}}

FYI.  If this isn't the place for this type of feedback, point me in the right direction, please.

[palica]

I think it is ok to discuss here. You can probably directly open an issue on github. Had to disable profile-info in /etc/report.conf and then it reports.

I have elasticsearch with kibana running here

http://elk.liguros.net

there you can check the results or data being submitted.