digifuzzy Posted December 12, 2014 Report Share Posted December 12, 2014 Looking into keep my bandwidth usage down, I found references to setting up rsync daemon in gentoo. The idea to have a local mirror is quite appealling. Since Funtoo uses git, is there an equivalent set up/process? Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 14, 2014 Author Report Share Posted December 14, 2014 Adding "--interpolated-path=/home/git-mirrors" to GITDAEMON_OPT values allows for git-daemon to "server" the repository independent of local server path. so the command git clone git://localhost/portage.git /usr/portage will actually clone the repository from /home/git-mirrors/portage.git into /usr/portage. Note: git checkout funtoo.org is VITAL! Emerge will go back to funtoo.org if you do not do this step. If you have DNS setup, you can repeat "Cloning from local git-daemon" on the local computer using git clone git://server.your.network/portage.git /usr/portage after git checkout, emerge will now work successfully. Link to comment Share on other sites More sharing options...
0 Peje Posted December 12, 2014 Report Share Posted December 12, 2014 just sync to one server in your network, then share /usr/portage and /usr/portage/distfiles with nfs over your network cu peje Link to comment Share on other sites More sharing options...
0 666threesixes666 Posted December 12, 2014 Report Share Posted December 12, 2014 git is distributed out of the box. it only fetches updates... there should be a way to fetch git, then serve git. i don't know git well enough to do that. we need an article for this... to git sync on a master tree box, then git share on a master sync box, then set everyone's configs behind the mirror to sync against the mirror instead of github. Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 12, 2014 Author Report Share Posted December 12, 2014 just sync to one server in your network, then share /usr/portage and /usr/portage/distfiles with nfs over your network cu peje Sharing protected directories across a network just seems like an extremely bad idea. Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 12, 2014 Author Report Share Posted December 12, 2014 git is distributed out of the box. it only fetches updates... there should be a way to fetch git, then serve git. i don't know git well enough to do that. we need an article for this... to git sync on a master tree box, then git share on a master sync box, then set everyone's configs behind the mirror to sync against the mirror instead of github. I agree with the premise. There must be a way. funtoo git server daemon? Link to comment Share on other sites More sharing options...
0 sputnik Posted December 12, 2014 Report Share Posted December 12, 2014 I have done exactly what peje suggests for over a year, works great here. If you can't trust other peers on your own local network, who can you trust? Each computer on the network keeps it's own set of distfiles and package files, as the architecture and usage varies. You can keep the distfiles on the server too, but it gets messy trying to keep things clean and not have huge growing out of control directories. Link to comment Share on other sites More sharing options...
0 Oleg Vinichenko Posted December 13, 2014 Report Share Posted December 13, 2014 we have a wiki for setting local git mirror. I'm not tested this myself and giving no guarantee it's a 100% working one. We could make it work as needed. http://www.funtoo.org/Portage_Git_Mirror Link to comment Share on other sites More sharing options...
0 Peje Posted December 13, 2014 Report Share Posted December 13, 2014 @digifuzzy Peje, on 12 Dec 2014 - 8:53 PM, said: just sync to one server in your network, then share /usr/portage and /usr/portage/distfiles with nfs over your network cu peje Sharing protected directories across a network just seems like an extremely bad idea. there are many ways to protect traffic and folders inside your own network, so I dont belive its an extremly bad idea... cu peje Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 13, 2014 Author Report Share Posted December 13, 2014 we have a wiki for setting local git mirror. I'm not tested this myself and giving no guarantee it's a 100% working one. We could make it work as needed. http://www.funtoo.org/Portage_Git_Mirror Reading through the wiki page, I can see the tree being cloned and a daemon set up on the server. Also, I see on the client the portage tree being set aside and the tree from the git daemon cloned. Is it implicit in how emerge/git works that emerge function will grab from the local tree from then on? Link to comment Share on other sites More sharing options...
0 Oleg Vinichenko Posted December 13, 2014 Report Share Posted December 13, 2014 i have no idea :) Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 13, 2014 Author Report Share Posted December 13, 2014 Time to experiment. If you don't see me in a while, you'll know what happened Going in... :) Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 14, 2014 Author Report Share Posted December 14, 2014 we have a wiki for setting local git mirror. I'm not tested this myself and giving no guarantee it's a 100% working one. We could make it work as needed. http://www.funtoo.org/Portage_Git_Mirror I think there is a step missing here. Doing a git clone git://[localhost | host] fails. Will look into SCM git daemon page. Note: edit made to wiki page. quotes needed around GITDAEMON_OPT values. Starting git-daemon service will fail with quotes missing. Link to comment Share on other sites More sharing options...
0 Peje Posted December 14, 2014 Report Share Posted December 14, 2014 And whats your solution for distfiles? -> this seems to produce much more traffic then the ebuilds! Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 14, 2014 Author Report Share Posted December 14, 2014 And whats your solution for distfiles? -> this seems to produce much more traffic then the ebuilds! Are not the distfiles only required when you emerge? Having a git-daemon mirror the funtoo.org github repo does work. I have server and a desktop sync'ing to the local mirror. No problem here. Link to comment Share on other sites More sharing options...
0 Peje Posted December 14, 2014 Report Share Posted December 14, 2014 Yes distfiles are only necesary if you install something, but if you keep your system up to date you will download much sources so it's at least for me about 1:12 (1gb portage -> 12gb distfiles for 1Server 3 Multimediapc's and one Desktop). So any ideas how to solve this? cu Peje Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 14, 2014 Author Report Share Posted December 14, 2014 My understanding is that the funtoo git mirrors (and gentoo rsync mirrors) are just mirrors of the portage tree containing ebuilds (the instructions to build) plus any supporting files/patches to make the ebuilds successful. Downloading dist files? I think that's outside of the scope of my original question. I know that Debain maintains a sources for downloads, but the ebuilds contain the url of where the user can get the source. Otherwise, funtoo's bandwidth would be enourmous (and I don't think Daniel makes as much as Mark Shuttleworth where he can dump money into funtoo ad nauseum ;-) ). A distribution archive/sync? Not sure if that's practical or useful on a large scale but certainly a considered need in a network environment. I can think of a couple of ideas, but it would have to be a development effort. - emerge client "registers" with emerge server (incorporates git-daemon) - any calls from client to download packages, server registers as "tracking" - downloads to archive and passes to client - server must consider hard drive space and report errors. - server must have clean up/maintenance functionality. - sercurity? server can be restricted to accept authenticated clients? I'm oversimplifying things here. Not a simple task, but not impossible. Link to comment Share on other sites More sharing options...
0 Peje Posted December 15, 2014 Report Share Posted December 15, 2014 Sorry I misunderstood your first question: Looking into keep my bandwidth usage down... My simple message is portage uses much less bandwith then the downloaded source code. So you really should find a good solution there And please dont take the time to explain how funtoo works best regards Peje Link to comment Share on other sites More sharing options...
0 digifuzzy Posted December 15, 2014 Author Report Share Posted December 15, 2014 My simple message is portage uses much less bandwith then the downloaded source code. So you really should find a good solution there And please dont take the time to explain how funtoo works Apologies if I offended. Not my intent. I was asking questions to ensure we both had the same understanding and were not getting terms confused. One of the "joys" of open source projects and multi-language efforts. ;) You're situation is certainly more severe than my in terms distfiles. I'm giving this one some thought... Thanks for speaking out. :) Scott Link to comment Share on other sites More sharing options...
0 Peje Posted December 15, 2014 Report Share Posted December 15, 2014 No problem sorry for the mess. If my english would be better... But back to topic the downloaded source code isnt arch specific so you can host all downloaded source code on one place. the insecure way is to share /usr/portage/distfiles over nfs a more secure approch would be to use sftp or sshfs cu Peje Link to comment Share on other sites More sharing options...
Question
digifuzzy
Looking into keep my bandwidth usage down, I found references to setting up rsync daemon in gentoo.
The idea to have a local mirror is quite appealling.
Since Funtoo uses git, is there an equivalent set up/process?
Link to comment
Share on other sites
19 answers to this question
Recommended Posts