Jump to content
funtoo forums
  • 0
digifuzzy

Funtoo rsync equivalent?

Question

Looking into keep my bandwidth usage down, I found references to setting up rsync daemon in gentoo.

The idea to have a local mirror is quite appealling.

 

Since Funtoo uses git, is there an equivalent set up/process?

 

Share this post


Link to post
Share on other sites

19 answers to this question

Recommended Posts

  • 0

Adding "--interpolated-path=/home/git-mirrors" to GITDAEMON_OPT values allows for git-daemon to "server" the repository independent of local server path.

 

so the command

 

git clone git://localhost/portage.git /usr/portage

will actually clone the repository from /home/git-mirrors/portage.git into /usr/portage.

 

Note: git checkout funtoo.org is VITAL! Emerge will go back to funtoo.org if you do not do this step.

 

If you have DNS setup, you can repeat "Cloning from local git-daemon" on the local computer using

 

git clone git://server.your.network/portage.git /usr/portage

 

after git checkout, emerge will now work successfully.

Share this post


Link to post
Share on other sites
  • 0

git is distributed out of the box.  it only fetches updates...  there should be a way to fetch git, then serve git.  i don't know git well enough to do that.

 

we need an article for this...  to git sync on a master tree box, then git share on a master sync box, then set everyone's configs behind the mirror to sync against the mirror instead of github.

Share this post


Link to post
Share on other sites
  • 0

just sync to one server in your network, then share /usr/portage and /usr/portage/distfiles with nfs over your network

cu peje

Sharing protected directories across a network just seems like an extremely bad idea.

Share this post


Link to post
Share on other sites
  • 0

git is distributed out of the box.  it only fetches updates...  there should be a way to fetch git, then serve git.  i don't know git well enough to do that.

 

we need an article for this...  to git sync on a master tree box, then git share on a master sync box, then set everyone's configs behind the mirror to sync against the mirror instead of github.

 

I agree with the premise. There must be a way.

funtoo git server daemon?

Share this post


Link to post
Share on other sites
  • 0

I have done exactly what peje suggests for over a year, works great here.  If you can't trust other peers on your own local network, who can you trust?

Each computer on the network keeps it's own set of distfiles and package files, as the architecture and usage varies.  You can keep the distfiles on the server too, but it gets messy trying to keep things clean and not have huge growing out of control directories.

Share this post


Link to post
Share on other sites
  • 0

@digifuzzy 

 

Peje, on 12 Dec 2014 - 8:53 PM, said:

snapback.png

just sync to one server in your network, then share /usr/portage and /usr/portage/distfiles with nfs over your network

cu peje

Sharing protected directories across a network just seems like an extremely bad idea.

 

there are many ways to protect traffic and folders inside your own network, so I dont belive its an extremly bad idea...

cu peje

Share this post


Link to post
Share on other sites
  • 0

we have a wiki for setting local git mirror. I'm not tested this myself and giving no guarantee it's a 100% working one. We could make it work as needed.

http://www.funtoo.org/Portage_Git_Mirror

Reading through the wiki page, I can see the tree being cloned and a daemon set up on the server.

Also, I see on the client the portage tree being set aside and the tree from the git daemon cloned.

Is it implicit in how emerge/git works that emerge function will grab from the local tree from then on?

Share this post


Link to post
Share on other sites
  • 0

we have a wiki for setting local git mirror. I'm not tested this myself and giving no guarantee it's a 100% working one. We could make it work as needed.

http://www.funtoo.org/Portage_Git_Mirror

I think there is a step missing here. Doing a git clone git://[localhost | host] fails.

Will look into SCM git daemon page.

 

Note: edit made to wiki page.

quotes needed around GITDAEMON_OPT values. Starting git-daemon service will fail with quotes missing.

Share this post


Link to post
Share on other sites
  • 0

And whats your solution for distfiles? -> this seems to produce much more traffic then the ebuilds!

Are not the distfiles only required when you emerge?

 

Having a git-daemon mirror the funtoo.org github repo does work.

I have server and a desktop sync'ing to the local mirror. No problem here.

Share this post


Link to post
Share on other sites
  • 0

Yes distfiles are only necesary if you install something, but if you keep your system up to date you will download much sources so it's at least for me about 1:12 (1gb portage -> 12gb distfiles for 1Server 3 Multimediapc's and one Desktop).

So any ideas how to solve this?

 

cu Peje

Share this post


Link to post
Share on other sites
  • 0

My understanding is that the funtoo git mirrors (and gentoo rsync mirrors) are just mirrors of the portage tree containing ebuilds (the instructions to build) plus any supporting files/patches to make the ebuilds successful.

 

Downloading dist files? I think that's outside of the scope of my original question.

I know that Debain maintains a sources for downloads, but the ebuilds contain the url of where the user can get the source. Otherwise, funtoo's bandwidth would be enourmous (and I don't think Daniel makes as much as Mark Shuttleworth where he can dump money into funtoo ad nauseum ;-) ).

 

A distribution archive/sync? Not sure if that's practical or useful on a large scale but certainly a considered need in a network environment.

I can think of a couple of ideas, but it would have to be a development effort.

 

- emerge client "registers" with emerge server (incorporates git-daemon)

- any calls from client to download packages, server registers as "tracking" - downloads to archive and passes to client

- server must consider hard drive space and report errors.

- server must have clean up/maintenance functionality.

- sercurity? server can be restricted to accept authenticated clients?

 

I'm oversimplifying things here. Not a simple task, but not impossible.

Share this post


Link to post
Share on other sites
  • 0

Sorry I misunderstood your first question:

Looking into keep my bandwidth usage down... 

My simple message is portage uses much less bandwith then the downloaded source code.

So you really should find a good solution there

And please dont take the time to explain how funtoo works 

best regards Peje

Share this post


Link to post
Share on other sites
  • 0

My simple message is portage uses much less bandwith then the downloaded source code.

So you really should find a good solution there

And please dont take the time to explain how funtoo works 

Apologies if I offended. Not my intent. I was asking questions to ensure we both had the same understanding and were not getting terms confused.

One of the "joys" of open source projects and multi-language efforts. ;)

 

You're situation is certainly more severe than my in terms distfiles. I'm giving this one some thought...

Thanks for speaking out. :)

Scott

Share this post


Link to post
Share on other sites
  • 0

No problem sorry for the mess. If my english would be better...

 

But back to topic the downloaded source code isnt arch specific so you can host all downloaded source code on one place.

the insecure way is to share /usr/portage/distfiles over nfs

a more secure approch would be to use sftp or sshfs

cu Peje

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×