[users] State of play

[Karanbir Singh]

hi guys,

Just to do a recap on things as they are w.r.t hosting setup.

At the moment we have :
1 machine doing the frontend with 2x73gb scsi disks ( C4 )
1 machine doing the storage with 2x500gb disks ( C5 )

So the plan was:

- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )

- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

- We have space for another 2 1U machines here in the UK, so if there is
any need to buy in machines that need hosting - we can put them here.
The Guys in the DC are very open source friendly, and available 24/7
with reaction times of between 3 to 5 minutes ( its a professional DC,
not a mass hosting facility ).

Finally, Fabian has access to the guys at Coreix DC - so if I am not
around and something breaks, he can get in touch with them and ask for
attention etc. I dont want too many people on that list, but if we can
get someone from Australia on there, that would mean we get fairly good
round the clock coverage.

- KB

[Yury V. Zaytsev]

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

I think VMs are definitively the way to go for service isolation. We had
a setup which was basically an OpenVZ cluster with isolated VMs for
website, database etc. and it was very nice to be able to upgrade them
separately, move the nodes around the machines without downtime whenever
the maintenance had to be performed etc.

And certainly a VM is a nice playground for testing by an unprivileged /
untrusted person such as I am. Also we can offload the administration of
different parts of the system (Redmine, SCM, mirrors, buildhosts) to
different groups of people without too much compromising the security of
the environment.

This is just a suggestion though. Who is going to administrate the thing
will make final decisions probably with the consultation of the list.

Post by Karanbir Singh
- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

Hey, I don't think we necessarily need to invent something on our own
here. There are several models that work fine, we just need to evaluate
them and chose which one will suit us better.

(1) I do maintain some packages for Debian. The way it works is that
basically any seemingly sane person can get access to any SCM.

There is a number of people that have their keys in the keyring (Debian
Developers). Once I want to get my package in the distro I need to ask
for sponsorship and one of them will review / push my package.
Eventually you can become a DD if you think it's worth it.

(2) The packages are signed by the build host and the privileges to push
packages and get them signed are distributed on a per-package basis. To
my mind, this works when you have a rather large number of people
responsible for individual packages and almost no jacks of all trades.

Which one is better? It's up to the discussion.

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

[Karanbir Singh]

Post by Yury V. Zaytsev

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

What makes you think that Hg isnt faster than svn for larger tree's ?
I've got a mozilla gecko checkout here that updates in about 6 seconds;
and I only update about once every few weeks.

- KB

[Karanbir Singh]

Post by Yury V. Zaytsev

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

What makes you think that Hg isnt faster than svn for larger tree's ?
I've got a mozilla gecko checkout here that updates in about 6 seconds;
and I only update about once every few weeks.

- KB

[Karanbir Singh]

Post by Yury V. Zaytsev

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

What makes you think that Hg isnt faster than svn for larger tree's ?
I've got a mozilla gecko checkout here that updates in about 6 seconds;
and I only update about once every few weeks.

- KB

[Karanbir Singh]

Post by Yury V. Zaytsev

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

What makes you think that Hg isnt faster than svn for larger tree's ?
I've got a mozilla gecko checkout here that updates in about 6 seconds;
and I only update about once every few weeks.

- KB

[Karanbir Singh]

Post by Yury V. Zaytsev

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

What makes you think that Hg isnt faster than svn for larger tree's ?
I've got a mozilla gecko checkout here that updates in about 6 seconds;
and I only update about once every few weeks.

- KB

[David Hrbáč]

Post by Karanbir Singh
- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

If Steve wouldn't be able, I can do it on our infrastructure.

Post by Karanbir Singh
- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

Maybe I can create a few VMs on one (95GHz) of our clusters.
DH

[Karanbir Singh]

Post by David Hrbáč
If Steve wouldn't be able, I can do it on our infrastructure.

The plan was that Steve would get something together that a couple of
people can look at and then decide on how we want to move from there.
Beyond that - I'll let Steve and you sync up.

Keeping that in mind, no reason why you should not also perhaps have a
go at this. All you would need is a complete rsync of the rpmforge
mirror, if you dont have that already. Having different people look at
the options might be a good idea anyway, we might create more options
that way.

Post by David Hrbáč

Post by Karanbir Singh
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Not really, since we can't control those points. And we have fairly good
b/w here. eg. we can deliver > 600 Mbps into aws from here. But its
mostly about control and being able to fix things quickly. So either an
ip or a login/pass based auth mechanism that lets public mirrors pull
from here would be good. They can then be the public face of the repo's
( == urls in baseurl= )

Post by David Hrbáč
Maybe I can create a few VMs on one (95GHz) of our clusters.

Sounds good. But just dont do something where if something is down, we
need to get in touch with you to fix things! I know this has been an
issue with me and the present setup, which is why I've now got Fabian on
the acl. Also, am actively looking for someone in Australia who might be
able to help.

- KB

[Karanbir Singh]

Post by David Hrbáč
If Steve wouldn't be able, I can do it on our infrastructure.

The plan was that Steve would get something together that a couple of
people can look at and then decide on how we want to move from there.
Beyond that - I'll let Steve and you sync up.

Keeping that in mind, no reason why you should not also perhaps have a
go at this. All you would need is a complete rsync of the rpmforge
mirror, if you dont have that already. Having different people look at
the options might be a good idea anyway, we might create more options
that way.

Post by David Hrbáč

Post by Karanbir Singh
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Not really, since we can't control those points. And we have fairly good
b/w here. eg. we can deliver > 600 Mbps into aws from here. But its
mostly about control and being able to fix things quickly. So either an
ip or a login/pass based auth mechanism that lets public mirrors pull
from here would be good. They can then be the public face of the repo's
( == urls in baseurl= )

Post by David Hrbáč
Maybe I can create a few VMs on one (95GHz) of our clusters.

Sounds good. But just dont do something where if something is down, we
need to get in touch with you to fix things! I know this has been an
issue with me and the present setup, which is why I've now got Fabian on
the acl. Also, am actively looking for someone in Australia who might be
able to help.

- KB

[Karanbir Singh]

Post by David Hrbáč
If Steve wouldn't be able, I can do it on our infrastructure.

The plan was that Steve would get something together that a couple of
people can look at and then decide on how we want to move from there.
Beyond that - I'll let Steve and you sync up.

Keeping that in mind, no reason why you should not also perhaps have a
go at this. All you would need is a complete rsync of the rpmforge
mirror, if you dont have that already. Having different people look at
the options might be a good idea anyway, we might create more options
that way.

Post by David Hrbáč

Post by Karanbir Singh
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Not really, since we can't control those points. And we have fairly good
b/w here. eg. we can deliver > 600 Mbps into aws from here. But its
mostly about control and being able to fix things quickly. So either an
ip or a login/pass based auth mechanism that lets public mirrors pull
from here would be good. They can then be the public face of the repo's
( == urls in baseurl= )

Post by David Hrbáč
Maybe I can create a few VMs on one (95GHz) of our clusters.

Sounds good. But just dont do something where if something is down, we
need to get in touch with you to fix things! I know this has been an
issue with me and the present setup, which is why I've now got Fabian on
the acl. Also, am actively looking for someone in Australia who might be
able to help.

- KB

[Karanbir Singh]

Post by David Hrbáč
If Steve wouldn't be able, I can do it on our infrastructure.

The plan was that Steve would get something together that a couple of
people can look at and then decide on how we want to move from there.
Beyond that - I'll let Steve and you sync up.

Keeping that in mind, no reason why you should not also perhaps have a
go at this. All you would need is a complete rsync of the rpmforge
mirror, if you dont have that already. Having different people look at
the options might be a good idea anyway, we might create more options
that way.

Post by David Hrbáč

Post by Karanbir Singh
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Not really, since we can't control those points. And we have fairly good
b/w here. eg. we can deliver > 600 Mbps into aws from here. But its
mostly about control and being able to fix things quickly. So either an
ip or a login/pass based auth mechanism that lets public mirrors pull
from here would be good. They can then be the public face of the repo's
( == urls in baseurl= )

Post by David Hrbáč
Maybe I can create a few VMs on one (95GHz) of our clusters.

Sounds good. But just dont do something where if something is down, we
need to get in touch with you to fix things! I know this has been an
issue with me and the present setup, which is why I've now got Fabian on
the acl. Also, am actively looking for someone in Australia who might be
able to help.

- KB

[Karanbir Singh]

Post by David Hrbáč
If Steve wouldn't be able, I can do it on our infrastructure.

The plan was that Steve would get something together that a couple of
people can look at and then decide on how we want to move from there.
Beyond that - I'll let Steve and you sync up.

Keeping that in mind, no reason why you should not also perhaps have a
go at this. All you would need is a complete rsync of the rpmforge
mirror, if you dont have that already. Having different people look at
the options might be a good idea anyway, we might create more options
that way.

Post by David Hrbáč

Post by Karanbir Singh
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Not really, since we can't control those points. And we have fairly good
b/w here. eg. we can deliver > 600 Mbps into aws from here. But its
mostly about control and being able to fix things quickly. So either an
ip or a login/pass based auth mechanism that lets public mirrors pull
from here would be good. They can then be the public face of the repo's
( == urls in baseurl= )

Post by David Hrbáč
Maybe I can create a few VMs on one (95GHz) of our clusters.

Sounds good. But just dont do something where if something is down, we
need to get in touch with you to fix things! I know this has been an
issue with me and the present setup, which is why I've now got Fabian on
the acl. Also, am actively looking for someone in Australia who might be
able to help.

- KB

[Karanbir Singh]

hi guys,

Just to do a recap on things as they are w.r.t hosting setup.

At the moment we have :
1 machine doing the frontend with 2x73gb scsi disks ( C4 )
1 machine doing the storage with 2x500gb disks ( C5 )

So the plan was:

- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )

- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

- We have space for another 2 1U machines here in the UK, so if there is
any need to buy in machines that need hosting - we can put them here.
The Guys in the DC are very open source friendly, and available 24/7
with reaction times of between 3 to 5 minutes ( its a professional DC,
not a mass hosting facility ).

Finally, Fabian has access to the guys at Coreix DC - so if I am not
around and something breaks, he can get in touch with them and ask for
attention etc. I dont want too many people on that list, but if we can
get someone from Australia on there, that would mean we get fairly good
round the clock coverage.

- KB

[Yury V. Zaytsev]

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

I think VMs are definitively the way to go for service isolation. We had
a setup which was basically an OpenVZ cluster with isolated VMs for
website, database etc. and it was very nice to be able to upgrade them
separately, move the nodes around the machines without downtime whenever
the maintenance had to be performed etc.

And certainly a VM is a nice playground for testing by an unprivileged /
untrusted person such as I am. Also we can offload the administration of
different parts of the system (Redmine, SCM, mirrors, buildhosts) to
different groups of people without too much compromising the security of
the environment.

This is just a suggestion though. Who is going to administrate the thing
will make final decisions probably with the consultation of the list.

Post by Karanbir Singh
- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

Hey, I don't think we necessarily need to invent something on our own
here. There are several models that work fine, we just need to evaluate
them and chose which one will suit us better.

(1) I do maintain some packages for Debian. The way it works is that
basically any seemingly sane person can get access to any SCM.

There is a number of people that have their keys in the keyring (Debian
Developers). Once I want to get my package in the distro I need to ask
for sponsorship and one of them will review / push my package.
Eventually you can become a DD if you think it's worth it.

(2) The packages are signed by the build host and the privileges to push
packages and get them signed are distributed on a per-package basis. To
my mind, this works when you have a rather large number of people
responsible for individual packages and almost no jacks of all trades.

Which one is better? It's up to the discussion.

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

[David Hrbáč]

Post by Karanbir Singh
- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

If Steve wouldn't be able, I can do it on our infrastructure.

Post by Karanbir Singh
- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

Maybe I can create a few VMs on one (95GHz) of our clusters.
DH

[Karanbir Singh]

hi guys,

Just to do a recap on things as they are w.r.t hosting setup.

At the moment we have :
1 machine doing the frontend with 2x73gb scsi disks ( C4 )
1 machine doing the storage with 2x500gb disks ( C5 )

So the plan was:

- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )

- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

- We have space for another 2 1U machines here in the UK, so if there is
any need to buy in machines that need hosting - we can put them here.
The Guys in the DC are very open source friendly, and available 24/7
with reaction times of between 3 to 5 minutes ( its a professional DC,
not a mass hosting facility ).

Finally, Fabian has access to the guys at Coreix DC - so if I am not
around and something breaks, he can get in touch with them and ask for
attention etc. I dont want too many people on that list, but if we can
get someone from Australia on there, that would mean we get fairly good
round the clock coverage.

- KB

[Yury V. Zaytsev]

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

I think VMs are definitively the way to go for service isolation. We had
a setup which was basically an OpenVZ cluster with isolated VMs for
website, database etc. and it was very nice to be able to upgrade them
separately, move the nodes around the machines without downtime whenever
the maintenance had to be performed etc.

And certainly a VM is a nice playground for testing by an unprivileged /
untrusted person such as I am. Also we can offload the administration of
different parts of the system (Redmine, SCM, mirrors, buildhosts) to
different groups of people without too much compromising the security of
the environment.

This is just a suggestion though. Who is going to administrate the thing
will make final decisions probably with the consultation of the list.

Post by Karanbir Singh
- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

Hey, I don't think we necessarily need to invent something on our own
here. There are several models that work fine, we just need to evaluate
them and chose which one will suit us better.

(1) I do maintain some packages for Debian. The way it works is that
basically any seemingly sane person can get access to any SCM.

There is a number of people that have their keys in the keyring (Debian
Developers). Once I want to get my package in the distro I need to ask
for sponsorship and one of them will review / push my package.
Eventually you can become a DD if you think it's worth it.

(2) The packages are signed by the build host and the privileges to push
packages and get them signed are distributed on a per-package basis. To
my mind, this works when you have a rather large number of people
responsible for individual packages and almost no jacks of all trades.

Which one is better? It's up to the discussion.

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

[David Hrbáč]

Post by Karanbir Singh
- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

If Steve wouldn't be able, I can do it on our infrastructure.

Post by Karanbir Singh
- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

Maybe I can create a few VMs on one (95GHz) of our clusters.
DH

[Karanbir Singh]

hi guys,

Just to do a recap on things as they are w.r.t hosting setup.

At the moment we have :
1 machine doing the frontend with 2x73gb scsi disks ( C4 )
1 machine doing the storage with 2x500gb disks ( C5 )

So the plan was:

- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )

- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

- We have space for another 2 1U machines here in the UK, so if there is
any need to buy in machines that need hosting - we can put them here.
The Guys in the DC are very open source friendly, and available 24/7
with reaction times of between 3 to 5 minutes ( its a professional DC,
not a mass hosting facility ).

Finally, Fabian has access to the guys at Coreix DC - so if I am not
around and something breaks, he can get in touch with them and ask for
attention etc. I dont want too many people on that list, but if we can
get someone from Australia on there, that would mean we get fairly good
round the clock coverage.

- KB

[Yury V. Zaytsev]

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

I think VMs are definitively the way to go for service isolation. We had
a setup which was basically an OpenVZ cluster with isolated VMs for
website, database etc. and it was very nice to be able to upgrade them
separately, move the nodes around the machines without downtime whenever
the maintenance had to be performed etc.

And certainly a VM is a nice playground for testing by an unprivileged /
untrusted person such as I am. Also we can offload the administration of
different parts of the system (Redmine, SCM, mirrors, buildhosts) to
different groups of people without too much compromising the security of
the environment.

This is just a suggestion though. Who is going to administrate the thing
will make final decisions probably with the consultation of the list.

Post by Karanbir Singh
- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

Hey, I don't think we necessarily need to invent something on our own
here. There are several models that work fine, we just need to evaluate
them and chose which one will suit us better.

(1) I do maintain some packages for Debian. The way it works is that
basically any seemingly sane person can get access to any SCM.

There is a number of people that have their keys in the keyring (Debian
Developers). Once I want to get my package in the distro I need to ask
for sponsorship and one of them will review / push my package.
Eventually you can become a DD if you think it's worth it.

(2) The packages are signed by the build host and the privileges to push
packages and get them signed are distributed on a per-package basis. To
my mind, this works when you have a rather large number of people
responsible for individual packages and almost no jacks of all trades.

Which one is better? It's up to the discussion.

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

[David Hrbáč]

Post by Karanbir Singh
- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

If Steve wouldn't be able, I can do it on our infrastructure.

Post by Karanbir Singh
- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

Maybe I can create a few VMs on one (95GHz) of our clusters.
DH

[Karanbir Singh]

hi guys,

Just to do a recap on things as they are w.r.t hosting setup.

At the moment we have :
1 machine doing the frontend with 2x73gb scsi disks ( C4 )
1 machine doing the storage with 2x500gb disks ( C5 )

So the plan was:

- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )

- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

- We have space for another 2 1U machines here in the UK, so if there is
any need to buy in machines that need hosting - we can put them here.
The Guys in the DC are very open source friendly, and available 24/7
with reaction times of between 3 to 5 minutes ( its a professional DC,
not a mass hosting facility ).

Finally, Fabian has access to the guys at Coreix DC - so if I am not
around and something breaks, he can get in touch with them and ask for
attention etc. I dont want too many people on that list, but if we can
get someone from Australia on there, that would mean we get fairly good
round the clock coverage.

- KB

[Yury V. Zaytsev]

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

I think VMs are definitively the way to go for service isolation. We had
a setup which was basically an OpenVZ cluster with isolated VMs for
website, database etc. and it was very nice to be able to upgrade them
separately, move the nodes around the machines without downtime whenever
the maintenance had to be performed etc.

And certainly a VM is a nice playground for testing by an unprivileged /
untrusted person such as I am. Also we can offload the administration of
different parts of the system (Redmine, SCM, mirrors, buildhosts) to
different groups of people without too much compromising the security of
the environment.

This is just a suggestion though. Who is going to administrate the thing
will make final decisions probably with the consultation of the list.

Post by Karanbir Singh
- w.r.t keysigning; thats a seperate issue and needs some level of
discussion.

Hey, I don't think we necessarily need to invent something on our own
here. There are several models that work fine, we just need to evaluate
them and chose which one will suit us better.

(1) I do maintain some packages for Debian. The way it works is that
basically any seemingly sane person can get access to any SCM.

There is a number of people that have their keys in the keyring (Debian
Developers). Once I want to get my package in the distro I need to ask
for sponsorship and one of them will review / push my package.
Eventually you can become a DD if you think it's worth it.

(2) The packages are signed by the build host and the privileges to push
packages and get them signed are distributed on a per-package basis. To
my mind, this works when you have a rather large number of people
responsible for individual packages and almost no jacks of all trades.

Which one is better? It's up to the discussion.

Post by Karanbir Singh
- version control is a policy issue, hosting it is as close to free as
free can get, either with svn or git or Hg ( which is an easier
migration from svn, and comes with most benefits that git has for the
'average' user )

Well, hg is not so different in terms of speed from svn, so in such a
case the question pops up why would we migrate from svn at all?, given
that as I outlined above we don't have a proper SCM workflow, but what
we're doing now is basically using it as an authenticated archiving
backend with integrated diff viewer.

[David Hrbáč]

Post by Karanbir Singh
- setup a mirror for the existing storage and see how we might be able
to reduce size and footprint for the main production rpm-repo ( Steve
has offered to do this and is already setup, he's waiting on my
finishing the mirror for the storage - which finished yesterday ),
unfortunately the outage from last week was caused due to my over
aggressive dd which took down the storage machine.

If Steve wouldn't be able, I can do it on our infrastructure.

Post by Karanbir Singh
- If we can get the main mirror down to 65GB or so - we can then host it
locally on the main machine, and remove the dependency on the second
machine. ( Fabian has offered to redo the setup / install on the main
machine once we are ready )
- The second machine could then become a rsync target for public mirrors
to pull from, and not impact resources on the main machine.

Isn't it better to use ie.freshrpms.net as the main source?

Post by Karanbir Singh
- w.r.t VM's - if we need to do that, the main box has 8 gigs of ram,
and dual opterons; so it could handle a few, I am just not sure we need
VM's as long as we are not doing builds on the same machine. I think the
hetzner machine that Dag has is a *much* better build host than surya
will be.

Maybe I can create a few VMs on one (95GHz) of our clusters.
DH