adding subgraph-feature to vis-output in dot-file-format

  Pfeh... long night again, but here you go. With this patch,
batman-adv now supports the subgraphing-feature of the
dot-file-format. The vis-output can then be parsed with 
"fdp -Tsvg -Gcharset=utf8 /proc/net/batman-adv/vis > test.svg"
for example. 

Interfaces belonging to one BATMAN-node can be found
inside a box, the primary interface of a BATMAN-node is
double-circled. The output also got more detailed, in that
connections between nodes are being displayed with the exact
mac address of the used interface instead of replacing one end
with the primary interface's mac (see [0] for the problems I had
posted before).

Cheers, Linus

PS: Please check if I got that kmalloc stuff right.
PPS: I had to introduce a src-field in the vis-packet-struct,
therefore the compatibility version had to be increased as well.

[0] https://lists.open-mesh.net/pipermail/b.a.t.m.a.n/2009-August/002832.html
  

And if someone prefers a visual example, a simple setup like
node1 <-> node2 <-> node3 results into a dot-file like this:

digraph {
        "22:05:f9:aa:77:1f" -> "a2:14:84:e5:e5:46" [label="1.15"]
        "22:05:f9:aa:77:1f" -> "00:ff:5b:62:9c:a3" [label="HNA"]
        subgraph "cluster_22:05:f9:aa:77:1f" {
                "22:05:f9:aa:77:1f" [peripheries=2]
        }   
        "6a:25:1f:a0:14:72" -> "4e:18:87:ef:80:12" [label="1.0"]
        "6a:25:1f:a0:14:72" -> "00:ff:2c:c8:bd:d5" [label="HNA"]
        subgraph "cluster_6a:25:1f:a0:14:72" {
                "6a:25:1f:a0:14:72" [peripheries=2]
        }   
        "a2:14:84:e5:e5:46" -> "22:05:f9:aa:77:1f" [label="1.0"]
        "4e:18:87:ef:80:12" -> "6a:25:1f:a0:14:72" [label="1.15"]
        "a2:14:84:e5:e5:46" -> "00:ff:5e:e0:23:15" [label="HNA"]
        subgraph "cluster_a2:14:84:e5:e5:46" {
                "a2:14:84:e5:e5:46" [peripheries=2]
                "4e:18:87:ef:80:12"
        }   
}

Which can then be parsed to a nice svg with fdp (or dot):
http://krtek.asta.uni-luebeck.de:8080/~tux/batman-vis-with-subgraph.svg
  

> Cheers, Linus
>
> PPS: I had to introduce a src-field in the vis-packet-struct,
> therefore the compatibility version had to be increased as well.

I'm thinking about linux-mainline here.

Does it make sense to have a version number per message type? Some
things we can do backward compatibility easily, some things not.

I guess it should not be too hard to do backward compatibility
here. If you receive a version 7 do the old way of getting the source
address. If its a version 8 packet, use the new way?

Allowing this sort of backward compatibility also makes deployment
easier. You currently have a real danger of loosing contact to some of
your nodes during software updates. If you do it in the wrong order,
you could end up with a disconnected island.

Also, once in mainline, the vis file becomes part of the well defined
API. It should not just change in ways which are not backward
compatible. I'm guessing this change breaks all my tools for parsing
vis, and it probably breaks other peoples tools. Ideally, we strip out
all graphviz and JSON formatting from the kernel and put it into
userspace, eg batctl. Tools should then use batcl. It then becomes
safer to change the proc vis format. The output of batctl stays the
same, unless you add a command line flag to enable the new feature.

This move to mainline requires some adaptations to the development
ways. We need to discuss this and need to get the basics sorted out
before we do go into mainline. However, it looks like we could be in
mainline very quickly if we wanted.

      Andrew
  

Hi,

On Sun, Aug 30, 2009 at 09:37:52AM +0200, Andrew Lunn wrote:
> > Cheers, Linus
> >
> > PPS: I had to introduce a src-field in the vis-packet-struct,
> > therefore the compatibility version had to be increased as well.
> 
> I'm thinking about linux-mainline here.
> 
> Does it make sense to have a version number per message type? Some
> things we can do backward compatibility easily, some things not.
> 
> I guess it should not be too hard to do backward compatibility
> here. If you receive a version 7 do the old way of getting the source
> address. If its a version 8 packet, use the new way?

Generally possible, but as we store the vis information locally 
in whole packets we have to distinguish everywhere between the versions
(packet handling, proc access, etc). This will not make the code
look pretty.

> 
> Allowing this sort of backward compatibility also makes deployment
> easier. You currently have a real danger of loosing contact to some of
> your nodes during software updates. If you do it in the wrong order,
> you could end up with a disconnected island.

Each software updates on meshed nodes is difficult. Even 
if you just change the channel, the ESSID/BSSID or other wifi parameters
you can end up with islands. Most setup i have seen tried to use software
firmwares as homogenous as possible, even if they are a little bit older.
Software Updates in mesh networks always require more sophisticated
strategies.

> 
> Also, once in mainline, the vis file becomes part of the well defined
> API. It should not just change in ways which are not backward
> compatible. I'm guessing this change breaks all my tools for parsing
> vis, and it probably breaks other peoples tools. Ideally, we strip out
> all graphviz and JSON formatting from the kernel and put it into
> userspace, eg batctl. Tools should then use batcl. It then becomes
> safer to change the proc vis format. The output of batctl stays the
> same, unless you add a command line flag to enable the new feature.

Mhm, even if we had transported the data from kernel to userspace, we
would have ended up in a change to this interface as the data format
changed (additional source field). 

However i've added a compile flag in the reviewed version of linus patch
which should give you the possibility to get the "old" vis output.

Generally i think that the patch was really neccesary as the original
vis did not consider nodes with multiple interfaces. Disconnected graphs
were the result of this, which makes the vis output pretty useless
if you employ nodes with multiple interfaces. I think the approach
by using clusters is quite good too, even if we (the toolwriters,
me included) need to change their parsers now. 

> 
> This move to mainline requires some adaptations to the development
> ways. We need to discuss this and need to get the basics sorted out
> before we do go into mainline. However, it looks like we could be in
> mainline very quickly if we wanted.

I agree, we should sort this out early. My personal opinion is that
the algorithm will be optimized further, and it will make the code
unmaintainable to add exceptions for all older versions (especially
the various intermediate versions).

As it was pointed out earlier by Marek and Elektra, using multiple versions
in one network will most probably lead to "side effects". IMHO it's better
to let users know that it won't work properly backward than risking these
side effects.

Usually we have stable versions (e.g. batman-adv 0.1) with a specific format,
and an ongoing development in the svn trunk. Maybe we should do the same
for future kernel development - maintaining one packet and algorithm version
in the kernel, and further develop the algorithm on our own (e.g. in a 
git tree or with sets of patches)? 
In this case, we should post the current batman-adv to the list when
0.2 is stabilized and finished. Mainline algorithm should only be changed
when we reach a new stable version.

best regards,
	Simon
  

On Monday 31 August 2009 04:23:06 Simon Wunderlich wrote:
> Usually we have stable versions (e.g. batman-adv 0.1) with a specific
> format, and an ongoing development in the svn trunk. Maybe we should do the
> same for future kernel development - maintaining one packet and algorithm
> version in the kernel, and further develop the algorithm on our own (e.g.
> in a git tree or with sets of patches)?
> In this case, we should post the current batman-adv to the list when
> 0.2 is stabilized and finished. Mainline algorithm should only be changed
> when we reach a new stable version.

Actually, I came to a similar conclusion:
We "only" submit stable versions to mainline (including stability fixes) 
whereas the development stays outside of the official linux tree. That way we 
can keep our development speed without worrying too much about compatibility. 
When we are getting to the next stable release we can think about how to 
soften the transition. 

More ideas / opinions ?

Regards,
Marek
  

On Mon, Aug 31, 2009 at 01:26:29PM +0800, Marek Lindner wrote:
> On Monday 31 August 2009 04:23:06 Simon Wunderlich wrote:
> > Usually we have stable versions (e.g. batman-adv 0.1) with a specific
> > format, and an ongoing development in the svn trunk. Maybe we should do the
> > same for future kernel development - maintaining one packet and algorithm
> > version in the kernel, and further develop the algorithm on our own (e.g.
> > in a git tree or with sets of patches)?
> > In this case, we should post the current batman-adv to the list when
> > 0.2 is stabilized and finished. Mainline algorithm should only be changed
> > when we reach a new stable version.
> 
> Actually, I came to a similar conclusion:
> We "only" submit stable versions to mainline (including stability fixes) 
> whereas the development stays outside of the official linux tree. That way we 
> can keep our development speed without worrying too much about compatibility. 
> When we are getting to the next stable release we can think about how to 
> soften the transition. 

The rules when in staging some somewhat different when in the main
part of the tree.

The point of staging is that it allows cleanup work to be done, fixing
bad practices etc. User space APIs are not baked in stone, since bad
practices in these APIs needs cleaning up etc. So we have some
flexibility here.

Gazing into my crystal ball:

2.6.31 is out 7th September.
2.6.32 merge window ends 21st September.

Maybe GregKH prioritizes main tree patches during the window. staging
is low priority. So batman is not in 2.6.32-rc1.

Batman appears in rc2.

We have around 2 1/2 months time for cleanup work.

Sometime in December 2.6.32 is released with batman in staging.

By the time 2.6.33 is released, March 2010, we are in the main tree.


I would say we have around three months to get 0.2 finished and into
2.6.32 staging. For 2.6.33 we don't break compatibility any more and
aim to finish the cleanup and get into the main tree.

I would also suggest we post our patches as soon as possible, so we
start getting feedback from the kernel community about what else needs
adding to the TODO list etc.

       Andrew
  

On Monday 31 August 2009 13:50:07 Andrew Lunn wrote:
> I would say we have around three months to get 0.2 finished and into
> 2.6.32 staging. For 2.6.33 we don't break compatibility any more and
> aim to finish the cleanup and get into the main tree.

That sounds like a plan. Let me add our own versions to make things clear:

* 2.6.32 gets our 0.2 stable
* 2.6.33 get 0.2.5 which is 0.2 stable + linux mainline changes (no packet 
format changes, etc will happen here)

After we released the 0.2 stable we start our 0.3 development branch which 
contains the new ideas to improve the routing and other stuff. When this branch 
gets included into mainline is unclear at this point.


> I would also suggest we post our patches as soon as possible, so we
> start getting feedback from the kernel community about what else needs
> adding to the TODO list etc.

Yes, when and where should we post it ? Should the mailing list be migrated at 
that point (I felt a consensus concerning the ml move) ?

It might be a good idea to maintain the ToDo list in the wiki - what do you 
think ?

Regards,
Marek
  

Ok, now with a larger setup I found two smaller problems. I didn't
have the time to look at the source code again yet, but I'll just post
the things we noticed here.
First of all, not all interfaces get added to a subgraph. As an example
vis-output I have this [0] file, but unfortunately the lines in brackets
are not there, but should to avoid splitted graphs.
Secondly, it seems that a vis-client is just sending to the
vis-server its prefered interface plus quality to a neighbour, not
all possible links plus qualities to the neighbour. As a
network-administrator I'd like to have these information as well
to be able to check if an interface might be broken for example.

Cheers, Linus

[0]:
--------
digraph {
        "00:50:bf:d6:fd:b0" -> "00:22:b0:44:94:02" [label="1.0"]
        "ca:5f:0b:67:03:c7" -> "00:ff:4d:f1:04:c5" [label="4.553"]
        "00:50:bf:d6:fd:b0" -> "00:ff:82:74:9c:eb" [label="HNA"]
        subgraph "cluster_00:50:bf:d6:fd:b0" {
                "00:50:bf:d6:fd:b0" [peripheries=2]
                "ca:5f:0b:67:03:c7"
        }
        "00:22:b0:98:87:de" -> "00:22:b0:98:87:fa" [label="1.15"]
        "06:22:b0:98:87:dd" -> "06:22:b0:44:94:5d" [label="1.7"]
        "00:22:b0:98:87:de" -> "00:e0:29:39:34:d7" [label="1.15"]
        "06:22:b0:98:87:dd" -> "00:16:cb:ba:4a:f4" [label="HNA"]
        "06:22:b0:98:87:dd" -> "00:19:d2:7a:ba:21" [label="HNA"]
        "06:22:b0:98:87:dd" -> "00:ff:ca:4d:a5:e7" [label="HNA"]
        "06:22:b0:98:87:dd" -> "00:22:b0:98:87:dd" [label="HNA"]
        "06:22:b0:98:87:dd" -> "00:19:7e:6a:8a:27" [label="HNA"]
        subgraph "cluster_06:22:b0:98:87:dd" {
              ( "00:22:b0:98:87:de" )
                "06:22:b0:98:87:dd" [peripheries=2]
        }
        "00:22:b0:98:87:fa" -> "00:22:b0:98:87:de" [label="1.15"]
        "06:22:b0:98:87:f9" -> "06:22:b0:44:c6:59" [label="1.3"]
        "06:22:b0:98:87:f9" -> "06:22:b0:98:85:5b" [label="1.0"]
        "06:22:b0:98:87:f9" -> "06:22:b0:98:8c:33" [label="1.0"]
        "06:22:b0:98:87:f9" -> "00:22:b0:98:87:f9" [label="HNA"]
        "06:22:b0:98:87:f9" -> "00:ff:1b:01:51:12" [label="HNA"]
        subgraph "cluster_06:22:b0:98:87:f9" {
              ( "00:22:b0:98:87:fa" )
                "06:22:b0:98:87:f9" [peripheries=2]
        }
        "06:22:b0:44:c6:59" -> "06:22:b0:98:87:f9" [label="1.15"]
        "00:22:b0:44:c6:5a" -> "00:22:b0:98:8c:34" [label="1.85"]
        "06:22:b0:44:c6:59" -> "06:22:b0:98:85:5b" [label="1.0"]
        "06:22:b0:44:c6:59" -> "06:22:b0:98:8c:33" [label="1.32"]
        "06:22:b0:44:c6:59" -> "00:22:b0:44:c6:59" [label="HNA"]
        "06:22:b0:44:c6:59" -> "00:ff:d8:85:d7:fa" [label="HNA"]
        subgraph "cluster_06:22:b0:44:c6:59" {
                "06:22:b0:44:c6:59" [peripheries=2]
                "00:22:b0:44:c6:5a"
        }
        "00:22:b0:44:94:02" -> "00:50:bf:d6:fd:b0" [label="1.15"]
        "06:22:b0:44:94:01" -> "00:22:b0:44:94:01" [label="HNA"]
        "06:22:b0:44:94:01" -> "00:ff:af:cb:d6:64" [label="HNA"]
        subgraph "cluster_06:22:b0:44:94:01" {
              ( "00:22:b0:44:94:02" )
                "06:22:b0:44:94:01" [peripheries=2]
        }
        "06:22:b0:98:85:5b" -> "06:22:b0:44:c6:59" [label="1.15"]
        "06:22:b0:98:85:5b" -> "06:22:b0:98:8c:33" [label="1.62"]
        "06:22:b0:98:85:5b" -> "00:ff:00:02:cb:82" [label="HNA"]
        "06:22:b0:98:85:5b" -> "00:22:b0:98:85:5b" [label="HNA"]
        subgraph "cluster_06:22:b0:98:85:5b" {
                "06:22:b0:98:85:5b" [peripheries=2]
        }
}
  

> > I would also suggest we post our patches as soon as possible, so we
> > start getting feedback from the kernel community about what else needs
> > adding to the TODO list etc.
> 
> Yes, when and where should we post it ? Should the mailing list be
> migrated at that point (I felt a consensus concerning the ml move) ?

We should post them to GregKH. That is what he said. What happens
after that i don't know. Maybe he has an internal team which will do a
review? Maybe it goes straight into staging and anybody who is
interested can review the code.

I would say the list needs to be open before we post the patches. We
don't want valuable comments going lost, or annoying people who do
reviews by bouncing there comments etc.

> It might be a good idea to maintain the ToDo list in the wiki - what do you 
> think ?

I say the master has to be in the sources, where anybody who reviews
staging will see. The Wiki copy is just a mirror, with a big comment
that its not the master, all changes should first go into the sources
TODO list which is then re-imported into the Wiki.

Maybe we need to branch SVN, or we setup a git tree. I already have
local changes just so that it builds with linux-next.git from
Saturday.

  Andrew
  

On Wednesday 02 September 2009 14:18:22 Andrew Lunn wrote:
> We should post them to GregKH. That is what he said. What happens
> after that i don't know. Maybe he has an internal team which will do a
> review? Maybe it goes straight into staging and anybody who is
> interested can review the code.
>
> I would say the list needs to be open before we post the patches. We
> don't want valuable comments going lost, or annoying people who do
> reviews by bouncing there comments etc.

Ok, then who is doing the migration ? I surely can help to provide the list of 
subscribed users.


> > It might be a good idea to maintain the ToDo list in the wiki - what do
> > you think ?
>
> I say the master has to be in the sources, where anybody who reviews
> staging will see. The Wiki copy is just a mirror, with a big comment
> that its not the master, all changes should first go into the sources
> TODO list which is then re-imported into the Wiki.

Fine with me.


> Maybe we need to branch SVN, or we setup a git tree. I already have
> local changes just so that it builds with linux-next.git from
> Saturday.

I can create git.open-mesh.com so that we have an official git to pull from ?

Regards,
Marek
  

Hi Folks

Sorry for suddenly disappearing after getting some momentum going for
getting batman into mainline. My employer sent me to Finland to work
on a project for 2-3 weeks and it has been hard to find time to work
on batman. I guess i will not be back to normal working situation for
at least another week.

> > I would say the list needs to be open before we post the patches. We
> > don't want valuable comments going lost, or annoying people who do
> > reviews by bouncing there comments etc.
> 
> Ok, then who is doing the migration ? I surely can help to provide
> the list of subscribed users.

The steps as i see it:

1) Make a decision about which list server to use.
2) Contact the sysadmin and ask for a list to be setup.
3) Export the list of subscribers.
4) Import the list of subscribers.
5) Modify the MX record.
 
I guess vger is probably the least flexible about domain names. Mail
coming from the list might not be using the lists.open-mesh.net
domain. So maybe someone should ask sourceforge and sourceware if they
could provide the list using this domain.

> I can create git.open-mesh.com so that we have an official git to pull from ?

I think this is a good idea. It is probably easier for GregKH to do a
pull then handle lots of patches. 

     Andrew
  

On Wednesday 09 September 2009 00:01:46 Andrew Lunn wrote:
> The steps as i see it:
> 
> 1) Make a decision about which list server to use.

I think as soon as we havea short list of options we can make the decision. At 
the moment I see no need/desire for a long discussion.  :-)


> 2) Contact the sysadmin and ask for a list to be setup.

Volunteer needed here.


> 3) Export the list of subscribers.

That will be me.


> 4) Import the list of subscribers.

Probably the same volunteer.


> 5) Modify the MX record.

That will be Elektra.


> I guess vger is probably the least flexible about domain names. Mail
> coming from the list might not be using the lists.open-mesh.net
> domain. So maybe someone should ask sourceforge and sourceware if they
> could provide the list using this domain.

If possible I'd like to keep the domain name. It will create less confusion.


> > I can create git.open-mesh.com so that we have an official git to pull
> > from ?
> 
> I think this is a good idea. It is probably easier for GregKH to do a
> pull then handle lots of patches.

Elektra already created the DNS record. I will setup the necessary stuff in the 
next days.

Regards,
Marek
  

On Tue, Sep 8, 2009 at 11:54 AM, Marek Lindner <lindner_marek@yahoo.de> wrote:
> On Wednesday 09 September 2009 00:01:46 Andrew Lunn wrote:
>> The steps as i see it:
>>
>> 1) Make a decision about which list server to use.
>
> I think as soon as we havea short list of options we can make the decision. At
> the moment I see no need/desire for a long discussion.  :-)
>
>
>> 2) Contact the sysadmin and ask for a list to be setup.
>
> Volunteer needed here.

I would like to volunteer.

>
>
>> 3) Export the list of subscribers.
>
> That will be me.
>
>
>> 4) Import the list of subscribers.
>
> Probably the same volunteer.
>
>

I don't have a server that would work for this, but I think it would
be nice to find a service that hosts FOSS projects for free, but
normally charges for the service.  That way, no one in the batman
project has to worry about servers failing/upgrading/migrating/etc.

The first example that comes to mind is XMission because they are my
ISP.  They use Mailman (same as we're using now), so we might even be
able to transfer the messages on the current Mailman server to the new
one.  I'm not at all familiar with Mailman, but I'll call them today
and see what they say.

http://mailman.xmission.com/

Jake
  

On Fri, Sep 18, 2009 at 3:14 PM, Jacob Marble <jacobmarble@gmail.com> wrote:
> On Tue, Sep 8, 2009 at 11:54 AM, Marek Lindner <lindner_marek@yahoo.de> wrote:
>> On Wednesday 09 September 2009 00:01:46 Andrew Lunn wrote:
>>> The steps as i see it:
>>>
>>> 1) Make a decision about which list server to use.
>>
>> I think as soon as we havea short list of options we can make the decision. At
>> the moment I see no need/desire for a long discussion.  :-)
>>
>>
>>> 2) Contact the sysadmin and ask for a list to be setup.
>>
>> Volunteer needed here.
>
> I would like to volunteer.
>
>>
>>
>>> 3) Export the list of subscribers.
>>
>> That will be me.
>>
>>
>>> 4) Import the list of subscribers.
>>
>> Probably the same volunteer.
>>
>>
>
> I don't have a server that would work for this, but I think it would
> be nice to find a service that hosts FOSS projects for free, but
> normally charges for the service.  That way, no one in the batman
> project has to worry about servers failing/upgrading/migrating/etc.
>
> The first example that comes to mind is XMission because they are my
> ISP.  They use Mailman (same as we're using now), so we might even be
> able to transfer the messages on the current Mailman server to the new
> one.  I'm not at all familiar with Mailman, but I'll call them today
> and see what they say.
>
> http://mailman.xmission.com/
>
> Jake
>

I just got off the phone with a sales guy at XMission.  He says
mailing lists are US$3/month.  I can cover this.
How does that sound, Marek?  Do you have something else in mind?
  

> > I don't have a server that would work for this, but I think it would
> > be nice to find a service that hosts FOSS projects for free, but
> > normally charges for the service. ??That way, no one in the batman
> > project has to worry about servers failing/upgrading/migrating/etc.
> >
> > The first example that comes to mind is XMission because they are my
> > ISP. ??They use Mailman (same as we're using now), so we might even be
> > able to transfer the messages on the current Mailman server to the new
> > one. ??I'm not at all familiar with Mailman, but I'll call them today
> > and see what they say.
> >
> > http://mailman.xmission.com/
> >
> > Jake
> >
> 
> I just got off the phone with a sales guy at XMission.  He says
> mailing lists are US$3/month.  I can cover this.
> How does that sound, Marek?  Do you have something else in mind?

What is important is the quality of its spam filtering. We need
something with very high quality, eg like sourceforge, vger, or
sourceware would provide. So i would suggest a provider with a proven
record of hosting open source project mailing lists which are open for
anybody to post to.

        Andrew
  

On Sat, Sep 19, 2009 at 1:59 AM, Andrew Lunn <andrew@lunn.ch> wrote:
> What is important is the quality of its spam filtering. We need
> something with very high quality, eg like sourceforge, vger, or
> sourceware would provide. So i would suggest a provider with a proven
> record of hosting open source project mailing lists which are open for
> anybody to post to.

Sorry for the delay, busy week.  So, I'm looking for a mailing host that:

a) has experience hosting, and so can filter spam without hassle
b) lets us use open-mesh.net instead of super-duper-mailing-host.com
c) hosts open source gratis

I've looked at sourceforge, but their mailing list service looks very
canned and they only allow <listname>@lists.sourceforge.net.  Also, SF
already has a project called batman, so I created one called batman2
(there are surely more creative project names available).  I have
created a test account:
https://lists.sourceforge.net/lists/admin/batman2-test

This week, I'll look at commercial entities as well as larger open
source projects.

Jake
  

On Sun, Sep 27, 2009 at 9:29 PM, Jacob Marble <jacobmarble@gmail.com> wrote:
> On Sat, Sep 19, 2009 at 1:59 AM, Andrew Lunn <andrew@lunn.ch> wrote:
>> What is important is the quality of its spam filtering. We need
>> something with very high quality, eg like sourceforge, vger, or
>> sourceware would provide. So i would suggest a provider with a proven
>> record of hosting open source project mailing lists which are open for
>> anybody to post to.
>
> Sorry for the delay, busy week.  So, I'm looking for a mailing host that:
>
> a) has experience hosting, and so can filter spam without hassle
> b) lets us use open-mesh.net instead of super-duper-mailing-host.com
> c) hosts open source gratis
>
> I've looked at sourceforge, but their mailing list service looks very
> canned and they only allow <listname>@lists.sourceforge.net.  Also, SF
> already has a project called batman, so I created one called batman2
> (there are surely more creative project names available).  I have
> created a test account:
> https://lists.sourceforge.net/lists/admin/batman2-test
>
> This week, I'll look at commercial entities as well as larger open
> source projects.
>
> Jake
>

Information about mailing lists at SF:
https://sourceforge.net/apps/trac/sourceforge/wiki/Mailing%20lists
  

> Sorry for the delay, busy week.  So, I'm looking for a mailing host that:
> 
> a) has experience hosting, and so can filter spam without hassle
> b) lets us use open-mesh.net instead of super-duper-mailing-host.com
> c) hosts open source gratis

a and b are must have, c would be nice. 
 
> I've looked at sourceforge, but their mailing list service looks very
> canned and they only allow <listname>@lists.sourceforge.net.  Also, SF
> already has a project called batman, so I created one called batman2
> (there are surely more creative project names available).  I have
> created a test account:
> https://lists.sourceforge.net/lists/admin/batman2-test

There was briefly a discussion about renaming batman-adv to something
else, but that discussion seemed to of died before reaching any
conclusion. So for the moment i would suggest for the moment we keep
with batman.

        Andrew
  

On Sun, Sep 27, 2009 at 11:03 PM, Andrew Lunn <andrew@lunn.ch> wrote:
>> a) has experience hosting, and so can filter spam without hassle
>> b) lets us use open-mesh.net instead of super-duper-mailing-host.com
>> c) hosts open source gratis
>
> a and b are must have, c would be nice.

I started going down the list at
http://en.wikipedia.org/wiki/Comparison_of_open_source_software_hosting_facilities#Features

Codendi looks like a very "comprehensive" solution, probably excessive.

The following require that every list subscriber have "yet another
username/password" and they all host mailing lists at mail-host.com:

sourceforge.net
tigris.org
launchpad.net

Another alternative is freelists.org, which only hosts @freelists.org,
but does not require subscribers to log in to their site, ever.  I
found them because the Haiku OS project uses them.

There are others listed, but I feel like this is a wrong path.  Now
I'm looking at services listed at the Mailman wiki:
http://wiki.list.org/display/COM/Mailman+hosting+services

A lot of these services are VPS, which tells me that they let the
customer handle details like spam.

JPBerlin (http://www.jpberlin.de/) looks good, but I don't know any
German.  The Mailman like says "CEO Peer Heinlein is the maintainer of
the german Mailman-translation and author of a book about secure
mail-server administration."

Thielpark Systeme GmbH (http://www.thielpark.de/) I will contact this
one.  "Reliable Mailman hosting on Linux with per list
templates/language files, MHonArc integration for archiving and
searching, customization of templates to match look of your website.
Address import from various data sources."

EMWD (http://www.emwd.com/mailman.html) looks like a regular webhost
with better-than-average mailing list support.  They claim to have
spam filters in place, and they charge US$4/month.  I emailed them
regarding hosting with open-mesh.net

More later.

Jake
  

On Mon, Sep 28, 2009 at 12:12 AM, Jacob Marble <jacobmarble@gmail.com> wrote:
>>> a) has experience hosting, and so can filter spam without hassle
>>> b) lets us use open-mesh.net instead of super-duper-mailing-host.com
>>> c) hosts open source gratis
>>
>> a and b are must have, c would be nice.
>
> EMWD (http://www.emwd.com/mailman.html) looks like a regular webhost
> with better-than-average mailing list support.  They claim to have
> spam filters in place, and they charge US$4/month.  I emailed them
> regarding hosting with open-mesh.net
>

I got an email this morning from Brian Carpenter at EMWD:

<quote>
Hi Jacob:

Thank you for your interest in our services.

> a) filters spam very well
> b) allows the list to be hosted using our domain name, not yours

We meet both of those requirement. Just keep in mind that since you
cannot have a domain point to two servers (Mailman requires a web
server), you will need to use a domain name that is not in current use
or you can use a sub-domain off of your main domain such as
list.open-mesh.net.

> We currently have about 100 message/month in traffic.  Does this
> describe your $4/month service?

How many members will be subscribed to your list?
</quote>

This is the best alternative that I have found.  They charge $4/month
for up to 1000 subscribers and 1-8 posts/day.  Looking at the
archives, I don't think this will be a problem for us.  We can pay
more for up to 20,000 subscribers with this service.  If you guys are
comfortable with this, then I need to work out details with Brian (how
many subscribers, how to pay, can we import the existing archive, DNS
records).

IMHO, I feel that better alternatives exist if we drop requirement
"b", but I'm happy to be sysadmin for the list, wherever we host it.

Jake
  

> > a) filters spam very well
> > b) allows the list to be hosted using our domain name, not yours
> 
> We meet both of those requirement. Just keep in mind that since you
> cannot have a domain point to two servers (Mailman requires a web
> server), you will need to use a domain name that is not in current use
> or you can use a sub-domain off of your main domain such as
> list.open-mesh.net.
> 
> > We currently have about 100 message/month in traffic.  Does this
> > describe your $4/month service?
> 
> How many members will be subscribed to your list?
> </quote>

> This is the best alternative that I have found.  They charge $4/month
> for up to 1000 subscribers and 1-8 posts/day.

I would read the small print. We currently have 100 messages/month to
a closed list. Once it is an open list, and well publicised, the
amount of spam will increase dramatically. I personally get 100-200
spams per day. Does this 1-8 posts per day include or exclude spam?

However, thanks for doing a good job so far.

      Andrew
  

On Tuesday 29 September 2009 13:25:11 Andrew Lunn wrote:
> > This is the best alternative that I have found.  They charge $4/month
> > for up to 1000 subscribers and 1-8 posts/day.
> 
> I would read the small print. We currently have 100 messages/month to
> a closed list. Once it is an open list, and well publicised, the
> amount of spam will increase dramatically. I personally get 100-200
> spams per day. Does this 1-8 posts per day include or exclude spam?

Ok, if this list is not going to make it I only see 2 options (feel free to 
add more):

* We drop the requirement to keep the domain name for the list and migrate 
everyone to the new list.
* We create the additional new list while keeping the old one. The new list is 
for the communication with the kernel developers.

Opinions ?

Regards,
Marek


PS: Big thanks to Jacob for digging into this.
  

On Mon, Sep 28, 2009 at 11:25 PM, Andrew Lunn <andrew@lunn.ch> wrote:
>> > a) filters spam very well
>> > b) allows the list to be hosted using our domain name, not yours
>>
>> We meet both of those requirement. Just keep in mind that since you
>> cannot have a domain point to two servers (Mailman requires a web
>> server), you will need to use a domain name that is not in current use
>> or you can use a sub-domain off of your main domain such as
>> list.open-mesh.net.
>>
>> > We currently have about 100 message/month in traffic.  Does this
>> > describe your $4/month service?
>>
>> How many members will be subscribed to your list?
>> </quote>
>
>> This is the best alternative that I have found.  They charge $4/month
>> for up to 1000 subscribers and 1-8 posts/day.
>
> I would read the small print. We currently have 100 messages/month to
> a closed list. Once it is an open list, and well publicised, the
> amount of spam will increase dramatically. I personally get 100-200
> spams per day. Does this 1-8 posts per day include or exclude spam?

More from Brian at EMWD:

<quote>
> We currently have far fewer than 1000 subscribers, but we need a
> public list, and we currently have a private list.  That's why spam
> filtering is important.  Once we switch to public, we expect to see
> more spam and more new ham as well.  Let's assume that we get 4 times
> as much hammy traffic (400 message/month = 12/day) as we currently do
> (less than 100/month = 3/day).
>
> How much would EMWD charge?  Do you count dropped spam towards our
> charged traffic?
>
> Thanks,
>
> Jake

We would start you at $4 per month for the list. After 2-3 months and
we will take a look at your list traffic and make any pricing
adjustments if necessary. A 1000 member mailing list that has 12
messages posted to it per day would be $8 per month. Spam does not
count as traffic as long as it doesn't get broadcast to your list. It
shouldn't if you have your list configuration settings setup
correctly.
</quote>

They are concerned with messages that get broadcast to the list, not
with blocked spam.  Also, I would say that their pricing is very
reasonable.
  

On Tue, Sep 29, 2009 at 3:19 AM, Marek Lindner <lindner_marek@yahoo.de> wrote:
> Ok, if this list is not going to make it I only see 2 options (feel free to
> add more):
>
> * We drop the requirement to keep the domain name for the list and migrate
> everyone to the new list.
> * We create the additional new list while keeping the old one. The new list is
> for the communication with the kernel developers.
>
> Opinions ?

I think that having two lists would be a hassle and a mistake.  To
have two lists at one host might work, but one list at VGER and
another somewhere else could be counter-productive, as "internal"
communication would be constantly forwarded to the kernel list, and
relevant kernel traffic would have to be forwarded back to the
"internal" list.  Think about that for a second.  Eventually, we would
probably just drop one list.

If one list is O(n), then two lists are O(n^2) for anyone who needs
both.  I vote for one list, no matter where it's hosted.

commercial solution (EMWD):
 - spam filtering
 - open-mesh.net
 - small price < US$20/month

open source host (VGER, freelists.org)
 - spam filtering
 - their-domain.com
 - no price
 - we get added to their project directory, so potential contributors see us

Jake
  

[...]
> > * We create the additional new list while keeping the old one. The new
> > list is for the communication with the kernel developers.
> >
> > Opinions ?
> 
> I think that having two lists would be a hassle and a mistake.  To
> have two lists at one host might work, but one list at VGER and
> another somewhere else could be counter-productive, as "internal"
> communication would be constantly forwarded to the kernel list, and
> relevant kernel traffic would have to be forwarded back to the
> "internal" list.  Think about that for a second.  Eventually, we would
> probably just drop one list.
> 
> If one list is O(n), then two lists are O(n^2) for anyone who needs
> both.  I vote for one list, no matter where it's hosted.
Don't understand the calculation as I would prefer + instead of * here, but 
this isn't the point (and if you would do program a bad matching algorithm 
here it is still O(M*N) ).
Don't think that two lists are good without a good topic split. I see a reason 
for -devel, -users and -announcements list in some projects, but not for a 
list for "kernel people" and for "other people".

> commercial solution (EMWD):
>  - spam filtering
>  - open-mesh.net
>  - small price < US$20/month
> 
> open source host (VGER, freelists.org)
>  - spam filtering
>  - their-domain.com
>  - no price
>  - we get added to their project directory, so potential contributors see
>  us
Haven't followed the whole discussion, but is it real such a big problem to 
change the mailing list address? OK, it shouldn't be done every week, but this 
is done for a good reason. Just do an announcement on this list and create an 
auto reply (smtp failure notice with further information) when someone writes 
to the old list.
Maybe more interesting is the fact that if we can import old information into 
the new list. I think mailman stores everything in simple mboxes, or am I 
wrong?
And also interesting if we can do backups of the data on vger and co.

Best regards,
	Sven
  

>> I think that having two lists would be a hassle and a mistake.  To
>> have two lists at one host might work, but one list at VGER and
>> another somewhere else could be counter-productive, as "internal"
>> communication would be constantly forwarded to the kernel list, and
>> relevant kernel traffic would have to be forwarded back to the
>> "internal" list.  Think about that for a second.  Eventually, we would
>> probably just drop one list.
>>
>> If one list is O(n), then two lists are O(n^2) for anyone who needs
>> both.  I vote for one list, no matter where it's hosted.
> Don't understand the calculation as I would prefer + instead of * here, but
> this isn't the point (and if you would do program a bad matching algorithm
> here it is still O(M*N) ).
> Don't think that two lists are good without a good topic split. I see a reason
> for -devel, -users and -announcements list in some projects, but not for a
> list for "kernel people" and for "other people".
...
> Haven't followed the whole discussion, but is it real such a big problem to
> change the mailing list address? OK, it shouldn't be done every week, but this
> is done for a good reason. Just do an announcement on this list and create an
> auto reply (smtp failure notice with further information) when someone writes
> to the old list.

I agree with all of this.  My point is that multiple lists should at
least be hosted at the same place.  The topic split is a good point,
too; if there were two -devel lists, I think all of the developers
would eventually use the "kernel people" list exclusively.

> Maybe more interesting is the fact that if we can import old information into
> the new list. I think mailman stores everything in simple mboxes, or am I
> wrong?
> And also interesting if we can do backups of the data on vger and co.

Just looked at the VGER website, and they use majordomo.  I don't know
much about these things; can mailman data be imported to majordomo?  A
quick look around reveals no email addresses to contact them, other
than some that seem to be spambot honeypots.

What do we need in order to make a decision?  I would like to get this
transition happening soon so that real work can move forward.

Jacob
  

On Wednesday 30 September 2009 13:39:21 Jacob Marble wrote:
> Just looked at the VGER website, and they use majordomo.  I don't know
> much about these things; can mailman data be imported to majordomo?  A
> quick look around reveals no email addresses to contact them, other
> than some that seem to be spambot honeypots.
> 
> What do we need in order to make a decision?  I would like to get this
> transition happening soon so that real work can move forward.

Ok, I feel a consensus that the open-mesh domain is not that important 
compared to having only one list.

Personally, I don't think paying for a list is a good solution because sooner 
or later that will become an issue (even if it is not much money). 
Other than that I'm fine with every list you think is best. We should be able 
to somehow import the existing users (I can provide a list of mail addresses 
as a text file) and we should have archives somewhere. 

Regards,
Marek