##Introduction

Dynamic sublists or dlists is a feature that was added by Systers to Mailman 2.1.10, this feature aims to provide flexibility to list subscribers which is discussed below in detail

Need for such a feature

Flexibility provided by dynamic lists to subscribers

Subscribe/Unsubcribe from new conversations

List subscribers can decide whether to be a part of new conversations or not. If the users decide to subscribe to new conversations, then they will receive all the messages of a new conversation unless they explicitly unsubscribe from it. If they otherwise decide to unsubscribe from new conversations,then they will receive only the first message of every new conversation unless they explicitly subscribe to it.

Advantages:

1. List Users depending upon the conversation can anytime opt to be a part of conversation or not.
2. It prevents the inbox to get unnecessarily filled up with conversations that are of no interest.
3. It prevents List Users to either switch over to Digest or worst Unsubscribe from the entire list if many uninteresting conversations start hitting their mailbox at some point of time.

##Workflow

List Administrator and Dlists

###Create a list as Dlist or Non-Dlist

While creating a list the List Administrator can choose to define a list as either a Dlist or Non-Dlist (original Mailman list). To enable this we add a new attribute in MailingList object as dlist which is True if the list has Dlists enabled and False if not.

All USE CASES mentioned at link

Architecture

###Message flow through mailman

Message moves from the IN queue to the default-posting-chain, to enable Dlist a new rule dlist is added to this chain. The characteristics of this rule are:

• If the mailing list is not dlist enabled, the rule misses

• If the mailing list is dlist enabled and the address in the To: or X-Original-To header has the address of the form listname+new@domain or listaname+new+subject@domain then the message is accepted and is copied to the dlist-posting-pipeline (i.e. dlist queue).

• If the mailing list is dlist-enabled and the address is not of the form specified in the above point, the message bouces back to the user.

Another important function of the dlist queue would be to calculate the thread information to the MailingList metadata object, ofcourse if the message is fit for posting to the list. If not appropriate errors are raised which are mentioned below. Calculation of thread information means to add the thread_id to the metadata if the message belongs to a previously created thread, or to create a new thread and add its id.

Users can subscribe/unsubscribe from the particular conversation by sending emails to listname+threadname+(un)subscribe.To process the email-commands for subscribing/ubsubscribing to threads, we have two possible solutions. Either we follow the current mechanism of processing the email-commands i.e. the administrivia rule checks for the -command at the end and moves the message to the hold queue where hold runner picks up the message and then again copies the message to command queue. From here the command runner picks up every message and takes the required action.

Note: In this case we may have some problem with two different delimiters i.e. - and +, but on the plus side we get to reuse some of the existing code.

Or we could just do the check in the dlist rule and move the message to the hold queue and the further action can be similar to those is the previous paragraph.

After the messages are acccepted for posting from the default-posting-chain the message is moved to dlist-posting-pipeline which is slighly different from the default-posting-pipeline since it has to create different footers with per-thread unsubscribe links.

Dlist-recipient handler calculates the list of the recipeients (replacing the job of calculate-recipients handler)

###(Un)Subscribing to the threads

As mentioned above, the users can (un)subscribe to the threads by sending email commands to the address listname+thread+(un)subscribe@domain. We could modify administrivia rule to check for the above mentioned form and hold the message. Then hold runner is modified to pick up the message and move it to the dlist queue where the dlist runner takes the appropriate action. Dlist runner calls the dlist handler (like autorespond handler in the command runner)to process the message and sends the autoreplies to the virgin queue.

The subscription options can be changed by modifying the overrirde table which keeps track of users’ (un)subscription to each thread. The details are mentioned below.

###Dlist Posting Pipeline

The handlers in this pipeline would be following.

- dlist-decorate
  	- RFC 2369
  	- to-archive
  	- to-digest
  	- to-unset
  	- after delivery
  	- acknowledge
  	- to-outgoing

####dlist-recipeints*

This is a simple handler like calculate-recipients and calculates which users are subscribed to the list and the thread. If it is a new thread the message is sent to all the subscribers, but if it is continuation of a previous thread it is sent only to the subscribers who are either subscribed to all the messages or who are unsubscribed from the messages after first post but have explicitly subscribed to the thread. The Override model is used to check for explicit subscriptions/unsubscriptions from a thread. The details about it are mentioned below.

####dlist-decorate

Also, the footer would be different for the two set of recipeints mentiond above and for that purpose decorate handler would be modified for the purpose. Or again a new handler dlist-decorate could be created to modify the messages for dlists.

Note: The implementation of dlists in mailman2.1 by systers uses a second pipeline for dlists. But it might be possible that we could re-use the existing default-posting-pipeline with a new dlist-recipeints handler. Or we could just modify calculate-recipients handler to check for dlists and calculate the recipeints like it is calculated in dlist-recipeints handler. Personally I would like to have a seperate pipeline and handler to keep the whole process clean and to reduce the errors caused by the checks (assuming the implementations of theory is not always perfect), but I am open to the opinions from the mentors.

Probable Implementation

These new models would be created:

class Thread(model):
	"""This is base model for threads which stores the message id of the first
	post and in linked to the `Message` model using the `Message.thread_id`
	attribute
    """
    __tablename__ = 'thread'
	thread_id = Column(Integer, primary_key=True)
	thread_name = Column(Unicode)
	base_message_id = Column(Integer, ForeignKey('message.id'), index=True)
	messages = relationship('Message')

	def create_thread(self, msg, msgdata, thread_name):
		# Create new thread considerting the messsage as the first post
		# to the thread
		....

	def new_thread(self, msg, msgdata, thread_name=None):
		# Add relevant headers to create a thread and pass the params to
		# create_thread() method.
		....

	def continue_thread(self, msg, msgdata, thread_id):
		# Add an existing message to a thread.
		...


class DlistsPrefernce(Enum):
	# Recieve all posts in all thread and then explicitly unsubscribe to
	# a particular thread
	all_posts = 1
	# Receive first posts from each thread and explicitly subscribe to the
	# conversation
	first_posts = 2


class Override(model):
    """ This model overrides the default user preferences for subscription or
    unsubscription from a particular thread. By default all 'First' posts are
    received by all the users and if a user wishes to unsubscribe (assuming its
    his/her default status "Receive all mails") `Override` will store this
    change in preference and will unsubscribe the user. If current status is
    "Only receive first post" and user particularly wishes to subscribe to a
    thread then Override will store that prefrence and subscribe the user.
    """
	__tablename__ = 'override'
	member_id = Column(Integer, primary_key=True)
	member = relationship('Member')
	# Above could be user_id and user, but need to read more about difference
	# between user and subscriber class
	thread_id = Column(Integer, ForeignKey('thread.id'))
	thread = relationship('Thread')
	# These preferences are picked up from member's default preferences for
	# this list and are updated if user subscribes/unsubscribes from a thread
	preference = Column(Enum(DlistPreference))

Also, the following models needs to be modified to accomodate the changes for dynamic sub-lists. Only the additional attributes of the models are mentioned.

class Message(model):
    __tablename__ = 'override'
    thread_id = Column(Integer, ForeginKey('thread.id'))
	thread = relationship('Thread')
	subject = Column(Unicode())


class Preferences(model):
	# This is global preference for a particular user, and will be the default
	# for a user.
    dlist_preference = Column(Enum(DlistPreference))


class Member(model):
	# This is a per-list preference for a particular membership, it picks the
	# default from above preferences.
	dlist_preference = Column(Enum(DlistPreference))

##Errors

• NewThreadError : When a user sends a email to the address listname+new+threadname@domain but a thread already exists with the same threadname this error is raised. Also, as a result of this error the email bounces back to the user asking him to either change the threadname or post to listname+threadname@domain if he meant to post to the existing thread.

• NonExistentError: This error is raised when a user sends an email to listname+threadname@domain and no such threadname exists in the database. In this case too the email bounces back to the user with the appropriate reasons.

• MalformedRequestError: Apart from the above two errors, it the commands are in any other format that is not supported this error is raised and again the email bounces back to the user.

Postorius

This project would obviously follow by another small project (which I promise to finish ASAP after the GSoC, as adding it to this very project would make it impossible for me to finish it in this summer) which would involve providing admins with options to create lists with dynamic sublists enabled or disabled.

I think adding the dlist flag (mentioned above) in the mailing_list resource and adding th UI options would be enough to create a Dlist. And then it would be followed by a new set of preferences in user’s preferences page which would enable him to set his dlist preferences.

Timeline

The above projects can be divided into three major parts so as to concretely define the schedule of this project.

• Models
• Rules
• Handlers

All the changes described in the implementation part come under these categories( and ofcourse wiring them together and throughout the mailman core).

• 27th Apr - 25th May (Community Bonding Period): This would mainly involve fixing errors in the viability of this project. I will try building a proof of concept set of rules/handlers/models which would at least enable finding out errors in the above implementation at a very early stage.

• Week 1 : Adding all the left out parts of the models and their attributes. This would involve Thread, Override models and new methods in Member, MailingList and other required models/

• Week 2 : Adding all the methods of the above mentioned models including create_thread, continue_thread. Adding some tests for these models too.

• Week 3 : By the end of this week, all the utility functions to create and add threads would be completed. Also, the documentation and tests for the same would be done.

• Week 4 : Send a pull request to simply group messages under threads. Even though this would not work obviously without rules to check for threadname, but a full features pull request with all tests passing should be working by now.

• Week 5-6 : Create interfaces for rules, handlers, and dlist/command runner and add their implementations. Design decisions for creating new rules/handlers or adding to existing handlers.

• Week 7-8 : Adding all the code for the smooth flowing of the message from IN queue to OUT queue. Basis dlist functionality should be working by the end of this week.

• Week 9-10 : Modify Member to enable the dlist_preferences for a user. Make the Override model functional so that users can subscribe and unsubscribe. Also dlist-posting-pipeline should be working by now.

• Week 11-13 : Writing tests and documentation and solving problems from previous weeks.