Implemented batch domain creation0

This commit is contained in:
Marc Aymerich 2015-03-11 20:01:08 +00:00
parent b36ca7a248
commit e80f921601
5 changed files with 103 additions and 98 deletions

44
TODO.md
View file

@ -11,21 +11,16 @@
* add `BackendLog` retry action * add `BackendLog` retry action
* webmail identities and addresses * webmail identities and addresses
* use Code: https://github.com/django/django/blob/master/django/forms/forms.py#L415 for domain.refresh_serial()
* Permissions .filter_queryset() * Permissions .filter_queryset()
* env vars instead of multiple settings files: https://devcenter.heroku.com/articles/config-vars ? * env vars instead of multiple settings files: https://devcenter.heroku.com/articles/config-vars ?
* Log changes from rest api (serialized objects) * Log changes from rest api (serialized objects)
* EMAIL backend operations which contain stderr messages (because under certain failures status code is still 0)
* Settings dictionary like DRF2 in order to better override large settings like WEBSITES_APPLICATIONS.etc
* backend logs with hal logo * backend logs with hal logo
* set_password orchestration method? * set_password orchestration method?
* make account_link to autoreplace account on change view.
* LAST version of this shit http://wkhtmltopdf.org/downloads.html * LAST version of this shit http://wkhtmltopdf.org/downloads.html
@ -48,22 +43,20 @@
* Maildir billing tests/ webdisk billing tests (avg metric) * Maildir billing tests/ webdisk billing tests (avg metric)
* move icons to apps, and use appconfig to cleanup config stuff
* when using modeladmin to store shit like self.account, make sure to have a cleanslate in each request? * when using modeladmin to store shit like self.account, make sure to have a cleanslate in each request? no, better reuse the last one
* jabber with mailbox accounts (dovecto mail notification) * jabber with mailbox accounts (dovecot mail notification)
* rename accounts register to "account", and reated api and admin references * rename accounts register to "account", and reated api and admin references
* take a look icons from ajenti ;)
* Disable services is_active should be computed on the fly in order to distinguish account.is_active from service.is_active when reactivation. * Disable services is_active should be computed on the fly in order to distinguish account.is_active from service.is_active when reactivation.
* Perhaps it is time to create a ServiceModel ? * Perhaps it is time to create a ServiceModel ?
* prevent deletion of main user by the user itself * prevent deletion of main user by the user itself
* AccountAdminMixin auto adds 'account__name' on searchfields and handle account_link on fieldsets * AccountAdminMixin auto adds 'account__name' on searchfields
* Separate panel from server passwords? Store passwords on panel? set_password special backend operation? * Separate panel from server passwords? Store passwords on panel? set_password special backend operation?
@ -75,16 +68,15 @@
* delete main user -> delete account or prevent delete main user * delete main user -> delete account or prevent delete main user
* Ansible orchestration *method* (methods.py)
* multiple domains creation; line separated domains
* Move MU webapps to SaaS?
* offer to create mailbox on account creation * multiple domains creation; line separated domains
* init.d celery scripts * init.d celery scripts
-# Required-Start: $network $local_fs $remote_fs postgresql celeryd -# Required-Start: $network $local_fs $remote_fs postgresql celeryd
-# Required-Stop: $network $local_fs $remote_fs postgresql celeryd -# Required-Stop: $network $local_fs $remote_fs postgresql celeryd
* for list virtual_domains cleaning up we need to know the old domain name when a list changes its address domain, but this is not possible with the current design.
* regenerate virtual_domains every time (configure a separate file for orchestra on postfix) * regenerate virtual_domains every time (configure a separate file for orchestra on postfix)
* update_fields=[] doesn't trigger post save! * update_fields=[] doesn't trigger post save!
@ -97,9 +89,10 @@
* proforma without billing contact? * proforma without billing contact?
* print open invoices as proforma?
* env ORCHESTRA_MASTER_SERVER='test1.orchestra.lan' ORCHESTRA_SECOND_SERVER='test2.orchestra.lan' ORCHESTRA_SLAVE_SERVER='test3.orchestra.lan' python manage.py test orchestra.apps.domains.tests.functional_tests.tests:AdminBind9BackendDomainTest * env ORCHESTRA_MASTER_SERVER='test1.orchestra.lan' ORCHESTRA_SECOND_SERVER='test2.orchestra.lan' ORCHESTRA_SLAVE_SERVER='test3.orchestra.lan' python manage.py test orchestra.apps.domains.tests.functional_tests.tests:AdminBind9BackendDomainTest
* Pangea modifications: domain registered/non-registered list_display and field with register link: inconsistent, what happen to related objects with a domain that is converted to register-only?
* ForeignKey.swappable * ForeignKey.swappable
* Field.editable * Field.editable
@ -111,16 +104,12 @@
* multiple files monitoring * multiple files monitoring
* Split plans into a separate app (plans and rates / services ) interface ?
* sync() ServiceController method that synchronizes orchestra and servers (delete or import) * sync() ServiceController method that synchronizes orchestra and servers (delete or import)
* consider removing mailbox support on forward (user@pangea.org instead) * consider removing mailbox support on forward (user@pangea.org instead)
* Databases.User add reverse M2M databases widget (like mailbox.addresses) * Databases.User add reverse M2M databases widget (like mailbox.addresses)
* Root owned logs on user's home ? yes
* reconsider binding webapps to systemusers (pangea multiple users wordpress-ftp, moodle-pangea, etc) * reconsider binding webapps to systemusers (pangea multiple users wordpress-ftp, moodle-pangea, etc)
* Secondary user home in /home/secondaryuser and simlink to /home/main/webapps/app so it can have private storage? * Secondary user home in /home/secondaryuser and simlink to /home/main/webapps/app so it can have private storage?
* Grant permissions to systemusers, the problem of creating a related permission model is out of sync with the server-side. evaluate tradeoff * Grant permissions to systemusers, the problem of creating a related permission model is out of sync with the server-side. evaluate tradeoff
@ -155,7 +144,6 @@
* Create an admin service_view with icons (like SaaS app) * Create an admin service_view with icons (like SaaS app)
* Fix ftp traffic
* Resource graph for each related object * Resource graph for each related object
@ -179,22 +167,15 @@ Multi-tenant WebApps
* SaaS - Those apps that can't use custom domain * SaaS - Those apps that can't use custom domain
* WebApp - Those apps that can use custom domain * WebApp - Those apps that can use custom domain
* Howto upgrade webapp PHP version? <FilesMatch \.php$> SetHandler php54-cgi</FilesMatch> ? or create a new app
* prevent @pangea.org email addresses on contacts, enforce at least one email without @pangea.org * prevent @pangea.org email addresses on contacts, enforce at least one email without @pangea.org
* fcgid kill instead of apache reload?
* username maximum as group user in UNIX
* forms autocomplete="off", doesn't work in chrome * forms autocomplete="off", doesn't work in chrome
ln -s /proc/self/fd /dev/fd ln -s /proc/self/fd /dev/fd
* http-https/https-only/http-only
POST INSTALL POST INSTALL
------------ ------------
@ -212,9 +193,10 @@ Php binaries should have this format: /usr/bin/php5.2-cgi
* <IfModule security2_module> and other IfModule on backend SecRule * <IfModule security2_module> and other IfModule on backend SecRule
* webalizer backend on webapps and check webapps.websites.all()
* monitor in batches doesnt work!!! * monitor in batches doesnt work!!!
* mv: cannot move `/home/marcay/webapps/webalizer/' to a subdirectory of itself, `/home/marcay/webapps/webalizer/.deleted' * Orchestra global search box on the header, based https://github.com/django/django/blob/master/django/contrib/admin/options.py#L866 and iterating over all registered services and inspectin its admin.search_fields
* Create utility for dealing with web paths '//', leading and ending '/'
* contain error on plugin missing key (plugin dissabled)

View file

@ -10,7 +10,7 @@ from orchestra.apps.accounts.admin import AccountAdminMixin
from orchestra.utils import apps from orchestra.utils import apps
from .actions import view_zone from .actions import view_zone
from .forms import RecordInlineFormSet, CreateDomainAdminForm from .forms import RecordInlineFormSet, BatchDomainCreationAdminForm
from .filters import TopDomainListFilter from .filters import TopDomainListFilter
from .models import Domain, Record from .models import Domain, Record
@ -68,7 +68,7 @@ class DomainAdmin(AccountAdminMixin, ExtendedModelAdmin):
list_filter = [TopDomainListFilter] list_filter = [TopDomainListFilter]
change_readonly_fields = ('name',) change_readonly_fields = ('name',)
search_fields = ['name',] search_fields = ['name',]
add_form = CreateDomainAdminForm add_form = BatchDomainCreationAdminForm
change_view_actions = [view_zone] change_view_actions = [view_zone]
def structured_name(self, domain): def structured_name(self, domain):
@ -115,11 +115,37 @@ class DomainAdmin(AccountAdminMixin, ExtendedModelAdmin):
qs = qs.prefetch_related('websites') qs = qs.prefetch_related('websites')
return qs return qs
# def save_related(self, request, form, formsets, change): def save_model(self, request, obj, form, change):
# super(DomainAdmin, self).save_related(request, form, formsets, change) """ batch domain creation support """
# if form.cleaned_data['migrate_subdomains']: super(DomainAdmin, self).save_model(request, obj, form, change)
# domain = form.instance self.extra_domains = []
# domain.subdomains.update(account_id=domain.account_id) if not change:
for name in form.extra_names:
domain = Domain.objects.create(name=name, account_id=obj.account_id)
self.extra_domains.append(domain)
def save_formset(self, request, form, formset, change):
"""
Given an inline formset save it to the database.
"""
formset.save()
def save_related(self, request, form, formsets, change):
""" batch domain creation support """
super(DomainAdmin, self).save_related(request, form, formsets, change)
if not change:
# Clone records to extra_domains, if any
for formset in formsets:
if formset.model is Record:
for domain in self.extra_domains:
# Reset pk value of the record instances to force creation of new ones
for record_form in formset.forms:
record = record_form.instance
if record.pk:
record.pk = None
formset.instance = domain
form.instance = domain
self.save_formset(request, form, formset, change=change)
admin.site.register(Domain, DomainAdmin) admin.site.register(Domain, DomainAdmin)

View file

@ -33,9 +33,9 @@ class Bind9MasterDomainBackend(ServiceController):
self.append(textwrap.dedent("""\ self.append(textwrap.dedent("""\
echo -e '%(zone)s' > %(zone_path)s.tmp echo -e '%(zone)s' > %(zone_path)s.tmp
diff -N -I'^\s*;;' %(zone_path)s %(zone_path)s.tmp || UPDATED=1 diff -N -I'^\s*;;' %(zone_path)s %(zone_path)s.tmp || UPDATED=1
# Because bind reload will not display any fucking error
named-checkzone -k fail -n fail %(name)s %(zone_path)s.tmp
mv %(zone_path)s.tmp %(zone_path)s mv %(zone_path)s.tmp %(zone_path)s
# Because bind realod will not display any fucking error
named-checkzone -k fail -n fail %(name)s %(zone_path)s
""") % context """) % context
) )
self.update_conf(context) self.update_conf(context)

View file

@ -7,15 +7,35 @@ from .helpers import domain_for_validation
from .models import Domain from .models import Domain
class CreateDomainAdminForm(forms.ModelForm): class BatchDomainCreationAdminForm(forms.ModelForm):
# migrate_subdomains = forms.BooleanField(label=_("Migrate subdomains"), required=False, name = forms.CharField(label=_("Names"), widget=forms.Textarea(attrs={'rows': 5, 'cols': 50}),
# initial=False, help_text=_("Propagate the account owner change to subdomains.")) help_text=_("Domain per line. All domains will share the same attributes."))
def clean_name(self):
self.extra_names = []
target = None
for name in self.cleaned_data['name'].strip().splitlines():
name = name.strip()
if not name:
continue
if target is None:
target = name
else:
domain = Domain(name=name)
try:
domain.full_clean(exclude=['top'])
except ValidationError as e:
raise ValidationError(e.error_dict['name'])
self.extra_names.append(name)
return target
def clean(self): def clean(self):
""" inherit related top domain account, when exists """ """ inherit related top domain account, when exists """
cleaned_data = super(CreateDomainAdminForm, self).clean() cleaned_data = super(BatchDomainCreationAdminForm, self).clean()
if not cleaned_data['account']: if not cleaned_data['account']:
domain = Domain(name=cleaned_data['name']) account = None
for name in [cleaned_data['name']] + self.extra_names:
domain = Domain(name=name)
top = domain.get_top() top = domain.get_top()
if not top: if not top:
# Fake an account to make django validation happy # Fake an account to make django validation happy
@ -24,50 +44,18 @@ class CreateDomainAdminForm(forms.ModelForm):
raise ValidationError({ raise ValidationError({
'account': _("An account should be provided for top domain names."), 'account': _("An account should be provided for top domain names."),
}) })
cleaned_data['account'] = top.account elif account and top.account != account:
# Fake an account to make django validation happy
account_model = self.fields['account']._queryset.model
cleaned_data['account'] = account_model()
raise ValidationError({
'account': _("Provided domain names belong to different accounts."),
})
account = top.account
cleaned_data['account'] = account
return cleaned_data return cleaned_data
#class BatchDomainCreationAdminForm(DomainAdminForm):
# # TODO
# name = forms.CharField(widget=forms.Textarea, label=_("Names"),
# help_text=_("Domain per line. All domains will share the same attributes."))
#
# def clean_name(self):
# self.names = []
# target = None
# for name in self.cleaned_data['name'].splitlines():
# name = name.strip()
# if target is None:
# target = name
# else:
# domain = Domain(name=name)
# try:
# domain.full_clean(exclude=['top'])
# except ValidationError as e:
# raise ValidationError(e.error_dict['name'])
# self.names.append(name)
# return target
#
# def save_model(self, request, obj, form, change):
# # TODO thsi is modeladmin
# """ batch domain creation support """
# super(DomainAdmin, self).save_model(request, obj, form, change)
# if not change:
# for name in form.names:
# domain = Domain.objects.create(name=name, account_id=obj.account_id)
#
# def save_related(self, request, form, formsets, change):
# # TODO thsi is modeladmin
# """ batch domain creation support """
# super(DomainAdmin, self).save_related(request, form, formsets, change)
# if not change:
# for name in form.names:
# for formset in formsets:
# formset.instance = form.instance
# self.save_formset(request, form, formset, change=change)
class RecordInlineFormSet(forms.models.BaseInlineFormSet): class RecordInlineFormSet(forms.models.BaseInlineFormSet):
# TODO # TODO
def clean(self): def clean(self):

View file

@ -57,16 +57,23 @@ class PHPAppType(AppType):
return init_vars return init_vars
help_message = _("Version of PHP used to execute this webapp. <br>"
"Changing the PHP version may result in application malfunction, "
"make sure that everything continue to work as expected.")
class PHPFPMAppForm(PluginDataForm): class PHPFPMAppForm(PluginDataForm):
php_version = forms.ChoiceField(label=_("PHP version"), php_version = forms.ChoiceField(label=_("PHP version"),
choices=settings.WEBAPPS_PHP_FPM_VERSIONS, choices=settings.WEBAPPS_PHP_FPM_VERSIONS,
initial=settings.WEBAPPS_PHP_FPM_DEFAULT_VERSION) initial=settings.WEBAPPS_PHP_FPM_DEFAULT_VERSION,
help_text=help_message)
class PHPFPMAppSerializer(serializers.Serializer): class PHPFPMAppSerializer(serializers.Serializer):
php_version = serializers.ChoiceField(label=_("PHP version"), php_version = serializers.ChoiceField(label=_("PHP version"),
choices=settings.WEBAPPS_PHP_FPM_VERSIONS, choices=settings.WEBAPPS_PHP_FPM_VERSIONS,
default=settings.WEBAPPS_PHP_FPM_DEFAULT_VERSION) default=settings.WEBAPPS_PHP_FPM_DEFAULT_VERSION,
help_text=help_message)
class PHPFPMApp(PHPAppType): class PHPFPMApp(PHPAppType):
@ -91,13 +98,15 @@ class PHPFPMApp(PHPAppType):
class PHPFCGIDAppForm(PluginDataForm): class PHPFCGIDAppForm(PluginDataForm):
php_version = forms.ChoiceField(label=_("PHP version"), php_version = forms.ChoiceField(label=_("PHP version"),
choices=settings.WEBAPPS_PHP_FCGID_VERSIONS, choices=settings.WEBAPPS_PHP_FCGID_VERSIONS,
initial=settings.WEBAPPS_PHP_FCGID_DEFAULT_VERSION) initial=settings.WEBAPPS_PHP_FCGID_DEFAULT_VERSION,
help_text=help_message)
class PHPFCGIDAppSerializer(serializers.Serializer): class PHPFCGIDAppSerializer(serializers.Serializer):
php_version = serializers.ChoiceField(label=_("PHP version"), php_version = serializers.ChoiceField(label=_("PHP version"),
choices=settings.WEBAPPS_PHP_FCGID_VERSIONS, choices=settings.WEBAPPS_PHP_FCGID_VERSIONS,
default=settings.WEBAPPS_PHP_FCGID_DEFAULT_VERSION) default=settings.WEBAPPS_PHP_FCGID_DEFAULT_VERSION,
help_text=help_message)
class PHPFCGIDApp(PHPAppType): class PHPFCGIDApp(PHPAppType):