Mozdev.org

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

• hg testing released; two projects have asked for testing repos already
• secure update.rdf generation released; there's been some additional interest on the list, but no direct questions
• sent policy update to PO list; no objections so far
• setup admin interface to select active VCS and initialize hg repository
• left for hg: logging of commits to db, writing documentation, figuring out how to deal w/ source.html, figuring out what to do w/ /source/browse

Discussed sysadmin priorities

• connection limitations are in place for CGIs (bugzilla, cvsweb, mailman)
• bots/spiders have been hard on our server recently so this was very helpful
• several software packages were updated on both servers

Web stats/Top 50 page

• still ongoing

Staging server migration

• no updates

Additional SSL Certs

• hg.mozdev.org will be needed
• look into bugzilla.mozdev.org?
• in the future we'll need svn.mozdev.org
• wildcard cert? We're not sure about the techincal details w/ that yet
• probably cheaper to keep buying individual certs for the near future (4x $30/yr < $200/yr wildcard)

Next meeting May 13th, 2008 @ 15:00 UTC in #mozdev

Present: cdn-work (Chris Neale), davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)
Community present: ccaygill, djc, JesperHansen

Discussion was held publically in #mozdev

Discussed developer priorities

• worked on the backend for choosing VCS and initializing mercurial
• didn't get much time to work on the admin interface; working on that now
• some more cleanup from the spam a week ago
• research into what the proper mime type for .rdf files should be (application/rdf+xml)
• fixed password reset/change forms (missed from php5 upgrade)
• been short on time the past couple weeks due to being down a car and being driver for the family
• Doug will look into any recent errors that should be fixed

Discussed sysadmin priorities

• mostly trying to find spam sources and potential spam sources on our server
• some minor non-user-visible config changes
• doesn't look like there is any spam being generated from hosted php scripts

drupal admin messages going to webmaster@

• spam notifications were fixed to go to the site email (which is set to the project owner) rather than user 1 (which is always webmaster@)

Web stats/Top 50 page

• top 50 page is still blank; web stats are missing for april
• processing choked on new log file format; restarted this morning

email delay

• email was blocked from Saturday to this morning
• virus scanner died

Staging server migration

• no updates

Minor updates for unmaintained projects

• still waiting for Doug to write a policy and post to the PO list; will try to address this week

pay drupal developer to write wiki page on configuring discussed defaults

• the documentation would help with existing projects and new installs until we have the time to update the default install profile
• more discussion needed

Next meeting May 6th, 2008 @ 15:00 UTC in #mozdev

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner), djc (autocopy extension owner)

Discussion was held publically in #mozdev

Discussed developer priorities

• spam attack last week has been cleaned up; old notes system is currently disabled
• mercurial is setup on staging at hg.vebzom.org; authentication is currently working against tigris database using existing CVS perms
• currently working on getting admin tools setup to select VCS and create repositories
• once admin tools are in place (hopefully this week) we can have devs test this on staging
• no response from community for secure updates testing; just going to release it as "beta" and deal with bugs later

Discussed sysadmin priorities

• spam attack ate up all the time last week
• logging of outgoing messages was broken since PHP 5 upgrade; fixed and improved
• gjm is monitoring abuse@mozdev now and replied to our upstreams about our recent spam attack
• some security patches were applied to servers
• copied production apache/nginx config to staging for Doug

Web stats/Top 50 page

• log analysis has been restarted (about a month behind)

Staging server migration

• no updates

Firefox updates server load handling

• handled firefox 2.0.0.14 release well

Cooperation between AMO and Mozdev.org on abandoned project adoption policy

• djc adopted auto-copy extension on mozdev.org bug would like to take over addon on AMO
• https://bugzilla.mozilla.org/show_bug.cgi?id=430048
• would be nice if Mozdev could work w/ Addons on this type of thing

ericjung would like to bring passwordmaker.org's mediawiki install back to mozdev.org

• should be possible; we'll need to work on the configuration a little bit

announce Mozdev.org sysadmin meetings?

• Doug will post the next meeting time in his minutes
• we should try to get the agenda for the upcoming week in a wiki for people to edit/comment

project status update form is being spammed

• gjm will post to sysadmin list to ensure it's safe to be disabled
• Doug will look into removing the form and links to it

Next meeting April 29th, 2008 @ 15:00 UTC

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

• got mercurial plan together; started getting mercurial setup on vebzom
• deployed update.rdf generation, but need some testers
• various bug triaging
• we have some mod_rewrite bugs still lingering; trying to get my test suite fixed to track them down
• otherwise working on mercurial and update.rdf testing

Discussed sysadmin priorities

• still need to work on log rotation scripts
• fixing mod_rewrite bugs as they come up
• apache2 is serving http and https

Review roadmap changes

• MXR tool (replaces LXR) support, CVS, SVN, and hg
• updated roadmap will be announced to POs later today

Staging server migration

• no updates

Firefox updates server load handling

• no updates

Present: cdn-work (Chris Neale), davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

• testing apache 2.2/php 5 changes
• created project tagging policy/docs
• started update.rdf generation for secure updates
• started setting up test suite for mozdev.org code
• this week is planned to be: finish up the hg setup plan (in progress), start working on hg, and try to get some sysadmin time to close some bugs (apache 2.2 rewrite bugs, web-visible cache directory setup)
• asked about working on mercurial before svn; davidwboswell says an update to the roadmap is coming soon

Discussed sysadmin priorities

• apache 2.2 and php 5 upgrade is complete; working on ironing out some bugs
• https is being setup and tested

Firefox updates server load handling

• no updates

Staging server migration

• no news on VMs
• server move should be highest priority now that apache/php setup is done

Other projects

• if staging server setup drags on too long, sysadmin might move on to cvs perms with pam auth
• project creation automation would be a good item to work on as well
• openid would be a nice authentication mechanism for users; integrating this with other auth mechanisms isn't understood well right now

Code testing

• mostly just unit tests now that PHP 5 is avaialble (PHP 4 couldn't do mock objects which was very limiting)
• Doug will be writing some tests to verify certain web paths are working correctly
• not really focused on full integration testing or continuous integration right now (the entire web stack isn't in version control, so we can't know when changes are made)

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

• project overview page/secure installs was generally well-received
• several minor bugs in file management, presentation, and extension parsing were handled quickly
• working on a tagging policy to help project owners know what to expect to be approved: http://www.mozdev.org/drupal/wiki/MozdevProjectTagging
• improved download counters offered by mozdev
• setup web-visible cache directory on the staging server w/ info for migration to production
• work on update.rdf generation is going well; script to generate files is mostly complete; need to add 'update info' url to file management and test the resulting update.rdf files; this will require the web-visible cache directory setup to finish
• should be starting on subversion setup plan this week and filing bugs using email I sent earlier to sysadmin@ as a base
• visiting family this weekend; offline Friday

Discussed sysadmin priorities

• nginx has been in production for about a week and has been handling load from updates fine
• web stats are currently broken (since the 25th) due to nginx setup and apache2/php5 update
• apache2/php5 is being setup in production today
• backups have not been occuring (reason isn't known yet); gjm is working w/ osuosl to get them working again

Firefox updates server load handling

• update was handled well; there were several spikes in bandwidth/requests but they were handled by nginx
• silfreed/gjm will continue to work on improving nginx setup by serving more files, but the solution worked incredibly well
• some projects get lots of 404 hits for their update.rdf requests; gjm will make a list of projects and silfreed will contact POs to make sure they're aware their users don't have an upgrade path

Staging server migration

• mozilla has said "yes" to 2U rack space and probably a VM
• osuosl is having disk capacity problems so a VM isn't immediately available
• first plans for for VM (hopefully from OSUOSL) is moving download master
• separating email and web traffic would probably be the next step
• figuring out how to split projects up across servers would also be useful from a security viewpoint (split mozdev.org projects from hosted projects)
• we'll still need to ask TWS to ship our server when we decide where its going (Mozilla or retirement)

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner), ccaygill (MyCroft project)

Discussion was held publically in #mozdev

Discussed developer priorities

• about ready to release the secure installation/project overview stuff; been working w/ a PO on testing things; trying to figure out a bug in the file release part, but otherwise things seem good
• plan to get the secure installation announced and in production this week
• did some testing of the lightweight web server last week and couldn't find any problems with the setup on vebzom
• fixed a bug in Drupal authentication module when editing mozdev cvs users
• changed the season starts for mozdev's themes to the equinoxes/solstices
• secure updates shouldn't take too long - it's mostly backend stuff that won't really have a UI anyway; hopefully only a week left

Database policy (bug#15661)

• we need to separate our policy decision from our technical limitations
• Doug will file a separate bug about needing to be able to monitor what our project usage is

Discussed sysadmin priorities

• preparations for apache2/php5 are underway
• setup php eaccelerator at the end of last week
• looking at spam filtering to reduce amount of cpu time used by increasing number of firewalled hosts

Firefox updates server load handling

• 2.0.0.13 release is scheduled for today
• still need to reduce amount of pages that get handled by apache/php

Staging server migration

• moco might be able to provide a VM; hopefully hear back later this week
• no news about VMs from OSUOSL yet

Present: davidwboswell (David Boswell), cdn (Chris Neale), ericjung (Eric Jung), gjm_home (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

• spring theme is updated
• the project overview page is mostly complete; just trying to get POs to help with testing
• started looking into subversion and mercurial configuration
• will be focusing on testing nginx to help get it deployed soon
• will get web-visible save directory setup soon

Discussed sysadmin priorities

• started filtering spam bots on production; blocking bots after 10 spam/week
• project creation scripts were updated to force lowercase names
• nginx is setup on the staging server; performs much better for serving static content than apache

Automated testing of Mozdev.org

• could be doable w/ Selenium
• would be neat to get the community to suggest/create tests for mozdev.org
• Doug will file a bug

Firefox updates server load handling

• covered in developer/sysadmin updates

Staging server migration

• no updates

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

• working on getting a project overview page
• this will have links for a project (bugs, source, etc) as well as list some basic information, stats, and downloads for a project
• mostly needed for the downloads section to give people a place to do secure installations from
• started working on the spring theme for Mozdev.org this week
• fixed up a small bug with the new apache rewrite rules
• fixed a bug on D.MD.o with recently uploaded files that prevented users from downloading them
• trying to tie bugzilla products to mozdev projects more tightly so its easier to tell if a project has a bugzilla product and link between them
• ericjung would like to see Doug's efforts be diverted into projects with a wider audience
• other than the spring theme, work is done from the roadmap. making sure the Mozdev server can handle the firefox updates and releases has been a recent priority and I've been working towards secure installation and updates as per the roadmap

Discussed sysadmin priorities

• focused on fixing small bugs that have been going on for awhile since the new server was setup (several years ago)
• root email was going unnoticed; gjm started going through the emails and trying to coordinate with Doug to get problems fixed
• bugzilla stats were broken; server has been working on this since yesterday
• FreeBSD security update was applied to staging & production
• web content was synced from production -> staging to help with Apache2/PHP5 testing
• trying to secure the server by limiting where mysql passwords are stored
• moved some more old things from /sandbox to cruft
• log rotation is being improved
• mysql tables are being backed up to individual files instead of large per-database files
• newsgroup creation was tested and updated
• minor reconfiguration of ntpd, dns
• lowered max # of smtp servers to help reduce load on server today

Firefox updates server load handling

• gjm and silfreed have a plan in place to start serving update.rdf requests from a light-weight web server to allow more requests to be handled
• Mozdev will also work with Mycroft to help them serve their update files directly from disk instead of going through PHP
• last will be making downloads.mozdev.org able to serve update.rdf files from the lightweight web server
• progress should be made by next week

Staging server migration

• OSUOSL might have VMs available; gjm is going to look into obtaining one

Subversion priority

• Mozdev is missing some high-profile projects due to only supporting CVS
• there is only a couple weeks worth of work to get secure installations and updates setup; server load handling is interspersed in that
• "subversion support" is actually a rather large problem due to Mozdev's CVS integration up to this point; Doug will write an email with the problems and migration plans he has so far
• current plan is to continue with the current roadmap plus fixing server load handling problems and get to subversion support ASAP

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

• working on the UI to allow project owners to verify and release their downloads and extensions
• working on mockups for a project overview page that would contain the publicly available downloads
• trying to cleanup the queries used on D.MD.o
  • reduce CPU usage a bit by throwing users to a random mirror for update requests instead of their best mirror
  • able to speed up the query that gave users their best mirror
• the extension list and tagged projects for an application are now merged into one list that shows both
• notified POs that the Apache upgrade will be happening March 31st and helping POs prepare for that

Discussed sysadmin priorities

• stats processing is up to date and the cause of the backlog is being addressed
• more spam blocks are in place
• there's a problem w/ spamassassin dying and letting spam through but the cause hasn't been found yet
• thinking about discarding all spam to lists that has a high spam score

Firefox 3 release

• mycroft is currently the biggest bottleneck when mass updates happen
• we should focus on helping them get static updates provided
• mycroft and downloads might have problems with updates not going through their scripts because stats will no longer be counted

Staging server migration

• no updates

SSL Cert renewal

• new cert is in place for 02/2008 - 03/2009

Protected wiki for sysadmin docs

• server docs were setup on a protected wiki page that only "mozdev" people can view/edit
• we can setup perms per-page on the www wiki or setup a protected wiki on admin

Per-project databases

• we don't know what projects currently have access to mysql
• projects currently can't configure db access w/o exposing the password
• need to work to move projects w/ existing mysql access to their own dbs and then work on improving the policy to allow general access

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

• skipped; silfreed not present

Discussed sysadmin priorities

• gjm has written some python classes to deal with mailman configs; this makes it easier to do validation & reconfiguration
• defaults for new mailman lists have been setup
• gjm has been reviewing site content and submitting bugs

Firefox 3 release

• no updates

Staging server migration

• migration proposals sent to sysadmin list; people will review and comment there

Dealing with newsgroups

• project creation scripts are now fixed to setup newsgroup/mailman sync

Present: davidwboswell (David Boswell), cdn-work (Chris Neale), ericjung (Eric Jung), gjm_home (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

• sick last week; didn't get a lot of stuff done
• mostly spent time fixing minor bugs with the new application/extension parsing
• started working on interface for managing download files (allowing public releases, verifying hashes) to get ready for secure installations
• for site reorganization I'm going to be working on getting a list of content put together in the wiki to see how it could be organized
• trying to figure out how to relieve some load from D.MD.o; best I can think of doing is disabling the new download tracking
• possibly disable the download tracking when server is over some specified threshold? possibly; will need to investigate

Discussed sysadmin priorities

• gjm was ill last week
• tanker helped fix permissions on project

Firefox 3 release

• looking for ideas to get our server through the update cycle that goes along with a release
• disable services dynamically under load
• offload some services to vebzom.org
• migrate postfix/mailman/bugzilla to another server
• looking up an IP's mirror is a heavy operation; silfreed might look into caching these results
• we need a better understanding of how much a user hits our site when performing updates; silfreed will analyze log files

Mailman defaults

• we'll change the defaults for new lists and point people to the wiki page if they need to reduce their list management time

Staging server migration

• proposal still needs typed
• OSUOSL admins say it would be possible to have a second [physical] machine hosted there

Dealing with newsgroups

• gjm worked with Pete to get newsgroup creation scripted, tested, and ready to integrate with project creation scripts. still needs implemented

Present: davidwboswell (David Boswell), cdn-work (Chris Neale), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

• project searching now searches words 3 characters and more (so searches on things like 'rss' or 'moz' work now)
• we're automatically parsing extensions and pulling extension information and supported application information
• the application list with descriptions, icons, and links to home pages is live
• the list of extensions per application is live; still needs some work to enable features such as the "latest version"
• project searching now includes applications and extensions
• will be spending some time soon thinking about the redesign and how to make finding things on our site easier

Discussed sysadmin priorities

• more research into the CVS mysql auth patch - it's going to be incredibly difficult to port this to a more recent CVS so that we could add additional features to it
• we will be investigating using pam_mysql/nss_mysql to manage CVS permissions on the filesystem instead
• pam_mysql will make implementing other services easier as well (ftp, ssh, VCSes)
• 15m downtime on Friday due to a spike in the server load caused by firefox release
• spam scanning has been improved to use in-memory partition for parsing messages (increases throughput)
• spam scanning has been stricter and is now blocking ~800k dailing (up from ~140k)
• web log analysis is moving faster; a caching DNS server was installed to reduce DNS latency
• firewall config was updated to allow more connections
• apache2/php5 update has been on hold; Doug says PHP code should be ready and will need more testers soon

Mailman defaults

• less spam should help a bit, but no new updates
• we should document the changes we are looking to change in the wiki so that we can point people there when they have questions

Staging server migration

• plan is still in the works; needs typed up an emailed out

Dealing with newsgroups

• no updates

Present: davidwboswell (David Boswell), cdn-work (Chris Neale), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

• project search interface is live; needs some tweaks and am coordinating with sysadmins
• parsing extensions for supported applications is underway; I'll be starting testing soon, then need to create a UI for searching and displaying the extensions per application

Roadmap updates

• What's left before subversion? What apps are supported by an extension (bug12486), secure installs (bug#17302)
• ericjung would like to see an online version of mccoy
• Doug will send a note to the PO list to see if anyone is interested in helping code the backend; if not, we'll try to fit it into the roadmap

Discussed sysadmin priorities

• stats processing is behind; logs are being generated quicker than being parsed
• checking on mailman default settings bug#17667

Staging server migration

• gjm and tanker have some proposals; discussion will continue in the sysadmin list to include all parties
• being able to support SSL for some subdomains would be helpful - for example, bugzilla - more discussion will occur on-list

Dealing with newsgroups

• gjm has created a "sanitycheck" python class to verify list/news setup and has lots of "issues found"

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), ericjung(Eric Jung), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

• Mozdev source code should be publically available now in hovercraft/sandbox/php
• admin tools are updated to work with PHP upgrade
• project search interface is almost complete; see http://www.mozdev.org/projects/search.html
• starting to track what applications are supported by projects this week

Discussed sysadmin priorities

• PHP on mozdev.org was updated from 4.4.7 to 4.4.8
• Bugzilla now supports UTF8 properly; there were some problems with migrating the data and will be dealt with on a case-by-case basis
• CVS hiccup this week that caused the cvs server to max out the CPU
• TWS is no longer authoritative for mozdev.org DNS
• some private lists were publically available via /mailarchives/
• quick way of putting the site in maintenance mode was implemented for the downtime this week

Staging server migration

• strg.at has started making plans for hardware/hosting/VMs
• OSUOSL is offering VMs, but doesn't have much disk space currently so that offering is currently on hold
• nothing definitive about using mozilla for hosting yet
• vebzom.org is still giving disk errors; the /mozdev/ partition may be on its way out
• will need to migrate mirror service off the vebzom server

Dealing with newsgroups

• gjm is looking into project and newsgroup creation; should have more info soon