Friday, September 02, 2005

Scrum

I finished reading the book Agile Software Development with Scrum by Ken Schwaber and Mike Beedle and discovered that there is a way to develop software that actually makes sense! Great book.

Having developed software for 13 years now for a few different companies, I've come across a number of different methodologies, each touting the reasons why they will work. One fundamental difference between all those and Scrum is that they all have adopted a "defined" process control model, whereas Scrum uses an "empirical" process control model to manage the development effort.

A defined process is one in which, given a set of inputs, you can pretty much guarantee a certain output. You can do this over and over again. This works great in the manufacturing industry. Years ago, someone or 'someones' thought this would work in software, and we've never really gotten away from it. The defined process can be used successfully when every piece of work is completely understood. A defined process can be started and run to completion with the same results every time.

How often is this the case with software development? ... never, yeah never. Software development is complex, unpredictable and never completely defined up front. This type of problem requires an empirical control model. An empirical process "provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs." (pg 25)

Another thing that Scrum fosters and promotes is the idea of self-organizing teams. Set a couple of goals for a team of 4-8 people to accomplish in 30 days and let them go for it. Check in with them each day to monitor progress and identify impediments (the empirical process). The team is 100% responsible for getting the job done and will organize themselves to be successful.

The Scrum process is essentially this:
1. Create a list of bugs, enhancements, features, etc. (The Product Backlog)
2. Prioritize the list.
3. Create a team and appoint a Scrum Master for the team who is responsible for enforcing Scrum practices.
4. Have the team select 30 days worth of the highest priority items in the list.
5. Begin the 30-day Sprint where the team works uninterrupted for 30 days.
6. Meet each morning for a 15 minute Daily Scrum Meeting to identify impediments, report on progress from the previous day, and identify what each team member will do today.
7. After 30 days, hold a Sprint Review where everyone gets to show off the executable product code they've created.

There's a whole lot more to it than this, so read the book! After you've read it and agree that it is a great way to development software, you can use ScrumTool (shameless plug), a great, web-based tool to help you manage your Scrum projects.

Monday, August 01, 2005

iDebtManager Goes Live

iDebtManager, which is Phase II of Ideal Financial Solutions' CashFlow Management System, has been completed and released for the general public.

iDebtManager allows users of the CashFlow Management system to stay up-to-date on the progress they are making in becoming debt free by tracking the balances on all debts being managed by the CashFlow Management System. All of the user's debts that are being managed by the CashFlow Management System can be viewed individually and payment amounts can be adjusted if desired. New debts can also be added to the system.

A Payoff Timeline is available which allows users to see when each debt is scheduled to be paid off as well as the progress they are making to become debt free. The Payoff Timeline gives the user the ability to compare how he is actually doing in his efforts to become debt free against the CashFlow Management System's proposed plan.

To see a demonstration of iDebtManager's functionality, visit the Ideal Financial Solutions' website and select the iDebtManager Demo link.

Friday, July 15, 2005

Running regular, periodic jobs with Quartz

Many applications have the need to run regular, periodic jobs/processes. Such is the case with the application I'm working on. One option is to write your own job processing engine, but why reinvent the wheel. Another option is to use classes provided in the Java SDK, java.util.Timer and TimerTask, but there are limitations here. For example, there is a very limited set of scheduling mechanisms and the timers don't use a thread pool.

I came across the Quartz Job Scheduler, an open-source project by OpenSymphony. What a great product! I have integrated it into two different Struts-based web applications and it was quite simple to do.

There are a couple of ways to integrate it. One way is by adding a reference to the Quartz servlet in your web.xml file. I did have a couple of problems with this approach, so I opted for integrating using a Struts plugin. The problem I encountered with the first approach was that I couldn't get the Scheduler to stop running after I stopped my servlet container (Tomcat 5.0.x). I suspect it may have been user error. If not, the problem could well have been resolved with the later releases.

The plugin approach was quite simple. Add a line like this to your struts-config.xml file:

<plug-in className="com.mycompany.plugin.QuartzPlugin"/>

Then create the QuartzPlugin class. It may look something like this:

public class QuartzPlugin implements PlugIn {
private String MY_JOB_GROUP = "MyJobGroup";
Scheduler sched = null;

public void init(ActionServlet servlet, ModuleConfig moduleConfig) throws ServletException {
log.info("Quartz starting");

try {
SchedulerFactory schedFact = new org.quartz.impl.StdSchedulerFactory();
sched = schedFact.getScheduler();
sched.start();

// Register job listeners with the scheduler
sched.addJobListener(new EmailWhenJobCompletesListener());

// Create the MyJob
JobDetail jobDetail = new JobDetail("MyJob", MY_JOB_GROUP, MyJob.class);

// Auto delete job from scheduler when no active triggers are associated with it.
jobDetail.setDurability(false);

// Do not persist between restarts of scheduler.
jobDetail.setVolatility(true);

// Pass stuff to the job with the JobDataMap.
String basePath = servlet.getServletContext().getRealPath("/");
jobDetail.getJobDataMap().put("basePath", basePath);
jobDetail.getJobDataMap().put("emailRecipients", MyConstants.JOB_STATUS_EMAIL_ADDRESS);
jobDetail.getJobDataMap().put("jobName", "My Job");

// Assign the EmailWhenJobCompletesListener to listen to this job and
// send an email when it is done.
jobDetail.addJobListener(EmailWhenJobCompletesListener.LISTENER_NAME);

// This is a trigger that will fire every night at 2:00am.
String cronExpression = "0 0 2 * * ?";
CronTrigger trigger = new CronTrigger("MyTrigger",MY_JOB_GROUP, "MyJob", MY_JOB_GROUP, cronExpression);

try {
// Schedule the job with the Scheduler.
sched.scheduleJob(jobDetail, trigger);
log.info("Scheduled the MyJob - " + cronExpression);
} catch (SchedulerException e) {
log.info("Quartz Scheduler failed to schedule the My Job: " + e.toString());
throw new ServletException(e);
}

} catch (Exception e) {
log.info("Quartz Scheduler failed to initialize: " + e.toString());
throw new ServletException(e); }
log.debug("Quartz started");
}

public void destroy() {
log.info("Quartz stopping");

try {
sched.shutdown();
} catch (SchedulerException ex) {
ex.printStackTrace();
}

sched = null;
}
}


And finally, create the job class, i.e., the class that will do what you want the job to do.

public class MyJob implements Job {
public MyJob() { }
public void execute(JobExecutionContext context) throws JobExecutionException {
JobDataMap dataMap = context.getJobDetail().getJobDataMap();
// Get a variable I passed to the job.
String basePath = dataMap.getString("basePath");
// Do stuff...

Tuesday, March 29, 2005

CashFlow Management Goes Live

After 7 months of development, the CashFlow Management project I have been working on with Ideal Financial Solutions has made it into production! There are, of course, a number of bugs that need to be worked out and a list of enhancements that will consume another 6 months of my nights and weekends, but it feels good to have something out in the market that is useable.

It has been quite a unique project to say the least. Ideal has a system they had created over the years and at the core was a monster of an Excel spreadsheet. I was asked to mimick the logic of the spreadsheet. Another unique aspect of the project was that the people involved in the project are separated geographically, so most meetings and communications were by phone or email. To make a long story short, everyone involved learned a lot about how to develop software and manage expectations.

The CashFlow Management system is one which helps people get out of debt and build wealth. A typical customer would provide Ideal, or one of its marketing call centers, with all of his financial data, including salary, monthly savings, and all debts and mortgages. This data is captured using the web-based application that my development team and I created. The data is then analyzed by the Ideal financial experts using tools provided in our web-based app. A proposal is created for the customer, which may include refinancing, credit card workout or settlement, adjustment of W2 withholdings, etc. The customer can then login to the web-based application and review it. If he chooses, he can sign-up for the program. His responsibility is simply to make sure that on a given day of the month, he has a certain amount of money in his checking account. This amount is the same amount, or less, than he is currently paying toward his debts. Ideal pays his debts off for him in the most effective way. It's not uncommon to see a 20+ year debt-free date cut to under 10 years.

Friday, February 25, 2005

iTracker - Open source bug tracker

I am working on a Debt Reduction/Wealth Builder software project for Ideal Financial Solutions where the team is dispersed geopgraphically. We have two developers, two testers, and some project manager types in four locations. We made it through the initial development phase with weekly phone calls, emails, and a few in-person meetings, but when we hit the testing phase, we had some big problems.

We started out by tracking the bugs in a couple of Word documents that we would email back and forth. The developers would add their comments on the issues and email the docs to the testers and vice versa. This got unwieldy after a couple of weeks, not to mention the attachments of the emails kept getting larger and larger.

I decided to look for an open-source (free) bug tracking tool and found iTracker. We've only been using it a couple of days now, but what a difference it has and will make for us.

I think my biggest gripe is the lack of detailed documentation, but that is to be expected with most open-source software. I originally wanted to use a MS SQL Server database as the backend, but quickly discovered I would have to spend a weekend pouring over JBoss installation/configuration documents in order to understand exactly what the limited documentation was asking me to do. (I only have Tomcat experience.)

Thank goodness that iTracker ships with an "Express Install". The Express Install uses HSQL, not sure if that is a JBoss specific persistence mechanism, or another open-source package. Using the Express Install I was up and running in less than an hour. I still need to figure out why the email notification isn't working. It appears to be an SMTP server problem. I contacted my ISP and they gave me the SMTP server info, but email notification is still not working.

I remember seeing some posts at the iTracker forum regarding troubleshooting email notifications. I'll have to do a little research and get it going.

Update - Email notification now working
Since I was running outside of my ISP's network, I had to provide authentication information. After I specified that in the itrackerApplication.properties file I still had problems. The last thing I had to do was provide valid email addresses in itrackerApplication.properties. I had some bogus email address for the from and reply. After changing them to real email addresses, email notifications started working.

Friday, February 04, 2005

jTDS - The fastest JDBC driver for MSSQL

jTDS is an open source 100% pure Java (type 4) JDBC driver for Microsoft SQL Server. It claims to be the fastest JDBC driver around and backs the claim with some benchmarks. All you have to do is drop a jar in the classpath, change your jdbc driver settings and url, and you're good to go, or so they say.

I dropped the jar file, modified my jdbc connection info as specified in their FAQs, but now my app won't start. I suspect it's my fault somehow, but it is a bit perplexing.

My app has some startup code that loops through all the resources in the environment context "java:comp/env". For some reason, my jTDS datasource is not found in the environment. If I replace the Driver and URL with the original Microsoft driver values, it does show up in the environment.

Here is my resource entry in Tomcat server.xml file:

<resource type="net.sourceforge.jtds.jdbcx.JtdsDataSource" auth="Container" name="mypool">
<resourceparams name="mypool">
<parameter>
<name>factory</name>
<value>org.apache.commons.dbcp.BasicDataSourceFactory</value>
</parameter>
<parameter>
<name>maxActive</name>
<value>50</value>
</parameter>
<parameter>
<name>maxIdle</name>
<value>25</value>
</parameter>
<parameter>
<name>maxWait</name>
<value>10000</value>
</parameter>
<parameter>
<name>username</name>
<value>the_user</value>
</parameter>
<parameter>
<name>password</name>
<value>the_password</value>
</parameter>
<parameter>
<name>driverClassName</name>
<value>net.sourceforge.jtds.jdbc.Driver</value>
</parameter>
<parameter>
<name>url</name>
<value>jdbc:jtds:sqlserver://server1:52500/dev_2.2;
user=the_user;password=the_password</value>
</parameter>
<parameter>
<name>validationQuery</name>
<value>SELECT 1</value>
</parameter>
<parameter>
<name>logAbandoned</name>
<value>true</value>
</parameter>
<parameter>
<name>removeAbandoned</name>
<value>true</value>
</parameter>
<parameter>
<name>removeAbandonedTimeout</name>
<value>300</value>
</parameter>
</resourceparams>


And the code that fails is:

Context initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");
Object obj = envCtx.lookup("mypool");

After the last line of code above, obj is null. For some reason, the datasource is not getting into the environment context.

I posted the above problem on a jTDS forum (http://sourceforge.net/forum/forum.php?thread_id=1226037&forum_id=104389) and received a suggestion that fixed the problem.

I needed to change "net.sourceforge.jtds.jdbcx.JtdsDataSource" to "javax.sql.DataSource" in the <resource> element. I think I misunderstood the jTDS documentation. There was a table that had two columns, one column had "javax.sql.DataSource" and the other had "net.sourceforge.jtds.jdbcx.JtdsDataSource". I thought it was a simple "search and replace" job, but not so.

Anyway, it's up and running now!

StrutsWS (Web Service)

StrutsWS is a relativley new open-source project on Sourceforge http://sourceforge.net/projects/strutsws/. Frank Zametti, the author, has mentioned it various times recently on the struts user list. He mentioned today that he had just released a new MAJOR update, so I decided to take a look.

Here is Frank's abstract of the project:

This project is an attempt to create a mechanism in Struts whereby existing business logic components of a Struts-based application can be exposed as rudimentary Web Services without the need to change any code. To do this, I have created a custom RequestProcessor class. This class recognizes a Web Service request, that is, a SOAP message over HTTP, performs some pre-processing on the request, and passes it along to the targeted Action in a form it will understand. It then generates a SOAP-based XML response to the request in place of the usual view components of the existing application.

I read through the overview document and it sounded like an interesting project. It's main strength, in my opinion, is that you can use your existing Actions and ActionForms. There is no need to rewrite business objects to make them web services. Unfortunately, if an application makes heavy use of Session data, like session-scoped ActionForms or for handling security, then the StrutsWS library is of little value, since you now cannot use existing Actions and ActionForms. To quote Frank, "And of course, anything requiring session will NOT work, since no session would exist for a one-off request like these. "

I need to investigate Apache Axis a bit more and see how complicated it is use install and use.

Thursday, February 03, 2005

Trying out formdef

I decided to try out Hubert Rabago's formdef Struts plugin. After working through some silly mistakes, like getting the correct version of the formdef library for Struts 1.1, I think I finally got it working.

The framework we originally put together did not address the issue of business objects and Struts ActionForms very well. We naively just added as a member variable to each ActionForm (ValidatorForm) a data transfer object (DTO). These DTOs were the same ones used by the data access layer, so the data types of the variables in these "beans" were data types that the database expected. This is not ideal, as web forms should typically only deal with strings and maybe booleans.

As we got further into implementation and development using this paradigm, we discovered that we had to worry about data type conversions for displaying the data on a web form. It got a little cumbersome as a developer.

I stumbled across the formdef Struts plugin and it sounded like a good option. It introduced us to Dyna forms, which we had read about, but never investigated much. They're actually quite nice and save the developer the time of having to create ActionForm classes for each form. formdef takes it one step further by creating your dyna ActionForms for you based on a DTO (bean) that you specify, so the ActionForms match the business objects.

formdef also provides built in conversion methods to move data from the DTO to the form and vice versa. It is a very flexible library, providing many hooks and overrideables, so it should serve us well.

There was a thread on the Struts user list Nov 2004 that talked about this subject a little. Here is the URL to the original post. You can step through the response using the "next in thread" option. http://marc.theaimsgroup.com/?l=struts-user&m=110001898604664&w=2

[UPDATE 27Aug05]
The past couple of days I have tried to use formdef in a real web application and have run into a few problems. First off, I don't think formdef supports multi-select lists, i.e., <select multiple="multiple">. Second, I'm not sure how to implement a one-to-many relationship using formdef. Let me explain...

I use iBATIS for the DAO and db connection functionality. One feature iBATIS provides is that I can populate a DTO object from a query and the DTO can have a List property that stores the "many" side of the relationship. iBATIS will populate that List property automatically for me with all the related db rows.

I have not had success in creating the DynaValidatorForm when I have a List object in my DTO. formdef seems to choke on it, because when I remove it, everything is fine. Maybe there is a better way to handle one-to-many relationships when using formdef, not sure ??

Tuesday, February 01, 2005

Initial Post

I decided to start my own blog! I'll be blogging stuff I learn and discover as I work on a variety of computer consulting/contracting jobs. It'll be nice to have a repository for such findings, but really, it just kinda makes me feel cool that I am a blogger now :)