MySQL A.B. has released MySQL Connector/J 5.0.7, the Type-IV all-Java JDBC driver for MySQL that's "suitable for use with any MySQL version including MySQL-4.1, MySQL-5.0, MySQL-5.1 beta or the MySQL-6.0 Falcon development releases." 5.0.7 adds some TCP configuration options and fixes bugs.
The Gnu Project has released version 4.2.1 of GCC, the GNU Compiler Collection. GCC contains frontends for C, C++, Objective C, Fortran, Ada, and Java as well as libraries for these languages. GCC's Java is a clean room implementation that doesn't use any Sun code, so it doesn't always exactly match Sun release versions, but this is roughly at the Java 1.4 level with some omissions. 4.2.1 is a bug fix release and "will be the last release of GCC covered by version 2 of the GNU General Public License. All future releases will be released under GPL version 3."
Friday kicks off with an information-packed keynote from two of eBay's architects, Randy Shoup and Dan Pritchett. I can't type fast enough to keep up with them. I cheated and talked to Randy earlier in the Speaker's lounge. They have a few petabytes of data going back to the site's founding, split across a couple of dozen Oracle databases. They use a custom object-relational mapping layer to manage queries across independent databases. (Please open source that. I need it.)
Dan starts with the database side:
- Scale out, not up.
- They also do functional decomposition: separate selling from searching from bidding, etc.
- Prefer asynchronous integration.
- Design for failure
They're on version 3 of the architecture. eBay started with commodity Fry's hardware, Perl, and open source software, every item in a separate file. Limited to 50,000 files. Now they use 4-way Solaris 2900 boxes, Java, and Oracle. Power and cooling is the biggest current issue.
Version 2 was C++ in IIS. Database was moved to Oracle. Microsoft Index Server was introduced for search. Used Resonate for load balancing and spread horizontally. Still a single database though. Failover database introduced here. November 1999 they're down for a day and on CNN. This motivated splitting the databases.
December 2002 current DB architecture introduced, many different databases. Application is out of control though. It's a two-tiered architecture. 2 million+ lines of code in a single ISAPI DLL. Hundreds of developers are working on this. Integration nightmare. This motivates a move to J2EE and Java. Presentation logic was moved out of C++ into XSL! Introduced developer API.
Version 3 is Java.
- Database segmentation by function: user hosts, item hosts, account hosts, feedback hosts, transaction hosts, and 70 more. Different functions have different availability and other requirements. Also helps enforce functional isolation and prevents coupling. Quotes Eric Billingsley: "If you can't split it, you can't scale it." Dan: "At eBay.com, vertical scaling is just not in our vocabulary."
- Horizontal splits within functions: multiple instances per database. Write master/read slaves.
- Limit the database work.
- No transactions. They use autocommit. They sometimes do transaction logic for a single database within the application Java code using Oracle specific features. They do recover from failed operations. Listing an item updates 73 tables. There's no reasonable way to do this in a transaction.
- "Not a single stored procedure. Not one." but they do have simple triggers. They do try to keep logic out of the database.
- Performance: CPU intensive work goes into the applications, including referential integrity checking! They do use unique key constraints, but that's it. They make extensive use of prepared statements and bind variables. They do joins, but not big ones, and mostly along primary key. They don't use ordering.
The horizontal splitting of the databases is transparent to the application developers. They don't need to know how many physical servers they're talking to. However, 15,000 app servers at four DB connections per server. Connection management is a problem. Oracle can't handle 60,000 connections. They have to cheat to handle this.
Different rows have very different characteristics; e.g. a user with one feedback and a user with half a million feedbacks.
Amazon has the exact same rules. Brewer's CAP theorem: Pick 2 out of 3: consistency, availability, scalability (partition tolerance). They and Amazon discovered this empirically.
Now Randy starts talking about scaling the application tier (as opposed to the database tier).
- Segmentation by function.
- Application tier is stateless, which enables horizontal scaling.
- Minimize dependencies between search, selling, etc.
- Virtualize data access
- Throw out most of J2EE. Use servlets, not much else.
- Keep application server completely stateless. No session state. Transient state maintained in a cookie or another database.
- Cache where possible.
They'd like to put more than 4K of data in a cookie, and they believe they can't do multipage flows without cookies. (They're wrong about that one, by the way.) They're in process of inventing server-side cookies to get around this. Now if they just threw out the cookies and stored the ID of the server side record in the URL, they'd be RESTful. They did have some scalability issues where the 4K cookies came back with each of 70 requests for 70 images on one page. They had to reorganize to avoid that. They do try to avoid session state, but they haven't achieved stateless nirvana yet (my opinion, not theirs).
The 4K limit in cookies is a surprise. I didn't know about that. URLs can hold more than 4K. I'm about to run out of battery power. I think I got most of the critical details down anyway.
I wanted to hear Granville Miller talk about Agile Architecture but that room is just too full. If the fire marshal shows up, we're in trouble. The conference has 5 class rooms running simultaneously and they're all about the same size. That has often meant the most popular session or two in any one time slot has people sitting in the aisles, and the others feel relatively empty.
First afternoon session Anthony Mattei talks about "When Every Click Counts: Building Mission Critical Enterprise Systems". "Fault tolerance is not a new concept." He cites fault tolerance architectures such as Enniac going back to 1946. Tandem et al. available in the 1980s.
SLA is an outward facing document to the customer about how your system will behave. However it also has internal value to the engineers. Distinguish between planned and unplanned downtime. Avoid single points of failure. Publish/subscribe has pretty much supplanted point-to-point messaging.
Thursday kicks off with IBM's Bill Lewis talking about "Data Driven Application Engineering." I was torn between this session and another one on Designing Domain Specific Languages in Static and Dynamic Languages, but this talk seemed a little more relevant to what I'm doing right now.
"Data wants to behave in a certain normalized way." I asked him how he defines data, because I know a lot of content that does not want to behave or be normalized. He defines it as "labeled values". "Normalization gives you a car factory in your garage."
Scott Ambler, Bob Martin, and myself had a really nice panel at lunch on Tough Problems in Unit Testing. (No notes from this one. It was all off the cuff.) Later in the afternoon, I talked about User Interface Principles in API Design to a standing room only crowd.
I've arrived at Architecture & Design World after an early morning flight. For future reference and anyone who hasn't arrived yet, to get here you take the Blue line from the airport to Jackson. Change at Jackson for the Red Line heading south to Dan Ryan/95th. Get off at Chinatown and walk east on West Cermak, away from Chinatown until you see the Hyatt. (There are also a couple of buses on Cermak Road but I didn't feel like waiting.) Total cost: $2.00. Time spent: about 90 minutes, roughly the same as the van I shared last year from the airport since it stopped at five other downtown hotels first. The subway would have been quicker except I missed the Chinatown stop and had to double back.
There's a standing-room only crowd for Scott Masumoto's Architectural Risk Analysis: Design Security Into Your Application. He warns us that the last chance error handler that dumps a stack trace to the user is a security risk. There's a lot of other good advice including writing "abuse cases" as well as use cases.
In Ivar Jacobsen's lunch keynote he reveals that he's pretty much given up on large, all-encompassing "processes" like RUP. He's now advocating picking and choosing from a grab-bag of individual "practices", which is what most shops were doing anyway.
My own talk on Testing Legacy Code went quite well. There was a lot of interest, both in legacy issues and general unit testing concerns. Surprisingly a lot of people still aren't yet test infected, but the virus is spreading.
CMP has posted the call for abstracts for Software Development 2008 West. The conference takes place March 3-7, 2008 in Santa Clara. Once again I'll be chairing the XML track. As well as the usual C++, Java, .NET, Security, Modeling & Design, Ruby, and Web Development. The latter is sort of a catch-all for AJAX, Web 2.0, PHP, JavaScript and and other web tech that isn't already addressed in one of the existing tracks. Submissions are due by August 27.
By the way, I'll be at CMP's Architecture & Design World in Chicago later this week. There's still room for one more person on the Tough Problems in Unit Testing Panel with myself, Scott Ambler, and Bob Martin on Thursday. If anyone would like to participate, please drop me a line. It's always a good time.
The Omni Group has released
OmniGraffle 4.2,
a general purpose Mac OS X diagramming tool and my
UML editor of choice. 4.2 adds a Document Settings Inspector, can
convert basic shapes to custom polygonal/bezier shapes and vice versa, merges layers, and
improves Visio import and export.
OmniGraffle ranges from $79.95 to $149.95. Upgrades from 3.x and earlier are $29.95. Mac OS X 10.3 or later is required.
Sun has submitted JSR-318, Enterprise JavaBeans 3.1, to the Java Community Process. According to the JSR,
The purpose of the Enterprise JavaBeans 3.1 specification is to further simplify the EJB architecture by reducing its complexity from the developer's point of view, while also adding new functionality in response to the needs of the community.
The focus will be on the core session bean and message-driven bean component models and their client API. Although the Java Persistence API was developed within EJB 3.0, it will evolve under a separate JSR rather than within EJB 3.1.
Aspects that should be considered by the Expert Group for inclusion in this work include, but are not limited to, the following:
- Removal of the requirement for a separate local business interface.
- Support for direct use of EJBs in the servlet container, including simplified packaging options.
- Singleton beans.
- Support for asynchronous session bean invocation.
- Support for stateful web services via stateful session bean web service endpoints.
- Specification of concurrency options for stateful session beans.
- Application-level callback notifications, including for container initialization and shutdown.
- EJB Timer Service enhancements to support cron-like scheduling, deployment-time timer creation, and stateful session bean timed objects.
- An ejb-jar level component environment to simplify the specification of shared dependencies among components.
Comments are due by July 30.
Sun has submitted JSR-317, Java Persistence 2.0, to the Java Community Process. According to the JSR,
The purpose of the Java Persistence 2.0 specification is to augment the Java Persistence API to include further features requested by the community, including additional object/relational mapping functionality and query language capabilities, a criteria-based query API, and standardization of features currently designated as optional, and to align it with related JSRs that are currently in-process and/on in-plan for the Java EE 6 timeframe.
Aspects that should be considered by the Expert Group for inclusion in this work include, but are not limited to, the following:
- Expanded object/relational mapping functionality, including greater flexibility in combining existing mapping options, support for collections of embedded objects, multiple levels of embedded objects, ordered lists, combinations of access types, etc.
- Additions to the Java Persistence query language
- An API for "criteria" queries
- Standardization of sets of "hints" for query configuration and for entity manager configuration
- Standardization of additional metadata to support DDL generation and "Java2DB" mapping
- Expanded pluggability contracts to support efficient passivation and replication of extended persistence contexts in Java EE environments
- Standardization of additional contracts for entity detachment and merge, and persistence context management
- Support for validation
Comments are due by July 30.
Atlassian has released Bamboo 1.2, a $960 payware continuous integration server
that "provides Build Telemetry to help identify and highlight trends, patterns, and linkages across builds — not just focusing on the results of a single build."
new features in 1.2 include "global permissions, external database support, Perforce triggering support, Regex labeling of build results, scheduled backups, new plugins and more."
Sun's posted the third update to the JDK 1.6.0 for Windows. The full internal version number for this update release is 1.6.0_02-b06 (where "b" means "build"). (The Linux and Solaris releases don't seem to have been updated yet.) It's not immediately clear what changed since b05 a couple of weeks ago, but I suspect this release plugs a couple of security holes.
Sun has posted the twelfth update release for Java 5.0 Tiger with fixes for a few dozen assorted bugs. "The full internal version number for this update release is 1.5.0_12-b04 (where "b" means "build"). The external version number is 5.0u12." As usual, it's available for Solaris, Windows, and Linux.
I got my author copy of Beautiful Code: Leading Programmers Explain How They Think this past week. It looks very nice, if a little heavy for reading on the subway. (I get all my best reading done on the subway these days. Maybe if I go to work a little early I can be guaranteed a seat.) Amazon has it in stock for immediate shipment with a 34% discount. It's eligible for free super-saver shipping. Author royalties on this book are dedicated to Amnesty International.
Day Software has posted the public review draft of JSR-283 Content Repository for Java Technology API 2.0 to the JCP. According to the JSR:
Since this JSR represents an enhancement of JSR-170, the same general goals apply to this JSR as to JSR-170 (from the JSR-170 proposal):
The aim is to produce a content repository API that provides an implementation independent way to access content bi-directionally on a granular level. A content repository is a high-level information management system that is a superset of traditional data repositories. A content repository implements "content services" such as: author based versioning, full textual searching, fine grained access control, content categorization and content event monitoring. It is these "content services" that differentiate a content repository from a data repository. Many of today's (web) applications interact with content repositories in various ways. This API proposes that content repositories have a dedicated, standard way of interaction with applications that deal with content. This API will focus on transactional read/write access, binary content (stream operations), textual content, full-text searching, filtering, observation, versioning, handling of hard and soft structured content.
In particular, the following functional areas will be reviewed by the expert group for possible inclusion in version 2.0:
- Extensions in the area of management of a content repository such as access control management, workspace and nodetype management, retention aspects of content or repository construction patterns.
- Improvement of content repository interoperability through the addition of new standardized node types, including node types for meta information and internationalization.
- Extensions to content modelling capabilities.
- Federation, cross-repository and cross-workspace functionality.
- Active development of existing query-languages, versioning and observation.
- Remoting and client/server protocol mappings.
- Possibly other enhancements.
The spec is over 400 pages long. While it appears relatively well written and thought out compared to other JCP specs, the sheer length gives me pause. I certainly don't have time to review it, but comments are due by September 20.
Atlassian has released version 3.10 of JIRA, a $1200-$4800 payware J2EE-based bug tracking and project management server application. 3.10 adds editable worklogs, start Dates for worklogs, new ways to browse components and versions, and an AJAX-based user picker and issue picker. I've been using Jira lately with Jaxen and Apache. It's a definite improvement over Bugzilla. I'm not sure it really does anything that Bugzilla doesn't do (at least not anything I use) but the user interface is about a hundred times cleaner.
Besides finally adding full-screen playback, Apple's recent QuickTime 7.2 update seems to be causing problems for people who've also installed the Java 6 developer preview on their Mac. This may not be immediately obvious because the application the break, such as Office, don't have any obvious connection to either QuickTime or Java. Deinstalling Java 6 will probably fix the problem.
The Apache Commons Project has released CLI 1.1, a "simple API for working with command line arguments and options." This is mostly a bug fix release with some API changes.
The Apache Jakarta Project has released POI 3.0.1, an open source Java library for "manipulating various file formats based upon Microsoft's OLE 2 Compound Document format. OLE 2 Compound Document Format based files include most Microsoft Office files such as XLS and DOC." Version 3.0.1 fixes bugs.
In what I think is a first for the JCP, the micro edition Executive Committee has unanimously voted not to approve JSR 190 - Event Tracking API for J2ME. The concerns seem to be related to the licensing of the spec and reference implementation, not any technical issues. Committee member Jean-Marie Dautelle commented, "It has been brought to my attention that the license terms are still not compliant (despite the latest changes). Specifically, there is no explicit right to create an independent implementation and right to redistribute the RI. In order to see a new ballot with compliant license, I have no choice but to vote No." Other comments were similar.
I haven't been following this JSR closely, so I'm not sure what happened. or if it is likely to be repaired.
JSR 280, XML API for Java ME, has also been rejected for similar reasons, though not unanimously. Intel voted No with the following comment:
The Spec Leads need to assure the community that the licensing regime for JSR 280 will meet the spirit and the letter of the JSPA. Our preference for accomplishing this would be for the Spec Leads to provide the text of TCK and RI licenses that they commit to offer to any interested party and that meet the requirements of the JSPA.
Questions on this JSR from the EC have focused on the RI license. The Spec Leads have distributed to the EC a sample RI license. While we have no objection to the Spec Leads offering that license as one alternative, its restrictions on redistribution make it unacceptable in the role of the fulfilling the Spec Leads obligations for RI licensing as expressed in the JSPA.
To avoid any possible confusion, we note that RI licensing is a separate and distinct issue from TCK licensing, which has potentially a more wide-ranging impact than RI licensing, as exemplified by the situation encountered with JSR 176 Java SE 5. The Apache Open Letter on JSR 176 Java SE 5 claimed that field of use restrictions are being imposed on how Independent Implementations of the spec can be used. JCP requirements for licensing compatibility tests must not be used to limit or restrict Independent Implementations of JCP specs. We would be more comfortable if the Spec Lead had supplied a complete sample TCK license, but we are hopeful that, nevertheless, TCK licensing will not be an issue for JSR 280.
Macromates has released Textmate 1.5.6, a €39 payware text editor for Mac OS X. 1.5.6 adds a database browser in the SQL bundle.
BlueJ 2.2, a free integrated development environment
(IDE) for Java aimed at education, has been released. "BlueJ now explicitly supports team work based on CVS (with subversion
support to be added later in the year). BlueJ presents a simple
interface to a subset of CVS functionality (sharing of projects,
concurrent editing and merging, history)."
Sun's posted the second update to the JDK 1.6.0 for Windows, Linux, and Solaris. "The full internal version number for this update release is 1.6.0_02-b05 (where "b" means "build"). The external version number is 6u05." (Just calling it 6.0.2 would have made way too much sense.) There are about 180 or so bug fixes in this release.
Sun's posted the tenth milestone of NetBeans 6.0, its pure Java IDE for various platforms. Additions include:
- New default color scheme and syntax coloring
- Ruby Debugger enhancements (global vars, watch view, locals view)
- Redesigned Find / Replace dialog
- Plugin Manager allows you to distribute groups of IDE components (for example Ruby, SOA)
- Integrated Visual Design for Web Applications
- New, Integrated UI for CLDC/MIDP and CDC development
Sun has submitted JSR-316, Java Platform, Enterprise Edition 6 (Java EE 6) Specification , to the Java Community Process. According to the JSR,
This JSR will not itself define any new APIs, rather it will enumerate APIs defined in other JSRs or through the JCP maintenance process.
Java EE 6 is the Enterprise Edition of version 6 of the Java platform, and thus will be built on Java SE 6.
We propose to include the following new JSRs in Java EE 6:
- JSR-196 Java Authentication SPI for Containers
- JSR-236 Timer for Application Servers
- JSR-237 Work Manager for Application Servers
- JSR-299 Web Beans
- JSR-311 JAX-RS: Java API for RESTful Web Services
In addition, we expect to update the following existing technologies for Java EE 6:
- Enterprise JavaBeans
- Java Persistence API
- Servlets
- JavaServer Faces
- JAX-WS
- Java EE Connector API
The exact set of technologies to be included will be determined by the expert group and will be based on partner and customer requirements.
Comments are due by July 16.
The Apache Commons Team has released Commons IO 1.3.2, a class library that provides utility classes and methods for copying streams, endianness conversion, safely closing streams, creating Strings and byte arrays from streams and Readers, filtering files, and querying and manipulating the file system. Version 1.3.1 fixes a couple of bugs. Commons IO is published under the Apache license.
Sun has released Java3D 1.5.1. New features in this release include:
- Windows Vista Support
- Automatic fallback to D3D pipeline if OpenGL unavailable
- JOALMixer-based audio engine
- Automatic mipmap generation
- Pack200 support for Java Web Start applications
- Logging Framework
- Support for new JNLPAppletLauncher utility (based on JOGLAppletLauncher)
Of course, there are many bug fixes too. Java 5 or later is required.
MySQL has posted the first beta of Connector/J 5.1.2, a Type-IV pure-Java JDBC driver for MySQL. "Setting the configuration property rewriteBatchedStatements to true will now cause the driver to rewrite batched prepared statements with more than 3 parameter sets in a batch into multi-statements (separated by ";") if they are not plain (i.e. without SELECT or ON DUPLICATE KEY UPDATE clauses) INSERT or REPLACE statements."
The Eclipse Project has released Eclipse 3.3, an open source integrated development environment (IDE) for Java and C++. It also doubles as a base platform for your own applications, an alternative to the AWT and Swing, and a powerful floor wax and dessert topping. New features version 3.3 include Vista support, an XML Editor, improved Undo, and Mylyn, a task-focused user interface that integrates with Bugzilla, Trac, and Jira. Java 5 or later is required, though it can compile for earlier versions of Java.
Andrei Kouznetsov has released Unified I/O 3.0, an open source (BSD license) class library that "allows random access to any data or stream (even over HTTP), and gives a clear difference between read only and read/write access." Version 3.0 This release adds a new buffering model and can create child streams with shared or independent stream position.
Websina has released BugZero 5.0, a $1299 payware (+$300 for maintenance) Web-based bug tracking system that supports multiple projects, group-based access, automatic bug assignment, file attachment, email notification, and metric reports. Bug Zero is written in Java and can run on top of various backend databases including MySQL. 5.0 adds field dependencies and an AJAX interface.