Sergio Del Piccolo

Not wanting to spoil any K2 announcements and presentations at the Microsoft SharePoint in Las Vegas, I decided to wait until now to start posting on some of the events and information at the K2 Insider Conference held last week.

Keynote Presentation

The conference started of with the Keynote Presentation by SourceCode CEO, Adriaan Van Wyk. Adriaan spoke about SourceCode’s need to ensure that software quality and performance were the number one priority for the K2 line of products. He went into the roadmap going forward and mentioned that even in the current economic climate SourceCode was performing well – profitable and cash flow positive.

He reiterated that K2 blackpearl was all about solving business problems, and was it was the natural progression from K2.net 2003  solving some of the latter’s challenges, a lack of proper event monitoring, policies and reporting functionality. In addition, it added the SmartObjects layer which provides a  single source of data for workflows.

Going forward, the next major release for K2 blackpearl is currently called 090x. It includes some new features and enhancements to the current version, including:

1. K2 blackpearl and K2 blackpoint will originate from the same code base. As a result K2 blackpearl will include all of the features that currently exist in the K2 blackpoint. K2 blackpoint will be a true subset of K2 blackpearl.
2. K2 connect will be updated for 90x compatibility (clustering made easier).
3. Enhanced installer platform allowing for:
   • Simple / full installation.
   • Custom installation.
   • A configuration manager and health checking feature to make for a simple installation process.
4. Platform enhancements including:
   • Workflow Server enhancements to improve scalability and resource management.
   • Smart Objects enhancements to improve performance and scalability.
   • New APIs and enhancements to existing ones.
5. An improved browser experience, including the following new features:
   • A new K2 Designer for Microsoft SharePoint.
   • Inline Functions, the ability to include more functions in workflows without needing to resort to code.
   • Process portals for Microsoft SharePoint.
   • Enhanced reporting.
6. An improved development experience including:
   • K2 Studio for K2 blackpearl development. The K2 Studio is an Office-like development tool geared at power-users.
   • Visual Studio 2008 and Visual Studio 2005 support.
   • Inline Functions.
   • SmartObject Filters.

While no official release date for the 090x version was announced, it can be assured that it will be after a stringent QA cycle has been completed, ensuring the highest quality with a much user experience being the top priority.

K2 Release Management

This provided a nice segue into the Release Management presentation by Olaf Wagner, Senior VP for Product Management. Olaf spoke about the four main areas of SourceCode’s release management cycle:

1. K2 Release Strategy:
   • Major releases every 3/4 years. The version being slated for 2011.
   • Minor releases twice per year. The next release being 090x. Minor releases will Includes new features.
   • Updates (hotfixes) to be released every 2nd Tuesday. They are cumulative but customers can access older updates.
2. Support strategy:
   • Major release support – includes the current and previous version (e.g. K2.net 2003)
   • Minor release support – includes the current and previous minor version of current major version, i.e. currently K2 blackpearl.
   • Updates release support – current updates are supported
3. Coexistence:
   • It is possible to export from older client to an updated server unless specifically stated in release notes.
4. Product Shipping Vehicle:
   • It will be simpler with the single codebase after the release of 090x.
   • K2 blackpoint will be a true subset of K2 blackpearl.
   • All products will ship on the same day, which means that new features will not ship in one product ahead of another.
   • The above mentioned point will allow for enhanced quality assurance from SourceCode for the entire product line.

What’s new in K2 blackpearl 090x

The next presentation by Holly Anderson and Renier Britz from SourceCode delved into the new features and enhancements around the upcoming 090x release. These include so many new features and enhancements, I’ve decided to list them in point form:

1. Installation
   • The Installation and Configuration wizard are merged into a single wizard for a simpler installation process.
   • New install options allowing for simple or custom installation.
   • A new Configuration Analysis Tool to check the K2 environment for potential issues.
   • The ability to install to a non-Active Directory environment. It must be noted that there is no support Forms based authentication in SharePoint at this time.
   • The ability to perform unattended installs.
   • The option to activate K2 features on default SharePoint site collections.
   • Auto-chaining with the latest K2 update.
2. Administration
   • K2 has been updated for SharePoint administration tools.
   • All K2 features are deployed as solutions to the SharePoint environment.
   • The ability to manage the K2 server from SharePoint.
   • Enhanced SharePoint SmartObject administration.
   • The Process Portal site will be available on any SharePoint site.
3. Infrastructure
   • Scalability enhancements.
   • SmartObject performance enhancements.
   • New Dynamic SQL ServiceObject supporting stored procedures (with Out parameter), views and tables.
   • Windows 2008 R2 and SQL Server 2008 support.
   • SSL support for SharePoint.
   • Internationalization support.
4. End User Experience
   • Process Portals with process and user reports.
   • Silverlight based View Flow.
5. Design and Development Experience
   • K2 Designer for Visual Studio
     • Visual Studio 2008 support
     • Enhancements to ADO.Net data provider including: Paging, ANSI SQL Support and Query performance enhancements.
     • ServiceObject APIs.
     • Instance Management APIs.
     • Inline Functions including Math, String formatting, Expressions, Integration with SmartObjects. These are also extendable.
   • K2 Studio
     • Targets non-developers and SharePoint power users. It’s a full breadth designer, minus the code.
     • K2 studio is an office-like think client designer.
     • No Visual Studio dependencies.
     • No additional licensing requirements.
     • Assemble item and non-item based workflow applications and solutions.
     • No code.
     • Line-Of-Business (LOB) integration.
     • A Reference Wizard which can call out to an external assembly/web service.
     • Use SharePoint records management and publishing.
   • K2 Designer for SharePoint
     • Silverlight based workflow designer aimed at less technical users and Microsoft Office power users.
     • It requires no client site installation and includes a rich design experience.
     • The layout is workflow controlled and is focused on item-based workflows
     • The ability to quickly create SharePoint based workflow applications and solutions with no need to burden IT resources.
     • No coding required.
6. General
   • InfoPath forms supports.
   • Inline Functions.
   • Custom task notifications.
   • Task groups allowing multiple destination sets.
   • Export process to K2 studio (downstream movement only).
   • Process data fields.
   • Use SmartObject in process design.
   • Ability to connect sibling nodes.
   • Save workflows and workflow steps into favourites.
   • New wizards, including:
     • Set folio and process data field.
     • Switch InfoPath view.
   • The ability to save email templates.
   • The ability to split workflow from the Start activity.
   • Process Portals – administration view of process and instances.

Custom Inline Functions

Koos Du Preez, SourceCode CTO, spent some time going over the Inline Functions to be introduced in the 090x version, and how to build custom ones.

Building custom Inline Functions are quite simple. The steps are:

1. Create .Net 3.5 project Visual Studio 2008.
2. Add references to:
   • SourceCode.Framework
   • SourceCode.Workflow.Functions
   • SourceCode.Workflow.Functions.Design
   • System.Drawing
3. Add Using statements for the referenced assemblies.
4. Declare the class with a Category (single category per class) – you can reuse K2 categories or create your own.
5. Declare the “Name”, “Description” and “Icon” methods for the inline function.
6. Declare method parameters as function parameters (inside the method signature).
7. Provide the function implementation (use static functions).

To install the function the Design Time steps are:

1. Quit the designer.
2. Copy the assembly to K2 Bin folder
3. Restart designer.

The Runtime steps are:

1. Stop the K2 Host Server.
2. Copy the assembly to K2 bin folder.
3. Start the K2 Host Server.

It is important to note that you will need to manually deploy on alls clients and servers for all environments and reference the assembly deployed manually (including in the GAC).

Debugging the functions is simple to. Simply start the K2 Host server, attach to  the K2HostServer.exe process and debug.

090x Performance & Stability Enhancements

Koos also spent some explaining some the performance and stability enhancements that will come with the 090x version. These include:

1. Design Experience
   • There was a big effort to analyze process memory allocation and usage patterns using tools like windbg with SOS, CLR profiler, perfmon and of course code reviews.
     • IDisposable pattern extensively
     • Freed memory early disposing large resources like images, icons, etc.
     • Replaced all string manipulations with StringBuilder
     • Remove unnecessary cloning and creation of objects
     • Wizard framework rewritten using XAML and control templates. No control via code.
2. Runtime
   • K2 Workflow Server
     • There was a big effort to analyze process memory allocation and usage patterns using tools like windbg with SOS, CLR profiler, perfmon, code review
       • Ensure unused objects disposed early
       • Process runtime state management improvements (kept current version only)
       • Bypassed socket transmissions for local IPC’s
       • IPC authentication cashing implemented
     • Used SQL Profiler to isolate longest and most expensive queries
       • Common Table Expressions rewrite of Worklist (not using temp tables anymore)
       • Actions per Item refactored to use Common Table Expressions for action union
       • Additional indexes on Transaction and Log databases
       • Added constraints to force integrity
     • Validating investment
       • The Microsoft Technology Centre experience, where SourceCode did some benchmarking using 0807 release.
3. SmartObjects & ADO.NET
   • High Level Enhancements
     • There was a great deal of SmartObject stack analysis, to minimise client/server calls.
     • “Join Discovery” identified as a bottleneck & implemented relationship caching.
     • Bypassed discovery and force/allow the use of SQL joins in reports.
     • Schema retrieval indentified as bottleneck & implemented object definition caching.
     • ADO data sets returns less than 15-20k results in batch (rather than row by row).
     • ADO Direct Execution (Including Bulk Insert) – option for Service Object to directly execute SQL code. Supported by
       • Dynamic SQL Service
       • SmartBox
       • K2 Reports
     • Compressed Double Buffering (more involved than a one-liner).
     • Double Buffered Data Readers.
     • Server Side Paging (SmartObject API & ADO).
     • Greatly enhanced SQL Syntax support for ADO.
4. Reporting
   • Pushed 50 GB database and analysed built-in reports.
   • SQL Profiler to most expensive queries.
     • Pushed for ADO based paging (SSRS does local paging)
     • Report Query optimizations
     • Altered log database indexes

With that we ended the first day of the conference. It made for a long but rewarding day of sessions. Next, to dinner and a chance to meet some of the new Insiders.

SourceCode, K2, K2 blackpearl, K2 blackpoint, 090x, Microsoft SharePoint

This week sees the start of the annual K2 Conference. It will be held at the MGM Grand in Las Vegas from October 16th – 18th. The conference is attended by senior management from SourceCode and K2 Insiders. For those of you unfamiliar with K2 Insiders, they are K2 advocates who share their expertise within the K2 community (and are not SourceCode employees).

This year I will have the opportunity to present the Partner Integrated Workflow program that I have been working on at Avantage Partners. It involves streamlining business processes among multiple organisations, allowing them to become more efficient in their business practices and thus gaining a competitive advantage in the market. K2 blackpearl makes for the perfect tool to implement a Partner Integrated Workflow solution.

As I did last year, I will be blogging on the developments at the conference for any interested readers.

Last year’s posts: Day 1 |  Day 2 |  Day 3

SourceCode, K2, K2 blackpearl, K2 blackpoint

The Microsoft SOA & Business Process Conference sessions for 2009 are available for download and viewing. The conference which was held at the end of January contained some interesting sessions on SOA, BizTalk, Oslo and Dublin.

The content can be viewed on Windows Live here: http://soaconference2009.spaces.live.com/default.aspx

One of the best sessions I attended was on the ESB Guidance 2.0 presented by Peter Kelcey. Peter’s session can be found here: http://soaconference2009.spaces.live.com/blog/cns!FA6FC7F5DB1C07!136.entry

Microsoft, Microsoft SOA Conference, BizTalk,   Enterprise Service Bus Guidance, ESB Guidance Toolkit, Peter Kelcey

For those of you attending the Connected Systems Conference in Vancouver, I’m presenting the session on ESB Guidance 2.0 and Service Virtualization with Jim Bowyer from Microsoft Canada. This will be an exciting (geek license invoked) look into the upcoming release of the ESB Guidance v 2.0 and the enhancements it promises to deliver. Also covered in the presentation will be the topic of service virtualization using the Managed Services Engine (MSE). If you haven’t heard of the MSE, it’s worth looking at as a part of a balanced and robust SOA infrastructure.

It’s not too late register. Click here to register for Vancouver and here to find links to the other Canadian conference venues.

If you do plan on coming to the conference, make sure to stop by and say hello.

Microsoft, BizTalk, BizTalk Server, SOA, Connected Systems, Vancouver, Canada, ESB, Enterprise Service Bus, MSE, Managed Services Engine, CodePlex

The annual Microsoft SOA & Business Process Conference 2009 will be held in Redmond from January 27th-30th 2009. There have been a couple of changes to the conference that is usually held every October but is being pushed back this year due to PDC 2008 running at the same time.

There are a couple of editions year, one for Partners and the other for Customers. The Partner briefing will be being held on January 26th-27th and the Customer briefing on January 28th-30th.

There are also a couple of tracks offered for the conference:

• SOA & BPM Best Practices – highlighting case studies and technical trends providing guidance and information on how to achieve a robust and sound long-term SOA strategy.
• Technology Offerings – deep dives into existing and future technologies including .Net Framework, BizTalk Server, Visual Studio, Cloud Services and Oslo.

Pricing is $599 for partners and $899 for customers (although customers who register by December 1st, 2008 will only pay $599).

Links:
Partner registration
Customer registration

Microsoft, Microsoft SOA Conference, SOA, Business Process, BizTalk

In a previous post I mentioned that the Canadian Connected Systems Team are planning on hosting a roadshow in three cities across Canada this fall.  I managed to get some information regarding the agenda and registration to the events.

The roadshow dates and registration details are:

Vancouver – 30th October 2008 – register
Toronto – 4th November 2008- register
Montreal – 6th November 2008 – register

Agenda

Microsoft, BizTalk, BizTalk Server, SOA, Connected Systems, Vancouver, Canada

The Canadian Connected Systems Team are planning on putting on a roadshow in three cities across Canada this fall. This would be a great opportunity to hear more about BizTalk and how it relates to broader SOA initiatives. The roadshow dates are as follows:

Vancouver – 30th October 2008
Toronto – 4th November 2008
Montreal – 6th November 2008

Watch this space for more information regarding the agenda and registration details.

Microsoft, BizTalk, BizTalk Server, SOA, Connected Systems, Vancouver, Canada

K2 blackpearl has been selected as the winner of the Best of Tech Ed 2008 Developers Awards for SharePoint Development.

“…K2 blackpearl is a prime example of a new breed of applications that helps developers get the most out of their SharePoint investment. Using a collection of easy to use tools and features, K2 blackpearl allows developers to quickly create applications that pull together disparate systems, information, and business processes. K2 blackpearl can leverage your existing infrastructure investment—including SharePoint, but also BizTalk and Forms Server—to surface the application mashups you create…”

For more see the article on Windows IT Pro.

K2, SourceCode, blackpearl, Microsoft Office SharePoint Server, MOSS, Tech Ed 2008

The final day of the conference came around to quickly. Soon it would be time to go. I can happily report that I won a handsome bet of $1 last night at the bowling alley!

Development Reference
Mike Talley, SourceCode

Mike went through the development reference that comes with K2 [blackpearl]. He went trough what existed and what was planned. The following is planned for release this year:

i – Database schemas of all 14 K2 databases.
i – Architecture diagrams and discussions.
ii – More How To topics e.g. how to debug.
iv – Event Bus topics e.g. calling another another assembly, starting a process.
v – Impersonate a user (and reverting to a user).

Training
Codi Kaji, Source Code

Codi discussed planned training content for the year, including:

i – A K2 Infrastructure course
ii – A Getting Started guide
iii – A K2 Developers course – once the Development Reference is complete

Smart Objects
Jason Apergis, Colin Murphy and Gabe Malherbe (K2 Insiders)

Jason, Colin and Gabe went through a discussion of Smart Objects, highlighting the advantages and mentioning pitfalls to avoid. They started with a quick overview of K2.net 2003 advantages and shortcomings:

i – It was easy to drop into code from wizards
ii – When  using code to access external systems, it was time consuming, complex is not generally considered a best practice.
iii – When storing workflow data in XML or process data fields, you could run into performance issues when storing large amounts of data.
iv – When creating custom wizards for non-technical users to consume your custom code, it was very time consuming and could lead to problems during deployment.

When using K2 [blackpearl], Smart Objects can solve many of the aforementioned shortcomings.

i – Smart Objects are reusable business entities which are deployed centrally and available to be consumed by non-technical business workflow authors.
ii – The SmartBox is the storage location for process data.
iii – Allows for quick access to external Line of Business (LOB) data.
iv – Allows for the aggregation of data from multiple back-end systems (e.g. AD, SAP, SQL Server) into a single composite object.
v – Can be created without writing code.
vi – Can be accessed by the workflow and can be used for reporting.

There was some discussion about the positioning of Smart Objects within the application layer (see the diagram below) and some things to consider:

i – Don’t circumvent the data warehouse.
ii – Think carefully about whether there is a need to save the data from the Smart Object in the workflow context to the database layer.
iii – Consider the reporting benefit that comes from the Smart Objects.

Diagram 1 – The Smart Object position within the application layer.

Next came a discussion about Smart Objects services. These services are wrappers to functionality that Smart Objects can call. The following are available out of the box:

i – SmartBox (although these can’t be used for custom SQL Server databases).
ii – Microsoft Office SharePoint Server 2007
iii – Microsoft Active Directory
iv – K2 [blackpearl]
v – SalesForce.com

There are more available on K2 [blackmarket]:

i – Dynamic SQL Server Stored Procedure Service
ii – Dynamic Smart Object Services Service
iii – Dynamic Web Services Service
iv – Microsoft Office Communication Server Service
v – SharePoint Users in Groups Service

K2 [blackmarket] is part of K2 Underground – the community site that allows community to create and share projects and code on the K2 Underground site.

Forms Discussion
Mike Talley, SourceCode

Mike led a discussion about the advantages and disadvantages of using InfoPath and ASP.net for forms development in a K2 Solution. The table below summarizes the discussion.

So after three action packed days, the inaugural K2 Insiders Conference comes to an end. It was a great opportunity to meet some of the people behind K2 and other Insiders. Thanks to Chris who went to a great deal of effort to organize this. Well done!

SourceCode, K2, K2[blackpearl], K2 [blackmarket], K2 Underground, Smart Object, SmartBox, Microsoft SharePoint Office Server, MOSS, ASP.net, InfoPath 2007

Solution Architecture Overview
Bob Maggio, Architect, SourceCode

Bob went through a K2 [blackpearl] solution showing much of the detailed architecture. A couple of noteworthy items to share here are:

1. When working with a Smart Objects, you can add a process level reference to the Smart Object in the solution. By doing this, each activity or event will point to the Smart Object in the reference. Not using the referenced Smart Object results in each call to the Smart Object adding to the operating cost of your solution and thus negatively affecting performance.

2. K2.Net 2003 and K2 [blackpearl] differ regarding activity instances. In K2.net 2003, there would be an activity instance for each person assigned to a task. In K2 [blackpearl] there is now a single activity instance regardless of the number of people assigned to a task.

K2 [blackpearl] Architecture Overview
Koos du Preez, CTO, SourceCode

Koos went through the some of K2 [blackpearl] architecture. The first thing he spoke about though was performance. Although not an architecture item, the product architecture is a major consideration in achieving optimal performance. Performance will be much improved with the upcoming 803 release. While the main goals of this release are 64 bit support and improving the K2 [server] installer, it includes significant performance enhancements. The workflow server performance is much improved; primarily process execution. The worklist and worklist item performance has been improved significantly as well.  

Koos also touched on the following items:

1. Workflow Authoring

K2 allows for authoring of processes using any designer, viz. Microsoft Visual Studio 2005, K2 Studio and Microsoft Visio 2007. It has a common authoring API which generates a KPRX file used by all designers. It also contains custom configuration data and customizable pre-built workflow schedules – which don’t require code generation. The schedules are created using XAML. One important point to note is that the extender files should not be checked into source control, as it will result in the extender files becoming corrupted.

When the solution is built, the result is compiled executable artifact which are contained in a MSBuild package ready for deployment. Deployment executes the MSBuild package which in turn results in the following steps being taken:
i – Obtaining environment setting
ii – Deploying artifacts using build tasks
    a – creates MOSS artifacts
    b – uploads MOSS artifacts
    c – deploys the workflow
    d – deploys the events
    e – deploys Smart Objects

One thing to remember is that Service Objects need to be manually deployed to other environments. Lastly, a great tip regarding deployment is that when deploying using the command line tool, detailed information is returned as well as error information, e.g. an error that might be returned using this method could be “Error creating MOSS site”.

2. Runtime

At runtime, a .Net AppDomain is created on the K2 [server]. All process definitions and behaviour is loaded from the K2 databases. Isolated and versioned execution of processes start.

3. K2 Layered Architecture

i – K2 [blackpearl] comes with a K2 SSO (Single Sign-On) – see below for more.
ii – Communications are pluggable by design. Currently the K2 API to the server uses TCP on port 5555. This can be replaced by WCF or an alternative protocol if needed.
iii – Roles can be authenticated using AD (Active Directory), LDAP or Microsoft SQL Server amongst others.
iv – The K2 [server] can listen for K2.Server 2003 and K2 Studio 2003 using TCP over port 5252.

4. K2 Single Sign-On (SSO)

The K2 SSO is based on a primary ID. Line of business (LOB) IDs are attached to the primary ID. It supports multiple backend LOB credential caching using pluggable authentication providers. These can have multiple instances of the authentication providers e.g. multiple SAP instances can be attached. The K2 SSO supports Windows SSPI (NTLM and Kerberos).

The K2 SSO ships with the following authentication providers: AD; SQL; Siebel; SAP; SalesForce. The K2 SSO is pluggable and an SDK exists for creation of custom authentication providers. It is highly secure, making use of SQL Server symmetric keys in turn protected by system credentials. Caching can be set to expire or turned off. Automatic prompting, validation and caching of credentials occur the first time an event results in a LOB being hit.

K2 SSO supports seamless Smart Object integration. The way it works is as follows:
i – The API requests the Smart Object method call
ii – Marshalling calls the Smart Object Exec with broker package
iii – The Service Instance requests the LOB credentials
iv – The Security Framework accesses the SSO cache
v – If the credentials are not cached then the Security Framework prompts the user for them

When a prompt is requested, it is the result of an error being thrown. If not handled, then the process will be in error state. This needs to be taken into account when using Microsoft InfoPath as a user interface.

5. K2 Databases

There can be up to 14 K2 databases. They can be configured for performance by adjusting the threads for the processing database and the log database. In the k2server.setup file, the <K2ThreadPool> tag is set at 20 thread by default for both databases. These numbers can be adjusted e.g. processing uses 25 or 30 threads and the log 15 or 10 threads. It is also possible to rename the databases as long as the names are updated in the configuration manager.

6. Common Logging Framework

The logging framework allows for logging events to be sent to multiple targets, including:
i – Microsoft SQL Server
ii – MSMQ
iii – File
iv – Windows Event Log
v – Console

The verbosity and severity levels for each target can be set. The message queue used to send logging events can be bypassed, resulting in synchronous real time logging of messages. This does affect overall performance though. This is done by opening the HostServerLogging.config and altering the <AsyncQueueEnabled> tag. When making this alteration, the service needs to be restarted.

It is also possible to create predefined messages taking parameters using text, etc.

7. Monitoring (Windows PerfMon)

The following are tracked using PerfMon:
i – Processes Started, per second
ii – Worklist Opened, per second
iii – Worklist items finished, per second
iv – Process Memory Usage
v – Process Thread Count
vi – Process Modules Loaded
vii – TCP Bytes Retrieved Total, per second
viii – TCP Bytes Sent Total, per second
ix – TCP Concurrent Connections
vii – TCP Connections Opened, per second

Business Process Management Today
Maureen Flemming, Program Director, Business Process, Integration and Deployment Software, IDC

The afternoon session had Maureen give some insight into the market today and Business Process Management. The session was highly informative, although I did not take too many notes.

As is clearly evident, the second day was busy and managed to live up to the high standards of the day 1. Here’s to day 3, but before that there are a few bets to win at the bowling alley this evening.

SourceCode, K2, K2[blackpearl], Process Management, K2 SSO, K2 Databases, Monitoring K2, K2 Logging Framework, K2 Workflow Authoring