Integrate Crafter CMS with Jenkins to Automate DevOps: Code Forward, Content Back Process

Great DevOps helps us build better software products faster. One of the key elements of DevOps is automation within the development process across lower development environments.  Jenkins, Bamboo, Travis and many others platforms like them are used by DevOps teams to help automate the process of Continous Integration and Continous Delivery (CI/CD). To a large degree, real support for CI/CD and agile development is something that is woefully missing in Content Management, a core component of digital experience platforms.  The toolset and the architecture of today’s content management platforms make it difficult, sometimes near impossible to support let alone automate. How easy is it for your team to take content from the Production CMS and update content in lower environments like Dev, QA?  How easy is it to roll out new features?  Often times this process is not only laborious for the DevOps team, it also halts the content authoring process.    Crafter CMS’s, an open source CMS Git-based repository and Java/Spring and Groovy-backed stack are a game changer in this regard.  Further, Crafter integrates directly into to your CI/CD process.  In this article, using Jenkins as our example we’ll demonstrate how you can connect your production CMS to your developer workflow to facilitate Code Forward, Content Back(TM) workflow.
You can read more about this process in detail here:

Although our example is based on Jenkins, the scripts and flow used in this blog are applicable to nearly any automation

Understanding Where Code Forward, Content Back Automation Fits

The fact is that every step of your DevOps process is open to automation.  In this section, we’ll cover the common points of integration, specifically:

  • The point in the process where you want to move content from your production CMS “back” to lower environments to support development and testing.
  • The point where you want to promote code “forward” from the development process to the production CMS so it can be published.

Both of these points in the process are illustrated and addressed in the diagram below by the double-headed arrow labeled Code Forward, Content back.   With respect to the CMS, Code Forward, Content Back is the most important aspect of the DevOps automation.

Crafter CMS’s Git-based repository is the foundation of the automation.  Our automation running in Jenkins is going to leverage API’s within the authoring environment (Crafter Studio) to sync code and content with the development process.  APIs will also be used to publish code synced to authoring to the Production delivery infrastructure.

Implementing Code Forward, Content Back Sync

Now that we’ve established what integration we’re addressing here is, let’s focus on configuring it.  Take a look at the diagram below, this elaborates the previous diagram showing how the sync occurs.

Note that both the Production authoring and the Development “environment” has a repository.  In authoring, this is a local Git repository.  In development, this is most often a centrally hosted Git repository that supports workflow and review (like Bitbucket, Gitlab, Github, and others.)   You can think of the repository under authoring as the Content Repository and the repository supporting developers as the Code Repository.  These names (Content Repository, Code Repository) are simply labels to help describe their purpose and assist us in addressing them in the context of this article.

To facilitate this flow, the Content Repository under authoring/Crafter Studio declares the Code Repository as a remote. The primary way of “syncing” work between Git repositories is through pull and push operations.  Before you can push your work to a remote, you must first pull merge the updates (if any) from the remote.  Once done, you can push your changes to the remote.

Automating the Pull / Push of Code and Content

To help automate the process described above Crafter Studio, the authoring and repository component of Crafter CMS supports a set of APIs.  You can find a full listing of Crafter Studio APIs for Crafter 3.0 here: http://docs.craftercms.org/en/3.0/developers/projects/studio/index.html

These APIs are easily invoked by a script.   You can use the following example script in your own implementation:

codeforward-contentback-sync.sh

#!/usr/bin/env bash
studioUsername=$1
studioPassword=$2
studioserver=$3
project=$4
remote=$5
branch=$6

echo "Authenticating with authoring"
rm session.txt
curl -d '{ "username":"'$studioUsername'", "password":"'$studioPassword'" }' --cookie-jar session.txt --cookie "XSRF-TOKEN=A_VALUE" --header "X-XSRF-TOKEN:A_VALUE" --header "Content-Type: application/json"  -X POST $studioserver/studio/api/1/services/api/1/security/login.json

echo "Pull from remote (get code waiting to come to sandbox)"
curl -d '{ "site_id" :"'$project'", "remote_name":"'$remote'", "remote_branch":"'$branch'" }' --cookie session.txt --cookie "XSRF-TOKEN=A_VALUE"  --header "Content-Type: application/json" --header "X-XSRF-TOKEN:A_VALUE" -X POST $studioserver/studio/api/1/services/api/1/repo/pull-from-remote.json

echo "Push to remote (send content waiting to go to development)"
curl -d '{ "site_id" :"'$project'", "remote_name":"'$remote'", "remote_branch":"'$branch'" }' --cookie session.txt --cookie "XSRF-TOKEN=A_VALUE"  --header "Content-Type: application/json" --header "X-XSRF-TOKEN:A_VALUE" -X POST $studioserver/studio/api/1/services/api/1/repo/push-to-remote.json

Use of the script:

codeforward-contentback-sync.sh [USERNAME] [PASSWORD] [AUTHOR_SERVER_AND_PORT]  [SITE_ID] [REMOTE_NAME] [BRANCH_NAME]

USER_NAME is the Studio user (application account)
PASSWORD is the Studio user password (application account)
AUTHOR_SERVER_AND_PORT the protocol server name and port of Studio
SITE_ID the ID of the site
REMOTE_NAME the name of the upstream (typically origin)
BRANCH_NAME the name of the branch (typically master)

Example:
codeforward-contentback-sync.sh devops mydevopspw http://localhost myprojectID origin master

The script is quite simple.  It authenticates to Crafter Studio, performs a pull from the Remote Code Repository and then if there are no conflicts, performs a push.  These two operations move code updates forward to the production Sandbox (not yet live) and content back to the development process.  Only approved code that’s been moved to the “master” branch with the intention to release is moved forward.

Calling the Script in Jenkins

The first step is to create a project.  Give the project a clear name and select the Freestyle project then click OK to continue.

There is no Source Code Management (SCM) aspect of the project.  The most typical use case for Content back workflow is a scheduled event: Every hour, day, week etc.

 

The next step is to define build triggers.  Since you are calling APIs here and content back is most likely based on some schedule you define you want to indicate that there is no Source Code Management (SCM) aspect of the project.

Select “Build Periodically” and define your schedule.  Schedule definitions user standard Cron/Quartz configuration.

Finally, you must define that you want Jenkins to call your script:

Once you have done these steps you are ready to go.  Manually invoke this build any time you want directly through the Jenkins console.  I recommend testing it to make sure your parameters and schedule are correct.

Publishing Code That’s Been Sync’d to Sandbox

When you run the code forward, content back process code in the remote code repository is moved to the production authoring sandbox (content repository.)  This code is now staged for publishing.  It is not yet live.  Crafter Studio must publish the code, making it available to your delivery servers.  This in-and-of-itself is awesome: global, elastic deployment at the touch of a button.

So how is it done?  Crafter Studio provides an API that allows you to publish commit IDs.  You can provide a single commit ID or you can provide a list.  It’s typical as part of your release process to “Squash” all of the commits in a given release into a single commit ID.  This allows you to address all of the work as a single ID/moniker which makes it very easy to move, publish and roll back without missing anything.

These APIs are easily invoked by a script.   You can use the following example script in your own implementation:

publish-code.sh

#!/usr/bin/env bash
studioUsername=$1
studioPassword=$2
xsrf=AUTOMATED
studioserver=$3
project=$4
env="Live"
commit=$5

echo "Authenticating with authoring"
rm session.txt
curl -d '{ "username":"'$studioUsername'", "password":"'$studioPassword'" }' --cookie-jar session.txt --cookie "XSRF-TOKEN=A_VALUE" --header "X-XSRF-TOKEN:A_VALUE" --header "Content-Type: application/json"  -X POST $studioserver/studio/api/1/services/api/1/security/login.json

echo "Publishing Commit $commit"
curl -d '{ "site_id" :"'$project'", "environment":"'$env'", "commit_ids": ["'$commit'"] }' --cookie session.txt --cookie "XSRF-TOKEN=A_VALUE"  --header "Content-Type: application/json" --header "X-XSRF-TOKEN:A_VALUE" -X POST $studioserver/studio/api/1/services/api/1/publish/commits.json

Use of the script:

publish-code.sh [USERNAME] [PASSWORD] [AUTHOR_SERVER_AND_PORT]  [SITE_ID] [COMMIT_ID] 

USER_NAME is the Studio user (application account)
PASSWORD is the Studio user password (application account)
AUTHOR_SERVER_AND_PORT the protocol server name and port of Studio
SITE_ID the ID of the site
COMMIT_ID the squashed commit ID of the items coming from the release branch

Example:
publish-code.sh devops mydevopspw http://localhost myprojectID 378d0fc4c495b66de9820bd9af6387a1dcf636b8

The script is quite simple.  It authenticates to Crafter Studio and invokes a publish for the provided commit.  This op

Calling the Script in Jenkins

See configuration of sync script above.  The steps are exactly the same with the following differences:

  1. You will call the publish-code script instead of the codeforward-contentback script.
  2. You will ask the user for a parameter  value COMMIT_ID via the UI on each invocation and pass that to the command line as the COMMIT_ID parameter value

 

 

That’s it!  You can now publish your code releases via commits to your entire delivery infrastructure regardless of its size or distribution.

Conclusion

CMS platforms are notorious for refusing to play nice with CI/CD and agile development practices and process, automation and tools like Jenkins, Bamboo, Travis and others.  Databases and JCR repositories are one component of several fundamental, architectural limitations that make supporting CI/CD difficult for CMS platforms. Crafter is an open source, dynamic CMS with a unique Git based repository specifically designed to fit neatly in to your development practices and bring your authoring and development teams together in a way never before possible to improve and increase the rate and volume of innovation!

Integrating Crafter CMS with GitLab for Better DevOps

Content authoring and software development are both a major part of producing today’s digital experiences. Unfortunately, development support is not something traditional CMS platforms handle very well at scale. Crafter CMS, a 100% open source CMS platform that includes a Git-based repository designed to handle not only authoring but also DevOps seamlessly.

Many development teams use cloud hosted developer platforms like GitLab to assist with their development process.   Crafter’s Git based CMS supports developers working against remote repositories like GitLab, Github, Bitbucket, and others.

By supporting this kind of architecture, Crafter provides a very simple way to flow code forward from a developer and her team all the way up through the CI/CD process to production.  We also support a very simple way for any developer or any environment to easily update itself with the latest content from production.  DevOps and CMS have never been easy — until now. Crafter CMS’s CODE FORWARD, CONTENT BACK™ process makes this process simple and allows you to leverage awesome platforms like GitLab to support your team with the tools they know and love.

In this article, I’ll show you how to create a new project in GitLab and then start a new project in Crafter CMS in a way that connects to GitLab as an upstream remote repository so that GitLab can be used to support your development process with your CMS related development! 

Create a New Project and Connect it to GitLab

Step 1: Create a Project in GitLab

Figure 1: Create a project in GitLab 

  1. Select Blank Project to create a bare project
  2. Enter your project name
  3. Provide a project description
  4. Choose your security level
  5. Click create site

Once your repository is created you will see a screen similar to the one below.  You want to make note of the Git URL for the site.  You will need this URL in the next step.

Figure 2: New Project in GitLab 

Step 2: Create Your Project In Crafter Studio

Next, you want to log in to Crafter Studio as the admin user. The admin user has the rights to create new projects (called sites.) Click Create Site.

Figure 3: Create site via Crafter Studio

Clicking Create Site will present you with the Create Site dialog. This dialog changes depending on what you choose. Below is an example of the dialog filled out in a way that creates your project locally, set the Github repository as its upstream remote and pushes the initial project contents to the upstream repository.

Let’s walk through each part of the dialog:

Figure 4: Create Site Dialog in Crafter Studio, populating a bare upstream Git repository.

  1. The first thing you need to do is give your site an ID. The ID itself doesn’t matter in a sense. It doesn’t need to match anything per se, technically speaking the only requirement is that it’s unique. That said, it’s a best practice to provide an ID that is meaningful/recognizable to the team. If your website is called Sweet.com a good ID might be “sweetdotcom”
  2. Next, because you plan to connect this project to an upstream repository you want to click the plus (+) on “Link to upstream remote Git repository” This will open a number of new fields.
  3. In the “Remote Git Repository Name” field you want to provide a repository name that makes sense. It’s common to use “origin” or “upstream.”
  4. In the “Remote Git Repository URL” field you must provide the link to the Git repository discussed in Step #1: https://gitlab.com/russdanner/sweet-dotcom.git
  5. Provide your credentials in Git Remote Repository Username and Password
  6. Choose the option: “Create site based on blueprint then push to  remote bare repository.” This means that Crafter Studio will create a new site based on the blueprint you choose, link the remote repository as an upstream and then once the blueprint is installed in the local Repositories it will be pushed automatically to the upstream remote.
  7. Choose your blueprint. There are several out of the box blueprints provided by default. Choose one of these or one of your own. For our example, we’ll choose Editorial which is the simple Article style website/project template.
  8. Click Create. Crafter CMS will create the local repositories, Solr core and internal data structures required to support the project and install the blueprint. Once complete it will connect to the upstream and push the contents of the Sandbox repository to the remote.

Figure 5: Site is created and the contents of the sandbox are automatically pushed to the upstream repository.

Step 3: Check GitLab to Make Sure Your Site is There

Go back to your GitLab project and refresh the screen.  You will see the contents of your CMS project in the repository.

Your project is there!  Now you are ready to set up your entire development process and CI/CD automation.  To learn more about how this is configured check out these blogs:

Creating a Project in Crafter CMS Based on an Existing GitLab Project

Let’s consider for a moment that you’re a new developer joining the team. The topology above is already set up and you just want to get a local environment up and going. Simple. Follow these instructions.

  1. Install Crafter Studio locally (Source build or Binaries bundle)
  2. Login as Admin
  3. Click Create Site
  4. Fill out the Create Site Form as in a similar fashion described in Step 2, except this time you chose the option to create your site based on an existing upstream repository. This can be your team’s branch or your own fork. The exact workflow is up to you.

Conclusion

Platforms like GitLab that support agile development and CI/CD are helping bring best practices to the enterprise with easy to use tools that make implementing these activities simple.

Crafter CMS’s Git-based repository and DevOps integration make it simple to build CMS and content related projects while adhering to DevOps best practices and leveraging today’s best development platforms.

“Code Forward, Content Back” is a trademark of Crafter Software Corporation

Using Crafter CMS. Github and Two-Factor Authentication

Crafter’s Git based CMS supports developers working against remote repositories like Github, Gitlab, Bitbucket, and others. Many organizations that use Github enforce a two-factor authentication for developers.  This article shows you how to create a site in Crafter when you have two-factor authentication in Github configured.

Step 1: Configure an Access Token in Github

If you are reading this article, it’s very likely you already have two-factor authentication configured and are using it with Github.  If you’re interested in setting it up you can learn more about that here:
https://help.github.com/articles/securing-your-account-with-two-factor-authentication-2fa/ 

The first step in getting Crafter Studio to work with Github when two-factor authentication is enabled is to create a personal access token.  Follow the instructions shown here to create and acquire your token:   https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/

Step 2: Create a Site based on a Remote Repository

  1. If Crafter CMS is not installed: Install it locally (Source build or Binaries bundle)
  2. Login as Admin user

Next, you want to log in to Crafter Studio as the admin user. The admin user has the rights to create new projects (called sites.) Click Create Site.

Figure 1: Create site via Crafter Studio

Clicking Create Site will present you with the Create Site dialog. This dialog changes depending on what you choose. Below is an example of the dialog filled out in a way that creates your project locally based on an upstream repository that becomes it’s remote.

Let’s walk through each part of the dialog:

Figure 2: Set up a project based on existing remote Git repository via Crafter Studio

  1. The first thing you need to do is give your site an ID. The ID itself doesn’t matter in a sense. It doesn’t need to match anything per se, technically speaking the only requirement is that it’s unique. That said, it’s a best practice to provide an ID that is meaningful/recognizable to the team. If your website is called FreshFlowers.com a good ID might be “freshflowerscom”
  2. Next, because you plan to connect this project to an upstream repository you want to click the plus (+) on “Link to upstream remote Git repository” This will open a number of new fields.
  3. In the “Remote Git Repository Name” field you want to provide a repository name that makes sense. It’s common to use “origin” or “upstream.”
  4. In the “Remote Git Repository URL” field you must provide the link to the Git repository discussed in Step #1: https://github.com/russdanner/devworkflowexample.git

  5. Provide your credentials in Git Remote Repository Username and Password
    USE THE PERSONAL ACCESS TOKEN
  6. Choose the option: “Create site based on remote git repository.” This means that Crafter Studio will create a new site by cloning the upstream repository locally and setting the upstream repository as a remote.
  7. Click Create. Crafter CMS will create the local repositories, Solr core and internal data structures required to support the project and pull in the project from the remote.

 

Content Management Meets DevOps (Part 2 of 2) How a Git-based CMS Supports Continuous Integration and Delivery

As we learned in Part 1 of this series: Content Authoring and Publishing; development and content authoring are both a major part of producing today’s digital experiences. Unfortunately, development support is not something traditional CMS platforms handle very well at scale. Crafter CMS, a 100% open source project not only supports authors but DevOps as well.

In this article, we’re going to expand on the authoring workflow and publishing mechanics we looked at in the last article and explain how Crafter CMS seamlessly supports DevOps tools and process.

Development Requires Environments

The CMS where authors work is typically considered a production system. That’s not to say that the edits they make are immediately live. Typically CMS platforms support a concept of a “draft” type workflow with the ability to publish approved content and make it “Live”.

Crafter CMS is no exception here. As we described in our last article in this series, Authors work in a decoupled authoring environment and content is published from authoring to delivery on approval.

In the authoring environment, content can be edited and previewed via our web-based authoring tool called Crafter Studio. Edits are saved in a Git-based repository called “Sandbox”. On approval, edits are moved to a repository called “Published.” Crafter Studio handles all of the mechanics of making editing and publishing safe and simple for authors. Authors simply click edit and make changes, perform reviews and publish work via the UI. In the background, Crafter Studio will lock content, perform commits and take any other actions necessary to perform the low-level repository tasks necessary.

The authoring environment is akin to a work and test area for the authors. Relative to the live site or app the content authors are deploying content to, the Crafter Studio (and the Sandbox repository) where they edit, preview and approve content is what we would call a lower environment. Work is promoted from the lower environment (authoring) to the live environment (delivery.)
Developers almost always work in lower environments. No real developer modifies the code in production. Experience shows that making changes in production leads to outages and hinders future upgrades. Interestingly, many headless CMS platforms making their way to the market today require developers managing content types to work directly in production! While the scope of development in these systems is limited, the end results are the same.

In Crafter CMS, because we decouple authoring from delivery, it is possible for developers to work directly with the production system via authoring. Changes can be made and tested in the authoring environment before being pushed live.

Figure 1: Production environment.  Showing DevOps as part of the production workflow.  DevOps do not want to touch production directly and should not have to.  Unfortunately, most CMS platforms force them to do so.

That said, developers do not want to work with CMS tools! They want to work with their own Integrated Development environments (IDE) and other tools. They want to work locally on their own machines where they have control and can debug. They want the ability to collaborate with their teammates on feature branches. They want to be able to work in parallel with other development teams without fear of stepping on each other or interfering with content authoring activities and they want to be able to easily move their work through the various environments of their certification and testing process out to production. Developer environments and process are complex.

To illustrated this I have included a diagram below of a typical developer process and set of environments which must be followed in order to get new functionality to production.

Figure 2: Typical DevOps workflow.  Today’s modern app development DevOps workflows include parallel development efforts on the same project and extend from the developer’s local environment and team sandbox all the way to production.

Traditional CMS Systems and Environments, Code Artifacts and Process Don’t Mix Well

It is difficult at best to support this kind of process with traditional CMS architecture. Server-side code is typically managed in a source code repository while CMS related code artifacts like templates, Javascript and CSS (that should be managed in a source code management) want to live in the CMS. Server-side code typically requires heavy packing and deployment systems like JARs, WARs and OSGi bundles. It takes a lot of effort on the part of the developer to get an environment setup locally so that they can develop and it’s equally difficult to get a specific set of code into one of the often many environments. Worse, content is always stored in a JCR repository or SQL database and it’s a huge burden to get the latest production content to a lower testing environment. The whole process of “code forward, content back” with traditional CMS is a nightmare for DevOps. Still worse, in some environments, going to production means interruptions in content authoring ability for content authors while their environment is updated with the latest code. These outages can last hours to days or in some cases even longer.

These conditions are unacceptable.  This is no way to support innovation at a time when the amount and speed of innovation an organization can sustain is a major key to competitive advantage.

Crafter CMS Provides the Solution

Crafter’s Git-based dynamic CMS tackles all of these challenges head-on with a set of technologies that incorporate lightweight development, integration with developer tools and process and elastic scalability that provides the ability to serve any front-end technology via API or markup with fully dynamic content.

First: Crafter CMS supports a fully featured, Spring-enabled, Groovy-based scripting layer. This is a subject for a whole blog of its own — but without getting too in-depth, it’s huge. You can create classes on your own in Groovy. You can leverage classes in the classpath. You can create your own beans based on Groovy or Java classes, you can inject existing beans and services. It’s a rich, robust powerful environment full of services for you to build on at the speed of scripting in the same languages and frameworks that you are used to.
Second: Crafter CMS sits on top of Git. Git is traditionally speaking, a source code version control system. That means that all of your code can live in the same version control system as the business content, together. It’s versioned together and it can be packaged and deployed together. Crafter CMS makes the world of CMS simple for content authors and developer native for DevOps.
Third: Because Crafter is Git-based you can natively support your ideal DevOps process for Continuous Integration and Delivery (CI/CD).

 

Let’s jump in and take a look at how it works.
Below is what you might think of as the archetype of a developer workflow that’s integrated with the production authoring environment. In this diagram, you will find a mechanism for both a “code forward” activity in which code can be moved through a development process through to production as well as a “content back” activity in which production content, including in-process content, can be brought back to lower development environments! including local development machines.

Figure 3: Archetypal DevOps workflow that describes in general how content flows back to the development process and code flows forward through the certification process to production.

The above diagram has a lot going on. Let’s break it down and explain things a step at a time.

Step 1: Create an Upstream Repository

In order to facilitate a developer workflow, you want to establish an upstream repository for your Sandbox in your production authoring environment. Any Git repository will work. It’s typical to use a Git repository that has a UI/Workflow atop of it such as Github, Bitbucket, Gitlab and other similar systems. The upstream repository is the root of the developer workflow.

Figure 4: A Git repository with support for “developer/team workflow” such as Github, GitLab or Bitbucket is configured as a remote upstream repository of the Production Authoring Sandbox repository.  This is the primary point of contact between the authoring process and the development and code release process come together.  

Step 2: Use Automation to Sync Sandbox with the Upstream

Note that the repository is what the Git community refers to as an “upstream” remote repository. That means in a sense the Production Sandbox becomes subordinate to it from a Git log perspective. This upstream repository is where content and development work will be merged before making its way to the production Sandbox and ultimately to the Published repository and the delivery nodes themselves. Also, note that nothing changes about the publishing and deployment configuration and topology of the Production environment.

You will want to keep the upstream repository up to date with authoring. The best way to accomplish this is to use a DevOps automation platform such as Jenkins or Bamboo to orchestrate a push to the upstream on a schedule. Keeping the upstream repository up to date with the authoring Sandbox repository provides downstream development and testing repositories with the latest content and helps to avoid conflicts when it’s time to promote from the upstream repository to the authoring Sandbox repository.

Figure 5: Content flows from Production Authoring Sandbox to the Developer repository via automated Git Push

Step 3: Use the Upstream Repository to Support Development and Testing Environments and Workflow

From this upstream repository, you will feed all lower environments and branch for each of your development efforts. From the upstream repository, you can support any development process and supply content to any development or testing environment. Lower environments may consist of Authoring and Delivery, or just Authoring, or just Delivery. It depends on the needs. For example, Development often contains both Authoring and Delivery, while QA tends to focus only on Delivery. Simple “Git pull” mechanics are used to move code and content from the upstream repository, typically from a branch.

Figure 6: Development branches and forks can be created to meet your specific workflow needs.  Lower environments use these repositories and branches as their upstream.

Step 4: Merge Code Updates Into the Upstream Master and Pull them to the Production Authoring Sandbox Repository

When you are ready to move code from a development branch to the authoring Sandbox you will first merge the work into the master of the upstream repository. You can do merge all of your commits or you can merge them into a single commit by using the Git rebase command. By Squashing all of the commits into a single commit you make it easier to move the workaround as a single unit. Merges are typically done via pull requests on repositories that support development workflow.
Once the merge operation is complete in the upstream repository your automation will carry that work to the production authoring Sandbox.

Figure 7:  Use “Pull Requests” and other Git workflow mechanics to promote code through the process.  When it’s ready to go live merge it to the “Master” of the Developer Repository.  At this point, it will flow via automation to the Production Authoring Sandbox.

Step 5: Use Crafter Studio’s Publish Commits API to Publish Code to Delivery

Once your development work is in the Production Sandbox you will want to publish it. To do this Crafter Studio provides a REST API that will trigger the publishing on one or more commits. Simply call Crafter Studio’s Publish Commits REST API (/api/1/services/api/1/publish/commits.json) via the DevOps automation platform passing the required parameters including the commit IDs to be published and Crafter Studio will move the work to the Published repository from which it will be replicated to your delivery nodes.

Figure 8: Move code from Sandbox to Published with a simple API call. 

Step 5: Build Tons of Amazing Things

Now that you have the basic mechanics of Crafter’s native Git-based distributed repository and development workflow you are ready to support any amount of parallel development you want with the kind of tools and process rigor you are used to. Because Crafter’s Git-based repository is distributed, authors and developers, even developers working locally are all working out of what is essentially the same repository.

The workflow enables the team to collaborate and work simultaneously without ever interrupting or stepping on each other’s toes. It’s time to start innovating!

How do I set up this workflow?

For the purpose of simplicity, we’ll assume you’ve got Crafter Studio up and running and you are about to create a new project.

Step 1: Create your upstream

Create an empty repository in your upstream (GitHub, Bitbucket etc.) The specific steps depend on the repository you are using. The key here is that you want to create an empty or what’s sometimes called a “bare” repository.

Figure 9: Create a bare repository in your developer Git. In this example, we’re using Github.

On Github, once created you will see the following screen. Here you will note the repository URL (https://github.com/russdanner/devworkflowexample.git) which you will need for the next step. Also, if you’re trying to create an upstream for an existing project (out of scope for this blog), you’ll find the instructions below in the “push an existing repository from the command line” section.

Figure 10: Bare repository created.  

Step 2: Create Your Project In Crafter Studio

Next, you want to log in to Crafter Studio as the admin user. The admin user has the rights to create new projects (called sites.) Click Create Site.

Figure 11: Create site via Crafter Studio

Clicking Create Site will present you with the Create Site dialog. This dialog changes depending on what you choose. Below is an example of the dialog filled out in a way that creates your project locally, set the Github repository as its upstream remote and pushes the initial project contents to the upstream repository.

Let’s walk through each part of the dialog:

Figure 12: Create Site Dialog in Crafter Studio, populating a bare upstream Git repository.

  1. The first thing you need to do is give your site an ID. The ID itself doesn’t matter in a sense. It doesn’t need to match anything per se, technically speaking the only requirement is that it’s unique. That said, it’s a best practice to provide an ID that is meaningful/recognizable to the team. If your website is called FreshFlowers.com a good ID might be “freshflowerscom”
  2. Next, because you plan to connect this project to an upstream repository you want to click the plus (+) on “Link to upstream remote Git repository” This will open a number of new fields.
  3. In the “Remote Git Repository Name” field you want to provide a repository name that makes sense. It’s common to use “origin” or “upstream.”
  4. In the “Remote Git Repository URL” field you must provide the link to the Git repository discussed in Step #1: https://github.com/russdanner/devworkflowexample.git
  5. Provide your credentials in Git Remote Repository Username and Password
  6. Choose the option: “Create site based on blueprint push to a remote bare repository.” This means that Crafter Studio will create a new site based on the blueprint you choose, link the remote repository as an upstream and then once the blueprint is installed in the local Repositories it will be pushed automatically to the upstream remote.
  7. Choose your blueprint. There are several out of the box blueprints provided by default. Choose one of these or one of your own. For our example, we’ll choose Empty which is the “Hello World” of blueprints.
  8. Click Create. Crafter CMS will create the local repositories, Solr core and internal data structures required to support the project and install the blueprint. Once complete it will connect to the upstream and push the contents of the Sandbox repository to the remote.

Figure 13: Site is created and the contents of the sandbox are automatically pushed to the upstream repository.

Step 3: Set up Your Delivery Nodes

Now that your project is created you can set up the rest of your production environment by initializing your delivery nodes to receive deployments from authoring. Remember these delivery nodes will pull from Crafter Studio’s repositories, not the upstream remote repository.

When you add a new delivery node a simple command line script is run on that node that configures it to replicate and process content from the “Published” repository from authoring.

  • Instructions for creating a site can be found here.
  • Instructions for initializing a delivery node can be found here.

Step 4: Set up your Developer Workflow and Lower Environments

Now that your upstream repository is initialized in GitHub you can set up any developer workflow you want. It’s typical to consider Master to be in-sync with the Production Authoring Sandbox. Given that, you don’t want to work in Master. Create branches to isolate development work from work that’s ready to move to Production Authoring. Below is an example topology that shows multiple environments and developer workflow that include feature branches, developer forms, and local developer clones.

Figure 14: Full DevOps “Code Forward, Content Back” workflow for CI/CD with Crafter CMS leveraging Git mechanics and DevOps automation.

I Am a Developer, I Want to Work Locally Against The Upstream

Let’s consider for a moment that your a new developer joining the team. The topology above is already set up and you just want to get a local environment up and going. Simple. Follow these instructions.

  1. Install Crafter Studio locally (Source build or Binaries bundle)
  2. Login as Admin
  3. Click Create Site
  4. Fill out the Create Site Form as in a similar fashion described in Step 2, except this time you chose the option to create your site based on an existing upstream repository. This can be your team’s branch or your own fork. The exact workflow is up to you.

Figure 15: Set up a project based on existing remote Git repository via Crafter Studio

Conclusion

Today’s digital marketplace is constantly evolving. Companies are always iterating on existing functionality with improvements and deeper integration or introducing new functionality and channels for their employees, partners, and customers. Crafter CMS provides the right technology and integrates seamlessly with your DevOps processes to enable you to achieve a high, sustainable continuous rate of constant, iterative development, integration and delivery (CI/CD.)

Digital experience teams finally have a toolset that allows authors to work continuously without being interrupted by developers.

Developers have a means for easily moving code forward through environments and pulling content back from production to lower environments.

Further, with Crafter’s distributed repository development shops can run as much parallel development as they want and developers are able to leverage the workflow and local tools they are accustomed to.

“Code Forward, Content Back” is a trademark of Crafter Software Corporation

Content Management Meets DevOps (Part 1 of 2) How a Git-based CMS Improves Content Authoring and Publishing

Traditional CMS platforms based on SQL and JCR repositories have begun to show major signs of weakness in keeping up with today’s demands for a high rate of innovation and rapid scalable deployment on modern elastic architectures. This is nowhere more evident than the move towards headless CMS. Many CMS platforms today push headless, or what some call Content as a Service (CaaS), as the one-stop-shop solution to the struggles most CMS platforms have in providing support for scalability, multi-channel, and development integration. It’s not. Headless capability is important but it has its own limitations.

Crafter CMS, an open source Git-based dynamic CMS tackles all of these challenges head-on with a set of technologies and that incorporate lightweight development, integration with developer tools and process, and elastic scalability for content delivery that provides the ability to serve any front-end technology via API or markup with fully dynamic content.

In this two-part series, we’ll explain the basic mechanics that support content authoring, publishing and developer workflow and demonstrate how these mechanics combined with Crafter’s architecture and developer stack set a new standard for what a CMS can provide in today’s competitive digital marketplace.

Content Management and Deployment Mechanics

In this section we’ll explore the mechanics of how (non-technical) content authors work with the CMS and how their changes, once reviewed and approved, are deployed from their authoring tools to a live content delivery system.

Crafter CMS is decoupled, composed of several microservices where content authoring and content delivery services are separated into their own distinct, subsystems. This model has many advantages related to security, scalability and delivery flexibility. In a decoupled architecture, content is published from authoring to delivery as shown in the diagram below. The delivery system may be any number of independent digital channels – enterprise website, mobile app, social, augmented reality, digital kiosk or signage, e-commerce front end, microsite, etc.

Crafter CMS supports authoring via Crafter Studio that sits on top of a headless Git-based repository and publishing system. Content authors don’t need to know anything about Git. They simply work with Crafter Studio, a web-based application. Crafter Studio provides beautiful content entry forms, in-context editing with multi-channel preview, drag-and-drop layout, component placement, image cropping, and more. While content authors are performing their work, Crafter is managing all of the Git mechanics, managing locking, creating a time-machine like, Git-based version history and audit trail for them behind the scenes, all accessible to them via the Studio UI.

 

Figure 1: Crafter CMS microservices applied to decoupled architecture

Crafter’s publish mechanism deploys content from the Authoring system to the Delivery system. Content logically flows from the authoring environment to the delivery environment. The mechanism for this, given the underlying Git repo, is a “pull” type interaction. Meaning the actual network conversation is initiated from the delivery infrastructure to the authoring infrastructure, as shown in Figure 2.

Each delivery node has a Deployer agent that coordinates deployment activities on the node for each site that is being delivered on that node.

  • Delivery nodes can initiate deployment pulls either on a scheduled interval (a “duty cycle”), on-demand via an API call, or both.
  • The Deployer performs a number of activities beyond receiving and updating content on the delivery node. A list of post-commit processors is run. These can be used to execute updates on search indexes, clear caches and perform other such operations.
  • The Delivery node maintains a clone of the Authoring Git-based repository.
  • The Crafter Deployer takes care of managing the synchronization of the delivery node’s clone authoring repository from the authoring environment.
  • Git-mechanics ensure content sync is 100% accurate.

Figure 2: Crafter’s Dynamic CMS Publishing via Git

Technically speaking, Authoring does not require knowledge of the Delivery nodes. This makes the architecture more elastic, globally scalable and even enables Crafter to support disconnected and intermittent content delivery.

  • Elastically add new nodes or revive dormant nodes and they will sync to the latest without any additional wiring.
  • Create region-based depots to avoid transferring data more than once over long distances for global deployments.
  • Airplanes, cruise ships, drilling/mining locations and other remote disconnected deployments can operate with their latest pull of content, and sync up with Authoring when connectivity is available.

Figure 3: Elastic Delivery

In Crafter CMS, only approved content is published to the delivery environment. Crafter manages this by using 2 repositories for each project. One called a “Sandbox” which contains work-in-progress and the other called “Published” which represents approved, published work and complete content history.

  • Authors use the Crafter Studio UI to review and approve content via workflow.
  • Crafter Studio takes care of moving approved work between Sandbox and Published repositories.
  • Delivery nodes monitor the published repository for updates.


Figure 4: Authors work in Sandbox. Delivery nodes pull from Published.

Benefits

Crafter’s Git-based publishing model provides your authoring team with a highly reliable, highly accurate publishing mechanism that is elastically scalable, globally distributable and supports multi-channel.  Crafter CMS’s architecture enables your team to reliably deliver your dynamic content on any channel, wherever and whenever it is needed.

Further, As we’ll see in in Part 2, this architecture enables content authors to work side-by-side with DevOps, while they continually introduce new features and functionality without any disruption to the authors.

How do I set up this workflow?

The underlying Git repositories and related workflow for Authoring require no setup at all. When you create a project in Crafter Studio it automatically creates the local “Sandbox” and “Published” repositories. When you add a new “Delivery” node a simple command line script is run on that node that configures the node’s deployer to replicate and process content from the “Published” repository from authoring.

  • Instructions for creating a site via Crafter Studio can be found here.
  • Instructions for initializing a delivery node can be found here.

Conclusion

Content authors are non-technical users who need powerful but easy-to-use tools to help create, maintain and manage their digital experiences. Crafter Studio provides these users with a web-based application that makes it easy for content authors to achieve their goals. Under the hood, Crafter Studio leverages a powerful Git-based repository and deployment engine that provides authors with next-generation versioning and auditing mechanics as well as robust, elastic and distributed deployment.

Today’s digital marketplace is constantly evolving. Companies are always iterating on existing functionality with improvements and deeper integration or introducing new functionality and channels for their audiences. For today’s most innovative and competitive organizations, ongoing development and the move to DevOps is a fact of life. The companies that have the greatest success are those that have the right technology and processes through which they are able to achieve a high, sustainable continuous rate of constant, iterative development, integration and delivery, i.e., Continuous Integration and Delivery (CI/CD).

In the second half of this blog series, we will take a deep dive into how Crafter CMS seamlessly integrates with your CI/CD processes to enable your entire team of developers and content authors to innovate collaboratively without interfering with each other’s workstream.

 

Working with Crafter Studio’s API

Crafter CMS is a decoupled CMS composed multiple microservices where content authoring and content delivery capabilities and services are separated into their own distinct, subsystems.

Organizations often want to interact with the content authoring and management system via APIs. In this article, we’ll show the basics of interacting with this API by example:

  • Authenticate
  • Get a list of projects under management
  • Write content to a project

To keep things really basic, we’ll use CURL, a ubiquitous Linux command tool as our client.

You can find the full Crafter Studio API for Crafter CMS version 3.0 here
http://docs.craftercms.org/en/3.0/developers/projects/studio/api/index.html

Step 1: Authenticate

We’ll use the authenticate API
http://docs.craftercms.org/en/3.0/developers/projects/studio/api/security/login.html

curl -d '{"username":"admin","password":"admin"}' --cookie "XSRF-TOKEN=A_VALUE" --header "X-XSRF-TOKEN:A_VALUE" --header "Content-Type: application/json" -v -X POST http://localhost:8080/studio/api/1/services/api/1/security/login.json

The first thing you’ll note is that we’re going to perform a POST, passing the username and password as a JSON object.  In a production environment, you will want to use HTTPS.

The next thing you will notice, we are passing a cookie “XSRF-TOKEN” and a header “X-XSRF-TOKEN”.  The value passed for these are arbitrary.  They must match and they must be passed in all future PUT and POST API calls.  These are used to protect against certain cross-browser scripting attacks.  If you are using Studio APIs as part of a web client you want to make sure these values randomly generated.

When you issue the curl command you will get back a response:
 * Trying ::1...
 * Trying 127.0.0.1...
 * Connected to localhost (127.0.0.1) port 8080 (#0)
 > POST /studio/api/1/services/api/1/security/login.json HTTP/1.1
 > Host: localhost:8080
 > User-Agent: curl/7.43.0
 > Accept: */*
 > Cookie: XSRF-TOKEN=A_VALUE
 > X-XSRF-TOKEN:A_VALUE
 > Content-Type: application/json
 > Content-Length: 39
 >
 * upload completely sent off: 39 out of 39 bytes
 < HTTP/1.1 200
 < Cache-Control: no-cache, no-store, max-age=0, must-revalidate
 < Pragma: no-cache
 < Expires: 0
 < Set-Cookie: JSESSIONID=2E114725C82F3EE44ADC04B578A3BE8F; Path=/studio; HttpOnly
 < Content-Type: application/json;charset=UTF-8
 < Content-Language: en-US
 < Transfer-Encoding: chunked
 < Date: Mon, 22 Jan 2018 21:32:48 GMT
 <
 * Connection #0 to host localhost left intact
 {"username":"admin","first_name":"admin","last_name":"admin","email":"evaladmin@example.com"}

Note the response returned is a successful 200 status code and the response contains JSON with details for the authenticated user.

Also found as part of the request is the JSESSION cookie.  You will need this value for all future requests.

Step 2: Get a list of sites the user is authorized to work with

http://docs.craftercms.org/en/3.0/developers/projects/studio/api/site/get-sites-per-user.html

curl --cookie "XSRF-TOKEN=A_VALUE;JSESSIONID=2E114725C82F3EE44ADC04B578A3BE8F" -H "X-XSRF-TOKEN:A_VALUE"  -X GET http://localhost:8080/studio/api/1/services/api/1/site/get-per-user.json?username=admin
Note the CURL command contains your session ID and XSRF tokens.
When you issue the CURL you will get a response that contains sites your user has access to:
{"sites":[{"id":9,"siteId":"ar","name":"ar","description":"","status":null,"liveUrl":null,"lastCommitId":"951004363449cc83209f307b1e9f110dab37fed7","publishingEnabled":1,"publishingStatusMessage":"idle|Idle","lastVerifiedGitlogCommitId":null},{"id":5,"siteId":"diiot","name":"diiot","description":"","status":null,"liveUrl":null,"lastCommitId":"92d543eaa164b1ebfbdd6ce538ae028d4d6421b7","publishingEnabled":0,"publishingStatusMessage":"idle|Idle","lastVerifiedGitlogCommitId":"92d543eaa164b1ebfbdd6ce538ae028d4d6421b7"},{"id":10,"siteId":"editorialcom","name":"editorialcom","description":"","status":null,"liveUrl":null,"lastCommitId":"503d922f226e8ab821073e23ef5a229f907212a0","publishingEnabled":1,"publishingStatusMessage":"","lastVerifiedGitlogCommitId":"503d922f226e8ab821073e23ef5a229f907212a0"},{"id":3,"siteId":"flow","name":"flow","description":"","status":null,"liveUrl":null,"lastCommitId":"21923775c3a1fc778a364d47884b9ee2bb4928a5","publishingEnabled":1,"publishingStatusMessage":"idle|Idle","lastVerifiedGitlogCommitId":"21923775c3a1fc778a364d47884b9ee2bb4928a5"},{"id":8,"siteId":"vr","name":"vr","description":"","status":null,"liveUrl":null,"lastCommitId":"c67fd9dd25d1aa59ff13e3fda2a4387be50dfc69","publishingEnabled":1,"publishingStatusMessage":"idle|Idle","lastVerifiedGitlogCommitId":null}],"total":6}

The response above contains a number of projects.  In the next call I want to write a content object to one of the projects (editorial.com.) To do this I need the site ID.  I get this from the response above: editorialcom

Step 3: Write content to the Editorial com Project

http://docs.craftercms.org/en/3.0/developers/projects/studio/api/content/write-content.html

curl -d "<page><content-type>/page/category-landing</content-type><display-template>/templates/web/pages/category-landing.ftl</display-template><merge-strategy>inherit-levels</merge-strategy><file-name>index.xml</file-name><folder-name>test3</folder-name><internal-name>test3</internal-name><disabled >false</disabled></page>" --cookie "XSRF-TOKEN=A_VALUE;JSESSIONID=2E114725C82F3EE44ADC04B578A3BE8F" -H "X-XSRF-TOKEN:A_VALUE"  -X POST "http://localhost:8080/studio/api/1/services/api/1/content/write-content.json?site=editorialcom&phase=onSave&path=/site/website/test3/index.xml&fileName=index.xml&user=admin&contentType=/page/category-landing&unlock=true"

In the call above note:

  • We are passing in content as the POST body.  The content is in XML format.  In Crafter content objects are stored as simple XML documents.
  • We are passing the Session ID and the XSRF tokens
  • We are passing a number of parameters that tell Crafter CMS where and how to store the content in the repository

Conclusion

In this article we covered the basic mechanics of connecting to and interacting with Crafter Studio, the authoring services of Crafter CMS.  We’ve avoided the nitty gritty details of each API call in favor of the macro mechanics.  You now have the basic skills and capability to interact with any Crafter Studio API found here: http://docs.craftercms.org/en/3.0/developers/projects/studio/api/index.html.  Get out there and integrate!

 

Five Reasons Why You Should Use a Git-based CMS (Part 5 of 5)

In our previous posts we looked at Crafter CMS and its Git-based versioning (part 1), distributed repository (part 2), dug in to Git’s underlying mechanics to see how it benefits deployment (part 3) and we looked at how the support for branching (part 4) can help your organization dramatically speed up development and deployment activities. In this post, we’re going to wrap the series up with one final reason.

There are so many advantages to the way that we’ve leveraged Git; It’s hard to pick just 5 things to talk about. Because I’ve arrived at our last reason, #5, I want to use this item to speak to something non-technical: familiarity.

Reason #5: Familiarity

Throughout this series, you have heard me talk about a “Git-based” CMS.  That’s intentional and I want to elaborate on why that’s important.  The entire series isn’t just about a better mousetrap.  We’ve tried to take a hard look at the kind of problems that remain in the CMS space and the needs that modern organizations have in terms of innovation at a competitive level with respect to ease, speed and scale of continuous delivery.  It turns out that, yeah, some of the biggest bottlenecks go all the way down to the core of today’s CMS platform: They way we store and manage content and code.

The features I spoke to in this series are the first steps in truly making a real and major difference. What we’re saying is, we need a repository with a specification that enables these kinds of capabilities.  Developers and DevOps should not have to dance around the technology to get their jobs done.  That said, the functional specifications of features and how you implement them are different things. We could have focused on a system with “Git-like” rather than “Git-based” functionality. We chose to do the latter. We did that on purpose.

Leveraging Git rather than re-inventing it has nothing to do with code-reuse or ease of implementation. If you look around, you will find a number of projects that enable non-technical content editing and publishing (basic CMS functionality) for statically generated static websites that leverage Git.  Developers and DevOps familiar with Git, already recognize many of the benefits of leveraging Git for these kinds of needs.  However, implementing a full enterprise-class CMS designed to support modern, dynamic and personalized multi-channel experiences isn’t as simple as sticking Git under your CMS. The full range of WCM authoring, development and DevOps use cases are much different and more complex than what you see with simple editing and publishing of statically generated websites and standard source code management.  There are a lot of decisions to make and problems to solve.  A Git-based implementation is important for a much bigger reason another reason:

Familiarity matters. Familiarity covers both adoption and integration. Git is used all over the world. It’s proven. Moreover, developers and operations teams know how to use it. A Git based CMS is part of the broader Git ecosystem — which means your existing Git-friendly toolchain natively works with the CMS.

“Git-like” is not enough. Its got to be Git-based. Familiarity matters.

Conclusion

Crafting great digital customer experience is a complex, multi-step, multi-environment and multi-disciplinary practice. Today, a CMS must be as good for development and operations/process as it is for the content authors that are traditionally thought of as the primary users.  Today’s contemporary CMS platforms still sit on a basic architecture that was designed over 20 years ago when needs were different.  To meet today’s challenges a new kind of CMS is required with a new architecture that starts at the root of the platform, the repository.  Today’s needs call for a distributed repository that supports branching, advanced content/code flow scenarios and a versioning model that is multi-file.  These are Git-like features.   While the functional specification of a modern CMS repository is “Git-like” a solution that is “Git-based” leverages the power and track record of the worlds most popular source code management platform and plugs into a vast ecosystem of developers and DevOps in a way that no other solution can. 

Crafter CMS is an open source CMS project and the world’s first enterprise-class CMS based on Git. Crafter CMS is on a mission to bring content authors, developers, and operations teams together in a single platform that makes innovation easy, fast and fun.

Check out the other articles in this series:

Five Reasons Why You Should Use a Git-based CMS (Part 4 of 5)

In our previous posts we looked at Crafter CMS and its Git-based versioning (part 1), distributed repository (part 2) and deployment mechanics along with its decoupled architecture (part 3)  In this post we’ll take a deeper dive into a feature of Crafter’s Git-based CMS that provides unparalleled support and speed for innovation: branching.

Imagine a single train track that stretches from Washington D.C. to New York.  How many trains can run simultaneously on this track?  How fast can the trains go? How much control over the order of arrival do you have?

  • You can run as many trains as you have room for on the track.
  • Anyone train can only go as fast as the train in front of it.
  • The trains always arrive in the order they departed in.
  • If a train breaks down or gets put on hold, it’s nearly impossible to reorder.

With only one track, the railroad’s ability to deliver success is throttled by the amount of volume they can handle and even when capacity is not an issue, they are completely dependent on good luck with respect to order and unplanned changes and breakdowns. With railroads, the solution to this problem is implemented with switches and sidings. Sidings are branches in that single track that allow a train to pull off the main line giving the controller the ability to regulate the order, speed, and quantity of trains on the main line.

The same problems of bandwidth, throughput, and order you see in the railroad example exist with projects that need to go through your traditional CMS.  Traditional CMS platforms have no ability to branch the content and code base. They are just like that single track from DC to New York.  This means you have very little control over the order, speed, and capacity of your development. Your CMS needs branching. Let’s explore this further.

Reason #4: Branching

Nearly every CMS still sits on an architecture and repository that was devised 20 years ago.  Back then innovation was important but the world moved a lot slower.  A single pipeline of features got the job done and the CMS was less of a bottleneck.  Today, in contrast, digital channels are at the core of many organizations’ strategy for serving the customer better and beating the competition.  The organization who moves the fastest often wins.  Agility is key.

Meanwhile, infrastructure costs have gone down while development costs continue to rise.  Balancing costs with volume and speed of innovation is a major challenge of our day.  Today it’s all about great DevOps. To make DevOps really work with CMS you need to be able to branch.

Traditional software development process has included branching for eons. Developers and DevOps have long since figured out that they need to be able to work in teams, isolate work and control the order in which work is merged into the critical path for go-live. The CMS track runs right alongside the traditional development track, and at some point, it merges and the last few steps require the CMS.  It’s only now that the demand for the speed of innovation has increased that the connective tissues between development and the CMS have been put under so much stress that they are completely failing.  The fact is that today, we need that same agility all the way through the CMS and right up to the very last step of production deployment.

Even if the majority of your development is outside the CMS you still need to integrate. Consider the following example:

Because the website needs to integrate with, and ultimately deliver the functionality of the microservice, we need to perform development.  We also need to support daily content edits and continuous publishing. With traditional CMS we have no option but to use multiple CMS environments to support this scenario.  Does this approach give us multiple tracks and control over my releases?  Yes, technically it does.  But practically speaking?  No.  Not at all.

As we learned before, moving content and code between CMS environments with traditional CMS architecture is extremely difficult. The process of spinning up environments and loading code and content into them is so difficult and time consuming for any DevOps team that most won’t even consider it unless absolutely forced to.  Even then the size of the team may not support the need. The rate of innovation crawls to a near stop.

To address this problem, we built Crafter CMS v3 on top of a Git-based repository. As a result, the Crafter CMS platform is built on repository store that not only branches but also is fully distributed.  Not only are you able to easily control the order and rate of work, but it’s a snap to move work from one environment to another.

Moreover, branching not only supports DevOps and but it makes development easier.  Developers and authors can experiment, work on major features and other site enhancements in the safety of branch-based sandboxes that keep them from stepping on each other’s toes.

Conclusion

If you want to quickly innovate with your website, mobile app and other content-rich digital experience apps, you will need multiple teams working on different features at the same time. You need control of who is working on what and the order in which projects will be delivered.  Having the capability to manage these concerns with agility is the key to innovating quickly.  Traditional CMS platforms don’t support the basic feature set that enables this. Moving most of the development outside the CMS only gets you so far. You need a CMS that supports your DevOps process with features like branching and distributed repository if you truly want to be able to move fast.

Stay tuned for our next blog entry to learn another major reason why you should use a Git-based CMS!

Five Reasons Why You Should Use a Git-based CMS (Part 2 of 5)

Since the birth of content management system (CMS) technology, well over 20 years ago, platforms have been leveraging “obvious backends” like SQL databases as a store for the content. Not because it’s the best or right store for the job, but because SQL databases are a commonly available, simple to use technology that (kinda) gets the job done. By the early 2000s, it was clear with many implementations that directly leveraged SQL and similar database stores do not provide the full range of features like versioning that a CMS requires. They can’t. They were not built to do it. The Java Content Repository (JCR) and other similar technologies entered the scene. The implementations of these technologies sit on top of the same old database stores and add a layer of capability to fill the gaps. This is good but not good enough. Ultimately, the fact that they sit on top of a database comes back to haunt them.

In Part 1, we looked at what kind versioning model is needed to support modern digital experiences. Today we focus on another critical capability that is missing in traditional CMS solutions: a distributed repository. More specifically, distributed versioning and workflow.

Reason #2: Distributed repository

Most databases are not easily distributable from a geographic sense, and more importantly, they are not distributable from a versioning and workflow sense.

I could spend a lot of time talking about how scaling and distributing a database geographically matters in the context of CMS and why it’s so difficult. I don’t have to. If you have the need for a CMS with high availability and global distribution you already know why it matters. If you have tried to make this work with a CMS based on a traditional database or a JCR repository, you already know it’s a difficult and sometimes impossible errand.

What is distributed versioning and workflow? The easiest way to get at this is by example. In the software development space, we’ve had Source Code Management (SCM) systems for a long time. These SCM systems allow teams of developers to work on a single code base as a team without stepping on each other’s toes by checking out work locally, working on it and then checking back in edits. Hint: This is not much different from what a CMS provides to content authors behind its UI.

Back to developers: In the past, we had CVS, SVN along with many others. These SCM systems provided basic version management as well as branching and tagging but fundamentally the system was a centralized model. With such solutions, there is a single central store and source of truth for the code base.

This SCM model worked well for smaller teams and smaller code bases but for large projects like the Linux operating system, it failed completely. Linux has so many developers spread out all over the world, working on many separate but related projects. A single, centralized system simply does not scale (in several ways) to meet this need. To make a long story short (collapsing a lot of history and detail), Linus Torvalds created Git as a lightning fast, open source solution to solve this problem. Git allows developers to have their own local and intermediary repositories that are all born from a parent repository. This makes distributing developers easy, it makes concurrency simple and most importantly to us, it distributes the versioning and workflow which makes “flowing” code to and from these independent repositories possible, fast and easy. Yes!

In the CMS space, for more than 20 years all the way up to this day, we’ve had repository solutions of various capabilities and quality. All of these solutions have no real, workable solutions for moving content back from production to lower environments like Staging, QA, Development, Load Testing and local developer machines. Yes, you can do it. But it’s a nightmare. You end up doing an export/import process and it’s not easy. Some systems are easier than others but they all stink. CMS consumers rig up all kinds of replication and publishing workarounds to try and deal with this problem. It’s all a hack. There’s no technical solution in the CMS space that was built to handle the problem specifically. For this reason and many others, development, and operations teams HATE the CMS options available today. They do nothing to help the team work — worse, they fight them in almost every way. The technical members of the team put up with CMS technology because their business counterparts need content creation and editing capabilities. That’s all.

Moreover, today we understand that to some degree, in the digital experience space, “code is content.” Just as we need to be able to move content back to environments, we also must be able to move code (templates, javascript, CSS, etc.) forward through the environments. Developers have processes that they use to ensure quality and performance. With traditional CMS, moving code forward through environments is even harder than moving content back. Wholesale export/import doesn’t work!

Because Crafter CMS is Git-based and because we’ve specifically built capability in Crafter CMS to handle these needs, the world finally has a CMS that solves this problem. The same approach developers use to make and promote source code changes with Git is used by Crafter CMS to move code forward and content back.

Every organization that uses a CMS for more than simple edits and blog posts know exactly what I am talking about. Today, it’s understood that customer experience is one of the biggest competitive advantages an organization can have. Further, beyond the human element, digital enablement and innovation is the most important component of delivering great customer experience. Because content and code are inseparable from customer experience, the CMS is a mission-critical component of any and all customer experience solutions. Here’s the kicker: nearly the entire world is using a CMS technology that not only fails to enable the organization to innovate faster — it actually fights them!

The Git-based distributed capabilities in Crafter CMS allow your organization to have many environments that are all related to one another — syncing and moving objects between them is natural and part and parcel to the technology itself. This means it’s easy to move content back and code forward.

Because the system is distributed and Git-based, developers can work locally and still be part of the CMS. That means they can use the tools they know and like, and they are not working on an island. The best way to make a developer love the CMS is to let them work with the CMS without having to work _in_ the CMS. Organizations that want to win, need to innovate without impedance.

Conclusion

Today’s CMS systems are rooted in 20-year-old architectures and technologies. As the demand for greater amounts of innovation and digital experience has grown and organizations are under pressure to deliver more at ever increasing rates CMS platforms have become more of a hindrance than a help. Crafter CMS, with its Git-based approach, not only solves these fundamental problems but also integrates very well with developer process and tools that innovation moves even faster. Finally, a CMS approach that accelerates development instead of blocking it.

Stay tuned for our next blog entry to learn another major reason why you should use a Git-based CMS!

Five Reasons Why You Should Use a Git-based CMS (Part 1 of 5)

Crafter CMS is a revolutionary open source digital experience platform based on Git. Crafter CMS solves problems from scalability and performance to ease of innovation that has existed in the CMS space for more than 20 years. What makes Crafter CMS so unique is its technical approach and underlying architecture. From its repository layer to its content delivery technology, Crafter CMS is designed to handle today’s most difficult content management challenges associated with creating and managing omnichannel digital experiences.

While there are many architectural advantages of Crafter CMS, in this series we will focus your attention on Crafter’s underlying repository technology: Git. Crafter CMS is the first and only enterprise-class CMS based on Git. We’ve based our CMS on Git for many reasons, and throughout this series, we’ll explore five of the most important.

Reason #1: Event-based multi-object versioning

Traditional CMS platforms like Drupal, WordPress, Adobe Experience Manager, Sitecore, and most others either have severely limited versioning or provide basic versioning capabilities that track single object graphs or maintain clunky data structures to track relationships.

Figure 1: Single file versioning model. Each object has its own version tree. How and whether relationships are tracked between objects differ from one system to the next.

Such simplistic approaches work for basic content management needs like blogs or boring websites but largely fall down in the face of managing today’s multi-object, multi-asset digital experiences. Today’s content models are component-based, and they have many relationships and dependencies. Further, there is often a relationship between the content and the code (CSS, Javascript, templates, etc.) that needs to be considered. Tracking the edits of any one specific object in isolation is simply not enough.

While simple object versioning does support basic editing, simple review, and basic reversion, these use cases are only the tip of the iceberg in a real-world environment. Scenarios like legal audits, company re-branding, and concurrent feature development drive the need for much more sophisticated CMS capabilities like a “time-machine” preview, multi-object reversion and content/codebase branching.

Instead of the single file versioning we see in the CMS space, what’s needed is a multi-object versioning approach like we see in the programming space. We require an approach that tracks “the entire state of the universe on each change.” With this level of version detail, a system can provide real previews at any point in time, make intelligent decisions about what must be reverted and support a host of branching and workflow needs.

Figure 2: Multi-object (“striped”) versioning model. Each event tracks the state of the entire repository at the time of the event.  

This type of solution already exists in the enterprise software development space. With software, one source file is often related to many others. Versions between objects matter. Modern Source Code Management (SCM) has evolved to support this need. Git is today’s most popular and widely used source code management system. It’s clear that the content and technical components of today’s digital experiences share many of the same needs that we see in the software development space. Rather than re-invent Git to achieve the same versioning capabilities in the context of content management, we’ve based Crafter CMS on Git’s versioning mechanics.

Because Crafter CMS is based on Git, every content change event is tracked with an event ID known as a “Commit ID.” Using this ID, it is possible to know the state of every content object in the system at the time of the event. For the sake of simplicity, we can say that we’ve created a version “stripe” across the entire repository at a given moment in time. The system does not make a copy of every object on every edit. That would be too slow and cost too much in terms of storage. Instead, this is done in a very efficient and effective manner by leveraging Git’s own proven versioning mechanics.

Moreover, because of the way Git stores and manages versions, traversals to any point in time are extremely fast. Performance is very important when it comes to the types of use cases we discussed earlier. Let’s take, for example, an auditing scenario: legal needs to see what the site looked like 46 days, 2 hours and 42 minutes ago. With most CMS platforms, this scenario is impossible to support. At best a systems group can attempt to restore a backup from that date and staff can be diverted to give the lawyers what they need. Even if your CMS claims to support this kind of review, the speed at which it can be provided is of key importance. If it’s too slow it won’t be practical. I’ve seen demos of CMS platforms that take minutes to render a previous version of a dynamic site. That’s too slow when you are doing a triage. It’s worse if you are traversing for editorial reasons. Crafter CMS simply doesn’t have this issue. Because of the way Git stores versions, traversal of versions n our Git-based CMS is extremely fast.

Finally, Crafter’s Git-based versioning approach itself hints at another important and related characteristic of Crafter CMS: content is managed in a document-oriented, file-based store. In short, content is stored as XML. Git is a file-based versioning system. Storing content as files are not only necessary, but the file-based approach has several major advantages. Because we’re dealing with files, content is easy to move among environments (Dev, QA, Prod, etc.) and migrate between systems. It’s much easier to integrate the content with other 3rd party systems, such as for language translation, e-commerce, and marketing automation. And because we store content in an XML format, it’s multi-byte character set friendly and totally extensible.

Conclusion

Most CMS’s lack the sophisticated versioning mechanics that are needed by today’s multi-disciplinary teams who are creating modern digital experiences. Today’s sophisticated digital experiences call for a much richer set of versioning mechanics similar to those we see in the software development space with Source Code Management Systems (SCMS.)  Git is today’s most powerful and popular SCMS. Because it’s based on Git, Crafter CMS is able to deliver the versioning needs for today’s most sophisticated needs and use cases.

Stay tuned for our next blog entry to learn more reasons why you should use a Git-based CMS!