Recently I’ve been piloting Xcode’s Continuous Integration server for a project at work. We’ve been using it over the last few months, and so far it’s been going well. I’ve had people both at work and outside of work ask me about it, so I thought I’d write up a review of my experiences so far.
Requirements/Installation
Xcode CI Server is a component of Maverick’s Server app, which is a $20 add on for Mavericks available from the app store (Edit: Guido Hendriks reminded me that it’s a free download for developers, available from the developer portal.) It requires an installation of Xcode (Xcode 5 or higher, as far as I know), along with access to your source repository. Like Xcode, it supports either Git or SVN.
Xcode CI is built around the concept of a “Bot.” Bots can execute the build, analyze, test, and archive phases of a scheme within your project. When you set up a bot, you can pick and choose if you want to execute the analyze, test or archive steps or your scheme. This provides some interesting use cases, as you could have a bot that is only used for archiving builds, making Xcode CI act only as a build server. The test phase of a project scheme is usually configured to run some, or all, of your unit tests that are part of your project. A single run of the phases of your bot is called an integration, and you can choose at what frequency you want to run an integration when you set up your bot.
The source control requirement caused a bit of an issue for us initially, as at work we’re a Perforce shop. We know that Perforce isn’t widely used in the iOS industry, but it’s the source control system the rest of the company and our build servers use, so until Xcode CI we had been content to continue using it. Fortunately we found a solution called Git Fusion that let’s us pull and push to Perforce via Git. I’ll cover strategies for using Git with Xcode CI later in this review.
iOS developers have had two different automated testing tools available to them: Unit Testing and UI Automation. Unit Tests are tests usually written in Obj-C and managed by Xcode, while UI Automation tests are written in JavaScript and run by Instruments. Unit Tests are handy for testing classes while UI Automation is intended for testing entire flows of an application and providing higher level UI validation. Unfortunately, only Unit Tests are currently compatible with Xcode CI. This is disappointing news for developers or QA engineers who’ve already invested time into UI Automation scripts. I’ve heard that the Xcode CI team is aware users would like to see support for UI Automation scripts added, which would be handy as Unit tests and UI Automation have two different use cases.(If you’re interested in UI Automation testing, Jonathan Penn has a book on the topic.)
I’ve found Xcode CI becomes much more effective with Test Driven Development practice. If you’re interested in learning more about Test Driven Development for iOS specifically, Graham Lee has an excellent book on the subject. At the very least, your projects are required to support unit tests to be tested with Xcode CI.
Setup – Xcode Install and Provisioning
Once Server.app is installed, the Xcode service can be found inside the application. When the service is enabled for the first time, it will need to be pointed to an Xcode installation, and to a developer account that has access to your or your team’s provisioning profiles. Xcode CI does support multiple teams working from one server.
There is an interesting catch with provisioning a CI Server. The CI Server will only download the automatically generated provisioning profiles from the provisioning portal. If you’re like us, and your project has been around for a very long time, you’re probably using provisioning profiles you manually created.
The Server application doesn’t offer an easy way to install provisioning profiles that weren’t automatically created by your developer account. But you can manually install provisioning profiles into the /Library/Server/Xcode/Data/ProvisioningProfiles directory, and they will be available for the Xcode CI Server at build time. Any related keys can be installed with Keychain Access into the System keychain.
Setup – Repositories
Xcode CI will also need to be pointed to the repositories that you’d like it to pull source from. You can tell Xcode CI to point either to an existing repository, or you can also have the Xcode CI host a repository for you. If you’re pointing to an existing repository via SSH, the Xcode service will maintain it’s own independent SSH key, which if you like you can replace with your own public/private key set.
A copy of Xcode that is configured to use an Xcode CI server can automatically add repositories that a newly created bot is dependent on, but I found this interaction awkward. When Xcode tries to automatically add access to a repository to the Xcode CI service, it doesn’t offer SSH key based authentication as an option, meaning the server administration has to log in to the Server application, and correct the authentication information for the repository. Apple has room for improvement here.
I spend most of my time with Xcode CI dealing with existing repositories, but in my limited time spent hosting repositories from Xcode CI, it seemed to act as a good, basic, no-frills Git host.
Setting up Xcode On Clients
Once the Xcode CI server is setup, you can link it to an installation of Xcode on a developer machine, or use many of the same features by pointing a web browser to the server. I’ll talk more about the web interface vs. the Xcode Interface, but to link Xcode to a remote Xcode CI Server, Go to the accounts tab in Xcode’s preferences, add a server account, and enter the address and a username and password for an account on the server. Our Xcode CI Server is on Active Directory, and the Xcode CI Service will let users log in with accounts through the Active Directory domain it is bound to, which makes our account management nice and easy. If your Xcode CI Server is bound to another service like Open Directory, that should work just as well.
Using Xcode CI
If your local Xcode installation has been bound to the Xcode CI Server, you can create a new bot from “gear” icon in the log navigation tab in Xcode. When you’re in Xcode, Xcode will list all the bots associated with your current project’s source control repository and branch in the log navigation tab, along with any results from testing. Also, any bot you create will automatically be associated with the current source control repository and branch.
Xcode will ask you several other questions when you create a bot. You can tell Xcode CI at what frequency you want to run a bot, including running a bot whenever there is a commit to the repository branch (Xcode will check the branch on a five minute interval for updates.) You can also tell Xcode CI what simulators and devices you want to run your tests on. Xcode CI will let you run tests on any device attached to the server, or any simulator available on the server.
Sadly, Xcode CI only currently will test on one device at a time. So keep in mind that every device you add will extend the length of your test. While your bot is testing, it will also block any other bot from executing (Xcode will queue up bot runs, so once the current bot is done, the next bot in line will start.) A great enhancement to Xcode CI would be testing on multiple devices in parallel in order to speed up testing time.
Xcode CI should support any device or OS version supported by the current version of Xcode. So if you want to test across multiple versions of iOS for each run of a bot, you totally can. If you support iOS 6 and 7, or you have bugs that may be dependent on minor revisions of iOS, you can test against these different OS versions all in the same test, and view the results side by side.
Bots will adopt all the settings specified in your schemes, including any configuration data or location data that you’ve set up for testing. If you haven’t used the “configuration” setting in schemes before, it’s worth checking out. You can download the on disk state of your iOS applications, and then reupload to an installation of your app when you’d like for testing. One use case is archiving states of previous versions of our applications so they can used for migration testing later. Archives could be paired with schemes for automatic testing of migrations from previous versions of an app.
Once a bot is set up, it will create and archive reports for each integration that you can view directly from Xcode. It will show each device it tested on, along with the results for each test.
A good branching seems to be key to an effective use of Xcode CI. Bots can be associated with different branches of your source repository, so if configured to, Xcode CI can provide continuous regression testing of multiple new or existing branches, which is something developers may not have time to do manually. If a bot if configured to archive builds, Xcode CI could also be used for installed IPAs from different branches, which is also very handy for testing different builds with new or experimental features at the same time.
Another use I’ve found for Xcode CI is testing for race conditions. If you have a test failure that only occurs 1/20 runs of a test, Xcode CI can provide continuous testing and build telemetry on the cause of an issue. If an issue only occurred 1/20 runs, and an Xcode CI bot was doing regression testing on an application every hour, it would only take approximately 16 hours for the issue to be found. If I feel suspicious about random failures in a component of the application, I’ll frequently create a bot to repeatedly test that component on a schedule, wait a day or two, and then review the testing history that has been built up to look for the frequency of the issue, or maybe find correlation to a specific set of variables that could be causing the issue.
Xcode CI can be obtuse at times. If your application crashes during testing, Xcode CI will report that “tests did not finish”. but it will only treat this as a warning, and there are no hints that your application crashed, such as a stack trace. You can track down the crash log by hand, but I’ve been unable to get these crash logs to symbolicate against the outputted xcarchive the CI Server can create.
Another issue is that Xcode CI can also suck time away from development and into system administration. Xcode CI is going to invoke the entire Xcode build chain every integration, so you’ll have to perform whatever setup you’d have to at a client machine. This includes any work for talking to source control, custom provisioning, and Xcode updates. For example, I don’t have CocoaPods installed on my OS X Server, which made many open source projects untestable unless I put in the extra setup work on the server. This could be a problem in a workgroup where not all developers have access to install tools on the server. And frustratingly, Xcode CI will give you precious little information when something goes wrong. While writing this review, I decided to get SparrowKit checked in to my Xcode CI Server, and create a bot to test it. The bot is executing tests, but each integration has a failed state due to a “internal-post-timeseries-error”, whatever that is. A Google search found no answers as to what this error means.
Xcode CI also exposes developers to the issues with OS X Server that have previously only been the concern of system administrators. I encountered several issues with account authentication include https certificate issues, and home directory-less users not being able to authenticate to repositories via SSH (accounts on OS X Server require a server side home folder to be created before SSH logins will work.) I was able to work through these issues because of previous experience debugging my OS X Server for other services, but to a novice system administrator these errors might be harder to track down.
The Web Front End
I should mention Xcode CI has a web front end. Just point any web browser to the Xcode CI Server’s address, and you’ll get a nice web page with most of the features of the Xcode Integration. This is useful for sharing data with others who may not have access to your project, may not have Xcode, or may not be on a Mac.
Another benefit is that the web front end will list all bots on the server, including all the bots for all the branches of a repository. This can be more useful than Xcode’s bot list, which will only display the bots for the current branch of the current project.
Adding a bot through the web front end is more frustrating that adding a bot through Xcode though. As part of the set up, you’ll need to supply the web front end with the path to your project from the root of your repository, along with the scheme name required to build/test the project.
The web front end does have one other handy feature that the Xcode client does not: the “big screen” feature. Big screen is a feature reminiscent of Panic’s Status Board, providing a heads up animated view of all the bots on the server. This is useful for team settings, and I’ve had this set up as a full screen web page on a spare monitor at my desk. It’s been a great conversation starter with other engineers or managers.
Conclusion
Overall, I’ve found Xcode CI to be a time saver and a very valuable tool, and I’d recommend it to developers considering deployment. But Xcode CI is still very rough around the edges. Missing features such as testing on devices in parallel, exception traces, no support for UI Automation, and clearer errors and easier setup could make the initial setup a frustrating experience. But my hope is that Apple continues development of this tool as it has the potential to be a critical cornerstone to iOS development practices. While it’s still a very 1.0 product, I’d recommend developers start working with it now and building a unit testing strategy, especially if a developer has not yet adopted unit testing or continuous integration.
Another valuable addition could be code review or push request tools other services like GitHub or Bitbucket have. It would certainly give me more of a reason to consider using Xcode CI as a host, and Apple’s user and Wiki tools that are already components of OS X Server seem like they could serve as a great starting point.
I’ve also been asked how I’d compare Xcode CI to Jenkins. I haven’t deployed Jenkins for testing on iOS, but our Android team has deployed it. I can’t make a technical comparison to Jenkins, but I can say we’ve enjoyed having Apple supported on device testing and Xcode integration. Jenkins does support UI Automation, and if Apple doesn’t support UIAutomation in the next major release I may have to consider deploying Jenkins for some projects where integrating UI Automation and our unit tests into one work flow is desirable, but I wouldn’t like sacrificing the other benefits Xcode CI brings.