As announced in the TeamForge Connector Server (TCS) forum, there will be a series of blog posts on how to use its REST API with different programming languages. In this post I want to start introducing the main TCS resources (aka CCF entities), the REST endpoints (ie how to navigate to those resources) and how to use them in Python. For this purpose, I created a small demo application that can be used to return all failed shipments (artifacts that could not be synched) which match a certain condition (e. g. belong to a certain tracker, project or error code). This tool comes quite handy for TCS administrators who want to monitor only certain projects for failed shipments and are only interested in errors that occurred after the last time, they called the script. The sample application has been inspired by a recent post to the CCF discussion forum. Before we explain the internals of the sample application, let’s back off one step and ask what else can be done with TCS REST API.
Why should I use this API?
With the REST API of TCS it is easy to automate workflows, to create mash-ups using the entities, or report things. As an example I’ll show you a tool that reports all new hospital entries for defined paths, by leading the user through a step for step guide. The code can be downloaded at the bottom of this post.
Request:
C:\scripts\ccf_notifier_cgui.py --url=http://url.to.my.ccf.master/CCFMaster/api/ --login=username:password --path=prpl1068.tracker1195.* --filter=gt(id,1585)Result:
-------\hospitalentries--------
1587 This hospital entry has been inserted by CCF Core.
---------------------------------------------------------------------------
reprocessed: false
sourceArtifactI: artf3208
errorCode: parentArtifactNotPresentError
causeExceptionC: No cause exception detected.
id: 1587
exceptionClassN: com.collabnet.ccf.core.CCFRuntimeException
targetLastModif: 1969-12-31T16:00:00-08:00
originatingComp: SWPEntityService
dataType: org.dom4j.tree.DefaultDocument
version: 0
sourceArtifactV: 101
causeExceptionM: No cause exception detected.
description: This hospital entry has been inserted by CCF Core.
genericArtifact:
exceptionMessag: Parent artifact artf3036 for artifact artf3208 is not yet created on the target system for combination artf3208-tracker1195-1-Bridge(-792309605
128688648)-Task-2. Since a project mapping exists for tracker1194 CCF bails out now.
timestamp: 2011-11-30 05:30:19 PST
sourceLastModif: 2011-11-25T02:53:28-08:00
data: org.dom4j.tree.DefaultDocument@184a6b9 [Document: name null]
stackTrace: com.collabnet.ccf.core.EntityService2.processXMLDocument(Entity
Service2.java:506)
com.collabnet.ccf.core.EntityService2.process(EntityService2.java:115)
org.openadaptor.core.node.Node.processSingleRecord(Node.java:151)
...
org.openadaptor.core.node.ReadNode.process(ReadNode.java:237)
org.openadaptor.core.node.ReadNode.run(ReadNode.java:193)
java.lang.Thread.run(Unknown Source)
targetArtifactI: unknown
targetArtifactV: unknown
adaptorName: TF2SWP
repositoryMappi: 37
artifactType: plainArtifact
fixed: false
---------------------------------------------------------------------------
1589 This hospital entry has been inserted by CCF Core.
---------------------------------------------------------------------------
...
-------\hospitalentries--------
(output truncated)
Before we jump to the internals of the program, first have a look what REST is all about. If you are already familiar with the REST principles, you can skip the next paragraph.
What is REST?
REST (=Representational state transfer) is a programming paradigm for web services. The main idea of REST is that one URL represents one result of a server-side action. This result is static, that means you’ll get the same resource, every time you request this url, assuming nothing on server-side has changed.
These are some main concepts which describe the architectural style of REST:
- adressability
- a REST conform server is reachable through a unique address
- various representations
- the service hands out the representation the client asks for, like html for a browser or XML and JSON for other rest clients
- stateless
- the communication between server and client is stateless, any data which you need for a request will be submitted
- operations
- there are different operations to access and modify resources – GET, POST, PUT and DELETE
In our case the API is not completely stateless. We use the session cookie to store the authentication to accelerate the request. But it is also possible to send the authentication header with every request as I’ll do in my examples.
Which TCS resources are exposed over the REST API?
As we learned in the previous section REST is resource based, so let’s have a look what entities the TCS server provides:
- Landscape
- A landscape describes the connection of a TeamForge instance with one other participant for synchronization. This participant can be Scrumworks Pro or HP ALM / Qualitiy Center. A landscape can be associated with several landscape config resources, which store additional information about the landscape like usernames and passwords.
- Participant
- A landscape consists of two participants. One participant is always TeamForge, the other one is HP ALM / Quality Center or Scrumworks Pro. Participant specific information is stored in participant config resources like the URL of the server where the participant is running.
- External app
- Instance of an integrated app (in this case TCS) in context of a TF project.
- Direction
- Direction of synchronization like TF -> SWP (FORWARD) and SWP -> TF (REVERSE). A direction belongs to a landscape and controls one CCF core.
- Repository mapping
- A repository mapping defines which TeamForge repository (tracker or planning folder) should be mapped to which partner repository (QC defect/requirement or SWP product/release/PBI/tasks). A repository mapping does not say anything about the direction, artifacts will be transported, this is where repository mapping directions come into play (children of repository mappings). Another child dependency exists to identity mappings since those are also independent of any synchronization direction.
- Hospital entry
- An artifact which couldn’t be synchronized without error. Hospital entries are stored in the hospital.
- Identity mapping
- An identity mapping defines which artifact of a TF repository is mapped to which artifact in the other participant’s repository. While a synchronization between repositories may be only unidirectional, an identity mapping is always bidirectional.
- Participant config, landscape config and direction config
- Those resources store additional information of participants, landscapes and directions in key/value format. A complete list of used key/value pairs can be found here.
The complete list of resources including their properties can be found here.
How can I access those resources?
TCS resources are exposed over several URL endpoints. A complete list of all available endpoints can be found here.
For this tutorial we will mostly be interested in getting access to the hospital entries. Hospital entries belong to repository mappings directions which in turn belong to repository mappings, which in turn belong to external apps which in turn belong to landscapes, so in order to navigate to hospital entries, we have to traverse quite deep into the TCS resource hierarchy, which is a good thing for a tutorial because it is a perfect example to show how the same resource can be identified/accessed using different paths/URL endpoints.
Before we can start our journey to a specific hospital entry, you need to setup proper permissions for the full path down there. In our case, we need to grant the user accessing the REST API permissions to access all hospital entries and repository mappings of a project you like to experiment with. You have to add a project specific role in TeamForge and add the TCS specific permissions for hospital entries and repository mappings and apply this role to the user that should access the API.
More information on access permissions needed for the REST API endpoints can be found here.
For the first contact with TCS REST API, you can use a REST-Client like “Advanced REST Client” for Google Chrome. All what you need is the URL and user/password for the TCS instance. For authorization you can use Http Basic Auth. If so, you have to encode “username:password” with base64 and add “Basic “ + base64 encoded username/password combination as “Authorization” header. For Testing you can use http://base64encode.org/ to encode this. To get the result as XML, you have to add the “Accept” header with “application/xml” as value.
baseurl: http://urltoccfserver/CCFMaster/api/
headers: Accept: application/xml
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
(“username:password”)
To get an overview of all possible endpoints in the API, you can take a look at the documentation for the REST endpoints. Some of those are limited to site admin access only like repositorymappingdirections/ or hospitalentrys/, but you can access most resources via the “linkid/{linkid}” part of the API. If you want to access hospital entries of a specific project in TeamForge, you can access those via linkid/{linkid}/hospitalentrys/.
To get the link id that corresponds to the TeamForge project, you have to access externalapps/. You’ll get a list of external apps. Each of these apps has a link id and a project path. Using the path you can find the link id related to the project you want to access.
Requesting externalapps/ will return you a list of all external apps, including the link ids as attribute:
<?xml version="1.0" encoding="UTF-8"?>
<externalAppList>
<externalApp>
<linkId>prpl1009</linkId>
<projectPath>projects.sample_product</projectPath>
<landscape>1</landscape>
<projectTitle>Sample Product</projectTitle>
<id>1</id>
<version>0</version>
</externalApp>
<externalApp>
<linkId>prpl1068</linkId>
<projectPath>projects.bridge</projectPath>
<landscape>1</landscape>
<projectTitle>Bridge</projectTitle>
<id>2</id>
<version>0</version>
</externalApp>
<externalApp>
<linkId>prpl1069</linkId>
<projectPath>projects.bridge2</projectPath>
<landscape>1</landscape>
<projectTitle>Bridge2</projectTitle>
<id>3</id>
<version>0</version>
</externalApp>
<externalApp>
<linkId>prpl1070</linkId>
<projectPath>projects.bridge3</projectPath>
<landscape>1</landscape>
<id>4</id>
<version>0</version>
</externalApp>
</externalAppList>
Now I search for the external app where the project path is the one I’m looking for. In this case it’s projects.bridge and the link id is prpl1068. With this information I can access linkid/prpl1068/hospitalentrys/.
When accessing a endpoint starting with “linkid”, it is essential that you add the link id to the username. The new auth header is composed of the base64 encoded string “username/prpl1068:password”.
Result will be a list of all hospital entries that belong to the bridge project.
It is also possible to get all hospital entries linked with a specific tracker. To get those you need the id of the tracker. You’ll get this from linkid/prpl1068/repositorymappings/. With this id you can request linkid/prpl1068/repositorymappings/{id}/hospitalentrys/.
<?xml version="1.0" encoding="UTF-8"?>
<repositoryMappingList>
<repositoryMapping>
<description>Task</description>
<externalApp>2</externalApp>
<teamForgeRepositoryId>tracker1195</teamForgeRepositoryId>
<participantRepositoryId>Bridge(-792309605128688648)-Task</participantRepositoryId>
<id><strong>22</strong></id>
<version>0</version>
</repositoryMapping>
<repositoryMapping>...</repositoryMapping>
</repositoryMappingList>
If you want to have all hospital entries corresponding to a task, you have to take 22, the id for tracker1195. To limit the result to one direction you can request just the hospital entries for one external app and tracker and direction. The endpoint for this is linkid/{linkid}/repositorymappingdirections/{directionid}/hospitalentrys/. To get the fitting direction id you need to request linkid/{linkid}/repositorymappingdirections/ and search for the right id. In this case it is 37 for tracker1195 and direction FORWARD.
<?xml version="1.0" encoding="UTF-8"?>
<repositoryMappingDirectionList>
<repositoryMappingDirection>
<direction>FORWARD</direction>
<repositoryMapping>22</repositoryMapping>
<status>PAUSED</status>
<lastSourceArtifactModificationDate>2011-11-25T02:49:36- 08:00</lastSourceArtifactModificationDate>
<lastSourceArtifactVersion>101</lastSourceArtifactVersion>
<lastSourceArtifactId>artf3103</lastSourceArtifactId>
<conflictResolutionPolicy>alwaysOverride</conflictResolutionPolicy>
<activeFieldMapping>37</activeFieldMapping>
<id>37</id>
<version>3</version>
</repositoryMappingDirection>
<repositoryMappingDirection>...</repositoryMappingDirection>
</repositoryMappingDirectionList>
There are many other endpoints that might be interesting, here is an overview of those I used in my sample application:
- hospitalentrys/
- all hospital entries (just for site admin)
- externalapps/
- List of all external apps = projects in TeamForge where our TCS integrated application has been enabled
- linkid/{linkid}/hospitalentrys/
- List of all hospital entries for the external app with the given link id
- inkid/{linkid}/hospitalentrys/count/
- Number of hospital entries for the external app with the given link id
- linkid/{linkid}/hospitalentrys/{id}
- The hospital entry with the given id
- linkid/{linkid}/repositorymappings/
- linkid/{linkid}/repositorymappings/hospitalentrys/
- linkid/{linkid}/repositorymappings/hospitalentrys/{id}
- linkid/{linkid}/repositorymappings/hospitalentrys/count/
- linkid/{linkid}/repositorymappingdirections/
- linkid/{linkid}/repositorymappingdirections/{directionid}/hospitalentrys/
- linkid/{linkid}/repositorymappingdirections/{directionid}/hospitalentrys/{id}
- linkid/{linkid}/repositorymappingdirections/{directionid}/hospitalentrys/count/
If you understood the semantics of the previous endpoints, those should be easy. If not, please leave a comment 
Please note that there is no trailing slash in the endpoints ending with {id}.
I would suggest that you try out some of those endpoints on your own TCS server to get a feeling when to use which endpoint. The trickiest part is probably to remember that you have to add the link id to the credentials you are using to authenticate (see example above). Once you understood how to navigate through the hierarchy, let’s move on with the promised sample application.
Access the TCS REST API via Python – The core part
Since you are very limited using the API with a simple REST client, it might be useful to write a script which accesses the API, gathers all information you need and filters them. I’ll explain the API in detail by using the example of an easy notifier, that checks for new hospital entries.
To make REST calls, I used urllib2 from the standard library. By extending the Request class from urllib2 it is possible to make requests using a specific method.
class RequestWithMethod(urllib2.Request):
def __init__(self, method, *args, **kwargs):
urllib2.Request.__init__(self, *args, **kwargs)
self._method = method
def get_method(self):
return self._method
In addition to this basic Request class, I wrote a RESTCall class, which automatically adds the right headers – the authorization and the accept header. This class has a request method where a RequestWithMethod is instantiated and it returns the result of urllib2.urlopen with this request object as argument.
As little helper, there is a CCFConfig class, which holds the current username/password combination and the base URL for the requests, as well as a class which represents a generic XML object (like a hospital entry).
The CCFNotifier class abstracts from the REST calls and provides an easy access to resources accessible through the REST API. The main method of the CCFNotifier class is getObjects(url):
def getObjects(self, url):
call = RESTCall(self.conf, url)
try:
response = call.request()
raw_xml = response.read()
xml = dom.parseString(raw_xml)
return xmlToObjects(xml)
except urllib2.HTTPError, e:
print e, call.url
return []
By using the xmlToObjects method, the getObjects method returns a list of GenericXMLObjects, based on the response of the RESTCall.
The other methods from CCFNotifer are based on this method and use the specific URL to access the desired resources. The two simplest methods are getProjects and getRepositoryMappings. Both just return the result of the getObjects method with a certain endpoint.
def getProjects(self):
return self.getObjects("externalapps/")
def getRepositoryMappings(self, linkid):
return self.getObjects("linkid/%s/repositorymappings/"%linkid)
As you can see there is an argument called linkid. It is necessary to add the link id to the URL, because every external app has its own repository mappings. As a non site admin user it is only possible to use this endpoint. As a site admin it is possible to use the repositorymappings/ endpoint to get all repository mappings.
To get an association between all trackers and the corresponding repository mapping id, I wrote the getTrackerNames method. In this method I take all repository mappings from getRepositoryMappings and return a dictionary with those ones where the teamForgeRepositoryId starts with “tracker” and the related id.
For every repository mapping, there are repository mapping directions. To get the id of this direction, I fetched all repository mapping directions belonging to the current repository mapping and external app and filtered for the direction string in the direction attribute.
There are four ways to get hospital entries, but only three are available for common users, because the hospitalentrys/ endpoint is only usable for site admins. The first endpoint only contains one variable part – the link id:
linkid/{linkid}/hospitalentrys/
Requesting this resource will return all hospital entries that belong to the specified external app.
The following query restricts the replied hospital entries to those belonging to a determined repository mapping:
linkid/{linkid}/repositorymappings/{repositorymapping}/hospitalentrys/
The last method makes use of the linkid and the repository mapping direction id:
linkid/{linkid}/repositorymappingdirections/{repositorymappingdirection}/hospitalentrys/
Two parts of the endpoints are common. All endpoints start with linkid/{linkid}/ and end with hospitalentrys/. Therefore its possible to put all requests in one method and handle all cases. If you want to have all hospital entries of the whole landscape, all the hospital entries for every external app you can access are collected.
def getHospitalEntries(self, linkid = None, tracker = None, direction = None):
url = ""
if linkid:
url += "linkid/%s/"%linkid
trackers = self.getTrackerNames(linkid)
if linkid and tracker and not direction:
url += "repositorymappings/%s/"%trackers[tracker]
elif linkid and tracker and direction:
url += "repositorymappingdirections/%s/"%self.getDirectionId(linkid, trackers[tracker], direction)
url += "hospitalentrys/"
return self.getObjects(url)
else:
hospitalentries = []
for project in self.getProjects():
self.conf.user = "%s/%s"%(self.conf.user.split("/")[0],project.linkId)
hospitalentries += self.getHospitalEntries(project.linkId)
return hospitalentries
The method for getting the hospital entry count is similar, except the endpoint is ending with hospitalentrys/count/.
So now that we have the basic infrastructure in place to get all the resources we need, we have to add some helper methods to expose them to our users in an interactive mode and via command line.
Access the TCS REST API via Python – The boiler plate
The interactive part of the script is the console “GUI”. There are two ways for getting the hospital entries you want. You can use arguments to specify username, password, base URL, paths and filters (command line mode), or use the step by step mode (interactive mode).
If there are no arguments, the script starts the step by step mode. In the first step, the script asks for the configuration, in fact the base URL to the TCS instance and the username/password combination of a TeamForge user. After specifying those details, you can choose between adding a path, adding a filter, changing the configuration, or show all hospital entries.
C:\scripts\ccf_notify_cgui.py
URL to CCF Master (including '/CCFMaster/api/': http://url.to.my.ccf.master/CCFMaster/api/
TF User: username
Password: password
Select Step:
p - add path
f - add filter
c - change config (username, password, url)
s - show hospitalentrys
next step: _
When choosing a path, the script requests all possible projects and displays them. After the user has chosen a project, all corresponding repository mappings are shown. As last step the user can choose the synch directions he is interested in. At every point the user can choose all of the displayed items. The resulting path looks like prpl1068.tracker1195.FORWARD, or with * as wildcard.
next step: p
(0) - Sample Product (prpl1009)
(1) - Bridge (prpl1068)
(2) - Bridge2 (prpl1069)
(3) - projectTitle (4) (prpl1070)
(-1) - all projects
select project: 1
(0) - Bridge(-792309605128688648)-Task (tracker1195)
(1) - Bridge(-792309605128688648)-PBI (tracker1194)
(2) - Bridge(-792309605128688648)-Product (proj1071-planningFolders)
(3) - Bridge(-792309605128688648)-Release (proj1071-planningFolders)
(4) - Bridge(-792309605128688648)-MetaData (tracker1194-MetaData)
(-1) - all tracker
select tracker: 0
(0) - FORWARD
(1) - REVERSE
(-1) all directions
select direction: -1
Select Step:
p - add path
f - add filter
c - change config (username, password, url)
s - show hospitalentrys
next step: _
In this case the path is:
prpl1068.tracker1195.*
I implemented three types of filters: equals, greater than and less than. These filters can applied to all attributes of objects in a list of objects. A hospital entry has those attributes:
reprocessed, sourceArtifactId, errorCode, causeExceptionClassName, id, exceptionClassName, targetLastModificationTime, originatingComponent, dataType, version, sourceArtifactVersion, causeExceptionMessage, description, genericArtifact, exceptionMessage, timestamp, sourceLastModificationTime, data, stackTrace, targetArtifactId, targetArtifactVersion, adaptorName, repositoryMappingDirection, artifactType, fixed
All filters look like a method call with the identifier for the filter (eq = equal, lt = less than, gt = greater than) and the attribute on what you want to filter and a value you want to filter with. For example “gt(id,1585)” applied on a list of hospital entries will only keep those hospital entries where the id is greater than “1585”.
next step: f
filter ('f(attribute, value)' f=eq(=), f=lt():gt(id,1585)
Select Step:
p - add path
f - add filter
c - change config (username, password, url)
s - show hospitalentrys
next step: f
filter ('f(attribute, value)' f=eq(=), f=lt():lt(id,1590)
Select Step:
p - add path
f - add filter
c - change config (username, password, url)
s - show hospitalentrys
next step: _
After setting all paths and filters you can request the corresponding hospital entries. In addition the equivalent command line to the user’s input will be generated and printed. Instead of using the wizard every time, the user can use this command line.
next step: s
To use the script with the same configuration again, you can use this command line:
C:\Users\Jan\Dropbox\Arbeit\collab.net\CCF_REST\ccfnotifier\ccf_notify_cgui.py --url=http://url.to.my.ccf.master/CCFMaster/api/ --login=username:password --path=prpl1068.tracker1195.* --filter=gt(id,1585) --filter=lt(id,1590)
--------hospitalentries--------
...
-------\hospitalentries--------(output truncated)
When processing the command line, the first step is to parse the arguments. Then all hospital entries are fetched – path for path. Because we just want to see the items newer than the last request, we filter those out, which are older than the newest one from the last request. For this purpose we save the path and the last id (if a hospital entry was fetched) and save them to a file.
After receiving the hospital entries the filters, defined in the command line arguments, are applied.
Now you know how to use the REST API to request resources. What I did not show is how to make other requests than GET. I also did not show how to store the authorization in a cookie, so if you are interested in this, please leave a comment. Also further ideas for extending the script or other examples are highly appreciated.
Links
- more about REST (Wikipedia)
- REST API documentation for TCS
- Advanced REST Client (for Chrome)
- online base64 encoder
- TCS project page
- source code for notifier
Problems?
- I get an HTTP 403 status code
-
- Check the spelling
- Check if you haven’t forgot the trailing slash
- Check the user’s permissions
- Check whether you added the linkid to the username when accessing a endpoint starting with “linkid”
- I can not start the script
-
- Python installed? (I used 2.7.2)
- unzipped both files?
